Anatomy of a failed AI project: an illustrative churn model story

Illustrative example, not a client. CustomerCo is a composite: an invented company built from failure patterns that are common in enterprise AI projects. It is not a DataTruth engagement, and every name, date and figure in it is made up. The dynamics are the point.

After eighteen months and a seven-figure budget, the customer churn prediction project at CustomerCo was being quietly discontinued. The internal announcement cited a strategic pivot and changing priorities. The real reason was simpler: the model did not work, and after months of struggling, leadership concluded there was no realistic path to making it work.

How had such a well-resourced project, with strong executive sponsorship and a clear business case, failed so completely? The answer reveals patterns that organisations contemplating AI initiatives need to understand.

The promise

CustomerCo was a mid-market B2B software company with a large base of enterprise customers. The company had grown rapidly through a combination of strong product-market fit and aggressive sales execution. Customer retention was good but not great.

The CEO had become convinced that AI could transform retention. The pitch from the AI vendor was compelling: by analysing customer behaviour patterns, the model would identify at-risk customers months before they churned, giving the customer success team time to intervene. Similar models had reportedly achieved remarkable results at other companies.

The maths looked straightforward. Every point of retention preserved meaningful recurring revenue, and a lift of a few points would repay the project several times over. On paper, it was a clear return.

The project kicked off with enthusiasm. A respected AI vendor was selected after a competitive evaluation. A cross-functional team was assembled, including customer success, sales, product, and IT stakeholders. Executive sponsors at the C-level committed to acting on model predictions and providing whatever resources were needed.

Early warning signs

The first hint of trouble came during the data discovery phase, about six weeks into the project. The vendor data scientists needed to understand what customer data was available for model training. What they found was concerning.

Customer data lived in multiple systems, each implemented at a different time by a different team:

Salesforce housed customer account information, sales history, and customer success activity logs. It had been implemented eight years earlier and modified extensively over time.

A homegrown product analytics system captured usage telemetry, tracking which features customers used and how frequently. This system was only four years old and had undergone two major schema changes.

Zendesk contained support ticket history, including issue categories, resolution times, and customer satisfaction scores. It had been implemented five years earlier.

NetSuite held financial data including billing history, payment patterns, and contract details. It was the oldest system, having been in place for over a decade.

HubSpot tracked marketing engagement including email opens, content downloads, and webinar attendance. It had been implemented three years earlier.

Each system used a different customer identifier. Salesforce used its internal Account ID. The product system used a separate Customer ID assigned at provisioning. Zendesk used email addresses as the primary identifier. NetSuite used its own customer numbers. HubSpot tracked contacts rather than accounts, creating many-to-one relationships.

Matching records across systems required complex logic that produced uncertain results. The data team estimated that a significant minority of customers could not be reliably matched across all five systems. For these customers, the model would have incomplete feature sets.

The second warning was the state of historical churn data. CustomerCo had been tracking churn for years, but the definition had evolved:

Pre-2022: Churn included both voluntary cancellations and non-renewals due to customer acquisition. If a customer was acquired by another company and that company did not continue the subscription, it counted as churn.

2022 to 2023: The definition was revised to exclude customers acquired by companies that remained CustomerCo customers under the acquiring company account.

2024: Customers who downgraded by more than half were reclassified as partial churn, a new category that was tracked separately.

These definitional changes meant that the historical churn labels were inconsistent. A customer with similar behaviour might be labelled as churned in one year and retained in another, depending on when the behaviour occurred.

Exit surveys, which would have provided insight into why customers left, had a low response rate. The survey questions had changed twice over the past four years, and the response categories were different in each version. Cancellation reason codes in Salesforce had been modified three times, with some categories split, others merged, and several added.

At this point, the project was six months in and had already consumed a substantial share of its budget in vendor fees, internal team time, and infrastructure setup. A rigorous assessment might have recommended pausing to address data issues. Instead, the team pushed forward, confident they could work around the problems.

The build

Model development took another eight months, far longer than the originally planned four months. The data scientists tried multiple approaches: logistic regression, random forests, gradient boosting, and eventually neural networks. Each approach required extensive feature engineering to handle the data quality issues.

The team developed elaborate workarounds. For customers who could not be matched across all systems, they imputed missing features using averages from similar customers. For inconsistent churn labels, they standardised to the most recent definition and relabelled historical records. For missing exit survey data, they inferred churn reasons from support ticket categories and usage patterns.

Performance on held-out test data was promising. The best model scored respectably, distinguishing between customers who churned and those who did not with reasonable accuracy. The precision and recall metrics were acceptable. The team was ready for production.

But troubling patterns had emerged during development that were not fully addressed. The model heavily weighted features that seemed like proxies rather than drivers: company size, industry vertical, and original contract value. These features did not change over time and therefore could not explain why a customer who was happy for three years suddenly became at risk.

Usage features, which the business believed should be highly predictive, showed surprisingly weak signal. Customers who churned did not have dramatically different usage patterns than customers who stayed. The data scientists hypothesised that usage data quality issues were masking true patterns, but by this point the project was over budget and behind schedule. There was pressure to move to production and show results.

The production disaster

The model went live in month fourteen. Customer success managers received weekly lists of at-risk customers with predicted churn probability scores. The team was optimistic.

Within six weeks, problems became undeniable.

The false positive rate was much higher than expected. Many customers flagged as high risk showed no signs of dissatisfaction when contacted. Customer success managers were wasting time on customers who had no intention of churning. Worse, the unnecessary check-in calls were annoying some customers. The sales team complained that CSM outreach was undermining relationships with healthy accounts.

The model was missing customers who actually churned. Post-mortem analysis of churned customers showed that many had been assigned low risk scores. These were often customers with data quality issues: incomplete matching across systems, missing usage data, or atypical characteristics that the model had not learned to recognise.

The predictions were not actionable. Knowing a customer was at risk did not tell the customer success team why or what to do about it. Was the customer unhappy with the product? Having implementation problems? Facing budget pressure? Under new leadership? The model provided a score but no insight into root cause. The promised integration with intervention playbooks had never materialised.

Trust eroded rapidly. After a few weeks of acting on predictions that proved wrong, customer success managers stopped trusting the model. They reverted to their own judgment and relationships, using the AI predictions only when convenient. The expensive AI system became shelfware.

The painful reckoning

By month sixteen, the executive team could no longer ignore the failure. Customer retention had not improved. In fact, it had slipped slightly, possibly because the misguided outreach had damaged some customer relationships. The investment had produced nothing of value.

A post-mortem analysis identified the root causes:

Data foundation was inadequate. The inability to reliably match customers across systems meant the model was trained on incomplete feature sets. The usage data that should have been most predictive was too noisy and inconsistent to provide signal.

Label quality undermined training. The changing definitions of churn meant the model was learning from inconsistent targets. It was trying to find patterns in what was partly noise.

The use case was poorly specified. The goal was to predict churn, but churn is an outcome, not a behaviour. The model could identify correlates of churn but could not identify the interventions that would prevent it. Prediction without actionability is worthless.

Warnings were ignored. The data quality issues were known early in the project but were treated as obstacles to work around rather than fundamental problems to solve. The pressure to show progress overrode the judgment to pause and fix the foundation.

What should have been different

In retrospect, several decision points could have changed the outcome:

Data assessment before project initiation. A rigorous assessment of data quality, completeness, and integration feasibility would have revealed the foundational issues before significant investment was made. The project could have been rescoped to a more achievable goal, or data remediation could have preceded model development.

Data remediation before model building. The identity resolution problem and the inconsistent churn definitions should have been addressed before any modelling began. Building sophisticated AI on a shaky data foundation guaranteed a shaky result.

Stage gates with clear criteria. The project proceeded past multiple warning signs because there was no formal checkpoint requiring data quality thresholds before advancing. Each phase flowed into the next on schedule rather than on readiness.

Use-case scoping aligned with data reality. Given the data limitations, a more modest use case might have succeeded. Perhaps identifying customers with support escalations or significant usage declines, rather than predicting churn broadly. A narrower use case with reliable data would have been more valuable than an ambitious use case with unreliable data.

Emphasis on actionability from the start. The project focused on prediction accuracy without equal attention to what would happen after predictions were made. A prediction system without an action system is incomplete.

The aftermath

CustomerCo eventually did improve retention, but through traditional methods: better onboarding processes, more proactive business reviews, improved support responsiveness, and closer alignment between product development and customer needs. These improvements were enabled by better use of existing data and closer customer relationships, not by AI.

The AI detour cost them the full project budget in direct expense, eighteen months of distraction, and significant organisational credibility damage. The executives who championed the project moved on. The data team was demoralised. The organisation became sceptical of AI, making future legitimate AI initiatives harder to fund.

The vendor moved on to the next customer, armed with lessons learned that would hopefully prevent similar failures. But the pattern continues across the industry: ambitious AI projects launched without adequate data assessment, proceeding past warning signs under schedule pressure, and ultimately failing when the data foundation proves inadequate.

The lesson is not that AI cannot work. The lesson is that AI cannot work without adequate data, and that organisations must honestly assess their data before committing to AI initiatives. The assessment might reveal that AI is possible with current data, that AI requires data remediation first, or that AI is not feasible for the intended use case. All of these are better outcomes than spending the budget to learn the hard way.