Every enterprise ML leader eventually faces the same question: should we build our own ML pipelines or buy a platform that handles predictions for us? The standard answer is "it depends." The honest answer is that most companies dramatically undercount what "build" actually costs.
Gartner estimates that 85% of ML projects fail to reach production. That number has not changed meaningfully since 2019. The failure is rarely technical. The models work in notebooks. The failure is economic: the total cost of getting a model from prototype to production, and keeping it there, exceeds the value it delivers.
This article walks through the real numbers. Not vendor marketing math. The actual line items that show up when you build a custom ML pipeline from raw data to production predictions.
The anatomy of a custom ML pipeline
A production ML system is not a model. It is a pipeline with at least seven distinct components, each requiring different skills and infrastructure.
Data ingestion and transformation. Connecting to source systems (data warehouses, transactional databases, streaming sources), handling schema changes, ensuring data freshness. This requires data engineers and typically 4-8 weeks of work.
Feature engineering. Transforming raw relational data into flat feature tables. The Stanford RelBench study measured this at 12.3 hours and 878 lines of code per prediction task for experienced data scientists. For a production system, this phase takes 6-12 weeks and produces 100-500 features.
Feature storage and serving. Feature stores like Tecton or Feast cost $100K-300K/year in licensing and infrastructure. You need them for consistency between training and serving, and for low-latency feature retrieval in production.
Model training and validation. The part most people think of as "ML." Experiment tracking, hyperparameter tuning, cross-validation, fairness auditing. This is typically 2-4 weeks and requires senior data scientists.
Model serving infrastructure. Deploying models behind APIs, managing compute scaling, ensuring sub-100ms latency for real-time use cases. Requires ML engineers and DevOps, plus GPU or high-CPU infrastructure.
Monitoring and observability. Tracking prediction quality, detecting data drift, alerting on feature pipeline failures. Tools like Evidently, Arize, or WhyLabs cost $50K-150K/year.
Retraining and maintenance. Models degrade. Features break when upstream schemas change. New data sources need integration. This is the cost that never stops.
custom_ml_pipeline — TCO breakdown per use case
| Cost Category | Year 1 | Year 2 | Year 3 | 3-Year Total |
|---|---|---|---|---|
| Data Science Team (2 DS + 1 MLE + 1 DE) | $300K-600K | $150K-300K | $150K-300K | $600K-1.2M |
| Cloud Compute (training + serving) | $20K-100K | $20K-100K | $20K-100K | $60K-300K |
| Feature Store (Tecton/Feast) | $100K-300K | $100K-300K | $100K-300K | $300K-900K |
| Monitoring & MLOps Tools | $50K-150K | $50K-150K | $50K-150K | $150K-450K |
| Ongoing Maintenance (30-50%) | — | $250K-600K | $250K-600K | $500K-1.2M |
| Total per Use Case | $500K-2M | $570K-1.45M | $570K-1.45M | $1.6M-4.9M |
Most enterprises underestimate by 3-5x because they only count the modeling phase. Maintenance alone exceeds the initial build cost by year 3.
scaling_costs — total cost by number of use cases
| Use Cases | Custom Build (3-Year) | Foundation Model (3-Year) | Savings |
|---|---|---|---|
| 1 | $1.6M-4.9M | $200K-500K | 69-90% |
| 5 | $8M-24.5M | $400K-1M | 88-96% |
| 10 | $16M-49M | $600K-1.5M | 94-97% |
| 20 | $32M-98M | $800K-2M | 97-98% |
Custom pipelines scale linearly. Foundation models scale sub-linearly because the same model serves all use cases.
The real numbers
Here is what a single production ML use case actually costs at a mid-to-large enterprise, broken down by category.
People costs
A typical ML team for one use case includes 2 data scientists ($180K each fully loaded), 1-2 ML engineers ($200K each), 1 data engineer ($170K), and partial allocation of a project manager and ML platform engineer. Total team cost for a 6-month build: $300K-600K.
That assumes the team already exists and is not being recruited. If you are hiring, add 3-6 months of recruiting time and $30K-50K per hire in recruiting costs. Senior ML engineers in the Bay Area command $350K-500K in total compensation.
Infrastructure and tooling
Cloud compute for training: $20K-100K per use case depending on data volume and model complexity. Feature store: $100K-300K/year. Monitoring tools: $50K-150K/year. Experiment tracking and MLOps platform: $50K-200K/year. Total infrastructure: $220K-750K/year.
Maintenance
Industry benchmarks consistently show that maintenance costs 30-50% of the initial build cost annually. For a $1M pipeline, that is $300K-500K per year in perpetuity. This covers model retraining, feature pipeline updates, infrastructure upgrades, and on-call engineering time.
Where the money actually goes
When you break down the time allocation, the distribution is striking.
time_allocation — typical 6-month ML project
| Activity | Weeks | % of Total | Typical Output |
|---|---|---|---|
| Data extraction & cleaning | 4-6 | 15-20% | SQL pipelines, schema mapping |
| Feature engineering | 8-14 | 40-55% | 100-500 aggregate features |
| Model training & tuning | 2-3 | 8-12% | Trained XGBoost/LightGBM model |
| Validation & testing | 1-2 | 5-8% | Fairness audit, holdout evaluation |
| Deployment & serving | 2-3 | 8-12% | Docker container, API endpoint |
| Documentation & handoff | 1-2 | 5-8% | Model cards, runbooks |
Feature engineering consumes 40-55% of project time. Combined with data extraction, the data plumbing stages take 60-75% of the total project.
This means that the "ML" part of an ML project is roughly 15% of the work. The rest is data plumbing. You are paying $400K/year ML engineers to write SQL joins and debug Airflow DAGs.
Here is what a typical feature engineering session produces for a lead scoring model. The raw data lives across three CRM tables:
leads
| lead_id | source | company_size | industry | created_date |
|---|---|---|---|---|
| L-201 | Webinar | 500-1K | SaaS | 2025-01-15 |
| L-202 | Organic | 50-100 | Retail | 2025-02-03 |
| L-203 | Paid Ad | 1K-5K | Finance | 2025-02-10 |
activities
| activity_id | lead_id | type | timestamp | duration_sec |
|---|---|---|---|---|
| A-301 | L-201 | Page view (pricing) | 2025-01-16 09:12 | 180 |
| A-302 | L-201 | Demo request | 2025-01-16 09:15 | — |
| A-303 | L-202 | Page view (blog) | 2025-02-04 14:30 | 22 |
| A-304 | L-203 | Page view (pricing) | 2025-02-11 10:00 | 45 |
| A-305 | L-203 | Page view (pricing) | 2025-02-12 10:05 | 120 |
| A-306 | L-203 | Page view (case study) | 2025-02-12 10:20 | 210 |
L-201 went from pricing page to demo request in 3 minutes (high intent). L-203 returned to pricing twice, then read a case study (researching). L-202 bounced from a blog post in 22 seconds.
flat_feature_table (after 12+ hours of engineering)
| lead_id | activity_count_7d | source | company_size | converted |
|---|---|---|---|---|
| L-201 | 2 | Webinar | 500-1K | ? |
| L-202 | 1 | Organic | 50-100 | ? |
| L-203 | 3 | Paid Ad | 1K-5K | ? |
The flat table compresses all activity into a count. L-201's pricing-to-demo sequence (3 minutes, extremely high intent) is indistinguishable from L-203's research-phase browsing. The behavioral sequence that best predicts conversion is gone.
It also means that the single highest-leverage improvement you can make is not a better model architecture or a faster GPU. It is eliminating the feature engineering and pipeline orchestration that dominate the cost structure.
The hidden multiplier: use case count
One use case is expensive. But the economics get worse as you scale. Traditional ML pipelines are bespoke. Each use case requires its own feature engineering, its own model, its own serving infrastructure, and its own maintenance budget.
A churn prediction model and a fraud detection model on the same database share almost no code. Different features, different targets, different time windows, different serving requirements. You are building from scratch each time.
At 10 use cases, you are looking at $5M-20M in year-one costs and $3M-10M in annual maintenance. At 20 use cases, the numbers double. Most enterprises cap out at 3-5 production models not because they lack ideas, but because they cannot afford to build more.
Build: custom ML pipeline
- $500K-2M per use case (year one)
- 6-18 months to first production prediction
- 4-8 person team per use case
- 30-50% annual maintenance overhead
- Each new use case starts from zero
Buy: foundation model approach
- Single platform cost covers all use cases
- Minutes to first prediction, not months
- No dedicated ML team required per use case
- Maintenance handled by the platform
- Same model serves any prediction task
PQL Query
-- Same model, five use cases, five queries PREDICT churn_90d FOR EACH customers.customer_id PREDICT fraud_probability FOR EACH transactions.txn_id PREDICT ltv_next_12m FOR EACH customers.customer_id PREDICT conversion FOR EACH leads.lead_id PREDICT demand_next_7d FOR EACH products.product_id
A foundation model serves all five prediction tasks from one platform. Each query is a new use case that would cost $500K-2M to build custom.
Output
| Use Case | Time to Deploy (Custom) | Time to Deploy (KumoRFM) | AUROC |
|---|---|---|---|
| Churn | 6-12 months | < 1 day | 81.1 |
| Fraud | 8-14 months | < 1 day | 79.3 |
| LTV | 4-8 months | < 1 day | 77.8 |
| Lead Scoring | 3-6 months | < 1 day | 82.4 |
| Demand Forecast | 6-10 months | < 1 day | 78.6 |
What foundation models change
The economics shift fundamentally when a single model can serve multiple prediction tasks on the same data. A relational foundation model like KumoRFM is pre-trained on relational patterns across thousands of databases. It connects directly to your data warehouse, understands the table structure through foreign keys, and generates predictions without feature engineering, model training, or pipeline orchestration.
The same model that predicts churn also predicts fraud, forecasts demand, scores leads, and estimates lifetime value. You are not building 10 separate pipelines. You are querying one model 10 different ways.
On the RelBench benchmark (7 databases, 30 tasks, 103 million rows), KumoRFM zero-shot outperforms custom pipelines built by Stanford data scientists on 11 of 12 classification tasks. The accuracy is 76.71 AUROC vs 62.44 for LightGBM with manually engineered features. With fine-tuning, it reaches 81.14 AUROC.
DoorDash deployed this approach and saw a 1.8% engagement lift across 30 million users. That single lift, across that user base, represents millions in annual revenue from a system that took days, not months, to deploy.
When build still wins
Foundation models are not the right answer for every ML problem. Build still makes sense in specific scenarios.
Unusual data modalities. If your core data is satellite imagery, genomic sequences, or proprietary sensor streams, relational foundation models will not help. These require domain-specific architectures trained on domain-specific data.
Custom optimization objectives. If your business requires a custom loss function that encodes specific operational constraints (multi-objective optimization for logistics routing, for example), you need a custom training setup.
Regulatory requirements for full model transparency. Some regulated industries require line-by-line model explainability. Foundation models offer feature attribution but may not satisfy the most stringent regulatory audits.
For the estimated 80% of enterprise ML that involves predicting outcomes from structured relational data (customer behavior, financial risk, operational forecasting, product recommendations), the TCO math favors foundation models by a wide margin.
The decision framework
When evaluating build vs buy, ask three questions:
1. How many use cases do you need? If you need one highly specialized model and have the team, build may be viable. If you need 5-20 prediction tasks across your relational data, the per-use-case cost of building makes the math prohibitive.
2. What is your time-to-value requirement? If you can wait 12 months for a first prediction, build gives you maximum control. If the business needs answers in weeks, you cannot afford a custom pipeline.
3. Do you have the team? A production ML pipeline requires data engineers, data scientists, ML engineers, and MLOps specialists. If that team exists and has capacity, build is feasible. If you are hiring, add 6-12 months and $200K-500K in recruiting costs before the first line of code.
For most enterprises, the honest answer is: build for your 1-2 most differentiated ML capabilities, and use a foundation model for everything else. The 80/20 rule applies. Let your expensive ML talent work on problems that genuinely require custom solutions, and let the platform handle the rest.