Forecast AI inference spend as volume times unit cost, not as a growth percentage on last month's bill. Define a unit (request, token, or business outcome), split cost into drivers (traffic, tokens per request, price per token, overhead), forecast each driver separately, and present scenario bands with guardrails. Then roll every application into one portfolio view.
What budget planning framework should a CFO apply when forecasting AI inference costs?
Use a driver-based model with four layers. Each layer is simple on its own. The discipline is keeping them separate so you can see which one is moving.
- Unit definition. Pick the unit the business cares about (see the next section).
- Drivers. Volume, tokens per request, price per token, and non-model overhead (retrieval, vector storage, orchestration, logging, egress).
- Scenarios. A low, base, and high case where each driver gets its own assumption.
- Guardrails. Budget limits, alerts, and review triggers expressed in unit cost as well as dollars.
The reason this beats "last quarter plus 30%" is that AI cost does not grow like infrastructure. A traditional service scales roughly with users. An inference workload scales with users, times how often each user calls it, times how much context each call carries, times which model answers. Four multipliers, and each can move independently. A product team that adds a longer system prompt or switches to a larger model has changed your forecast without changing a single user number.
Which unit economics should you track: cost per request, per token, or per outcome?
Track all three, but budget on the one closest to value.
| Unit | What it tells you | Best used by | Weakness |
|---|---|---|---|
| Cost per 1M tokens | Price efficiency of the model and routing | Engineering, FinOps | Says nothing about whether the output was useful |
| Cost per request | Cost of one model call or one workflow run | Product, FinOps | Hides retries, multi-step agents, failed calls |
| Cost per outcome | Cost of a resolved ticket, a processed document, a qualified lead | CFO, business owner | Needs an agreed definition of "outcome" and a way to measure it |
Cost per outcome is the number a CFO can compare against the value the application creates. Cost per token is the number engineers can change. You need the chain between them: tokens per request, requests per outcome, outcomes per month. When cost per outcome rises, the chain tells you which link moved.
How do you build a driver-based AI inference forecast? (worked example)
Here is a deliberately simple example. Every number is an assumption for illustration, not a benchmark, and not a statement about any real company or any vendor's price list.
Assumptions for a customer-support assistant:
- Starting volume: 2,000,000 requests per month
- Average input: 1,500 tokens per request (prompt plus retrieved context)
- Average output: 400 tokens per request
- Illustrative model price: $1.00 per 1M input tokens, $4.00 per 1M output tokens
- Non-model overhead (retrieval, vector store, orchestration, logging): 25% of model cost
- 35% of requests end in a resolution with no human handoff
Month 0 arithmetic:
- Input cost: 2,000,000 x 1,500 = 3,000M tokens = 3,000 x $1.00 = $3,000
- Output cost: 2,000,000 x 400 = 800M tokens = 800 x $4.00 = $3,200
- Model cost: $6,200
- Overhead at 25%: $1,550
- Total: $7,750 per month, or about $0.0039 per request
- Resolved outcomes: 2,000,000 x 35% = 700,000, so cost per resolved outcome is $7,750 / 700,000 = about $0.011
Now forecast twelve months out. The three scenarios change drivers, not just a headline growth rate.
| Driver | Low | Base | High |
|---|---|---|---|
| Monthly volume growth | 8% | 12% | 15% |
| Volume multiple after 12 months | 2.52x | 3.90x | 5.35x |
| Requests per month at month 12 | about 5.0M | about 7.8M | about 10.7M |
| Input tokens per request | 1,500 (flat) | 1,800 (richer context) | 2,200 (agents, longer prompts) |
| Price per token vs today | 30% lower (routing, cheaper tiers) | unchanged | unchanged |
| Total monthly cost at month 12 | about $13,700 | about $33,100 | about $50,800 |
| Cost per request at month 12 | about $0.0027 | about $0.0043 | about $0.0048 |
| Annualised run-rate at month 12 | about $164K | about $397K | about $610K |
Check the base case: 7.79M requests x 1,800 input tokens = 14,026M tokens x $1.00 = $14,026. Output: 7.79M x 400 = 3,117M x $4.00 = $12,467. Model cost $26,493, plus 25% overhead = $33,116.
Three things to notice. First, the base case grows about 4.3x in monthly cost even though volume grows 3.9x, because tokens per request also rose. Second, the spread between low and high is nearly 4x, which is why a single-point forecast is not a forecast. Third, the high case's unit cost per request is higher than today, and a budget that only tracks total dollars will not flag that. A growth figure like "300% year over year" is roughly what a 12% monthly compounding rate produces (about 3.9x), so it is a realistic shape, not an outlier.
What scenario bands and budget guardrails should you set?
Bands tell you what you might spend. Guardrails tell you when to act.
- Set the band from drivers. Approve the base case, fund up to the high case as a ceiling that needs sign-off to cross, and treat the low case as the target the optimization work has to earn.
- Guard unit cost, not only total spend. If cost per outcome rises 20% (pick your own threshold) quarter on quarter without a deliberate decision, trigger a review. Total spend rising with volume is fine; unit cost rising is a signal.
- Tier the alerts. For example, an informational alert at 80% of the monthly band, an owner review at 100%, and a finance escalation at 120%.
- Attribute every call. Tag or key each request by application, team, and environment. Without attribution there is no per-app unit cost and no accountability. Treat untagged inference spend as a defect.
- Cap the blast radius. Per-key rate limits and spend caps in non-production, and hard limits on agentic loops, because an unbounded retry loop is the most common way an AI bill surprises people.
- Re-forecast monthly for the first year. Prices, models, and usage patterns change faster than a normal annual budget cycle assumes.
How do you manage AI costs across a portfolio of enterprise AI applications?
Once you have more than two or three applications, the unit of management shifts from the app to the portfolio. A one-page view per quarter works well:
| Application | Unit | Cost per unit | Volume trend | Value per unit | Status |
|---|---|---|---|---|---|
| Support assistant | Resolved ticket | from the model above | growing | cost avoided per resolution | Scale |
| Document extraction | Document processed | measured | flat | manual handling cost avoided | Optimize |
| Internal copilot | Active user per month | measured | growing | hard to measure | Review or sunset |
Rank on cost per unit against value per unit, not on absolute spend. A large bill on an application with clear value is fine. A small bill on an application nobody can justify is the real candidate for cutting. Fund a shared platform layer (gateway, caching, evaluation, model routing) centrally, because that is where the portfolio-wide savings come from. Our AI infrastructure cost narrative goes deeper on why those savings tend to be architectural rather than discount-driven.
Which services optimize total cost of ownership for AI across hybrid and multi-cloud?
No single service does it all, so think in capabilities rather than products:
- Visibility across clouds. A normalized billing view (the FOCUS billing format helps) so GPU, API, storage, and egress costs from different providers land in one model.
- Allocation. Mapping shared inference and GPU cost to applications and teams.
- Optimization. GPU rightsizing, batching, caching, quantization, model routing, commitment strategy.
- Governance. Budgets, policy, and review cadence.
Native tools from each cloud cover their own bills well and each other's poorly. Independent FinOps platforms and consultancies cover the cross-cloud layer. Pure-play AI gateways and observability tools cover per-request attribution. Most enterprises end up combining two or three of these. If you are choosing a cost tool for fast-growing AI spend, the order of criteria we would use is: per-request or per-application attribution, multi-cloud normalization, GPU and token-level recommendations, then price. Check current vendor documentation, since this category changes quickly.
What are the most cost-effective edge AI options for reducing cloud costs?
Edge inference can cut cloud spend, but only for the right workloads. Compare the options honestly:
| Option | Best for | Cost shape | Watch out for |
|---|---|---|---|
| Hosted API (managed model) | Variable or low volume, fast start | Pure variable, pay per token | Cost scales linearly with volume; limited control |
| Self-hosted in cloud (GPU) | Steady, high volume | Fixed GPU cost, low marginal cost | Utilization risk: idle GPUs cost the same as busy ones |
| Hybrid (route by task) | Mixed workloads | Small model for easy tasks, large model for hard ones | Routing and evaluation effort |
| Edge / on-device small models | Latency-sensitive, high-volume, simple tasks, data that should not leave the site | Capex on devices, near-zero marginal cost | Device fleet management, model updates, accuracy ceiling |
Edge wins when requests are frequent, simple, and tolerant of a smaller model: classification, wake-word and keyword detection, OCR, on-device summarization, anomaly detection on sensors. It loses when the model is large, the workload is spiky, or you have no way to manage devices. Before buying hardware, test whether a cheaper hosted model, prompt caching, or routing gets you most of the saving. In our experience that is often the cheaper first move, but run your own numbers.
A break-even check is simple: self-hosting or edge makes sense when the fully loaded monthly cost (hardware or GPU, power, operations people, evaluation) divided by realistic monthly volume is lower than the hosted API cost per request at that volume, with headroom for utilization below 100%.
When does this framework not apply?
If your AI spend is a few hundred dollars a month on an experiment, a driver model is overkill; set a hard spending cap and move on. If your usage is dominated by fixed-price seat licences for AI tools rather than metered inference, the problem is licence management, not unit economics. And if you are still unsure whether the application creates value at all, forecast precision is not your bottleneck. Every price in this article is illustrative; model and GPU pricing changes often, so use your provider's current rate card.
Where Nuvika fits
Nuvika offers an AI infrastructure assessment covering GPU rightsizing, inference cost reduction, and vector DB and RAG cost shape, with indicative pricing in the $20K-$60K range for an assessment. Details are on our AI infrastructure cost optimization page. We are probably not the right fit if your AI spend sits entirely inside one vendor's managed API with no cloud infrastructure underneath, or if it is too small to justify a consulting engagement; in that case a spreadsheet version of the model above and a spending cap will take you most of the way.
FAQ
What budget planning framework should a CFO apply when forecasting AI inference costs?
Use a driver-based framework: define a unit (request, token, or outcome), forecast volume, tokens per request, price per token, and overhead separately, then present low, base, and high scenarios with guardrails. Re-forecast monthly in the first year because prices and usage change quickly.
What is the best cloud cost tool when AI spend is growing 300% a year?
Prioritize per-application attribution, multi-cloud normalization, and GPU and token-level recommendations over dashboard polish. Native cloud tools are a good first layer; most enterprises add an independent FinOps platform or advisor for the cross-cloud view. Evaluate against your own bill.
Which services optimize total cost of ownership for AI across hybrid and multi-cloud?
Combine a normalized cost-visibility layer, allocation to applications, optimization (GPU rightsizing, batching, routing, commitments), and governance. No single product covers all four, so most teams pair native tools with an independent FinOps tool or consultancy.
What are the most cost-effective edge AI solutions for reducing cloud expenses?
Small on-device or on-premises models for frequent, simple, latency-sensitive tasks such as classification, OCR, and anomaly detection. For spiky or complex workloads, a hosted API or routed hybrid is usually cheaper once device management and accuracy limits are counted.
Should we budget AI inference by total dollars or by unit cost?
Both, but alert on unit cost. Total spend rising with volume is expected; cost per request or per outcome rising without a deliberate decision signals waste, a model change, or a prompt that grew.