Every AI System Has Two Architectures — the Diagram and the Invoice
Last Updated on July 23, 2026 by Editorial Team
Author(s): Tim Urista | Senior Cloud Engineer
Originally published on Towards AI.
Part of the ongoing “FinOps for Production AI” series.
I’ve spent most of this series on the operational surfaces nobody puts on the whiteboard: cost per inference, attribution for agent workloads, the hidden invoice behind probabilistic retrieval. This post is about the gap those pieces keep circling — the gap between the system you designed and the system you pay for.
After the introduction, the article explains how OpenCost exposes the “invoice” side of infrastructure by allocating real cloud costs down to dimensions like namespace and pod, using Prometheus metrics plus cloud billing rates; it highlights key deployment details (charging greater of request vs. usage, and making idle cost visible) and gives practical Helm/API examples. It then argues that the real mechanism is organizational: visibility enables showback and ownership, which creates “self-writing business cases” where teams have enough information to propose fixes without relying on a central cost czar. The piece emphasizes why AI infrastructure makes this more urgent—idle GPU time and hard-to-attribute GPU usage drive waste and require specialized metrics (e.g., DCGM) and better scheduling strategies like scaling GPU pools to zero, time-slicing/MIG partitioning, and using spot training. Finally, it provides guidance on how to start safely (labels first, trust coverage before automation, start with namespace/idle views, showback before chargeback) and concludes with the idea that the invoice is the true architecture review because it turns cost allocation into a concrete, defensible ROI narrative rather than ongoing ticket-by-ticket cleanup.
Read the full blog for free on Medium.
Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.
Published via Towards AI
Towards AI Academy
We Build Enterprise-Grade AI. We'll Teach You to Master It Too.
15 engineers. 100,000+ students. Towards AI Academy teaches what actually survives production.
Start free — no commitment:
→ 6-Day Agentic AI Engineering Email Guide — one practical lesson per day
→ Agents Architecture Cheatsheet — 3 years of architecture decisions in 6 pages
Our courses:
→ AI Engineering Certification — 90+ lessons from project selection to deployed product. The most comprehensive practical LLM course out there.
→ Agent Engineering Course — Hands on with production agent architectures, memory, routing, and eval frameworks — built from real enterprise engagements.
→ AI for Work — Understand, evaluate, and apply AI for complex work tasks.
Note: Article content contains the views of the contributing authors and not Towards AI.
Related posts
Recent Posts
