Author(s): Dr Swarneendu AI

Originally published on Towards AI.

The AI industry sells you tokens. It bills you for tokens. It throttles you by tokens. And almost nobody has explained what a token actually costs, mathematically, and why the cost is not what you think it is.

Open any LLM pricing page.

After the pricing page opener, the article argues that token prices—especially why output tokens cost more than input tokens—follow directly from the mathematics of autoregressive Transformers and the geometry of attention, not from arbitrary business choices. It explains the two phases of inference (prefill vs. decode), showing why prefill can be parallelized while decode is intrinsically sequential and memory-bandwidth-bound due to the need to repeatedly read key/value (KV) vectors from GPU memory. The author then connects attention’s softmax computation to the MacLaurin (Taylor) series and uses this to frame long-context costs as a “geometry tax” that scales quadratically with context length, largely driven by KV cache growth and memory/communication limits. The piece also highlights optimization mechanisms like KV caching and batching as methods of reusing or sharing computed geometry, and it extends the theme to inference-time strategy (beam search, sampling) as navigation on a curved statistical manifold rather than flat Euclidean space. Overall, the conclusion is that while hardware and systems improve throughput, the underlying geometric scaling remains, so the industry’s token economics are ultimately payments for mathematical and geometric constraints.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI