Author(s): Anubhav

Originally published on Towards AI.

2.8 trillion parameters is about 1.4TB of weights. Your GPU has 24GBs.

Days after shipping the largest open model ever built, Moonshot paused new subscriptions. Not a billing glitch, a GPU crunch, it could not keep up with demand for its own model. Kimi K3 has 2.8 trillion parameters, and if the lab that trained it, sitting on a data center, is scrambling for the compute to serve it, the advice to self-host it that filled your feed last week is worth a second look.

The article argues that Kimi K3’s massive parameter count turns “open weights” into something most people can’t practically run locally: even at 4-bit, the model is ~1.4TB, far beyond typical consumer VRAM, and the mixture-of-experts design doesn’t eliminate the memory wall because most experts still need to reside in active memory. It explains why common strategies like offloading to slower buses, or aggressively quantizing down to 2-bit, fail for large MoE models by introducing errors that break routing and degrade outputs. To run K3 properly requires a data-center-style multi-node GPU cluster with high-bandwidth interconnects (e.g., NVLink and InfiniBand). As a result, “open” at this scale mostly translates into: you can rent/access it via APIs, or you can use it indirectly through distillation or enterprise fine-tuning after buying/operating the necessary compute. The conclusion is that K3 may be an excellent frontier model, but it is “the least runnable,” so the practical beneficiaries are those who already have the compute, or those who wait for smaller distilled derivatives that fit within consumer hardware constraints.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI