Author(s): Codebook Fusion

Originally published on Towards AI.

Every time someone asked me how to add AI to a Java backend, I gave the same answer in one breath: pick Spring AI or LangChain4j, done.

I was answering the wrong question.

The article argues that focusing only on the framework (Spring AI vs. LangChain4j) misses the more stable, higher-value parts of the Java AI stack. It outlines a four-layer stack: a modern JVM runtime (virtual threads and structured concurrency) to make blocking calls and parallel tool workflows efficient and safe; agent frameworks that sit above the runtime but churn quickly; protocol layers like MCP and A2A that let typed, testable enterprise services become reusable agent tools without rewriting business logic; and an optional “model inside the JVM” layer for in-process inference (e.g., Jlama, GPU-accelerated options, and Llama3.java). It highlights key Java 27 capabilities (like JFR redaction, quantum-safe TLS, and compact object headers), lists noteworthy agent frameworks beyond the big two, and explains practical guidance for building in order—pin a runtime, choose a framework based on your stack, use structured concurrency and scoped identity, expose existing services via MCP, add observability early, and optionally start local inference on smaller workloads. It concludes with honest downsides (Java isn’t for training, runtime features may still be preview, churn and limited local inference can be trade-offs) and a final recommendation to invest attention in the layers underneath the framework.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI