🦀 Building AI Agents in Rust – part 10
Author(s): Enzo Lombardi
Originally published on Towards AI.
A provider with no socket
Every provider Eugene has spoken to so far, going all the way back to Part 6, ends the same way: a URL, a header, a JSON body over HTTP. Anthropic’s Messages API, OpenAI’s Chat Completions, and even Ollama running on the same laptop as the agent all get the same treatment, because the Provider trait was built around one assumption: somewhere, there is a socket. Ollama already narrows the distance to zero latency-wise, but the shape of the call is still “make an HTTP request to localhost and wait.” This closing post asks what happens when you drop that assumption entirely and drive a local model the way you’d drive any other child process: stdin in, stdout out, no port to bind, nothing to curl while it’s still warming up.
The article explains how to implement the existing Provider abstraction without any HTTP socket by keeping a local model process (DwarfStar’s ds4 REPL) alive and communicating via stdin/stdout. It contrasts the usual HTTP-style “one request per turn” approach with a pipe-based design that preserves session state and avoids re-sending the full transcript each turn, using a mutex-protected child process and logic to read until the REPL prompt reappears. It also clarifies a key boundary: this pipe integration outputs plain text only and doesn’t support tool calling, so the agent loop behaves correctly by taking the “no ToolCall emitted” path. Finally, it argues that ds4-server is preferable when tool calling and multiple clients are needed, while the no-socket pipe route fits narrower use cases like CI jobs, sandboxed evaluations, or fast local inference for “Fast-tier” tasks, and notes how the new adapter is added to Eugene’s provider workspace.
Read the full blog for free on Medium.
Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.
Published via Towards AI
Towards AI Academy
We Build Enterprise-Grade AI. We'll Teach You to Master It Too.
15 engineers. 100,000+ students. Towards AI Academy teaches what actually survives production.
Start free — no commitment:
→ 6-Day Agentic AI Engineering Email Guide — one practical lesson per day
→ Agents Architecture Cheatsheet — 3 years of architecture decisions in 6 pages
Our courses:
→ AI Engineering Certification — 90+ lessons from project selection to deployed product. The most comprehensive practical LLM course out there.
→ Agent Engineering Course — Hands on with production agent architectures, memory, routing, and eval frameworks — built from real enterprise engagements.
→ AI for Work — Understand, evaluate, and apply AI for complex work tasks.
Note: Article content contains the views of the contributing authors and not Towards AI.
Recent Posts
