On July 8, 2026, the Paris-based startup ZML released LLMD, an inference server that runs large language models across five chip families (NVIDIA, AMD, Google, Intel, Apple) from a single codebase. SFEIR reads this as a signal: as training cedes the spotlight to inference, the real battleground — and cost center — shifts toward serving, where cost per token, latency, and dependence on silicon are decided.
ZML's bet comes down to three words, model to metal: not offering yet another model, but a layer that decouples the model from the hardware. The stack has four layers. At the top, models (Qwen, Gemma, Mistral, LLaMa) loaded zero-copy via a virtual file system from Hugging Face, S3, or GCS. Then LLMD, a server exposing an OpenAI-compatible API (drop-in) with continuous batching, paged attention, prefix caching, tool calling, and Prometheus metrics. Below that, ZML compiles the graph upfront, once and for all, into a hermetic native binary in Zig + MLIR, with no Python in the execution path. This binary runs on five backends: CUDA, ROCm, TPU, oneAPI, Metal. The elegance lies in being "portable, not leveled" — chip-specific paths (FlashAttention, AITER) are preserved. Figures announced (by the vendor): images from 1.7 GB (CUDA) to ~140 MB (Apple), cold start of 1-2 s on an 8B model, and the DFlash accelerator (claimed "up to 10×," ~6.17× in the underlying research).
Two components, two licenses: ZML (the framework) is open source (Apache-2.0, >90% Zig); LLMD (the server) is not, free at launch while usage data is collected. The demo runs in two commands on Apple Silicon Macs; a 27B model in BF16 requires ≥ 64 GB of unified memory.
SFEIR reads the object through three client-facing lenses: FinOps (choosing the cheapest chip → acting on the cost per token), architectural freedom (Design to Exit, built-in reversibility, cf. France Télévisions/ALIX) and sovereignty (European chips Axelera, Kalray, SiPearl, VSORA; a VivaTech 2026 partnership with Scaleway, VSORA, and the Île-de-France Region, integration into the Jotunn8 processor).
Unsparing verdict: it is an alpha, not for production; support for specific local machines (DGX Spark, Ryzen AI Max+) is neither named nor benchmarked. Against vLLM (server-GPU throughput) and llama.cpp (single-user local), LLMD aims for the middle ground. Not to switch to today, but to place "under active watch": a serious, made in France candidate to become the "docker run of inference."