DuckDB and the changing physics of analytics
Guest post by **Andy Warfield**, an engineer on the **S3** team at **AWS**, published on **August 26, 2026** on *All Things Distributed*, **Werner Vogels**'s blog, who introduces it in a few lines signed "--W": **3,554 words** per the page. The text serves as the vehicle for the announcement that **DuckLabs**, the team behind **DuckDB**, is joining **AWS**. (A) The thesis: systems computing is about seeking the elegant trade-off against a moving "physics" — the ratios between memory speed, network, and compute — and that physics has changed. Warfield quantifies the gap: an **m1.xlarge** from 2007 offered **15 GB of RAM**, **4 virtual cores**, and **~1 Gb/s** of network; an **m8g.48xlarge** today offers roughly **50×** more of each of the three. Dataset growth, meanwhile, follows a distribution whose tail consists of very large volumes. (B) The consequence: distributed processing — **MapReduce**, **Spark**'s **RDDs** — was designed under the I/O constraints of the early 2000s, and much of the work assigned to it no longer needs to leave the application. Hence the embedded, in-process library engine, running in the application's address space, of which **DuckDB** is the example. Warfield anchors this in the *Scalability! But at what COST?* paper (2015) and **Paul Barham**'s epigraph: "You can have a second computer once you've shown you know how to use the first one." He states an explicit caveat: "When a job genuinely needs a thousand machines, it needs a thousand machines." The corpus already holds [[vogels-tech-predictions-2026-allthingsdistributed-2025-11-25]] from the same blog and [[anthropic-self-service-data-analytics-claude-agentic-stack-2026-06-03]] on self-service analytics.
Andy Warfield · ingénieur du service S3 chez AWS · en billet invité sur *All Things Distributed* ; introduction de Werner Vogels · CTO d'Amazon.