# warfield-duckdb-changing-physics-analytics-2026-08-26

## Veille

Guest post by **Andy Warfield**, an engineer on the **S3** team at **AWS**, published on **August 26, 2026** on *All Things Distributed*, **Werner Vogels**'s blog, who introduces it in a few lines signed "--W": **3,554 words** per the page. The text serves as the vehicle for the announcement that **DuckLabs**, the team behind **DuckDB**, is joining **AWS**. (A) The thesis: systems computing is about seeking the elegant trade-off against a moving "physics" — the ratios between memory speed, network, and compute — and that physics has changed. Warfield quantifies the gap: an **m1.xlarge** from 2007 offered **15 GB of RAM**, **4 virtual cores**, and **~1 Gb/s** of network; an **m8g.48xlarge** today offers roughly **50×** more of each of the three. Dataset growth, meanwhile, follows a distribution whose tail consists of very large volumes. (B) The consequence: distributed processing — **MapReduce**, **Spark**'s **RDDs** — was designed under the I/O constraints of the early 2000s, and much of the work assigned to it no longer needs to leave the application. Hence the embedded, in-process library engine, running in the application's address space, of which **DuckDB** is the example. Warfield anchors this in the *Scalability! But at what COST?* paper (2015) and **Paul Barham**'s epigraph: "You can have a second computer once you've shown you know how to use the first one." He states an explicit caveat: "When a job genuinely needs a thousand machines, it needs a thousand machines." The corpus already holds [[vogels-tech-predictions-2026-allthingsdistributed-2025-11-25]] from the same blog and [[anthropic-self-service-data-analytics-claude-agentic-stack-2026-06-03]] on self-service analytics.

## Titre Article

DuckDB and the changing physics of analytics

## Date

2026-08-26

## URL

https://www.allthingsdistributed.com/2026/08/duckdb-and-the-changing-physics-of-analytics.html

## Keywords

DuckDB, DuckLabs, AWS acquisition, embedded analytical engine, in-process library, changing physics of systems, COST paper, Paul Barham, MapReduce, Spark RDD, distributed processing, MonetDB, X100, CWI, vectorization, SQLite, S3 Tables, Apache Iceberg, Iceberg extension v2 v3, asynchronous I/O, NIC saturation, AWS Lambda, ATTACH, CONNECT, WebAssembly, glibc of structured data, DuckDB Foundation, MIT license, m1.xlarge, m8g.48xlarge, per-core efficiency, continuous analytics

## Authors

Andy Warfield, ingénieur du service S3 chez AWS, en billet invité sur *All Things Distributed* ; introduction de Werner Vogels, CTO d'Amazon.

## Ton

Profile: long-form first-person technical post, in the register of an engineer's narrative rather than a press release, medium-to-high technical level, aimed at developers and data-systems architects. The structure moves from personal recollection to the announcement: an anecdote from a British pub where physicist friends mock computer science — "any discipline that needs to put science in its name probably isn't a science" — is used to set up the thesis that this absence of an immutable truth is precisely what makes the field interesting. Three examples situated in time follow (Berkeley's **NOW** project, the author's own work on **Xen**, the **MonetDB**/**X100** research at CWI), the observation that these constraints recur in cycles — Xen's virtualization ideas had already been laid out on IBM mainframes in the 1960s — and then the application to data processing. Warfield qualifies his reading of distributed systems: "I mean this much more as an observation than as a criticism of these systems, because they were building for their own physics." The section headings play out the duck proverb (*If it walks like a duck… / …and quacks like a duck / …it must be a duck*), and the anecdote of Hannes's pet duck, Wilbur, is given as the origin of the name. Directly quotable: the phrase "the glibc of structured data," the statement that "'analytics' is becoming less of a separate activity that happens to data somewhere else, and more of something you do continuously as you build," Hannes Mühleisen's reported goal — enabling "anyone to work with data confidently" — and the line from the SIGMOD paper about the absence of any revolutionary component in DuckDB.

## Pense-betes

- **The argument is a ratio, not an absolute value**: what has shifted are the ratios between compute, memory, and network on a single machine. Reference points given: the 2007 m1.xlarge (15 GB, 4 vCPUs, ~1 Gb/s) versus the m8g.48xlarge (~50× across all three axes); the author's MacBook Pro claims 3 to 5× the cores and RAM, ~40× the memory bandwidth, and more than 100× the I/O bandwidth of the m1.xlarge. A server today outperforms the clusters on which many people ran Hadoop and Spark.
- **Data growth is a distribution, not a uniform trend.** The largest datasets grow exponentially, but they form the tail; many others follow human-scale magnitudes — a company's size, its number of customers, the number of daily bank transactions. It is the gap between the two that fuels the renewed interest in single-host engines.
- **The COST paper as a reference point**: *Scalability! But at what COST?* (McSherry, Isard, Murray, 2015) compares a well-optimized single-thread implementation to distributed frameworks on the same tasks — on graph processing workloads, the single thread beats distributed systems running on **128 cores**, and it takes **512 cores** for the distributed system to pull back ahead. Warfield points out that the authors themselves worked on distributed systems: the point is per-core efficiency, not a rejection of distribution.
- **What the "library" form changes**: the engine runs in the application's address space, on memory structures already present, and cares as much about its own overhead as about the queries it executes. It doesn't have to sit in the client — it becomes a component that can be placed wherever it's useful in the stack, up to compiling to **WebAssembly** and running in a browser tab (shell.duckdb.org). Claimed lineage: the SIGMOD 2019 demo paper leaned on **SQLite**'s popularity, and the two founders come from the CWI lab that produced MonetDB and X100.
- **The AWS ↔ DuckLabs timeline, as reported**: AWS becomes a DuckLabs customer and sponsors the **Iceberg** extension during the work on **S3 Tables**, with the goal of extending Iceberg beyond the Spark world; the extension now covers the **v2 and v3** specifications and drove the **asynchronous I/O** expected in version **2.0**, whose design goal is to saturate the NIC while scanning tables on S3. The article's single figure: more than **800,000 downloads per week** for the extension. Other avenues mentioned: **Lambda** as a primitive for launching queries, and the **ATTACH**/**CONNECT** commands toward AWS engines.
- **Terms of the deal**: DuckLabs joins AWS **as a subsidiary**; the DuckDB project remains open source under the stewardship of the **DuckDB Foundation**, developed by the DuckLabs team, under the **MIT** license; the team stays in Amsterdam. Hannes Mühleisen and Mark Raasveldt explain their reasoning in a separate post on the DuckLabs blog, not covered here. AWS states it is targeting developers "and increasingly agents," and says it already uses DuckDB internally for dashboards, CLI tooling, server-side accelerators, and bridges between systems.
- ⚠️ **What the text does not provide**: no amount or financial structure for the deal, no comparative performance data on DuckDB itself, and no quantified governance details for the Foundation (composition, rights, term commitments). The post's only quantified reference point remains the extension's download volume.
- **Related**: [[netflix-uda-unified-data-architecture-knowledge-graph-2025-06-12]] on a single data model consumed by multiple engines, and [[clouded-judgement-121225-long-live]] on the shift of value toward systems of record.

## RésuméDe400mots

Andy Warfield, an engineer on the S3 team at AWS, published a guest post on All Things Distributed on August 26, 2026, introduced by Werner Vogels. In it, he explains why embedded analytical engines like DuckDB are gaining importance, and announces that DuckLabs, the team that develops DuckDB, is joining AWS.

His reading grid is one of a moving "physics." Where the physical sciences explore invariants, systems computing seeks the elegant trade-off against ratios that shift: memory speed versus network speed, richness of abstractions versus available power. He cites three moments — Berkeley's NOW project, his own work on Xen, and the MonetDB and X100 research at Amsterdam's CWI, where the bottleneck of query processing had shifted from disk to CPU — and notes that these constraints recur in cycles.

Applied to data, this grid explains distributed processing. Processing is always simpler and more efficient on a single fast machine, but when a server's disk or network card can no longer read the desired volume, one partitions. That was the constraint of the early 2000s, the one that produced MapReduce and then Spark's RDDs. Warfield notes two qualities of these systems: they innovated heavily on developer ergonomics, and they accepted a fixed cost of planning and distribution, betting on throughput gained by adding machines rather than on per-unit efficiency.

But the ratios have changed. A current instance offers roughly fifty times the memory, cores, and network bandwidth of the largest EC2 instance from 2007, while dataset growth follows a distribution whose extreme cases form the tail. The 2015 Scalability! But at what COST? paper had already shown that a carefully optimized single-thread implementation could beat distributed frameworks running on one hundred twenty-eight cores.

DuckDB, launched in 2018 by Hannes Mühleisen and Mark Raasveldt, applies this logic: an in-process library analytical engine, running in the application's address space, following SQLite's distribution model. AWS became a DuckLabs customer and then a sponsor of the Iceberg extension, alongside its work on S3 Tables; the extension now supports Iceberg v2 and v3 and exceeds 800,000 downloads per week.

Warfield does not present the embedded model as a replacement: when a job requires a thousand machines, it requires them. What is changing, he writes, is that much of the work done on data never actually needed a cluster. DuckLabs joins AWS as a subsidiary, with the project remaining open source under the MIT license and under the stewardship of the DuckDB Foundation.

## GrapheDeConnaissance

- Andy Warfield —a_créé→ DuckDB and the changing physics of analytics (DOCUMENT, 0.97)
- Werner Vogels —publie→ DuckDB and the changing physics of analytics (DOCUMENT, 0.93)
- Andy Warfield —travaille_chez→ AWS (ORGANISATION, 0.96)
- AWS —collabore_avec→ DuckLabs (ORGANISATION, 0.97)
- DuckLabs —fait_partie_de→ AWS (ORGANISATION, 0.96)
- Hannes Mühleisen —a_créé→ DuckDB (TECHNOLOGIE, 0.97)
- Mark Raasveldt —a_créé→ DuckDB (TECHNOLOGIE, 0.97)
- DuckDB —est_instance_de→ moteur analytique embarqué (CONCEPT, 0.95)
- DuckDB —s_inspire_de→ SQLite (TECHNOLOGIE, 0.92)
- DuckDB —est_basé_sur→ MonetDB (TECHNOLOGIE, 0.87)
- DuckDB Foundation —permet→ maintien de DuckDB en open source sous licence MIT après l'entrée de DuckLabs chez AWS (AFFIRMATION, 0.94)
- Andy Warfield —affirme_que→ les rapports entre calcul, mémoire et réseau sur une seule machine ne sont plus les contraintes qu'ils étaient (AFFIRMATION, 0.95)
- m8g.48xlarge —mesure→ environ 50× la mémoire, les cœurs et la bande passante réseau d'une m1.xlarge de 2007 (15 Go, 4 vCPU, ~1 Gb/s) (MESURE, 0.93)
- Andy Warfield —affirme_que→ la croissance des jeux de données suit une distribution dont les très grands volumes sont la queue, beaucoup d'autres suivant des grandeurs humaines (AFFIRMATION, 0.9)
- MapReduce —résout→ contrainte de bande passante d'I/O des grands jeux de données du début des années 2000 (CONCEPT, 0.93)
- Spark —utilise→ Resilient Distributed Datasets (CONCEPT, 0.94)
- Scalability! But at what COST? —mesure→ une implémentation mono-thread optimisée bat des systèmes de graphe distribués sur 128 cœurs, le distribué ne repassant devant qu'à 512 cœurs (MESURE, 0.93)
- Paul Barham —affirme_que→ « You can have a second computer once you've shown you know how to use the first one » (CITATION, 0.94)
- moteur analytique embarqué —réduit→ surcoût de planification, d'expédition de tâches et d'aller-retour réseau du traitement distribué (CONCEPT, 0.9)
- Andy Warfield —affirme_que→ l'embarqué ne remplace pas le distribué : un travail qui exige mille machines les exige toujours (AFFIRMATION, 0.94)
- extension Iceberg de DuckDB —mesure→ plus de 800 000 téléchargements par semaine (MESURE, 0.92)
- extension Iceberg de DuckDB —utilise→ Apache Iceberg (TECHNOLOGIE, 0.95)
- DuckDB —s_applique_à→ S3 Tables (TECHNOLOGIE, 0.93)
- S3 Tables —fait_partie_de→ S3 (TECHNOLOGIE, 0.95)
- I/O asynchrone —permet→ saturer le NIC lors du scan de tables stockées sur S3, attendu en DuckDB 2.0 (AFFIRMATION, 0.9)
- DuckDB —s_applique_à→ WebAssembly (TECHNOLOGIE, 0.91)
- Andy Warfield —affirme_que→ DuckDB est « the glibc of structured data » : une dépendance sobre et ubiquitaire à laquelle beaucoup de logiciels se lient sans y penser (CITATION, 0.92)
- Andy Warfield —prédit→ l'analytique cesse d'être une activité séparée pour devenir quelque chose que l'on fait en continu pendant qu'on construit (AFFIRMATION, 0.9)
- AWS —utilise→ DuckDB en interne pour des tableaux de bord, de l'outillage CLI, des accélérateurs côté serveur et des ponts entre systèmes (AFFIRMATION, 0.89)
- Xen —s_inspire_de→ virtualisation des mainframes IBM des années 60 (CONCEPT, 0.88)

---
Canonical: https://www.thekb.eu/en/fiches/warfield-duckdb-changing-physics-analytics-2026-08-26/