Testing, review, reliability and security of AI-produced software.
24 fiches · 47 entities · Updated
Can software written largely by agents be trusted to work, and to resist misuse? That question anchors this collection. Defects and vulnerabilities enter differently when code is generated; output can pass tests while quietly degrading a codebase; and only some verification practices — automated gates, human review, adversarial checks — actually hold up. Recurring topics include review that scales to machine-speed output and the new attack surface that generated dependencies open. Fiches trace risks particular to autonomous generation, from silent quality erosion to injection and supply-chain exposure, alongside the controls proposed to contain them. The through-line is trust in AI-produced software across its whole lifecycle.
Engineering article published on **Uber**'s blog by six engineers (Matt Mathew, Prasad Borole, Meng Huang, Sergey Burykin, Gaurav Goel, Bayard Walsh) on **May 21, 2026**, laying out the **AI agent identity and access-control doctrine** deployed in production at Uber for several thousand internal agents. **Pivotal thesis**: existing identity models (human + workload) do not describe **agency** — *« an agent is best defined as an entity that is authorized to act for or in the place of another »* — and lose **provenance** across the hops of an agentic workflow. **Two operational problems identified**: (1) ***« Current Identity Model Doesn't Describe Agency »*** — delegation is the default mode, workflows are compositional (agents calling agents calling tools), behavior is dynamic (plans evolve based on intermediate results); (2) ***« Original Provenance Isn't Effectively Carried Forward Across Agents to Systems »*** — *« Execution context (originating user, intermediate agents) is dropped across agent hops. »* **Proposed architecture** as an extension of Uber's Zero Trust Architecture: **Agent Registry** (source of truth for agent↔workload mappings) + **AI Agent Mesh** (inter-agent data plane) + **STS (Security Token Service)** (issuance of short scoped JWTs) + **MCP Gateway** (policy enforcement point for tool invocation) + **AI Gateway** (mediation of external LLM calls with guardrails) + **SPIRE** (workload credential provider). **Cryptographic mechanics**: workloads fetch cryptographically signed **SVIDs (SPIFFE Verifiable IDs)** from SPIRE → the SDK requests a JWT from the STS via the workload identity → the STS verifies the agent's authorization against the Agent Registry → a short token (TTL on the order of minutes) is issued for a **specific single-hop destination** (targeted `Audience` claim). **Pivotal doctrine**: ***« Single-hop, short-lived tokens. Every JWT minted by the STS is intended for a single hop, with a specific Audience claim and a short time-to-live in the order of minutes. »*** **Actor-chain preservation**: multi-hop example with on-call engineer `user1` → Oncall Agent (Workload-1) → Investigation Agent (Workload-2) → MCP Gateway; the final JWT carries a verifiable **actor chain `[user1, oncall-agent, investigation-agent]`**, enabling tool-level access decisions based on the **full request history**. **Standardization**: a **Standardized A2A (Agent-to-Agent) Client** that automates STS exchanges and actor-chain propagation — *« the secure path is also the easiest path for developers to implement A2A calls »* — phased migration of legacy agents. **Production metrics**: ***« P99 latency for the STS Token Exchange API is consistently below 40 milliseconds »***, thousands of internal agents onboarded, a real-time observability dashboard tracing multi-agent sessions. **Long-term vision — three-layer framework**: (1) Identity & Trust Foundation (verifiable agent identity + delegation chains), (2) Dynamic Access Control (context-based permissions + human-in-the-loop), (3) Unified Enforcement Plane (centralized, observable policy). **Standards alignment**: the IETF **WIMSE** working group + draft `draft-klrc-aiagent-auth-01` *AI Agent Authentication and Authorization*, conceptually grounded in **OAuth 2.0 Token Exchange (RFC 8693)** and **SPIFFE/SPIRE** (CNCF graduated). The first reference publication from a non-AI-lab hyperscaler (logistics/mobility) that industrializes agent security at the infrastructure level, bridging the doctrinal gap between skills/harness frameworks (Vincent, Lattice, PROJ-AI) and enterprise-grade identity questions.
Empirical study by the **Compare the Market** engineering team (Meerkat Careers, UK) evaluating four approaches to **context retrieval for AI code review**: Baseline (no additional context), **RAG** (vector search), **GKG** (GitLab Knowledge Graph, AST-based knowledge graph), and **GKG+RAG** (hybrid). Evaluation on **79 real merge requests** with **MLflow on Databricks**. Striking result: **RAG performs worse than the baseline** on almost every metric — vector noise is counterproductive for code review. **GKG outperforms RAG by +21%** in inline comments coverage (0.696 vs 0.577) through structural AST understanding (Tree-sitter + Kuzu graph database). Code requires **structural** understanding (callers, signatures, hierarchies), not mere semantic similarity. GKG costs 4× the baseline but delivers measurable improvements; RAG costs 3× with no improvement. Implemented as a **Docker sidecar** in CI/CD wrapping the GKG binary (still in GitLab beta) with a local MCP server.
#Compare the Market#Meerkat Careers#AI code review
Équipe Engineering Compare the Market (Meerkat Careers, UK — site de comparaison d'assurances et services financiers).
Wharton study (Generative AI Labs): expert personas don't improve LLM factual accuracy - GPQA Diamond and MMLU-Pro benchmarks - SSRN
#AI prompting#personas#LLM accuracy
Savir Basil · Ina Shapiro · Dan Shapiro · Ethan Mollick · Lilach Mollick · Lennart Meincke (Generative AI Labs, The Wharton School, University of Pennsylvania)
First AI-orchestrated cyber espionage campaign - Claude Code manipulated - Chinese state actor - 30 global targets - 80-90% automated - Jailbreaking - Anthropic Threat Intelligence
**Mathieu Eveillard** publishes on his personal blog on **December 7, 2022** (last updated March 17, 2025) a **point-by-point counter-argument** to the famous essay by **David Heinemeier Hansson (DHH)** *"TDD is dead. Long live testing."* (RailsConf 2014). Article categorized **craft / best-of**, a **software craftsman** stance that defends **Test-Driven Development** without dogmatism. **Pivotal distinction** that DHH misses according to Eveillard: ***"Test-first"*** (writing all the tests before any code) vs ***"Test-Driven Development"*** (tests **guide** me in writing code, so each time I write a bit of code *"in reaction"* to a new test). DHH actually criticizes *Test-first* while calling it TDD — a confusion that **hides an entirely different way of programming**. **Point-by-point responses**: (1) *"TDD as hammer to beat down the nonbelievers"* — Eveillard concedes the deontological point but redefines *"good code"*: not just the absence of bugs but **fine-grained unit tests** documenting behavior at the lowest level, co-located with the code, a **safety net**; (2) *"Rebalance from unit to system"* — TDD **says nothing** about system tests and **does not say** there is nothing outside TDD; system tests do **not replace** unit tests (an income tax return tested end-to-end makes no sense); **test pyramid** — each type contributes its share, unit tests for **millisecond** feedback + early bug detection; (3) *"Horrendous monstrosities of architecture (service objects, command patterns)"* — Eveillard responds that he **does not see these effects in functional programming**, so the effect is likely due to **OOP**, not TDD; but concedes that excessive dependency injection can couple test and implementation. **Balanced conclusion**: *"TDD is not a religion, it's a tool"*. TDD is particularly well suited to **domain code** (the functional core of a *bounded context*, the *core of the hexagon*) — calculation engines, fine-grained business rules, edge cases galore — ***"30% of the codebase at most"***. Mentions the **Law of the Instrument** (if the tool doesn't help, it's because you've fallen into it). **Relevance to the corpus**: a **craft article outside the AI corpus** but worth archiving to position current debates on coding agents (Beck's *Augmented Coding Beyond Vibes*, 2025-06-25, Vibe Coding vs TDD, Frizzo's *writing muscle atrophy*) within the historical lineage of craft debates around TDD. To be used as a **library foundation** for training sessions.
#Mathieu Eveillard#TDD#Test-driven development
**Mathieu Eveillard** — développeur / coach craft / formateur (blog personnel mathieueveillard.com, services *Accompagnement* et *Office hours*). Identité publique : *artisan logiciel* avec une pratique pédagogique autour du TDD · du DDD et du craft. Newsletter hebdomadaire (*"Chaque mercredi, une idée pour démarrer la journée"*).