Compare the Market

Compare the Market's engineering team published an empirical evaluation on 6 March 2026 comparing four context-retrieval strategies for AI code review, measured across 79 real merge requests with MLflow on Databricks. The four arms were a baseline with no added context, RAG via embedding search, «GKG» (GitLab Knowledge Graph, built on Tree-sitter AST parsing and the Kuzu graph database), and a GKG+RAG hybrid.

The finding that gives the study its weight is negative: RAG performed worse than the baseline on nearly every metric, including inline comment coverage, summary coverage, and score accuracy. The team names four causes: noise from vector similarity retrieving code that merely looks alike, false positives, no grasp of cross-file relationships, and a distraction effect that misleads the model. GKG landed at 0.696 inline comment coverage against RAG's 0.577, a 21 percent gap. The structural reason stated is that review requires knowing who calls a function and how it sits in the architecture, which an AST captures and semantic similarity does not.

The implementation was a workaround: GKG being in beta and not integrated into GitLab CI/CD, the team wrapped its binary in a Docker sidecar that indexes the codebase on each merge request pipeline and exposes tooling through a local «MCP» server. Cost is 4x baseline, versus 3x for RAG's worse results.

SFEIR's fifth SDLC review episode cites this as a warning: a reviewer built on vector RAG degrades the review.