DevIQ — an AI engineering assistant, grounded in your code
DevIQ is an AI engineering assistant built on retrieval-augmented generation (RAG): it answers questions about a codebase with citations to exact line ranges, and says so when its sources don’t establish an answer — instead of guessing.
One pipeline.
Every step traced.
A question runs through one explicit RAG pipeline — keyword and vector search fused by rank, a bounded set of passages handed to an AI answer model, and every citation validated before an answer ships.
Ingestion
Reads only allowlisted folders, skips anything .gitignore excludes, and quarantines files that look like secrets before anything reaches an embedding provider.
Retrieval
Keyword and vector search run in parallel, always filtered to the asker’s organization, and merged by rank so the two scoring scales never need calibrating.
Answering
The model sees only the selected passages, each tagged with a citation token tied to a file, line range, and document version. Retrieved text is treated as data, never as instructions.
Validation
Any citation pointing at nothing the model was given gets removed. An answer with no valid citation is withheld — DevIQ says what’s missing instead of guessing.
Five choices
that shaped it.
| Decision | Why |
|---|---|
| Tenant scope comes from the session, never the client | A dev login stub sits behind the real session interface, so cross-tenant tests were real before auth existed. Composite foreign keys make the database itself reject a mismatched row. |
| One fixed embedding dimension, one model per index | pgvector needs a declared dimension to index, and vectors from different models aren’t comparable. A check constraint and a trigger reject mixing. |
| Keyword search tuned for code | Postgres doesn’t split camelCase, so ingestion adds split identifiers (proposeTests → propose tests) to the search column. |
| Rank fusion, not score blending | Reciprocal rank fusion (k = 60) combines keyword and cosine results without calibrating their scales. |
| No similarity-based “confidence” | Vector search always returns neighbors, so the model judges sufficiency directly rather than trusting a similarity score that would mislead. |
The reasoning behind each decision is recorded in five ADRs in the repository.
One clean run,
four gates.
Run on 2026-09-23 at commit b40328f: 15 labeled questions against the project’s own repository, claude-sonnet-5 (prompt answer-v3), voyage-code-3 embeddings, 96 indexed documents. It’s a baseline, not a claim of general accuracy.
| Metric | Result | Gate |
|---|---|---|
| Recall@5 | 82% (9/11) | ≥ 80% |
| Citation validity | 100% (62/62) | 100% |
| Declined unanswerable questions | 100% (3/3) | 100% |
| Declined answerable questions | 0% (0/11) | — |
| Cross-organization leaks | 0 | 0 |
A --sabotage-retrieval mode reverses rankings to prove the recall gate actually fails (it drops from 64% to 18% in a keyword-only run), and a preflight confirms every labeled file is indexed and current before anything is scored. Whether cited text genuinely supports each claim isn’t auto-scored — answers are stored beside each question’s rubric for human review.
Eight defects
the unit tests missed.
Running the eval against live models surfaced problems unit tests couldn’t catch. Each fix shipped with a test.
| Defect | How it showed up | Fix |
|---|---|---|
| Ingestion indexed the eval’s own reports | Retrieval could score itself against its previous answers | Anything .gitignore excludes is never indexed |
| Leaked tool-call markup | All gates passed while users would have seen raw markup in 9 of 12 answers | Markup is cut and counted as a metric; any answer still showing it fails the eval |
| Model sometimes omitted the abstain field | 4 of 15 answers errored, 2 on questions it should decline | Strict, API-enforced tool schema |
| A stale index looked like bad retrieval | New files were missing, so the model correctly declined and got blamed | Preflight checks every labeled file against the index and disk |
| Citations from a nested source lost their folder | A shortened path failed two cases | Paths are qualified by their source |
| A label pointed at a never-indexed file type | The freshness preflight flagged a SQL migration | Migrations are now indexed and still secret-scanned |
| A model error erased the retrieval record | Retrieval was reported as a miss when the model call had failed | Retrieval is saved before the model is called |
| The isolation check flagged a safe answer | Repeating the question’s own term counted as a leak | The check looks for a detail planted only in the other organization |
What it isn’t,
yet.
- Not built yet: real sign-in, roles, rate limits, and a working hosted deployment. The dev login is deliberately refused in production, so the hosted build can’t answer questions until real auth exists.
- Small, self-referential corpus: 15 questions about the project’s own repository. A larger labeled set on a second codebase is the next test of whether the numbers hold.
- Latency isn’t representative yet — 9.0s median and 43.6s p95 include waiting on the embedding provider’s free-tier limit of 3 requests per minute.
- Output repairs run at 53% (8 of 15): leaked markup is stripped before display, but it should be fixed at the source — for example with JSON structured output.
- Both recall misses share a cause: README and test chunks crowd implementation files out of the top five. Limiting their share is the next retrieval experiment.
- Next feature: propose_tests, a read-only workflow that proposes tests for a file as a diff for human review, never writing or running code.
DevIQ is an internal engineering demo — the app itself isn’t publicly deployed. The repository (README, five ADRs, the labeled eval set, and 238 tests) is public.