01
Verify, don't trust, citations
The model cites passages and verbatim quotes; the server finds each quote in the document before the interface is allowed to call it verified.
← All work
AI product · rebuild of my own v1
Margin
A document assistant you can check.
Upload a contract, paper, policy or report. Margin summarises it, answers questions, pulls out key facts and compares documents, and every claim comes with a quote that is verified and highlighted on the real page in one click.
- Role
- Product direction, UX and interface, retrieval and evaluation design
- Status
- Live working prototype · evaluations run
- Stack
- Python, FastAPI, Gemini, local embeddings, Next.js
1The problem
AI summaries of contracts and reports sound confident, but you can’t tell which parts are true.
2My role
Product direction, UX and UI, retrieval and evaluation design. Built with Claude Code under my direction.
3Outcome
Live. 46 of 47 answers correct, 6 of 7 unanswerable questions refused, and every cited quote verified.
01
The problem
AI summaries of contracts and reports sound confident, but you can't tell which parts are true.
Version 1 was one endpoint that sent a PDF's text to a model: no chunking, no retrieval, no citations, no interface.
Who it's for
People who have to act on long documents (contracts, policies, papers) and need to point at the sentence an answer came from.
02
Research
Status, honestly
Desk research and a v1 audit. No usability testing yet; a study plan is in the case study.
- Four synthetic documents (15 pages, 47 chunks) written as test material
- 47 answerable questions labelled with the page holding the answer, plus unanswerable ones
- An ablation of dense, BM25 and hybrid retrieval
03 · User flow
From a document to answers you can point at
Step 1 / 5
04
Key decisions
02
Fact and judgement, labelled
Every claim is marked stated (the document says it) or interpretation, and every quote verified, approximate or not found.
03
Show the failure, don't hide it
When 27 of 28 claims verify, the screen says so in an amber banner and flags the one it couldn't find in red.
04
“The document doesn't say”
Questions are answered only from the document. When it has no answer, Margin says so, and shows which passages it searched.
05
Final interface
Real screens from the live product.
06
Technical architecture
Verify, don't trust, citations
On upload
On request
On upload · step 1
Type & limits
File type from bytes, size and page limits, path-safe storage.
- Fact and interpretation labelled on every claim
- Failures shown, not hidden
- Anonymous workspaces scoped by a hashed token
07
AI implementation
How the AI is kept honest
01
Stated or interpretation
Every claim says whether the document states it or the model inferred it.
02
Three-level quote check
Verified (word for word), approximate, or not found, shown on each claim.
03
Refusal by design
Questions the document can't answer get “the document doesn't say”, with the passages searched.
04
End-to-end eval
The real Ask pipeline run over 47 answerable and 7 unanswerable questions, failures written up.
08
Before & after
Retrieval: which search finds the right page
hit@10.787from 0.681
hit@30.915from 0.872
hit@61from 0.936
MRR0.866from 0.787
From the README's retrieval table. Dense search alone is as good or better than hybrid except at hit@6; hybrid is kept for identifier-heavy queries. A small regression suite, not a benchmark.
What v1 actually was
v1v2
One endpoint sending a PDF's text to a modelSummary, Ask, Key facts, Compare, Library
No citationsEvery claim quoted, verified and highlighted on the page
Uploads saved with the client's filename (path traversal)Validated uploads, workspace-scoped storage
Crashed without a key; every error a 500Clear errors; works without a key for reading and search
09
Challenges
The one wrong answer
It inferred “at least twelve months” from a termination clause instead of the stated 24-month term. Written up, not hidden.
Free-tier latency
p50 1.9 s but p95 14.5 s on the free tier, so caching and a waking notice matter.
Scanned PDFs
No OCR yet: scanned files are detected and explained rather than read badly.
10
Outcomes
46 / 47
Answers containing every expected fact
6 / 7
Unanswerable questions correctly refused
100%
Cited quotes verified in the source
1.0
Right page in the top 6 results (hit@6)
A small synthetic set written by the person who built the pipeline: evidence that grounding holds, not a benchmark. The one wrong answer and the failed refusal are described in the README.
11
What's next
- →OCR for scanned PDFs
- →Accounts and sharing
- →A usability study (the plan is in the case study)
Built with Claude Code under my direction: I set the product direction and requirements, and reviewed the design, code and evaluations.
Next project
FITFLOW AI →
Every lead followed up. Every client kept.