PolicyRAG

The problem

LLMs answer questions about SEC filings fluently, and sometimes wrongly. For analysts and compliance teams a confident wrong answer is worse than no answer, so the system has to show receipts.

Approach

PolicyRAG is a full RAG pipeline behind a chat interface. Documents are chunked into ChromaDB, retrieval goes through a reranker, and generation is constrained to cite the numbered context passages it used. The backend is FastAPI with SSE streaming and per-request JWT verification through Supabase auth; the LLM provider is swappable behind one interface.

Results

No published numbers yet: the honest version of this section. What exists is the scoring methodology: every answer gets a weighted Trust Score built from faithfulness (40%, NLI entailment of each claim against the retrieved context), citation precision (20%, does the cited chunk actually entail the citing sentence), citation recall (10%, share of substantive sentences with a citation), context relevance (15%, cosine similarity between query and retrieved chunks), and completeness (15%, LLM-as-judge). The repo ships an automated benchmark suite to produce the real distribution: running it and reporting the output is next.