Hybrid Retrieval | Deterministic Guardrails | AI Guardrails | Automated Lead Conversion
Project Executive Snapshot
| Industry / Niche | Enterprise knowledge retrieval, lead qualification and automated scheduling |
| Solution | OrionSoft Enterprise Conversational Agent and RAG Pipeline |
| Architecture | Autonomous multi-path router, hybrid dense + sparse retrieval, cross-encoder reranker, deterministic AST math engine |
| Tech Stack | Python 3.11, Flask, FAISS, BM25Okapi, Sentence-Transformers, React 19, TypeScript, Supabase PostgreSQL, Resend |
| Routing Accuracy | 100% (1.000) across 15 benchmark test cases |
| Retrieval Hit Rate @ 3 | 100% (1.000) across 29 domain test queries |
| Mean Reciprocal Rank | 1.0000 |
| Average Response Time | 1,137.14 ms end to end |
| Abstention Correctness | 100%: no made-up answers on questions the documents cannot support |
| Data Leak Block Rate | 100% of internal confidential terms blocked from external web search |
About the Project
While standard business chatbots perform well in canned demos, they routinely fail under enterprise production scrutiny—hallucinating pricing, fabricating policies, producing arithmetic errors, and risking proprietary data leaks through external tools.
This project set out to fix that. OrionSoft built an enterprise conversational agent on a Retrieval-Augmented Generation (RAG) pipeline that answers only from verified company knowledge. The same chat widget also qualifies website visitors as leads and helps them book a discovery call, so one tool handles support, sales and scheduling.
The Challenge
Moving a chatbot from prototype to a mission-critical production system means solving five problems at the same time:
- Made-up answers: When documents are unclear or incomplete, language models tend to fill the gaps. In areas like pricing, legal terms and compliance, one wrong answer can turn into a real liability.
- Data leakage: An agent that can use external search may accidentally send confidential terms, such as executive salaries, internal budgets or leave policies, to third-party search APIs while expanding a prompt.
- Retrieval blind spots: Pure semantic search misses exact product codes, acronyms and identifiers, while pure keyword search misses meaning, paraphrasing and multi-step questions.
- Unreliable maths: Language models cannot be trusted with percentage reductions, multi-tier discounts or budget calculations. They often return numbers that sound right but are wrong.
- Lead qualification friction & drop-off: Standard contact forms attract spam and get abandoned, and ordinary chat widgets rarely verify who they are talking to or capture buying intent in the moment.
Our Approach
We built the system around one principle: the chatbot should be as careful as a good employee. It checks where an answer comes from, uses the right tool for the job and says so when it does not know.
Every question first passes through an autonomous router that picks the safest path. Retrieval combines meaning-based and keyword-based search, a reranker cleans up the results, and an evidence check decides whether the material is strong enough to answer from. Only then does the model write a reply, with citations attached
System Architecture & Information Flow
The platform splits every incoming query into a dedicated decision path. It pairs hybrid retrieval with a deterministic calculation engine and an evidence sufficiency gate that sits in front of the final answer.
Figure 1: How a query moves from the router, through retrieval and evidence checks, to a grounded answer and lead capture.
Key Features Delivered
1. Autonomous Routing and the Anti-Leak Gate
Incoming questions are classified into one of five execution paths:
| Route | What it does |
| internal_rag | Sends the question to the hybrid dense + sparse retrieval pipeline |
| google_drive | Searches the connected Google Drive workspace with role-based access control (RBAC) |
| calculator | Solves maths with a deterministic engine built on Python Abstract Syntax Trees |
| web_search | Searches only an authoritative whitelist of domains for external technology releases |
| abstain | Politely refuses questions that are unanswerable or out of scope |
The Anti-Exfiltration Gate sits in front of all of them. Any query that touches internal employee policies, compensation data or other confidential keywords is strictly barred from reaching external search tools. This shuts the door on prompt-injection attacks designed to leak private information.
2. Hybrid Retrieval With Cross-Encoder Reranking
No single search method is good enough on its own, so every document is indexed in two ways:
- Dense vector index: all-MiniLM-L6-v2 creates 384-dimensional vectors that capture meaning and context, compared using cosine similarity.
- Sparse keyword index: A BM25Okapi inverted index handles exact tokens, acronyms and product terminology.
- Cross-encoder reranker: Top results from both streams are merged and re-scored with cross-encoder/ms-marco-MiniLM-L-6-v2, which reads the question and each chunk together. False positives are removed before the model generates anything.
3. Exact Maths, Not Estimates
A question like “If our cloud budget is $184,500 and we reduce spend by 17%, what is the net budget?” never reaches the language model. It is parsed into an isolated Abstract Syntax Tree and solved using only approved operations: add, subtract, multiply and divide. The use of eval() or exec() is banned outright, so calculations are 100% accurate with no code-injection risk.
4. Honest “I Don’t Know” Responses (Rule 6)
An Evidence Sufficiency Evaluator scores the retrieved chunks on query-term overlap, reranker confidence and agreement between sources. If the evidence comes back INSUFFICIENT or CONFLICTING, the assistant triggers negative abstention and replies that the available company documents do not specify this information. This contract keeps ungrounded speculation from ever reaching the user.
5. Verified Lead Qualification and One-Click Calendar Booking
- Step 1: The visitor shares their name, company and email. These are checked for valid syntax, disposable domains (30+ providers blocked) and live DNS existence.
- Step 2: Qualified inquiries are verified in real time, screened against 30+ disposable email domains, and instantly dispatched to sales leadership inboxes with an AI conversational summary.
- High-intent booking: When a conversation shows buying intent, the agent asks for contact details, saves the record to Supabase PostgreSQL (with a local SQLite backup) and generates one-click links for Google Calendar and Outlook / Office 365, plus an RFC 5545 .ics invite.
Technology Stack
A stack chosen for speed, privacy and reliability under real traffic:
- Backend: Python 3.11 with Flask
- Retrieval: FAISS dense index, BM25Okapi keyword index and Sentence-Transformers models, all running locally
- Frontend: React 19 and TypeScript, delivered as an embeddable widget
- Database: Supabase PostgreSQL with a local SQLite dual-write backup
- Email & Scheduling: Resend for one-time codes, plus Google Calendar, Outlook and .ics support
Engineering Trade-offs & Key Decisions
Good architecture is a series of honest trade-offs. These are the three biggest decisions on this project and the reasoning behind each.
| Decision | The Trade-off | Why It Was Worth It |
| Local models vs. cloud API embeddings | Packaging the embedding and reranker models inside the Docker image adds about 1.2 GB to the build. | No third-party rate limits, retrieval under 150 ms, no external data transmission and fully reproducible offline results. |
| Deterministic AST parser vs. LLM function calling for maths | Needs dedicated parsing grammars for natural-language maths expressions. | Zero arithmetic errors, hundreds of milliseconds saved on model round-trips and no code-injection surface. |
| Dual-write storage (SQLite + Supabase PostgreSQL) | Adds some write-coordination logic across local and cloud storage. | If the cloud connection drops, leads, conversations and bookings keep saving locally with no data loss and no visible downtime. |
Benchmark Results
The pipeline was tested with automated evaluation suites that measure routing accuracy, retrieval quality and speed.
Benchmark A: Autonomous Decision & Route Evaluation
Tested against 15 diverse, real-world scenario prompts.
| Evaluation Dimension | Achieved | Target Baseline |
| Overall Pass Rate | 100% (15/15) | 90% or higher |
| Route Accuracy | 100% (1.000) | 95% or higher |
| Tool Selection Accuracy | 100% (1.000) | 90% or higher |
| Unnecessary Tool Rate | 0.0% (0.000) | 5% or lower |
| Negative Abstention Correctness | 100% (1.000) | 100% |
| Calculation Accuracy | 100% (1.000) | 100% |
| Structured Output Validity | 100% (1.000) | 95% or higher |
| Average End-to-End Latency | 1,137.14 ms | 1,500 ms or lower |
Benchmark B: Retrieval Quality (29 Domain Queries @ K=3)
Tested across the OrionSoft service and policy corpus.
| Retrieval Metric | Score | What It Means |
| Hit Rate @ 3 | 1.0000 (100%) | The correct reference chunk appeared in the top 3 results for every query |
| Mean Reciprocal Rank (MRR) | 1.0000 | The most relevant chunk was ranked #1 across all 29 queries |
| Precision @ 3 | 1.0000 (100%) | No irrelevant context was pushed into the model’s prompt |
| Recall @ 3 | 1.0000 (100%) | All ground-truth evidence was successfully captured |
The Result
OrionSoft now has a conversational agent that businesses can actually put in front of customers. It answers from verified documents, calculates exactly, protects confidential data and turns visitors into qualified, scheduled conversations.
✓ Zero hallucination on ungrounded questions: Rule 6 abstention keeps unauthorised policy statements and incorrect pricing out of replies
✓ Sub-1.2 second responses, fast enough for a floating live-chat widget
✓ 100% routing accuracy, 100% retrieval hit rate and 100% correct refusals across the benchmark suites
✓ 100% of internal confidential terms blocked from external search
✓ Visitors verify themselves with a one-time code and book a discovery call in one click
✓ Deployable on WordPress, Webflow, React or any custom site with a single script tag (widget-loader.js) and an isolated iframe, with no stylesheet conflicts
What’s Next: The Architectural Roadmap
- Streaming token delivery: Moving from single JSON responses to Server-Sent Events (SSE), so replies start appearing sooner.
- Multi-tenant document sharding: Separate pgvector namespaces so each department can have its own isolated knowledge base.
- Async cross-encoder distillation: Compressing the reranker into a lightweight bi-encoder to scale past 100,000 indexed chunks within a 50 ms latency budget.
Conclusion
A trustworthy business chatbot is not the one that always has an answer. It is the one that knows when to answer, when to calculate and when to stay quiet. By combining hybrid retrieval, deterministic guardrails and verified lead capture, OrionSoft turned a standard chat widget into a dependable front desk that informs visitors, protects company data and sends qualified leads straight to the calendar.
Ready to deploy zero-hallucination conversational AI in your business? Schedule a Technical Architecture Discovery Call with OrionSoft Technologies.
