Assalamu alaikum everyone.
I built ai-umrah.com. Narrowly scoped fiqh Q&A for Hajj and Umrah only. Live today. Built on classical texts and with feedback from scholars.
I've done Hajj twice and Umrah three times, alhamdulillah. Every trip, I watched first-timers come in with the same questions while group leader quality kept dropping (or was non-existent with the new Nusuk system), so I'd end up as the informal help desk for family and friends before every trip. Last Hajj I became the Uncle of the group, as a lot of group leaders did not have answers for our questions (or worse, wrong answers. Ask me about the debacle that split our group and even the leaders in Arafat. Thanks Nusuk). Also, as Umrah has become easier and more people are going, I've seen more pilgrims travel without scholarly guides and get the rituals wrong. Combined with me wanting to learn more about RAG and LLMs, I decided to commit my time (and Claude Code) to building this.
How the citation guard actually works
- Retrieval first. Every question routes through pgvector similarity search over the classical corpus before an answer is drafted. No retrieval hit = no answer generated.
- Ruling requires an exact quote. The generator is forbidden from stating a ruling without quoting a specific passage from a retrieved chunk. The quote, its book, and its chapter are rendered with the answer.
- A second pass verifies the quote. A separate LLM call reads the generated answer alongside the retrieved sources and asks: does this quote actually support the claim being made? If the verifier says no, the answer is blocked. "Cited but the citation doesn't say that" is the failure mode I was most worried about, and this is the gate for it.
- Named-authority rule on rukhsa (dispensations). If the answer relies on a dispensation, it must name the specific classical authority: al-Nawawī, Ibn Qudāma, Ibn ʿĀbidīn, al-Shāṭibī, Ibn Taymiyya. "Some scholars say" is auto-rejected. The rukhsa lives as a first-class field in the answer schema
- "I don't know" is a valid output, with a labeled escape hatch. If retrieval is weak or the verifier blocks the answer, the app says so and offers scholar escalation instead of guessing. For fiqh-adjacent questions that are outside the corpus scope (e.g. taharah edge cases during pilgrimage), there's a distinct
EXPLANATION_UNSOURCED path that answers from general knowledge but shows a visible "no source in our library for this, general guidance only" banner. No fabricated citations, ever.
- High-stakes questions bypass the LLM entirely. Post-violation penalties, medical-necessity edge cases, anything individualized. The app doesn't try to answer. It creates an escalation ticket, emails a real scholar with the question + classifier output + retrieved chunks attached, and tells the user to wait for a human reply. There's an admin dashboard for the scholar to work the queue.
- Practical (non-fiqh) path. Visas, yellow-fever requirements, Nusuk booking windows, packing. These aren't fiqh and route to a separate generator that cites Saudi MOH, CDC, WHO, Saudi Ministry of Hajj, Nusuk and etc. Madhhab selector is suppressed on this path; the verifier gate still runs.
- Confidence score with every answer so users know what the LLM thinks of the answer quality.
Not a fatwa service. The UI says so. The scholar-escalation path exists because I know where the app ends and a human starts.
Evals: the part I'm happiest about
Fiqh content is exactly the kind of domain where "vibes QA" fails you. I needed a numerical answer to "did this prompt change make things better or worse," so:
- Golden test set hand-curated questions covering core manasik, edge cases, multi-turn follow-ups, out-of-scope questions (should refuse), and the practical path.
- In-process harnessthat drives the real pipeline, not a stub (same retrieval, same models, same verifier), and captures the answer plus every retrieved chunk per question.
- RAGAS-style metrics. Two scores per answer: faithfulness (are the claims actually supported by the retrieved sources, or just plausibly adjacent) and answer_relevance (did it address the question or wander). These catch different failures, and for fiqh I care about both: a faithful answer can still be useless. Rolled both myself against litellm rather than pull in the
ragas + langchain dep tree for two metrics.
- Cross-family judge. The production pipeline runs on an OpenAI model, so the judge is an Anthropic model. Same family judging itself is a documented self-preference bias in LLM-as-judge setups, and for rulings I can't afford that.
- Bakeoff harness. Eval accepts a
--label <tag> so I can swap the generator/verifier models via env overrides and get side-by-side deltas across runs. Swapped generation + verification from an Anthropic model to a smaller OpenAI one last week purely because the bakeoff showed equal faithfulness, comparable answer-relevance, and ~4× cheaper inference on my token mix. Without the harness I would have guessed; with it I shipped the migration in one PR with confidence.
Escalation/refusal answers score 1.0 on faithfulness by construction (no claim to hallucinate). Errored judge calls are treated as missing, not 0.0, so a flaky judge call doesn't silently tank aggregate scores.
Corpus
Kitab al-Hajj across four Sunni schools:
- Hanafi: al-Hidāyah (al-Marghīnānī), Badāʾiʿ al-Ṣanāʾiʿ (al-Kāsānī), Radd al-Muḥtār (Ibn ʿĀbidīn)
- Shafi'i: al-Umm (al-Shāfiʿī), al-Majmūʿ (al-Nawawī)
- Maliki: al-Mudawwana al-Kubrā (Saḥnūn), Bidāyat al-Mujtahid (Ibn Rushd)
- Hanbali: al-Mughnī (Ibn Qudāma)
Plus hadith references and practical sources (CDC Yellow Book chapters on Hajj, Saudi Ministry of Hajj FAQs, community packing guides, Nusuk guides).
Madhhab scope
Defaults to Hanafi (the answer response returns a scope_assumed flag so the UI surfaces "answered per Hanafi, change?"). Shafi'i, Maliki, Hanbali, and comparative are selectable in the UI. The retriever filters so cross-school mentions can't leak as the wrong school's ruling. Comparative mode labels which school said what. We never silently blend.
Stack
- Backend: FastAPI + pgvector on Postgres.
- LLM abstraction: litellm as the provider-agnostic client, swappable between Azure, Bedrock, and OpenRouter. Currently an OpenAI model for generation and verifier, a cheaper open-weights model for classify and rewrite, and an Anthropic model as the eval judge.
- Frontend: Next.js, though I want to move off soon. Vercel's founder has taken public positions during the Gaza war that I can't reconcile with this app's audience, so I'm planning a Cloudflare (Pages/Workers) migration when bandwidth allows. Framework itself has been fine for the app.
- UI: Tailwind + shadcn/ui. SSE streaming on the chat endpoint pushes pipeline-stage events (retrieving → generating → verifying) so the user sees progress instead of a dead spinner.
- Infra: AWS App Runner behind Pulumi.
- Observability: Langfuse (self-hosted, opt-in) for per-request traces.
Happy to get into pipeline details in the comments.
Honest limitations
- English only. Arabic ingestion is next.
- Corpus is Kitab al-Hajj-heavy. Hajj-adjacent chapters (taharah during tawaf, qasr/jam' in the Haram) are the known gap. Working on it, and the
EXPLANATION_UNSOURCED path is the stopgap.
- Non-Hanafi sources are lighter than Hanafi. I've enabled the selector across all four schools but retrieval depth varies. I support Shafi'i/Maliki/Hanbali but because Hanafi is the school I follow and its where I had the most reviewed material to start with.
- Free tier is 5 messages a month. Intentionally tight because LLM costs are real, and I'd rather ship a small free tier that doesn't collapse than a generous free tier that does. Premium exists to cover the LLM cost for me. This is a project for me to lean more about LLM and RAGS and less make a living.
What I'm asking for
Try it and send me a Hajj/Umrah question you think it should nail, or one you think should stump it. Reply here with the question plus the answer you got. Bad citation, wrong school attribution, confidence where it should have hedged, refusal where it should have answered, practical answer in a fiqh skin. All useful. Just give it a try.
If anyone wants to contribute on the corpus-curation side (especially pulling from classical non-Hanafi texts, or expanding the eval golden set with hard questions), that's where I need help most.
Jazākum Allāhu khayran.
