Banking Chat Widget — example built on the Amex KB

Why this exists

The askthe assignment

A pre-screen assignment for a Technical Product Owner interview with American Express Digital Banking: define the product strategy and phased roadmap for an enterprise knowledge-base chatbot — problem, users, priorities, trade-offs, and success metrics.

Why we built it livethe thesis

A strategy document can claim anything. This page proves one claim instead: that a large share of general servicing questions can be answered from public knowledge alone — no core-system integration — as long as the assistant is grounded, cited, and honest when it isn't sure. So it runs live. The widget in the corner answers real banking questions from the real public help center, and shows its sources every time. (Snapshot indexed July 18, 2026; every citation links to the current live page.)

Two bots, two lessonswhat to try

The banking widget (bottom right) — ask a real question, click a citation chip to check it against the live source, then ask something off-topic and watch it decline instead of guessing.
The "Ask about Udit" widget (bottom left) — a separate assistant that answers questions about the builder — or how this site was built — and turns banking questions away.

That same Udit knowledge also powers "Ava," a packaged voice agent that screens recruiter calls on his résumé line — one database, two channels, two different build decisions (see the story below).

The assignment, answered

The pre-screen posed six questions. Each is answered here in brief — and, wherever possible, proven live elsewhere on this page — with the full write-up in the accompanying strategy document, which covers all six in depth.

1 · Problem & userswho hurts, and how

Knowledge exists — in help centers, wikis, policy docs — but answers don't arrive fast enough.

Primary usersFrontline servicing agents mid-interaction · operations teams executing procedures · new hires (the heaviest question-askers).

Secondary usersKnowledge managers and content owners · compliance-risk partners.

The costLonger handle times · inconsistent customer answers · slow onboarding · repeated escalations.

Key challengesTrust (one confidently wrong answer kills adoption) · coverage vs. quality · freshness · no learning loop.

2 · Product strategytrusted answers at the speed of conversation

The product's job is not to answer every question — it's to be right when it answers, honest when it isn't sure, and measurably better every week. Search returns documents and makes the user do the reading; basic chatbots answer fluently but unaccountably — this product closes that gap.

Three pillars:

Grounded, cited, or silent — proven by the widget in the corner.
The feedback loop is the product — the rating ask.
Guardrails as a feature, not a constraint — the topic firewall.

3 · Roadmap, phasedtrust → coverage → scale

Four phases, gated in order.

Phase 0 · Agent-assist on-rampPut the grounded assistant in front of care professionals first — proof on real traffic, zero customer-facing risk.

Phase 1 · Trust FoundationGrounding, citations, an evaluation harness, confidence fallback, rating capture, and the sensitive-topic firewall (this demo is a running Phase-1 instance).

Phase 2 · Coverage & FreshnessEvaluation-gated source onboarding, freshness SLAs, a knowledge-owner dashboard, and gap-mining.

Phase 3 · Scale & ActionAuthenticated, account-aware answers under full model-risk validation, action-taking on top intents, and more channels.

The rule: trust before coverage, coverage before scale.

4 · Top-5 prioritiesand the criteria

Grounded answers with citations
The evaluation harness
The confidence fallback
The feedback loop
The freshness pipeline

Criteria: impact on user trust (the adoption currency) · reach · risk reduction · effort · dependency order — measure before optimizing, trust before scale.

5 · Trade-offseach with a mechanism, not a vibe

Accuracy vs. speedCache the frequent head, stream everything, spend retrieval depth on the long tail — correct-in-6s beats wrong-in-2s.

Coverage vs. noiseA source ships only if it raises answer quality on the golden set, and retrieval weights source authority and recency — a definitive policy document outranks a five-year-old slide deck (applied for real here: 164 pages crawled, 80 shipped).

Delivery vs. reliabilityAnswer quality is an SLO with an error budget — when the budget burns, capacity pivots to quality; new capabilities roll out canary-first with automatic rollback on rating drops.

6 · Success metricswhat matters most

North starSelf-serve resolution rate (agent-assist deflection in Phase 0, evolving to customer self-serve resolution as the assistant goes customer-facing).

LeadingGroundedness on an audited sample (the trust number that gates releases) · retrieval hit rate (isolates "can't find it" from "can't say it") · rating scores · repeat usage.

LaggingEscalation deflection · time-to-answer vs. baseline.

GuardrailsZero sensitive-topic violations · freshness SLA · p95 latency.

The two that matter most: groundedness (leading trust) and self-serve resolution (lagging proof of value).

📄 Read the full strategy — PDF · 6 sections · 4 pages

How it works

The full answer path, in four steps — retrieve, gate, generate, learn.

Retrieve: your question is scored against 80 indexed help articles by keyword relevance; the top passages become the only grounding the model may use.
Gate: weak retrieval → confidence fallback (sources only, no generation). Sensitive topic → firewall routing. Both fire in code.
Generate: the model answers strictly from the retrieved passages and must cite them — plain text, tight length, low temperature.
Learn: every conversation ends with a rating ask; in production this stream drives content gap-mining, retrieval tuning, and the roadmap.

Try these:

See a grounded answer

Dispute a charge Fund a checking account Transfer MR points CD maturity Travel notice

Watch it refuse

Off-topic (watch the fallback) Complaint (watch the firewall)

The second bot

Ask the other bot about Udit How was this site built?

Buy for the Udit Résumé chat. Build for Amex.

Udit has shipped this decision twice, in opposite directions — and that's the point. His own résumé line runs "Ava": an AI screening agent answering recruiter calls about his background, grounded in a GoHighLevel knowledge base — a packaged product, bought. This banking widget is custom-built. Same builder, same discipline, two different right answers.

How the Amex call was made — evidence, not taste. Before building, the packaged option was probed live for this brief:

its knowledge base has no content-export API — the export endpoints simply do not exist
its widget shows citations to operators but never to customers
its retrieval is a fixed pipeline that can't be tuned

Those probe findings decided it, not a preference: a structured four-lens evaluation — interviewer impact, technical risk, compliance optics, strategy narrative — weighed both paths against that evidence and landed on build.

Buy when chat is a commodity feature — as it is for the Udit Résumé assistant. Build when trust mechanics are the product — as they are in regulated banking. Knowing which is which is the job.

Udit Résumé "Ava" — the right call was BUY (GHL KB)

✓Live in an afternoon on the same platform that already runs the phone line, SMS, and CRM — zero integration work
✓Voice-native: call routing, ring-first-then-answer, email handoff come with the platform, not built from scratch
✓Low-stakes domain: a miss means a recruiter emails instead — prompt-level guardrails are genuinely enough
✓Nobody needs citations read aloud on a phone call — the packaged retrieval quality is fit for purpose
✕The trade accepted: knowledge is locked in (no export API), retrieval untunable, behavior prompt-deep only

This Amex widget — the right call was BUILD

✓Citations rendered to the user, linking to the live source page — visible, verifiable trust is the whole thesis
✓Confidence gate + topic firewall enforced in code before any model call — what a bank's compliance team can audit
✓Database is one inspectable JSON file we own — swappable, versionable, dateable (snapshot: July 18, 2026)
✓Retrieval is tunable — the 20-question QA below went 17/20 → 20/20 after retrieval tuning, with a held-out eval set as the next gate; impossible on the packaged stack
✕The trade accepted: more to build and test — the QA table below is the honest price of control

System architecture — answer path

Every question passes through deterministic control layers before any model is called. The trust mechanics fire in code, not in a prompt.

Runtime request flow (Cloudflare Pages Function)

User questionchat widget · 500-char cap · last 6 turns of history

1 · Topic firewalldeterministic rule classes: legal · complaint · PII · financial advice — pattern-matched in code, fully auditable

2 · Lexical retrievalIDF-weighted token + title + alias scoring over 80 indexed articles → top-4 passages

3 · Confidence gatebest-score below floor → no generation at all

4 · Grounded LLManswers ONLY from retrieved passages · must cite [KB-###] · plain text, max 3 sentences · temp 0.2 · dual-provider chain → built-in cached-answer layer if both fail

5 · Answer + receiptsinline citations + chips linking to the live source page

6 · Rate & learn"Did that answer your question?" → in-widget 1–5 star rating · in production this stream feeds the quality dashboard + content gap-mining

⚠ Firewall hit (step 1)No generation. The topic is named and routed to a human channel — in production, straight into the right servicing queue with context attached.

◦ Weak retrieval (step 3)No generation. The assistant says it isn't confident and returns the closest source articles — worst case becomes a search result, never a wrong answer.

✓ Strong retrieval (steps 4–5)Generation is allowed but caged: only the retrieved passages exist as far as the model is concerned, and every claim carries a clickable citation.

Build pipeline — how the knowledge base was made · 164 crawled → 80 curated · 0 failures · every drop recorded

1 · Public help centerthe main customer-service section + the separate checking & savings FAQ centers

2 · Map + crawlFirecrawl · 164 pages · 2s pacing · 0 failures · related-question links followed one hop

3 · Cleanstrip identical footer boilerplate · remove raw links from body text · drop test/legal/niche pages

4 · Curate164 → 80 by topic quota (coverage vs noise, applied for real) · every entry keeps its canonical source URL

5 · kb.json80 articles · median 668 chars · stats file records what was dropped and why

checking 11savings/CD 12card mgmt 12payments/AutoPay 10rewards 10account/security 8disputes 5statements 5travel/foreign 4gift cards 3

The process at a glance

How it started, the decisions taken, the testing done — including the AI evaluation council that weighed both paths on live probe evidence and settled the build-vs-buy call.

It started with a thesis Pre-screen brief: strategy for an enterprise KB chatbot. Decision: don't just write it — run it live. assignment → working demo

AI agent council 4 live recon probes + 4 judging lenses + synthesis. Findings: no export API in the packaged product, widget hides citations, help center crawls clean. probe evidence → build

Decisions locked Build for Amex (buy already proven right for the Udit Résumé assistant Ava)
database scoped to 80 curated articles incl. banking
neutral unofficial skin
healthy LLM provider primary. evidence over taste

Built + tested Database crawled (164 pages, 0 failures), serving layer shipped, 7 automated checks passed — all 3 answer modes verified. same-day build

Human UAT + retrieval QA Owner UAT found 2 real defects (fixed same session). A 20-question retrieval QA then went 17/20 → tuned → 20/20. found · fixed · re-verified

The build, step by step — the 5 nodes above, expanded

1 · Frame the assignment

Pre-screen brief: product strategy + roadmap for an enterprise KB chatbot. Thesis to prove: a large share of general servicing questions can be answered from public knowledge alone — no core-system integration — if the assistant is grounded, cited, and honest about uncertainty. Decision: don't just write that; run it live.

2 · The AI agent council

The build-vs-buy call was made by a structured multi-agent evaluation, not a hunch: four fact-finding probes ran live (API probes of the packaged product's knowledge-base endpoints, crawlability tests against the help center, panel/positioning research, crawler-capability research), feeding four independent judging lenses — interviewer impact, technical risk, compliance optics, strategy narrative — and a synthesis chair. Full story in Build vs. buy.

four evaluation lensesevidence → builddecisive: no export API · hidden citations

3 · Build the database

Mapped the help center (the obvious path variant returns zero — found the correct one), crawled politely at 2 pages/sec, followed related-question links, and discovered the checking/savings content lives in two separate FAQ centers the main map never reaches — critical, since the audience is a digital-banking team. Curated 164 pages down to 80 by topic quota with drop-reasons recorded.

164 scraped0 failures80 curated23 banking articles

4 · Build the serving layer

Single Cloudflare Pages Function: deterministic firewall → lexical retrieval → confidence gate → grounded generation with mandatory citations → a rating ask that closes every conversation.

Dual LLM provider with the healthy one primary — the fallback chain was observed firing for real, not assumed.
Static page, no build step, one JSON database file — every moving part is inspectable.

The second, separately-scoped assistant (bottom left) reuses the same pattern on the same public bio that powers the Udit Résumé packaged GHL voice agent — one database, a custom web channel and a bought voice channel side by side.

5 · Verify — automated, then human

AutomatedAPI-level tests of all three answer modes, end-to-end UI verification, citation-link spot checks, provider-failure kill test.

Human UATFound real defects the automated checks missed — verbose markdown answers and URL overflow — fixed same-session.

Retrieval QA20 questions across all 10 topics: 17/20 on first run; the 3 misses (paraphrases with no lexical bridge) drove IDF term-weighting + curated aliases; re-run 20/20 with fallback and firewall re-verified.

3 modes testedUAT: 2 defects → fixedretrieval QA 17/20 → 20/20

6 · Ship behind gates

Deploy to production is human-gated (owner-approved release window), preceded by an offline-resilience layer, owner UAT sign-off, and a full dress rehearsal on the real meeting software. Code freeze after rehearsal.

Release log — 9 versions shipped in one day · July 18, 2026

v0.1Working prototype: full trust pipeline (retrieval → confidence gate → cited generation → firewall) on a fictional-bank skin

v0.2Real Amex public help-center database swapped in (164 pages crawled → 80 curated) + unofficial-demo reskin

v0.3Process, architecture diagrams, and QA table documented on-site

v0.4Owner UAT round 1: single-page merge, plain-text answer contract, URL-overflow fixes

v0.5Build-vs-buy story, agent-council journey chart, design-system pass

v0.620-question retrieval QA: 17/20 → IDF weighting + aliases → 20/20

v0.7Second bot (Udit Résumé Chat), star-rating flow, click-outside collapse, /home demos hub

v0.8Buy-for-résumé / build-for-Amex reframe, two-bots hero, database snapshot dating

v0.9Assignment-answers section, release log, terminology + copy sweep (owner UAT round 3)

QA process

Answer quality is the release gate: nothing ships on "it looked fine." Every PASS below ran on July 18, 2026, against the build in the release log above. The SCHEDULED rows are the gates still ahead of deploy. Each row is a real check with a real result — including human UAT, which caught what automation didn't.

12 quality checks10 PASS2 SCHEDULED pre-deploy gates

Check	Method	Result
Database integrity	Schema validation of all 80 entries: ids, titles, canonical source URLs, length caps; stats file cross-check	PASS — 80/80 valid, 0 non-source URLs
Crawl honesty	Per-page success log; failures counted, not hidden; drop-reasons recorded for every curation cut	PASS — 164/164 pages scraped; curated to 80 articles, every cut recorded with a reason
Grounded mode	API-level tests: disputes, checking funding, CD maturity, rewards transfers — answers checked against source passages, citation ids verified	PASS — correct content, correct cites
Fallback mode	Deliberate out-of-database question — must decline to generate and return sources	PASS — declined, sources shown
Firewall mode	Sensitive prompts (regulator complaint, legal, PII) — must route to human with zero generation	PASS — routed, no model call made
Citation links	UI spot-check: citation chips resolve to the live source article pages	PASS — verified in-browser
Provider failure	Kill test of the LLM chain — with one provider forced to fail, the request fell through to the backup provider in production-identical code (July 18).	PASS — fall-through observed firingThe backup provider is being provisioned before deploy; the offline answer layer below covers a total outage of both.
Human UAT — exploratory	Owner free-form testing on the live build, no script: judge answer quality, tone, layout, trust signals as a real user would	PASS w/ findings — verbose markdown answers + URL overflow found → fixed → re-verified
Retrieval QA at scale	20 paraphrased questions across all 10 topics: grounded mode + expected citation in top-4 + plain text + length cap, asserted by script (qa/retrieval_qa.py)	PASS — 17/20 first run → IDF weighting + aliases → 20/20 on the tuned set; a held-out golden set is the Phase-1 evaluation-harness gate. Fallback + firewall re-verified
Offline resilience	Cached answers for the scripted demo questions are built into the page and served automatically if the API fails; backup screen recording lands at Monday's rehearsal	PASS — cached layer live (recording: Monday)
Human UAT — sign-off	Owner accepts the deployed build end-to-end before it's shown to anyone; deploy itself is human-gated	SCHEDULED — release gate
Dress rehearsal	Full run over the real meeting software on the deployed site; code freeze after	SCHEDULED — day before