How to Train a Site Assistant on Your Own Articles

Training Means Retrieval, Not a New Model

Ask AI

Once the goal is a site assistant that answers from your articles, “training” means indexing the pages you own and supplying retrieved passages as evidence at question time. It does not retrain or fine-tune the underlying model.

Fine-tuning is the wrong tool for facts. The working rule is retrieval for facts and fine-tuning for style: injecting claims by fine-tuning blends them into the model’s original training and multiplies errors, instead of pointing at a page you can correct.

Pasting website copy into a system prompt is not a substitute. That approach still hallucinated specific pricing. A retrieval crawl of service, case-study, and blog pages reduced those hallucinations sharply, though not to zero — what remained had to be managed with guardrails, not assumed away.

A custom name does not change the model underneath. Without a site-specific knowledge base, a named assistant is still a generic model with a custom name tag.

This article is the training contract only: what gets indexed, how passages are written, and what counts as citable. How an answer is refused or escalated belongs to the query-contract article; that decision tree is not restated here.

Admit Only Owned, Current Articles

Ask AI

With retrieval as the training method, admission is the next decision. Reliable training is a limited crawl, useful documents, and repeatable questions that later prove the right passage was retrieved—not a sitemap dump. The first index is pages you own, starting at crawl depth 0, and preferably articles that already cleared editorial QA.

Design that set from about two weeks of real visitor logs. The questions people type in chat are not the questions teams put on FAQ pages, so a corpus built from the FAQ misses the passages the assistant is actually asked to retrieve.

Leave out carts, tag archives, print views, translated duplicates, and more than one outdated version of the same policy. Conflicting copies make retrieval unpredictable. A focused set of roughly 30–50 strong articles typically beats about 500 poorly organized documents. A production assistant can run on a modest mix: 63 sources split into 271 chunks, including manual entries for facts that never became a page.

Apply that checklist to the next article before it is crawled. The content-automation workflow remains the QA gate; this list only decides whether a cleared page may enter the index.

Write Each Section So a Retrieved Chunk Can Stand Alone

Ask AI

Getting a page into the index is only half the job. Retrieval cites a chunk, not the article around it. A passage that says “as noted above,” or that needs a price, exception, or definition living under another heading, is not citable.

Keep one topic per section. Lead with the fact, name the product or policy, state the condition, and leave the supporting detail in that same section. A product rule, price condition, or process has to travel with its context so the retrieved passage can answer on its own.

Labels do as much as the prose. Metadata enrichment alone raised RAG precision from 73.3% to 82.5%, with no change to the retriever, embedding model, or chunking, and at sub-30ms P95 latency. Put the title, canonical URL, updated date, and section label on every indexed chunk. That is the on-page structure retrieval needs—nothing more.

Why stale and contradictory pages stay out of the index

Ask AI

A clean section label still fails if the page is outdated, duplicated, or at odds with another page you published. Source quality dominates model choice. A stronger model does not fix missing facts or contradictions.

That readiness gap is why the hold exists, not a reason to shop for a larger model. The same retrieval stack that fabricated answers on 52% of questions over unvetted data fell to near zero once it was limited to curated content. Stale evidence is worse than a gap: it flipped 30% of Llama and 37% of Qwen answers that were correct without retrieval, even with no instruction to trust the document, and rose to 66% and 75% when the model was told to follow it. Across models and domains, poisoning ranged from 17% to 91%, while matched current evidence was followed in 97–100% of trials. A recency-aware reranker helped only 4.6–10.0 points when dates were accurate, so a real date is not a substitute for removing the stale page.

What to hold, and what never to quote

• Hold the page if it is outdated, duplicated, or disagrees with another published page.
• Do not index both versions and hope the model picks.
• Bar the assistant from quoting a specific dollar amount the current indexed page does not state.

An unsanitized knowledge-base price was quoted to a visitor and set the wrong expectation on a later sales call. When an article changes, re-read and re-index that page on a set cadence. Quality depends on clean source content, a sensible scope, and realistic tests—not on training a new model for a content edit.

Prove Retrieval Before You Widen the Corpus

Ask AI

The hold only works if the assistant cannot paper over a missing page with a made-up link. Retrieval decides what the model may know. Never ask it to write arbitrary source URLs. Every citation is an allowlist of known chunk IDs that already point at indexed canonical URLs.

That allowlist is non-negotiable. Across 16 million unique URLs, major AI assistants sent visitors to 404s 2.87 times more often than Google Search. ChatGPT’s clicked-URL 404 rate was 1.01% versus a 0.15% Google baseline, and 2.38% of ChatGPT-cited URLs checked against a crawler database returned a 404—the highest rate measured.

Prove the index before you add pages. Keep a fixed set of 10 to 20 real visitor questions, each paired with the expected source fact, plus unique facts that exist on only one page. Rerun that set after a content edit or a model change. A miss means the article is missing, stale, or contradictory—not that you need a larger model. Stop at the hold. The unanswered-question-to-brief loop belongs to the sibling article; link it, and do not rebuild it here.