Reducing fabricated chatbot answers through application design

When we first enabled the website assistant, we asked a straightforward question: “Who is on the team?”

It listed six people, complete with names, titles and convincing descriptions. The actual team had four people. None of the six names was real.

Why another instruction is not enough

The instinctive response is to add: “Never invent people's names.” That can help, but it is not an application-enforced guarantee. A differently worded question can still produce an unsupported answer.

Keep this distinction clear: a model cannot accurately quote data it has not received, but it can still invent data it has not received.

The design question is therefore how the application can prevent unsupported claims from reaching the user, rather than how emphatically it can ask the model to behave.

Three complementary controls

1. Instructions

Do not invent people's names, team sizes or roles.

We retain instructions like this, while documenting their limitations. A guardrail people wrongly believe to be certain is especially dangerous.

2. Minimize sensitive context

Suppose the assistant is not allowed to quote prices. Keeping the internal price list out of its context reduces the risk of exposing that particular list. It does not stop the model inventing a plausible price.

If the product must not give quotations, the application needs to intercept that intent or return an approved fixed response. Output checks, tool access controls and retrieved-data permissions still matter.

Data minimization and fabrication prevention solve different problems.

3. Retrieve facts and validate their use

For information the assistant needs, such as approved products or staff, provide a controlled lookup rather than relying on a static prompt:

Use tool results for product and team information.
If the tool does not provide it, say that the information is unavailable.
Use list_members for team questions.

A tool declaration alone does not ensure the model will call it. It may still answer from its own generated assumptions, as happened with the six names.

A stronger design makes the lookup mandatory for that question type, validates IDs and counts, and refuses or hands off when evidence is missing. For a staff list, the application can render approved records directly instead of asking the model to rewrite names.

These controls are not a universal ladder of certainty. They address different risks and work best together.

Smaller decisions that still matter

Avoid wide tables in a narrow chat

Our chat panel is about 360 pixels wide. Multi-column tables are often awkward there without responsive layout, so we ask the assistant to use short lists. That is a product choice, not a claim that tables can never work on small screens.

Render formatting safely

An assistant may return Markdown such as bold. Leaving it untouched shows the punctuation. Some rendering approaches produce an HTML string and insert it with dangerouslySetInnerHTML; others render interface elements directly. Raw HTML insertion is not required by every Markdown library.

Treat model output as untrusted input. Visitors can influence it, including by asking the assistant to repeat material they supplied.

We use a small renderer for bold, italic, code and headings that returns UI elements rather than raw HTML. That removes the raw-HTML insertion path from this component. It does not prove the entire application is immune to XSS: links, URL protocols and extensions still need appropriate validation.

Match the reader's language

The site has Vietnamese and English versions. An originally fixed Vietnamese instruction caused English-page visitors to receive Vietnamese answers.

Instead of maintaining two diverging rule sets, keep the underlying rules in one language and add a response-language instruction:

RESPONSE LANGUAGE — security and access rules still apply
- The brief below is written in Vietnamese. Follow its rules and answer in English.
- If the visitor requests another language, use that language.

A language preference should never override access or security rules.

Before enabling a business chatbot

1. List prohibited claims or disclosures: prices, delivery commitments, sensitive data.

2. Remove unnecessary sensitive information from context.

3. Put required facts behind authorized lookup tools.

4. Enforce retrieval and validate IDs, counts and permissions in the application.

5. Render safely; validate links as well as text.

6. Test ordinary and adversarial phrasings of the questions you most worry about.

The six invented names were discovered through an ordinary question. Testing should include those, not just elaborate attacks.

The six-person incident is retained from the October 2026 Zukatek draft. Its original incident log is not available here for independent verification.

Reference: OWASP — LLM Misinformation.