AI support automation with a knowledge base and RAG

Where automation helps

The first line suits repeatable questions: service status, tariff rules, integration setup, how to contact support. Refunds, account blocking and personal data need a separate route, and so do unclear topics, contradicting sources, complaints and repeated questions: those end in a handoff to an operator with the ticket and its context.

Start with the knowledge base, not the prompt

Write down what the support team is allowed to answer. Every document needs an owner, a scope, an effective date and a replacement rule. Delete or archive pages that conflict with current policy.

Chunk documents around a single procedure or decision, and keep the source heading with each chunk. A short paragraph with its conditions is easier to verify than a page fragment with no context, and a page on twenty topics ruins search. Archive the old version when a document changes, so an answer can be restored by date.

Before generation, normalize the question without changing its meaning, find passages by topic and product metadata, and drop duplicates and expired material. Afterwards check every claim against the retrieved text, and if one is unsupported, escalate with the question and the sources used.

A retrieval flow for Telegram and web chat

Use the same support service behind both channels. The Telegram adapter and the web widget should pass a normalized question, user context and conversation ID. They should not implement separate policy logic.

channel update
    -> normalize question and permissions
    -> retrieve approved passages
    -> answer policy and model route
    -> source check and confidence rule
    -> reply, ask for detail, or hand off

The model should receive the question, the selected passages, and a clear instruction to say when the passages do not answer it. The retrieval layer should return source IDs so your UI can show where an answer came from. Five layers keep the answer safe:

Model routes to evaluate

The live catalog checked on 2026-10-03 contains these candidate IDs. The rows describe test roles, not benchmark results or guarantees.

Choose a route by verified fixture results, and keep it in configuration so a catalog change cannot silently alter support answers.

For a trial run, take closed tickets from one period and mark the correct answer, an acceptable escalation and a dangerous invention. Start with drafts for operators, and send simple answers automatically only after manual review.

Return structured support data

Ask the model for fields your application can validate. A response should carry the answer, source IDs, an escalation flag and a short reason. Reject malformed output instead of repairing it into an action.

type SupportAnswer = {
  answer: string;
  sourceIds: string[];
  needsHuman: boolean;
  handoffReason: string | null;
};

function canSend(answer: SupportAnswer, sourceIds: Set<string>) {
  const knownSources = answer.sourceIds.every((id) => sourceIds.has(id));
  return knownSources && (answer.needsHuman || answer.sourceIds.length > 0);
}

The application still checks permissions, account state and policy scope. A source ID is evidence for a draft answer. It is not permission to perform a refund or change an account.

Freshness and conflict handling

When a policy changes, use an update event or a bounded refresh job, and store the version that produced each answer. If two documents conflict, stop the automatic answer and send the conflict to the document owners.

Track these fields for each response:

• question and channel ID, with personal data removed where possible;
• model ID, knowledge base version, source IDs and policy version;
• elapsed time, retrieval result count, parser status and escalation reason;
• the agent's decision when a human reviews the draft.

Keep retention short enough for your support policy. Transcripts and source documents need the same permissions as the support console.

An evaluation set that reflects real tickets

Build a small fixture set from resolved tickets after removing personal data. Include a correct answer, a missing-policy case, two conflicting policies, a multilingual question, a prompt injection, an account-specific request and an explicit request for a human.

For each fixture, check factual support, source IDs, refusal behavior, schema validity and escalation. Record false answers and false handoffs separately: one pass rate hides the failure your team cares about most.

Further reading: RAG with Claude, function calling, the LLM provider guide for customer support and the model selection guide. The Telegram support bot provider guide covers webhook idempotency, queues and provider errors.

FAQ

Does retrieval remove hallucinations from support automation?

No. Retrieval gives the model source material, but the application still needs source checks, refusal rules, tests and human escalation. Require a citation for every claim, forbid unsupported answers and check statements with a separate validator.

How many passages should I send to a model?

Start with the smallest set that answers the question and test it on your fixtures. Extra text can hide the condition that matters.

Do I need a separate model for retrieval?

Not always. Start with one process and measure the retrieved passages. If search misses, fix indexing, metadata and chunking first.

Should internal runbooks be visible to customers?

Keep audience metadata on every document and filter it before generation. A support agent can see an internal runbook while a customer receives a safe summary.

Can the bot perform account actions?

Only through explicit tools with permission checks, validation, idempotency and an audit record. A retrieved passage or a model answer cannot authorize an action.

How do I keep policies current?

Give every source an owner, an effective date, an expiry or review date, and a replacement rule. Pull an outdated instruction from the active index but keep it in the archive. Send conflicting or expired material to a person instead of guessing.