Ask Rusty

The public assistant at /chatbot/. It answers questions about training, ratings, currency, tunnels and membership, for anyone from an anonymous visitor to an examiner — and what it will tell you depends on who you are.

Two pages carry the detail:

The shape of it

Every question follows the same path:

1. Resolve who is asking. The member is read from the JWT and their access is worked out from live database state — not from the token, and not from a cache.
2. Assemble what they may see. Content is queried from the CMS at that moment and filtered to their tier.
3. Answer, optionally calling tools for material too large to carry in every request.
4. Log it — the resolved tier, which tools ran, what the prompt carried, tokens, latency.

There is no ingestion step and no search index over documents. Content is read live, which is why a change in the CMS takes effect on the very next question and why retiring something removes it immediately.

Rules that hold regardless of the question

Access is resolved live. Currency and payment status change without the member signing in again. A cached access decision would keep a lapsed instructor at instructor level until it expired, so the decision is made fresh every time.

The prompt is never the gate. An instruction telling the model not to reveal something is not access control. Gating happens in three places the model cannot talk its way past: which tools are registered for the request, the WHERE clause inside each of those tools, and which content is assembled in the first place. A member is never sent material they may not see.

Out-of-scope content is omitted, not refused. If a document is above a member’s tier it is not listed, not named and not linked — it simply is not there. This differs from pages, which are always described: being told the Member Directory exists, where it lives and that it needs a coach or instructor rating is a useful answer, and far better than silence.

Unmapped means invisible. A content category with no tier mapping resolves to nobody, including administrators. A new category added in the CMS has to be mapped deliberately before its documents are surfaced — the failure direction is closed, not open.

What it will not do

• Look up other members. It cannot read anyone’s account, ratings or contact details. Asked to, it says so and points at the Member Directory if that member can use it.
• Quote a document it cannot read. Reference-material and tutorial PDFs are named and linked, never summarised from guesswork. See Content sources.
• Answer at all for a banned or deleted account. No model call is made.

Operational notes

It can be switched off without a deploy. Setting CHATBOT_DISABLED=1 withdraws the feature: every route returns a maintenance response, including the read-only history endpoints. It is deliberately all-or-nothing — suggestion chips that load and then fail on click are worse than a page that is plainly unavailable. While it is on, previously shared answer links stop resolving.

Rate limits are tighter here than elsewhere on the API, because each question costs money. Anonymous and signed-in visitors get separate allowances, keyed per member where one can be identified so a shared tunnel IP does not throttle everyone on it.

Every answer is recorded in chatbot_query_log, including refusals and denials. Historically 56% of questions went unanswered with no way to tell why; the log exists so that number can be broken down into retrieval gaps, policy refusals and out-of-scope questions rather than staying a single figure.

Reporting

Two views in the admin panel read that log. Both are restricted to super administrators, because between them they expose every question a member has typed and the whole of what the assistant is told about the most senior tiers.

Ask Rusty answers how is it doing: volume over time, failure rate by role, the questions that came back with nothing, which tools the model reached for, and what it cost. Clicking a question opens the whole conversation it belonged to, which is usually where the useful reading is — a single turn rarely explains itself.

Ask Rusty Prefix answers what does it know about this member. Pick a tier and it renders the exact material assembled for someone at that tier, alongside the access decision that produced it.

Three outcomes, not two

Reporting separates a question that failed from one that came back unanswered, and the distinction is the point. A failure means the request never produced an answer at all — a timeout, a provider error, something broken. Unanswered means the machinery worked and the assistant had nothing useful to say. The first is an engineering problem and the second is a content problem, and a single “failures” number would hide which one you have.

That separation is also why failures are logged at all. A request that threw used to return without writing a row, so outages were invisible in exactly the figures meant to reveal them.

A request that completes but produces no text is a failure too, not an unanswered question. The member is shown a short message asking them to rephrase, and the row records empty answer (stop_reason: …). Before this, such rows were written with a blank answer and no error, so they were counted as unanswered and read as a content gap when the cause was the machinery.

The assistant may call tools a limited number of rounds per question. If it reaches the limit still searching, it is made to answer from what it has already found rather than stopping empty-handed.

The prefix inspector is a safety control

It exists to make a content-visibility mistake visible. The check that matters is comparing a lapsed instructor against a current one: a lapsed instructor should be handed precisely what a flyer gets, and if that ever stops being true, instructor material is reaching someone whose currency has gone.

It builds what it shows by running the real access resolver over a synthetic member, rather than by describing what the rules ought to produce. A view that approximated the decision could show a version nobody is ever given — a control that reassures without checking anything.

History comes from shape, not text

The prefix is assembled fresh for every question, so there is no stored copy of it and deliberately so: keeping the text of each one would duplicate the CMS and still not be reproducible once the content moved on.

What is kept, on every row, is its shape — how many items and characters each section contributed. That is enough to answer the questions worth asking. When did documents drop to zero for a role, which is a gating regression. When did the prefix jump, which usually means someone added a lot of copy. And whether it is drifting toward the size ceiling, which matters because exceeding it drops content silently: the assistant simply stops being told things, with nothing in the answer to say so.

Cost is estimated here and billed elsewhere

Spend shown in the admin panel is calculated from the token counts on each row. It is a model of the bill, not the bill — the authoritative figure lives with the provider, and the two are worth reconciling occasionally rather than assuming they agree.