Ask Rusty rebuilt on Claude, with role-gated content
Shipped 2026-08-19
OpenAI’s Assistants API shuts down on 26 August 2026, and Ask Rusty was built entirely on it. This is the replacement: Claude Sonnet 5, answering from content read live out of the database rather than from a hardcoded list of 117 S3 URLs.
The original plan called for a vector pipeline — extract the PDFs, chunk, embed, search by nearest neighbour. That is deferred. Re-reading 4,271 logged questions showed the failures were never retrieval failures: ~840 directory lookups (79% unanswered), 265 navigation questions (63.8%), 92 asking what the assistant can even do (82.6%). None of those are fixed by a better document index. They are fixed by tools, a site map, and refusals that say something useful.
Content is read live, so the CMS is the source of truth
FAQs in full, the skills index, tunnels, the reference-document catalogue, news headlines and the site map are assembled per request. Nothing is ingested.
Two consequences worth knowing:
- Adding or retiring content needs no deploy. An FAQ added in the CMS is answerable on the next question.
- Setting content to
closedremoves it immediately, not on the next ingest run. There is no index to fall out of date.
Content too large to carry in every request sits behind tools instead —
search_skills, get_skill (which also returns a skill’s videos and its
Flight Tutorial PDFs), and search_news.
One access resolver, gating three surfaces
Every source is filtered through a single decision about who is asking. The
ladder is public < flyer < coach < instructor < trainer < examiner < manager < admin, and it is cumulative — an instructor sees instructor, coach, flyer and
public material.
It gates which tools are registered, the WHERE clause inside each
tool, and which catalogue entries exist at all. A member is never offered
a document they cannot open, and a tool they may not use is not merely
discouraged — it is never sent to the model, so it cannot be called.
Three behaviours that were wrong before:
- Banned and deleted accounts fell through to the public corpus, because the old role map had no case for them and treated forbidden the same as unrecognised. They now receive no answer at all, and the refusal is logged.
- Lapsed currency downgrades access. An instructor whose currency has lapsed is treated as a flyer until recurrent. A confirmed coach keeps coach access, because that rating is earned separately.
- Access is resolved live, never cached. Currency and payment change without the member signing in again, so a cached decision would keep a lapsed instructor current until it expired.
The old answer cache is gone
chatbot_history was quietly serving stored answers keyed on the question text
and role id, before any retrieval ran — 1,857 reusable rows, 382 of them
keyed to instructor tier and above, with no expiry and answers re-served up to
571 days after they were written.
It bypassed access control completely: a lapsed instructor matching one of those rows received the instructor-tier answer without the resolver ever running, and a document retired months ago was still quoted verbatim.
That path is removed. Both legacy tables are now frozen — read-only, never written again — which keeps pre-cutover share links resolving while making it impossible for a stale answer to be served.
The 56% is now measurable
Every answer writes a row recording the resolved role, which tools ran, what the prompt actually carried, token counts and latency. Refusals and denials are recorded too. For the first time the unanswered rate can be decomposed into retrieval gaps, policy refusals and out-of-scope questions rather than being a single number.
Known trade-off
Reference-material and tutorial bodies are not readable in this version — roughly 3.9M of a 5.03M-character corpus. On deep instructor questions this is a regression against the old system, which searched them.
The mitigation is signposting: the assistant names the document that answers the question and links it, rather than guessing at contents it cannot read. Measured against 444 historical questions the old engine answered while citing a PDF, a sample of 8 produced 8 substantive answers, 4 naming or linking the source document, and no silent failures — which was the outcome most feared. Document bodies return in v2.