This is the third post in a series on preparing for Anthropic’s Claude Certified Architect – Foundations exam in the open. The first set the target and the two study tracks. The second went inside the stack and reported an honest 5 out of 10 on the first drill. This one covers the part I promised back in June: the gap between operating these tools every day and being tested on them formally, and what I did about it.
The exam is booked. Tuesday, September 15, 8:00 AM, online proctored: 60 questions in 120 minutes, and a scaled score of 720 out of 1,000 to pass. As I write this, the study artifacts are finished. What’s left is recall.
The gap had a shape
The 5/10 in July turned into a summer of drills, and the drills produced a pattern. The first six sessions, between mid-July and early August, went 5, 8, 6, 6, 9, and then, on a set built deliberately from the exam’s favorite traps, 4. The domains where I ship daily held up every time. The misses clustered in one place: prompt engineering and structured output, and the cross-cutting questions about which fix belongs to which failure. I knew a problem needed a fix; I kept reaching for the wrong layer. The drill tutor’s note after that 4/10 named it in one line, which mechanism, not whether a mechanism, and that line became the thesis of the last month. It’s exactly the gap the first post predicted: when you operate a tool fluently, you stop noticing the decisions underneath it, and the exam tests the decisions.
So the question for the last stretch was: how do you study a decision you make by reflex?
I’ll come back to the answer, but the drilling didn’t stop while I worked on it. Twenty-two tutor sessions are on the ledger now: six over the summer and sixteen in the two days of September 7 and 8. The first session on the 7th scored 3 out of 10, the lowest on the record, and it came the morning after the study artifacts were finished. By the evening of the 8th the day’s eight sessions read 9, 9, 8, 10, 9, 9, 8, 7, including the first clean sheet on a traps set. The concept I had missed most often, eight times across five weeks, whether a tool call should be forced or left to the model’s judgment, cleared on the last session of the day. The tutor keeps a per-session score log and a per-concept misses file, and the two reconcile into one ledger. This is it.
The study map
The answer we landed on was to stop studying the material and start studying the firm.
Prism runs on six repositories: the dashboard, the workspace governance files, the public chief-of-staff plugin, the configuration-management framework, this website, and the shared memory that two Claude instances sync between machines. Together they hold every real decision I’ve made about running agents in production: the hooks I kept and the one I retired, the escalation policy, the sub-agent contracts, the headless runner and its allowlist, the incident write-ups behind all of it.
On September 2, I had Claude survey all six against the exam’s 35 topics, seven per domain, and build a study map. For each topic: which Prism artifact already embodies it, in which repo, what design decision it encodes, and a one-line cue to recall it cold. The status for each topic was one of three words. Built meant a Prism artifact teaches it. Partial meant it’s touched, but a piece is missing. Gap meant nothing in Prism covers it.
The first count was 25 built, 9 partial, 1 gap.
Two things about that count mattered more than the number. First, eight of the nine partials shared one shape: Prism had the schema, the guard, or the loop in documents and tooling, but nothing that actually called the Claude API and handled a tool result in code. My whole structured-output gap, in other words, was a place where the firm had the idea and not the implementation. Second, the one true gap was writing an MCP server. I’d been consuming ten of them for months and had never published one.
That map is the most useful study artifact I’ve produced, and not because of the topics it covered. It’s because of the topics it said I was pretending to cover.
Five mornings, five builds
The map came with a plan: five days, September 2 through 6. Each morning, a 90-minute build to close a partial or a gap, on a study branch in the real codebase, under the same workflow rules as any other ticket. Each afternoon, a cold-recall block: a blank page, write the whole domain from memory, compare to the map, and study only the gaps.
Here’s what got built.
Structured output, Wednesday. The compliance chat on the Prism dashboard had always returned prose. It now returns a forced tool call with a defined schema: a required answer, required citations, a confidence enum, a required list of what it couldn’t reach, and a nullable list of what it couldn’t resolve. The response is validated against that schema, the validation errors are fed back to the model exactly once, and a second failure returns a structured error instead of a third request. Two contrasting examples went into the system prompt: one fully covered answer, one honest partial. Fourteen tests. And a live run against real benchmark data, where the model, given no relevant rules, returned confidence none, empty citations, and named the missing section instead of inventing a rule number.
That single build closed three topics I’d been missing in drills, schema design, validation-retry loops, and the agentic loop at the API level, and put a fourth, few-shot construction, into code where it had only been a note. The thing I couldn’t recall on Tuesday was something I could read in my own code by Thursday.
An MCP server, Thursday. The one gap. A small Python server that exposes the compliance rule search as a tool, the governance file as a resource, and the system prompt as a prompt, all three primitives from code that already existed unlabeled. Read-only by construction: the database opens in read-only mode and no write tool exists. An empty result is an answer. A missing database is an error that names the fix. Thursday evening, the reading on transports produced the one-line rule the map asked for: a local subprocess that borrows the client’s credentials talks over stdio; a shared service that must authenticate its callers talks over HTTP and demands OAuth.
Claude Code configuration, Friday. Two things Prism had only by reference. A path-scoped rule that loads only when files under the public folder are touched, so a backend-only session never carries it. And a read-only sub-agent, a repository auditor whose tool list is the enforcement (it cannot write because Write is not in its tools) and whose return is a machine-checked report contract. I ran it once. It came back valid on the first try, passed four of its five checks, and escalated two real findings: a claim in the project’s own instructions file that was false, and a line count that was documented at 7,270 and actually 8,402. The study exercise audited the firm.
Context and cost, Saturday. Prompt caching on the chat: the tool definition and the system prompt go to the API as one cached prefix, roughly 1,400 tokens that are byte-identical on every call, and only the retrieved context and the question are paid for in full. Beside it, a running summary: the last ten turns verbatim, everything older folded into one deterministic paragraph with no model call, labeled as data rather than instructions. The endpoint now returns its token usage, so the proof is two numbers on two consecutive calls.
One topic closed without code. The map had asked me to move the CIS benchmark extraction to the Batches API "or write down exactly why not." Reading the pipeline end to end settled it: the extraction is a PDF text pass plus a deterministic parser, the import is a spreadsheet into SQLite, and neither one calls a model. The only Claude API call in the repository has a person waiting on it. So the ledger says, in writing, that Prism has no batch candidate today and will not add one to look complete. Writing that down taught me the batch-versus-synchronous rule better than building it would have.
By Sunday evening the map read 35 built, 0 partial, 0 gap.
What the builds found
I want to be honest about something the plan didn’t predict. Every build turned up defects in production code that had nothing to do with the exam.
The structured-output live test found that the chat’s model had been retired and production had been quietly returning errors. It found a search that let archived benchmarks outvote the product a user was actually asking about. It found a rule importer that crashed on a missing parameter. The MCP server port found that the search never stripped punctuation, so a question ending in a question mark matched nothing. The sub-agent audit found two documentation claims that were wrong.
All of it got fixed, ticketed, and merged under the normal workflow. None of it was the point. But it’s a useful reminder of what an exam like this is actually for: the topics aren’t trivia. Each one is a place where a real system either has the discipline or doesn’t. I found out where mine didn’t by trying to teach it.
The last week
With the artifacts done, the last week is recall only. No new material. The plan is the one the map wrote for itself.
The misses file comes first. It’s one line per miss: the concept, what I thought, what is true. Three rows are still open, one each in three domains: when to resume a session and when to start fresh with a summary, when interacting problems have to go into one message, and the exit a fixed list of allowed answers needs so a required field never forces a made-up one. The drill tutor is instructed to quiz every open row before anything else and to clear a row only after two consecutive correct answers across sessions. Never delete a row. Two of the three had cleared once and came back, and every reopening on the ledger so far has been the same shape: an undrilled row or sub-point of a concept whose main rule I had learned. That is the last week’s actual target, and it is smaller than the pattern I started with.
Then the cold-recall pages. Two of them are still blank on purpose. Each starts with an empty page for a domain, then the seven "When … choose … because …" rules, then a compare checklist. The rule for ticking a topic off is strict: it counts as recalled only when I can name the artifact, the repo, and the decision without looking.
Then a timed mixed set: thirty questions across all five domains at exam pace, two minutes each, and sixteen cross-domain questions that only have answers if I can retrieve them without scrolling up. Why was one hook retired and another kept? What does the sub-agent contract require even when it’s an empty array, and what does that prove? Which of citations, confidence, and unresolved questions is required, which is an enum, which is nullable, and why?
The ledger also projects a score, weighting each domain’s recent accuracy by its published share of the exam. I’m not putting much weight on the number itself, since the last three sessions are a thin sample and two domains were barely asked. What I do trust is where it says the points still at stake are: more than half of them sit in the heaviest domain, agentic architecture, on the one open row there. That tells me what to drill on the morning of the 14th.
And one piece of plumbing, in keeping with how this series started: the proctor’s system test on the exam-day machine, run before the day it matters.
What I’d tell someone starting this
Study the weighting, not the syllabus. That was the first post’s lesson and it still holds.
Build the study system so it pushes back. That was the second post’s lesson: a passive artifact can’t tell you what you don’t know.
And the lesson from this stretch: when the gap is between operating a tool and being tested on it, close it by making the tool teach you. Find the places where your real systems have the idea but not the implementation, and build the implementation. The exam becomes a review of decisions you’ve now made on purpose, in code you own, with tests that prove them.
One more post comes before the exam, written from the other side of the desk: the study system’s own account of the last three months, including the parts it got wrong. Then the result, whatever it is.
Putting Agents to Work in Production?
Prism AI Analytics builds AI-run operations for small businesses, ours included. Everything in this post, the forced schema, the read-only server, the auditor that can’t write, the cached prefix, is now running in the firm’s own systems. That’s the method: take the official capability, build the tooling around it that makes it dependable, and keep a log honest enough to trust.
