Last month’s post opened this series: I’m preparing for Anthropic’s Claude Certified Architect – Foundations exam, in the open, before I know the result. I promised the next post would go inside the study stack itself — how the pieces get built, where they earn their place, and where they don’t.
Since that post, the stack grew a whole new layer. And two nights ago the newest piece handed me a score of 5 out of 10 and a named list of exactly what I don’t know. That number is the most useful thing my preparation has produced so far, and explaining why is really the story of the whole stack.
Layer one: the artifacts
The first post described the NotebookLM system: audio overviews I listen to away from the desk, one-page infographics that compress a domain, flashcards and quizzes for repetition. A month in, I can report where those artifacts earn their keep.
The audio is genuinely good at second passes — a domain I’ve already studied gets replayed on a walk, and things settle. The infographics are good as a gap test in one specific way: if I can’t redraw the one-pager from memory, I don’t know the domain yet.
But a month of use exposed the ceiling: passive artifacts can’t tell you what you don’t know. They test recognition — "yes, I’ve seen this" — and an architecture exam tests production under pressure: given a scenario, produce the right decision and defend it against three plausible wrong ones. I could listen to the audio all summer and never find out that two entire domains hadn’t actually stuck. Something in the stack had to push back.
Layer two: a wiki an agent maintains
So the stack grew a second layer: a study wiki that an agent maintains and I read.
The pattern is adapted from Andrej Karpathy’s llm-wiki idea, and the division of labor is strict. I curate raw source material into one folder the agent may read but never touch. The agent compiles it into a clean, interlinked wiki — sixty-two pages now — and keeps a change log of everything it does. The schema file that governs all this is written for the agent, not for me: it defines the page types, the linking rules, and the hygiene passes it runs on its own work.
Two design choices matter more than the rest.
First, the wiki is organized the way the exam is weighted, not the way the material is published. Five folders, one per exam domain, with the weights on the label — agentic architecture carries 27 percent, context management 15. Every page lives where the exam says it lives. Studying for the weighting instead of the syllabus was the first post’s small lesson; the wiki hard-codes it.
Second, the wiki stores the traps, not just the truths. Alongside concept pages there are comparison pages — resume versus fork, schema validation versus semantic validation — and a catalog bluntly titled "never right answers." Certification distractors aren’t random; they’re the plausible-but-wrong neighbors of the right answer. The wiki treats those neighbors as first-class content.
The agent also audits its own work, and the audit has caught real problems — including a domain hub that had silently omitted an entire task area of the exam. A study system with a lint pass sounds like overkill until the lint pass finds the hole that would have cost you the question.
Layer three: the examiner
The newest layer is the one that produced the 5/10: a small drill tutor that turns the wiki into an examiner.
It writes scenario-based multiple-choice questions in the exam’s format — a realistic situation, four options, one correct. The three wrong options are drawn deliberately from the trap catalog and the comparison pages, so the distractors are the same ones the real exam would reach for. It samples questions according to the exam’s own domain weights, asks one at a time, and on every answer explains why the right answer is right, why each wrong one is wrong, and which wiki page to reread.
Then it does the thing no passive artifact can: it writes down what I missed, by name, and holds a grudge. Every miss becomes a link into the wiki. Session scores accumulate in a log. Future sessions bias toward whatever the log says I’m weak at, and a review mode re-quizzes only the concepts I’ve gotten wrong. The study system studies me back.
The honest number
First weighted drill, two nights ago: 5 out of 10.
The shape of the misses is the interesting part. Tool design and MCP integration: perfect — that’s where my daily client work lives. Agentic architecture: solid. But prompt engineering and structured output: zero for two. Context management and reliability: zero for one. The domains I operate every day held up; the conceptual scaffolding underneath the tools I use fluently is exactly where the gaps are.
That is precisely the gap the first post predicted — being expert in an adjacent field and tested on this one. The difference is that a month ago it was a hypothesis, and now it’s a log entry with named concepts attached: the difference between resuming and forking a session, between schema validation and semantic validation, between escalating and clarifying when an agent hits ambiguity. I don’t have a vague feeling of underpreparedness anymore. I have a reading list.
A 5/10 three weeks into a two-month runway, with every miss named and a system that automatically drills my weak side harder — I’ll take that trade over a flattering practice score with no diagnosis attached.
What each layer earns
The stack, honestly assessed after a month: the official Academy curriculum owns coverage — it’s still the spine, and nothing I built replaces it. The NotebookLM artifacts own the passive passes — audio for repetition, one-pagers for compression — and stop there. The wiki owns the truth: one interlinked, source-cited place where the material lives, weighted like the exam. The drill owns the pressure: production instead of recognition, with a memory of my misses.
And the honest costs are real too. The DIY layers carry an operational tax — expired sessions, rate limits, the plumbing repairs the first post opened with. And the wiki is built from community material, not an official syllabus, so the tutor carries a standing instruction: when the wiki contradicts current product knowledge, flag the discrepancy rather than teach it as fact. A study system you built yourself deserves the same skepticism as any other system you built yourself.
What’s next
The exam target is set: before mid-September, ahead of a travel commitment that makes the deadline pleasantly non-negotiable. The next post in the series is the one I flagged at the start — the gap between operating these tools daily and being tested on them formally, which the 5/10 has now made concrete. And then the result, whatever it is.
Putting Agents to Work in Production?
Prism AI Analytics builds AI-run operations for small businesses — and runs on one itself. The study stack in this post is the firm’s method in miniature: take the official capability, build the tooling around it that makes it dependable, and keep a log honest enough to trust.
