Instrumentation Design

Instrumentation decisions outlive the incident. What you choose to log today decides what you can reconstruct tomorrow, and no amount of post-incident effort recovers a signal nobody recorded.

That makes instrumentation a design discipline. The academy teaches it as one.

Log for the postmortem you have not had yet

Every incident review starts with the same question: what do we wish we had recorded? The answer is never more volume. High-volume logging produces haystacks, and haystacks are where signals go to hide.

The right question is narrower. Which doorways did the agent walk through, and what would have been sufficient to reconstruct the crossing? A tool call's arguments, its target, the threshold it crossed, the moment it was denied. A handful of records per decision beats a torrent of telemetry per second.

The layering rule

Instrumentation stacks like everything else in operations. Data, aggregation, model, policy, control. Each layer needs to rest before the next one goes on top, because a layer added blind cannot be monitored, and an unmonitored layer is an unobserved surface.

Practitioners learn to add one layer at a time and let it settle. The temptation is to instrument everything at once after an incident. The discipline is to instrument deliberately before one.

Design, not compliance

Checkbox instrumentation records what an auditor asks about. Designed instrumentation records what an investigator needs. The two overlap far less than expected.

The academy's test is simple. Take a hypothetical incident, walk it backwards through your logging, and see how much of the story survives. Whatever gaps the walk reveals are the design debt. Paying it down before the incident is the entire discipline.