Observability for AI systems: what to log and why
Traditional monitoring assumes a deterministic system: the same input gives the same output, and an error is either thrown or not. AI features break both assumptions. They fail softly, plausibly, and without an exception in the logs. So what should you actually record?
Log the whole decision, not just the answer
For every AI interaction we store one structured record containing:
- The user's input, the resolved prompt, and the template version that produced it.
- Model name and version, temperature and other parameters.
- Retrieved context: document ids, chunk ids and retrieval scores.
- The raw response, plus whatever was parsed out of it.
- Tool or function calls, with arguments and results.
- Latency broken down by stage, and token counts in and out.
- A trace id that ties all of it to the originating request.
The prompt template version is the field teams forget, and it is the one that answers "did quality drop because of our change last Tuesday?"
Three numbers worth alerting on
Dashboards full of token counts look impressive and tell you very little. These earn their place:
- Refusal and fallback rate. A sudden rise usually means retrieval broke, not that users started asking harder questions.
- Citation validity. The share of answers whose citations resolve to chunks that were actually retrieved. It catches hallucination without a human in the loop.
- p95 end-to-end latency. Averages hide the requests that made users give up.
Capture feedback where the answer is
A thumbs-up control next to the answer costs nothing and gives you a labelled dataset. Store the feedback against the interaction id so a negative rating carries its full context: prompt, retrieved chunks, model version. Without that link you have a number; with it you have a reproducible failure.
Evaluate on a schedule, not just on deploy
Model providers update models underneath you, your documents change, and users find new phrasings. We run the evaluation set nightly and on every prompt change, and treat a regression like a failing test.
If you cannot tell whether last week's change made the assistant better, you are not iterating. You are guessing in public.
Privacy is part of the design
Prompt logs contain whatever users typed, which in most products means personal data. Decide deliberately: redact before storage, keep a short retention window, restrict access, and document it. Doing this at the start is cheap. Doing it after a security review is not.
Start here
If you have an AI feature in production with no instrumentation, add the interaction record first, then the citation check, then feedback capture. Each takes about a day, and together they turn "the assistant feels worse lately" into a question you can answer.
