In brief
What should be monitored in a production LLM application?
Answer quality on sampled conversations, grounding in sources, retrieval health, safety and policy events, refusals and escalations, latency, cost per request and per user, user feedback, and every change to prompts, models, documents and configuration, so shifts in quality can be traced to their cause.
Traditional application monitoring focuses on availability, errors and response times. Those still matter for LLM applications, but they miss the most important failure mode: the application keeps running and answers keep arriving, but the answers become less accurate, less grounded or less appropriate.
Why LLM quality drifts
- Model providers update models, sometimes changing behaviour in subtle ways.
- Source documents are added, edited or become outdated.
- Users start asking different kinds of questions than the ones the system was tested on.
- Prompts and configuration are adjusted to fix one issue and affect others.
- Retrieval indexes fall out of sync with document stores.
A monitoring plan in six areas
| Area | What to track | How |
|---|---|---|
| Answer quality | Correctness, completeness and helpfulness | Daily or weekly samples graded against a rubric by experts or calibrated model graders |
| Grounding | Claims unsupported by retrieved sources | Automated faithfulness checks on samples, citation validity checks |
| Retrieval health | Empty or low-relevance results, index freshness | Retrieval scores, no-result rates, document sync checks |
| Safety and policy | Harmful content, data leakage, injection attempts | Filters and classifiers with alerting and review queues |
| Performance and cost | Latency, token usage, cost per request and per user | Tracing and cost dashboards with budget alerts |
| User behaviour | Feedback, rephrasing, escalations, abandonment | Thumbs ratings, follow-up patterns, handover rates |
Trace every request
A trace records the user input, retrieved passages, prompt version, model version, output, latency, tokens and any tool calls. Traces make it possible to investigate a complaint precisely and to find patterns across many conversations. Handle traces as sensitive data: apply access controls, masking and retention limits.
Treat every change as a release
Prompts, model versions, retrieval settings and knowledge sources should be versioned and changed through a release process that runs the evaluation suite. Record each change alongside monitoring data so that a quality dip can be linked to its cause. Pin model versions where providers allow it, and test new versions before switching.
Close the loop
Monitoring is only useful if it leads to action. Establish a regular review of sampled outputs, negative feedback and safety events. Add difficult real questions to the evaluation set. Fix knowledge gaps at the source. Adjust thresholds and escalation rules as evidence builds.
Sources and further reading
About the author: Dr. AarzooCo-Founder and Chief Operating Officer, AARHIT Systems. Writes on delivery, quality assurance, data readiness, monitoring and human review. More articles by Dr. Aarzoo