AI Agent Monitoring Basics (what to log on every run)

A free guide to monitoring AI agents through their logs: six fields worth recording on every run, a write-in alert-threshold box, and a review cadence that fits around real work. The fields are generic mechanics; they apply whether your agent is a cron script calling a model API or a framework orchestrating a dozen tools.

Six fields to log on every run

One row per run, written at run time. Six fields cover most of what you'll need when a run misbehaves and someone asks what happened.

Field	What to record	What it tells you later
Trigger time	A timestamp with timezone, written when the run starts.	Groups failures by schedule. Ties a bad batch to a deploy, a cron change, or a provider incident at the same minute.
Input size	Bytes or tokens of everything the run consumed: message, files, tool output.	Rising input size with flat output is the usual signature of a context that never gets trimmed.
Output summary	One line: a success flag plus a fixed slice of the result, such as the first 100 characters or the field names returned.	Answers whether a run produced something sane without storing every byte. Keep secrets and personal data out of this slice.
Duration	Wall-clock seconds from start to finish.	Slow drift points at provider latency or a retry loop that moved in.
Error class	A short label from a fixed vocabulary you define: timeout, malformed, empty, auth, rate-limit.	A failure count says how many. Classes say which fix to reach for first.
Cost	Tokens times price, or whatever the provider's usage report says the run cost.	The first number that goes wrong without an error. A loop that doubles its turns doubles the bill.

Log the fields as plain text or CSV rows. Formats that a spreadsheet opens directly get read; formats that need a parser get skipped.

Alert thresholds (write these in)

Three lines, written down before the first production run. The blanks are yours to fill; the starting numbers are opinions, not rules:

Run rate. Alert when an hour exceeds ______ runs. A number around twice your busiest expected hour catches runaway loops without paging you for ordinary spikes.
Daily cost. Alert when a day exceeds $______. Start at what a bad day would actually hurt, then tighten or loosen after a month of real numbers.
Error rate. Alert when more than ______% of the last 20 runs fail. An error class you can't explain counts as a failure even when the run reported success.

Write the filled-in numbers where the agent's config lives, not only in a doc. A threshold nobody can find at 2am is a paragraph, not an alert.

A review cadence that fits real weeks

The table below is one opinion on scope, sized for a single person maintaining a handful of agents:

Cadence	Scope	Rough time
Daily	Scan the error-class counts and yesterday's cost total. Any class that appears for the first time gets looked at the same day.	About two minutes
Weekly	Read ten sampled outputs end to end, and glance at the duration trend for drift.	Ten to twenty minutes
Monthly	Re-check the three thresholds against the month's actuals, and delete any log field nobody has read.	Half an hour

Four opinions

Opinions from logging agent runs this way, not rules. Disagree with any of them once your own log says otherwise.

(Opinion.) Log cost from run one. Every other field explains what happened; cost is what it takes to keep happening. Backfilling cost later means running blind for as long as the gap.
(Opinion.) Keep the log line flat. One row per run in a CSV or plain text file beats a nested structure you have to dig through at 2am. Flat rows sort, filter, and paste into a spreadsheet.
(Opinion.) Fix the error vocabulary early. Five or six class names, written down and shared with anyone who reads the log. Free-form error messages resist counting, and counting is the point.
(Opinion.) A text file is enough for months. Real observability tooling earns its keep when you run multiple agents, need alert routing, or have teammates asking questions your file can't answer. Until then, the log line matters more than the log system.

What this page is

This page is a generic mechanics guide for logging and monitoring agent runs, written to be useful before you pick any tooling. It is not a review, ranking, or comparison of observability platforms, and it isn't affiliated with any of them. The thresholds and cadence are opinions, not rules, and nothing here promises an agent will behave; the log makes misbehavior cheaper to explain when it happens.

Still designing the agent? The free Agent Workflow Checklist covers eight checks to run before the first line of code, and the full catalog lists everything else Matchbook Labs ships.

The paperwork around the log

When a run misbehaves, the follow-up is paperwork: the incident report, the runbook update, the fix nobody documented. The Agent-OS Template Pack is 20 markdown templates for that side of agent building. Two of them pair directly with a log: a post-incident report format and a one-page triage runbook. The pack has no monitoring template, and the logging itself stays in your code. The product README lists the full contents verbatim:

Product specs (3): full v1 spec, single-feature spec, launch spec with success bar

System prompts (4): general agent, support agent, research agent, workflow automation — each with boundaries, refusal lines, and example blocks

Eval checklists (3): pre-release checklist, prompt quality grading, user testing script

Incident/runbook (2): post-incident report format, one-page triage runbook

Onboarding (2): new-user guide, one-screen quick start card

Changelog/release notes (2): running changelog format, public release notes template

Pricing copy (2): pricing page skeleton, purchase-unblocking FAQ

Support macros (2): seven common-ticket replies, six hard-conversation replies

According to the same README, index.md maps all 20 templates to the moment you need each one, and searching for `[` in any template surfaces every placeholder. License: personal + commercial use, including products you sell; no resale or redistribution of the template files as-is.

If the paperwork after an incident tends to go unwritten, the Agent-OS Template Pack puts the structure there in advance, 20 markdown templates that open in any editor or repo.

Get the Agent-OS Template Pack — 20 templates incl. incident report + runbook →

Plain markdown, no tooling. Personal + commercial use.