Failure detectors & incidents

The seven deterministic span detectors rendfly runs over your traces, and what an incident contains.

rendfly captures your agent’s production runs as traces made of spans — one span per LLM call, tool call, retrieval step, or step you instrument yourself. A sweep runs every few minutes over settled traces and per-project windows and applies deterministic detectors to them. When a detector fires, rendfly raises an incident: firings for the same failure are grouped into one record instead of one alert per trace.

Detection reads only span facts — token counts, durations, finish reasons, status, cost, step names. There is no model call in the detection path, so a detector costs nothing to run and returns the same verdict every time on the same spans.

The detector catalogue

Seven detectors ship enabled. Five run per trace, three run over a per-project window (cost_spike does both).

Per-trace detectors

truncation — an LLM span in the trace finished because it hit its token budget (finish_reason of max_tokens or length). The reply was cut off mid-answer. Severity: hard.

retry_storm — three or more adjacent LLM spans, within two minutes, that re-send the same input: same request model and the same prompt_tokens. A real multi-turn loop accumulates context and its prompt tokens change; identical prompt tokens mean the same call is being repeated. Severity: hard.

agent_loop — the trace’s tool/LLM step names form an adjacent cycle of 2 to 4 steps that repeats at least three times, and at least one LLM position inside the cycle has known, non-increasing prompt_tokens across the repetitions. Flat or shrinking prompt tokens are the evidence of non-progress: the agent is re-sending the same context rather than accumulating new information. A single repeated step is not an agent_loop — that is retry_storm’s job. Severity: soft.

token_explosion — an LLM span whose prompt_tokens are at least 3× the project’s trailing p95 for that model and above an absolute floor of 4,000 tokens. Usually unbounded context growth. Severity: soft.

cost_spike (per trace) — one trace’s summed LLM cost is at least 3× the project’s trailing per-trace p95 and above $0.50. Severity: hard.

Per-window detectors

tool_error_rate — over a 24-hour window, a tool whose spans failed on more than 20% of at least 10 calls. One incident per failing tool, so a broken dependency shows up as itself rather than as scattered trace noise. Severity: hard.

latency_regression — a model whose p95 span duration over the last hour is at least 1.5× its 7-day baseline p95, above 2,000 ms, with at least 20 recent samples. Only exactly-timed spans are counted, so proxy-inferred tool timing can never trigger it. Severity: soft.

cost_spike (daily) — the project’s most recent UTC day of cost is at least 3× the median of the preceding days in the window and above $1.00. Severity: hard.

What an incident is

A firing does not become a notification on its own. It is folded into an incident, keyed on (project, detector, failure class) — the failure class is the detector’s sub-kind, for example the model name for latency_regression, tool:<name> for tool_error_rate, or cost_spike_daily vs cost_spike_trace. There is at most one active incident per key, so a failure mode that keeps firing produces one record to work through instead of an alert storm.

An incident carries:

Detector and failure class — what fired, and on what.
Severity — soft, hard, or critical, set by the detector.
Evidence — the deterministic numbers behind the firing, in the detector’s own words: "4 identical gpt-4o calls (18,214 prompt tokens) within 2m0s", "search_orders failed 7/22 calls (32%)", "today $14.80 vs $3.10 median (4.8x)".
A trace link — per-trace detectors record the trace that fired, so you can open the waterfall and see the exact spans. Window detectors are project-level and have no single trace to point at.
Occurrences — every individual firing folded into the incident, each with its own evidence, timestamp and (where applicable) trace.
Status — open, resolved, or regressed, plus first-seen / last-seen timestamps and an occurrence count.

Resolving an incident is explicit — from the dashboard, or with the rendfly_resolve_incident MCP tool. If the same failure fires again after that, rendfly reopens the incident as regressed rather than creating a new one, so the history of a recurring problem stays in one place.

Alerts are dispatched per incident to email, Slack, or a webhook, depending on your plan and the project’s alert settings.

What changed in v4

If you used rendfly before July 2026, the detection engine was replaced. The following are gone from the product:

The LLM judge — detection no longer calls a model to grade a conversation. There is no judge, no rubric, no per-detector verdict prompt, and no prompt-injection surface in the detection path.
BYOK / judge keys — because nothing in detection calls a model, there is no judge provider key to bring. Your provider key is still used for one thing only: forwarding your own traffic upstream in proxy mode.
Custom checks — natural-language checks authored via MCP were tied to the judge and were removed with it.
Per-rule scoring, overall_score, rule extraction — “the system message is the contract” and the 0–100 conversation score no longer exist.
Suggested fixes and one-click fix PRs — incidents no longer carry a generated system-prompt edit, and rendfly no longer opens pull requests. An incident points you at the evidence and the trace.
Score-threshold drift alerting — replaced by the window detectors above, which compare recent behavior against a trailing baseline using span facts.
The conversations table — spans is the canonical data model. A conversation is now a trace.

What is rendfly — the overview and what problem rendfly solves
Proxy mode — the one-line ingestion path that produces the spans detectors read
Python SDK — instrumented ingestion for exact tool, retrieval, and business-logic spans