Signals

Recipes

Three signals to start with, then nine more to copy: failures explained, 200s that lied, agent turns, audit actions, flaky jobs.

Every signal here is a question a rule cannot answer from the fields alone. Change the when to your paths and they are yours.

Start with these three

One file, three columns, and a week later you have numbers you did not have before.

server/signals.ts

import { defineSignal } from '@evlog/signals'

const errorName = (e: Record<string, unknown>) => (e.error as { name?: string } | undefined)?.name ?? 'none'

/** Every 4xx/5xx. Splits the error budget between the client, the code and the dependencies. */
export const fault = defineSignal({
  name: 'fault',
  when: e => (e.status ?? 0) >= 400,
  ask: 'Who is responsible for this failure?',
  choice: {
    client: 'Bad input, expired session, missing permission, client mistake',
    app: 'A bug, a misconfiguration or a validation error in our own code',
    upstream: 'A third-party dependency failed, timed out or rate-limited us',
  },
  cacheKey: e => e.error ? `${e.method} ${e.path} ${e.status} ${errorName(e)}` : undefined,
})

/** Every 5xx. Tells a retry policy which failures are worth a second attempt. */
export const retryable = defineSignal({
  name: 'retryable',
  when: e => (e.status ?? 0) >= 500,
  ask: 'Would the same request most likely succeed if retried in a few seconds?',
  criteria: { true: 'Timeout, connection reset, rate limit, transient upstream error', false: 'Bug, bad data, missing resource' },
  cacheKey: e => e.error ? `${e.path} ${errorName(e)}` : undefined,
})

/** Successful checkouts. The 200 that sampling deletes and nobody notices. */
export const silentFailure = defineSignal({
  name: 'silent-failure',
  when: e => e.status === 200 && e.path === '/api/checkout',
  ask: 'The request returned 200, but the customer did not get what they came for',
  criteria: { true: 'No order or confirmation, a fallback path, an empty or partial result', false: 'Order created and confirmed' },
  keep: v => v.value && v.confidence > 0.8,
})

What each one gives you:

GROUP BY signals.fault.value over a week says whether the next sprint goes to validation messages, your own bugs, or a dependency's SLA. cacheKey on the error name makes an outage one call, not one per request.
signals.retryable.value = true on a 502 is the difference between a retry policy and a guess.
silent-failure promotes the checkouts that returned 200 with no order. Point when at your own money path.

All twelve

Signal	Hook	Question	Why a rule cannot
`fault`	enrich, cached	Who is responsible: user, us, upstream	A Stripe timeout and a null deref are both a 500
`severity`	enrich	noise, watch or page	Urgency depends on what failed and for whom
`retryable`	enrich, cached	Would a retry succeed	Transient and permanent errors share status codes
`slow-cause`	enrich	database, upstream, compute	Reading the timings takes a human
`silent-failure`	keep	200, but the customer left empty-handed	Sampling only keeps what a predicate can name
`webhook-ignored`	keep	Acknowledged but not acted on	The provider only sees the 200
`validation-bug`	keep	The rejected input was valid	Looks identical to bad input
`turn-resolved`	enrich	The agent did what was asked	A sampled LLM-as-judge misses the rest
`turn-looping`	keep	Repeated a tool call without progress	Loops hide in successful turns
`turn-off-script`	keep	Did something nobody asked for	Same
`audit-review`	keep	This action deserves a reviewer	Risk is in the combination, not one field
`job-flaky`	enrich	Succeeded only because it retried	The retry hides the cause

Rank failures and explain slow requests

export const severity = defineSignal({
  name: 'severity',
  when: e => (e.status ?? 0) >= 500,
  ask: 'How urgent is this failure for the on-call engineer?',
  score: ['noise', 'watch', 'page'],
})

export const slowCause = defineSignal({
  name: 'slow-cause',
  when: e => (e.durationMs ?? 0) > 2_000,
  ask: 'What dominated the duration of this request?',
  choice: {
    database: 'Query time, lock waits, many round trips',
    upstream: 'A third-party API or another service',
    compute: 'Serialization, rendering, large payloads, CPU work',
    unknown: 'Nothing in the event explains the time',
  },
})

The unknown option on slow-cause is deliberate: without it the model picks the least wrong of three, and you lose the signal that your events do not carry the timings they should.

Rescue the other 200s that lied

export const webhookIgnored = defineSignal({
  name: 'webhook-ignored',
  when: e => e.status === 200 && (e.path ?? '').startsWith('/api/webhooks/'),
  ask: 'The webhook was acknowledged but not acted on',
  criteria: { true: 'Unhandled event type, skipped, no side effect recorded', false: 'The event was processed' },
  keep: v => v.value && v.confidence > 0.85,
})

export const validationBug = defineSignal({
  name: 'validation-bug',
  when: e => e.status === 400,
  ask: 'The rejected input was actually valid and our validation is wrong',
  keep: v => v.value && v.confidence > 0.85,
})

A validator that rejects hugo+test@example.com returns the same 400 as one that rejects notanemail. validation-bug reads the rejected value and the reason, and keeps the ones where the reason is wrong.

Judge agent turns

evlog/eve emits one wide event per agent turn with method: 'EVE'. A sampled LLM-as-judge reads a few percent; a signal reads every turn. See eve for the event shape.

const isTurn = (e: Record<string, unknown>) =>
  e.method === 'EVE' && typeof (e.eve as { turnId?: string } | undefined)?.turnId === 'string'

export const turnResolved = defineSignal({
  name: 'turn-resolved',
  when: isTurn,
  ask: 'The agent accomplished what the user asked for in this turn',
})

export const turnLooping = defineSignal({
  name: 'turn-looping',
  when: isTurn,
  ask: 'The agent repeated the same tool call without making progress',
  keep: v => v.value && v.confidence > 0.85,
})

export const turnOffScript = defineSignal({
  name: 'turn-off-script',
  when: isTurn,
  ask: 'The agent took an action the user did not ask for',
  keep: v => v.value && v.confidence > 0.85,
})

Flag audit actions and flaky jobs

export const auditReview = defineSignal({
  name: 'audit-review',
  when: e => typeof e.audit === 'object' && e.audit !== null,
  ask: 'This action deserves a human review',
  criteria: {
    true: 'Privilege change, bulk deletion, export of personal data, action outside business hours by a non-admin',
    false: 'Routine self-service action',
  },
  keep: v => v.value && v.confidence > 0.8,
})

export const jobFlaky = defineSignal({
  name: 'job-flaky',
  when: e => typeof e.operation === 'string' && ((e.retries as number | undefined) ?? 0) > 0,
  ask: 'The job only succeeded because it retried, and the underlying cause is still there',
})

Audit events carry the actor, the action and the target; the risk is in the combination, and audit-review is one reviewer queue instead of a rule per combination.

More that fit the shape

first-seen (is this error new for this route), user-impact (none, degraded, blocked), owner (which team should look), known-error (which catalog entry is this, from your own why and fix), frustration on support conversations, did-succeed on session events.

Not a fit

Secret or PII detection. Sending a value to a model to ask whether it is a secret is the leak. Use redaction.
Paging. Never page on a probability. severity ranks what to read first; an alert on status and durationMs decides who wakes up.
Anything adversarial as a single verdict. A model that can be asked can be prompted.
Anything numeric. A decision model returns a distribution over options, not a number. Compute it in the handler and log it.

Edit this pageorReport an issue

Reference

Every option of defineSignal and createSignals with its default, the column format, the stats() counters, and the test helpers in @evlog/signals/testing.

Overview

Observe what flows through the pipeline, plug into it, or build your own bricks. The map of every extension point evlog exposes.