---
title: "Reference"
description: "Every option of defineSignal and createSignals with its default, the column format, the stats() counters, and the test helpers in @evlog/signals/testing."
canonical_url: "https://www.evlog.dev/use-cases/signals/reference"
---
# Reference

> Every option of defineSignal and createSignals with its default, the column format, the stats() counters, and the test helpers in @evlog/signals/testing.

Everything configurable, on one page. The other pages link here instead of repeating it.

```bash [Terminal]
pnpm add @evlog/signals ai
```

`ai` 7.0.105 or later is a peer dependency. Entry points: `@evlog/signals` and `@evlog/signals/testing`.

## Options

<tabs>
<tabs-item label="defineSignal">

`defineSignal(options)` validates and returns a `Signal`. The shape of the answer follows the input: `ask` alone is yes/no, `ask` + `choice` picks an option, `ask` + `score` positions on a rubric.

<table>
<thead>
  <tr>
    <th>
      Option
    </th>
    
    <th>
      Type
    </th>
    
    <th>
      Notes
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        name
      </code>
    </td>
    
    <td>
      <code>
        string
      </code>
    </td>
    
    <td>
      Column name under <code>
        event.signals
      </code>
      
      . Letters, digits, dashes, starting with a letter. Unique across registered signals.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        ask
      </code>
    </td>
    
    <td>
      <code>
        string
      </code>
    </td>
    
    <td>
      The question, in plain English.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        when
      </code>
    </td>
    
    <td>
      <code>
        (event) => boolean
      </code>
    </td>
    
    <td>
      Predicate over the event. No model call when <code>
        false
      </code>
      
      . Required with <code>
        keep
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        choice
      </code>
    </td>
    
    <td>
      <code>
        Record<option, description>
      </code>
    </td>
    
    <td>
      At least two options. Makes a choice signal.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        score
      </code>
    </td>
    
    <td>
      <code>
        [level, level, ...]
      </code>
    </td>
    
    <td>
      Ordered lowest first, at least two. Makes a score signal.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        criteria
      </code>
    </td>
    
    <td>
      <code>
        { true?, false? }
      </code>
    </td>
    
    <td>
      Boolean signals only. What makes each answer true.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        keep
      </code>
    </td>
    
    <td>
      <code>
        (verdict) => boolean
      </code>
    </td>
    
    <td>
      Promote the event past sampling when <code>
        true
      </code>
      
      . Never drops. Typed on the verdict the shape produces.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        cacheKey
      </code>
    </td>
    
    <td>
      <code>
        (event) => string | undefined
      </code>
    </td>
    
    <td>
      Reuse the verdict for events sharing a key. <code>
        undefined
      </code>
      
       skips caching. Last 1,000 verdicts per process.
    </td>
  </tr>
</tbody>
</table>

`defineSignal` throws at definition time on an invalid name, an empty `ask`, `keep` without `when`, fewer than two options or levels.

</tabs-item>

<tabs-item label="createSignals">

`createSignals(options)` returns a `SignalsPlugin`: an evlog plugin with `keep`, `enrich` and `stats()`. The only required field is `signals`.

<table>
<thead>
  <tr>
    <th>
      Option
    </th>
    
    <th>
      Default
    </th>
    
    <th>
      Notes
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        signals
      </code>
    </td>
    
    <td>
      
    </td>
    
    <td>
      The signals to run. Duplicate names throw at startup.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        model
      </code>
    </td>
    
    <td>
      <code>
        'typesafe-ai/jev'
      </code>
    </td>
    
    <td>
      AI SDK evaluation model: a gateway id or a provider instance.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        budget.perMinute
      </code>
    </td>
    
    <td>
      <code>
        600
      </code>
    </td>
    
    <td>
      Model calls per minute, shared by all signals, per process.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        budget.cooldownMs
      </code>
    </td>
    
    <td>
      <code>
        30000
      </code>
    </td>
    
    <td>
      Pause after a failed call.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        timeoutMs
      </code>
    </td>
    
    <td>
      <code>
        2000
      </code>
    </td>
    
    <td>
      Per call. An overrun aborts the call and counts as an error.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        state
      </code>
    </td>
    
    <td>
      whole event minus <code>
        signals
      </code>
      
       and <code>
        signalsModel
      </code>
    </td>
    
    <td>
      What the model reads. May return a string.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        maxStateChars
      </code>
    </td>
    
    <td>
      <code>
        100000
      </code>
    </td>
    
    <td>
      Largest state sent, in characters of JSON. Larger events are skipped.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        providerOptions
      </code>
    </td>
    
    <td>
      
    </td>
    
    <td>
      Forwarded to the model call, e.g. <code>
        { gateway: { zeroDataRetention: true } }
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        evaluate
      </code>
    </td>
    
    <td>
      AI SDK <code>
        experimental_evaluate
      </code>
    </td>
    
    <td>
      Replace the model call. Tests, record and replay.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        stampModel
      </code>
    </td>
    
    <td>
      <code>
        false
      </code>
    </td>
    
    <td>
      Also write the answering model's id on <code>
        event.signalsModel
      </code>
      
      . Set it while comparing two models so their verdicts can be told apart in the drain; one more column on every judged event otherwise.
    </td>
  </tr>
</tbody>
</table>

`model` takes any AI SDK evaluation model. See [Pick the model](#pick-the-model).

</tabs-item>

<tabs-item label="Registration">

`SignalsPlugin` is a standard evlog [plugin](https://www.evlog.dev/extend/plugins). Where it goes depends on how the framework exposes plugins:

<table>
<thead>
  <tr>
    <th>
      Framework
    </th>
    
    <th>
      Register
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Nuxt, Nitro
    </td>
    
    <td>
      In a server plugin: <code>
        nitroApp.hooks.hook('evlog:emit:keep', plugin.keep)
      </code>
      
       and <code>
        nitroApp.hooks.hook('evlog:enrich', plugin.enrich)
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      Next.js, standalone
    </td>
    
    <td>
      <code>
        initLogger({ plugins: [plugin] })
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      Hono, Express, Fastify, Elysia, oRPC, SvelteKit, React Router
    </td>
    
    <td>
      The middleware's <code>
        plugins
      </code>
      
       option: <code>
        evlog({ plugins: [plugin] })
      </code>
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        createMiddlewareLogger
      </code>
      
       (toolkit)
    </td>
    
    <td>
      <code>
        plugins: [plugin]
      </code>
      
       on the call
    </td>
  </tr>
</tbody>
</table>
</tabs-item>
</tabs>

## Pick the model

The default goes through AI Gateway: one key, every decision model, swap by changing a string. A direct provider is the same option with an instance instead of a string.

<tabs>
<tabs-item label="AI Gateway">

The default. Reads `AI_GATEWAY_API_KEY`, or OIDC on Vercel with no key at all.

```typescript
createSignals({
  model: 'typesafe-ai/jev', // the default; 'liquid/d1' is the other decision model
  signals: [fault, silentFailure],
})
```

Both gateway models return a probability distribution, so every column has a `confidence`. The id string is the only thing to change to compare them; set `stampModel: true` while you do, so verdicts from each model can be told apart in the drain.

</tabs-item>

<tabs-item label="TypeSafe AI">

A direct key to the model behind the default. Install `@ai-sdk/typesafe-ai`; it reads `TYPESAFE_AI_API_KEY`.

```typescript
import { typeSafeAi } from '@ai-sdk/typesafe-ai'

createSignals({
  model: typeSafeAi.evaluationModel('jev-latest'),
  signals: [fault, silentFailure],
})
```

Same distributions, same columns as through the gateway.

</tabs-item>

<tabs-item label="OpenAI">

A general model prompted to answer. Install `@ai-sdk/openai`; it reads `OPENAI_API_KEY`.

```typescript
import { openai } from '@ai-sdk/openai'

createSignals({
  model: openai.evaluationModel('gpt-6-luna'),
  signals: [fault, silentFailure],
})
```

A prompted model returns P(true) for yes/no questions and a bare answer for choice and score, so those columns carry `value` without `confidence`. Expect a slower call than a decision model; raise `timeoutMs` if `stats().errors` climbs.

</tabs-item>
</tabs>

Any other provider that exposes `evaluationModel()` plugs in the same way. Options the provider understands go through `providerOptions`, keyed by provider name.

## The column

Verdicts land on `event.signals.<name>`. With `stampModel: true`, the answering model's id is on `event.signalsModel`.

<table>
<thead>
  <tr>
    <th>
      Shape
    </th>
    
    <th>
      <code>
        value
      </code>
    </th>
    
    <th>
      <code>
        confidence
      </code>
    </th>
    
    <th>
      Extra
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      Boolean
    </td>
    
    <td>
      <code>
        boolean
      </code>
    </td>
    
    <td>
      probability of <code>
        value
      </code>
    </td>
    
    <td>
      
    </td>
  </tr>
  
  <tr>
    <td>
      Choice
    </td>
    
    <td>
      one of the option names
    </td>
    
    <td>
      probability of that option, when the model returns a distribution
    </td>
    
    <td>
      
    </td>
  </tr>
  
  <tr>
    <td>
      Score
    </td>
    
    <td>
      one of the level names (most likely)
    </td>
    
    <td>
      probability of that level, when the model returns a distribution
    </td>
    
    <td>
      <code>
        score: number
      </code>
      
      , probability-weighted position from <code>
        0
      </code>
      
       to <code>
        levels - 1
      </code>
    </td>
  </tr>
</tbody>
</table>

A signal that promoted the event past sampling adds `kept: true` to its column.

```ts
interface BooleanVerdict { value: boolean; confidence: number }
interface ChoiceVerdict<Option extends string> { value: Option; confidence?: number }
interface ScoreVerdict<Level extends string> { value: Level; score: number; confidence?: number }
type SignalColumn = Verdict & { kept?: true }
```

Columns are written in `enrich`, before the console line. They reach stdout, platform logs such as Vercel, and every [drain](https://www.evlog.dev/integrate/adapters/overview). The response is already sent by then; what waits for the evaluation is the console line, bounded by `timeoutMs`, so a late line in the dev terminal is the model, not a slow request.

## stats()

`plugin.stats()` returns a snapshot of per-process counters:

<table>
<thead>
  <tr>
    <th>
      Field
    </th>
    
    <th>
      Counts
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        calls
      </code>
    </td>
    
    <td>
      Model calls made.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        skipped
      </code>
    </td>
    
    <td>
      Events with due signals not judged: budget spent, breaker open, or state over <code>
        maxStateChars
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        errors
      </code>
    </td>
    
    <td>
      Calls that failed or timed out.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        cached
      </code>
    </td>
    
    <td>
      Verdicts served from a <code>
        cacheKey
      </code>
      
       without a call.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        inputTokens
      </code>
    </td>
    
    <td>
      Tokens billed, as reported by the model.
    </td>
  </tr>
</tbody>
</table>

`cached` next to `calls` is what `cacheKey` saved. `skipped` climbing while `errors` stays flat means the budget is below your traffic, not that the model is down.

## Testing

`@evlog/signals/testing` replaces the model with a script. No key, no network, no budget:

```typescript [signals.test.ts]
import { createSignals } from '@evlog/signals'
import { answers, scriptedEvaluate } from '@evlog/signals/testing'

const mock = scriptedEvaluate((name, question) => {
  if (name === 'fault') return answers.choice(question, 'upstream', 0.93)
  return answers.first(question)
})

const plugin = createSignals({ signals, evaluate: mock.evaluate })

// after a request has run through evlog
expect(mock.calls).toHaveLength(1)
expect(mock.calls[0].questions).toHaveProperty('fault')
```

<table>
<thead>
  <tr>
    <th>
      Export
    </th>
    
    <th>
      Does
    </th>
  </tr>
</thead>

<tbody>
  <tr>
    <td>
      <code>
        scriptedEvaluate(answer, modelId?)
      </code>
    </td>
    
    <td>
      Returns <code>
        { evaluate, calls }
      </code>
      
      . <code>
        answer(name, question, state)
      </code>
      
       runs once per question; <code>
        calls
      </code>
      
       records every request.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        answers.boolean(p)
      </code>
    </td>
    
    <td>
      A yes/no answer with P(true) = <code>
        p
      </code>
      
      .
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        answers.choice(question, pick, p)
      </code>
    </td>
    
    <td>
      Picks <code>
        pick
      </code>
      
       at <code>
        p
      </code>
      
      , spreads the rest over the other options.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        answers.score(question, levelIndex, p)
      </code>
    </td>
    
    <td>
      Level <code>
        levelIndex
      </code>
      
       at <code>
        p
      </code>
      
      , with the weighted <code>
        score
      </code>
      
       computed.
    </td>
  </tr>
  
  <tr>
    <td>
      <code>
        answers.first(question)
      </code>
    </td>
    
    <td>
      A confident default of the right type: first option at 0.9, last level at 0.8, yes at 0.94.
    </td>
  </tr>
</tbody>
</table>

The same hook replays recorded verdicts: a `scriptedEvaluate` that looks answers up by `requestId` runs your signals against yesterday's events without a call.

---

- [What You Can Ask](https://www.evlog.dev/use-cases/signals/ask)
- [What It Costs](https://www.evlog.dev/use-cases/signals/cost)


## Sitemap

See the full [sitemap](https://www.evlog.dev/sitemap.md) for all pages.
