Tool telemetry

@evlog/telemetry brings evlog's wide-event model to tools that run on other people's machines — CLIs, GitHub Actions, dev scripts, and CI jobs. Same philosophy as HTTP logging: one command execution → one structured event, not a stream of analytics calls.

pnpm add @evlog/telemetry

You get command name, sanitized flags, duration, and outcome automatically. Call telemetry.set() only when you have extra counters — numbers and booleans by default. Raw argv is never read; disclosure is generated from your runtime config so it cannot drift from what you actually collect.

From CLI to your backend

The disk outbox (~/.config/{toolName}/telemetry/outbox.ndjson) is a reliability buffer, not where telemetry lives long-term. Each run appends one event locally, then the HTTP drain POSTs the backlog to your ingestion endpoint. Short-lived CI jobs and offline machines drain on the next invocation.

my-tool doctor
     │
     ▼
┌─────────────────────────────────────────┐
│  @evlog/telemetry (on the user's machine) │
│  sanitize flags · consent · one RunEvent   │
└─────────────────────────────────────────┘
     │ append (always)
     ▼
  outbox.ndjson          POST { events: RunEvent[] }
     │ flush ───────────────────────────────►  your /api/telemetry/ingest
     │                                              │
     │                                              ▼
     │                                    validate · allowlist · dedupe
     │                                              │
     │                                              ▼
     │                                    DB · warehouse · evlog drain
     ▼
  remove delivered keys on 2xx

Nothing is sent until you configure an endpoint and the server returns a success status. Until then, events stay in the outbox (or are purged on opt-out).

Configure delivery

Point the CLI at your ingestion URL when you ship it — baked into the binary, overridable per environment:

withTelemetry(command, {
  name: TOOL,
  version: VERSION,
  endpoint: 'https://telemetry.my-tool.dev/api/telemetry/ingest',
})

The built-in HTTP drain POSTs Content-Type: application/json with body { events: RunEvent[] } and a 400ms timeout. On success (res.ok), delivered keys are removed from the outbox. flush() is capped at 500ms and never throws.

Build an ingestion endpoint

You ship a CLI. Users run it on their machines. You want to know which commands run, how often they fail, and which versions are out there — without building a separate analytics SDK or reading raw argv.

That means a server endpoint your CLI POSTs to. The local outbox is only there so short CI jobs and offline runs do not lose data before the POST succeeds.

The security question (read this first)

"I put the ingest URL in my CLI — isn't that a secret?"

No. The URL lives in your npm package or binary. Anyone can read it with strings, by installing the package, or by watching network traffic. The same applies to an API key or Authorization header baked into the CLI — extractable, not a real barrier.

"So can't anyone send fake events?"

In theory, yes — someone could curl your endpoint with forged JSON. That is why you treat the endpoint as semi-public (like a browser analytics key) and defend on the server:

What "good enough" looks like: validate every POST, store idempotently, rate-limit at the edge (CDN, API gateway, or middleware), and keep the payload boring (no PII by design). That is the same playbook as client-side analytics — not banking-grade auth, but reliable product insight.

Step 1 — Validate with parseIngestBody()

Mirror the CLI config on the server: same name, same custom keys you allow via collect / telemetry.set().

import { parseIngestBody } from '@evlog/telemetry/ingest'

export const ingestOptions = {
  allowedTools: ['my-tool'],
  allowedCustomKeys: {
    'my-tool': ['checksFailed', 'checksWarn', 'itemsSynced'],
  },
} as const

export function parseTelemetryBody(raw: string) {
  return parseIngestBody(raw, ingestOptions)
}

parseIngestBody() checks batch size, JSON shape, event: 'run', tool allowlist, envelope types, ISO timestamp, and strips custom keys you did not declare. Throws IngestValidationError on failure — return 400 from your route.

Step 2 — Wire your route

Same contract for any framework — read the raw body, validate, store, return 204 so the CLI clears its outbox:

import { defineEventHandler, readRawBody, setResponseStatus } from 'h3'
import { IngestValidationError } from '@evlog/telemetry/ingest'
import { parseTelemetryBody } from '~/lib/telemetry-ingest'
import { storeRunEvents } from '~/lib/telemetry-store'

export default defineEventHandler(async (event) => {
  const raw = await readRawBody(event, 'utf8')
  if (!raw) {
    setResponseStatus(event, 400)
    return { error: 'empty body' }
  }

  try {
    const events = parseTelemetryBody(raw)
    await storeRunEvents(events)
    setResponseStatus(event, 204)
    return null
  } catch (err) {
    setResponseStatus(event, err instanceof IngestValidationError ? 400 : 500)
    return { error: 'invalid payload' }
  }
})

import { IngestValidationError } from '@evlog/telemetry/ingest'
import { parseTelemetryBody } from '@/lib/telemetry-ingest'
import { storeRunEvents } from '@/lib/telemetry-store'

export async function POST(request: Request) {
  const raw = await request.text()
  try {
    const events = parseTelemetryBody(raw)
    await storeRunEvents(events)
    return new Response(null, { status: 204 })
  } catch (err) {
    const status = err instanceof IngestValidationError ? 400 : 500
    return Response.json({ error: 'invalid payload' }, { status })
  }
}

import { Hono } from 'hono'
import { IngestValidationError } from '@evlog/telemetry/ingest'
import { parseTelemetryBody } from '../lib/telemetry-ingest'
import { storeRunEvents } from '../lib/telemetry-store'

const app = new Hono()

app.post('/api/telemetry/ingest', async (c) => {
  const raw = await c.req.text()
  try {
    const events = parseTelemetryBody(raw)
    await storeRunEvents(events)
    return c.body(null, 204)
  } catch (err) {
    const status = err instanceof IngestValidationError ? 400 : 500
    return c.json({ error: 'invalid payload' }, status)
  }
})

export default app

Step 3 — Store idempotently

Retries and backlog drains can POST the same idempotencyKey twice. Upsert (or skip) so metrics stay correct:

import type { RunEvent } from '@evlog/telemetry'

export async function storeRunEvents(events: RunEvent[]): Promise<void> {
  for (const run of events) {
    await db.telemetryRun.upsert({
      where: { idempotencyKey: run.idempotencyKey },
      create: {
        tool: run.tool.name,
        version: run.tool.version,
        command: run.command,
        outcome: run.outcome,
        durationMs: run.durationMs,
        custom: run.custom,
        recordedAt: run.timestamp,
      },
      update: {},
    })
  }

  // Or forward into evlog's drain pipeline for Axiom / Datadog / OTLP:
  // await ingestWideEvents(events.map(run => ({ source: 'telemetry', ...run })))
}

Step 4 — Rate-limit at the edge

parseIngestBody() is your last line of schema defense, not your only one. Add rate limits where traffic enters — CDN, API gateway, or middleware — especially per IP. Optional: also bucket by machineId hash inside the handler if you need tighter control.

Non-2xx responses keep events in the user's outbox; they will retry on the next CLI invocation. Return 204 (or any 2xx) only when storage succeeded.

Setup with citty

Wrap your root command in withTelemetry() — typically in src/index.ts, the file that calls runMain(). Subcommands can live in the same file or in src/commands/*.ts; only the entrypoint needs the wrapper.

import { defineCommand, runMain } from 'citty'
import { withTelemetry, defineTelemetryCommands } from '@evlog/telemetry'
import { doctorCommand } from './commands/doctor'
import { syncCommand } from './commands/sync'

const TOOL = 'my-tool'
const VERSION = '1.0.0'

export const main = withTelemetry(
  defineCommand({
    meta: { name: 'my-tool', description: '…', version: VERSION },
    subCommands: {
      doctor: doctorCommand,
      sync: syncCommand,
      telemetry: defineTelemetryCommands({ name: TOOL }),
    },
  }),
  {
    name: TOOL,
    version: VERSION,
    endpoint: 'https://telemetry.my-tool.dev/api/telemetry/ingest',
    collect: {
      flags: { format: ['json', 'csv'] },
      fields: { framework: ['nuxt', 'next'] },
    },
  },
)

runMain(main)

What gets recorded automatically

withTelemetry walks your citty tree. Each run handler produces one event — no per-command telemetry boilerplate.

The root meta.name is not prefixed when the root only delegates to subCommands.

Enriching a run

Call telemetry.set() anywhere inside a command handler — or in helpers that run on the same async stack (the run context is preserved via AsyncLocalStorage).

import { telemetry } from '@evlog/telemetry'
import { existsSync } from 'node:fs'
import { resolveConfigPath } from '../lib/config'

async function runHealthChecks() {
  let checksFailed = 0
  let checksWarn = 0

  const configPath = resolveConfigPath()
  if (!existsSync(configPath)) {
    checksFailed++
  }

  try {
    await fetch('https://registry.npmjs.org/my-tool')
  } catch {
    checksWarn++
  }

  // Counters land in event.custom on the wide event for this run
  telemetry.set({ checksFailed, checksWarn })
  return checksFailed === 0
}

export const doctorCommand = {
  meta: { name: 'doctor', description: 'Check environment' },
  args: {
    json: { type: 'boolean', alias: 'j' },
  },
  async run({ args }: { args: { json?: boolean } }) {
    const ok = await runHealthChecks()
    if (!ok) {
      throw Object.assign(new Error('Health checks failed'), { code: 'DOCTOR_FAILED' })
    }
    if (!args.json) process.stdout.write('ok\n')
  },
}

Throw an error with a code property and outcome: "error" plus errorCode are recorded automatically:

throw Object.assign(new Error('Config missing'), { code: 'CONFIG_NOT_FOUND' })

A minimal sync handler for comparison — flags are auto-captured, you only set() business counters:

import { telemetry } from '@evlog/telemetry'

export const syncCommand = {
  meta: { name: 'sync', description: 'Pull remote state' },
  args: {
    dryRun: { type: 'boolean' },
    output: { type: 'string', description: 'Output path' },
  },
  async run() {
    const itemsSynced = await pullRemoteState()
    telemetry.set({ itemsSynced })
  },
}

Setup without citty

For scripts, migrators, or custom CLIs, use createTelemetry() and wrap each logical run with t.run():

import { createTelemetry, telemetry } from '@evlog/telemetry'

const t = createTelemetry({ name: 'my-migrator', version: '2.0.0' })

await t.run('migrate', async () => {
  const rows = await migrateBatch()
  telemetry.set({ rowsMigrated: rows.length, batchSize: 500 })
})

await t.flush() // optional — also runs at end of each t.run()

GitHub Actions: swap createTelemetry for createGitHubActionsTelemetry() in the same file — it adds ghaAction and ghaEvent to custom from GITHUB_ACTION / GITHUB_EVENT_NAME only (never repo content).

import { createGitHubActionsTelemetry, telemetry } from '@evlog/telemetry'

const t = createGitHubActionsTelemetry({ name: 'my-action', version: '1.0.0' })

await t.run('report', async () => {
  const artifacts = await collectArtifacts()
  telemetry.set({ artifactCount: artifacts.length })
})

Standard envelope

Every run shares the same shape. You do not declare per-command schemas.

{
  "event": "run",
  "command": "sync",
  "durationMs": 412,
  "outcome": "success",
  "flags": { "dryRun": true, "output": true },
  "tool": { "name": "my-tool", "version": "1.0.0" },
  "env": {
    "node": "20.11",
    "ci": false,
    "provider": null,
    "tty": true,
    "agent": "cursor"   // std-env: cursor, claude, codex, … or null
  },
  "machineId": "ab3f…", // hashed; omitted in ephemeral CI
  "custom": { "itemsSynced": 42 }
}

Extending the schema

The envelope above is fixed — every tool sends the same top-level shape so parseIngestBody() and disclosure stay predictable. You do not add fields next to command or durationMs.

Your product schema lives in two extension zones:

Think of it as two contracts:

• Transport contract (RunEvent) — owned by @evlog/telemetry, stable across tools
• Product contract (custom + declared flags) — owned by you, declared in collect, mirrored on the server

Declare your extensions once

Everything you collect beyond the standard envelope is declared inline in the same withTelemetry() / createTelemetry() call:

withTelemetry(command, {
  name: 'acme-cli',
  version: '2.1.0',
  endpoint: 'https://telemetry.acme.dev/api/ingest',
  collect: {
    flags: {
      format: ['json', 'yaml', 'table'],
      target: ['staging', 'production'],
    },
    fields: {
      product: ['cli', 'action', 'migrator'],
      plan: ['free', 'pro', 'enterprise'],
      framework: ['nuxt', 'next', 'remix'],
    },
  },
})

Inside handlers, add counters and dimensions:

await t.run('deploy', async () => {
  const result = await deployServices()
  telemetry.set({
    servicesDeployed: result.count,   // number — always allowed
    rollbackUsed: false,              // boolean — always allowed
    framework: 'nuxt',                // string — allowed (in collect.fields)
    schemaVersion: 2,                 // number — version your custom shape
  })
})

Undeclared strings are dropped at runtime, never thrown. That is the privacy guarantee — no accidental paths, tokens, or free-form PII in custom.

Mirror on the server

Your ingest handler is the second gate. Pass the same tool name and custom keys you declared on the CLI:

import { parseIngestBody } from '@evlog/telemetry/ingest'

const ALLOWED_CUSTOM = [
  'servicesDeployed', 'rollbackUsed', 'framework',
  'product', 'plan', 'schemaVersion',
  'ghaAction', 'ghaEvent',
] as const

export async function POST(req: Request) {
  const raw = await req.text()
  const events = parseIngestBody(raw, {
    allowedTools: ['acme-cli'],
    allowedCustomKeys: { 'acme-cli': ALLOWED_CUSTOM },
  })
  // dedupe on idempotencyKey, then persist
}

parseIngestBody() strips any custom key you did not list — even if a client somehow sent it.

Map to your warehouse schema

After validation, shape the event for your database. The evlog envelope is the wire format; your tables are your choice:

function toRow(event: RunEvent) {
  return {
    command: event.command,
    duration_ms: event.durationMs,
    outcome: event.outcome,
    tool_version: event.tool.version,
    format: event.flags.format,
    target: event.flags.target,
    services_deployed: event.custom.servicesDeployed,
    framework: event.custom.framework,
    plan: event.custom.plan,
    schema_version: event.custom.schemaVersion ?? 1,
    is_ci: event.env.ci,
    node: event.env.node,
    received_at: new Date(),
  }
}

This is where a company "modifies the schema" in practice — not by changing RunEvent, but by defining how custom maps into internal analytics tables.

Version breaking changes in custom

When your product counters evolve, bump a numeric schemaVersion in custom (no allowlist needed) and branch in the ingest mapper:

const version = (event.custom.schemaVersion as number | undefined) ?? 1
if (version === 1) return mapV1(event)
return mapV2(event)

Commit an updated TELEMETRY.md whenever collect changes so disclosure stays aligned with releases.

Multiple tools, one endpoint

If several CLIs POST to the same ingest URL, namespace custom keys or rely on tool.name:

// Option A — prefix keys per product line
telemetry.set({ acme_product: 'cli', deployCount: 3 })

// Option B — separate allowedCustomKeys per tool.name in parseIngestBody
allowedCustomKeys: {
  'acme-cli': ['deployCount', 'framework'],
  'acme-action': ['jobDuration', 'ghaAction', 'ghaEvent'],
}

What v1 does not support

Planned v2 extensions

Not shipped yet — design targets for when the v1 extension zones are not enough:

enrich hook — run after sanitization, before the outbox append. Lets you inject computed fields without forking the package:

// Planned API — not available in v1
createTelemetry({
  name: 'acme-cli',
  version: '3.0.0',
  collect: { fields: { product: ['cli', 'action'] } },
  enrich(event) {
    return {
      ...event,
      custom: {
        ...event.custom,
        schemaVersion: 2,
      },
    }
  },
})

The hook cannot bypass sanitization rules — strings still require collect.fields. Server-side allowedCustomKeys stays the source of truth.

Namespaced keys — optional acme.product convention for multi-tool endpoints without prefix collisions. Would be documented in disclosure as acme.product: cli | action.

If you need one of these before v2 lands, open a discussion on the repo — the v1 path (custom + server mapping) covers most product analytics needs.

Privacy

Raw argv is never read. Sanitization applies to citty-parsed flags only:

• Booleans / numbers → value stored (json: true, limit: 50)
• Strings → presence only (output: true) unless allowlisted in collect.flags
• telemetry.set() → numbers and booleans always; strings only via collect.fields (undeclared values are dropped at runtime, never thrown)

collect: {
  flags: { format: ['json', 'csv'] },           // --format json → "json"; --format yaml → true
  fields: { framework: ['nuxt', 'next'] },      // telemetry.set({ framework: 'nuxt' }) ok
}

Declare allowlists in the same withTelemetry() / createTelemetry() call as collect — no separate config file.

Disclosure

generateDisclosure() produces markdown + JSON from the standard envelope plus your collect extensions. Commit the output (e.g. TELEMETRY.md) so it stays in sync with releases:

import { writeFile } from 'node:fs/promises'
import { generateDisclosure } from '@evlog/telemetry'

const { markdown } = generateDisclosure('my-tool', {
  flags: { format: ['json', 'csv'] },
})

await writeFile('TELEMETRY.md', markdown)

Users can also read it at runtime via my-tool telemetry status when you wire defineTelemetryCommands().

Consent and reliability

Opt-out priority: DO_NOT_TRACK=1 → EVLOG_TELEMETRY=0 → persisted preference (disableTelemetry() / telemetry disable). Opt-out purges the undelivered outbox.

Never harms the host: telemetry never throws, never blocks exit; flush() has a 500ms hard cap.

Outbox: events append locally before any network attempt. Offline machines and CI matrix jobs drain the backlog on the next invocation once your ingestion endpoint returns 2xx.

Endpoint: EVLOG_TELEMETRY_ENDPOINT env → endpoint option baked into the CLI. Omit both during local development; ship the URL when your ingest handler is live.

Debug

EVLOG_TELEMETRY_DEBUG=1 my-tool doctor
EVLOG_TELEMETRY=0 my-tool sync

Debug mode prints would-be payloads to stderr. Nothing is sent unless an endpoint is configured and delivery succeeds.

Why install it

You can wire anonymous usage telemetry yourself — outbox file, consent flags, flag sanitization, disclosure markdown, ingest validation, first-run notice. Or you add one dependency and get the evlog playbook baked in.

What you get for ~28 KB

Fits wherever your code runs

No framework lock-in. No PostHog / Segment / custom analytics SDK. No second schema to maintain — disclosure is generated from the same collect config the CLI uses at runtime.

What you skip building

• Privacy-safe flag capture (never reads raw argv)
• DO_NOT_TRACK / EVLOG_TELEMETRY / persisted opt-out with outbox purge
• Disk outbox with lockfile, stale-lock recovery, and backlog drain
• Auto-generated disclosure (TELEMETRY.md + telemetry status)
• First-run notice with opt-out instructions
• Typed telemetry.set() with allowlisted string fields
• Server-side ingest validation aligned with the CLI envelope
• Hard guarantee: telemetry never throws, never blocks exit

The payoff

Once ingest is live you can answer product questions that CLI authors usually guess at:

• Which commands are actually used (doctor vs sync vs telemetry status)
• Where runs fail (outcome, errorCode, version breakdown)
• How long operations take (durationMs per command)
• Which versions are still in the wild (tool.version on every event)
• Whether CI or local dev dominates (env.ci, env.agent, env.tty)

All from one wide event per run — the same mental model as evlog on the server, without bolting a browser analytics stack onto a terminal tool.

See also

• Audit — wide events for security-sensitive actions
• Drain pipeline — forward ingested events into Axiom, Datadog, OTLP, or a custom store