kevinmurphywebdev.com · live demo

forge

A multi-agent debugging concierge. Paste a stack trace; four specialist subagents fan out in parallel, each with a focused tool set, then a coordinator merges their structured findings into ranked, calibration-aware hypotheses.

how this demo works

~30s
  1. 1. Click run sample investigation below. The input is a hardcoded Sentry-style payload (a TypeError stack trace, a deployed commit SHA, a timestamp) — see .
  2. 2. Four lanes fire in parallel. Watch them transition queued → running → done. Each lane calls Claude Sonnet 4.6 with its own focused system prompt and a small tool set (fixture-backed in this demo; real GitHub/Sentry adapters are a follow-up).
  3. 3. When all lanes finish, the coordinator merges into ranked hypotheses with calibration-weighted confidences. You'll see panels for calibration (Brier scores per lane) and cost (per-call token usage in USD, with cache hit rates).
  4. 4. Try refreshing the page mid-run — the session is durable, the UI rehydrates from server state via the resumable-stream layer.

what's real vs fixtures

The LLM calls, structured outputs, parallel fan-out, abort plumbing, calibration log, prompt caching, and cost math are all real. The data sources the subagents call into are fixture-backed for this public demo:

  • fixture src/auth/session.ts — a small TypeScript snippet shaped like a real auth handler with a null-deref bug. Returned by fetch_file when source-reader navigates to it.
  • fixture the candidate commits ( a3f1c92, 1b8e44d) and their diffs. Returned by git_log and git_diff when blame-correlator queries the history.
  • fixture the error metrics (1,247 occurrences, 184 affected users, hourly spike). Returned by the error-tracking query tools when frequency-analyzer asks for blast radius.
  • real everything else — the LLM's reasoning, the structured output coercion, the confidence values, the merge logic, the per-lane cost in USD, the speculator hit rate. Swap the fixtures for real GitHub + Sentry adapters and the rest of the system works identically.

concepts demonstrated

  • parallel tool use
  • structured concurrency
  • discriminated unions
  • resumable streams (SSE)
  • cooperative abort
  • speculative prefetch
  • Brier-score calibration
  • Anthropic prompt caching
  • graded eval rubric
  • n-runs aggregation
stackNext.js 16·Vercel AI SDK 6·Anthropic Claude Sonnet 4.6·Upstash (rate-limit)·source

source-reader

reads the implicated source file

queued

waiting...

blame-correlator

ranks suspect commits

queued

waiting...

frequency-analyzer

quantifies blast radius

queued

waiting...

repro-drafter

drafts a minimal repro

queued

waiting...

eval harness

Forge ships with a CLI eval harness (scripts/eval.ts) that runs each of 5 reproducible bug scenarios N times and grades the output against a per-lane rubric (lib/eval/rubric.ts). The rubric scores correctness components (file match, line-range intersection-over-union, top suspect match, severity match) plus process components (snippet present, reasoning length, candidate explanations). The runner reports mean and stddev per scenario so prompt changes can be compared for statistical significance, not single-run noise. Snapshots write to .forge/evals/<timestamp>.json.

snapshot below is illustrative · from a 3-run sweep on 2026-05-20 · run pnpm eval --runs=3 to regenerate

total mean rubric score
78%
auth-null-session
87% · ±11.4pts
Null session deref after migration removed null guard
3 runs · mean 312/360pts · mean duration 22.5s
checkout-undefined-price
79% · ±18.2pts
Undefined price field crash after price service refactor
3 runs · mean 286/360pts · mean duration 24.9s
rate-limit-loop
74% · ±22.6pts
Infinite retry loop after rate-limit threshold removed
3 runs · mean 268/360pts · mean duration 26.3s
cron-missed-schedule
68% · ±24.1pts
Cron job misses schedule due to off-by-one in cron expression
3 runs · mean 244/360pts · mean duration 21.1s
webhook-replay
82% · ±15.3pts
Webhook replay causes duplicate charge due to missing idempotency
3 runs · mean 296/360pts · mean duration 25.6s

Why an eval harness matters: LLM output is non-deterministic. A single "correct" answer is a sample from a distribution. n-runs aggregation makes variance visible — a v2 that scores 17/20 once isn't proof of improvement if v1's mean is 15 ± 2. Without this layer, every prompt change is a vibes-based judgment.