Handbook

PDCA campaigns

Sequential Plan → Do → Check → Act runs over ordered campaign items. Each item drives the L1 driver in an isolated git worktree, runs quality gates, writes a model reliability report, and optionally promotes to a live…

Updated 2026-07-26

Quick start

bash scripts/setup.sh
bash scripts/run-pdca-campaign.sh campaigns/poc-boundary.yaml
bash scripts/run-pdca-campaign.sh campaigns/lenses-production.yaml

Campaign outputs live under runs/campaigns/<campaign_id>/<item_id>/.

Campaign YAML schema

Top-level keys:

Field	Type	Required	Default	Purpose
`campaign_id`	string	no	YAML stem	Campaign directory name under `runs/campaigns/`
`items`	list	yes	—	Ordered campaign items

Per-item fields (parsed by pdca.load_campaign):

Field	Type	Required	Default	Purpose
`id`	string	yes	—	Stable item id (directory name)
`goal`	string	yes	—	PLAN goal passed to the driver
`target`	string	yes	—	Sandbox or repo path (worktree root)
`level`	string	no	`L1`	Assay target level (`L1`, `L2`, `L3`, …)
`sublevel`	string	no	—	Assay sub-level id (`L1.1`–`L4.3`); see AUTONOMY-LEVELS.md
`worker_ladder`	list of strings	no	`["local", "cursor"]`	Ordered backends to try (see worker matrix below)
`patch_units`	list	no	—	Ordered units: `goal`, optional `allowed_files`, optional `layer` (`logic` / `docs` / `ui`)
`promote_live`	string	no	—	Live checkout path to copy into when CHECK passes and tree is clean
`verification_argv`	list	no	—	Extra argv run after driver pass (pytest, link checker, …)
`fixture`	string	no	—	Fixture JSON for `fake` backend
`pre_pytest_red`	bool	no	`false`	Require red pytest before the run
`allowed_files`	list	no	—	Path allowlist for single-unit items
`verification_cwd`	string	no	—	Working directory for `verification_argv`
`verification_argv_extra`	list of lists	no	—	Additional verification argv tuples
`promote_scope`	string	no	`repo`	Promotion scope hint

Example fragment from campaigns/poc-boundary.yaml:

campaign_id: poc-boundary
items:
  - id: poc-sandbox-offline
    goal: fix failing multiply
    target: sandbox/calculator
    level: L1
    worker_ladder: [fake]
    fixture: sandbox/fixtures/multiply_fix.json
    pre_pytest_red: true

  - id: poc-sandbox-granite
    goal: fix failing multiply test in calculator.py
    target: sandbox/calculator
    level: L1
    worker_ladder: [local, cursor]
    pre_pytest_red: false

Worker ladder

Backend	Role	Env / skip conditions	Typical ladder position
`fake`	Fixture-backed patch (`FORGE_DARK_FACTORY_FIXTURE` or item `fixture`)	No LLM, no network	Offline CI (`poc-boundary` item 1); forced when `worker_backend=fake` and no custom ladder
`deterministic`	Rule/fixture map for clear-domain tasks	No LLM	CI campaigns (`lenses-production-*-ci.yaml`); router tier `deterministic`
`local`	Granite / OpenAI-compatible via `forge_workcells.local_llm_worker`	`FORGE_DARK_FACTORY_GRANITE_TIMEOUT_SEC` (default 180s, max 300); `FORGE_DARK_FACTORY_LLM_TIMEOUT_SEC` for other profiles	Default sign-off ladder step 1
`cursor`	Cursor CLI `agent -p --model <CURSOR_MODEL>`	`CURSOR_AGENT_BIN`, `CURSOR_MODEL`, `CURSOR_AGENT_NON_INTERACTIVE=1`; skipped with `cursor_skipped` if `agent` missing	Escalation after `local` failure or Granite timeout

Escalation order: campaign worker_ladder is tried left-to-right (next_backend in tier_backend.py). On CHECK fail, PDCA retries once with the next tier. Router-only default for local tier: ("local", "cursor"); deterministic tier: ("deterministic",); escalate tier: empty ladder (human).

Campaign pattern	Typical `worker_ladder`
Offline boundary	`[fake]` then `[local, cursor]`
Sign-off (Granite/Cursor)	`[local, cursor]`
CI deterministic	`[deterministic]`

Sign-off gate: Granite or Cursor verify_ok per item (see scripts/campaign-signoff.sh).

Fail-fast LLM (campaign runs)

export FORGE_LCDL_CHAT_HTTP_RETRIES=0          # default in run-pdca-campaign.sh
export FORGE_DARK_FACTORY_LLM_TIMEOUT_SEC=45   # default 45s (max 120)
export FORGE_DARK_FACTORY_GRANITE_TIMEOUT_SEC=180  # local worker + harness only (max 300)

On Granite timeout the driver skips further local attempts and escalates to Cursor.

For long-timeout Granite calibration (no Cursor), see GRANITE-HARNESS.md and campaigns/granite-calibration.yaml.

Set CURSOR_AGENT_BIN, CURSOR_MODEL, CURSOR_AGENT_NON_INTERACTIVE=1 for Cursor. If agent is missing, the cursor step is skipped (cursor_skipped in model report).

Per-item artifacts

Path	Purpose
`plan.json`	PLAN: goal, target, level
`run-*/machine/`	Driver machine record (M)
`run-*/human/report.md`	Generated human narrative (H)
`run-*/machine/model_report.json`	Model usage + reliability scores
`run-*/machine/assay.json`	Assay gate (tests_pass, acceptance, risks)
`run-*/machine/promote.json`	Promotion result (when `promote_live` set)
`check.json`	CHECK gate summary

Model reliability report

Written to machine/model_report.json and summarized in the H report.

Reliability per attempt (deterministic):

verify_ok → 1.0
draft_ok only → 0.5
else → 0.0

Campaign summary aggregates granite_success_rate and winner backend per item.

CHECK gates

Dual-wiki freeze gate (H matches M)
Assay gate (forge/forge.config.yaml core evidence)
Driver final_status == pass
Optional item verification_argv (pytest, link checker, inline asserts)

On CHECK fail: ACT retries with the next worker tier once; if still failing, the item is marked blocked or fail and the campaign continues (unless P1/P8 infra breaks).

Model reliability (PoC boundary campaign)

Example from a green poc-boundary Granite run:

Item	Winner	Granite success	Notes
poc-sandbox-offline	fake	n/a	Offline fixture
poc-sandbox-granite	local	1.0	Granite drafted + verified on first attempt

Per-item JSON: runs/campaigns/<campaign>/<item>/run-*/machine/model_report.json.

Promotion policy

When promote_live is set (lenses campaigns):

All CHECK gates passed on the worktree
Live repo git status --porcelain is clean
Changed files copied from worktree → live (allowed paths only)
No auto-commit — operator commits manually

If the live tree is dirty, promotion is skipped and recorded in promote.json.

Campaign files

File	Items
`campaigns/poc-boundary.yaml`	Offline sandbox + Granite sandbox
`campaigns/lenses-production.yaml`	Sign-off — Granite/Cursor on docs-health + handbook slice (L1)
`campaigns/lenses-production-ci.yaml`	CI only — deterministic L1 fixtures
`campaigns/lenses-production-l2.yaml`	Sign-off — L2 multi-file docs-health (Granite or Cursor)
`campaigns/lenses-production-l2-ci.yaml`	CI only — deterministic L2 multi-unit fixtures
`campaigns/lenses-production-l3.yaml`	Sign-off — L3 use-case slice (Granite or Cursor + Playwright)
`campaigns/lenses-production-l3-ci.yaml`	CI only — deterministic L3 layered fixtures + Playwright

L2 multi-unit semantics

Campaign items may declare explicit patch_units (ordered goals + per-unit allowed_files). The driver runs draft → apply → verify sequentially; only the final unit runs verification_argv / verification_argv_extra. L2 assay requires ≥2 distinct changed files in the proof union.

L3 layered use-case semantics

L3 items add optional layer per patch unit (logic, docs, ui). Final verification uses scripts/verify-docs-health-l3.py (pytest + Playwright via scripts/run-l3-e2e.sh). L3 assay requires ≥2 layers, cross-layer files, and E2E pass.

Handbook slice: sandbox/lenses-handbook-slice/ (sync via bash scripts/sync-handbook-slice.sh). Docs-health L2 seed: sandbox/docs-health-l2/ (copy of forge-lenses fixture). Docs-health L3 seed: sandbox/docs-health-l3/ (bootstrap via bash scripts/bootstrap-docs-health-l3.sh).

L1 sign-off

bash scripts/campaign-signoff.sh           # pytest + campaigns + reliability doc
bash scripts/campaign-signoff.sh --granite-demo   # includes poc-demo-granite.sh
bash scripts/campaign-signoff.sh --granite-demo --granite-harness  # + probe/matrix harness

Writes docs/RELIABILITY-SIGNOFF.md from campaign model reports (includes L2 deterministic CI when run via sign-off script).

Granite harness

See GRANITE-HARNESS.md. Quick probe:

bash scripts/granite-harness.sh --probe-only
bash scripts/granite-harness.sh --skip-readme   # calculator calibration, long timeout

L2 complete when sign-off includes green lenses-production-l2-ci (offline) and, for full sign-off, green lenses-production-l2.yaml with Granite or Cursor winners per item.

L3 complete when sign-off includes green lenses-production-l3-ci (offline + Playwright) and, for full sign-off, green lenses-production-l3.yaml with Granite or Cursor per item.

CI

GitHub Actions (.github/workflows/ci.yml): setup.sh, pytest, dark_factory_check.py, poc-demo.sh, lenses-production-ci.yaml, lenses-production-l2-ci.yaml, bootstrap-docs-health-l3.sh, lenses-production-l3-ci.yaml (deterministic + Playwright).

Forge Dark Factory