Handbook

Design: lmeta bridge for the Dark Factory driver loop

Status: signed off (Program C, P1). Author: Fable (orchestrator) with Architecture Versona review notes inline. Implementation: Program C P2–P5.

Updated

Purpose

Express the sequential driver loop (driver.py: classify → route → context → plan → draft → apply+verify → repair → assay → promote) as an lmeta (AutomationFlow v3) flow executed by forge_lcdl.flow.FlowExecutor, keeping lmeta orchestration-only: every side effect (worktree, file writes, wiki trace, promote) stays in dark-factory code reached via forgeConsumerHook, mirroring forge-cockpit-web/cockpit_server/flow_ingest_runner.py.

Rollback: env flag FORGE_DARK_FACTORY_VIA_LMETA (default off) selects the lmeta path; driver.py is untouched and remains the default.

Boundary decisions (Architecture Versona review)

  1. Single persist owner — dark-factory. The flow never writes files; hooks do. The lmeta trace (FlowContext.trace) is orchestration metadata only and is not written into machine/ (the machine wiki keeps its existing writers, called from hooks).
  2. run_lcdl routing, not task registry entries — dark-factory stage functions are exposed to the flow through the run_lcdl callable bound in driver_lmeta.py, routing df.* task ids to local functions (same pattern as Cockpit's lcdl_teams_bridge.run_task). We do not register global TaskSpec entries in forge_lcdl.tasks.registry: these operations are consumer-local, not governed reusable tasks. This keeps forge-lcdl deltas to the loop_while export only.
  3. No new lmeta refs. v3 composition (switch, forgeOperatorFallback, forgeLoopWhile via states, forgeConsumerHook) expresses the loop. The v4 guard state is windowed-signal monitoring — wrong shape for a one-shot assay verdict — so assay is a hook + switch, not a guard.
  4. Flow granularity — one flow (df-driver-loop) per campaign item / driver invocation. Decompose-and-rerun (route.decompose) stays host-side in driver_lmeta.run (it spawns a second flow run), matching driver.run's two-run behavior.

Stage → lmeta mapping

# Driver stage (PhaseRecord name) lmeta state (type) Mechanism Task id / hook name
1 classify Classify (operation) forgeLcdlRun task df.classify
2 route Route (operation) + RouteSwitch (switch) forgeLcdlRun, then switch on ${ .route.tier } = escalate → FinalizeEscalated task df.route
3 context BuildContext (operation) forgeConsumerHook (creates target copy + worktree + context pack — side effects) hook df.prepare_run
4 plan Plan (operation) forgeLcdlRun (pure planning over pack) + forgeConsumerHook to persist plan.json task df.plan; hook df.write_plan
5 draft-unit* DraftUnit (operation, inside foreach over units) forgeOperatorFallback over one step per ladder backend, each df.draft with input.backend task df.draft
6 apply+verify, repair-N ApplyVerifyRepair (operation) forgeLoopWhile (max_iterations = max_repair_attempts + 1) wrapping hook df.apply_verify + fallback re-draft df.draft with repair_context hook df.apply_verify; task df.draft
7 proof Proof (operation) forgeConsumerHook (writes proof.md/proof.json) hook df.write_proof
8 assay Assay (operation) + AssaySwitch (switch) forgeConsumerHook (runs check_assay, writes assay.json) then switch on ${ .assay.ok } hook df.assay
9 promote Promote (operation) forgeOptional around forgeConsumerHook (only when pass + promote_live_path) hook df.promote
10 finalize / dual-wiki Finalize (operation) forgeConsumerHook (write_run, generate_and_write, freeze_verify) hook df.finalize

Phase recording: each hook internally calls _record_phase-equivalent writers so phases.json matches the legacy driver's phase-name set (parity requirement). The flow does not emit phases itself.

Task vs hook matrix

Operation Kind Why
df.classify (classify.classify) task via run_lcdl Pure, deterministic, JSON-in/out
df.route (router.route) task Pure decision over classification
df.plan (planner.plan_for_target) task Pure over context pack (reads worktree via pre-built pack)
df.draft (worker.draft_patch) task Backend call, JSON proposal out; ladder = forgeOperatorFallback steps with input.backend per rung; transport-failure skip logic preserved in the task wrapper
df.prepare_run (target copy, worktree, build_context_pack) hook Filesystem side effects
df.write_plan, df.write_proof hook Machine-wiki writes
df.apply_verify (execute_patch_unit + worktree apply) hook Filesystem + subprocess verification
df.assay (check_assay + write) hook Persist + verdict
df.promote (promote_worktree_to_live) hook Live-repo side effect, human-gated by campaign config
df.finalize (wiki write, H report, freeze gate) hook Single persist owner; freeze gate raises on drift

FlowContext I/O contract

Inputs (FlowExecutor.run(canonical, inputs=...)) — JSON-safe serialization of DriverConfig:

{
  "goal": "...", "target": "/abs/path", "out_dir": "/abs/run-dir",
  "level": "L1", "sublevel": "L2.2|null", "item_id": "...",
  "worker_ladder": ["fake"], "fixture_path": "/abs|null",
  "allowed_files": ["..."], "patch_units": [{"goal": "...", "allowed_files": ["..."], "layer": "logic"}],
  "max_repair_attempts": 2, "promote_live_path": null, "verification_argv": null
}

State I/O (written to ctx under the state's output key; hooks receive resolved args.data):

State Reads Writes
Classify inputs.goal, inputs.target .classification (domain/size/value/task_class dict)
Route .classification, inputs.max_repair_attempts .route (tier/decompose/quality/reasons)
BuildContext inputs.* .prepared (worktree path, pack summary, run_id)
Plan .prepared, inputs .plan (units: patch_id/goal/allowed_files/layer)
DraftUnit .plan.units[i], .prepared .draft (backend, proposal file list, model, notes)
ApplyVerifyRepair .draft, .plan.units[i] .verify (status, changed_files, attempts, errors)
Proof .verify, .plan .proof (final_status)
Assay .proof, .verify, inputs.level/sublevel .assay (ok, messages, tests_pass, …)
Promote .assay.ok, inputs.promote_live_path .promote (ok, messages)
Finalize everything above .final (final_status, run_dir, report_path)

WorkerRequest/WorkerResult and PatchUnitSpec cross the boundary as dicts with exactly the fields listed above; driver_lmeta.py owns the dataclass↔dict conversion (hooks receive/return dicts only).

Executor wiring (driver_lmeta.py)

  • Loads flows/df-driver-loop.lmeta (+ flows/units/*.lmeta) with a local _dir_loader (no forge-cdp-manager runtime dependency); resolve() flattens; FlowExecutor(run_lcdl=..., consumer_hooks=..., max_iterations=...) runs it in-process.
  • run_lcdl(task_id, version, input) dispatch table: df.classify, df.route, df.plan, df.draft → thin wrappers over existing functions. Unknown ids raise RuntimeError (so forgeOperatorFallback treats a missing rung as failure and resolve-time validation catches typos).
  • Returns a DriverResult assembled from .final so pdca.run_campaign is agnostic to the path.
  • Flag selection in pdca.py: _driver_run = driver_lmeta.run if os.environ.get("FORGE_DARK_FACTORY_VIA_LMETA", "") in ("1","true","yes","on") else driver.run — resolved lazily at call time so importing pdca never imports driver_lmeta when the flag is off (negative-test requirement).

Parity definition (P4 gate)

Same campaign item (poc-sandbox-offline, worker_ladder: [fake]) through both paths must yield: identical assay.json verdict fields (ok, tests_pass, acceptance_criteria_met, risks_reviewed, level, sublevel), identical phase-name set in phases.json (order may differ only where the design table above documents it — currently no differences expected), and freeze_gate.verify pass on the lmeta run dir.

Explicitly rejected alternatives

  • v4 guard for assay — windowed signal sampling, wrong semantics for a one-shot verdict (see boundary decision 3).
  • Global TaskSpec registration — pollutes the shared registry with consumer-local operations; run_lcdl routing is the Cockpit-proven pattern.
  • HTTP flow run via cdp-manager — forgeConsumerHook passes Python callables; must run in-process (same constraint documented in flow_ingest_runner.py).