Design: Fleet-distributed campaign dispatch

Purpose

Let a PDCA campaign fan independent items out to a prioritized list of Forge Fleet servers as docker_argv jobs, run each item inside a container with the dark-factory worker stack, return artifacts to the host, and keep every trust gate (assay, freeze) host-side. Rollback: FORGE_DARK_FACTORY_DISPATCH unset → the existing sequential local path runs unchanged.

Dispatch contract — forge_lcdl.dispatch.fleet

New leaf module src/forge_lcdl/dispatch/fleet.py (+ empty-ish dispatch/__init__.py). Boundary rules (Architecture Versona):

• stdlib urllib/json/time only; no new dependencies.
• No imports from forge_lcdl.flow, runner, tasks, transport — and nothing in forge-lcdl core imports dispatch. Consumers inject it (mirrors RunLcdlFn injection); the lmeta flow reaches it only through a consumer hook the host binds.

API surface (shapes mirror forge-lenses/lenses/sandbox/fleet_client.py and forge-certificators/.../fleet_client.py):

@dataclass(frozen=True)
class FleetNode:      # id, base_url, bearer_token, enabled, priority, max_cpu_percent, max_memory_percent
@dataclass(frozen=True)
class FleetJobRef:    # node_id, base_url, job_id  (NO token stored)

load_fleet_servers(path=None, environ=None) -> list[FleetNode]
select_node(nodes, *, exclude_ids=()) -> tuple[FleetNode, dict] | None   # priority asc, /v1/health probe, ceiling skip
submit_docker_argv_job(node, argv, *, session_id="", meta=None) -> FleetJobRef
upload_workspace(node, job_id, tar_gz_bytes) -> dict                     # PUT /v1/jobs/{id}/workspace + sha256 header
poll_job(node, job_id, *, timeout_s, interval_s) -> dict                 # GET /v1/jobs/{id} until terminal
cancel_job(node, job_id) -> dict

Config resolution order for load_fleet_servers: FORGE_FLEET_SERVERS_FILE env → <cwd>/config/fleet-servers.json → single-node fallback from FORGE_FLEET_BASE_URL + FORGE_FLEET_BEARER_TOKEN (priority 0). Node JSON shape is identical to Lenses fleet-settings.json nodes[] entries so Studio-configured nodes can be pasted verbatim:

{"nodes": [{"id": "granite-host", "base_url": "http://host:18766", "bearer_token": "…",
            "enabled": true, "priority": 100, "max_cpu_percent": null, "max_memory_percent": null}]}

Executor integration

No new lmeta ref. The host (pdca.py fan-out mode) performs dispatch outside the flow: the container runs the full lmeta driver loop for one item; the host submits/polls/ingests. A future per-state dispatch (single flow state remoted) would use a df.dispatch consumer hook bound to this module — the hook name is reserved, not implemented in Program D.

Job shape

{"kind": "docker_argv",
 "argv": ["docker", "run", "--rm", "--network", "host",
           "-v", "{fleet_workspace}:/workspace",
           "forge-agent-worker:latest",
           "python", "-m", "forge_dark_factory.container_worker",
           "--item-spec", "/workspace/item-spec.json", "--out", "/workspace/out"],
 "session_id": "dfd-{campaign_id}-{item_id}",
 "meta": {"container_class": "forge-agent-worker",
          "workspace_upload_required": true,
          "use_fleet_template_image": true,
          "requirements": ["forge-agent-worker"],
          "build_template_if_missing": true}}

The Fleet runner substitutes its workspace mount; container_worker reads the bundle, runs one campaign item via the lmeta driver path (FORGE_DARK_FACTORY_VIA_LMETA=1 inside the container), posts progress via POST /v1/jobs/{id}/workspace-worker-progress (header X-Workspace-Worker-Token, token delivered inside the workspace bundle by Fleet — never by us), and writes results to /workspace/out.

Workspace tarball (host → node)

Contents (built by a packer helper in dark-factory):

1. target/ — target repo snapshot: tracked files only, excluding .git, virtualenvs, runs/, and any *.env / *secrets* paths (deny-glob list in the packer, tested).
2. item-spec.json — serialized campaign item (goal, level, sublevel, worker_ladder minus cursor, allowed_files, patch_units, fixture inline if small).
3. llm-env.json — LLM endpoint config for the container: LLM_BASE_URL, LLM_MODEL, LLM_API_KEY only (Security Versona: LLM key is the only secret that enters the container; it is required for drafting; it is never echoed into artifacts — existing model_report.json records env file name, not contents).

Explicitly excluded: Fleet bearer tokens (host-only), Cursor credentials (the cursor ladder rung is stripped from remote item specs; cursor-backend items are not eligible for dispatch), git credentials, .ssh.

Result retrieval and host-side gates

1. Terminal job → host reads worker_result JSON (small summary: final_status, assay verdict, artifact manifest) and downloads the out-bundle (Fleet workspace round-trip; same channel as certificators source-ingest).
2. Host unpacks machine/ + human/ trace into runs/campaigns/<cid>/<item>/run-*/ exactly where the local path would write them.
3. Host re-runs check_assay on the returned evidence and freeze_gate.verify on the run dir — a remote worker cannot self-certify (trust boundary; Security Versona requirement).
4. machine/run.json gains dispatch: {"mode": "fleet", "node_id": "...", "job_id": "...", "base_host": "host:port"} — no token, no full URL with credentials; H report renders one dispatch line.

Fan-out semantics (pdca.py)

• Flags: FORGE_DARK_FACTORY_DISPATCH=fleet (default unset = local), FORGE_DARK_FACTORY_DISPATCH_MAX_PARALLEL (default 2), FORGE_DARK_FACTORY_DISPATCH_REQUIRE_REMOTE=1 → zero eligible nodes is a hard error; unset → documented local fallback with a warning line.
• Scheduler: items are independent by campaign contract; submit up to N in flight; re-select node per item (exclude_ids of busy nodes when possible) so healthy nodes share load; poll round-robin; aggregate into the existing campaign summary shape.
• Item eligibility: worker_ladder must not require cursor; promote_live items run promote host-side after ingest, never in-container.

Container image (docker/agent-worker/)

Single Dockerfile: python:3.12-slim, non-root worker user, editable installs of vendored forge-lcdl, forge-workcells, and forge_dark_factory from the build context, no secrets baked in. Registered as Fleet container template id forge-agent-worker (PUT /v1/container-templates, POST /v1/container-templates/build — operator step documented in docs/FLEET-DISPATCH.md). No forge-fleet server changes.

Security posture (Security Versona summary)

Rejected alternatives

• New lmeta forgeDispatch ref — dispatch is a consumer capability, not an orchestration primitive; hooks/host-side scheduling suffice (lmeta boundary rule).
• Fleet server API changes (job groups, node federation) — client-side priority selection over N single-orchestrator hosts is proven in Lenses and sufficient for campaign fan-out.
• Remoting the Granite HTTP call only (no container) — would distribute the cheapest part (LLM call) while keeping verify/apply local; containerizing the whole item run is what actually parallelizes campaigns.