Handbook

Granite / Qwen local-model harness

Probe, calibrate, and measure Granite-hosted Qwen models for forge-dark-factory local worker runs. Slow inference is expected; the harness uses long timeouts and records partial success (draft-only vs verify_ok).

Updated

Prerequisites

  1. Python venv: bash scripts/setup.sh
  2. Granite env file (never commit secrets). Auto-discovery order is documented in src/forge_dark_factory/llm_env.py. Typical on this workspace:

forge-composer-workbench/project-management-certification/certification-llm.local.env

  1. Copy config/granite-env.example.env to config/granite.local.env when you need a repo-local override.

  2. Optional multi-endpoint failover: add config/llm-priority.json (see Multi-endpoint registry below). The discovered Granite env file is always priority 0; JSON entries are fallbacks with higher priority numbers.

Multi-endpoint registry

forge-dark-factory uses forge-lcdl load_llm_registry() to combine:

Source Priority Role
First discovered Granite env (load_llm_profile) 0 Primary endpoint (discovered-<source> id)
config/llm-priority.json 1+ Ordered fallbacks (transport errors only)

Example config/llm-priority.json:

{
  "profiles": [
    {
      "id": "granite-backup",
      "priority": 1,
      "base_url": "https://backup-gateway.example/v1",
      "api_key": "from-secrets",
      "model": "ctx-unlim-qwen3-8b:latest",
      "timeout_sec": 180
    }
  ]
}

Failover scope: timeouts, connection errors, HTTP 5xx across profiles. JSON parse / content errors stay inside each profile (chat_with_json_mode_then_plain); they do not advance to the next endpoint.

Redaction: aggregated failure strings include profile id and gateway host only — never api_key. PDCA model_report.json records profile_id and endpoint per attempt.

Canonical LCDL reference: forge-lcdl llm-priority.md.

Rollback: remove config/llm-priority.json and unset FORGE_LCDL_LLM_PRIORITY_FILE — behavior reverts to single discovered env (same as before this feature).

Quick start

# Phase 0 only — smoke + JSON roundtrip (no worktree writes)
bash scripts/granite-harness.sh --probe-only

# Full matrix (calculator + optional readme link), long timeout
export FORGE_DARK_FACTORY_GRANITE_TIMEOUT_SEC=180
bash scripts/granite-harness.sh

# Calculator only (faster)
bash scripts/granite-harness.sh --skip-readme

# Dry run (offline CI)
bash scripts/granite-harness.sh --dry-run --out-dir /tmp/granite-harness-dry

# Save full request/response I/O for every Granite call
bash scripts/granite-harness.sh --save-raw --skip-readme
# → runs/granite-harness/<utc>/llm-io.jsonl (consolidated)
# → per-task .../calc-multiply/llm-io.jsonl and run-*/machine/llm-io.jsonl

Outputs: runs/granite-harness/<utc>/matrix_report.json, report.md, and optionally llm-io.jsonl.

Model id Role
ctx-unlim-qwen3-8b:latest Primary cheap model (integration default)
ctx-unlim-qwen25-coder-7b:latest Coder-oriented variant

Override for a run: bash scripts/granite-harness.sh --model ctx-unlim-qwen25-coder-7b:latest

Timeouts

Variable Default Used by
FORGE_DARK_FACTORY_LLM_TIMEOUT_SEC 45 (max 120) Campaign sign-off (run-pdca-campaign.sh)
FORGE_DARK_FACTORY_GRANITE_TIMEOUT_SEC 180 (max 300) Local worker + harness

Campaign runs stay fast (45s) and escalate to Cursor. Harness and campaigns/granite-calibration.yaml use the granite timeout so Qwen can finish (calculator often needs 60–120s).

Calibration campaign

Local-only ladder (no Cursor fallback):

export FORGE_DARK_FACTORY_GRANITE_TIMEOUT_SEC=180
bash scripts/run-pdca-campaign.sh campaigns/granite-calibration.yaml

Or from the harness:

bash scripts/granite-harness.sh --run-calibration-campaign

Success: at least one task with verify_ok in matrix_report.json.

Best practices for local models

Practice Why
Narrow allowed_files Reduces wrong-path edits
One patch unit (L1) Matches cheap-model operating guide
Long granite timeout Avoid false timeouts on slow host
Verify before escalate Deterministic pytest/link checks
Repair loop Driver re-drafts with LCDL repair hints + verify output
Optional two-phase draft FORGE_DARK_FACTORY_LLM_TWO_PHASE=1 — plan bullets then JSON
JSON parse-repair Automatic strict JSON retry on parse failure

Inference note

forge-lcdl inference (forge_lcdl.inference.planner) is for RAG retrieval planning in ExecutionEngine, not coding patches. forge-dark-factory does not use it. The practical substitute for Qwen is two-phase draft + repair loops + JSON-mode-then-plain (via chat_with_json_mode_then_plain).

Reading matrix_report.json

{
  "verify_ok_count": 1,
  "probe": { "smoke_ok": true, "json_ok": true, "model": "..." },
  "tasks": [
    {
      "task_id": "calc-multiply",
      "verify_ok": true,
      "draft_ok": true,
      "latency_sec": 82.4,
      "failure_reason": ""
    }
  ]
}

Reliability in PDCA model_report.json:

  • verify_ok → score 1.0
  • draft_ok only → 0.5
  • failed draft (recorded via abandon_draft before Cursor) → 0.0

Sign-off integration

Optional during campaign sign-off:

bash scripts/campaign-signoff.sh --granite-demo --granite-harness

Default sign-off does not require Granite-only wins on lenses-production.

See also