qwen38-lab

A 27B reasoning model with an agentic harness on a single consumer GPU under WSL2, kept alive unattended by a watchdog/supervisor layer, with a CPU docker fleet for memory and support services. Every number below is measured on this box (2026-08-15 … 17), not quoted from a spec sheet.

110.5 t/sgreedy, Q5 @32k, MTP depth 5

~50 t/sreal agentic use, temp 1.0

1546 t/sprefill (pp512)

22.5 GBof 24 GB VRAM, all on-GPU

~15 scold model load (DirectIO)

1The system on one screen

The one-image version — the whole lab on a single card.

Blue: the inference path (CLI ↔ native llama-server ↔ GPU). Green: memory & support (recall fleet on CPU, kali sandbox). Amber: the self-healing loop (gate, watchdog, status file). Red: the one thing that actually breaks it — desktop VRAM creep.

The self-healing loop, validated live twice on 17-08 — a wedged GPU run recovers with zero manual intervention beyond freeing the memory it names.

source: tools/qwen38.sh · tools/health-sampler.sh · cli/client.py · ~/recall/docker-compose.yml — every element in the diagram exists in the repo

2Hardware & host

Windows 11 desktop · AMD Ryzen 9 7950X3D (16C/32T) · 32 GB RAM · RTX 4090 24 GB (driver 596.49).
WSL2 Ubuntu; .wslconfig: memory=24GB, swap=16GB, autoMemoryReclaim=gradual.
The GPU reaches WSL through WDDM paravirtualization — this layer is where the failure modes live (§6): the Windows desktop shares the same 24 GB of VRAM, and WDDM silently pages GPU memory instead of failing.

3Model & serving

Qwen3.8-27B (qwen35): dense 27B, hybrid attention — 64 layers, only every 4th with full attention, the rest linear/recurrent → KV ≈ 64 KiB/token, so long context is cheap. Block 65 is an MTP head → self-speculation without a separate draft model (--spec-type draft-mtp). Vision tower is a separate 0.93 GB mmproj, opt-in.
Quants: UD-Q5_K_XL 20.2 GB (chosen) · UD-Q4_K_XL 17.9 GB (fallback; frees 2.4 GB → 64k ctx) · Q6_K rejected (20-25 t/s at 16k ctx).
Q4 vs Q5 on wikitext-2 perplexity: 6.4259 vs 6.4442 (±0.042) — inside the error bars. Weak proxy, but this test cannot tell them apart.
llama.cpp built from source with stock apt nvcc 12.0 (-DCMAKE_CUDA_ARCHITECTURES=89 compiles sm_89 fine).
Config: Q5 @32k, q8_0 KV cache both directions (+0.16% ppl = noise, saves ~2 GB), flash attention, MTP depth 5, Qwen's own sampling defaults (temp 1.0 / top-p 0.95 / top-k 20).
The depth/context cliff: at 64k ctx, depth 5 collapses 110 → 35 t/s — not the draft depth but VRAM spill (WDDM pages the draft batch to host RAM, silently). Rule, enforced by the launcher: 64k → depth ≤ 4; 32k → depth 5-6.
--load-mode dio: default mmap parked the whole 20 GB gguf in WSL's page cache and drove Windows to 227 MB free; DirectIO loads in ~15 s with a 2 GB cache peak.

source: tools/qwen38.sh · README.md measurement tables · results/RESULTS.txt (repo: git.wbd-rd.nl/r.de.ren/qwen_lab)

4The ops layer — what keeps it alive unattended

1qwen38.sh supervisor→ 2preflight.sh gate→ 3llama-server+ 4health-sampler watchdog

Gate before start

preflight.sh refuses to launch without ≥ 1.2 GB VRAM headroom and ≥ 4 GB free Windows RAM — the two preconditions of every observed wedge and bugcheck.

Watchdog during

1 Hz fsync'd health CSV (post-crash forensics), plus a wedge detector on the one signature nothing healthy shares: slot processing + frozen progress + GPU power < 150 W for 120 s → log + SIGKILL. SIGTERM provably cannot reach a wedged server.

Pre-warning

One log line when free VRAM drops under 1.5 GB: both observed wedges struck at ~23.2 GB total while the healthy state sits at ~22.7 GB.

Paused, not dead

A preflight refusal becomes a PAUSED state that names the blocker in run/launcher.status ("1832 MiB VRAM held on the desktop — close what holds it") and retries every 30 s. The server starts by itself the moment headroom returns; an unexpected server death restarts through the same gate.

Client survives too

A 120 s stream-stall trip and mid-stream connection loss raise a recoverable error: completed rounds are kept, only the half-finished exchange is dropped, and the turn resumes when the server is usable again.

/health lies

A wedged server still answers /health ok — its HTTP thread is fine, only CUDA is stuck. The usable-check therefore reads /slots: a stuck is_processing slot is the tell.

Net effect, validated live twice on 17-08: wedge → kill within 2 minutes → explained pause → automatic restart when memory frees → the turn resumes. Zero manual recovery.

source: tools/qwen38.sh · tools/preflight.sh · tools/health-sampler.sh · cli/client.py (StreamStalled, server_usable) · cli/qwen.py (_heal, _wait_until_usable)

5The CLI — governed thinking + a multi-pass tracker

Entry: qwen in a terminal (starts the server if it is down) or headless cli/qwen.py "<prompt>" = one autonomous multi-round turn. Every turn lands in sessions/*.jsonl including the full thinking trace.
Thinking policy: think freely; cut only on proven repetition (12-gram detector, threshold 0.55 — calibrated: genuine exploration 0-3%, legitimate case enumeration 31%, an actual loop 100%) or at the context ceiling. Every cut is a harvest: the model gets its own reasoning back with "write it down now" — a mid-word cut still yields a complete answer.
Tools (each returns text; an error is a tool error: string the model repairs from, never an exception): python execution, dimensional analysis (pint), symbolic math, plotting, files, bash/git, browser_check (headless Chromium: screenshot + console + page errors), and security_scan — recon tools inside an isolated kali container, allowlisted, refusing private ranges without explicit authorization.
Tracker, breadth-first and enforced in code: pass 1 gives every task its bare minimum; discovered work parks in the next pass; a pass closes only with a written insight; every task carries a falsifiable done_means committed before the work starts. A slice budget injects a ship/park/block menu when one task starts eating the run. The round cap measures futility (rounds without tracker progress), not length.
Memory: completing a task writes a disk ledger and its tool exchanges leave the context window whole; task_recall reads them back deterministically. Raw thinking never enters the vector store; only verified/falsified findings are promoted to global scope.

source: cli/qwen.py · cli/client.py · cli/tracker.py · cli/tools.py · docs/align/2026-08-16-qwen-cot-stack.md

6Failure modes worth knowing (all observed, none theoretical)

The GPU-PV wedge

Decode freezes mid-stream; GPU "utilization" pegs 100% while power collapses to ~85 W — a stuck kernel spinning, not work. No guest-visible error. Trigger both times: desktop apps grew their VRAM share mid-run until WDDM started paging.

The silent spill

Past the VRAM line, WDDM pages GPU memory to host RAM with no error — throughput quietly drops 3×. Windows-side mitigation: NVIDIA Control Panel → CUDA → Prefer No Sysmem Fallback turns the spill into a clean OOM.

Host bugcheck 0x1E

In winhvr.sys — the WSL2 hypervisor runtime, not the NVIDIA driver (confirmed from the minidump) — while the box was in the spilled state. Same layer as the wedges.

The end-game ceiling spiral

Near a full context window the thinking ceiling shrinks every round while harvests feed thinking back in; the turn dies with work done but unbooked. The crash-resume flow (fresh window + tracker re-injection) recovers it; a structural fix is parked in the harness plan.

Rule of thumb On a 24 GB card under WSL2, treat free VRAM < 1.5 GB as the danger zone. The desktop shares the card; a browser tab with WebGL is enough to cross the line mid-run.

source: docs/CRASH-2026-08-16.md · logs/stall.log · logs/health.csv (the wedge signature is in the samples)

7The docker stack — CPU fleet, GPU stays exclusive

The 27B runs native (llama.cpp build, not docker) and owns the GPU. Everything supporting runs in docker on CPU; the compose has CUDA variants of the model services, but the CPU profile is the one in use — deliberately, so the 4090 stays free.

service	port	what
qdrant v1.13.4	6333	vector store behind the memory
recall-embed (llama.cpp server)	8181	Qwen3-Embedding-0.6B Q8
recall-rerank (llama.cpp server)	8182	Qwen3-reranker-0.6B Q8
recall-chat (llama.cpp server)	8188	Qwen3-1.7B Q4 — utility model
recall-docparse / ingest / retrieve / grep	8183-8186	document pipeline
recall-mcp	8187	the MCP bridge the CLI's memory tools talk to
recall-agent	8189	agent endpoint
qwen-kali	—	isolated recon container for `security_scan` (3.2 GB image)

source: ~/recall/docker-compose.yml · qwen38-lab/docker/kali/

8Daily use

Interactive: qwen in a terminal, or the qwen.cmd double-click on the Windows desktop.
Long unattended jobs: headless cli/qwen.py "<prompt>"; watch run/launcher.status and logs/stall.log — if anything pauses, the status file says why, and it resumes alone.
After a crash or stall: relaunch with "continue" — the tracker re-injects the plan into a fresh window; ledgers survive everything.
Knobs (env): CTX · DEPTH · LOAD_MODE · VISION=1; ops: QWEN_STALL_TIMEOUT_S · QWEN_STALL_KILL_S · QWEN_VRAM_WARN_MB · QWEN_RESUME_MAX · QWEN_SLICE_BUDGET · QWEN_ONESHOT=1 (old refuse-and-exit semantics) · QWEN_FORCE=1 (override the gate, at your own risk).
Everything is in git (git.wbd-rd.nl/r.de.ren/qwen_lab): harness in cli/, ops in tools/, measurements in README.md + results/, plans in plans/. Sessions and runtime state are gitignored.