A real distributed AI cluster on DGX Spark
Two NVIDIA DGX Spark GB10 boxes are fused into a single local LLM backend: a running cluster, marked IMPLEMENTED AND WORKING in its own transfer notes.
And the first question for any "running cluster" is: does it actually serve?
Running AWQ 4-bit Qwen3-Coder-Next via vLLM, the distributed run loads in ~6 minutes, holds tens of GiB of KV cache, and generates ~200 tokens in ~15s on a coding task.
Real numbers — so what hardware are they coming from?
Each DGX Spark node is an NVIDIA GB10 with 128GB unified memory on Ubuntu 24.04 aarch64. Both sit on a Tailscale mesh alongside a Windows dev machine — a topology read by Amplifier via fleet-awareness context.
Two capable boxes — but why does the model need both of them?
A single node can't serve the whole model cleanly, and the easy quantization shortcuts are closed: on GB10, FP8 produces garbage output and MXFP4 crashes on load. Only AWQ 4-bit is verified good.
If shortcuts are ruled out, the two boxes have to genuinely cooperate.
The two nodes fuse over a 200Gbps QSFP direct link (ConnectX-7) into a single Ray cluster: tensor parallel size 2, one shard per GPU, ~256GB combined unified memory serving one model.
But someone still has to stand up that second node — by hand?
The cluster kit's install.sh installs head and worker roles, and auto-detects spark-2's worker address (from /etc/hosts, or by guessing the head IP + 1) before templating it into the configs.
Which sets up the real moment: one box reaching across to configure the other.
./install.sh --node-type head on spark-1 and --node-type worker on spark-2./etc/hosts, or guess head IP + 1.{{WORKER_IP}} into the configs; add spark-1 and spark-2 to /etc/hosts.spark-1's spark-mode script SSHes into spark-2, runs mkdir -p /etc/vllm, then writes node.env onto it — a genuine remote config write added in commit bbc9fe9, so the two GPUs come up serving one model.
The distributed part isn't just the inference — it's the setup itself.
The amplifier-module-provider-cluster speaks the OpenAI Chat Completions API to the vLLM endpoint, with a mount-time health check, tool calling, and Qwen3 thinking-token support — two self-provisioning boxes become one brain you can point work at.
That's the whole arc: two boxes, one usable brain behind Amplifier.
Data as of: Local repos under /home/ramparte/dev/ANext and /home/ramparte/dgx-spark-cluster; latest cluster commit 2026-03-04.
Feature status: Distributed cluster WORKING (AWQ Qwen3-Coder-Next verified on spark-1/spark-2); provider module PRESENT and usable.
Commands run:
cat /home/ramparte/dev/ANext/dgx-spark-setup/fleet.yaml — machines, Tailscale IPs, fleet-awareness notecat /home/ramparte/dev/ANext/cluster/CLUSTER-NOTES.md and CONTEXT-TRANSFER.md — GB10 / 128GB, Ray+QSFP, AWQ perf statscat /home/ramparte/dev/ANext/cluster/cluster-qwen.md — model, vLLM container, AWQ 4-bitwc -l .../amplifier_module_provider_cluster/__init__.py — 655 linesgrep -n 'spark-2|WORKER|ssh' /home/ramparte/dgx-spark-cluster/install.sh; git show bbc9fe9 — cross-node config writegit -C /home/ramparte/dgx-spark-cluster log (18 commits) and git -C .../dgx-spark-setup log (6 commits)Gaps / unverified: Narrative framing found elsewhere — gaming pauses, spoken-goal builds, a Raspberry Pi "voicebox", port 8420, and autonomous agent-to-agent self-deploy — is presenter narration only and is NOT documented in any code, commit, or config; it is excluded from this deck. MoE parameter count is inconsistent across sources, so no count is asserted.
Primary contributors: Brian Krabach (dgx-spark-cluster kit, all 18 commits); Sam Schillace (dgx-spark-setup fleet tooling, all 6 commits).