Running Qwen2.5-32B on a Free 6-Node Cluster

The Stack

Six Linux VMs from a free compute provider, 8 GB RAM and 20 GB disk each. Connected via Tailscale in a full mesh VPN — no NAT traversal issues, clean peer-to-peer connections. The model: Qwen2.5-32B-Instruct at Q4_K_M quantization (19.8 GB total), split into five GGUF files downloaded from Hugging Face.

The inference engine: llama.cpp, specifically its RPC (Remote Procedure Call) mode — a relatively obscure feature that lets you distribute a model's tensors across multiple machines and run inference collaboratively.

llama.cpp RPCQwen2.5-32B-Q4_K_MTailscaleSSHFSOpen WebUI6× Linux VMs

Architecture

llama.cpp's RPC model is elegant: a central coordinator (llama-server) reads the model files, shards tensors across multiple RPC workers (llama-rpc-server), and each worker holds its share in RAM.

click to zoom

Master coordinator distributes tensors via RPC to all 6 nodes. Parts 4+5 on m5, via SSHFS.

Qwen2.5-32B-Instruct weighs in at 19.8 GB at Q4_K_M — split into five GGUF files. Master only has 20 GB of disk, so parts 4 and 5 had to live on m5's local disk.

File	Size	Location
...00001-of-00005.gguf	3.7 GB	master (local disk)
...00002-of-00005.gguf	3.7 GB	master (local disk)
...00003-of-00005.gguf	3.8 GB	master (local disk)
...00004-of-00005.gguf	3.7 GB	m5 → SSHFS mount
...00005-of-00005.gguf	3.8 GB	m5 → SSHFS mount

The lsyncd Disaster

This is the part nobody tells you about: the hidden daemon running in the background, quietly waiting to ruin your day.

The cluster came pre-configured with lsyncd — a live sync daemon that uses inotify to watch for file changes and rsync them immediately to all worker nodes. Its config had one line that destroyed us:

/data/home/.lsyncd.conf.lua

-- sync master to all workers
local workers = {"m2","m3","m4","m5","m6"}
for _, host in ipairs(workers) do
sync {
default.rsync,
source = "/data/home/pool/work/",
target = host .. ":/data/home/pool/work/",
rsync = {
archive = true,
compress = true,
delete = true, -- ← the problem
},
},
end

delete = true tells rsync to make the destination an exact mirror of the source. We downloaded parts 4 and 5 directly to m5's local disk. lsyncd saw they weren't on master, and obediently deleted them. We lost ~40 GB of downloads across multiple attempts before finding this.

The fix: systemctl --user stop lsyncd && systemctl --user disable lsyncd. It ran as a user systemd service — killing the process wasn't enough because systemd restarted it every time.

SSHFS Deadlock

With lsyncd dead, SSHFS was the right tool — mount m5's filesystem over SSH, let llama-server read files normally. But under sustained IO load, the daemon would enter D state (uninterruptible sleep) and never return.

symptoms

# sshfs daemon — stuck, can't be killed
$ cat /proc/[pid]/wchan
__futex_wait

# llama-server — waiting forever
$ cat /proc/[llama-pid]/wchan
fuse_dev_do_read

Root cause: libfuse3 3.17.2 signal handler deadlock. The signal handler tries to acquire se→lock which the FUSE main loop already holds. The fix: mask signals before exec-ing sshfs so the handler never runs.

click to zoom

libfuse3 3.17.2 signal handler deadlock — and the signal-masking fix

signal-masked sshfs

nohup bash -c 'trap "" HUP TERM INT QUIT; exec sshfs -f -s \
-o allow_other,reconnect,direct_io \
m5:/data/home /data/home/mounts/m5' \
> /tmp/sshfs-m5.log 2>&1 &

# verify — after 128 MB read test
134217728 bytes (134 MB) copied, 16.3 s, 8.2 MB/s

trap "" SIGNAL sets the handler to SIG_IGN in the shell, inherited by the child after exec. No handler means no deadlock. With direct_io, SSHFS achieves a steady 8.2 MB/s.

Loading the Model

start_llama.sh

/usr/local/bin/llama-server \
--model "$MDIR/qwen2.5-32b-...-00001-of-00005.gguf" \
--rpc "100.122.91.81:50052,100.125.71.55:50052, \
100.85.214.27:50052,100.87.57.57:50052, \
100.116.9.45:50052,100.107.105.2:50052" \
--host 0.0.0.0 --port 8080 \
--ctx-size 8192 --threads 4 --parallel 2 \
--load-mode none

--load-mode none means: put nothing on master's CPU. All tensors go to the RPC workers. llama-server reads all five GGUF files, distributes tensor weights across the six workers' RAM, and then acts purely as a coordinator. The whole load takes about 48 minutes — dominated by reading parts 4+5 over SSHFS.

Results

4.67

tok/s

Prompt eval speed

0.52

tok/s

Generation speed

4,096

tokens

Context per slot

19.8

Total model size

~48

minutes

Loading time

VMs · 8 GB each

Nodes

0.52 tokens per second. For a 20-token response, that's 38 seconds. Prompt eval is 9× faster (4.67 tok/s) because prefill is embarrassingly parallel, while autoregressive generation requires sequential forward passes — each one a full round-trip across all RPC workers via Tailscale.

Lessons

Check for file sync daemons first. lsyncd with delete = true will silently destroy any file that doesn't exist on the master. Always audit systemd user services on a managed cluster before writing any important data.
Smaller model, same cluster. Qwen2.5-14B-Q4_K_M is ~8 GB — fits on two nodes, eliminates SSHFS, and would hit 2–4 tok/s.
Larger master disk. Reading 7.7 GB over SSHFS at 8 MB/s is a 16-minute IO bottleneck on every restart.
The speed bottleneck is the network. Tailscale connections are already direct (not relay). The 0.52 tok/s is the fundamental limit of TCP round-trips per transformer layer. RDMA would change this entirely.

0.52 tokens per second is not useful for real work. But it runs. And sometimes that's the point.