conduct

A small LAN router for a 3-machine llama.cpp cluster. Exposes one OpenAI-compatible endpoint (http://<server-host>:8100/v1) that routes each request to the right node by model name, starts the gaming PC's llama-server on demand, and skips it instantly when someone games.

Config

~/.config/conduct/config.json (created on first run):

{
  "listen": ":8100",
  "nodes": {
    "ws-router":   {"type":"static","baseURL":"http://<workstation>:11434",
                    "discover":true,"discoverSources":["preset"]},
    "headless-gemma": {"type":"static","baseURL":"http://localhost:11434"},
    "bazzite-gemma":  {"type":"bazinga","baseURL":"http://<gaming-box-host>:8089","llmURL":"http://<gaming-box-host>:11434"}
  },
  "models": {
    "gemma-4-12b": ["bazzite-gemma","headless-gemma"],
    "qwen3.8-27b": ["ws-router"]
  },
  "idleStopSeconds": 600,
  "startTimeoutSec": 90
}

bazinga nodes are driven through the bazinga API on the gaming box (start on demand, drained stop after idle, instant skip while a game or Remote Play session is active).

A node that serves a model under a different name than the config key gets a per-node modelAliases map: incoming name → name that node actually serves. conduct rewrites the model field of the forwarded request (and its /slots probes) for that node only; clients always use the models key, and /v1/models lists the model back under that key.

/v1/models also reports, per model, the token window the cluster will really serve: the node's live n_ctx once the model is loaded, falling back to the --ctx-size it launches with. Nodes otherwise advertise the model's native (training) window, which a preset-sized --ctx-size does not match.

Besides the models declared above, /v1/models and routing also accept the models a node with "discover": true advertises (see below).

"nodes": {
  "ws-router": {"type":"static","baseURL":"http://<workstation>:11434",
                "modelAliases": {"qwen3.8-27b": "Qwen3.8-27B-IQ4"}}
}

Model preset

Every llama-server node serves gemma-4-12b in router mode from one canonical preset: deploy/models-preset.ini in this repo. It holds the full parameter set for the model (context size, cache quants, chat template, sampling, reasoning off) plus the absolute model/mmproj/template paths under the uniform model dir /var/mnt/d6174ce5-701c-4276-aeb2-58f01a3930ce/llm-models (a real mount on the gaming box, a symlink to ~/llm-models on the headless server). Same model, same parameters, whichever node answers.

The serve command is identical on both nodes:

llama-server --models-dir /var/mnt/d6174…/llm-models \
    --models-preset /var/mnt/d6174…/llm-models/models-preset.ini \
    --models-max 1 --no-webui --host 0.0.0.0 --port 11434

Replicate after editing the preset (the gaming PC's copy lives in the same model dir, wrapped in its distrobox):

make sync-preset

Adding a model

discoverSources selects which of llama.cpp's model sources a discoverable node may contribute: preset (the default), models_dir (any .gguf in the node's --models-dir) and cache (HF-cache downloads). Preset-only is the default because a preset is the only source that carries tuned serving parameters — a bare .gguf is advertised with llama.cpp's defaults and the model's native context window. Adding "discoverSources":["preset","models_dir"] makes loose GGUFs join the cluster too; an unrecognised value fails the config load. Entries a node advertises without a source field are kept, so a server that omits the label cannot silently un-discover a whole node.

Add a preset section to the node that should serve it and tell that node to re-read its preset — no conduct config edit and no conduct restart:

# on the workstation, add a [section] to ~/llm-models/models-preset.ini
curl 'http://127.0.0.1:<node-port>/models?reload=1'
# in OpenCode: /models-discovery-refresh

Reloading the node is your step, not conduct's: ?reload=1 can unload a running model whose preset changed, and on a node with a large model directory it is slow enough that the listing times out. Conduct only reads the list the node already advertises.

Only a node with "discover": true contributes models; the pinned headless node and the gaming node stay off unless opted in, so a discovered model can never displace the always-loaded Gemma. Declared models entries and modelAliases always win: they keep their priority order, client-facing name and aliases, and a discovered id that collides with one is ignored.

A model added as a bare .gguf (no preset section) is not part of the cluster — it is only discovered when the node's discoverSources says so.

The workstation node runs --models-max 1: two clients on two different models will swap the model between requests, and each swap pays a load delay. Conduct probes with autoload=false, so listing or resolving a model never loads or evicts one — only the client's own request may.

Client

Point any OpenAI client at http://<server-host>:8100/v1. Model ids are the keys of models, plus the ids a discover: true node advertises — every node serves them under the same --alias, unless the node needs a different name, in which case modelAliases maps it per node.

Routing prefers an idle node: when the primary's slot is busy the next node takes the request (this is what allows two concurrent users of one model), and only when every node is busy does a request queue. Requests touching the gaming box keep its idle timer fresh so it is not stopped mid-conversation.

Deploy

make deploy HOST=<server-host>

Development

go test ./...