conduct

A small LAN router for a 3-machine llama.cpp cluster. Exposes one OpenAI-compatible endpoint (http://<server-host>:8100/v1) that routes each request to the right node by model name, starts the gaming PC's llama-server on demand, and skips it instantly when someone games.

Config

~/.config/conduct/config.json (created on first run):

{
  "listen": ":8100",
  "nodes": {
    "ws-router":   {"type":"static","baseURL":"http://<workstation>:11434"},
    "headless-gemma": {"type":"static","baseURL":"http://localhost:11434"},
    "bazzite-gemma":  {"type":"bazinga","baseURL":"http://<gaming-box-host>:8089","llmURL":"http://<gaming-box-host>:11434"}
  },
  "models": {
    "gemma-4-12b": ["bazzite-gemma","headless-gemma"],
    "qwen3.8-27b": ["ws-router"]
  },
  "idleStopSeconds": 600,
  "startTimeoutSec": 90
}

bazinga nodes are driven through the bazinga API on the gaming box (start on demand, drained stop after idle, instant skip while a game or Remote Play session is active).

Model preset

Every llama-server node serves gemma-4-12b in router mode from one canonical preset: deploy/models-preset.ini in this repo. It holds the full parameter set for the model (context size, cache quants, chat template, sampling, reasoning off) plus the absolute model/mmproj/template paths under the uniform model dir /var/mnt/d6174ce5-701c-4276-aeb2-58f01a3930ce/llm-models (a real mount on the gaming box, a symlink to ~/llm-models on the headless server). Same model, same parameters, whichever node answers.

The serve command is identical on both nodes:

llama-server --models-dir /var/mnt/d6174…/llm-models \
    --models-preset /var/mnt/d6174…/llm-models/models-preset.ini \
    --models-max 1 --no-webui --host 0.0.0.0 --port 11434

Replicate after editing the preset (the gaming PC's copy lives in the same model dir, wrapped in its distrobox):

make sync-preset

Client

Point any OpenAI client at http://<server-host>:8100/v1. Model ids are the keys of models — every node serves them under the same --alias.

Routing prefers an idle node: when the primary's slot is busy the next node takes the request (this is what allows two concurrent users of one model), and only when every node is busy does a request queue. Requests touching the gaming box keep its idle timer fresh so it is not stopped mid-conversation.

Deploy

make deploy HOST=<server-host>

Development

go test ./...