Inferno

Selected checkpoints. Maximum single-GPU inference performance.

Inferno is a fork of NInfer, a from-scratch C++/CUDA inference engine for explicitly registered Qwen checkpoints on one NVIDIA sm_120a GPU. Upstream NInfer targets the RTX 5090 and keeps models fully resident in VRAM; this fork targets the RTX 5060 and optimizes offloading weights to CPU/RAM. It runs text, image, and video prompts through a local CLI or OpenAI-/Anthropic-compatible HTTP APIs.

Current delta between Inferno and NInfer:

• ffn layer offloading to CPU/RAM ("run dense models on too low VRAM")
• additional KV cache quant: FP6
• model parameters exposed on API (automatic context size discovery)
• KV cache activation rotation: applies to quantized KV caches (int8-g64 and fp6-g64), off for BF16, disable via EngineOptions::kv_rotation

The supported identities are qwen3.6-27b/groupwise-int, qwen3.6-27b/nvfp4, qwen3.8-27b/groupwise-int, and qwen3.6-35b-a3b/groupwise-int. All identities execute Text, image/video Vision, MTP, prefix reuse, CLI, OpenAI/Anthropic serving, and measurement through the same public .ninfer Engine route; the 35B-A3B target additionally supports text-only DFlash.

NInfer deliberately supports a closed set of model artifacts instead of acting as a general model runtime:

Model	Weights	NInfer artifact	Size	SHA-256
Qwen3.6-27B	groupwise-int	qwen3_6_27b.ninfer	17,495,365,888 bytes (16.29 GiB)	7b51600ffd10632b9660f56085efdd9b751d79733ad32036a652234b64bebe7b
Qwen3.6-27B NVFP4	nvfp4	qwen3_6_27b_nvfp4.ninfer	18,324,064,000 bytes (17.07 GiB)	bce5f00d066c0f20f1317bf1fdcb458264cf95837c3b1f3fbec163694627893a
Qwen3.8-27B	groupwise-int	qwen3_8_27b.ninfer	18,210,531,328 bytes (16.96 GiB)	eec39564993d6e9c7d5e383382a760f093465c9d163ec9a1bd6b80199514bf3e
Qwen3.6-35B-A3B	groupwise-int	qwen3_6_35b_a3b.ninfer	22,783,246,080 bytes (21.22 GiB)	1fb9ea0b5b8561e49d9604115ec89e5d9f2b6f6434e32c37c57fffd480a325d2

The two Qwen3.6-27B weight profiles bind to the registered qwen3_6_27b target; the version-2 artifact identity selects the profile without a separate runtime flag. Qwen3.8-27B is separately registered as qwen3_8_27b and shares the 27B execution package while using W8 token-embedding and full-output-head weights. The nvfp4 profile uses W4A4 Tensor Core MMA for prefill and A16 NVFP4 kernels for decode. All three 27B artifacts retain the same Text, Vision, MTP, prefix-reuse, CLI, and serving routes.

Performance

The published measurements currently cover the three Qwen3.6 artifact profiles. Qwen3.8-27B is supported by current NInfer builds but is not yet included in the benchmark campaign.

Measurements on smaller sm_120a cards with host weight offload are reported in docs/performance.md under "Host weight offload on smaller sm_120a cards". On a 16 GB RTX 5060 Ti the Qwen3.8-27B groupwise-int identity reaches about 10.6 decode tok/s with MTP3 and 41 of 64 FFN layers device-resident.

Concurrent MTP3 decode

Saturated decode was measured on an RTX 5090 with INT8 group-64 KV cache, CUDA Graphs, MTP3, and one 8,192-token generation per active request. The values below are aggregate committed decode throughput from complete one-second intervals in which the actual decode batch remained equal to the configured concurrency. Each profile should be read independently.

Model profile	C=1	C=2	C=4	C=8	C8 / C1
Qwen3.6-27B groupwise-int	185.8 tok/s	247.0 tok/s	309.5 tok/s	535.0 tok/s	2.88×
Qwen3.6-27B nvfp4	202.4 tok/s	399.7 tok/s	699.7 tok/s	1,146.9 tok/s	5.67×
Qwen3.6-35B-A3B groupwise-int	593.0 tok/s	877.7 tok/s	1,166.0 tok/s	1,313.8 tok/s	2.22×

At C=8, Qwen3.6-35B-A3B reaches 1,313.8 aggregate decode tok/s. The 27B NVFP4 profile reaches 1,146.9 tok/s and 5.67× its C=1 throughput.

Single-request serving

The single-request corpus was measured on the same GPU with INT8 group-64 KV cache, CUDA Graphs, and a 1,024-token prefill chunk. Each reported fixture uses five fixed seeds after server warm-up. The two measured targets are reported independently and are not cross-target comparisons. The two 27B weight profiles are reported separately. Requests were submitted serially to a persistent server.

Qwen3.6-35B-A3B

• MTP0 at a 7,680-token prompt: 15,544.3 prefill tok/s and 271.1 decode tok/s.
• MTP0 at a 260,096-token prompt: 5,157.1 prefill tok/s and 188.2 decode tok/s.
• MTP3 long reasoning: 620.3–726.2 decode tok/s with 72.7–82.8% acceptance.
• MTP3 structured output: 770.9 decode tok/s, 89.1% acceptance, and 3.67 tokens/round.

Qwen3.6-27B (groupwise-int)

• MTP0 at a 7,680-token prompt: 3,218.1 prefill tok/s and 77.6 decode tok/s.
• MTP0 at a 260,096-token prompt: 1,614.8 prefill tok/s and 54.8 decode tok/s.
• MTP3 long reasoning: 161.9–175.4 decode tok/s with 73.4–78.8% acceptance.
• MTP3 structured output: 193.0 decode tok/s, 88.7% acceptance, and 3.66 tokens/round.

Qwen3.6-27B (nvfp4)

• MTP0 at a 7,680-token prompt: 11,191.5 prefill tok/s and 86.4 decode tok/s.
• MTP0 at a 260,096-token prompt: 2,510.6 prefill tok/s and 59.9 decode tok/s.
• MTP3 long reasoning: 213.1–231.0 decode tok/s with 76.3–81.1% acceptance.
• MTP3 structured output: 252.2 decode tok/s, 89.8% acceptance, and 3.69 tokens/round.
• Against groupwise-int on the same corpus and runtime options: 3.48× the 7,680-token prefill throughput, 1.55× the 260,096-token prefill throughput, and 30–32% higher MTP3 decode throughput.

See Performance for the full methodology, variability, reproduction command, and per-fixture results.

Evaluation

Capability scores were measured through NInfer's OpenAI-compatible serving route with thinking enabled, MTP=3, and EvalScope 1.9.0 (0-shot, rule scoring, one sample per problem):

Model profile	AIME 2025	AIME 2026	GPQA-Diamond
Qwen3.6-27B groupwise-int	86.67%	93.33%	86.87%
Qwen3.6-27B NVFP4	93.33%	93.33%	84.34%
Qwen3.6-35B-A3B groupwise-int	90.00%	90.00%	85.35%

Qwen3.8-27B is supported but has not yet been added to this published evaluation campaign.

These are single-sample results under that NInfer evaluation profile, not pass@k. See the model cards and full performance document for correct/total counts and evaluation notes.

Requirements

NInfer currently requires:

• 64-bit Linux;
• one NVIDIA sm_120a GPU; the RTX 5090's 32 GiB runs every identity fully resident, and the 27B dense identities also run on 16 GB-class sm_120a cards with --weight-residency ffn;
• NVIDIA driver support for CUDA 13.1 and the CUDA Toolkit 13.1 or newer;
• CMake 3.28 or newer and a C++20-capable host compiler;
• pkg-config;
• FFmpeg development libraries: libavformat >= 60, libavcodec >= 60, libavutil >= 58, and libswscale >= 7;
• libcurl >= 7.85;
• Ninja, when using the commands below.

The build rejects CUDA architectures other than 120a. There is no install target or packaged binary distribution; NInfer is run from its source build tree.

Build

git clone https://github.com/Neroued/ninfer.git
cd ninfer

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release
cmake --build build --parallel

The default configuration builds:

build/apps/ninfer
build/apps/ninfer-serve

Tests, benchmarks, and maintainer tools are excluded from the default build.

Docker

Build the runtime image on a 64-bit Linux host with an RTX 5090, a CUDA 13.1-compatible NVIDIA driver, Docker, and the NVIDIA Container Toolkit.

docker build --tag ninfer:local .

Download a model into models/ as described below, then run the HTTP server:

docker run --rm \
  --gpus '"device=0"' \
  --publish 8080:8080 \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer-serve /models/qwen3_6_27b.ninfer \
  --host 0.0.0.0

Run the CLI from the same image:

docker run --rm \
  --gpus '"device=0"' \
  --volume "$PWD/models:/models:ro" \
  ninfer:local \
  ninfer /models/qwen3_6_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-new 256

Download a model

Use the Hugging Face CLI to download one of the registered artifacts:

hf download neroued/Qwen3.6-27B-NInfer \
  qwen3_6_27b.ninfer \
  --local-dir models

# Or the 27B NVFP4 weight variant:
hf download neroued/Qwen3.6-27B-nvfp4-NInfer \
  qwen3_6_27b_nvfp4.ninfer \
  --local-dir models

# Or Qwen3.8-27B:
hf download neroued/Qwen3.8-27B-NInfer \
  qwen3_8_27b.ninfer \
  --local-dir models

# Or:
hf download neroued/Qwen3.6-35B-A3B-NInfer \
  qwen3_6_35b_a3b.ninfer \
  --local-dir models

Current NInfer builds accept only the version-2 artifact container, and all four downloads above are version 2. Migration applies only to Qwen3.6 artifacts downloaded before their version-2 publication; Qwen3.8-27B was published directly as version 2. Migrate an older exact local file in place:

python3 -m tools.artifact.migrate_v1_to_v2 models/qwen3_6_27b.ninfer

Use the same command with qwen3_6_27b_nvfp4.ninfer or qwen3_6_35b_a3b.ninfer for those artifacts. The migration updates only container metadata; it does not rewrite the weight payload. Alternatively, download the current version-2 file again from its Hugging Face repository.

Each .ninfer file contains the weights and frontend resources needed by NInfer. It is not a Transformers checkpoint, Safetensors distribution, or GGUF file.

Each artifact is complete, while GPU residency is fixed at process startup. Speculative decoding is disabled by default, so MTP/DFlash state and the optimized proposal head are not uploaded. Vision is also disabled by default, so its weights, Vision scratch phase, and frozen request-transient allocation are omitted. Add --vision to the CLI or server process that must accept image or video input. Disabled capabilities cannot be enabled by a later request. DFlash is available only for the 35B-A3B target and is text-only.

Run the CLI

./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --prompt "Explain prefill and decode in three sentences." \
  --max-context 16384 \
  --max-new 256 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

Use --messages FILE instead of --prompt for chat history, images, or videos:

./build/apps/ninfer models/qwen3_6_27b.ninfer \
  --messages examples/cli/messages/image_chart.json \
  --max-context 8192 \
  --max-new 128 \
  --vision

Answer content is written to stdout. Loading progress, reasoning, timing, throughput, memory, and speculative-decoding statistics are written to stderr. See the CLI guide and committed examples for structured input and runtime options.

Run the HTTP server

./build/apps/ninfer-serve models/qwen3_6_27b.ninfer \
  --max-context 16384 \
  --kv-capacity auto \
  --max-concurrency 2 \
  --spec mtp --draft-tokens 3 \
  --lm-head-draft

The public model ID defaults to the artifact's identity.model_id; use --model-id only to publish a deployment-specific alias.

Then send an OpenAI-style request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "qwen3.6-27b",
    "messages": [{"role": "user", "content": "Reply with one short sentence."}],
    "max_tokens": 64
  }'

The server also implements OpenAI Responses Core (typed Items, semantic SSE, local continuation state, and function calls) plus Anthropic Messages, token counting, and multimodal input. See HTTP serving.

Capabilities

All three registered model IDs support:

• text generation with thinking and non-thinking prompt modes;
• image, multi-image, video, and mixed multimodal messages;
• chunked prefill and CUDA Graph decode;
• startup-bounded small-scale concurrent serving with true batched decode;
• MTP speculative decoding with draft windows from one to five;
• BF16 and INT8 group-64 KV cache;
• host weight offload for the 27B dense identities (--weight-residency ffn), with a partial-residency knob (--n-ffn-layers N) that keeps the first N FFN layers resident to trade VRAM against throughput;
• model- and thinking-mode-aware official sampling defaults, with explicit greedy, temperature, top-k, top-p, min-p, and presence/frequency-penalty overrides;
• compatible-prefix reuse;
• OpenAI Responses Core, OpenAI Chat Completions, and Anthropic Messages, including streaming and usage accounting;
• prompt-rendered function tools and parsed tool calls.

The 35B-A3B target additionally supports text-only DFlash speculative decoding with draft windows from one to fifteen.

Current limits

• Only the four (model_id, weights_id) artifact identities listed above are accepted product identities.
• Execution is specialized for one NVIDIA sm_120a GPU and one CUDA device. The RTX 5090 runs every registered identity fully resident; cooperative kernels query the live GPU's multiprocessor count and step down their split-K schedule when the grid cannot be co-resident, so smaller sm_120a cards (for example the 16 GB RTX 5060 Ti) run correctly.
• One Engine owns one resident model and supports a startup-fixed capacity of 1–8 active requests. Decode-ready requests are compacted at round boundaries and executed in one batched model traversal.
• NInfer does not provide large-scale or preemptive continuous batching, priority/QoS scheduling, multi-GPU execution, CPU-compute offload, or distributed serving. Weight offload is supported: the 27B dense identities can place their per-layer FFN matrices in pinned host memory (--weight-residency ffn) and stream them through a fixed device staging arena during decode, which is how they load on GPUs with less VRAM than the resident weights. A partial-residency knob (--n-ffn-layers N) keeps the first N FFN layers resident to trade VRAM against throughput.
• --max-context is the logical ceiling of each sequence and is configurable up to the registered models' native 262,144-token limit. --kv-capacity N explicitly sizes the shared Main Text KV pool for all active and retained sequences, while --kv-capacity auto selects the largest usable capacity from the memory remaining after weights are loaded while preserving 1 GiB of sizing headroom. Omission defaults to one --max-context worth of pages. The resolved pool is fixed at startup and is not divided statically among request lanes.
• Tool calls are parsed and returned to the client; NInfer does not execute tools.
• The C++ headers are used by the in-tree applications and are not distributed as an installed SDK.

Documentation

• Contributing
• Documentation index
• CLI
• HTTP serving
• Performance
• CLI examples

License

NInfer is licensed under the Apache License 2.0.

The published artifacts are derived from Qwen/Qwen3.6-27B, Qwen/Qwen3.8-27B, and Qwen/Qwen3.6-35B-A3B. The 27B NVFP4 artifact also uses the fixed packed weights from rdtand/Qwen3.6-27B-PrismaSCOUT-Blackwell-NVFP4-BF16-vllm. These source repositories are distributed under Apache-2.0. Vendored dependencies retain their own license files under third_party/.