How It Works

VibeThinker runs a quantized 3.1B parameter model on CPU via llama-cpp-python. This page explains the internals: model loading, inference, and performance characteristics.

Model Specs

Inference

Inference runs on CPU. Typical throughput is ~2 tokens/second on a 2-core CPU. The model outputs structured responses:

<think>The user wants a fibonacci generator in Rust.
I'll write a function using an iterative approach
with pattern matching for the base cases.</think>
fn fib(n: u32) -> u32 {
    match n {
        0 => 0,
        1 => 1,
        _ => fib(n - 1) + fib(n - 2),
    }
}

The <think> section contains the model's reasoning. Most commands strip this tag. Use --no-think to suppress, or vibe think to show only reasoning.

Performance

• First request: ~70s (model load + inference)
• Cache hit: ~2ms (instant via LRU cache)
• Throughput: ~2 tokens/second on 2-core CPU

The LRU cache (128 entries, 5min TTL) provides dramatic speedups for repeated queries.