// api
3 API surfaces on one port.
// api surface
/v1/models List available models with detected capabilities/v1/models/{id} Single-model lookup with full metadata/v1/chat/completions Chat completion — streaming + non-streaming, think / thinking_budget/v1/completions Text completion/v1/embeddings Generate embeddings — batched, auto-chunked/v1/rerank Rerank documents (llama.cpp only — 501 elsewhere)// api in action
OpenAI-compatible. Drop in OPENAI_API_BASE=http://localhost:11430/v1 and call from any OpenAI SDK.
curl http://localhost:11430/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3:8b:4bit",
"messages": [
{"role": "user", "content": "Why is Rust good for inference servers?"}
],
"stream": true
}'data: {
"id": "chatcmpl-...",
"object": "chat.completion.chunk",
"model": "qwen3:8b:4bit",
"choices": [{
"index": 0,
"delta": { "role": "assistant", "content": "Zero-cost" },
"finish_reason": null,
"logprobs": null
}]
}Reasoning streams live up to thinking_budget, then the final answer streams with reasoning frozen — reasoning_content and content stay separate.
curl http://localhost:11430/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3.5:4b:4bit",
"messages": [{"role": "user", "content": "Prove the Pythagorean theorem."}],
"think": true,
"thinking_budget": 4096,
"stream": true
}'Inputs over embed_batch_size are auto-chunked across multiple engine calls; usage.prompt_tokens and indices are re-merged transparently.
curl http://localhost:11430/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3-embed:0.6b:4bit",
"input": ["doc one", "doc two"]
}'{
"object": "list",
"model": "qwen3-embed:0.6b:4bit",
"data": [
{ "object": "embedding", "index": 0, "embedding": [0.012, -0.034, ...] },
{ "object": "embedding", "index": 1, "embedding": [0.045, 0.018, ...] }
],
"usage": { "prompt_tokens": 6, "total_tokens": 6 }
}Detected capabilities ride along on /v1/models — and stay current after upgrades thanks to detector self-heal.
curl -s http://localhost:11430/v1/models \
| jq '.data[] | {id, capabilities}'{
"id": "qwen3.5:4b:mtp:4bit",
"capabilities": {
"chat": true,
"thinking": true,
"mtp": true,
"vision": false,
"embeddings": false
}
}Idempotent — no-op if the model is already resident. Load progress streams on GET /lf/status/stream (SSE).
for m in qwen3.5:4b:4bit qwen3-vl:2b:4bit qwen3-embed:0.6b:8bit; do
curl -sS -X POST http://localhost:11430/lf/model/switch \
-H 'Content-Type: application/json' \
-d "{\"model\":\"$m\"}"
done// model catalog
A sample of the 130+ entry curated catalog. Each shortcut resolves to the right artifact for your engine — MLX on Apple Silicon, GGUF elsewhere — and you can always paste a Hugging Face repo directly.
| Shortcut | Role | Why pull it |
|---|---|---|
qwen3.5:9b:4bit | chat · thinking | flagship quality that fits consumer GPUs |
qwen3:8b:4bit | chat · thinking | solid all-rounder, the 5-minute-eval default |
gemma3:12b:4bit | chat | strong general chat at 12B |
qwen3:4b:thinking:4bit | reasoning | native reasoning — think control locked on |
phi4:4b:reasoning:4bit | reasoning | compact dedicated reasoner |
qwen3.5:4b:mtp:4bit | chat · MTP | speculative decoding — higher tokens/sec |
qwen3-vl:2b:4bit | vision | image chat — URL or base64 inputs |
qwen3-embed:0.6b:8bit | embeddings | RAG building block, batched /v1/embeddings |
qwen3-reranker:0.6b:8bit | rerank | /v1/rerank for retrieval pipelines |
Ready when you are