Hardware-Aware LLM Inference Orchestrator
LMForge is a persistent daemon that picks the right engine for your hardware (Apple MLX, SGLang, llama.cpp), manages VRAM, and exposes a single OpenAI-compatible REST API to every app on your machine. Closing the desktop UI never stops your models.
// by the numbers
// install
curl -fsSL https://github.com/phoenixtb/lmforge/releases/latest/download/install-core.sh | bashcurl -fsSL https://github.com/phoenixtb/lmforge/releases/latest/download/install-ui.sh | bashlmforge pull qwen3:8b:4bit
lmforge run qwen3:8b:4bitThe installer drops the binary in /usr/local/bin, registers a launchd agent (macOS) or systemd --user unit (Linux), and starts the daemon. No sudo required.
// screenshots
// features
Probes the host at startup and picks the best inference backend automatically — no engine config to maintain.
Apple Silicon → oMLX (Metal/MLX, OpenAI-compatible server). Linux NVIDIA → SGLang (CUDA, high-concurrency). Windows NVIDIA, ARM Linux, or CPU-only → llama.cpp with auto -ngl tuning. Engine choice is hardware-driven, not user-configured; users get one OpenAI API regardless of what's underneath.
Run chat, embedding, vision, and reranker models simultaneously — each in its own engine subprocess, each with its own keep-alive lifecycle.
One engine subprocess per loaded model on a unique TCP port. The EngineManager owns a slot table, supervises the children, broadcasts state changes over SSE + Tauri IPC, and evicts the LRU slot when a new load would exceed the VRAM budget. oMLX manages its own residency; the Rust keepalive timer is skipped for that engine.
Native two-call workflow for Qwen3 / DeepSeek-R1 style reasoners — control how long the model thinks before it answers.
Call 1 streams reasoning tokens up to `thinking_budget`. When the budget is exhausted, LMForge appends the accumulated reasoning as a closed `<think>…</think>` turn and issues call 2 with `enable_thinking: false`. Live reasoning deltas stream to the client during call 1; `call2_prefill` SSE event marks the answer phase for UI feedback.
Multimodal requests with server-side URL fetching, real User-Agent, size caps, and capability gating before the engine spins up.
Remote `image_url` references are fetched server-side and rewritten as inline `data:` URLs before reaching the engine — hosts that block empty UAs (Wikimedia, several CDNs) no longer cause silent hallucinations. Sending an image to a non-vision model returns a 400 with `vision_not_supported` before any subprocess work happens. Live counters for accepted, rejected, and data-URL image inputs surface in the observability dashboard.
Loads models up to the detected VRAM budget; evicts least-recently-used when new loads don't fit.
Drop-in replacement for both ecosystems. Point `OPENAI_API_BASE=http://127.0.0.1:11430/v1` and you're done.
launchd on macOS, systemd --user on Linux, Scheduled Task on Windows — all via `lmforge service install`. No root.
`/metrics` for scrapers, `/lf/status/stream` for UIs. Per-endpoint latency, model load history, image preflight mix, auth rejections.
`lmforge start` is safe to call from any script — no-ops if the daemon is already running.
Compared against HuggingFace's `X-Linked-Etag` for LFS files. Corrupt downloads are auto-deleted.
// api in action
OpenAI-compatible. Drop in `OPENAI_API_BASE=http://127.0.0.1:11430/v1` and call from any OpenAI SDK.
curl http://127.0.0.1:11430/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3:8b:4bit",
"messages": [
{"role": "user", "content": "Why is Rust good for inference servers?"}
],
"stream": true
}'data: {
"id": "chatcmpl-...",
"object": "chat.completion.chunk",
"model": "qwen3:8b:4bit",
"choices": [{
"index": 0,
"delta": { "role": "assistant", "content": "Zero-cost" },
"finish_reason": null,
"logprobs": null
}]
}Two-call workflow — call 1 streams reasoning tokens up to `thinking_budget`; call 2 streams the final answer with reasoning frozen.
curl http://127.0.0.1:11430/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3.5:4b:4bit",
"messages": [{"role": "user", "content": "Prove the Pythagorean theorem."}],
"think": true,
"thinking_budget": 4096,
"stream_reasoning_deltas": true,
"stream": true
}'Inputs over `embed_batch_size` are auto-chunked across multiple engine calls; `usage.prompt_tokens` and indices are re-merged transparently.
curl http://127.0.0.1:11430/v1/embeddings \
-H 'Content-Type: application/json' \
-d '{
"model": "qwen3-embed:0.6b:4bit",
"input": ["doc one", "doc two"]
}'{
"object": "list",
"model": "qwen3-embed:0.6b:4bit",
"data": [
{ "object": "embedding", "index": 0, "embedding": [0.012, -0.034, ...] },
{ "object": "embedding", "index": 1, "embedding": [0.045, 0.018, ...] }
],
"usage": { "prompt_tokens": 6, "total_tokens": 6 }
}Idempotent — no-op if the model is already resident. Load progress streams on `GET /lf/status/stream` (SSE).
for m in qwen3.5:4b:4bit qwen2.5-vl:7b:4bit qwen3-embed:0.6b:8bit; do
curl -sS -X POST http://127.0.0.1:11430/lf/model/switch \
-H 'Content-Type: application/json' \
-d "{\"model\":\"$m\"}"
done// architecture
A single Rust binary hosts an axum HTTP server, an EngineManager that supervises engine subprocesses (one per loaded model), and a ModelIndex over the on-disk model store. Clients of every flavour speak HTTP to 127.0.0.1:11430; the daemon proxies their requests to the appropriate subprocess. The diagram-as-code source lives in the LMForge repo at `docs/architecture/ARCHITECTURE.md`.
// platform support
| Platform | Architecture | Engine | Core | Desktop UI |
|---|---|---|---|---|
| macOS 13+ | Apple Silicon (arm64) | oMLX (Metal/MLX) | ✓ | ✓ DMG |
| Ubuntu 22.04+ | x86_64 | SGLang (NVIDIA, 24GB+) / llama.cpp | ✓ | ✓ AppImage |
| Ubuntu 22.04+ | arm64 | llama.cpp | ✓ | 🔜 Planned |
| Windows 10/11 | x86_64 | llama.cpp (CPU + NVIDIA CUDA) | ✓ | ✓ NSIS |
| Windows + WSL2 | x86_64 | SGLang (CUDA via WSL) | ✓ (inside WSL) | via Linux build |
// model catalog
Same shortcut, hardware-specific repo. MLX on Apple Silicon; GGUF elsewhere.
| Shortcut | macOS (MLX) | Linux / Windows (GGUF) |
|---|---|---|
qwen3:8b:4bit | mlx-community/Qwen3-8B-4bit | bartowski/Qwen3-8B-GGUF |
qwen3.5:4b:4bit | mlx-community/Qwen3.5-4B-4bit | Qwen/Qwen3.5-4B-GGUF |
qwen3.5:9b:4bit | mlx-community/Qwen3.5-9B-OptiQ-4bit | bartowski/Qwen3.5-9B-OptiQ-GGUF |
gemma3:4b:4bit | mlx-community/gemma-3-4b-it-4bit | bartowski/gemma-3-4b-it-GGUF |
gemma3:12b:4bit | mlx-community/gemma-3-12b-it-4bit | bartowski/gemma-3-12b-it-GGUF |
llama3.1:8b:4bit | mlx-community/Meta-Llama-3.1-8B-Instruct-4bit | bartowski/Meta-Llama-3.1-8B-Instruct-GGUF |
llama4:17b:4bit:scout | mlx-community/Llama-4-Scout-17B-16E-Instruct-4bit | bartowski/Llama-4-Scout-17B-16E-Instruct-GGUF |
phi4:4b:4bit | mlx-community/Phi-4-mini-instruct-4bit | bartowski/Phi-4-mini-instruct-GGUF |
deepseek_r1:8b:4bit:distill-qwen | mlx-community/DeepSeek-R1-Distill-Qwen-8B-4bit | unsloth/DeepSeek-R1-Distill-Qwen-8B-GGUF |
// api surface
/v1/models List available models with capabilities/v1/models/{id} Single-model lookup with full metadata/v1/chat/completions Chat completion — streaming + non-streaming/v1/completions Text completion/v1/embeddings Generate embeddings — batched, auto-chunked/v1/rerank Rerank documents (llama.cpp only — 501 elsewhere)// user journeys
Zero-API integration — just install and chat.
Reuse your existing OpenAI SDK code.
Run multiple models concurrently for a RAG pipeline.
// tech stack
Ready when you are