jason-fxz
12ed695386
The request ring only carried whole-request duration, so /v1/stats exposed no first-token latency — the number that says how responsive a local model feels, independent of decode speed. Streaming generations now stamp the first delta they emit; non-streaming rows carry None and are skipped by the mean, so a /health probe can never drag it toward zero. Surfaced as requests.ttft_mean_ms in /v1/stats and in `ft ctl stats`.
| Download | Paper | Developer Slack | Community Discord |
A local, MoE-offload inference runtime with an OpenAI- and Anthropic-compatible HTTP API — Run DeepSeek-V4-Flash on your 5090 with 20+ TPS.
Quick start
See docs/install.md for requirements and installation.
ft serve --model ~/models/Qwen3.6-35B-A3B # API server on http://127.0.0.1:1919
ft launch claude # point an agent at it (codex / dsh / hermes / opencode / openclaw)
ft shell # or chat in the terminal
Documentation
- Install — requirements and setup
- Supported models — model × quantization
- CLI reference —
ftcommands and environment variables
Acknowledgment
FreeToken was deeply inspired by mini-sglang, and learned the design and reused code from the following projects: SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp.