freetoken/README.md

A local, MoE-offload inference runtime with an OpenAI- and Anthropic-compatible HTTP API — Run DeepSeek-V4-Flash on your 5090 with 20+ TPS.

Quick start

See docs/install.md for requirements and installation.

ft serve --model ~/models/Qwen3.6-35B-A3B   # API server on http://127.0.0.1:1919
ft launch claude                            # point an agent at it (codex / opencode / openclaw)
ft shell                                    # or chat in the terminal

Documentation

• Install — requirements and setup
• Supported models — model × quantization
• CLI reference — ft commands and environment variables

Acknowledgment

FreeToken was deeply inspired by mini-sglang, and learned the design and reused code from the following projects: SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp.

License

Apache License 2.0.