Xiaoze Fan
2757bb5f91
* feat(server)!: add --gpu to choose the GPU on multi-GPU machines --gpu takes a GPU UUID (as nvidia-smi -L prints) or an nvidia-smi index. It is applied as CUDA_VISIBLE_DEVICES in the parent before the workers spawn, so the engine still binds cuda:<rank>. /v1/stats reports the engine's GPU under "gpus". ft bench bw takes the same --gpu and writes one profile per GPU (benchbw/<gpu-uuid>.json); the legacy benchbw.json is still read by GPU name. The daemon's /bench/profile returns the running serve's GPU profile. BREAKING CHANGE: ft checkpoint --device is removed; use --gpu. * refactor!: resolve --gpu via NVML and bind by UUID, not CUDA_VISIBLE_DEVICES BREAKING CHANGE: ft bench bw --device is removed; use --gpu. * refactor: name the id namespaces and drop the multi-device e4m3 scan Splits the published id into _assigned_physical (UUID) and _assigned_visible (CUDA ordinal). One process runs on one GPU, so e4m3_native() judges that card instead of scanning every visible device.
benchmarks
Run from the repo root with PYTHONPATH=python:., pinned to one GPU
(CUDA_VISIBLE_DEVICES=0). Each script's --help / docstring has the details.
bench_decode_moe.py — bs=1 decode tok/s of a served MoE model. Spawns ft serve
per backend and times token arrivals over streamed /v1/chat/completions, so numbers
include the full serving path. AIME-25 prompt, checkpoint-recommended sampling.
python benchmarks/bench_decode_moe.py --model /path/to/model --backend offload,cpu,hybrid
bench_load_weight_generic.py — expert-bank load time: serial vs parallel O_DIRECT
vs pre-repacked FTW, each mode in its own subprocess. Linux-only; stages the FTW under
/var/tmp (--ftw-dir overrides; roughly checkpoint-sized).
python benchmarks/bench_load_weight_generic.py --model /path/to/model
bench_offload_cache_copy.py — synthetic (no checkpoint): per-layer decode expert
copy cost (ensure_experts + copy_missing), swept over bank layout x cache slots x
batch size x miss rate.
python benchmarks/bench_offload_cache_copy.py
For host RAM vs PCIe bandwidth and the offload/hybrid backend pick, use ft bench bw
instead — it writes the JSON profile the engine reads.