Edge0-35B-A3B-preview

Edge0-35b-a3b Preview

A 35B-class sparse MoE that runs in phone-class memory.

3 GiB active memory · 15 tok/s · 4-bit

GitHub Hugging Face Hugging Face ModelScope ModelScope arXiv License

Edge0-35b-a3b — a 35B MoE LLM that runs at viable speed in under 3 GiB of active memory, via the edge0 streaming inference framework.

Preview status: this is an early preview release of the edge0 pipeline. The checkpoint ships as int4 quantization plus LoRA and prerouter adapters trained for this framework.

Highlights

• Runs in phone-class memory: the full 4-bit checkpoint stays on storage and experts are streamed on demand, so only the active weights are in RAM — under 3 GiB, with no sharding and no upfront download of the weights into memory.
• Fast enough for interactive use: 15 tok/s decode; long prompts fill in at 140 tok/s.
• Quality kept after quantization: Recover-LoRA distillation keeps the int4 model within 3.9 points of its fp16 base.
• Works out of the box: base, LoRA and prerouter adapters ship together and load automatically via edge0.

Three mechanisms make this work:

• SSD expert offload: expert weights are streamed from storage on demand — fetched only as routed, so RAM holds just the active weights. Peak memory is bounded by the active set, not the parameter count.
• Prerouter: a trained head predicts expert routing one step ahead, so expert loads overlap the forward pass instead of stalling it — up to +59% decode throughput; the gain grows with storage latency, model size, and routed width K.
• Recover-LoRA: the int4 base is frozen and LoRA adapters are trained by distillation from the FP teacher, recovering most of the quantization loss at 4-bit (see Quality below). Adapters stay unmerged: one read-only base serves multiple adapter sets.

Model summary

The LoRA and prerouter adapters are co-located with the base checkpoint and load automatically — this repository is a complete, ready-to-run model directory for edge0.

Quality

All benchmarks were run by us with OpenCompass under identical settings and parameters for both models. The loss of the edge0 pipeline (int4 + adapters) relative to the fp16 base model is small: 3.9 points on average. Max 100:

Performance

Measured with examples/bench.py on a Mac mini M4 Pro, 24 GB:

*Short contexts; long contexts add KV cache. Expert weights stream from SSD on demand and are not resident.

Use cases

• Edge / on-device inference where GPU VRAM is scarce and storage is fast (NVMe, internal flash).
• Batch serving on a single commodity machine — one read-only base serves many LoRA adapter sets without re-quantization.
• Multilingual chat and reasoning with thinking mode enabled by the bundled chat template.

Limitations

• Preview release: coverage and quality are still being extended; the model is primarily tuned for the languages of the base model.
• Agent capability: this preview release is not yet optimized for agentic tasks — tool use, multi-step planning, and long-horizon autonomy are currently weak. The full release will substantially strengthen agent capability.
• The MLX backend currently targets Apple Silicon; other backends are on the edge0 roadmap.
• Long contexts grow the KV cache; use shorter contexts to keep peak memory at 3 GiB.

Quick start

pip install -e 'git+https://github.com/Edge0-AI/edge0.git#egg=edge0[fetch]'

# Download this repository into a local directory
huggingface-cli download Edge0/Edge0-35b-a3b-preview --local-dir ./Edge0-35b-a3b-preview

# Run it
export EDGE0_35B_MODEL=$PWD/Edge0-35b-a3b-preview
edge0 chat --name edge0-35b --prompt "Introduce yourself"

# Or serve an OpenAI-compatible HTTP API
edge0 serve --name edge0-35b --port 8085

For full usage (Python API, streaming options, prerouter details), see the edge0 documentation.

License

Apache 2.0. See LICENSE.

Citation

If you find Edge0 useful in your research, please cite our paper:

@misc{lin2026halfmemorywallserving,
      title={The Other Half of the Memory Wall: Serving 35B MoEs from SSD with Trained Routing Prediction},
      author={Yu Lin and Yiming Wang and Runyuan Cai and Hanze Liu and Xiaodong Zeng},
      year={2026},
      eprint={2609.18063},
      archivePrefix={arXiv},
      primaryClass={cs.AI},
      url={https://arxiv.org/abs/2609.18063},
}