Edge0 releases open-source 145B and 120B on-device AI models
Reported by Hacker News and r/LocalLLaMA3 sources ↓
Edge0 says it has open-sourced its 145B and 120B models, along with an inference framework and deployment tools for running them locally.
Key points
- Edge0 reports 80+ tokens per second and 28GB peak memory for the 145B model on a laptop
- The 145B model has about 10B parameters active per inference step
- Edge0 says the releases include model weights, an inference framework and deployment tools
Sources
Details
Edge0 says the models are intended to run on personal computers, and that developers can reproduce results, evaluate or fine-tune them, and integrate them into products. The company also points users to an Edge0 app; the articles do not provide pricing or specific availability details.
According to Edge0's internal testing, the 145B model reaches more than 80 tokens per second with 28GB peak memory on a laptop. Its report describes a design that shares matrices across sparse experts and coordinates GPU memory, system memory and SSD access; the reported measurements are company figures, not independent tests.
Edge0 reports scores of 86.52 on Terminal-Bench 2.1 and 79.3 on AA-LCR, comparing the results with selected cloud models. The supplied text says these evaluations are internal and does not provide enough of the benchmark table to assess all comparisons.
Written from coverage as of
Images
- Hacker News
- Hacker News
- Hacker News
- Hacker News