Pete warden on local AI, open models, voice interfaces, the future of hardware
Reported by Hacker News1 source ↓
In a podcast interview, Pete Warden discussed running AI locally, open-weight models, on-device speech and hardware. He said local models are now capable of common tasks and can trail leading commercial models by only a few months.
Key points
- Warden said 9-billion-parameter models can fit in about 4 GB of graphics memory with four-bit quantization
- Warden said local models perform well on benchmarks and can be a few months behind leading commercial models
- He named Ollama as a tool that makes running models locally relatively straightforward
Sources
Details
The interview covered running models on laptops and in browsers, open-weight models, voice interfaces, the economics of local AI and potential alternatives to GPU-focused hardware.
Warden said model choice and graphics memory determine what a laptop can run. He gave a 9-billion-parameter model as an example: at eight bits, its weights require about 9 GB, while four-bit quantization can bring the requirement to roughly 4 GB. He said that amount of graphics memory is widely available, including in some older laptops.
Warden said local models perform well on benchmarks and are only a few months behind leading commercial models, while being capable of tasks people want to use them for. He also argued that commercial AI companies have less incentive to promote local models because they do not generate subscription revenue in the same way as cloud services.
Written from coverage as of
Images
- Hacker News