KittenML says it released inferencekit, a free, open-source plugin that uses coding agents to optimize model inference for users’ hardware. The project reports about 10 ms streaming TTFB for Kitten-TTS-2 on an RTX Pro 6000.
KittenML describes inferencekit as a way to find inference settings suited to a user’s hardware, rather than relying on settings tuned for the hardware used by a model’s authors. It says users can submit optimized configurations to the inferencekit-engine repository for others with the same model and hardware to reuse.
The project reports that it used inferencekit to reduce streaming time to first byte for Kitten-TTS-2 to about 10 milliseconds on an RTX Pro 6000. The post does not provide details on how that result was measured.