Tokasaurus: An LLM Inference Engine for High-Throughput WorkloadsTL;DRWe’re releasing Tokasaurus, a new LLM inference engine optimized for throughput-intensive workloads. With small models, Tokasaurus benefits from very low CPU overhead and dynamic Hydragen grouping to exploit shared prefixes. For larger models, Tokasaurus supports async tensor parallelism for GPUs with NVLink and a fast implementation of pipeline parallelism for GPUs without. Open-source inference engines (i.e. Optimizing Small ModelsTo benchmark Tokasaurus with small models, we’ll use two workloads:Completing chatbot prompts from the ShareGPT dataset (this is a common benchmark for testing inference engines).