Image missing.
Distillation Can Make AI Models Smaller and Cheaper

Amos Zeeberg

created: Sept. 20, 2025, 11 a.m. | updated: Oct. 2, 2025, 8:58 a.m.

Sources alleged that DeepSeek had obtained, without permission, knowledge from OpenAI’s proprietary o1 model by using a technique known as distillation. But distillation, also called knowledge distillation, is a widely used tool in AI, a subject of computer science research going back a decade and a tool that big tech companies use on their own models. Perhaps a smaller “student” model could use the information from the large “teacher” model to more quickly grasp the categories it was supposed to sort pictures into. After discussing this possibility with Hinton, Vinyals developed a way to get the large teacher model to pass more information about the image categories to a smaller student model. Many researchers turned to distillation as a way to make smaller models.

5 months, 1 week ago: Science Latest