| Title |
Inference acceleration for large language models using “stairs” assisted greedy generation / |
| Authors |
Grigaliūnas, Domas ; Lukoševičius, Mantas |
| Full Text |
|
| Is Part of |
CEUR workshop proceedings: IVUS 2024: Information society and university studies 2024: proceedings of the 29th international conference on information society and university studies (IVUS 2023) Kaunas, Lithuania, May 17, 2024 / edited by: I. Veitaitė, A. Lopata, T. Krilavičius, M. Woźniak.. Aachen : CEUR-WS. 2024, vol. 3885, p. 165-174.. ISSN 1613-0073 |
| Keywords [eng] |
large language model ; inference ; assisted generation |
| Abstract [eng] |
Large Language Models (LLMs) with billions of parameters are known for their impressive predicting capabilities but require lots of resources to run. With their massive rise in popularity, even a small reduction in required resources could have an impact on environment. On the other hand, smaller models require fewer resources but may sacrifice accuracy. In this work, we are proposing an implementation of “stairs” assisted greedy generation. It is a modified assisted generation methodology that makes use of a smaller model’s fast generation, large model’s batch prediction, and “stairs” validation in order to achieve a speed up in prediction generation. Results show between 9.58 and 17.24 percent inference time reduction compared to a stand-alone large LLM prediction in a text generation task without a loss in accuracy. |
| Published |
Aachen : CEUR-WS |
| Type |
Conference paper |
| Language |
English |
| Publication date |
2024 |
| CC license |
|
