Meet Medusa: An Efficient Machine Learning Framework for Accelerating Large Language Models (LLMs) Inference with Multiple Decoding Heads

The newest development within the area of Synthetic Intelligence (AI), i.e., Giant Language Fashions (LLMs), has demonstrated some nice enchancment in language manufacturing. With mannequin sizes reaching billions of parameters, these fashions are moving into each area, starting from healthcare and finance to training.

Although these fashions have proven wonderful capabilities, the event of the mannequin’s dimension has led to an elevated inference latency, which poses an issue for real-world purposes. Reminiscence-bound operations signify the principle bottleneck in LLM inference, as it’s inefficient to move all mannequin parameters from Excessive Bandwidth Reminiscence (HBM) to the accelerator’s cache throughout auto-regressive decoding.

Researchers have been placing in efforts to discover a answer to those limitations, one in all which is to lower the variety of decoding steps and improve the arithmetic depth of the decoding course of. Utilizing a smaller draft mannequin for speculative decoding, which produces a collection of tokens which are then improved upon by the larger unique mannequin, has been recommended. Nonetheless, there are difficulties with incorporating a draft mannequin right into a distributed system.

To beat these challenges, a staff of researchers in a current examine has offered MEDUSA, an environment friendly strategy that enhances LLM inference by incorporating extra decoding heads to foretell a number of subsequent tokens in parallel. It makes use of the spine mannequin’s quite a few decoding heads to hurry up inference. These heads overcome the difficulties of speculative decoding by concurrently predicting quite a few tokens. 

MEDUSA doesn’t require a separate draft mannequin like speculative decoding requires, which makes it able to getting simply built-in into present LLM programs, even in dispersed conditions. The staff has shared that MEDUSA builds a number of candidate continuations in every decoding section and verifies them concurrently utilizing a tree-based consideration mechanism. By using parallel processing, MEDUSA lowers the variety of mandatory decoding steps whereas introducing little or no overhead when it comes to single-step latency.

Two new insights have been added to MEDUSA. First, quite a few candidate continuations have been generated utilizing MEDUSA heads, they usually have been verified concurrently. Secondly, an acceptance process has been used to decide on appropriate candidates. The staff has shared the rejection sampling technique utilized in speculative decoding, which a temperature-based threshold can successfully substitute to deal with deviations.

The examine has recommended two strategies for fine-tuning LLMs’ predictive MEDUSA heads, that are as follows.

1. MEDUSA-1: This enables lossless inference acceleration by immediately fine-tuning MEDUSA on high of a frozen spine LLM. MEDUSA-1 has been recommended for use when incorporating MEDUSA into an current mannequin or in settings with restricted computational assets. It makes use of much less reminiscence and might be made much more environment friendly by making use of quantization strategies.

2. MEDUSA-2: This methodology adjusts MEDUSA and the principle LLM concurrently. Whereas it affords a higher speedup and improved prediction accuracy for MEDUSA heads, it necessitates a singular coaching recipe to keep up the spine mannequin’s performance. MEDUSA-2 is suitable when assets are plentiful and permits simultaneous coaching of MEDUSA heads and the spine mannequin with out sacrificing output high quality or next-token prediction skill.

The analysis has additionally recommended a number of additions to reinforce or broaden the usage of MEDUSA. These embrace a traditional acceptance scheme to extend the acceptance fee with out sacrificing era high quality and a self-distillation methodology within the absence of coaching information. The staff has shared that the analysis technique of MEDUSA included testing on fashions of various sizes and coaching protocols. The outcomes have demonstrated that MEDUSA-1 can speed up information by greater than 2.2 occasions with out sacrificing era high quality. Furthermore, the acceleration is improved to 2.3-3.6× utilizing MEDUSA-2. 

Try the Paper and Github. All credit score for this analysis goes to the researchers of this venture. Additionally, don’t overlook to observe us on Twitter. Be part of our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.

For those who like our work, you’ll love our newsletter..

Don’t Overlook to hitch our Telegram Channel

Tanya Malhotra is a closing yr undergrad from the College of Petroleum & Power Research, Dehradun, pursuing BTech in Pc Science Engineering with a specialization in Synthetic Intelligence and Machine Studying.
She is a Information Science fanatic with good analytical and demanding considering, together with an ardent curiosity in buying new abilities, main teams, and managing work in an organized method.