Researchers at Stanford Introduce Score Entropy Discrete Diffusion (SEDD): A Machine Learning Model that Challenges the Autoregressive Language Paradigm and Beats GPT-2 on Perplexity and Quality

Recent advancements in the field of Artificial Intelligence and Deep Learning have made remarkable strides, especially in generative modelling, which is a subfield of Machine Learning where models are trained to produce new data samples that match the training data. Significant progress has been made with this strategy, in the creation of generative AI systems. These systems have demonstrated amazing capabilities, such as creating images from written descriptions and figuring out challenging problems.

The idea of probabilistic modeling is essential to the performance of deep generative models. Autoregressive modeling has been significant in the field of Natural Language Processing (NLP). This technique is based on the probabilistic chain rule and breaks down a sequence into the probabilities of each of its individual components in order to forecast the likelihood of the sequence. However, autoregressive transformers have several intrinsic drawbacks, like the output’s difficult control and delayed text production.

Researchers have been looking into different text generation models in an effort to overcome these restrictions. Text generation has been adopted from diffusion models, which have demonstrated tremendous promise in image production. These models replicate the opposite process of diffusion by gradually converting random noise into organized data. But in terms of speed, quality, and efficiency, these methods have not yet been able to outperform autoregressive models despite significant attempts.

In order to address the limitations of both autoregressive and diffusion models in text generation, a team of researchers has introduced a unique model named Score Entropy Discrete Diffusion models (SEDD). Using a loss function called score entropy, SEDD innovates by parameterizing a reverse discrete diffusion process based on ratios in the data distribution. This approach has been adapted for discrete data such as text and has been inspired by score-matching algorithms seen in typical diffusion models.

SEDD performs as well as existing language diffusion models for essential language modeling tasks and can even compete with conventional autoregressive models. In zero-shot perplexity challenges, it outperforms models such as GPT-2, proving its amazing efficiency. The team has shared that it performs exceptionally well in producing unconditionally high-quality text samples, enabling a compromise between processing capacity and output quality. SEDD is remarkably efficient as it can accomplish results that are comparable to those of GPT-2 with a lot less computational power.

SEDD also provides previously unheard-of control over the text production process by explicitly parameterizing probability ratios. It performs remarkably well in conventional and infill text generation scenarios compared to both diffusion models and autoregressive models using strategies like nucleus sampling. It allows text generation from any starting point without the requirement for specialized training.

In conclusion, the SEDD model challenges the long-standing supremacy of autoregressive models and marks a significant improvement in generative modeling for Natural Language Processing. Its capacity to produce text of excellent quality quickly and with more control creates new opportunities for AI.

Check out the Paper, Github, and Blog. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 38k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.