Researchers from Qualcomm AI Research Introduced CodeIt: Combining Program Sampling and Hindsight Relabeling for Program Synthesis

Programming by example is one of the diverse fields of Artificial intelligence (AI) in automation processes. The goal is to generate programs to solve tasks based on input-output examples. This domain presents a unique challenge as it demands a system that can understand the underlying patterns in the data and apply reasoning to extrapolate these patterns to unseen examples.

Despite their advancements, current methods for programming-by-example often fall short when faced with tasks that require high levels of abstraction and reasoning. The complexity of these tasks lies in their requirement for a solution that can generalize from a limited set of examples to a broad range of unseen scenarios. This problem is exemplified in benchmarks like the Abstraction and Reasoning Corpus (ARC), which tests AI systems’ ability to apply core knowledge systems—objects, actions, numbers, and space—in novel ways.

Existing approaches to tackle these challenges can be categorized into neural and neuro-symbolic methods. Neural approaches attempt to directly predict output grids from input grids using deep learning models. On the other hand, neuro-symbolic methods first aim to understand the mapping between input and output grids through symbolic representations, such as programs, before generating the desired outputs. Each approach has its merits but often needs help with task generalization due to the sparsity of rewards in program synthesis.

Researchers from the University of Amsterdam have introduced a novel method called Code Iteration (CodeIt) to address these challenges. CodeIt iterates between program sampling with hindsight relabeling and learning from prioritized experience replay. This method enables the model to refine its understanding and improve its predictions through self-improvement, leveraging the vast capabilities of pre-trained language models while addressing the issue of reward sparsity.

The study tackles the ARC challenge by framing it as a programming-by-examples issue. It employs a two-stage method: program generation through policy application with hindsight relabeling and iterative learning from input-output pairs. The approach emphasizes object-centric grid representation for efficient learning by utilizing Hodel’s open-source Domain-specific language (DSL) for grid manipulation and the pretrained CodeT5+ LLM for policy creation. The CodeIt Algorithm, underpinned by a robust training regimen involving 400 ARC training examples and an expanded dataset of 19,200 program samples, demonstrates notable efficacy.

The implementation of CodeIt on the ARC dataset showcased remarkable results. With its state-of-the-art performance, CodeIt solved 15% of the ARC evaluation tasks, outperforming existing neural and symbolic baselines. The method of iterating between program sampling, hindsight relabeling, and learning from prioritized experience replay effectively dealt with the extreme sparsity of rewards in program synthesis.

The exploration and development of self-improving AI systems like CodeIt represent a promising direction in addressing complex problem-solving tasks that require abstract reasoning. By harnessing the power of hindsight replay and prioritized learning, CodeIt illustrates the potential of neuro-symbolic approaches in advancing our understanding and capabilities in AI. As the field continues to evolve, the principles underlying CodeIt could pave the way for more intelligent and adaptable AI systems.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 37k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.