Salesforce AI Researchers Propose BootPIG: A Novel Architecture that Allows a User to Provide Reference Images of an Object in Order to Guide the Appearance of a Concept in the Generated Images

Personalized image generation is the process of generating images of certain personal objects in different user-specified contexts. For example, one may want to visualize the different ways their pet dog would look in different scenarios. Apart from personal experiences, this method also has use cases in personalized storytelling, interactive designs, etc. Although current text-to-image generation models have demonstrated exceptional performance, they fail to personalize the image generation as per the specific subject and often fall short in terms of faithfulness to the reference object.

In this research paper, a team of researchers from Salesforce AI have tried to address the above issues and have introduced a novel architecture, BootPIG, which enables personalized image generation capabilities in text-to-image models. The idea behind the architecture is to insert the appearance of the reference object into the features of a pretrained diffusion model so that the generated images mimic the reference object. This process is done by replacing all the self-attention (SA) layers with an operation that the authors refer to as reference self-attention (RSA).

BootPIG has been built on top of existing diffusion models, and its architecture consists of two replicas of a latent diffusion model: Reference UNet and Base UNet. The former is used to process the reference image and collect its features before each SA layer. The SA layers of the Base UNet are modified to RSA layers, and it uses the reference features as input and guides the image generation toward the reference object.

For training BootPIG, the researchers used an automated synthetic data generation pipeline leveraging the capabilities of ChatGPT, Stable Diffusion, and the Segment Anything model. ChatGPT is used to generate captions, Stable Diffusion for image generation, and the Segment Anything model to segment the image’s foreground, which is then used as the reference image. Most importantly, it can be trained in just 1 hour, approximately.

For evaluation, the authors compared BootPIG’s performance with that of existing methods like BLIP-Diffusion, ELITE, and Dreambooth. Qualitative comparison results show that BootPIG outperforms the other methods regarding subject and prompt fidelity and avoids test-time finetuning. Furthermore, human evaluation highlights the superiority of BootPIG over other methods. Human evaluators consistently preferred the framework’s generated images and found a significantly greater subject and caption fidelity.

BootPIG also has some limitations that are common to existing methods. In many cases, it fails to render the fine details of the subject and struggles to adhere strictly to the user prompt. However, some of its failures are also inherited from underlying models. Nevertheless, BootPIG shows impressive results when it comes to personalized image generation. The authors believe that their method can help learn new capabilities and unlock other modalities of image generation.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 36k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.