Amazon AI Research Introduces BioBRIDGE: A Parameter-Efficient Machine Learning Framework to Bridge Independently Trained Unimodal Foundation Models to Establish Multimodal Behavior

In the interdisciplinary field of biomedical research, the advent of foundation models (FMs) has significantly enhanced our ability to process and analyze large volumes of unlabeled data across various tasks. Despite their prowess, FMs in the biomedical domain have largely been confined to unimodal applications, focusing on either protein sequences, small molecule structures, or clinical data in isolation. This narrow scope limits their potential, especially when considering the interconnected nature of biomedical knowledge.

Researchers from the University of Illinois Urbana-Champaign and Amazon AWS AI have developed BioBRIDGE, a parameter-efficient learning framework designed to unify independently trained unimodal FMs and establish multimodal behavior. This innovation is achieved by employing Knowledge Graphs (KGs) to learn transformations between unimodal FMs without fine-tuning the underlying models. The research demonstrates that BioBRIDGE can significantly outperform baseline KG embedding methods in cross-modal retrieval tasks by approximately 76.3%, showcasing an impressive ability to generalize across unseen modalities or relations.

The cornerstone of BioBRIDGE’s methodology is its use of biomedical KGs, which contain rich structural information represented by triplets of head and tail biomedical entities and their relationships. This structure enables the comprehensive analysis of various modalities such as proteins, molecules, and diseases. By aligning the embedding space of unimodal FMs through cross-modal transformation models utilizing KG triplets, BioBRIDGE maintains data sufficiency and efficiency and navigates the challenges posed by computational costs and data scarcity that hinder the scalability of multimodal approaches.

BioBRIDGE’s performance is evaluated through experiments demonstrating its competency in diverse cross-modal prediction tasks. It can extrapolate to nodes not present in the training KG and generalize to relationships absent from the training data. It introduces a novel application as a general-purpose retriever aiding in biomedical multimodal question answering and the guided generation of novel drugs.

BioBRIDGE efficiently bridges the gap between unimodal FMs, leveraging the rich structural information from KGs to facilitate cross-modal transformations. It demonstrates remarkable out-of-domain generalization ability, offering new pathways for integrating and analyzing multimodal biomedical data. The framework is a versatile tool that could significantly impact biomedical research, from enhancing question-answering systems to facilitating drug discovery.

In conclusion, BioBRIDGE represents a significant leap forward in applying foundation models for biomedical research, offering a scalable and efficient approach to integrating multimodal data. By bridging the gap between unimodal FMs and enabling their application across various domains without extensive retraining or data collection, this research paves the way for more holistic and interconnected analyses in the biomedical field. The potential of BioBRIDGE to extend to other domains, given a structured representation in KGs, sets the stage for future explorations and innovations in multimodal data integration and analysis.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, don’t forget to follow us on Twitter and Google News. Join our 38k+ ML SubReddit, 41k+ Facebook Community, Discord Channel, and LinkedIn Group.