Overcoming catastrophic forgetting in molecular property prediction using continual learning of sequential episodes

Abstract

Continual Learning requires Large Language Models (LLM) to adapt to new episodes and data over time without forgetting the knowledge acquired from previous episodes. This dynamic approach to learning is crucial in molecular property prediction, where data is not static but arrives in streams. LLMs are prone to Catastrophic Forgetting (CF), where learning new information leads to erosion of previously acquired knowledge. This is particularly problematic in scenarios where the chemical, genomic, and proteomic data distribution shifts or new episodes differ significantly from prior ones. In response to the above issue, this work proposes a Multi-task Learner with Online Elastic Weight Consolidation and LLMs (Bidirectional Encoder Representation of Transformer (BERT) and Bidirectional Autoregressive Transformers (BART)) called Bidirectional Multi-Task Learner Elastic Weight Consolidation (B-MTLEWC). Two molecular datasets used for sequential learning of episodes are trained for 2000 epochs, which generated acceptable results on the B-MTLEWC model (1. Blood Brain Barrier Peptides (BBBP): Accuracy 89.16% and 88.81% and 2. Bitter: Accuracy 85.82% and 87.06% on unmasked and masked sets, respectively). The B-MTLEWC model is evaluated against BERT and BART across twelve augmented test datasets spanning three distinct episodes, focusing on knowledge retention from previous episodes. The model shows a maximum performance drop of only 1% in accuracy and Area Under the Curve, and in some cases, its performance remained consistent, demonstrating effective mitigation of CF. The empirical analysis combined with various explainability techniques highlighted significant performance improvements in sequential learning of episodes compared to State-of-the-Art methods, hence addressing the stability-plasticity trade-off. © 2024 Elsevier Ltd