Leveraging State Space Models in Long Range Genomics
Published in ICLR 2025 (LMRL) Spotlight , 2025
Long-range dependencies are crucial for interpreting genomic structure and function, yet conventional transformer-based genomics models often fail to generalize beyond their training window even when employing sophisticated positional embeddings. We show that State-Space Models (SSMs) can perform as well as the best transformer based models on a range of biologically relevant tasks, while also being able to zero-shot extrapolate two orders of magnitude beyond their original context length, thus capturing distal regulatory interactions required for gene expressions without specialized fine-tuning. With our hidden-state transfer mechanism, we can efficiently process ultralong genomic sequences (1Mbp) on a single GPU—providing a scalable, generalizable, and resource-efficient approach to push the frontier in genomic modeling.
Project Website: https://anirudharamesh.github.io/iclr-long-range-genomics/
Video: Watch Talk