A unified transformer architecture for 3D MSI-H&E fusion

Hassan Imani; Niladri Das; Iain Styles

doi:10.1109/ISBI61048.2026.11515580

A unified transformer architecture for 3D MSI-H&E fusion

School of Electronics, Electrical Engineering and Computer Science

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

31 Downloads (Pure)

Abstract

Mass spectrometry imaging (MSI) enables unlabeled, untargeted mapping of biomolecular distributions in tissue but is limited by low spatial resolution and high data complexity. We address these challenges with a generative model for MSI peak learning that produces compact, chemically meaningful embeddings, and a transformer-based framework for multimodal fusion of low-resolution MSI with high-resolution H&E microscopy across serial sections. The model tokenizes MSI and H&E patches and employs deformable cross-attention to transfer histological structure while preserving MSI chemistry, yielding slice-consistent high-resolution 3D reconstructions. Experiments on 3D DESI Orbitrap MSI/H&E data from human colorectal adenocarcinoma demonstrate anatomically accurate and chemicallyfaithful reconstruction without manual intervention.
Code is available at https://github.com/H-deep/TransUniFERE-3D.

Original language	English
Title of host publication	2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI 2026): Proceedings
Publisher	IEEE
Number of pages	5
ISBN (Electronic)	9798331577636
ISBN (Print)	9798331577643
DOIs	https://doi.org/10.1109/ISBI61048.2026.11515580
Publication status	Published - May 2026
Event	23rd International Symposium on Biomedical Imaging (ISBI 2026) - London, United Kingdom Duration: 08 Apr 2026 → 11 Apr 2026

Publication series

Name	IEEE International Symposium on Biomedical Imaging (ISBI): Proceedings
ISSN (Electronic)	1945-8452

Conference

Conference	23rd International Symposium on Biomedical Imaging (ISBI 2026)
Country/Territory	United Kingdom
City	London
Period	08/04/2026 → 11/04/2026

Publications and Copyright Policy

This work is licensed under Queen’s Research Publications and Copyright Policy. 

Access to Document

10.1109/ISBI61048.2026.11515580Licence: Unspecified

A unified transformer architecture for 3D MSI-H&E fusion
Copyright 2026 the authors. This is an accepted manuscript distributed under a Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution and reproduction in any medium, provided the author and source are cited.
Accepted author manuscript, 9.44 MBLicence: CC BY

R1129ECI: Kelvin-2 - The High Performance Computing Centre in Northern Ireland (HPC-NI)
Chevallier, O. (CoI), Gillan, C. (CoI), Hu, P. (CoI), Rafferty, K. (CoI), Salto-Tellez, M. (CoI), Tikhonova, I. (CoI), Vandierendonck, H. (CoI) & Woods, R. (CoI)
04/12/2019 → …
Project: Research

Cite this

@inproceedings{a871b07491364602b31833a733812327,

title = "A unified transformer architecture for 3D MSI-H\&E fusion",

abstract = "Mass spectrometry imaging (MSI) enables unlabeled, untargeted mapping of biomolecular distributions in tissue but is limited by low spatial resolution and high data complexity. We address these challenges with a generative model for MSI peak learning that produces compact, chemically meaningful embeddings, and a transformer-based framework for multimodal fusion of low-resolution MSI with high-resolution H\&E microscopy across serial sections. The model tokenizes MSI and H\&E patches and employs deformable cross-attention to transfer histological structure while preserving MSI chemistry, yielding slice-consistent high-resolution 3D reconstructions. Experiments on 3D DESI Orbitrap MSI/H\&E data from human colorectal adenocarcinoma demonstrate anatomically accurate and chemicallyfaithful reconstruction without manual intervention. Code is available at https://github.com/H-deep/TransUniFERE-3D.",

author = "Hassan Imani and Niladri Das and Iain Styles",

year = "2026",

month = may,

doi = "10.1109/ISBI61048.2026.11515580",

language = "English",

isbn = "9798331577643",

series = "IEEE International Symposium on Biomedical Imaging (ISBI): Proceedings",

publisher = "IEEE",

booktitle = "2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI 2026): Proceedings",

note = "23rd International Symposium on Biomedical Imaging (ISBI 2026) ; Conference date: 08-04-2026 Through 11-04-2026",

}

Imani, H , Das, N & Styles, I 2026, A unified transformer architecture for 3D MSI-H&E fusion. in 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI 2026): Proceedings. IEEE International Symposium on Biomedical Imaging (ISBI): Proceedings, IEEE, 23rd International Symposium on Biomedical Imaging (ISBI 2026), London, United Kingdom, 08/04/2026. https://doi.org/10.1109/ISBI61048.2026.11515580

TY - GEN

T1 - A unified transformer architecture for 3D MSI-H&E fusion

AU - Imani, Hassan

AU - Das, Niladri

AU - Styles, Iain

PY - 2026/5

Y1 - 2026/5

N2 - Mass spectrometry imaging (MSI) enables unlabeled, untargeted mapping of biomolecular distributions in tissue but is limited by low spatial resolution and high data complexity. We address these challenges with a generative model for MSI peak learning that produces compact, chemically meaningful embeddings, and a transformer-based framework for multimodal fusion of low-resolution MSI with high-resolution H&E microscopy across serial sections. The model tokenizes MSI and H&E patches and employs deformable cross-attention to transfer histological structure while preserving MSI chemistry, yielding slice-consistent high-resolution 3D reconstructions. Experiments on 3D DESI Orbitrap MSI/H&E data from human colorectal adenocarcinoma demonstrate anatomically accurate and chemicallyfaithful reconstruction without manual intervention. Code is available at https://github.com/H-deep/TransUniFERE-3D.

AB - Mass spectrometry imaging (MSI) enables unlabeled, untargeted mapping of biomolecular distributions in tissue but is limited by low spatial resolution and high data complexity. We address these challenges with a generative model for MSI peak learning that produces compact, chemically meaningful embeddings, and a transformer-based framework for multimodal fusion of low-resolution MSI with high-resolution H&E microscopy across serial sections. The model tokenizes MSI and H&E patches and employs deformable cross-attention to transfer histological structure while preserving MSI chemistry, yielding slice-consistent high-resolution 3D reconstructions. Experiments on 3D DESI Orbitrap MSI/H&E data from human colorectal adenocarcinoma demonstrate anatomically accurate and chemicallyfaithful reconstruction without manual intervention. Code is available at https://github.com/H-deep/TransUniFERE-3D.

U2 - 10.1109/ISBI61048.2026.11515580

DO - 10.1109/ISBI61048.2026.11515580

M3 - Conference contribution

SN - 9798331577643

T3 - IEEE International Symposium on Biomedical Imaging (ISBI): Proceedings

BT - 2026 IEEE 23rd International Symposium on Biomedical Imaging (ISBI 2026): Proceedings

PB - IEEE

T2 - 23rd International Symposium on Biomedical Imaging (ISBI 2026)

Y2 - 8 April 2026 through 11 April 2026

ER -

A unified transformer architecture for 3D MSI-H&E fusion

Abstract

Publication series

Conference

Publications and Copyright Policy

Access to Document

Fingerprint

Projects

R1129ECI: Kelvin-2 - The High Performance Computing Centre in Northern Ireland (HPC-NI)

Cite this