Multi(Modal) and Longitudinal Data in Healthcare – Opportunities, Challenges, and Future

Catarina Barata

Instituto Superior Técnico, Univ. Lisboa, Portugal

Resumen/Abstract

We live in an increasingly digital world, where large amounts of information are generated and stored every day. One such example is the medical field, where technological advances have played a significant role, making it possible to digitalize and store healthcare data. Electronic health records (EHRs), which serve as digital repositories, allow the tracking of patients’ entire healthcare journeys across various settings and providers. Large biobanks offer complete populational healthcare views, covering specific areas like oncology and cardiology. Finally, the rising of public repositories for medical images and genome/microbiome sequencing facilitated the open access to healthcare data. All combined, we are living an exciting era for computer vision and machine learning (ML) in medicine, where researchers are moving from narrowly defined tasks, which used a single type of healthcare data, to broader challenges, such as predicting treatment responses or patient modeling across long time spans. This is only accomplished when integrating data from multiple sources, as well as from different time stamps, which poses new and exciting challenges for ML systems. Foundation models became a core element in data representation, while LLMs and VLMs are emerging as tools to integrate multiple types of data and model time progression. In this talk I will discuss the integration of multi healthcare data, from data representation to its fusion in multimodal and longitudinal ML systems, using examples of our recent work to showcase opportunities and obstacles currently faced by ML methods, while challenging the audience to consider what will be the future of healthcare and ML.

Curriculum ponente

Catarina Barata (CB) is an Assistant Professor at IST and Researcher at ISR, where she is currently a member of its scientific committee. She is a Scholar Member of the European Laboratory for Learning and Intelligent Systems (ELLIS) and its Lisbon Unit (LUMLIS). Her main line of research focuses on the application of machine learning models to societal problems, mainly healthcare, aiming for development of explainable artificial intelligence models. In 2021 she received a Google Research Award for her work in personalizing treatment for skin cancer patients, where she addressed the identification of molecular biomarkers across various imaging modalities. She participated in several research projects, leading two of them (DeepMutation and OptSurgAI) and co-PI of ISR for the Center for Responsible AI, a consortium of companies, startups and research units funded by the RRP. Presently she is the PI of one national project (MMIST) and the ISR-coordination of two European Projects (NextGen and MULTISCOPE), all focusing on the development of multimodal systems in different healthcare applications She currently supervises four PhD students (plus one concluded), having also supervised +45 master theses and one visiting professor, published +20 articles in scientific journals, and presented +40 papers at international conferences. She is involved in the organization of the annual ISIC Workshop at top conferences, including CVPR, ECCV, MICCAI.