Research

LLM-Supported Exploration of Highly Multiplexed Imaging

Description: Multiplexed immunofluorescence (mIF) imaging enables visualization of a large number of biomarkers simultaneously, offering high-resolution insight into spatial tissue architecture and immune landscapes. Current visualization tools make it hard for non-expert users to interpret biological relevance of diverse biomarkers and the volume of data makes it difficult to locate regions containing patterns of interest within the tissue. To address this challenge, we present an approach to assist users in exploring mIF datasets through the use of a natural language-driven multi-agent system. This agentic system interprets user queries, retrieves relevant biomarker information, and identifies spatial regions relevant to the user query within the imaging data. By integrating SQL-based querying, dynamic tool execution, and contextual reasoning, our system allows users to explore biomarker relationships, highlight spatially distinct regions, and dynamically update views based on natural language input. Our implementation builds on Vitessce (https://vitessce.io), a visual integration tool for spatial and single-cell datasets, and can be deployed as an interactive widget within a Jupyter notebook. We demonstrate our approach on multiple mIF datasets spanning diverse tissue and cancer types, demonstrating its adaptability and utility in discovering spatial patterns and anomalies. By reducing the effort required to navigate and interpret mIF datasets, our approach has the potential to aid researchers in quickly identifying regions of interest for downstream segmentation tasks and exploratory analysis.

Individual contributions:

• Designed and implemented the chatbot interface for Vitessce.
• Created a multi-agent orchestration layer for biomarker and ROI search.
• Integrated few-shot learning and RAG for accurate retrieval and structured output.

Multimodal pathology image search between H&E slides and multiplexed immunofluorescent images

Description: We present an approach for multimodal pathology image search, using dynamic time warping (DTW) on Variational Autoencoder (VAE) latent space that is fed into a ranked choice voting scheme to retrieve multiplexed immunofluorescent imaging (mIF) that is most similar to a query H&E slide. Through training the VAE and applying DTW, we align and compare mIF and H&E slides. Our method improves differential diagnosis and therapeutic decisions by integrating morphological H&E data with immunophenotyping from mIF, providing clinicians a rich perspective of disease states. This facilitates an understanding of the spatial relationships in tissue samples and could revolutionize the diagnostic process, enhancing precision and enabling personalized therapy selection. Our technique demonstrates feasibility using colorectal cancer and healthy tonsil samples. An exhaustive ablation study was conducted on a search engine designed to explore the correlation between multiplexed Immunofluorescence (mIF) and Hematoxylin and Eosin (H&E) staining, in order to validate its ability to map these distinct modalities into a unified vector space. Despite extreme class imbalance, the system demonstrated robustness and utility by returning similar results across various data features, which suggests potential for future use in multimodal histopathology data analysis.

Individual contributions:

• Developed and trained the VAE on HPC clusters to generate patch-level latent vectors.
• Implemented DTW-based alignment and ranked-choice retrieval pipeline.
• Conducted validation and ablation studies to assess robustness despite extreme class imbalance.

Trace to Touch: Eliciting Gestures from Capacitive Touch Electrodes

Description: Capacitive touch relies on electrodes to detect and interpret touch gestures. These electrodes are conventionally designed as rigid, grid-like structures, optimized for manufacturing efficiency. However, the advent of diverse conductive materials opens new avenues for enhancing the way electrodes are designed. In this paper, we develop a textile-silicone sensor composite using embedded conductive yarn as a capacitive touch electrode. By deviating from the grid pattern, we explore how alternative patterns can inspire novel, playful, and expressive gestures. We describe our design process for conceptualizing gestures from electrode design principles and iteratively test gesture detection using an off-the-shelf CNN model. Our approach in developing a textile-silicone sensor with unique electrode designs enables the development of creative, comfortable and customizable haptic interfaces.

Individual contributions:

• Fabricated layered silicone composites with unique electrode designs.
• Collected and processed gesture datasets for CNN-based classification.
• Achieved 97% accuracy in gesture recognition from sensor heatmaps.