Duration: 1–2 years

Abstract

Firefighting operations require rapid perception–action coupling under conditions of extreme uncertainty, limited visibility, and high cognitive load. In such environments, traditional training and in-field support systems are often insufficient to provide timely, context-aware decision support.

Mixed Reality (MR) technologies offer new opportunities for enhancing situational awareness; however, most current systems rely primarily on visual overlays and do not incorporate embodied sensory feedback that reflects the physical intensity of fire environments.

This project develops and evaluates an AI-enhanced multimodal MR system for firefighting support, integrating real-time hazard detection, explainable AI (XAI), and thermal haptic feedback. The system aims to augment human perception through spatially anchored visualisation and proximity-based thermal cues, enabling more intuitive understanding of environmental risk in low-visibility scenarios.

The study contributes to the development of human-centred emergency response technologies that combine artificial intelligence with embodied interaction, supporting safer and more effective operational decision-making.

Research Objectives

The project pursues the following core objectives:

• Development of multimodal MR systems:

  Design and implement MR interfaces that integrate visual augmentation, AI-based hazard detection, and thermal haptic feedback for immersive firefighting support.

• Integration of explainable AI (XAI):

  Develop transparent AI models that provide interpretable hazard detection outputs to support firefighter trust and situational understanding in high-risk environments.

• Embodied sensory augmentation:

  Investigate the role of thermal haptics in improving proximity perception, spatial awareness, and decision-making under low-visibility conditions.

• Evaluation of operational performance:

  Conduct controlled experimental studies with professional firefighters to assess system impact on navigation accuracy, hazard detection, and cognitive workload.

• Human-centred system design:

  Develop design principles for MR systems that align with cognitive, perceptual, and physical constraints in emergency response contexts.

Expected Impact

The project is expected to improve the effectiveness and safety of firefighting operations by enhancing real-time situational awareness and decision support in complex environments.

By integrating AI, MR visualisation, and thermal haptic feedback, the system enables more intuitive perception of hazards and improves response accuracy under stress and limited visibility conditions.

In addition, the project advances human-centred AI design in safety-critical domains, contributing to the development of next-generation emergency response systems that combine cognitive support with embodied interaction.

Overall, the outcomes support improved operational safety, faster decision-making, and enhanced training quality for firefighting professionals.

Contact Details
Dr. Sara Scheffer
Email: sara.scheffer@tuwien.ac.at
© TU Wien