Mixed Reality (MR) technologies are increasingly emerging as a key enabler for next-generation firefighting support systems. By combining immersive visualisation with artificial intelligence and real-time data integration, MR has the potential to fundamentally enhance situational awareness, decision-making, and operational safety in highly dynamic and hazardous environments.

At the intersection of human-centred AI and emergency response, the following two research projects explore how MR can support firefighters both during operations and in the critical preparation phase. While one project focuses on in-field, multimodal decision support under extreme conditions, the other addresses AI-supported pre-incident briefing to improve preparedness and strategic planning before deployment. Together, they contribute to the development of intelligent, context-aware support systems that strengthen both operational performance and safety in firefighting scenarios.

X-MINDER

Multimodal Decision Support in Emergency Response

Duration: 1–2 years

Abstract

Firefighting operations require rapid perception–action coupling under conditions of extreme uncertainty, limited visibility, and high cognitive load. In such environments, traditional training and in-field support systems are often insufficient to provide timely, context-aware decision support.

Mixed Reality (MR) technologies offer new opportunities for enhancing situational awareness; however, most current systems rely primarily on visual overlays and do not incorporate embodied sensory feedback that reflects the physical intensity of fire environments.

This project develops and evaluates an AI-enhanced multimodal MR system for firefighting support, integrating real-time hazard detection, explainable AI (XAI), and thermal haptic feedback. The system aims to augment human perception through spatially anchored visualisation and proximity-based thermal cues, enabling more intuitive understanding of environmental risk in low-visibility scenarios.

The study contributes to the development of human-centred emergency response technologies that combine artificial intelligence with embodied interaction, supporting safer and more effective operational decision-making.

Research Objectives

The project pursues the following core objectives:

• Development of multimodal MR systems:

  Design and implement MR interfaces that integrate visual augmentation, AI-based hazard detection, and thermal haptic feedback for immersive firefighting support.

• Integration of explainable AI (XAI):

  Develop transparent AI models that provide interpretable hazard detection outputs to support firefighter trust and situational understanding in high-risk environments.

• Embodied sensory augmentation:

  Investigate the role of thermal haptics in improving proximity perception, spatial awareness, and decision-making under low-visibility conditions.

• Evaluation of operational performance:

  Conduct controlled experimental studies with professional firefighters to assess system impact on navigation accuracy, hazard detection, and cognitive workload.

• Human-centred system design:

  Develop design principles for MR systems that align with cognitive, perceptual, and physical constraints in emergency response contexts.

Expected Impact

The project is expected to improve the effectiveness and safety of firefighting operations by enhancing real-time situational awareness and decision support in complex environments.

By integrating AI, MR visualisation, and thermal haptic feedback, the system enables more intuitive perception of hazards and improves response accuracy under stress and limited visibility conditions.

In addition, the project advances human-centred AI design in safety-critical domains, contributing to the development of next-generation emergency response systems that combine cognitive support with embodied interaction.

Overall, the outcomes support improved operational safety, faster decision-making, and enhanced training quality for firefighting professionals.

Contact Details
Dr. Sara Scheffer
Email: sara.scheffer@tuwien.ac.at
© TU Wien

FIREBRIEF

AI- and MR-supported Pre-Incident Briefing for Firefighting Operations

Duration: 1–2 years

Abstract

Effective pre-incident briefing is critical for ensuring situational awareness and operational readiness in firefighting missions. However, current briefing practices are often based on static maps, fragmented building documentation, and limited real-time data integration.

This project develops an AI-enhanced Mixed Reality (MR) system for pre-incident briefing that integrates Internet of Things (IoT) sensor data, building information models (BIM), and real-time hazard intelligence into an immersive spatial environment.

The system enables firefighters to explore building layouts in MR prior to deployment while receiving context-aware insights, including occupancy estimates, fire safety infrastructure, hazardous zones, and sensor-derived environmental conditions. Explainable AI components support the interpretation of risk indicators and the generation of data-driven recommendations.

The objective is to improve preparedness, reduce cognitive load during entry, and enhance strategic decision-making before and during emergency response operations.

Research Objectives

The project pursues the following core objectives:

• Development of MR-based briefing environments:

  Design immersive pre-incident briefing tools that combine building models, IoT sensor streams, and AI-generated situational insights.

• Integration of real-time IoT and environmental data:

  Fuse distributed sensor networks with spatial building representations to provide up-to-date information on fire risk, occupancy, and environmental conditions.

• Explainable AI for decision support:

  Develop interpretable AI modules that translate complex sensor and model data into actionable insights for firefighters.

• Support for spatial and cognitive preparedness:

  Investigate how MR-based pre-briefing improves spatial understanding, route planning, and cognitive readiness before entering incident environments.

• Evaluation in operationally relevant scenarios:

  Assess system effectiveness in simulated pre-incident briefing exercises with professional firefighters, focusing on situational awareness and decision confidence.

Expected Impact

The project is expected to significantly enhance preparedness and decision-making quality in firefighting operations through improved pre-incident situational awareness.

By integrating IoT data, building information models, and AI-driven analysis into MR environments, firefighters can access a comprehensive and spatially grounded understanding of incident sites before deployment.

This leads to improved operational planning, reduced uncertainty during entry, and enhanced safety for both responders and affected populations.

In the long term, the project contributes to the development of intelligent, data-driven emergency preparedness systems that support faster, safer, and more informed intervention strategies.

Contact
Dr. Sara Scheffer
Email: sara.scheffer@tuwien.ac.at
© TU Wien