Explainable Reinforcement Learning for Multi-Agent Systems

Abstract

Multi-Agent Reinforcement Learning (MARL) has emerged as a powerful paradigm for solving complex tasks involving both cooperation and competition among agents. However, the decision-making processes of MARL models often difficult to interpret, presenting challenges for transparency and user trust. While existing explainable reinforcement learning techniques predominantly focus on single-agent settings, they fall short in addressing the intricacies of multi-agent environments. To bridge this gap, we propose PEIM (Policy Explanation in MARL), a temporally aware framework for explaining decision-making in MARL using local and model-agnostic techniques. PEIM captures and visualizes recent decision histories of agents, highlighting influential features and actions to provide both real-time and temporal insights. These explanations support temporal pattern mining, strategic behavior analysis, and behavior modeling, thereby facilitating a deeper understanding of emergent multiagent dynamics. We validate the proposed framework through experiments on competitive multi-agent scenarios in the StarCraft Multi-Agent Challenge using pre-trained MARL models. The results show that PEIM enabled the identification of individual behavior patterns (e.g., fleeing, target selection) and collaborative strategies (e.g., focus fire, grouping). These interpretable insights align agent decisions with human reasoning, effectively reducing the black-box nature of MARL systems and fostering trust through enhanced transparency and interpretability.

Department(s)

Computer Science

Document Type

Conference Proceeding

DOI

10.1109/ICTAI66417.2025.00194

Keywords

Explainable AI, Multi-Agent Reinforcement Learning, SMACv2, StarCraft II

Publication Date

1-1-2025

Journal Title

Proceedings International Conference on Tools with Artificial Intelligence Ictai

Link to Full Text

Share

COinS