Abstract
Traditional centralized operations struggle to scale with the complexity of large satellite constellations. This work presents a decentralized multi-agent deep reinforcement learning approach for operating an autonomous Space Situational Awareness constellation of 20 satellites. Four reward functions, with different reward scaling factors, are evaluated using a mission goal term across three coordination topologies. Developed strategies obtain similar or even outperform centralized rule-based and Mixed Integer Programming benchmarks in tested scenarios, and generalize across coordination topologies despite being trained in fully decentralized environments. Finally, space hardware inference confirms feasibility in modern systems, with execution times in the microsecond range.
Key Findings
Performance
60-70%
Remaining resources maintained across all coordination topologies, demonstrating efficient resource management while maintaining high mission performance.
Inference Speed
~4μs
Average inference time per decision on NVIDIA Jetson Orin Nano (7W power mode), confirming real-time feasibility for onboard deployment.
Generalization
Cross-Topology
Policies trained in fully decentralized environments generalize effectively to centralized and constrained decentralized topologies, demonstrating robust coordination strategies.
Hardware Validation: The trained policies were evaluated on an NVIDIA Jetson Orin Nano operating in 7W power mode, a configuration specifically designed for resource-constrained applications. This computing platform has been identified as a promising candidate for AI-enabled nanosatellite missions due to its balance of size, computational capability, and power efficiency. The empirical results demonstrate that the developed neural network policies can be executed in real-time on low-power onboard systems without compromising mission-critical response times.
Reward Function Insights: Negative reward functions maintain more balanced operational modes even under communication limitations, with observation activities remaining above 20% in constrained scenarios. This aligns with research suggesting that shifting reward functions towards values equal to or lower than zero leads to faster convergence and better optimal decisions.
Methodology
Problem Space
Our problem space consists of a LEO constellation of small satellites, acting as collaborative agents, equipped with imaging and inter-satellite link (ISL) capabilities. Their environment is populated with other orbiting objects in their vicinity. Their main objective is to locate and track these objects and spread the collected information among them to improve Space Situational Awareness (SSA). This aims to provide better outcome information and monitoring of space objects while optimizing the use of resources within the constellation. This optimization of shared resources and onboard capabilities is especially interesting in small satellites with limited onboard resources that perform missions with opportunistic events.
Network Architectures
Centralized: One random observer is initialized as a central node, being the only one in the constellation able to propagate the information it receives from the rest of the constellation.
Constrained Decentralized: Satellites can only communicate between them if they share the same communication band (4 alternatives).
Fully Decentralized: Every node can communicate with each other.
Multi-Agent Reinforcement Learning
AUTOPS-RL uses distributed Proximal Policy Optimization (PPO) to train coordination policies for multiple observer satellites. Each satellite acts as an independent agent making real-time decisions about which targets to observe, considering battery constraints, communication limitations, and mission objectives. The training process is computationally intensive but produces lightweight policies that can be executed in real-time on resource-constrained systems, with inference times in the microsecond range.
Reward Function Design
The success of RL techniques heavily depends on effective reward function design. This work implements PPO with four different reward functions to determine which type leads to better learning policies. Each reward function has three main components: resources, actions, and mission goal terms.
Common Components
Resource Penalty: A proportional penalty based on available energy and storage to encourage efficient resource management.
Action-specific Rewards: Rewards and penalties for different actions:
- Standby: Small penalty to discourage idleness
- Communication (successful): Reward proportional to data transmitted (with cap to normalize and encourage balanced communications)
- Communication/Observation (failed): Penalty for failed attempts
- Observation (successful): Fixed reward plus quality-based bonus
Four Reward Function Cases
Case 1: Individual Positive Reinforcement
Positive rewards for monitored targets based on individual observation matrices. Each agent receives rewards proportional to the fraction of targets it has observed.
Case 2: Individual Negative Reinforcement
Negative rewards for undetected targets based on individual observation matrices. Agents are penalized for targets they have not observed, encouraging comprehensive coverage.
Case 3: Collective Positive Reinforcement
Individual terms for resources and actions, plus a global collective bonus shared by all agents when any target is observed by any agent in the constellation. Encourages collaborative behavior.
Case 4: Collective Negative Reinforcement
Individual terms for resources and actions, plus a global collective penalty shared by all agents when targets remain unobserved by the entire constellation. Strong incentive for coordination.
Sensitivity Analysis
To assess the effect of bonus parameters on the final mission goal, the scaling factor α for the mission bonus term is varied. Comprehensive evaluation across scaling factors (α = 0.0, 0.01, 0.05, 1.0, 2.0, 5.0) to analyze the impact of reward scaling on policy performance and coordination behavior. Values are adjusted so that the total return per agent has an order of magnitude not too small for gradients to be significant, nor too big to avoid unstable behaviors and non-stationarity.
Constellation Configuration
Space Situational Awareness Exemple Scenario: 20 observer satellites arranged in a Walker Delta constellation to track and monitor 100 target objects in various orbital configurations.
Observer Satellite Parameters
Results
Mission Accomplishment
Comparison of mission accomplishment rates across RL policies (with different scaling factors), rule-based, and MIP baseline methods. Results show the effectiveness of learned coordination strategies compared to traditional approaches.
Resource Utilization
Analysis of resource utilization efficiency, including battery consumption and observation coverage. RL policies demonstrate efficient resource management while maintaining high mission performance.
Action Distribution Analysis
Detailed breakdown of action distributions across different policies, revealing coordination patterns and decision-making strategies learned by the RL agents compared to rule-based and optimization-based approaches.
Paper
EUCASS 2025 - 11th European Conference for Aeronautics and Aerospace Sciences (EUCASS)
Presentation
EUCASS 2025 - Conference Presentation Slides
Download the conference presentation slides from EUCASS 2025. The presentation covers the methodology, experimental setup, results, and key findings of the AUTOPS-RL framework for autonomous satellite coordination.
BibTeX
@inproceedings{autops-rl2025,
title={Autonomous Decision-Making for Large Satellite Constellations: a Multi-Agent Reinforcement Learning Approach to Space Situational Awareness in Partially Observable Dynamic Environments},
author={Juan Oliver, Clemente J. and Messina, Vincenzo and Dolan, Sydney and Golkar, Alessandro},
booktitle={11th European Conference for Aeronautics and Aerospace Sciences (EUCASS)},
year={2025},
url={https://gitlab.lrz.de/sps/autops/autops-rl},
}