Author(s): Aarush Mahajan

Originally published on Towards AI.

How I Built a Satellite Collision Risk Classifier Using Random Forest and XGBoost on Conjunction Data Messages

By Aarush Mahajan | Focus: Artificial Intelligence

Project Deliverables: 5-Minute Video Showcase · Interactive Colab Notebook

TL;DR

I built a supervised pipeline that flags close approaches between satellites and debris whose collision probability exceeds the screening threshold of 10⁻⁴. I generated 5,000 synthetic Conjunction Data Message (CDM) records from a 2D Gaussian encounter model and trained Random Forest and XGBoost classifiers. XGBoost reached a test ROC-AUC of 0.98, but the labels are a known function of the inputs the models see, so that score mostly measures how well the generator can be learned. Three results matter more. Raw inputs did as well as my engineered features. Raw classifier scores are rankings, not probabilities, and isotonic calibration fixes that. But even after calibration, the model flags the high-risk region only barely, and the raw model never flags it at all.

1. Why This Matters

Low Earth orbit is getting crowded. The ESA Annual Space Environment Report (2024 edition) estimates about 900,000 objects between 1 and 10 cm and about 128 million between 1 mm and 1 cm. Objects in the 1 to 10 cm range are too small to track reliably, yet at orbital speeds they can seriously damage a spacecraft.

Today, collision avoidance is mostly manual. A tracking network detects a close approach and issues a Conjunction Data Message under CCSDS 508.0-B-1, typically days before the Time of Closest Approach (TCA). Flight dynamics analysts then review the message, estimate the risk, decide whether the event crosses the screening threshold, and plan any avoidance burn.

Large constellations generate hundreds of these messages per satellite each week. Automated triage could help separate routine near-misses from events that need action, so operators spend their attention and propellant where it matters.

2. What I Built

The project was inspired by the ESA Collision Avoidance Challenge, documented by Uriot et al. (2022). That challenge used real CDM-derived data and a different prediction target, so this project does not reproduce it.

My goal was a Python pipeline that answers two questions:

1. Can tree models flag high-risk conjunctions from CDM-style features?
2. How much of any apparent skill comes from the label being a simple function of those features?

3. How the Pipeline Works

Generate encounter geometry: miss components (dx, dy, dz), relative speed,
 1-sigma position uncertainties for primary and secondary, TCA
 │
 ▼
Compute Pc from the 2D encounter model, then apply lognormal noise (σ = 0.3)
 │
 ▼
Label: high-risk if Pc > 1e-4
 │
 ▼
Features: raw CDM columns + miss distance + covariance trace + log transforms (15 total)
 │
 ▼
80/20 stratified split → Random Forest (200 trees) and XGBoost
 │
 ▼
Evaluate: test-set metrics, 5-fold CV, calibration, sensitivity sweep, seed stability

Encounter geometry. Each record has miss components dx (along-track), dy, and dz, drawn from zero-mean normals with scales of 1,000 m, 300 m, and 200 m. Relative speed is uniform between 7 and 14 km/s. Each object has three 1-sigma position uncertainties, drawn uniformly over fixed ranges.

Collision probability. The encounter-plane approximation uses a combined hard-body radius of 10 m and combined per-axis standard deviations:

σ_y = √(σ_y,pri² + σ_y,sec²), σ_z = √(σ_z,pri² + σ_z,sec²)

Pc = (HBR² / (2 σ_y σ_z)) · exp(−½ (dy²/σ_y² + dz²/σ_z²))

This decreases as the miss distance grows, as Figure 1 shows. The sample record used throughout this article (dy = 15 m, dz = 8 m) has a physics Pc of 4.02 × 10⁻².

Labels. The generator multiplies Pc by lognormal noise, exp(N(0, 0.3)), and labels records with the noisy value above 10⁻⁴ as high-risk (y = 1). All others are y = 0. The high-risk class is 81% of records.

Features. The classifier sees the exact miss components and sigmas, not noisy measurements. Only the label carries noise. This is a simplification discussed in Section 6.

Models. A Random Forest (200 trees, max depth 10, balanced class weights) and an XGBoost classifier (200 trees, depth 5, learning rate 0.05, positive-class weight set from the training split) are trained on an 80/20 stratified split. A majority-class predictor is the baseline.

4. Results

The majority-class baseline already scores 81% accuracy, so accuracy alone says little here. ROC-AUC and the confusion matrices are more informative.

Held-out test set (seed 42, n = 1,000)

Model Accuracy Precision Recall F1 ROC-AUC Majority-class baseline 0.810 0.00 0.00 0.00 0.500 Random Forest 0.911 0.932 0.961 0.946 0.961 XGBoost 0.931 0.978 0.936 0.957 0.982

5-fold stratified cross-validation (seed 42, mean ± SD across folds)

Model Accuracy ROC-AUC Random Forest 0.909 ± 0.015 0.947 ± 0.010 XGBoost 0.937 ± 0.004 0.980 ± 0.004

XGBoost’s weak spot is the low-risk class. Its recall on the 190 low-risk test records is 0.91, and its precision for that class is 0.77.

Seed stability (5 independent datasets, n = 5,000 each). The high-risk fraction ranged from 0.806 to 0.819 across seeds.

Model ROC-AUC Accuracy Random Forest 0.946 ± 0.010 0.913 ± 0.008 XGBoost 0.981 ± 0.002 0.937 ± 0.004

A reference rule. Thresholding the noise-free physics Pc at 10⁻⁴ gives 96.8% accuracy against the noisy labels, using the same inputs. XGBoost, at 93.1%, does not match this rule. The 158 records where label noise moves the outcome across the threshold account for most of the gap. These records sit near the boundary, so they are hard for any model. The models are approximating a formula, not discovering anything beyond it.

Feature importance. For the Random Forest, the top features are dz (0.21) and dy (0.20), followed by sigma_z_sec (0.07), log miss distance (0.07), and miss distance (0.07).

4.1 Calibration

The raw XGBoost outputs are not calibrated. I measured this on the 1,000-record test split, with the model trained on the 4,000-record training split, using 10 uniform-width bins for expected calibration error (ECE).

Isotonic calibration reduces expected calibration error about threefold (0.054 to 0.016) and lowers the Brier score from 0.048 to 0.039. Ranking quality is not re-measured here, since the notebook does not print calibrated AUC. The raw model is systematically optimistic in the middle range: its reliability curve sits above the diagonal across the middle of the range, as Figure 6 shows.

An important distinction: the calibrated output is the probability that the label equals 1 given the features. It is not the physical collision probability Pc. Calibration makes the number honest about the label, but the label is itself a threshold on Pc.

4.2 Sensitivity Sweep

I held the sample record’s geometry and covariances fixed and varied the transverse miss distance dy from 0 to 1,500 m, recording both models’ scores at each value.

• The physics Pc falls below 10⁻⁴ at 143 m.
• Raw XGBoost never exceeds 0.5. Its score falls from 0.25 at dy = 0 to 0.007 at 1,500 m.
• The isotonic-calibrated model is above 0.5 for every grid point from 0 to 113 m, at a plateau near 0.51. It falls below 0.5 for larger miss distances.
• The calibrated model therefore flags most of the high-risk region, but only just. It misses the stretch from about 113 m to 143 m, and its margin above the cut-off is about 0.01.

The sample record has a physics Pc of 4.02 × 10⁻², which sits inside the high-risk region. Raw XGBoost gives it 0.25 and labels it nominal. The calibrated model gives it 0.508 (printed by the notebook for the sample record), which is above the cut-off by a hair. Calibration changes the scale of the scores, but the classifier’s separation of the high-risk region is still weak, and a cut-off of 0.5 is not a reliable operating point.

5. What Went Wrong and How I Fixed It

Challenge 1: I expected engineered features to help, and they did not. My first design gave the classifiers raw position and velocity, and I expected that projecting the inputs into the encounter plane would help. The engineered version adds miss distance, a covariance trace, and log transforms to the raw CDM columns. To test whether these additions matter, I trained XGBoost on the raw columns alone.

XGBoost input Test ROC-AUC Test accuracy Raw columns only (11 features) 0.984 0.934 Raw columns + engineered features (15 features) 0.982 0.931

The raw-only model is as good or slightly better. The derived features add no information, because the label is already a function of the raw columns, and trees can learn the relevant combinations themselves. I report this as a finding. The earlier claim that projection improved the model is not supported.

Challenge 2: Scores are rankings, not probabilities. In the sample record above, a raw score of 0.25 for a Pc of 0.04 is a calibration failure, and a 0.5 threshold misses it. Isotonic calibration brings expected calibration error from 0.054 to 0.016 and the Brier score from 0.048 to 0.039 (Section 4.1). But the sweep shows calibration is not sufficient: the calibrated model separates the high-risk region only marginally, so an operational threshold would need to be chosen against costs, not set at 0.5.

Challenge 3: Feature design made the task easier than it looks. The label depends on dy, dz, and the sigmas, which the classifier receives directly. A tree that learns a threshold on those quantities can score well without learning anything beyond the threshold. The high AUC reflects how well the generator can be learned, not how well real risk can be detected.

6. Limitations

The task is synthetic. Labels come from an analytical 2D Gaussian model with multiplicative noise, not from orbit determination against real tracking data. A high score on this dataset shows that the pipeline runs. It does not show that the model is ready for operations.

The features are noise-free. The classifier never sees a noisy measurement. Real CDMs carry tracking errors that are correlated, non-Gaussian, and not captured by one lognormal factor on Pc. A model that performs well on noise-free inputs can degrade on real ones.

Single-snapshot evaluation. The model analyzes one CDM at one time. Pavanello et al. (2025) note that conjunction risk evolves across a sequence of messages as tracking improves near TCA. A sequential model would be more realistic.

Classification is not decision-making. A high-risk flag tells an operator that a collision might occur. It does not say how to maneuver without spending too much propellant or creating a new conflict.

Limited scope of the validation. Results come from five synthetic datasets from the same generator. None of the experiments add noise to the features, and I have not tested the models on real CDM data.

7. What I’m Doing Next

This build showed that a tree model can triage synthetic conjunction data, and that the result can come from the setup itself. The next steps are:

• Add measurement noise to the features, so the classifier has to recover the label from noisy inputs.
• Test on real CDM data from the ESA challenge or NASA CARA, keeping the feature pipeline unchanged.
• Choose the operating threshold by cost, weighing missed collisions against false alarms, rather than by a fixed 0.5 cut-off.
• Move to sequential decisions: a Deep Recurrent Q-Network (DRQN) based on Bourriez et al. (2023), tested against feed-forward policies under Clohessy-Wiltshire dynamics and a propellant budget of 1.0 m/s.

References

1. Uriot, T., Izzo, D., Simões, L. F., et al. (2022). Spacecraft collision avoidance challenge: Design and results of a machine learning competition. Astrodynamics, 6(2), 121–140.
2. Bourriez, N., Loizeau, A., & Abdin, A. F. (2023). Spacecraft Autonomous Decision-Planning for Collision Avoidance: a Reinforcement Learning Approach. 74th International Astronautical Congress. arXiv:2310.18966.
3. Pavanello, Z., et al. (2025). CAMmary: A Review of Spacecraft Collision Avoidance Manoeuvre Design Methods. arXiv:2503.22555.
4. Catulo, J. S., Soares, C., & Guimarães, M. (2023). Predicting the Probability of Collision of a Satellite with Space Debris: A Bayesian Machine Learning Approach. arXiv:2311.10633.
5. Consultative Committee for Space Data Systems (CCSDS). (2013). Conjunction Data Message. CCSDS Recommended Standard 508.0-B-1.
6. European Space Agency. Annual Space Environment Report, 2024 edition.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI