Astrophysical Discovery Pipeline for Multi-Wavelength Time-Domain Surveys

AGX-PIPE-01 is a sophisticated, multi-modal, deep-learning-driven pipeline designed to discover previously unknown astrophysical objects using multi-wavelength time-domain survey data. It is optimized for a single RTX 3050 GPU (~8GB VRAM) while ensuring top-tier accuracy and minimal false positives.

• NEOWISE infrared time-series data (dense, preprocessed)
• ZTF optical time-series data (g, r, i bands) integrated for multi-wavelength analysis
• Hubble photometry via MAST API (lazy-loaded for top candidates)
• Modular interfaces for Gaia, Pan-STARRS, 2MASS cross-validation
• Extensible survey module architecture

• Quality filtering using survey flags and uncertainty thresholds
• Normalization of NEOWISE delta magnitudes and Hubble fluxes
• Missing data masking and noisy epoch down-weighting for irregular multi-band data

• Time-Series Transformer Autoencoders for temporal pattern extraction
  • Natively handles sparse, irregular sampling via Time2Vec continuous time-step embeddings and attention masking
  • Replaces legacy LSTM architectures for superior temporal modeling
• Contrastive embedding alignment for overlapping multi-survey objects
• Mixed precision (FP16) training for GPU efficiency

• HDBSCAN + Isolation Forest ensemble
• Multi-factor discovery confidence scoring:
  • HDBSCAN anomaly score (25%)
  • Isolation Forest anomaly score (25%)
  • Multi-survey agreement (20%)
  • Multi-epoch persistence (15%)
  • Catalog novelty (15%)

• Advanced Physics Analytics: Gaussian Process (GP) regression on sparse, irregular light curves to continuously extract variability timescales and amplitudes
• Physics-Aware Feature Extraction: Extensive multi-wavelength features covering variability, periodicity (Lomb–Scargle), irregularity, and spatial context
• Supervised ML Classification (XGBoost): XGBoost mapping Transformer embeddings and physics features directly to known SIMBAD types, overriding heuristic pipelines natively
• Active Learning: High-confidence unknown anomalies are automatically exported to local training sets to progressively improve the XGBoost model
• Uncertainty & Novelty Scoring: Quantified via Shannon entropy and Mahalanobis distance

• Ranking by discovery confidence (0-1 scale)
• Enrichment with scientific hypotheses and meta-categories
• Lazy-loading of sparse survey data for top candidates
• Comprehensive flagging system and generated discovery notes

• Real-time generation of JSON alert payloads for highly confident anomalies and specific target classes (e.g., Transients)
• Ready for integration with robotic telescope networks (LCOGT, Swift)

• Multi-wavelength light curves with uncertainty regions
• Latent space projections (UMAP/t-SNE) colored by hypothesis
• Feature contribution bar charts (interpretability)
• Hypothesis probability summary stacked charts
• Sky maps (RA/Dec and Galactic coordinates)
• Score distribution histograms
• Publication-ready figures (300 DPI)

# Clone or download the repository
cd AGX-PIPE-01

# Create virtual environment (recommended)
python -m venv venv
source venv/bin/activate  # Linux/Mac
# or
venv\Scripts\activate  # Windows

# Install dependencies
pip install -r requirements.txt

For CUDA support, install PyTorch with CUDA:

pip install torch --index-url https://download.pytorch.org/whl/cu118

python run_example.py

This generates synthetic NEOWISE-like data and runs the complete pipeline.

python -m agx_pipe_01.pipeline \
    --data path/to/neowise_data.parquet \
    --output ./outputs \
    --mast-token YOUR_MAST_TOKEN

from agx_pipe_01.pipeline import AGXPipeline
from agx_pipe_01.config import PipelineConfig
import pandas as pd

# Load your data
df = pd.read_parquet('neowise_data.parquet')

# Configure pipeline
config = PipelineConfig()
config.model.num_epochs = 100
config.candidate.top_k_candidates = 100

# Run pipeline
pipeline = AGXPipeline(config=config, output_dir='./outputs')
candidates = pipeline.run(neowise_df=df)

# Access results
print(candidates.to_dataframe().head(10))

AGX-PIPE-01/
├── agx_pipe_01/
│   ├── __init__.py          # Package initialization
│   ├── config.py            # Configuration classes
│   ├── pipeline.py          # Main pipeline orchestrator
│   ├── data/
│   │   ├── __init__.py
│   │   ├── neowise.py       # NEOWISE data loader
│   │   └── hubble.py        # Hubble/MAST client
│   ├── preprocessing/
│   │   └── __init__.py      # Preprocessing pipeline
│   ├── representation/
│   │   └── __init__.py      # LSTM autoencoders, TCN
│   ├── anomaly/
│   │   └── __init__.py      # HDBSCAN + Isolation Forest
│   ├── categorization/
│   │   ├── __init__.py      # Subpackage entry point
│   │   ├── features.py      # Physics feature extraction
│   │   ├── reference.py     # Latent-space population management
│   │   ├── classifier.py    # Soft classification & meta-categories
│   │   ├── uncertainty.py   # Entropy & novelty metrics
│   │   └── output.py        # Report generation & discovery notes
│   ├── utils/
│   │   └── __init__.py      # Candidate ranking, export
│   ├── surveys/
│   │   └── __init__.py      # Gaia, Pan-STARRS, 2MASS
│   └── visualization/
│       └── __init__.py      # Publication-ready figures
├── outputs/
│   ├── candidates/          # Candidate tables (CSV, JSON)
│   └── figures/             # Visualization outputs
├── run_example.py           # Example runner
└── requirements.txt         # Dependencies

The pipeline is configured via PipelineConfig:

from agx_pipe_01.config import PipelineConfig

config = PipelineConfig()

# Data settings
config.data.neowise_epochs = 64
config.data.hubble_max_epochs = 8
config.data.neowise_uncertainty_threshold = 0.2

# Model settings (optimized for RTX 3050)
config.model.device = 'cuda'
config.model.mixed_precision = True
config.model.neowise_hidden_dim = 96
config.model.neowise_latent_dim = 20
config.model.batch_size = 32

# Anomaly detection
config.anomaly.hdbscan_min_cluster_size = 15
config.anomaly.iforest_n_estimators = 150

# Candidate selection
config.candidate.confidence_threshold = 0.5
config.candidate.top_k_candidates = 100

# Probabilistic Categorization
config.categorization.enabled = True
config.categorization.top_k_hypotheses = 3
config.categorization.unknown_entropy_threshold = 1.5

• candidates.csv - Full candidate table with all scores
• candidates.json - JSON format with metadata
• summary.txt - Human-readable summary statistics

• latent_space_anomalies.png - UMAP projection with anomalies highlighted
• latent_space_hypotheses.png - UMAP projection colored by scientific hypothesis
• feature_contributions.png - Interpretability bar charts for top candidates
• hypothesis_summary.png - Probability distributions for top candidates
• sky_map.png - Sky distribution of candidates
• score_distributions.png - Histograms of all score components
• lightcurve_*.png - Multi-wavelength light curves for top candidates
• categorization_summary.csv - Scientific typing results for all objects
• categorization_notes.txt - Publication-ready discovery notes

• Small batch sizes (32)
• Mixed precision training (FP16)
• Gradient accumulation for effective larger batches
• CPU-based clustering (HDBSCAN, Isolation Forest)

• Hubble data only loaded for top candidates (confidence > 0.7)
• Optional surveys loaded on-demand
• MAST API response caching

For Hubble data access, obtain a token from MAST:

python -m agx_pipe_01.pipeline --data data.parquet --mast-token YOUR_TOKEN

Or set in config:

config.mast_token = "your_token_here"

from agx_pipe_01.surveys import SurveyDataLoader

class MySurveyLoader(SurveyDataLoader):
    survey_name = "mysurvey"

    def load_for_candidate(self, object_id, ra, dec, **kwargs):
        # Implement loading logic
        pass

    def create_sequences(self, object_ids=None):
        # Implement sequence creation
        pass

from agx_pipe_01.anomaly import EnsembleAnomalyDetector

detector = EnsembleAnomalyDetector()
detector.weights['custom_score'] = 0.2  # Add custom weight

If you use AGX-PIPE-01 in your research, please cite:

@software{agx_pipe_01,
  title = {AGX-PIPE-01: Astrophysical Discovery Pipeline},
  version = {2.0.0},
  year = {2026},
}

• NEOWISE data from NASA/JPL-Caltech
• Hubble data from MAST/STScI
• Built with PyTorch, scikit-learn, HDBSCAN, and UMAP