MAGIC – Microsoft Azure Graph Information Crawler

This repository is no longer actively maintained.

➡️ The project has been moved to a new repository:
👉 https://github.com/SVA-Cybersecurity/magic

MAGIC is a wrapper around the Microsoft Graph Python SDK designed to extract incident‑response‑relevant data from Microsoft 365 environments. It assists analysts in exporting log data efficiently and produces a consolidated .jsonl file for ingestion into OpenSearch or Timesketch.

# 1. Create and activate a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# 2. Install MAGIC
pip install git+https://github.com/magic-tool/magic.git

# 3. Initialize workspace (creates config + folders)
magic-init

# 4. Edit config.yaml:
#   - Add your M365 App credentials
#   - Add crawl modules (e.g. m365_signin) from available_crawls.yaml

# 5. Run MAGIC
magic

That’s it — you now have standardized M365 incident logs ready for analysis.

MAGIC requires an Entra ID Application Registration with:

• Application (client) ID
• Client secret
• Tenant ID
• Appropriate Microsoft Graph permissions depending on the crawl modules you want to run

MAGIC can automatically generate the application manifest with the required permissions for your configuration using:

magic --manifest

A full permissions manifest is available at the end of this document.

• Python 3.11+
• A virtual environment is recommended

python3 -m venv .venv
source .venv/bin/activate

pip install git+https://github.com/magic-tool/magic/.git

magic-init

This creates the following structure:

.
├── logs                    # Log folder
├── output                  # Output folder
├── available_crawls.yaml   # All crawl modules
├── config.yaml             # Active configuration

This is required to download any information using the Microsoft Graph API.

settings:
  auth:
    client_secret: "<your-secret>"
    client_id: "<application-id>"
    tenant_id: "<tenant-id>"

Default parameters can be used to configure settings for all crawl blocks. Defaults can be overwritten within individual crawl blocks.

Parameter priority:

1. Crawl‑level value
2. Default value
3. Internal fallback

settings:
  defaults:
    user_principal_names:
      - jdoe@tenant.com
    date_start: 2026-01-01 12:00:00
    date_end: 2026-01-12

The permissions of the configured application will be validated before downloading logs. Enable or disable via configuration:

settings:
  permission_preflight_check: True

To enrich events using the IpAPI enricher module, configure the credentials:

settings:
  ipapi:
    key: ExampleKey
    endpoint: "https://your-ipapi-endpoint"
    cert: False

To download logs, multiple crawl modules can be configured. A module may appear multiple times with different parameters.

Example:

crawl:
  - type: m365_signin
    sign_in_type: user

  - type: m365_message_traces
    recipient_addresses:
      - john@tenant.com

All modules and parameters are documented in available_crawls.yaml.

magic [-h] [-c CONFIG] [-o OUTPUT_DIR] [--reports-dir REPORTS_DIR] [--debug] [--manifest]

MAGIC - Microsoft Azure Graph Informations Crawler

options:
  -h, --help            show this help message and exit
  -c CONFIG, --config CONFIG
                        Path to configuration file (default: config.yaml in working directory or module directory)
  -o OUTPUT_DIR, --output-dir OUTPUT_DIR
                        Directory for results (default: output in working directory or module directory)
  --reports-dir REPORTS_DIR
                        Directory for logs (default: logs in working directory or module directory)
  --debug               Enable debug logging
  --manifest            Generate permissions manifest

Each crawl module creates its own output directory under the base output folder. Each crawl configuration is identified by a computed file identifier based on configured filters and parameters.

The base output is consolidated into a single base.jsonl file.

To ensure consistency:

• Output JSON contains only one layer
• Nested JSON objects are stored as JSON strings

Enricher modules can transform or enrich the output. Currently available:

• Timesketch transformer
• IPAPI enrichment
• Hash generator for integrity validation

Example:

enrich:
  timesketch:
    enabled: True
    output_filename: timesketch.jsonl

  ipapi:
    enabled: True
    input_filename: timesketch.jsonl
    output_filename: ipapi.jsonl

  hash:
    enabled: True
    output_filename: hash.jsonl
    output_filename_csv: hash.csv

To illustrate the tool’s workflow, here is a common M365 forensics scenario.

Collect all sign‑ins of user1@contoso.com between 2025‑12‑01 and 2025‑12‑10, and convert the results into a Timesketch-compatible format.

settings:
  auth:
    client_id: "<app-id>"
    client_secret: "<app-secret>"
    tenant_id: "<tenant-id>"

  defaults:
    date_start: 2025-12-01
    date_end: 2025-12-10
    user_principal_names:
      - user1@contoso.com

crawl:
  - type: m365_signin
    sign_in_type: user

enrich:
  timesketch:
    enabled: True
    output_filename: timesketch.jsonl

magic

1. MAGIC loads and merges configuration values
2. OAuth2 client credential flow obtains a Microsoft Graph API token
3. Each crawl module calls its Graph API endpoint
4. All data is merged into base.jsonl
5. Enrichment modules run (Timesketch)
6. Final timesketch.jsonl file is ready for ingestion

Some modules are limited by Microsoft Graph API restrictions. If no date range is specified, default retention periods apply. Available data may depend on the tenant’s license.

All crawl modules declare the Microsoft Graph permissions they require. You can validate your app permissions using:

permission_preflight_check: True

To generate a permission manifest containing required permissions for your configuration:

magic --manifest

Contributions are very welcome!

1. Report bugs via GitHub Issues
2. Propose new features via Issues
3. Submit Pull Requests:
   • Use feature branches
   • Write clear commit messages
   • Update documentation when needed