Official implementation of "A Unified Framework for Detecting Point and Collective Anomalies in Operating System Logs via Collaborative Transformers"

• Introduction
• Key Features
• Architecture
• Installation
• Quick Start
• Datasets
• Results
  • Point Anomaly Detection
  • Point and Collective Anomaly Detection
  • Method Ranking
• Citation
• License
• Contact

Log anomaly detection is crucial for preserving the security of operating systems. Depending on the source of log data collection, various information is recorded in logs that can be considered log modalities. In light of this intuition, unimodal methods often struggle by ignoring the different modalities of log data. Meanwhile, multimodal methods fail to handle the interactions between these modalities. Applying multimodal sentiment analysis to log anomaly detection, we propose CoLog, a framework that collaboratively encodes logs utilizing various modalities. CoLog utilizes collaborative transformers and multi-head impressed attention to learn interactions among several modalities, ensuring comprehensive anomaly detection. To handle the heterogeneity caused by these interactions, CoLog incorporates a modality adaptation layer, which adapts the representations from different log modalities. This methodology enables CoLog to learn nuanced patterns and dependencies within the data, enhancing its anomaly detection capabilities. Extensive experiments demonstrate CoLog’s superiority over existing state-of-the-art methods. Furthermore, in detecting both point and collective anomalies, CoLog achieves a mean precision of 99.63%, a mean recall of 99.59%, and a mean F1 score of 99.61% across seven benchmark datasets for log-based anomaly detection. The comprehensive detection capabilities of CoLog make it highly suitable for cybersecurity, system monitoring, and operational efficiency. CoLog represents a significant advancement in log anomaly detection, providing a sophisticated and effective solution to point and collective anomaly detection through a unified framework and a solution to the complex challenges automatic log data analysis poses.

• Python 3.8 or higher
• CUDA-compatible GPU (recommended for training)

1. Clone the repository

git clone https://github.com/NasirzadehMoh/CoLog.git
cd CoLog

2. Create a virtual environment (recommended)

python -m venv venv
source venv/bin/activate  # On Windows: venv\Scripts\activate

3. Install dependencies

pip install -r requirements.txt

4. Download spaCy language model

python -m spacy download en_core_web_lg

# Preprocess a specific dataset
python groundtruth/main.py \
    --dataset hadoop \
    --sequence-type context \
    --window-size 1 \
    --model all-MiniLM-L6-v2 \
    --batch-size 128 \
    --device auto \
    --resample \
    --resample-method TomekLinks \
    --verbose

# Train CoLog on a specific dataset
python train.py --dataset casper-rw --epochs 100 --batch_size 32

# Test the trained model
python test.py --dataset casper-rw --model_path runs/logs/best_model.pth

# Launch the interactive notebook
jupyter notebook run_colog.ipynb

CoLog has been evaluated on seven benchmark datasets:

• Casper-RW: System logs from Casper RW environment
• DFRWS 2009 Jhuisi: Forensic challenge dataset
• DFRWS 2009 Nssal: Network security logs
• Honeynet Challenge 7: Honeypot network logs
• Zookeeper: Apache Zookeeper distributed system logs
• Hadoop: Apache Hadoop distributed computing logs
• BlueGene/L (BGL): Supercomputer system logs

All datasets are located in the datasets/ directory. For more information, see datasets/README.md.

@inproceedings{CoLog2024,
  title={A Unified Framework for Detecting Point and Collective Anomalies in Operating System Logs via Collaborative Transformers},
  author={CoLog Authors},
  booktitle={Proceedings of [Conference Name]},
  year={2024},
  url={https://arxiv.org}
}