CogScale: Scalable Benchmark for Sequence Processing

A lightweight, fully synthetic benchmark designed as a cognitive "sanity check" for evaluating the sequence modeling capabilities of neural networks. CogScale isolates specific cognitive and memory abilities to let you rapidly validate architectural innovations before committing to massive, environmentally costly large-scale training.

• 14 Diverse Tasks: Signal Forecasting, Memory & Retention, Pattern Recognition, and Manipulation & Reasoning.
• Parametrizable Scalability: Test architectures across different difficulty levels (Small, Medium, Large) by scaling sequence lengths, delays, and vocabulary sizes.
• Unified Interface: Consistent API across all tasks with standardized evaluation metrics (MSE, Error Rate).
• Zero Disk Storage: Data is generated dynamically during training, bypassing loading bottlenecks and preventing simple memorization.
• Strict Baseline Evaluation: Evaluate how your model compares against established architectures (Transformers, Mamba, xLSTM, LSTM, GRU, ESN) under strict parameter budgets.

We provide a solid baseline by evaluating 7 distinct architectures under strict parameter budgets (1k, 10k, and 100k). The Cognitive Radar visualizes their peak performances (accuracy) across a selective subset of tasks and scales, demonstrating how attention models (Transfomers) and state space models (Mamba) maintain robust reasoning under increased cognitive loads, while simple reservoir models (ESN) offer striking efficiency for basic retention tasks at an ultra-low parameter scale.

(See the paper for detailed insights on scalability and parameter efficiency).

pip install cogscale

Or install from source:

git clone [https://github.com/Naowak/cogscale.git](https://github.com/Naowak/cogscale.git)
cd cogscale
pip install -e .

import cogscale as cog

# Build a task
task_data = cog.build_task('simple_copy', difficulty='small', seed=42)

# Access the dynamically generated data
X_train = task_data['X_train']  # Training inputs
Y_train = task_data['Y_train']  # Training targets

# Train your model (example with dummy predictions)
Y_pred = your_model.predict(X_train)

# Evaluate performance using the unified metric
score = cog.compute_score(
    Y=Y_train, 
    Y_hat=Y_pred, 
    category=task_data['category']
)
print(f"Score: {score}")

Predict the future evolution of a sinusoidal signal.

Forecast the future state of a three-dimensional chaotic Lorenz system.

Reproduce a discrete sequence identically after a specified time shift.

Reproduce a continuous sequence identically after a specified time shift.

Read a sequence, hold it in memory during a silent delay, and reproduce it after a trigger token.

Memorize only a specific subset of marked tokens amidst distractions and output them at the end.

Memorize a sequence of key-value pairs and retrieve the correct value when queried with a seen key.

Identify and infer missing components within a masked discrete periodic motif.

Identify and infer missing components within a masked continuous periodic motif.

Recognize in-context duplicated sequence structures to predict the next token.

Compute and output the sum of only the marked numbers within a random sequence.

Output a randomized sequence sorted into the correct positional order after a trigger.

Determine if a mutated string of parentheses represents a valid hierarchy.

Infer logical roles and attributes (objects, colors, positions) from a simplified natural language reasoning problem encoded in one-hot vectors.

Each task supports three modular difficulty levels, designed to verify if scaling a model's parameters genuinely translates to better cognitive capabilities:

• Reduced sequence lengths, delays, and vocabulary sizes.
• Ideal for quick experiments, debugging, and testing models at the 1k-10k parameter scale.

• Realistic problem sizes with increased cognitive load.
• Suitable for thorough model evaluation and scalability testing at the 10k-100k parameter scale.

• Highly complex configurations.
• Designed to push high-performance and large-scale architectures to their representational limits.

# Small configuration (fast & lightweight)
task_small = cog.build_task('bracket_matching', difficulty='small')

# Medium configuration (thorough scalability test)
task_medium = cog.build_task('bracket_matching', difficulty='medium')

All tasks return a standardized dictionary containing NumPy arrays:

{
    'X_train': np.ndarray,      # Training inputs [batch, time, features]
    'Y_train': np.ndarray,      # Training targets [batch, time, outputs]
    'X_valid': np.ndarray,      # Validation inputs
    'Y_valid': np.ndarray,      # Validation targets  
    'X_test': np.ndarray,       # Test inputs
    'Y_test': np.ndarray,       # Test targets
    'category': str             # 'classification', 'multi_classification', or 'regression'
}

import cogscale as cog
from MyModel import MyModel

def evaluate_model_on_all_tasks(model, difficulty='small'):
    """Evaluate an architecture across the full CogScale cognitive spectrum."""
    
    results = {}
    task_names = [
        'sinus_forecasting', 'chaotic_forecasting',
        'discrete_postcasting', 'continuous_postcasting', 
        'simple_copy', 'selective_copy', 'associative_recall',
        'discrete_pattern_completion', 'continuous_pattern_completion', 'induction_heads',
        'adding_problem', 'sorting_problem', 'bracket_matching', 'cross_situation'
    ]
    
    for task_name in task_names:
        print(f"Evaluating on {task_name}...")
        
        # Load dynamically generated task
        task_data = cog.build_task(task_name, difficulty=difficulty)
        
        # Train model
        model = MyModel(...)
        model.train(task_data['X_train'], task_data['Y_train'])

        # Predict on test set
        Y_pred = model.predict(task_data['X_test'])
        
        # Compute unified score
        score = cog.compute_score(
            Y=task_data['Y_test'],
            Y_hat=Y_pred,
            category=task_data['category']
        )
        
        results[task_name] = score
        print(f"  Score: {score:.4f}")
    
    return results

The evaluation metrics automatically adapt based on the task category:

• Regression tasks: Mean Squared Error (MSE)
• Classification tasks: Error rate (1 - accuracy)
• Multi-class classification tasks: Label-based error rate (1 - label-based accuracy)

Lower scores indicate better performance across all tasks.

This project is licensed under the MIT License - see the LICENSE file for details.

If you use CogScale in your research or find it helpful as a sanity check for your architectures, please cite:

@inproceedings{cogscale2026,
  title={CogScale: Scalable Benchmark for Sequence Processing},
  author={Bendi-Ouis Yannis, De Coudenhove Romain and Hinaut Xavier},
  year={2026},
  url={https://github.com/Naowak/cogscale}
}

• 🐛 Issues: GitHub Issues
• 💬 Discussions: GitHub Discussions

CogScale - Democratizing architectural research by ensuring foundational cognitive abilities before massive scaling.