Modular neural architecture for growing knowledge, selective unlearning, and reversible knowledge operations via low-rank submatrices.

Author: Victor Alejandro Cano Jaramillo -- April 2026

With task-ID routing, removing a knowledge module leaves all other modules perfectly intact:

7/7 remaining tasks: exactly 0.0% degradation. Tested on a 110M-param transformer with 8 knowledge modules.

RSM-Net stores knowledge in modular low-rank submatrices that can be independently added, frozen, removed, or inspected. Each submatrix is a pair (A_k, B_k) that adds a task-specific perturbation to frozen base weights:

W_eff = W_base + sum_k gate_k * A_k @ B_k

Knowledge operations:

• Add: create a new submatrix for a new task (no retraining needed)
• Remove: delete a submatrix to forget specific knowledge (other modules unaffected)
• Disable: temporarily deactivate a module (reversible)
• Inspect: measure each module's contribution to any task

RSM-Net MLP (400x200) on MNIST-Family and Multi-Domain benchmarks.

RSM-Net reduces forgetting 4.4x vs Naive on multi-domain.

Sparsemax vs softmax: no difference. Protection comes from modular structure, not sparse routing.

RSM-Net as adapter on a pretrained LLM. Code -> Math -> Creative.

Negative result (honest): RSM-Net gates don't help on LLMs. When the frozen backbone is massive (1.5B params) and the adapter is tiny (r=4), Naive LoRA is sufficient. The adapter-to-backbone ratio is too small for interference to occur.

Train from scratch on WikiText-103, then add NovaTech/Lucas/Bias via submatrices.

Selective knowledge removal works: Lucas is forgotten (+110%), NovaTech is retained (-4%).

8 submatrices: WikiText, NovaTech, Lucas, Bias, Valar Syndrome, Terranova, History, Logic.

With task-ID routing (activate only the relevant submatrix): 0.0% cross-contamination on removal (see headline table).

Specialization confirmed: Sub 1 is 12.65x more critical for math than for other tasks. Each submatrix develops internal specialization even though soft gates distribute uniformly.

Input x --> [ConvEncoder] (optional)
  |
  v
W_base(x) ---- frozen after task 0 --------+
  |                                          |
  v                                          |
Routing context --> query_proj --> gates      |
                                  |          |
                    [Sub 0] [Sub 1] [Sub 2]  |
                         \    |    /         |
                    sum_k gate_k * A_k@B_k --+---> output

Two routing modes:

• Soft gating (no task ID): gates distribute uniformly, protection via load distribution
• Task-ID routing (task ID known): activate only the relevant submatrix, perfect isolation

See docs/architecture.md for details.

pip install -e ".[dev]"  # Python 3.10+

RSM-Net/
|-- src/rsm_net/           # Core implementation
|-- experiments/
|   |-- mlp/               # MLP benchmark results
|   |-- llm_adapter/       # Qwen2.5-1.5B adapter results
|   |-- from_scratch/      # 110M transformer results
|   |-- unlearning/        # NovaTech/Lucas/Bias removal
|   |-- growth/            # 8-module growth + task-ID routing
|   |-- figures/           # All plots and heatmaps
|-- docs/
|   |-- architecture.md    # Architecture + routing modes
|   |-- experiments.md     # All 5 experimental phases
|   |-- findings.md        # Key findings and analysis
|   |-- paper.md           # Mathematical formalization
|-- tests/                 # 45 tests (pytest)

1. Soft gates always uniform (0.33-0.34) across MLPs, transformers, and LLMs
2. Protection via load distribution, not selective routing
3. Internal specialization is real despite uniform gates (12.65x ratio)
4. Task-ID routing gives perfect isolation (0.0% cross-contamination)
5. Adapter-to-backbone ratio predicts effectiveness (high ratio = RSM-Net helps, low ratio = Naive LoRA sufficient)
6. Unlearning works: selective knowledge removal with minimal collateral damage

See docs/findings.md for full analysis.

pytest tests/ -v

• Kirkpatrick et al. (2017) "Overcoming catastrophic forgetting in neural networks"
• Hu et al. (2021) "LoRA: Low-Rank Adaptation of Large Language Models"
• Martins & Astudillo (2016) "From Softmax to Sparsemax"

Apache 2.0