Skip to content

Add Qwen3.5 model implementation - #6

Merged
dfalbel merged 1 commit into
add-ministralfrom
add-qwen3
Mar 4, 2026
Merged

Add Qwen3.5 model implementation#6
dfalbel merged 1 commit into
add-ministralfrom
add-qwen3

Conversation

@dfalbel

@dfalbel dfalbel commented Mar 4, 2026

Copy link
Copy Markdown
Member

Summary

  • Implements the Qwen3.5 text decoder with hybrid linear/full attention (Gated Delta Net)
  • Supports loading pretrained weights from Qwen/Qwen3.5-0.8B
  • Follows the same single-file, self-contained pattern as other models in minhub

Architecture highlights

  • Hybrid attention: 18 linear attention layers (Gated Delta Net) + 6 full attention layers (every 4th)
  • Full attention: GQA with output sigmoid gate, QK RMSNorm, partial RoPE (25% of head_dim)
  • Linear attention: Causal conv1d, recurrent gated delta rule with learned decay (A_log) and step size (dt_bias)
  • On-the-fly computation: Causal masks and RoPE frequencies computed per-forward to support 262K context without large buffer allocation

Validation

  • Logits match HuggingFace transformers reference (Python, float32) to ~1e-3 tolerance

Test plan

  • Custom small config forward pass (runs on CI)
  • Pretrained model logit comparison against Python reference (skip_on_ci)

Implements the Qwen3.5 text decoder architecture featuring:
- Gated Delta Net linear attention (recurrent implementation)
- Full attention with output sigmoid gate and QK norms
- RMSNorm with unit offset, partial rotary embeddings (0.25 factor)
- On-the-fly causal mask and RoPE computation for 262K context support

Validated against HuggingFace transformers on Qwen/Qwen3.5-0.8B.
@dfalbel
dfalbel merged commit 48de9ea into add-ministral Mar 4, 2026
@jrosell

jrosell commented Apr 11, 2026

Copy link
Copy Markdown

Why qwen3 is not merged to the main branch?

@dfalbel

dfalbel commented Apr 11, 2026

Copy link
Copy Markdown
Member Author

@jrosell this was my bad. just merged in #7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants