Add Qwen3 ASR to Keras-Hub - #2942
buildwithsuhana wants to merge 21 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces the Qwen3-ASR model architecture to KerasHub, adding components for audio preprocessing, encoding, tokenization, and causal language modeling, along with a checkpoint conversion script. The code review identifies several critical issues for backend compatibility and robustness: in Qwen3ASRAudioEncoder, dynamic shape retrieval and dynamic attention mask construction must be avoided to prevent JAX JIT compilation failures, and positional embeddings should be computed dynamically to avoid PyTorch device placement issues. In Qwen3ASRCausalLMPreprocessor, a batch-safe contiguous block approach should replace the fragile ops.where logic to prevent runtime errors with varying batch lengths. Additionally, the audio converter should register its window and filters as weights in build(), and the checkpoint conversion script must remove the invalid gated MLP weight mapping to prevent crashes.
665c573 to
def3368
Compare
def3368 to
1aa7c9a
Compare
…q2Seq and fix lint
cb79ad5 to
045d436
Compare
…ocessing - Update Qwen3ASRTokenizer to include <|im_end|> and <|endoftext|> as end tokens. - Add unsplittable_tokens to Qwen3ASRTokenizer for correct special token handling. - Fix Qwen3ASRPreprocessor to avoid adding EOS token to prompt during generation. - Override Qwen3ASRCausalLM.generate to handle multiple stop tokens automatically. - Refactor conversion script for cleaner output comparison and faster testing.
045d436 to
50bd68f
Compare
ee5be5f to
cd2ddd9
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request implements the Qwen3-ASR model architecture, including the audio converter, audio encoder, backbone, causal LM, and tokenizer components. The review feedback highlights several technical improvements: correcting a method name typo in the tokenizer, optimizing audio sequence padding to avoid slow map functions, adding input validation to the causal LM for better error handling, simplifying shape manipulation in the audio converter, improving JIT-compatibility in the audio encoder, and refactoring the decoder layer to fix a bug in the base class rather than using subclassing.
434d102 to
73e593c
Compare
|
It looks like you didn't specify a command after
Please let me know if you would like me to perform one of these actions! |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces the Qwen3-ASR model to KerasHub, adding the audio converter, audio encoder, multi-modal projector, backbone, causal LM, preprocessor, tokenizer, and HuggingFace conversion utilities. The review feedback focuses on improving robustness and compatibility with JAX JIT compilation. Specifically, the reviewer recommends using static shapes instead of symbolic operations in the audio encoder, simplifying the audio scattering logic in the backbone using ops.take_along_axis to avoid dynamic arange operations, adopting backend-agnostic ops.squeeze in the preprocessor, and adding explicit input validation in the causal LM.
b2c29fa to
905b0d2
Compare
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces the Qwen3-ASR model architecture to KerasHub, adding the necessary components for multimodal speech recognition and causal language modeling. This includes the audio converter, audio encoder, multimodal projector, backbone, preprocessor, tokenizer, and causal LM task model, along with presets, conversion utilities, and tests. The review feedback focuses on improving robustness and backend compatibility, specifically by clipping indices in the audio scattering logic to prevent out-of-bounds crashes, using -1 in ops.reshape to avoid JAX/JIT tracing issues, optimizing static constants, and cleaning up overly verbose mathematical operations (AI slop) in favor of standard Python operators.
d2686c0 to
8afeb97
Compare
…into feat/qwen3-asr
8afeb97 to
ec374b0
Compare
Fixes #2600
Description of the change
This PR implements the Qwen3-ASR model family (0.6B and 1.7B) in KerasHub. Qwen3-ASR is a Large Audio-Language Model (LALM) that leverages the Qwen3-Omni foundation for state-of-the-art multilingual speech recognition and language identification.
Colab Notebook
https://colab.research.google.com/drive/1P7hyLcYX0_132Y7JB4REpyInCiYEEq2h?resourcekey=0-8YE-PpzIJjNgNP0Xx8Y3dg&usp=sharing
Checklist