[recipe] feat: DeepSeek-R1-Zero on Ascend NPU by johnjunjun7 · Pull Request #3427 · verl-project/verl

johnjunjun7 · 2025-09-09T13:38:00Z

What does this PR do?

Add concise overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review.
This recipe is a training example based on the Deepseek-V3-Base model after RLHF (Reinforcement Learning from Human Feedback) training on Ascend A3 NPU, utilizing GRPO and rule-based rewards with the deepscaler dataset.

Checklist Before Starting

Search for similar PRs. Paste at least one query link here: ...
Format the PR title as [{modules}] {type}: {description} (This will be checked by the CI)
- {modules} include fsdp, megatron, sglang, vllm, rollout, trainer, ci, training_utils, recipe, hardware, deployment, ray, worker, single_controller, misc, perf, model, algo, env, tool, ckpt, doc, data
- If this PR involves multiple modules, separate them with , like [megatron, fsdp, doc]
- {type} is in feat, fix, refactor, chore, test
- If this PR breaks any API (CLI arguments, config, function signature, etc.), add [BREAKING] to the beginning of the title.
- Example: [BREAKING][fsdp, megatron] feat: dynamic batching

Test

For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc.

API and Usage Example

Demonstrate how the API changes if any, and provide usage example(s) if possible.

# Add code snippet or script demonstrating how to use this

Design & Code Changes

Demonstrate the high-level design if this PR is complex, and list the specific changes.

Checklist Before Submitting

Important

Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review.

Read the Contribute Guide.
Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always
Add / Update the documentation.
Add unit or end-to-end test(s) to the CI workflow to cover all the code. If not feasible, explain why: ...
Once your PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace. (If not accessible, please try the Feishu group (飞书群).)

CLAassistant · 2025-09-09T13:38:13Z

All committers have signed the CLA.

gemini-code-assist

Code Review

This pull request introduces a recipe for training DeepSeek-R1-Zero on Ascend NPUs. The changes are comprehensive, including a new Dockerfile, Python scripts with NPU-specific adaptations, and shell scripts for execution. However, I've identified several critical issues that will likely cause build or runtime failures. These include placeholders and typos in the Dockerfile, NameError exceptions in Python scripts due to undefined variables or missing imports, and a logic error in a model conversion script. Additionally, there are high-severity issues such as inconsistencies between the documentation and the implementation, and a confusing exit code in a shell script. Addressing these issues is crucial for the recipe to be functional and usable.

HzZHoO · 2025-11-10T06:11:22Z

Hi @FightingZhen @ji-huazhong , could you please help review this PR when you have a moment?

ji-huazhong · 2025-11-14T01:20:06Z

+            if self.inference_engine.llm_engine.model_executor.driver_worker.worker.cache_engine is None:
+                self.inference_engine.llm_engine.model_executor.driver_worker.worker._init_cache_engine()
+
+    def onload_model_weights(self):


Please add a comment explaining why we need to manually on/off load the weights instead of directly using VLLM's sleep mode, to help other users understand this change.

Thanks for the suggestion. I will add a comment for it.

ji-huazhong · 2025-11-14T01:22:59Z

+                    attn_impl.value_cache = None
+
+        gc.collect()
+        torch.npu.empty_cache()


please use aggressive_empty_cache instead.

https://github.com/volcengine/verl/blob/65f9badfe4e3c73e86dd9b6c6dab1a21400a4c4e/verl/utils/memory_utils.py#L31

FightingZhen · 2025-11-15T09:09:45Z

@@ -0,0 +1,69 @@
+# Adapted from


Suggest moving it below copyright with blank line split, same advice in following files :)

### What does this PR do? > Add **concise** overview of what this PR aims to achieve or accomplish. Reference related GitHub issues and PRs that help with the review. This recipe is a training example based on the Deepseek-V3-Base model after RLHF (Reinforcement Learning from Human Feedback) training on Ascend A3 NPU, utilizing GRPO and rule-based rewards with the deepscaler dataset. ### Checklist Before Starting - [x] Search for similar PRs. Paste at least one query link here: ... - [x] Format the PR title as `[{modules}] {type}: {description}` (This will be checked by the CI) - `{modules}` include `fsdp`, `megatron`, `sglang`, `vllm`, `rollout`, `trainer`, `ci`, `training_utils`, `recipe`, `hardware`, `deployment`, `ray`, `worker`, `single_controller`, `misc`, `perf`, `model`, `algo`, `env`, `tool`, `ckpt`, `doc`, `data` - If this PR involves multiple modules, separate them with `,` like `[megatron, fsdp, doc]` - `{type}` is in `feat`, `fix`, `refactor`, `chore`, `test` - If this PR breaks any API (CLI arguments, config, function signature, etc.), add `[BREAKING]` to the beginning of the title. - Example: `[BREAKING][fsdp, megatron] feat: dynamic batching` ### Test > For changes that can not be tested by CI (e.g., algorithm implementation, new model support), validate by experiment(s) and show results like training curve plots, evaluation results, etc. ### API and Usage Example > Demonstrate how the API changes if any, and provide usage example(s) if possible. ```python # Add code snippet or script demonstrating how to use this ``` ### Design & Code Changes > Demonstrate the high-level design if this PR is complex, and list the specific changes. ### Checklist Before Submitting > [!IMPORTANT] > Please check all the following items before requesting a review, otherwise the reviewer might deprioritize this PR for review. - [x] Read the [Contribute Guide](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md). - [x] Apply [pre-commit checks](https://github.com/volcengine/verl/blob/main/CONTRIBUTING.md#code-linting-and-formatting): `pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always` - [x] Add / Update [the documentation](https://github.com/volcengine/verl/tree/main/docs). - [x] Add unit or end-to-end test(s) to [the CI workflow](https://github.com/volcengine/verl/tree/main/.github/workflows) to cover all the code. If not feasible, explain why: ... - [x] Once your PR is ready for CI, send a message in [the `ci-request` channel](https://verl-project.slack.com/archives/C091TCESWB1) in [the `verl` Slack workspace](https://join.slack.com/t/verl-project/shared_invite/zt-3855yhg8g-CTkqXu~hKojPCmo7k_yXTQ). (If not accessible, please try [the Feishu group (飞书群)](https://applink.larkoffice.com/client/chat/chatter/add_by_link?link_token=772jd4f1-cd91-441e-a820-498c6614126a).) --------- Co-authored-by: ZhangHZ <zhuoer33@gmail.com>

gemini-code-assist Bot reviewed Sep 9, 2025

View reviewed changes

johnjunjun7 requested review from PeterSH6, eric-haibin-lin, tongyx361 and vermouth1992 as code owners September 29, 2025 07:28

HzZHoO force-pushed the main branch 2 times, most recently from 49387f4 to 910b344 Compare November 6, 2025 09:16

support Deepseek-R1 Zero on Ascend NPU

65a2a69

HzZHoO force-pushed the main branch from 910b344 to 65a2a69 Compare November 6, 2025 12:34

add code modifications and training results in readme

9f65116

update dockerfile and copyright in code files

0705190

FightingZhen reviewed Nov 13, 2025

View reviewed changes

ji-huazhong reviewed Nov 14, 2025

View reviewed changes

clean codes

b68ab22

FightingZhen reviewed Nov 15, 2025

View reviewed changes

wuxibin89 approved these changes Nov 17, 2025

View reviewed changes

HzZHoO added 4 commits November 17, 2025 10:48

update license

f753f93

fix ci issues

0a6fec0

fix ci issues

bbd1c56

fix device name

8ca1e66

ji-huazhong approved these changes Nov 18, 2025

View reviewed changes

FightingZhen merged commit 448c6c3 into verl-project:main Nov 18, 2025
75 of 77 checks passed

Conversation

johnjunjun7 commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Checklist Before Starting

Test

API and Usage Example

Design & Code Changes

Checklist Before Submitting

Uh oh!

CLAassistant commented Sep 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

HzZHoO commented Nov 10, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ji-huazhong Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

HzZHoO Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

ji-huazhong Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

HzZHoO Nov 14, 2025

Choose a reason for hiding this comment

Uh oh!

FightingZhen Nov 15, 2025

Choose a reason for hiding this comment

Uh oh!

HzZHoO Nov 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

johnjunjun7 commented Sep 9, 2025 •

edited

Loading

CLAassistant commented Sep 9, 2025 •

edited

Loading