[rollout] fix: render mapping tool-call arguments through DeepSeek chat templates - #7768

What does this PR do?

Fixes #7664.

verl keeps tool-call arguments as a mapping through the message history (the OpenAIFunctionCallSchema shape), which templates render with | tojson. The chat templates shipped with DeepSeek-R1, DeepSeek-V3 and DeepSeek-V3.1 concatenate tool['function']['arguments'] into the prompt instead, so every full-history render of a conversation with a tool-call turn raised TypeError: can only concatenate str (not "dict") to str: the initial-prompt render in ContinuousTokenBuilder, MultiTurnSFTDataset, and scripts/chat_template_checker.py, which therefore could not validate the DeepSeek family at all. The dummy-user retry in apply_chat_template then failed on the same conversation with its own error and hid the first one.

apply_chat_template now retries a TypeError once with the arguments serialized as JSON strings (json.dumps(..., ensure_ascii=False)), the form these models emit and the DeepSeek API returns for the same call. Templates that render mappings themselves never raise, so they keep seeing the original messages; a TypeError with nothing to convert propagates as before; the history is copied, never mutated.

Two things this PR deliberately does not do:

• The R1 and V3 templates only enter their tool-call branch when the assistant message's content is None. With content: "", which is what ToolAgentLoop._build_assistant_message stores, those two templates render an empty assistant turn and skip the tool calls without raising. That is a property of the templates rather than of the argument type, and the rollout path never re-renders assistant turns (it keeps the generated tokens), so it is left alone here and noted in the test. V3.1 keys its branch on tool_calls and is fully covered.
• RLHFDataset.filter_overlong_prompts calls processor.apply_chat_template directly (the deepseek.py module docstring already records this) and its except branch counts a failed render as overlong. Not touched here.

Checklist Before Starting

• Search for similar PRs: https://github.com/verl-project/verl/pulls?q=is%3Apr+deepseek+chat+template+arguments
• PR title formatted as [{modules}] {type}: {description}

Test

• tests/utils/test_chat_template_tool_arguments_on_cpu.py (new, 10 tests): the helper copies rather than mutates and leaves untouched messages as the same objects; a concatenating fake template renders {"city": "Seattle"} with exactly one retry (tokenize on and off); a tojson-style fake receives the mapping in a single call; the V3.1 tool-call branch under jinja2.sandbox.ImmutableSandboxedEnvironment raises can only concatenate str on its own and renders through the wrapper; the DeepSeek-R1 chat template (vendored verbatim from the R1 distill checkpoints' tokenizer_config.json) rendered through transformers.utils.chat_template_utils.render_jinja_template; a TypeError unrelated to the arguments still propagates.
• tests/utils/test_continuous_token_on_cpu.py: one new test renders build_initial_tokens for a history that already contains a tool-call turn with mapping arguments; the existing append test now checks that the default builder renders the same append as DeepSeekContinuousTokenBuilder instead of expecting the TypeError.
• pytest tests/utils/test_chat_template_tool_arguments_on_cpu.py tests/utils/test_continuous_token_on_cpu.py tests/utils/test_turn_separator_on_cpu.py: 142 passed; the agent-loop CPU tests (test_tool_call_id_on_cpu.py, test_call_tool_on_cpu.py, test_agent_loop_extra_fields_schema_on_cpu.py): 18 passed. ruff check, ruff format --check and mypy pass on the touched files.
• The template facts above were checked against the current tokenizer_config.json of deepseek-ai/DeepSeek-V3, deepseek-ai/DeepSeek-R1 and deepseek-ai/DeepSeek-V3.1 on the Hub (concatenated arguments in all three; content is none gating in R1 and V3; tool_calls gating in V3.1). I have not rerun the real-tokenizer harness from [rollout] fix: DeepSeek continuous token builder cannot render tool appends #7630 on this branch.

API and Usage Example

No API change. verl.utils.tokenizer.chat_template.tool_call_arguments_as_json(messages) is new and returns (messages, changed).

Design & Code Changes

• verl/utils/tokenizer/chat_template.py: tool_call_arguments_as_json and _apply_chat_template_with_json_arguments; the wrapper's first render goes through the latter. The dummy-user fallback is untouched, so this is independent of [misc] fix: handle strict system-message chat templates #7726, which reworks that block.
• DeepSeekContinuousTokenBuilder keeps stringifying its synthetic tool call; it no longer depends on it.

Checklist Before Submitting

• Read the Contribute Guide.
• ruff check / ruff format pass on the touched files.
• Docs: internal tokenizer utility, no user-facing change.
• Unit tests under tests/utils/, covered by the CPU unit-test workflow.
• CI request in ci-request once ready.