Enable TogetherAI api to output reasoning traces consistently #2065
Conversation
|
I don't see any logic changes in this PR, only test changes |
| ) | ||
|
|
||
| @override | ||
| def _convert_conversation_to_api_input( |
There was a problem hiding this comment.
Per DRY, and to make sure this implementation doesn't diverge from the parent class over time, could you call super._convert_conversation_to_api_input() and then add any modifications you need after that? Ditto for _convert_api_output_to_conversation.
| api_kwargs: Optional[dict[str, Any]] = None | ||
| """Additional keyword arguments to pass to the API. | ||
| This allows for passing any API-specific parameters that are not |
There was a problem hiding this comment.
Could you add a comment here that this is currently only used for the together inference engine?
| Returns: | ||
| Conversation: The conversation including the generated response. | ||
| """ | ||
| try: |
There was a problem hiding this comment.
I don't quite get what this try-except is trying to do. If the try-block fails, then super()._convert_api_output_to_conversation will also fail because it will also try to extract message the exact same way right?
| response, original_conversation | ||
| ) | ||
|
|
||
| if "reasoning" in message and "</think>" not in message.get("content", ""): |
There was a problem hiding this comment.
Do you know if any other inference provider may include a "reasoning" field? If this is a common convention, we should move the logic in this function into the parent class RemoteInferenceEngine.
|
|
||
| # Then layer on Together-specific / remote-specific kwargs | ||
| remote_params = self._remote_params | ||
| if remote_params.api_kwargs: |
There was a problem hiding this comment.
Ditto here; do you know of any other inference engines that could benefit from this? Should this logic be in RemoteInferenceEngine?
Description
reasoningargument ofChatCompletionsobjectReasoning/hybrid models like Kimi K2, DeepSeek v3.1, GPT OSS 120B returns thinking traces in the
reasoningargument ofChatCompletionMessageobject.Currently, Oumi conversations only consumes the
contentargument ofChatCompletionMessageobject. This PR modifies TogetherAI inference engine API to concatenatereasoningandcontentfields of theChatCompletionMessagefor thecontentfield of oumiConversationsoutput.Hybrid models like DeepSeek-V3.1 need
"reasoning": {"enabled": True},to be passed in via API input to turn on reasoning. This PR adds optionalapi_kwargstoRemoteParamsto allow users to pass in such arguments.Related issues
Fixes # (issue)
Before submitting
Reviewers
At least one review from a member of
oumi-ai/oumi-staffis required.