Skip to content

fix(llama-index-llms-siliconflow): parse SSE frames line by line in astream_chat - #23668

Open
lester076 wants to merge 1 commit into
run-llama:mainfrom
lester076:fix-siliconflow-sse-stream-parsing
Open

lester076 wants to merge 1 commit into
run-llama:mainfrom
lester076:fix-siliconflow-sse-stream-parsing

Conversation

@lester076

Copy link
Copy Markdown

What this PR does

SiliconFlow.astream_chat parsed the streaming response by splitting each raw iter_any() chunk on the literal string "data: ". That breaks in two ways that stream_chat (the sync path) already handles correctly:

  1. A delta whose text contains data: — for example a log excerpt or a code snippet — is split into two fragments and json.loads fails mid-stream, so the whole async stream dies.
  2. An SSE frame spanning two network reads — iter_any() yields arbitrary chunk boundaries, so an event split across reads is decoded as two invalid JSON fragments.

This PR reads the stream the same way the sync path does: buffer bytes, split on newlines, and only parse complete data: lines. SSE comment lines (: keepalive) and blank separators are ignored, [DONE] ends the stream, and a trailing unterminated line is still processed so a stream that ends without a final newline does not lose its last event.

raw on the yielded ChatResponse is now the parsed event payload (matching stream_chat and chat) instead of the raw network chunk.

Why this is the right fix

  • The sync path in the same module already iterates lines and slices off the data: prefix; the async path was the only place that re-derived framing from a byte-chunk split.
  • The parser now depends on SSE framing rather than on how the transport happens to chunk the response, which is what the SSE event stream interpretation rules describe.

Tests

Three regressions cover the reported failure modes; each is RED on main and GREEN with this change, and the existing test_astream_chat still passes.

Test On main With this PR
test_astream_chat_content_containing_data_marker JSONDecodeError passes
test_astream_chat_event_split_across_chunks JSONDecodeError passes
test_astream_chat_ignores_comments_and_blank_lines JSONDecodeError passes
test_astream_chat_multiple_events_in_one_chunk passes (guards no regression) passes
$ uv run -- pytest tests -q
..........                                                               [100%]
10 passed in 1.65s

$ uv run -- ruff check .
All checks passed!

$ uv run -- ruff format --check .
5 files already formatted

Without the source change, the same suite reports 3 failed, 7 passed.

Package version bumped to 0.6.1.

AI assistance disclosure

Per the repository's AI contribution guidelines: this change was prepared with an AI coding agent. I reviewed the diff, reproduced both failure modes locally against main, confirmed the three regressions fail without the fix and pass with it, and ran the package's test and lint commands above.

Closes #22223

…stream_chat

astream_chat split each raw iter_any() chunk on the literal string 'data: ', so a delta whose text contains 'data: ' (log excerpts, code snippets) and any SSE frame spanning two network reads failed with JSONDecodeError and killed the stream. The sync stream_chat path already iterates lines and slices off the data: prefix.

Buffer the decoded bytes, split on newlines, skip SSE comments and blank separators, and parse only complete data: lines. A trailing unterminated line is still processed so a stream without a final newline keeps its last event. raw now holds the parsed event payload, matching stream_chat and chat.

Adds three regressions for the reported failure modes plus one guard for multiple events in a single chunk; existing test_astream_chat still passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: SiliconFlow astream_chat raises JSONDecodeError when streamed content contains "data: "

1 participant