Add new Chat Transform for dataset-level prompt formatting - #401
Add new Chat Transform for dataset-level prompt formatting#401dameikle wants to merge 5 commits into
Conversation
| ids, tokens = encoding.ids, encoding.tokens | ||
| if self.max_length is not None: | ||
| max_length = self.max_length | ||
| decoder_start_token = self.full_config.decoder_start_token | ||
| if side == "tgt" and getattr(self.full_config.model, "encoder", None) is None and decoder_start_token == "": | ||
| # Decoder-only LM targets are shifted later, so keep one extra token to preserve legacy truncation. | ||
| max_length += 1 | ||
| ids, tokens = ids[:max_length], tokens[:max_length] | ||
| return ids, tokens |
There was a problem hiding this comment.
@vince62s I noticed the max_length from config was being ignored, having got removed in previous commits. Not sure if it was intentional or not, so reintroduced, but can remove if preferred
There was a problem hiding this comment.
I haven't looked at the PR yet but:
- the existing HF streaming support did not work properly for you ?
- yes I removed it because it was conflicting with our max length, the point was maximum generated token including or not including the prompt length, so it was easier to remove it.
There was a problem hiding this comment.
HF streaming needed some extra dataset/config/split URI handling and arbitrary column selection for the datasets I’ve been using. This PR mostly adds on-the-fly chat prompt formatting plus metadata passthrough for prompts.
Thanks, I get the max_length context now. I’d noticed it in the EuroLLM recipes and went down the rabbit hole. I’ve pulled the huggingface_tokenize.max_length restoration out of this PR.
|
maybe it can be done in another PR, but bear in mind that https://github.com/eole-nlp/eole/blob/main/eole/bin/run/serve.py uses a "apply_chat_template" that could be refactored with the chat transform. That may simplify a little bit the work in serve.py |
|
I started doing the above and have ended up with a bigger refactor to accommodate other shapes, so I'll probably close this PR and come back with another. Sorry for the wasted time on this one. |
This pull request introduces support for Hugging Face streaming datasets and dataset-level transform configuration overrides, along with a new
chattransform for rendering chat-style prompts. It also improves tokenizer truncation and updates documentation to reflect these new features. The most important changes are grouped below:Hugging Face Streaming Dataset Support:
hf://URIs in config files, allowing direct streaming from Hugging Face with flexible URI forms for split and config selection. All fields (source, target, score) must reference the same dataset/split except for the last field name. [1] [2] [3] [4]hf://URIs to ensure correct dataset, config, and split alignment between source and target fields. [1] [2]Dataset-Level Transform Overrides:
transforms_configsin the data config, enabling dataset-level overrides for supported transforms (currently onlychat). Validation ensures overrides are only used with enabled transforms that support them. [1] [2] [3] [4]New Chat Transform:
chattransform (eole.transforms.chat.ChatTransform) to render chat-style prompts using a configurable message template, with support for dataset-level message overrides. [1] [2] [3]Tokenizer Improvements:
max_length, with special handling for decoder-only models to preserve legacy truncation behavior.Documentation and Dependency Updates:
chattransform with configuration examples. [1] [2]jinja2as a new dependency for chat template rendering.