Skip to content

v1.3.7

tagged this 13 Jun 22:29
Bug fixes
- Fix inconsistent behavior of `utf::segmenter` (and thus `icu_tokenizer`) for
    different locales. Thanks @CanoeFZH and @tng-konrad for helping debug
    this!

Enhancements
- Allow for specifying the language and country for locale generation in
    setting up `utf::segmenter` (and thus `icu_tokenizer`)
- Allow for suppression of `<s>` and `</s>` tags within `icu_tokenizer`,
    mostly useful for information retrieval experiments with unigram words.
    Thanks @husseinhazimeh for the suggestion!
- Add a `default-unigram-chain` filter chain preset which is suitable for
    information retrieval experiments using unigram words. Thanks
    @husseinhazimeh for the suggestion!
Assets 2
Loading