Skip to content

epic: dictionary-based word-breakers 🔬 #12142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
mcdurdin wants to merge 92 commits into
base: master
Choose a base branch
from
Draft

epic: dictionary-based word-breakers 🔬 #12142

mcdurdin wants to merge 92 commits into from
+993 −283

Conversation

@mcdurdin
Copy link
Member

@mcdurdin mcdurdin commented Aug 9, 2024 •
edited
Loading

jahorton and others added 23 commits August 2, 2024 10:55
@jahorton
refactor(common/models): split trie-model implementation into separat…
7644d38 
…e components
@jahorton @mcdurdin
chore(common/models): Apply suggestions from code review
98ccdef 
Co-authored-by: Marc Durdin <marc@durdin.net>
@jahorton
feat(common/models/wordbreakers): starting on dictionary-based wordbr…
425a0a0 
…eaking
@jahorton
feat(common/models/wordbreakers): actual first-pass implementation
7b456c1 
Only wordbreaks anything AFTER the last space / ZWNJ.  Doesn't bother with anything before it.
@jahorton
feat(common/models/wordbreakers): pass 2 - should now tokenize full c…
2da7b7c 
…ontext
@jahorton
feat(common/models/wordbreakers): dict-breaker helper unit tests (BMP)
5568167
@jahorton
feat(common/models/wordbreakers): dict-breaker helper unit tests (non…
d40a09d 
…-BMP)
@jahorton
feat(common/models/wordbreakers): dict-breaker unit tests with simple…
10f91e0 
… mocked-fixture
@jahorton
fix(common/models/wordbreakers): blocks empty span output on empty co…
1d07e9f 
…ntext
@jahorton
fix(common/models): update re base branch change
3e82434
@jahorton
docs(common/models): updates dict-breaker comments
56e4052
@jahorton
change(common/models/wordbreakers): allows wordbreaker unit tests to …
66c156e 
…use Tries
@jahorton
feat(common/models/wordbreakers): baby's first khmer wordbreaking test
3181a4c
@jahorton
chore(common/models/wordbreakers): comment tweak
07f6747
@jahorton
feat(common/models/wordbreakers): rejoins adjacent single-point spans…
8c5e54a 
… from unmatched chars in path
@jahorton
fix(common/models/wordbreakers): handling of penalty transitions
766992d
@jahorton
change(common/models): use spread operator to split on codepoints
99e1f4b 
Addresses #10973 (comment)
@jahorton
chore(common/models/wordbreakers): Merge branch 'feat/common/models/w…
51b62e6 
…ordbreakers/dict-breaker-start' into change/common/models/wordbreakers/unit-test-trie-access
@jahorton
chore(common/models/wordbreakers): Merge base branch into feat/common…
295c70b 
…/models/wordbreakers/fuse-dict-unmatched-chars
@jahorton
chore(common/models): drops unit tests for replaced func
8ae6a62
@jahorton
chore(common/models/wordbreakers): Merge branch 'feat/common/models/w…
258763b 
…ordbreakers/dict-breaker-start' into change/common/models/wordbreakers/unit-test-trie-access
@jahorton
chore(common/models/wordbreakers): Merge base branch fixes into feat/…
8942883 
…common/models/wordbreakers/fuse-dict-unmatched-chars
@mcdurdin
chore: establish dictionary breakers epic
001126b
@keymanapp-test-bot keymanapp-test-bot bot added the user-test-missing User tests have not yet been defined for the PR label Aug 9, 2024
@keymanapp-test-bot
Copy link

keymanapp-test-bot bot commented Aug 9, 2024 •
edited
Loading

User Test Results

Test specification and instructions

User tests are not required

Test Artifacts

  • Android
    • Keyman for Android apk - build : all tests passed (no artifacts on BuildLevel "build")
    • FirstVoices Keyboards for Android apk - build : all tests passed (no artifacts on BuildLevel "build")
    • FirstVoices Keyboards for Android apk (old PRs) - build : all tests passed (no artifacts on BuildLevel "build")
    • KeyboardHarness apk - build : all tests passed (no artifacts on BuildLevel "build")
    • Keyman for Android apk (old PRs) - build : all tests passed (no artifacts on BuildLevel "build")
    • KMSample1 apk - build : all tests passed (no artifacts on BuildLevel "build")
    • KMSample2 apk - build : all tests passed (no artifacts on BuildLevel "build")
  • Developer
    • Keyman Developer - build : all tests passed (no artifacts on BuildLevel "build")
    • Compiler Regression Tests - build : all tests passed (no artifacts on BuildLevel "build")
    • Keyman Developer (old PRs) - build : all tests passed (no artifacts on BuildLevel "build")
    • kmcomp.zip - build : all tests passed (no artifacts on BuildLevel "build")
    • kmcomp.zip (old PRs) - build : all tests passed (no artifacts on BuildLevel "build")
  • iOS
    • Keyman for iOS (simulator image) - build : all tests passed (no artifacts on BuildLevel "build")
    • FirstVoices Keyboards for iOS (simulator image) - build : all tests passed (no artifacts on BuildLevel "build")
    • FirstVoices Keyboards for iOS (simulator image) (old PRs) - build : all tests passed (no artifacts on BuildLevel "build")
    • Keyman for iOS (simulator image) (old PRs) - build : all tests passed (no artifacts on BuildLevel "build")
  • Keyboards
    • Test Keyboards - build : all tests passed (no artifacts on BuildLevel "build")
  • Web
    • KeymanWeb Test Home - build : all tests passed (no artifacts on BuildLevel "build")
  • Windows
    • Keyman for Windows - build : all tests passed (no artifacts on BuildLevel "build")
    • FirstVoices Keyboards for Windows - build : all tests passed (no artifacts on BuildLevel "build")
    • FirstVoices Keyboards for Windows (old PRs) - build : all tests passed (no artifacts on BuildLevel "build")
    • Keyman for Windows (old PRs) - build : all tests passed (no artifacts on BuildLevel "build")
    • Text Editor (32 bit) - build : all tests passed (no artifacts on BuildLevel "build")
    • Text Editor (64 bit) - build : all tests passed (no artifacts on BuildLevel "build")

@keymanapp-test-bot keymanapp-test-bot bot added this to the A18S8 milestone Aug 9, 2024
@keyman-server keyman-server modified the milestones: A19S15, A19S16 Nov 8, 2025
@keyman-server keyman-server modified the milestones: A19S16, A19S17 Nov 22, 2025
@keyman-server keyman-server modified the milestones: A19S17, A19S18 Dec 6, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

common/ developer/ epic-dict-breaker web/predictive-text/ web/ windows/engine/ windows/

Projects

Keyman
Status: No status

Milestone

A19S18

Development

Successfully merging this pull request may close these issues.

6 participants

@mcdurdin @jahorton @keyman-server @darcywong00 @rc-swag