Rule-based C++ tokenizer and sentence segmenter. Feed it raw text; get back typed tokens grouped into sentences — handling contractions, abbreviations, hyphenation, ellipses, and numbers without any model weights or dependencies.
Input → "Dr. Jones can't visit on Jan. 31st. Amazing!"
Sentence 1: [Dr.] ABBREVIATION [Jones] WORD [can't] CONTRACTION
[visit] WORD [on] WORD [Jan.] ABBREVIATION
[31] NUMBER [st.] ABBREVIATION [.] PUNCT
Sentence 2: [Amazing] WORD [!] PUNCT
┌───────────────────────────────┐
│ Raw text (std::string) │
└───────────────┬───────────────┘
│ char-by-char
▼
┌───────────────────────────────────────────────────────────┐
│ Tokenizer (FSM) │
│ │
│ START ──letter──▶ IN_WORD ──apostrophe──▶ IN_CONTRACTION│
│ │ │ │
│ │ hyphen │
│ │ ▼ │
│ │ IN_HYPHENATED │
│ │ │
│ │ IN_WORD ──period (≤3 chars)──▶ IN_ABBREV │
│ │ │
│ ├──digit──▶ IN_NUMBER │
│ │ │
│ └──punct/period──▶ IN_PUNCT (merges '...' ellipsis) │
└───────────────────────────────┬───────────────────────────┘
│ std::vector<Token>
▼
┌───────────────────────────────────────────────────────────┐
│ SentenceSegmenter │
│ │
│ Walk tokens; push to current sentence; │
│ on PUNCT('.','!','?') or SENTENCE_END → flush sentence │
│ Abbreviations (ABBREVIATION type) never split sentences │
└───────────────────────────────┬───────────────────────────┘
│ std::vector<std::vector<Token>>
▼
Labelled sentences
Character-by-character FSM processing means a single linear pass over the input with O(1) lookahead. There is no backtracking, no regex engine, and no heap allocation per character — just a state variable, a string buffer, and a switch. This makes boundary cases (e.g. U.S. vs dogs.) explicit and testable rather than buried in regex alternations.
The tokenizer assigns types (ABBREVIATION, CONTRACTION, etc.) that the segmenter uses to decide whether a period ends a sentence. Keeping the two stages separate means each can be tested independently, and the segmenter can be swapped out without touching tokenization logic.
| Type | Examples | Notes |
|---|---|---|
WORD |
Hello, fox, 31st |
Default alphabetic token |
NUMBER |
3, 42 |
Pure digit sequences |
PUNCT |
., !, ?, ... |
Punctuation; ./!/? trigger sentence boundaries |
ABBREVIATION |
Dr., Jan., U.S. |
Word ≤ 3 chars followed by a period; never splits sentences |
CONTRACTION |
don't, I'll, can't |
Word + apostrophe + continuation |
HYPHENATED |
state-of-the-art |
Word segments joined by hyphens |
SENTENCE_END |
— | Explicit marker; can be injected directly into the token stream |
| Tool | Version |
|---|---|
| C++ compiler | C++11 or later (g++ 4.8+, clang++ 3.3+) |
No external libraries. No build system required.
g++ -std=c++11 -o tss \
tss_main.cpp \
tokenizer.cpp \
char_classes.cpp \
sentence_segmenter.cppg++ -std=c++11 -o tokenizer_tests \
tokenizer_tests.cpp \
tokenizer.cpp \
char_classes.cpp \
-I.g++ -std=c++11 -o sentence_segmenter_tests \
sentence_segmenter_tests.cpp \
sentence_segmenter.cpp \
-I.g++ -std=c++11 -o tss_tests \
tss_tests.cpp \
tokenizer.cpp \
char_classes.cpp \
-I.Run the program, type (or pipe) input, then press Ctrl-D to signal end of input:
./tss
The quick brown fox jumps over 3 lazy dogs. Amazing!
^DSentence 1:
[The] WORD
[quick] WORD
[brown] WORD
[fox] WORD
[jumps] WORD
[over] WORD
[3] NUMBER
[lazy] WORD
[dogs] WORD
[.] PUNCT
Sentence 2:
[Amazing] WORD
[!] PUNCT
Pipe text directly:
echo "Dr. Meeden doesn't like state-of-the-art models. Does she?" | ./tss./tokenizer_tests[PASS]simple sentence
[PASS]contractions
[PASS]hyphenation
[PASS]abbreviations
[PASS]ellipsis
Tests run:5
Tests failed: 0
./sentence_segmenter_testsRunning Sentence Segmenter Tests
===============================
[PASS] basic sentence period: sentence count
[PASS] basic sentence period: content
... (21 tests total)
Tests run: 21
Tests failed: 0
rule-based-tss/
├── tokenizer.h/.cpp # FSM tokenizer — core of the pipeline
├── char_classes.h/.cpp # Character classification (LETTER, DIGIT, PERIOD, etc.)
├── token.h # Token struct and TokenType enum
├── sentence_segmenter.h/.cpp # Sentence boundary detection
├── utils.h # String utility declarations
├── tss_main.cpp # CLI entry point (stdin → labelled sentences)
├── tokenizer_tests.cpp # Tokenizer unit tests
├── sentence_segmenter_tests.cpp# Sentence segmenter unit tests
├── tss_tests.cpp # Integration tests (tokenizer end-to-end)
└── test_input.txt # Sample input file
| State | Entered when | Exits when |
|---|---|---|
START |
Initial / after any token is emitted | Any non-whitespace character |
IN_WORD |
Letter seen in START |
Non-letter (emits WORD), or apostrophe / hyphen / short-word period |
IN_NUMBER |
Digit seen in START |
Non-digit (emits NUMBER) |
IN_PUNCT |
Punct or period seen in START |
Non-punct, non-period (emits PUNCT; ... merges naturally) |
IN_CONTRACTION |
Apostrophe seen in IN_WORD |
Non-letter (emits CONTRACTION) |
IN_HYPHENATED |
Hyphen seen in IN_WORD |
Non-letter (emits HYPHENATED) |
IN_ABBREVIATION |
Period seen in IN_WORD with buffer ≤ 3 chars |
Non-letter, non-period (emits ABBREVIATION) |
POSSIBLE_SENTENCE_END |
Reserved for future lookahead disambiguation | — |
- Abbreviation threshold: the
buffer_.length() <= 3check intokenizer.cppcontrols what counts as an abbreviation candidate. Raise it to catch longer abbreviations likeProf.(4 chars before the period). - Known abbreviations list:
sentence_segmenter.hhas anabbreviationsvector stub ready to hold an explicit list for finer-grained sentence boundary control. - Additional token types: add a new
TokenTypeintoken.h, a matchingStateintokenizer.h, and handle the transition inprocess_char.