Parser Building Toolkit for JavaScript
-
Updated
Aug 1, 2025 - TypeScript
#
tokenizer
A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Here are 1,433 public repositories matching this topic...
Persian NLP Toolkit
python nlp natural-language-processing tokenizer embeddings persian text-processing dependency-parser farsi pos-tagging persian-nlp normalization lemmatization
-
Updated
Jul 16, 2024 - Python
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
-
Updated
May 7, 2025 - Python
CogComp's Natural Language Processing Libraries and Demos: Modules include lemmatizer, ner, pos, prep-srl, quantifier, question type, relation-extraction, similarity, temporal normalizer, tokenizer, transliteration, verb-sense, and more.
nlp natural-language-processing data-mining big-data tokenizer transliteration similarity named-entity-recognition pos lemmatizer ner pos-tagging dependency-parsing lemmatization relation-extraction natural-language-understanding cogcomp parts-of-speech-tagging
-
Updated
Jul 7, 2023 - Java
The fast scanner generator for Java™ with full Unicode support
java flex parsing cup scanner regexp tokenizer grammar maven-plugin bazel-rules lexer yacc lexer-generator nfa dfa lexical-analyzer dfa-minimization scanner-generator
-
Updated
Jan 1, 2025 - Java
专注于可解释的NLP技术 An NLP Toolset With A Focus on Explainable Inference
-
Updated
Feb 3, 2021 - Java
Solves basic Russian NLP tasks, API for lower level Natasha projects
-
Updated
Oct 17, 2024 - Python
数据标注是一款专门对文本数据进行处理和标注的工具，通过简化快捷的文本标注流程和动态的算法反馈，支持用户快速标注关键词并能通过算法持续减少人工标注的成本和时间。数据标注的过程先由人工标注构建基础，再由自动标注反哺人工标注，最后由人工标注进行纠偏，从而大幅度提高标注的精准度和高效性。数据标注需要依赖开源的数字底座进行人员岗位管控。
java docker elasticsearch tokenizer chinese data-annotations springboot2 vue3 nacos tokenizer-parser data-annotation-tools
-
Updated
Jun 23, 2025 - Java
Open Korean Text Processor - An Open-source Korean Text Processor
natural-language-processing tokenizer korean text-processing korean-text-processing korean-tokenizer
-
Updated
Mar 12, 2024 - Scala
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
translation tokenizer corpus linguistics tagger literature dependency-parser corpus-linguistics lemmatizer corpus-tools corpus-processing corpus-search corpus-statistics stopword corpus-analysis
-
Updated
Jul 27, 2025 - Python
High performance Chinese tokenizer with both GBK and UTF-8 charset support based on MMSEG algorithm developed by ANSI C. Completely based on modular implementation and can be easily embedded in other programs, like: MySQL, PostgreSQL, PHP, etc.
c tokenizer full-text-search chinese-word-segmentation chinese-tokenizer php-tokenizer korean-tokenizer japanese-tokenizer cjk-tokenizer
-
Updated
Oct 29, 2023 - C
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
nlp tokenizer text-processing semeval nlp-library word-segmentation spelling-correction tokenization text-segmentation spell-corrector word-normalization
-
Updated
Jun 2, 2025 - Python
Fast and customizable text tokenization library with BPE and SentencePiece support
python unicode natural-language-processing cpp icu tokenizer machine-translation tokenization bpe sentencepiece
-
Updated
Apr 15, 2025 - C++
🌿 NodeJS PHP Parser - extract AST or tokens
-
Updated
Jul 18, 2025 - JavaScript
Colin's ALM Corner Custom Build Tasks
-
Updated
Nov 7, 2024 - PowerShell
Optimised tokenizer/lexer generator! 🐄 Uses /y for performance. Moo.
-
Updated
May 16, 2023 - JavaScript
Python port of Moses tokenizer, truecaser and normalizer
-
Updated
May 26, 2024 - Python
- Followers
- 11k followers
- Website
- github.com/topics/parsing
- Wikipedia
- Wikipedia