A grammar describes the syntax of a programming language, and might be defined in Backus-Naur form (BNF). A lexer performs lexical analysis, turning text into tokens. A parser takes tokens and builds a data structure like an abstract syntax tree (AST). The parser is concerned with context: does the sequence of tokens fit the grammar? A compiler is a combined lexer and parser, built for a specific grammar.
Persian NLP Toolkit
한국어 자연어처리를 위한 파이썬 라이브러리입니다. 단어 추출/ 토크나이저 / 품사판별/ 전처리의 기능을 제공합니다.
Solves basic Russian NLP tasks, API for lower level Natasha projects
Ekphrasis is a text processing tool, geared towards text from social networks, such as Twitter or Facebook. Ekphrasis performs tokenization, word normalization, word segmentation (for splitting hashtags) and spell correction, using word statistics from 2 big corpora (english Wikipedia, twitter - 330mil english tweets).
An Integrated Corpus Tool With Multilingual Support for the Study of Language, Literature, and Translation
Python port of Moses tokenizer, truecaser and normalizer
DadmaTools is a Persian NLP tools developed by Dadmatech Co.
Bitextor generates translation memories from multilingual websites
Text2Text Language Modeling Toolkit
Vietnamese tokenizer (Maximum Matching and CRF)
Text tokenization and sentence segmentation (segtok v2)
Text to sentence splitter using heuristic algorithm by Philipp Koehn and Josh Schroeder.
A Japanese tokenizer based on recurrent neural networks
A Python implementation of Farasa toolkit
一个轻量且功能全面的中文分词器，帮助学生了解分词器的工作原理。MicroTokenizer: A lightweight Chinese tokenizer designed for educational and research purposes. Provides a practical, hands-on approach to understanding NLP concepts, featuring multiple tokenization algorithms and customizable models. Ideal for students, researchers, and NLP enthusiasts..
The aim of this repository is to show a baseline model for text classification by implementing a LSTM-based model coded in PyTorch. In order to provide a better understanding of the model, it will be used a Tweets dataset provided by Kaggle.
A tokenizer and sentence splitter for German and English web and social media texts.
aim to use JapaneseTokenizer as easy as possible
Essential NLP & ML, short & fast pure Python code
A library for advanced Natural Language Processing towards multi-modal educational items.