IPA transcription tokenizer

A library for mentions on Android

Vietnamese tokenizer (Maximum Matching and CRF)

Natural Language Text Processing, NLTK, Data Analysis, Regular Expression, Lexicon Normalization, Statistical Features, Text to Features, Tokenize

Neural Networks: zero to hero

fine-tuned BERT and scikit-learn models for real-time classification of disaster-related tweets, using TensorFlow, Keras, and Transformers. .

Objectron uses string template to design classes

Application to analyze a tweet's positivity using deep learning.

Regular Expression Preprocessor

Coronavirus tweets NLP - Text Classification mini-project work for Data Science course, FCSE, Skopje

A simple brainf**k interpreter made in rust.

Trent + Chippi = TRIPPI Programming Language (Project for CS451)

📄 | Recursive descent parser | Abstract Syntax Trees | Tokenizer

Text to tokes

Node based tokenizers for open source LLMs hosted on HuggingFace.

This is the result of my scientific work on the creation of a question-answer system for the Kazakh language. Google drive presents 4 models with different learning depths, as well as with different datasets on which we trained the models.

A web app to compare pre-built or self-built tokenizers

Train a model using LSTM(Long short-term memory) to classify whether hotel reviews are positive or negative

simple implementation of LLM Tokenizer

🀄 The Jieba Chinese Analyzer for INFINI Pizza.