Build unigram and bigram language models, implement Laplace smoothing and use the models to compute the perplexity of test corpora.
-
Updated
Jun 24, 2017 - Python
#
unigram
Here are 28 public repositories matching this topic...
High-precision, Byte-Fallback Unigram tokenizer with dual-offset tracking, arithmetic isolation, and multilingual Unicode protection.
nlp rust deep-learning tokenizer help-wanted up-for-grabs hacktoberfest unigram good-first-issue pyo3 subwords byte-fallback
-
Updated
Sep 8, 2026 - Python
NLP extra project at AUT Artificial Intelligence course (Fall 2020)
-
Updated
Feb 15, 2021 - Python
Word2Vec using Hierarchy Softmax and Negative Sampling with Unigram & Subsampling
-
Updated
Feb 7, 2020 - Python
Python Web Crawler implementing Iterative Deepening Depth Search
-
Updated
Aug 24, 2023 - Python
Performance evaluation of sentiment classification on movie reviews
natural-language-processing sentiment classification tf-idf bigrams unigram ensemble-machine-learning
-
Updated
May 26, 2018 - Python
UNB Fall-2018 NLP Assignments 💬
nlp sentiment-analysis baseline bigrams language-models hidden-markov-model python27 unb unigram information-retrieval-based-chatbot
-
Updated
Feb 25, 2019 - Python
Word segmentation to create unigrams in Portuguese (pt-br)
-
Updated
Jun 17, 2023 - Python
Global NIPS Paper Implementation Challenge - Plagiarism Detection on Electronic Text Based Assignments Using Vector Space Model (iciafs14)
natural-language-processing trigrams vector-space-model cosine-similarity jaccard-similarity unigram plagiarism-detection bigram nips-challenge
-
Updated
Jan 31, 2018 - Python
Final AI course of CE department at Amirkabir University of Technology (Tehran Polytechnic) - Winter 2020.
-
Updated
Sep 20, 2022 - Python
easy to use mixture of unigram topic modeling tool
-
Updated
Nov 20, 2019 - Python
Sentiment Classification exercise with perceptron, feed-forward multilayer net, LSTM RNN, and RCNN!
-
Updated
Mar 2, 2020 - Python
Community-maintained UnigramPlus fork for current NVDA releases. Code maintenance is driven by GPT-5.5 powered Codex; the human maintainer reviews functionality and participates in discussions.
-
Updated
May 17, 2026 - Python
Build a bigram language model from scratch with PyTorch to understand the basics behind modern AI language models like GPT and LLaMA.
nlp statistics vocabulary language-detection smoothing bigrams preprocessing language-models tokenization classification-algorithm cross-entropy unigram probabilities trigram-model laplace-smoothing perplexity
-
Updated
Sep 22, 2026 - Python
a probabilistic language identification system that identifies the language of a sentence
-
Updated
Apr 12, 2024 - Python
200k-vocab SentencePiece (Unigram) tokenizer for German-primary LLMs — German/English/code, low fertility, byte-fallback, chat-template tokens. From the Auralis/Helix project.
-
Updated
Jun 19, 2026 - Python
SentencePiece Tokenizer Wrapper implementation for PLDR-LLM with KV cache and G-cache
machine-learning natural-language-processing deep-learning tokenizer pytorch transformer unigram bpe sentencepiece large-language-models llm pldr-llm
-
Updated
Feb 20, 2025 - Python
Add this topic to your repo
To associate your repository with the unigram topic, visit your repo's landing page and select "manage topics."