openai GPT2TokenizerFast的C#实现

Minimal Tokenizer implementation of BertJapanese(cl-tohoku/bert-base-japanese) in C#

.NET Windows Forms app applying concepts of Tokenization, Custom File extension, and Computer Graphics.

tokeniser for math in c#

Token based word generator

This application analyzes sentences. Normalizes sentences and separates them into words. It divides the words into their attachments. ITU uses NLP API for this.

The project is an identity server which has three endpoints to register the user, login the user and to get the refresh token. The user will get the jwt bearer token with authetication scheme. The token is signed with HmacSha256 algorithm. The code contains to add a custom claim to the payload of the token. Implementation of refresh token is als…

This repository provides a lexer and parser for the Pascal programming language. Both lexer and parser can generate errors indicating the location in the code.

Compress Visual Studio solutions into token-lite format for AI workflows. Scrunched in, clean out.

A simple parser for a custom markup language written in C# targeting .NET Core 8.0.

A command-line calculator

A simple tokenizer, parser and AST generator for s-expressions(ish) and CIL (.NET) Arithmetic OpCodes.

A modular generic compiler (tokenizer, parser, generator) for utf-8, utf-16 and utf-32 strings

HTML.NET is an HTML Parser.

Math Expressions

A C# math expression parser and evaluator

Minimal .NET Windows Forms app applying concepts of Tokenization, Custom File extension, and Computer Graphics.

Simple to use parser capable of parsing a usable time object from human input

A tokenizer made to tokenize tweets and get hashtags and mentions as tokens

Pure C# implementation of GPT-style Byte Pair Encoding tokenizer and tiny Transformer playground, CPU-only, no external ML libraries.