Hands-on project for data collection & extraction (Week 2). Implements OCR (Tesseract), web scraping (arXiv), PDF text extraction, automatic speech recognition (Whisper), and dataset cleaning/deduplication.

Big data homework solutions

Aurora karton for calculating minhash from input dataset.

中文文本数据去重小工具

MPEI Project - Search repeated news (using Bloom filter) and similar news (using MinHash) from a news API.

Fast Sketching for Weighted Sets

Rust implementation of alignment-free similarity estimation methods.

Implementation of the paper "Finding Highly Correlated Pairs with Powerful Pruning" in Java.

Trabalho Prático da UC de Métodos Probabilísticos para Engenharia Informática, UA 2019/2020

Sparkling Water is a scalable system for detecting, merging, and clustering similar server processes based on interaction logs. Using Apache Spark, MinHash, LSH, and time-series hashing (SSH, BSeSH), it efficiently identifies behavior patterns in large server infrastructures for performance optimization, anomaly detection, and system analysis.

SpellChecker: an application to check for spell errors.

(WIP) HTTP server that deploy distributes Vokter (https://github.com/vokter/vokter) through a REST API.

Finding Similar Items: Textually Similar Documents

Implementation for the attacks of the paper "Locality-Sensitive Hashing Does Not Guarantee Privacy! Attacks on Google's FLoC and the MinHash Hierarchy System"

An implementation of the MinHashing algorithm in C using POSIX threads.

This project implements Locality-Sensitive Hashing (LSH) for efficient document similarity detection. Instead of performing exhaustive pairwise comparisons between documents, LSH uses probabilistic techniques to quickly identify similar document pairs, making it particularly effective for large document collections.

A family of high-quality 64-bit hash functions delivering outstanding performance across diverse applications.

Probability Methods for Informatics Engineering | UA 2018/2019

Assignment-2 for CS F469 Information Retrieval Course