PyPI
moj-analytical-services/splink
Fast, accurate and scalable probabilistic data linkage with support for multiple SQL backends
MITSep 17, 2026 · metrics 2.10.0
All tags
Catalogue tag
#deduplication
Every repository in the public record carrying this tag — from its GitHub topics or the keywords its package registries publish. Health is measured under the same versioned methodology as the rest of the record.
34 records
Go · npm
prometheus/alertmanager
Prometheus Alertmanager
Apache-2.0Aug 28, 2026 · metrics 2.10.0
PyPI · npm
NVIDIA-NeMo/Curator
Scalable data pre processing and curation toolkit for LLMs
Apache-2.0Sep 25, 2026 · metrics 2.10.0
PyPI
borgbackup/borg
Deduplicating archiver with compression and authenticated encryption.
Custom licenseAug 27, 2026 · metrics 2.10.0
PyPI · crates.io
huggingface/xet-core
xet client tech, used in huggingface_hub
Apache-2.0Sep 16, 2026 · metrics 2.10.0
Go
PlakarKorp/plakar
plakar is a backup solution powered by Kloset and ptar
ISCSep 18, 2026 · metrics 2.10.0
PyPI
opensanctions/nomenklatura
Framework and command-line tools for integrating FollowTheMoney data streams from multiple sources
MITOct 19, 2026 · metrics 2.10.0
npm · PyPI
benseverndev-oss/goldenmatch
Zero-config entity resolution feeding a durable identity layer: messy records from any source become stable golden entities, a Customer 360 with provenance, merge/split and audit. Fellegi-Sunter beats hand-tuned Splink. Arrow-native/Rust, 250M rows in 11.2 min. Python + edge TypeScript (WASM), SQL-native in Postgres & DuckDB, 97 MCP tools + REST.
MITSep 6, 2026 · metrics 2.10.0
crates.io
rustic-rs/rustic_core
rustic_core - library for fast, encrypted, deduplicated backups that powers rustic-rs
Apache-2.0Oct 19, 2026 · metrics 2.10.0
npm
trieb-work/nextjs-turbo-redis-cache
Designed for speed, scalability, and optimized performance, nextjs-turbo-redis-cache is your go-to solution Next.js cache handler for demanding production environments. If you use the app router and self-host, you will probably need this. Development sponsored by https://TRWK.de
MITSep 15, 2026 · metrics 2.10.0
PyPI
sebastienrousseau/bankstatementparser
Parse bank statements — CAMT/ISO 20022, PAIN.001, CSV (EN/DE/FR/ES), OFX/QFX, MT940, PDFs (digital + scanned) — into unified Transactions. Deterministic parsers, LLM/vision fallback, Golden Rule + multi-currency + continuity verification, page provenance, dedupe, review, accuracy evals, CLI + REST API. 100% test coverage.
Custom licenseSep 14, 2026 · metrics 2.10.0
crates.io · npm · PyPI
WithAutonomi/self_encryption
Self-encrypting, content-addressed storage - convergent encryption that splits data into obfuscated chunks. The peer-reviewed core of Autonomi.
Apache-2.0Oct 10, 2026 · metrics 2.10.0
Go
cplieger/docker-fclones-scheduler
Find and deduplicate files on a schedule — reclaim wasted disk space automatically
Apache-2.0Sep 5, 2026 · metrics 2.10.0
Go
franchoy/coldkeep
Coldkeep is a correctness-first, content-addressed cold-storage engine with deterministic restore, integrity verification, safe garbage collection, deduplication, transform-aware packed storage, and a reusable engine/catalog core. v1.x is complete and frozen; v2.x is planned for local-first productization.
Apache-2.0Oct 19, 2026 · metrics 2.10.0
npm
git-stunts/git-cas
Git, freebased: pure CAS that’ll knock your SHAs off. LFS hates this repo!
Apache-2.0Sep 13, 2026 · metrics 2.10.0
PyPI
crunchtools/petit
Log analysis program for use from the command line
Custom licenseOct 3, 2026 · metrics 2.10.0
Go
LarsArtmann/go-idempotency
Go idempotency/deduplication store for CQRS command keys
MITOct 18, 2026 · metrics 2.10.0
npm
MikkelScheike/email-provider-links
Email Provider Links automatically detects which email service (Gmail, Outlook, Yahoo, etc.) someone uses and provides a direct link to their login page, making password reset flows and email-related UX seamless.
MITSep 11, 2026 · metrics 2.10.0
crates.io
nlfiedler/fastcdc-rs
FastCDC implementation in Rust
MITAug 28, 2026 · metrics 2.10.0
PyPI · crates.io · Go
The-Billy-Company/relate
Code similarity search by compression - find near-duplicate files, clone families, and where a pasted snippet came from. No embeddings, no model, no vector database.
Apache-2.0Oct 1, 2026 · metrics 2.10.0
—
mhx/dwarfs
A fast high-compression read-only file system for Linux, FreeBSD, macOS and Windows
Custom licenseSep 9, 2026 · metrics 2.10.0
npm
SIARHEI-SMANTSAR/redux-requests-factory
No repository description published.
MITOct 21, 2026 · metrics 2.10.0
Go
samekind/codexfold
Local-first deduplication and transparent storage for Codex sessions
Apache-2.0Oct 19, 2026 · metrics 2.10.0
PyPI
beshu-tech/deltaglider
Store 4TB in 5GB: S3-compatible storage with 99.9% compression for versioned files
MITOct 19, 2026 · metrics 2.10.0
crates.io
revidiumhq/biblib
Parse, manage, and deduplicate academic citations
Apache-2.0Sep 12, 2026 · metrics 2.10.0
crates.io
nootr/tracing-throttle
High-performance log deduplication and rate limiting for the Rust tracing ecosystem.
MITSep 15, 2026 · metrics 2.10.0
RubyGems
rosette-api/ruby
Babel Street Analytics Client Library for Ruby
Custom licenseOct 6, 2026 · metrics 2.10.0
Go · npm
alexander-kolodka/crestic
Crestic simplifies working with restic. Restic is fast and secure, but managing multiple folders and backup destinations via its CLI can get repetitive. Crestic lets you define repositories and jobs in a single YAML config and run them with simple commands — no magic, just a cleaner way to use restic.
MITSep 11, 2026 · metrics 2.10.0
crates.io
jRimbault/yadf
Yet Another Dupes Finder
MITSep 19, 2026 · metrics 2.10.0
PyPI
Fallen-Breath/pyfastcdc
A high-performance FastCDC 2020 implementation written in Python + Cython
MITSep 5, 2026 · metrics 2.10.0