🦛 CHONK docs with Chonkie ✨ — The lightweight ingestion library for fast, efficient and robust RAG pipelines
🦛 CHONK your texts with Chonkie ✨ Type-friendly, light-weight, fast and super-simple chunking library
Open-source toolkit for reliable RAG pipelines: convert PDFs to Markdown, clean documents, inspect chunks, compare chunking strategies, and enrich metadata for LLM applications.
FastCDC implementation in Python https://pypi.org/project/fastcdc/
SmartChunk is a lightweight, structure-aware semantic chunking toolkit designed to supercharge RAG (Retrieval-Augmented Generation) and LLM pipelines. Unlike naive splitters that break text arbitrarily, SmartChunk respects document structure (headings, lists, tables, code blocks) and semantic flow, ensuring cleaner, more coherent chunks.
Chunkpad is designed to prepare documents for Retrieval-Augmented Generation (RAG) pipelines and AI applications.
Go implementation of the AE chunking algorithm.
Android Resumable Uploads SDK from Fastpix
A high-performance FastCDC 2020 implementation written in Python + Cython
A Rust library for Content-Defined Chunking (CDC).
A nodejs chunking system
Implementation of an interactive chatbot for summarizing legal and policy documents. Includes data preprocessing (cleaning, tokenization, hierarchical chunking), extractive TF-IDF baselines, and fine-tuned abstractive models (DistilBART, LED). Integrates a retrieval layer for document relevance and uses ROUGE, BLEU, and cosine similarity metrics.
Explore and benchmark the world of data chunking algorithms in 'ChunkingChampions' - a competitive arena to determine the most efficient and effective chunking strategies for varied data sizes.
[2024-2] AI 기반 회의 지원 플랫폼 서비스 "Clerker"
A smol Go package for splitting text into chunks while preserving semantic meaning.
Documind.AI is a full-stack Generative AI application that allows users to interact with PDF documents through a chat-based interface.
Cross-industry RAG benchmarking platform, 10 chunking strategies, 3 retrieval modes (BM25/Vector/Hybrid RRF), 5-node LangGraph agent, NDCG@K · MRR · MAP · Ragas · adversarial robustness evaluation across healthcare, finance & legal domains
This is my curated repo regarding Retrieval Augmented Generation(RAG).
Add a description, image, and links to the chunking-algorithm topic page so that developers can more easily learn about it.
To associate your repository with the chunking-algorithm topic, visit your repo's landing page and select "manage topics."