Changelog

All notable changes to Chunx are documented in this file.

The format follows Keep a Changelog, and this project follows Semantic Versioning.

Unreleased

Added Chunx.Chunker.Recursive, which progressively splits text at paragraph, sentence, punctuation, whitespace, and token boundaries.
Added custom tokenizer adapters through Chunx.Tokenizer.
Added an example script covering the available non-semantic chunkers.
Added opt-in real-model embedding integration tests, runnable with mix test --include integration or mix test --only integration.
Added property tests for reconstruction, offsets, overlap, Unicode, tokenizer windows, and semantic metadata.

Standardized token_count across all chunkers to mean content tokens, excluding tokenizer entries without a byte span.
Made tokenizer-derived chunk boundaries grapheme-safe.
Reduced intermediate allocations and repeated traversal in the chunkers.
Updated Nx, EXLA, Bumblebee, Scholar, StreamData, and related dependencies.

Preserved exact byte offsets for repeated text and Unicode content.
Prevented Sentence from stalling when overlap contains the entire previous chunk.
Preserved Token's trailing overlap window while keeping indivisible graphemes intact.
Propagated tokenizer failures instead of crashing on failed encodings.
Rejected malformed tokenizer responses and offsets with tagged errors.
Prevented Semantic from silently dropping sentences when an embedding function returns the wrong number of embeddings.
Preserved all input text when grouping sentences and placing chunk boundaries.

Initial Hex.pm release.
Added Token, Word, Sentence, and Semantic chunking strategies.
Added configurable chunk sizes, overlap, sentence grouping, and semantic similarity thresholds.
Added byte-offset and token-count metadata through Chunx.Chunk and Chunx.SentenceChunk.
Added caller-provided Nx embedding support for semantic chunking.