Chunx.Chunker.Sentence (chunx v0.2.0)

Splits text at sentence boundaries, with optional whole-sentence overlap.

Summary

Types

chunk_opts()

Functions

chunk(text, tokenizer, opts \\ [])

Splits text into overlapping chunks using sentence boundaries.

Types

chunk_opts()

@type chunk_opts() :: [
  chunk_size: pos_integer(),
  chunk_overlap: non_neg_integer(),
  min_sentences_per_chunk: pos_integer(),
  delimiters: [String.t(), ...],
  short_sentence_threshold: pos_integer()
]

Functions

chunk(text, tokenizer, opts \\ [])

@spec chunk(binary(), Chunx.Tokenizer.t(), chunk_opts()) ::
  {:ok, [Chunx.SentenceChunk.t()]} | {:error, term()}

Splits text into overlapping chunks using sentence boundaries.

Options

:chunk_size - Target maximum content-token count (default: 512). Whole sentences and :min_sentences_per_chunk take precedence over this target.
:chunk_overlap - Token budget for whole sentences repeated between consecutive chunks (default: 128). Must be non-negative and less than :chunk_size.
:min_sentences_per_chunk - Minimum sentence count before applying the size target (default: 1).
:delimiters - Strings that end sentences (default: [".", "!", "?", "\n"]).
:short_sentence_threshold - Sentences shorter than this byte count are joined to an adjacent sentence (default: 6).