Splits text through an ordered sequence of structural boundaries.
Text is split using each configured level in order. Segments that still exceed
:chunk_size are passed to the next level, while adjacent segments are merged
whenever they fit. The default hierarchy tries paragraphs, sentences,
punctuation, whitespace, and finally token boundaries.
Summary
Functions
Recursively splits text toward the :chunk_size target.
Types
@type chunk_opts() :: [chunk_size: pos_integer(), levels: [level(), ...]]
@type level() :: [String.t(), ...] | :whitespace | :tokens
Functions
@spec chunk(binary(), Chunx.Tokenizer.t(), chunk_opts()) :: {:ok, [Chunx.Chunk.t()]} | {:error, term()}
Recursively splits text toward the :chunk_size target.
Options
:chunk_size- Target maximum content-token count (default: 512). An indivisible grapheme may exceed the target.:levels- Ordered splitting levels. Each level is a non-empty list of delimiters,:whitespace, or:tokens. Token splitting is always used as a final fallback when custom levels are exhausted.
Examples
iex> {:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("gpt2")
iex> text = "First paragraph." <> <<10, 10>> <> "Second paragraph."
iex> {:ok, chunks} = Chunx.Chunker.Recursive.chunk(text, tokenizer, chunk_size: 4)
iex> Enum.map(chunks, & &1.text)
["First paragraph." <> <<10, 10>>, "Second paragraph."]