@spec chunk(binary(), Chunx.Tokenizer.t(), chunk_opts()) ::
  {:ok, [Chunx.Chunk.t()]} | {:error, term()}

Splits text into overlapping chunks using word boundaries.

Options

• :chunk_size - Target maximum content-token count (default: 512). A word that exceeds the target is kept intact.
• :chunk_overlap - Overlap as a token count or a fraction in the range [0.0, 1.0) (default: 0.25).

Examples

iex> {:ok, tokenizer} = Tokenizers.Tokenizer.from_pretrained("gpt2")
iex> Chunx.Chunker.Word.chunk("Some text to split", tokenizer, chunk_size: 3, chunk_overlap: 1)
{
  :ok,
  [
    %Chunx.Chunk{end_byte: 12, start_byte: 0, text: "Some text to", token_count: 3},
    %Chunx.Chunk{end_byte: 18, start_byte: 9, text: " to split", token_count: 2}
  ]
}