Chunx.Tokenizer behaviour (chunx v0.2.0)

Copy Markdown View Source

Defines the tokenizer interface used by chunkers.

Chunkers accept either a native Tokenizers.Tokenizer or an adapter tuple containing a module that implements this behaviour and its state:

defmodule MyTokenizer do
  @behaviour Chunx.Tokenizer

  @impl true
  def offsets(state, text) do
    # Return half-open byte offsets for every content token.
    {:ok, state.offsets.(text)}
  end
end

tokenizer = {MyTokenizer, %{offsets: &token_offsets/1}}

Offsets are half-open byte ranges. Empty spans are ignored. Other spans are expanded to grapheme boundaries so a chunk cannot contain part of a grapheme. Adapter errors are returned unchanged.

Summary

Types

offset()

A half-open content-token byte range.

t()

A native tokenizer or a {module, state} tokenizer adapter.

Callbacks

offsets(state, text)

Adapter callback returning half-open byte offsets for text.

Functions

count(tokenizer, text)

Returns the number of content tokens in text.

offsets(tokenizer, text)

Returns validated content-token offsets normalized to grapheme boundaries.

Types

offset()

@type offset() :: {non_neg_integer(), non_neg_integer()}

A half-open content-token byte range.

t()

@type t() :: Tokenizers.Tokenizer.t() | {module(), term()}

A native tokenizer or a {module, state} tokenizer adapter.

Callbacks

offsets(state, text)

@callback offsets(state :: term(), text :: binary()) ::
  {:ok, [offset()]} | {:error, term()}

Adapter callback returning half-open byte offsets for text.

Functions

count(tokenizer, text)

@spec count(t(), binary()) :: {:ok, non_neg_integer()} | {:error, term()}

Returns the number of content tokens in text.

offsets(tokenizer, text)

@spec offsets(t(), binary()) :: {:ok, [offset()]} | {:error, term()}

Returns validated content-token offsets normalized to grapheme boundaries.