# `Chunx.Tokenizer`
[🔗](https://github.com/preciz/chunx/blob/v0.2.0/lib/chunx/tokenizer.ex#L1)

Defines the tokenizer interface used by chunkers.

Chunkers accept either a native `Tokenizers.Tokenizer` or an adapter tuple
containing a module that implements this behaviour and its state:

    defmodule MyTokenizer do
      @behaviour Chunx.Tokenizer

      @impl true
      def offsets(state, text) do
        # Return half-open byte offsets for every content token.
        {:ok, state.offsets.(text)}
      end
    end

    tokenizer = {MyTokenizer, %{offsets: &token_offsets/1}}

Offsets are half-open byte ranges. Empty spans are ignored. Other spans are
expanded to grapheme boundaries so a chunk cannot contain part of a grapheme.
Adapter errors are returned unchanged.

# `offset`

```elixir
@type offset() :: {non_neg_integer(), non_neg_integer()}
```

A half-open content-token byte range.

# `t`

```elixir
@type t() :: Tokenizers.Tokenizer.t() | {module(), term()}
```

A native tokenizer or a `{module, state}` tokenizer adapter.

# `offsets`

```elixir
@callback offsets(state :: term(), text :: binary()) ::
  {:ok, [offset()]} | {:error, term()}
```

Adapter callback returning half-open byte offsets for `text`.

# `count`

```elixir
@spec count(t(), binary()) :: {:ok, non_neg_integer()} | {:error, term()}
```

Returns the number of content tokens in `text`.

# `offsets`

```elixir
@spec offsets(t(), binary()) :: {:ok, [offset()]} | {:error, term()}
```

Returns validated content-token offsets normalized to grapheme boundaries.

---

*Consult [api-reference.md](api-reference.md) for complete listing*