# `Chunx.Chunker`
[🔗](https://github.com/preciz/chunx/blob/v0.2.1/lib/chunx/chunker.ex#L1)

Defines the shared callback types for chunkers.

Chunker input must be valid UTF-8. Invalid text is returned as
`{:error, {:invalid_text, :invalid_utf8}}` before tokenization begins.

# `chunk_result`

```elixir
@type chunk_result() ::
  {:ok, [Chunx.Chunk.t()] | [Chunx.SentenceChunk.t()]} | {:error, term()}
```

# `embedding_fun`

```elixir
@type embedding_fun() :: ([String.t()] -> [Nx.Tensor.t()])
```

# `chunk`

```elixir
@callback chunk(
  text :: String.t(),
  tokenizer :: Chunx.Tokenizer.t(),
  opts_or_embedding_fun :: keyword() | embedding_fun()
) :: chunk_result()
```

Splits text using a tokenizer.

Semantic chunkers take an embedding function instead of options as the third
argument and accept options as a fourth argument.

## Parameters
  * `text` - The text to chunk
  * `tokenizer` - The tokenizer to use
  * `opts_or_embedding_fun` - Options specific to the chunking strategy, or
    the embedding function used by a semantic chunker

# `chunk`
*optional* 

```elixir
@callback chunk(
  text :: String.t(),
  tokenizer :: Chunx.Tokenizer.t(),
  embedding_fun(),
  opts :: keyword()
) :: chunk_result()
```

---

*Consult [api-reference.md](api-reference.md) for complete listing*