Chunx.Chunk (chunx v0.2.0)

Struct representing a text chunk with metadata.

Fields

:text - The text content of the chunk
:start_byte - The starting byte offset of the chunk in the original text
:end_byte - The ending byte offset of the chunk in the original text
:token_count - The number of content tokens in the chunk, excluding tokenizer entries without a byte span
:embedding - The embedding vector for the chunk (optional)

Summary

Types

t()

Functions

new(text, start_byte, end_byte, token_count, embedding \\ nil)

Creates a chunk. Byte offsets use a half-open range.

Types

t()

@type t() :: %Chunx.Chunk{
  embedding: Nx.Tensor.t() | nil,
  end_byte: non_neg_integer(),
  start_byte: non_neg_integer(),
  text: String.t(),
  token_count: non_neg_integer()
}

Functions

new(text, start_byte, end_byte, token_count, embedding \\ nil)

@spec new(
  String.t(),
  non_neg_integer(),
  non_neg_integer(),
  pos_integer(),
  Nx.Tensor.t() | nil
) :: t()

Creates a chunk. Byte offsets use a half-open range.