Chunx.SentenceChunk (chunx v0.2.0)

Struct representing a sentence chunk with metadata.

Fields

:text - The text content of the chunk
:start_byte - The starting byte offset of the chunk in the original text
:end_byte - The ending byte offset of the chunk in the original text
:token_count - The number of content tokens in the chunk, excluding tokenizer entries without a byte span
:sentences - The list of Chunx.Chunk structs in the chunk

Summary

Types

t()

Functions

new(text, start_byte, end_byte, token_count, sentences)

Creates a sentence chunk. Byte offsets use a half-open range.

Types

t()

@type t() :: %Chunx.SentenceChunk{
  end_byte: non_neg_integer(),
  sentences: [Chunk.t()],
  start_byte: non_neg_integer(),
  text: String.t(),
  token_count: pos_integer()
}

Functions

new(text, start_byte, end_byte, token_count, sentences)

@spec new(String.t(), non_neg_integer(), non_neg_integer(), pos_integer(), [Chunk.t()]) ::
  t()

Creates a sentence chunk. Byte offsets use a half-open range.