unicode

Branch: main

Switch branches or tags

Branch: main
Go to file t

413 files indexed

History
Code

segmentation/sentence.mbt
1///|2/// A forward cursor over default UAX #29 sentence segments. Create with3/// `sentences`; trailing spaces and punctuation remain in the returned views.4struct Sentences(TextSegments)56///|7/// Split text at the default Unicode 18.0.0 sentence boundaries, retaining8/// every character. Closing punctuation, trailing spaces, and paragraph9/// separators are included according to UAX #29. No abbreviation dictionary10/// or locale tailoring is applied. The supplied view is an independent text.11/// Construction takes O(n) time and space; traversal does not copy text.12/// Unpaired surrogates have Sentence_Break=Other; valid pairs stay together.13///14/// ```mbt check15/// test {16///   debug_inspect(17///     @segmentation.sentences("Hello! Next.")18///     .iter()19///     .map(s => s.to_owned())20///     .collect(),21///     content="[\"Hello! \", \"Next.\"]",22///   )23/// }24/// ```25pub fn sentences(text : StringView) -> Sentences {26  Sentences({ text, ends: sentence_segment_ends(text), index: 0, offset: 0, })27}2829///|30/// Return the next segment, or an empty view at the input's end on exhaustion.31pub fn Sentences::next(self : Sentences) -> StringView {32  self.0.next()33}3435///|36/// Iterate over remaining segments, consuming the same cursor as `next`.37pub fn Sentences::iter(self : Sentences) -> Iter[StringView] {38  self.0.iter()39}40