unicode

Branch: main

Switch branches or tags

Branch: main
Go to file t

413 files indexed

History
Code

README.md

unicode

Unicode support for MoonBit.

This library implements selected Unicode standards for normalization, case and category lookup, Punycode, IDNA domain processing, bidirectional text, and text segmentation. The generated tables target Unicode 18.0.0.

Features

  • Unicode Normalization Forms from UAX #15: NFD, NFC, NFKD, and NFKC
  • Default full case folding, canonical caseless matching, and NFKC_Casefold identifier matching from Unicode section 3.13
  • Punycode encoding and decoding from RFC 3492
  • IDNA processing from UTS #46, including mapping, validation, Bidi checks, joiner checks, and DNS length checks, with optional IDNA2008 character and contextual validation
  • Unicode Bidirectional Algorithm support from UAX #9
  • Default line breaking opportunities from UAX #14
  • Extended grapheme cluster, word, and sentence segmentation from UAX #29
  • General_Category, East_Asian_Width, Line_Break, Joining_Type, Joining_Group, and Hangul_Syllable_Type lookup, binary property predicates (White_Space, Alphabetic, Lowercase, Uppercase, XID_Start, XID_Continue, Default_Ignorable_Code_Point, Grapheme_Extend, Regional_Indicator, Prepended_Concatenation_Mark, Emoji_Presentation, Emoji_Modifier, Emoji_Modifier_Base, Extended_Pictographic), emoji variation sequences, and simple/full case mapping from the Unicode Character Database

Installation

Add only the feature modules your application needs:

moon add moonbit-community/unicode-data
moon add moonbit-community/unicode-normalization
moon add moonbit-community/unicode-casefold
moon add moonbit-community/punycode
moon add moonbit-community/unicode-bidi
moon add moonbit-community/unicode-segmentation
moon add moonbit-community/unicode-idna

Import the packages you need in moon.pkg:

import {
  "moonbit-community/unicode-data" @ucd,
  "moonbit-community/unicode-normalization" @normalization,
  "moonbit-community/unicode-casefold" @casefold,
  "moonbit-community/punycode",
  "moonbit-community/unicode-idna" @idna,
  "moonbit-community/unicode-bidi" @bidi,
  "moonbit-community/unicode-segmentation" @segmentation,
}

MoonBit uses the last package path segment as the default alias, which is not a valid identifier for the hyphenated unicode-* modules, so give those imports an explicit alias. The examples below use @ucd, @normalization, @casefold, @punycode, @idna, @bidi, and @segmentation. Existing users may continue depending on moonbit-community/unicode; it is an umbrella compatibility module that preserves the old package paths.

Usage

Normalization and case folding accept StringView, so callers can pass slices without copying them first. String arguments convert automatically; text-producing operations still return String.

Normalization

let composed = @normalization.nfc("e\u{0301}") // "é"
let decomposed = @normalization.nfd("é") // "e" + combining acute
let compatible = @normalization.nfkc("fi") // "fi"

let already_nfc = @normalization.is_normalized(
  composed,
  @normalization.NFC,
)

let normalized = @normalization.normalize(
  "text",
  @normalization.NFKC,
)

Case folding and caseless matching

let folded = @casefold.case_fold("Straße") // "strasse"
let same_case = @casefold.caseless_match("Straße", "STRASSE") // true
let same_text = @casefold.canonical_caseless_match("Å", "a\u{030A}") // true
let search_key = @casefold.canonical_case_fold("Å") // "a\u{030A}"
let folded_id = @casefold.nfkc_casefold("A\u{00AD}") // "a"
let same_id = @casefold.identifier_caseless_match("A\u{00AD}", "a") // true

These operations follow Unicode §3.13 and are locale-independent. Default full folding uses CaseFolding.txt statuses C and F, with expansions such as ß → ss; it does not normalize text or apply Turkic tailoring. I folds to i, while İ folds to i plus U+0307. Cherokee folds to uppercase, so a folded string is not necessarily lowercase.

For canonically equivalent text, canonical_case_fold(s) computes NFD(case_fold(NFD(s))) (D145). Store and compare these keys, or search folded text with a folded query. Folding can change text length, so result offsets do not directly identify positions in the original text.

nfkc_casefold(s) applies the official NFKC_CF mappings and then NFC (R5). This merges compatibility differences and removes default-ignorable characters. identifier_caseless_match(a, b) first applies NFD to both inputs before NFKC_Casefold (D147), including the Greek iota-subscript edge cases. It tests equality; identifier validation and confusable detection are separate operations.

Punycode

encode and decode use checked errors. Use try! for examples or handle the error explicitly in application code.

let encoded = try! @punycode.encode("münchen") // "mnchen-3ya"
let decoded = try! @punycode.decode("mnchen-3ya") // "münchen"

let chinese = try! @punycode.encode("中文") // "fiq228c"

IDNA

let ascii = try! @idna.to_ascii("münchen.de")
// "xn--mnchen-3ya.de"

let unicode = try! @idna.to_unicode("xn--mnchen-3ya.de")
// "münchen.de"

let checked = try! @idna.to_ascii(
  "example.com",
  use_std3_ascii_rules=true,
  check_hyphens=true,
  check_bidi=true,
  check_joiners=true,
  verify_dns_length=true,
)

let idna2008_checked = try! @idna.to_ascii(
  "ÖBB.at",
  check_idna2008=true,
  check_contexto=true,
)
// "xn--bb-eka.at" — UTS #46 mapping lowercases the input.

Both functions support IDNA2008 label validation through the opt-in check_idna2008 and check_contexto options, which default to false. Enable both alongside the default checks. Validation applies after UTS #46 mapping and NFC normalization, including to Unicode labels decoded from Punycode.

To handle validation failures:

try @idna.to_ascii("example..com") catch {
  err => println("invalid domain: \{err}")
} noraise {
  ascii => println(ascii)
}

Bidi

let direction = @bidi.detect_direction("Hello World") // LTR
let needs_bidi = @bidi.requires_bidi("Hello \u{05E9}\u{05DC}\u{05D5}\u{05DD}")

let paragraph = @bidi.process("abc\u{05D0}\u{05D1}")
let visual = @bidi.reorder_string(paragraph)
let order = @bidi.reorder(paragraph)

let forced = @bidi.process(
  "\u{05D0}\u{05D1}\u{05D2}",
  direction=@bidi.Direction::LTR,
)

Segmentation

// "é" (e + combining acute), a flag, and a family emoji are one cluster each.
let count = @segmentation.graphemes("e\u{0301}🇺🇳👨‍👩‍👧").iter().count() // 3

for cluster in @segmentation.graphemes("e\u{0301}x") {
  // Each cluster is a view into the input.
  println("\{cluster.start_offset()}: \{cluster}")
}

// `next` returns an empty view at the end instead of an `Option`, which
// avoids an allocation per cluster in hot loops.
let clusters = @segmentation.graphemes("e\u{0301}x")
while clusters.next() is cluster && !cluster.is_empty() {
  println(cluster)
}

Case And Category Data

let category = @ucd.general_category('A') // Lu
let group = category.group() // L
let is_punct = @ucd.is_punctuation('!') // true: General_Category group P
let width = @ucd.east_asian_width('世') // W
let can_start = @ucd.is_xid_start('A') // true
let can_continue = @ucd.is_xid_continue('0') // true
let is_space = @ucd.is_white_space('\u{3000}') // true
let is_letter = @ucd.is_alphabetic('\u{093E}') // true (Other_Alphabetic)
let is_lower = @ucd.is_lowercase('\u{00AA}') // true, although gc is Lo
let is_upper = @ucd.is_uppercase('\u{2160}') // true, although gc is Nl
let joining = @ucd.joining_type('\u{0628}') // DualJoining
let is_ignorable = @ucd.is_default_ignorable_code_point('\u{200D}') // true
let is_emoji = @ucd.is_emoji_presentation('\u{2764}') // false: text by default
let has_vs = @ucd.has_emoji_variation_sequence('\u{2764}') // true: U+FE0F selects emoji
let group = @ucd.joining_group('\u{062A}') // Beh
let break_class = @ucd.line_break('\u{4E2D}') // ID
let hangul = @ucd.hangul_syllable_type('\u{AC00}') // LVSyllable

let simple = @ucd.to_simple_uppercase('a') // 'A'
let full = @ucd.to_uppercase('\u{00DF}') // "SS"
let lower = @ucd.to_lowercase('\u{0130}') // "i" + combining dot above
let shout = @ucd.uppercase("Straße") // "STRASSE": one string, no per-character allocation

Public Packages

moonbit-community/unicode-data

Root package for Unicode Character Database helpers.

API Description
general_category(Char) -> GeneralCategory Return the two-letter Unicode General_Category value.
GeneralCategory::group() -> GeneralCategoryGroup Return the one-letter category group.
is_punctuation(Char) -> Bool Test whether the character is punctuation (General_Category group P), using a compact table independent of the General_Category trie.
east_asian_width(Char) -> EastAsianWidth Return the UAX #11 East_Asian_Width value: A, F, H, N, Na, or W.
is_white_space(Char) -> Bool Test the Unicode White_Space property.
is_alphabetic(Char) -> Bool Test the Unicode Alphabetic property.
is_lowercase(Char) -> Bool Test the Unicode Lowercase property (broader than General_Category Ll).
is_uppercase(Char) -> Bool Test the Unicode Uppercase property (broader than General_Category Lu).
is_xid_start(Char) -> Bool Test the Unicode XID_Start identifier property.
is_xid_continue(Char) -> Bool Test the Unicode XID_Continue identifier property.
is_default_ignorable_code_point(Char) -> Bool Test the Unicode Default_Ignorable_Code_Point property.
is_grapheme_extend(Char) -> Bool Test the Unicode Grapheme_Extend property.
is_regional_indicator(Char) -> Bool Test the Unicode Regional_Indicator property.
is_prepended_concatenation_mark(Char) -> Bool Test the Unicode Prepended_Concatenation_Mark property.
is_emoji_presentation(Char) -> Bool Test the UTS #51 Emoji_Presentation property (emoji by default).
is_emoji_modifier(Char) -> Bool Test the UTS #51 Emoji_Modifier property (skin tone modifiers).
is_emoji_modifier_base(Char) -> Bool Test the UTS #51 Emoji_Modifier_Base property.
has_emoji_variation_sequence(Char) -> Bool Test whether emoji-variation-sequences.txt lists the character, so that U+FE0E or U+FE0F may follow it to select a text or emoji presentation.
joining_type(Char) -> JoiningType Return the Joining_Type value: NonJoining, JoinCausing, DualJoining, LeftJoining, RightJoining, or Transparent.
joining_group(Char) -> JoiningGroup Return the Joining_Group value, such as Beh, Alef, or Lam; NoJoiningGroup for characters outside the groups.
hangul_syllable_type(Char) -> HangulSyllableType Return the Hangul_Syllable_Type value: NotApplicable, LeadingJamo, VowelJamo, TrailingJamo, LVSyllable, or LVTSyllable.
is_extended_pictographic(Char) -> Bool Test Extended_Pictographic, including reserved potential emoji.
line_break(Char) -> LineBreak Return the UAX #14 Line_Break class, such as AL, ID, or OP, as the Unicode Character Database assigns it. Rule LB1 is not applied here; use unicode-segmentation for the default line breaking algorithm.
to_simple_uppercase(Char) -> Char Simple uppercase mapping.
to_simple_lowercase(Char) -> Char Simple lowercase mapping.
to_simple_titlecase(Char) -> Char Simple titlecase mapping.
to_uppercase(Char) -> String Full uppercase mapping.
to_lowercase(Char) -> String Full lowercase mapping.
to_titlecase(Char) -> String Full titlecase mapping.
uppercase(StringView) -> String Full uppercase mapping of a string, built at once. Context-free: conditional SpecialCasing.txt mappings are not applied.
lowercase(StringView) -> String Unicode default full lowercase conversion, including Final_Sigma: ΟΣ becomes ος. Language-specific mappings are not applied.

The Joining_Group value Pe shares its name with a General_Category value. Where the type is not known from the context, write @ucd.JoiningGroup::Pe or @ucd.GeneralCategory::Pe.

moonbit-community/unicode-normalization

API Description
nfd(StringView) -> String Canonical decomposition.
nfc(StringView) -> String Canonical decomposition followed by canonical composition.
nfkd(StringView) -> String Compatibility decomposition.
nfkc(StringView) -> String Compatibility decomposition followed by canonical composition.
normalize(StringView, NormalizationForm) -> String Normalize with a selected form.
is_normalized(StringView, NormalizationForm) -> Bool Check whether text is already in a selected form.

The available forms are NFD, NFC, NFKD, and NFKC.

moonbit-community/unicode-casefold

API Description
case_fold(StringView) -> String Default full case folding (R4).
caseless_match(StringView, StringView) -> Bool Default caseless equality, without normalization (D144).
canonical_case_fold(StringView) -> String Canonical caseless comparison/search key (D145).
canonical_caseless_match(StringView, StringView) -> Bool Caseless equality under canonical equivalence (D145).
nfkc_casefold(StringView) -> String Official NFKC_CF mapping followed by NFC (R5).
identifier_caseless_match(StringView, StringView) -> Bool Identifier caseless equality after initial NFD (D147).

moonbit-community/punycode

API Description
encode(StringView) -> String raise PunycodeError Encode Unicode text as Punycode.
encode_to(StringView, StringBuilder) -> Int32 raise PunycodeError Append the Punycode encoding to a builder and return its length.
decode(StringView) -> String raise PunycodeError Decode Punycode text back to Unicode.

PunycodeError variants are Overflow, InvalidInput, and BadInput.

moonbit-community/unicode-idna

API Description
to_ascii(String, ...) -> String raise IdnaError Convert a domain name to ASCII form for DNS use.
to_unicode(String, ...) -> String raise IdnaError Convert an ASCII or ACE domain name to Unicode form for display.

to_ascii accepts the following independent boolean options. to_unicode accepts the same options except verify_dns_length and does not check DNS lengths.

Option Default Validation
use_std3_ascii_rules true Restrict ASCII label characters to letters, digits, and hyphens.
check_hyphens true Reject leading or trailing hyphens and labels with hyphens in both the third and fourth positions.
check_bidi true Apply the RFC 5893 Bidi rule.
check_joiners true Check the RFC 5892 CONTEXTJ rules for ZWNJ (U+200C) and ZWJ (U+200D).
check_idna2008 false Restrict Unicode label characters to IDNA2008 PVALID, CONTEXTJ, and CONTEXTO. Other code points raise DisallowedCodePoint(cp~).
check_contexto false Check the RFC 5892 CONTEXTO rules. Violations raise InvalidContextO(cp~).
verify_dns_length true Check ASCII label and domain lengths against DNS limits.

These options apply after UTS #46 mapping and NFC normalization, and to decoded ACE labels. For example, ÖBB.at is mapped to öbb.at, while ☕.example is rejected when check_idna2008=true. Enable check_idna2008 and check_contexto alongside the default checks for IDNA2008 label validation.

moonbit-community/unicode-bidi

API Description
detect_direction(String) -> Direction Detect the base direction from the first strong character.
requires_bidi(String) -> Bool Check whether text contains right-to-left characters.
process(String, direction? : Direction) -> BidiParagraph Resolve classes and levels, optionally forcing the base direction.
reorder(BidiParagraph) -> Array[Int] Return visual-order indexes.
reorder_string(BidiParagraph) -> String Return visually reordered text.
bidi_class(Char) -> BidiClass Return the Unicode Bidi_Class value.
get_mirrored(Char, Int) -> Char Return the mirrored character at an RTL level when one exists.
direction_from_level(Int) -> Direction Convert an embedding level to LTR or RTL.

process_with_direction and process_with_base_level remain available as deprecated compatibility wrappers.

moonbit-community/unicode-segmentation

API Description
graphemes(StringView) -> Graphemes Create a cursor over extended grapheme clusters (UAX #29).
Graphemes::iter() -> Iter[StringView] Iterate over remaining clusters; enables for .. in.
Graphemes::next() -> StringView Return the next cluster, or an empty view at the end.
Graphemes::line_break() -> BreakOpportunity Query Prohibited, Allowed, or Mandatory after the last returned cluster. Initializes line breaking context only when queried.
word_segments(StringView) -> WordSegments All default UAX #29 word segments, retaining whitespace and punctuation.
words(StringView) -> Words Word segments containing at least one Unicode Alphabetic or Number character.
sentences(StringView) -> Sentences Default UAX #29 sentence segments, retaining trailing whitespace and punctuation.
line_breaks(StringView) -> LineBreaks Create a cursor over segments ending at default UAX #14 opportunities.
LineBreaks::next() -> StringView Return the next segment, or an empty view at the end.
LineBreaks::iter() -> Iter[StringView] Iterate over remaining segments; enables for .. in.
LineBreaks::opportunity() -> BreakOpportunity Query the break kind at the end of the last returned segment.

All cursors expose next() -> StringView and iter() -> Iter[StringView], which consume the same cursor. Returned views retain UTF-16 offsets in the backing string; next() returns an empty view at the input's end on exhaustion. word_segments and sentences retain all input characters; words filters complete word segments. They use default rules without locale or dictionary tailoring and prepare boundaries in O(n) time and space.

The grapheme and line cursors retain spaces and newlines. Before the first result their break queries return Prohibited; after exhaustion they preserve the last kind. Nonempty text ends with Mandatory, even without a newline. Empty text has no results.

LineBreaks exposes every default UAX #14 opportunity, including ones inside grapheme clusters. Graphemes::line_break() queries only complete cluster ends, without moving internal opportunities to those ends. See segmentation for examples and complexity. The workspace conformance suite runs every Unicode 18.0.0 LineBreakTest.txt case directly from tests/data/, along with all 1,944 word and 512 sentence boundary cases in WordBreakTest.txt and SentenceBreakTest.txt.

Development

Common commands:

moon check
moon test
moon test -p moonbit-community/unicode-normalization
moon fmt
moon info
moon build

Run moon info after public API changes to refresh pkg.generated.mbti files.

Unicode data tables are generated from official Unicode files:

moon run --target native tools/gen data

# Or regenerate everything in one pass:
moon run --target native tools/gen all

Individual commands are ucd, casefold, idna, bidi, and segmentation. casefold regenerates just the folding tables; it is also included in data / all. Downloaded Unicode source files are cached per version in tools/.cache/<version>/.

Conformance tests live in the workspace-only moonbit-community/unicode-tests module (tests/). They read the official Unicode test data files committed under tests/data/ directly at test time with moonbitlang/core/async, so no test code is generated from those files. Verify a module's contents from its directory, for example:

moon -C unicode package --list  # the compatibility umbrella
moon -C bidi package --list     # an individual feature module

Normalization benchmarks

Run moon bench ./bench/normperf to measure normalization using the 14 full ICU normperf corpora. The benchmark reads committed data directly and supports Native, Wasm and Wasm-GC.

Cross-language benchmark

Run moon run --target native bench/xlang to compare every module with ICU4C, ICU4X and the usual Rust, Go, Node and Python libraries on the same inputs. See bench/xlang for the harnesses and how to read the report.

Releasing

The native release tool keeps the eight published module versions aligned, validates the workspace, and checks every package archive. Preparation publishes nothing and never dry-runs an archive against the registry, so it needs no mooncakes.io credentials:

moon run --target native tools/release -- prepare --version <MAJOR.MINOR.PATCH>

The version is a plain MAJOR.MINOR.PATCH with no prerelease suffix. prepare refuses to run unless the Git worktree is clean and the eight published manifests already agree on one version, which it then rewrites to the new one.

Review and commit the resulting manifest changes, push the branch, open a pull request, and wait for CI to pass. Publication is a separate, explicit step that does not require merging into main or creating a Git tag. In GitHub Actions, run the publish-package workflow and select the branch to publish under Use workflow from. The selected branch must contain this workflow version; the run checks out its exact triggering commit (github.sha), even if the branch moves afterward. The run name includes the branch and commit SHA.

Alternatively, run the underlying command from a clean checkout of the commit to publish:

moon run --target native tools/release -- publish \
  --execute

The tool publishes unicode-data, punycode, unicode-bidi, unicode-segmentation, unicode-normalization, unicode-casefold, unicode-idna, and the unicode compatibility umbrella in dependency order. It reads the release version from the umbrella manifest unicode/moon.mod and verifies that all eight published modules use that version. The modules that do not depend on this release are dry-run before anything is published, so a rejected archive cannot leave a partial release behind; each remaining module is dry-run immediately before its own publication, after the registry is refreshed for its same-release dependencies. It never publishes the workspace-only tests or tools modules. If publication stops partway there is no resume step: the modules that were already published stay at that version, so fix the problem and release the whole set again at a new version.

Standards

License

Apache-2.0. See LICENSE.

1# unicode23Unicode support for MoonBit.45This library implements selected Unicode standards for normalization, case and6category lookup, Punycode, IDNA domain processing, bidirectional text, and text7segmentation. The generated tables target Unicode 18.0.0.89## Features1011- Unicode Normalization Forms from UAX #15: NFD, NFC, NFKD, and NFKC12- Default full case folding, canonical caseless matching, and NFKC_Casefold13  identifier matching from Unicode section 3.1314- Punycode encoding and decoding from RFC 349215- IDNA processing from UTS #46, including mapping, validation, Bidi checks,16  joiner checks, and DNS length checks, with optional IDNA2008 character and17  contextual validation18- Unicode Bidirectional Algorithm support from UAX #919- Default line breaking opportunities from UAX #1420- Extended grapheme cluster, word, and sentence segmentation from UAX #2921- General_Category, East_Asian_Width, Line_Break, Joining_Type, Joining_Group,22  and Hangul_Syllable_Type lookup, binary property predicates (White_Space,23  Alphabetic, Lowercase, Uppercase, XID_Start, XID_Continue,24  Default_Ignorable_Code_Point, Grapheme_Extend, Regional_Indicator,25  Prepended_Concatenation_Mark, Emoji_Presentation, Emoji_Modifier,26  Emoji_Modifier_Base, Extended_Pictographic), emoji variation sequences, and simple/full case mapping27  from the Unicode Character Database2829## Installation3031Add only the feature modules your application needs:3233```bash34moon add moonbit-community/unicode-data35moon add moonbit-community/unicode-normalization36moon add moonbit-community/unicode-casefold37moon add moonbit-community/punycode38moon add moonbit-community/unicode-bidi39moon add moonbit-community/unicode-segmentation40moon add moonbit-community/unicode-idna41```4243Import the packages you need in `moon.pkg`:4445```moonbit46import {47  "moonbit-community/unicode-data" @ucd,48  "moonbit-community/unicode-normalization" @normalization,49  "moonbit-community/unicode-casefold" @casefold,50  "moonbit-community/punycode",51  "moonbit-community/unicode-idna" @idna,52  "moonbit-community/unicode-bidi" @bidi,53  "moonbit-community/unicode-segmentation" @segmentation,54}55```5657MoonBit uses the last package path segment as the default alias, which is not a58valid identifier for the hyphenated `unicode-*` modules, so give those imports59an explicit alias. The examples below use `@ucd`, `@normalization`, `@casefold`,60`@punycode`, `@idna`, `@bidi`, and `@segmentation`. Existing users may continue depending on61`moonbit-community/unicode`; it is an umbrella compatibility module that62preserves the old package paths.6364## Usage6566Normalization and case folding accept `StringView`, so callers can pass slices67without copying them first. `String` arguments convert automatically;68text-producing operations still return `String`.6970### Normalization7172```moonbit73let composed = @normalization.nfc("e\u{0301}") // "é"74let decomposed = @normalization.nfd("é") // "e" + combining acute75let compatible = @normalization.nfkc("fi") // "fi"7677let already_nfc = @normalization.is_normalized(78  composed,79  @normalization.NFC,80)8182let normalized = @normalization.normalize(83  "text",84  @normalization.NFKC,85)86```8788### Case folding and caseless matching8990```moonbit91let folded = @casefold.case_fold("Straße") // "strasse"92let same_case = @casefold.caseless_match("Straße", "STRASSE") // true93let same_text = @casefold.canonical_caseless_match("Å", "a\u{030A}") // true94let search_key = @casefold.canonical_case_fold("Å") // "a\u{030A}"95let folded_id = @casefold.nfkc_casefold("A\u{00AD}") // "a"96let same_id = @casefold.identifier_caseless_match("A\u{00AD}", "a") // true97```9899These operations follow [Unicode §3.13](https://www.unicode.org/versions/Unicode18.0.0/core-spec/chapter-3/)100and are locale-independent. Default full folding uses `CaseFolding.txt` statuses101`C` and `F`, with expansions such as `ß → ss`; it does not normalize text or102apply Turkic tailoring. `I` folds to `i`, while `İ` folds to `i` plus U+0307.103Cherokee folds to uppercase, so a folded string is not necessarily lowercase.104105For canonically equivalent text, `canonical_case_fold(s)` computes106`NFD(case_fold(NFD(s)))` (D145). Store and compare these keys, or search folded107text with a folded query. Folding can change text length, so result offsets do108not directly identify positions in the original text.109110`nfkc_casefold(s)` applies the official `NFKC_CF` mappings and then NFC (R5).111This merges compatibility differences and removes default-ignorable characters.112`identifier_caseless_match(a, b)` first applies NFD to both inputs before113NFKC_Casefold (D147), including the Greek iota-subscript edge cases. It tests114equality; identifier validation and confusable detection are separate operations.115116### Punycode117118`encode` and `decode` use checked errors. Use `try!` for examples or handle the119error explicitly in application code.120121```moonbit122let encoded = try! @punycode.encode("münchen") // "mnchen-3ya"123let decoded = try! @punycode.decode("mnchen-3ya") // "münchen"124125let chinese = try! @punycode.encode("中文") // "fiq228c"126```127128### IDNA129130```moonbit131let ascii = try! @idna.to_ascii("münchen.de")132// "xn--mnchen-3ya.de"133134let unicode = try! @idna.to_unicode("xn--mnchen-3ya.de")135// "münchen.de"136137let checked = try! @idna.to_ascii(138  "example.com",139  use_std3_ascii_rules=true,140  check_hyphens=true,141  check_bidi=true,142  check_joiners=true,143  verify_dns_length=true,144)145146let idna2008_checked = try! @idna.to_ascii(147  "ÖBB.at",148  check_idna2008=true,149  check_contexto=true,150)151// "xn--bb-eka.at" — UTS #46 mapping lowercases the input.152```153154Both functions support IDNA2008 label validation through the opt-in155`check_idna2008` and `check_contexto` options, which default to `false`. Enable156both alongside the default checks. Validation applies after UTS #46 mapping157and NFC normalization, including to Unicode labels decoded from Punycode.158159To handle validation failures:160161```moonbit162try @idna.to_ascii("example..com") catch {163  err => println("invalid domain: \{err}")164} noraise {165  ascii => println(ascii)166}167```168169### Bidi170171```moonbit172let direction = @bidi.detect_direction("Hello World") // LTR173let needs_bidi = @bidi.requires_bidi("Hello \u{05E9}\u{05DC}\u{05D5}\u{05DD}")174175let paragraph = @bidi.process("abc\u{05D0}\u{05D1}")176let visual = @bidi.reorder_string(paragraph)177let order = @bidi.reorder(paragraph)178179let forced = @bidi.process(180  "\u{05D0}\u{05D1}\u{05D2}",181  direction=@bidi.Direction::LTR,182)183```184185### Segmentation186187```moonbit188// "é" (e + combining acute), a flag, and a family emoji are one cluster each.189let count = @segmentation.graphemes("e\u{0301}🇺🇳👨‍👩‍👧").iter().count() // 3190191for cluster in @segmentation.graphemes("e\u{0301}x") {192  // Each cluster is a view into the input.193  println("\{cluster.start_offset()}: \{cluster}")194}195196// `next` returns an empty view at the end instead of an `Option`, which197// avoids an allocation per cluster in hot loops.198let clusters = @segmentation.graphemes("e\u{0301}x")199while clusters.next() is cluster && !cluster.is_empty() {200  println(cluster)201}202```203204### Case And Category Data205206```moonbit207let category = @ucd.general_category('A') // Lu208let group = category.group() // L209let is_punct = @ucd.is_punctuation('!') // true: General_Category group P210let width = @ucd.east_asian_width('世') // W211let can_start = @ucd.is_xid_start('A') // true212let can_continue = @ucd.is_xid_continue('0') // true213let is_space = @ucd.is_white_space('\u{3000}') // true214let is_letter = @ucd.is_alphabetic('\u{093E}') // true (Other_Alphabetic)215let is_lower = @ucd.is_lowercase('\u{00AA}') // true, although gc is Lo216let is_upper = @ucd.is_uppercase('\u{2160}') // true, although gc is Nl217let joining = @ucd.joining_type('\u{0628}') // DualJoining218let is_ignorable = @ucd.is_default_ignorable_code_point('\u{200D}') // true219let is_emoji = @ucd.is_emoji_presentation('\u{2764}') // false: text by default220let has_vs = @ucd.has_emoji_variation_sequence('\u{2764}') // true: U+FE0F selects emoji221let group = @ucd.joining_group('\u{062A}') // Beh222let break_class = @ucd.line_break('\u{4E2D}') // ID223let hangul = @ucd.hangul_syllable_type('\u{AC00}') // LVSyllable224225let simple = @ucd.to_simple_uppercase('a') // 'A'226let full = @ucd.to_uppercase('\u{00DF}') // "SS"227let lower = @ucd.to_lowercase('\u{0130}') // "i" + combining dot above228let shout = @ucd.uppercase("Straße") // "STRASSE": one string, no per-character allocation229```230231## Public Packages232233### `moonbit-community/unicode-data`234235Root package for Unicode Character Database helpers.236237| API | Description |238| --- | --- |239| `general_category(Char) -> GeneralCategory` | Return the two-letter Unicode General_Category value. |240| `GeneralCategory::group() -> GeneralCategoryGroup` | Return the one-letter category group. |241| `is_punctuation(Char) -> Bool` | Test whether the character is punctuation (General_Category group P), using a compact table independent of the General_Category trie. |242| `east_asian_width(Char) -> EastAsianWidth` | Return the UAX #11 East_Asian_Width value: `A`, `F`, `H`, `N`, `Na`, or `W`. |243| `is_white_space(Char) -> Bool` | Test the Unicode White_Space property. |244| `is_alphabetic(Char) -> Bool` | Test the Unicode Alphabetic property. |245| `is_lowercase(Char) -> Bool` | Test the Unicode Lowercase property (broader than General_Category Ll). |246| `is_uppercase(Char) -> Bool` | Test the Unicode Uppercase property (broader than General_Category Lu). |247| `is_xid_start(Char) -> Bool` | Test the Unicode XID_Start identifier property. |248| `is_xid_continue(Char) -> Bool` | Test the Unicode XID_Continue identifier property. |249| `is_default_ignorable_code_point(Char) -> Bool` | Test the Unicode Default_Ignorable_Code_Point property. |250| `is_grapheme_extend(Char) -> Bool` | Test the Unicode Grapheme_Extend property. |251| `is_regional_indicator(Char) -> Bool` | Test the Unicode Regional_Indicator property. |252| `is_prepended_concatenation_mark(Char) -> Bool` | Test the Unicode Prepended_Concatenation_Mark property. |253| `is_emoji_presentation(Char) -> Bool` | Test the UTS #51 Emoji_Presentation property (emoji by default). |254| `is_emoji_modifier(Char) -> Bool` | Test the UTS #51 Emoji_Modifier property (skin tone modifiers). |255| `is_emoji_modifier_base(Char) -> Bool` | Test the UTS #51 Emoji_Modifier_Base property. |256| `has_emoji_variation_sequence(Char) -> Bool` | Test whether `emoji-variation-sequences.txt` lists the character, so that U+FE0E or U+FE0F may follow it to select a text or emoji presentation. |257| `joining_type(Char) -> JoiningType` | Return the Joining_Type value: `NonJoining`, `JoinCausing`, `DualJoining`, `LeftJoining`, `RightJoining`, or `Transparent`. |258| `joining_group(Char) -> JoiningGroup` | Return the Joining_Group value, such as `Beh`, `Alef`, or `Lam`; `NoJoiningGroup` for characters outside the groups. |259| `hangul_syllable_type(Char) -> HangulSyllableType` | Return the Hangul_Syllable_Type value: `NotApplicable`, `LeadingJamo`, `VowelJamo`, `TrailingJamo`, `LVSyllable`, or `LVTSyllable`. |260| `is_extended_pictographic(Char) -> Bool` | Test Extended_Pictographic, including reserved potential emoji. |261| `line_break(Char) -> LineBreak` | Return the UAX #14 Line_Break class, such as `AL`, `ID`, or `OP`, as the Unicode Character Database assigns it. Rule LB1 is not applied here; use `unicode-segmentation` for the default line breaking algorithm. |262| `to_simple_uppercase(Char) -> Char` | Simple uppercase mapping. |263| `to_simple_lowercase(Char) -> Char` | Simple lowercase mapping. |264| `to_simple_titlecase(Char) -> Char` | Simple titlecase mapping. |265| `to_uppercase(Char) -> String` | Full uppercase mapping. |266| `to_lowercase(Char) -> String` | Full lowercase mapping. |267| `to_titlecase(Char) -> String` | Full titlecase mapping. |268| `uppercase(StringView) -> String` | Full uppercase mapping of a string, built at once. Context-free: conditional SpecialCasing.txt mappings are not applied. |269| `lowercase(StringView) -> String` | Unicode default full lowercase conversion, including Final_Sigma: `ΟΣ` becomes `ος`. Language-specific mappings are not applied. |270271The Joining_Group value `Pe` shares its name with a General_Category value.272Where the type is not known from the context, write `@ucd.JoiningGroup::Pe` or273`@ucd.GeneralCategory::Pe`.274275### `moonbit-community/unicode-normalization`276277| API | Description |278| --- | --- |279| `nfd(StringView) -> String` | Canonical decomposition. |280| `nfc(StringView) -> String` | Canonical decomposition followed by canonical composition. |281| `nfkd(StringView) -> String` | Compatibility decomposition. |282| `nfkc(StringView) -> String` | Compatibility decomposition followed by canonical composition. |283| `normalize(StringView, NormalizationForm) -> String` | Normalize with a selected form. |284| `is_normalized(StringView, NormalizationForm) -> Bool` | Check whether text is already in a selected form. |285286The available forms are `NFD`, `NFC`, `NFKD`, and `NFKC`.287288### `moonbit-community/unicode-casefold`289290| API | Description |291| --- | --- |292| `case_fold(StringView) -> String` | Default full case folding (R4). |293| `caseless_match(StringView, StringView) -> Bool` | Default caseless equality, without normalization (D144). |294| `canonical_case_fold(StringView) -> String` | Canonical caseless comparison/search key (D145). |295| `canonical_caseless_match(StringView, StringView) -> Bool` | Caseless equality under canonical equivalence (D145). |296| `nfkc_casefold(StringView) -> String` | Official NFKC_CF mapping followed by NFC (R5). |297| `identifier_caseless_match(StringView, StringView) -> Bool` | Identifier caseless equality after initial NFD (D147). |298299### `moonbit-community/punycode`300301| API | Description |302| --- | --- |303| `encode(StringView) -> String raise PunycodeError` | Encode Unicode text as Punycode. |304| `encode_to(StringView, StringBuilder) -> Int32 raise PunycodeError` | Append the Punycode encoding to a builder and return its length. |305| `decode(StringView) -> String raise PunycodeError` | Decode Punycode text back to Unicode. |306307`PunycodeError` variants are `Overflow`, `InvalidInput`, and `BadInput`.308309### `moonbit-community/unicode-idna`310311| API | Description |312| --- | --- |313| `to_ascii(String, ...) -> String raise IdnaError` | Convert a domain name to ASCII form for DNS use. |314| `to_unicode(String, ...) -> String raise IdnaError` | Convert an ASCII or ACE domain name to Unicode form for display. |315316`to_ascii` accepts the following independent boolean options. `to_unicode`317accepts the same options except `verify_dns_length` and does not check DNS318lengths.319320| Option | Default | Validation |321| --- | --- | --- |322| `use_std3_ascii_rules` | `true` | Restrict ASCII label characters to letters, digits, and hyphens. |323| `check_hyphens` | `true` | Reject leading or trailing hyphens and labels with hyphens in both the third and fourth positions. |324| `check_bidi` | `true` | Apply the RFC 5893 Bidi rule. |325| `check_joiners` | `true` | Check the RFC 5892 CONTEXTJ rules for ZWNJ (U+200C) and ZWJ (U+200D). |326| `check_idna2008` | `false` | Restrict Unicode label characters to IDNA2008 `PVALID`, `CONTEXTJ`, and `CONTEXTO`. Other code points raise `DisallowedCodePoint(cp~)`. |327| `check_contexto` | `false` | Check the RFC 5892 CONTEXTO rules. Violations raise `InvalidContextO(cp~)`. |328| `verify_dns_length` | `true` | Check ASCII label and domain lengths against DNS limits. |329330These options apply after UTS #46 mapping and NFC normalization, and to decoded331ACE labels. For example, `ÖBB.at` is mapped to `öbb.at`, while `☕.example` is332rejected when `check_idna2008=true`. Enable `check_idna2008` and `check_contexto`333alongside the default checks for IDNA2008 label validation.334335### `moonbit-community/unicode-bidi`336337| API | Description |338| --- | --- |339| `detect_direction(String) -> Direction` | Detect the base direction from the first strong character. |340| `requires_bidi(String) -> Bool` | Check whether text contains right-to-left characters. |341| `process(String, direction? : Direction) -> BidiParagraph` | Resolve classes and levels, optionally forcing the base direction. |342| `reorder(BidiParagraph) -> Array[Int]` | Return visual-order indexes. |343| `reorder_string(BidiParagraph) -> String` | Return visually reordered text. |344| `bidi_class(Char) -> BidiClass` | Return the Unicode Bidi_Class value. |345| `get_mirrored(Char, Int) -> Char` | Return the mirrored character at an RTL level when one exists. |346| `direction_from_level(Int) -> Direction` | Convert an embedding level to `LTR` or `RTL`. |347348`process_with_direction` and `process_with_base_level` remain available as349deprecated compatibility wrappers.350351### `moonbit-community/unicode-segmentation`352353| API | Description |354| --- | --- |355| `graphemes(StringView) -> Graphemes` | Create a cursor over extended grapheme clusters (UAX #29). |356| `Graphemes::iter() -> Iter[StringView]` | Iterate over remaining clusters; enables `for .. in`. |357| `Graphemes::next() -> StringView` | Return the next cluster, or an empty view at the end. |358| `Graphemes::line_break() -> BreakOpportunity` | Query `Prohibited`, `Allowed`, or `Mandatory` after the last returned cluster. Initializes line breaking context only when queried. |359| `word_segments(StringView) -> WordSegments` | All default UAX #29 word segments, retaining whitespace and punctuation. |360| `words(StringView) -> Words` | Word segments containing at least one Unicode Alphabetic or Number character. |361| `sentences(StringView) -> Sentences` | Default UAX #29 sentence segments, retaining trailing whitespace and punctuation. |362| `line_breaks(StringView) -> LineBreaks` | Create a cursor over segments ending at default UAX #14 opportunities. |363| `LineBreaks::next() -> StringView` | Return the next segment, or an empty view at the end. |364| `LineBreaks::iter() -> Iter[StringView]` | Iterate over remaining segments; enables `for .. in`. |365| `LineBreaks::opportunity() -> BreakOpportunity` | Query the break kind at the end of the last returned segment. |366367All cursors expose `next() -> StringView` and `iter() -> Iter[StringView]`,368which consume the same cursor. Returned views retain UTF-16 offsets in the369backing string; `next()` returns an empty view at the input's end on370exhaustion. `word_segments` and `sentences` retain all input characters;371`words` filters complete word segments. They use default rules without372locale or dictionary tailoring and prepare boundaries in O(n) time and space.373374The grapheme and line cursors retain spaces and newlines.375Before the first result their break queries return `Prohibited`;376after exhaustion they preserve the last kind. Nonempty text ends with377`Mandatory`, even without a newline. Empty text has no results.378379`LineBreaks` exposes every default UAX #14 opportunity, including ones inside380grapheme clusters. `Graphemes::line_break()` queries only complete cluster ends,381without moving internal opportunities to those ends. See382[segmentation](segmentation/README.md) for examples and complexity.383The workspace conformance suite runs every Unicode 18.0.0 `LineBreakTest.txt`384case directly from `tests/data/`, along with all 1,944 word and 512 sentence385boundary cases in `WordBreakTest.txt` and `SentenceBreakTest.txt`.386387## Development388389Common commands:390391```bash392moon check393moon test394moon test -p moonbit-community/unicode-normalization395moon fmt396moon info397moon build398```399400Run `moon info` after public API changes to refresh `pkg.generated.mbti` files.401402Unicode data tables are generated from official Unicode files:403404```bash405moon run --target native tools/gen data406407# Or regenerate everything in one pass:408moon run --target native tools/gen all409```410411Individual commands are `ucd`, `casefold`, `idna`, `bidi`, and `segmentation`.412`casefold` regenerates just the folding tables; it is also included in `data` / `all`.413Downloaded Unicode source files are cached per version in `tools/.cache/<version>/`.414415Conformance tests live in the workspace-only `moonbit-community/unicode-tests`416module (`tests/`). They read the official Unicode test data files committed417under `tests/data/` directly at test time with `moonbitlang/core/async`, so no418test code is generated from those files. Verify a module's contents419from its directory, for example:420421```bash422moon -C unicode package --list  # the compatibility umbrella423moon -C bidi package --list     # an individual feature module424```425426### Normalization benchmarks427428Run `moon bench ./bench/normperf` to measure normalization using the429[14 full ICU normperf corpora](bench/normperf/README.md). The benchmark430reads committed data directly and supports Native, Wasm and Wasm-GC.431432### Cross-language benchmark433434Run `moon run --target native bench/xlang` to compare every module with ICU4C,435ICU4X and the usual Rust, Go, Node and Python libraries on the same inputs.436See [bench/xlang](bench/xlang/README.md) for the harnesses and how to read the437report.438439### Releasing440441The native release tool keeps the eight published module versions aligned,442validates the workspace, and checks every package archive. Preparation443publishes nothing and never dry-runs an archive against the registry, so it444needs no mooncakes.io credentials:445446```bash447moon run --target native tools/release -- prepare --version <MAJOR.MINOR.PATCH>448```449450The version is a plain `MAJOR.MINOR.PATCH` with no prerelease suffix. `prepare`451refuses to run unless the Git worktree is clean and the eight published452manifests already agree on one version, which it then rewrites to the new one.453454Review and commit the resulting manifest changes, push the branch, open a pull455request, and wait for CI to pass. Publication is a separate, explicit step that456does not require merging into `main` or creating a Git tag. In GitHub Actions,457run the `publish-package` workflow and select the branch to publish under458**Use workflow from**. The selected branch must contain this workflow version;459the run checks out its exact triggering commit (`github.sha`), even if the branch460moves afterward. The run name includes the branch and commit SHA.461462Alternatively, run the underlying command from a clean checkout of the commit463to publish:464465```bash466moon run --target native tools/release -- publish \467  --execute468```469470The tool publishes `unicode-data`, `punycode`, `unicode-bidi`,471`unicode-segmentation`, `unicode-normalization`, `unicode-casefold`, `unicode-idna`, and the472`unicode` compatibility umbrella in dependency order. It reads473the release version from the umbrella manifest `unicode/moon.mod` and verifies474that all eight published modules use that version. The modules that do not475depend on this release are dry-run before anything is published, so a rejected476archive cannot leave a partial release behind; each remaining module is477dry-run immediately before its own publication, after the registry is478refreshed for its same-release dependencies. It never publishes the479workspace-only `tests` or `tools` modules. If publication stops partway there480is no resume step: the modules that were already published stay at that481version, so fix the problem and release the whole set again at a new version.482483## Standards484485- [UAX #9: Unicode Bidirectional Algorithm](https://unicode.org/reports/tr9/)486- [UAX #14: Unicode Line Breaking Algorithm](https://unicode.org/reports/tr14/)487- [UAX #15: Unicode Normalization Forms](https://unicode.org/reports/tr15/)488- [UAX #29: Unicode Text Segmentation](https://unicode.org/reports/tr29/)489- [UAX #44: Unicode Character Database](https://unicode.org/reports/tr44/)490- [RFC 3492: Punycode](https://datatracker.ietf.org/doc/html/rfc3492)491- [UTS #46: Unicode IDNA Compatibility Processing](https://unicode.org/reports/tr46/)492- [RFC 5891: Internationalized Domain Names in Applications](https://datatracker.ietf.org/doc/html/rfc5891)493- [RFC 5892: The Unicode Code Points and IDNA](https://datatracker.ietf.org/doc/html/rfc5892)494- [RFC 5893: Right-to-Left Scripts for IDNA](https://datatracker.ietf.org/doc/html/rfc5893)495496## License497498Apache-2.0. See [LICENSE](LICENSE).499