unicode

Branch: main

Switch branches or tags

Branch: main
Go to file t

413 files indexed

History
Code

AGENTS.md

AGENTS.md

This file provides guidance to code agents (Claude Code, Codex, etc.) when working with code in this repository.

Project Overview

A MoonBit library implementing Unicode standards: text normalization (UAX #15), Punycode (RFC 3492), IDNA (UTS #46), bidirectional text (UAX #9), and text segmentation (UAX #29). Targets Unicode 18.0.0.

Build Commands

moon check          # Type check
moon test           # Run all tests
moon test -p <pkg>  # Test specific package (e.g., moon test -p punycode)
moon fmt            # Format code
moon build          # Build project
moon info           # Regenerate pkg.generated.mbti files

Architecture

Public Modules

  • ucd/ (moonbit-community/unicode-data): Unicode Character Database lookup, case mapping, and general category APIs

  • normalization/ (moonbit-community/unicode-normalization): UAX #15 Unicode normalization

  • casefold/ (moonbit-community/unicode-casefold): full case folding, canonical caseless matching, and NFKC_Casefold

  • punycode/ (moonbit-community/punycode): RFC 3492 Punycode encoding/decoding

    • encode(StringView) -> String raise PunycodeError
    • encode_to(StringView, StringBuilder) -> Int32 raise PunycodeError - appends the encoding and returns its length
    • decode(StringView) -> String raise PunycodeError
  • bidi/ (moonbit-community/unicode-bidi): UAX #9 bidirectional text processing

  • segmentation/ (moonbit-community/unicode-segmentation): UAX #29 text segmentation

    • graphemes(StringView) -> Graphemes - cursor over extended grapheme clusters (iter() for for .. in, next() returning an empty view at the end)
    • word_segments(StringView) -> WordSegments - all default word segments
    • words(StringView) -> Words - word segments containing Alphabetic or Number characters
    • sentences(StringView) -> Sentences - all default sentence segments
    • line_breaks(StringView) -> LineBreaks - default UAX #14 break opportunities
    • All cursors provide next() and iter() consuming the same state
  • idna/ (moonbit-community/unicode-idna): UTS #46 IDNA processing

    • to_ascii() - convert domain to ASCII (Punycode)
    • to_unicode() - convert domain from Punycode
  • unicode/ (moonbit-community/unicode): compatibility umbrella preserving the old package paths

  • tests/ (moonbit-community/unicode-tests): workspace-only conformance tests. They read the official Unicode test data files committed under tests/data/ directly at run time with moonbitlang/core/async; no test code is generated

  • bench/ (local/unicode-bench): workspace-only benchmarks. normperf/ measures normalization with moon bench; xlang/ compares every module with other languages' libraries (moon run --target native bench/xlang), with one standalone harness per language in its subdirectories

Internal Data Packages (auto-generated)

  • ucd/data/: Unicode Character Database lookup tables
  • casefold/case_folding_data.mbt, casefold/nfkc_casefold_data.mbt: full default case folding and NFKC_CF mappings
  • ucd/line_break.mbt, ucd/joining_group.mbt: the LineBreak and JoiningGroup enums of the root package, generated so that they stay in step with the ordinals in ucd/line_break/ and ucd/joining_group/
  • idna/internal/idna/: IDNA mapping table (Joining_Type comes from unicode-data)
  • bidi/internal/bidi/: Bidi mirroring and bracket tables
  • segmentation/grapheme_break_property.mbt: grapheme cluster break properties
  • segmentation/word_break_property.mbt, segmentation/sentence_break_property.mbt: word and sentence break properties

Dependencies

unicode-normalization -> unicode-data
unicode-casefold -> unicode-normalization
unicode-idna -> unicode-data
unicode-idna -> unicode-normalization
unicode-idna -> punycode
unicode-idna -> unicode-bidi

Code Generation

Unicode data tables are generated from official Unicode source files:

moon run --target native tools/gen data   # Generate all Unicode lookup tables
moon run --target native tools/gen all    # Alias for `data`

Individual commands are ucd, casefold, idna, bidi, and segmentation. casefold regenerates folding tables only and is also included in data / all. Downloaded Unicode data is cached per version in tools/.cache/<version>/.

Conformance Tests

Conformance tests live in the workspace-only tests/ module and read the official Unicode test files (BidiTest.txt, BidiCharacterTest.txt, NormalizationTest.txt, IdnaTestV2.txt, GraphemeBreakTest.txt, WordBreakTest.txt, SentenceBreakTest.txt, LineBreakTest.txt) directly at test time via moonbitlang/core/async. The data files are committed under tests/data/, and the tests use the standard library's async runtime, so they need no dependency beyond the published modules.

Case folding conformance tests also read CaseFolding.txt and DerivedNormalizationProps.txt, checking every Unicode scalar value against the full default folding and NFKC_CF mappings.

Publishing

moon -C unicode package --list  # Verify the compatibility umbrella
moon -C bidi package --list     # Verify an individual feature module

Testing Notes

  • Unit tests: *_test.mbt files with test blocks and inspect() assertions
  • Conformance tests: tests/ module (moonbit-community/unicode-tests), which reads the Unicode test data files in tests/data/ at run time
  • Fuzz tests: fuzz_test.mbt files for property-based testing

Commit Convention

Uses conventional commits: fix:, feat:, refactor:, test:, chore:, release:

1# AGENTS.md23This file provides guidance to code agents (Claude Code, Codex, etc.) when working with code in this repository.45## Project Overview67A MoonBit library implementing Unicode standards: text normalization (UAX #15), Punycode (RFC 3492), IDNA (UTS #46), bidirectional text (UAX #9), and text segmentation (UAX #29). Targets Unicode 18.0.0.89## Build Commands1011```bash12moon check          # Type check13moon test           # Run all tests14moon test -p <pkg>  # Test specific package (e.g., moon test -p punycode)15moon fmt            # Format code16moon build          # Build project17moon info           # Regenerate pkg.generated.mbti files18```1920## Architecture2122### Public Modules2324- **ucd/** (`moonbit-community/unicode-data`): Unicode Character Database lookup,25  case mapping, and general category APIs26- **normalization/** (`moonbit-community/unicode-normalization`): UAX #15 Unicode27  normalization28- **casefold/** (`moonbit-community/unicode-casefold`): full case folding,29  canonical caseless matching, and NFKC_Casefold30- **punycode/** (`moonbit-community/punycode`): RFC 3492 Punycode encoding/decoding31  - `encode(StringView) -> String raise PunycodeError`32  - `encode_to(StringView, StringBuilder) -> Int32 raise PunycodeError` - appends33    the encoding and returns its length34  - `decode(StringView) -> String raise PunycodeError`35- **bidi/** (`moonbit-community/unicode-bidi`): UAX #9 bidirectional text processing36- **segmentation/** (`moonbit-community/unicode-segmentation`): UAX #29 text segmentation37  - `graphemes(StringView) -> Graphemes` - cursor over extended grapheme clusters38    (`iter()` for `for .. in`, `next()` returning an empty view at the end)39  - `word_segments(StringView) -> WordSegments` - all default word segments40  - `words(StringView) -> Words` - word segments containing Alphabetic or Number characters41  - `sentences(StringView) -> Sentences` - all default sentence segments42  - `line_breaks(StringView) -> LineBreaks` - default UAX #14 break opportunities43  - All cursors provide `next()` and `iter()` consuming the same state44- **idna/** (`moonbit-community/unicode-idna`): UTS #46 IDNA processing45  - `to_ascii()` - convert domain to ASCII (Punycode)46  - `to_unicode()` - convert domain from Punycode47- **unicode/** (`moonbit-community/unicode`): compatibility umbrella preserving the old48  package paths49- **tests/** (`moonbit-community/unicode-tests`): workspace-only conformance tests.50  They read the official Unicode test data files committed under `tests/data/`51  directly at run time with `moonbitlang/core/async`; no test code is generated5253- **bench/** (`local/unicode-bench`): workspace-only benchmarks. `normperf/`54  measures normalization with `moon bench`; `xlang/` compares every module with55  other languages' libraries (`moon run --target native bench/xlang`), with one56  standalone harness per language in its subdirectories5758### Internal Data Packages (auto-generated)5960- **ucd/data/**: Unicode Character Database lookup tables61- **casefold/case_folding_data.mbt**, **casefold/nfkc_casefold_data.mbt**:62  full default case folding and NFKC_CF mappings63- **ucd/line_break.mbt**, **ucd/joining_group.mbt**: the `LineBreak` and64  `JoiningGroup` enums of the root package, generated so that they stay in65  step with the ordinals in `ucd/line_break/` and `ucd/joining_group/`66- **idna/internal/idna/**: IDNA mapping table (Joining_Type comes from67  `unicode-data`)68- **bidi/internal/bidi/**: Bidi mirroring and bracket tables69- **segmentation/grapheme_break_property.mbt**: grapheme cluster break properties70- **segmentation/word_break_property.mbt**, **segmentation/sentence_break_property.mbt**:71  word and sentence break properties7273### Dependencies7475```76unicode-normalization -> unicode-data77unicode-casefold -> unicode-normalization78unicode-idna -> unicode-data79unicode-idna -> unicode-normalization80unicode-idna -> punycode81unicode-idna -> unicode-bidi82```8384## Code Generation8586Unicode data tables are generated from official Unicode source files:8788```bash89moon run --target native tools/gen data   # Generate all Unicode lookup tables90moon run --target native tools/gen all    # Alias for `data`91```9293Individual commands are `ucd`, `casefold`, `idna`, `bidi`, and `segmentation`.94`casefold` regenerates folding tables only and is also included in `data` / `all`. Downloaded95Unicode data is cached per version in `tools/.cache/<version>/`.9697## Conformance Tests9899Conformance tests live in the workspace-only `tests/` module and read the100official Unicode test files (`BidiTest.txt`, `BidiCharacterTest.txt`,101`NormalizationTest.txt`, `IdnaTestV2.txt`, `GraphemeBreakTest.txt`,102`WordBreakTest.txt`, `SentenceBreakTest.txt`, `LineBreakTest.txt`) directly at103test time via104`moonbitlang/core/async`. The data files are committed under `tests/data/`, and105the tests use the standard library's async runtime, so they need no dependency106beyond the published modules.107108Case folding conformance tests also read `CaseFolding.txt` and109`DerivedNormalizationProps.txt`, checking every Unicode scalar value against110the full default folding and NFKC_CF mappings.111112## Publishing113114```bash115moon -C unicode package --list  # Verify the compatibility umbrella116moon -C bidi package --list     # Verify an individual feature module117```118119## Testing Notes120121- Unit tests: `*_test.mbt` files with `test` blocks and `inspect()` assertions122- Conformance tests: `tests/` module (`moonbit-community/unicode-tests`), which123  reads the Unicode test data files in `tests/data/` at run time124- Fuzz tests: `fuzz_test.mbt` files for property-based testing125126## Commit Convention127128Uses conventional commits: `fix:`, `feat:`, `refactor:`, `test:`, `chore:`, `release:`129