unicode
AGENTS.md
This file provides guidance to code agents (Claude Code, Codex, etc.) when working with code in this repository.
Project Overview
A MoonBit library implementing Unicode standards: text normalization (UAX #15), Punycode (RFC 3492), IDNA (UTS #46), bidirectional text (UAX #9), and text segmentation (UAX #29). Targets Unicode 18.0.0.
Build Commands
moon check # Type check
moon test # Run all tests
moon test -p <pkg> # Test specific package (e.g., moon test -p punycode)
moon fmt # Format code
moon build # Build project
moon info # Regenerate pkg.generated.mbti files
Architecture
Public Modules
-
ucd/ (
moonbit-community/unicode-data): Unicode Character Database lookup, case mapping, and general category APIs -
normalization/ (
moonbit-community/unicode-normalization): UAX #15 Unicode normalization -
casefold/ (
moonbit-community/unicode-casefold): full case folding, canonical caseless matching, and NFKC_Casefold -
punycode/ (
moonbit-community/punycode): RFC 3492 Punycode encoding/decodingencode(StringView) -> String raise PunycodeErrorencode_to(StringView, StringBuilder) -> Int32 raise PunycodeError- appends the encoding and returns its lengthdecode(StringView) -> String raise PunycodeError
-
bidi/ (
moonbit-community/unicode-bidi): UAX #9 bidirectional text processing -
segmentation/ (
moonbit-community/unicode-segmentation): UAX #29 text segmentationgraphemes(StringView) -> Graphemes- cursor over extended grapheme clusters (iter()forfor .. in,next()returning an empty view at the end)word_segments(StringView) -> WordSegments- all default word segmentswords(StringView) -> Words- word segments containing Alphabetic or Number characterssentences(StringView) -> Sentences- all default sentence segmentsline_breaks(StringView) -> LineBreaks- default UAX #14 break opportunities- All cursors provide
next()anditer()consuming the same state
-
idna/ (
moonbit-community/unicode-idna): UTS #46 IDNA processingto_ascii()- convert domain to ASCII (Punycode)to_unicode()- convert domain from Punycode
-
unicode/ (
moonbit-community/unicode): compatibility umbrella preserving the old package paths -
tests/ (
moonbit-community/unicode-tests): workspace-only conformance tests. They read the official Unicode test data files committed undertests/data/directly at run time withmoonbitlang/core/async; no test code is generated -
bench/ (
local/unicode-bench): workspace-only benchmarks.normperf/measures normalization withmoon bench;xlang/compares every module with other languages' libraries (moon run --target native bench/xlang), with one standalone harness per language in its subdirectories
Internal Data Packages (auto-generated)
- ucd/data/: Unicode Character Database lookup tables
- casefold/case_folding_data.mbt, casefold/nfkc_casefold_data.mbt: full default case folding and NFKC_CF mappings
- ucd/line_break.mbt, ucd/joining_group.mbt: the
LineBreakandJoiningGroupenums of the root package, generated so that they stay in step with the ordinals inucd/line_break/anducd/joining_group/ - idna/internal/idna/: IDNA mapping table (Joining_Type comes from
unicode-data) - bidi/internal/bidi/: Bidi mirroring and bracket tables
- segmentation/grapheme_break_property.mbt: grapheme cluster break properties
- segmentation/word_break_property.mbt, segmentation/sentence_break_property.mbt: word and sentence break properties
Dependencies
unicode-normalization -> unicode-data
unicode-casefold -> unicode-normalization
unicode-idna -> unicode-data
unicode-idna -> unicode-normalization
unicode-idna -> punycode
unicode-idna -> unicode-bidi
Code Generation
Unicode data tables are generated from official Unicode source files:
moon run --target native tools/gen data # Generate all Unicode lookup tables
moon run --target native tools/gen all # Alias for `data`
Individual commands are ucd, casefold, idna, bidi, and segmentation.
casefold regenerates folding tables only and is also included in data / all. Downloaded
Unicode data is cached per version in tools/.cache/<version>/.
Conformance Tests
Conformance tests live in the workspace-only tests/ module and read the
official Unicode test files (BidiTest.txt, BidiCharacterTest.txt,
NormalizationTest.txt, IdnaTestV2.txt, GraphemeBreakTest.txt,
WordBreakTest.txt, SentenceBreakTest.txt, LineBreakTest.txt) directly at
test time via
moonbitlang/core/async. The data files are committed under tests/data/, and
the tests use the standard library's async runtime, so they need no dependency
beyond the published modules.
Case folding conformance tests also read CaseFolding.txt and
DerivedNormalizationProps.txt, checking every Unicode scalar value against
the full default folding and NFKC_CF mappings.
Publishing
moon -C unicode package --list # Verify the compatibility umbrella
moon -C bidi package --list # Verify an individual feature module
Testing Notes
- Unit tests:
*_test.mbtfiles withtestblocks andinspect()assertions - Conformance tests:
tests/module (moonbit-community/unicode-tests), which reads the Unicode test data files intests/data/at run time - Fuzz tests:
fuzz_test.mbtfiles for property-based testing
Commit Convention
Uses conventional commits: fix:, feat:, refactor:, test:, chore:, release:
1# AGENTS.md23This file provides guidance to code agents (Claude Code, Codex, etc.) when working with code in this repository.45## Project Overview67A MoonBit library implementing Unicode standards: text normalization (UAX #15), Punycode (RFC 3492), IDNA (UTS #46), bidirectional text (UAX #9), and text segmentation (UAX #29). Targets Unicode 18.0.0.89## Build Commands1011```bash12moon check # Type check13moon test # Run all tests14moon test -p <pkg> # Test specific package (e.g., moon test -p punycode)15moon fmt # Format code16moon build # Build project17moon info # Regenerate pkg.generated.mbti files18```1920## Architecture2122### Public Modules2324- **ucd/** (`moonbit-community/unicode-data`): Unicode Character Database lookup,25 case mapping, and general category APIs26- **normalization/** (`moonbit-community/unicode-normalization`): UAX #15 Unicode27 normalization28- **casefold/** (`moonbit-community/unicode-casefold`): full case folding,29 canonical caseless matching, and NFKC_Casefold30- **punycode/** (`moonbit-community/punycode`): RFC 3492 Punycode encoding/decoding31 - `encode(StringView) -> String raise PunycodeError`32 - `encode_to(StringView, StringBuilder) -> Int32 raise PunycodeError` - appends33 the encoding and returns its length34 - `decode(StringView) -> String raise PunycodeError`35- **bidi/** (`moonbit-community/unicode-bidi`): UAX #9 bidirectional text processing36- **segmentation/** (`moonbit-community/unicode-segmentation`): UAX #29 text segmentation37 - `graphemes(StringView) -> Graphemes` - cursor over extended grapheme clusters38 (`iter()` for `for .. in`, `next()` returning an empty view at the end)39 - `word_segments(StringView) -> WordSegments` - all default word segments40 - `words(StringView) -> Words` - word segments containing Alphabetic or Number characters41 - `sentences(StringView) -> Sentences` - all default sentence segments42 - `line_breaks(StringView) -> LineBreaks` - default UAX #14 break opportunities43 - All cursors provide `next()` and `iter()` consuming the same state44- **idna/** (`moonbit-community/unicode-idna`): UTS #46 IDNA processing45 - `to_ascii()` - convert domain to ASCII (Punycode)46 - `to_unicode()` - convert domain from Punycode47- **unicode/** (`moonbit-community/unicode`): compatibility umbrella preserving the old48 package paths49- **tests/** (`moonbit-community/unicode-tests`): workspace-only conformance tests.50 They read the official Unicode test data files committed under `tests/data/`51 directly at run time with `moonbitlang/core/async`; no test code is generated5253- **bench/** (`local/unicode-bench`): workspace-only benchmarks. `normperf/`54 measures normalization with `moon bench`; `xlang/` compares every module with55 other languages' libraries (`moon run --target native bench/xlang`), with one56 standalone harness per language in its subdirectories5758### Internal Data Packages (auto-generated)5960- **ucd/data/**: Unicode Character Database lookup tables61- **casefold/case_folding_data.mbt**, **casefold/nfkc_casefold_data.mbt**:62 full default case folding and NFKC_CF mappings63- **ucd/line_break.mbt**, **ucd/joining_group.mbt**: the `LineBreak` and64 `JoiningGroup` enums of the root package, generated so that they stay in65 step with the ordinals in `ucd/line_break/` and `ucd/joining_group/`66- **idna/internal/idna/**: IDNA mapping table (Joining_Type comes from67 `unicode-data`)68- **bidi/internal/bidi/**: Bidi mirroring and bracket tables69- **segmentation/grapheme_break_property.mbt**: grapheme cluster break properties70- **segmentation/word_break_property.mbt**, **segmentation/sentence_break_property.mbt**:71 word and sentence break properties7273### Dependencies7475```76unicode-normalization -> unicode-data77unicode-casefold -> unicode-normalization78unicode-idna -> unicode-data79unicode-idna -> unicode-normalization80unicode-idna -> punycode81unicode-idna -> unicode-bidi82```8384## Code Generation8586Unicode data tables are generated from official Unicode source files:8788```bash89moon run --target native tools/gen data # Generate all Unicode lookup tables90moon run --target native tools/gen all # Alias for `data`91```9293Individual commands are `ucd`, `casefold`, `idna`, `bidi`, and `segmentation`.94`casefold` regenerates folding tables only and is also included in `data` / `all`. Downloaded95Unicode data is cached per version in `tools/.cache/<version>/`.9697## Conformance Tests9899Conformance tests live in the workspace-only `tests/` module and read the100official Unicode test files (`BidiTest.txt`, `BidiCharacterTest.txt`,101`NormalizationTest.txt`, `IdnaTestV2.txt`, `GraphemeBreakTest.txt`,102`WordBreakTest.txt`, `SentenceBreakTest.txt`, `LineBreakTest.txt`) directly at103test time via104`moonbitlang/core/async`. The data files are committed under `tests/data/`, and105the tests use the standard library's async runtime, so they need no dependency106beyond the published modules.107108Case folding conformance tests also read `CaseFolding.txt` and109`DerivedNormalizationProps.txt`, checking every Unicode scalar value against110the full default folding and NFKC_CF mappings.111112## Publishing113114```bash115moon -C unicode package --list # Verify the compatibility umbrella116moon -C bidi package --list # Verify an individual feature module117```118119## Testing Notes120121- Unit tests: `*_test.mbt` files with `test` blocks and `inspect()` assertions122- Conformance tests: `tests/` module (`moonbit-community/unicode-tests`), which123 reads the Unicode test data files in `tests/data/` at run time124- Fuzz tests: `fuzz_test.mbt` files for property-based testing125126## Commit Convention127128Uses conventional commits: `fix:`, `feat:`, `refactor:`, `test:`, `chore:`, `release:`129# AGENTS.md
This file provides guidance to code agents (Claude Code, Codex, etc.) when working with code in this repository.
## Project Overview
A MoonBit library implementing Unicode standards: text normalization (UAX #15), Punycode (RFC 3492), IDNA (UTS #46), bidirectional text (UAX #9), and text segmentation (UAX #29). Targets Unicode 18.0.0.
## Build Commands
```bash
moon check # Type check
moon test # Run all tests
moon test -p <pkg> # Test specific package (e.g., moon test -p punycode)
moon fmt # Format code
moon build # Build project
moon info # Regenerate pkg.generated.mbti files
```
## Architecture
### Public Modules
- **ucd/** (`moonbit-community/unicode-data`): Unicode Character Database lookup,
case mapping, and general category APIs
- **normalization/** (`moonbit-community/unicode-normalization`): UAX #15 Unicode
normalization
- **casefold/** (`moonbit-community/unicode-casefold`): full case folding,
canonical caseless matching, and NFKC_Casefold
- **punycode/** (`moonbit-community/punycode`): RFC 3492 Punycode encoding/decoding
- `encode(StringView) -> String raise PunycodeError`
- `encode_to(StringView, StringBuilder) -> Int32 raise PunycodeError` - appends
the encoding and returns its length
- `decode(StringView) -> String raise PunycodeError`
- **bidi/** (`moonbit-community/unicode-bidi`): UAX #9 bidirectional text processing
- **segmentation/** (`moonbit-community/unicode-segmentation`): UAX #29 text segmentation
- `graphemes(StringView) -> Graphemes` - cursor over extended grapheme clusters
(`iter()` for `for .. in`, `next()` returning an empty view at the end)
- `word_segments(StringView) -> WordSegments` - all default word segments
- `words(StringView) -> Words` - word segments containing Alphabetic or Number characters
- `sentences(StringView) -> Sentences` - all default sentence segments
- `line_breaks(StringView) -> LineBreaks` - default UAX #14 break opportunities
- All cursors provide `next()` and `iter()` consuming the same state
- **idna/** (`moonbit-community/unicode-idna`): UTS #46 IDNA processing
- `to_ascii()` - convert domain to ASCII (Punycode)
- `to_unicode()` - convert domain from Punycode
- **unicode/** (`moonbit-community/unicode`): compatibility umbrella preserving the old
package paths
- **tests/** (`moonbit-community/unicode-tests`): workspace-only conformance tests.
They read the official Unicode test data files committed under `tests/data/`
directly at run time with `moonbitlang/core/async`; no test code is generated
- **bench/** (`local/unicode-bench`): workspace-only benchmarks. `normperf/`
measures normalization with `moon bench`; `xlang/` compares every module with
other languages' libraries (`moon run --target native bench/xlang`), with one
standalone harness per language in its subdirectories
### Internal Data Packages (auto-generated)
- **ucd/data/**: Unicode Character Database lookup tables
- **casefold/case_folding_data.mbt**, **casefold/nfkc_casefold_data.mbt**:
full default case folding and NFKC_CF mappings
- **ucd/line_break.mbt**, **ucd/joining_group.mbt**: the `LineBreak` and
`JoiningGroup` enums of the root package, generated so that they stay in
step with the ordinals in `ucd/line_break/` and `ucd/joining_group/`
- **idna/internal/idna/**: IDNA mapping table (Joining_Type comes from
`unicode-data`)
- **bidi/internal/bidi/**: Bidi mirroring and bracket tables
- **segmentation/grapheme_break_property.mbt**: grapheme cluster break properties
- **segmentation/word_break_property.mbt**, **segmentation/sentence_break_property.mbt**:
word and sentence break properties
### Dependencies
```
unicode-normalization -> unicode-data
unicode-casefold -> unicode-normalization
unicode-idna -> unicode-data
unicode-idna -> unicode-normalization
unicode-idna -> punycode
unicode-idna -> unicode-bidi
```
## Code Generation
Unicode data tables are generated from official Unicode source files:
```bash
moon run --target native tools/gen data # Generate all Unicode lookup tables
moon run --target native tools/gen all # Alias for `data`
```
Individual commands are `ucd`, `casefold`, `idna`, `bidi`, and `segmentation`.
`casefold` regenerates folding tables only and is also included in `data` / `all`. Downloaded
Unicode data is cached per version in `tools/.cache/<version>/`.
## Conformance Tests
Conformance tests live in the workspace-only `tests/` module and read the
official Unicode test files (`BidiTest.txt`, `BidiCharacterTest.txt`,
`NormalizationTest.txt`, `IdnaTestV2.txt`, `GraphemeBreakTest.txt`,
`WordBreakTest.txt`, `SentenceBreakTest.txt`, `LineBreakTest.txt`) directly at
test time via
`moonbitlang/core/async`. The data files are committed under `tests/data/`, and
the tests use the standard library's async runtime, so they need no dependency
beyond the published modules.
Case folding conformance tests also read `CaseFolding.txt` and
`DerivedNormalizationProps.txt`, checking every Unicode scalar value against
the full default folding and NFKC_CF mappings.
## Publishing
```bash
moon -C unicode package --list # Verify the compatibility umbrella
moon -C bidi package --list # Verify an individual feature module
```
## Testing Notes
- Unit tests: `*_test.mbt` files with `test` blocks and `inspect()` assertions
- Conformance tests: `tests/` module (`moonbit-community/unicode-tests`), which
reads the Unicode test data files in `tests/data/` at run time
- Fuzz tests: `fuzz_test.mbt` files for property-based testing
## Commit Convention
Uses conventional commits: `fix:`, `feat:`, `refactor:`, `test:`, `chore:`, `release:`