unicode

Branch: main

Switch branches or tags

Branch: main
Go to file t

413 files indexed

Code

README.md

Normalization benchmarks

Run from the repository root:

moon bench ./bench/normperf
moon bench ./bench/normperf --target wasm
moon bench ./bench/normperf --target wasm-gc

Native is the default. The benchmark reads the committed files in data/ directly with moonbitlang/x/fs, verifies their sizes and SHA-256 hashes against corpus.json, removes the UTF-8 BOM where present, and preserves all remaining characters and line endings. After MoonBit dependencies are installed, running the benchmark needs no network access, ICU installation, or preparation command. There is no code generation or temporary runner project; Moon manages build artifacts in _build and prints timing statistics directly.

The matrix contains 168 cases: 14 full corpora × original/NFC/NFD input × NFC/NFD target × normalize/is_normalized. NFC and NFD inputs are produced with the library under test before timing. File IO, decoding, hashing, input preparation and consistency checks are outside the b.bench closures. All cases run sequentially in one benchmark block, using the standard moon bench sampling and statistics. The bench module belongs to the root workspace, which uses normalization and UCD from this checkout. The explicit package path selects only normperf; root workspace checks and tests also cover this module.

Already-normalized input may take the unchanged-string path. False checks may exit near the beginning, so they do not represent whole-input throughput. Generated NFC/NFD inputs and consistency checks are not an independent correctness oracle; the workspace's normalization conformance tests use the official Unicode test data. This benchmark measures MoonBit only.

To validate the corpus and input matrix without running timers:

moon test ./bench/normperf --target native
moon test ./bench/normperf --target wasm
moon test ./bench/normperf --target wasm-gc

The 14 source files are those named by ICU's NormPerf.pl, from the same pinned revision's collation corpus directory. They are committed unchanged (about 6.5 MiB), with their source URLs and hashes in corpus.json and the upstream license.