|
Ada 4.1.1
Fast spec-compliant URL parser
|
Classes | |
| class | UnicodeData |
Functions | |
| str | get_ucd_file (str name, str version=UNICODE_VERSION) |
| set[int] | parse_composition_exclusions (str text) |
| dict[int, int] | parse_joining_types (str text) |
| tuple[list[int], list[int]] | build_multistage (list[int] entries, int cols, list[bool] block_is_empty) |
| dict[str, list[int]] | build_decomposition (UnicodeData ucd) |
| dict[str, list[int]] | build_ccc (UnicodeData ucd) |
| dict[str, list[int]] | build_composition (UnicodeData ucd, set[int] exclusions) |
| list[tuple[int, int, int]] | build_ranges (dict[int, int] values) |
| dict[str, list[int]] | build_validity (UnicodeData ucd, dict[int, int] joining_types) |
| dict[str, list[int]] | build_sections (str unicode_data_text, str composition_exclusions_text, str joining_type_text) |
| str | get_versioned_ucd_file (str name, str version) |
| dict[str, list[int]] | generate (str version=UNICODE_VERSION) |
| None | update_in_blob (dict[str, list[int]] generated) |
| int | main (list[str] argv) |
Variables | |
| str | UNICODE_VERSION = "18.0.0" |
| str | UCD_URL = "https://www.unicode.org/Public/{version}/ucd/{name}" |
| int | MAX_CODE_POINT = 0x10FFFF |
| tuple | BLOCK_COUNT = (MAX_CODE_POINT + 1) >> 8 |
| int | HANGUL_SBASE = 0xAC00 |
| int | HANGUL_SCOUNT = 11172 |
| dict | COMBINING_CATEGORIES = {"Mn", "Mc", "Me"} |
| list | BIDI_CLASSES |
| list | JOINING_TYPES = ["U", "C", "D", "L", "R", "T"] |
Generate the normalization and label validity tables from UnicodeData.txt,
CompositionExclusions.txt and DerivedJoiningType.txt, and pack them into
src/table_blob.inc.
Use the same Unicode version as the mapping table (scripts/idna_table.py).
python3 scripts/unicode_data_table.py # print a summary
python3 scripts/unicode_data_table.py --write # update src/table_blob.inc
Sections
--------
Normalization (src/normalization.cpp). Each table is indexed by cp >> 8 into
rows of 256 entries; blocks without data share one all-zero row.
decomposition_index/block/data
Row entry = (offset << 2) | flags. The decomposition of cp is
data[entry(cp) >> 2 .. entry(cp + 1) >> 2), so rows have 257 columns.
The data holds the full (recursive) canonical decomposition, or the full
compatibility decomposition when only that exists. Flags: 1 = the
decomposition is compatibility-only; 2 = canonical, but the full
compatibility decomposition differs. data[0] is unused. Hangul syllables
are decomposed algorithmically and have no entries.
ccc_index/block
Canonical_Combining_Class.
composition_index/block/data
Row entry = index into data, which holds (second, composite) pairs
sorted by second for each first code point; [entry(cp), entry(cp + 1))
are the pairs starting with cp. Only primary composites (canonical
decompositions of length two, not Full_Composition_Exclusion) appear.
data[0] is unused.
Validity (src/validity.cpp).
combining_flat
General_Category Mn, Mc or Me ranges: a label must not begin with a mark.
dir_start/dir_final/dir_value
Bidi_Class ranges of assigned code points for the RFC 5893 Bidi rule.
Unassigned code points are rejected by the mapping table first and look
up as NONE.
joining_start/joining_final/joining_value
Joining_Type ranges for the RFC 5892 ContextJ rule for U+200C. Code
points without a range are U (Non_Joining). The rule's Virama test
uses Canonical_Combining_Class 9 from the ccc tables.
| dict[str, list[int]] build_ccc | ( | UnicodeData | ucd | ) |
Definition at line 221 of file unicode_data_table.py.
References build_multistage().
Referenced by build_sections().
| dict[str, list[int]] build_composition | ( | UnicodeData | ucd, |
| set[int] | exclusions ) |
Definition at line 231 of file unicode_data_table.py.
References build_multistage().
Referenced by build_sections().
| dict[str, list[int]] build_decomposition | ( | UnicodeData | ucd | ) |
Definition at line 190 of file unicode_data_table.py.
References build_multistage().
Referenced by build_sections().
| tuple[list[int], list[int]] build_multistage | ( | list[int] | entries, |
| int | cols, | ||
| list[bool] | block_is_empty ) |
Split per-code-point entries into an index and deduplicated rows. Row b covers entries[b * 256 : b * 256 + cols]. Empty blocks get a zero row.
Definition at line 165 of file unicode_data_table.py.
Referenced by build_ccc(), build_composition(), and build_decomposition().
| list[tuple[int, int, int]] build_ranges | ( | dict[int, int] | values | ) |
Merge code points with equal values into (first, last, value) ranges.
Definition at line 263 of file unicode_data_table.py.
Referenced by build_validity().
| dict[str, list[int]] build_sections | ( | str | unicode_data_text, |
| str | composition_exclusions_text, | ||
| str | joining_type_text ) |
Definition at line 300 of file unicode_data_table.py.
References build_ccc(), build_composition(), build_decomposition(), build_validity(), parse_composition_exclusions(), and parse_joining_types().
Referenced by generate().
| dict[str, list[int]] build_validity | ( | UnicodeData | ucd, |
| dict[int, int] | joining_types ) |
Definition at line 275 of file unicode_data_table.py.
References build_ranges().
Referenced by build_sections().
| dict[str, list[int]] generate | ( | str | version = UNICODE_VERSION | ) |
Definition at line 326 of file unicode_data_table.py.
References build_sections(), get_ucd_file(), and get_versioned_ucd_file().
Referenced by main().
| str get_ucd_file | ( | str | name, |
| str | version = UNICODE_VERSION ) |
Return a UCD file, downloading it once into the working directory.
Definition at line 76 of file unicode_data_table.py.
Referenced by generate(), and get_versioned_ucd_file().
| str get_versioned_ucd_file | ( | str | name, |
| str | version ) |
Return a UCD file whose '# Name-version.txt' header matches version.
Definition at line 316 of file unicode_data_table.py.
References get_ucd_file().
Referenced by generate().
| int main | ( | list[str] | argv | ) |
Definition at line 345 of file unicode_data_table.py.
References generate(), main(), and update_in_blob().
Referenced by main().
| set[int] parse_composition_exclusions | ( | str | text | ) |
Definition at line 136 of file unicode_data_table.py.
Referenced by build_sections().
| dict[int, int] parse_joining_types | ( | str | text | ) |
Map code points to JOINING_TYPES indexes; U is left out.
Definition at line 147 of file unicode_data_table.py.
Referenced by build_sections().
| None update_in_blob | ( | dict[str, list[int]] | generated | ) |
Definition at line 334 of file unicode_data_table.py.
References pack_tables.load_blob_sections(), and pack_tables.write_blob().
Referenced by main().
| list unicode_data_table.BIDI_CLASSES |
Definition at line 66 of file unicode_data_table.py.
| tuple unicode_data_table.BLOCK_COUNT = (MAX_CODE_POINT + 1) >> 8 |
Definition at line 58 of file unicode_data_table.py.
| dict unicode_data_table.COMBINING_CATEGORIES = {"Mn", "Mc", "Me"} |
Definition at line 63 of file unicode_data_table.py.
| int unicode_data_table.HANGUL_SBASE = 0xAC00 |
Definition at line 60 of file unicode_data_table.py.
| int unicode_data_table.HANGUL_SCOUNT = 11172 |
Definition at line 61 of file unicode_data_table.py.
| list unicode_data_table.JOINING_TYPES = ["U", "C", "D", "L", "R", "T"] |
Definition at line 73 of file unicode_data_table.py.
| int unicode_data_table.MAX_CODE_POINT = 0x10FFFF |
Definition at line 57 of file unicode_data_table.py.
| str unicode_data_table.UCD_URL = "https://www.unicode.org/Public/{version}/ucd/{name}" |
Definition at line 55 of file unicode_data_table.py.
| str unicode_data_table.UNICODE_VERSION = "18.0.0" |
Definition at line 54 of file unicode_data_table.py.