Ada 4.1.1
Fast spec-compliant URL parser
Loading...
Searching...
No Matches
unicode_data_table Namespace Reference

Classes

class  UnicodeData

Functions

str get_ucd_file (str name, str version=UNICODE_VERSION)
set[int] parse_composition_exclusions (str text)
dict[int, int] parse_joining_types (str text)
tuple[list[int], list[int]] build_multistage (list[int] entries, int cols, list[bool] block_is_empty)
dict[str, list[int]] build_decomposition (UnicodeData ucd)
dict[str, list[int]] build_ccc (UnicodeData ucd)
dict[str, list[int]] build_composition (UnicodeData ucd, set[int] exclusions)
list[tuple[int, int, int]] build_ranges (dict[int, int] values)
dict[str, list[int]] build_validity (UnicodeData ucd, dict[int, int] joining_types)
dict[str, list[int]] build_sections (str unicode_data_text, str composition_exclusions_text, str joining_type_text)
str get_versioned_ucd_file (str name, str version)
dict[str, list[int]] generate (str version=UNICODE_VERSION)
None update_in_blob (dict[str, list[int]] generated)
int main (list[str] argv)

Variables

str UNICODE_VERSION = "18.0.0"
str UCD_URL = "https://www.unicode.org/Public/{version}/ucd/{name}"
int MAX_CODE_POINT = 0x10FFFF
tuple BLOCK_COUNT = (MAX_CODE_POINT + 1) >> 8
int HANGUL_SBASE = 0xAC00
int HANGUL_SCOUNT = 11172
dict COMBINING_CATEGORIES = {"Mn", "Mc", "Me"}
list BIDI_CLASSES
list JOINING_TYPES = ["U", "C", "D", "L", "R", "T"]

Detailed Description

Generate the normalization and label validity tables from UnicodeData.txt,
CompositionExclusions.txt and DerivedJoiningType.txt, and pack them into
src/table_blob.inc.

Use the same Unicode version as the mapping table (scripts/idna_table.py).

  python3 scripts/unicode_data_table.py          # print a summary
  python3 scripts/unicode_data_table.py --write  # update src/table_blob.inc

Sections
--------
Normalization (src/normalization.cpp). Each table is indexed by cp >> 8 into
rows of 256 entries; blocks without data share one all-zero row.

  decomposition_index/block/data
      Row entry = (offset << 2) | flags. The decomposition of cp is
      data[entry(cp) >> 2 .. entry(cp + 1) >> 2), so rows have 257 columns.
      The data holds the full (recursive) canonical decomposition, or the full
      compatibility decomposition when only that exists. Flags: 1 = the
      decomposition is compatibility-only; 2 = canonical, but the full
      compatibility decomposition differs. data[0] is unused. Hangul syllables
      are decomposed algorithmically and have no entries.
  ccc_index/block
      Canonical_Combining_Class.
  composition_index/block/data
      Row entry = index into data, which holds (second, composite) pairs
      sorted by second for each first code point; [entry(cp), entry(cp + 1))
      are the pairs starting with cp. Only primary composites (canonical
      decompositions of length two, not Full_Composition_Exclusion) appear.
      data[0] is unused.

Validity (src/validity.cpp).

  combining_flat
      General_Category Mn, Mc or Me ranges: a label must not begin with a mark.
  dir_start/dir_final/dir_value
      Bidi_Class ranges of assigned code points for the RFC 5893 Bidi rule.
      Unassigned code points are rejected by the mapping table first and look
      up as NONE.
  joining_start/joining_final/joining_value
      Joining_Type ranges for the RFC 5892 ContextJ rule for U+200C. Code
      points without a range are U (Non_Joining). The rule's Virama test
      uses Canonical_Combining_Class 9 from the ccc tables.

Function Documentation

◆ build_ccc()

dict[str, list[int]] build_ccc ( UnicodeData ucd)

Definition at line 221 of file unicode_data_table.py.

References build_multistage().

Referenced by build_sections().

◆ build_composition()

dict[str, list[int]] build_composition ( UnicodeData ucd,
set[int] exclusions )

Definition at line 231 of file unicode_data_table.py.

References build_multistage().

Referenced by build_sections().

◆ build_decomposition()

dict[str, list[int]] build_decomposition ( UnicodeData ucd)

Definition at line 190 of file unicode_data_table.py.

References build_multistage().

Referenced by build_sections().

◆ build_multistage()

tuple[list[int], list[int]] build_multistage ( list[int] entries,
int cols,
list[bool] block_is_empty )
Split per-code-point entries into an index and deduplicated rows.

Row b covers entries[b * 256 : b * 256 + cols]. Empty blocks get a zero
row.

Definition at line 165 of file unicode_data_table.py.

Referenced by build_ccc(), build_composition(), and build_decomposition().

◆ build_ranges()

list[tuple[int, int, int]] build_ranges ( dict[int, int] values)
Merge code points with equal values into (first, last, value) ranges.

Definition at line 263 of file unicode_data_table.py.

Referenced by build_validity().

◆ build_sections()

dict[str, list[int]] build_sections ( str unicode_data_text,
str composition_exclusions_text,
str joining_type_text )

◆ build_validity()

dict[str, list[int]] build_validity ( UnicodeData ucd,
dict[int, int] joining_types )

Definition at line 275 of file unicode_data_table.py.

References build_ranges().

Referenced by build_sections().

◆ generate()

dict[str, list[int]] generate ( str version = UNICODE_VERSION)

Definition at line 326 of file unicode_data_table.py.

References build_sections(), get_ucd_file(), and get_versioned_ucd_file().

Referenced by main().

◆ get_ucd_file()

str get_ucd_file ( str name,
str version = UNICODE_VERSION )
Return a UCD file, downloading it once into the working directory.

Definition at line 76 of file unicode_data_table.py.

Referenced by generate(), and get_versioned_ucd_file().

◆ get_versioned_ucd_file()

str get_versioned_ucd_file ( str name,
str version )
Return a UCD file whose '# Name-version.txt' header matches version.

Definition at line 316 of file unicode_data_table.py.

References get_ucd_file().

Referenced by generate().

◆ main()

int main ( list[str] argv)

Definition at line 345 of file unicode_data_table.py.

References generate(), main(), and update_in_blob().

Referenced by main().

◆ parse_composition_exclusions()

set[int] parse_composition_exclusions ( str text)

Definition at line 136 of file unicode_data_table.py.

Referenced by build_sections().

◆ parse_joining_types()

dict[int, int] parse_joining_types ( str text)
Map code points to JOINING_TYPES indexes; U is left out.

Definition at line 147 of file unicode_data_table.py.

Referenced by build_sections().

◆ update_in_blob()

None update_in_blob ( dict[str, list[int]] generated)

Definition at line 334 of file unicode_data_table.py.

References pack_tables.load_blob_sections(), and pack_tables.write_blob().

Referenced by main().

Variable Documentation

◆ BIDI_CLASSES

list unicode_data_table.BIDI_CLASSES
Initial value:
= [
"NONE", "BN", "CS", "ES", "ON", "EN", "L", "R", "NSM", "AL", "AN", "ET",
"WS", "RLO", "LRO", "PDF", "RLE", "RLI", "FSI", "PDI", "LRI", "B", "S",
"LRE",
]

Definition at line 66 of file unicode_data_table.py.

◆ BLOCK_COUNT

tuple unicode_data_table.BLOCK_COUNT = (MAX_CODE_POINT + 1) >> 8

Definition at line 58 of file unicode_data_table.py.

◆ COMBINING_CATEGORIES

dict unicode_data_table.COMBINING_CATEGORIES = {"Mn", "Mc", "Me"}

Definition at line 63 of file unicode_data_table.py.

◆ HANGUL_SBASE

int unicode_data_table.HANGUL_SBASE = 0xAC00

Definition at line 60 of file unicode_data_table.py.

◆ HANGUL_SCOUNT

int unicode_data_table.HANGUL_SCOUNT = 11172

Definition at line 61 of file unicode_data_table.py.

◆ JOINING_TYPES

list unicode_data_table.JOINING_TYPES = ["U", "C", "D", "L", "R", "T"]

Definition at line 73 of file unicode_data_table.py.

◆ MAX_CODE_POINT

int unicode_data_table.MAX_CODE_POINT = 0x10FFFF

Definition at line 57 of file unicode_data_table.py.

◆ UCD_URL

str unicode_data_table.UCD_URL = "https://www.unicode.org/Public/{version}/ucd/{name}"

Definition at line 55 of file unicode_data_table.py.

◆ UNICODE_VERSION

str unicode_data_table.UNICODE_VERSION = "18.0.0"

Definition at line 54 of file unicode_data_table.py.