Database changes have finished applying - please report any issues you're (still) seeing to support@shoutwiki.com.
Module:Unicode data
Template:High riskTemplate:Cascade-protected template
![]() | Uses Lua: |
Usage
This module provides functions that access information on Unicode code points. The information is retrieved from data modules generated from the Unicode Character Database, or derived by rules given in the Unicode Specification. It and its submodules were copied from English Wiktionary and then modified; see there for more information.
Parameters and functions
code point
The code point is to be entered as hexadecimal value. For example, Expression error: Unrecognized punctuation character "[".:
- |A9hex
- |0xA9hex
- |0x00A9hex
- |0x00a9hex
{{#invoke:Unicode data|lookup|name|0x00A9}}
→ <reserved-00A9>
Incorrect or unintended results:
- 169dec:
{{#invoke:Unicode data|lookup|name|169}}
→ <reserved-0169> Template:Nay —"U+00A9"©
expected; but is read as 00A9hex (that is, Template:Hex2decdec - U+00A9
{{#invoke:Unicode data|lookup|name|U+00A9}}
Template:Nay —do not use "U+" prefix - غ
{{#invoke:Unicode data|lookup|name|غ}}
Template:Nay —cannot enter a character as codepoint
"lookup" and "is" functions
lookup, is
- Template-invokable functions that allow access to the functions starting with
lookup
andis
.For most of the functions, add the code point in hexadecimal base as the next parameter. Foris|Latin
,is|rtl
, andis|valid_pagename
, add character string. HTML character references in the text are decoded by the module into code points. - For example, {{#invoke:Unicode data|is|Latin|àzàhàr̃iyyā̀}} → Lua error at line 297: attempt to index local 'data_module' (a boolean value)..
- Internally, in modules, these functions are named using underscore:
lookup_name|code point
←lookup_name
- For
&A9;
©: {{#invoke:Unicode data|lookup|name|A9}} → <reserved-00A9>
Functions overview
- Code points: enter hexadecimal value, for example Template:Mxt or Template:Mxt; not Template:!mxt.
Topic | Function | Parameter type (string=by character(s); c.p. by 0xHex value) |
Example | Returns | Character |
---|---|---|---|---|---|
Unicode character name | |lookup|name
|
code point | Template:Ubl | Template:Ubl | Template:Ubl |
Scripts | |lookup|script
|
code point | lookup|script|A061}} | Lua error at line 297: attempt to index local 'data_module' (a boolean value). | Template:Ubl |
Blocks | |lookup|block
|
code point | lookup|block|A061}} | Lua error at line 15: attempt to index local 'ranges' (a boolean value). | Template:Ubl |
Planes | |lookup|plane
|
code point | Template:Ubl | Template:Ubl | Template:Ubl
|
General Category | |lookup|category
|
code point | Template:Ubl | Template:Ubl | Template:Ubl |
Controls | |is|control
|
code point | Template:Ubl | Template:Ubl | Template:Ubl |
Unihan properties | |lookup|kCantonese
|
code point | Template:Ubl | Template:Ubl | Template:Ubl |
Latin script | |is|Latin
|
string | Template:Ubl | Template:Ubl | |
WP:Article title (WP:NCTR) | |is|valid_pagename
|
string | Template:Ubli | Template:Ubl | |
Bidirectionality, right-to-left scripts | |is|rtl
|
string | Template:Ubl | Template:Ubli | Template:Ubl |
Combining character | |is|combining
|
code point | Template:Ubl | Template:Ubl | Template:Ubl |
Character assignation | |is|assigned
|
code point | Template:Ubl | Template:Ubl | Template:Ubl |
Printable | |is|printable
|
code point | Template:Ubl | Template:Ubl | Template:Ubl |
Template:Slink | |is|whitespace
|
code point | Template:Ubl | Template:Ubl | Template:Ubl |
Alias names | |aliases
|
[application unknown] | Template:Ubl | ||
Combining class | |
|
[application unknown] | Template:Ubl | ||
Age | |
|
[application unknown] | Template:Ubl | ||
get_best_script | |get_best_script
|
[application unknown] | Template:Ubl |
Data modules
The data used by functions in this module is found in submodules. Some are generated by AWK scripts shown at User:Kephir/Unicode on English Wiktionary, others by Lua scripts on the /make
subpages of the submodules.
- Module:Unicode data/age: 'Age' of a character, that is: version introduction number.
- Module:Unicode data/aliases: the formal name aliases for characters (from NameAliases.txt)
- Module:Unicode data/blocks: the list of Unicode blocks (from Blocks.txt)
- Module:Unicode data/category: data mapping characters to their General Category (from DerivedGeneralCategory.txt)
- Module:Unicode data/combining: data mapping characters to their Combining Classes (from DerivedCombiningClass.txt)
- Module:Unicode data/control: data for identifying characters that belong to the General Categories of Separator and Other (from DerivedGeneralCategory.txt)
- Module:Unicode data/derived core properties:
- Module:Unicode data/Hangul: data used to generate the names of Hangul syllables (from Jamo.txt)
- Module:Unicode data/Unihan/kCantonese/*: kCantonese pronunciation property
- Module:Unicode data/names/* (000hh, .., 0E0hh; e.g. ../names/000): names
- Module:Unicode data/scripts: data mapping characters to their Unicode script properties (from Scripts.txt).
The name data modules (Module:Unicode data/names/xxx) were compiled from UnicodeData.txt. Each one contains, at maximum, code points U+xxx000 to U+xxxFFF. Script error: No such module "Unicode data/documentation functions".
Copyright
The Unicode database is released by Unicode Inc. under the following terms:
Copyright © 1991-2018 Unicode, Inc. All rights reserved. Distributed under the Terms of Use in https://www.unicode.org/copyright.html.
Permission is hereby granted, free of charge, to any person obtaining a copy of the Unicode data files and any associated documentation (the "Data Files") or Unicode software and any associated documentation (the "Software") to deal in the Data Files or Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, and/or sell copies of the Data Files or Software, and to permit persons to whom the Data Files or Software are furnished to do so, provided that either (a) this copyright and permission notice appear with all copies of the Data Files or Software, or (b) this copyright and permission notice appear in associated Documentation.
THE DATA FILES AND SOFTWARE ARE PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF THIRD PARTY RIGHTS. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR HOLDERS INCLUDED IN THIS NOTICE BE LIABLE FOR ANY CLAIM, OR ANY SPECIAL INDIRECT OR CONSEQUENTIAL DAMAGES, OR ANY DAMAGES WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THE DATA FILES OR SOFTWARE.
Except as contained in this notice, the name of a copyright holder shall not be used in advertising or otherwise to promote the sale, use or other dealings in these Data Files or Software without prior written authorization of the copyright holder.
Known issues
- Reading data like
Module:Unicode data/aliases
not provided nor documented - Test fail:
lookup_category
U+FFFF (<noncharacter-FFFF>) expected: Cn.
{{#invoke:Unicode data|lookup|category|0xFFFF}}
→ Lua error at line 297: attempt to index local 'data_module' (a boolean value). [Nil]
See also
- Named entities: for example, Expression error: Unrecognized punctuation character "[".:
{{#invoke:LoadData|Numcr2namecr|0x22C1}}
→ ⋁, ⋁, ⋁
The above documentation is transcluded from Module:Unicode data/doc. (edit | history) Editors can experiment in this module's sandbox (create | mirror) and testcases (create) pages. Subpages of this module. |
1 local p = {}
2
3 local floor = math.floor
4
5 local function errorf(level, ...)
6 if type(level) == "number" then
7 return error(string.format(...), level + 1)
8 else -- level is actually the format string.
9 return error(string.format(level, ...), 2)
10 end
11 end
12
13 local function binary_range_search(codepoint, ranges)
14 local low, mid, high
15 low, high = 1, ranges.length or require "Module:TableTools".length(ranges)
16 while low <= high do
17 mid = floor((low + high) / 2)
18 local range = ranges[mid]
19 if codepoint < range[1] then
20 high = mid - 1
21 elseif codepoint <= range[2] then
22 return range, mid
23 else
24 low = mid + 1
25 end
26 end
27 return nil, mid
28 end
29 p.binary_range_search = binary_range_search
30
31 --[[
32 local function linear_range_search(codepoint, ranges)
33 for i, range in ipairs(ranges) do
34 if range[1] <= codepoint and codepoint <= range[2] then
35 return range
36 end
37 end
38 end
39 --]]
40
41 -- Load a module by indexing "loader" with the name of the module minus the
42 -- "Module:Unicode data/" part. For instance, loader.blocks returns
43 -- [[Module:Unicode data/blocks]]. If a module cannot be loaded, false will be
44 -- returned.
45 local loader = setmetatable({}, {
46 __index = function (self, key)
47 local success, data = pcall(mw.loadData, "Module:Unicode data/" .. key)
48 if not success then
49 data = false
50 end
51 self[key] = data
52 return data
53 end
54 })
55
56 -- For the algorithm used to generate Hangul Syllable names,
57 -- see "Hangul Syllable Name Generation" in section 3.12 of the
58 -- Unicode Specification:
59 -- https://www.unicode.org/versions/Unicode11.0.0/ch03.pdf
60 local name_hooks = {
61 { 0x00, 0x1F, "<control-%04X>" }, -- C0 control characters
62 { 0x7F, 0x9F, "<control-%04X>" }, -- DEL and C1 control characters
63 { 0x3400, 0x4DBF, "CJK UNIFIED IDEOGRAPH-%04X" }, -- CJK Ideograph Extension A
64 { 0x4E00, 0x9FFF, "CJK UNIFIED IDEOGRAPH-%04X" }, -- CJK Ideograph
65 { 0xAC00, 0xD7A3, function (codepoint) -- Hangul Syllables
66 local Hangul_data = loader.Hangul
67 local syllable_index = codepoint - 0xAC00
68
69 return ("HANGUL SYLLABLE %s%s%s"):format(
70 Hangul_data.leads[floor(syllable_index / Hangul_data.final_count)],
71 Hangul_data.vowels[floor((syllable_index % Hangul_data.final_count)
72 / Hangul_data.trail_count)],
73 Hangul_data.trails[syllable_index % Hangul_data.trail_count]
74 )
75 end },
76 -- High Surrogates, High Private Use Surrogates, Low Surrogates
77 { 0xD800, 0xDFFF, "<surrogate-%04X>" },
78 { 0xE000, 0xF8FF, "<private-use-%04X>" }, -- Private Use
79 -- CJK Compatibility Ideographs
80 { 0xF900, 0xFA6D, "CJK COMPATIBILITY IDEOGRAPH-%04X" },
81 { 0xFA70, 0xFAD9, "CJK COMPATIBILITY IDEOGRAPH-%04X" },
82 { 0x17000, 0x187F7, "TANGUT IDEOGRAPH-%04X" }, -- Tangut Ideograph
83 { 0x18800, 0x18AFF, function (codepoint)
84 return ("TANGUT COMPONENT-%03d"):format(codepoint - 0x187FF)
85 end },
86 { 0x18D00, 0x18D08, "TANGUT IDEOGRAPH-%04X" }, -- Tangut Ideograph Supplement
87 { 0x1B170, 0x1B2FB, "NUSHU CHARACTER-%04X" }, -- Nushu
88 { 0x20000, 0x2A6DF, "CJK UNIFIED IDEOGRAPH-%04X" }, -- CJK Ideograph Extension B
89 { 0x2A700, 0x2B739, "CJK UNIFIED IDEOGRAPH-%04X" }, -- CJK Ideograph Extension C
90 { 0x2B740, 0x2B81D, "CJK UNIFIED IDEOGRAPH-%04X" }, -- CJK Ideograph Extension D
91 { 0x2B820, 0x2CEA1, "CJK UNIFIED IDEOGRAPH-%04X" }, -- CJK Ideograph Extension E
92 { 0x2CEB0, 0x2EBE0, "CJK UNIFIED IDEOGRAPH-%04X" }, -- CJK Ideograph Extension F
93 -- CJK Compatibility Ideographs Supplement (Supplementary Ideographic Plane)
94 { 0x2F800, 0x2FA1D, "CJK COMPATIBILITY IDEOGRAPH-%04X" },
95 { 0xE0100, 0xE01EF, function (codepoint) -- Variation Selectors Supplement
96 return ("VARIATION SELECTOR-%d"):format(codepoint - 0xE0100 + 17)
97 end},
98 { 0x30000, 0x3134A, "CJK UNIFIED IDEOGRAPH-%04X" }, -- CJK Ideograph Extension G
99 { 0x31350, 0x323AF, "CJK UNIFIED IDEOGRAPH-%04X" }, -- CJK Ideograph Extension H
100 { 0x2EBF0, 0x2EE5D, "CJK UNIFIED IDEOGRAPH-%04X" }, -- CJK Ideograph Extension I
101 { 0xF0000, 0xFFFFD, "<private-use-%04X>" }, -- Plane 15 Private Use
102 { 0x100000, 0x10FFFD, "<private-use-%04X>" } -- Plane 16 Private Use
103 }
104 name_hooks.length = #name_hooks
105
106 local name_range_cache
107
108 local function generate_name(data, codepoint)
109 if type(data) == "string" then
110 return data:format(codepoint)
111 else
112 return data(codepoint)
113 end
114 end
115
116 --[[
117 -- Checks that the code point is a number and in range.
118 -- Does not check whether code point is an integer.
119 -- Not used
120 local function check_codepoint(funcName, argIdx, val)
121 require 'libraryUtil'.checkType(funcName, argIdx, val, 'number')
122 if codepoint < 0 or 0x10FFFF < codepoint then
123 errorf("Codepoint %04X out of range", codepoint)
124 end
125 end
126 --]]
127
128 function p.is_noncharacter(codepoint)
129 -- U+FDD0-U+FDEF and all code points ending in FFFE or FFFF are Unassigned
130 -- (Cn) and specifically noncharacters:
131
132 -- https://www.unicode.org/faq/private_use.html#nonchar4
133 return 0xFDD0 <= codepoint and (codepoint <= 0xFDEF
134 or floor(codepoint % 0x10000) >= 0xFFFE)
135 end
136
137 -- https://www.unicode.org/versions/Unicode11.0.0/ch04.pdf, section 4.8
138 function p.lookup_name(codepoint)
139 if p.is_noncharacter(codepoint) then
140 return ("<noncharacter-%04X>"):format(codepoint)
141 end
142
143 if name_range_cache -- Check if previously used "name hook" applies to this code point.
144 and codepoint >= name_range_cache[1]
145 and codepoint <= name_range_cache[2] then
146 return generate_name(name_range_cache[3], codepoint)
147 end
148
149 local range = binary_range_search(codepoint, name_hooks)
150 if range then
151 name_range_cache = range
152 return generate_name(range[3], codepoint)
153 end
154
155 local data = loader[('names/%03X'):format(codepoint / 0x1000)]
156
157 if data and data[codepoint] then
158 return data[codepoint]
159
160 -- Unassigned (Cn) consists of noncharacters and reserved characters.
161 -- The character has been established not to be a noncharacter,
162 -- and if it were assigned, its name would already been retrieved,
163 -- so it must be reserved.
164 else
165 return ("<reserved-%04X>"):format(codepoint)
166 end
167 end
168
169 function p.lookup_image(codepoint)
170 local data = loader[('images/%03X'):format(codepoint / 0x1000)]
171
172 if data then
173 return data[codepoint]
174 end
175 end
176
177 local planes = {
178 [ 0] = "Basic Multilingual Plane";
179 [ 1] = "Supplementary Multilingual Plane";
180 [ 2] = "Supplementary Ideographic Plane";
181 [ 3] = "Tertiary Ideographic Plane";
182 [14] = "Supplementary Special-purpose Plane";
183 [15] = "Supplementary Private Use Area-A";
184 [16] = "Supplementary Private Use Area-B";
185 }
186
187 -- Load [[Module:Unicode data/blocks]] if needed and assign it to this variable.
188 local blocks
189
190 local function block_iter(blocks, i)
191 i = i + 1
192 local data = blocks[i]
193 if data then
194 -- Unpack doesn't work on tables loaded with mw.loadData.
195 return i, data[1], data[2], data[3]
196 end
197 end
198
199 -- An ipairs-type iterator generator for the list of blocks.
200 function p.enum_blocks()
201 local blocks = loader.blocks
202 return block_iter, blocks, 0
203 end
204
205 function p.lookup_plane(codepoint)
206 local i = floor(codepoint / 0x10000)
207 return planes[i] or ("Plane %u"):format(i)
208 end
209
210 function p.lookup_block(codepoint)
211 local blocks = loader.blocks
212 local range = binary_range_search(codepoint, blocks)
213 if range then
214 return range[3]
215 else
216 return "No Block"
217 end
218 end
219
220 function p.get_block_info(name)
221 for i, block in ipairs(loader.blocks) do
222 if block[3] == name then
223 return block
224 end
225 end
226 end
227
228 function p.is_valid_pagename(pagename)
229 local has_nonws = false
230
231 for cp in mw.ustring.gcodepoint(pagename) do
232 if (cp == 0x0023) -- #
233 or (cp == 0x005B) -- [
234 or (cp == 0x005D) -- ]
235 or (cp == 0x007B) -- {
236 or (cp == 0x007C) -- |
237 or (cp == 0x007D) -- }
238 or (cp == 0x180E) -- MONGOLIAN VOWEL SEPARATOR
239 or ((cp >= 0x2000) and (cp <= 0x200A)) -- spaces in General Punctuation block
240 or (cp == 0xFFFD) -- REPLACEMENT CHARACTER
241 then
242 return false
243 end
244
245 local printable, result = p.is_printable(cp)
246 if not printable then
247 return false
248 end
249
250 if result ~= "space-separator" then
251 has_nonws = true
252 end
253 end
254
255 return has_nonws
256 end
257
258 local function manual_unpack(what, from)
259 if what[from + 1] == nil then
260 return what[from]
261 end
262
263 local result = {}
264 from = from or 1
265 for i, item in ipairs(what) do
266 if i >= from then
267 table.insert(result, item)
268 end
269 end
270 return unpack(result)
271 end
272
273 local function compare_ranges(range1, range2)
274 return range1[1] < range2[1]
275 end
276
277 -- Creates a function to look up data in a module that contains "singles" (a
278 -- code point-to-data map) and "ranges" (an array containing arrays that contain
279 -- the low and high code points of a range and the data associated with that
280 -- range).
281 -- "loader" loads and returns the "singles" and "ranges" tables.
282 -- "match_func" is passed the code point and either the data or the "dots", and
283 -- generates the final result of the function.
284 -- The varargs ("dots") describes the default data to be returned if there wasn't
285 -- a match.
286 -- In case the function is used more than once, "cache" saves ranges that have
287 -- already been found to match, or a range whose data is the default if there
288 -- was no match.
289 local function memo_lookup(data_module_subpage, match_func, ...)
290 local dots = { ... }
291 local cache = {}
292 local singles, ranges
293
294 return function (codepoint)
295 if not singles then
296 local data_module = loader[data_module_subpage]
297 singles, ranges = data_module.singles, data_module.ranges
298 end
299
300 if singles[codepoint] then
301 return match_func(codepoint, singles[codepoint])
302 end
303
304 local range = binary_range_search(codepoint, cache)
305 if range then
306 return match_func(codepoint, manual_unpack(range, 3))
307 end
308
309 local range, index = binary_range_search(codepoint, ranges)
310 if range then
311 table.insert(cache, range)
312 table.sort(cache, compare_ranges)
313 return match_func(codepoint, manual_unpack(range, 3))
314 end
315
316 if ranges[index] then
317 local dots_range
318 if codepoint > ranges[index][2] then
319 dots_range = {
320 ranges[index][2] + 1,
321 ranges[index + 1] and ranges[index + 1][1] - 1 or 0x10FFFF,
322 unpack(dots)
323 }
324 else -- codepoint < range[index][1]
325 dots_range = {
326 ranges[index - 1] and ranges[index - 1][2] + 1 or 0,
327 ranges[index][1] - 1,
328 unpack(dots)
329 }
330 end
331 table.sort(cache, compare_ranges)
332 end
333
334 return match_func(codepoint)
335 end
336 end
337
338 -- Get a code point's combining class value in [[Module:Unicode data/combining]],
339 -- and return whether this value is not zero. Zero is assigned as the default
340 -- if the combining class value is not found in this data module.
341 -- That is, return true if character is combining, or false if it is not.
342 -- See https://www.unicode.org/reports/tr44/#Canonical_Combining_Class_Values for
343 -- more information.
344 p.is_combining = memo_lookup(
345 "combining",
346 function (codepoint, combining_class)
347 return combining_class and combining_class ~= 0 or false
348 end,
349 0)
350
351 function p.add_dotted_circle(str)
352 return (mw.ustring.gsub(str, ".",
353 function(char)
354 if p.is_combining(mw.ustring.codepoint(char)) then
355 return '◌' .. char
356 end
357 end))
358 end
359
360 local lookup_control = memo_lookup(
361 "control",
362 function (codepoint, ccc)
363 return ccc or "assigned"
364 end,
365 "assigned")
366 p.lookup_control = lookup_control
367
368 function p.is_assigned(codepoint)
369 return lookup_control(codepoint) ~= "unassigned"
370 end
371
372 function p.is_printable(codepoint)
373 local result = lookup_control(codepoint)
374 return (result == "assigned") or (result == "space-separator"), result
375 end
376
377 function p.is_whitespace(codepoint)
378 local result = lookup_control(codepoint)
379 return (result == "space-separator"), result
380 end
381
382 p.lookup_category = memo_lookup(
383 "category",
384 function (codepoint, category)
385 return category
386 end,
387 "Cn")
388
389 local lookup_script = memo_lookup(
390 "scripts",
391 function (codepoint, script_code)
392 return script_code or 'Zzzz'
393 end,
394 "Zzzz")
395 p.lookup_script = lookup_script
396
397 function p.get_best_script(str)
398 -- Check type of argument, because mw.text.decode coerces numbers to strings!
399 require "libraryUtil".checkType("get_best_script", 1, str, "string")
400
401 -- Convert HTML character references (including named character references,
402 -- or character entities) to characters.
403 str = mw.text.decode(str, true)
404
405 local scripts = {}
406 for codepoint in mw.ustring.gcodepoint(str) do
407 local script = lookup_script(codepoint)
408
409 -- Ignore "Inherited", "Undetermined", or "Uncoded" scripts.
410 if not (script == "Zyyy" or script == "Zinh" or script == "Zzzz") then
411 scripts[script] = true
412 end
413 end
414
415 -- If scripts does not contain two or more keys,
416 -- return first and only key (script code) in table.
417 if not next(scripts, next(scripts)) then
418 return next(scripts)
419 end -- else return majority script, or else "Zzzz"?
420 end
421
422 function p.is_Latin(str)
423 require "libraryUtil".checkType("get_best_script", 1, str, "string")
424 str = mw.text.decode(str, true)
425
426 -- Search for the leading bytes that introduce the UTF-8 encoding of the
427 -- code points U+0340-U+10FFFF. If they are not found and there is at least
428 -- one Latin-script character, the string counts as Latin, because the rest
429 -- of the characters can only be Zyyy, Zinh, and Zzzz.
430 -- The only scripts found below U+0370 (the first code point of the Greek
431 -- and Coptic block) are Latn, Zyyy, Zinh, and Zzzz.
432 -- See the codepage in the [[UTF-8]] article.
433 if not str:find "[\205-\244]" then
434 for codepoint in mw.ustring.gcodepoint(str) do
435 if lookup_script(codepoint) == "Latn" then
436 return true
437 end
438 end
439 end
440
441 local Latn = false
442 local i = 0; -- indexer for use in error messages
443
444 for codepoint in mw.ustring.gcodepoint(str) do
445 i = i + 1; -- bump the indexer
446 local script = lookup_script(codepoint)
447
448 if script == "Latn" then
449 Latn = true
450 elseif not (script == "Zyyy" or script == "Zinh"
451 or script == "Zzzz") then
452 return false, i -- abandon as not Latn; identify the offending character's position
453 end
454 end
455
456 return Latn, (not Latn and i) or nil -- when <Latn> false, return offending charactor's position as second return value; nil else
457 end
458
459 -- Checks that a string contains only characters belonging to right-to-left
460 -- scripts, or characters of ignorable scripts.
461 function p.is_rtl(str)
462 require "libraryUtil".checkType("get_best_script", 1, str, "string")
463 str = mw.text.decode(str, true)
464
465 -- Search for the leading bytes that introduce the UTF-8 encoding of the
466 -- code points U+0580-U+10FFFF. If they are not found, the string can only
467 -- have characters from a left-to-right script, because the first code point
468 -- in a right-to-left script is U+0591, in the Hebrew block.
469 if not str:find "[\214-\244]" then
470 return false
471 end
472
473 local result = false
474 local rtl = loader.scripts.rtl
475 for codepoint in mw.ustring.gcodepoint(str) do
476 local script = lookup_script(codepoint)
477
478 if rtl[script] then
479 result = true
480 elseif not (script == "Zyyy" or script == "Zinh"
481 or script == "Zzzz") then
482 return false
483 end
484 end
485
486 return result
487 end
488
489
490 --[[--------------------------< I S _ R T L _ F R A M E >------------------------------------------------------
491
492 external entry from an {{#invoke:}} to determine if a string of text is rtl. Strips html and html-like tags so
493 that those tags don't corrupt the is-rtl-is-not-rtl determination; this added for the cases where the rtl text
494 has <br /> tags.
495
496 ]]
497
498 function p.is_rtl_frame (frame)
499 local str = frame.args[1]; -- get the string from the {{#invoke:}} frame
500 str = str:gsub ('%b<>', ''); -- strip any html and html-like tags
501 return p.is_rtl (str); -- return if whatever remains rtl; false else
502 end
503
504
505 local function get_codepoint(args, arg)
506 local codepoint_string = args[arg]
507 or errorf(2, "Parameter %s is required", tostring(arg))
508 local codepoint = tonumber(codepoint_string, 16)
509 or errorf(2, "Parameter %s is not a code point in hexadecimal base",
510 tostring(arg))
511 if not (0 <= codepoint and codepoint <= 0x10FFFF) then
512 errorf(2, "code point in parameter %s out of range", tostring(arg))
513 end
514 return codepoint
515 end
516
517 local function get_func(args, arg, prefix)
518 local suffix = args[arg]
519 or errorf(2, "Parameter %s is required", tostring(arg))
520 suffix = mw.text.trim(suffix)
521 local func_name = prefix .. suffix
522 local func = p[func_name]
523 or errorf(2, "There is no function '%s'", func_name)
524 return func
525 end
526
527 -- This function allows any of the "lookup" functions to be invoked. The first
528 -- parameter is the word after "lookup_"; the second parameter is the code point
529 -- in hexadecimal base.
530 function p.lookup(frame)
531 local func = get_func(frame.args, 1, "lookup_")
532 local codepoint = get_codepoint(frame.args, 2)
533 local result = func(codepoint)
534 if func == p.lookup_name then
535 -- Prevent code point labels such as <control-0000> from being
536 -- interpreted as HTML tags.
537 result = result:gsub("<", "<")
538 end
539 return result
540 end
541
542 function p.is(frame)
543 local func = get_func(frame.args, 1, "is_")
544
545 -- is_Latin and is_valid_pagename take strings.
546 if func == p.is_Latin or func == p.is_valid_pagename or func == p.is_rtl then
547 return (func(frame.args[2]))
548 else -- The rest take code points.
549 local codepoint = get_codepoint(frame.args, 2)
550 return (func(codepoint)) -- Adjust to one result.
551 end
552 end
553
554 function p.lookup_kCantonese(codepoint)
555 local data = loader[('Unihan/kCantonese/%02X'):format(floor(codepoint / 0x1000))]
556 if data then
557 return data[codepoint]
558 end
559 end
560
561 return p