0xA3 0xA0 in GB 18030 #338
Comments
xfq
added
i18n-tracker
|
Some preliminary analysis: Usage statistics of GB 18030, GBK, and GB 2312 for websitesSee:
Bing search engine resultsSearching [] (U+E5E5) in Bing China, most of the web pages were in 200x. Some websites may quote text from other websites, resulting in some recent results:
Exampleshttps://www.chinanews.com.cn/n/2004-01-13/26/391173.html (GB 2312) 
This website uses both U+3000 and U+E5E5. https://blog.sina.com.cn/s/blog_44c67f2c0102v4at.html (UTF-8)
This website uses both U+3000 and U+E5E5. https://news.sina.com.cn/c/2004-01-21/09272687351.shtml (GB 2312) 
This website only uses U+3000. https://edu.sina.com.cn/focus/wq3/index.html (GB2312, date 2008) 
Although it's a U+E5E5 result, the source code only contains U+3000. |
We've had the current behavior for years and years. What practical problem does the current state of things cause? Changing it would affect the relationship of GBK and GB18030 in the Encoding Standard, right? |
Because GB 18030 is a compulsory standard, according to Article 14 of CHAPTER III of the Standardization Law of the People's Republic of China:
IANAL, but non-conformance to GB 18030 could be seen as a risk. |
|
We already diverged from GB18030-2022 spec because of the non-round trip mapping proposed by UTC. There is no point in changing 0xA3 0xA0 mapping from the spec conformance perspective. |
|
What supports your claim about ICU? Did https://unicode-org.atlassian.net/browse/ICU-22420 get reverted? I created web-platform-tests/wpt#49137 to ensure we test this code point. I haven't seen any credible argument in this thread to change this mapping so I'm inclined to close this. |
Sorry, I'm not using the latest version of ICU, you are right. I updated the description above. I think before closing this issue, at least we should analyze the impact of updating and not updating the mapping. |
|
I don't think you have convinced anyone thus far that we collectively need to do that. You are certainly welcome to perform such an analysis of course, but given all browsers agree it's extremely unlikely that making a change here would improve anything. |
|
I was actioned by I18N with responding to this issue, which we discussed in our 2024-11-21 call. As near as I can tell, the code unit sequence 0xA3 0xA0 is not actually assigned in GB18030. A look at CJKV Information Processing suggests that the code space for two-byte sequences does not use the bytes 0xA0 and 0xFF. I do not have a copy of GB18030 handy to look at myself and don't have any direct experience implementing this encoding. My local encoders (Oracle JVM 23.0.1 and ICU4J v76.1) produce U+E5E5 for this byte sequence. The reverse (encoding U+E5E5 to GB18030-2022) produces 0xA3 0xA0. I reproduce my code below, in case this is useful. I did not test ICU4C. I've written to Ken Lunde to ask his advice. I do think U+3000 is a tiny bit weird, although the logical character before I do not think that this represents a critical problem, since no data should exist in GB18303 that uses this byte sequence for anything meaningful. Replacing the sequence with one character or another should produce no meaningful difference, unless I'm not understanding something. But past experience with sequences in a legacy encoding producing different results in different coders have generally been that this becomes a problem at a later date. In this case, I don't think any graphical character will ever be assigned to this specific sequence, so it probably makes no difference. My code: public static void encoding338() {
try {
Charset gb18030 = Charset.forName("GB18030-2022");
CharsetDecoder decoder = gb18030.newDecoder();
ByteBuffer bb = ByteBuffer.wrap(new byte[] { (byte) 0xA3, (byte) 0xA0 });
CharBuffer cb = decoder.decode(bb);
System.out.println(Util.native2ascii(cb.toString())); // not standard code but does what you think it does
Charset icu = CharsetICU.forNameICU("GB18030-2022");
decoder = gb18030.newDecoder();
bb = ByteBuffer.wrap(new byte[] { (byte) 0xA3, (byte) 0xA0 });
cb = decoder.decode(bb);
System.out.println(Util.native2ascii(cb.toString()));
ByteBuffer out = gb18030.encode(cb);
byte[] bytes = out.array();
for (byte b : bytes) {
System.out.print(Integer.toHexString((int) (b &0xFF)));
System.out.print(' ');
}
System.out.println();
cb.rewind();
out = icu.encode(cb);
bytes = out.array();
for (byte b : bytes) {
System.out.print(Integer.toHexString((int) (b &0xFF)));
System.out.print(' ');
}
} catch (Throwable t) {
t.printStackTrace();
}
}Produces: |
|
Although I don't agree with OP's request, I have to correct the factual error.
Yes, the code unit sequence 0xA3 0xA0 is assigned in GB18030.
Why don't you read the spec itself instead of a secondary source?
🤔 Here is a quote from the GB18030-2022 spec. From the spec compliance perspective, 0xA3 0xA0 must be mapped to U+E5E5, period. We are intentionally violating the spec for web-compat.
Even one character mapping change makes GB18030 not a UTF. It breaks round-trip conversion.
Yes, it makes a difference. U+E5E5 will render a white box (on Chromium) or a hexbox (on Firefox) and it will be a visual glitch if the page author intended to use an IDEPGRAPHIC SPACE (U+3000). This is the very reason we violated the spec. |
What is the issue with the Encoding Standard?
https://encoding.spec.whatwg.org/#gb18030-encoder
We didn't update this in #336 , so I filed this issue to track it.
https://bugzilla.mozilla.org/show_bug.cgi?id=131837 , a bug filed in 2002 mentioned this. The reason behind this mapping was that some websites use 0xA3 0xA0 as space characters, which causes display abnormalities, so Mozilla changed the mapping to
U+3000 IDEOGRAPHIC SPACE.In the Hong Kong Supplementary Character Set, U+E5E5 was used to encode
𨪜(U+2A89Cin Unicode CJK Unified Ideographs Extension I).We need to analyze how many websites using GB 18030 are still using 0xA3 0xA0 to represent U+3000.
Currently, iconv
and ICUseems to map 0xA3 0xA0 to U+E5E5. ICU 74.1+ maps it to U+3000.The following is some information about this misuse (mostly translated from a Chinese website).
The 0xA3A1 ~ 0xA3FE part of GB18030-2022 is inherited from row 3 of GB 2312, and contains the G0 set of GB/T 1988-80 (ISO 646-CN). GB 2312 does not specify the width of these characters, but subsequent standards (such as GB 5007.1-85) made it clear that characters in row 3 are full-width, which are mapped to the Halfwidth and Fullwidth Forms Unicode block.
However, the G0 set of GB/T 1988-80 does not include spaces, but influenced by ASCII, people often consider spaces together with the remaining 94 characters. Now let's assume that someone thinks that 0xA3A1 ~ 0xA3FE are full-width ASCII characters (although "＄" has been replaced by "￥"), then this person is likely to think that 0xA3 0xA0 should be a full-width space (although the actual full-width space is at 0xA1A1). Because some fonts display .notdef as a 1 em wide space, even when the corresponding Unicode code point of the two are different, the rendering is the same (undefined PUA code points in GB encoding will be displayed as .notdef).
The text was updated successfully, but these errors were encountered: