Main Page » Mailing List » June 2012

normalize-unicode function

Thursday, 28 June 2012 · 6 messages · last reply 2 Jul 2012

Toshio HIRAI28 Jun 2012 04:41 #

Hi, I tried some Japanese letters in fn:normalize-unicode. FULLWIDTH DIGIT, FULLWIDTH LATIN CAPITAL LETTER: fn:normalize-unicode("１２３４５６７８９０ａｂｃｄｅｆｇ", "NFKC") returns 1234567890abcdefg FULLWIDTH EXCLAMATION MARK: fn:normalize-unicode("！", "NFKC") returns ! FULLWIDTH LESS-THAN SIGN, ?FULLWIDTH GREATER-THAN SIGN:

  fn:normalize-unicode("＜＞", "NFKC")
  returns &lt;&gt;

FULLWIDTH AMPERSAND;
  fn:normalize-unicode("＆", "NFKC")
  returns?&amp;

These are normal. but, FULLWIDTH QUOTATION MARK, FULLWIDTH APOSTROPHE: fn:normalize-unicode("”’", "NFKC") returns ”’ In this case, I think that "' should be returned. Best regards, Toshio HIRAI

Christian Grün28 Jun 2012 14:08 #

Toshio san, thanks for your e-mail.

Quoted text · Toshio HIRAI · 3 lines

> FULLWIDTH QUOTATION MARK, FULLWIDTH APOSTROPHE: > fn:normalize-unicode("”’", "NFKC") > returns ”’

As far as I can judge, the result is actually correct; it’s returned by Java’s standard Unicode algorithms, and also returned by other XQuery (Saxon, Zorba, XMLPrime, etc). I may need to do more research on how to normalize quotes the way you’d like them to have, though. Hope this helps (at least a little), Christian

Leonard Wörtelerin reply to Toshio HIRAI, 28 Jun 2012, 04:4128 Jun 2012 15:00 #

Dear Toshio,

Quoted text · Toshio HIRAI · 6 lines

Am 28.06.2012 04:41, schrieb Toshio HIRAI: > FULLWIDTH QUOTATION MARK, FULLWIDTH APOSTROPHE: > fn:normalize-unicode("”’", "NFKC") > returns ”’ > > In this case, I think that "' should be returned.

this is just a different representation for the same characters, BaseX only escapes quotes when necessary (e.g. inside attributes). this query for example returns true():

Quoted text · Toshio HIRAI · 1 line

> normalize-unicode('＂＇', 'NFKC') eq '"''

The first two character entities are *FULLWIDTH QUOTATION MARK* [1] and *FULLWIDTH APOSTROPHE* [2]. Hope that helps, cheers, Leo [1] http://www.fileformat.info/info/unicode/char/ff02/index.htm [2] http://www.fileformat.info/info/unicode/char/ff07/index.htm

Toshio HIRAI29 Jun 2012 04:59 #

Hi Leo, Thanks for your help, And I understood that, The thing which believed that I was 'FULL WIDTH QUOTATION MARK' so far was 'RIGHT DOUBLE QUOTATION MARK'. 'RIGHT DOUBLE QUOTATION MARK' (U+201D): [1] 'RIGHT SINGLE QUOTATION MARK' (U+2019): [2] However, It is *RIGHT DOUBLE QUOTATION MARK* to be input when a Japanese inputs with a Japanese keyboard. normalize-unicode('＂＇', 'NFKC') eq '"'' returns true Yes, I understood. But, If it is Japanese Griff, normalize-unicode('”’', 'NFKC') eq '"'' returns false This sample is the same as that of the following. normalize-unicode('”’', 'NFKC') eq '"'' returns false Although before was afflicted, This seems to UNICODE mapping problem in Java (in Japanese). Thing that I want to do is to normalize, such as symbols and numbers that appear in the context of Japanese. However, It is unavoidable when it is a problem peculiar only to Japanese, I could not but exceed by the following solution. normalize-unicode(fn:translate('”’', '”’' , '＂＇'), "NFKC") eq '"'' returns true There will be a good way or something else? Thanks, Toshio [1] http://www.fileformat.info/info/unicode/char/201d/index.htm [2] http://www.fileformat.info/info/unicode/char/2019/index.htm 2012/6/28 Leonard Wörteler <leonard.woerteler at uni-konstanz.de>:

Quoted text · Leonard Wörteler · 23 lines

> Dear Toshio, > > Am 28.06.2012 04:41, schrieb Toshio HIRAI: >> FULLWIDTH QUOTATION MARK, FULLWIDTH APOSTROPHE: >> fn:normalize-unicode("”’", "NFKC") >> returns ”’ >> >> In this case, I think that "' should be returned. > > this is just a different representation for the same characters, BaseX > only escapes quotes when necessary (e.g. inside attributes). this query > for example returns `true()`: > >> normalize-unicode('＂＇', 'NFKC') eq '"'' > > The first two character entities are *FULLWIDTH QUOTATION MARK* [1] and > *FULLWIDTH APOSTROPHE* [2]. > > Hope that helps, > cheers, Leo > > [1] http://www.fileformat.info/info/unicode/char/ff02/index.htm > [2] http://www.fileformat.info/info/unicode/char/ff07/index.htm

Toshio HIRAI29 Jun 2012 12:07 #

Now noticed, That part in the Japanese tokenizer had been hard-coded... https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/util/ft/JapaneseTokenizer.java#L395 2012/6/29 Toshio HIRAI <toshio.hirai at gmail.com>:

Quoted text · Toshio HIRAI · 73 lines

> Hi Leo, > > Thanks for your help, > And I understood that, The thing which believed that I was 'FULL WIDTH > QUOTATION MARK' so far was 'RIGHT DOUBLE QUOTATION MARK'. > > 'RIGHT DOUBLE QUOTATION MARK' (U+201D): [1] > 'RIGHT SINGLE QUOTATION MARK' (U+2019): [2] > > However, It is *RIGHT DOUBLE QUOTATION MARK* to be input when a > Japanese inputs with a Japanese keyboard. > > normalize-unicode('＂＇', 'NFKC') eq '"'' > returns true > > Yes, I understood. > But, If it is Japanese Griff, > > normalize-unicode('”’', 'NFKC') eq '"'' > returns false > > This sample is the same as that of the following. > > normalize-unicode('”’', 'NFKC') eq '"'' > returns false > > Although before was afflicted, This seems to UNICODE mapping problem > in Java (in Japanese). > > Thing that I want to do is to normalize, such as symbols and numbers > that appear in the context of Japanese. > However, It is unavoidable when it is a problem peculiar only to > Japanese, I could not but exceed by the following solution. > > normalize-unicode(fn:translate('”’', '”’' , > '＂＇'), "NFKC") eq '"'' > returns true > > There will be a good way or something else? > > Thanks, > Toshio > > [1] http://www.fileformat.info/info/unicode/char/201d/index.htm > [2] http://www.fileformat.info/info/unicode/char/2019/index.htm > > > > > 2012/6/28 Leonard Wörteler <leonard.woerteler at uni-konstanz.de>: >> Dear Toshio, >> >> Am 28.06.2012 04:41, schrieb Toshio HIRAI: >>> FULLWIDTH QUOTATION MARK, FULLWIDTH APOSTROPHE: >>> fn:normalize-unicode("”’", "NFKC") >>> returns ”’ >>> >>> In this case, I think that "' should be returned. >> >> this is just a different representation for the same characters, BaseX >> only escapes quotes when necessary (e.g. inside attributes). this query >> for example returns `true()`: >> >>> normalize-unicode('＂＇', 'NFKC') eq '"'' >> >> The first two character entities are *FULLWIDTH QUOTATION MARK* [1] and >> *FULLWIDTH APOSTROPHE* [2]. >> >> Hope that helps, >> cheers, Leo >> >> [1] http://www.fileformat.info/info/unicode/char/ff02/index.htm >> [2] http://www.fileformat.info/info/unicode/char/ff07/index.htm

Toshio HIRAIin reply to Christian Grün, 28 Jun 2012, 14:082 Jul 2012 12:52 #

Hi Christian, Thanks for your help. I confirmed it about the relevant part of a program. And, I confirmed that java.text.Normalizer.normalize() returned it without converting a full-width double quotation mark that we usually used (U+201D). At present, As my hope, When U+201C, U+201D, returns U+0022. When U+2018, U+2019, returns U+0027. However, these will not be enough for users of the whole world (U+201A, U+201E... Other punctuation mark... etc...). Furthermore, I cannot be convinced whether you should implement it its changing the standard specifications of the Java. Best regards, Toshio 2012/6/28 Christian Grün <christian.gruen at gmail.com>:

Quoted text · Christian Grün · 15 lines

> Toshio san, > > thanks for your e-mail. > >> FULLWIDTH QUOTATION MARK, FULLWIDTH APOSTROPHE: >> fn:normalize-unicode("”’", "NFKC") >> returns ”’ > > As far as I can judge, the result is actually correct; it’s returned > by Java’s standard Unicode algorithms, and also returned by other > XQuery (Saxon, Zorba, XMLPrime, etc). I may need to do more research > on how to normalize quotes the way you’d like them to have, though. > > Hope this helps (at least a little), > Christian

Written for BaseX 7.3. BaseX may have changed since: please check the current documentation.

⚡Generated with XQuery

`J`	Select the next message
`K`	Select the previous message
`N`	Next thread
`P`	Previous thread
`U`	Go up one level
`/`	Jump to the search field