Encoding hassle ....
Friday, 27 May 2022 · 6 messages · last reply 13 Jun 2022
ML
Marco Lettere27 May 2022 19:00 #
Dear all,
after wrapping our heads around this for hours today, we don't know how to get rid of this inconsistency. Thus I ask for help ...
SSCE:
BaseX 9.6.4 [Standalone] Try 'help' to get more information.
Quoted text · 1 line
> xquery file:write-text("a1.txt", "°" || out:nl()) (: Same with
codepoints-to-string(176) instead of "°" :)
Query executed in 183.94 ms.
Quoted text · 1 line
> xquery file:read-text("a1.txt")
°
Query executed in 1.49 ms.
Quoted text · 1 line
> xquery file:write-text("a2.txt", file:read-text("a1.txt"))
Query executed in 3.4 ms.
Quoted text · 1 line
> xquery file:read-text("a2.txt")
[file:io-error] Decoding error: xb0
Testing the files with linux command-line tool "file", this is the output:
Quoted text · 1 line
> file a1.txt
a1.txt: Unicode text, UTF-8 text
Quoted text · 1 line
> file a2.txt
a2.txt: ISO-8859 text
Reading the file after "copying" it seems to change the encoding. How is this supposed to be handled?
Regards,
Marco.
CG
Definitely looks like a bug. I’m currently on the road, but I’ll get to the bottom of this once I’m back.
Bridger Dyson-Smith <bdysonsmith at gmail.com> schrieb am Fr., 27. Mai 2022, 19:27:
Quoted text · Bridger Dyson-Smith · 72 lines
> Marco - > I'm sorry but I can only corroborate your findings, and that trying to > force UTF-8 by adding the encoding parameter to the functions doesn't seem > to help; e.g. > > ) ./bin/basex > BaseX 9.7.1 [Standalone] > Try 'help' to get more information. > > xquery file:current-dir() > /usr/home/bridger/bin/basex/ > Query executed in 886.62 ms. > > xquery file:write-text("a1.txt", "°" || out:nl(), "UTF-8") > > Query executed in 4.32 ms. > > xquery file:read-text("a1.txt") > ° > > Query executed in 1.99 ms. > > xquery file:write-text("a2.txt", file:read-text("a1.txt", "UTF-8"), > "UTF-8") > > Query executed in 1.83 ms. > > xquery file:read-text("a2.txt") > [file:io-error] Decoding error: xb0 > > xquery file:read-text("a2.txt", "UTF-8") > [file:io-error] Decoding error: xb0 > > xquery file:read-text("a2.txt", "ISO-8859-1") > ° > > Query executed in 2.01 ms. > > On Fri, May 27, 2022 at 1:00 PM Marco Lettere <m.lettere at gmail.com> wrote: > >> Dear all, >> >> after wrapping our heads around this for hours today, we don't know how >> to get rid of this inconsistency. Thus I ask for help ... >> >> SSCE: >> >> BaseX 9.6.4 [Standalone] >> Try 'help' to get more information. >> > xquery file:write-text("a1.txt", "°" || out:nl()) (: Same with >> codepoints-to-string(176) instead of "°" :) >> >> Query executed in 183.94 ms. >> > xquery file:read-text("a1.txt") >> ° >> >> Query executed in 1.49 ms. >> > xquery file:write-text("a2.txt", file:read-text("a1.txt")) >> Query executed in 3.4 ms. >> >> > xquery file:read-text("a2.txt") >> [file:io-error] Decoding error: xb0 >> >> Testing the files with linux command-line tool "file", this is the output: >> >> > file a1.txt >> a1.txt: Unicode text, UTF-8 text >> >> > file a2.txt >> a2.txt: ISO-8859 text >> >> Reading the file after "copying" it seems to change the encoding. How is >> this supposed to be handled? >> >> Regards, >> >> Marco. >> >>
BD
Marco - I'm sorry but I can only corroborate your findings, and that trying to force UTF-8 by adding the encoding parameter to the functions doesn't seem to help; e.g.
) ./bin/basex BaseX 9.7.1 [Standalone] Try 'help' to get more information.
Quoted text · Marco Lettere · 1 line
> xquery file:current-dir()
/usr/home/bridger/bin/basex/ Query executed in 886.62 ms.
Quoted text · Marco Lettere · 1 line
> xquery file:write-text("a1.txt", "°" || out:nl(), "UTF-8")
Query executed in 4.32 ms.
Quoted text · Marco Lettere · 1 line
> xquery file:read-text("a1.txt")
°
Query executed in 1.99 ms.
Quoted text · Marco Lettere · 1 line
> xquery file:write-text("a2.txt", file:read-text("a1.txt", "UTF-8"),
"UTF-8")
Query executed in 1.83 ms.
Quoted text · Marco Lettere · 5 lines
> xquery file:read-text("a2.txt") [file:io-error] Decoding error: xb0 > xquery file:read-text("a2.txt", "UTF-8") [file:io-error] Decoding error: xb0 > xquery file:read-text("a2.txt", "ISO-8859-1")
°
Query executed in 2.01 ms.
Quoted text · Marco Lettere · 40 lines
On Fri, May 27, 2022 at 1:00 PM Marco Lettere <m.lettere at gmail.com> wrote: > Dear all, > > after wrapping our heads around this for hours today, we don't know how > to get rid of this inconsistency. Thus I ask for help ... > > SSCE: > > BaseX 9.6.4 [Standalone] > Try 'help' to get more information. > > xquery file:write-text("a1.txt", "°" || out:nl()) (: Same with > codepoints-to-string(176) instead of "°" :) > > Query executed in 183.94 ms. > > xquery file:read-text("a1.txt") > ° > > Query executed in 1.49 ms. > > xquery file:write-text("a2.txt", file:read-text("a1.txt")) > Query executed in 3.4 ms. > > > xquery file:read-text("a2.txt") > [file:io-error] Decoding error: xb0 > > Testing the files with linux command-line tool "file", this is the output: > > > file a1.txt > a1.txt: Unicode text, UTF-8 text > > > file a2.txt > a2.txt: ISO-8859 text > > Reading the file after "copying" it seems to change the encoding. How is > this supposed to be handled? > > Regards, > > Marco. > >
ML
Marco Lettere27 May 2022 23:40 #
Oh yes thanks. Forgot to mention this. Forcing utf8 doesn't help.
Il ven 27 mag 2022, 19:26 Bridger Dyson-Smith <bdysonsmith at gmail.com> ha scritto:
Quoted text · Bridger Dyson-Smith · 72 lines
> Marco - > I'm sorry but I can only corroborate your findings, and that trying to > force UTF-8 by adding the encoding parameter to the functions doesn't seem > to help; e.g. > > ) ./bin/basex > BaseX 9.7.1 [Standalone] > Try 'help' to get more information. > > xquery file:current-dir() > /usr/home/bridger/bin/basex/ > Query executed in 886.62 ms. > > xquery file:write-text("a1.txt", "°" || out:nl(), "UTF-8") > > Query executed in 4.32 ms. > > xquery file:read-text("a1.txt") > ° > > Query executed in 1.99 ms. > > xquery file:write-text("a2.txt", file:read-text("a1.txt", "UTF-8"), > "UTF-8") > > Query executed in 1.83 ms. > > xquery file:read-text("a2.txt") > [file:io-error] Decoding error: xb0 > > xquery file:read-text("a2.txt", "UTF-8") > [file:io-error] Decoding error: xb0 > > xquery file:read-text("a2.txt", "ISO-8859-1") > ° > > Query executed in 2.01 ms. > > On Fri, May 27, 2022 at 1:00 PM Marco Lettere <m.lettere at gmail.com> wrote: > >> Dear all, >> >> after wrapping our heads around this for hours today, we don't know how >> to get rid of this inconsistency. Thus I ask for help ... >> >> SSCE: >> >> BaseX 9.6.4 [Standalone] >> Try 'help' to get more information. >> > xquery file:write-text("a1.txt", "°" || out:nl()) (: Same with >> codepoints-to-string(176) instead of "°" :) >> >> Query executed in 183.94 ms. >> > xquery file:read-text("a1.txt") >> ° >> >> Query executed in 1.49 ms. >> > xquery file:write-text("a2.txt", file:read-text("a1.txt")) >> Query executed in 3.4 ms. >> >> > xquery file:read-text("a2.txt") >> [file:io-error] Decoding error: xb0 >> >> Testing the files with linux command-line tool "file", this is the output: >> >> > file a1.txt >> a1.txt: Unicode text, UTF-8 text >> >> > file a2.txt >> a2.txt: ISO-8859 text >> >> Reading the file after "copying" it seems to change the encoding. How is >> this supposed to be handled? >> >> Regards, >> >> Marco. >> >>
CG
Christian Grün13 Jun 2022 16:45 #
Hi Marco,
If the content of a file is written to another file without intermediate steps, it is streamed and consumes constant memory. The implementation for streaming the data was deficient.
The bug has been fixed; a new snapshot is available [1,2].
Grazie e ciao, Christian
[1] https://github.com/BaseXdb/basex/issues/2117 [2] https://files.basex.org/releases/latest/
Quoted text · Marco Lettere · 74 lines
On Fri, May 27, 2022 at 11:40 PM Marco Lettere <m.lettere at gmail.com> wrote: > > Oh yes thanks. Forgot to mention this. Forcing utf8 doesn't help. > > Il ven 27 mag 2022, 19:26 Bridger Dyson-Smith <bdysonsmith at gmail.com> ha scritto: >> >> Marco - >> I'm sorry but I can only corroborate your findings, and that trying to force UTF-8 by adding the encoding parameter to the functions doesn't seem to help; e.g. >> >> ) ./bin/basex >> BaseX 9.7.1 [Standalone] >> Try 'help' to get more information. >> > xquery file:current-dir() >> /usr/home/bridger/bin/basex/ >> Query executed in 886.62 ms. >> > xquery file:write-text("a1.txt", "°" || out:nl(), "UTF-8") >> >> Query executed in 4.32 ms. >> > xquery file:read-text("a1.txt") >> ° >> >> Query executed in 1.99 ms. >> > xquery file:write-text("a2.txt", file:read-text("a1.txt", "UTF-8"), "UTF-8") >> >> Query executed in 1.83 ms. >> > xquery file:read-text("a2.txt") >> [file:io-error] Decoding error: xb0 >> > xquery file:read-text("a2.txt", "UTF-8") >> [file:io-error] Decoding error: xb0 >> > xquery file:read-text("a2.txt", "ISO-8859-1") >> ° >> >> Query executed in 2.01 ms. >> >> On Fri, May 27, 2022 at 1:00 PM Marco Lettere <m.lettere at gmail.com> wrote: >>> >>> Dear all, >>> >>> after wrapping our heads around this for hours today, we don't know how >>> to get rid of this inconsistency. Thus I ask for help ... >>> >>> SSCE: >>> >>> BaseX 9.6.4 [Standalone] >>> Try 'help' to get more information. >>> > xquery file:write-text("a1.txt", "°" || out:nl()) (: Same with >>> codepoints-to-string(176) instead of "°" :) >>> >>> Query executed in 183.94 ms. >>> > xquery file:read-text("a1.txt") >>> ° >>> >>> Query executed in 1.49 ms. >>> > xquery file:write-text("a2.txt", file:read-text("a1.txt")) >>> Query executed in 3.4 ms. >>> >>> > xquery file:read-text("a2.txt") >>> [file:io-error] Decoding error: xb0 >>> >>> Testing the files with linux command-line tool "file", this is the output: >>> >>> > file a1.txt >>> a1.txt: Unicode text, UTF-8 text >>> >>> > file a2.txt >>> a2.txt: ISO-8859 text >>> >>> Reading the file after "copying" it seems to change the encoding. How is >>> this supposed to be handled? >>> >>> Regards, >>> >>> Marco. >>>
ML
Marco Lettere13 Jun 2022 16:49 #
Thank you very much.
We'll give it a try ASAP.
M.
Quoted text · Christian Grün · 4 lines
On 13/06/22 16:45, Christian Grün wrote: > If the content of a file is written to another file without > intermediate steps, it is streamed and consumes constant memory. The > implementation for streaming the data was deficient.
Written for BaseX 9.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery