Main Page » Mailing List » February 2011

Different default encodings based on output param's media-type

Monday, 14 February 2011 · 9 messages · last reply 15 Feb 2011
CG
Christian Grün14 Feb 2011 17:19 #
I couldn't reproduce this behavior via curl. How did you test the requests? Maybe your browser is performing some additional conversions. Christian
Quoted text · software developer · 37 lines
On Mon, Feb 14, 2011 at 4:43 PM, software developer <computer.software.developer at gmail.com> wrote: > Hi, > > I have noted that: > http://localhost:8984/basex/jax-rx?wrap=no&query=%3Cdata%3E%C3%A4%3C/data%3E&output=media-type=text%2Fxml&output=encoding=utf-8 > leads to > <data>ä</data> > > http://localhost:8984/basex/jax-rx?wrap=no&query=%3Cdata%3E%C3%A4%3C/data%3E&output=media-type=text%2Fxml&output=encoding=iso-8859-1 > leads to > > <data>�</data> > > http://localhost:8984/basex/jax-rx?wrap=no&query=%3Cdata%3E%C3%A4%3C/data%3E&output=media-type=text%2Fplain&output=encoding=iso-8859-1 > leads to > > <data>ä</data> > > > http://localhost:8984/basex/jax-rx?wrap=no&query=%3Cdata%3E%C3%A4%3C/data%3E&output=media-type=text%2Fplain&output=encoding=utf-8 > leads to > > <data>ä</data> > > Does this mean that for text/plain, the query submitted is assumed to be in > iso-8859-1 and for text/xml, it is assumed to be in utf-8? If so, why > different treatment? > > Thanks. > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
SD
software developer15 Feb 2011 10:31 #
Yes, you are right. It was a red-herring. I had tested using Firefox, which did its own interpretation of received data. I did a wireshark capture and for the first and the last query, the hex dump shows c3 a4, which is utf-8: http://www.ltg.ed.ac.uk/~richard/utf-8.cgi?input=%E4&mode=char For the other queries (2nd and 3rd), the hex dump shows e4, which is iso-8859-1: http://www.neuroinformatik.ruhr-uni-bochum.de/PEOPLE/rolf/iso_table.html Hex dumps (wireshark captures) show the same characters for both Firefox invocation and curl invocation (but curl rendering is different from that of Firefox). For the 4th query above, firefox shows strange characters because it interprets utf-8 as Latin-1 in that case; it could be happening because of one of the plugins I have or it might be a subtle bug - the outgoing request says firefox will accept both iso-8859-1 and utf-8 charsets and then firefox assumes the response is in utf-8 if the media-type is text/xml and in iso-8859-1 if the media-type is text/plain. Thanks for your inputs.
SD
software developer15 Feb 2011 10:39 #
Actually, thinking about it, although it might be happening due to one of the plugins/assumptions in firefox in case explicit preferences are not available in the response headers; I believe basex can help the situation by setting the charset in the response. Well, firefox or any other client will have to assume things if not available explicitly and assumptions in such a situation are bound to fail in one case or the other, so it does help if the response specifies the charset. In particular, I do not think Firefox is to blame for the rendering in 4th query above, because the URL below states that by default assume iso-8859-1 encoding... http://www.w3.org/International/O-HTTP-charset
CG
Christian Grün15 Feb 2011 12:36 #
Thanks for the hint. If you have in mind how the character specification could be realized the easiest way, feel free to tell us (otherwise, it might take a while…) Christian
Quoted text · software developer · 18 lines
On Tue, Feb 15, 2011 at 10:39 AM, software developer <computer.software.developer at gmail.com> wrote: > Actually, thinking about it, although it might be happening due to one > of the plugins/assumptions in firefox in case explicit preferences are > not available in the response headers; I believe basex can help the > situation by setting the charset in the response. > > Well, firefox or any other client will have to assume things if not > available explicitly and assumptions in such a situation are bound to > fail in one case or the other, so it does help if the response > specifies the charset. > > In particular, I do not think Firefox is to blame for the rendering in > 4th query above, because the URL below states that by default assume > iso-8859-1 encoding... > > http://www.w3.org/International/O-HTTP-charset >
SD
software developer15 Feb 2011 13:13 #
It should be set by jax-rx server depending upon the value of the output param's encoding key. For example, if the jax-rx server receives a query like http://localhost:8984/basex/jax-rx?wrap=no&query=%3Cdata%3E%C3%A4%3C/data%3E&output=media-type=text%2Fxml&output=encoding=utf-8, then it should set the Content-Type header to include the charset value as utf-8, as in "Content-Type: text/xml; charset=utf-8" Please note that jax-rx server already sets the Content-Type header according to the value of media-type key (or some other keys, as detailed in the documentation at http://docs.basex.org/wiki/JAX-RX_API under heading "Response Media Type"), so it perhaps will not be hard to augment that and also take into account if the user specified a desired encoding via output param (e.g., output=encoding=utf-8). Please note that jax-rx/basex already actually encodes the response content in line with the user specified encoding (via output param), so it is just a matter of declaring (by augmenting the Content-Type header as stated above) that the content has been encoded using the encoding specified by the user. Thanks.
Quoted text · Christian Grün · 7 lines
On Tue, Feb 15, 2011 at 11:36 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Thanks for the hint. If you have in mind how the character > specification could be realized the easiest way, feel free to tell us > (otherwise, it might take a while…) > > Christian
CG
Christian Grün15 Feb 2011 13:18 #
Thanks; I've added your suggestions to the JAX-RX issue tracker. Christian
Quoted text · software developer · 31 lines
On Tue, Feb 15, 2011 at 1:13 PM, software developer <computer.software.developer at gmail.com> wrote: > It should be set by jax-rx server depending upon the value of the > output param's encoding key. > For example, if the jax-rx server receives a query like > http://localhost:8984/basex/jax-rx?wrap=no&query=%3Cdata%3E%C3%A4%3C/data%3E&output=media-type=text%2Fxml&output=encoding=utf-8, > then it should set the Content-Type header to include the charset > value as utf-8, as in "Content-Type: text/xml; charset=utf-8" > > Please note that jax-rx server already sets the Content-Type header > according to the value of media-type key (or some other keys, as > detailed in the documentation at http://docs.basex.org/wiki/JAX-RX_API > under heading "Response Media Type"), so it perhaps will not be hard > to augment that and also take into account if the user specified a > desired encoding via output param (e.g., output=encoding=utf-8). > Please note that jax-rx/basex already actually encodes the response > content in line with the user specified encoding (via output param), > so it is just a matter of declaring (by augmenting the Content-Type > header as stated above) that the content has been encoded using the > encoding specified by the user. > > Thanks. > > On Tue, Feb 15, 2011 at 11:36 AM, Christian Grün > <christian.gruen at gmail.com> wrote: >> Thanks for the hint. If you have in mind how the character >> specification could be realized the easiest way, feel free to tell us >> (otherwise, it might take a while…) >> >> Christian >
SD
software developerin reply to software developer, 15 Feb 2011, 13:1315 Feb 2011 13:19 #
Further to the email below, in case there are established rules in jax-rx regarding determining content encoding of the response content, then the Content-Type header should be augmented to make those rules explicit to the client which made the request. The goal is to be able to specify (via Content-Type header) the encoding of the returned content in all applicable cases where it is possible to do so.
JC
Jan Vlčinský (CAD)15 Feb 2011 13:22 #
Hi What about using standard header in the request Accept-Charset? http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.2 Jan 2011/2/15 software developer <computer.software.developer at gmail.com>
Quoted text · software developer · 12 lines
> Further to the email below, in case there are established rules in > jax-rx regarding determining content encoding of the response content, > then the Content-Type header should be augmented to make those rules > explicit to the client which made the request. > The goal is to be able to specify (via Content-Type header) the > encoding of the returned content in all applicable cases where it is > possible to do so. > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
-- *Ing. Jan Vlčinský* CAD programy Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic tel: +420-597 602 024; mob: +420-608 979 040 skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com http://cz.linkedin.com/in/vlcinsky

Written for BaseX 6. BaseX may have changed since: please check the current documentation.

« Probably a bug: replace value of nodeRelease of BaseX 6.5.1 »
⚡Generated with XQuery