Nonstandard CSV in RESTXQ

On Sun, Jun 12, 2016 at 4:03 PM, Alexander Shpack <shadowkin at gmail.com> wrote: > Hi, > > I have nonstandard CSV file with semicolon instead of comma as a separator > and cp1251 encoded. Documentation is not clear in this case, and the sample > doesn't work properly > > declare > %rest:path("/store.csv") > %rest:POST("{$csv}") > %input:csv("header=true") > function page:store-csv($csv as document-node()) > { > "Number of rows: " || count($csv/csv/record) > }; > > Even if I changed the encoding manually the next code does nothing > %input:csv("separator=semicolon") > and returns text instead of document-node() > > Could you check it? > > It would be nice if you add additional encoding parameter to csv module. > > Thanks!

Am 12.06.2016 8:05 nachm. schrieb "Alexander Shpack" <shadowkin at gmail.com>: > Thank you, Christian! > > But it doesn't work as expected > > declare > %rest:path("/upload") > %rest:POST("{$csv}") > %input:csv("header=false,encoding=CP1251,separator=semicolon") > %output:method("html") > function page:uploadFile($csv as document-node()) > { > $csv > }; > > > [XPTY0004] Cannot cast xs:string to document-node(): "?????????? ?.... > > > On Sun, Jun 12, 2016 at 6:48 PM, Christian Grün <christian.gruen at gmail.com > > wrote: > >> Hi Alex, >> >> The following annotation should do what you want: >> >> %input:csv("header=true,encoding=CP1252") >> >> You were completely right, the existing documentation was a bit >> scrappy. I have slightly rewritten it (see [1] and outgoing links). >> >> Hope thie helps, >> Christian >> >> [1] http://docs.basex.org/wiki/RESTXQ#Input_options >> >> >> >> >> On Sun, Jun 12, 2016 at 4:03 PM, Alexander Shpack <shadowkin at gmail.com> >> wrote: >> > Hi, >> > >> > I have nonstandard CSV file with semicolon instead of comma as a >> separator >> > and cp1251 encoded. Documentation is not clear in this case, and the >> sample >> > doesn't work properly >> > >> > declare >> > %rest:path("/store.csv") >> > %rest:POST("{$csv}") >> > %input:csv("header=true") >> > function page:store-csv($csv as document-node()) >> > { >> > "Number of rows: " || count($csv/csv/record) >> > }; >> > >> > Even if I changed the encoding manually the next code does nothing >> > %input:csv("separator=semicolon") >> > and returns text instead of document-node() >> > >> > Could you check it? >> > >> > It would be nice if you add additional encoding parameter to csv module. >> > >> > Thanks! >> > > > > -- > s0rr0w >

On Sun, Jun 12, 2016 at 9:41 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Did you specify "text/comma-separated-values" as Content-Type? > Am 12.06.2016 8:05 nachm. schrieb "Alexander Shpack" <shadowkin at gmail.com > >: > >> Thank you, Christian! >> >> But it doesn't work as expected >> >> declare >> %rest:path("/upload") >> %rest:POST("{$csv}") >> %input:csv("header=false,encoding=CP1251,separator=semicolon") >> %output:method("html") >> function page:uploadFile($csv as document-node()) >> { >> $csv >> }; >> >> >> [XPTY0004] Cannot cast xs:string to document-node(): "?????????? ?.... >> >> >> On Sun, Jun 12, 2016 at 6:48 PM, Christian Grün < >> christian.gruen at gmail.com> wrote: >> >>> Hi Alex, >>> >>> The following annotation should do what you want: >>> >>> %input:csv("header=true,encoding=CP1252") >>> >>> You were completely right, the existing documentation was a bit >>> scrappy. I have slightly rewritten it (see [1] and outgoing links). >>> >>> Hope thie helps, >>> Christian >>> >>> [1] http://docs.basex.org/wiki/RESTXQ#Input_options >>> >>> >>> >>> >>> On Sun, Jun 12, 2016 at 4:03 PM, Alexander Shpack <shadowkin at gmail.com> >>> wrote: >>> > Hi, >>> > >>> > I have nonstandard CSV file with semicolon instead of comma as a >>> separator >>> > and cp1251 encoded. Documentation is not clear in this case, and the >>> sample >>> > doesn't work properly >>> > >>> > declare >>> > %rest:path("/store.csv") >>> > %rest:POST("{$csv}") >>> > %input:csv("header=true") >>> > function page:store-csv($csv as document-node()) >>> > { >>> > "Number of rows: " || count($csv/csv/record) >>> > }; >>> > >>> > Even if I changed the encoding manually the next code does nothing >>> > %input:csv("separator=semicolon") >>> > and returns text instead of document-node() >>> > >>> > Could you check it? >>> > >>> > It would be nice if you add additional encoding parameter to csv >>> module. >>> > >>> > Thanks! >>> >> >> >> >> -- >> s0rr0w >> >

> No, browser doesn't want to send file content with this mime type.

> RFC describes "text/csv" mime type. Are you sure that > "text/comma-separated-values" is the correct one?

On 12 June 2016 at 20:26, Christian Grün <christian.gruen at gmail.com> wrote: > > No, browser doesn't want to send file content with this mime type. > > If you use the browser to send requests, it may be better to convert > your data to CSV via XQuery: > > csv:parse($input, map { > "header": false(), "separator": "semicolon" > }) > > How does your html look like, which you use for sending the request > via the browser? > > "header=false,encoding=CP1251,separator=semicolon" > > > RFC describes "text/csv" mime type. Are you sure that > > "text/comma-separated-values" is the correct one? > > True, text/csv should work as well! >

On Sun, Jun 12, 2016 at 10:26 PM, Christian Grün <christian.gruen at gmail.com> wrote: > > No, browser doesn't want to send file content with this mime type. > > If you use the browser to send requests, it may be better to convert > your data to CSV via XQuery: > > csv:parse($input, map { > "header": false(), "separator": "semicolon" > }) > >

How does your html look like, which you use for sending the request > via the browser? > >

> Not really up to speed with this but I would expect the separator to be the > symbol, no?

> > "header=false,encoding=CP1251,separator=;" > or > map{"header": false(), "separator": ";"} > /Andy > > On 12 June 2016 at 20:26, Christian Grün <christian.gruen at gmail.com> wrote: >> >> > No, browser doesn't want to send file content with this mime type. >> >> If you use the browser to send requests, it may be better to convert >> your data to CSV via XQuery: >> >> csv:parse($input, map { >> "header": false(), "separator": "semicolon" >> }) >> >> How does your html look like, which you use for sending the request >> via the browser? >> >> "header=false,encoding=CP1251,separator=semicolon" >> >> > RFC describes "text/csv" mime type. Are you sure that >> > "text/comma-separated-values" is the correct one? >> >> True, text/csv should work as well! > >

> I've tried it, it works. Almost fine. Now I have an issue with encoding. > Seems like 'encoding' parameter was not applied > > let $csv := csv:parse($csv, map{ 'separator': ';', 'header': false(), > 'encoding': 'CP1251' } )

On Sun, Jun 12, 2016 at 10:50 PM, Christian Grün <christian.gruen at gmail.com> wrote: > > I've tried it, it works. Almost fine. Now I have an issue with encoding. > > Seems like 'encoding' parameter was not applied > > > > let $csv := csv:parse($csv, map{ 'separator': ';', 'header': false(), > > 'encoding': 'CP1251' } ) > > This should work: > > declare > %rest:path("/upload") > %rest:POST("{$body}") > function local:store-csv($body) > { > let $string := convert:binary-to-string($body, 'CP1251') > return csv:parse($string, map{ 'separator': ';', 'header': false(), > }; >

On Sun, Jun 12, 2016 at 9:55 PM, Alexander Shpack <shadowkin at gmail.com> wrote: > No luck with you code > > [XPTY0004] Binary expected, xs:string found. > > > > On Sun, Jun 12, 2016 at 10:50 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> > I've tried it, it works. Almost fine. Now I have an issue with encoding. >> > Seems like 'encoding' parameter was not applied >> > >> > let $csv := csv:parse($csv, map{ 'separator': ';', 'header': false(), >> > 'encoding': 'CP1251' } ) >> >> This should work: >> >> declare >> %rest:path("/upload") >> %rest:POST("{$body}") >> function local:store-csv($body) >> { >> let $string := convert:binary-to-string($body, 'CP1251') >> return csv:parse($string, map{ 'separator': ';', 'header': false(), >> }; > > > > > -- > s0rr0w

> OMG.... > Basex 8.4.4 returns different error for the browsers

> As the error indicates, it seems that your CSV input seems to contain > characters that are not valid in XML. There are various ways to tackle > this; one looks as follows: > > (: interpret client data as Base64 :) > declare > %rest:path("/upload") > %rest:POST("{$body}") > function local:store-csv($body as xs:base64Binary) > { > (: replace invalid characters with a question mark :) > let $input := bin:from-octets( > bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . else > 63) > ) > (: convert to XQuery Unicode string; convert to XML :) > let $string := bin:decode-string($input, 'CP1251') > return csv:parse($string) > }; > > There may be easier solutions as well (I’ll give you an update once I > remember them ;). >

On Sun, Jun 12, 2016 at 11:45 PM, Alexander Shpack <shadowkin at gmail.com> wrote: > Did you always upload the same file? > > Yeah > > >> As the error indicates, it seems that your CSV input seems to contain >> characters that are not valid in XML. There are various ways to tackle >> this; one looks as follows: >> >> (: interpret client data as Base64 :) >> declare >> %rest:path("/upload") >> %rest:POST("{$body}") >> function local:store-csv($body as xs:base64Binary) >> { >> (: replace invalid characters with a question mark :) >> let $input := bin:from-octets( >> bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . else >> 63) >> ) >> (: convert to XQuery Unicode string; convert to XML :) >> let $string := bin:decode-string($input, 'CP1251') >> return csv:parse($string) >> }; >> >> There may be easier solutions as well (I’ll give you an update once I >> remember them ;). >> > > Strange solution... Ok, I'll wait for a better one. > > -- > s0rr0w >

> let $content := file:read-text($fName, "cp1251")

> let $csv := csv:parse($content, map{ 'separator': ';', 'header': false() } ) > return $csv > > What the difference between post data and raw file? > > > On Sun, Jun 12, 2016 at 11:45 PM, Alexander Shpack <shadowkin at gmail.com> > wrote: >> >> Did you always upload the same file? >> >> Yeah >> >>> >>> As the error indicates, it seems that your CSV input seems to contain >>> characters that are not valid in XML. There are various ways to tackle >>> this; one looks as follows: >>> >>> (: interpret client data as Base64 :) >>> declare >>> %rest:path("/upload") >>> %rest:POST("{$body}") >>> function local:store-csv($body as xs:base64Binary) >>> { >>> (: replace invalid characters with a question mark :) >>> let $input := bin:from-octets( >>> bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . else >>> 63) >>> ) >>> (: convert to XQuery Unicode string; convert to XML :) >>> let $string := bin:decode-string($input, 'CP1251') >>> return csv:parse($string) >>> }; >>> >>> There may be easier solutions as well (I’ll give you an update once I >>> remember them ;). >> >> >> Strange solution... Ok, I'll wait for a better one. >> >> -- >> s0rr0w > > > > > -- > s0rr0w

On Mon, Jun 13, 2016 at 12:02 AM, Christian Grün <christian.gruen at gmail.com> wrote: > This line tells you the difference: > > > let $content := file:read-text($fName, "cp1251") > > You are reading the file contents as text (i.e., string). You could > also use file:read-binary. Using the browser, you can also upload > images or other binary contents. > > > > let $csv := csv:parse($content, map{ 'separator': ';', 'header': false() > } ) > > return $csv > > > > What the difference between post data and raw file? > > > > > > On Sun, Jun 12, 2016 at 11:45 PM, Alexander Shpack <shadowkin at gmail.com> > > wrote: > >> > >> Did you always upload the same file? > >> > >> Yeah > >> > >>> > >>> As the error indicates, it seems that your CSV input seems to contain > >>> characters that are not valid in XML. There are various ways to tackle > >>> this; one looks as follows: > >>> > >>> (: interpret client data as Base64 :) > >>> declare > >>> %rest:path("/upload") > >>> %rest:POST("{$body}") > >>> function local:store-csv($body as xs:base64Binary) > >>> { > >>> (: replace invalid characters with a question mark :) > >>> let $input := bin:from-octets( > >>> bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . > else > >>> 63) > >>> ) > >>> (: convert to XQuery Unicode string; convert to XML :) > >>> let $string := bin:decode-string($input, 'CP1251') > >>> return csv:parse($string) > >>> }; > >>> > >>> There may be easier solutions as well (I’ll give you an update once I > >>> remember them ;). > >> > >> > >> Strange solution... Ok, I'll wait for a better one. > >> > >> -- > >> s0rr0w > > > > > > > > > > -- > > s0rr0w >

On Sun, Jun 12, 2016 at 11:06 PM, Alexander Shpack <shadowkin at gmail.com> wrote: > Oh.. right. I forgot about it > > On Mon, Jun 13, 2016 at 12:02 AM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> This line tells you the difference: >> >> > let $content := file:read-text($fName, "cp1251") >> >> You are reading the file contents as text (i.e., string). You could >> also use file:read-binary. Using the browser, you can also upload >> images or other binary contents. >> >> >> > let $csv := csv:parse($content, map{ 'separator': ';', 'header': false() >> > } ) >> > return $csv >> > >> > What the difference between post data and raw file? >> > >> > >> > On Sun, Jun 12, 2016 at 11:45 PM, Alexander Shpack <shadowkin at gmail.com> >> > wrote: >> >> >> >> Did you always upload the same file? >> >> >> >> Yeah >> >> >> >>> >> >>> As the error indicates, it seems that your CSV input seems to contain >> >>> characters that are not valid in XML. There are various ways to tackle >> >>> this; one looks as follows: >> >>> >> >>> (: interpret client data as Base64 :) >> >>> declare >> >>> %rest:path("/upload") >> >>> %rest:POST("{$body}") >> >>> function local:store-csv($body as xs:base64Binary) >> >>> { >> >>> (: replace invalid characters with a question mark :) >> >>> let $input := bin:from-octets( >> >>> bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . >> >>> else >> >>> 63) >> >>> ) >> >>> (: convert to XQuery Unicode string; convert to XML :) >> >>> let $string := bin:decode-string($input, 'CP1251') >> >>> return csv:parse($string) >> >>> }; >> >>> >> >>> There may be easier solutions as well (I’ll give you an update once I >> >>> remember them ;). >> >> >> >> >> >> Strange solution... Ok, I'll wait for a better one. >> >> >> >> -- >> >> s0rr0w >> > >> > >> > >> > >> > -- >> > s0rr0w > > > > > -- > s0rr0w

On Mon, Jun 13, 2016 at 12:49 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Alex, > > I decided to add another function signature to the > convert:binary-to-string function [1]: If you set the third parameter > to true(), invalid XML characters will automatically be replaced with > \uFFFD (the so-called Unicode replacement character): > > declare > %rest:path("/upload") > %rest:POST("{$body}") > function local:store-csv($body as xs:base64Binary) > { > let $string := convert:binary-to-string($input, 'CP1251', true()) > return csv:parse($string) > }; > > A new stable snapshot is available [2]; the final version of BaseX 8.5 > will be released in June or July. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/Conversion_Module#convert:binary-to-string > [2] http://files.basex.org/releases/latest/ > > > > On Sun, Jun 12, 2016 at 11:06 PM, Alexander Shpack <shadowkin at gmail.com> > wrote: > > Oh.. right. I forgot about it > > > > On Mon, Jun 13, 2016 at 12:02 AM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> This line tells you the difference: > >> > >> > let $content := file:read-text($fName, "cp1251") > >> > >> You are reading the file contents as text (i.e., string). You could > >> also use file:read-binary. Using the browser, you can also upload > >> images or other binary contents. > >> > >> > >> > let $csv := csv:parse($content, map{ 'separator': ';', 'header': > false() > >> > } ) > >> > return $csv > >> > > >> > What the difference between post data and raw file? > >> > > >> > > >> > On Sun, Jun 12, 2016 at 11:45 PM, Alexander Shpack < > shadowkin at gmail.com> > >> > wrote: > >> >> > >> >> Did you always upload the same file? > >> >> > >> >> Yeah > >> >> > >> >>> > >> >>> As the error indicates, it seems that your CSV input seems to > contain > >> >>> characters that are not valid in XML. There are various ways to > tackle > >> >>> this; one looks as follows: > >> >>> > >> >>> (: interpret client data as Base64 :) > >> >>> declare > >> >>> %rest:path("/upload") > >> >>> %rest:POST("{$body}") > >> >>> function local:store-csv($body as xs:base64Binary) > >> >>> { > >> >>> (: replace invalid characters with a question mark :) > >> >>> let $input := bin:from-octets( > >> >>> bin:to-octets($body) ! (if(. >= 32 or . = (9, 10, 13)) then . > >> >>> else > >> >>> 63) > >> >>> ) > >> >>> (: convert to XQuery Unicode string; convert to XML :) > >> >>> let $string := bin:decode-string($input, 'CP1251') > >> >>> return csv:parse($string) > >> >>> }; > >> >>> > >> >>> There may be easier solutions as well (I’ll give you an update once > I > >> >>> remember them ;). > >> >> > >> >> > >> >> Strange solution... Ok, I'll wait for a better one. > >> >> > >> >> -- > >> >> s0rr0w > >> > > >> > > >> > > >> > > >> > -- > >> > s0rr0w > > > > > > > > > > -- > > s0rr0w >