Main Page » Mailing List » August 2022

changes in BaseX-10.0 w.r.t. http:send-request() / fetch:doc() functions?

Thursday, 4 August 2022 · 9 messages · last reply 5 Aug 2022

Ron Van den Branden4 Aug 2022 16:42 #

Hi, After upgrading BaseX to 10.0 (yay!), I notice how http:send-request() calls that used to work in 9.7 are now failing. I didn't find any changes documented between version 9.7 and 10.0 at https://docs.basex.org/wiki/HTTP_Client_Module. Yet (tested with the same JDK), I'm observing differences between BaseX-9.7 and BaseX-10.0. Test case:

   let $uri := 'http://vocab.getty.edu/aat/300027473.rdf'
   returnhttp:send-request(<http:request  method="get" status-only="true" href="{$uri}"/>)

Result: * BaseX-9.7: valid response

    <http:response  xmlns:http="http://expath.org/ns/http-client"  status="200" message="OK">
       <http:header  name="Server" value="Apache-Coyote/1.1"/>
       <http:header  name="Content-Disposition" value="attachment; filename=&quot;aat_300027473.rdf&quot;"/>
       <http:header  name="Set-Cookie" value="BIGipServerForest=!VhOKP5OjryEOuPhyk3Vb4l5ey77Ecb9C7h36RwDl7h0P+12yggqrxtSz6mEpwv8Du1qdIeJTRfFkqw==; path=/; Httponly"/>
       <http:header  name="Content-Length" value="55888"/>
       <http:header  name="Date" value="Thu, 04 Aug 2022 14:32:02 GMT"/>
       <http:header  name="Content-Type" value="application/rdf+xml;charset=UTF-8"/>
       <http:body  media-type="application/rdf+xml"/>
    </http:response>

* BaseX-10.0: "Bad Request" error

    <http:response  xmlns:http="http://expath.org/ns/http-client"  status="400" message="Bad Request">
       <http:header  name="connection" value="close"/>
       <http:header  name="content-length" value="968"/>
       <http:header  name="content-type" value="text/html;charset=utf-8"/>
       <http:header  name="date" value="Thu, 04 Aug 2022 14:31:56 GMT"/>
       <http:header  name="server" value="Apache-Coyote/1.1"/>
       <http:header  name="set-cookie" value="BIGipServerForest=!mBtphfxjdsqnz+Vyk3Vb4l5ey77EcVUpOTC6kqayu7tj3BpCNxN68oQTs+W4wrEO4+vQ2pECmIF6rA==; path=/; Httponly"/>
       <http:body  media-type="text/html"/>
    </http:response>

The same results are obtained with fetch:xml() (BaseX-9.7 - valid response) and fetch:doc() (BaseX-10.0 - bad request). Apologies if I'm overlooking the obvious, but has anything changed w.r.t. these http / fetch module functions or their underlying methods of network access that would require changes in my XQuery code or BaseX-10.0 configuration? Best, Ron

Christian Grün4 Aug 2022 16:55 #

Dear Ron, There has indeed been a substantial change in the way how http:send-request works; it’s now based on the contemporary Java HTTP Client API, which provides a better overall performance [1]. We’ll additionally mention that in the article on the HTTP Client Module. It should yield the same results as the old implementation, though; so thanks for your example, we’ll see what we can do. We’ll keep you updated. Christian [1] https://docs.basex.org/wiki/BaseX_10#HTTP_Requests

Quoted text · Ron Van den Branden · 45 lines

On Thu, Aug 4, 2022 at 4:42 PM Ron Van den Branden <ron.vdbranden at gmail.com> wrote: > > Hi, > > After upgrading BaseX to 10.0 (yay!), I notice how http:send-request() calls that used to work in 9.7 are now failing. I didn't find any changes documented between version 9.7 and 10.0 at https://docs.basex.org/wiki/HTTP_Client_Module. Yet (tested with the same JDK), I'm observing differences between BaseX-9.7 and BaseX-10.0. > > Test case: > > let $uri := 'http://vocab.getty.edu/aat/300027473.rdf' > return http:send-request(<http:request method="get" status-only="true" href="{$uri}"/>) > > Result: > > BaseX-9.7: valid response > > <http:response xmlns:http="http://expath.org/ns/http-client" status="200" message="OK"> > <http:header name="Server" value="Apache-Coyote/1.1"/> > <http:header name="Content-Disposition" value="attachment; filename="aat_300027473.rdf""/> > <http:header name="Set-Cookie" value="BIGipServerForest=!VhOKP5OjryEOuPhyk3Vb4l5ey77Ecb9C7h36RwDl7h0P+12yggqrxtSz6mEpwv8Du1qdIeJTRfFkqw==; path=/; Httponly"/> > <http:header name="Content-Length" value="55888"/> > <http:header name="Date" value="Thu, 04 Aug 2022 14:32:02 GMT"/> > <http:header name="Content-Type" value="application/rdf+xml;charset=UTF-8"/> > <http:body media-type="application/rdf+xml"/> > </http:response> > > BaseX-10.0: "Bad Request" error > > <http:response xmlns:http="http://expath.org/ns/http-client" status="400" message="Bad Request"> > <http:header name="connection" value="close"/> > <http:header name="content-length" value="968"/> > <http:header name="content-type" value="text/html;charset=utf-8"/> > <http:header name="date" value="Thu, 04 Aug 2022 14:31:56 GMT"/> > <http:header name="server" value="Apache-Coyote/1.1"/> > <http:header name="set-cookie" value="BIGipServerForest=!mBtphfxjdsqnz+Vyk3Vb4l5ey77EcVUpOTC6kqayu7tj3BpCNxN68oQTs+W4wrEO4+vQ2pECmIF6rA==; path=/; Httponly"/> > <http:body media-type="text/html"/> > </http:response> > > The same results are obtained with fetch:xml() (BaseX-9.7 - valid response) and fetch:doc() (BaseX-10.0 - bad request). > > Apologies if I'm overlooking the obvious, but has anything changed w.r.t. these http / fetch module functions or their underlying methods of network access that would require changes in my XQuery code or BaseX-10.0 configuration? > > Best, > > Ron

Ron Van den Branden4 Aug 2022 17:01 #

Dear Christian, Whoops, the obvious, after all; thanks for kindly (and lightning fast) pointing that out, and looking into this! Best, Ron

Quoted text · Christian Grün · 61 lines

On 4/08/2022 16:55, Christian Grün wrote: > Dear Ron, > > There has indeed been a substantial change in the way how > http:send-request works; it’s now based on the contemporary Java HTTP > Client API, which provides a better overall performance [1]. We’ll > additionally mention that in the article on the HTTP Client Module. > > It should yield the same results as the old implementation, though; so > thanks for your example, we’ll see what we can do. > > We’ll keep you updated. > Christian > > [1] https://docs.basex.org/wiki/BaseX_10#HTTP_Requests > > > On Thu, Aug 4, 2022 at 4:42 PM Ron Van den Branden > <ron.vdbranden at gmail.com> wrote: >> Hi, >> >> After upgrading BaseX to 10.0 (yay!), I notice how http:send-request() calls that used to work in 9.7 are now failing. I didn't find any changes documented between version 9.7 and 10.0 at https://docs.basex.org/wiki/HTTP_Client_Module. Yet (tested with the same JDK), I'm observing differences between BaseX-9.7 and BaseX-10.0. >> >> Test case: >> >> let $uri := 'http://vocab.getty.edu/aat/300027473.rdf' >> return http:send-request(<http:request method="get" status-only="true" href="{$uri}"/>) >> >> Result: >> >> BaseX-9.7: valid response >> >> <http:response xmlns:http="http://expath.org/ns/http-client" status="200" message="OK"> >> <http:header name="Server" value="Apache-Coyote/1.1"/> >> <http:header name="Content-Disposition" value="attachment; filename="aat_300027473.rdf""/> >> <http:header name="Set-Cookie" value="BIGipServerForest=!VhOKP5OjryEOuPhyk3Vb4l5ey77Ecb9C7h36RwDl7h0P+12yggqrxtSz6mEpwv8Du1qdIeJTRfFkqw==; path=/; Httponly"/> >> <http:header name="Content-Length" value="55888"/> >> <http:header name="Date" value="Thu, 04 Aug 2022 14:32:02 GMT"/> >> <http:header name="Content-Type" value="application/rdf+xml;charset=UTF-8"/> >> <http:body media-type="application/rdf+xml"/> >> </http:response> >> >> BaseX-10.0: "Bad Request" error >> >> <http:response xmlns:http="http://expath.org/ns/http-client" status="400" message="Bad Request"> >> <http:header name="connection" value="close"/> >> <http:header name="content-length" value="968"/> >> <http:header name="content-type" value="text/html;charset=utf-8"/> >> <http:header name="date" value="Thu, 04 Aug 2022 14:31:56 GMT"/> >> <http:header name="server" value="Apache-Coyote/1.1"/> >> <http:header name="set-cookie" value="BIGipServerForest=!mBtphfxjdsqnz+Vyk3Vb4l5ey77EcVUpOTC6kqayu7tj3BpCNxN68oQTs+W4wrEO4+vQ2pECmIF6rA==; path=/; Httponly"/> >> <http:body media-type="text/html"/> >> </http:response> >> >> The same results are obtained with fetch:xml() (BaseX-9.7 - valid response) and fetch:doc() (BaseX-10.0 - bad request). >> >> Apologies if I'm overlooking the obvious, but has anything changed w.r.t. these http / fetch module functions or their underlying methods of network access that would require changes in my XQuery code or BaseX-10.0 configuration? >> >> Best, >> >> Ron

Christian Grün4 Aug 2022 17:19 #

What I have assessed so far is that it’s the Java Client that fails to retrieve the result. It’s the same response that’s returned by BaseX.

String uri = "http://vocab.getty.edu/aat/300027473.rdf";
HttpClient client = HttpClient.newBuilder().build();
HttpRequest request = HttpRequest.newBuilder(URI.create(uri)).build();
BodyHandler<String> handler = HttpResponse.BodyHandlers.ofString();
HttpResponse<String> result = client.send(request, handler);
System.out.println(result.statusCode());
System.out.println(result.body());

400
<html><head><title>Apache Tomcat/7.0.42 - Error
report</title><style><!--H1
{font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
H2 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
H3 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
B {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
P {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}A
{color : black;}A.name {color : black;}HR {color :
#525D76;}--></style> </head><body><h1>HTTP Status 400 - </h1><HR
size="1" noshade="noshade"><p><b>type</b> Status
report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The
request sent by the client was syntactically incorrect.</u></p><HR
size="1" noshade="noshade"><h3>Apache Tomcat/7.0.42</h3></body></html>

So we need to find out why the server thinks the Java request is »syntactically incorrect«. Maybe we can compare the low-level representation of the requests with Java 9 and 10 (?).

Andy Bunce4 Aug 2022 18:34 #

There seems to be a 303 redirect. Maybe this is relevant https://stackoverflow.com/a/66325588/3210344 /Andy [image: image.png]

Quoted text · Christian Grün · 39 lines

On Thu, 4 Aug 2022 at 16:19, Christian Grün <christian.gruen at gmail.com> wrote: > What I have assessed so far is that it’s the Java Client that fails to > retrieve the result. It’s the same response that’s returned by BaseX. > > String uri = "http://vocab.getty.edu/aat/300027473.rdf"; > HttpClient client = HttpClient.newBuilder().build(); > HttpRequest request = HttpRequest.newBuilder(URI.create(uri)).build(); > BodyHandler<String> handler = HttpResponse.BodyHandlers.ofString(); > HttpResponse<String> result = client.send(request, handler); > System.out.println(result.statusCode()); > System.out.println(result.body()); > > 400 > <html><head><title>Apache Tomcat/7.0.42 - Error > report</title><style></style> </head><body><h1>HTTP Status 400 - </h1><HR > size="1" noshade="noshade"><p><b>type</b> Status > report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The > request sent by the client was syntactically incorrect.</u></p><HR > size="1" noshade="noshade"><h3>Apache Tomcat/7.0.42</h3></body></html> > > So we need to find out why the server thinks the Java request is > »syntactically incorrect«. Maybe we can compare the low-level > representation of the requests with Java 9 and 10 (?). >

Ron Van den Branden5 Aug 2022 09:22 #

Hi, Thanks for chiming in, Andy! I realized yesterday that I should have added how some URLs can be retrieved without problem in BaseX-10.0, e.g.:

   let $uri := 'https://www.w3.org'
   returnhttp:send-request(<http:request  method="get" status-only="true" href="{$uri}"/>)

...which is well-formed (to rule out non-XML parser issues), and indeed has no redirection, which seems consistent with Andy's observation. Yet, https://w3.org also is retrieved successfully, which has an initial 301 response (instead of 303). Best, Ron

Quoted text · Andy Bunce · 47 lines

On 4/08/2022 18:34, Andy Bunce wrote: > There seems to be a 303 redirect. Maybe this is relevant > https://stackoverflow.com/a/66325588/3210344 > /Andy > image.png > > > > On Thu, 4 Aug 2022 at 16:19, Christian Grün > <christian.gruen at gmail.com> wrote: > > What I have assessed so far is that it’s the Java Client that fails to > retrieve the result. It’s the same response that’s returned by BaseX. > > String uri = "http://vocab.getty.edu/aat/300027473.rdf"; > HttpClient client = HttpClient.newBuilder().build(); > HttpRequest request = HttpRequest.newBuilder(URI.create(uri)).build(); > BodyHandler<String> handler = HttpResponse.BodyHandlers.ofString(); > HttpResponse<String> result = client.send(request, handler); > System.out.println(result.statusCode()); > System.out.println(result.body()); > > 400 > <html><head><title>Apache Tomcat/7.0.42 - Error > report</title><style></style> </head><body><h1>HTTP Status 400 - </h1><HR > size="1" noshade="noshade"><p><b>type</b> Status > report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The > request sent by the client was syntactically incorrect.</u></p><HR > size="1" noshade="noshade"><h3>Apache Tomcat/7.0.42</h3></body></html> > > So we need to find out why the server thinks the Java request is > »syntactically incorrect«. Maybe we can compare the low-level > representation of the requests with Java 9 and 10 (?). >

Christian Grün5 Aug 2022 11:05 #

This is what we found out (by the help of Wireshark, and some online resources): • The new JDK HTTP Client does not attach a default "Accept" header to the HTTP Request. • The getty.edu web server (Tomcat?) returns a syntax error when this header is missing in the request. • We also had a look at the 303 redirection. It works fine; with BaseX 10, redirection could even be improved, as protocol changes (http → https) are now supported, too. A new snapshot with a workaround is online [1,2]. Thanks for the observation. Christian [1] https://github.com/BaseXdb/basex/issues/2133 [2] https://files.basex.org/releases/latest/

Quoted text · Ron Van den Branden · 68 lines

On Fri, Aug 5, 2022 at 9:22 AM Ron Van den Branden <ron.vdbranden at gmail.com> wrote: > Hi, > > Thanks for chiming in, Andy! I realized yesterday that I should have added > how some URLs can be retrieved without problem in BaseX-10.0, e.g.: > > let $uri := 'https://www.w3.org' > return http:send-request(<http:request method="get" status-only="true" href="{$uri}"/>) > > ...which is well-formed (to rule out non-XML parser issues), and indeed > has no redirection, which seems consistent with Andy's observation. Yet, > https://w3.org also is retrieved successfully, which has an initial 301 > response (instead of 303). > > Best, > > Ron > On 4/08/2022 18:34, Andy Bunce wrote: > > There seems to be a 303 redirect. Maybe this is relevant > https://stackoverflow.com/a/66325588/3210344 > /Andy > [image: image.png] > > > > On Thu, 4 Aug 2022 at 16:19, Christian Grün <christian.gruen at gmail.com> > wrote: > >> What I have assessed so far is that it’s the Java Client that fails to >> retrieve the result. It’s the same response that’s returned by BaseX. >> >> String uri = "http://vocab.getty.edu/aat/300027473.rdf"; >> HttpClient client = HttpClient.newBuilder().build(); >> HttpRequest request = HttpRequest.newBuilder(URI.create(uri)).build(); >> BodyHandler<String> handler = HttpResponse.BodyHandlers.ofString(); >> HttpResponse<String> result = client.send(request, handler); >> System.out.println(result.statusCode()); >> System.out.println(result.body()); >> >> 400 >> <html><head><title>Apache Tomcat/7.0.42 - Error >> report</title><style></style> </head><body><h1>HTTP Status 400 - </h1><HR >> size="1" noshade="noshade"><p><b>type</b> Status >> report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The >> request sent by the client was syntactically incorrect.</u></p><HR >> size="1" noshade="noshade"><h3>Apache Tomcat/7.0.42</h3></body></html> >> >> So we need to find out why the server thinks the Java request is >> »syntactically incorrect«. Maybe we can compare the low-level >> representation of the requests with Java 9 and 10 (?). >> >

Ron Van den Branden5 Aug 2022 11:13 #

I'm stunned, thanks so much! Best, Ron

Quoted text · Christian Grün · 97 lines

On 5/08/2022 11:05, Christian Grün wrote: > This is what we found out (by the help of Wireshark, and some online > resources): > > • The new JDK HTTP Client does not attach a default "Accept" header to > the HTTP Request. > • The getty.edu <http://getty.edu> web server (Tomcat?) returns a > syntax error when this header is missing in the request. > • We also had a look at the 303 redirection. It works fine; with BaseX > 10, redirection could even be improved, as protocol changes (http → > https) are now supported, too. > > A new snapshot with a workaround is online [1,2]. > > Thanks for the observation. > Christian > > [1] https://github.com/BaseXdb/basex/issues/2133 > [2] https://files.basex.org/releases/latest/ > > > > On Fri, Aug 5, 2022 at 9:22 AM Ron Van den Branden > <ron.vdbranden at gmail.com> wrote: > > Hi, > > Thanks for chiming in, Andy! I realized yesterday that I should > have added how some URLs can be retrieved without problem in > BaseX-10.0, e.g.: > > let $uri := 'https://www.w3.org' > returnhttp:send-request( <http:send-request(><http:request <http:request> method="get" status-only="true" href="{$uri}"/>) > > ...which is well-formed (to rule out non-XML parser issues), and > indeed has no redirection, which seems consistent with Andy's > observation. Yet, https://w3.org also is retrieved successfully, > which has an initial 301 response (instead of 303). > > Best, > > Ron > > On 4/08/2022 18:34, Andy Bunce wrote: >> There seems to be a 303 redirect. Maybe this is relevant >> https://stackoverflow.com/a/66325588/3210344 >> /Andy >> image.png >> >> >> >> On Thu, 4 Aug 2022 at 16:19, Christian Grün >> <christian.gruen at gmail.com> wrote: >> >> What I have assessed so far is that it’s the Java Client that >> fails to >> retrieve the result. It’s the same response that’s returned >> by BaseX. >> >> String uri = "http://vocab.getty.edu/aat/300027473.rdf"; >> HttpClient client = HttpClient.newBuilder().build(); >> HttpRequest request = >> HttpRequest.newBuilder(URI.create(uri)).build(); >> BodyHandler<String> handler = >> HttpResponse.BodyHandlers.ofString(); >> HttpResponse<String> result = client.send(request, handler); >> System.out.println(result.statusCode()); >> System.out.println(result.body()); >> >> 400 >> <html><head><title>Apache Tomcat/7.0.42 - Error >> report</title><style></style> </head><body><h1>HTTP Status 400 - </h1><HR >> size="1" noshade="noshade"><p><b>type</b> Status >> report</p><p><b>message</b> <u></u></p><p><b>description</b> >> <u>The >> request sent by the client was syntactically >> incorrect.</u></p><HR >> size="1" noshade="noshade"><h3>Apache >> Tomcat/7.0.42</h3></body></html> >> >> So we need to find out why the server thinks the Java request is >> »syntactically incorrect«. Maybe we can compare the low-level >> representation of the requests with Java 9 and 10 (?). >>

Andy Bunce5 Aug 2022 11:20 #

Great! I just spotted that the home page was failing in the same way. It has no redirect. /Andy

  let $uri := 'http://vocab.getty.edu/'
  return http:send-request(<http:request method="get"  href="{$uri}"/>)

Quoted text · Ron Van den Branden · 99 lines

On Fri, 5 Aug 2022 at 10:13, Ron Van den Branden <ron.vdbranden at gmail.com> wrote: > I'm stunned, thanks so much! > > Best, > > Ron > On 5/08/2022 11:05, Christian Grün wrote: > > This is what we found out (by the help of Wireshark, and some online > resources): > > • The new JDK HTTP Client does not attach a default "Accept" header to the > HTTP Request. > • The getty.edu web server (Tomcat?) returns a syntax error when this > header is missing in the request. > • We also had a look at the 303 redirection. It works fine; with BaseX 10, > redirection could even be improved, as protocol changes (http → https) are > now supported, too. > > A new snapshot with a workaround is online [1,2]. > > Thanks for the observation. > Christian > > [1] https://github.com/BaseXdb/basex/issues/2133 > [2] https://files.basex.org/releases/latest/ > > > > On Fri, Aug 5, 2022 at 9:22 AM Ron Van den Branden < > ron.vdbranden at gmail.com> wrote: > >> Hi, >> >> Thanks for chiming in, Andy! I realized yesterday that I should have >> added how some URLs can be retrieved without problem in BaseX-10.0, e.g.: >> >> let $uri := 'https://www.w3.org' >> return http:send-request(<http:request method="get" status-only="true" href="{$uri}"/>) >> >> ...which is well-formed (to rule out non-XML parser issues), and indeed >> has no redirection, which seems consistent with Andy's observation. Yet, >> https://w3.org also is retrieved successfully, which has an initial 301 >> response (instead of 303). >> >> Best, >> >> Ron >> On 4/08/2022 18:34, Andy Bunce wrote: >> >> There seems to be a 303 redirect. Maybe this is relevant >> https://stackoverflow.com/a/66325588/3210344 >> /Andy >> [image: image.png] >> >> >> >> On Thu, 4 Aug 2022 at 16:19, Christian Grün <christian.gruen at gmail.com> >> wrote: >> >>> What I have assessed so far is that it’s the Java Client that fails to >>> retrieve the result. It’s the same response that’s returned by BaseX. >>> >>> String uri = "http://vocab.getty.edu/aat/300027473.rdf"; >>> HttpClient client = HttpClient.newBuilder().build(); >>> HttpRequest request = HttpRequest.newBuilder(URI.create(uri)).build(); >>> BodyHandler<String> handler = HttpResponse.BodyHandlers.ofString(); >>> HttpResponse<String> result = client.send(request, handler); >>> System.out.println(result.statusCode()); >>> System.out.println(result.body()); >>> >>> 400 >>> <html><head><title>Apache Tomcat/7.0.42 - Error >>> report</title><style></style> </head><body><h1>HTTP Status 400 - </h1><HR >>> size="1" noshade="noshade"><p><b>type</b> Status >>> report</p><p><b>message</b> <u></u></p><p><b>description</b> <u>The >>> request sent by the client was syntactically incorrect.</u></p><HR >>> size="1" noshade="noshade"><h3>Apache Tomcat/7.0.42</h3></body></html> >>> >>> So we need to find out why the server thinks the Java request is >>> »syntactically incorrect«. Maybe we can compare the low-level >>> representation of the requests with Java 9 and 10 (?). >>> >>

Written for BaseX 10.0. BaseX may have changed since: please check the current documentation.

⚡Generated with XQuery

`J`	Select the next message
`K`	Select the previous message
`N`	Next thread
`P`	Previous thread
`U`	Go up one level
`/`	Jump to the search field