Issue with HTTP client and authentication

On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> wrote: > > Hello, > > I'm trying to post a SPARQL query to an endpoint using Digest authentication with the HTTP client. > > The query works fine using curl: > > curl --digest --user user:pass -X POST -d@'test.rq' \ > -H "Content-type: application/sparql-query" \ > 'http://example.org/sparql' > > But the equivalent request in BaseX fails with 401 Unauthorized: > > let $endpoint := "http://example.org/sparql" > let $user := "user" > let $pass := "pass" > let $type := "application/sparql-query" > > let $response := ( > http:send-request( > <http:request > method="POST" > href="{$endpoint}" > username="{$user}" > password="{$pass}" > auth-method="Digest" > send-authorization="true"> > <http:header > name="Content-Type" > value="{$type}; charset=utf-8"/> > <http:body > media-type="{$type}">{ > ``[ > select * where {?s ?p ?o} limit 1 > ]`` > }</http:body> > </http:request> > ) > ) > return > $response > > Any ideas about what might be causing the BaseX HTTP client to be denied here? > > Thanks in advance, > Tim > > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library >

Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:48: > Thanks, Christian. Right, that wasn't a working example. The server I'm > querying is behind a firewall, so I can't really reproduce the issue > unfortunately. > > The BaseX documentation states that the HTTP client does support Digest > authentication. However, I did some poking around using Wireshark and > limited knowledge of network traffic. When I run the request in curl, the > expected protocol is played out, and I see the evidence in Wireshark: > > (1) The server responds with "401 Unauthorized" and provides the nonce > value. > (2) The client (curl) then does its business (supplies the username and > password, resends the request, etc.) > > But when I submit the request via BaseX, it never gets past step (1). It > never seems to send the username and password at all. Is Digest > authentication still supported? > > Best, > Tim > > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library > > > On Wed, Feb 2, 2022 at 10:13 AM Christian Grün <christian.gruen at gmail.com> > wrote: > >> Hi Tim, >> >> Difficult to tell; both the curl and the XQuery variants give me 404. >> Do you think you could provide us with an example that works out of >> the box? >> >> Best, >> Christian >> >> >> >> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> wrote: >> > >> > Hello, >> > >> > I'm trying to post a SPARQL query to an endpoint using Digest >> authentication with the HTTP client. >> > >> > The query works fine using curl: >> > >> > curl --digest --user user:pass -X POST -d@'test.rq' \ >> > -H "Content-type: application/sparql-query" \ >> > 'http://example.org/sparql' >> > >> > But the equivalent request in BaseX fails with 401 Unauthorized: >> > >> > let $endpoint := "http://example.org/sparql" >> > let $user := "user" >> > let $pass := "pass" >> > let $type := "application/sparql-query" >> > >> > let $response := ( >> > http:send-request( >> > <http:request >> > method="POST" >> > href="{$endpoint}" >> > username="{$user}" >> > password="{$pass}" >> > auth-method="Digest" >> > send-authorization="true"> >> > <http:header >> > name="Content-Type" >> > value="{$type}; charset=utf-8"/> >> > <http:body >> > media-type="{$type}">{ >> > ``[ >> > select * where {?s ?p ?o} limit 1 >> > ]`` >> > }</http:body> >> > </http:request> >> > ) >> > ) >> > return >> > $response >> > >> > Any ideas about what might be causing the BaseX HTTP client to be >> denied here? >> > >> > Thanks in advance, >> > Tim >> > >> > >> > -- >> > Tim A. Thompson >> > Metadata Librarian >> > Yale University Library >> > >> >

Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:54: > With "digest" I get: "[experr:HC0004] Invalid authentication method: > digest." > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library > > On Wed, Feb 2, 2022 at 11:52 AM Christian Grün <christian.gruen at gmail.com> > wrote: > >> Yes, it should still be supported. I assume it doesn't make a difference >> if you use "Digest" or "digest"? >> >> >> >> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:48: >> >>> Thanks, Christian. Right, that wasn't a working example. The server I'm >>> querying is behind a firewall, so I can't really reproduce the issue >>> unfortunately. >>> >>> The BaseX documentation states that the HTTP client does support Digest >>> authentication. However, I did some poking around using Wireshark and >>> limited knowledge of network traffic. When I run the request in curl, the >>> expected protocol is played out, and I see the evidence in Wireshark: >>> >>> (1) The server responds with "401 Unauthorized" and provides the nonce >>> value. >>> (2) The client (curl) then does its business (supplies the username and >>> password, resends the request, etc.) >>> >>> But when I submit the request via BaseX, it never gets past step (1). It >>> never seems to send the username and password at all. Is Digest >>> authentication still supported? >>> >>> Best, >>> Tim >>> >>> >>> -- >>> Tim A. Thompson >>> Metadata Librarian >>> Yale University Library >>> >>> >>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün < >>> christian.gruen at gmail.com> wrote: >>> >>>> Hi Tim, >>>> >>>> Difficult to tell; both the curl and the XQuery variants give me 404. >>>> Do you think you could provide us with an example that works out of >>>> the box? >>>> >>>> Best, >>>> Christian >>>> >>>> >>>> >>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> >>>> wrote: >>>> > >>>> > Hello, >>>> > >>>> > I'm trying to post a SPARQL query to an endpoint using Digest >>>> authentication with the HTTP client. >>>> > >>>> > The query works fine using curl: >>>> > >>>> > curl --digest --user user:pass -X POST -d@'test.rq' \ >>>> > -H "Content-type: application/sparql-query" \ >>>> > 'http://example.org/sparql' >>>> > >>>> > But the equivalent request in BaseX fails with 401 Unauthorized: >>>> > >>>> > let $endpoint := "http://example.org/sparql" >>>> > let $user := "user" >>>> > let $pass := "pass" >>>> > let $type := "application/sparql-query" >>>> > >>>> > let $response := ( >>>> > http:send-request( >>>> > <http:request >>>> > method="POST" >>>> > href="{$endpoint}" >>>> > username="{$user}" >>>> > password="{$pass}" >>>> > auth-method="Digest" >>>> > send-authorization="true"> >>>> > <http:header >>>> > name="Content-Type" >>>> > value="{$type}; charset=utf-8"/> >>>> > <http:body >>>> > media-type="{$type}">{ >>>> > ``[ >>>> > select * where {?s ?p ?o} limit 1 >>>> > ]`` >>>> > }</http:body> >>>> > </http:request> >>>> > ) >>>> > ) >>>> > return >>>> > $response >>>> > >>>> > Any ideas about what might be causing the BaseX HTTP client to be >>>> denied here? >>>> > >>>> > Thanks in advance, >>>> > Tim >>>> > >>>> > >>>> > -- >>>> > Tim A. Thompson >>>> > Metadata Librarian >>>> > Yale University Library >>>> > >>>> >>>

On Wed, 2 Feb 2022 at 16:58, Christian Grün <christian.gruen at gmail.com> wrote: > Good to hear; so at least Digest is detected at this stage ;) > > > Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:54: > >> With "digest" I get: "[experr:HC0004] Invalid authentication method: >> digest." >> >> -- >> Tim A. Thompson >> Metadata Librarian >> Yale University Library >> >> On Wed, Feb 2, 2022 at 11:52 AM Christian Grün <christian.gruen at gmail.com> >> wrote: >> >>> Yes, it should still be supported. I assume it doesn't make a difference >>> if you use "Digest" or "digest"? >>> >>> >>> >>> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:48: >>> >>>> Thanks, Christian. Right, that wasn't a working example. The server I'm >>>> querying is behind a firewall, so I can't really reproduce the issue >>>> unfortunately. >>>> >>>> The BaseX documentation states that the HTTP client does support Digest >>>> authentication. However, I did some poking around using Wireshark and >>>> limited knowledge of network traffic. When I run the request in curl, the >>>> expected protocol is played out, and I see the evidence in Wireshark: >>>> >>>> (1) The server responds with "401 Unauthorized" and provides the nonce >>>> value. >>>> (2) The client (curl) then does its business (supplies the username and >>>> password, resends the request, etc.) >>>> >>>> But when I submit the request via BaseX, it never gets past step (1). >>>> It never seems to send the username and password at all. Is Digest >>>> authentication still supported? >>>> >>>> Best, >>>> Tim >>>> >>>> >>>> -- >>>> Tim A. Thompson >>>> Metadata Librarian >>>> Yale University Library >>>> >>>> >>>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün < >>>> christian.gruen at gmail.com> wrote: >>>> >>>>> Hi Tim, >>>>> >>>>> Difficult to tell; both the curl and the XQuery variants give me 404. >>>>> Do you think you could provide us with an example that works out of >>>>> the box? >>>>> >>>>> Best, >>>>> Christian >>>>> >>>>> >>>>> >>>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> >>>>> wrote: >>>>> > >>>>> > Hello, >>>>> > >>>>> > I'm trying to post a SPARQL query to an endpoint using Digest >>>>> authentication with the HTTP client. >>>>> > >>>>> > The query works fine using curl: >>>>> > >>>>> > curl --digest --user user:pass -X POST -d@'test.rq' \ >>>>> > -H "Content-type: application/sparql-query" \ >>>>> > 'http://example.org/sparql' >>>>> > >>>>> > But the equivalent request in BaseX fails with 401 Unauthorized: >>>>> > >>>>> > let $endpoint := "http://example.org/sparql" >>>>> > let $user := "user" >>>>> > let $pass := "pass" >>>>> > let $type := "application/sparql-query" >>>>> > >>>>> > let $response := ( >>>>> > http:send-request( >>>>> > <http:request >>>>> > method="POST" >>>>> > href="{$endpoint}" >>>>> > username="{$user}" >>>>> > password="{$pass}" >>>>> > auth-method="Digest" >>>>> > send-authorization="true"> >>>>> > <http:header >>>>> > name="Content-Type" >>>>> > value="{$type}; charset=utf-8"/> >>>>> > <http:body >>>>> > media-type="{$type}">{ >>>>> > ``[ >>>>> > select * where {?s ?p ?o} limit 1 >>>>> > ]`` >>>>> > }</http:body> >>>>> > </http:request> >>>>> > ) >>>>> > ) >>>>> > return >>>>> > $response >>>>> > >>>>> > Any ideas about what might be causing the BaseX HTTP client to be >>>>> denied here? >>>>> > >>>>> > Thanks in advance, >>>>> > Tim >>>>> > >>>>> > >>>>> > -- >>>>> > Tim A. Thompson >>>>> > Metadata Librarian >>>>> > Yale University Library >>>>> > >>>>> >>>>

Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 18:19: > Thanks, Andy! That's very handy. So, when I test against httpbin, the same > thing happens. > > curl request: > curl --digest --user user:pw -X GET ' > http://httpbin.org/digest-auth/auth/user/pw' > > response: > { > "authenticated": true, > "user": "user" > } > > BaseX request: > let $endpoint := "http://httpbin.org/digest-auth/auth/user/pw" > > let $response := ( > http:send-request( > <http:request > method="GET" > href="{$endpoint}" > username="user" > password="pw" > auth-method="Digest" > send-authorization="true"> > <http:header > name="Content-Type" > value="{$type}; charset=utf-8"/> > </http:request> > ) > ) > return > $response > > response: > <http:response xmlns:http="http://expath.org/ns/http-client" status="401" > message="UNAUTHORIZED"> > <http:header name="Server" value="gunicorn/19.9.0"/> > <http:header name="Access-Control-Allow-Origin" value="*"/> > <http:header name="Access-Control-Allow-Credentials" value="true"/> > <http:header name="WWW-Authenticate" value="Digest realm=& > quot;me at kennethreitz.com&quot;, > nonce=&quot;36d59385e96e1689595f4f14fa2921ca&quot;, qop=&quot;auth&quot;, > opaque=&quot;a21df3287b5cb9cbf8b6a68b2c15fa1d&quot;, algorithm=MD5, > stale=FALSE"/> > <http:header name="Connection" value="keep-alive"/> > <http:header name="Set-Cookie" value="fake=fake_value; Path=/"/> > <http:header name="Set-Cookie" value="stale_after=never; Path=/"/> > <http:header name="Content-Length" value="0"/> > <http:header name="Date" value="Wed, 02 Feb 2022 17:14:13 GMT"/> > <http:header name="Content-Type" value="text/html; charset=utf-8"/> > </http:response> > > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library > > > On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce <bunce.andy at gmail.com> wrote: > >> I find httpbin.org a useful resource to test this kind of thing [1] and >> [2] >> /Andy >> >> [1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_ >> [2] http://httpbin.org/basic-auth/user1/mypass >> >> On Wed, 2 Feb 2022 at 16:58, Christian Grün <christian.gruen at gmail.com> >> wrote: >> >>> Good to hear; so at least Digest is detected at this stage ;) >>> >>> >>> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:54: >>> >>>> With "digest" I get: "[experr:HC0004] Invalid authentication method: >>>> digest." >>>> >>>> -- >>>> Tim A. Thompson >>>> Metadata Librarian >>>> Yale University Library >>>> >>>> On Wed, Feb 2, 2022 at 11:52 AM Christian Grün < >>>> christian.gruen at gmail.com> wrote: >>>> >>>>> Yes, it should still be supported. I assume it doesn't make a >>>>> difference if you use "Digest" or "digest"? >>>>> >>>>> >>>>> >>>>> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:48: >>>>> >>>>>> Thanks, Christian. Right, that wasn't a working example. The server >>>>>> I'm querying is behind a firewall, so I can't really reproduce the issue >>>>>> unfortunately. >>>>>> >>>>>> The BaseX documentation states that the HTTP client does support >>>>>> Digest authentication. However, I did some poking around using Wireshark >>>>>> and limited knowledge of network traffic. When I run the request in curl, >>>>>> the expected protocol is played out, and I see the evidence in Wireshark: >>>>>> >>>>>> (1) The server responds with "401 Unauthorized" and provides the >>>>>> nonce value. >>>>>> (2) The client (curl) then does its business (supplies the username >>>>>> and password, resends the request, etc.) >>>>>> >>>>>> But when I submit the request via BaseX, it never gets past step (1). >>>>>> It never seems to send the username and password at all. Is Digest >>>>>> authentication still supported? >>>>>> >>>>>> Best, >>>>>> Tim >>>>>> >>>>>> >>>>>> -- >>>>>> Tim A. Thompson >>>>>> Metadata Librarian >>>>>> Yale University Library >>>>>> >>>>>> >>>>>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün < >>>>>> christian.gruen at gmail.com> wrote: >>>>>> >>>>>>> Hi Tim, >>>>>>> >>>>>>> Difficult to tell; both the curl and the XQuery variants give me 404. >>>>>>> Do you think you could provide us with an example that works out of >>>>>>> the box? >>>>>>> >>>>>>> Best, >>>>>>> Christian >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> >>>>>>> wrote: >>>>>>> > >>>>>>> > Hello, >>>>>>> > >>>>>>> > I'm trying to post a SPARQL query to an endpoint using Digest >>>>>>> authentication with the HTTP client. >>>>>>> > >>>>>>> > The query works fine using curl: >>>>>>> > >>>>>>> > curl --digest --user user:pass -X POST -d@'test.rq' \ >>>>>>> > -H "Content-type: application/sparql-query" \ >>>>>>> > 'http://example.org/sparql' >>>>>>> > >>>>>>> > But the equivalent request in BaseX fails with 401 Unauthorized: >>>>>>> > >>>>>>> > let $endpoint := "http://example.org/sparql" >>>>>>> > let $user := "user" >>>>>>> > let $pass := "pass" >>>>>>> > let $type := "application/sparql-query" >>>>>>> > >>>>>>> > let $response := ( >>>>>>> > http:send-request( >>>>>>> > <http:request >>>>>>> > method="POST" >>>>>>> > href="{$endpoint}" >>>>>>> > username="{$user}" >>>>>>> > password="{$pass}" >>>>>>> > auth-method="Digest" >>>>>>> > send-authorization="true"> >>>>>>> > <http:header >>>>>>> > name="Content-Type" >>>>>>> > value="{$type}; charset=utf-8"/> >>>>>>> > <http:body >>>>>>> > media-type="{$type}">{ >>>>>>> > ``[ >>>>>>> > select * where {?s ?p ?o} limit 1 >>>>>>> > ]`` >>>>>>> > }</http:body> >>>>>>> > </http:request> >>>>>>> > ) >>>>>>> > ) >>>>>>> > return >>>>>>> > $response >>>>>>> > >>>>>>> > Any ideas about what might be causing the BaseX HTTP client to be >>>>>>> denied here? >>>>>>> > >>>>>>> > Thanks in advance, >>>>>>> > Tim >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > Tim A. Thompson >>>>>>> > Metadata Librarian >>>>>>> > Yale University Library >>>>>>> > >>>>>>> >>>>>>

On Wed, Feb 2, 2022 at 6:19 PM Tim Thompson <timathom at gmail.com> wrote: > > Thanks, Andy! That's very handy. So, when I test against httpbin, the same thing happens. > > curl request: > curl --digest --user user:pw -X GET 'http://httpbin.org/digest-auth/auth/user/pw' > > response: > { > "authenticated": true, > "user": "user" > } > > BaseX request: > let $endpoint := "http://httpbin.org/digest-auth/auth/user/pw" > > let $response := ( > http:send-request( > <http:request > method="GET" > href="{$endpoint}" > username="user" > password="pw" > auth-method="Digest" > send-authorization="true"> > <http:header > name="Content-Type" > value="{$type}; charset=utf-8"/> > </http:request> > ) > ) > return > $response > > response: > <http:response xmlns:http="http://expath.org/ns/http-client" status="401" message="UNAUTHORIZED"> > <http:header name="Server" value="gunicorn/19.9.0"/> > <http:header name="Access-Control-Allow-Origin" value="*"/> > <http:header name="Access-Control-Allow-Credentials" value="true"/> > <http:header name="WWW-Authenticate" value="Digest realm=&quot;me at kennethreitz.com&quot;, nonce=&quot;36d59385e96e1689595f4f14fa2921ca&quot;, qop=&quot;auth&quot;, opaque=&quot;a21df3287b5cb9cbf8b6a68b2c15fa1d&quot;, algorithm=MD5, stale=FALSE"/> > <http:header name="Connection" value="keep-alive"/> > <http:header name="Set-Cookie" value="fake=fake_value; Path=/"/> > <http:header name="Set-Cookie" value="stale_after=never; Path=/"/> > <http:header name="Content-Length" value="0"/> > <http:header name="Date" value="Wed, 02 Feb 2022 17:14:13 GMT"/> > <http:header name="Content-Type" value="text/html; charset=utf-8"/> > </http:response> > > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library > > > On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce <bunce.andy at gmail.com> wrote: >> >> I find httpbin.org a useful resource to test this kind of thing [1] and [2] >> /Andy >> >> [1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_ >> [2] http://httpbin.org/basic-auth/user1/mypass >> >> On Wed, 2 Feb 2022 at 16:58, Christian Grün <christian.gruen at gmail.com> wrote: >>> >>> Good to hear; so at least Digest is detected at this stage ;) >>> >>> >>> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:54: >>>> >>>> With "digest" I get: "[experr:HC0004] Invalid authentication method: digest." >>>> >>>> -- >>>> Tim A. Thompson >>>> Metadata Librarian >>>> Yale University Library >>>> >>>> On Wed, Feb 2, 2022 at 11:52 AM Christian Grün <christian.gruen at gmail.com> wrote: >>>>> >>>>> Yes, it should still be supported. I assume it doesn't make a difference if you use "Digest" or "digest"? >>>>> >>>>> >>>>> >>>>> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:48: >>>>>> >>>>>> Thanks, Christian. Right, that wasn't a working example. The server I'm querying is behind a firewall, so I can't really reproduce the issue unfortunately. >>>>>> >>>>>> The BaseX documentation states that the HTTP client does support Digest authentication. However, I did some poking around using Wireshark and limited knowledge of network traffic. When I run the request in curl, the expected protocol is played out, and I see the evidence in Wireshark: >>>>>> >>>>>> (1) The server responds with "401 Unauthorized" and provides the nonce value. >>>>>> (2) The client (curl) then does its business (supplies the username and password, resends the request, etc.) >>>>>> >>>>>> But when I submit the request via BaseX, it never gets past step (1). It never seems to send the username and password at all. Is Digest authentication still supported? >>>>>> >>>>>> Best, >>>>>> Tim >>>>>> >>>>>> >>>>>> -- >>>>>> Tim A. Thompson >>>>>> Metadata Librarian >>>>>> Yale University Library >>>>>> >>>>>> >>>>>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün <christian.gruen at gmail.com> wrote: >>>>>>> >>>>>>> Hi Tim, >>>>>>> >>>>>>> Difficult to tell; both the curl and the XQuery variants give me 404. >>>>>>> Do you think you could provide us with an example that works out of >>>>>>> the box? >>>>>>> >>>>>>> Best, >>>>>>> Christian >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> wrote: >>>>>>> > >>>>>>> > Hello, >>>>>>> > >>>>>>> > I'm trying to post a SPARQL query to an endpoint using Digest authentication with the HTTP client. >>>>>>> > >>>>>>> > The query works fine using curl: >>>>>>> > >>>>>>> > curl --digest --user user:pass -X POST -d@'test.rq' \ >>>>>>> > -H "Content-type: application/sparql-query" \ >>>>>>> > 'http://example.org/sparql' >>>>>>> > >>>>>>> > But the equivalent request in BaseX fails with 401 Unauthorized: >>>>>>> > >>>>>>> > let $endpoint := "http://example.org/sparql" >>>>>>> > let $user := "user" >>>>>>> > let $pass := "pass" >>>>>>> > let $type := "application/sparql-query" >>>>>>> > >>>>>>> > let $response := ( >>>>>>> > http:send-request( >>>>>>> > <http:request >>>>>>> > method="POST" >>>>>>> > href="{$endpoint}" >>>>>>> > username="{$user}" >>>>>>> > password="{$pass}" >>>>>>> > auth-method="Digest" >>>>>>> > send-authorization="true"> >>>>>>> > <http:header >>>>>>> > name="Content-Type" >>>>>>> > value="{$type}; charset=utf-8"/> >>>>>>> > <http:body >>>>>>> > media-type="{$type}">{ >>>>>>> > ``[ >>>>>>> > select * where {?s ?p ?o} limit 1 >>>>>>> > ]`` >>>>>>> > }</http:body> >>>>>>> > </http:request> >>>>>>> > ) >>>>>>> > ) >>>>>>> > return >>>>>>> > $response >>>>>>> > >>>>>>> > Any ideas about what might be causing the BaseX HTTP client to be denied here? >>>>>>> > >>>>>>> > Thanks in advance, >>>>>>> > Tim >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > Tim A. Thompson >>>>>>> > Metadata Librarian >>>>>>> > Yale University Library >>>>>>> >

On Wed, Feb 2, 2022 at 10:13 AM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Tim, > > Difficult to tell; both the curl and the XQuery variants give me 404. > Do you think you could provide us with an example that works out of > the box? > > Best, > Christian > > > > On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> wrote: > > > > Hello, > > > > I'm trying to post a SPARQL query to an endpoint using Digest > authentication with the HTTP client. > > > > The query works fine using curl: > > > > curl --digest --user user:pass -X POST -d@'test.rq' \ > > -H "Content-type: application/sparql-query" \ > > 'http://example.org/sparql' > > > > But the equivalent request in BaseX fails with 401 Unauthorized: > > > > let $endpoint := "http://example.org/sparql" > > let $user := "user" > > let $pass := "pass" > > let $type := "application/sparql-query" > > > > let $response := ( > > http:send-request( > > <http:request > > method="POST" > > href="{$endpoint}" > > username="{$user}" > > password="{$pass}" > > auth-method="Digest" > > send-authorization="true"> > > <http:header > > name="Content-Type" > > value="{$type}; charset=utf-8"/> > > <http:body > > media-type="{$type}">{ > > ``[ > > select * where {?s ?p ?o} limit 1 > > ]`` > > }</http:body> > > </http:request> > > ) > > ) > > return > > $response > > > > Any ideas about what might be causing the BaseX HTTP client to be denied > here? > > > > Thanks in advance, > > Tim > > > > > > -- > > Tim A. Thompson > > Metadata Librarian > > Yale University Library > > >

On Wed, Feb 2, 2022 at 11:52 AM Christian Grün <christian.gruen at gmail.com> wrote: > Yes, it should still be supported. I assume it doesn't make a difference > if you use "Digest" or "digest"? > > > > Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:48: > >> Thanks, Christian. Right, that wasn't a working example. The server I'm >> querying is behind a firewall, so I can't really reproduce the issue >> unfortunately. >> >> The BaseX documentation states that the HTTP client does support Digest >> authentication. However, I did some poking around using Wireshark and >> limited knowledge of network traffic. When I run the request in curl, the >> expected protocol is played out, and I see the evidence in Wireshark: >> >> (1) The server responds with "401 Unauthorized" and provides the nonce >> value. >> (2) The client (curl) then does its business (supplies the username and >> password, resends the request, etc.) >> >> But when I submit the request via BaseX, it never gets past step (1). It >> never seems to send the username and password at all. Is Digest >> authentication still supported? >> >> Best, >> Tim >> >> >> -- >> Tim A. Thompson >> Metadata Librarian >> Yale University Library >> >> >> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün <christian.gruen at gmail.com> >> wrote: >> >>> Hi Tim, >>> >>> Difficult to tell; both the curl and the XQuery variants give me 404. >>> Do you think you could provide us with an example that works out of >>> the box? >>> >>> Best, >>> Christian >>> >>> >>> >>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> wrote: >>> > >>> > Hello, >>> > >>> > I'm trying to post a SPARQL query to an endpoint using Digest >>> authentication with the HTTP client. >>> > >>> > The query works fine using curl: >>> > >>> > curl --digest --user user:pass -X POST -d@'test.rq' \ >>> > -H "Content-type: application/sparql-query" \ >>> > 'http://example.org/sparql' >>> > >>> > But the equivalent request in BaseX fails with 401 Unauthorized: >>> > >>> > let $endpoint := "http://example.org/sparql" >>> > let $user := "user" >>> > let $pass := "pass" >>> > let $type := "application/sparql-query" >>> > >>> > let $response := ( >>> > http:send-request( >>> > <http:request >>> > method="POST" >>> > href="{$endpoint}" >>> > username="{$user}" >>> > password="{$pass}" >>> > auth-method="Digest" >>> > send-authorization="true"> >>> > <http:header >>> > name="Content-Type" >>> > value="{$type}; charset=utf-8"/> >>> > <http:body >>> > media-type="{$type}">{ >>> > ``[ >>> > select * where {?s ?p ?o} limit 1 >>> > ]`` >>> > }</http:body> >>> > </http:request> >>> > ) >>> > ) >>> > return >>> > $response >>> > >>> > Any ideas about what might be causing the BaseX HTTP client to be >>> denied here? >>> > >>> > Thanks in advance, >>> > Tim >>> > >>> > >>> > -- >>> > Tim A. Thompson >>> > Metadata Librarian >>> > Yale University Library >>> > >>> >>

On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce <bunce.andy at gmail.com> wrote: > I find httpbin.org a useful resource to test this kind of thing [1] and > [2] > /Andy > > [1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_ > [2] http://httpbin.org/basic-auth/user1/mypass > > On Wed, 2 Feb 2022 at 16:58, Christian Grün <christian.gruen at gmail.com> > wrote: > >> Good to hear; so at least Digest is detected at this stage ;) >> >> >> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:54: >> >>> With "digest" I get: "[experr:HC0004] Invalid authentication method: >>> digest." >>> >>> -- >>> Tim A. Thompson >>> Metadata Librarian >>> Yale University Library >>> >>> On Wed, Feb 2, 2022 at 11:52 AM Christian Grün < >>> christian.gruen at gmail.com> wrote: >>> >>>> Yes, it should still be supported. I assume it doesn't make a >>>> difference if you use "Digest" or "digest"? >>>> >>>> >>>> >>>> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:48: >>>> >>>>> Thanks, Christian. Right, that wasn't a working example. The server >>>>> I'm querying is behind a firewall, so I can't really reproduce the issue >>>>> unfortunately. >>>>> >>>>> The BaseX documentation states that the HTTP client does support >>>>> Digest authentication. However, I did some poking around using Wireshark >>>>> and limited knowledge of network traffic. When I run the request in curl, >>>>> the expected protocol is played out, and I see the evidence in Wireshark: >>>>> >>>>> (1) The server responds with "401 Unauthorized" and provides the nonce >>>>> value. >>>>> (2) The client (curl) then does its business (supplies the username >>>>> and password, resends the request, etc.) >>>>> >>>>> But when I submit the request via BaseX, it never gets past step (1). >>>>> It never seems to send the username and password at all. Is Digest >>>>> authentication still supported? >>>>> >>>>> Best, >>>>> Tim >>>>> >>>>> >>>>> -- >>>>> Tim A. Thompson >>>>> Metadata Librarian >>>>> Yale University Library >>>>> >>>>> >>>>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün < >>>>> christian.gruen at gmail.com> wrote: >>>>> >>>>>> Hi Tim, >>>>>> >>>>>> Difficult to tell; both the curl and the XQuery variants give me 404. >>>>>> Do you think you could provide us with an example that works out of >>>>>> the box? >>>>>> >>>>>> Best, >>>>>> Christian >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> >>>>>> wrote: >>>>>> > >>>>>> > Hello, >>>>>> > >>>>>> > I'm trying to post a SPARQL query to an endpoint using Digest >>>>>> authentication with the HTTP client. >>>>>> > >>>>>> > The query works fine using curl: >>>>>> > >>>>>> > curl --digest --user user:pass -X POST -d@'test.rq' \ >>>>>> > -H "Content-type: application/sparql-query" \ >>>>>> > 'http://example.org/sparql' >>>>>> > >>>>>> > But the equivalent request in BaseX fails with 401 Unauthorized: >>>>>> > >>>>>> > let $endpoint := "http://example.org/sparql" >>>>>> > let $user := "user" >>>>>> > let $pass := "pass" >>>>>> > let $type := "application/sparql-query" >>>>>> > >>>>>> > let $response := ( >>>>>> > http:send-request( >>>>>> > <http:request >>>>>> > method="POST" >>>>>> > href="{$endpoint}" >>>>>> > username="{$user}" >>>>>> > password="{$pass}" >>>>>> > auth-method="Digest" >>>>>> > send-authorization="true"> >>>>>> > <http:header >>>>>> > name="Content-Type" >>>>>> > value="{$type}; charset=utf-8"/> >>>>>> > <http:body >>>>>> > media-type="{$type}">{ >>>>>> > ``[ >>>>>> > select * where {?s ?p ?o} limit 1 >>>>>> > ]`` >>>>>> > }</http:body> >>>>>> > </http:request> >>>>>> > ) >>>>>> > ) >>>>>> > return >>>>>> > $response >>>>>> > >>>>>> > Any ideas about what might be causing the BaseX HTTP client to be >>>>>> denied here? >>>>>> > >>>>>> > Thanks in advance, >>>>>> > Tim >>>>>> > >>>>>> > >>>>>> > -- >>>>>> > Tim A. Thompson >>>>>> > Metadata Librarian >>>>>> > Yale University Library >>>>>> > >>>>>> >>>>>

> I notice that curl doesn't include the Authorization header on the first request, but I am out of my depth here.

From: BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on behalf of Tim Thompson <timathom at gmail.com> Sent: Wednesday, February 2, 2022 1:59 PM To: Christian Grün <christian.gruen at gmail.com>; BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Issue with HTTP client and authentication Ah! Thanks for clarifying that. So, now it is working for GET requests in BaseX, but still not for POST. So, this works: http:send-request( <http:request method="GET" href="{$endpoint}?query={encode-for-uri('select * where {?s ?p ?o} limit 1')}" username="{$user}" password="{$pass}" auth-method="Digest"> </http:request> ) But when I try to POST a request body with Digest auth, it is still failing (httpbin doesn't seem to support testing Digest with POST, unfortunately). The request hangs for a while, then fails with 401 Unauthorized. Wireshark reveals that the first network response from the server is actually a 502 Bad Gateway. Attached are two screenshots from Wireshark: first, the request/response sequence with curl, then with BaseX. Here's what Wireshark shows for the first request via curl: Hypertext Transfer Protocol POST /v1/graphs/sparql HTTP/1.1\r\n [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n] [POST /v1/graphs/sparql HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: POST Request URI: /v1/graphs/sparql Request Version: HTTP/1.1 Host: example.org<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D&reserved=0>\r\n User-Agent: curl/7.71.1\r\n Accept: */*\r\n Content-type: application/sparql-query\r\n Content-Length: 0\r\n [Content length: 0] \r\n [Full request URI: example.org/v1/graphs/sparql<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D&reserved=0>] Then via BaseX: Hypertext Transfer Protocol POST /v1/graphs/sparql HTTP/1.1\r\n [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n] [POST /v1/graphs/sparql HTTP/1.1\r\n] [Severity level: Chat] [Group: Sequence] Request Method: POST Request URI: /v1/graphs/sparql Request Version: HTTP/1.1 Content-type: application/sparql-query\r\n Authorization: Digest\r\n User-Agent: Java/1.8.0_121\r\n Host: example.org<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D&reserved=0>\r\n Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n Connection: keep-alive\r\n \r\n [Full request URI: example.org/v1/graphs/sparql<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D&reserved=0>] I notice that curl doesn't include the Authorization header on the first request, but I am out of my depth here. Best, Tim -- Tim A. Thompson Metadata Librarian Yale University Library On Wed, Feb 2, 2022 at 12:40 PM Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: Hi Tim, Your request should be successful if you omit the send-authorization attribute: “If send-authorization is true (default value is false) and the authentication method supports generating the header Authorization without challenge, the request contains this header. The default value is to send a non-authenticated request, and if the response is an authentication challenge, then only send the credentials in a second message.” [1] If you use Basic authentication, you can reduce the communication to a single request by sending the authorization with the first request. With Digest authentication, there will always be two requests. With the (still to be finalized) HTTP Client module 2.0, send-authorization will disappear, and it will be up to the implementation to decide how many requests will be sent [2]. Hope this helps, Christian [1] http://expath.org/spec/http-client#d2e430<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexpath.org%2Fspec%2Fhttp-client%23d2e430&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RREH6BeyWOV8juK%2Brf0UuofvBiC2JOv7B1%2BWXLYCjUQ%3D&reserved=0> [2] https://expath.github.io/expath-cg/specs/http-client-2/<https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fexpath.github.io%2Fexpath-cg%2Fspecs%2Fhttp-client-2%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=owfYuuSXSpkkPTnQhBIEJ1wBcKPCs4oFnaaVEvndvMA%3D&reserved=0> On Wed, Feb 2, 2022 at 6:19 PM Tim Thompson <timathom at gmail.com<mailto:timathom at gmail.com>> wrote: > > Thanks, Andy! That's very handy. So, when I test against httpbin, the same thing happens. > > curl request: > curl --digest --user user:pw -X GET 'http://httpbin.org/digest-auth/auth/user/pw<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fdigest-auth%2Fauth%2Fuser%2Fpw&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SWqtnqsC98VlfYr7TeIUggFdoR7dJDZtkNJxz9051jw%3D&reserved=0>' > > response: > { > "authenticated": true, > "user": "user" > } > > BaseX request: > let $endpoint := "http://httpbin.org/digest-auth/auth/user/pw<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fdigest-auth%2Fauth%2Fuser%2Fpw&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SWqtnqsC98VlfYr7TeIUggFdoR7dJDZtkNJxz9051jw%3D&reserved=0>" > > let $response := ( > http:send-request( > <http:request > method="GET" > href="{$endpoint}" > username="user" > password="pw" > auth-method="Digest" > send-authorization="true"> > <http:header > name="Content-Type" > value="{$type}; charset=utf-8"/> > </http:request> > ) > ) > return > $response > > response: > <http:response xmlns:http="http://expath.org/ns/http-client<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexpath.org%2Fns%2Fhttp-client&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FNS%2BIceZOx6yP%2Fp4ikxkNqAhYq44F49nUaE3u8iijr0%3D&reserved=0>" status="401" message="UNAUTHORIZED"> > <http:header name="Server" value="gunicorn/19.9.0"/> > <http:header name="Access-Control-Allow-Origin" value="*"/> > <http:header name="Access-Control-Allow-Credentials" value="true"/> > <http:header name="WWW-Authenticate" value="Digest realm=&quot;me at kennethreitz.com<mailto:quot%3Bme at kennethreitz.com>&quot;, nonce=&quot;36d59385e96e1689595f4f14fa2921ca&quot;, qop=&quot;auth&quot;, opaque=&quot;a21df3287b5cb9cbf8b6a68b2c15fa1d&quot;, algorithm=MD5, stale=FALSE"/> > <http:header name="Connection" value="keep-alive"/> > <http:header name="Set-Cookie" value="fake=fake_value; Path=/"/> > <http:header name="Set-Cookie" value="stale_after=never; Path=/"/> > <http:header name="Content-Length" value="0"/> > <http:header name="Date" value="Wed, 02 Feb 2022 17:14:13 GMT"/> > <http:header name="Content-Type" value="text/html; charset=utf-8"/> > </http:response> > > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library > > > On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce <bunce.andy at gmail.com<mailto:bunce.andy at gmail.com>> wrote: >> >> I find httpbin.org<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5ZA6A7rJL%2B2YD%2BPvtA9wtLUjJVf3EAZw3Fb5ifo4H%2BU%3D&reserved=0> a useful resource to test this kind of thing [1] and [2] >> /Andy >> >> [1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2F%23%2FAuth%2Fget_basic_auth__user___passwd_&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TU9Itw0Bs4Wbk51bnStj2oFqWMq%2BrdIOxgUwq5fFFrM%3D&reserved=0> >> [2] http://httpbin.org/basic-auth/user1/mypass<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fbasic-auth%2Fuser1%2Fmypass&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lH8HeRKkVu%2Fev4Er8lokTCZ0xqyS3HOYtBBqgTSw0PY%3D&reserved=0> >> >> On Wed, 2 Feb 2022 at 16:58, Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: >>> >>> Good to hear; so at least Digest is detected at this stage ;) >>> >>> >>> Tim Thompson <timathom at gmail.com<mailto:timathom at gmail.com>> schrieb am Mi., 2. Feb. 2022, 17:54: >>>> >>>> With "digest" I get: "[experr:HC0004] Invalid authentication method: digest." >>>> >>>> -- >>>> Tim A. Thompson >>>> Metadata Librarian >>>> Yale University Library >>>> >>>> On Wed, Feb 2, 2022 at 11:52 AM Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: >>>>> >>>>> Yes, it should still be supported. I assume it doesn't make a difference if you use "Digest" or "digest"? >>>>> >>>>> >>>>> >>>>> Tim Thompson <timathom at gmail.com<mailto:timathom at gmail.com>> schrieb am Mi., 2. Feb. 2022, 17:48: >>>>>> >>>>>> Thanks, Christian. Right, that wasn't a working example. The server I'm querying is behind a firewall, so I can't really reproduce the issue unfortunately. >>>>>> >>>>>> The BaseX documentation states that the HTTP client does support Digest authentication. However, I did some poking around using Wireshark and limited knowledge of network traffic. When I run the request in curl, the expected protocol is played out, and I see the evidence in Wireshark: >>>>>> >>>>>> (1) The server responds with "401 Unauthorized" and provides the nonce value. >>>>>> (2) The client (curl) then does its business (supplies the username and password, resends the request, etc.) >>>>>> >>>>>> But when I submit the request via BaseX, it never gets past step (1). It never seems to send the username and password at all. Is Digest authentication still supported? >>>>>> >>>>>> Best, >>>>>> Tim >>>>>> >>>>>> >>>>>> -- >>>>>> Tim A. Thompson >>>>>> Metadata Librarian >>>>>> Yale University Library >>>>>> >>>>>> >>>>>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: >>>>>>> >>>>>>> Hi Tim, >>>>>>> >>>>>>> Difficult to tell; both the curl and the XQuery variants give me 404. >>>>>>> Do you think you could provide us with an example that works out of >>>>>>> the box? >>>>>>> >>>>>>> Best, >>>>>>> Christian >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com<mailto:timathom at gmail.com>> wrote: >>>>>>> > >>>>>>> > Hello, >>>>>>> > >>>>>>> > I'm trying to post a SPARQL query to an endpoint using Digest authentication with the HTTP client. >>>>>>> > >>>>>>> > The query works fine using curl: >>>>>>> > >>>>>>> > curl --digest --user user:pass -X POST -d@'test.rq' \ >>>>>>> > -H "Content-type: application/sparql-query" \ >>>>>>> > 'http://example.org/sparql<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8uhboyxkhGWHk8HbZRKgDXOAbDRa2UwIigI1H5vWxt0%3D&reserved=0>' >>>>>>> > >>>>>>> > But the equivalent request in BaseX fails with 401 Unauthorized: >>>>>>> > >>>>>>> > let $endpoint := "http://example.org/sparql<https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8uhboyxkhGWHk8HbZRKgDXOAbDRa2UwIigI1H5vWxt0%3D&reserved=0>" >>>>>>> > let $user := "user" >>>>>>> > let $pass := "pass" >>>>>>> > let $type := "application/sparql-query" >>>>>>> > >>>>>>> > let $response := ( >>>>>>> > http:send-request( >>>>>>> > <http:request >>>>>>> > method="POST" >>>>>>> > href="{$endpoint}" >>>>>>> > username="{$user}" >>>>>>> > password="{$pass}" >>>>>>> > auth-method="Digest" >>>>>>> > send-authorization="true"> >>>>>>> > <http:header >>>>>>> > name="Content-Type" >>>>>>> > value="{$type}; charset=utf-8"/> >>>>>>> > <http:body >>>>>>> > media-type="{$type}">{ >>>>>>> > ``[ >>>>>>> > select * where {?s ?p ?o} limit 1 >>>>>>> > ]`` >>>>>>> > }</http:body> >>>>>>> > </http:request> >>>>>>> > ) >>>>>>> > ) >>>>>>> > return >>>>>>> > $response >>>>>>> > >>>>>>> > Any ideas about what might be causing the BaseX HTTP client to be denied here? >>>>>>> > >>>>>>> > Thanks in advance, >>>>>>> > Tim >>>>>>> > >>>>>>> > >>>>>>> > -- >>>>>>> > Tim A. Thompson >>>>>>> > Metadata Librarian >>>>>>> > Yale University Library >>>>>>> >

On Wed, Feb 2, 2022 at 12:40 PM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Tim, > > Your request should be successful if you omit the send-authorization > attribute: > > “If send-authorization is true (default value is false) and the > authentication method supports generating the header Authorization > without challenge, the request contains this header. The default value > is to send a non-authenticated request, and if the response is an > authentication challenge, then only send the credentials in a second > message.” [1] > > If you use Basic authentication, you can reduce the communication to a > single request by sending the authorization with the first request. > With Digest authentication, there will always be two requests. > > With the (still to be finalized) HTTP Client module 2.0, > send-authorization will disappear, and it will be up to the > implementation to decide how many requests will be sent [2]. > > Hope this helps, > Christian > > [1] http://expath.org/spec/http-client#d2e430 > [2] https://expath.github.io/expath-cg/specs/http-client-2/ > > > > On Wed, Feb 2, 2022 at 6:19 PM Tim Thompson <timathom at gmail.com> wrote: > > > > Thanks, Andy! That's very handy. So, when I test against httpbin, the > same thing happens. > > > > curl request: > > curl --digest --user user:pw -X GET ' > http://httpbin.org/digest-auth/auth/user/pw' > > > > response: > > { > > "authenticated": true, > > "user": "user" > > } > > > > BaseX request: > > let $endpoint := "http://httpbin.org/digest-auth/auth/user/pw" > > > > let $response := ( > > http:send-request( > > <http:request > > method="GET" > > href="{$endpoint}" > > username="user" > > password="pw" > > auth-method="Digest" > > send-authorization="true"> > > <http:header > > name="Content-Type" > > value="{$type}; charset=utf-8"/> > > </http:request> > > ) > > ) > > return > > $response > > > > response: > > <http:response xmlns:http="http://expath.org/ns/http-client" > status="401" message="UNAUTHORIZED"> > > <http:header name="Server" value="gunicorn/19.9.0"/> > > <http:header name="Access-Control-Allow-Origin" value="*"/> > > <http:header name="Access-Control-Allow-Credentials" value="true"/> > > <http:header name="WWW-Authenticate" value="Digest realm=& > quot;me at kennethreitz.com&quot;, > nonce=&quot;36d59385e96e1689595f4f14fa2921ca&quot;, qop=&quot;auth&quot;, > opaque=&quot;a21df3287b5cb9cbf8b6a68b2c15fa1d&quot;, algorithm=MD5, > stale=FALSE"/> > > <http:header name="Connection" value="keep-alive"/> > > <http:header name="Set-Cookie" value="fake=fake_value; Path=/"/> > > <http:header name="Set-Cookie" value="stale_after=never; Path=/"/> > > <http:header name="Content-Length" value="0"/> > > <http:header name="Date" value="Wed, 02 Feb 2022 17:14:13 GMT"/> > > <http:header name="Content-Type" value="text/html; charset=utf-8"/> > > </http:response> > > > > > > -- > > Tim A. Thompson > > Metadata Librarian > > Yale University Library > > > > > > On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce <bunce.andy at gmail.com> wrote: > >> > >> I find httpbin.org a useful resource to test this kind of thing [1] > and [2] > >> /Andy > >> > >> [1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_ > >> [2] http://httpbin.org/basic-auth/user1/mypass > >> > >> On Wed, 2 Feb 2022 at 16:58, Christian Grün <christian.gruen at gmail.com> > wrote: > >>> > >>> Good to hear; so at least Digest is detected at this stage ;) > >>> > >>> > >>> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:54: > >>>> > >>>> With "digest" I get: "[experr:HC0004] Invalid authentication method: > digest." > >>>> > >>>> -- > >>>> Tim A. Thompson > >>>> Metadata Librarian > >>>> Yale University Library > >>>> > >>>> On Wed, Feb 2, 2022 at 11:52 AM Christian Grün < > christian.gruen at gmail.com> wrote: > >>>>> > >>>>> Yes, it should still be supported. I assume it doesn't make a > difference if you use "Digest" or "digest"? > >>>>> > >>>>> > >>>>> > >>>>> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, > 17:48: > >>>>>> > >>>>>> Thanks, Christian. Right, that wasn't a working example. The server > I'm querying is behind a firewall, so I can't really reproduce the issue > unfortunately. > >>>>>> > >>>>>> The BaseX documentation states that the HTTP client does support > Digest authentication. However, I did some poking around using Wireshark > and limited knowledge of network traffic. When I run the request in curl, > the expected protocol is played out, and I see the evidence in Wireshark: > >>>>>> > >>>>>> (1) The server responds with "401 Unauthorized" and provides the > nonce value. > >>>>>> (2) The client (curl) then does its business (supplies the username > and password, resends the request, etc.) > >>>>>> > >>>>>> But when I submit the request via BaseX, it never gets past step > (1). It never seems to send the username and password at all. Is Digest > authentication still supported? > >>>>>> > >>>>>> Best, > >>>>>> Tim > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Tim A. Thompson > >>>>>> Metadata Librarian > >>>>>> Yale University Library > >>>>>> > >>>>>> > >>>>>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün < > christian.gruen at gmail.com> wrote: > >>>>>>> > >>>>>>> Hi Tim, > >>>>>>> > >>>>>>> Difficult to tell; both the curl and the XQuery variants give me > 404. > >>>>>>> Do you think you could provide us with an example that works out of > >>>>>>> the box? > >>>>>>> > >>>>>>> Best, > >>>>>>> Christian > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> > wrote: > >>>>>>> > > >>>>>>> > Hello, > >>>>>>> > > >>>>>>> > I'm trying to post a SPARQL query to an endpoint using Digest > authentication with the HTTP client. > >>>>>>> > > >>>>>>> > The query works fine using curl: > >>>>>>> > > >>>>>>> > curl --digest --user user:pass -X POST -d@'test.rq' \ > >>>>>>> > -H "Content-type: application/sparql-query" \ > >>>>>>> > 'http://example.org/sparql' > >>>>>>> > > >>>>>>> > But the equivalent request in BaseX fails with 401 Unauthorized: > >>>>>>> > > >>>>>>> > let $endpoint := "http://example.org/sparql" > >>>>>>> > let $user := "user" > >>>>>>> > let $pass := "pass" > >>>>>>> > let $type := "application/sparql-query" > >>>>>>> > > >>>>>>> > let $response := ( > >>>>>>> > http:send-request( > >>>>>>> > <http:request > >>>>>>> > method="POST" > >>>>>>> > href="{$endpoint}" > >>>>>>> > username="{$user}" > >>>>>>> > password="{$pass}" > >>>>>>> > auth-method="Digest" > >>>>>>> > send-authorization="true"> > >>>>>>> > <http:header > >>>>>>> > name="Content-Type" > >>>>>>> > value="{$type}; charset=utf-8"/> > >>>>>>> > <http:body > >>>>>>> > media-type="{$type}">{ > >>>>>>> > ``[ > >>>>>>> > select * where {?s ?p ?o} limit 1 > >>>>>>> > ]`` > >>>>>>> > }</http:body> > >>>>>>> > </http:request> > >>>>>>> > ) > >>>>>>> > ) > >>>>>>> > return > >>>>>>> > $response > >>>>>>> > > >>>>>>> > Any ideas about what might be causing the BaseX HTTP client to > be denied here? > >>>>>>> > > >>>>>>> > Thanks in advance, > >>>>>>> > Tim > >>>>>>> > > >>>>>>> > > >>>>>>> > -- > >>>>>>> > Tim A. Thompson > >>>>>>> > Metadata Librarian > >>>>>>> > Yale University Library > >>>>>>> > >

On 02.02.2022 21:22, Tim Thompson wrote: > Thanks for that. Still no luck, I'm afraid. Attached is the full HTTP > info for the requests from BaseX (hostname info replaced with > "example.org <http://example.org>"). > > I should be able to stick with GET requests for now, but would be nice > to have access to POST as well. > > All best, > Tim > >

Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 21:22: > Thanks for that. Still no luck, I'm afraid. Attached is the full HTTP info > for the requests from BaseX (hostname info replaced with "example.org"). > > I should be able to stick with GET requests for now, but would be nice to > have access to POST as well. > > All best, > Tim > > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library > > > On Wed, Feb 2, 2022 at 2:20 PM Christian Grün <christian.gruen at gmail.com> > wrote: > >> > I notice that curl doesn't include the Authorization header on the >> first request, but I am out of my depth here. >> >> That could be an interesting hint. In the latest snapshot [1], I have >> removed the Authorization header from the first request. Can you check >> if it makes a difference? >> >> If not, could you share the headers of the second requests with us? >> >> [1] https://files.basex.org/releases/latest/ >> >

On Wed, Feb 2, 2022 at 2:20 PM Christian Grün <christian.gruen at gmail.com> wrote: > > I notice that curl doesn't include the Authorization header on the first > request, but I am out of my depth here. > > That could be an interesting hint. In the latest snapshot [1], I have > removed the Authorization header from the first request. Can you check > if it makes a difference? > > If not, could you share the headers of the second requests with us? > > [1] https://files.basex.org/releases/latest/ >

On Wed, Feb 2, 2022 at 2:21 PM Clark, Ash <as.clark at northeastern.edu> wrote: > In case it helps: I had a similar problem in eXist some years ago. My fix > was to put the attribute `http-version="1.0"` on the request element. The > HTTP client defaults to HTTP v1.1, but for whatever reason, only HTTP 1.0 > worked for me. > > ~Ash > ------------------------------ > *From:* BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on behalf > of Tim Thompson <timathom at gmail.com> > *Sent:* Wednesday, February 2, 2022 1:59 PM > *To:* Christian Grün <christian.gruen at gmail.com>; BaseX < > basex-talk at mailman.uni-konstanz.de> > *Subject:* Re: [basex-talk] Issue with HTTP client and authentication > > Ah! Thanks for clarifying that. So, now it is working for GET requests in > BaseX, but still not for POST. > > So, this works: > http:send-request( > <http:request > method="GET" > href="{$endpoint}?query={encode-for-uri('select * where {?s ?p ?o} > limit 1')}" > username="{$user}" > password="{$pass}" > auth-method="Digest"> > </http:request> > ) > > But when I try to POST a request body with Digest auth, it is still > failing (httpbin doesn't seem to support testing Digest with POST, > unfortunately). The request hangs for a while, then fails with 401 > Unauthorized. Wireshark reveals that the first network response from the > server is actually a 502 Bad Gateway. > > Attached are two screenshots from Wireshark: first, the request/response > sequence with curl, then with BaseX. > > Here's what Wireshark shows for the first request via curl: > > Hypertext Transfer Protocol > POST /v1/graphs/sparql HTTP/1.1\r\n > [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n] > [POST /v1/graphs/sparql HTTP/1.1\r\n] > [Severity level: Chat] > [Group: Sequence] > Request Method: POST > Request URI: /v1/graphs/sparql > Request Version: HTTP/1.1 > Host: example.org > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D&reserved=0> > \r\n > User-Agent: curl/7.71.1\r\n > Accept: */*\r\n > Content-type: application/sparql-query\r\n > Content-Length: 0\r\n > [Content length: 0] > \r\n > [Full request URI: example.org/v1/graphs/sparql > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D&reserved=0> > ] > > Then via BaseX: > > Hypertext Transfer Protocol > POST /v1/graphs/sparql HTTP/1.1\r\n > [Expert Info (Chat/Sequence): POST /v1/graphs/sparql HTTP/1.1\r\n] > [POST /v1/graphs/sparql HTTP/1.1\r\n] > [Severity level: Chat] > [Group: Sequence] > Request Method: POST > Request URI: /v1/graphs/sparql > Request Version: HTTP/1.1 > Content-type: application/sparql-query\r\n > Authorization: Digest\r\n > User-Agent: Java/1.8.0_121\r\n > Host: example.org > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=j3Wipf2PLgF3njCsb6BTN1J%2BwrOOfe1%2Fcm%2F%2BUiGTMQk%3D&reserved=0> > \r\n > Accept: text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2\r\n > Connection: keep-alive\r\n > \r\n > [Full request URI: example.org/v1/graphs/sparql > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fv1%2Fgraphs%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=PeCGztKkndWthFeKyUrYUljM2oq4ivkyP1ugAYL64QU%3D&reserved=0> > ] > > I notice that curl doesn't include the Authorization header on the first > request, but I am out of my depth here. > > Best, > Tim > > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library > > > On Wed, Feb 2, 2022 at 12:40 PM Christian Grün <christian.gruen at gmail.com> > wrote: > > Hi Tim, > > Your request should be successful if you omit the send-authorization > attribute: > > “If send-authorization is true (default value is false) and the > authentication method supports generating the header Authorization > without challenge, the request contains this header. The default value > is to send a non-authenticated request, and if the response is an > authentication challenge, then only send the credentials in a second > message.” [1] > > If you use Basic authentication, you can reduce the communication to a > single request by sending the authorization with the first request. > With Digest authentication, there will always be two requests. > > With the (still to be finalized) HTTP Client module 2.0, > send-authorization will disappear, and it will be up to the > implementation to decide how many requests will be sent [2]. > > Hope this helps, > Christian > > [1] http://expath.org/spec/http-client#d2e430 > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexpath.org%2Fspec%2Fhttp-client%23d2e430&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=RREH6BeyWOV8juK%2Brf0UuofvBiC2JOv7B1%2BWXLYCjUQ%3D&reserved=0> > [2] https://expath.github.io/expath-cg/specs/http-client-2/ > <https://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fexpath.github.io%2Fexpath-cg%2Fspecs%2Fhttp-client-2%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=owfYuuSXSpkkPTnQhBIEJ1wBcKPCs4oFnaaVEvndvMA%3D&reserved=0> > > > > On Wed, Feb 2, 2022 at 6:19 PM Tim Thompson <timathom at gmail.com> wrote: > > > > Thanks, Andy! That's very handy. So, when I test against httpbin, the > same thing happens. > > > > curl request: > > curl --digest --user user:pw -X GET ' > http://httpbin.org/digest-auth/auth/user/pw > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fdigest-auth%2Fauth%2Fuser%2Fpw&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SWqtnqsC98VlfYr7TeIUggFdoR7dJDZtkNJxz9051jw%3D&reserved=0> > ' > > > > response: > > { > > "authenticated": true, > > "user": "user" > > } > > > > BaseX request: > > let $endpoint := "http://httpbin.org/digest-auth/auth/user/pw > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fdigest-auth%2Fauth%2Fuser%2Fpw&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=SWqtnqsC98VlfYr7TeIUggFdoR7dJDZtkNJxz9051jw%3D&reserved=0> > " > > > > let $response := ( > > http:send-request( > > <http:request > > method="GET" > > href="{$endpoint}" > > username="user" > > password="pw" > > auth-method="Digest" > > send-authorization="true"> > > <http:header > > name="Content-Type" > > value="{$type}; charset=utf-8"/> > > </http:request> > > ) > > ) > > return > > $response > > > > response: > > <http:response xmlns:http="http://expath.org/ns/http-client > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexpath.org%2Fns%2Fhttp-client&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2FNS%2BIceZOx6yP%2Fp4ikxkNqAhYq44F49nUaE3u8iijr0%3D&reserved=0>" > status="401" message="UNAUTHORIZED"> > > <http:header name="Server" value="gunicorn/19.9.0"/> > > <http:header name="Access-Control-Allow-Origin" value="*"/> > > <http:header name="Access-Control-Allow-Credentials" value="true"/> > > <http:header name="WWW-Authenticate" value="Digest realm=& > quot;me at kennethreitz.com&quot;, > nonce=&quot;36d59385e96e1689595f4f14fa2921ca&quot;, qop=&quot;auth&quot;, > opaque=&quot;a21df3287b5cb9cbf8b6a68b2c15fa1d&quot;, algorithm=MD5, > stale=FALSE"/> > > <http:header name="Connection" value="keep-alive"/> > > <http:header name="Set-Cookie" value="fake=fake_value; Path=/"/> > > <http:header name="Set-Cookie" value="stale_after=never; Path=/"/> > > <http:header name="Content-Length" value="0"/> > > <http:header name="Date" value="Wed, 02 Feb 2022 17:14:13 GMT"/> > > <http:header name="Content-Type" value="text/html; charset=utf-8"/> > > </http:response> > > > > > > -- > > Tim A. Thompson > > Metadata Librarian > > Yale University Library > > > > > > On Wed, Feb 2, 2022 at 12:06 PM Andy Bunce <bunce.andy at gmail.com> wrote: > >> > >> I find httpbin.org > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2F&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=5ZA6A7rJL%2B2YD%2BPvtA9wtLUjJVf3EAZw3Fb5ifo4H%2BU%3D&reserved=0> > a useful resource to test this kind of thing [1] and [2] > >> /Andy > >> > >> [1] http://httpbin.org/#/Auth/get_basic_auth__user___passwd_ > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2F%23%2FAuth%2Fget_basic_auth__user___passwd_&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=TU9Itw0Bs4Wbk51bnStj2oFqWMq%2BrdIOxgUwq5fFFrM%3D&reserved=0> > >> [2] http://httpbin.org/basic-auth/user1/mypass > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fhttpbin.org%2Fbasic-auth%2Fuser1%2Fmypass&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=lH8HeRKkVu%2Fev4Er8lokTCZ0xqyS3HOYtBBqgTSw0PY%3D&reserved=0> > >> > >> On Wed, 2 Feb 2022 at 16:58, Christian Grün <christian.gruen at gmail.com> > wrote: > >>> > >>> Good to hear; so at least Digest is detected at this stage ;) > >>> > >>> > >>> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 17:54: > >>>> > >>>> With "digest" I get: "[experr:HC0004] Invalid authentication method: > digest." > >>>> > >>>> -- > >>>> Tim A. Thompson > >>>> Metadata Librarian > >>>> Yale University Library > >>>> > >>>> On Wed, Feb 2, 2022 at 11:52 AM Christian Grün < > christian.gruen at gmail.com> wrote: > >>>>> > >>>>> Yes, it should still be supported. I assume it doesn't make a > difference if you use "Digest" or "digest"? > >>>>> > >>>>> > >>>>> > >>>>> Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, > 17:48: > >>>>>> > >>>>>> Thanks, Christian. Right, that wasn't a working example. The server > I'm querying is behind a firewall, so I can't really reproduce the issue > unfortunately. > >>>>>> > >>>>>> The BaseX documentation states that the HTTP client does support > Digest authentication. However, I did some poking around using Wireshark > and limited knowledge of network traffic. When I run the request in curl, > the expected protocol is played out, and I see the evidence in Wireshark: > >>>>>> > >>>>>> (1) The server responds with "401 Unauthorized" and provides the > nonce value. > >>>>>> (2) The client (curl) then does its business (supplies the username > and password, resends the request, etc.) > >>>>>> > >>>>>> But when I submit the request via BaseX, it never gets past step > (1). It never seems to send the username and password at all. Is Digest > authentication still supported? > >>>>>> > >>>>>> Best, > >>>>>> Tim > >>>>>> > >>>>>> > >>>>>> -- > >>>>>> Tim A. Thompson > >>>>>> Metadata Librarian > >>>>>> Yale University Library > >>>>>> > >>>>>> > >>>>>> On Wed, Feb 2, 2022 at 10:13 AM Christian Grün < > christian.gruen at gmail.com> wrote: > >>>>>>> > >>>>>>> Hi Tim, > >>>>>>> > >>>>>>> Difficult to tell; both the curl and the XQuery variants give me > 404. > >>>>>>> Do you think you could provide us with an example that works out of > >>>>>>> the box? > >>>>>>> > >>>>>>> Best, > >>>>>>> Christian > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Wed, Feb 2, 2022 at 12:54 AM Tim Thompson <timathom at gmail.com> > wrote: > >>>>>>> > > >>>>>>> > Hello, > >>>>>>> > > >>>>>>> > I'm trying to post a SPARQL query to an endpoint using Digest > authentication with the HTTP client. > >>>>>>> > > >>>>>>> > The query works fine using curl: > >>>>>>> > > >>>>>>> > curl --digest --user user:pass -X POST -d@'test.rq' \ > >>>>>>> > -H "Content-type: application/sparql-query" \ > >>>>>>> > 'http://example.org/sparql > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8uhboyxkhGWHk8HbZRKgDXOAbDRa2UwIigI1H5vWxt0%3D&reserved=0> > ' > >>>>>>> > > >>>>>>> > But the equivalent request in BaseX fails with 401 Unauthorized: > >>>>>>> > > >>>>>>> > let $endpoint := "http://example.org/sparql > <https://nam12.safelinks.protection.outlook.com/?url=http%3A%2F%2Fexample.org%2Fsparql&data=04%7C01%7Cas.clark%40northeastern.edu%7Cdf41115443c24ad1a67a08d9e67e4565%7Ca8eec281aaa34daeac9b9a398b9215e7%7C0%7C0%7C637794253163659084%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=8uhboyxkhGWHk8HbZRKgDXOAbDRa2UwIigI1H5vWxt0%3D&reserved=0> > " > >>>>>>> > let $user := "user" > >>>>>>> > let $pass := "pass" > >>>>>>> > let $type := "application/sparql-query" > >>>>>>> > > >>>>>>> > let $response := ( > >>>>>>> > http:send-request( > >>>>>>> > <http:request > >>>>>>> > method="POST" > >>>>>>> > href="{$endpoint}" > >>>>>>> > username="{$user}" > >>>>>>> > password="{$pass}" > >>>>>>> > auth-method="Digest" > >>>>>>> > send-authorization="true"> > >>>>>>> > <http:header > >>>>>>> > name="Content-Type" > >>>>>>> > value="{$type}; charset=utf-8"/> > >>>>>>> > <http:body > >>>>>>> > media-type="{$type}">{ > >>>>>>> > ``[ > >>>>>>> > select * where {?s ?p ?o} limit 1 > >>>>>>> > ]`` > >>>>>>> > }</http:body> > >>>>>>> > </http:request> > >>>>>>> > ) > >>>>>>> > ) > >>>>>>> > return > >>>>>>> > $response > >>>>>>> > > >>>>>>> > Any ideas about what might be causing the BaseX HTTP client to > be denied here? > >>>>>>> > > >>>>>>> > Thanks in advance, > >>>>>>> > Tim > >>>>>>> > > >>>>>>> > > >>>>>>> > -- > >>>>>>> > Tim A. Thompson > >>>>>>> > Metadata Librarian > >>>>>>> > Yale University Library > >>>>>>> > > >

On Wed, Feb 2, 2022 at 3:38 PM Christian Grün <christian.gruen at gmail.com> wrote: > Dommage. > > Do you know more about the server you are communicating with? Maybe it > rejects http (POST?) requests sent by Java? > > You could try to supply the curl-specific request headers (such as > User-Agent, and others) with http:send-request. > > Could you additionally forward the full Wireshark logs for curl? > > I remember that Adam (Retter) added the http-version option to the new > version of the HTTP Client Module (it’s not available in 1.0 yet). In the > given case, I’d assume that http-version=1.1 should be okay, because curl > uses it as well. > > > > Tim Thompson <timathom at gmail.com> schrieb am Mi., 2. Feb. 2022, 21:22: > >> Thanks for that. Still no luck, I'm afraid. Attached is the full HTTP >> info for the requests from BaseX (hostname info replaced with " >> example.org"). >> >> I should be able to stick with GET requests for now, but would be nice to >> have access to POST as well. >> >> All best, >> Tim >> >> >> -- >> Tim A. Thompson >> Metadata Librarian >> Yale University Library >> >> >> On Wed, Feb 2, 2022 at 2:20 PM Christian Grün <christian.gruen at gmail.com> >> wrote: >> >>> > I notice that curl doesn't include the Authorization header on the >>> first request, but I am out of my depth here. >>> >>> That could be an interesting hint. In the latest snapshot [1], I have >>> removed the Authorization header from the first request. Can you check >>> if it makes a difference? >>> >>> If not, could you share the headers of the second requests with us? >>> >>> [1] https://files.basex.org/releases/latest/ >>> >>

> The server is MarkLogic, s maybe they are biased against BaseX ;) I'm not sure about the configuration, but replacing the User-Agent header with "curl/7.71.1" didn't help.

> I noticed that in both curl and Postman the first request says, "[HTTP request 1/2]," whereas BaseX says, "[HTTP request 1/1]"

On Wed, Feb 2, 2022 at 6:08 PM Christian Grün <christian.gruen at gmail.com> wrote: > Thanks. > > > The server is MarkLogic, s maybe they are biased against BaseX ;) I'm > not sure about the configuration, but replacing the User-Agent header with > "curl/7.71.1" didn't help. > > Fraudulent competition ;·) > > I’m particularly confused by the response code: 502 usually indicates > a server-side configuration error, which should not be triggerable > from a client. > > I noticed it would be more correct if BaseX had not sent a second > request if the response code is unequal to 401. In our case, it might > have proven to be helpful, as we can see that > > a) the response to the second request is 401, and > b) it contains the Digest WWW-Authenticate header. > > In addition, the second request contains a Content-Length header, > which is missing in the first request. Presumably, due to the missing > Content-Length:0 header, the server tries to retrieve a body, but > gives up after the timeout and returns 502. > > With the latest snapshot [1], I’m now creating an explicit > 0-bytes-body for all methods except for GET and TRACE. Does this help? > > > I noticed that in both curl and Postman the first request says, "[HTTP > request 1/2]," whereas BaseX says, "[HTTP request 1/1]" > > My hope is that numerous of these irregularities will disappear once > we’ll have switched to the new JDK 11 HTTP Client [2] with BaseX 10. > > [1] https://files.basex.org/releases/latest/ > [2] https://openjdk.java.net/groups/net/httpclient/intro.html >

On Thu, Feb 3, 2022 at 3:04 AM Tim Thompson <timathom at gmail.com> wrote: > > That solved it! Thank you, Christian, for the outstanding support, as always. I always panic a bit when I have to think about using something other than BaseX for my programming needs ;) > > Tim > > > > On Wed, Feb 2, 2022 at 6:08 PM Christian Grün <christian.gruen at gmail.com> wrote: >> >> Thanks. >> >> > The server is MarkLogic, s maybe they are biased against BaseX ;) I'm not sure about the configuration, but replacing the User-Agent header with "curl/7.71.1" didn't help. >> >> Fraudulent competition ;·) >> >> I’m particularly confused by the response code: 502 usually indicates >> a server-side configuration error, which should not be triggerable >> from a client. >> >> I noticed it would be more correct if BaseX had not sent a second >> request if the response code is unequal to 401. In our case, it might >> have proven to be helpful, as we can see that >> >> a) the response to the second request is 401, and >> b) it contains the Digest WWW-Authenticate header. >> >> In addition, the second request contains a Content-Length header, >> which is missing in the first request. Presumably, due to the missing >> Content-Length:0 header, the server tries to retrieve a body, but >> gives up after the timeout and returns 502. >> >> With the latest snapshot [1], I’m now creating an explicit >> 0-bytes-body for all methods except for GET and TRACE. Does this help? >> >> > I noticed that in both curl and Postman the first request says, "[HTTP request 1/2]," whereas BaseX says, "[HTTP request 1/1]" >> >> My hope is that numerous of these irregularities will disappear once >> we’ll have switched to the new JDK 11 HTTP Client [2] with BaseX 10. >> >> [1] https://files.basex.org/releases/latest/ >> [2] https://openjdk.java.net/groups/net/httpclient/intro.html