http:send-request - problem with response

On Fri, Oct 25, 2019 at 1:52 PM Bogdan Bogucki <bbogucki at hussar.pl> wrote: > > > Hello, > > I encounter problem with http:send-request function. I need handle multiple forward requests (3) with cookies. > > First request returns cookies information which are required with next requests. Output from http:send-request dosen't contains all fields which are returned by server. > > Response from browser is flowing: > cache-control > private > cf-cache-status > DYNAMIC > cf-ray > 52b3cedc5b9bcc9f-WAW > content-type > text/html; charset=utf-8 > date > Fri, 25 Oct 2019 11:21:37 GMT > expect-ct > max-age=604800, report-uri="ht….com/cdn-cgi/beacon/expect-ct" > location > http://www.pracuj.pl/praca/jav…eloper-warszawa,oferta,7171988 > server > cloudflare > set-cookie > __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly > set-cookie > _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ > set-cookie > _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly > x-aspnet-version > 4.0.30319 > x-aspnetmvc-version > 5.2 > X-Firefox-Spdy > h2 > x-powered-by > ASP.NET > x-ua-compatible > IE=edge > > > Request: > > http:send-request( > <http:request method='get' follow-redirect='false'> > <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"/> > <http:header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/> > <http:header name="Accept-Languag" value="pl,en-US;q=0.7,en;q=0.3"/> > <http:header name="DNT" value="1"/> > <http:header name="Connection" value="keep-alive"/> > <http:header name="Upgrade-Insecure-Requests" value="1"/> > <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> > </http:request>, 'https://www.pracuj.pl/praca/java-developer-warszawa,oferta,7171988') > > Response form http:send-request is: > > <http:response xmlns:http="http://expath.org/ns/http-client" status="302" message="Found"> > <http:header name="Transfer-Encoding" value="chunked"/> > <http:header name="CF-RAY" value="52b3d0de097fb839-WAW"/> > <http:header name="Server" value="cloudflare"/> > <http:header name="Connection" value="keep-alive"/> > <http:header name="Date" value="Fri, 25 Oct 2019 11:23:00 GMT"/> > <http:header name="X-AspNetMvc-Version" value="5.2"/> > <http:header name="X-UA-Compatible" value="IE=edge"/> > <http:header name="CF-Cache-Status" value="DYNAMIC"/> > <http:header name="Cache-Control" value="private"/> > <http:header name="X-AspNet-Version" value="4.0.30319"/> > <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> > <http:header name="Expect-CT" value="max-age=604800, report-uri=&quot;https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct&quot;"/> > <http:header name="X-Powered-By" value="ASP.NET"/> > <http:header name="Content-Type" value="text/html; charset=utf-8"/> > <http:body media-type="text/html"/> > </http:response> > > Two set-cookie fields is missing. > > How should I invoke http:send-request to receive proper result ? > > Regards > > Bogdan

On Wed, Nov 13, 2019 at 4:26 PM Bogdan Bogucki <bbogucki at hussar.pl> wrote: > > Hi Christian, > Lack of many redirection is not a big problem. I can handle it manually. > Problem is not complete response header form first request. > > Browser make 3 redirection I am talking about first. > > Please take a look. > > > Response header from browser (first request) - contains three fields Set > Cookies: > > set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly > > set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ > > set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly > > but response header form http:send-request contains only last one: > > <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; > expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> > > I need information from previous set-cookie fields to make manual > redirection passing information about sessions to next requests. > > Regards > > Bogdan Bogucki > > W dniu 29.10.2019 o 03:06, Christian Grün pisze: > > Hi Bogdan, > > > > The current http:send-request implementation is based on the default > > Java HttpURLConnection, which does not resolve redirects that use > > different protocols [1]. This is the reason why your request will not > > be fully processed (as it e.g. happens when you use the browser). > > > > It seems that your initial request to the https protocol returns a 302 > > redirect to a (now unsafe) http URL, which returns another redirect to > > https. I don’t know who maintains the discussed web site, but it could > > be worth contacting the admins and asking them if they could update > > and simplify their redirect policy. > > > > Hope this helps, > > Christian > > > > [1] https://stackoverflow.com/a/1884427/1018790 > > > > > > > > > > On Fri, Oct 25, 2019 at 1:52 PM Bogdan Bogucki <bbogucki at hussar.pl> wrote: > >> > >> Hello, > >> > >> I encounter problem with http:send-request function. I need handle multiple forward requests (3) with cookies. > >> > >> First request returns cookies information which are required with next requests. Output from http:send-request dosen't contains all fields which are returned by server. > >> > >> Response from browser is flowing: > >> cache-control > >> private > >> cf-cache-status > >> DYNAMIC > >> cf-ray > >> 52b3cedc5b9bcc9f-WAW > >> content-type > >> text/html; charset=utf-8 > >> date > >> Fri, 25 Oct 2019 11:21:37 GMT > >> expect-ct > >> max-age=604800, report-uri="ht….com/cdn-cgi/beacon/expect-ct" > >> location > >> http://www.pracuj.pl/praca/jav…eloper-warszawa,oferta,7171988 > >> server > >> cloudflare > >> set-cookie > >> __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly > >> set-cookie > >> _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ > >> set-cookie > >> _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly > >> x-aspnet-version > >> 4.0.30319 > >> x-aspnetmvc-version > >> 5.2 > >> X-Firefox-Spdy > >> h2 > >> x-powered-by > >> ASP.NET > >> x-ua-compatible > >> IE=edge > >> > >> > >> Request: > >> > >> http:send-request( > >> <http:request method='get' follow-redirect='false'> > >> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"/> > >> <http:header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/> > >> <http:header name="Accept-Languag" value="pl,en-US;q=0.7,en;q=0.3"/> > >> <http:header name="DNT" value="1"/> > >> <http:header name="Connection" value="keep-alive"/> > >> <http:header name="Upgrade-Insecure-Requests" value="1"/> > >> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> > >> </http:request>, 'https://www.pracuj.pl/praca/java-developer-warszawa,oferta,7171988') > >> > >> Response form http:send-request is: > >> > >> <http:response xmlns:http="http://expath.org/ns/http-client" status="302" message="Found"> > >> <http:header name="Transfer-Encoding" value="chunked"/> > >> <http:header name="CF-RAY" value="52b3d0de097fb839-WAW"/> > >> <http:header name="Server" value="cloudflare"/> > >> <http:header name="Connection" value="keep-alive"/> > >> <http:header name="Date" value="Fri, 25 Oct 2019 11:23:00 GMT"/> > >> <http:header name="X-AspNetMvc-Version" value="5.2"/> > >> <http:header name="X-UA-Compatible" value="IE=edge"/> > >> <http:header name="CF-Cache-Status" value="DYNAMIC"/> > >> <http:header name="Cache-Control" value="private"/> > >> <http:header name="X-AspNet-Version" value="4.0.30319"/> > >> <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> > >> <http:header name="Expect-CT" value="max-age=604800, report-uri=&quot;https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct&quot;"/> > >> <http:header name="X-Powered-By" value="ASP.NET"/> > >> <http:header name="Content-Type" value="text/html; charset=utf-8"/> > >> <http:body media-type="text/html"/> > >> </http:response> > >> > >> Two set-cookie fields is missing. > >> > >> How should I invoke http:send-request to receive proper result ? > >> > >> Regards > >> > >> Bogdan

> Hi Bogdan (cc to the list), > > Thanks for digging deeper. I noticed that the standard Java function > that we used returned only one value per header field, and dropped the > others [1]. > > I managed to fix this in the latest stable snapshot; could you give it > a try [2]? > > BaseX 9.3 will be released end of November. > > Cheers, > Christian > > [1] http://files.basex.org/releases/latest/ > [2] https://github.com/BaseXdb/basex/issues/1751 > > > > > On Wed, Nov 13, 2019 at 4:26 PM Bogdan Bogucki <bbogucki at hussar.pl> wrote: >> Hi Christian, >> Lack of many redirection is not a big problem. I can handle it manually. >> Problem is not complete response header form first request. >> >> Browser make 3 redirection I am talking about first. >> >> Please take a look. >> >> >> Response header from browser (first request) - contains three fields Set >> Cookies: >> >> set-cookie __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly >> >> set-cookie _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ >> >> set-cookie _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly >> >> but response header form http:send-request contains only last one: >> >> <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; >> expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> >> >> I need information from previous set-cookie fields to make manual >> redirection passing information about sessions to next requests. >> >> Regards >> >> Bogdan Bogucki >> >> W dniu 29.10.2019 o 03:06, Christian Grün pisze: >>> Hi Bogdan, >>> >>> The current http:send-request implementation is based on the default >>> Java HttpURLConnection, which does not resolve redirects that use >>> different protocols [1]. This is the reason why your request will not >>> be fully processed (as it e.g. happens when you use the browser). >>> >>> It seems that your initial request to the https protocol returns a 302 >>> redirect to a (now unsafe) http URL, which returns another redirect to >>> https. I don’t know who maintains the discussed web site, but it could >>> be worth contacting the admins and asking them if they could update >>> and simplify their redirect policy. >>> >>> Hope this helps, >>> Christian >>> >>> [1] https://stackoverflow.com/a/1884427/1018790 >>> >>> >>> >>> >>> On Fri, Oct 25, 2019 at 1:52 PM Bogdan Bogucki <bbogucki at hussar.pl> wrote: >>>> Hello, >>>> >>>> I encounter problem with http:send-request function. I need handle multiple forward requests (3) with cookies. >>>> >>>> First request returns cookies information which are required with next requests. Output from http:send-request dosen't contains all fields which are returned by server. >>>> >>>> Response from browser is flowing: >>>> cache-control >>>> private >>>> cf-cache-status >>>> DYNAMIC >>>> cf-ray >>>> 52b3cedc5b9bcc9f-WAW >>>> content-type >>>> text/html; charset=utf-8 >>>> date >>>> Fri, 25 Oct 2019 11:21:37 GMT >>>> expect-ct >>>> max-age=604800, report-uri="ht….com/cdn-cgi/beacon/expect-ct" >>>> location >>>> http://www.pracuj.pl/praca/jav…eloper-warszawa,oferta,7171988 >>>> server >>>> cloudflare >>>> set-cookie >>>> __cfduid=daf54110b2a87d66c2a53…; domain=.pracuj.pl; HttpOnly >>>> set-cookie >>>> _yaic=13; expires=Fri, 31-Dec-9999 23:59:59 GMT; path=/ >>>> set-cookie >>>> _urnadiam=A; domain=.pracuj.pl…3:00:00 GMT; path=/; HttpOnly >>>> x-aspnet-version >>>> 4.0.30319 >>>> x-aspnetmvc-version >>>> 5.2 >>>> X-Firefox-Spdy >>>> h2 >>>> x-powered-by >>>> ASP.NET >>>> x-ua-compatible >>>> IE=edge >>>> >>>> >>>> Request: >>>> >>>> http:send-request( >>>> <http:request method='get' follow-redirect='false'> >>>> <http:header name="User-Agent" value="Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:69.0) Gecko/20100101 Firefox/69.0"/> >>>> <http:header name="Accept" value="text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8"/> >>>> <http:header name="Accept-Languag" value="pl,en-US;q=0.7,en;q=0.3"/> >>>> <http:header name="DNT" value="1"/> >>>> <http:header name="Connection" value="keep-alive"/> >>>> <http:header name="Upgrade-Insecure-Requests" value="1"/> >>>> <http:body media-type='application/x-www-form-urlencoded; charset=UTF-8'/> >>>> </http:request>, 'https://www.pracuj.pl/praca/java-developer-warszawa,oferta,7171988') >>>> >>>> Response form http:send-request is: >>>> >>>> <http:response xmlns:http="http://expath.org/ns/http-client" status="302" message="Found"> >>>> <http:header name="Transfer-Encoding" value="chunked"/> >>>> <http:header name="CF-RAY" value="52b3d0de097fb839-WAW"/> >>>> <http:header name="Server" value="cloudflare"/> >>>> <http:header name="Connection" value="keep-alive"/> >>>> <http:header name="Date" value="Fri, 25 Oct 2019 11:23:00 GMT"/> >>>> <http:header name="X-AspNetMvc-Version" value="5.2"/> >>>> <http:header name="X-UA-Compatible" value="IE=edge"/> >>>> <http:header name="CF-Cache-Status" value="DYNAMIC"/> >>>> <http:header name="Cache-Control" value="private"/> >>>> <http:header name="X-AspNet-Version" value="4.0.30319"/> >>>> <http:header name="Set-Cookie" value="_urnadiam=B; domain=.pracuj.pl; expires=Tue, 31-Dec-2019 23:00:00 GMT; path=/; HttpOnly"/> >>>> <http:header name="Expect-CT" value="max-age=604800, report-uri=&quot;https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct&quot;"/> >>>> <http:header name="X-Powered-By" value="ASP.NET"/> >>>> <http:header name="Content-Type" value="text/html; charset=utf-8"/> >>>> <http:body media-type="text/html"/> >>>> </http:response> >>>> >>>> Two set-cookie fields is missing. >>>> >>>> How should I invoke http:send-request to receive proper result ? >>>> >>>> Regards >>>> >>>> Bogdan