stream binary responses from http:module

On Thu, Mar 27, 2014 at 1:06 PM, Johan Mörén <hutchkintoot at gmail.com> wrote: > Hi > > I have built a small client in XQuery that fetches zip files from another > remote service using the http-module and stores them on disk with the help > of the file-module. > > Some of the returned zip files are really large and BaseX seems to need to > materialize the files in memory before writing them to disk resulting in out > of memory errors for some files. > > Is there some way read the response in a more streamable fashion when using > the http-module? > > I tried using the fetch:binary function successfully but i really need the > more extended functionality of the http-module. > > #results in out of memory if the zip file is too large > file:write-binary("file.zip", http:sendrequest(...)[2]) > > #works all the time. > file:write-binary("file.zip", fetch:binary(...)) > > Regards, > Johan > > > > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

> Hi Johan, > > the HTTP Module is pretty magic, because it automatically tries to > convert the input to the expected result format. This makes it > difficult to stream. > > We have already pondered two options to circumvent this restriction: > > * Fetch Module: extend the function signatures with additional options > * HTTP Module (our favorite): add additional functions (e.g. > http:get(), http:post(), etc.) with xs:base64 as return type. > > Which options of the HTTP Module do you currently use? > > Christian > ______________________________ > > On Thu, Mar 27, 2014 at 1:06 PM, Johan Mörén <hutchkintoot at gmail.com> > wrote: > > Hi > > > > I have built a small client in XQuery that fetches zip files from another > > remote service using the http-module and stores them on disk with the > help > > of the file-module. > > > > Some of the returned zip files are really large and BaseX seems to need > to > > materialize the files in memory before writing them to disk resulting in > out > > of memory errors for some files. > > > > Is there some way read the response in a more streamable fashion when > using > > the http-module? > > > > I tried using the fetch:binary function successfully but i really need > the > > more extended functionality of the http-module. > > > > #results in out of memory if the zip file is too large > > file:write-binary("file.zip", http:sendrequest(...)[2]) > > > > #works all the time. > > file:write-binary("file.zip", fetch:binary(...)) > > > > Regards, > > Johan > > > > > > > > > > > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > >

On Thu, Mar 27, 2014 at 3:37 PM, Johan Mörén <hutchkintoot at gmail.com> wrote: > Hi Christian > > The options i need from the http-module are at the moment mainly the ability > to authenticate. Getting hold of response/request headers and status code > are also very useful if you want to do some more detailed error handling. > > Extending the HTTP-module sounds like my favourite as well. > > /Johan > > > On Thu, Mar 27, 2014 at 1:57 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Johan, >> >> the HTTP Module is pretty magic, because it automatically tries to >> convert the input to the expected result format. This makes it >> difficult to stream. >> >> We have already pondered two options to circumvent this restriction: >> >> * Fetch Module: extend the function signatures with additional options >> * HTTP Module (our favorite): add additional functions (e.g. >> http:get(), http:post(), etc.) with xs:base64 as return type. >> >> Which options of the HTTP Module do you currently use? >> >> Christian >> ______________________________ >> >> On Thu, Mar 27, 2014 at 1:06 PM, Johan Mörén <hutchkintoot at gmail.com> >> wrote: >> > Hi >> > >> > I have built a small client in XQuery that fetches zip files from >> > another >> > remote service using the http-module and stores them on disk with the >> > help >> > of the file-module. >> > >> > Some of the returned zip files are really large and BaseX seems to need >> > to >> > materialize the files in memory before writing them to disk resulting in >> > out >> > of memory errors for some files. >> > >> > Is there some way read the response in a more streamable fashion when >> > using >> > the http-module? >> > >> > I tried using the fetch:binary function successfully but i really need >> > the >> > more extended functionality of the http-module. >> > >> > #results in out of memory if the zip file is too large >> > file:write-binary("file.zip", http:sendrequest(...)[2]) >> > >> > #works all the time. >> > file:write-binary("file.zip", fetch:binary(...)) >> > >> > Regards, >> > Johan >> > >> > >> > >> > >> > >> > >> > _______________________________________________ >> > BaseX-Talk mailing list >> > BaseX-Talk at mailman.uni-konstanz.de >> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> > > >

> Hi Johan, > > that's something you may know anyway, but... you can as well specify the > authentication data in your URL: > > http://name:password@... > > Hope this helps, > Christian > > > > On Thu, Mar 27, 2014 at 3:37 PM, Johan Mörén <hutchkintoot at gmail.com> > wrote: > > Hi Christian > > > > The options i need from the http-module are at the moment mainly the > ability > > to authenticate. Getting hold of response/request headers and status code > > are also very useful if you want to do some more detailed error handling. > > > > Extending the HTTP-module sounds like my favourite as well. > > > > /Johan > > > > > > On Thu, Mar 27, 2014 at 1:57 PM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi Johan, > >> > >> the HTTP Module is pretty magic, because it automatically tries to > >> convert the input to the expected result format. This makes it > >> difficult to stream. > >> > >> We have already pondered two options to circumvent this restriction: > >> > >> * Fetch Module: extend the function signatures with additional options > >> * HTTP Module (our favorite): add additional functions (e.g. > >> http:get(), http:post(), etc.) with xs:base64 as return type. > >> > >> Which options of the HTTP Module do you currently use? > >> > >> Christian > >> ______________________________ > >> > >> On Thu, Mar 27, 2014 at 1:06 PM, Johan Mörén <hutchkintoot at gmail.com> > >> wrote: > >> > Hi > >> > > >> > I have built a small client in XQuery that fetches zip files from > >> > another > >> > remote service using the http-module and stores them on disk with the > >> > help > >> > of the file-module. > >> > > >> > Some of the returned zip files are really large and BaseX seems to > need > >> > to > >> > materialize the files in memory before writing them to disk resulting > in > >> > out > >> > of memory errors for some files. > >> > > >> > Is there some way read the response in a more streamable fashion when > >> > using > >> > the http-module? > >> > > >> > I tried using the fetch:binary function successfully but i really need > >> > the > >> > more extended functionality of the http-module. > >> > > >> > #results in out of memory if the zip file is too large > >> > file:write-binary("file.zip", http:sendrequest(...)[2]) > >> > > >> > #works all the time. > >> > file:write-binary("file.zip", fetch:binary(...)) > >> > > >> > Regards, > >> > Johan > >> > > >> > > >> > > >> > > >> > > >> > > >> > _______________________________________________ > >> > BaseX-Talk mailing list > >> > BaseX-Talk at mailman.uni-konstanz.de > >> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >> > > > > > >

On 27 March 2014 13:57, Christian Grün wrote:

> * HTTP Module (our favorite): add additional functions (e.g. > http:get(), http:post(), etc.) with xs:base64 as return type.

> Is there any intrinsic difference, from an implementation point of > view, in the context of streamability of the returned content, between > a function returning always an xs:base64Binary item, and another one > returning sometimes an xs:base64Binary item, sometimes a string, and > sometimes a document node?

> I am pretty sure Adam implemented streaming for the HTTP Client in > eXist, so it might be interesting to have a look there as well.

On Fri, Mar 28, 2014 at 5:31 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Florent, > >> Is there any intrinsic difference, from an implementation point of >> view, in the context of streamability of the returned content, between >> a function returning always an xs:base64Binary item, and another one >> returning sometimes an xs:base64Binary item, sometimes a string, and >> sometimes a document node? > > Yes there is, at least in our architecture: a specific function will > always be streamable or not [1]. This static distinction is necessary > because various optimizations rely on it. > > I remember you talked about a second version of the HTTP Client > Module... Have you thought about removing the "magic parts" of the spec > and provide explicit functions for retrieving binary or string data, > similar to the EXPath Archive and File Modules? > >> I am pretty sure Adam implemented streaming for the HTTP Client in >> eXist, so it might be interesting to have a look there as well. > > Sounds interesting, but I couldn't find any hints in their > documentation. Do you have any reference that confirms this > assumption? Or some general info on how streamable results are handled > in eXist? What about the first http:response item, do you know if it > will also be streamed? > > Best, > Christian > > [1] http://docs.basex.org/wiki/Streaming_Module

> The main features was authentication

> and the ability to control headers of > the request and response.

On Tue, Nov 11, 2014 at 10:17 PM, Johan Mörén <hutchkintoot at gmail.com> wrote: > Hi Christian! > > The main features was authentication and the ability to control headers of > the request and response. > > Regards, > Johan > > On Mon, Nov 10, 2014 at 3:00 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Johan, hi Florent, >> >> Maybe 8 months ago, I have opened a new issue for extending HTTP >> client functionality in BaseX. The main requirement was that HTTP >> responses should be streamable. As HTTP responses are currently >> returned as elements, it is not possible (at least in BaseX) to >> include streamable sub elements. We could either… >> >> * provide HTTP client functions that only return the body itself, or >> * wrap all results in maps and arrays. >> >> In both cases, we could avoid wrapping binary items as Base64, which >> would save us both memory and processing time. >> >> My questions to you guys are: >> >> @Florent: Have you thought about using maps and arrays in a new >> version of the HTTP Client spec? >> >> @Johan (in case you remember): What HTTP client features do you need >> that prevent you from using the fetch:binary function? >> >> Thanks in advance, >> Christian >> >> [1] https://github.com/BaseXdb/basex/issues/914 >> >> >> >> On Fri, Mar 28, 2014 at 5:31 PM, Christian Grün >> <christian.gruen at gmail.com> wrote: >> > Hi Florent, >> > >> >> Is there any intrinsic difference, from an implementation point of >> >> view, in the context of streamability of the returned content, between >> >> a function returning always an xs:base64Binary item, and another one >> >> returning sometimes an xs:base64Binary item, sometimes a string, and >> >> sometimes a document node? >> > >> > Yes there is, at least in our architecture: a specific function will >> > always be streamable or not [1]. This static distinction is necessary >> > because various optimizations rely on it. >> > >> > I remember you talked about a second version of the HTTP Client >> > Module... Have you thought about removing the "magic parts" of the spec >> > and provide explicit functions for retrieving binary or string data, >> > similar to the EXPath Archive and File Modules? >> > >> >> I am pretty sure Adam implemented streaming for the HTTP Client in >> >> eXist, so it might be interesting to have a look there as well. >> > >> > Sounds interesting, but I couldn't find any hints in their >> > documentation. Do you have any reference that confirms this >> > assumption? Or some general info on how streamable results are handled >> > in eXist? What about the first http:response item, do you know if it >> > will also be streamed? >> > >> > Best, >> > Christian >> > >> > [1] http://docs.basex.org/wiki/Streaming_Module > >