RESTXQ, multipart/form-data, out of memory saving to file

On 20. Mar 2019, at 13:34, James Ball <basex-talk at jamesball.co.uk<mailto:basex-talk at jamesball.co.uk>> wrote:

> However, what is the actual reason to put the binary into a map?

> I would try to use the streaming capabilities and to store the binary as-is.

On Wed, Mar 20, 2019 at 3:02 PM James Ball <basex-talk at jamesball.co.uk> wrote: > > Hello Dirk, > > Thank you for such a quick response. > > > However, what is the actual reason to put the binary into a map? > > I don’t know - it’s BaseX that is doing it. For files uploaded via POST they are put into a map. I’m using the information here: http://docs.basex.org/wiki/RESTXQ#File_Uploads > > "The file contents are placed in a map, with the filename serving as key.” > > > I would try to use the streaming capabilities and to store the binary as-is. > > > This would be perfect but I’m not sure how to get the binary from the POST request without using the map. > > I have a function something like this: > > declare > %rest:POST("{$data}") > %rest:path("/test2.htm") > %rest:consumes("multipart/form-data") > %rest:form-param("zip", "{$files}") > function _:test($data,$files, $database) { > ... > }; > > If I just save $data to disk I don’t get a valid Zip file. $data type is xs:base64Binary but is not the same as the xs:base64Binary from $files. > To save the file from $files I have to use the map and so get the error. > > Is there a trick to get $data in the right format to match what I get from $files? > > Many thanks, James

> increase the memory that’s assigned to the JVM.

> You could try to interpret the incoming POST data with XQuery, but > then you might also struggle with memory constraints.

> You could try to interpret the incoming POST data with XQuery

On Sat, Mar 23, 2019 at 1:02 PM James Ball <basex-talk at jamesball.co.uk> wrote: > > Hi Christian, > > Thank you for your ideas - lots for me to consider. > > > increase the memory that’s assigned to the JVM. > > I’ve currently assigned it 4GB and the file that’s failing is 350MB so I don’t think that’s going to be an easy fix. > > > You could try to interpret the incoming POST data with XQuery, but > > then you might also struggle with memory constraints. > > What’s puzzling me is that the ‘upload’ part works - the data gets added to the variables and the RESTXQ function is called. > > The following does NOT cause a memory issue: > > declare > %rest:POST("{$data}") > %rest:path("/test2.htm”) > %rest:form-param("zip", "{$files}") > function _:test($data,$files) { > file:write-binary(“the path”,$data) > }; > > But the file I get isn’t valid. > > The following DOES cause the issue with larger files: > > declare > %rest:POST("{$data}") > %rest:path("/test2.htm”) > %rest:form-param("zip", "{$files}") > function _:test($data,$files) { > file:write-binary(“the path”,$files(map:keys($files)[1])) > }; > > But the file I get is valid. > > Perhaps the issue isn’t anything to do with RESTXQ and is a limitation on the size of xs:base64Binary from a map that can be written to a file? > > > You could try to interpret the incoming POST data with XQuery > > > I was trying to read the source code to understand what processes happen to convert the POST data to a map. > > I don’t understand what format or encoding the $data variable has that makes the file different to the one I get from $files. > > Because if I can get $data -> run decoding -> save to file that would work perfectly for me. > > I shall keep looking but any pointers in the right direction are much appreciated. > > Thank you for your continued help. > > Regards, James >

> On 25 Mar 2019, at 17:22, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi James, > > Thanks for your persistence. Your observation (4 GB assigned, failing > with 350 MB) made me think, and indeed the difference between handling > raw post and map data was caused by an internal info output generation > of map structures that does not contribute to the eventual result. > With the latest snapshot [1], you should be able to upload and save > your file as requested! > > Cheers, > Christian > > [1] http://files.basex.org/releases/latest/ > > > > > On Sat, Mar 23, 2019 at 1:02 PM James Ball <basex-talk at jamesball.co.uk> wrote: >> >> Hi Christian, >> >> Thank you for your ideas - lots for me to consider. >> >>> increase the memory that’s assigned to the JVM. >> >> I’ve currently assigned it 4GB and the file that’s failing is 350MB so I don’t think that’s going to be an easy fix. >> >>> You could try to interpret the incoming POST data with XQuery, but >>> then you might also struggle with memory constraints. >> >> What’s puzzling me is that the ‘upload’ part works - the data gets added to the variables and the RESTXQ function is called. >> >> The following does NOT cause a memory issue: >> >> declare >> %rest:POST("{$data}") >> %rest:path("/test2.htm”) >> %rest:form-param("zip", "{$files}") >> function _:test($data,$files) { >> file:write-binary(“the path”,$data) >> }; >> >> But the file I get isn’t valid. >> >> The following DOES cause the issue with larger files: >> >> declare >> %rest:POST("{$data}") >> %rest:path("/test2.htm”) >> %rest:form-param("zip", "{$files}") >> function _:test($data,$files) { >> file:write-binary(“the path”,$files(map:keys($files)[1])) >> }; >> >> But the file I get is valid. >> >> Perhaps the issue isn’t anything to do with RESTXQ and is a limitation on the size of xs:base64Binary from a map that can be written to a file? >> >>> You could try to interpret the incoming POST data with XQuery >> >> >> I was trying to read the source code to understand what processes happen to convert the POST data to a map. >> >> I don’t understand what format or encoding the $data variable has that makes the file different to the one I get from $files. >> >> Because if I can get $data -> run decoding -> save to file that would work perfectly for me. >> >> I shall keep looking but any pointers in the right direction are much appreciated. >> >> Thank you for your continued help. >> >> Regards, James >>