archive:extract:-binary can cause OutOfMemoryException
Saturday, 2 January 2016 · 2 messages
CG
Quoted text · E. Wray Johnson · 5 lines
> I have written an XQuery that properly extracts binary(raw) archive resource > contents to binary(raw) resources. However since archive:extract:-binary > returns the contents as xs:base64Binary and not streamable items, large > archive contents are loaded completely into memory and will often cause an > OutOfMemoryException.
Right; see [1].
Quoted text · E. Wray Johnson · 2 lines
> this is not acceptable in a system where many users > may be doing this concurrently.
Well, I only partially agree. Similar to Java, and other full-fledged languages, it’s fairly easy to write code that cause out-of-memory errors, and – as far as I know – it’s generally hard to restrict memory usage to a single thread within the same JVM. We try to do so with
xquery:eval [2], but there is no guarantee that it will always work out.
Christian
[1] https://www.mail-archive.com/basex-talk%40mailman.uni-konstanz.de/msg07078.html [2] http://docs.basex.org/wiki/XQuery_Module#xquery:evalQuoted text · E. Wray Johnson · 16 lines
archive:extract-to seems to only extract to > an external file-system location and not to binary(raw) resources. Example > code is provided below that extracts to the same database folder as the > archive, however the target location of the contents is not pertinent to the > problem. > > let $database := 'db' > let $archivePath := 'myfolder/myarchive.zip' > let $archive := db:retrieve($database, $archivePath) > let $basePath := replace($archivePath, '[^/]+$', '') > let $entries := archive:entries($archive) > let $contents := archive:extract-binary($archive) > for $entry at $pos in $entries > let $content := $contents[$pos] > let $target := $basePath || '/' || $entry/text() > return db:store($database, $target, $content)
EJ
E. Wray Johnson2 Jan 2016 21:07 #
I have written an XQuery that properly extracts binary(raw) archive resource contents to binary(raw) resources. However since
archive:extract:-binary returns the contents as xs:base64Binary and not streamable items, large archive contents are loaded completely into memory and will often cause an OutOfMemoryException. I have increased the BaseX JVM heap size and the problem goes away, but this is not acceptable in a system where many users may be doing this concurrently.
archive:extract-to seems to only extract to an external file-system location and not to binary(raw) resources. Example code is provided below that extracts to the same database folder as the archive, however the target location of the contents is not pertinent to the problem.let $database := 'db'
let $archivePath := 'myfolder/myarchive.zip'
let $archive := db:retrieve($database, $archivePath)
let $basePath := replace($archivePath, '[^/]+$', '')
let $entries := archive:entries($archive)
let $contents := archive:extract-binary($archive)
for $entry at $pos in $entries
let $content := $contents[$pos]
let $target := $basePath || '/' || $entry/text()
return db:store($database, $target, $content)Written for BaseX 8.3. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery