Increasing memory allocated to BaseX
Tuesday, 12 May 2015 · 2 messages
SB
Steve Baskauf12 May 2015 18:17 #
I am running BaseX 8.1 on a Win7 machine with 8GB memory. I have a query that loads data from several CSV files, then generates a RDF/XML dump into a single, very large file (1370000 lines). When I run the query using the BaseX GUI, I get an "Out of Main Memory" error. I am able to get the query to complete via the command line, which is fine for the moment, but if the database gets larger, I'm anticipating that I won't be able to use it. An obvious solution would be to generate several smaller files, but the real problem is that BaseX doesn't seem to be utilizing all of the available memory. When I monitored memory utilization, it was only utilizing a small fraction of the available memory. I tried increasing the allocated memory by changing the BASEX_JVM=-Xmx512m
to
BASEX_JVM=-Xmx1024m
line in the basex.bat and basexgui.bat files, but that didn't seem to have any effect - I still run out of memory and there is plenty of unused memory in the system. I must be missing something. Is there a simple change that I can make to make more memory available to BaseX?
For reference purposes, the query I'm running is at
https://github.com/baskaufs/test/blob/master/image-all-hack.xq
Steve Baskauf
CG
Christian Grün12 May 2015 21:37 #
Hi Steve,
It can happen that a query leads to Out Of Memory when using the GUI, whereas it runs fine on command-line. The reason is that results on command-line (or various other APIs) are returned one by one, and in the GUI they will be cached.
However, if you assign more memory to the JVM, it should always be utilized by BaseX. If you want to observe memory consumption while the query is running, you could open the memory dialog by pressing on the memory bar in the lower right corner. This dialog will also tell you if your JVM option was correctly detected. You could also open the Java tool jvisualvm while the query is running and see what's going on. If OOM occurs too soon, you can slow down processing by adding some
prof:sleep(...) in your query.
Hope this helps,
ChristianQuoted text · Steve Baskauf · 26 lines
On Tue, May 12, 2015 at 4:17 PM, Steve Baskauf <steve.baskauf at vanderbilt.edu> wrote: > I am running BaseX 8.1 on a Win7 machine with 8GB memory. I have a query > that loads data from several CSV files, then generates a RDF/XML dump into a > single, very large file (1370000 lines). When I run the query using the > BaseX GUI, I get an "Out of Main Memory" error. I am able to get the query > to complete via the command line, which is fine for the moment, but if the > database gets larger, I'm anticipating that I won't be able to use it. An > obvious solution would be to generate several smaller files, but the real > problem is that BaseX doesn't seem to be utilizing all of the available > memory. When I monitored memory utilization, it was only utilizing a small > fraction of the available memory. I tried increasing the allocated memory > by changing the > BASEX_JVM=-Xmx512m > to > BASEX_JVM=-Xmx1024m > line in the basex.bat and basexgui.bat files, but that didn't seem to have > any effect - I still run out of memory and there is plenty of unused memory > in the system. I must be missing something. Is there a simple change that > I can make to make more memory available to BaseX? > > For reference purposes, the query I'm running is at > https://github.com/baskaufs/test/blob/master/image-all-hack.xq > > Steve Baskauf >
Written for BaseX 8.1. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery