basexhttp out of memory where basexgui suceeds
Hi,
I have a database about 200 MB in size made up of approximately 150 000 documents of similar size and structure as children of the root node.
When I run the following script in basexgui a significant amount of memory is consumed (over 1 GB if I'm reading the display correctly), but I do get a result:
for $doc in db:get('docs')
return $doc update {
delete node .//*[local-name() = ('A', 'B', 'C', 'D')]
}When I run it over basexhttp I get a java.lang.OutOfMemoryError: Java heap space. I have increased the memory available to the JVM to 4 GB but this has not affected the failure of the script.
How can I resolve this? Is there some rewriting of the script that would help, or is it more specific to basexhttp?
Many thanks,
Jack
Sorry, the first line of the script should end with
/*/*
Could the problem be related to UPDINDEX, AUTOOPTIMIZE and/or FTINDEX all being set to true on basexhttp but false on basexgui? I thought it wouldn't make a difference because the updates occur in main memory, but perhaps it does.
Best,
Jack
Quoted text · Jack Steyn · 27 lines
On Wed, 7 Feb 2024, 4:03 pm Jack Steyn, <steynjack at gmail.com> wrote: > Hi, > > I have a database about 200 MB in size made up of approximately 150 000 > documents of similar size and structure as children of the root node. > > When I run the following script in basexgui a significant amount of memory > is consumed (over 1 GB if I'm reading the display correctly), but I do get > a result: > > for $doc in db:get('docs') > return $doc update { > delete node .//*[local-name() = ('A', 'B', 'C', 'D')] > } > > When I run it over basexhttp I get a java.lang.OutOfMemoryError: Java heap > space. I have increased the memory available to the JVM to 4 GB but this > has not affected the failure of the script. > > How can I resolve this? Is there some rewriting of the script that would > help, or is it more specific to basexhttp? > > Many thanks, > > Jack > >
Hi Jack,
If you run the query via basexhttp, how do you retrieve the results, i.e., which client do you use?
Can you possibly provide us with a self-contained example, something like…
for $i in 1 to 500000
return <xml/> update {
insert node <a/> into .
}…and some steps to reproduce the behavior?
Thanks in advance, Christian
Quoted text · Jack Steyn · 27 lines
On Wed, Feb 7, 2024 at 6:04 AM Jack Steyn <steynjack at gmail.com> wrote: > Hi, > > I have a database about 200 MB in size made up of approximately 150 000 > documents of similar size and structure as children of the root node. > > When I run the following script in basexgui a significant amount of memory > is consumed (over 1 GB if I'm reading the display correctly), but I do get > a result: > > for $doc in db:get('docs') > return $doc update { > delete node .//*[local-name() = ('A', 'B', 'C', 'D')] > } > > When I run it over basexhttp I get a java.lang.OutOfMemoryError: Java heap > space. I have increased the memory available to the JVM to 4 GB but this > has not affected the failure of the script. > > How can I resolve this? Is there some rewriting of the script that would > help, or is it more specific to basexhttp? > > Many thanks, > > Jack > >
Hi Christian,
Sorry, I should have provided a self-contained example to begin with.
In any case, I was running BaseX 10.0; after noticing that 10.6 boasts 'Much more memory-efficient representation of XML fragments', I upgraded to 10.7 and the problem appears to be resolved (and wow, there is a big difference in performance – kudos!).
Many thanks,
Jack
Quoted text · Christian Grün · 48 lines
On Wed, 7 Feb 2024, 6:09 pm Christian Grün, <christian.gruen at gmail.com> wrote: > Hi Jack, > > If you run the query via basexhttp, how do you retrieve the results, i.e., > which client do you use? > > Can you possibly provide us with a self-contained example, something like… > > for $i in 1 to 500000 > return <xml/> update { > insert node <a/> into . > } > > …and some steps to reproduce the behavior? > > Thanks in advance, > Christian > > > On Wed, Feb 7, 2024 at 6:04 AM Jack Steyn <steynjack at gmail.com> wrote: > >> Hi, >> >> I have a database about 200 MB in size made up of approximately 150 000 >> documents of similar size and structure as children of the root node. >> >> When I run the following script in basexgui a significant amount of >> memory is consumed (over 1 GB if I'm reading the display correctly), but I >> do get a result: >> >> for $doc in db:get('docs') >> return $doc update { >> delete node .//*[local-name() = ('A', 'B', 'C', 'D')] >> } >> >> When I run it over basexhttp I get a java.lang.OutOfMemoryError: Java >> heap space. I have increased the memory available to the JVM to 4 GB but >> this has not affected the failure of the script. >> >> How can I resolve this? Is there some rewriting of the script that would >> help, or is it more specific to basexhttp? >> >> Many thanks, >> >> Jack >> >>
…great to hear, thanks.
Quoted text · Jack Steyn · 63 lines
On Thu, Feb 8, 2024 at 7:30 AM Jack Steyn <steynjack at gmail.com> wrote: > Hi Christian, > > Sorry, I should have provided a self-contained example to begin with. > > In any case, I was running BaseX 10.0; after noticing that 10.6 boasts > 'Much more memory-efficient representation of XML fragments', I upgraded to > 10.7 and the problem appears to be resolved (and wow, there is a big > difference in performance – kudos!). > > Many thanks, > > Jack > > On Wed, 7 Feb 2024, 6:09 pm Christian Grün, <christian.gruen at gmail.com> > wrote: > >> Hi Jack, >> >> If you run the query via basexhttp, how do you retrieve the results, >> i.e., which client do you use? >> >> Can you possibly provide us with a self-contained example, something like… >> >> for $i in 1 to 500000 >> return <xml/> update { >> insert node <a/> into . >> } >> >> …and some steps to reproduce the behavior? >> >> Thanks in advance, >> Christian >> >> >> On Wed, Feb 7, 2024 at 6:04 AM Jack Steyn <steynjack at gmail.com> wrote: >> >>> Hi, >>> >>> I have a database about 200 MB in size made up of approximately 150 000 >>> documents of similar size and structure as children of the root node. >>> >>> When I run the following script in basexgui a significant amount of >>> memory is consumed (over 1 GB if I'm reading the display correctly), but I >>> do get a result: >>> >>> for $doc in db:get('docs') >>> return $doc update { >>> delete node .//*[local-name() = ('A', 'B', 'C', 'D')] >>> } >>> >>> When I run it over basexhttp I get a java.lang.OutOfMemoryError: Java >>> heap space. I have increased the memory available to the JVM to 4 GB but >>> this has not affected the failure of the script. >>> >>> How can I resolve this? Is there some rewriting of the script that would >>> help, or is it more specific to basexhttp? >>> >>> Many thanks, >>> >>> Jack >>> >>>
Written for BaseX 10.7. BaseX may have changed since: please check the current documentation.