Memory footprint for a certain query is much higher usisng BaseX10 compared to 9
Friday, 12 August 2022 · 6 messages · last reply 16 Aug 2022
AW
Alexander Witzigmann12 Aug 2022 17:33 #
Hi,
BaseX 10 has a different behavior in memory footprint for the following situation.
To avoid passing around data you can create db using the following
create (i did this with BaseX 9)
declare variable $input := <root><sheet name="SAP">{for $idx in (1 to
10000) return <row>{for $e in $enames return element {$e} { attribute
{"content"} {$idx}}}</row>}</sheet></root>;
db:create("test-db",$input,"test.xml")Then using the following query on the created db:
declare variable $input := db:open("test-db");
<root>{
for $s in $input//sheet[./@name => contains("SAP")]
return
<sheet>{$s/@*}{
for $r in $s//row
let $no := $r/Objektkurztext/@content
return
<row>{$r/@*}{($r/*,<partno>{$no}</partno>)}</row>
}</sheet>
}</root>Using BaseX 9 the used memory for the query is round about 2 GB which is ok. Using BaseX 10 Out-of-Memory exception happens in my case used memory is > 12 GB
Any thoughts what the reason is - any additional information required?
Alex
CG
Christian Grün12 Aug 2022 18:01 #
Hi Alex,
Quoted text · Alexander Witzigmann · 4 lines
> declare variable $input := <root><sheet name="SAP">{for $idx in (1 to > 10000) return <row>{for $e in $enames return element {$e} { attribute > {"content"} {$idx}}}</row>}</sheet></root>; > db:create("test-db",$input,"test.xml")
Looking at the subsequent query, I assume there must be additional "Objektkurztext" elements. Do you have a query for us that we can run out of the box?
Thanks in advance,
Christian
AW
Alexander Witzigmann12 Aug 2022 18:18 #
Ah I see -for the create db part I missed the list of element names. Stored in variable $enames One of them is the referenced one and the rest are any name 5-15 chars long. There are round about 30 elements / I assume the real name doesn’t make a difference… each of them are unique in the list of enames. but I can send you those next week The created db using the same characteristic and volume than real world data and reproducing the issue at least on my test client
Alex
Quoted text · Christian Grün · 18 lines
> Am 12.08.2022 um 18:01 schrieb Christian Grün <christian.gruen at gmail.com>: > > > Diese Nachricht stammt von einem nicht vertrauenswürdigen Absender > Sie haben bisher nicht mit diesem Absender korrespondiert. > Hi Alex, > > > declare variable $input := <root><sheet name="SAP">{for $idx in (1 to > > 10000) return <row>{for $e in $enames return element {$e} { attribute > > {"content"} {$idx}}}</row>}</sheet></root>; > > db:create("test-db",$input,"test.xml") > > Looking at the subsequent query, I assume there must be additional > "Objektkurztext" elements. Do you have a query for us that we can run > out of the box? > > Thanks in advance, > Christian
CG
Christian Grün12 Aug 2022 18:34 #
I managed to get this reproduced [1]. A first JMC analysis indicates this seems to be related to the revised data structures for
main-memory updates [2]. A small overhead would be expectable and acceptable, but definitely not on this scale. Thanks for the observation.
[1] https://github.com/BaseXdb/basex/issues/2134
[2] https://docs.basex.org/wiki/BaseX_10#Main-Memory_Updates
CG
Christian Grün12 Aug 2022 20:28 #
…and I’m looking forward to your feedback on the latest snapshot [1].
Best,
Christian
[1] https://files.basex.org/releases/latest/
AW
Alexander Witzigmann16 Aug 2022 09:35 #
works wonderful - appr. same memory footprint as using 9.x release and similar and fast response time ...
thanks
alex
Quoted text · Christian Grün · 13 lines
Am 12.08.2022 um 20:28 schrieb Christian Grün: > …and I’m looking forward to your feedback on the latest snapshot [1]. > Best, Christian [1] https: //files. basex. org/releases/latest/ > > > > > …and I’m looking forward to your feedback on the latest snapshot [1]. > > Best, > Christian > > [1] https://files.basex.org/releases/latest/
Written for BaseX 10.0. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery