Xquery update question
Monday, 29 December 2014 · 3 messages · last reply 6 Jan 2015
EA
Erol Akarsu29 Dec 2014 20:40 #
Hi,
I have several xml document index keeping products features and have one master index file of products.. Currently, we have about 25 million products that have size about 25G xquery index size.
My xquery application is reading features data from auxiliary xml document and enhancing master document.
In order to do it, I am inserting data into master document with a generic insert command:
insert node <FEATURES_NODE> into <PRODUCT_NODE>
But xquery update is executing update process after all insert requests collected. But with huge size of my index, the memory of machine (35G) is exhausted easily and can NOT finish process.
Is there way to execute individual xquery update requests one by one so that process can finish?
Erol Akarsu
CG
Christian Grün6 Jan 2015 10:22 #
Hi Erol,
Quoted text · Erol Akarsu · 3 lines
> I have several xml document index keeping products features and have one > master index file of products.. Currently, we have about 25 million products > that have size about 25G xquery index size.
One solution is to generate a BaseX command script, which contains multiple subsequent update operations. It can e.g. be created as follows:
let $records := count(db:open('db')//product-node)
let $commands :=
for tumbling window $w in (1 to $records)
start at $s when true()
only end at $e when $e - $s = 999999
return 'for $product-node in //product-node' ||
'[position() = ' || $s || ' to ' || $e || '] ' ||
'return insert node <a/> into $product-node'
return file:write-text-lines('script.bxs', $commands)..and run via "basex script.bxs".
Hope this helps,
Christian
Quoted text · Erol Akarsu · 19 lines
> > My xquery application is reading features data from auxiliary xml document > and enhancing master document. > > In order to do it, I am inserting data into master document with a generic > insert command: > > insert node <FEATURES_NODE> into <PRODUCT_NODE> > > But xquery update is executing update process after all insert requests > collected. But with huge size of my index, the memory of machine (35G) is > exhausted easily and can NOT finish process. > > Is there way to execute individual xquery update requests one by one so that > process can finish? > > Erol Akarsu > >
EA
Erol Akarsu6 Jan 2015 14:39 #
Thanks Christian
That was I what I was thinking
Hopefully, it will resolve the issue,
Erol Akarsu
Quoted text · Christian Grün · 52 lines
On Tue, Jan 6, 2015 at 4:22 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Erol, > > > I have several xml document index keeping products features and have one > > master index file of products.. Currently, we have about 25 million > products > > that have size about 25G xquery index size. > > One solution is to generate a BaseX command script, which contains > multiple subsequent update operations. It can e.g. be created as > follows: > > let $records := count(db:open('db')//product-node) > let $commands := > for tumbling window $w in (1 to $records) > start at $s when true() > only end at $e when $e - $s = 999999 > return 'for $product-node in //product-node' || > '[position() = ' || $s || ' to ' || $e || '] ' || > 'return insert node <a/> into $product-node' > return file:write-text-lines('script.bxs', $commands) > > ..and run via "basex script.bxs". > > Hope this helps, > Christian > > > > > > My xquery application is reading features data from auxiliary xml > document > > and enhancing master document. > > > > In order to do it, I am inserting data into master document with a > generic > > insert command: > > > > insert node <FEATURES_NODE> into <PRODUCT_NODE> > > > > But xquery update is executing update process after all insert requests > > collected. But with huge size of my index, the memory of machine (35G) is > > exhausted easily and can NOT finish process. > > > > Is there way to execute individual xquery update requests one by one so > that > > process can finish? > > > > Erol Akarsu > > > > >
Written for BaseX 7.9. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery