Large updating query

On Fri, Oct 29, 2010 at 5:23 PM, Huib Verweij <huib.verwey at mpi.nl> wrote: > Hi BaseX-ers, > > I'm running into an out-of-memory error when running an updating query over (almost) every document in the database. > > Every doc has the form > > <root> > <entry><node3/><node2/>...</entry> > <entry><node2/><node1/>...</entry> > ... > </root> > > I am reordering the elements in every <entry/> element in every document (e.g. <node1/><node2/><node3/>). The query is easy enough to write, it reorders the elements in every <entry/> and replaces the <entry/> in the document with the reordered <entry/>. It works for just one document, but as I said, the db keels over when applying the query to every document. (I am not using any doc() function by the way, just matching on the top level <root/> node.) > > Do you have a suggestion how I could make this work? I've looked at the documentation on the BaseX website about XQuery Update, but frankly, I do not really understand the bit about documents not being updated, I just select a node from the database, update it, works fine... I must be missing something... (the point probably ;-)). > > I am using the basex REST api. > Huib Verweij.

> Huib, > > thanks for your mail. Our XQUF expert is currently on vacation, so it > might take a while until this will be looked at. Apart from that, how > does your query look like? And... ideally... could you provide us with > an example that allows us to reproduce the problem? >

> PS: The index build problem you mentioned earlier should have been > fixed with 6.3. I managed to build text and attribute indexes for 2 GB > XML instances with 50 MB RAM. >

On Sat, Oct 30, 2010 at 9:21 AM, Huib Verweij <huib.verwey at mpi.nl> wrote: > Hi Christian, > > Op 29 okt 2010, om 17:34 heeft Christian Grün het volgende geschreven: > >> Huib, >> >> thanks for your mail. Our XQUF expert is currently on vacation, so it >> might take a while until this will be looked at. Apart from that, how >> does your query look like? And... ideally... could you provide us with >> an example that allows us to reproduce the problem? >> > > the query ls this: > > <query> > <text> > > declare namespace lexus="http://www.mpi.nl/lat/lexus"; > > > (: > Order the container and data elements in the lexical-entry/container by schema-order. > :) > declare function lexus:orderNodes($le as node()*, $schema as node()*) as node()* { > for $sc in $schema > let $containers := $le[@schema-ref eq $sc/@id] > return > for $lc in $containers > return element container {$lc/@*, lexus:orderNodes($lc/*, $sc/*)} > }; > > (: > Return a lexical-entry with ordered container and data elements. > :) > declare function lexus:orderLE($le as node(), $schema as node()*) as node() { > element lexical-entry { > $le/@*, lexus:orderNodes($le/*, $schema) > } > }; > > (: replace the lexical entry with an ordered one :) > declare updating function lexus:updateLE($le as node(), $schema as node()*) { > replace node $le with lexus:orderLE($le, $schema) > }; > > (: process all lexical entries in a lexicon :) > declare updating function lexus:updateLexicon($lexus as node()) { > let $schema := $lexus/meta/schema//container[@type eq 'lexical-entry']/* > for $le in $lexus/lexicon/lexical-entry > return lexus:updateLE($le, $schema) > }; > > for $lexus in /lexus return lexus:updateLexicon($lexus) > </text> > </query> > > Short description: > > In lexus:updateLexicon() the schema for a lexical entry is retrieved (the ordered sequence of container elements that make up a lexical entry). > > For each lexical entry in the lexicon element the update function updateLE is called where the lexical entry is replaced by an order version of it. > > The ordering function orders the elements in the lexical entry by schema order. > > I have about 830Mb of data in the db. Most lexica are 1Mb or less, one is 41Mb, around 35 lexica are larger than 10Mb. > > >> PS: The index build problem you mentioned earlier should have been >> fixed with 6.3. I managed to build text and attribute indexes for 2 GB >> XML instances with 50 MB RAM. >> > > YES! I noticed! Well done! > > Hartelijke groet, > > Huib. > >

> With XQuery Update, all > updates are performed at the end of query evaluation. A so-called > "pending update list" is created, which includes all update operations > to be performed. Probably, this list will get too large to fit into > main memory, as it contains all temporary element fragments, which are > created by your query.

> > I'll pass this on to Lukas, when he's back; maybe he's got some ideas > to reduce memory consumption for your query. It might take a while..