Main Page » Mailing List » February 2015

Design pattern ...

Wednesday, 18 February 2015 · 3 messages · last reply 20 Feb 2015
ML
Marco Lettere18 Feb 2015 12:32 #
Hi all, I start from the following condition: - 2 dbs (let's say A and B) with around 200K documents each - every document in B refers to exactly one document in A (many to one is possible) - I have to apply a filter on every document of A and depending on the boolean outcome "move" the document to anoter db A'. - According to this the correlated documents of B need to be "moved" to a database B'. - The solution should be made automatic in the mid term What is the best design pattern and tool (I use the GUI by now) for this finding the right compromise for limiting "Out of memory (huge PUL)", "too much time", "complexity introduced by chunking" and "absence of a _db:move-document_ operation for moving one document in one single operation? Every suggestion is more then warmly welcome! Thanks and best regards, Marco
CG
Christian Grün20 Feb 2015 12:18 #
Hi Marco, Finally some feedback. As you may already have guessed, there is no obvious answer to your question. A straightforward query could look as follows..
  let $docs-a := db:open('A1')[ ..condition.. ]
  let $paths := $docs-a ! db:path(.)
  let $docs-b := $paths ! db:open('A1', .)
  return (
    db:delete('A1', $paths),
    db:add('A2', $docs-a),
    db:delete('B1', $paths),
    db:add('B2', $docs-b)
  )
..but it probably leads to out of memory exceptions. An alternative could be to serialize the documents to disk and try to add it in a second step. If this doesn't work out neither, I would indeed propose to move the documents in chunks. A BaseX command script for that could be generated via XQuery. I will think about the implications of db:copy-doc and db:move-doc functions. It's right that such scripts may not be obvious candidates for OOM errors, because we already have database representations for the documents to be moved/copied. Another solution to avoid OOM would be to write the data structures of copied documents to disk instead of keeping them in main memory. Mostly a matter of when we find enough time to make it happen.. Cheers, Christian
Quoted text · Marco Lettere · 20 lines
> - 2 dbs (let's say A and B) with around 200K documents each > - every document in B refers to exactly one document in A (many to one is > possible) > > - I have to apply a filter on every document of A and depending on the > boolean outcome "move" the document to anoter db A'. > - According to this the correlated documents of B need to be "moved" to a > database B'. > - The solution should be made automatic in the mid term > > What is the best design pattern and tool (I use the GUI by now) for this > finding the right compromise for limiting "Out of memory (huge PUL)", "too > much time", "complexity introduced by chunking" and "absence of a > _db:move-document_ operation for moving one document in one single > operation? > > Every suggestion is more then warmly welcome! > > Thanks and best regards, > Marco
ML
Marco Lettere20 Feb 2015 13:02 #
Thank you Christian, my plans were to serialize to disk but I'll give a try to the COMMAND generation approach. Separating the work in two phases could anyway improve the process from a safety point of view ... Will keep you ligned up. Regards, Marco.
Quoted text · Christian Grün · 56 lines
On 20/02/2015 12:18, Christian Grün wrote: > Hi Marco, > > Finally some feedback. As you may already have guessed, there is no > obvious answer to your question. A straightforward query could look as > follows.. > > let $docs-a := db:open('A1')[ ..condition.. ] > let $paths := $docs-a ! db:path(.) > let $docs-b := $paths ! db:open('A1', .) > return ( > db:delete('A1', $paths), > db:add('A2', $docs-a), > db:delete('B1', $paths), > db:add('B2', $docs-b) > ) > > ..but it probably leads to out of memory exceptions. An alternative > could be to serialize the documents to disk and try to add it in a > second step. If this doesn't work out neither, I would indeed propose > to move the documents in chunks. A BaseX command script for that could > be generated via XQuery. > > I will think about the implications of db:copy-doc and db:move-doc > functions. It's right that such scripts may not be obvious candidates > for OOM errors, because we already have database representations for > the documents to be moved/copied. Another solution to avoid OOM would > be to write the data structures of copied documents to disk instead of > keeping them in main memory. Mostly a matter of when we find enough > time to make it happen.. > > Cheers, > Christian > > > >> - 2 dbs (let's say A and B) with around 200K documents each >> - every document in B refers to exactly one document in A (many to one is >> possible) >> >> - I have to apply a filter on every document of A and depending on the >> boolean outcome "move" the document to anoter db A'. >> - According to this the correlated documents of B need to be "moved" to a >> database B'. >> - The solution should be made automatic in the mid term >> >> What is the best design pattern and tool (I use the GUI by now) for this >> finding the right compromise for limiting "Out of memory (huge PUL)", "too >> much time", "complexity introduced by chunking" and "absence of a >> _db:move-document_ operation for moving one document in one single >> operation? >> >> Every suggestion is more then warmly welcome! >> >> Thanks and best regards, >> Marco

Written for BaseX 8.0. BaseX may have changed since: please check the current documentation.

« Regression? Unknown parameter: 'wrap' in 8.0.xBug in XQuery archive module »
⚡Generated with XQuery