Incremental updates: Best practices?
Wednesday, 12 March 2025 · 3 messages · last reply 14 Mar 2025
FS
Florian Schmitt12 Mar 2025 15:34 #
Hi all,
I’m searching for a way to apply “incremental updates” to a database. By “incremental updates”, I refer to the following situation:
1. Starting point is a database containing several xml resources, e.g.
/coll1/res1.xml
/coll1/res2.xml
/coll2/res3.xml.
2. The update consists of a file structure like
/coll1/res2.xml
/coll2/res4.xml
/coll3/res5.xml
- so I want to update the existing resource /coll1/res2.xml and add some additional resources /coll2.res4.xml, /coll3/res5.xml.
The desired result is a structure like this:
/coll1/res1.xml (untouched)
/coll1/res2.xml (updated)
/coll2/res3.xml (untouched)
/coll2/res4.xml (added)
/coll3/res5.xml (added)
Using the command line client and the “ADD” command, I’m able to apply the update content in a single step, but (as the documentation predicted) /coll1/res2.xml now appears twice. This isn’t the desired result - /coll1.res2.xml should be overwritten by the content of / coll1.res2.xml from the update.
Using the command line client and the “PUT” command replaces / coll1.res2.xml and adds the additional resource. But it deletes / coll1/res1.xml and /coll2/res3.xml - again, not the desired result, since resources missing in the update should simply be kept untouched.
In my real use case, the initial dataset contains about 18,000 resources, updates may contain hundreds of updated or new resources, so separating ADD and PUT operations manually isn't possible.
A simply but ugly solution would be to keep the initial set of resources in the file system, "apply" the update there and import the complete set of files as new Database. Another solution would be to write a dedicated xquery that checks for each single resource to apply a PUT or ADD operation.
Is there a better way for such a use case?
Florian
CG
Christian Grün13 Mar 2025 15:13 #
Hi Florian,
The most flexible solution is an XQuery script that traverses all files:
let $root := 'local-root-folder/'
let $db := 'your-fb'
for $path in file:list($root, true())
return db:put($db, doc($root || $path), $path)If you like, you can do different things if a file exists or not:
let $root := 'local-root-folder/'
let $db := 'your-fb'
for $path in file:list($root, true())
return if(db:exists($db, $path)) then (
db:put($db, doc($root || $path), $path)
) else (
db:add($db, doc($root || $path), $path)
)Hope this helps,
Christian
Quoted text · Florian Schmitt · 55 lines
On Wed, Mar 12, 2025 at 3:34 PM Florian Schmitt < ml-basex-talk at florian-schmitt.net> wrote: > Hi all, > > I’m searching for a way to apply “incremental updates” to a database. > By “incremental updates”, I refer to the following situation: > > 1. Starting point is a database containing several xml resources, e.g. > > /coll1/res1.xml > /coll1/res2.xml > /coll2/res3.xml. > > 2. The update consists of a file structure like > > /coll1/res2.xml > /coll2/res4.xml > /coll3/res5.xml > > - so I want to update the existing resource /coll1/res2.xml and add > some additional resources /coll2.res4.xml, /coll3/res5.xml. > > The desired result is a structure like this: > > /coll1/res1.xml (untouched) > /coll1/res2.xml (updated) > /coll2/res3.xml (untouched) > /coll2/res4.xml (added) > /coll3/res5.xml (added) > > Using the command line client and the “ADD” command, I’m able to apply > the update content in a single step, but (as the documentation > predicted) /coll1/res2.xml now appears twice. This isn’t the desired > result - /coll1.res2.xml should be overwritten by the content of / > coll1.res2.xml from the update. > > Using the command line client and the “PUT” command replaces / > coll1.res2.xml and adds the additional resource. But it deletes / > coll1/res1.xml and /coll2/res3.xml - again, not the desired result, > since resources missing in the update should simply be kept untouched. > > In my real use case, the initial dataset contains about 18,000 > resources, updates may contain hundreds of updated or new resources, > so separating ADD and PUT operations manually isn't possible. > > A simply but ugly solution would be to keep the initial set of > resources in the file system, "apply" the update there and import the > complete set of files as new Database. Another solution would be to > write a dedicated xquery that checks for each single resource to apply > a PUT or ADD operation. > > Is there a better way for such a use case? > > Florian >
M
ml-basex-talk14 Mar 2025 15:49 #
Hi Christian,
thanks a lot, looks fine! I've just added a where clause to filter for files only, leaving out directories:
let $root := '/mnt/baseX_files/iu_test2/'
let $db := 'IU_Test'
for $path in file:list($root, true())
*where file:is-file($root || $path) *
return if(db:exists($db, $path)) then (
db:put($db, doc($root || $path), $path)
) else (
db:add($db, doc($root || $path), $path)
)Works like a charm!
Yours
Florian
Written for BaseX 11.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery