Main Page » Mailing List » April 2019

Add & replace documents

Thursday, 11 April 2019 · 5 messages
JB
Johannes Bauer11 Apr 2019 09:13 #
Hi, this may seem like a dumb question, but how can I add & replace a number of documents in a database? I have a directory with some files that I want to import. Some of the documents may already exist in the database. If I use the db:replace('db', '', '/path/to/my/files') function it adds the missing docs and overwrites the existing ones. But it also deletes all other documents from the database. I cannot work with db:add() because it does not overwrite existing docs. Any suggestions? Best regards Johannes
CG
Christian Grün11 Apr 2019 09:33 #
Hi Johannes, We allow file paths for db:add and db:replace as a basic convenience feature. To have full control over your imports, it is often better to work with the functions provided by the File Module. Herre is a simple example that might already cover your requirements (provided that the input directory contains only XML documents):
  let $root := '/path/to/your/files/'
  for $path in file:list($root, true())
  return db:replace('your-db', $path, $root || $path)
Hope this helps, Christian
Quoted text · Johannes Bauer · 26 lines
On Thu, Apr 11, 2019 at 9:13 AM Johannes Bauer <johannes.bauer at tanner.de> wrote: > > Hi, > > this may seem like a dumb question, but how can I add & replace a number > of documents in a database? > > I have a directory with some files that I want to import. Some of the > documents may already exist in the database. > > If I use the db:replace('db', '', '/path/to/my/files') function it adds > the missing docs and overwrites the existing ones. > But it also deletes all other documents from the database. > > I cannot work with db:add() because it does not overwrite existing docs. > > Any suggestions? > > Best regards > Johannes > > > > > >
JB
Johannes Bauer11 Apr 2019 09:57 #
Hello Christian, thank you for the fast reply. With your example I will actually make a db:replace() call for each file. I've choosen the approach with just passing the directory to db:replace() for performance reasons. My experience is that this call is extremely fast even for a large number of files (let's say > 30.000). Will the iterative approach have a comparable performance? Or are there some optimizations for directory structures in db:replace(). Thanks Johannes
Quoted text · Christian Grün · 42 lines
Am 11.04.2019 um 09:33 schrieb Christian Grün: > Hi Johannes, > > We allow file paths for db:add and db:replace as a basic convenience > feature. To have full control over your imports, it is often better to > work with the functions provided by the File Module. Herre is a simple > example that might already cover your requirements (provided that the > input directory contains only XML documents): > > let $root := '/path/to/your/files/' > for $path in file:list($root, true()) > return db:replace('your-db', $path, $root || $path) > > Hope this helps, > Christian > > > On Thu, Apr 11, 2019 at 9:13 AM Johannes Bauer <johannes.bauer at tanner.de> wrote: >> Hi, >> >> this may seem like a dumb question, but how can I add & replace a number >> of documents in a database? >> >> I have a directory with some files that I want to import. Some of the >> documents may already exist in the database. >> >> If I use the db:replace('db', '', '/path/to/my/files') function it adds >> the missing docs and overwrites the existing ones. >> But it also deletes all other documents from the database. >> >> I cannot work with db:add() because it does not overwrite existing docs. >> >> Any suggestions? >> >> Best regards >> Johannes >> >> >> >> >> >>
CG
Christian Grün11 Apr 2019 10:00 #
Quoted text · Johannes Bauer · 2 lines
> Will the iterative approach have a comparable performance? Or are there > some optimizations for directory structures in db:replace().
This always depends on various factors (on the input sizes, the directory structure, possibly even the file system). If it turns out that it would be much faster to add a specific option to db:add and db:replace, we could think about that as well. Cheers, Christian
Quoted text · Johannes Bauer · 44 lines
> Am 11.04.2019 um 09:33 schrieb Christian Grün: > > Hi Johannes, > > > > We allow file paths for db:add and db:replace as a basic convenience > > feature. To have full control over your imports, it is often better to > > work with the functions provided by the File Module. Herre is a simple > > example that might already cover your requirements (provided that the > > input directory contains only XML documents): > > > > let $root := '/path/to/your/files/' > > for $path in file:list($root, true()) > > return db:replace('your-db', $path, $root || $path) > > > > Hope this helps, > > Christian > > > > > > On Thu, Apr 11, 2019 at 9:13 AM Johannes Bauer <johannes.bauer at tanner.de> wrote: > >> Hi, > >> > >> this may seem like a dumb question, but how can I add & replace a number > >> of documents in a database? > >> > >> I have a directory with some files that I want to import. Some of the > >> documents may already exist in the database. > >> > >> If I use the db:replace('db', '', '/path/to/my/files') function it adds > >> the missing docs and overwrites the existing ones. > >> But it also deletes all other documents from the database. > >> > >> I cannot work with db:add() because it does not overwrite existing docs. > >> > >> Any suggestions? > >> > >> Best regards > >> Johannes > >> > >> > >> > >> > >> > >> > >
JB
Johannes Bauer11 Apr 2019 10:04 #
ok, I will run some tests and see if it works and get back to you. Thanks Johannes
Quoted text · Christian Grün · 56 lines
Am 11.04.2019 um 10:00 schrieb Christian Grün: >> Will the iterative approach have a comparable performance? Or are there >> some optimizations for directory structures in db:replace(). > This always depends on various factors (on the input sizes, the > directory structure, possibly even the file system). If it turns out > that it would be much faster to add a specific option to db:add and > db:replace, we could think about that as well. > > Cheers, > Christian > > > >> Am 11.04.2019 um 09:33 schrieb Christian Grün: >>> Hi Johannes, >>> >>> We allow file paths for db:add and db:replace as a basic convenience >>> feature. To have full control over your imports, it is often better to >>> work with the functions provided by the File Module. Herre is a simple >>> example that might already cover your requirements (provided that the >>> input directory contains only XML documents): >>> >>> let $root := '/path/to/your/files/' >>> for $path in file:list($root, true()) >>> return db:replace('your-db', $path, $root || $path) >>> >>> Hope this helps, >>> Christian >>> >>> >>> On Thu, Apr 11, 2019 at 9:13 AM Johannes Bauer <johannes.bauer at tanner.de> wrote: >>>> Hi, >>>> >>>> this may seem like a dumb question, but how can I add & replace a number >>>> of documents in a database? >>>> >>>> I have a directory with some files that I want to import. Some of the >>>> documents may already exist in the database. >>>> >>>> If I use the db:replace('db', '', '/path/to/my/files') function it adds >>>> the missing docs and overwrites the existing ones. >>>> But it also deletes all other documents from the database. >>>> >>>> I cannot work with db:add() because it does not overwrite existing docs. >>>> >>>> Any suggestions? >>>> >>>> Best regards >>>> Johannes >>>> >>>> >>>> >>>> >>>> >>>> >>

Written for BaseX 9.1. BaseX may have changed since: please check the current documentation.

« Func def & performance: element()* vs item()*FTP client module »
⚡Generated with XQuery