BaseX performance improvements

> 1. each XML node => 500 bytes > 2. total size of XML DB => 10million nodes(which will grow continuously as > new files are added).

> I have to do a update of the nodes as I need to replace them. I chose delete > + insert instead of replace

> Also, I am thinking of indexing on the node id, as that is like the primary > key on which I base my operations, any suggestions might also be helpful,

> Thanks again, > Regards > Martin Lourduswamy > > On Sun, May 20, 2018 at 9:28 AM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Martin, >> >> > I am new to BaseX, I would like ot speed up XQuery of insert and delete >> > and >> > replace through options. >> >> Welcome to the list. As there are numerous ways to do updates in >> BaseX, feel free to give us more information on your insert and delete >> operations. Do you work with large single documents or many small >> documents? How large is your database? >> >> > While I query basex through perl, and try to >> > connect through GUI, the perl connection aborts. Is there a parameters >> > for >> > parallel connections, please let me know. >> >> I can’t tell why your perl connection is interrupted by opening the >> GUI (because they should be completely independent from each other). A >> step-by-step description on how you proceeded might be helpful. >> >> Because there is no coupling between GUI and the client/server >> architecture, however, you should avoid running updates outside the >> GUI. Please check out [1] for more information. >> >> Best, >> Christian >> >> [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations >> >> >> >> > >> > Thanks, >> > Regards >> > Martin Lourduswamy > >

> Java heap space at C:\Users\lourduswamym\oracle_scripts/BaseXClient.pm line > 213.

> Actually the error on the server log. The insert operation was taking 1+minutes due to slow network and then the error happened. I have been running this server instance continuously for 1 day with continuous insert from a per script.

> So I think I should be careful to run only one single process of client at a > time or make sure memory is enough to handle multiple client operations, > please let me know your thoughts,

> Thanks again, > Regards > Martin Lourduswamy > > On Sun, May 20, 2018 at 9:48 AM, Martin Lourduswamy <martin.louis at gmail.com> > wrote: >> >> Hi, >> >> Thanks for the clarifications. My database is >> >> DB Size: >> ====== >> 1. each XML node => 500 bytes >> 2. total size of XML DB => 10million nodes(which will grow continuously as >> new files are added). >> 3. I will be having multiple DB's of same size and more size as the DB >> continuously grows >> >> I have to do a update of the nodes as I need to replace them. I chose >> delete + insert instead of replace for each of the node through perl, as I >> will make sure there are no duplicate nodes left in the database.(just a >> precaution that even if some nodes get duplicated, through some other >> processes I will be able to remove them by delete) >> >> DB Speed: >> ======== >> I will disable logging as it might speed up the database operations. >> >> DB Architecture >> ============ >> 1 server(windows or Linux) running baseX >> multiple client computers trying to query the same machine for select >> through fn:doc(...) API to get the data through perl >> then do a delete + insert from perl >> >> I would like to know if you think this might work for production and any >> suggestion for data retrieval and update >> Any changes to the way I architecture it might help >> >> Also, I am thinking of indexing on the node id, as that is like the >> primary key on which I base my operations, any suggestions might also be >> helpful, >> Thanks again, >> Regards >> Martin Lourduswamy >> >> On Sun, May 20, 2018 at 9:28 AM, Christian Grün >> <christian.gruen at gmail.com> wrote: >>> >>> Hi Martin, >>> >>> > I am new to BaseX, I would like ot speed up XQuery of insert and delete >>> > and >>> > replace through options. >>> >>> Welcome to the list. As there are numerous ways to do updates in >>> BaseX, feel free to give us more information on your insert and delete >>> operations. Do you work with large single documents or many small >>> documents? How large is your database? >>> >>> > While I query basex through perl, and try to >>> > connect through GUI, the perl connection aborts. Is there a parameters >>> > for >>> > parallel connections, please let me know. >>> >>> I can’t tell why your perl connection is interrupted by opening the >>> GUI (because they should be completely independent from each other). A >>> step-by-step description on how you proceeded might be helpful. >>> >>> Because there is no coupling between GUI and the client/server >>> architecture, however, you should avoid running updates outside the >>> GUI. Please check out [1] for more information. >>> >>> Best, >>> Christian >>> >>> [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations >>> >>> >>> >>> > >>> > Thanks, >>> > Regards >>> > Martin Lourduswamy >> >> >

On Sun, May 20, 2018 at 12:51 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Martin (cc to the list), > > > 1. each XML node => 500 bytes > > 2. total size of XML DB => 10million nodes(which will grow continuously > as > > new files are added). > > So one node is one document, right? I’m just asking because, in the > XML terminology, each XML document has nodes itself: element nodes, > attributes nodes, text nodes, etc. > > If you are working with millions of documents, it can be helpful to > work with a "daily" database, which contains auxiliary references on > documents that have been deleted or updated. If there is a time > interval in which no people access your database (e.g. each night), > you can merge these databases and recreate your index structures. If > you query your database, you can a few more lines of XQuery: first > check your daily database, and (if it doesn’t contain the required > document) look up the document in your fully indexed database. > > > I have to do a update of the nodes as I need to replace them. I chose > delete > > + insert instead of replace > > Please note that a replace operation can be much faster: If the > structure of the updated document is similar to the old document, the > document can be replace in-place. > > > Also, I am thinking of indexing on the node id, as that is like the > primary > > key on which I base my operations, any suggestions might also be helpful, > > Here it may also be interesting to know if you will only address > documents or arbitrary nodes of your document. In the first case, > addressing the document path via db:open should be sufficient [1]. In > the latter case, you could possibly use the existing function > db:node-id [2]. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/Databases > [2] http://docs.basex.org/wiki/Database_Module#db:node-id > > > > > Thanks again, > > Regards > > Martin Lourduswamy > > > > On Sun, May 20, 2018 at 9:28 AM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi Martin, > >> > >> > I am new to BaseX, I would like ot speed up XQuery of insert and > delete > >> > and > >> > replace through options. > >> > >> Welcome to the list. As there are numerous ways to do updates in > >> BaseX, feel free to give us more information on your insert and delete > >> operations. Do you work with large single documents or many small > >> documents? How large is your database? > >> > >> > While I query basex through perl, and try to > >> > connect through GUI, the perl connection aborts. Is there a parameters > >> > for > >> > parallel connections, please let me know. > >> > >> I can’t tell why your perl connection is interrupted by opening the > >> GUI (because they should be completely independent from each other). A > >> step-by-step description on how you proceeded might be helpful. > >> > >> Because there is no coupling between GUI and the client/server > >> architecture, however, you should avoid running updates outside the > >> GUI. Please check out [1] for more information. > >> > >> Best, > >> Christian > >> > >> [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations > >> > >> > >> > >> > > >> > Thanks, > >> > Regards > >> > Martin Lourduswamy > > > > >

> Hi, > Can we delete a node with a node id got from a query like this > > Query: db:node-id(db:text("martin1", > "23b2513e6118bd32d53b128d19d3390f6374abac")) > I got a id 54210971 > Can I use it to delete it form database, > Thanks, > Regards > > On Sun, May 20, 2018 at 12:51 PM, Christian Grün < > christian.gruen at gmail.com> wrote: > >> Hi Martin (cc to the list), >> >> > 1. each XML node => 500 bytes >> > 2. total size of XML DB => 10million nodes(which will grow continuously >> as >> > new files are added). >> >> So one node is one document, right? I’m just asking because, in the >> XML terminology, each XML document has nodes itself: element nodes, >> attributes nodes, text nodes, etc. >> >> If you are working with millions of documents, it can be helpful to >> work with a "daily" database, which contains auxiliary references on >> documents that have been deleted or updated. If there is a time >> interval in which no people access your database (e.g. each night), >> you can merge these databases and recreate your index structures. If >> you query your database, you can a few more lines of XQuery: first >> check your daily database, and (if it doesn’t contain the required >> document) look up the document in your fully indexed database. >> >> > I have to do a update of the nodes as I need to replace them. I chose >> delete >> > + insert instead of replace >> >> Please note that a replace operation can be much faster: If the >> structure of the updated document is similar to the old document, the >> document can be replace in-place. >> >> > Also, I am thinking of indexing on the node id, as that is like the >> primary >> > key on which I base my operations, any suggestions might also be >> helpful, >> >> Here it may also be interesting to know if you will only address >> documents or arbitrary nodes of your document. In the first case, >> addressing the document path via db:open should be sufficient [1]. In >> the latter case, you could possibly use the existing function >> db:node-id [2]. >> >> Hope this helps, >> Christian >> >> [1] http://docs.basex.org/wiki/Databases >> [2] http://docs.basex.org/wiki/Database_Module#db:node-id >> >> >> >> > Thanks again, >> > Regards >> > Martin Lourduswamy >> > >> > On Sun, May 20, 2018 at 9:28 AM, Christian Grün < >> christian.gruen at gmail.com> >> > wrote: >> >> >> >> Hi Martin, >> >> >> >> > I am new to BaseX, I would like ot speed up XQuery of insert and >> delete >> >> > and >> >> > replace through options. >> >> >> >> Welcome to the list. As there are numerous ways to do updates in >> >> BaseX, feel free to give us more information on your insert and delete >> >> operations. Do you work with large single documents or many small >> >> documents? How large is your database? >> >> >> >> > While I query basex through perl, and try to >> >> > connect through GUI, the perl connection aborts. Is there a >> parameters >> >> > for >> >> > parallel connections, please let me know. >> >> >> >> I can’t tell why your perl connection is interrupted by opening the >> >> GUI (because they should be completely independent from each other). A >> >> step-by-step description on how you proceeded might be helpful. >> >> >> >> Because there is no coupling between GUI and the client/server >> >> architecture, however, you should avoid running updates outside the >> >> GUI. Please check out [1] for more information. >> >> >> >> Best, >> >> Christian >> >> >> >> [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations >> >> >> >> >> >> >> >> > >> >> > Thanks, >> >> > Regards >> >> > Martin Lourduswamy >> > >> > >> > >