Main Page » Mailing List » July 2016

db:optimize questions

Friday, 29 July 2016 · 8 messages · last reply 2 Aug 2016
BJ
Baysdon, Joseph29 Jul 2016 20:43 #
I a new baseX user and I have a few questions regarding the db:optimize command. Does 'optimize all' perform all available optimizations or are there other parameters (indexing or full-text as mentioned at http://docs.basex.org/wiki/Database_Module#db:optimize) to supply? Is read/write possible to a database undergoing optimization? I’m guessing no because I get an error (“Database ‘foo’ is currently opened by another process”) whenever another connection to the database is open. On what size or content of databases does optimization have a significant impact? Currently on a 400Mb database, I don't see any performance benefits after optimization. (Note: all of my performance testing is using a simple db:list command.) Thanks in advance for any assistance you can provide. Joe
CG
Christian Grün30 Jul 2016 22:22 #
Hi Joe, Welcome to the list!
Quoted text · Baysdon, Joseph · 3 lines
> Does 'optimize all' perform all available optimizations or are there other > parameters (indexing or full-text as mentioned at > http://docs.basex.org/wiki/Database_Module#db:optimize) to supply?
If optimize is called (with and without all), the existing index structures will be updated. Via db:optimize, you can additionally, enable or disable specific index structures. If you run 'optimize all', the database will be completely rebuilt.
Quoted text · Baysdon, Joseph · 3 lines
> Is read/write possible to a database undergoing optimization? I’m guessing > no because I get an error (“Database ‘foo’ is currently opened by another > process”) whenever another connection to the database is open.
Right. This error indicates that you should always access a database within the same JVM context (unless you restrict yourself to read operations) [1].
Quoted text · Baysdon, Joseph · 3 lines
> On what size or content of databases does optimization have a significant > impact? Currently on a 400Mb database, I don't see any performance benefits > after optimization.
Optimizations are only required after update operations. A newly created database is fully optimized. If you believe that specific queries should run faster, feel free to provide us with the query strings. Cheers Christian [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations
BJ
Baysdon, Joseph1 Aug 2016 15:25 #
Thanks for the guidance, Christian. I do have two short follow-up questions: Does an optimized db (via db:optimize) perform any differently than an un-optimized db for document retrieval, as opposed to XQuery execution over document(s)? Also, I noticed that I would see the “Database ‘foo’ is currently opened by another process” error message even if there was only had one *other* db connection, within the same JVM. Is this expected? Or would you think that there must be another process (separate JVM) connected to the db? Thanks again, Joe
Quoted text · Christian Grün · 1 line
On 7/30/16, 4:22 PM, "Christian Grün" <christian.gruen at gmail.com> wrote:
Hi Joe, Welcome to the list!
Quoted text · Christian Grün · 3 lines
> Does 'optimize all' perform all available optimizations or are there other > parameters (indexing or full-text as mentioned at > http://docs.basex.org/wiki/Database_Module#db:optimize) to supply?
If optimize is called (with and without all), the existing index structures will be updated. Via db:optimize, you can additionally, enable or disable specific index structures. If you run 'optimize all', the database will be completely rebuilt.
Quoted text · Christian Grün · 3 lines
> Is read/write possible to a database undergoing optimization? I’m guessing > no because I get an error (“Database ‘foo’ is currently opened by another > process”) whenever another connection to the database is open.
Right. This error indicates that you should always access a database within the same JVM context (unless you restrict yourself to read operations) [1].
Quoted text · Christian Grün · 3 lines
> On what size or content of databases does optimization have a significant > impact? Currently on a 400Mb database, I don't see any performance benefits > after optimization.
Optimizations are only required after update operations. A newly created database is fully optimized. If you believe that specific queries should run faster, feel free to provide us with the query strings. Cheers Christian [1] http://docs.basex.org/wiki/Startup#Concurrent_Operations
CG
Christian Grün1 Aug 2016 22:51 #
Hi Joe,
Quoted text · Baysdon, Joseph · 1 line
> Does an optimized db (via db:optimize) perform any differently than an un-optimized db for document retrieval, as opposed to XQuery execution over document(s)?
If you perform updates on a database, your queries may get slower over time. db:optimize can be used to speed up queries again. Does that answer your query? I must confess I do not exactly understand which difference you see between "document retrieval" and "XQuery execution over document(s)"; could you please give me some more hints?
Quoted text · Baysdon, Joseph · 1 line
> Also, I noticed that I would see the “Database ‘foo’ is currently opened by another process” error message even if there was only had one *other* db connection, within the same JVM. Is this expected? Or would you think that there must be another process (separate JVM) connected to the db?
There must be at least one other process locking the accessed database (see [1] for more info). Did you open the GUI at the same time? Christian [1] http://docs.basex.org/wiki/Transaction_Management#Database_Locks
BJ
Baysdon, Joseph2 Aug 2016 17:45 #
Thanks, Christian. I’ll try to be more specific… Does an optimized database perform better than an unoptimized database for db:open? How about for db:list? Or db:retrieve? As for the another process locking the accessed database, is it possible to have multiple, open db connections, within the same JVM, and use one of them to optimize and another to read from the db? I’m asking because my testing indicates that I can only have one open connection if I want to run db:optimize. If I have two open connections (again, within the same JVM), then my db:optimize will fail. Thanks, Joe
Quoted text · Christian Grün · 1 line
On 8/1/16, 4:51 PM, "Christian Grün" <christian.gruen at gmail.com> wrote:
Hi Joe,
Quoted text · Christian Grün · 1 line
> Does an optimized db (via db:optimize) perform any differently than an un-optimized db for document retrieval, as opposed to XQuery execution over document(s)?
If you perform updates on a database, your queries may get slower over time. db:optimize can be used to speed up queries again. Does that answer your query? I must confess I do not exactly understand which difference you see between "document retrieval" and "XQuery execution over document(s)"; could you please give me some more hints?
Quoted text · Christian Grün · 1 line
> Also, I noticed that I would see the “Database ‘foo’ is currently opened by another process” error message even if there was only had one *other* db connection, within the same JVM. Is this expected? Or would you think that there must be another process (separate JVM) connected to the db?
There must be at least one other process locking the accessed database (see [1] for more info). Did you open the GUI at the same time? Christian [1] http://docs.basex.org/wiki/Transaction_Management#Database_Locks
CG
Christian Grün2 Aug 2016 17:51 #
Hi Joseph,
Quoted text · Baysdon, Joseph · 1 line
> Does an optimized database perform better than an unoptimized database for db:open? How about for db:list? Or db:retrieve?
It strongly depends on the update operations you perform. The most tangible difference between optimized and updated databases will be that your (non-incremental) index structures get lost [1]. When it comes to pure retrieval of database resources, there should hardly be a difference. In our article on indexes [2], you will see which index structures are affected by update operations.
Quoted text · Baysdon, Joseph · 1 line
> As for the another process locking the accessed database, is it possible to have multiple, open db connections, within the same JVM.
When talking about open connections, do you use the client/server of BaseX [3]? Christian [1] http://docs.basex.org/wiki/XQuery_Update#Indexes [2] http://docs.basex.org/wiki/Indexes [3] http://docs.basex.org/wiki/Startup
BJ
Baysdon, Joseph2 Aug 2016 18:05 #
Great answer regarding resource retrieval and optimization. Thanks, Christian! Regarding the possibility of multiple db connections (w/in same JVM) and using one of them to db:optimize, we are using the client/server configuration. Joe
Quoted text · Christian Grün · 1 line
On 8/2/16, 11:51 AM, "Christian Grün" <christian.gruen at gmail.com> wrote:
Hi Joseph,
Quoted text · Christian Grün · 1 line
> Does an optimized database perform better than an unoptimized database for db:open? How about for db:list? Or db:retrieve?
It strongly depends on the update operations you perform. The most tangible difference between optimized and updated databases will be that your (non-incremental) index structures get lost [1]. When it comes to pure retrieval of database resources, there should hardly be a difference. In our article on indexes [2], you will see which index structures are affected by update operations.
Quoted text · Christian Grün · 1 line
> As for the another process locking the accessed database, is it possible to have multiple, open db connections, within the same JVM.
When talking about open connections, do you use the client/server of BaseX [3]? Christian [1] http://docs.basex.org/wiki/XQuery_Update#Indexes [2] http://docs.basex.org/wiki/Indexes [3] http://docs.basex.org/wiki/Startup
CG
Christian Grün2 Aug 2016 18:10 #
Quoted text · Baysdon, Joseph · 1 line
> Regarding the possibility of multiple db connections (w/in same JVM) and using one of them to db:optimize, we are using the client/server configuration.
If you use database clients, all conflicting transactions will be queued. Here’s yet another Wiki article that gives you insight into the details [1]. Christian [1] http://docs.basex.org/wiki/Transaction_Management
Quoted text · Christian Grün · 24 lines
> On 8/2/16, 11:51 AM, "Christian Grün" <christian.gruen at gmail.com> wrote: > > Hi Joseph, > > > Does an optimized database perform better than an unoptimized database for db:open? How about for db:list? Or db:retrieve? > > It strongly depends on the update operations you perform. The most > tangible difference between optimized and updated databases will be > that your (non-incremental) index structures get lost [1]. When it > comes to pure retrieval of database resources, there should hardly be > a difference. In our article on indexes [2], you will see which index > structures are affected by update operations. > > > As for the another process locking the accessed database, is it possible to have multiple, open db connections, within the same JVM. > > When talking about open connections, do you use the client/server of BaseX [3]? > > Christian > > [1] http://docs.basex.org/wiki/XQuery_Update#Indexes > [2] http://docs.basex.org/wiki/Indexes > [3] http://docs.basex.org/wiki/Startup > >

Written for BaseX 8.5. BaseX may have changed since: please check the current documentation.

« users.xml change outside basexInconsistency of regular expression support in BaseX and GUI »
⚡Generated with XQuery