BaseX db questions
Saturday, 23 January 2010 · 8 messages · last reply 29 Jan 2010
A
aleks23 Jan 2010 00:49 #
Hello,
I'm evaluating baseX database and trying to figure out few details:
1) What is the concept of the Context ? Do I need to create a new context for every database operation or same context should be used within open database ?
2) Is it true that I have to open and close database every time I run query or I can keep database open.
3) How does commit occur, with basex6 it seems like data being committed to disk with some delay?
Thanks a lot in advance!!
--Aleks
CG
Christian Grün24 Jan 2010 15:55 #
Dear Aleks,
thanks for your questions:
Quoted text · aleks · 1 line
> 1) What is the concept of the Context ? Do I need to create a new context for every database operation or same context should be used within open database ?
One Context instance should be used for all database operations. It contains a reference to the currently opened database. I would only recommend to create multiple Context instances if you need to open several databases at the same time.
Quoted text · aleks · 1 line
> 2) Is it true that I have to open and close database every time I run query or I can keep database open.
The database will be kept open until you send a Close command, or open another database. The following code snippet might give you some more hints:
http://www.inf.uni-konstanz.de/dbis/basex/code.php?code=DatabaseExample
Quoted text · aleks · 1 line
> 3) How does commit occur, with basex6 it seems like data being committed to disk with some delay?
All modified data is flushed to disk as soon as an XQuery Update expression has been completely processed, so no delays should occur.
Hope this helps,
Christian
CG
Dear Aleks,
Quoted text · Alexander Sheynkman · 1 line
> Thank you very much for your time, Just to clarify on point #2 regarding close. What I understood from you if I have an application that is managing multiple embedded database ( let say 3 ) at the same time and needs to constantly run query / updates on these db's , then I just need to Open it once on startup and the close it whenever I'm done with database.
That's true. Opening database is a very fast operation, however, so a second, easier alternative is to implicitly open and query the database within XQuery expressions, as shown in the following two sample queries:
-- doc('dbname')/path/to/elements
-- for $x in basex:db('dbname')/to/be/deleted return delete node $x
Quoted text · Alexander Sheynkman · 1 line
> How about OPTIMIZE , I see you suggested to run it after all kinds of database updates, is it true ?
The optimize command refreshes meta data and index structures of a database, which might speed up the evaluation of new XQuery requests. We decided to make database optimization an optional feature to allow for a much faster evaluation of update queries.
Feel free to ask for more,
Christian
AS
Hi Christian,
Thank you very much for your time, Just to clarify on point #2 regarding close. What I understood from you if I have an application that is managing multiple embedded database ( let say 3 ) at the same time and needs to constantly run query / updates on these db's , then I just need to Open it once on startup and the close it whenever I'm done with database.
How about OPTIMIZE , I see you suggested to run it after all kinds of database updates, is it true ?
Thank you,
Regards,
--Aleks
Quoted text · Christian Grün · 23 lines
-----Original Message----- From: Christian Grün [mailto:christian.gruen at gmail.com] Sent: Sunday, January 24, 2010 6:56 AM To: Alexander Sheynkman Cc: basex-talk at mailman.uni-konstanz.de Subject: Re: [basex-talk] BaseX db questions Dear Aleks, thanks for your questions: > 1) What is the concept of the Context ? Do I need to create a new context for every database operation or same context should be used within open database ? One Context instance should be used for all database operations. It contains a reference to the currently opened database. I would only recommend to create multiple Context instances if you need to open several databases at the same time. > 2) Is it true that I have to open and close database every time I run query or I can keep database open. The database will be kept open until you send a Close command, or open another database. The following code snippet might give you some more hints: http://www.inf.uni-konstanz.de/dbis/basex/code.php?code=DatabaseExample > 3) How does commit occur, with basex6 it seems like data being committed to disk with some delay? All modified data is flushed to disk as soon as an XQuery Update expression has been completely processed, so no delays should occur. Hope this helps, Christian
CG
Hi Aleks,
Quoted text · currently BaseX does not provide any good way to perform XSD validation and all · 1 line
> 1) XSD validation, currently BaseX does not provide any good way to perform XSD validation and all writes are always flushed to the disk. This will potentially can cause a problem, because incorrect changes can break consistency of the database. In addition to this it's a bit heavy to keep complete xml in the memory just for the purpose of validation due to the fact that database can grow to a large size.
It's true we don't offer schema validation, so documents won't be automatically revalidated after updates. I.e., the application must guarantee that updates won't violate the schema.
Quoted text · Alexander Sheynkman · 1 line
> 2) Currently all writes to database only reflect in internal BaseX file system, the changes does not apply to XML file right away, unless you specifically copy it over, is there are a good way where you can force BaseX also update xml file with the changes.
Currently, this has to be done by explicitly calling the
fn:put() function or using the export command. But let me think.. We could add a new database flag to automatically write changes back to the original files. This flag could be switched on by default. Just tell me if this would be helpful in your context (note, however, that we'll have to check this solution for potential side effects).
Hope this helps,
ChristianAS
Hi Christian,
Follow our conversation regarding Basex. Can you please advice me on how to best perform the following functionality with BaseX.
1) XSD validation, currently BaseX does not provide any good way to perform XSD validation and all writes are always flushed to the disk. This will potentially can cause a problem, because incorrect changes can break consistency of the database. In addition to this it's a bit heavy to keep complete xml in the memory just for the purpose of validation due to the fact that database can grow to a large size.
2) Currently all writes to database only reflect in internal BaseX file system, the changes does not apply to XML file right away, unless you specifically copy it over, is there are a good way where you can force BaseX also update xml file with the changes.
Thank you for your time!!
--Aleks
Quoted text · Christian Grün · 21 lines
-----Original Message----- From: Christian Grün [mailto:christian.gruen at gmail.com] Sent: Sunday, January 24, 2010 8:21 AM To: Alexander Sheynkman Cc: basex-talk at mailman.uni-konstanz.de Subject: Re: [basex-talk] BaseX db questions Dear Aleks, > Thank you very much for your time, Just to clarify on point #2 regarding close. What I understood from you if I have an application that is managing multiple embedded database ( let say 3 ) at the same time and needs to constantly run query / updates on these db's , then I just need to Open it once on startup and the close it whenever I'm done with database. That's true. Opening database is a very fast operation, however, so a second, easier alternative is to implicitly open and query the database within XQuery expressions, as shown in the following two sample queries: -- doc('dbname')/path/to/elements -- for $x in basex:db('dbname')/to/be/deleted return delete node $x > How about OPTIMIZE , I see you suggested to run it after all kinds of database updates, is it true ? The optimize command refreshes meta data and index structures of a database, which might speed up the evaluation of new XQuery requests. We decided to make database optimization an optional feature to allow for a much faster evaluation of update queries. Feel free to ask for more, Christian
AS
Hi Christian,
Adding a flag would be great , this will helps out so that we can sync database changes into xml file in one call.
I have one more question:
We are getting the following exception while performing query against database : Main.notexpected(Object...) line: 205
TableDiskAccess.cursor(int) line: 326
TableDiskAccess.read1(int, int) line: 95
DiskData(Data).kind(int) line: 277
DBNode$3.next() line: 234
SimpleIterStep$1.next() line: 51
AxisPath.iter(int, NodIter, QueryContext) line: 425 AxisPath.iter(int, NodIter, QueryContext) line: 428 AxisPath.iter(int, NodIter, QueryContext) line: 428 AxisPath.iter(int, NodIter, QueryContext) line: 428 AxisPath.iter(QueryContext) line: 398
QueryContext.iter(Expr) line: 338
QueryContext.iter() line: 296
QueryContext.eval() line: 255
QueryProcessor.query() line: 92
BXQueryService.query(Nodes, String) line: 121 BXQueryService.query(String) line: 69
BXQueryService.query(String) line: 1
Can you shade some light why would this happen ? this problem happens randomly.
We currently create a single Context object for every open database and keep passing this context for every call, do you think this will cause the problem ?
Thank you!!!
--Aleks
Quoted text · Christian Grün · 20 lines
-----Original Message----- From: Christian Grün [mailto:christian.gruen at gmail.com] Sent: Wednesday, January 27, 2010 8:13 AM To: Alexander Sheynkman Cc: basex-talk at mailman.uni-konstanz.de Subject: Re: [basex-talk] BaseX db questions Hi Aleks, > 1) XSD validation, currently BaseX does not provide any good way to perform XSD validation and all writes are always flushed to the disk. This will potentially can cause a problem, because incorrect changes can break consistency of the database. In addition to this it's a bit heavy to keep complete xml in the memory just for the purpose of validation due to the fact that database can grow to a large size. It's true we don't offer schema validation, so documents won't be automatically revalidated after updates. I.e., the application must guarantee that updates won't violate the schema. > 2) Currently all writes to database only reflect in internal BaseX file system, the changes does not apply to XML file right away, unless you specifically copy it over, is there are a good way where you can force BaseX also update xml file with the changes. Currently, this has to be done by explicitly calling the fn:put() function or using the export command. But let me think.. We could add a new database flag to automatically write changes back to the original files. This flag could be switched on by default. Just tell me if this would be helpful in your context (note, however, that we'll have to check this solution for potential side effects). Hope this helps, Christian
CG
Christian Grün29 Jan 2010 00:27 #
Dear Aleks,
Quoted text · Alexander Sheynkman · 1 line
> Adding a flag would be great , this will helps out so that we can sync database changes into xml file in one call.
I've added the flag which allows updates files to be written back to disk. Note that, currently, all files of a modified database will be written back to disk after an update operation has been completed. Note that this is experimental; you're invited to test the latest version of the sources, which you find in our Subversion repository.
The flag can be activated via the SET command in BaseX:
Command Line:
set writeback on; create db input.xml; xquery delete node //text()
BaseX API:
Context ctx = new Context();
new Set("WRITEBACK", true).execute(ctx);
new CreateDB("input.xml").execute(ctx);
new XQuery("xquery delete node //text()").execute(ctx);Quoted text · Alexander Sheynkman · 5 lines
> We are getting the following exception while performing query against database : > Main.notexpected(Object...) line: 205 > TableDiskAccess.cursor(int) line: 326 > [...] > Can you shade some light why would this happen ? this problem happens randomly.
This exception might (but need not) indicate an inconsistency in the data storage. It would be great if you could try to create a reproducible example – although I guess this might not be easy if the issue occurs only randomly.
Quoted text · Alexander Sheynkman · 1 line
> We currently create a single Context object for every open database and keep passing this context for every call, do you think this will cause the problem ?
That should be fine. Do you close the database/context – data.close() / context.close() – after all updates have been processed?
All the best,
Christian
Written for BaseX 6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery