Empty Initial Document Inconsistencies

Thursday, 29 December 2011 · 23 messages · last reply 7 Jul 2012
DG
Dave Glick29 Dec 2011 17:36 #
Hello again, I've noticed some inconsistencies with regard to the initial document in an empty database. I understand that it is often treated as an indication that the database is "empty". I.e., if the database has only one node and it's a document at the first pre value, then Data.empty() returns true. The problem arises because the database isn't actually empty - it contains one empty document - and some commands/methods view it as an empty database while others treat it as a database with one empty document. Several examples: - The method Data.doc(name) will return -1 (indicating the document doesn't exist) while Data.docs() will return an array of size 1 with the first value a 0 (indicating there is one document at pre value 0). This is actually my own problem because the API I'm writing relies on these two methods being internally consistent - I can't have one telling me there is one document called "XYZ" but then have the other refuse to give me a pre value for the "XYZ" document I was just told exists. - The GUI shows a single doc node in the tree view but issuing the command "list [database name]" shows the database as having no resources. - I can evaluate the query "insert node 'text' into doc('[initial document name]')" and it executes. I can see the new text node under the initial document in the GUI tree view. Now when I issue the command "list [database name]" I see 1 resource. Having the list command report different numbers of resources before and after a XQuery insertion seems odd. - If I issue the command "ADD TO newdoc <xyz/>", I can see the new doc("newdoc") node with the child <xyz> element in the GUI but the previously empty document is now gone. This also seems odd - I'm not sure how I feel about an ADD command removing data from the database (even if it was just intended to be a placeholder). In the end, I think BaseX should support the notion of a database with one empty document as being valid - there are probably cases where one might want to start their session in that state. I'm not sure what the solution is or should be (or if it's even something that is worth or needs solving). My own preference would be for the internal indication of an empty database to use a different node kind dedicated for that purpose so there is no confusion whether the single node at pre 0 is an empty initial document or an indication of an empty database. At the very least, I think the inconsistencies above should be corrected - if a single empty document node is intended to signify an empty database and is not actually intended to be part of the database, then you should not see it in the GUI, not be allowed to insert content to it, etc. Dave
CG
Christian Grün7 Jan 2012 14:23 #
Hi Dave, sorry for the long delay, caused by the occidental winter break.. You've analyzed very well what are the inconsistencies between an "empty database" and a database with one empty document – and I know it's no real excuse that this issue hasn't caused any problems in our own use cases. The basic reason why we have equalized an empty document node with an "empty database" some time ago was that we wanted to avoid too many new code snippets that dealt with the special case of having stored no data at all. In a nutshell, I completely agree that the solution you are proposing would be more consistent, but I suspect that we'd need quite some time to rework all the relevant lines of code. If you are interested in tackling this challenge, I'll be more than glad! Best, Christian
Quoted text · Dave Glick · 20 lines
On Thu, Dec 29, 2011 at 4:36 PM, Dave Glick <dglick at dracorp.com> wrote: > Hello again, > > I've noticed some inconsistencies with regard to the initial document in an empty database. I understand that it is often treated as an indication that the database is "empty". I.e., if the database has only one node and it's a document at the first pre value, then Data.empty() returns true. The problem arises because the database isn't actually empty - it contains one empty document - and some commands/methods view it as an empty database while others treat it as a database with one empty document. Several examples: > > - The method Data.doc(name) will return -1 (indicating the document doesn't exist) while Data.docs() will return an array of size 1 with the first value a 0 (indicating there is one document at pre value 0). This is actually my own problem because the API I'm writing relies on these two methods being internally consistent - I can't have one telling me there is one document called "XYZ" but then have the other refuse to give me a pre value for the "XYZ" document I was just told exists. > - The GUI shows a single doc node in the tree view but issuing the command "list [database name]" shows the database as having no resources. > - I can evaluate the query "insert node 'text' into doc('[initial document name]')" and it executes. I can see the new text node under the initial document in the GUI tree view. Now when I issue the command "list [database name]" I see 1 resource. Having the list command report different numbers of resources before and after a XQuery insertion seems odd. > - If I issue the command "ADD TO newdoc <xyz/>", I can see the new doc("newdoc") node with the child <xyz> element in the GUI but the previously empty document is now gone. This also seems odd - I'm not sure how I feel about an ADD command removing data from the database (even if it was just intended to be a placeholder). > > In the end, I think BaseX should support the notion of a database with one empty document as being valid - there are probably cases where one might want to start their session in that state. I'm not sure what the solution is or should be (or if it's even something that is worth or needs solving). My own preference would be for the internal indication of an empty database to use a different node kind dedicated for that purpose so there is no confusion whether the single node at pre 0 is an empty initial document or an indication of an empty database. > > At the very least, I think the inconsistencies above should be corrected - if a single empty document node is intended to signify an empty database and is not actually intended to be part of the database, then you should not see it in the GUI, not be allowed to insert content to it, etc. > > Dave > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
DG
Dave Glick9 Jan 2012 19:00 #
Christian, No problem - I had sent this right before leaving on vacation myself (I didn't want to totally forget everything while I was gone). Your explanation is pretty much what I figured. The current implementation works fine and makes sense in nearly every case - it's just the rare edge cases that the behavior seems odd or contradictory. I certainly don't mind taking a look at the problem and submitting a patch if I get it resolved - just wanted to ensure there weren't already plans in the pipeline to address this before I started hacking. Also, it might take me a little while - the "real" job is consuming a lot of my time right now. However, look for something from me on this in about a month... Thanks, Dave
Quoted text · Christian Grün · 32 lines
-----Original Message----- From: Christian Grün [mailto:christian.gruen at gmail.com] Sent: Saturday, January 07, 2012 8:23 AM To: Dave Glick Cc: BaseX Subject: Re: [basex-talk] Empty Initial Document Inconsistencies Hi Dave, sorry for the long delay, caused by the occidental winter break.. You've analyzed very well what are the inconsistencies between an "empty database" and a database with one empty document - and I know it's no real excuse that this issue hasn't caused any problems in our own use cases. The basic reason why we have equalized an empty document node with an "empty database" some time ago was that we wanted to avoid too many new code snippets that dealt with the special case of having stored no data at all. In a nutshell, I completely agree that the solution you are proposing would be more consistent, but I suspect that we'd need quite some time to rework all the relevant lines of code. If you are interested in tackling this challenge, I'll be more than glad! Best, Christian On Thu, Dec 29, 2011 at 4:36 PM, Dave Glick <dglick at dracorp.com> wrote: > Hello again, > > I've noticed some inconsistencies with regard to the initial document in an empty database. I understand that it is often treated as an indication that the database is "empty". I.e., if the database has only one node and it's a document at the first pre value, then Data.empty() returns true. The problem arises because the database isn't actually empty - it contains one empty document - and some commands/methods view it as an empty database while others treat it as a database with one empty document. Several examples: > > - The method Data.doc(name) will return -1 (indicating the document doesn't exist) while Data.docs() will return an array of size 1 with the first value a 0 (indicating there is one document at pre value 0). This is actually my own problem because the API I'm writing relies on these two methods being internally consistent - I can't have one telling me there is one document called "XYZ" but then have the other refuse to give me a pre value for the "XYZ" document I was just told exists. > - The GUI shows a single doc node in the tree view but issuing the command "list [database name]" shows the database as having no resources. > - I can evaluate the query "insert node 'text' into doc('[initial document name]')" and it executes. I can see the new text node under the initial document in the GUI tree view. Now when I issue the command "list [database name]" I see 1 resource. Having the list command report different numbers of resources before and after a XQuery insertion seems odd. > - If I issue the command "ADD TO newdoc <xyz/>", I can see the new doc("newdoc") node with the child <xyz> element in the GUI but the previously empty document is now gone. This also seems odd - I'm not sure how I feel about an ADD command removing data from the database (even if it was just intended to be a placeholder). > > In the end, I think BaseX should support the notion of a database with one empty document as being valid - there are probably cases where one might want to start their session in that state. I'm not sure what the solution is or should be (or if it's even something that is worth or needs solving). My own preference would be for the internal indication of an empty database to use a different node kind dedicated for that purpose so there is no confusion whether the single node at pre 0 is an empty initial document or an indication of an empty database. > > At the very least, I think the inconsistencies above should be corrected - if a single empty document node is intended to signify an empty database and is not actually intended to be part of the database, then you should not see it in the GUI, not be allowed to insert content to it, etc. > > Dave > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
CG
Christian Grünin reply to Dave Glick, 29 Dec 2011, 17:363 Jul 2012 02:13 #
Hi Dave, here's finally some public feedback for your outstanding feature request: I've rewritten our internal database structures, such that an empty database will now really be empty (i.e., contain no dummy document node anymore). As a result, some convenience methods have disappeared from the Data class (Data.isEmpty(), Data.single()), as they were only offered to hide the former inconsistency. The changes have now been merged into the BaseX master branch. While I believe that we have tested the changes quite well, I am always glad for any feedback. Christian
Quoted text · Dave Glick · 20 lines
On Thu, Dec 29, 2011 at 4:36 PM, Dave Glick <dglick at dracorp.com> wrote: > Hello again, > > I've noticed some inconsistencies with regard to the initial document in an empty database. I understand that it is often treated as an indication that the database is "empty". I.e., if the database has only one node and it's a document at the first pre value, then Data.empty() returns true. The problem arises because the database isn't actually empty - it contains one empty document - and some commands/methods view it as an empty database while others treat it as a database with one empty document. Several examples: > > - The method Data.doc(name) will return -1 (indicating the document doesn't exist) while Data.docs() will return an array of size 1 with the first value a 0 (indicating there is one document at pre value 0). This is actually my own problem because the API I'm writing relies on these two methods being internally consistent - I can't have one telling me there is one document called "XYZ" but then have the other refuse to give me a pre value for the "XYZ" document I was just told exists. > - The GUI shows a single doc node in the tree view but issuing the command "list [database name]" shows the database as having no resources. > - I can evaluate the query "insert node 'text' into doc('[initial document name]')" and it executes. I can see the new text node under the initial document in the GUI tree view. Now when I issue the command "list [database name]" I see 1 resource. Having the list command report different numbers of resources before and after a XQuery insertion seems odd. > - If I issue the command "ADD TO newdoc <xyz/>", I can see the new doc("newdoc") node with the child <xyz> element in the GUI but the previously empty document is now gone. This also seems odd - I'm not sure how I feel about an ADD command removing data from the database (even if it was just intended to be a placeholder). > > In the end, I think BaseX should support the notion of a database with one empty document as being valid - there are probably cases where one might want to start their session in that state. I'm not sure what the solution is or should be (or if it's even something that is worth or needs solving). My own preference would be for the internal indication of an empty database to use a different node kind dedicated for that purpose so there is no confusion whether the single node at pre 0 is an empty initial document or an indication of an empty database. > > At the very least, I think the inconsistencies above should be corrected - if a single empty document node is intended to signify an empty database and is not actually intended to be part of the database, then you should not see it in the GUI, not be allowed to insert content to it, etc. > > Dave > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
FE
Fabrice Etanchaud3 Jul 2012 09:48 #
Dear all at BaseX, Dear Christian, Does that mean that current databases will not work anymore with the next releases ? Best regards, Fabrice
Quoted text · Christian Grün · 31 lines
-----Original Message----- From: Christian Grün [mailto:christian.gruen at gmail.com] Sent: mardi 3 juillet 2012 02:13 To: Dave Glick Cc: BaseX Subject: Re: [basex-talk] Empty Initial Document Inconsistencies Hi Dave, here's finally some public feedback for your outstanding feature request: I've rewritten our internal database structures, such that an empty database will now really be empty (i.e., contain no dummy document node anymore). As a result, some convenience methods have disappeared from the Data class (Data.isEmpty(), Data.single()), as they were only offered to hide the former inconsistency. The changes have now been merged into the BaseX master branch. While I believe that we have tested the changes quite well, I am always glad for any feedback. Christian On Thu, Dec 29, 2011 at 4:36 PM, Dave Glick <dglick at dracorp.com> wrote: > Hello again, > > I've noticed some inconsistencies with regard to the initial document in an empty database. I understand that it is often treated as an indication that the database is "empty". I.e., if the database has only one node and it's a document at the first pre value, then Data.empty() returns true. The problem arises because the database isn't actually empty - it contains one empty document - and some commands/methods view it as an empty database while others treat it as a database with one empty document. Several examples: > > - The method Data.doc(name) will return -1 (indicating the document doesn't exist) while Data.docs() will return an array of size 1 with the first value a 0 (indicating there is one document at pre value 0). This is actually my own problem because the API I'm writing relies on these two methods being internally consistent - I can't have one telling me there is one document called "XYZ" but then have the other refuse to give me a pre value for the "XYZ" document I was just told exists. > - The GUI shows a single doc node in the tree view but issuing the command "list [database name]" shows the database as having no resources. > - I can evaluate the query "insert node 'text' into doc('[initial document name]')" and it executes. I can see the new text node under the initial document in the GUI tree view. Now when I issue the command "list [database name]" I see 1 resource. Having the list command report different numbers of resources before and after a XQuery insertion seems odd. > - If I issue the command "ADD TO newdoc <xyz/>", I can see the new doc("newdoc") node with the child <xyz> element in the GUI but the previously empty document is now gone. This also seems odd - I'm not sure how I feel about an ADD command removing data from the database (even if it was just intended to be a placeholder). > > In the end, I think BaseX should support the notion of a database with one empty document as being valid - there are probably cases where one might want to start their session in that state. I'm not sure what the solution is or should be (or if it's even something that is worth or needs solving). My own preference would be for the internal indication of an empty database to use a different node kind dedicated for that purpose so there is no confusion whether the single node at pre 0 is an empty initial document or an indication of an empty database. > > At the very least, I think the inconsistencies above should be corrected - if a single empty document node is intended to signify an empty database and is not actually intended to be part of the database, then you should not see it in the GUI, not be allowed to insert content to it, etc. > > Dave > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
CG
Christian Grün3 Jul 2012 13:40 #
Hi Fabrice,
Quoted text · Fabrice Etanchaud · 1 line
> Does that mean that current databases will not work anymore with the next releases ?
no worries, the database format will only (potentially) in an upcoming major release (BaseX 8 or 9). The only change is that formerly empty databases will now contain a single, empty document. Best, Christian
Quoted text · Fabrice Etanchaud · 46 lines
> > Best regards, > Fabrice > > -----Original Message----- > From: Christian Grün [mailto:christian.gruen at gmail.com] > Sent: mardi 3 juillet 2012 02:13 > To: Dave Glick > Cc: BaseX > Subject: Re: [basex-talk] Empty Initial Document Inconsistencies > > Hi Dave, > > here's finally some public feedback for your outstanding feature > request: I've rewritten our internal database structures, such that an empty database will now really be empty (i.e., contain no dummy document node anymore). As a result, some convenience methods have disappeared from the Data class (Data.isEmpty(), Data.single()), as they were only offered to hide the former inconsistency. > > The changes have now been merged into the BaseX master branch. While I believe that we have tested the changes quite well, I am always glad for any feedback. > > Christian > ___________________________ > > On Thu, Dec 29, 2011 at 4:36 PM, Dave Glick <dglick at dracorp.com> wrote: >> Hello again, >> >> I've noticed some inconsistencies with regard to the initial document in an empty database. I understand that it is often treated as an indication that the database is "empty". I.e., if the database has only one node and it's a document at the first pre value, then Data.empty() returns true. The problem arises because the database isn't actually empty - it contains one empty document - and some commands/methods view it as an empty database while others treat it as a database with one empty document. Several examples: >> >> - The method Data.doc(name) will return -1 (indicating the document doesn't exist) while Data.docs() will return an array of size 1 with the first value a 0 (indicating there is one document at pre value 0). This is actually my own problem because the API I'm writing relies on these two methods being internally consistent - I can't have one telling me there is one document called "XYZ" but then have the other refuse to give me a pre value for the "XYZ" document I was just told exists. >> - The GUI shows a single doc node in the tree view but issuing the command "list [database name]" shows the database as having no resources. >> - I can evaluate the query "insert node 'text' into doc('[initial document name]')" and it executes. I can see the new text node under the initial document in the GUI tree view. Now when I issue the command "list [database name]" I see 1 resource. Having the list command report different numbers of resources before and after a XQuery insertion seems odd. >> - If I issue the command "ADD TO newdoc <xyz/>", I can see the new doc("newdoc") node with the child <xyz> element in the GUI but the previously empty document is now gone. This also seems odd - I'm not sure how I feel about an ADD command removing data from the database (even if it was just intended to be a placeholder). >> >> In the end, I think BaseX should support the notion of a database with one empty document as being valid - there are probably cases where one might want to start their session in that state. I'm not sure what the solution is or should be (or if it's even something that is worth or needs solving). My own preference would be for the internal indication of an empty database to use a different node kind dedicated for that purpose so there is no confusion whether the single node at pre 0 is an empty initial document or an indication of an empty database. >> >> At the very least, I think the inconsistencies above should be corrected - if a single empty document node is intended to signify an empty database and is not actually intended to be part of the database, then you should not see it in the GUI, not be allowed to insert content to it, etc. >> >> Dave >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
DG
Dave Glick3 Jul 2012 14:13 #
Christian, Thanks again for this! I still haven't had a chance to do any rigorous testing, but in my initial use everything looks great (and all my unit tests still pass). I think more importantly, this resolves an undocumented inconsistency for users of the low-level API - with the guarantee that the database will behave the same way regardless of state, a lot of conditional code can now be removed. Proof that good things come to those who wait :). Dave
Quoted text · Christian Grün · 37 lines
-----Original message----- From: "Christian Grün" <christian.gruen at gmail.com> To: Dave Glick <dglick at dracorp.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Sent: Tue, Jul 3, 2012 00:13:58 GMT+00:00 Subject: Re: [basex-talk] Empty Initial Document Inconsistencies Hi Dave, here's finally some public feedback for your outstanding feature request: I've rewritten our internal database structures, such that an empty database will now really be empty (i.e., contain no dummy document node anymore). As a result, some convenience methods have disappeared from the Data class (Data.isEmpty(), Data.single()), as they were only offered to hide the former inconsistency. The changes have now been merged into the BaseX master branch. While I believe that we have tested the changes quite well, I am always glad for any feedback. Christian On Thu, Dec 29, 2011 at 4:36 PM, Dave Glick <dglick at dracorp.com> wrote: > Hello again, > > I've noticed some inconsistencies with regard to the initial document in an empty database. I understand that it is often treated as an indication that the database is "empty". I.e., if the database has only one node and it's a document at the first pre value, then Data.empty() returns true. The problem arises because the database isn't actually empty - it contains one empty document - and some commands/methods view it as an empty database while others treat it as a database with one empty document. Several examples: > > - The method Data.doc(name) will return -1 (indicating the document doesn't exist) while Data.docs() will return an array of size 1 with the first value a 0 (indicating there is one document at pre value 0). This is actually my own problem because the API I'm writing relies on these two methods being internally consistent - I can't have one telling me there is one document called "XYZ" but then have the other refuse to give me a pre value for the "XYZ" document I was just told exists. > - The GUI shows a single doc node in the tree view but issuing the command "list [database name]" shows the database as having no resources. > - I can evaluate the query "insert node 'text' into doc('[initial document name]')" and it executes. I can see the new text node under the initial document in the GUI tree view. Now when I issue the command "list [database name]" I see 1 resource. Having the list command report different numbers of resources before and after a XQuery insertion seems odd. > - If I issue the command "ADD TO newdoc <xyz/>", I can see the new doc("newdoc") node with the child <xyz> element in the GUI but the previously empty document is now gone. This also seems odd - I'm not sure how I feel about an ADD command removing data from the database (even if it was just intended to be a placeholder). > > In the end, I think BaseX should support the notion of a database with one empty document as being valid - there are probably cases where one might want to start their session in that state. I'm not sure what the solution is or should be (or if it's even something that is worth or needs solving). My own preference would be for the internal indication of an empty database to use a different node kind dedicated for that purpose so there is no confusion whether the single node at pre 0 is an empty initial document or an indication of an empty database. > > At the very least, I think the inconsistencies above should be corrected - if a single empty document node is intended to signify an empty database and is not actually intended to be part of the database, then you should not see it in the GUI, not be allowed to insert content to it, etc. > > Dave > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
HR
Hans-Juergen Rennau3 Jul 2012 15:30 #
Hello, this is a general question as to whether in a given scenario BaseX might be an appropriate instrument. Per second approximately 1000 XML log messages must be stored and thus made available for querying. The messages are expected to be <= 1MB. The log messages may be sent by any number of clients simultaneously. The clients are probably not able to specify unique document URIs, as they are working independently of each other. So the database would have to create unique URIs - perhaps concatenating a semantic part supplied by the client and a generated unique identifier - as part of the storage processing. Would this be possible already now? If not, perhaps in the near future? Thank you, kind regards, Hans-Juergen
AW
Andreas Weiler3 Jul 2012 15:51 #
Hello Hans-Juergen, here are some details about my use case, which is similar to yours. I'm using BaseX to insert the live public Twitter Stream into databases (see Wiki Entry [1]). One Twitter message is around 4 kb of size and i'm able to insert about 2000 of them per second using single XQuery Update inserts. So that would probably be working out for you, too. If you use bulk inserts, like caching the items in a item list and running one XQuery Update for all of them, the amount of inserts would also increase.
Quoted text · Hans-Juergen Rennau · 1 line
> thus made available for querying
this could be a bigger problem, cause as long as you are writing items into the database (which will never stop in your use case), the readers are blocked. And if one of your readers will be running, the writers are blocked. Hope this helps, Andreas [1] http://docs.basex.org/wiki/Twitter
Quoted text · Hans-Juergen Rennau · 21 lines
Am 03.07.2012 um 15:30 schrieb Hans-Juergen Rennau: > Hello, > > this is a general question as to whether in a given scenario BaseX might be an appropriate instrument. > > Per second approximately 1000 XML log messages must be stored and thus made available for querying. The messages are expected to be <= 1MB. > > The log messages may be sent by any number of clients simultaneously. The clients are probably not able to specify unique document URIs, as they are working independently of each other. So the database would have to create unique URIs - perhaps concatenating a semantic part supplied by the client and a generated unique identifier - as part of the storage processing. > > Would this be possible already now? If not, perhaps in the near future? > > Thank you, > kind regards, > > Hans-Juergen > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
HR
Hans-Juergen Rennau3 Jul 2012 23:58 #
Hello Andreas, thank you very much for these informations! Indeed, the use-cases are similar. I try to understand how exactly you stored the messages. The Wiki says: "the initial database just contained a root node <tweets/>". So my understanding is that the messages are inserted as child elements into this root element - and the end result is one document with one root element and millions of child elements representing the invidual messages, yes? Therefore you do not have to come up with URIs, as there is only one single document. A monster document, but I conclude from your approach that this is no problem, and not worse (or even better) than having a million individual, small documents. Is it correct - would you recommend to store the messages in one single document? If the loading process cannot concur with queries - would there be any way how one could periodically "shift" packages of messages into a "read only" database? Or perhaps better the other way around, let the server periodically interrupt its loading activity, close the database, rename it, open and initialize a new base and then continue to load? Or is there presently simply no solution available? Kind regards, Hans-Juergen ________________________________ Von: Andreas Weiler <andreas.weiler at uni-konstanz.de> An: Hans-Juergen Rennau <hrennau at yahoo.de> CC: "basex-talk at mailman.uni-konstanz.de" <basex-talk at mailman.uni-konstanz.de> Gesendet: 15:51 Dienstag, 3.Juli 2012 Betreff: Re: [basex-talk] BaseX as a log msg store? Hello Hans-Juergen, here are some details about my use case, which is similar to yours. I'm using BaseX to insert the live public Twitter Stream into databases (see Wiki Entry [1]). One Twitter message is around 4 kb of size and i'm able to insert about 2000 of them per second using single XQuery Update inserts. So that would probably be working out for you, too. If you use bulk inserts, like caching the items in a item list and running one XQuery Update for all of them, the amount of inserts would also increase. thus made available for querying this could be a bigger problem, cause as long as you are writing items into the database (which will never stop in your use case), the readers are blocked. And if one of your readers will be running, the writers are blocked. Hope this helps, Andreas [1] http://docs.basex.org/wiki/Twitter
PH
Pascal Heusin reply to Dave Glick, 9 Jan 2012, 19:004 Jul 2012 01:44 #
All: Anyone has a little xquery / basex tip in getting rid of the extra white space that shows up at the beginning of each line after the first in the following xquery for $i in 1 to 10 return concat("x","&#10;") which in the UI returns x x x x x Just need a text output going to csv but this extra space is causing me headaches. tks *P
CG
Christian Grün4 Jul 2012 04:24 #
Hi Pascal, you can either create a new, single string via the fn:string-join() function…
  string-join(for $i in 1 to 10 return "x","&#10;")
…or use the "output:separator" option of BaseX: declare option output:separator '\n'; (1 to 10) ! "x" The second option, or a variation of it, may eventually find entrance into the official W3 XQuery Serialization Specification (see [1] for future details). Hope this helps, Christian [1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=16311
Quoted text · Pascal Heus · 22 lines
> Anyone has a little xquery / basex tip in getting rid of the extra white > space that shows up at the beginning of each line after the first in the > following xquery > for $i in 1 to 10 return concat("x","&#10;") > which in the UI returns > x > x > x > x > x > Just need a text output going to csv but this extra space is causing me > headaches. > tks > *P > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
AH
Alexander Holupirek4 Jul 2012 08:17 #
Hi Pascal, if you use BaseX inside a shell script or as a command line xquery processing tool, you may also use the -L flag: $ basex -h 2>&1 | grep L Usage: basex [-bcdiLosuvVwxz] [query] -L Append newlines to query results $ basex -L 'for $i in 1 to 10 return $i' 1 2 3 ... All the best, Alex
Quoted text · Christian Grün · 48 lines
On 04.07.2012, at 04:24, Christian Grün wrote: > Hi Pascal, > > you can either create a new, single string via the fn:string-join() function… > > string-join(for $i in 1 to 10 return "x","&#10;") > > …or use the "output:separator" option of BaseX: > > declare option output:separator '\n'; > (1 to 10) ! "x" > > The second option, or a variation of it, may eventually find entrance > into the official W3 XQuery Serialization Specification (see [1] for > future details). > > Hope this helps, > Christian > > [1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=16311 > > >> Anyone has a little xquery / basex tip in getting rid of the extra white >> space that shows up at the beginning of each line after the first in the >> following xquery >> for $i in 1 to 10 return concat("x","&#10;") >> which in the UI returns >> x >> x >> x >> x >> x >> Just need a text output going to csv but this extra space is causing me >> headaches. >> tks >> *P >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
-- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- -- Alexander Holupirek |-- http://www.informatik.uni-konstanz.de/~holupire |-- Database & Information Systems Group, U Konstanz `-- Room E 221, 0049 7531 88 2188 (phone) 3577 (fax)
AW
Andreas Weilerin reply to Hans-Juergen Rennau, 3 Jul 2012, 23:584 Jul 2012 09:45 #
Hello Hans-Juergen,
Quoted text · Hans-Juergen Rennau · 1 line
> So my understanding is that the messages are inserted as child elements into this root element - and the end result is one document with one root element and millions of child elements representing the invidual messages, yes?
Yes that is correct, i have one root element at the beginning and insert the incoming items as child nodes of the root.
Quoted text · Hans-Juergen Rennau · 1 line
> Therefore you do not have to come up with URIs, as there is only one single document. A monster document, but I conclude from your approach that this is no problem, and not worse (or even better) than having a million individual, small documents. Is it correct - would you recommend to store the messages in one single document?
In my use case, tweets have unique id attributes, so i don't need any URIs to identify them. Probably, it is a good idea if you describe your further querying process so it is easier to understand what you want to do.
Quoted text · Hans-Juergen Rennau · 1 line
> If the loading process cannot concur with queries - would there be any way how one could periodically "shift" packages of messages into a "read only" database? Or perhaps better the other way around, let the server periodically interrupt its loading activity, close the database, rename it, open and initialize a new base and then continue to load? Or is there presently simply no solution available?
Thats exactly what i do after each hour. I rename the current db with the current date_hour and create a new database for the next incoming items. Shifting is not really an alternative, cause it will probably take too long to insert the items into a second database and delete them from the "main" database. Kind regards, Andreas
Quoted text · Hans-Juergen Rennau · 38 lines
Am 03.07.2012 um 23:58 schrieb Hans-Juergen Rennau: > Hello Andreas, > > thank you very much for these informations! Indeed, the use-cases are similar. > > I try to understand how exactly you stored the messages. The Wiki says: "the initial database just contained a root node <tweets/>". So my understanding is that the messages are inserted as child elements into this root element - and the end result is one document with one root element and millions of child elements representing the invidual messages, yes? Therefore you do not have to come up with URIs, as there is only one single document. A monster document, but I conclude from your approach that this is no problem, and not worse (or even better) than having a million individual, small documents. Is it correct - would you recommend to store the messages in one single document? > > If the loading process cannot concur with queries - would there be any way how one could periodically "shift" packages of messages into a "read only" database? Or perhaps better the other way around, let the server periodically interrupt its loading activity, close the database, rename it, open and initialize a new base and then continue to load? Or is there presently simply no solution available? > > Kind regards, > Hans-Juergen > > Von: Andreas Weiler <andreas.weiler at uni-konstanz.de> > An: Hans-Juergen Rennau <hrennau at yahoo.de> > CC: "basex-talk at mailman.uni-konstanz.de" <basex-talk at mailman.uni-konstanz.de> > Gesendet: 15:51 Dienstag, 3.Juli 2012 > Betreff: Re: [basex-talk] BaseX as a log msg store? > > Hello Hans-Juergen, > > here are some details about my use case, which is similar to yours. > I'm using BaseX to insert the live public Twitter Stream into databases (see Wiki Entry [1]). > > One Twitter message is around 4 kb of size and i'm able to insert about 2000 of them per second > using single XQuery Update inserts. So that would probably be working out for you, too. > If you use bulk inserts, like caching the items in a item list and running one XQuery Update for all of them, the amount of inserts would also increase. > >> thus made available for querying > > this could be a bigger problem, cause as long as you are writing items into the database (which will never stop in your use case), the readers are blocked. > And if one of your readers will be running, the writers are blocked. > > Hope this helps, > Andreas > > [1] http://docs.basex.org/wiki/Twitter > >
HR
Hans-Juergen Rennau4 Jul 2012 19:54 #
Hello Andreas, was this database renaming+creation triggered by the client or autonomously done by the server? The latter alternative would require some scheduling feature, so that the database server could be configured to perform such actions according to a schedule. I think this would be a very desirable feature, especially as long as high frequency storage and querying cannot be done well concurrently. Anyway, do you see a way to implement this periodic renaming+creation behaviour in a multi-client environment? Concerning the querying tasks to be expected I know very little myself - except for the need to group messages by transaction IDs and then somehow evaluate the execution of transactions. But at the moment I suppose that your "monster document" approach might be appropriate in our case, too. Kind regards, Hans-Juergen ________________________________ Von: Andreas Weiler <andreas.weiler at uni-konstanz.de> An: Hans-Juergen Rennau <hrennau at yahoo.de> CC: Base X <basex-talk at mailman.uni-konstanz.de> Gesendet: 9:45 Mittwoch, 4.Juli 2012 Betreff: Re: [basex-talk] BaseX as a log msg store? Hello Hans-Juergen, So my understanding is that the messages are inserted as child elements into this root element - and the end result is one document with one root element and millions of child elements representing the invidual messages, yes? Yes that is correct, i have one root element at the beginning and insert the incoming items as child nodes of the root. Therefore you do not have to come up with URIs, as there is only one single document. A monster document, but I conclude from your approach that this is no problem, and not worse (or even better) than having a million individual, small documents. Is it correct - would you recommend to store the messages in one single document? In my use case, tweets have unique id attributes, so i don't need any URIs to identify them. Probably, it is a good idea if you describe your further querying process so it is easier to understand what you want to do. If the loading process cannot concur with queries - would there be any way how one could periodically "shift" packages of messages into a "read only" database? Or perhaps better the other way around, let the server periodically interrupt its loading activity, close the database, rename it, open and initialize a new base and then continue to load? Or is there presently simply no solution available? Thats exactly what i do after each hour. I rename the current db with the current date_hour and create a new database for the next incoming items. Shifting is not really an alternative, cause it will probably take too long to insert the items into a second database and delete them from the "main" database. Kind regards, Andreas
AW
Andreas Weiler5 Jul 2012 08:57 #
Hello Hans-Juergen,
Quoted text · Hans-Juergen Rennau · 1 line
> was this database renaming+creation triggered by the client or autonomously done by the server?
I have a kind of "manager" client running, which executes the database renaming+creation for each hour. Since we have single writer transactions, all other transactions are blocked during that time. So the next client transactions will automatically use the "new" database. I have it running like this: create db tmp <root/> .... insert items .... close alter db tmp date_hour create db tmp <root/> ..... insert items ....
Quoted text · Hans-Juergen Rennau · 1 line
> Concerning the querying tasks to be expected I know very little myself - except for the need to group messages by transaction IDs and then somehow evaluate the execution of transactions. But at the moment I suppose that your "monster document" approach might be appropriate in our case, too.
I guess a "monster document" will be fine. However i would not create too large database, cause you probably need to create indexes afterwards and that could take a long time for *very* large databases. Let me know if you need more information, Andreas
Quoted text · Hans-Juergen Rennau · 36 lines
Am 04.07.2012 um 19:54 schrieb Hans-Juergen Rennau: > Hello Andreas, > > was this database renaming+creation triggered by the client or autonomously done by the server? The latter alternative would require some scheduling feature, so that the database server could be configured to perform such actions according to a schedule. I think this would be a very desirable feature, especially as long as high frequency storage and querying cannot be done well concurrently. Anyway, do you see a way to implement this periodic renaming+creation behaviour in a multi-client environment? > > Concerning the querying tasks to be expected I know very little myself - except for the need to group messages by transaction IDs and then somehow evaluate the execution of transactions. But at the moment I suppose that your "monster document" approach might be appropriate in our case, too. > > Kind regards, > Hans-Juergen > > Von: Andreas Weiler <andreas.weiler at uni-konstanz.de> > An: Hans-Juergen Rennau <hrennau at yahoo.de> > CC: Base X <basex-talk at mailman.uni-konstanz.de> > Gesendet: 9:45 Mittwoch, 4.Juli 2012 > Betreff: Re: [basex-talk] BaseX as a log msg store? > > Hello Hans-Juergen, > >> So my understanding is that the messages are inserted as child elements into this root element - and the end result is one document with one root element and millions of child elements representing the invidual messages, yes? > > > Yes that is correct, i have one root element at the beginning and insert the incoming items as child nodes of the root. > >> Therefore you do not have to come up with URIs, as there is only one single document. A monster document, but I conclude from your approach that this is no problem, and not worse (or even better) than having a million individual, small documents. Is it correct - would you recommend to store the messages in one single document? > > In my use case, tweets have unique id attributes, so i don't need any URIs to identify them. Probably, it is a good idea if you describe your further querying process so it is easier to understand what you want to do. > >> If the loading process cannot concur with queries - would there be any way how one could periodically "shift" packages of messages into a "read only" database? Or perhaps better the other way around, let the server periodically interrupt its loading activity, close the database, rename it, open and initialize a new base and then continue to load? Or is there presently simply no solution available? > > > Thats exactly what i do after each hour. I rename the current db with the current date_hour and create a new database for the next incoming items. Shifting is not really an alternative, cause it will probably take too long to insert the items into a second database and delete them from the "main" database. > > Kind regards, > Andreas > >
HR
Hans-Juergen Rennau5 Jul 2012 09:47 #
Hello Andreas, cordial thanks for the additional information. To summarize my understanding: a feasible way to effect a "rolling database" which is periodically renamed and replaced by a newly created database is feasible in a multi-client environment - when one provides a "manager client" triggering theses actions. I shall gladly come back for more information should the need arise. Cheers, Hans-Juergen ________________________________ Von: Andreas Weiler <andreas.weiler at uni-konstanz.de> An: Hans-Juergen Rennau <hrennau at yahoo.de> CC: Base X <basex-talk at mailman.uni-konstanz.de> Gesendet: 8:57 Donnerstag, 5.Juli 2012 Betreff: Re: [basex-talk] BaseX as a log msg store? Hello Hans-Juergen, was this database renaming+creation triggered by the client or autonomously done by the server? I have a kind of "manager" client running, which executes the database renaming+creation for each hour. Since we have single writer transactions, all other transactions are blocked during that time. So the next client transactions will automatically use the "new" database. I have it running like this: create db tmp <root/> .... insert items .... close alter db tmp date_hour create db tmp <root/> ..... insert items .... Concerning the querying tasks to be expected I know very little myself - except for the need to group messages by transaction IDs and then somehow evaluate the execution of transactions. But at the moment I suppose that your "monster document" approach might be appropriate in our case, too. I guess a "monster document" will be fine. However i would not create too large database, cause you probably need to create indexes afterwards and that could take a long time for *very* large databases. Let me know if you need more information, Andreas
AW
Andreas Weiler5 Jul 2012 09:55 #
Hello Hans-Juergen, thats correct, have fun with your project. -- Andreas
Quoted text · Hans-Juergen Rennau · 39 lines
Am 05.07.2012 um 09:47 schrieb Hans-Juergen Rennau: > Hello Andreas, cordial thanks for the additional information. To summarize my understanding: a feasible way to effect a "rolling database" which is periodically renamed and replaced by a newly created database is feasible in a multi-client environment - when one provides a "manager client" triggering theses actions. I shall gladly come back for more information should the need arise. > > Cheers, > Hans-Juergen > > Von: Andreas Weiler <andreas.weiler at uni-konstanz.de> > An: Hans-Juergen Rennau <hrennau at yahoo.de> > CC: Base X <basex-talk at mailman.uni-konstanz.de> > Gesendet: 8:57 Donnerstag, 5.Juli 2012 > Betreff: Re: [basex-talk] BaseX as a log msg store? > > Hello Hans-Juergen, > >> was this database renaming+creation triggered by the client or autonomously done by the server? > > I have a kind of "manager" client running, which executes the database renaming+creation for each hour. Since we have > single writer transactions, all other transactions are blocked during that time. So the next client transactions will automatically > use the "new" database. > > I have it running like this: > > create db tmp <root/> > .... insert items .... > > close > alter db tmp date_hour > create db tmp <root/> > ..... insert items .... > >> Concerning the querying tasks to be expected I know very little myself - except for the need to group messages by transaction IDs and then somehow evaluate the execution of transactions. But at the moment I suppose that your "monster document" approach might be appropriate in our case, too. > > I guess a "monster document" will be fine. However i would not create too large database, cause you probably need to create indexes afterwards and that could take a long time for *very* large databases. > > Let me know if you need more information, > Andreas > > >
PH
Pascal Heusin reply to Christian Grün, 4 Jul 2012, 04:245 Jul 2012 14:28 #
Christian: option#2 is great, hope it find its way in the standard (add my vote to it) best *P
Quoted text · Christian Grün · 44 lines
On 7/3/12 10:24 PM, Christian Grün wrote: > Hi Pascal, > > you can either create a new, single string via the fn:string-join() function… > > string-join(for $i in 1 to 10 return "x","&#10;") > > …or use the "output:separator" option of BaseX: > > declare option output:separator '\n'; > (1 to 10) ! "x" > > The second option, or a variation of it, may eventually find entrance > into the official W3 XQuery Serialization Specification (see [1] for > future details). > > Hope this helps, > Christian > > [1] https://www.w3.org/Bugs/Public/show_bug.cgi?id=16311 > > >> Anyone has a little xquery / basex tip in getting rid of the extra white >> space that shows up at the beginning of each line after the first in the >> following xquery >> for $i in 1 to 10 return concat("x","&#10;") >> which in the UI returns >> x >> x >> x >> x >> x >> Just need a text output going to csv but this extra space is causing me >> headaches. >> tks >> *P >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>
HR
Hans-Juergen Rennauin reply to Andreas Weiler, 5 Jul 2012, 09:556 Jul 2012 23:19 #
Hello, it seems to me that function db:open() and collection() are equivalent. db:open - in spite of its name - does not change any state, but just returns the document nodes contained in a database. Which is what collection() does (in BaseX). The only differences are differences of parameter style: (a) a database path (if any) is an extra parameter in db:open(), whereas with the collection() function it is appended to the db name (e.g. collection('db/data'); and (b) db:open() lets one specify the database by name or by node, whereas with collection() it is (of course) always only by name. If something is wrong with this summary, I would appreciate a correction. Kind regards, Hans-Juergen
CG
Christian Grün7 Jul 2012 04:07 #
Hi Hans-Jürgen,
Quoted text · Hans-Juergen Rennau · 2 lines
> it seems to me that function db:open() and collection() are equivalent. > [...]
the main difference of the collection() and doc() functions is that they can also access local resources: you may as well use them to create a temporary main-memory collection from all files found at the given local file path [1]. In other words, db:open() will be a little bit more efficient, and cause no surprises, if you know that your files are stored in a database. Hope this helps, Christian [1] http://docs.basex.org/wiki/Database#XML_Documents
HR
Hans-Juergen Rennau7 Jul 2012 08:27 #
Christian, thank you very much - this difference is what I missed, and it is really important. So - doc() and collection() work in two steps: (a) try to interpret the argument URI as a "database resource URI" (defined to be the concatenation of database name and database path), locating a database document or directory (b) if no database documents were located (that is - the argument is no valid database resource URI): interpret the URI as a file:// URI, locating a file or directory. Concerning the default collection, I only just now discovered the difference between the function db:open and the command <open name="...">: the command changes the dynamic context for a subsequent query, making the documents of the opened db the default collection - whereas db:open does not change the dynamic context. (Well - otherwise it would not be side-effect free.) One more question: Does the open-command change the dynamic context of subsequent queries in any other way except for setting the default collection? Cheers, Hans-Juergen ________________________________ Von: Christian Grün <christian.gruen at gmail.com> An: Hans-Juergen Rennau <hrennau at yahoo.de> CC: Base X <basex-talk at mailman.uni-konstanz.de> Gesendet: 4:07 Samstag, 7.Juli 2012 Betreff: Re: [basex-talk] db:open and collection() Hi Hans-Jürgen,
Quoted text · Christian Grün · 2 lines
> it seems to me that function db:open() and collection() are equivalent. > [...]
the main difference of the collection() and doc() functions is that they can also access local resources: you may as well use them to create a temporary main-memory collection from all files found at the given local file path [1]. In other words, db:open() will be a little bit more efficient, and cause no surprises, if you know that your files are stored in a database. Hope this helps, Christian [1] http://docs.basex.org/wiki/Database#XML_Documents
CG
Christian Grün7 Jul 2012 08:38 #
Quoted text · Hans-Juergen Rennau · 3 lines
> One more question: > Does the open-command change the dynamic context of subsequent queries in > any other way except for setting the default collection?
From the XQuery perspective, this should indeed be all. Christian

⚡Generated with XQuery