Selective indexes

From: ferrari_martin at hotmail.com To: basex-talk at mailman.uni-konstanz.de Subject: Selective indexes Date: Tue, 1 Sep 2015 01:37:31 +0000 Hi Dirk, I didn't want to hijack the other thread, so I started a new one :) I've read https://github.com/BaseXdb/basex/issues/59 and I think it's great!. I have a slightly different request though that I was holding back because I didn't want to sound abusive, but now that you brought it up... I might be mistaken as I haven't really debugged the code, but I believe that indexing could maybe take so long (for our use case anyway) because it indexes all the values in the xml documents. So I wonder whether it would be beneficial for the indexing speed to be able to state which specific fields to index instead of all of them. This would allow for having fast regularly used queries, without impacting insertions that much. I can explain our use case quickly: we store XML files that are interchanged with third parties. We have lots of them all the time, so insertion speed is critical. Query speed on the other hand is not critical, though obviously having fast queries is a lot better than having slow ones. But it becomes more important as the queries lock the db for writing too, aside from feeling "if I just could index this one field", hehe. If we could just index the fields that we run most of our queries against, it would be perfect. Right now, I think we can get away with organizing resources inside directories and that would be enough as that seems to work perfectly (we'll perform a couple more tests with high volume), but I thought it would be really cool if we could have a similar insertion speed to what's achieved without indexing, by just indexing some fields (again, *guessing* that the cause of the decrease in speed is due to indexing all fields in the documents, which may or may not be the case). Thanks! Martín.

On 09/01/2015 03:55 AM, Martín Ferrari wrote: > Haha, does the selective index feature just do what I spent 30 minutes > writing about? > Sorry, I think my reading abilities were impaired, or I was too > impatient to read until the end of the issue. > > I'll give it a try when I have some moments, it looks pretty cool! > > Thanks, > Martín. > > ------------------------------------------------------------------------ > From: ferrari_martin at hotmail.com > To: basex-talk at mailman.uni-konstanz.de > Subject: Selective indexes > Date: Tue, 1 Sep 2015 01:37:31 +0000 > > Hi Dirk, > I didn't want to hijack the other thread, so I started a new one :) > > I've read https://github.com/BaseXdb/basex/issues/59 and I think > it's great!. I have a slightly different request though that I was > holding back because I didn't want to sound abusive, but now that you > brought it up... > I might be mistaken as I haven't really debugged the code, but I > believe that indexing could maybe take so long (for our use case > anyway) because it indexes all the values in the xml documents. So I > wonder whether it would be beneficial for the indexing speed to be > able to state which specific fields to index instead of all of them. > This would allow for having fast regularly used queries, without > impacting insertions that much. > > I can explain our use case quickly: we store XML files that are > interchanged with third parties. We have lots of them all the time, so > insertion speed is critical. Query speed on the other hand is not > critical, though obviously having fast queries is a lot better than > having slow ones. But it becomes more important as the queries lock > the db for writing too, aside from feeling "if I just could index this > one field", hehe. If we could just index the fields that we run most > of our queries against, it would be perfect. > Right now, I think we can get away with organizing resources > inside directories and that would be enough as that seems to work > perfectly (we'll perform a couple more tests with high volume), but I > thought it would be really cool if we could have a similar insertion > speed to what's achieved without indexing, by just indexing some > fields (again, *guessing* that the cause of the decrease in speed is > due to indexing all fields in the documents, which may or may not be > the case). > > Thanks! > Martín.