Removing document from collection erases indexes
Monday, 25 October 2010 · 6 messages · last reply 26 Oct 2010
IL
Ivan Lagunov25 Oct 2010 16:26 #
Hello,
Is it the correct behavior that indexes are automatically removed when I remove a document from a collection (version 6.3)? I mean text and attribute indexes, I haven't tried full-text index.
Best regards,
Ivan Lagunov
CG
Christian Grün26 Oct 2010 15:46 #
Dear Ivan,
Quoted text · Ivan Lagunov · 3 lines
> Is it the correct behavior that indexes are automatically removed when I > remove a document from a collection (version 6.3)? I mean text and attribute > indexes, I haven't tried full-text index.
Yes, that was one more deliberate design decision to make updates as fast as possible in BaseX. The indexes are rebuilt whenever the "optimize" command is called; even so, tests with different systems have shown that our solution is often superior to implementations that refresh the index after each update operation. Still, one of our team members is currently integrating incremental index updates, which will be offered as an optional feature.
Christian
Quoted text · Ivan Lagunov · 12 lines
On Mon, Oct 25, 2010 at 4:26 PM, Ivan Lagunov <ilagunov at evelopers.com> wrote: > Hello, > > > Best regards, > Ivan Lagunov > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
IL
Ivan Lagunov26 Oct 2010 16:01 #
Dear Christian,
Ok. However, it doesn't seem comfortable to me that I'll have to execute optimize now after every update/delete operation. Is there any estimated date when incremental index updates will be ready?
Best regards,
Ivan Lagunov
Quoted text · Christian Grün · 32 lines
On 10/26/2010 05:46 PM, Christian Grün wrote: > Dear Ivan, > >> Is it the correct behavior that indexes are automatically removed when I >> remove a document from a collection (version 6.3)? I mean text and attribute >> indexes, I haven't tried full-text index. > Yes, that was one more deliberate design decision to make updates as > fast as possible in BaseX. The indexes are rebuilt whenever the > "optimize" command is called; even so, tests with different systems > have shown that our solution is often superior to implementations that > refresh the index after each update operation. Still, one of our team > members is currently integrating incremental index updates, which will > be offered as an optional feature. > > Christian > > > > > On Mon, Oct 25, 2010 at 4:26 PM, Ivan Lagunov<ilagunov at evelopers.com> wrote: >> Hello, >> >> >> Best regards, >> Ivan Lagunov >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >
JC
Hi Ivan
I am currently planning to implement solution, which will insert new documents each minute.
Question1: does the optimize recreate complete index, or it only does some small update?
Question2: if the index becomes invalid after insertion of new document, how does it affect query performance? A bit? or possibly a lot?
There are scenarios, like bulk updates, when it is established practice even in SQL world to switch off indices, import/update data and then rebuild the index.
However, rebuilding index once a minute seems as something, what I would like not to do, if it would take too much time.
Jan
On Tue, Oct 26, 2010 at 3:46 PM, Christian Grün
<christian.gruen at gmail.com>wrote:
Quoted text · Christian Grün · 38 lines
> Dear Ivan, > > > Is it the correct behavior that indexes are automatically removed when I > > remove a document from a collection (version 6.3)? I mean text and > attribute > > indexes, I haven't tried full-text index. > > Yes, that was one more deliberate design decision to make updates as > fast as possible in BaseX. The indexes are rebuilt whenever the > "optimize" command is called; even so, tests with different systems > have shown that our solution is often superior to implementations that > refresh the index after each update operation. Still, one of our team > members is currently integrating incremental index updates, which will > be offered as an optional feature. > > Christian > > > > > On Mon, Oct 25, 2010 at 4:26 PM, Ivan Lagunov <ilagunov at evelopers.com> > wrote: > > Hello, > > > > > > Best regards, > > Ivan Lagunov > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
--
*Ing. Jan Vlčinský*
CAD programy
Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic
tel: +420-597 602 024; mob: +420-608 979 040
skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com
http://cz.linkedin.com/in/vlcinsky
CG
Quoted text · Ivan Lagunov · 3 lines
> Ok. However, it doesn't seem comfortable to me that I'll have to execute > optimize now after every update/delete operation. Is there any estimated > date when incremental index updates will be ready?
Sorry – not really, as our resources are limited.
We'll soon offer the priorized (but paid) integration of new features, if that helps.
Christian
Quoted text · Christian Grün · 40 lines
> On 10/26/2010 05:46 PM, Christian Grün wrote: >> >> Dear Ivan, >> >>> Is it the correct behavior that indexes are automatically removed when I >>> remove a document from a collection (version 6.3)? I mean text and >>> attribute >>> indexes, I haven't tried full-text index. >> >> Yes, that was one more deliberate design decision to make updates as >> fast as possible in BaseX. The indexes are rebuilt whenever the >> "optimize" command is called; even so, tests with different systems >> have shown that our solution is often superior to implementations that >> refresh the index after each update operation. Still, one of our team >> members is currently integrating incremental index updates, which will >> be offered as an optional feature. >> >> Christian >> >> >> >> >> On Mon, Oct 25, 2010 at 4:26 PM, Ivan Lagunov<ilagunov at evelopers.com> >> wrote: >>> >>> Hello, >>> >>> >>> Best regards, >>> Ivan Lagunov >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> >> > > >
IL
Hi Jan,
Ok, I've checked what you ask about.
Question1 - the optimize function recreates complete index (takes the same 7 seconds every time via GUI and 5 seconds via terminal client). Question2 - it affects query performance a lot as if index isn't used any more at all.
I completely agree with your remark. And it'd be great to have optional setting of automatic incremental index.
Best regards,
Ivan
Quoted text · Jan Vlčinský (CAD) · 73 lines
On 10/26/2010 06:06 PM, Jan Vlčinský (CAD) wrote: > Hi Ivan > I am currently planning to implement solution, which will insert new > documents each minute. > Question1: does the optimize recreate complete index, or it only does > some small update? > Question2: if the index becomes invalid after insertion of new > document, how does it affect query performance? A bit? or possibly a lot? > > There are scenarios, like bulk updates, when it is established > practice even in SQL world to switch off indices, import/update data > and then rebuild the index. > However, rebuilding index once a minute seems as something, what I > would like not to do, if it would take too much time. > > Jan > > On Tue, Oct 26, 2010 at 3:46 PM, Christian Grün > <christian.gruen at gmail.com <mailto:christian.gruen at gmail.com>> wrote: > > Dear Ivan, > > > Is it the correct behavior that indexes are automatically > removed when I > > remove a document from a collection (version 6.3)? I mean text > and attribute > > indexes, I haven't tried full-text index. > > Yes, that was one more deliberate design decision to make updates as > fast as possible in BaseX. The indexes are rebuilt whenever the > "optimize" command is called; even so, tests with different systems > have shown that our solution is often superior to implementations that > refresh the index after each update operation. Still, one of our team > members is currently integrating incremental index updates, which will > be offered as an optional feature. > > Christian > > > > > On Mon, Oct 25, 2010 at 4:26 PM, Ivan Lagunov > <ilagunov at evelopers.com <mailto:ilagunov at evelopers.com>> wrote: > > Hello, > > > > > > Best regards, > > Ivan Lagunov > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > <mailto:BaseX-Talk at mailman.uni-konstanz.de> > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > <mailto:BaseX-Talk at mailman.uni-konstanz.de> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > -- > *Ing. Jan Vlčinský* > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > <mailto:jan.vlcinsky at gmail.com> > http://cz.linkedin.com/in/vlcinsky >
Written for BaseX 6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery