Initial observation on BaseX 6.0 release

On Sun, Jan 10, 2010 at 11:28 PM, Phil <newintellectual at gmail.com> wrote: > I've been looking forward to the new release because I really want to use an > XQuery Fulltext system for a search engine for a reasonably large (200,000 > printed page equivalent) XML structured text-oriented database. However, my > initial test, on a large fraction of the total database (certainly over > 120,000 pages printed equivalent) yields disappointing results. > > The XML schema is roughly based on Docbook, so it is not unusual. Most of > the actual text is within PARA elements. > > My test query was: > > //para[. ftcontains "sumter"] > > On a dual quad Xeon system the query took about 28 seconds to run. This is > completely unusable in my context. > > I am pretty sure I know the reason for the poor performance. The derived > query plan resolves to: > > <QueryPlan> > <IterPath> > <Root/> > <IterStep axis="descendant" test="*:para"> > <FTContains> > <Context/> > <FTWords>sumter</FTWords> > </FTContains> > </IterStep> > </IterPath> > </QueryPlan> > > Now, there are *millions* of PARA elements in the database - but not so many > (hundreds) of references to the word "sumter". And this is by no means an > uncommon XML structure for a text-oriented system. > > The problem is obvious. There are an absolutely enormous number of PARA > elements and very few actual text hits. The IterStep is evidently hitting > every single PARA element. The query optimization should be checking the > fulltext search index and using those results as an initial filter. Then the > IterStep over PARAs from that filtered set would be highly productive. > > Is there a way to get that kind of (properly) optimized query from the > system, or is this hopelessly built into the current architecture? > > I do hope that I get some kind of response this time, because at least two > previous queries on earlier releases of BaseX went unanswered. > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

On Tue, Jan 12, 2010 at 12:04 AM, Phil Oliver <phil3 at olivercomputing.com> wrote: > After some experimentation, I've identified the fulltext search index > problem. Hopefully it's something easily fixed. > > I did have the FT indexing turned on, but basically with all of the various > options. I decided to experiment with the minimal index and then to > systematically add options. It turns out that the query does use the FT > index in the instance where there are no options, or with wildcards turned > on. However, the index is no longer used once stemming, case sensitivity, or > diacritics are turned on in the FT database indexing options. > > The speed is impressive with the FT indexing enabled, I think good enough > for me to plan to use it in development. However, the other FT options are > potentially important (to me and others) and I hope it's a priority to fix > the indexing problem. > > Thanks for your help. > > Phil