Incremental fulltext update?
Thursday, 2 January 2014 · 2 messages · last reply 11 Jan 2014
FH
Florent Heyworth2 Jan 2014 19:14 #
Hi all
I've been experimenting with BaseX - and I'm quite pleased with its capabilities - however I'm hitting a wall where the fulltext index is concerned. Here's the use-case:
- Web application analyzing media (news, etc.) on a large scale
- Frequent inserts (metadata about news) by many clients (ca. 40+ million documents per year)
- Online Full-text search is a must
Every time data is inserted into a database the full-text index is invalidated. Because of the online full-text search capability requirement a 2-stage update process (store to temporary database, batch update to main database and optimize full-text index afterwards) is unfortunately not an option.
Are there plans to provide for incremental full-text update capacity or is this not an option?
Thanks for any feedback and kind regards
Florent
CG
Christian Grün11 Jan 2014 13:28 #
Hi Florent,
sorry for letting you wait.
Incremental full-text updates are on our agenda, but they currently have low priority. As you already indicated, one solution with the current architecture would be to split up your data into more than one database and. Recently updated documents would then be stored in and queried from a temporary db without full-text index, and data that is large and/or static would be indexed, and only optimized if merged with temporary instances.
Christian
Quoted text · Florent Heyworth · 28 lines
On Thu, Jan 2, 2014 at 7:14 PM, Florent Heyworth <florent at napalu.ch> wrote: > Hi all > > I've been experimenting with BaseX - and I'm quite pleased with its > capabilities - however I'm hitting a wall where the fulltext index is > concerned. Here's the use-case: > > - Web application analyzing media (news, etc.) on a large scale > - Frequent inserts (metadata about news) by many clients (ca. 40+ million > documents per year) > - Online Full-text search is a must > > Every time data is inserted into a database the full-text index is > invalidated. Because of the online full-text search capability requirement a > 2-stage update process (store to temporary database, batch update to main > database and optimize full-text index afterwards) is unfortunately not an > option. > > Are there plans to provide for incremental full-text update capacity or is > this not an option? > > Thanks for any feedback and kind regards > > Florent > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Written for BaseX 7.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery