seeking for a document in a collection with a million documents is very slow
Friday, 15 March 2013 · 5 messages · last reply 18 Mar 2013
FE
Fabrice Etanchaud15 Mar 2013 14:18 #
Dear all,
Does the document index rely on text or attribute index ?
One can experiment very slow response time when looking for a specific document name in an optimized collection of a million documents (in dynamic execution, without optimizations, for example within a loop with a name being the concatenation of string values). Is there a way to speed things up ?
Best regards,
Fabrice Etanchaud
Questel-orbit
FE
Fabrice Etanchaud17 Mar 2013 20:34 #
Dear all,
From what I read in the documentation,
My problem seems to be related to the update of the resource index.
Is this index updated after each add/replace/delete command, Or at the end of the commands' list ?
Last, could you please tell me if replace is equivalent to delete+add ?
Best,
Fabrice
Quoted text · Fabrice Etanchaud · 11 lines
De : Fabrice Etanchaud Envoyé : vendredi 15 mars 2013 14:18 À : 'basex-talk at mailman.uni-konstanz.de' Objet : seeking for a document in a collection with a million documents is very slow Dear all, Does the document index rely on text or attribute index ? One can experiment very slow response time when looking for a specific document name in an optimized collection of a million documents (in dynamic execution, without optimizations, for example within a loop with a name being the concatenation of string values). Is there a way to speed things up ? Best regards, Fabrice Etanchaud Questel-orbit
CG
Christian Grün18 Mar 2013 15:32 #
Hi Fabrice,
yes, the document index is updated with each updating command. If you perform numerous updates, you may get better performance by switching
AUTOFLUSH off [1]. Another alternative to speed up multiple update operations is to use XQuery for updates. Due to the pending update list semantics, however, It will require more main memory.
Christian
[1] http://docs.basex.org/wiki/Options#AUTOFLUSHQuoted text · Fabrice Etanchaud · 12 lines
> Dear all, > > From what I read in the documentation, > My problem seems to be related to the update of the resource index. > > Is this index updated after each add/replace/delete command, > Or at the end of the commands’ list ? > > Last, could you please tell me if replace is equivalent to delete+add ? > > Best, > Fabrice
FE
Fabrice Etanchaud18 Mar 2013 21:47 #
Thank you Christian.
It seems that the resource index is not persistent,
But rebuilt in memory at the first index access after the collection is opened. For my 3 M documents collection, it takes about 250 secs to respond to the first db:open('mycoll','mydoc') query. Following queries responds in millisecs before the collection gets opened again.
From your experience,
What could be the good way to handle a collection of several millions documents, With about ten thousands inserted/updated documents once a week ?
Best regards,
Fabrice
-----Message d'origine-----
Quoted text · Christian Grün · 21 lines
De : Christian Grün [mailto:christian.gruen at gmail.com] Envoyé : lundi 18 mars 2013 15:32 À : Fabrice Etanchaud Cc : basex-talk at mailman.uni-konstanz.de Objet : Re: [basex-talk] seeking for a document in a collection with a million documents is very slow Hi Fabrice, yes, the document index is updated with each updating command. If you perform numerous updates, you may get better performance by switching AUTOFLUSH off [1]. Another alternative to speed up multiple update operations is to use XQuery for updates. Due to the pending update list semantics, however, It will require more main memory. Christian [1] http://docs.basex.org/wiki/Options#AUTOFLUSH > Dear all, > > From what I read in the documentation, My problem seems to be related > to the update of the resource index. > > Is this index updated after each add/replace/delete command, Or at the > end of the commands' list ? > > Last, could you please tell me if replace is equivalent to delete+add ? > > Best, > Fabrice
CG
Christian Grün18 Mar 2013 23:47 #
Hi Fabrice,
Quoted text · Fabrice Etanchaud · 3 lines
> From your experience, > What could be the good way to handle a collection of several millions documents, > With about ten thousands inserted/updated documents once a week ?
The article on the Twitter use case may give you some hints how updates can be sped up [1]. Apart from that, I would propose to do some profiling in order to find our which operations require most time or memory. Have you already tried the autoflush option? Do you use XQuery or the commands for your updates?
Regarding your last question..
Quoted text · Fabrice Etanchaud · 1 line
>> Last, could you please tell me if replace is equivalent to delete+add ?
The operations should be quite comparable. If you know the names of all documents to be deleted in advance, you could first delete all commands in a
db:delete loop and then add all new documents.
[1] http://docs.basex.org/wiki/TwitterQuoted text · Fabrice Etanchaud · 32 lines
> -----Message d'origine----- > De : Christian Grün [mailto:christian.gruen at gmail.com] > Envoyé : lundi 18 mars 2013 15:32 > À : Fabrice Etanchaud > Cc : basex-talk at mailman.uni-konstanz.de > Objet : Re: [basex-talk] seeking for a document in a collection with a million documents is very slow > > Hi Fabrice, > > yes, the document index is updated with each updating command. If you perform numerous updates, you may get better performance by switching AUTOFLUSH off [1]. Another alternative to speed up multiple update operations is to use XQuery for updates. Due to the pending update list semantics, however, It will require more main memory. > > Christian > > [1] http://docs.basex.org/wiki/Options#AUTOFLUSH > ___________________________ > >> Dear all, >> >> From what I read in the documentation, My problem seems to be related >> to the update of the resource index. >> >> Is this index updated after each add/replace/delete command, Or at the >> end of the commands' list ? >> >> Last, could you please tell me if replace is equivalent to delete+add ? >> >> Best, >> Fabrice > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
Written for BaseX 7.6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery