Indexes creation fails on big database
Tuesday, 23 April 2013 · 4 messages
EK
Erdal Karaca23 Apr 2013 19:29 #
Hi all,
I have downloaded/started the latest dev snapshot of 7.7.
My database consists of an extract of the OpenStreetMap data (only Germany) which is 29GB in size (xml/OSM format).
I have managed to load the file into the database but the indexes creation procedure fails. I have tried -Xmx4g, but if the progress bar of the GUI is right, I would need the 10x of heap space to create the attribute indexes.
Any ideas/workarounds to create the attribute indexes for such a big database?
Thanks!
CG
Christian Grün23 Apr 2013 19:35 #
Hi Erdal,
the index split heuristic seems to fail in your case, so you could try to set some fixed index split sizes [1]. I agree this requires some manual fine tuning.
Best,
Christian
[1] http://docs.basex.org/wiki/Indexes#Index_Construction
Quoted text · Erdal Karaca · 20 lines
On Tue, Apr 23, 2013 at 7:29 PM, Erdal Karaca <erdal.karaca.de at gmail.com> wrote: > > Hi all, > I have downloaded/started the latest dev snapshot of 7.7. > My database consists of an extract of the OpenStreetMap data (only Germany) > which is 29GB in size (xml/OSM format). > I have managed to load the file into the database but the indexes creation > procedure fails. I have tried -Xmx4g, but if the progress bar of the GUI is > right, I would need the 10x of heap space to create the attribute indexes. > > Any ideas/workarounds to create the attribute indexes for such a big > database? > > Thanks! > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
EK
Erdal Karaca23 Apr 2013 19:58 #
Hi Christian,
Thanks for the quick reply!
What value should I set the option INDEXSPLITSIZE to?
Thanks!
2013/4/23 Christian Grün <christian.gruen at gmail.com>
Quoted text · Christian Grün · 38 lines
> Hi Erdal, > > the index split heuristic seems to fail in your case, so you could try > to set some fixed index split sizes [1]. I agree this requires some > manual fine tuning. > > Best, > Christian > > [1] http://docs.basex.org/wiki/Indexes#Index_Construction > ___________________________ > > On Tue, Apr 23, 2013 at 7:29 PM, Erdal Karaca <erdal.karaca.de at gmail.com> > wrote: > > > > Hi all, > > I have downloaded/started the latest dev snapshot of 7.7. > > My database consists of an extract of the OpenStreetMap data (only > Germany) > > which is 29GB in size (xml/OSM format). > > I have managed to load the file into the database but the indexes > creation > > procedure fails. I have tried -Xmx4g, but if the progress bar of the GUI > is > > right, I would need the 10x of heap space to create the attribute > indexes. > > > > Any ideas/workarounds to create the attribute indexes for such a big > > database? > > > > Thanks! > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > >
CG
Christian Grün23 Apr 2013 20:02 #
Please first check out my link; you’ll have to try a little.
Quoted text · Erdal Karaca · 50 lines
On Tue, Apr 23, 2013 at 7:58 PM, Erdal Karaca <erdal.karaca.de at gmail.com> wrote: > Hi Christian, > Thanks for the quick reply! > > What value should I set the option INDEXSPLITSIZE to? > > Thanks! > > > 2013/4/23 Christian Grün <christian.gruen at gmail.com> >> >> Hi Erdal, >> >> the index split heuristic seems to fail in your case, so you could try >> to set some fixed index split sizes [1]. I agree this requires some >> manual fine tuning. >> >> Best, >> Christian >> >> [1] http://docs.basex.org/wiki/Indexes#Index_Construction >> ___________________________ >> >> On Tue, Apr 23, 2013 at 7:29 PM, Erdal Karaca <erdal.karaca.de at gmail.com> >> wrote: >> > >> > Hi all, >> > I have downloaded/started the latest dev snapshot of 7.7. >> > My database consists of an extract of the OpenStreetMap data (only >> > Germany) >> > which is 29GB in size (xml/OSM format). >> > I have managed to load the file into the database but the indexes >> > creation >> > procedure fails. I have tried -Xmx4g, but if the progress bar of the GUI >> > is >> > right, I would need the 10x of heap space to create the attribute >> > indexes. >> > >> > Any ideas/workarounds to create the attribute indexes for such a big >> > database? >> > >> > Thanks! >> > >> > _______________________________________________ >> > BaseX-Talk mailing list >> > BaseX-Talk at mailman.uni-konstanz.de >> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> > > >
Written for BaseX 7.6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery