Full Text Indexing 8 GB DB !!
Saturday, 25 May 2013 · 2 messages · last reply 27 May 2013
JB
John Best25 May 2013 08:10 #
Hi Team,
I have a XML DB with approx 8 GB in size. The size of the DB will go on increasing through monthly updates. The size of monthly update will be around 10 MB in size.As the DB is storing text documents, I will have to index the full-text index again, after updating to speed up the process. Sometimes, I get Heap Space error too !!
My DB will be stored at the user end. Then how come I will be able to Optimize it?
Also the DB is created using a huge XML file.
After viewing the Statistics (http://docs.basex.org/wiki/Statistics), I came to understand that the size of my DB is stands no where, when compared to at least, first 15-16 entries !!.
This situation has always troubled me a lot. May be I am doing it in wrong way.. Can you guys put light on this?
--
Have a nice day
JBest
CG
Christian Grün27 May 2013 08:29 #
Hi John,
creating a full-text index for a database of 8GB size may be a challenge if not enough main memory is available. I recommend you to play around with the FTINDEXSPLITSIZE argument [1]. If this doesn’t help, I would recommend you to split your data into several database instances.
Please note that we have turned off the index structures while creating the database statistics. I have added a sentence to make this more explicit; beside that, feel free to check out the original source [2].
Hope this helps,
Christian
[1] http://docs.basex.org/wiki/Indexes#Index_Construction
[2] http://files.basex.org/publications/Gruen%20[2010],%20Storing%20and%20Querying%20Large%20XML%20Instances.pdf
Quoted text · John Best · 30 lines
> I have a XML DB with approx 8 GB in size. The size of the DB will go on > increasing through monthly updates. The size of monthly update will be > around 10 MB in size.As the DB is storing text documents, I will have to > index the full-text index again, after updating to speed up the process. > Sometimes, I get Heap Space error too !! > > My DB will be stored at the user end. Then how come I will be able to > Optimize it? > > Also the DB is created using a huge XML file. > > After viewing the Statistics (http://docs.basex.org/wiki/Statistics), I came > to understand that the size of my DB is stands no where, when compared to at > least, first 15-16 entries !!. > > This situation has always troubled me a lot. May be I am doing it in wrong > way.. Can you guys put light on this? > > > -- > Have a nice day > JBest > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Written for BaseX 7.6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery