BaseX Evaluation for Big Data
Friday, 6 December 2013 · 6 messages
JM
Hi Gary
I am a BaseX team member, but have you sized up your JVM to run "Big Data" ?
See the files BaseX/basex*.bat. I am currently using set VM=-Xmx6g
Hope this helps
Jean-Marc
2013/12/6 Huband, Gary W *HS <GWH2SJ at hscmail.mcc.virginia.edu>
Quoted text · Huband, Gary W *HS · 20 lines
> I'm evaluating BaseX to use in a project that involves 10s of TB of xml > data. Our goal is to be able to return a small amount of data from a query > within a few seconds. Is BaseX capable of handling this much data? > > My initial evaluation of 1GB of data met our requirements. But, when I > evaluated BaseX using about 20GB of data I get an out of memory message > when I run a query. > > I'm running Windows 7 Pro on a Core i7, 8GB memory, and using the BaseX > GUI. > > Thanks for your time > > Gary > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
JM
jean-marc Mercier6 Dec 2013 15:37 #
I meant I am NOT a baseX member ! sorry for the mistyping !
2013/12/6 jean-marc Mercier <jeanmarc.mercier at gmail.com>
Quoted text · jean-marc Mercier · 36 lines
> Hi Gary > > I am a BaseX team member, but have you sized up your JVM to run "Big Data" > ? > > See the files BaseX/basex*.bat. I am currently using set VM=-Xmx6g > > Hope this helps > > Jean-Marc > > > 2013/12/6 Huband, Gary W *HS <GWH2SJ at hscmail.mcc.virginia.edu> > > I'm evaluating BaseX to use in a project that involves 10s of TB of xml >> data. Our goal is to be able to return a small amount of data from a query >> within a few seconds. Is BaseX capable of handling this much data? >> >> My initial evaluation of 1GB of data met our requirements. But, when I >> evaluated BaseX using about 20GB of data I get an out of memory message >> when I run a query. >> >> I'm running Windows 7 Pro on a Core i7, 8GB memory, and using the BaseX >> GUI. >> >> Thanks for your time >> >> Gary >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> > >
HH
Huband, Gary W *HS6 Dec 2013 16:22 #
I'm evaluating BaseX to use in a project that involves 10s of TB of xml data. Our goal is to be able to return a small amount of data from a query within a few seconds. Is BaseX capable of handling this much data?
My initial evaluation of 1GB of data met our requirements. But, when I evaluated BaseX using about 20GB of data I get an out of memory message when I run a query.
I'm running Windows 7 Pro on a Core i7, 8GB memory, and using the BaseX GUI.
Thanks for your time
Gary
JM
jean-marc Mercier6 Dec 2013 17:42 #
Hi Gary,
Maybe you should wait for BaseX team members advises over this topic.
Telling from my own experience, I can remember also having met some problem trying to insert 20 Gb of data into BaseX through the GUI.
I had to achieve this into smaller steps, using batch commands (see http://docs.basex.org/wiki/Commands), in order to free memory. i.e. I first inserted the xml into 4 or 5 passes. Then created some indexes, etc...In that way it was possible to run the insertion, even with 4Gb machines.
Hope this helps
Jean-Marc
2013/12/6 Huband, Gary W *HS <GWH2SJ at hscmail.mcc.virginia.edu>
Quoted text · Huband, Gary W *HS · 57 lines
> I still get an out of main memory when I try to create attribute indexes. > > I originally added the xml data with no indexes. > > Gary > > ________________________________ > From: jean-marc Mercier [jeanmarc.mercier at gmail.com] > Sent: Friday, December 06, 2013 9:37 AM > To: Huband, Gary W *HS > Cc: basex-talk at mailman.uni-konstanz.de > Subject: Re: [basex-talk] BaseX Evaluation for Big Data > > I meant I am NOT a baseX member ! sorry for the mistyping ! > > > 2013/12/6 jean-marc Mercier <jeanmarc.mercier at gmail.com<mailto: > jeanmarc.mercier at gmail.com>> > Hi Gary > > I am a BaseX team member, but have you sized up your JVM to run "Big Data" > ? > > See the files BaseX/basex*.bat. I am currently using set VM=-Xmx6g > > Hope this helps > > Jean-Marc > > > 2013/12/6 Huband, Gary W *HS <GWH2SJ at hscmail.mcc.virginia.edu<mailto: > GWH2SJ at hscmail.mcc.virginia.edu>> > > I'm evaluating BaseX to use in a project that involves 10s of TB of xml > data. Our goal is to be able to return a small amount of data from a query > within a few seconds. Is BaseX capable of handling this much data? > > My initial evaluation of 1GB of data met our requirements. But, when I > evaluated BaseX using about 20GB of data I get an out of memory message > when I run a query. > > I'm running Windows 7 Pro on a Core i7, 8GB memory, and using the BaseX > GUI. > > Thanks for your time > > Gary > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de<mailto: > BaseX-Talk at mailman.uni-konstanz.de> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > >
JM
jean-marc Mercier6 Dec 2013 18:10 #
Gary,
A remark over this topic, since you are worried about performances for 20 Gb of data with BaseX and might be interested by a user experience. Once you will be successful in inserting, simple XPATH queries over such a collection should take time below milliseconds.
2013/12/6 jean-marc Mercier <jeanmarc.mercier at gmail.com>
Quoted text · jean-marc Mercier · 82 lines
> Hi Gary, > > Maybe you should wait for BaseX team members advises over this topic. > > Telling from my own experience, I can remember also having met some > problem trying to insert 20 Gb of data into BaseX through the GUI. > > I had to achieve this into smaller steps, using batch commands (see > http://docs.basex.org/wiki/Commands), in order to free memory. i.e. I > first inserted the xml into 4 or 5 passes. Then created some indexes, > etc...In that way it was possible to run the insertion, even with 4Gb > machines. > > Hope this helps > > Jean-Marc > > > > > > > 2013/12/6 Huband, Gary W *HS <GWH2SJ at hscmail.mcc.virginia.edu> > >> I still get an out of main memory when I try to create attribute indexes. >> >> I originally added the xml data with no indexes. >> >> Gary >> >> ________________________________ >> From: jean-marc Mercier [jeanmarc.mercier at gmail.com] >> Sent: Friday, December 06, 2013 9:37 AM >> To: Huband, Gary W *HS >> Cc: basex-talk at mailman.uni-konstanz.de >> Subject: Re: [basex-talk] BaseX Evaluation for Big Data >> >> I meant I am NOT a baseX member ! sorry for the mistyping ! >> >> >> 2013/12/6 jean-marc Mercier <jeanmarc.mercier at gmail.com<mailto: >> jeanmarc.mercier at gmail.com>> >> Hi Gary >> >> I am a BaseX team member, but have you sized up your JVM to run "Big >> Data" ? >> >> See the files BaseX/basex*.bat. I am currently using set VM=-Xmx6g >> >> Hope this helps >> >> Jean-Marc >> >> >> 2013/12/6 Huband, Gary W *HS <GWH2SJ at hscmail.mcc.virginia.edu<mailto: >> GWH2SJ at hscmail.mcc.virginia.edu>> >> >> I'm evaluating BaseX to use in a project that involves 10s of TB of xml >> data. Our goal is to be able to return a small amount of data from a query >> within a few seconds. Is BaseX capable of handling this much data? >> >> My initial evaluation of 1GB of data met our requirements. But, when I >> evaluated BaseX using about 20GB of data I get an out of memory message >> when I run a query. >> >> I'm running Windows 7 Pro on a Core i7, 8GB memory, and using the BaseX >> GUI. >> >> Thanks for your time >> >> Gary >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de<mailto: >> BaseX-Talk at mailman.uni-konstanz.de> >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> >> >> >
CG
Dear Gary,
and thanks, Jean-Marc, for the feedback.
Quoted text · Huband, Gary W *HS · 1 line
> I'm evaluating BaseX to use in a project that involves 10s of TB of xml data. Our goal is to be able to return a small amount of data from a query within a few seconds. Is BaseX capable of handling this much data?
BaseX can in fact be used to handle data in the size of terabytes. Some database statistics are listed in our Wiki [1], but in most cases, a single database should not exceed 10-50 gigabytes (the optimal size is always dependent on the actual structure of your input data). An important thing to mention here is that BaseX allows you to address any number of databases with a single XQuery expression, so the limit is not as restrictive as it may sound.
Quoted text · Huband, Gary W *HS · 1 line
> My initial evaluation of 1GB of data met our requirements. But, when I evaluated BaseX using about 20GB of data I get an out of memory message when I run a query.
This is something that shouldn’t happen of course. Could you please provide us with the query that triggered the error?
Thanks,
Christian
[1] http://docs.basex.org/wiki/Statistics
Written for BaseX 7.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery