Student project based on BaseX

On Sun, Jun 27, 2010 at 4:06 PM, Antonis Thomas <iorulezz at gmail.com> wrote: > Hello all, > > we are two master students at the university of Utrecht. We have a DB > Architecture class that includes a small project (15%*7.5 ECTS= 1.125 > ECTS). So, BaseX looks like a good chance to work on such a project. > > Our idea so far is based on some benchmarking experiments and small > summary on how it works (maybe its relation with Monet) including any > experiments we run (5-6 pages). Do you have anything better for us to do > (maybe something you can get advantage of yourselves) or any > guidelines/hints/whatever? > > Take into account that out time limit is approximately a week > (full-time) and that we cannot do more work that what deserves 2.25ECTS > for the 2 of us. > > I look forward to an answer (so we can decide and start working -the > deadline is 9/7-. > > > Best regards, > -- > > Antonis Thomas > Information & Computing Sciences > Utrecht University > Utrecht, The Netherlands > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

On 06/28/2010 01:34 AM, Christian Grün wrote: > Dear Antonis, > > thanks for your e-mail and your interest to use BaseX for your class. > > We have performed numerous comparisons with other XML databases by > ourselves, but your results could include helpful hints as well, so > your focus sounds fine to us. Keep in mind, however, that the given > time frame is very tough, as long as you haven't done serious > benchmarking before: you will most likely encounter side effects, > which can lead to completely wrong interpretations of performance > results. > > As first hint, I'd recommend you to use the latest BaseX version and > the client/server architecture for all tests, and the -v option to > display the BaseX timing output: > > $ java -Xmx1g -cp basex-6.1.7.jar org.basex.BaseXServer > BaseX 6.1.7 [Server] > Server was started. > > $ java -cp basex-6.1.7.jar org.basex.BaseXClient -Uadmin -Padmin -v query.xq > <xml/> > Parsing: 0.04 ms > Compiling: 0.02 ms > Evaluating: 0.01 ms > Printing: 0.05 ms > Total Time: 0.19 ms > Results: 1 Item > Printed: 42 Bytes > Memory: 2180 KB > > You might use the hidden "-r" option to increase the number of runs: > > $ java -cp basex-6.1.7.jar org.basex.BaseXClient -Uadmin -Padmin -r100 > -v query.xq > <xml/> > Parsing: 0.02 ms (avg) > Compiling: 0.0 ms (avg) > Evaluating: 0.0 ms (avg) > Printing: 0.02 ms (avg) > Total Time: 0.04 ms (avg) > Results: 1 Item > Printed: 42 Bytes > Memory: 2604 KB > > You can also use BaseX commands: > > $ java -cp basex-6.1.7.jar org.basex.BaseXClient -Uadmin -Padmin -c > "set info on; set runs 100; run query.xq" > … > > Check out http://basex.org/server for details on using c/s. Next, the > following post from Michael Kay on benchmarking is well worth reading: > > http://x-query.com/pipermail/talk/2009-October/003613.html > > Best, > Christian > > > > On Sun, Jun 27, 2010 at 4:06 PM, Antonis Thomas <iorulezz at gmail.com> wrote: > >> Hello all, >> >> we are two master students at the university of Utrecht. We have a DB >> Architecture class that includes a small project (15%*7.5 ECTS= 1.125 >> ECTS). So, BaseX looks like a good chance to work on such a project. >> >> Our idea so far is based on some benchmarking experiments and small >> summary on how it works (maybe its relation with Monet) including any >> experiments we run (5-6 pages). Do you have anything better for us to do >> (maybe something you can get advantage of yourselves) or any >> guidelines/hints/whatever? >> >> Take into account that out time limit is approximately a week >> (full-time) and that we cannot do more work that what deserves 2.25ECTS >> for the 2 of us. >> >> I look forward to an answer (so we can decide and start working -the >> deadline is 9/7-. >> >> >> Best regards, >> -- >> >> Antonis Thomas >> Information & Computing Sciences >> Utrecht University >> Utrecht, The Netherlands >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >>

> we have started working in this small project as I informed you. We have > set up server/client problem and we have run many test queries on the > standard Xmark database.

> With options -z -r100 and I get: > [...] > Printing: 140.66 ms (avg) > [...] > Do you think this output is normal? I do not understand why printing > time is not 0 since I use '-z'. Should we include printing in timing > our benchmarks?

> In addition what exactly does OPTIMIZE? If we just create a DB based on > an .xml file and we dont actually UPDATE anything but we only perform > queries, then optimize does not change anything, right? I mean when > created a DB is already 'optimized'?

> I tried to find all this answers in documentation but I couldn't.

> I would indeed suggest to use the total times for benchmarking, and to > minimize the time for printing the result. You could e.g. move all > output to /dev/null, or to a temporary file on disk, which is ok with > the performance of today's hard disk. To play fair with the > competition, you could omit the -z flag for BaseX. >

> This is fine. We will run some query examples based on Xmark. We will > compare the results of several queries with eXist. It will be similar to > here (unfortunately smaller scale): > http://monetdb.cwi.nl/XQuery/Benchmark/XMark/

> I am thinking to run it on three different sizes of Xmark generated > database: 1.1MB, 11MB, 110MB. Any hints/suggestions/requests here? > Note that both systems are running on client/server mode.

On 07/06/2010 08:18 PM, Christian Grün wrote: > To get new insights, it would be nice if you could include some other > DBs as well, but the given time restrictions won't probably allow to > do more. > > You could think about including at least 1GB instances (as a side > note, we'd created XMark instances up to 55GB).

> http://www.ins.cwi.nl/projects/xmark/Assets/xmlquery.txt > Since there was not enough time to run them all, I chose queries > 1,3,6,8,13,14 as an indicative subset.

> In addition I tried to run on 1g instance and baseX was the only one to > successfully run all the queries. For both eXist & Monet my 3gb of ram was > not enough to open such a large instance. Do you think I should report that? > (I still I am not exactly sure that I configured Monet the best way > possible.)

> If you want to review the benchmark results before the final version -maybe > you spot an anomaly and some query needs rerun- ask for it and I will send > you asap.

On 07/09/2010 12:17 PM, Christian Grün wrote: > It might be interesting to swap the order in which you're running the > queries. A query might be executed faster in Java if it's executed at > the end of the test, as the JIT compiler will delay initial processes; > garbage collection might influence runtimes as well, etc. If you don't > restart the database server every time you run new queries, the > differences should be less significant. >

> Feel free to send me a preview.

On 07/10/2010 12:53 PM, Antonis Thomas wrote: > On 07/09/2010 12:17 PM, Christian Grün wrote: >> It might be interesting to swap the order in which you're running the >> queries. A query might be executed faster in Java if it's executed at >> the end of the test, as the JIT compiler will delay initial processes; >> garbage collection might influence runtimes as well, etc. If you don't >> restart the database server every time you run new queries, the >> differences should be less significant. >> > I wasn't turning up and down the servers after each query. What I did > every time before testing each query I was running a couple of random > ones. This way the initial execution of each query took more time and > it smoothed out the more I executed the same one. In addition I didnt > use the '-r' of basex because the results were always much faster -is > there any kind of pipelining teke place there?- and since no other > system had such an option I went with normal runs. Usually basex > wanted 10 executions of a query in a row to reach its minimum -then it > flunctuates reasonably- similarly for exist. Monet was more stable in > the sense that it could reach its minimum sometimes from the first > query. That might be differences in C and Java though. Note that I had > to compile Monet on my system, no binary worked for me. > > >> Feel free to send me a preview. > > I am including the results. The files are simple CSVs. There is an > info file that explains a bit. > > > We are at the stage of finishing our writing. Again, please do not > expect much from our project, it's only a small one and databases is > not the main interest for neither of us. Although I can say it was > kind of enjoyable! > > > greetings, > Antonis

>>> It might be interesting to swap the order in which you're running the >>> queries. A query might be executed faster in Java if it's executed at >>> the end of the test, as the JIT compiler will delay initial processes; >>> garbage collection might influence runtimes as well, etc. If you don't >>> restart the database server every time you run new queries, the >>> differences should be less significant. >>> >> I wasn't turning up and down the servers after each query. What I did >> every time before testing each query I was running a couple of random ones. >> This way the initial execution of each query took more time and it smoothed >> out the more I executed the same one. In addition I didnt use the '-r' of >> basex because the results were always much faster -is there any kind of >> pipelining teke place there?- and since no other system had such an option I >> went with normal runs. Usually basex wanted 10 executions of a query in a >> row to reach its minimum -then it flunctuates reasonably- similarly for >> exist. Monet was more stable in the sense that it could reach its minimum >> sometimes from the first query. That might be differences in C and Java >> though. Note that I had to compile Monet on my system, no binary worked for >> me.

On 07/10/2010 01:14 PM, Christian Grün wrote: > Thanks for the details. Your test environment sounds reasonable, as > well as the results (mabe except for the xmall.txt-results, which seem > to be somewhat influenced by other factors. Side effects are easy to > see if the results are visualized on a logarithmic scale). If you're > using the -r option in BaseX, the results will always be much faster, > as the same code is run again and again, and numerous side-effects are > this minimized. In fact, the results wouldn't be comparable if the > option had been used; it should only be used for tests inside BaseX. > > Just a hint: in the small.txt file, one result for eXist is missing. > > Looking forward to your final results, > Christian >

> Do you think the best option to visualize the results is in logarithmic > scale anyways? And if yes how do you justify this?

On 07/10/2010 03:15 PM, Christian Grün wrote: >> Do you think the best option to visualize the results is in logarithmic >> scale anyways? And if yes how do you justify this? > Logarithm scales are particularly useful if scalability is tested, > such as in your scenario. You might check out some online information > first, such as e.g.: > > http://en.wikipedia.org/wiki/Logarithmic_scale > > Christian