bulk-load
Monday, 4 July 2011 · 3 messages · last reply 17 Jul 2011
TM
Tomaso Musitelli4 Jul 2011 18:16 #
Hello dear BaseX Team,
I would like to load a big number of small documents in Basex. What I do is to use a little Java program:
Context ctx = new Context();
new CreateDB("basex1").execute(ctx);
new CreateIndex("fulltext").execute(ctx);
System.err.println("\n* Show database information:");
System.err.print(new InfoDB().execute(ctx));
for( ... ) {
new Add(somedata, miname, collection).execute(ctx);
}I dont know if it is correct , efficient ?
Do you have a notion of transaction, like begin/commit ?
( I did not see something like that)
My concern is that it seems not to behave well when
the number of documents goes big:
(each doc is about 2500 bytes)
10000 => 54 seconds ; 5.4 millis / document
20000 => 202 s ; 10 ms
50000 => 862 s ; 17.2 ms
And in fact I want to store more than 1 million!
Actually is there a limit in size ? Is 1 million x 2.5Kb OK?
Thank you very much
Tomaso
CG
Christian Grün4 Jul 2011 18:39 #
Tomaso,
thanks for your mail. Your transaction will be much faster if you specify the input within the CreateDB command (..provided that your input documents are located on disk):
new CreateDB("basex1", "/path/to/docs").execute(ctx);
There is no factual limit in terms of the number of documents, but the document size and number of total XML nodes might represent a limit. Please have a look at http://docs.basex.org/wiki/Statistics for some databases that we've parsed with BaseX.
Christian
Quoted text · Tomaso Musitelli · 36 lines
On Mon, Jul 4, 2011 at 6:16 PM, Tomaso Musitelli <tomasomusi at gmail.com> wrote: > Hello dear BaseX Team, > I would like to load a big number of small documents in Basex. > What I do is to use a little Java program: > Context ctx = new Context(); > new CreateDB("basex1").execute(ctx); > new CreateIndex("fulltext").execute(ctx); > System.err.println("\n* Show database information:"); > System.err.print(new InfoDB().execute(ctx)); > > for( ... ) { > new Add(somedata, miname, collection).execute(ctx); > } > > I dont know if it is correct , efficient ? > Do you have a notion of transaction, like begin/commit ? > ( I did not see something like that) > My concern is that it seems not to behave well when > the number of documents goes big: > (each doc is about 2500 bytes) > 10000 => 54 seconds ; 5.4 millis / document > 20000 => 202 s ; 10 ms > 50000 => 862 s ; 17.2 ms > And in fact I want to store more than 1 million! > Actually is there a limit in size ? Is 1 million x 2.5Kb OK? > Thank you very much > Tomaso > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
CG
Christian Grün17 Jul 2011 00:06 #
Dear Tomaso, dear Michael,
today, I had a closer look into the BaseX routines that are
responsible for adding new documents to the database, and I tweaked the update code of our document index to avoid linear costs for adding single documents. You are invited to check out the latest stable snapshot [1] and give us your valuable feedback.
There are still some bottlenecks:
- the default XML parser takes some time for initialization, which is particularly noticable for small documents. You'll get some
performance boost by switching to the internal parser (Command: set intparse; see [2] for details).
- as each BaseX database command is atomic, the data is flushed to disk after each update to avoid data loss. You may either specify a directory on disk to add multiple files at once, or choose to insert nodes instead of documents, which will give you better performance.
Hope this helps,
Christian
[1] http://files.basex.org/releases/latest/basex-6.7.1-SNAPSHOT.jar [2] http://docs.basex.org/wiki/Parsers
Quoted text · Christian Grün · 20 lines
On Mon, Jul 4, 2011 at 8:22 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Tomaso, > >> From your answer I suppose there is something slower >> when Add() is called many times, and faster if we use CreateDb. >> Can you explain why the times increase? Is it because Add >> updates the index each time? > > Exactly; as the ADD operation is an atomic operation, there is > currently no way to define a batch operation (other than going more > low level, and looking at the Add command [1]). It might be that some > update operations could be delayed though; I've added this as an issue > (feel free to include more details) [2]. > > Christian > > [1] https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/core/cmd/Add.java > [2] https://github.com/BaseXdb/basex/issues/137 >
Written for BaseX 6.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery