bulk-load
Hello dear BaseX Team,
I would like to load a big number of small documents in Basex. What I do is to use a little Java program:
Context ctx = new Context();
new CreateDB("basex1").execute(ctx);
new CreateIndex("fulltext").execute(ctx);
System.err.println("\n* Show database information:");
System.err.print(new InfoDB().execute(ctx));
for( ... ) {
new Add(somedata, miname, collection).execute(ctx);
}I dont know if it is correct , efficient ? Do you have a notion of transaction, like begin/commit ? ( I did not see something like that)
My concern is that it seems not to behave well when the number of documents goes big: (each doc is about 2500 bytes)
10000 => 54 seconds ; 5.4 millis / document 20000 => 202 s ; 10 ms 50000 => 862 s ; 17.2 ms
And in fact I want to store more than 1 million!
Actually is there a limit in size ? Is 1 million x 2.5Kb OK?
Thank you very much Tomaso
Tomaso,
thanks for your mail. Your transaction will be much faster if you specify the input within the CreateDB command (..provided that your input documents are located on disk):
new CreateDB("basex1", "/path/to/docs").execute(ctx);
There is no factual limit in terms of the number of documents, but the document size and number of total XML nodes might represent a limit. Please have a look at http://docs.basex.org/wiki/Statistics for some databases that we've parsed with BaseX.
Christian
Quoted text · Tomaso Musitelli · 36 lines
On Mon, Jul 4, 2011 at 6:16 PM, Tomaso Musitelli <tomasomusi at gmail.com> wrote: > Hello dear BaseX Team, > I would like to load a big number of small documents in Basex. > What I do is to use a little Java program: > Context ctx = new Context(); > new CreateDB("basex1").execute(ctx); > new CreateIndex("fulltext").execute(ctx); > System.err.println("\n* Show database information:"); > System.err.print(new InfoDB().execute(ctx)); > > for( ... ) { > new Add(somedata, miname, collection).execute(ctx); > } > > I dont know if it is correct , efficient ? > Do you have a notion of transaction, like begin/commit ? > ( I did not see something like that) > My concern is that it seems not to behave well when > the number of documents goes big: > (each doc is about 2500 bytes) > 10000 => 54 seconds ; 5.4 millis / document > 20000 => 202 s ; 10 ms > 50000 => 862 s ; 17.2 ms > And in fact I want to store more than 1 million! > Actually is there a limit in size ? Is 1 million x 2.5Kb OK? > Thank you very much > Tomaso > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
Dear Tomaso, dear Michael,
today, I had a closer look into the BaseX routines that are responsible for adding new documents to the database, and I tweaked the update code of our document index to avoid linear costs for adding single documents. You are invited to check out the latest stable snapshot [1] and give us your valuable feedback.
There are still some bottlenecks: - the default XML parser takes some time for initialization, which is particularly noticable for small documents. You'll get some performance boost by switching to the internal parser (Command: set intparse; see [2] for details). - as each BaseX database command is atomic, the data is flushed to disk after each update to avoid data loss. You may either specify a directory on disk to add multiple files at once, or choose to insert nodes instead of documents, which will give you better performance.
Hope this helps, Christian
[1] http://files.basex.org/releases/latest/basex-6.7.1-SNAPSHOT.jar [2] http://docs.basex.org/wiki/Parsers
Quoted text · Christian Grün · 20 lines
On Mon, Jul 4, 2011 at 8:22 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Tomaso, > >> From your answer I suppose there is something slower >> when Add() is called many times, and faster if we use CreateDb. >> Can you explain why the times increase? Is it because Add >> updates the index each time? > > Exactly; as the ADD operation is an atomic operation, there is > currently no way to define a batch operation (other than going more > low level, and looking at the Add command [1]). It might be that some > update operations could be delayed though; I've added this as an issue > (feel free to include more details) [2]. > > Christian > > [1] https://github.com/BaseXdb/basex/blob/master/src/main/java/org/basex/core/cmd/Add.java > [2] https://github.com/BaseXdb/basex/issues/137 >
Written for BaseX 6.7. BaseX may have changed since: please check the current documentation.