Bulk Loading XML documents with Add(docPath).execute(CONTEXT)
Sunday, 22 August 2010 · 2 messages
P
pw22 Aug 2010 15:22 #
When adding documents from a directory I get things working using a few small documents and
Add(docPath).execute(CONTEXT)
but when I want to add hundreds of documents (average size 1Mb) I get a memory error. I am using Jython and have -J-Xms1200M and
-J-Xmx1200m
should I use an iterator and only add one document at a time? [this doesn't seem to me to be the way to use Add() ]
TIA
Peter
CG
Christian Grün22 Aug 2010 19:11 #
Dear Peter,
thanks for your feedback. Indeed the Add command is supposed to be used for smaller-sized inputs. To minimize the risk of runtime failures, the command caches the parsed input in main-memory before it's inserted in the database. In your scenario, I'd advise you to either…
1. Create a new database and add all documents via CreateDB
2. Adding all documents using your own iterator.
Option 1 shoud be the preferred choice as long as you don't plan to perform regular updates.
Hope this helps,
Christian
___________________________
Christian Grün
University of Konstanz
Department of Computer & Information Science
D-78457 Konstanz, Germany
Tel: +49-7531-88-4449, Fax: +49-7531-88-3577
http://www.inf.uni-konstanz.de/~gruen
Quoted text · pw · 22 lines
On Sun, Aug 22, 2010 at 3:22 PM, <pw at themail.co.uk> wrote: > When adding documents from a directory I get things working using a > few small documents and > > Add(docPath).execute(CONTEXT) > > but when I want to add hundreds of documents (average size 1Mb) I get > a memory error. I am using Jython and have -J-Xms1200M and > -J-Xmx1200m > > should I use an iterator and only add one document at a time? [this > doesn't seem to me to be the way to use Add() ] > > TIA > > Peter > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Written for BaseX 6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery