best way to process a collection
Monday, 24 November 2014 · 3 messages · last reply 25 Nov 2014
P
pw24 Nov 2014 16:33 #
Hello list
I have a large collection of documents that I want to process and was wanting to get peoples' views on the best design patterns for this. I guess I'm looking at how to map-reduce or similar for a FLOWR process, which AFAIK is a single thread operation.
Are there any suitable approaches within a single programme? Could I pre-process the programme to define collection subsequences and then run multiple clients against a basex server?
TIA
Peter
CG
Christian Grün25 Nov 2014 18:40 #
Hi Peter,
Could you tell us what "large" means in your case? Do you need to cope with gigabytes or terabytes of data? Or did you already spot current bottlenecks (such as query time, IO time, bandwidth...) that makes you look for better solutions?
Quoted text · pw · 1 line
> TIA
HTH
Christian
CG
Christian Grün25 Nov 2014 20:33 #
Hi Peter,
Quoted text · Christian Grün · 4 lines
> In this case there are about 23k documents. My approach is given in the attached file. I have a basex client and server running and also an apache tika server. it's all on a 32 bit machine so heap size is restricted. I initially get a JSON file with the URLs to a set of Excel or PDF docs, PUT those to Tika and then PUT those to basex. > > I have had to find a way of chunking (using subsequence) otherwise I get Out of Memory errors. > However, doing a small subsequence seems to take ages. I have had this programme running over 3 days and it didn't complete.
Hm, 23k documents does not sound that much. I know too less about Tika. What part of the code is responsible for the error? Could you provide me with the stack trace of the OOM error (you may need to use -d to get it on screen)?
Christian
Written for BaseX 7.9. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery