best way to process a collection
Hello list
I have a large collection of documents that I want to process and was wanting to get peoples' views on the best design patterns for this. I guess I'm looking at how to map-reduce or similar for a FLOWR process, which AFAIK is a single thread operation.
Are there any suitable approaches within a single programme? Could I pre-process the programme to define collection subsequences and then run multiple clients against a basex server?
TIA
Peter
Hi Peter,
Could you tell us what "large" means in your case? Do you need to cope with gigabytes or terabytes of data? Or did you already spot current bottlenecks (such as query time, IO time, bandwidth...) that makes you look for better solutions?
Quoted text · pw · 1 line
> TIA
HTH Christian
Hi Peter,
Quoted text · Christian Grün · 4 lines
> In this case there are about 23k documents. My approach is given in the attached file. I have a basex client and server running and also an apache tika server. it's all on a 32 bit machine so heap size is restricted. I initially get a JSON file with the URLs to a set of Excel or PDF docs, PUT those to Tika and then PUT those to basex. > > I have had to find a way of chunking (using subsequence) otherwise I get Out of Memory errors. > However, doing a small subsequence seems to take ages. I have had this programme running over 3 days and it didn't complete.
Hm, 23k documents does not sound that much. I know too less about Tika. What part of the code is responsible for the error? Could you provide me with the stack trace of the OOM error (you may need to use -d to get it on screen)?
Christian
Written for BaseX 7.9. BaseX may have changed since: please check the current documentation.