Coding help
Monday, 5 August 2019 · 5 messages · last reply 8 Aug 2019
GK
Greg Kawchuk5 Aug 2019 10:42 #
Hi everyone,
I'm wondering if someone could provide what I think is a brief script for a scientific project to do the following.
The goal is to identify XML documents from a very large collection that would be too big to load into a database all at once.
Here is how I see the functions provided by the code.
1. In the script, the user could enter the path of the target folder (with millions of XML documents).
2. In the script, the user would enter the number of documents to load into a database at a given time (i =. 1,000) depending on memory limitations.
3. The code would then create a temporary database from the first (i) xml files in the target folder.
4. The code would then search the 1000 xml documents in the database for a pre-defined text string.
5. If hits exist for the text string, the code would write those documents to a unique XML file.
6. Clear the database.
7. Read in the next 1000 files (or remaining files in the folder).
8. Return to #4.
There would be no need to append XML files in step 5. The resulting XML files could be concatenated afterwards.
Thank you in advance. If you have any questions, please feel free to email me here.
Greg
***************************************************
Greg Kawchuk BSC, DC, MSc, PhD.
Professor, Faculty of Rehabilitation Medicine
University of Alberta
greg.kawchuk at ualberta.ca
780-492-6891
MH
Martin Honnen5 Aug 2019 12:36 #
Quoted text · Greg Kawchuk · 17 lines
Am 05.08.2019 um 08:41 schrieb Greg Kawchuk: > Hi everyone, > I'm wondering if someone could provide what I think is a brief script > for a scientific project to do the following. > The goal is to identify XML documents from a very large collection > that would be too big to load into a database all at once. > > Here is how I see the functions provided by the code. > 1. In the script, the user could enter the path of the target folder > (with millions of XML documents). > 2. In the script, the user would enter the number of documents to load > into a database at a given time (i =. 1,000) depending on memory > limitations. > 3. The code would then create a temporary database from the first (i) > xml files in the target folder. > 4. The code would then search the 1000 xml documents in the database > for a pre-defined text string.
What kind of search is that exactly? Does it depend on any database related features at all or can't you just use BaseX as a standalone XQuery processor?
Quoted text · Greg Kawchuk · 2 lines
> 5. If hits exist for the text string, the code would write those > documents to a unique XML file.
What kind of structure would that unique file have, simply
<root>{collection('foo')[1 to 1000][condition]}</root>Quoted text · Greg Kawchuk · 8 lines
> 6. Clear the database. > 7. Read in the next 1000 files (or remaining files in the folder). > 8. Return to #4. > > There would be no need to append XML files in step 5. The resulting > XML files could be concatenated afterwards. > Thank you in advance. If you have any questions, please feel free to > email me here.
RG
Hi Greg,
So, to be clear and succinct, the goal is to create a single XML file containing all XML files that have a predefined text string match in them, yes?
If so, I'm wondering if creating any database is necessary. A single pass through all the files, searching for the text string, and appending matched files as you go seems sufficient.
R
Quoted text · Greg Kawchuk · 36 lines
On Mon, Aug 5, 2019, 08:42 Greg Kawchuk <greg.kawchuk at ualberta.ca> wrote: > Hi everyone, > I'm wondering if someone could provide what I think is a brief script for > a scientific project to do the following. > The goal is to identify XML documents from a very large collection that > would be too big to load into a database all at once. > > Here is how I see the functions provided by the code. > 1. In the script, the user could enter the path of the target folder (with > millions of XML documents). > 2. In the script, the user would enter the number of documents to load > into a database at a given time (i =. 1,000) depending on memory > limitations. > 3. The code would then create a temporary database from the first (i) xml > files in the target folder. > 4. The code would then search the 1000 xml documents in the database for a > pre-defined text string. > 5. If hits exist for the text string, the code would write those documents > to a unique XML file. > 6. Clear the database. > 7. Read in the next 1000 files (or remaining files in the folder). > 8. Return to #4. > > There would be no need to append XML files in step 5. The resulting XML > files could be concatenated afterwards. > Thank you in advance. If you have any questions, please feel free to email > me here. > Greg > > *************************************************** > Greg Kawchuk BSC, DC, MSc, PhD. > Professor, Faculty of Rehabilitation Medicine > University of Alberta > greg.kawchuk at ualberta.ca > 780-492-6891 >
MS
Creating a bunch of temporary databases that you’re going to delete doesn’t sound like the most efficient way to process this data. But it hard to tell what alternative to recommend without more info about what your desired end result is.
Is this something that you’re going to do once, or do repeatedly for different strings ?
Taking your description literally, I would use ‘grep -l ‘ to generate a list of files with the specific string, and either feed that list into ‘cat’ or else use it to build a database of a subset of files for further investigation.
There are also other ways to filter the data through a stream to select a subset if that is what you want to do.
But if you’re going to do this repeatedly for different subsets, then it might make more sense to try to get everything parsed and indexed into the database once. If it really is too large for a single database after adjusting the java memory parameters in the basex scripts, you could try sharding the data into several databases, repeat the search on each collection, and concatenate the results.
Quoted text · Greg Kawchuk · 26 lines
> On Aug 5, 2019, at 2:41 AM, Greg Kawchuk <greg.kawchuk at ualberta.ca> wrote: > > Hi everyone, > I'm wondering if someone could provide what I think is a brief script for a scientific project to do the following. > The goal is to identify XML documents from a very large collection that would be too big to load into a database all at once. > > Here is how I see the functions provided by the code. > 1. In the script, the user could enter the path of the target folder (with millions of XML documents). > 2. In the script, the user would enter the number of documents to load into a database at a given time (i =. 1,000) depending on memory limitations. > 3. The code would then create a temporary database from the first (i) xml files in the target folder. > 4. The code would then search the 1000 xml documents in the database for a pre-defined text string. > 5. If hits exist for the text string, the code would write those documents to a unique XML file. > 6. Clear the database. > 7. Read in the next 1000 files (or remaining files in the folder). > 8. Return to #4. > > There would be no need to append XML files in step 5. The resulting XML files could be concatenated afterwards. > Thank you in advance. If you have any questions, please feel free to email me here. > Greg > > *************************************************** > Greg Kawchuk BSC, DC, MSc, PhD. > Professor, Faculty of Rehabilitation Medicine > University of Alberta > greg.kawchuk at ualberta.ca <mailto:greg.kawchuk at ualberta.ca> > 780-492-6891
M
Marc8 Aug 2019 09:08 #
For this type of project, I regard into the solution with a
file:list and a stream use of XSLT, or a use of the saxon:discard-document or the use of collection as in https://ajwelch.blogspot.com/2006/11/using-collection-and-saxondiscard.html
If the query to select file doesn'y need Xquery but XPath, it can be done like that. If the collection is to big you can do more than one pass with the file selector (files started by a, next b, etc....)
MarcQuoted text · Steven Dennis (sdm7g) · 66 lines
Le 06/08/2019 à 18:12, Majewski, Steven Dennis (sdm7g) a écrit : > > Creating a bunch of temporary databases that you’re going to delete > doesn’t sound like the most efficient way to process this data. > But it hard to tell what alternative to recommend without more info > about what your desired end result is. > > Is this something that you’re going to do once, or do repeatedly for > different strings ? > > Taking your description literally, I would use ‘grep -l ‘ to generate a > list of files with the specific string, and either feed that list into > ‘cat’ or else use it to build a database of a subset of files for > further investigation. > > There are also other ways to filter the data through a stream to select > a subset if that is what you want to do. > > But if you’re going to do this repeatedly for different subsets, then it > might make more sense to try to get everything parsed and indexed into > the database once. If it really is too large for a single database after > adjusting the java memory parameters in the basex scripts, you could try > sharding the data into several databases, repeat the search on each > collection, and concatenate the results. > > > > > > On Aug 5, 2019, at 2:41 AM, Greg Kawchuk <greg.kawchuk at ualberta.ca > > <mailto:greg.kawchuk at ualberta.ca>> wrote: > > > > Hi everyone, > > I'm wondering if someone could provide what I think is a brief script > > for a scientific project to do the following. > > The goal is to identify XML documents from a very large collection > > that would be too big to load into a database all at once. > > > > Here is how I see the functions provided by the code. > > 1. In the script, the user could enter the path of the target folder > > (with millions of XML documents). > > 2. In the script, the user would enter the number of documents to load > > into a database at a given time (i =. 1,000) depending on memory > > limitations. > > 3. The code would then create a temporary database from the first (i) > > xml files in the target folder. > > 4. The code would then search the 1000 xml documents in the database > > for a pre-defined text string. > > 5. If hits exist for the text string, the code would write those > > documents to a unique XML file. > > 6. Clear the database. > > 7. Read in the next 1000 files (or remaining files in the folder). > > 8. Return to #4. > > > > There would be no need to append XML files in step 5. The resulting > > XML files could be concatenated afterwards. > > Thank you in advance. If you have any questions, please feel free to > > email me here. > > Greg > > > > *************************************************** > > Greg Kawchuk BSC, DC, MSc, PhD. > > Professor, Faculty of Rehabilitation Medicine > > University of Alberta > > greg.kawchuk at ualberta.ca <mailto:greg.kawchuk at ualberta.ca> > > 780-492-6891 >
Written for BaseX 9.2. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery