Main Page » Mailing List » August 2015

Finding document based on filename

Saturday, 29 August 2015 · 6 messages · last reply 2 Sep 2015
MS
Mansi Sheth29 Aug 2015 00:52 #
Hello, I would be having 100s of databases, with each database having 100 XML documents. I want to devise an algorithm, where given a part of XML file name, i want to know which database(s) contains it, or null if document is not currently present in any database. Based on that, add current document into the database. This is to always maintain latest version of a document in DB, and remove the older version, while adding newer version. So far, only way I could come up with is:
for $db in all-databases:
      open $db
      $fileNames = list $db
            for eachFileName in $fileNames:
                   if $eachFileName.contains(sub-xml filename):
                            add to ret-list-db
return ret-list-db Above algorithm, seems highly inefficient, Is there any indexing, which can be done ? Do you suggest, for each document insert, I should maintain a separate XML document, which lists each file inserted etc. Once, i get hold of above list of db, I would be eventually deleting that file and inserting a latest version of that file(which would have same sub-xml file name). So, constant updating of this external document also seems painful (Map be ?). Also, would it be faster, using XQUERY script files, thru java code, or using Java API for such operations ? How do you all deal with such operations ? - Mansi
MF
Martín Ferrari31 Aug 2015 18:35 #
Hi Mansi, I have a similar situation. I don't think there's a fast way to get documents by only knowing a part of their names. It seems you need to know the exact name. In my case, we might be able to group documents by a common id, so we might create subfolders inside the DB and store/get the contents of the subfolder directly, which is pretty fast. I've also tried indexing, but insertions got really slow (I assume maybe because indexing is not granular, it indexes all values) and we need performance. Oh, I've also tried using starts-with() instead of contains(), but it seems it does not pick up indexes. Martín. Date: Fri, 28 Aug 2015 16:52:37 -0400
Quoted text · mansi.sheth at gmail.com · 13 lines
From: mansi.sheth at gmail.com To: basex-talk at mailman.uni-konstanz.de Subject: [basex-talk] Finding document based on filename Hello, I would be having 100s of databases, with each database having 100 XML documents. I want to devise an algorithm, where given a part of XML file name, i want to know which database(s) contains it, or null if document is not currently present in any database. Based on that, add current document into the database. This is to always maintain latest version of a document in DB, and remove the older version, while adding newer version. So far, only way I could come up with is: for $db in all-databases: open $db $fileNames = list $db for eachFileName in $fileNames: if $eachFileName.contains(sub-xml filename): add to ret-list-db return ret-list-db Above algorithm, seems highly inefficient, Is there any indexing, which can be done ? Do you suggest, for each document insert, I should maintain a separate XML document, which lists each file inserted etc. Once, i get hold of above list of db, I would be eventually deleting that file and inserting a latest version of that file(which would have same sub-xml file name). So, constant updating of this external document also seems painful (Map be ?). Also, would it be faster, using XQUERY script files, thru java code, or using Java API for such operations ? How do you all deal with such operations ? - Mansi
DK
Dirk Kirsten31 Aug 2015 20:20 #
Hello Martin, I would like to add that Christian just implemented selective indexes, so if you want to index in a more granular fashion this should now be possible. See https://github.com/BaseXdb/basex/issues/59 for more details and of course this is not stable software yet, so use with care. But we are happy about feedback, as always. Regarding the initial issue: I do think using supfolders in the database should be the easiest and fastest way. Is there any reason not to use distinct directories instead of naming the file using some pattern? Cheers Dirk
Quoted text · Martín Ferrari · 59 lines
On 08/31/2015 06:35 PM, Martín Ferrari wrote: > Hi Mansi, > I have a similar situation. I don't think there's a fast way to > get documents by only knowing a part of their names. It seems you need > to know the exact name. In my case, we might be able to group > documents by a common id, so we might create subfolders inside the DB > and store/get the contents of the subfolder directly, which is pretty > fast. > I've also tried indexing, but insertions got really slow (I > assume maybe because indexing is not granular, it indexes all > values) and we need performance. > > Oh, I've also tried using starts-with() instead of contains(), > but it seems it does not pick up indexes. > > Martín. > > ------------------------------------------------------------------------ > Date: Fri, 28 Aug 2015 16:52:37 -0400 > From: mansi.sheth at gmail.com > To: basex-talk at mailman.uni-konstanz.de > Subject: [basex-talk] Finding document based on filename > > Hello, > > I would be having 100s of databases, with each database having 100 XML > documents. I want to devise an algorithm, where given a part of XML > file name, i want to know which database(s) contains it, or null if > document is not currently present in any database. Based on that, add > current document into the database. This is to always maintain latest > version of a document in DB, and remove the older version, while > adding newer version. > > So far, only way I could come up with is: > > for $db in all-databases: > open $db > $fileNames = list $db > for eachFileName in $fileNames: > if $eachFileName.contains(sub-xml filename): > add to ret-list-db > > return ret-list-db > > Above algorithm, seems highly inefficient, Is there any indexing, > which can be done ? Do you suggest, for each document insert, I should > maintain a separate XML document, which lists each file inserted etc. > > Once, i get hold of above list of db, I would be eventually deleting > that file and inserting a latest version of that file(which would have > same sub-xml file name). So, constant updating of this external > document also seems painful (Map be ?). > > Also, would it be faster, using XQUERY script files, thru java code, > or using Java API for such operations ? > > How do you all deal with such operations ? > > - Mansi
-- Dirk Kirsten, BaseX GmbH, http://basexgmbh.de |-- Firmensitz: Blarerstrasse 56, 78462 Konstanz |-- Registergericht Freiburg, HRB: 708285, Geschäftsführer: | Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22
EK
Eliot Kimberin reply to Martín Ferrari, 31 Aug 2015, 18:3531 Aug 2015 21:56 #
How about (ignore the bad casing--that's Outlook's autocorrect and I'm too lazy to go back and correct every line):
Let $docs := collection('/mydir')/*
For $doc in $docs
    Return if (matches(document-uri(root($doc)), '^.+somestring$'))
           Then $doc
           Else ()
Cheers, Eliot ---- Eliot Kimber, Owner Contrext, LLC http://contrext.com On 8/31/15, 11:35 AM, "Martín Ferrari" <basex-talk-bounces at mailman.uni-konstanz.de on behalf of
Quoted text · Martín Ferrari · 63 lines
ferrari_martin at hotmail.com> wrote: >Hi Mansi, I have a similar situation. I don't think there's a fast >way to get documents by only knowing a part of their names. It seems you >need to know the exact name. In my case, we might be able to group >documents by a common id, so we might create subfolders inside the DB and >store/get the contents of the subfolder directly, which is pretty fast. > I've also tried indexing, but insertions got really slow (I assume >maybe because indexing is not granular, it indexes all values) and we >need performance. > > Oh, I've also tried using starts-with() instead of contains(), but >it seems it does not pick up indexes. > >Martín. > >________________________________________ >Date: Fri, 28 Aug 2015 16:52:37 -0400 >From: mansi.sheth at gmail.com >To: basex-talk at mailman.uni-konstanz.de >Subject: [basex-talk] Finding document based on filename > >Hello, >I would be having 100s of databases, with each database having 100 XML >documents. I want to devise an algorithm, where given a part of XML file >name, i want to know which database(s) contains it, or null if document >is not currently present in any database. Based on that, add current >document into the database. This is to always maintain latest version of >a document in DB, and remove the older version, while adding newer >version. > >So far, only way I could come up with is: > >for $db in all-databases: > open $db > $fileNames = list $db > for eachFileName in $fileNames: > if $eachFileName.contains(sub-xml filename): > add to ret-list-db > >return ret-list-db > >Above algorithm, seems highly inefficient, Is there any indexing, which >can be done ? Do you suggest, for each document insert, I should maintain >a separate XML document, which lists each file inserted etc. > >Once, i get hold of above list of db, I would be eventually deleting that >file and inserting a latest version of that file(which would have same >sub-xml file name). So, constant updating of this external document also >seems painful (Map be ?). > >Also, would it be faster, using XQUERY script files, thru java code, or >using Java API for such operations ? > >How do you all deal with such operations ? > >- Mansi > > > > > > >
MF
Martín Ferrariin reply to Martín Ferrari, 31 Aug 2015, 18:351 Sep 2015 03:45 #
I forgot one thing, I got much better performance by just calling replace rather than delete and insert, but this is a db with more than one million records. If performance is not important, I believe either way will do. Martín.
Quoted text · ferrari_martin at hotmail.com · 21 lines
From: ferrari_martin at hotmail.com To: mansi.sheth at gmail.com; basex-talk at mailman.uni-konstanz.de Date: Mon, 31 Aug 2015 16:35:33 +0000 Subject: Re: [basex-talk] Finding document based on filename Hi Mansi, I have a similar situation. I don't think there's a fast way to get documents by only knowing a part of their names. It seems you need to know the exact name. In my case, we might be able to group documents by a common id, so we might create subfolders inside the DB and store/get the contents of the subfolder directly, which is pretty fast. I've also tried indexing, but insertions got really slow (I assume maybe because indexing is not granular, it indexes all values) and we need performance. Oh, I've also tried using starts-with() instead of contains(), but it seems it does not pick up indexes. Martín. Date: Fri, 28 Aug 2015 16:52:37 -0400 From: mansi.sheth at gmail.com To: basex-talk at mailman.uni-konstanz.de Subject: [basex-talk] Finding document based on filename Hello, I would be having 100s of databases, with each database having 100 XML documents. I want to devise an algorithm, where given a part of XML file name, i want to know which database(s) contains it, or null if document is not currently present in any database. Based on that, add current document into the database. This is to always maintain latest version of a document in DB, and remove the older version, while adding newer version. So far, only way I could come up with is: for $db in all-databases: open $db $fileNames = list $db for eachFileName in $fileNames: if $eachFileName.contains(sub-xml filename): add to ret-list-db return ret-list-db Above algorithm, seems highly inefficient, Is there any indexing, which can be done ? Do you suggest, for each document insert, I should maintain a separate XML document, which lists each file inserted etc. Once, i get hold of above list of db, I would be eventually deleting that file and inserting a latest version of that file(which would have same sub-xml file name). So, constant updating of this external document also seems painful (Map be ?). Also, would it be faster, using XQUERY script files, thru java code, or using Java API for such operations ? How do you all deal with such operations ? - Mansi
MS
Mansi Sheth2 Sep 2015 01:07 #
Thanks guys for all expert comments. Currently, I am going experimenting performance with just deleting and inserting using Java API. If this process takes a tiny bit longer, i don't really care is what I figured :) If i becomes unacceptable, I will use one of these suggestions. Thanks once again. StringList databases = List.list(context) ; String query = "" ;
for(String database : databases ) {
query = "db:list('" + database + "')" ;
try {

for (String fileName: query(query).split(" ")) {
query = "db:delete('" + database + "','" + fileName + "')" ;
if(fileName.contains(XMLFileName.split("_")[1])) {
query(query) ; logger.info("Deleted " + fileName + " from " + database) ; retVal = true; break; } }
} catch (BaseXException e) {
e.printStackTrace(); } }
Quoted text · Martín Ferrari · 73 lines
On Mon, Aug 31, 2015 at 9:45 PM, Martín Ferrari <ferrari_martin at hotmail.com> wrote: > I forgot one thing, I got much better performance by just calling > replace rather than delete and insert, but this is a db with more than one > million records. If performance is not important, I believe either way will > do. > > Martín. > > ------------------------------ > From: ferrari_martin at hotmail.com > To: mansi.sheth at gmail.com; basex-talk at mailman.uni-konstanz.de > Date: Mon, 31 Aug 2015 16:35:33 +0000 > Subject: Re: [basex-talk] Finding document based on filename > > > Hi Mansi, > I have a similar situation. I don't think there's a fast way to get > documents by only knowing a part of their names. It seems you need to know > the exact name. In my case, we might be able to group documents by a common > id, so we might create subfolders inside the DB and store/get the contents > of the subfolder directly, which is pretty fast. > I've also tried indexing, but insertions got really slow (I assume > maybe because indexing is not granular, it indexes all values) and we > need performance. > > Oh, I've also tried using starts-with() instead of contains(), but it > seems it does not pick up indexes. > > Martín. > > ------------------------------ > Date: Fri, 28 Aug 2015 16:52:37 -0400 > From: mansi.sheth at gmail.com > To: basex-talk at mailman.uni-konstanz.de > Subject: [basex-talk] Finding document based on filename > > Hello, > > I would be having 100s of databases, with each database having 100 XML > documents. I want to devise an algorithm, where given a part of XML file > name, i want to know which database(s) contains it, or null if document is > not currently present in any database. Based on that, add current document > into the database. This is to always maintain latest version of a document > in DB, and remove the older version, while adding newer version. > > So far, only way I could come up with is: > > for $db in all-databases: > open $db > $fileNames = list $db > for eachFileName in $fileNames: > if $eachFileName.contains(sub-xml filename): > add to ret-list-db > > return ret-list-db > > Above algorithm, seems highly inefficient, Is there any indexing, which > can be done ? Do you suggest, for each document insert, I should maintain a > separate XML document, which lists each file inserted etc. > > Once, i get hold of above list of db, I would be eventually deleting that > file and inserting a latest version of that file(which would have same > sub-xml file name). So, constant updating of this external document also > seems painful (Map be ?). > > Also, would it be faster, using XQUERY script files, thru java code, or > using Java API for such operations ? > > How do you all deal with such operations ? > > - Mansi >
-- - Mansi

Written for BaseX 8.2. BaseX may have changed since: please check the current documentation.

« Fastest way to get resources starting with pathSchematron validation »
⚡Generated with XQuery