Main Page » Mailing List » November 2011

Whatever happened to DeepFS

Monday, 14 November 2011 · 9 messages · last reply 13 Dec 2011
AB
Andy Bunce14 Nov 2011 13:28 #
Hi, I was looking for this feature in version 7. https://mailman.uni-konstanz.de/pipermail/basex-talk/2011-January/000994.html says it was temporarily removed from the GUI but create fs [name] [path] gives an error in 7.0.1. Is this feature gone for good now? /Andy
AH
Alexander Holupirek14 Nov 2011 20:17 #
Hi Andy,
Quoted text · Andy Bunce · 9 lines
On 14.11.2011, at 13:28, Andy Bunce wrote: > Hi, > > I was looking for this feature in version 7. > https://mailman.uni-konstanz.de/pipermail/basex-talk/2011-January/000994.html > says it was temporarily removed from the GUI but > create fs [name] [path] > > gives an error in 7.0.1. Is this feature gone for good now?
depends on what you want to achieve. The 'create fs' command walked a given file hierarchy and produced a FSML database (Filesystem Markup Language). Whenever a 'known file type' has been encountered, a file-type specific extractor took care of it and added information about that file (ID3 tags for mp3 files, full text for pdf files etc.) to the FSML mapping. This provided the basis to later on 'query' the filesystem data. That code has been separated from BaseX, in order to keep the core clean. Moreover, the need for some external libraries to extract metadata has been against our wish to keep BaseX as independent as possible. If you just wish to produce a FSML mapping, XQuery using EXPath File Module [1] functionality can pretty much do the job:
declare namespace fs = "http://basex.org/fs";

declare function fs:parse($path as xs:string) as element() {
 let $name := replace($path, ".*[\\/]", "")
 return
   if(file:is-directory($path))
   then <dir name="{ $name }">{
     for $f in file:list($path)
     return fs:parse($f)
   }</dir>
   else <file name="{ $name }"
            size="{ file:size($path) }" />
};
If you also want to have the extractor functionality ... we thought about packaging [2] it for BaseX and make it available as XQuery functions. Just give us a hint and we will get going. Thanks, Alex [1] http://docs.basex.org/wiki/File_Functions [2] http://docs.basex.org/wiki/Packaging
J
Johannes.Lichtenberger14 Nov 2011 21:03 #
Quoted text · Alexander Holupirek · 2 lines
On 11/14/2011 08:17 PM, Alexander Holupirek wrote: > If you also want to have the extractor functionality ... we thought
about packaging [2] it for BaseX and make it available as XQuery functions. Just give us a hint and we will get going. Hi Alex, BTW: I would maybe provide a switch or something to disable the traversal of hidden files (for instance .svn folders :-)). For my own requirenments it seems to be sufficient to ignore hidden files at all. Hopefully my little changes are sufficient ;-) dents = os.listdir(dpath) if depth != 0: if not dents: visit_empty_directory(dpath, depth) return if not dents[0].startswith('.'): visit_enter_directory(dpath, depth) for file_ in dents: if not file_.startswith('.'): path = os.path.join(dpath, file_) if os.path.islink(path): visit_link(path, depth + 1) elif os.path.isdir(path): descend(path, depth + 1, xdcr_map) else: visit_file(path, depth + 1, xdcr_map) if depth != 0: if not dents[0].startswith('.'): visit_leave_directory(depth) kind regards, Johannes
JM
John D. Mitchellin reply to Alexander Holupirek, 14 Nov 2011, 20:1714 Nov 2011 21:22 #
Quoted text · Alexander Holupirek · 1 line
On Nov 14, 2011, at 11:17 , Alexander Holupirek wrote:
[...]
Quoted text · Alexander Holupirek · 1 line
> If you also want to have the extractor functionality ... we thought about packaging [2] it for BaseX and make it available as XQuery functions. Just give us a hint and we will get going.
++ Cheers, John
AB
Andy Bunce14 Nov 2011 21:48 #
It is the metadata extraction part that is non trivial. So packaging the libraries and calls for that sounds like a great way to go. /Andy On Mon, Nov 14, 2011 at 7:22 PM, John D. Mitchell <jdmitchell at gmail.com>wrote:
Quoted text · Alexander Holupirek · 12 lines
> On Nov 14, 2011, at 11:17 , Alexander Holupirek wrote: > [...] > > If you also want to have the extractor functionality ... we thought > about packaging [2] it for BaseX and make it available as XQuery functions. > Just give us a hint and we will get going. > > ++ > > Cheers, > John > >
KC
Kevin S. Clarkein reply to Johannes.Lichtenberger, 14 Nov 2011, 21:0314 Nov 2011 23:13 #
Haven't really been following the conversation so if this is already there ignore me, but... ... or, a regexp pattern of files/directories to ignore/include. Kevin
Quoted text · Johannes.Lichtenberger · 43 lines
On Mon, Nov 14, 2011 at 12:03 PM, Johannes.Lichtenberger <Johannes.Lichtenberger at uni-konstanz.de> wrote: > On 11/14/2011 08:17 PM, Alexander Holupirek wrote: >> If you also want to have the extractor functionality ... we thought > about packaging [2] it for BaseX and make it available as XQuery > functions. Just give us a hint and we will get going. > > Hi Alex, > > BTW: I would maybe provide a switch or something to disable the > traversal of hidden files (for instance .svn folders :-)). For my own > requirenments it seems to be sufficient to ignore hidden files at all. > Hopefully my little changes are sufficient ;-) > > dents = os.listdir(dpath) > if depth != 0: > if not dents: > visit_empty_directory(dpath, depth) > return > if not dents[0].startswith('.'): > visit_enter_directory(dpath, depth) > for file_ in dents: > if not file_.startswith('.'): > path = os.path.join(dpath, file_) > if os.path.islink(path): > visit_link(path, depth + 1) > elif os.path.isdir(path): > descend(path, depth + 1, xdcr_map) > else: > visit_file(path, depth + 1, xdcr_map) > if depth != 0: > if not dents[0].startswith('.'): > visit_leave_directory(depth) > > kind regards, > Johannes > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
AH
Alexander Holupirekin reply to Andy Bunce, 14 Nov 2011, 21:4815 Nov 2011 15:48 #
Quoted text · Andy Bunce · 15 lines
On 14.11.2011, at 21:48, Andy Bunce wrote: > It is the metadata extraction part that is non trivial. > So packaging the libraries and calls for that sounds like a great way to go. > > /Andy > > On Mon, Nov 14, 2011 at 7:22 PM, John D. Mitchell <jdmitchell at gmail.com> wrote: > On Nov 14, 2011, at 11:17 , Alexander Holupirek wrote: > [...] > > If you also want to have the extractor functionality ... we thought about packaging [2] it for BaseX and make it available as XQuery functions. Just give us a hint and we will get going. > > ++ > > Cheers, > John
Thanks for your feedback. We decided to go for the packaging approach and to provide an EXPath package [0] in order to produce a FSML database of a given file hierarchy. It would be interesting to hear what kind of file types are relevant for you. The idea is to have transducer code [1] that, for example, extracts ID3 information for audio files:
   <file name="LockerBleiben.mp3" suffix="mp3" st_mode="0100644" st_size="4585915" st_mtime="1320945388000" st_uid="1000" st_gid="1000" st_nlink="1" bsid="70622d84-f4f7-4b90-95e2-9e1821e8d283">
      <folder name="ID3v2">
        <fact name="Title">Locker Bleiben</fact>
        <fact name="Artist">Die Fantastischen Vier</fact>
        <fact name="Composer">Andreas Rieke/Michael DJ Beck/Thomas Dürr/Michael B. Schmidt</fact>
        <fact name="Album">Lauschgift</fact>
        <fact name="Track">15/20</fact>
        <fact name="PartOfSet">1/1</fact>
        <fact name="Year">1995</fact>
        <fact name="Genre">Hip Hop/Rap</fact>
        <fact name="Compilation">1</fact>
        <fact name="Comment">(iTunPGAP) 0</fact>
        <fact name="EncodedBy">iTunes 8.0.2</fact>
      </folder>
      <folder name="Cover">
	...
      </folder>
    </file>
Currently I think about using exiftool[1] by Phil Harvey to include metadata about numerous multi-media files. Extract full text and publisher metadata from PDF files, etc. If you have something special or want to comment on this, I'm all ears. Thanks, Alex [0] EXPath Packaging: http://docs.basex.org/wiki/Packaging [1] Transducer coined by Gifford et.al. Semantic File System: http://dl.acm.org/citation.cfm?id=121138 [1] http://www.sno.phy.queensu.ca/~phil/exiftool/
AB
Andy Bunce16 Nov 2011 00:34 #
I am mainly interested in image, (usually jpg ), and audio (usually mp3) I dont know much about Exiftool but it seems to be a Perl library. Nothing wrong with that :-), but sounds an heavy choice to wrap in a java package? xmlcalabash has cx:metadata-extractor extension step; for images a thin shell around Drew Noakes' library of the same name<http://www.drewnoakes.com/code/exif/> . <http://xmlcalabash.com/download/>Mentioned athttp://xmlcalabash.com/download/ Mp3 is more tricky, but https://github.com/mpatric/mp3agic looks like a possible candidate to me. /Andy
Quoted text · Alexander Holupirek · 69 lines
On Tue, Nov 15, 2011 at 2:48 PM, Alexander Holupirek < alexander.holupirek at uni-konstanz.de> wrote: > > On 14.11.2011, at 21:48, Andy Bunce wrote: > > > It is the metadata extraction part that is non trivial. > > So packaging the libraries and calls for that sounds like a great way to > go. > > > > /Andy > > > > On Mon, Nov 14, 2011 at 7:22 PM, John D. Mitchell <jdmitchell at gmail.com> > wrote: > > On Nov 14, 2011, at 11:17 , Alexander Holupirek wrote: > > [...] > > > If you also want to have the extractor functionality ... we thought > about packaging [2] it for BaseX and make it available as XQuery functions. > Just give us a hint and we will get going. > > > > ++ > > > > Cheers, > > John > > Thanks for your feedback. We decided to go for the packaging approach and > to provide an EXPath package [0] in order to produce a FSML database of a > given file hierarchy. > > It would be interesting to hear what kind of file types are relevant for > you. > The idea is to have transducer code [1] that, for example, extracts ID3 > information for audio files: > > <file name="LockerBleiben.mp3" suffix="mp3" st_mode="0100644" > st_size="4585915" st_mtime="1320945388000" st_uid="1000" st_gid="1000" > st_nlink="1" bsid="70622d84-f4f7-4b90-95e2-9e1821e8d283"> > <folder name="ID3v2"> > <fact name="Title">Locker Bleiben</fact> > <fact name="Artist">Die Fantastischen Vier</fact> > <fact name="Composer">Andreas Rieke/Michael DJ Beck/Thomas > Dürr/Michael B. Schmidt</fact> > <fact name="Album">Lauschgift</fact> > <fact name="Track">15/20</fact> > <fact name="PartOfSet">1/1</fact> > <fact name="Year">1995</fact> > <fact name="Genre">Hip Hop/Rap</fact> > <fact name="Compilation">1</fact> > <fact name="Comment">(iTunPGAP) 0</fact> > <fact name="EncodedBy">iTunes 8.0.2</fact> > </folder> > <folder name="Cover"> > ... > </folder> > </file> > > Currently I think about using exiftool[1] by Phil Harvey to include > metadata about numerous multi-media files. > Extract full text and publisher metadata from PDF files, etc. > > If you have something special or want to comment on this, I'm all ears. > > Thanks, > Alex > > > [0] EXPath Packaging: http://docs.basex.org/wiki/Packaging > [1] Transducer coined by Gifford et.al. Semantic File System: > http://dl.acm.org/citation.cfm?id=121138 > [1] http://www.sno.phy.queensu.ca/~phil/exiftool/
AB
Andy Bunce13 Dec 2011 13:24 #
Or apache tika covers a lot of ground... http://tika.apache.org/1.0/formats.html#Supported_Document_Formats
Quoted text · Andy Bunce · 88 lines
On Tue, Nov 15, 2011 at 11:34 PM, Andy Bunce <bunce.andy at gmail.com> wrote: > I am mainly interested in image, (usually jpg ), and audio (usually mp3) > I dont know much about Exiftool but it seems to be a Perl library. Nothing > wrong with that :-), but sounds an heavy choice to wrap in a java package? > > xmlcalabash has cx:metadata-extractor extension step; for images a thin > shell around Drew Noakes' library of the same name<http://www.drewnoakes.com/code/exif/> > . <http://xmlcalabash.com/download/>Mentioned athttp://xmlcalabash.com/download/ > > Mp3 is more tricky, but https://github.com/mpatric/mp3agic looks like a > possible candidate to me. > > /Andy > > > On Tue, Nov 15, 2011 at 2:48 PM, Alexander Holupirek < > alexander.holupirek at uni-konstanz.de> wrote: > >> >> On 14.11.2011, at 21:48, Andy Bunce wrote: >> >> > It is the metadata extraction part that is non trivial. >> > So packaging the libraries and calls for that sounds like a great way >> to go. >> > >> > /Andy >> > >> > On Mon, Nov 14, 2011 at 7:22 PM, John D. Mitchell <jdmitchell at gmail.com> >> wrote: >> > On Nov 14, 2011, at 11:17 , Alexander Holupirek wrote: >> > [...] >> > > If you also want to have the extractor functionality ... we thought >> about packaging [2] it for BaseX and make it available as XQuery functions. >> Just give us a hint and we will get going. >> > >> > ++ >> > >> > Cheers, >> > John >> >> Thanks for your feedback. We decided to go for the packaging approach >> and to provide an EXPath package [0] in order to produce a FSML database of >> a given file hierarchy. >> >> It would be interesting to hear what kind of file types are relevant for >> you. >> The idea is to have transducer code [1] that, for example, extracts ID3 >> information for audio files: >> >> <file name="LockerBleiben.mp3" suffix="mp3" st_mode="0100644" >> st_size="4585915" st_mtime="1320945388000" st_uid="1000" st_gid="1000" >> st_nlink="1" bsid="70622d84-f4f7-4b90-95e2-9e1821e8d283"> >> <folder name="ID3v2"> >> <fact name="Title">Locker Bleiben</fact> >> <fact name="Artist">Die Fantastischen Vier</fact> >> <fact name="Composer">Andreas Rieke/Michael DJ Beck/Thomas >> Dürr/Michael B. Schmidt</fact> >> <fact name="Album">Lauschgift</fact> >> <fact name="Track">15/20</fact> >> <fact name="PartOfSet">1/1</fact> >> <fact name="Year">1995</fact> >> <fact name="Genre">Hip Hop/Rap</fact> >> <fact name="Compilation">1</fact> >> <fact name="Comment">(iTunPGAP) 0</fact> >> <fact name="EncodedBy">iTunes 8.0.2</fact> >> </folder> >> <folder name="Cover"> >> ... >> </folder> >> </file> >> >> Currently I think about using exiftool[1] by Phil Harvey to include >> metadata about numerous multi-media files. >> Extract full text and publisher metadata from PDF files, etc. >> >> If you have something special or want to comment on this, I'm all ears. >> >> Thanks, >> Alex >> >> >> [0] EXPath Packaging: http://docs.basex.org/wiki/Packaging >> [1] Transducer coined by Gifford et.al. Semantic File System: >> http://dl.acm.org/citation.cfm?id=121138 >> [1] http://www.sno.phy.queensu.ca/~phil/exiftool/ > > >

Written for BaseX 7.0. BaseX may have changed since: please check the current documentation.

« Revised command-line APIEmbedded API »
⚡Generated with XQuery