Help adding "bulk" txt files
Monday, 23 March 2026 · 3 messages · last reply 24 Mar 2026
BD
Bridger Dyson-Smith23 Mar 2026 21:43 #
Hi all -
I'm wondering if you all would share better ways to approach the following problem: I have a directory of files but I only want to put a subset of the files into a BaseX database. After some fumbling around, I have these two approaches, but... what are your thoughts? Can I simplify this somehow?
I struggled initially with filtering the files down - while I know I could have thrown everything in the directory into a db, there are some fairly large binaries and disk space is precious :) - so I couldn't get a way forward through using commands, but maybe I overlooked something.
Thanks for your time!
Best,
Bridger
#1
(: relies on running 'CREATE DATABASE cco' as a first step :)
declare variable $path := "/home/bridger/Downloads/cco/";
for $m in file:list($path, false(), '*.txt')[starts-with(., 'retrospect-')]
let $n := substring-after(substring-before($m, '.txt'), 'cob-')
let $t := element text { file:read-text($path || $m, 'UTF-8', true())
return updating db:add(?, ?, ?)('cco', $t, $n)
#2
declare variable $path := "/home/bridger/Downloads/cco/";
declare variable $filepaths := file:list($path, false(),
'*.txt')[starts-with(., 'retrospect-')];
declare variable $files := (
for $f in $filepaths
let $t := element text { file:read-text($path || $f, 'UTF-8', true()) }
return $t
);
declare variable $names := (
for $f in $filepaths
let $n := substring-after(substring-before($f, '.txt'), 'cob-')
return $n
);
db:create(
'nothertest',
$files,
$names,
{ 'ftindex': true(), 'textindex': true() }
)Y
ykhabins24 Mar 2026 16:26 #
Hello,
You can reduce:
file:list($path, false(), '*.txt')[starts-with(., 'retrospect-')]to the following:
file:list($path, false(), 'retrospect-*.txt')It is up to you to test and compare both approaches: #1 vs. #2.
Regards,
Yitzhak Khabinsky
BD
Bridger Dyson-Smith24 Mar 2026 18:37 #
Great point Yitzhak, thank you. I wasn't thinking about how much I could abuse the glob syntax for the `$pattern`.
Best,
Bridger
Quoted text · Yitzhak Khabinsky · 31 lines
On Tue, Mar 24, 2026 at 11:26 AM Yitzhak Khabinsky via BaseX-Talk <basex-talk at mailman.uni-konstanz.de> wrote: > > Hello, > > > > You can reduce: > > file:list($path, false(), '*.txt')[starts-with(., 'retrospect-')] > > > > to the following: > > file:list($path, false(), 'retrospect-*.txt') > > > > It is up to you to test and compare both approaches: #1 vs. #2. > > > > > > > > Regards, > Yitzhak Khabinsky > >
Written for BaseX 12.2. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery