Importing TSV files with a command file

On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: > I have a command file that configures a database, throwing a bunch of > sources into various paths. > > I have a set of TSV files. I want to do the equivalent of ADD TO, pointing > to a directory of TSV files, so that they are parsed and converted to XML > and placed in the appropriate path. > > How do I do that? > > Jonathan >

On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <bdysonsmith at gmail.com> wrote: > Hi Jonathan, > > for $T in file:list($path-to-your-files) > let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), > 'format': 'direct' }) > return db:add($your-db, $TSV) > > maybe? I'm probably getting something wrong, semantics-wise, w/ > `db:add()`, but maybe it's close to get started? > Best, > Bridger > > On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie at gmail.com> > wrote: > >> I have a command file that configures a database, throwing a bunch of >> sources into various paths. >> >> I have a set of TSV files. I want to do the equivalent of ADD TO, >> pointing to a directory of TSV files, so that they are parsed and converted >> to XML and placed in the appropriate path. >> >> How do I do that? >> >> Jonathan >> >

On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: > > > On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith < > bdysonsmith at gmail.com> wrote: > >> Hi Jonathan, >> >> for $T in file:list($path-to-your-files) >> let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), >> 'format': 'direct' }) >> return db:add($your-db, $TSV) >> >> maybe? I'm probably getting something wrong, semantics-wise, w/ >> `db:add()`, but maybe it's close to get started? >> Best, >> Bridger >> >> On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie at gmail.com> >> wrote: >> >>> I have a command file that configures a database, throwing a bunch of >>> sources into various paths. >>> >>> I have a set of TSV files. I want to do the equivalent of ADD TO, >>> pointing to a directory of TSV files, so that they are parsed and converted >>> to XML and placed in the appropriate path. >>> >>> How do I do that? >>> >>> Jonathan >>> >>

On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: > Thanks! I was getting hung up trying to do this in a command. > > Jonathan > > On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <jonathan.robie at gmail.com> > wrote: > >> >> >> On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith < >> bdysonsmith at gmail.com> wrote: >> >>> Hi Jonathan, >>> >>> for $T in file:list($path-to-your-files) >>> let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), >>> 'format': 'direct' }) >>> return db:add($your-db, $TSV) >>> >>> maybe? I'm probably getting something wrong, semantics-wise, w/ >>> `db:add()`, but maybe it's close to get started? >>> Best, >>> Bridger >>> >>> On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie at gmail.com> >>> wrote: >>> >>>> I have a command file that configures a database, throwing a bunch of >>>> sources into various paths. >>>> >>>> I have a set of TSV files. I want to do the equivalent of ADD TO, >>>> pointing to a directory of TSV files, so that they are parsed and converted >>>> to XML and placed in the appropriate path. >>>> >>>> How do I do that? >>>> >>>> Jonathan >>>> >>>

On Tue, Oct 12, 2021 at 7:54 PM Bridger Dyson-Smith <bdysonsmith at gmail.com> wrote: > > Jonathan - > apologies for misreading your email! I'm not well-versed in the .bxs COMMAND syntax, but maybe this is closer? > I couldn't quite get a handle on using the EXECUTE syntax for this, but I'm under-caffeinated :) > > Another piece of uncertainty: can you pass variables from the .bxs to a script you are RUNning? Maybe Christian can help us both understand some things! > > Best, > Bridger > > test-tsv.bxs > ``` > CREATE DB test-tsv > RUN /home/bridger/test-tsv.xq > LIST test-tsv > ``` > test-tsv.xq > ``` > for $T in file:children("/home/bridger/tsvs/") > let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", "header": true() }) > let $TSV-NAME := file:name($T) > return db:add("test-tsv", $TSV, "/" || $TSV-NAME) > ``` > note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, "/tsv-files/" || $TSV-NAME) > > I've attached my two sample TSV files if that helps. > > > On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: >> >> Thanks! I was getting hung up trying to do this in a command. >> >> Jonathan >> >> On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: >>> >>> >>> >>> On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <bdysonsmith at gmail.com> wrote: >>>> >>>> Hi Jonathan, >>>> >>>> for $T in file:list($path-to-your-files) >>>> let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) >>>> return db:add($your-db, $TSV) >>>> >>>> maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? >>>> Best, >>>> Bridger >>>> >>>> On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: >>>>> >>>>> I have a command file that configures a database, throwing a bunch of sources into various paths. >>>>> >>>>> I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path. >>>>> >>>>> How do I do that? >>>>> >>>>> Jonathan

> Is there a similar incantation for JSON?

> On Tue, Oct 12, 2021 at 6:22 PM Christian Grün <christian.gruen at gmail.com> wrote: >> >> Hi Bridger, hi Jonathan, >> >> When using the graphical user interface, you can learn more about commands by >> >> a) enabling the Info View panel >> b) performing a GUI operation, such as adding a directory with >> specific options, and >> c) checking the contents of the Info View. >> >> Here’s a .bxs solution for adding TSV files: >> >> SET CREATEFILTER *.tsv >> SET PARSER tsv >> SET CSVPARSER header=true,separator=tab >> CREATE DB test >> ADD csv-dir >> >> As usual, CREATE and ADD can be combined whenever appropriate: >> >> CREATE DB test tsv-dir >> >> Hope this helps, >> Christian >> >> >> On Tue, Oct 12, 2021 at 7:54 PM Bridger Dyson-Smith >> <bdysonsmith at gmail.com> wrote: >> > >> > Jonathan - >> > apologies for misreading your email! I'm not well-versed in the .bxs COMMAND syntax, but maybe this is closer? >> > I couldn't quite get a handle on using the EXECUTE syntax for this, but I'm under-caffeinated :) >> > >> > Another piece of uncertainty: can you pass variables from the .bxs to a script you are RUNning? Maybe Christian can help us both understand some things! >> > >> > Best, >> > Bridger >> > >> > test-tsv.bxs >> > ``` >> > CREATE DB test-tsv >> > RUN /home/bridger/test-tsv.xq >> > LIST test-tsv >> > ``` >> > test-tsv.xq >> > ``` >> > for $T in file:children("/home/bridger/tsvs/") >> > let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", "header": true() }) >> > let $TSV-NAME := file:name($T) >> > return db:add("test-tsv", $TSV, "/" || $TSV-NAME) >> > ``` >> > note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, "/tsv-files/" || $TSV-NAME) >> > >> > I've attached my two sample TSV files if that helps. >> > >> > >> > On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: >> >> >> >> Thanks! I was getting hung up trying to do this in a command. >> >> >> >> Jonathan >> >> >> >> On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: >> >>> >> >>> >> >>> >> >>> On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <bdysonsmith at gmail.com> wrote: >> >>>> >> >>>> Hi Jonathan, >> >>>> >> >>>> for $T in file:list($path-to-your-files) >> >>>> let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) >> >>>> return db:add($your-db, $TSV) >> >>>> >> >>>> maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? >> >>>> Best, >> >>>> Bridger >> >>>> >> >>>> On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: >> >>>>> >> >>>>> I have a command file that configures a database, throwing a bunch of sources into various paths. >> >>>>> >> >>>>> I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path. >> >>>>> >> >>>>> How do I do that? >> >>>>> >> >>>>> Jonathan

On Tue, Oct 12, 2021 at 6:22 PM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Bridger, hi Jonathan, > > When using the graphical user interface, you can learn more about commands > by > > a) enabling the Info View panel > b) performing a GUI operation, such as adding a directory with > specific options, and > c) checking the contents of the Info View. > > Here’s a .bxs solution for adding TSV files: > > SET CREATEFILTER *.tsv > SET PARSER tsv > SET CSVPARSER header=true,separator=tab > CREATE DB test > ADD csv-dir > > As usual, CREATE and ADD can be combined whenever appropriate: > > CREATE DB test tsv-dir > > Hope this helps, > Christian > > > On Tue, Oct 12, 2021 at 7:54 PM Bridger Dyson-Smith > <bdysonsmith at gmail.com> wrote: > > > > Jonathan - > > apologies for misreading your email! I'm not well-versed in the .bxs > COMMAND syntax, but maybe this is closer? > > I couldn't quite get a handle on using the EXECUTE syntax for this, but > I'm under-caffeinated :) > > > > Another piece of uncertainty: can you pass variables from the .bxs to a > script you are RUNning? Maybe Christian can help us both understand some > things! > > > > Best, > > Bridger > > > > test-tsv.bxs > > ``` > > CREATE DB test-tsv > > RUN /home/bridger/test-tsv.xq > > LIST test-tsv > > ``` > > test-tsv.xq > > ``` > > for $T in file:children("/home/bridger/tsvs/") > > let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", > "header": true() }) > > let $TSV-NAME := file:name($T) > > return db:add("test-tsv", $TSV, "/" || $TSV-NAME) > > ``` > > note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, > "/tsv-files/" || $TSV-NAME) > > > > I've attached my two sample TSV files if that helps. > > > > > > On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie < > jonathan.robie at gmail.com> wrote: > >> > >> Thanks! I was getting hung up trying to do this in a command. > >> > >> Jonathan > >> > >> On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie < > jonathan.robie at gmail.com> wrote: > >>> > >>> > >>> > >>> On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith < > bdysonsmith at gmail.com> wrote: > >>>> > >>>> Hi Jonathan, > >>>> > >>>> for $T in file:list($path-to-your-files) > >>>> let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), > 'format': 'direct' }) > >>>> return db:add($your-db, $TSV) > >>>> > >>>> maybe? I'm probably getting something wrong, semantics-wise, w/ > `db:add()`, but maybe it's close to get started? > >>>> Best, > >>>> Bridger > >>>> > >>>> On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie < > jonathan.robie at gmail.com> wrote: > >>>>> > >>>>> I have a command file that configures a database, throwing a bunch > of sources into various paths. > >>>>> > >>>>> I have a set of TSV files. I want to do the equivalent of ADD TO, > pointing to a directory of TSV files, so that they are parsed and converted > to XML and placed in the appropriate path. > >>>>> > >>>>> How do I do that? > >>>>> > >>>>> Jonathan >

On Wed, Oct 13, 2021 at 9:35 AM Christian Grün <christian.gruen at gmail.com> wrote: > Is there a similar incantation for JSON?

> On Tue, Oct 12, 2021 at 6:22 PM Christian Grün <christian.gruen at gmail.com> wrote: >> >> Hi Bridger, hi Jonathan, >> >> When using the graphical user interface, you can learn more about commands by >> >> a) enabling the Info View panel >> b) performing a GUI operation, such as adding a directory with >> specific options, and >> c) checking the contents of the Info View. >> >> Here’s a .bxs solution for adding TSV files: >> >> SET CREATEFILTER *.tsv >> SET PARSER tsv >> SET CSVPARSER header=true,separator=tab >> CREATE DB test >> ADD csv-dir >> >> As usual, CREATE and ADD can be combined whenever appropriate: >> >> CREATE DB test tsv-dir >> >> Hope this helps, >> Christian >> >> >> On Tue, Oct 12, 2021 at 7:54 PM Bridger Dyson-Smith >> <bdysonsmith at gmail.com> wrote: >> > >> > Jonathan - >> > apologies for misreading your email! I'm not well-versed in the .bxs COMMAND syntax, but maybe this is closer? >> > I couldn't quite get a handle on using the EXECUTE syntax for this, but I'm under-caffeinated :) >> > >> > Another piece of uncertainty: can you pass variables from the .bxs to a script you are RUNning? Maybe Christian can help us both understand some things! >> > >> > Best, >> > Bridger >> > >> > test-tsv.bxs >> > ``` >> > CREATE DB test-tsv >> > RUN /home/bridger/test-tsv.xq >> > LIST test-tsv >> > ``` >> > test-tsv.xq >> > ``` >> > for $T in file:children("/home/bridger/tsvs/") >> > let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", "header": true() }) >> > let $TSV-NAME := file:name($T) >> > return db:add("test-tsv", $TSV, "/" || $TSV-NAME) >> > ``` >> > note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, "/tsv-files/" || $TSV-NAME) >> > >> > I've attached my two sample TSV files if that helps. >> > >> > >> > On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: >> >> >> >> Thanks! I was getting hung up trying to do this in a command. >> >> >> >> Jonathan >> >> >> >> On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: >> >>> >> >>> >> >>> >> >>> On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith <bdysonsmith at gmail.com> wrote: >> >>>> >> >>>> Hi Jonathan, >> >>>> >> >>>> for $T in file:list($path-to-your-files) >> >>>> let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': true(), 'format': 'direct' }) >> >>>> return db:add($your-db, $TSV) >> >>>> >> >>>> maybe? I'm probably getting something wrong, semantics-wise, w/ `db:add()`, but maybe it's close to get started? >> >>>> Best, >> >>>> Bridger >> >>>> >> >>>> On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie <jonathan.robie at gmail.com> wrote: >> >>>>> >> >>>>> I have a command file that configures a database, throwing a bunch of sources into various paths. >> >>>>> >> >>>>> I have a set of TSV files. I want to do the equivalent of ADD TO, pointing to a directory of TSV files, so that they are parsed and converted to XML and placed in the appropriate path. >> >>>>> >> >>>>> How do I do that? >> >>>>> >> >>>>> Jonathan

On Wed, Oct 13, 2021 at 9:35 AM Christian Grün <christian.gruen at gmail.com> wrote: > > Is there a similar incantation for JSON? > > Absolutely; it’s *.json and JSONPARSER ;) > > > > On Tue, Oct 12, 2021 at 6:22 PM Christian Grün < > christian.gruen at gmail.com> wrote: > >> > >> Hi Bridger, hi Jonathan, > >> > >> When using the graphical user interface, you can learn more about > commands by > >> > >> a) enabling the Info View panel > >> b) performing a GUI operation, such as adding a directory with > >> specific options, and > >> c) checking the contents of the Info View. > >> > >> Here’s a .bxs solution for adding TSV files: > >> > >> SET CREATEFILTER *.tsv > >> SET PARSER tsv > >> SET CSVPARSER header=true,separator=tab > >> CREATE DB test > >> ADD csv-dir > >> > >> As usual, CREATE and ADD can be combined whenever appropriate: > >> > >> CREATE DB test tsv-dir > >> > >> Hope this helps, > >> Christian > >> > >> > >> On Tue, Oct 12, 2021 at 7:54 PM Bridger Dyson-Smith > >> <bdysonsmith at gmail.com> wrote: > >> > > >> > Jonathan - > >> > apologies for misreading your email! I'm not well-versed in the .bxs > COMMAND syntax, but maybe this is closer? > >> > I couldn't quite get a handle on using the EXECUTE syntax for this, > but I'm under-caffeinated :) > >> > > >> > Another piece of uncertainty: can you pass variables from the .bxs to > a script you are RUNning? Maybe Christian can help us both understand some > things! > >> > > >> > Best, > >> > Bridger > >> > > >> > test-tsv.bxs > >> > ``` > >> > CREATE DB test-tsv > >> > RUN /home/bridger/test-tsv.xq > >> > LIST test-tsv > >> > ``` > >> > test-tsv.xq > >> > ``` > >> > for $T in file:children("/home/bridger/tsvs/") > >> > let $TSV := csv:doc($T, map { "format": "direct", "separator": "tab", > "header": true() }) > >> > let $TSV-NAME := file:name($T) > >> > return db:add("test-tsv", $TSV, "/" || $TSV-NAME) > >> > ``` > >> > note: you could specify a db path here; e.g. db:add("test-tsv", $TSV, > "/tsv-files/" || $TSV-NAME) > >> > > >> > I've attached my two sample TSV files if that helps. > >> > > >> > > >> > On Tue, Oct 12, 2021 at 11:45 AM Jonathan Robie < > jonathan.robie at gmail.com> wrote: > >> >> > >> >> Thanks! I was getting hung up trying to do this in a command. > >> >> > >> >> Jonathan > >> >> > >> >> On Tue, Oct 12, 2021 at 11:44 AM Jonathan Robie < > jonathan.robie at gmail.com> wrote: > >> >>> > >> >>> > >> >>> > >> >>> On Tue, Oct 12, 2021 at 10:33 AM Bridger Dyson-Smith < > bdysonsmith at gmail.com> wrote: > >> >>>> > >> >>>> Hi Jonathan, > >> >>>> > >> >>>> for $T in file:list($path-to-your-files) > >> >>>> let $TSV := csv:parse($T, map { 'separator': 'tab', 'header': > true(), 'format': 'direct' }) > >> >>>> return db:add($your-db, $TSV) > >> >>>> > >> >>>> maybe? I'm probably getting something wrong, semantics-wise, w/ > `db:add()`, but maybe it's close to get started? > >> >>>> Best, > >> >>>> Bridger > >> >>>> > >> >>>> On Tue, Oct 12, 2021 at 9:31 AM Jonathan Robie < > jonathan.robie at gmail.com> wrote: > >> >>>>> > >> >>>>> I have a command file that configures a database, throwing a > bunch of sources into various paths. > >> >>>>> > >> >>>>> I have a set of TSV files. I want to do the equivalent of ADD TO, > pointing to a directory of TSV files, so that they are parsed and converted > to XML and placed in the appropriate path. > >> >>>>> > >> >>>>> How do I do that? > >> >>>>> > >> >>>>> Jonathan >