combining queries from mutiple sources

On Sun, Feb 13, 2022 at 11:26 PM Eric Levy <contact at ericlevy.name> wrote: > > I am completely new to BaseX, and trying to conceive of a way to use it > to process a compound query that combines an external query with a > static one. > > The core idea is a very simple application, expressed as an XQuery > file, that results in the following sequence of actions: > > 1. Extraction of data from a database, by some query. > 2. Modification of the results of (1), by some query. > 3. Persistence of the results from (2), back into the database from > which the data was originally extracted. > > The modifications given in (2) would be expressed external to the > application, supplied as input per invocation, in a separate file or in > a command-line parameter. > > Thus, the overall operation would entail static components, the fixed > parts of the application, represented by (1) and (3), and dynamic > components, passed by the invocation, represented by (2). > > I realize such a processing pipeline may be achieved through queries > supplied in a full application accessing the database by one of the > language-specific APIs, but I am trying to consider an approach that > requires use only of XQuery statements, without involving a separate > programming language or formal build process. > > Does the design of BaseX accommodate an application having the > structure as described? > >

From: BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on behalf of Jonathan Robie <jonathan.robie at gmail.com> Date: Monday, February 14, 2022 at 8:17 AM To: Christian Grün <christian.gruen at gmail.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] combining queries from mutiple sources [External Email] Also, you can have the best of both worlds, saving results in a pipeline if it's helpful for debugging, just running the updates in place once you know it all works well. This is particularly helpful if your data is complex and your datasets are largish. I am currently merging various sources that were not originally meant to work together, creating a common reference system for fairly complex data. I find it helpful to write a bunch of simple updates that can be executed sequentially, running them with a .bxs file. If I want to see the output after any of these updates, I can do so with an OUTPUT statement. Most of these stages are only useful for debugging purposes, so once a pipeline is working well, I get rid of the OUTPUT statement. If your data is large, it can also be helpful to have two different statements for importing the data, one that imports everything, another that imports only a small test subset that runs quickly. Comment one of them out and use the other. For rapid development, use the small, quick subset, then run it on the whole dataset once it works. Jonathan On Sun, Feb 13, 2022 at 6:19 PM Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> wrote: Hi Eric, That’s perfectly feasible. You can bind the dynamic query string to an external variable and invoke it on extracted data via xquery:eval. Here’s one way to do it: (: db.xml :) <path><to><source><data/></source></to></path> (: query.xq :) declare variable $query external; for $data in db:open('db')/path/to/source/data let $updated := xquery:eval($query, map { '': $data }) return replace node $data with $updated The query could e.g. be invoked via basex -b query="<new>{ . }</new>" query.xq Hope this helps, Christian On Sun, Feb 13, 2022 at 11:26 PM Eric Levy <contact at ericlevy.name<mailto:contact at ericlevy.name>> wrote: > > I am completely new to BaseX, and trying to conceive of a way to use it > to process a compound query that combines an external query with a > static one. > > The core idea is a very simple application, expressed as an XQuery > file, that results in the following sequence of actions: > > 1. Extraction of data from a database, by some query. > 2. Modification of the results of (1), by some query. > 3. Persistence of the results from (2), back into the database from > which the data was originally extracted. > > The modifications given in (2) would be expressed external to the > application, supplied as input per invocation, in a separate file or in > a command-line parameter. > > Thus, the overall operation would entail static components, the fixed > parts of the application, represented by (1) and (3), and dynamic > components, passed by the invocation, represented by (2). > > I realize such a processing pipeline may be achieved through queries > supplied in a full application accessing the database by one of the > language-specific APIs, but I am trying to consider an approach that > requires use only of XQuery statements, without involving a separate > programming language or formal build process. > > Does the design of BaseX accommodate an application having the > structure as described? > >

On Sun, Feb 13, 2022 at 6:19 PM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Eric, > > That’s perfectly feasible. You can bind the dynamic query string to an > external variable and invoke it on extracted data via xquery:eval. > Here’s one way to do it: > > (: db.xml :) > <path><to><source><data/></source></to></path> > > (: query.xq :) > declare variable $query external; > > for $data in db:open('db')/path/to/source/data > let $updated := xquery:eval($query, map { '': $data }) > return replace node $data with $updated > > The query could e.g. be invoked via > > basex -b query="<new>{ . }</new>" query.xq > > Hope this helps, > Christian > > > > On Sun, Feb 13, 2022 at 11:26 PM Eric Levy <contact at ericlevy.name> wrote: > > > > I am completely new to BaseX, and trying to conceive of a way to use it > > to process a compound query that combines an external query with a > > static one. > > > > The core idea is a very simple application, expressed as an XQuery > > file, that results in the following sequence of actions: > > > > 1. Extraction of data from a database, by some query. > > 2. Modification of the results of (1), by some query. > > 3. Persistence of the results from (2), back into the database from > > which the data was originally extracted. > > > > The modifications given in (2) would be expressed external to the > > application, supplied as input per invocation, in a separate file or in > > a command-line parameter. > > > > Thus, the overall operation would entail static components, the fixed > > parts of the application, represented by (1) and (3), and dynamic > > components, passed by the invocation, represented by (2). > > > > I realize such a processing pipeline may be achieved through queries > > supplied in a full application accessing the database by one of the > > language-specific APIs, but I am trying to consider an approach that > > requires use only of XQuery statements, without involving a separate > > programming language or formal build process. > > > > Does the design of BaseX accommodate an application having the > > structure as described? > > > > >

On Mon, Feb 14, 2022 at 9:49 AM Eliot Kimber <eliot.kimber at servicenow.com> wrote: > I’m doing more less the same thing. I create various reports and indexes > over content stored in a primary database and use small XQuery scripts to > perform actions that must be individual transactions (db:create, db:drop, > db:alter, etc.), do the heavy data processing and large document loading in > temporary databases and then swap the temporary databases with the > production databases when appropriate. > > > > I’m doing the orchestration with bash scripts that call the basex > client—maybe I’ve made it too complicated? But I didn’t see a way to pass > parameters to BaseX scripts, thus shell scripts that just call the basex > client to run small .xq files, specifying any needed parameters. > > > > Cheers, > > > > E. > > > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> > > > > *From: *BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on behalf > of Jonathan Robie <jonathan.robie at gmail.com> > *Date: *Monday, February 14, 2022 at 8:17 AM > *To: *Christian Grün <christian.gruen at gmail.com> > *Cc: *BaseX <basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] combining queries from mutiple sources > > *[External Email]* > > > > Also, you can have the best of both worlds, saving results in a pipeline > if it's helpful for debugging, just running the updates in place once you > know it all works well. This is particularly helpful if your data is > complex and your datasets are largish. > > > > I am currently merging various sources that were not originally meant to > work together, creating a common reference system for fairly complex data. > I find it helpful to write a bunch of simple updates that can be executed > sequentially, running them with a .bxs file. If I want to see the > output after any of these updates, I can do so with an OUTPUT statement. > Most of these stages are only useful for debugging purposes, so once a > pipeline is working well, I get rid of the OUTPUT statement. > > > > If your data is large, it can also be helpful to have two different > statements for importing the data, one that imports everything, another > that imports only a small test subset that runs quickly. Comment one of > them out and use the other. For rapid development, use the small, quick > subset, then run it on the whole dataset once it works. > > > > Jonathan > > > > On Sun, Feb 13, 2022 at 6:19 PM Christian Grün <christian.gruen at gmail.com> > wrote: > > Hi Eric, > > That’s perfectly feasible. You can bind the dynamic query string to an > external variable and invoke it on extracted data via xquery:eval. > Here’s one way to do it: > > (: db.xml :) > <path><to><source><data/></source></to></path> > > (: query.xq :) > declare variable $query external; > > for $data in db:open('db')/path/to/source/data > let $updated := xquery:eval($query, map { '': $data }) > return replace node $data with $updated > > The query could e.g. be invoked via > > basex -b query="<new>{ . }</new>" query.xq > > Hope this helps, > Christian > > > > On Sun, Feb 13, 2022 at 11:26 PM Eric Levy <contact at ericlevy.name> wrote: > > > > I am completely new to BaseX, and trying to conceive of a way to use it > > to process a compound query that combines an external query with a > > static one. > > > > The core idea is a very simple application, expressed as an XQuery > > file, that results in the following sequence of actions: > > > > 1. Extraction of data from a database, by some query. > > 2. Modification of the results of (1), by some query. > > 3. Persistence of the results from (2), back into the database from > > which the data was originally extracted. > > > > The modifications given in (2) would be expressed external to the > > application, supplied as input per invocation, in a separate file or in > > a command-line parameter. > > > > Thus, the overall operation would entail static components, the fixed > > parts of the application, represented by (1) and (3), and dynamic > > components, passed by the invocation, represented by (2). > > > > I realize such a processing pipeline may be achieved through queries > > supplied in a full application accessing the database by one of the > > language-specific APIs, but I am trying to consider an approach that > > requires use only of XQuery statements, without involving a separate > > programming language or formal build process. > > > > Does the design of BaseX accommodate an application having the > > structure as described? > > > > > >