Main Page » Mailing List » January 2022

Managing Interactions with Long-Running Processes

Wednesday, 26 January 2022 · 3 messages · last reply 27 Jan 2022
EK
Eliot Kimber26 Jan 2022 22:33 #
I’m making good progress on our BaseX-based validation dashboard application. The basic process here is we use Oxygen’s scripting support to do DITA map validation and then ingest the result into a database (along with the content documents that were validated) and provide reports on the validation result. The practical challenge here seems to be running the Oxygen process successfully from BaseX—because our content is so huge it can take 10s of minutes for the Oxygen process to run. I set the command time-out to be much longer than the process should run but running it from the HTTP app’s query panel it eventually failed with an error that wasn’t a time out (my code had earlier reported legit errors so I know errors will be properly reported). As soon as the Oxygen process ends I want to ingest the resulting XML file, which is why I started with doing it from within BaseX. But I’m wondering if this is a bad idea and I should really be doing it with i.e., a shell script run via cron or some such? I was trying to keep everything in BaseX as much as possible just to keep it simple. Any general tips or cautions for this time of integration of BaseX with the outside world? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
IL
Imsieke, Gerrit, le-tex26 Jan 2022 22:54 #
Hi Eliot, Can you be a bit more specific about how you set the timeout? If you only set it as an option on one of the proc:*() functions but the the action gets triggered by a RESTXQ request, then still the default timeout of 30 seconds may kick in (https://docs.basex.org/wiki/Options#TIMEOUT). See the possibilities to alter this value at the top of said page. For example, you can set it in webapp/WEB-INF/web.xml or with -Dorg.basex.TIMEOUT=7200 in your BaseX startup script. Gerrit
Quoted text · Eliot Kimber · 51 lines
On 26.01.2022 22:33, Eliot Kimber wrote: > I’m making good progress on our BaseX-based validation dashboard > application. > > The basic process here is we use Oxygen’s scripting support to do DITA > map validation and then ingest the result into a database (along with > the content documents that were validated) and provide reports on the > validation result. > > The practical challenge here seems to be running the Oxygen process > successfully from BaseX—because our content is so huge it can take 10s > of minutes for the Oxygen process to run. > > I set the command time-out to be much longer than the process should run > but running it from the HTTP app’s query panel it eventually failed with > an error that wasn’t a time out (my code had earlier reported legit > errors so I know errors will be properly reported). > > As soon as the Oxygen process ends I want to ingest the resulting XML > file, which is why I started with doing it from within BaseX. > > But I’m wondering if this is a bad idea and I should really be doing it > with i.e., a shell script run via cron or some such? > > I was trying to keep everything in BaseX as much as possible just to > keep it simple. > > Any general tips or cautions for this time of integration of BaseX with > the outside world? > > Thanks, > > E. > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://www.linkedin.com/company/servicenow> | Twitter > <https://twitter.com/servicenow> | YouTube > <https://www.youtube.com/user/servicenowinc> | Facebook > <https://www.facebook.com/servicenow> >
-- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke at le-tex.de, http://www.le-tex.de Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930 Geschäftsführer / Managing Directors: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt
EK
Eliot Kimber27 Jan 2022 00:12 #
For my initial attempt I’m just using proc:execute() from a manually-run XQuery script run from the BaseX admin query panel:
  let $timeout as xs:integer := 60 * 30
  let $result := proc:execute($oxygenCmd,
     ('-i', $inputPath,
      '-r', $reportPath,
      '-s', $settingsPath
     ),
     map{
       'timeout' : $timeout
     }
  )
Subsequent tests seemed to work OK, in that the process ran until Oxygen ran out of memory. I haven’t been able to figure out why my attempts to set the JVM memory for Oxygen are failing and have contacted Oxygen support for guidance. But it’s good to know about these other ways to set the timeout—I know I’ll need to attend to more considered BaseX settings once I get everything hooked up. Cheers, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Quoted text · BaseX-Talk · 77 lines
From: BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on behalf of Imsieke, Gerrit, le-tex <gerrit.imsieke at le-tex.de> Date: Wednesday, January 26, 2022 at 3:54 PM To: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Managing Interactions with Long-Running Processes [External Email] Hi Eliot, Can you be a bit more specific about how you set the timeout? If you only set it as an option on one of the proc:*() functions but the the action gets triggered by a RESTXQ request, then still the default timeout of 30 seconds may kick in (https://urldefense.com/v3/__https://docs.basex.org/wiki/Options*TIMEOUT__;Iw!!N4vogdjhuJM!XON4qC6R8fNj3sdIfTp9rLyz4U2wPMQRUkUD_R98BtbZGDCeu0h8HtKZ9q0l6tNkSxdLrA$ ). See the possibilities to alter this value at the top of said page. For example, you can set it in webapp/WEB-INF/web.xml or with -Dorg.basex.TIMEOUT=7200 in your BaseX startup script. Gerrit On 26.01.2022 22:33, Eliot Kimber wrote: > I’m making good progress on our BaseX-based validation dashboard > application. > > The basic process here is we use Oxygen’s scripting support to do DITA > map validation and then ingest the result into a database (along with > the content documents that were validated) and provide reports on the > validation result. > > The practical challenge here seems to be running the Oxygen process > successfully from BaseX—because our content is so huge it can take 10s > of minutes for the Oxygen process to run. > > I set the command time-out to be much longer than the process should run > but running it from the HTTP app’s query panel it eventually failed with > an error that wasn’t a time out (my code had earlier reported legit > errors so I know errors will be properly reported). > > As soon as the Oxygen process ends I want to ingest the resulting XML > file, which is why I started with doing it from within BaseX. > > But I’m wondering if this is a bad idea and I should really be doing it > with i.e., a shell script run via cron or some such? > > I was trying to keep everything in BaseX as much as possible just to > keep it simple. > > Any general tips or cautions for this time of integration of BaseX with > the outside world? > > Thanks, > > E. > > _____________________________________________ > > *Eliot Kimber* > > Sr Staff Content Engineer > > O: 512 554 9368 > > M: 512 554 9368 > > servicenow.com <https://www.servicenow.com> > > LinkedIn <https://urldefense.com/v3/__https://www.linkedin.com/company/servicenow__;!!N4vogdjhuJM!XON4qC6R8fNj3sdIfTp9rLyz4U2wPMQRUkUD_R98BtbZGDCeu0h8HtKZ9q0l6tMVMAJcEA$ > | Twitter > <https://urldefense.com/v3/__https://twitter.com/servicenow__;!!N4vogdjhuJM!XON4qC6R8fNj3sdIfTp9rLyz4U2wPMQRUkUD_R98BtbZGDCeu0h8HtKZ9q0l6tOQDUB8iQ$ > | YouTube > <https://urldefense.com/v3/__https://www.youtube.com/user/servicenowinc__;!!N4vogdjhuJM!XON4qC6R8fNj3sdIfTp9rLyz4U2wPMQRUkUD_R98BtbZGDCeu0h8HtKZ9q0l6tN4xc5sJA$ > | Facebook > <https://urldefense.com/v3/__https://www.facebook.com/servicenow__;!!N4vogdjhuJM!XON4qC6R8fNj3sdIfTp9rLyz4U2wPMQRUkUD_R98BtbZGDCeu0h8HtKZ9q0l6tPFlGs32Q$ > > -- Gerrit Imsieke Geschäftsführer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke at le-tex.de, https://urldefense.com/v3/__http://www.le-tex.de__;!!N4vogdjhuJM!XON4qC6R8fNj3sdIfTp9rLyz4U2wPMQRUkUD_R98BtbZGDCeu0h8HtKZ9q0l6tM2UK0Wqw$<https://urldefense.com/v3/__http:/www.le-tex.de__;!!N4vogdjhuJM!XON4qC6R8fNj3sdIfTp9rLyz4U2wPMQRUkUD_R98BtbZGDCeu0h8HtKZ9q0l6tM2UK0Wqw$> Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930 Geschäftsführer / Managing Directors: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt

Written for BaseX 9.6. BaseX may have changed since: please check the current documentation.

« Unable to Make My Custom EXPath Module WorkTechniques for Unit Testing Updating Operations »
⚡Generated with XQuery