Main Page » Mailing List » February 2022

Understanding Job Execution and Verification

Friday, 18 February 2022 · 4 messages
EK
Eliot Kimber18 Feb 2022 16:51 #
I’m working out an “orchestration” module that manages running jobs that are functions from different modules, where the orchestrator dynamically constructs an XQuery that runs the specified function with the specified string parameters. I’m running jobs through a function that takes a sequence of jobs (XQuery strings) and calls jobs:eval() on each one in sequence:
declare function orch:runJobs($jobs as xs:string*) as xs:string* {
  let $debug := util:logToLog('Have ' || count($jobs) || ' jobs:')
  let $debug := prof:dump($jobs)
  return
  for $job at $i in $jobs
  let $debug := (util:logToLog('Running job'), prof:dump($job))
  return
  try {
    jobs:eval($job, (), map{'cache' : true(), 'log' : 'Running job: ' || $job})
  } catch * {
    util:logToLog('orch:runJobs',
      ('Error running job ' || $err:code || ': ' || $err:description)
    )
  }
};
What I’m trying to achieve is a sequence of jobs run such that job 2 does not start until job 1 is finished and I don’t think that’s actually what I’m producing here. Most of these jobs are manipulating databases: dropping, creating, renaming, copying, etc., but a few will be longer-running—constructing indexes and whatnot. My question: What is the (best?) way to produce a sequence of jobs where each job waits for the preceding? Is it simply to do a jobs:wait(jobs:eval()) on each job or is there something better? Or am I missing a simpler way to perform this kind of job orchestration? Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
CG
Christian Grün18 Feb 2022 17:35 #
Quoted text · Eliot Kimber · 1 line
> What I’m trying to achieve is a sequence of jobs run such that job 2 does not start until job 1 is finished and I don’t think that’s actually what I’m producing here.
If you schedule jobs, you cannot control when a job will eventually be evaluated, as the start time is e.g. influenced by other jobs that are run in parallel, and by databases that will be accessed by your query. An example: prof:dump(jobs:eval("delete nodes db:open('DB')/*")), insert node <a/> into db:open('DB')/* This query will update DB, and the scheduled query can only be started after your current query has finished. And there are options such as FAIRLOCK or PARALLEL that further complicate the manual orchestration of job executions.
Quoted text · Eliot Kimber · 1 line
> Is it simply to do a jobs:wait(jobs:eval()) on each job or is there something better?
The wait function will immediately return if the job id is unknown; but it could be that the job has not even been started. The following code would probably achieve what you’re trying to do:
let $query := 'prof:sleep(1000)'
let $job-id := jobs:eval($query, (), map { 'start': 'PT0.1S' })
return hof:until(
  function($_) { not(jobs:list() = $job-id) },
  function($_) { prof:sleep(100) },
  ()
),
'finished'
I wouldn’t recommend that because of the transactional semantics, and because of side effects that can take place in a productive environment and that might block your orchestration code forever. A cleaner solution is schedule the next job from inside your query:
(: query1.xq :)
db:create('db'),
jobs:eval(xs:anyURI('query2.xq')

(: query2.xq :)
...,
jobs:eval(xs:anyURI('query3.xq')
(: query3.xq :) ... And the cleanest solution is define as many updates as possible in one single query. This way, you can benefit most from numerous update optimizations performed by BaseX (e.g., 100 single insert queries can be much slower than one large insert query). Best, Christian
EK
Eliot Kimber18 Feb 2022 18:55 #
So if I understand your final recommendation, I should be “chaining” the jobs so that each job does its thing and then schedules the next job in the sequence. I can do that easily enough. Thanks, E. _____________________________________________ Eliot Kimber Sr Staff Content Engineer O: 512 554 9368 M: 512 554 9368 servicenow.com<https://www.servicenow.com> LinkedIn<https://www.linkedin.com/company/servicenow> | Twitter<https://twitter.com/servicenow> | YouTube<https://www.youtube.com/user/servicenowinc> | Facebook<https://www.facebook.com/servicenow>
Quoted text · Christian Grün · 47 lines
From: Christian Grün <christian.gruen at gmail.com> Date: Friday, February 18, 2022 at 10:35 AM To: Eliot Kimber <eliot.kimber at servicenow.com> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Understanding Job Execution and Verification [External Email] > What I’m trying to achieve is a sequence of jobs run such that job 2 does not start until job 1 is finished and I don’t think that’s actually what I’m producing here. If you schedule jobs, you cannot control when a job will eventually be evaluated, as the start time is e.g. influenced by other jobs that are run in parallel, and by databases that will be accessed by your query. An example: prof:dump(jobs:eval("delete nodes db:open('DB')/*")), insert node <a/> into db:open('DB')/* This query will update DB, and the scheduled query can only be started after your current query has finished. And there are options such as FAIRLOCK or PARALLEL that further complicate the manual orchestration of job executions. > Is it simply to do a jobs:wait(jobs:eval()) on each job or is there something better? The wait function will immediately return if the job id is unknown; but it could be that the job has not even been started. The following code would probably achieve what you’re trying to do: let $query := 'prof:sleep(1000)' let $job-id := jobs:eval($query, (), map { 'start': 'PT0.1S' }) return hof:until( function($_) { not(jobs:list() = $job-id) }, function($_) { prof:sleep(100) }, () ), 'finished' I wouldn’t recommend that because of the transactional semantics, and because of side effects that can take place in a productive environment and that might block your orchestration code forever. A cleaner solution is schedule the next job from inside your query: (: query1.xq :) db:create('db'), jobs:eval(xs:anyURI('query2.xq') (: query2.xq :) ..., jobs:eval(xs:anyURI('query3.xq') (: query3.xq :) ... And the cleanest solution is define as many updates as possible in one single query. This way, you can benefit most from numerous update optimizations performed by BaseX (e.g., 100 single insert queries can be much slower than one large insert query). Best, Christian
CG
Christian Grün18 Feb 2022 19:21 #
Quoted text · Eliot Kimber · 1 line
> So if I understand your final recommendation, I should be “chaining” the jobs so that each job does its thing and then schedules the next job in the sequence.
ちょうど！

Written for BaseX 9.6. BaseX may have changed since: please check the current documentation.

« Namespaces, replace, and performanceUsing %unit:after to verify database creation »
⚡Generated with XQuery