Difference between revisions of "Job Module"

From BaseX Documentation
Jump to navigation Jump to search
m (Text replacement - "syntaxhighlight" to "pre")
 
(55 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This [[Module Library|XQuery Module]] provides functions for organizing scheduled, queued, running and cached jobs. Jobs can be commands, queries, client or HTTP requests.
+
This [[Module Library|XQuery Module]] provides functions for registering new query jobs and orchestrating existing jobs. Jobs can be queries, commands, operations performed by a database client, and HTTP requests.
  
 
=Conventions=
 
=Conventions=
  
All functions in this module are assigned to the <code><nowiki>http://basex.org/modules/jobs</nowiki></code> namespace, which is statically bound to the {{Code|jobs}} prefix. Errors will be bound to the same prefix.
+
All functions in this module are assigned to the <code><nowiki>http://basex.org/modules/job</nowiki></code> namespace, which is statically bound to the {{Code|job}} prefix. Errors will be bound to the same prefix.
  
 
=Services=
 
=Services=
  
Jobs can additionally be registered as persistent services. An additional {{Code|service}} option has been added to the options list of {{Function|Jobs|jobs:eval}} and {{Function|Jobs|jobs:invoke}}:
+
A job can be registered as ''service'' by supplying the {{Code|service}} option to {{Function||job:eval}}:
  
<pre class="brush:xquery">
+
<pre lang='xquery'>
(: register job, which will be run every day at 1 am :)
+
(: register job as service; will be run every day at 1 am :)
jobs:eval('db:drop("tmp")', (), map { 'id':'cleanup', 'start':'01:00:00', 'interval':'P1D', 'service': true() }),
+
job:eval('db:drop("tmp")', (), map { 'id':'cleanup', 'start':'01:00:00', 'interval':'P1D', 'service': true() }),
  
 
(: list registered services :)
 
(: list registered services :)
jobs:services(),
+
job:services(),
 
(: result: <job base-uri="..." id="cleanup" interval="P1D" start="01:00:00">db:drop("tmp")</job> :)
 
(: result: <job base-uri="..." id="cleanup" interval="P1D" start="01:00:00">db:drop("tmp")</job> :)
  
 
(: unregister job :)
 
(: unregister job :)
jobs:stop('cleanup', map { 'service': true() })
+
job:remove('cleanup', map { 'service': true() })
 
</pre>
 
</pre>
  
 
'''Some more notes:'''
 
'''Some more notes:'''
  
* All registered jobs will be scheduled for evaluation when the BaseX server or BaseX HTTP server is started.
+
* All job services will be scheduled for evaluation when the BaseX server or BaseX HTTP server is started.
 
* If a job service is outdated (e.g. because a supplied end time has been exceeded), it will be removed from the jobs file at startup time.
 
* If a job service is outdated (e.g. because a supplied end time has been exceeded), it will be removed from the jobs file at startup time.
* Job services can be updated: If a new job is registered, and if there is already a job with the same id, the old entry will be replaced.
 
 
* The job definitions are stored in a {{Code|jobs.xml}} file in the database directory. It can also be edited manually.
 
* The job definitions are stored in a {{Code|jobs.xml}} file in the database directory. It can also be edited manually.
  
=Basic Functions=
+
=Executing Jobs=
  
==jobs:current==
+
There are cases in which a client does not, or cannot, wait until a request is fully processed. The client may be a browser, which sends an HTTP request to the server to start another time-consuming query job. The functions in this section allow you to register new query jobs and access existing ones. Jobs can be executed immediately (i.e., as soon as a [[Transaction Management#Concurrency Control|free slot is available]]) or scheduled for repeated execution. Each registered job gets a job ID, and the ID can be used to retrieve a query result, stop a job, or wait for its termination.
  
{| width='100%'
+
==job:eval==
|-
 
| width='120' | '''Signatures'''
 
|{{Func|jobs:current||xs:string}}
 
|-
 
| '''Summary'''
 
|Returns the id of the current job.
 
|}
 
 
 
==jobs:list==
 
 
 
{| width='100%'
 
|-
 
| width='120' | '''Signatures'''
 
|{{Func|jobs:list||xs:string*}}
 
|-
 
| '''Summary'''
 
|Returns the ids of all jobs that are currently registered.  The list includes scheduled, queued, running, stopped, and finished jobs with cached results.
 
|-
 
| '''Examples'''
 
| <code>jobs:list()</code> returns the same job id as {{Function|Jobs|jobs:current}} if no other job is registered.
 
|}
 
 
 
==jobs:list-details==
 
 
 
{| width='100%'
 
|-
 
| width='120' | '''Signatures'''
 
|{{Func|jobs:list-details||element(job)*}}<br/>{{Func|jobs:list-details|$id as xs:string|element(job)*}}
 
|-
 
| '''Summary'''
 
|Returns information on all jobs that are currently registered, or on a job with the specified {{Code|$id}} (or an empty sequence if this job is not found). The list includes scheduled, queued, running jobs, and cached jobs. A string representation of the job, or its URI, will be returned as value. The returned elements have additional attributes:
 
* <code>id</code>: job id
 
* <code>type</code>: type of the job (command, query, REST, RESTXQ, etc.)
 
* <code>state</code>: current state of the job (scheduled, queued, running, or cached)
 
* <code>user</code>: the user who started the job
 
* <code>duration</code>: evaluation time (for running and cached jobs)
 
* <code>start</code>: dateTime string with next start (for jobs that will be executed repeatedly)
 
|-
 
| '''Examples'''
 
| <code>jobs:list-details()</code> returns information on the currently running job and possibly others:
 
<pre class="brush:xml">
 
<job id="job1" type="XQuery" state="running" user="admin" duration="PT0.001S">
 
  XQUERY jobs:list-details()
 
</job>
 
</pre>
 
|}
 
  
 
{| width='100%'
 
{| width='100%'
|-
+
|- valign="top"
| width='120' | '''Signatures'''
+
| width='120' | '''Signature'''
|{{Func|jobs:services||element(job)*}}
+
|<pre>job:eval(
|-
+
  $query    as xs:anyAtomicType,
 +
  $bindings  as map(*)?          := map { },
 +
  $options  as map(*)?          := map { }
 +
) as xs:string</pre>
 +
|- valign="top"
 
| '''Summary'''
 
| '''Summary'''
|Returns a list of all jobs that have been persistently registered as [[#Services|Services]].
+
|Schedules the evaluation a new query job for the supplied {{Code|$query}} (of type {{Code|xs:string}}, or of type {{Code|xs:anyURI}} if points to a resource), and returns a job ID. The job will be queued until a free slot is available, and the query result can be cached. Queries can be updating, and variables and the context value can be declared via {{Code|$bindings}} (see {{Function|XQuery|xquery:eval}} for more details). The following {{Code|$options}} can be supplied:
|-
+
* {{Code|cache}}: indicates if the query result will be cached or ignored (default: {{Code|false}}):
| '''Errors'''
+
** The result will be cached in main-memory until it is fetched via {{Function||job:result}}, or until {{Option|CACHETIMEOUT}} is exceeded.
|{{Error|services|#Errors}} Registered services cannot be parsed.<br/>
 
|}
 
 
 
==jobs:finished==
 
 
 
{| width='100%'
 
|-
 
| width='120' | '''Signatures'''
 
|{{Func|jobs:finished|$id as xs:string|xs:boolean}}
 
|-
 
| '''Summary'''
 
|Indicates if the evaluation of an already running job with the specified {{Code|$id}} has finished. As the ids of finished jobs will usually be discarded, unless caching is enabled, the function will also return <code>true</code> for unknown jobs.
 
* <code>false</code> indicates that the job id is scheduled, queued, or currently running.
 
* <code>true</code> will be returned if the job has either finished, or if the id is unknown (because the ids of all finished jobs will not be cached).
 
|}
 
 
 
=Execution=
 
 
 
Asynchronous query execution is recommendable if a client does not, or cannot, wait until a request is fully processed. This is e. g. the case with web browsers, which will usually cancel a request after a specific timeout. In such cases, you can use asynchronous execution to trigger another server-side process, which will start the time-consuming process, and fetch the result later on as soon as it is available.
 
 
 
==jobs:eval==
 
 
 
{| width='100%'
 
|-
 
| width='120' | '''Signatures'''
 
|{{Func|jobs:eval|$query as xs:string|xs:string}}<br />{{Func|jobs:eval|$query as xs:string, $bindings as map(*)?|xs:string}}<br />{{Func|jobs:eval|$query as xs:string, $bindings as map(*)?, $options as map(*)?|xs:string}}<br />
 
|-
 
| '''Summary'''
 
|Schedules the evaluation of the supplied {{Code|$query}} and returns a query id. The query will be queued, and the result will optionally be cached. Queries can be updating. Variables and context items can be declared via {{Code|$bindings}} (see [[XQuery Module#xquery:eval|xquery:eval]] for more details). The following {{Code|$options}} can be supplied:
 
* {{Code|cache}}: indicates if the query result will be cached or ignored (default: <code>false</code>):
 
** The result will be cached in main-memory until it is fetched via [[#jobs:result|jobs:result]], or until {{Option|CACHETIMEOUT}} is exceeded.
 
 
** If the query raises an error, it will be cached and returned instead.
 
** If the query raises an error, it will be cached and returned instead.
* {{Code|start}}: a dayTimeDuration, time or dateTime can be specified to delay the execution of the query:
+
* {{Code|start}}: a dayTimeDuration, time, dateTime or integer can be specified to delay the execution of the query:
** If a dayTimeDuration is specified, the query will be queued after the specified duration has passed. Examples for valid values are: <code>P1D</code> (1 day), <code>PT5M</code> (5 minutes), <code>PT0.1S</code> (100 ms). An error will be raised if a negative value is specified.
+
** If a dayTimeDuration is specified, the query will be queued after the specified duration has passed. Examples of valid values are: <code>P1D</code> (1 day), <code>PT5M</code> (5 minutes), <code>PT0.1S</code> (100 ms). An error will be raised if a negative value is specified.
** If a time is specified, the query will be executed at this time of the day. Examples for valid times are: <code>02:00:00</code> (2am local time), <code>12:00:00Z</code> (noon, UTC). If the time lies in the past, the query will be executed the next day.
 
 
** If a dateTime is specified, the query will be executed at this date. Examples for valid values are: <code>2018-12-31T23:59:59</code> (New Year's Eve 2018, close to midnight). An error will be raised if the specified time lies in the past.
 
** If a dateTime is specified, the query will be executed at this date. Examples for valid values are: <code>2018-12-31T23:59:59</code> (New Year's Eve 2018, close to midnight). An error will be raised if the specified time lies in the past.
* {{Code|interval}}: a dayTimeDuration string can be specified to execute the query periodically. An error is raised if the specified interval is less than one second (<code>PT1S</code>). If the next scheduled call is due, and if a query with the same id is still running, it will be skipped.
+
** If a time is specified, the query will be executed at this time of the day. Examples of valid times are: <code>02:00:00</code> (2am local time), <code>12:00:00Z</code> (noon, UTC). If the time lies in the past, the query will be executed the next day.
 +
** An integer will be interpreted as minutes. If the specified number is greater than the elapsed minutes of the current hour, the query will be executed one hour later.
 +
* {{Code|interval}}: a dayTimeDuration string can be specified to execute the query periodically. An error is raised if the specified interval is less than one second (<code>PT1S</code>). If the next scheduled call is due, and if a query with the same ID is still running, it will be skipped.
 
* {{Code|end}}: scheduling can be stopped after a given time or duration. The string format is the same as for {{Code|start}}. An error is raised if the resulting end time is smaller than the start time.
 
* {{Code|end}}: scheduling can be stopped after a given time or duration. The string format is the same as for {{Code|start}}. An error is raised if the resulting end time is smaller than the start time.
 
* {{Code|base-uri}}: sets the [https://www.w3.org/TR/xquery-31/#dt-static-base-uri base-uri property] for the query. This URI will be used when resolving relative URIs, such as with {{Code|fn:doc}}.
 
* {{Code|base-uri}}: sets the [https://www.w3.org/TR/xquery-31/#dt-static-base-uri base-uri property] for the query. This URI will be used when resolving relative URIs, such as with {{Code|fn:doc}}.
* {{Code|id}}: sets a custom job id. The id must not start with the standard <code>job</code> prefix, and it can only be assigned if no job with the same name exists.
+
* {{Code|id}}: sets a custom job ID. The ID must not start with the standard <code>job</code> prefix, and it can only be assigned if no job with the same name exists.
* {{Code|service}}: additionally registers the job as [[#Services|service]].
+
* {{Code|service}}: additionally registers the job as [[#Services|service]]. Registered services must have no variable bindings.
|-
+
* {{Code|log}}: writes the specified string to the [[Logging|database logs]]. Two log entries are stored, one at the beginning and another one after the execution of the job.
 +
|- valign="top"
 
| '''Errors'''
 
| '''Errors'''
|{{Error|overflow|#Errors}} Query execution is rejected, because too many jobs are queued or being executed. {{Option|CACHETIMEOUT}} can be decreased if the default setting is too restrictive.<br/>{{Error|range|#Errors}} A specified time or duration is out of range.<br/>{{Error|id|#Errors}} The specified id is invalid or has already been assigned.<br/>{{Error|options|#Errors}} The specified options are conflicting.
+
|{{Error|overflow|#Errors}} Query execution is rejected because too many jobs are queued or being executed. {{Option|CACHETIMEOUT}} can be decreased if the default setting is too restrictive.<br/>{{Error|range|#Errors}} A specified time or duration is out of range.<br/>{{Error|id|#Errors}} The specified ID is invalid or has already been assigned.<br/>{{Error|options|#Errors}} The specified options are conflicting.
|-
+
|- valign="top"
 
| '''Examples'''
 
| '''Examples'''
 
|
 
|
* Cache query result. The returned id can be used to pick up the result with [[#jobs:result|jobs:result]]:
+
* Cache query result. The returned ID can be used to pick up the result with {{Function||job:result}}:
<pre class='brush:xquery'>
+
<pre lang='xquery'>
jobs:eval("1+3", (), map { 'cache': true() })
+
job:eval("1+3", (), map { 'cache': true() })
 
</pre>
 
</pre>
 
* A happy birthday mail will be sent at the given date:
 
* A happy birthday mail will be sent at the given date:
<pre class="brush:xquery">
+
<pre lang='xquery'>
jobs:eval("import module namespace mail='mail'; mail:send('Happy birthday!')",
+
job:eval("import module namespace mail='mail'; mail:send('Happy birthday!')",
 
   (), map { 'start': '2018-09-01T06:00:00' })}}
 
   (), map { 'start': '2018-09-01T06:00:00' })}}
 
</pre>
 
</pre>
* The following [[RESTXQ]] functions can be called to execute a query at 2 am every day. An id will be returned by the first function, which can be used to stop the scheduler via the second function:
+
* The following [[RESTXQ]] functions can be called to execute a query at 2 am every day. An ID will be returned by the first function, which can be used to stop the scheduler via the second function:
<pre class='brush:xquery'>
+
<pre lang='xquery'>
 
declare %rest:POST("{$query}") %rest:path('/start-scheduling') function local:start($query) {
 
declare %rest:POST("{$query}") %rest:path('/start-scheduling') function local:start($query) {
   jobs:eval($query, (), map { 'start': '02:00:00', 'interval': 'P1D' })
+
   job:eval($query, (), map { 'start': '02:00:00', 'interval': 'P1D' })
 
};
 
};
 
declare %rest:path('/stop-scheduling/{$id}') function local:stop($id) {
 
declare %rest:path('/stop-scheduling/{$id}') function local:stop($id) {
   jobs:stop($id)
+
   job:remove($id)
 
};
 
};
 
</pre>
 
</pre>
 
* Query execution is scheduled for every second, and for 10 seconds in total. As the query itself will take 1.5 seconds, it will only be executed every second time:
 
* Query execution is scheduled for every second, and for 10 seconds in total. As the query itself will take 1.5 seconds, it will only be executed every second time:
<pre class="brush:xquery">
+
<pre lang='xquery'>
jobs:eval("prof:sleep(1500)", (), map { 'interval': 'PT1S', 'end': 'PT10S' })
+
job:eval("prof:sleep(1500)", (), map { 'interval': 'PT1S', 'end': 'PT10S' })
 
</pre>
 
</pre>
* The following expression, if stored as a file, calls and evaluates itself every 5 seconds:
+
* The query in the specified file will be evaluated once:
<pre class="brush:xquery">
+
<pre lang='xquery'>
jobs:eval(
+
job:eval(xs:anyURI('cleanup.xq'))
   file:read-text(static-base-uri()),
+
</pre>
 +
* The following expression, if stored in a file, will be evaluated every 5 seconds:
 +
<pre lang='xquery'>
 +
job:eval(
 +
   static-base-uri(),
 
   map { },
 
   map { },
 
   map { 'start': 'PT5S' }
 
   map { 'start': 'PT5S' }
Line 168: Line 100:
 
|}
 
|}
  
==jobs:invoke==
+
==job:result==
  
 
{| width='100%'
 
{| width='100%'
|-
+
|- valign="top"
| width='120' | '''Signatures'''
+
| width='120' | '''Signature'''
|{{Func|jobs:invoke|$uri as xs:string|xs:string}}<br />{{Func|jobs:invoke|$uri as xs:string, $bindings as map(*)?|xs:string}}<br />{{Func|jobs:invoke|$uri as xs:string, $bindings as map(*)?, $options as map(*)?|xs:string}}<br />
+
|<pre>job:result(
|-
+
  $id      as xs:string,
| '''Summary'''
+
  $options as map(*)?   := map { }
|Schedules the evaluation of the XQuery expression located at {{Code|$uri}} and returns a query id. For further details, see [[#jobs:eval|jobs:eval]].
+
) as item()*</pre>
|-
+
|- valign="top"
| '''Errors'''
 
|{{Error|overflow|#Errors}} Query execution is rejected, because too many jobs are queued or being executed. {{Option|CACHETIMEOUT}} can be decreased if the default setting is too restrictive.<br/>{{Error|range|#Errors}} A specified time or duration is out of range.<br/>{{Error|id|#Errors}} The specified id is invalid or has already been assigned.<br/>{{Error|options|#Errors}} The specified options are conflicting.
 
|-
 
| '''Examples'''
 
| Run XQuery expression that may perform some cleanups:
 
<pre class='brush:xquery'>
 
jobs:invoke("cleanup.xq", (), ())
 
</pre>
 
|}
 
 
 
==jobs:result==
 
 
 
{| width='100%'
 
|-
 
| width='120' | '''Signatures'''
 
|{{Func|jobs:result|$id as xs:string|item()*}}
 
|-
 
 
| '''Summary'''
 
| '''Summary'''
 
|Returns the cached result of a job with the specified job {{Code|$id}}:
 
|Returns the cached result of a job with the specified job {{Code|$id}}:
* Results can only be retrieved once. After retrieval, the cached result will be dropped.
 
 
* If the original job has raised an error, the cached error will be raised instead.
 
* If the original job has raised an error, the cached error will be raised instead.
|-
+
* The cached result or error will be dropped after it has been retrieved.
| '''Errors'''
+
* If the result has not been cached or if it has been dropped, an empty sequence is returned.
|{{Error|running|#Errors}} the job is still running.<br/>{{Error|unknown|#Errors}} the supplied id is unknown: The id is unknown, or the result has already been retrieved.<br/>
+
The following {{Code|$options}} can be supplied:
|-
+
* {{Code|keep}}: Keep the cached result or error after retrieval.
 +
 
 +
|- valign="top"
 
| '''Examples'''
 
| '''Examples'''
 
|
 
|
 
* The following [[RESTXQ]] function will either return the result of a previously started job or raise an error:
 
* The following [[RESTXQ]] function will either return the result of a previously started job or raise an error:
<pre class='brush:xquery'>
+
<pre lang='xquery'>
 
declare %rest:path('/result/{$id}') function local:result($id) {
 
declare %rest:path('/result/{$id}') function local:result($id) {
   jobs:result($id)
+
   job:result($id)
 
};
 
};
 
</pre>
 
</pre>
* The following query demonstrates how the results of an asynchronously executed query can be returned within the same query:
+
* The following query demonstrates how the results of an executed query can be returned within the same query (see below why you should avoid this pattern in practice):
<pre class='brush:xquery'>
+
<pre lang='xquery'>
let $query := jobs:eval('(1 to 10000000)[. = 1]', map{}, map{ 'cache': true() })
+
let $query := job:eval('(1 to 10000000)[. = 1]', map { }, map { 'cache': true() })
 
return (
 
return (
   jobs:wait($query),
+
   job:wait($query),
   jobs:result($query)
+
   job:result($query)
 
)
 
)
 
</pre>
 
</pre>
Please note that queries of this kind can cause deadlocks. For example, if both the original query and the query to be executed asynchronously perform updates on the same database, the second query would only be run after the first one has been executed, and the first query will wait forever. This is why you should avoid this pattern in practice and resort to [[XQuery Module#xquery:fork-join|xquery:fork-join]] if you want to do things in parallel.
+
Queries of this kind can cause deadlocks! If the original query and the new query perform updates on the same database, the second query will only be run after the first one has been executed, and the first query will wait for the second query forever. You should resort to {{Function|XQuery|xquery:fork-join}} if you want to have full control on parallel query execution.
 
|}
 
|}
  
==jobs:stop==
+
==job:remove==
  
 
{| width='100%'
 
{| width='100%'
|-
+
|- valign="top"
| width='120' | '''Signatures'''
+
| width='120' | '''Signature'''
|{{Func|jobs:stop|$id as xs:string|empty-sequence()}}
+
|<pre>job:remove(
|-
+
  $id       as xs:string,
 +
  $options  as map(*)?    := map { }
 +
) as empty-sequence()</pre>
 +
|- valign="top"
 
| '''Summary'''
 
| '''Summary'''
|Triggers the cancelation of a job with the specified {{Code|$id}}, drops the cached result of a query, or cancels a scheduled job. Unknown ids are ignored. All jobs are gracefully stopped; it is up to the process to decide when it is safe to shut down. The following {{Code|$options}} can be supplied:
+
|Triggers the cancelation of a job with the specified {{Code|$id}}, cancels a scheduled job or removes a cached result. Unknown IDs are ignored. All jobs are gracefully stopped; it is up to the process to decide when it is safe to shut down. The following {{Code|$options}} can be supplied:
 
* {{Code|service}}: additionally removes the job from the [[#Services|job services]] list.
 
* {{Code|service}}: additionally removes the job from the [[#Services|job services]] list.
|-
+
|- valign="top"
 
| '''Examples'''
 
| '''Examples'''
| <code>jobs:list()[. != jobs:current()] ! jobs:stop(.)</code> stops and discards all jobs except for the current one.
+
|
 +
* <code>job:list()[. != job:current()] ! job:remove(.)</code> stops and discards all jobs except for the current one.
 +
* <code>job:remove(job:current())</code> interrupts the current job.
 
|}
 
|}
  
==jobs:wait==
+
==job:wait==
  
 
{| width='100%'
 
{| width='100%'
|-
+
|- valign="top"
| width='120' | '''Signatures'''
+
| width='120' | '''Signature'''
|{{Func|jobs:wait|$id as xs:string|empty-sequence()}}
+
|<pre>job:wait(
|-
+
  $id as xs:string
 +
) as empty-sequence()</pre>
 +
|- valign="top"
 
| '''Summary'''
 
| '''Summary'''
 
|Waits for the completion of a job with the specified {{Code|$id}}:
 
|Waits for the completion of a job with the specified {{Code|$id}}:
* The function will terminate immediately if the job id is unknown. This is the case if a future job has not been queued yet, or if the id has already been discarded after job evaluation.
+
* The function will terminate immediately if the job ID is unknown. This is the case if a future job has not been queued yet, or if the ID has already been discarded after job evaluation.
* If the function is called with the id of a queued job, or repeatedly executed job, it may stall and never terminate.
+
* If the function is called with the ID of a queued job, or repeatedly executed job, it may stall and never terminate.
|-
+
|- valign="top"
 
| '''Errors'''
 
| '''Errors'''
 
|{{Error|self|#Errors}} The current job is addressed.<br/>
 
|{{Error|self|#Errors}} The current job is addressed.<br/>
 +
|}
 +
 +
=Listing Jobs=
 +
 +
==job:current==
 +
 +
{| width='100%'
 +
|- valign="top"
 +
| width='120' | '''Signature'''
 +
|<pre>job:current() as xs:string</pre>
 +
|- valign="top"
 +
| '''Summary'''
 +
|Returns the ID of the current job.
 +
|}
 +
 +
==job:list==
 +
 +
{| width='100%'
 +
|- valign="top"
 +
| width='120' | '''Signature'''
 +
|<pre>job:list() as xs:string*</pre>
 +
|- valign="top"
 +
| '''Summary'''
 +
|Returns the IDs of all jobs that are currently registered. The list includes scheduled, queued, running, stopped, and finished jobs with cached results.
 +
|- valign="top"
 +
| '''Examples'''
 +
| <code>job:list()</code> returns the same job ID as {{Function||job:current}} if no other job is registered.
 +
|}
 +
 +
==job:list-details==
 +
 +
{| width='100%'
 +
|- valign="top"
 +
| width='120' | '''Signature'''
 +
|<pre>job:list-details(
 +
  $id  as xs:string  := ()
 +
) as element(job)*</pre>
 +
|- valign="top"
 +
| '''Summary'''
 +
|Returns information on all jobs that are currently registered, or on a job with the specified {{Code|$id}} (or an empty sequence if this job is not found). The list includes scheduled, queued, running jobs, and cached jobs. A string representation of the job, or its URI, will be returned as a value. The returned elements have additional attributes:
 +
* <code>id</code>: job ID
 +
* <code>type</code>: type of the job (command, query, REST, RESTXQ, etc.)
 +
* <code>state</code>: current state of the job: <code>scheduled</code>, <code>queued</code>, <code>running</code>, <code>cached</code>
 +
* <code>user</code>: user who started the job
 +
* <code>duration</code>: evaluation time (included if a job is running or if the result was cached)
 +
* <code>start</code>: next start of job (included if a job will be executed repeatedly)
 +
* <code>time</code>: time when job was registered
 +
|- valign="top"
 +
| '''Examples'''
 +
| <code>job:list-details()</code> returns information on the currently running job and possibly others:
 +
<pre lang="xml">
 +
<job id="job1" type="XQuery" state="running" user="admin" duration="PT0.001S">
 +
  XQUERY job:list-details()
 +
</job>
 +
</pre>
 +
|}
 +
 +
==job:bindings==
 +
 +
{| width='100%'
 +
|- valign="top"
 +
| width='120' | '''Signature'''
 +
|<pre>job:bindings(
 +
  $id  as xs:string
 +
) as map(*)</pre>
 +
|- valign="top"
 +
| '''Summary'''
 +
|Returns the variable bindings of an existing job with the specified {{Code|$id}}. If no variables have been bound to this job, an empty map is returned.
 +
|}
 +
 +
==job:finished==
 +
 +
{| width='100%'
 +
|- valign="top"
 +
| width='120' | '''Signature'''
 +
|<pre>job:finished(
 +
  $id  as xs:string
 +
) as xs:boolean</pre>
 +
|- valign="top"
 +
| '''Summary'''
 +
|Indicates if the evaluation of an already running job with the specified {{Code|$id}} has finished. As the IDs of finished jobs will usually be discarded, unless caching is enabled, the function will also return <code>true</code> for unknown jobs.
 +
* <code>false</code> indicates that the job ID is scheduled, queued, or currently running.
 +
* <code>true</code> will be returned if the job has either finished, or if the ID is unknown (because the IDs of all finished jobs will not be cached).
 +
|}
 +
 +
==job:services==
 +
 +
{| width='100%'
 +
|- valign="top"
 +
| width='120' | '''Signature'''
 +
|<pre>job:services() as element(job)*</pre>
 +
|- valign="top"
 +
| '''Summary'''
 +
|Returns a list of all jobs that have been persistently registered as [[#Services|Services]].
 +
|- valign="top"
 +
| '''Errors'''
 +
|{{Error|services|#Errors}} Registered services cannot be parsed.<br/>
 
|}
 
|}
  
Line 258: Line 278:
 
! width="110"|Code
 
! width="110"|Code
 
|Description
 
|Description
|-
+
|- valign="top"
 
|{{Code|options}}
 
|{{Code|options}}
 
| The specified options are conflicting.
 
| The specified options are conflicting.
|-
+
|- valign="top"
 
|{{Code|id}}
 
|{{Code|id}}
| The specified id is invalid or has already been assigned.
+
| The specified ID is invalid or has already been assigned.
|-
+
|- valign="top"
 
|{{Code|overflow}}
 
|{{Code|overflow}}
 
| Too many queries or query results are queued.
 
| Too many queries or query results are queued.
|-
+
|- valign="top"
 
|{{Code|range}}
 
|{{Code|range}}
 
| A specified time or duration is out of range.
 
| A specified time or duration is out of range.
|-
+
|- valign="top"
 
|{{Code|running}}
 
|{{Code|running}}
 
| A query is still running.
 
| A query is still running.
|-
+
|- valign="top"
 
|{{Code|self}}
 
|{{Code|self}}
 
| The current job cannot be addressed.
 
| The current job cannot be addressed.
|-
+
|- valign="top"
 
|{{Code|service}}
 
|{{Code|service}}
 
| Registered services cannot be parsed, added or removed.
 
| Registered services cannot be parsed, added or removed.
|-
 
|{{Code|unknown}}
 
| The supplied query id is unknown or not available anymore.
 
 
|}
 
|}
  
 
=Changelog=
 
=Changelog=
 +
 +
;Version 10.0
 +
* Updated: Renamed from ''Jobs Module'' to ''Job Module''. The namespace URI has been updated as well.
 +
* Updated: {{Function||job:remove}} renamed from {{Code|jobs:stop}}.
 +
* Updated: {{Function||job:result}}: options argument added.
 +
* Added: {{Function||job:bindings}}
 +
 +
;Version 9.7
 +
* Updated: {{Function||job:result}}: return empty sequence if no result is cached.
 +
 +
;Version 9.5
 +
* Updated: {{Function||job:eval}}: integers added as valid start and end times.
 +
 +
;Version 9.4
 +
* Updated: {{Function||job:eval}}: option added for writing log entries.
 +
* Updated: {{Function||job:list-details}}: interval added.
 +
 +
;Version 9.2
 +
* Deleted: job:invoke (merged with {{Function||job:eval}})
 +
 +
;Version 9.1
 +
* Updated: {{Function||job:list-details}}: registration time added.
  
 
;Version 9.0
 
;Version 9.0
 
+
* Added: {{Function||job:invoke}}, [[#Services|Services]]
* Added: {{Function|Jobs|jobs:invoke}}, [[#Services|Services]]
 
  
 
;Version 8.6
 
;Version 8.6
 
+
* Updated: {{Function||job:eval}}: <code>id</code> option added.
* Updated: [[#jobs:eval|jobs:eval]]: <code>id</code> option added.
 
  
 
The module was introduced with Version 8.5.
 
The module was introduced with Version 8.5.

Latest revision as of 17:39, 1 December 2023

This XQuery Module provides functions for registering new query jobs and orchestrating existing jobs. Jobs can be queries, commands, operations performed by a database client, and HTTP requests.

Conventions[edit]

All functions in this module are assigned to the http://basex.org/modules/job namespace, which is statically bound to the job prefix. Errors will be bound to the same prefix.

Services[edit]

A job can be registered as service by supplying the service option to job:eval:

(: register job as service; will be run every day at 1 am :)
job:eval('db:drop("tmp")', (), map { 'id':'cleanup', 'start':'01:00:00', 'interval':'P1D', 'service': true() }),

(: list registered services :)
job:services(),
(: result: <job base-uri="..." id="cleanup" interval="P1D" start="01:00:00">db:drop("tmp")</job> :)

(: unregister job :)
job:remove('cleanup', map { 'service': true() })

Some more notes:

  • All job services will be scheduled for evaluation when the BaseX server or BaseX HTTP server is started.
  • If a job service is outdated (e.g. because a supplied end time has been exceeded), it will be removed from the jobs file at startup time.
  • The job definitions are stored in a jobs.xml file in the database directory. It can also be edited manually.

Executing Jobs[edit]

There are cases in which a client does not, or cannot, wait until a request is fully processed. The client may be a browser, which sends an HTTP request to the server to start another time-consuming query job. The functions in this section allow you to register new query jobs and access existing ones. Jobs can be executed immediately (i.e., as soon as a free slot is available) or scheduled for repeated execution. Each registered job gets a job ID, and the ID can be used to retrieve a query result, stop a job, or wait for its termination.

job:eval[edit]

Signature
job:eval(
  $query     as xs:anyAtomicType,
  $bindings  as map(*)?           := map { },
  $options   as map(*)?           := map { }
) as xs:string
Summary Schedules the evaluation a new query job for the supplied $query (of type xs:string, or of type xs:anyURI if points to a resource), and returns a job ID. The job will be queued until a free slot is available, and the query result can be cached. Queries can be updating, and variables and the context value can be declared via $bindings (see xquery:eval for more details). The following $options can be supplied:
  • cache: indicates if the query result will be cached or ignored (default: false):
    • The result will be cached in main-memory until it is fetched via job:result, or until CACHETIMEOUT is exceeded.
    • If the query raises an error, it will be cached and returned instead.
  • start: a dayTimeDuration, time, dateTime or integer can be specified to delay the execution of the query:
    • If a dayTimeDuration is specified, the query will be queued after the specified duration has passed. Examples of valid values are: P1D (1 day), PT5M (5 minutes), PT0.1S (100 ms). An error will be raised if a negative value is specified.
    • If a dateTime is specified, the query will be executed at this date. Examples for valid values are: 2018-12-31T23:59:59 (New Year's Eve 2018, close to midnight). An error will be raised if the specified time lies in the past.
    • If a time is specified, the query will be executed at this time of the day. Examples of valid times are: 02:00:00 (2am local time), 12:00:00Z (noon, UTC). If the time lies in the past, the query will be executed the next day.
    • An integer will be interpreted as minutes. If the specified number is greater than the elapsed minutes of the current hour, the query will be executed one hour later.
  • interval: a dayTimeDuration string can be specified to execute the query periodically. An error is raised if the specified interval is less than one second (PT1S). If the next scheduled call is due, and if a query with the same ID is still running, it will be skipped.
  • end: scheduling can be stopped after a given time or duration. The string format is the same as for start. An error is raised if the resulting end time is smaller than the start time.
  • base-uri: sets the base-uri property for the query. This URI will be used when resolving relative URIs, such as with fn:doc.
  • id: sets a custom job ID. The ID must not start with the standard job prefix, and it can only be assigned if no job with the same name exists.
  • service: additionally registers the job as service. Registered services must have no variable bindings.
  • log: writes the specified string to the database logs. Two log entries are stored, one at the beginning and another one after the execution of the job.
Errors overflow: Query execution is rejected because too many jobs are queued or being executed. CACHETIMEOUT can be decreased if the default setting is too restrictive.
range: A specified time or duration is out of range.
id: The specified ID is invalid or has already been assigned.
options: The specified options are conflicting.
Examples
  • Cache query result. The returned ID can be used to pick up the result with job:result:
job:eval("1+3", (), map { 'cache': true() })
  • A happy birthday mail will be sent at the given date:
job:eval("import module namespace mail='mail'; mail:send('Happy birthday!')",
  (), map { 'start': '2018-09-01T06:00:00' })}}
  • The following RESTXQ functions can be called to execute a query at 2 am every day. An ID will be returned by the first function, which can be used to stop the scheduler via the second function:
declare %rest:POST("{$query}") %rest:path('/start-scheduling') function local:start($query) {
  job:eval($query, (), map { 'start': '02:00:00', 'interval': 'P1D' })
};
declare %rest:path('/stop-scheduling/{$id}') function local:stop($id) {
  job:remove($id)
};
  • Query execution is scheduled for every second, and for 10 seconds in total. As the query itself will take 1.5 seconds, it will only be executed every second time:
job:eval("prof:sleep(1500)", (), map { 'interval': 'PT1S', 'end': 'PT10S' })
  • The query in the specified file will be evaluated once:
job:eval(xs:anyURI('cleanup.xq'))
  • The following expression, if stored in a file, will be evaluated every 5 seconds:
job:eval(
  static-base-uri(),
  map { },
  map { 'start': 'PT5S' }
)

job:result[edit]

Signature
job:result(
  $id       as xs:string,
  $options  as map(*)?    := map { }
) as item()*
Summary Returns the cached result of a job with the specified job $id:
  • If the original job has raised an error, the cached error will be raised instead.
  • The cached result or error will be dropped after it has been retrieved.
  • If the result has not been cached or if it has been dropped, an empty sequence is returned.

The following $options can be supplied:

  • keep: Keep the cached result or error after retrieval.
Examples
  • The following RESTXQ function will either return the result of a previously started job or raise an error:
declare %rest:path('/result/{$id}') function local:result($id) {
  job:result($id)
};
  • The following query demonstrates how the results of an executed query can be returned within the same query (see below why you should avoid this pattern in practice):
let $query := job:eval('(1 to 10000000)[. = 1]', map { }, map { 'cache': true() })
return (
  job:wait($query),
  job:result($query)
)

Queries of this kind can cause deadlocks! If the original query and the new query perform updates on the same database, the second query will only be run after the first one has been executed, and the first query will wait for the second query forever. You should resort to xquery:fork-join if you want to have full control on parallel query execution.

job:remove[edit]

Signature
job:remove(
  $id       as xs:string,
  $options  as map(*)?    := map { }
) as empty-sequence()
Summary Triggers the cancelation of a job with the specified $id, cancels a scheduled job or removes a cached result. Unknown IDs are ignored. All jobs are gracefully stopped; it is up to the process to decide when it is safe to shut down. The following $options can be supplied:
  • service: additionally removes the job from the job services list.
Examples
  • job:list()[. != job:current()] ! job:remove(.) stops and discards all jobs except for the current one.
  • job:remove(job:current()) interrupts the current job.

job:wait[edit]

Signature
job:wait(
  $id  as xs:string
) as empty-sequence()
Summary Waits for the completion of a job with the specified $id:
  • The function will terminate immediately if the job ID is unknown. This is the case if a future job has not been queued yet, or if the ID has already been discarded after job evaluation.
  • If the function is called with the ID of a queued job, or repeatedly executed job, it may stall and never terminate.
Errors self: The current job is addressed.

Listing Jobs[edit]

job:current[edit]

Signature
job:current() as xs:string
Summary Returns the ID of the current job.

job:list[edit]

Signature
job:list() as xs:string*
Summary Returns the IDs of all jobs that are currently registered. The list includes scheduled, queued, running, stopped, and finished jobs with cached results.
Examples job:list() returns the same job ID as job:current if no other job is registered.

job:list-details[edit]

Signature
job:list-details(
  $id  as xs:string  := ()
) as element(job)*
Summary Returns information on all jobs that are currently registered, or on a job with the specified $id (or an empty sequence if this job is not found). The list includes scheduled, queued, running jobs, and cached jobs. A string representation of the job, or its URI, will be returned as a value. The returned elements have additional attributes:
  • id: job ID
  • type: type of the job (command, query, REST, RESTXQ, etc.)
  • state: current state of the job: scheduled, queued, running, cached
  • user: user who started the job
  • duration: evaluation time (included if a job is running or if the result was cached)
  • start: next start of job (included if a job will be executed repeatedly)
  • time: time when job was registered
Examples job:list-details() returns information on the currently running job and possibly others:
<job id="job1" type="XQuery" state="running" user="admin" duration="PT0.001S">
  XQUERY job:list-details()
</job>

job:bindings[edit]

Signature
job:bindings(
  $id  as xs:string
) as map(*)
Summary Returns the variable bindings of an existing job with the specified $id. If no variables have been bound to this job, an empty map is returned.

job:finished[edit]

Signature
job:finished(
  $id  as xs:string
) as xs:boolean
Summary Indicates if the evaluation of an already running job with the specified $id has finished. As the IDs of finished jobs will usually be discarded, unless caching is enabled, the function will also return true for unknown jobs.
  • false indicates that the job ID is scheduled, queued, or currently running.
  • true will be returned if the job has either finished, or if the ID is unknown (because the IDs of all finished jobs will not be cached).

job:services[edit]

Signature
job:services() as element(job)*
Summary Returns a list of all jobs that have been persistently registered as Services.
Errors services: Registered services cannot be parsed.

Errors[edit]

Code Description
options The specified options are conflicting.
id The specified ID is invalid or has already been assigned.
overflow Too many queries or query results are queued.
range A specified time or duration is out of range.
running A query is still running.
self The current job cannot be addressed.
service Registered services cannot be parsed, added or removed.

Changelog[edit]

Version 10.0
  • Updated: Renamed from Jobs Module to Job Module. The namespace URI has been updated as well.
  • Updated: job:remove renamed from jobs:stop.
  • Updated: job:result: options argument added.
  • Added: job:bindings
Version 9.7
  • Updated: job:result: return empty sequence if no result is cached.
Version 9.5
  • Updated: job:eval: integers added as valid start and end times.
Version 9.4
Version 9.2
  • Deleted: job:invoke (merged with job:eval)
Version 9.1
Version 9.0
Version 8.6

The module was introduced with Version 8.5.