Jobs and file modules

On Sat, Feb 6, 2021 at 10:00 AM Tim Thompson <timathom at gmail.com> wrote: > Hello, > > I was experimenting with the jobs module and wondering why there's a > difference between *file:write() *and *file:write-text()* in the query > below: > > for $i in 1 to 5 > return ( > jobs:eval(' > declare variable $iter external; > file:write-text("~/Desktop/job"|| $iter ||".txt", (prof:sleep(5000), > string(current-dateTime()))) > ', map {"iter": $i} > ) > ) > > With *file:write-text()*, if I don't wrap *current-dateTime()* in > *string()*, nothing is written and no files are created. With > *file:write()*, text is always written and files always get created. > > Thanks! > Tim > > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library > >

> Ah, never mind. When I run the file:write-text() without jobs:eval(), I get an error, "Cannot convert xs:dateTime to xs:string." Is it possible to return the error from a job call?

On Sat, Feb 6, 2021 at 5:22 PM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Tim, > > file:write uses the default W3 serialization method "XML". This means > that the standard entities (&, <, etc.) will be encoded. This can be > circumvented by using the 'text' output method… > > file:write(..., ..., map { 'method': 'text' }) > > …or file:write-text. > > In BaseX, we introduced our own serialization method 'basex', which > serializes strings as strings and basex64 and hex data as bytes. With > this method (if it had been part of the official standard), > file:write-text and file:write-binary could actually have been > dropped. > > > Ah, never mind. When I run the file:write-text() without jobs:eval(), I > get an error, "Cannot convert xs:dateTime to xs:string." Is it possible to > return the error from a job call? > > You can cache the result of a query… > > let $job-id := jobs:eval(..., ..., map { 'cache': true() }) > > …and retrieve the result or the error with jobs:result($job-id). > > Hope this helps, > Christian >

On Wed, Feb 10, 2021 at 1:56 AM Tim Thompson <timathom at gmail.com> wrote: > > Thank you, Christian, for the detailed explanation! > > One more question, if I may. Is it possible to run updating jobs on different databases in parallel? Or can database update operations only be run sequentially, one db at a time? I have a query that calls a function to perform a series of operations: > > for $i in (0 to 9) > return ( > jobs:eval(' > declare variable $iter external; > local:add-uris("marc.exp.20210115."||$iter) > ', map {"iter": $i}) > ) > > The function: > > opens a database > iterates through its records > performs lookups against an index > inserts any matches into the database > calls file:append-text-lines() to write the results of the lookups > > Based on some simple tests, it doesn't seem possible to run the jobs in parallel, but I thought I would ask--to see whether there was something I was missing. > > Thanks again, > Tim

On Wed, Feb 10, 2021 at 3:27 AM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Tim, > > Updates can be run in parallel if the name of the database is directly > specified in the query [1]: > > jobs:eval('delete node db:open("db1")//abc'), > jobs:eval('delete node db:open("db2")//def') > > In a future version of BaseX, we might split up our compilation phase > into multiple ones. After this, we could statically detect that a > passed on variable will be the name of a database. > > Until then, you could try to build a query string that included > hard-coded database names. > > Hope this helps, > Christian > > [1] https://docs.basex.org/wiki/Transaction_Management#XQuery > > > > On Wed, Feb 10, 2021 at 1:56 AM Tim Thompson <timathom at gmail.com> wrote: > > > > Thank you, Christian, for the detailed explanation! > > > > One more question, if I may. Is it possible to run updating jobs on > different databases in parallel? Or can database update operations only be > run sequentially, one db at a time? I have a query that calls a function to > perform a series of operations: > > > > for $i in (0 to 9) > > return ( > > jobs:eval(' > > declare variable $iter external; > > local:add-uris("marc.exp.20210115."||$iter) > > ', map {"iter": $i}) > > ) > > > > The function: > > > > opens a database > > iterates through its records > > performs lookups against an index > > inserts any matches into the database > > calls file:append-text-lines() to write the results of the lookups > > > > Based on some simple tests, it doesn't seem possible to run the jobs in > parallel, but I thought I would ask--to see whether there was something I > was missing. > > > > Thanks again, > > Tim >

> Is it possible to put updating expressions in a library module function (with the name of the database hard coded) and then call from the function within jobs:eval() in a main module?

On Tue, Feb 16, 2021 at 3:21 PM Tim Thompson <timathom at gmail.com> wrote: > > Yes, the insert operations also involve lookups against a couple of full-text indexes (using ft:search()). The entries in the full-text indexes include corresponding unique identifiers (URIs). The query: > > iterates through a set of library catalog records > filters on certain elements > runs full-text lookups on the text (which are name or subject strings) > inserts a new element containing the URI from a match in the index. > Logs the result of the each lookup (using file:append-text-lines()) > > > I'm guessing that the index access is what slows things down the most. What would be the approach for combining updates? > > Thank you, > Tim > > > -- > Tim A. Thompson > Discovery Metadata Librarian > Yale University Library > > > > On Tue, Feb 16, 2021 at 4:53 AM Christian Grün <christian.gruen at gmail.com> wrote: >> >> Hi Tim, >> >> > Is it possible to put updating expressions in a library module function (with the name of the database hard coded) and then call from the function within jobs:eval() in a main module? >> >> Yes, it should be. Here’s a little example (create database 'x' first): >> >> lib.xqm: >> module namespace lib = 'lib'; >> declare %updating function lib:delete() { >> prof:sleep(1000), >> delete node db:open('x')//x >> }; >> >> query.xq: >> let $id := jobs:eval(" >> import module namespace lib='lib' at 'lib.xqm'; >> lib:delete() >> ") >> return (prof:sleep(500), jobs:list-details($id)) >> >> The query result looks something like this: >> >> <job id="job29" type="QueryJob" state="running" user="admin" >> duration="PT0.498S" reads="(none)" writes="x" >> time="2021-02-16T10:40:57.605+01:00">import module namespace lib='lib' >> at 'lib.xqm'; lib:delete()</job> >> >> If the name of the database is found in the "writes" attribute, you’ll >> know that only this database will be write-locked. >> >> However, please note that, due to random file access patterns, >> parallel writes are often slower than consecutive ones (even with >> SSDs), so my guess is that you won’t save a lot. >> >> 48 hours sounds a lot indeed. It’s usually much, much faster to run a >> single XQuery expression that performs 1000 updates than running 1000 >> independent queries. Could you give us more information on the insert >> operations you need to perform? Is there any chance to combine >> updates? >> >> Best, >> Christian