Global locks

> Note @reads="BEP" > > Is this correct and/or of any consequence?

>Variables will only be inlined at compile time, i.e., after the locks have

On 20 September 2017 at 18:27, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Andy, > > > Note @reads="BEP" > > > > Is this correct and/or of any consequence? > > This is correct: If the argument of fn:collection, fn:doc, etc. is > directly supplied as string argument, it is possible to statically > detect (at parse time) which database needs to be locked. Variables > will only be inlined at compile time, i.e., after the locks have > already been assigned. As a result, your first query will cause a > global lock. > > Currently, we only have one compilation step. In an ideal BaseX world, > we would have at least 2 steps: > > 1. Static compilation, in which a query would be typed and rewritten > without considering dynamic input (external variables, databases, …). > The result could be reused several times, and in your case, the > database name would have been inlined. > > 2. Dynamic compilation, which binds external variables (from clients, > RESTXQ calls, etc.), checks databases for index structures, etc. > > It might turn out that Step 1 is too expensive in order to be > evaluated before locking a query. On the other hand, static > compilation needs not be locked, it can always run in parallel with > queries that are currently evaluated. > > Does this answer your question? > Christian >

> bxcode:inline-variables($xqcode-uri) as xs:string > @returns xquery code that is the source code from $xqcode-uri but with > inlining "constant" variables and operations applied > > This could be used as a preprocessor. > Is this something that could be created from the bits and pieces in the > current BaseX Java code?

On 20 September 2017 at 23:28, Christian Grün <christian.gruen at gmail.com> wrote: > > bxcode:inline-variables($xqcode-uri) as xs:string > > @returns xquery code that is the source code from $xqcode-uri but with > > inlining "constant" variables and operations applied > > > > This could be used as a preprocessor. > > Is this something that could be created from the bits and pieces in the > > current BaseX Java code? > > This reminds me of the discussion with Omar [1]: There is currently no > way to generate a correct string representation for compiled code. In > the GUI Info View, we output a string representation of the optimized > query. In simple cases, this string is a valid and equivalent to the > original query string, but it’s fairly easy to generate invalid > results. > > If it’s an option for you to generate code in XQuery, something like > the following could be done: > > _ bxcode.xqm _______ > > module namespace bxcode = 'bxcode'; > declare function bxcode:bind-external-variables( > $uri as xs:string, > $vars as map(xs:string, xs:string) > ) as xs:string { > let $xqcode := unparsed-text($uri) > return fold-left( > map:keys($vars), $xqcode, function($string, $name) { > let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') > return $string > => replace('declare variable $' || $name || ' external;', '', 'q') > => replace('$' || $name, '``[' || $value || ']``', 'q') > } > ) > }; > > _ example.xq _______ > > declare variable $db external; > count(collection($db)) > > _ run-bxcode.xq _______ > > import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; > bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' }) > > _ output _______ > > count(collection(``[BEP]``)) > > Cheers, > Christian > > [1] https://www.mail-archive.com/basex-talk at mailman.uni- > konstanz.de/msg09863.html >

> The use of String Constructor does not avoid the global read lock

> <job id="job13" type="ScheduledXQuery" state="cached" user="admin" > duration="PT0.007S" reads="(global)" > writes="(none)">count(collection(``[BEP]``))</job> > > On 20 September 2017 at 23:28, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> > bxcode:inline-variables($xqcode-uri) as xs:string >> > @returns xquery code that is the source code from $xqcode-uri but with >> > inlining "constant" variables and operations applied >> > >> > This could be used as a preprocessor. >> > Is this something that could be created from the bits and pieces in the >> > current BaseX Java code? >> >> This reminds me of the discussion with Omar [1]: There is currently no >> way to generate a correct string representation for compiled code. In >> the GUI Info View, we output a string representation of the optimized >> query. In simple cases, this string is a valid and equivalent to the >> original query string, but it’s fairly easy to generate invalid >> results. >> >> If it’s an option for you to generate code in XQuery, something like >> the following could be done: >> >> _ bxcode.xqm _______ >> >> module namespace bxcode = 'bxcode'; >> declare function bxcode:bind-external-variables( >> $uri as xs:string, >> $vars as map(xs:string, xs:string) >> ) as xs:string { >> let $xqcode := unparsed-text($uri) >> return fold-left( >> map:keys($vars), $xqcode, function($string, $name) { >> let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') >> return $string >> => replace('declare variable $' || $name || ' external;', '', 'q') >> => replace('$' || $name, '``[' || $value || ']``', 'q') >> } >> ) >> }; >> >> _ example.xq _______ >> >> declare variable $db external; >> count(collection($db)) >> >> _ run-bxcode.xq _______ >> >> import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; >> bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' }) >> >> _ output _______ >> >> count(collection(``[BEP]``)) >> >> Cheers, >> Christian >> >> [1] >> https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg09863.html > >

On Thu, 21 Sep 2017 at 16:26, Christian Grün <christian.gruen at gmail.com> wrote: > > The use of String Constructor does not avoid the global read lock > > Good to know, thanks. It now does [1], provided that the constructor > consists of a single static strings. > > [1] http://files.basex.org/releases/latest/ > > > > > <job id="job13" type="ScheduledXQuery" state="cached" user="admin" > > duration="PT0.007S" reads="(global)" > > writes="(none)">count(collection(``[BEP]``))</job> > > > > On 20 September 2017 at 23:28, Christian Grün <christian.gruen at gmail.com > > > > wrote: > >> > >> > bxcode:inline-variables($xqcode-uri) as xs:string > >> > @returns xquery code that is the source code from $xqcode-uri but with > >> > inlining "constant" variables and operations applied > >> > > >> > This could be used as a preprocessor. > >> > Is this something that could be created from the bits and pieces in > the > >> > current BaseX Java code? > >> > >> This reminds me of the discussion with Omar [1]: There is currently no > >> way to generate a correct string representation for compiled code. In > >> the GUI Info View, we output a string representation of the optimized > >> query. In simple cases, this string is a valid and equivalent to the > >> original query string, but it’s fairly easy to generate invalid > >> results. > >> > >> If it’s an option for you to generate code in XQuery, something like > >> the following could be done: > >> > >> _ bxcode.xqm _______ > >> > >> module namespace bxcode = 'bxcode'; > >> declare function bxcode:bind-external-variables( > >> $uri as xs:string, > >> $vars as map(xs:string, xs:string) > >> ) as xs:string { > >> let $xqcode := unparsed-text($uri) > >> return fold-left( > >> map:keys($vars), $xqcode, function($string, $name) { > >> let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') > >> return $string > >> => replace('declare variable $' || $name || ' external;', '', > 'q') > >> => replace('$' || $name, '``[' || $value || ']``', 'q') > >> } > >> ) > >> }; > >> > >> _ example.xq _______ > >> > >> declare variable $db external; > >> count(collection($db)) > >> > >> _ run-bxcode.xq _______ > >> > >> import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; > >> bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' }) > >> > >> _ output _______ > >> > >> count(collection(``[BEP]``)) > >> > >> Cheers, > >> Christian > >> > >> [1] > >> > https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg09863.html > > > > >

On Fri, 8 Feb 2019 at 16:03, Andy Bunce <bunce.andy at gmail.com> wrote: > Using 9.1.2 > > let $d:=db:open(``[BEP]``) > return db:create(``[BEP-staging]``,$d,$d!base-uri(.)) > > The GUI query info reports: > - Read Locking: (none) > - Write Locking: BEP,BEP-staging > > I expected BEP-staging to appear only in the Read locking, as it does if I > remove the db:create call > Is this correct? > > /Andy > > > On Thu, 21 Sep 2017 at 16:26, Christian Grün <christian.gruen at gmail.com> > wrote: > >> > The use of String Constructor does not avoid the global read lock >> >> Good to know, thanks. It now does [1], provided that the constructor >> consists of a single static strings. >> >> [1] http://files.basex.org/releases/latest/ >> >> >> >> > <job id="job13" type="ScheduledXQuery" state="cached" user="admin" >> > duration="PT0.007S" reads="(global)" >> > writes="(none)">count(collection(``[BEP]``))</job> >> > >> > On 20 September 2017 at 23:28, Christian Grün < >> christian.gruen at gmail.com> >> > wrote: >> >> >> >> > bxcode:inline-variables($xqcode-uri) as xs:string >> >> > @returns xquery code that is the source code from $xqcode-uri but >> with >> >> > inlining "constant" variables and operations applied >> >> > >> >> > This could be used as a preprocessor. >> >> > Is this something that could be created from the bits and pieces in >> the >> >> > current BaseX Java code? >> >> >> >> This reminds me of the discussion with Omar [1]: There is currently no >> >> way to generate a correct string representation for compiled code. In >> >> the GUI Info View, we output a string representation of the optimized >> >> query. In simple cases, this string is a valid and equivalent to the >> >> original query string, but it’s fairly easy to generate invalid >> >> results. >> >> >> >> If it’s an option for you to generate code in XQuery, something like >> >> the following could be done: >> >> >> >> _ bxcode.xqm _______ >> >> >> >> module namespace bxcode = 'bxcode'; >> >> declare function bxcode:bind-external-variables( >> >> $uri as xs:string, >> >> $vars as map(xs:string, xs:string) >> >> ) as xs:string { >> >> let $xqcode := unparsed-text($uri) >> >> return fold-left( >> >> map:keys($vars), $xqcode, function($string, $name) { >> >> let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') >> >> return $string >> >> => replace('declare variable $' || $name || ' external;', '', >> 'q') >> >> => replace('$' || $name, '``[' || $value || ']``', 'q') >> >> } >> >> ) >> >> }; >> >> >> >> _ example.xq _______ >> >> >> >> declare variable $db external; >> >> count(collection($db)) >> >> >> >> _ run-bxcode.xq _______ >> >> >> >> import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; >> >> bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' }) >> >> >> >> _ output _______ >> >> >> >> count(collection(``[BEP]``)) >> >> >> >> Cheers, >> >> Christian >> >> >> >> [1] >> >> >> https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg09863.html >> > >> > >> >

On Fri, Feb 8, 2019 at 5:45 PM Andy Bunce <bunce.andy at gmail.com> wrote: > > sorry typo there, should be: > I expected BEP to appear only in the Read locking, as it does if I remove the db:create call > > > > On Fri, 8 Feb 2019 at 16:03, Andy Bunce <bunce.andy at gmail.com> wrote: >> >> Using 9.1.2 >> >> let $d:=db:open(``[BEP]``) >> return db:create(``[BEP-staging]``,$d,$d!base-uri(.)) >> >> The GUI query info reports: >> - Read Locking: (none) >> - Write Locking: BEP,BEP-staging >> >> I expected BEP-staging to appear only in the Read locking, as it does if I remove the db:create call >> Is this correct? >> >> /Andy >> >> >> On Thu, 21 Sep 2017 at 16:26, Christian Grün <christian.gruen at gmail.com> wrote: >>> >>> > The use of String Constructor does not avoid the global read lock >>> >>> Good to know, thanks. It now does [1], provided that the constructor >>> consists of a single static strings. >>> >>> [1] http://files.basex.org/releases/latest/ >>> >>> >>> >>> > <job id="job13" type="ScheduledXQuery" state="cached" user="admin" >>> > duration="PT0.007S" reads="(global)" >>> > writes="(none)">count(collection(``[BEP]``))</job> >>> > >>> > On 20 September 2017 at 23:28, Christian Grün <christian.gruen at gmail.com> >>> > wrote: >>> >> >>> >> > bxcode:inline-variables($xqcode-uri) as xs:string >>> >> > @returns xquery code that is the source code from $xqcode-uri but with >>> >> > inlining "constant" variables and operations applied >>> >> > >>> >> > This could be used as a preprocessor. >>> >> > Is this something that could be created from the bits and pieces in the >>> >> > current BaseX Java code? >>> >> >>> >> This reminds me of the discussion with Omar [1]: There is currently no >>> >> way to generate a correct string representation for compiled code. In >>> >> the GUI Info View, we output a string representation of the optimized >>> >> query. In simple cases, this string is a valid and equivalent to the >>> >> original query string, but it’s fairly easy to generate invalid >>> >> results. >>> >> >>> >> If it’s an option for you to generate code in XQuery, something like >>> >> the following could be done: >>> >> >>> >> _ bxcode.xqm _______ >>> >> >>> >> module namespace bxcode = 'bxcode'; >>> >> declare function bxcode:bind-external-variables( >>> >> $uri as xs:string, >>> >> $vars as map(xs:string, xs:string) >>> >> ) as xs:string { >>> >> let $xqcode := unparsed-text($uri) >>> >> return fold-left( >>> >> map:keys($vars), $xqcode, function($string, $name) { >>> >> let $value := replace($vars($name), ']``', '`{"]``"}`', 'q') >>> >> return $string >>> >> => replace('declare variable $' || $name || ' external;', '', 'q') >>> >> => replace('$' || $name, '``[' || $value || ']``', 'q') >>> >> } >>> >> ) >>> >> }; >>> >> >>> >> _ example.xq _______ >>> >> >>> >> declare variable $db external; >>> >> count(collection($db)) >>> >> >>> >> _ run-bxcode.xq _______ >>> >> >>> >> import module namespace bxcode = 'bxcode' at 'bxcode.xqm'; >>> >> bxcode:bind-external-variables('query.xq', map { 'db': 'BEP' }) >>> >> >>> >> _ output _______ >>> >> >>> >> count(collection(``[BEP]``)) >>> >> >>> >> Cheers, >>> >> Christian >>> >> >>> >> [1] >>> >> https://www.mail-archive.com/basex-talk at mailman.uni-konstanz.de/msg09863.html >>> > >>> >

> From: Christian Grün <christian.gruen at gmail.com> > Subject: Re: [basex-talk] Global locks > Date: 11 February 2019 at 12:27:37 GMT > To: Andy Bunce <bunce.andy at gmail.com> > Cc: "basex-talk at mailman.uni-konstanz.de" <basex-talk at mailman.uni-konstanz.de> > > > Hi Andy, > > The current behavior is correct indeed – but it might not be what one > expects. Currently, we are… > > a) collecting all static database references in the query and > b) assigning either read or write locks to these databases, depending > if the overall query is updating or not. > > The reason is that it’s often tricky to determine statically (i.e., > while parsing the query and before compiling and optimizing it) which > databases will be accessed for read or write operations without > analyzing the query in more detail. An arbitrary example: > > let $db := db:open('db1') > return insert node <new/> into $db/* > > We would need to follow the variable reference in order to find out if > db1 will be updated. In simple queries such as yours, however, this > might be possible; I’ll have some more thoughts on that. > > Cheers > Christian

On Mon, Feb 11, 2019 at 3:22 PM James Ball <basex-talk at jamesball.co.uk> wrote: > > > I think I’m probably doing something similar on a current project with but with some careful query writes and use of jobs I’ve been able to keep everything running within acceptable time. > > However while optimising my code it did get me thinking about possible improvements. Clearly it’s difficult (or even impossible) to determine all the databases that will be used ahead of running the query. But I wonder how many times the calling function already knows? > > Then I thought of the (# db:enforceindex #) that was introduced for cases where the query writer knows that the databases will have indexes. I wondered if something similar might be possible for databases. > > A pragma or a function wrapper that would allow the name of a database (or databases) to be supplied and that would restrict access only to that database for the rest of the query. Returning an error if the query tries to address another. > > I’m sure this wouldn’t be simple - but might be easier and more reliable than trying to find more optimisations to the locking algorithm. > > Just thinking aloud… > > Kindest regards, James > > From: Christian Grün <christian.gruen at gmail.com> > Subject: Re: [basex-talk] Global locks > Date: 11 February 2019 at 12:27:37 GMT > To: Andy Bunce <bunce.andy at gmail.com> > Cc: "basex-talk at mailman.uni-konstanz.de" <basex-talk at mailman.uni-konstanz.de> > > > Hi Andy, > > The current behavior is correct indeed – but it might not be what one > expects. Currently, we are… > > a) collecting all static database references in the query and > b) assigning either read or write locks to these databases, depending > if the overall query is updating or not. > > The reason is that it’s often tricky to determine statically (i.e., > while parsing the query and before compiling and optimizing it) which > databases will be accessed for read or write operations without > analyzing the query in more detail. An arbitrary example: > > let $db := db:open('db1') > return insert node <new/> into $db/* > > We would need to follow the variable reference in order to find out if > db1 will be updated. In simple queries such as yours, however, this > might be possible; I’ll have some more thoughts on that. > > Cheers > Christian > >

> On 13 Feb 2019, at 16:38, Christian Grün <christian.gruen at gmail.com> wrote: > Manual locks could either be assigned globally or > within a query: > > (# db:manuallock #) { > (# basex:write-lock BEP-Staging #) { > (# basex:read-lock BEP #) { > let $d:=db:open('BEP') > return db:create('BEP-staging',$d,$d!base-uri(.)) > } > } > } > > I wouldn’t call this syntax particularly appealing

> Would this work where the database names are only known at run time?

>If manual locking is enabled, an error will be raised if a database is

On Wed, 13 Feb 2019 at 16:56, Christian Grün <christian.gruen at gmail.com> wrote: > > Would this work where the database names are only known at run time? > > Right now, our locking is based on static code analysis; runtime > information won’t be included. In the longer term, we would like to > extend our compiler to perform multiple steps: > > 1. Static parsing > 2. Static optimizations (pre-evaluate 1+2 to 3, etc.) > 3. Dynamic optimizations (reoptimize the query with query parameters > and other context information) > 4. Physical optimizations (open databases, utilize available index > structures) > > Queries that have been parsed and statically compiled (Step 1+2) could > be stored as compiled queries, or kept in main-memory and duplicated > before dynamic optimizations take place. This could further reduce > access times in RESTXQ applications – and it would allow us to improve > locking, as we could do the lock check after Step 3. >

> Is it required to specify all databases accessed within the manualock scope or could it work just specifying specific databases as readonly? > And the execution would error if these were violated.