Feature Request: serializable functions
Wednesday, 5 November 2014 · 8 messages · last reply 7 Nov 2014
ML
Marco Lettere5 Nov 2014 15:23 #
Hello all,
just out of curiosity, would it be too hard to have a serialization tool for functions that does more or less the inverse of
inspect:functions i. e. given a function item returns the full function code?
The use case is that I want to send out a client:execute request that is a XQuery script where functions can be attached at runtime selecting them out of a "toolbox" xqm module. And of course I don't want to manually parse the xqm to cut out the required function which could be done but is not very elegant.
Example:declare function local:f($i as xs:integer) as xs:integer{ $i + 1 };
inspect:serialize(local:f#1)
would yield the string "declare function local:f($i as xs:integer) as
xs:integer{ $i + 1 };"I've seen that even if it is not possible to serialize or cast to string a function item, if the function is assigned to a map, calling map:serialize yields something similar to the full function code. So I thought maybe the serialization mechanism is already in place and would just need a bit of generalization.
Regards,
Marco.
CG
Christian Grün6 Nov 2014 14:26 #
Hi Marco,
You may be surprised (or not) that I thought about a similar functionality. Indeed I wondered if we could allow function items as input for
client:query and xquery:evaluate.
I like your idea of providing an explicit serialization function. For several reasons, we cannot guarantee that the serialized
representation of an XQuery expression will always be executable. In some case, it's even not possible because the optimizer creates new expressions that have no counterpart in the original XQuery language. If we provide an explicit function for serializing function code, however, we'd be able to point this out.
I have added a new GitHub issue [1].
Christian
[1] https://github.com/BaseXdb/basex/issues/1021Quoted text · Marco Lettere · 27 lines
On Wed, Nov 5, 2014 at 3:23 PM, Marco Lettere <marco.lettere at dedalus.eu> wrote: > Hello all, > just out of curiosity, would it be too hard to have a serialization tool for > functions that does more or less the inverse of inspect:functions i. e. > given a function item returns the full function code? > The use case is that I want to send out a client:execute request that is a > XQuery script where functions can be attached at runtime selecting them out > of a "toolbox" xqm module. And of course I don't want to manually parse the > xqm to cut out the required function which could be done but is not very > elegant. > > Example: > > declare function local:f($i as xs:integer) as xs:integer{ $i + 1 }; > > inspect:serialize(local:f#1) > > would yield the string "declare function local:f($i as xs:integer) as > xs:integer{ $i + 1 };" > > I've seen that even if it is not possible to serialize or cast to string a > function item, if the function is assigned to a map, calling map:serialize > yields something similar to the full function code. So I thought maybe the > serialization mechanism is already in place and would just need a bit of > generalization. > Regards, > Marco.
CG
Christian Grün6 Nov 2014 15:20 #
Hi Carl,
In the BaseX query parser, tokenization and parsing is a joint operation. in the past, we thought about splitting these steps, as the tokens could also be used in the GUI to provide better highlighting. However, we'd probably need some more coders interested in making that happen..
However, thanks for the suggestion,
Christian
Quoted text · Carl Leitner · 72 lines
On Thu, Nov 6, 2014 at 3:12 PM, Carl Leitner <litlfred at gmail.com> wrote: > On a somewhat related note, I assume that when compiling xquery you are > creating some kind of token stream. Would it be possible to add a function > like > xquery:tokenize ($query as xs:string) as item()* > which takes as input an xquery string and output the token string. > > Presumably this could be another route used to meet the use case discussed > below to select functions from a toolbox module. > > Here is an example of similar functionality in PHP: > http://php.net/manual/en/function.token-get-all.php > > Cheers, > -carl > > > On Nov 6, 2014, at 8:26 AM, Christian Grün <christian.gruen at gmail.com> > wrote: > > Hi Marco, > > You may be surprised (or not) that I thought about a similar > functionality. Indeed I wondered if we could allow function items as > input for client:query and xquery:evaluate. > > I like your idea of providing an explicit serialization function. For > several reasons, we cannot guarantee that the serialized > representation of an XQuery expression will always be executable. In > some case, it's even not possible because the optimizer creates new > expressions that have no counterpart in the original XQuery language. > If we provide an explicit function for serializing function code, > however, we'd be able to point this out. > > I have added a new GitHub issue [1]. > > Christian > > [1] https://github.com/BaseXdb/basex/issues/1021 > > > On Wed, Nov 5, 2014 at 3:23 PM, Marco Lettere <marco.lettere at dedalus.eu> > wrote: > > Hello all, > just out of curiosity, would it be too hard to have a serialization tool for > functions that does more or less the inverse of inspect:functions i. e. > given a function item returns the full function code? > The use case is that I want to send out a client:execute request that is a > XQuery script where functions can be attached at runtime selecting them out > of a "toolbox" xqm module. And of course I don't want to manually parse the > xqm to cut out the required function which could be done but is not very > elegant. > > Example: > > declare function local:f($i as xs:integer) as xs:integer{ $i + 1 }; > > inspect:serialize(local:f#1) > > would yield the string "declare function local:f($i as xs:integer) as > xs:integer{ $i + 1 };" > > I've seen that even if it is not possible to serialize or cast to string a > function item, if the function is assigned to a map, calling map:serialize > yields something similar to the full function code. So I thought maybe the > serialization mechanism is already in place and would just need a bit of > generalization. > Regards, > Marco. > >
CG
One more thing we have to be aware of: In XQuery, we have lots of different function items. I wonder if we should support all of these?
declare function local:f() { 123 };
inspect:serialize(map {}),
inspect:serialize(local:f#0),
inspect:serialize(substring#2),
inspect:serialize(function() {1})Quoted text · Marco Lettere · 27 lines
On Wed, Nov 5, 2014 at 3:23 PM, Marco Lettere <marco.lettere at dedalus.eu> wrote: > Hello all, > just out of curiosity, would it be too hard to have a serialization tool for > functions that does more or less the inverse of inspect:functions i. e. > given a function item returns the full function code? > The use case is that I want to send out a client:execute request that is a > XQuery script where functions can be attached at runtime selecting them out > of a "toolbox" xqm module. And of course I don't want to manually parse the > xqm to cut out the required function which could be done but is not very > elegant. > > Example: > > declare function local:f($i as xs:integer) as xs:integer{ $i + 1 }; > > inspect:serialize(local:f#1) > > would yield the string "declare function local:f($i as xs:integer) as > xs:integer{ $i + 1 };" > > I've seen that even if it is not possible to serialize or cast to string a > function item, if the function is assigned to a map, calling map:serialize > yields something similar to the full function code. So I thought maybe the > serialization mechanism is already in place and would just need a bit of > generalization. > Regards, > Marco.
ML
Marco Lettere6 Nov 2014 17:26 #
If I had to decide, and luckily I don't have to ;-), I would skip inbuilt functions.
According to the described usecase the reason is that they are supposed to be "natively available" at the executor side. So they could be written in the code directly or wrapped in a user function.
From a more general viewpoint because they're supposed to be native and thus could possibly be implemented at a lower level (rts) which would mean that there is possibly no Xquery code to serialize.
Thus out of your list my choice would fall on
inspect:serialize(local:f#0)
inspect:serialize(function() {1})But honestly I don't know how hard it is to make the function selectively applicable to only user functions and anonymous functions. So please consider this to be just loud thinking.
Cheers,
Marco.
Quoted text · Christian Grün · 39 lines
On 06/11/2014 15:47, Christian Grün wrote: > One more thing we have to be aware of: In XQuery, we have lots of > different function items. I wonder if we should support all of these? > > declare function local:f() { 123 }; > inspect:serialize(map {}), > inspect:serialize(local:f#0), > inspect:serialize(substring#2), > inspect:serialize(function() {1}) > > > > On Wed, Nov 5, 2014 at 3:23 PM, Marco Lettere <marco.lettere at dedalus.eu> wrote: >> Hello all, >> just out of curiosity, would it be too hard to have a serialization tool for >> functions that does more or less the inverse of inspect:functions i. e. >> given a function item returns the full function code? >> The use case is that I want to send out a client:execute request that is a >> XQuery script where functions can be attached at runtime selecting them out >> of a "toolbox" xqm module. And of course I don't want to manually parse the >> xqm to cut out the required function which could be done but is not very >> elegant. >> >> Example: >> >> declare function local:f($i as xs:integer) as xs:integer{ $i + 1 }; >> >> inspect:serialize(local:f#1) >> >> would yield the string "declare function local:f($i as xs:integer) as >> xs:integer{ $i + 1 };" >> >> I've seen that even if it is not possible to serialize or cast to string a >> function item, if the function is assigned to a map, calling map:serialize >> yields something similar to the full function code. So I thought maybe the >> serialization mechanism is already in place and would just need a bit of >> generalization. >> Regards, >> Marco.
CG
To complement this: In a straightforward implementation, the following query…
declare function local:f() { 123 };
inspect:serialize(map {}),
inspect:serialize(local:f#0),
inspect:serialize(substring#2),
inspect:serialize(function() {1})…would yield the following result:
map { }
function() as item()* { local:f() }
function($string_0 as xs:string?, $start_1 as xs:double) as
xs:string { fn:substring($string_0, $start_1) }
function() as xs:integer { 1 }AB
Andy Bunce6 Nov 2014 22:18 #
Quoted text · Christian Grün · 1 line
>In some case, it's even not possible because the optimizer creates new
expressions that have no counterpart in the original XQuery language
Do you mean they invoke internal BaseX functions? Also to help me understand what BaseX optimization involves:
If these serializations are post optimization does mean they MAY embody assumptions about the context that are only valid at the moment they were made? Or could one expect to be able to deserialize them later and always get the same answers the original xquery would generate?
/Andy
Quoted text · Christian Grün · 18 lines
On 6 November 2014 16:34, Christian Grün <christian.gruen at gmail.com> wrote: > To complement this: In a straightforward implementation, the following > query… > > declare function local:f() { 123 }; > inspect:serialize(map {}), > inspect:serialize(local:f#0), > inspect:serialize(substring#2), > inspect:serialize(function() {1}) > > …would yield the following result: > > map { } > function() as item()* { local:f() } > function($string_0 as xs:string?, $start_1 as xs:double) as > xs:string { fn:substring($string_0, $start_1) } > function() as xs:integer { 1 } >
CG
Christian Grün7 Nov 2014 17:36 #
Hi Andy,
Quoted text · Andy Bunce · 1 line
> Do you mean they invoke internal BaseX functions?
Yes, exactly. One example are range checks. The following query…
<x>123</x>[text() > 1 and text() < 400]…will result in the following optimized query string:
element x { ("123") }[1.0 < text() < 400.0]
The query plan looks as follows:
<CmpR min="1.0" max="400.0">
<CachedPath>
<IterStep axis="child" test="text()"/>
</CachedPath>
</CmpR>Quoted text · Andy Bunce · 3 lines
> If these serializations are post optimization does mean they MAY embody > assumptions about the context that are only valid at the moment they were > made?
They will absolutely do so. The following query…
map:serialize(map{
'x': try {
db:open('i-do-not-exist')
} catch * { '-' }
})…result in { "x": "-" } (if "i-do-not-exist" does not exist).
If we decide to introduce a function like inspect:serialize, we should by all means stress that the output is no equivalent representation of the input function, and point out that it will be the compiled query result in the given context. As the
inspect:function would always be evaluated at runtime, we don't have a representation of the original query anymore (it's simply not required anywhere else), so this is actually the only thing we can currently offer at this stage.
Thanks for asking,
ChristianWritten for BaseX 7.9. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery