Main Page » Mailing List » February 2014

Performance Degraded In function-lookup

Friday, 21 February 2014 · 7 messages · last reply 24 Feb 2014
CL
Carl Leitner21 Feb 2014 18:06 #
Hi, I noticed a significant performance degradation in the function-lookup method introduced at: https://github.com/BaseXdb/basex/commit/4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 Where as at: b8b3dd2bee205303aeab3e8a0967b371a2ad7ef7 I have:
Quoted text · 1 line
> xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return ()
Query executed in 22.57 ms. At 4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 I have:
Quoted text · 1 line
>xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; ()
Query executed in 17.48 ms.
Quoted text · 1 line
>xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= xs:QName('csd_bsq:facility_search') return ()
Query executed in 20.44 ms.
Quoted text · 1 line
>xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return ()
Query executed in 716.18 ms. Any suggestions on how to avoid this? Thanks so much. Cheers, -carl
CG
Christian Grün22 Feb 2014 17:24 #
Hi Carl, it would be generally possible to optimize the current implementation of function-lookup and related reflective functions, but I think it will take us some more time. Instead, I rather recommend you to use function items, because these will always give you better performance and are less error-prone (they can be optimized much better than function calls based on string references). I'm not sure how your complete code looks like (and it would probably be too complex to serve as example here), but the following code might give you a hint how this could look like:
  let $add := function($a) { $a + $a }
  let $multiply := function($a) { $a * 2 }
  for $f in ($add, $multiply)
  return $f(2)
If you have already worked with function items... Sorry for teaching ;) Christian
Quoted text · Carl Leitner · 30 lines
On Fri, Feb 21, 2014 at 5:06 PM, Carl Leitner <litlfred at ibiblio.org> wrote: > Hi, > I noticed a significant performance degradation in the function-lookup method introduced at: > https://github.com/BaseXdb/basex/commit/4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 > > Where as at: b8b3dd2bee205303aeab3e8a0967b371a2ad7ef7 I have: >> xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return () > Query executed in 22.57 ms. > > At 4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 I have: >>xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; () > Query executed in 17.48 ms. >>xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= xs:QName('csd_bsq:facility_search') return () > Query executed in 20.44 ms. >>xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return () > Query executed in 716.18 ms. > > Any suggestions on how to avoid this? > > Thanks so much. > > Cheers, > -carl > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
CL
Carl Leitner22 Feb 2014 20:09 #
Dear Christian, Thanks for your suggestion. With the function-lookup, my essential requirement is to process an incoming XML message containing a UUID and execute a particular function based on that UUID. As I would ideally like to be able to manage the associations between UUID and function names dynamically, I do not see immediately how to apply your suggestion. There is a work-around along the lines of:
let $function_name := local:lookup-function-name($uuid)
let $def := concat(
	"import module namespace ‘blah’  at ‘blah.xml’;
	declare variable $a  external;
        declare variable $b  external;
       ”, $function_name , “($a,$b)”)
return xquery:evalulate($def)
Although this is not particularly ideal as it uses an eval/evaluate statement (evil [1]). Moreover, I was trying to replace an eval/evaluate statements with a function-lookup, as I assumed that this would be a more proper way to handle it, when I ran into the performance issue. Besides reverting back to the eval statements, only one other option comes to mind: for a given set of pairs ($uuid,$function_name) generate the needed xquery to invoke the functions with a switch statement. If I wanted to maintain at least a dynamic nature, I suppose I could write this generated xquery to a file and use the repository module [2]. There are some immediate drawbacks that I see to this though. So maybe my question is, how evil is evil? If the eval’ed code is largely just calling existing functions, will there be a big performance impact? If there are plans for optimize function-lookup in the future, I suppose I could live with a bit of short-term evil. Cheers, -carl [1] https://mailman.uni-konstanz.de/pipermail/basex-talk/2013-November/005707.html [2] http://docs.basex.org/wiki/Repository_Module
Quoted text · Christian Grün · 57 lines
On Feb 22, 2014, at 11:24 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Carl, > > it would be generally possible to optimize the current implementation > of function-lookup and related reflective functions, but I think it > will take us some more time. > > Instead, I rather recommend you to use function items, because these > will always give you better performance and are less error-prone (they > can be optimized much better than function calls based on string > references). > > I'm not sure how your complete code looks like (and it would probably > be too complex to serve as example here), but the following code might > give you a hint how this could look like: > > let $add := function($a) { $a + $a } > let $multiply := function($a) { $a * 2 } > for $f in ($add, $multiply) > return $f(2) > > If you have already worked with function items... Sorry for teaching ;) > > Christian > > > > On Fri, Feb 21, 2014 at 5:06 PM, Carl Leitner <litlfred at ibiblio.org> wrote: >> Hi, >> I noticed a significant performance degradation in the function-lookup method introduced at: >> https://github.com/BaseXdb/basex/commit/4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 >> >> Where as at: b8b3dd2bee205303aeab3e8a0967b371a2ad7ef7 I have: >>> xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return () >> Query executed in 22.57 ms. >> >> At 4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 I have: >>> xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; () >> Query executed in 17.48 ms. >>> xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= xs:QName('csd_bsq:facility_search') return () >> Query executed in 20.44 ms. >>> xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return () >> Query executed in 716.18 ms. >> >> Any suggestions on how to avoid this? >> >> Thanks so much. >> >> Cheers, >> -carl >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
CL
Carl Leitnerin reply to Christian Grün, 22 Feb 2014, 17:2422 Feb 2014 20:10 #
Dear Christian, Thanks for your suggestion. With the function-lookup, my essential requirement is to process an incoming XML message containing a UUID and execute a particular function based on that UUID. As I would ideally like to be able to manage the associations between UUID and function names dynamically, I do not see immediately how to apply your suggestion. There is a work-around along the lines of:
let $function_name := local:lookup-function-name($uuid)
let $def := concat(
	"import module namespace ‘blah’  at ‘blah.xml’;
	declare variable $a  external;
        declare variable $b  external;
       ”, $function_name , “($a,$b)”)
return xquery:evalulate($def)
Although this is not particularly ideal as it uses an eval/evaluate statement (evil [1]). Moreover, I was trying to replace an eval/evaluate statements with a function-lookup, as I assumed that this would be a more proper way to handle it, when I ran into the performance issue. Besides reverting back to the eval statements, only one other option comes to mind: for a given set of pairs ($uuid,$function_name) generate the needed xquery to invoke the functions with a switch statement. If I wanted to maintain at least a dynamic nature, I suppose I could write this generated xquery to a file and use the repository module [2]. There are some immediate drawbacks that I see to this though. So maybe my question is, how evil is evil? If the eval’ed code is largely just calling existing functions, will there be a big performance impact? If there are plans for optimize function-lookup in the future, I suppose I could live with a bit of short-term evil. Cheers, -carl [1] https://mailman.uni-konstanz.de/pipermail/basex-talk/2013-November/005707.html [2] http://docs.basex.org/wiki/Repository_Module
Quoted text · Christian Grün · 57 lines
On Feb 22, 2014, at 11:24 AM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Carl, > > it would be generally possible to optimize the current implementation > of function-lookup and related reflective functions, but I think it > will take us some more time. > > Instead, I rather recommend you to use function items, because these > will always give you better performance and are less error-prone (they > can be optimized much better than function calls based on string > references). > > I'm not sure how your complete code looks like (and it would probably > be too complex to serve as example here), but the following code might > give you a hint how this could look like: > > let $add := function($a) { $a + $a } > let $multiply := function($a) { $a * 2 } > for $f in ($add, $multiply) > return $f(2) > > If you have already worked with function items... Sorry for teaching ;) > > Christian > > > > On Fri, Feb 21, 2014 at 5:06 PM, Carl Leitner <litlfred at ibiblio.org> wrote: >> Hi, >> I noticed a significant performance degradation in the function-lookup method introduced at: >> https://github.com/BaseXdb/basex/commit/4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 >> >> Where as at: b8b3dd2bee205303aeab3e8a0967b371a2ad7ef7 I have: >>> xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return () >> Query executed in 22.57 ms. >> >> At 4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 I have: >>> xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; () >> Query executed in 17.48 ms. >>> xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= xs:QName('csd_bsq:facility_search') return () >> Query executed in 20.44 ms. >>> xquery import module namespace csd_bsq = "https://github.com/his-interop/openinfoman/csd_bsq" at "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return () >> Query executed in 716.18 ms. >> >> Any suggestions on how to avoid this? >> >> Thanks so much. >> >> Cheers, >> -carl >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
CG
Christian Grünin reply to Carl Leitner, 22 Feb 2014, 20:0922 Feb 2014 22:12 #
Hi Carl,
Quoted text · Carl Leitner · 3 lines
> As I would ideally like to be able to manage the associations > between UUID and function names dynamically, I do not see immediately how to > apply your suggestion.
Would sth. like the following code help?
  declare variable $FUNCTIONS := map {
    '69bfbe73-23ac-4033-b60b-6672fd67b937': function() { 12345 },
    '0d22d6a8-035b-46bf-a396-00471c840a08': function() { 67890 }
  };

  let $uuid := '69bfbe73-23ac-4033-b60b-6672fd67b937'
  return $FUNCTIONS($uuid)()
You could as well move this map, or the functions declared inside, into another module, etc.
Quoted text · Carl Leitner · 1 line
> So maybe my question is, how evil is evil?
One drawback of using xquery:eval (and xquery:evaluate) is that your code will only be evaluated at runtime, which is why the compiler has no chance to detect syntax and type errors at compile time. This is also the reason why updates operations are not allowed inside xquery:eval, because an updating eval query may change nodes that are referenced in the calling query. However, if you know exactly what your code is doing, the xquery:eval function won't cause any trouble (this is also the reason why we are offering the functions at all... And it may even be officially introduced with XQuery 3.1). Christian
Quoted text · Christian Grün · 84 lines
> On Feb 22, 2014, at 11:24 AM, Christian Grün <christian.gruen at gmail.com> > wrote: > > Hi Carl, > > it would be generally possible to optimize the current implementation > of function-lookup and related reflective functions, but I think it > will take us some more time. > > Instead, I rather recommend you to use function items, because these > will always give you better performance and are less error-prone (they > can be optimized much better than function calls based on string > references). > > I'm not sure how your complete code looks like (and it would probably > be too complex to serve as example here), but the following code might > give you a hint how this could look like: > > let $add := function($a) { $a + $a } > let $multiply := function($a) { $a * 2 } > for $f in ($add, $multiply) > return $f(2) > > If you have already worked with function items... Sorry for teaching ;) > > Christian > > > > On Fri, Feb 21, 2014 at 5:06 PM, Carl Leitner <litlfred at ibiblio.org> wrote: > > Hi, > I noticed a significant performance degradation in the function-lookup > method introduced at: > > https://github.com/BaseXdb/basex/commit/4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 > > Where as at: b8b3dd2bee205303aeab3e8a0967b371a2ad7ef7 I have: > > xquery import module namespace csd_bsq = > "https://github.com/his-interop/openinfoman/csd_bsq" at > "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; > let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return () > > Query executed in 22.57 ms. > > At 4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 I have: > > xquery import module namespace csd_bsq = > "https://github.com/his-interop/openinfoman/csd_bsq" at > "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; > () > > Query executed in 17.48 ms. > > xquery import module namespace csd_bsq = > "https://github.com/his-interop/openinfoman/csd_bsq" at > "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; > let $r:= xs:QName('csd_bsq:facility_search') return () > > Query executed in 20.44 ms. > > xquery import module namespace csd_bsq = > "https://github.com/his-interop/openinfoman/csd_bsq" at > "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; > let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return () > > Query executed in 716.18 ms. > > Any suggestions on how to avoid this? > > Thanks so much. > > Cheers, > -carl > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
CL
Carl Leitner24 Feb 2014 12:10 #
Dear Christian, Thanks for the suggestion, the map is certainly cleaner looking than the switch statement. Is support for updating statements in an eval something that would be considered for the future, for example by postponing the application of the updates to after the calling function finishes? Cheers, -carl
Quoted text · Christian Grün · 121 lines
On Feb 22, 2014, at 4:12 PM, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Carl, > >> As I would ideally like to be able to manage the associations >> between UUID and function names dynamically, I do not see immediately how to >> apply your suggestion. > > Would sth. like the following code help? > > declare variable $FUNCTIONS := map { > '69bfbe73-23ac-4033-b60b-6672fd67b937': function() { 12345 }, > '0d22d6a8-035b-46bf-a396-00471c840a08': function() { 67890 } > }; > > let $uuid := '69bfbe73-23ac-4033-b60b-6672fd67b937' > return $FUNCTIONS($uuid)() > > You could as well move this map, or the functions declared inside, > into another module, etc. > >> So maybe my question is, how evil is evil? > > One drawback of using xquery:eval (and xquery:evaluate) is that your > code will only be evaluated at runtime, which is why the compiler has > no chance to detect syntax and type errors at compile time. This is > also the reason why updates operations are not allowed inside > xquery:eval, because an updating eval query may change nodes that are > referenced in the calling query. > > However, if you know exactly what your code is doing, the xquery:eval > function won't cause any trouble (this is also the reason why we are > offering the functions at all... And it may even be officially > introduced with XQuery 3.1). > > Christian > > >> On Feb 22, 2014, at 11:24 AM, Christian Grün <christian.gruen at gmail.com> >> wrote: >> >> Hi Carl, >> >> it would be generally possible to optimize the current implementation >> of function-lookup and related reflective functions, but I think it >> will take us some more time. >> >> Instead, I rather recommend you to use function items, because these >> will always give you better performance and are less error-prone (they >> can be optimized much better than function calls based on string >> references). >> >> I'm not sure how your complete code looks like (and it would probably >> be too complex to serve as example here), but the following code might >> give you a hint how this could look like: >> >> let $add := function($a) { $a + $a } >> let $multiply := function($a) { $a * 2 } >> for $f in ($add, $multiply) >> return $f(2) >> >> If you have already worked with function items... Sorry for teaching ;) >> >> Christian >> >> >> >> On Fri, Feb 21, 2014 at 5:06 PM, Carl Leitner <litlfred at ibiblio.org> wrote: >> >> Hi, >> I noticed a significant performance degradation in the function-lookup >> method introduced at: >> >> https://github.com/BaseXdb/basex/commit/4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 >> >> Where as at: b8b3dd2bee205303aeab3e8a0967b371a2ad7ef7 I have: >> >> xquery import module namespace csd_bsq = >> "https://github.com/his-interop/openinfoman/csd_bsq" at >> "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; >> let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return () >> >> Query executed in 22.57 ms. >> >> At 4aeb4bdc74d980b4c8f2e27546d21fdc1ebaeb77 I have: >> >> xquery import module namespace csd_bsq = >> "https://github.com/his-interop/openinfoman/csd_bsq" at >> "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; >> () >> >> Query executed in 17.48 ms. >> >> xquery import module namespace csd_bsq = >> "https://github.com/his-interop/openinfoman/csd_bsq" at >> "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; >> let $r:= xs:QName('csd_bsq:facility_search') return () >> >> Query executed in 20.44 ms. >> >> xquery import module namespace csd_bsq = >> "https://github.com/his-interop/openinfoman/csd_bsq" at >> "/Users/litlfred/pulsar_cats/openinfoman/repo/csd_base_stored_queries.xqm"; >> let $r:= function-lookup(xs:QName('csd_bsq:facility_search'),2) return () >> >> Query executed in 716.18 ms. >> >> Any suggestions on how to avoid this? >> >> Thanks so much. >> >> Cheers, >> -carl >> >> >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >>
CG
Christian Grün24 Feb 2014 14:22 #
Quoted text · Carl Leitner · 1 line
> Is support for updating statements in an eval something that would be considered for the future, for example by postponing the application of the updates to after the calling function finishes?
We would probably need to merge the PUL (pending update list) of the dynamically evaluated function with the PUL from the main expression, as the updates cannot be executed before completing the evaluation of the main expression (otherwise, we would run into all kinds of new side-effects).

Written for BaseX 7.8. BaseX may have changed since: please check the current documentation.

« basex xsl:fo (Apache FOP) module?Question - Xquery with Basex »
⚡Generated with XQuery