Optimization for procedure call ?

On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier <jeanmarc.mercier at gmail.com> wrote: > Hello, > > I am facing some performances issues, probably due to unexpected query > reevaluation. Might it be due to the optimizer ? Here is a code to test > > declare function local:test($i){ > let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) > (: let $data := xquery:eval("fn:for-each(1 to 10000000, > function($a){2*$a} )") :) > return (1 to $i) ! local:test2($data,.) > }; > > declare function local:test2($data, $dummy) { count($data) }; > > (1 to 10) ! prof:time(local:test(.)) > > Output : > > 683.76 ms > 1137.95 ms > 1727.68 ms > 2151.06 ms > 2694.84 ms > 3189.57 ms > 3725.1 ms > 4277.51 ms > 4815.74 ms > 5417.79 ms > > Note that using xquery:eval (toggling the comment inside local:test). seems > to perform correctly. However I can't use this trick as a workaround. > > (1 to 10) ! prof:time(local:test(.)) > Ouput : > > 2472.71 ms > 825.61 ms > 3013.14 ms > 986.81 ms > 2982.97 ms > 778.24 ms > 1195.33 ms > 3280.7 ms > 976.96 ms > 913.69 ms

> Hi Jean-Marc, > > interesting one; it seems that $data is inlined and thus evaluated > more than once. This doesn't happen if a FLWOR expression is used: > > let $data := fn:for-each ... > for $n in 1 to $i > return local:test2($data, $n) > > Should be easy to resolve. > > Thanks for your (always concise) examples, > Christian > > > On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier > <jeanmarc.mercier at gmail.com> wrote: > > Hello, > > > > I am facing some performances issues, probably due to unexpected query > > reevaluation. Might it be due to the optimizer ? Here is a code to test > > > > declare function local:test($i){ > > let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) > > (: let $data := xquery:eval("fn:for-each(1 to 10000000, > > function($a){2*$a} )") :) > > return (1 to $i) ! local:test2($data,.) > > }; > > > > declare function local:test2($data, $dummy) { count($data) }; > > > > (1 to 10) ! prof:time(local:test(.)) > > > > Output : > > > > 683.76 ms > > 1137.95 ms > > 1727.68 ms > > 2151.06 ms > > 2694.84 ms > > 3189.57 ms > > 3725.1 ms > > 4277.51 ms > > 4815.74 ms > > 5417.79 ms > > > > Note that using xquery:eval (toggling the comment inside local:test). > seems > > to perform correctly. However I can't use this trick as a workaround. > > > > (1 to 10) ! prof:time(local:test(.)) > > Ouput : > > > > 2472.71 ms > > 825.61 ms > > 3013.14 ms > > 986.81 ms > > 2982.97 ms > > 778.24 ms > > 1195.33 ms > > 3280.7 ms > > 976.96 ms > > 913.69 ms >

> Hi. Thanks for the FLWOR trick, behaving in constant time. However it seems > to generates an unexpected 5X time overhead

On Tue, Jan 13, 2015 at 9:19 AM, jean-marc Mercier <jeanmarc.mercier at gmail.com> wrote: > Christian, > > Hi. Thanks for the FLWOR trick, behaving in constant time. However it seems > to generates an unexpected 5X time overhead, and I can't use it > straightforwardly as workaround. Can you reproduce it, since this overhead > might be due to eclipse ? > > declare function local:test(){fn:for-each(1 to 10000000, > function($a){2*$a})}; > declare function local:test1(){count(local:test())}; > declare function local:test2(){let $ data := local:test() return for $n in > (1 to 10) return count($data)}; > > prof:time(local:test1()) ouput : 639.12 ms > prof:time(local:test2()) ouput : 2431.67 ms > > > > 2015-01-13 1:59 GMT+01:00 Christian Grün <christian.gruen at gmail.com>: >> >> Hi Jean-Marc, >> >> interesting one; it seems that $data is inlined and thus evaluated >> more than once. This doesn't happen if a FLWOR expression is used: >> >> let $data := fn:for-each ... >> for $n in 1 to $i >> return local:test2($data, $n) >> >> Should be easy to resolve. >> >> Thanks for your (always concise) examples, >> Christian >> >> >> On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier >> <jeanmarc.mercier at gmail.com> wrote: >> > Hello, >> > >> > I am facing some performances issues, probably due to unexpected query >> > reevaluation. Might it be due to the optimizer ? Here is a code to test >> > >> > declare function local:test($i){ >> > let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) >> > (: let $data := xquery:eval("fn:for-each(1 to 10000000, >> > function($a){2*$a} )") :) >> > return (1 to $i) ! local:test2($data,.) >> > }; >> > >> > declare function local:test2($data, $dummy) { count($data) }; >> > >> > (1 to 10) ! prof:time(local:test(.)) >> > >> > Output : >> > >> > 683.76 ms >> > 1137.95 ms >> > 1727.68 ms >> > 2151.06 ms >> > 2694.84 ms >> > 3189.57 ms >> > 3725.1 ms >> > 4277.51 ms >> > 4815.74 ms >> > 5417.79 ms >> > >> > Note that using xquery:eval (toggling the comment inside local:test). >> > seems >> > to perform correctly. However I can't use this trick as a workaround. >> > >> > (1 to 10) ! prof:time(local:test(.)) >> > Ouput : >> > >> > 2472.71 ms >> > 825.61 ms >> > 3013.14 ms >> > 986.81 ms >> > 2982.97 ms >> > 778.24 ms >> > 1195.33 ms >> > 3280.7 ms >> > 976.96 ms >> > 913.69 ms > >

> Hi Jean-Marc, > > beforehand: The first issue you reported back to us recently [1] has > been fixed by our functional guru Leo! A new snapshot is available > [2]. > > > Hi. Thanks for the FLWOR trick, behaving in constant time. However it > seems > > to generates an unexpected 5X time overhead > > Most probably this is because the results of for-each will be cached, > while in the previous case (if no inlining takes place), all items > will be iterated. > > Hope this helps, > Christian > > [1] https://github.com/BaseXdb/basex/issues/1052 > [2] http://files.basex.org/releases/latest/ > > > On Tue, Jan 13, 2015 at 9:19 AM, jean-marc Mercier > <jeanmarc.mercier at gmail.com> wrote: > > Christian, > > > > Hi. Thanks for the FLWOR trick, behaving in constant time. However it > seems > > to generates an unexpected 5X time overhead, and I can't use it > > straightforwardly as workaround. Can you reproduce it, since this > overhead > > might be due to eclipse ? > > > > declare function local:test(){fn:for-each(1 to 10000000, > > function($a){2*$a})}; > > declare function local:test1(){count(local:test())}; > > declare function local:test2(){let $ data := local:test() return for $n > in > > (1 to 10) return count($data)}; > > > > prof:time(local:test1()) ouput : 639.12 ms > > prof:time(local:test2()) ouput : 2431.67 ms > > > > > > > > 2015-01-13 1:59 GMT+01:00 Christian Grün <christian.gruen at gmail.com>: > >> > >> Hi Jean-Marc, > >> > >> interesting one; it seems that $data is inlined and thus evaluated > >> more than once. This doesn't happen if a FLWOR expression is used: > >> > >> let $data := fn:for-each ... > >> for $n in 1 to $i > >> return local:test2($data, $n) > >> > >> Should be easy to resolve. > >> > >> Thanks for your (always concise) examples, > >> Christian > >> > >> > >> On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier > >> <jeanmarc.mercier at gmail.com> wrote: > >> > Hello, > >> > > >> > I am facing some performances issues, probably due to unexpected query > >> > reevaluation. Might it be due to the optimizer ? Here is a code to > test > >> > > >> > declare function local:test($i){ > >> > let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) > >> > (: let $data := xquery:eval("fn:for-each(1 to 10000000, > >> > function($a){2*$a} )") :) > >> > return (1 to $i) ! local:test2($data,.) > >> > }; > >> > > >> > declare function local:test2($data, $dummy) { count($data) }; > >> > > >> > (1 to 10) ! prof:time(local:test(.)) > >> > > >> > Output : > >> > > >> > 683.76 ms > >> > 1137.95 ms > >> > 1727.68 ms > >> > 2151.06 ms > >> > 2694.84 ms > >> > 3189.57 ms > >> > 3725.1 ms > >> > 4277.51 ms > >> > 4815.74 ms > >> > 5417.79 ms > >> > > >> > Note that using xquery:eval (toggling the comment inside local:test). > >> > seems > >> > to perform correctly. However I can't use this trick as a workaround. > >> > > >> > (1 to 10) ! prof:time(local:test(.)) > >> > Ouput : > >> > > >> > 2472.71 ms > >> > 825.61 ms > >> > 3013.14 ms > >> > 986.81 ms > >> > 2982.97 ms > >> > 778.24 ms > >> > 1195.33 ms > >> > 3280.7 ms > >> > 976.96 ms > >> > 913.69 ms > > > > >

> Glory to our holy Leo for transmitting *brahmavidya* to shishya !! > > Turning back to this optimization problem, this might be trickier since > execution time does not seem to behave linearly. For instance, consider > > declare function local:test1($i){let $data_3 as item() *:= fn:for-each((1 > to 1000000*$i), function($a_0) as item()? { (2 * $a_0) }) return for $n in > (1 to 2) return fn:count($data_3)}; > declare function local:test2($i){count(fn:for-each((1 to 1000000*$i), > function($a_0) as item()? { (2 * $a_0) }))}; > > Then execution time is ( the results are stable while repeating the > experience ) > > prof:time(local:test1(7)) : 631.32 ms prof:time(local:test2(7)) : 451.32 > ms > prof:time(local:test1(8)) : 2550.17 ms prof:time(local:test2(8)) : > 507.32 ms > > > > 2015-01-13 22:43 GMT+01:00 Christian Grün <christian.gruen at gmail.com>: > >> Hi Jean-Marc, >> >> beforehand: The first issue you reported back to us recently [1] has >> been fixed by our functional guru Leo! A new snapshot is available >> [2]. >> >> > Hi. Thanks for the FLWOR trick, behaving in constant time. However it >> seems >> > to generates an unexpected 5X time overhead >> >> Most probably this is because the results of for-each will be cached, >> while in the previous case (if no inlining takes place), all items >> will be iterated. >> >> Hope this helps, >> Christian >> >> [1] https://github.com/BaseXdb/basex/issues/1052 >> [2] http://files.basex.org/releases/latest/ >> >> >> On Tue, Jan 13, 2015 at 9:19 AM, jean-marc Mercier >> <jeanmarc.mercier at gmail.com> wrote: >> > Christian, >> > >> > Hi. Thanks for the FLWOR trick, behaving in constant time. However it >> seems >> > to generates an unexpected 5X time overhead, and I can't use it >> > straightforwardly as workaround. Can you reproduce it, since this >> overhead >> > might be due to eclipse ? >> > >> > declare function local:test(){fn:for-each(1 to 10000000, >> > function($a){2*$a})}; >> > declare function local:test1(){count(local:test())}; >> > declare function local:test2(){let $ data := local:test() return for $n >> in >> > (1 to 10) return count($data)}; >> > >> > prof:time(local:test1()) ouput : 639.12 ms >> > prof:time(local:test2()) ouput : 2431.67 ms >> > >> > >> > >> > 2015-01-13 1:59 GMT+01:00 Christian Grün <christian.gruen at gmail.com>: >> >> >> >> Hi Jean-Marc, >> >> >> >> interesting one; it seems that $data is inlined and thus evaluated >> >> more than once. This doesn't happen if a FLWOR expression is used: >> >> >> >> let $data := fn:for-each ... >> >> for $n in 1 to $i >> >> return local:test2($data, $n) >> >> >> >> Should be easy to resolve. >> >> >> >> Thanks for your (always concise) examples, >> >> Christian >> >> >> >> >> >> On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier >> >> <jeanmarc.mercier at gmail.com> wrote: >> >> > Hello, >> >> > >> >> > I am facing some performances issues, probably due to unexpected >> query >> >> > reevaluation. Might it be due to the optimizer ? Here is a code to >> test >> >> > >> >> > declare function local:test($i){ >> >> > let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) >> >> > (: let $data := xquery:eval("fn:for-each(1 to 10000000, >> >> > function($a){2*$a} )") :) >> >> > return (1 to $i) ! local:test2($data,.) >> >> > }; >> >> > >> >> > declare function local:test2($data, $dummy) { count($data) }; >> >> > >> >> > (1 to 10) ! prof:time(local:test(.)) >> >> > >> >> > Output : >> >> > >> >> > 683.76 ms >> >> > 1137.95 ms >> >> > 1727.68 ms >> >> > 2151.06 ms >> >> > 2694.84 ms >> >> > 3189.57 ms >> >> > 3725.1 ms >> >> > 4277.51 ms >> >> > 4815.74 ms >> >> > 5417.79 ms >> >> > >> >> > Note that using xquery:eval (toggling the comment inside local:test). >> >> > seems >> >> > to perform correctly. However I can't use this trick as a workaround. >> >> > >> >> > (1 to 10) ! prof:time(local:test(.)) >> >> > Ouput : >> >> > >> >> > 2472.71 ms >> >> > 825.61 ms >> >> > 3013.14 ms >> >> > 986.81 ms >> >> > 2982.97 ms >> >> > 778.24 ms >> >> > 1195.33 ms >> >> > 3280.7 ms >> >> > 976.96 ms >> >> > 913.69 ms >> > >> > >> > >

> Might it be garbage collector stuff ?

> > 2015-01-14 8:18 GMT+01:00 jean-marc Mercier <jeanmarc.mercier at gmail.com>: >> >> Glory to our holy Leo for transmitting brahmavidya to shishya !! >> >> Turning back to this optimization problem, this might be trickier since

>> >> declare function local:test1($i){let $data_3 as item() *:=

return for $n in (1 to 2) return fn:count($data_3)}; >> declare function local:test2($i){count(fn:for-each((1 to 1000000*$i),

>> >> Then execution time is ( the results are stable while repeating the

>> >> prof:time(local:test1(7)) : 631.32 ms prof:time(local:test2(7)) : 451.32 ms >> prof:time(local:test1(8)) : 2550.17 ms prof:time(local:test2(8)) :

>> >> >> >> 2015-01-13 22:43 GMT+01:00 Christian Grün <christian.gruen at gmail.com>: >>> >>> Hi Jean-Marc, >>> >>> beforehand: The first issue you reported back to us recently [1] has >>> been fixed by our functional guru Leo! A new snapshot is available >>> [2]. >>> >>> > Hi. Thanks for the FLWOR trick, behaving in constant time. However it

>>> > to generates an unexpected 5X time overhead >>> >>> Most probably this is because the results of for-each will be cached, >>> while in the previous case (if no inlining takes place), all items >>> will be iterated. >>> >>> Hope this helps, >>> Christian >>> >>> [1] https://github.com/BaseXdb/basex/issues/1052 >>> [2] http://files.basex.org/releases/latest/ >>> >>> >>> On Tue, Jan 13, 2015 at 9:19 AM, jean-marc Mercier >>> <jeanmarc.mercier at gmail.com> wrote: >>> > Christian, >>> > >>> > Hi. Thanks for the FLWOR trick, behaving in constant time. However it

>>> > to generates an unexpected 5X time overhead, and I can't use it >>> > straightforwardly as workaround. Can you reproduce it, since this

>>> > might be due to eclipse ? >>> > >>> > declare function local:test(){fn:for-each(1 to 10000000, >>> > function($a){2*$a})}; >>> > declare function local:test1(){count(local:test())}; >>> > declare function local:test2(){let $ data := local:test() return for

>>> > (1 to 10) return count($data)}; >>> > >>> > prof:time(local:test1()) ouput : 639.12 ms >>> > prof:time(local:test2()) ouput : 2431.67 ms >>> > >>> > >>> > >>> > 2015-01-13 1:59 GMT+01:00 Christian Grün <christian.gruen at gmail.com>: >>> >> >>> >> Hi Jean-Marc, >>> >> >>> >> interesting one; it seems that $data is inlined and thus evaluated >>> >> more than once. This doesn't happen if a FLWOR expression is used: >>> >> >>> >> let $data := fn:for-each ... >>> >> for $n in 1 to $i >>> >> return local:test2($data, $n) >>> >> >>> >> Should be easy to resolve. >>> >> >>> >> Thanks for your (always concise) examples, >>> >> Christian >>> >> >>> >> >>> >> On Mon, Jan 12, 2015 at 9:08 PM, jean-marc Mercier >>> >> <jeanmarc.mercier at gmail.com> wrote: >>> >> > Hello, >>> >> > >>> >> > I am facing some performances issues, probably due to unexpected

>>> >> > reevaluation. Might it be due to the optimizer ? Here is a code to

>>> >> > >>> >> > declare function local:test($i){ >>> >> > let $data := fn:for-each(1 to 10000000, function($a){2*$a} ) >>> >> > (: let $data := xquery:eval("fn:for-each(1 to 10000000, >>> >> > function($a){2*$a} )") :) >>> >> > return (1 to $i) ! local:test2($data,.) >>> >> > }; >>> >> > >>> >> > declare function local:test2($data, $dummy) { count($data) }; >>> >> > >>> >> > (1 to 10) ! prof:time(local:test(.)) >>> >> > >>> >> > Output : >>> >> > >>> >> > 683.76 ms >>> >> > 1137.95 ms >>> >> > 1727.68 ms >>> >> > 2151.06 ms >>> >> > 2694.84 ms >>> >> > 3189.57 ms >>> >> > 3725.1 ms >>> >> > 4277.51 ms >>> >> > 4815.74 ms >>> >> > 5417.79 ms >>> >> > >>> >> > Note that using xquery:eval (toggling the comment inside

>>> >> > seems >>> >> > to perform correctly. However I can't use this trick as a

>>> >> > >>> >> > (1 to 10) ! prof:time(local:test(.)) >>> >> > Ouput : >>> >> > >>> >> > 2472.71 ms >>> >> > 825.61 ms >>> >> > 3013.14 ms >>> >> > 986.81 ms >>> >> > 2982.97 ms >>> >> > 778.24 ms >>> >> > 1195.33 ms >>> >> > 3280.7 ms >>> >> > 976.96 ms >>> >> > 913.69 ms >>> > >>> > >> >> >