Main Page » Mailing List » January 2015

Pre-compilled Query in Base-X

Friday, 23 January 2015 · 8 messages
AK
ankit kumar23 Jan 2015 08:26 #
Hi, I was exploring BaseX for my XML based product. I was wondering whether there is Prepared Statement Like facility incorporated in BaseX, With the same QueryProcessor instance, is it possible to execute it multiple times with different binding values for given variable. I am executing following code.
QueryProcessor pr = new QueryProcessor("declare variable $a external ;
$a",context);
pr.context(itr);
pr.bind("a","ABC");
Resultfinalr=pr.execute();
System.out.println("Final Execution Result 1 ::" + finalr);
// rebinding  $a with different value
pr.bind("a", "XYZ");
Result out1 = pr.execute();
System.out.println("Final Execution Result 2 ::" + out1);
******************************************************************************* I am getting "ABC" when I am executing second time. I need this because I would be executing the same query multiple times. Thanks Anky
CG
Christian Grün23 Jan 2015 11:43 #
Hi Ankit,
Quoted text · ankit kumar · 5 lines
> I was exploring BaseX for my XML based product. I was wondering whether > there is Prepared Statement Like facility incorporated in BaseX, > > With the same QueryProcessor instance, is it possible to execute it multiple > times with different binding values for given variable.
We are working on caching precompiled XQuery expressions; for now, you need to create new QueryProcessor instances. In practice, however, this rarely leads to problems, because QueryProcessor instances are very light-weight. The following BaseX call creates 100,000 instances of the QueryProcessor, it takes around 2 seconds:
Quoted text · ankit kumar · 1 line
> basex -V -r100000 "1"
Parsing: 0.01 ms (avg) Compiling: 0.0 ms (avg) Evaluating: 0.0 ms (avg) Printing: 0.01 ms (avg) Total Time: 0.02 ms (avg) Obviously, the query "1" is very simple, but usually, parsing only gets relevant if the queries to be parsed take many KBs or even MBs (such as can be the case in RESTXQ applications). Best, Christian
AK
ankit kumar23 Jan 2015 12:46 #
Hi, That's great , thank you very much Appreciate your prompt response. Hoping caching functionality will be added in the next release. In my queries I am also including a module function library in each of my xqueries, so is there a way I can directly set the compiled module function library for all the QueryProcessor instance ,instead of adding library every time instance gets created. Thanks Anky
Quoted text · Christian Grün · 32 lines
On 23 January 2015 at 16:13, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Ankit, > > > I was exploring BaseX for my XML based product. I was wondering whether > > there is Prepared Statement Like facility incorporated in BaseX, > > > > With the same QueryProcessor instance, is it possible to execute it > multiple > > times with different binding values for given variable. > > We are working on caching precompiled XQuery expressions; for now, you > need to create new QueryProcessor instances. In practice, however, > this rarely leads to problems, because QueryProcessor instances are > very light-weight. The following BaseX call creates 100,000 instances > of the QueryProcessor, it takes around 2 seconds: > > > basex -V -r100000 "1" > > Parsing: 0.01 ms (avg) > Compiling: 0.0 ms (avg) > Evaluating: 0.0 ms (avg) > Printing: 0.01 ms (avg) > Total Time: 0.02 ms (avg) > > Obviously, the query "1" is very simple, but usually, parsing only > gets relevant if the queries to be parsed take many KBs or even MBs > (such as can be the case in RESTXQ applications). > > Best, > Christian >
CG
Christian Grün23 Jan 2015 12:51 #
Hi Ankit,
Quoted text · ankit kumar · 5 lines
> In my queries I am also including a module function library in each of my > xqueries, > so is there a way I can directly set the compiled module function library > for all the QueryProcessor instance ,instead of adding library > every time instance gets created.
I guess you are using the import module statement of XQuery? That's the only way to do it (but it will only be parsed once per call). However, if you copy your module to the repository, you won't have to specify the file path. I invite you to check out our documentation on the XQuery repository [1]. Regarding your request of compiling the module, I would be interested to hear if you have already done some benchmarking in order to find out how much time is spent for parsing the module? Best, Christian [1] http://docs.basex.org/wiki/Repository
Quoted text · ankit kumar · 39 lines
> > > Thanks > Anky > > On 23 January 2015 at 16:13, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Ankit, >> >> > I was exploring BaseX for my XML based product. I was wondering whether >> > there is Prepared Statement Like facility incorporated in BaseX, >> > >> > With the same QueryProcessor instance, is it possible to execute it >> > multiple >> > times with different binding values for given variable. >> >> We are working on caching precompiled XQuery expressions; for now, you >> need to create new QueryProcessor instances. In practice, however, >> this rarely leads to problems, because QueryProcessor instances are >> very light-weight. The following BaseX call creates 100,000 instances >> of the QueryProcessor, it takes around 2 seconds: >> >> > basex -V -r100000 "1" >> >> Parsing: 0.01 ms (avg) >> Compiling: 0.0 ms (avg) >> Evaluating: 0.0 ms (avg) >> Printing: 0.01 ms (avg) >> Total Time: 0.02 ms (avg) >> >> Obviously, the query "1" is very simple, but usually, parsing only >> gets relevant if the queries to be parsed take many KBs or even MBs >> (such as can be the case in RESTXQ applications). >> >> Best, >> Christian > >
AK
ankit kumar23 Jan 2015 13:18 #
Hi, These are the benchmarking result. basex -V -r100000 "import module 'http://www.xyz.org/2008/function/ins' at 'file:/C:/avc/FUNCTION_LIBRARY.xq';'a'" a Query: import module 'http://www.xyz.org/2008/function/ins' at 'file:/C:/avc/FUNCTION_LIBRARY.xq';'a' Parsing: 2.71 ms (avg) Compiling: 0.0 ms (avg) Evaluating: 0.0 ms (avg) Printing: 0.01 ms (avg) Total Time: 2.73 ms (avg) Hit(s): 1 Item Updated: 0 Items Printed: 1 Byte Read Locking: none Write Locking: none Query executed in 2.73 ms (avg). I have one more doubt regarding module compilation. Are the modules compiled lazily ?.
Quoted text · Christian Grün · 68 lines
On 23 January 2015 at 17:21, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Ankit, > > > In my queries I am also including a module function library in each of > my > > xqueries, > > so is there a way I can directly set the compiled module function library > > for all the QueryProcessor instance ,instead of adding library > > every time instance gets created. > > I guess you are using the import module statement of XQuery? That's > the only way to do it (but it will only be parsed once per call). > However, if you copy your module to the repository, you won't have to > specify the file path. I invite you to check out our documentation on > the XQuery repository [1]. > > Regarding your request of compiling the module, I would be interested > to hear if you have already done some benchmarking in order to find > out how much time is spent for parsing the module? > > Best, > Christian > > [1] http://docs.basex.org/wiki/Repository > > > > > > > > Thanks > > Anky > > > > On 23 January 2015 at 16:13, Christian Grün <christian.gruen at gmail.com> > > wrote: > >> > >> Hi Ankit, > >> > >> > I was exploring BaseX for my XML based product. I was wondering > whether > >> > there is Prepared Statement Like facility incorporated in BaseX, > >> > > >> > With the same QueryProcessor instance, is it possible to execute it > >> > multiple > >> > times with different binding values for given variable. > >> > >> We are working on caching precompiled XQuery expressions; for now, you > >> need to create new QueryProcessor instances. In practice, however, > >> this rarely leads to problems, because QueryProcessor instances are > >> very light-weight. The following BaseX call creates 100,000 instances > >> of the QueryProcessor, it takes around 2 seconds: > >> > >> > basex -V -r100000 "1" > >> > >> Parsing: 0.01 ms (avg) > >> Compiling: 0.0 ms (avg) > >> Evaluating: 0.0 ms (avg) > >> Printing: 0.01 ms (avg) > >> Total Time: 0.02 ms (avg) > >> > >> Obviously, the query "1" is very simple, but usually, parsing only > >> gets relevant if the queries to be parsed take many KBs or even MBs > >> (such as can be the case in RESTXQ applications). > >> > >> Best, > >> Christian > > > > >
CG
Christian Grün23 Jan 2015 13:26 #
Quoted text · ankit kumar · 1 line
> These are the benchmarking result.
Thanks. The overhead of 2-3 ms could surely be reduced once the modules will be precompiled. Does the current overhead represent a bottleneck in your scenario?
Quoted text · ankit kumar · 2 lines
> I have one more doubt regarding module compilation. Are the modules compiled > lazily ?.
Library modules are parsed every time they are imported. C.
Quoted text · ankit kumar · 73 lines
> > > On 23 January 2015 at 17:21, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Ankit, >> >> > In my queries I am also including a module function library in each of >> > my >> > xqueries, >> > so is there a way I can directly set the compiled module function >> > library >> > for all the QueryProcessor instance ,instead of adding library >> > every time instance gets created. >> >> I guess you are using the import module statement of XQuery? That's >> the only way to do it (but it will only be parsed once per call). >> However, if you copy your module to the repository, you won't have to >> specify the file path. I invite you to check out our documentation on >> the XQuery repository [1]. >> >> Regarding your request of compiling the module, I would be interested >> to hear if you have already done some benchmarking in order to find >> out how much time is spent for parsing the module? >> >> Best, >> Christian >> >> [1] http://docs.basex.org/wiki/Repository >> >> >> > >> > >> > Thanks >> > Anky >> > >> > On 23 January 2015 at 16:13, Christian Grün <christian.gruen at gmail.com> >> > wrote: >> >> >> >> Hi Ankit, >> >> >> >> > I was exploring BaseX for my XML based product. I was wondering >> >> > whether >> >> > there is Prepared Statement Like facility incorporated in BaseX, >> >> > >> >> > With the same QueryProcessor instance, is it possible to execute it >> >> > multiple >> >> > times with different binding values for given variable. >> >> >> >> We are working on caching precompiled XQuery expressions; for now, you >> >> need to create new QueryProcessor instances. In practice, however, >> >> this rarely leads to problems, because QueryProcessor instances are >> >> very light-weight. The following BaseX call creates 100,000 instances >> >> of the QueryProcessor, it takes around 2 seconds: >> >> >> >> > basex -V -r100000 "1" >> >> >> >> Parsing: 0.01 ms (avg) >> >> Compiling: 0.0 ms (avg) >> >> Evaluating: 0.0 ms (avg) >> >> Printing: 0.01 ms (avg) >> >> Total Time: 0.02 ms (avg) >> >> >> >> Obviously, the query "1" is very simple, but usually, parsing only >> >> gets relevant if the queries to be parsed take many KBs or even MBs >> >> (such as can be the case in RESTXQ applications). >> >> >> >> Best, >> >> Christian >> > >> > > >
AK
ankit kumar23 Jan 2015 14:09 #
Hi, I tried similar thing using java. It is taking 86 ms for parsing.
QueryProcessor mod = new QueryProcessor("import module '
http://www.xyz.org/2008/function/ins' at 'file:/C:/avc/FUNCTION_
LIBRARY.xq';'a' ", context);
long time=System.currentTimeMillis();
mod.parse();
System.out.println(" Time ::" + (System.currentTimeMillis() - time));
Output :: Time :: 86 Does the current overhead represent a bottleneck in your scenario? We have a scenario where we need to execute queries thousand times for each user, that will hamper the performance. Thanks Anky
Quoted text · Christian Grün · 93 lines
On 23 January 2015 at 17:56, Christian Grün <christian.gruen at gmail.com> wrote: > > These are the benchmarking result. > > Thanks. The overhead of 2-3 ms could surely be reduced once the > modules will be precompiled. Does the current overhead represent a > bottleneck in your scenario? > > > I have one more doubt regarding module compilation. Are the modules > compiled > > lazily ?. > > Library modules are parsed every time they are imported. > > C. > > > > > > > On 23 January 2015 at 17:21, Christian Grün <christian.gruen at gmail.com> > > wrote: > >> > >> Hi Ankit, > >> > >> > In my queries I am also including a module function library in each > of > >> > my > >> > xqueries, > >> > so is there a way I can directly set the compiled module function > >> > library > >> > for all the QueryProcessor instance ,instead of adding library > >> > every time instance gets created. > >> > >> I guess you are using the import module statement of XQuery? That's > >> the only way to do it (but it will only be parsed once per call). > >> However, if you copy your module to the repository, you won't have to > >> specify the file path. I invite you to check out our documentation on > >> the XQuery repository [1]. > >> > >> Regarding your request of compiling the module, I would be interested > >> to hear if you have already done some benchmarking in order to find > >> out how much time is spent for parsing the module? > >> > >> Best, > >> Christian > >> > >> [1] http://docs.basex.org/wiki/Repository > >> > >> > >> > > >> > > >> > Thanks > >> > Anky > >> > > >> > On 23 January 2015 at 16:13, Christian Grün < > christian.gruen at gmail.com> > >> > wrote: > >> >> > >> >> Hi Ankit, > >> >> > >> >> > I was exploring BaseX for my XML based product. I was wondering > >> >> > whether > >> >> > there is Prepared Statement Like facility incorporated in BaseX, > >> >> > > >> >> > With the same QueryProcessor instance, is it possible to execute it > >> >> > multiple > >> >> > times with different binding values for given variable. > >> >> > >> >> We are working on caching precompiled XQuery expressions; for now, > you > >> >> need to create new QueryProcessor instances. In practice, however, > >> >> this rarely leads to problems, because QueryProcessor instances are > >> >> very light-weight. The following BaseX call creates 100,000 instances > >> >> of the QueryProcessor, it takes around 2 seconds: > >> >> > >> >> > basex -V -r100000 "1" > >> >> > >> >> Parsing: 0.01 ms (avg) > >> >> Compiling: 0.0 ms (avg) > >> >> Evaluating: 0.0 ms (avg) > >> >> Printing: 0.01 ms (avg) > >> >> Total Time: 0.02 ms (avg) > >> >> > >> >> Obviously, the query "1" is very simple, but usually, parsing only > >> >> gets relevant if the queries to be parsed take many KBs or even MBs > >> >> (such as can be the case in RESTXQ applications). > >> >> > >> >> Best, > >> >> Christian > >> > > >> > > > > > >
CG
Christian Grün23 Jan 2015 14:22 #
Quoted text · ankit kumar · 1 line
> I tried similar thing using java. It is taking 86 ms for parsing.
This is most probably due to the startup of the JVM. I would be surprised if you shouldn't get similar times as shown on command line (btw, you'll get more exact measurements by using System.nanoTime()).
Quoted text · ankit kumar · 2 lines
> We have a scenario where we need to execute queries thousand times for each > user, that will hamper the performance.
I'll be interested to find out how much of the total time of your application will be spent for parsing the query, or if it's not other components that will be predominating. Feel free to send me some profiling data (e.g. generated via the JVM flags -Xrunhprof:cpu=samples) once you have it running.

Written for BaseX 7.9. BaseX may have changed since: please check the current documentation.

« text() in XQueryBug in declare base-uri »
⚡Generated with XQuery