Multiple queries in client API
Friday, 5 April 2013 · 5 messages · last reply 14 Apr 2013
MC
Michael CLEMENT5 Apr 2013 15:12 #
Hello,
I am building a Django application using BaseX as its database, using a custom model layer with the Python API.
Currently, I am using a single Session instance to connect to the BaseX server, but I am facing problems when multiple queries are sent at the same time.
I didn't check how the client API works in details, but is it made in such a way that only one query at a time can be sent? Should I use a new Session instance for every single query sent to the server?
Thanks in advance.
Regards,
Michael Clement
CG
Christian Grün12 Apr 2013 10:30 #
Hi Michael,
Quoted text · Michael CLEMENT · 3 lines
> I didn’t check how the client API works in details, but is it made in such a > way that only one query at a time can be sent? Should I use a new Session > instance for every single query sent to the server?
exactly, that’s the way our sessions work: If you want to run queries in parallel, you’ll have to create multiple sessions. In the use cases we are familiar with, this works out fine, because creating a new session is a pretty light-weight operation.
Hope this helps,
Christian
SS
Stu Salsbury12 Apr 2013 19:03 #
One use case where this might be a problem is when you want all of your queries compiled at your application start time (using
QueryProcessor::compile()). This is a speed/CPU load issue.
Please tell me I'm wrong it this is done automagically for modules stored in the database, but my impression is it doesn't happen until you call compile or execute (which calls compile).
I'm working in Java and plan to use Apache Commons Pool to store a bunch of sessions with associated start-time-compiled queries for the threads of my website requests. Can anyone see any problems with this approach?
On Fri, Apr 12, 2013 at 1:30 AM, Christian Grün
<christian.gruen at gmail.com>wrote:
Quoted text · Christian Grün · 19 lines
> Hi Michael, > > > I didn’t check how the client API works in details, but is it made in > such a > > way that only one query at a time can be sent? Should I use a new Session > > instance for every single query sent to the server? > > exactly, that’s the way our sessions work: If you want to run queries > in parallel, you’ll have to create multiple sessions. In the use cases > we are familiar with, this works out fine, because creating a new > session is a pretty light-weight operation. > > Hope this helps, > Christian > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
CG
Christian Grün13 Apr 2013 22:58 #
Quoted text · Stu Salsbury · 3 lines
> One use case where this might be a problem is when you want all of your > queries compiled at your application start time (using > QueryProcessor::compile()). This is a speed/CPU load issue.
This is something we want to tackle within the query processor architecture: query modules that are requested multiple times will automatically be cached and reused. As you may have seen, however, parsing and compilation time is usually pretty fast unless the number of linked library modules isn’t that extensive.
It would be interesting to hear if you (or someone else) has already encountered a real bottleneck of the current client/server
architecture or the XQuery parsing/compilation step?
Christian
Quoted text · Stu Salsbury · 42 lines
On Fri, Apr 12, 2013 at 5:03 PM, Stu Salsbury <stu.salsbury at gmail.com> wrote: > > Please tell me I'm wrong it this is done automagically for modules stored in > the database, but my impression is it doesn't happen until you call compile > or execute (which calls compile). > > I'm working in Java and plan to use Apache Commons Pool to store a bunch of > sessions with associated start-time-compiled queries for the threads of my > website requests. Can anyone see any problems with this approach? > > > > On Fri, Apr 12, 2013 at 1:30 AM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> Hi Michael, >> >> > I didn’t check how the client API works in details, but is it made in >> > such a >> > way that only one query at a time can be sent? Should I use a new >> > Session >> > instance for every single query sent to the server? >> >> exactly, that’s the way our sessions work: If you want to run queries >> in parallel, you’ll have to create multiple sessions. In the use cases >> we are familiar with, this works out fine, because creating a new >> session is a pretty light-weight operation. >> >> Hope this helps, >> Christian >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
SS
Stu Salsbury14 Apr 2013 01:21 #
Thanks, Christian. I have not run into anything yet. In fact, I'm still deciding between Marklogic and BaseX and this is one area that I want to make sure I understand before I take the plunge. I haven't even looked at how I would handle web request threads or compiled query caching in Marklogic yet, so I'm still pretty naive. I appreciate your response. It looks like I read the situation right. I might be over-thinking the parse/compile overhead -- I'm underexposed to interpreted languages on servers.
Aside: the race is close, with BaseX in the lead. BaseX supports Xquery Update Facility and Java function imports, Marklogic has rollbacks and schema awareness (to what extent I don't yet know). I need to figure out how/if I can efficiently integrate BaseX with Saxon as a middle tier so that they can share DOM and SAX objects. The servlex project is interesting: https://code.google.com/p/servlex/
On Sat, Apr 13, 2013 at 1:58 PM, Christian Grün
<christian.gruen at gmail.com>wrote:
Quoted text · Christian Grün · 66 lines
> > One use case where this might be a problem is when you want all of your > > queries compiled at your application start time (using > > QueryProcessor::compile()). This is a speed/CPU load issue. > > This is something we want to tackle within the query processor > architecture: query modules that are requested multiple times will > automatically be cached and reused. As you may have seen, however, > parsing and compilation time is usually pretty fast unless the number > of linked library modules isn’t that extensive. > > It would be interesting to hear if you (or someone else) has already > encountered a real bottleneck of the current client/server > architecture or the XQuery parsing/compilation step? > > Christian > ___________________________ > > On Fri, Apr 12, 2013 at 5:03 PM, Stu Salsbury <stu.salsbury at gmail.com> > wrote: > > > > Please tell me I'm wrong it this is done automagically for modules > stored in > > the database, but my impression is it doesn't happen until you call > compile > > or execute (which calls compile). > > > > I'm working in Java and plan to use Apache Commons Pool to store a bunch > of > > sessions with associated start-time-compiled queries for the threads of > my > > website requests. Can anyone see any problems with this approach? > > > > > > > > On Fri, Apr 12, 2013 at 1:30 AM, Christian Grün < > christian.gruen at gmail.com> > > wrote: > >> > >> Hi Michael, > >> > >> > I didn’t check how the client API works in details, but is it made in > >> > such a > >> > way that only one query at a time can be sent? Should I use a new > >> > Session > >> > instance for every single query sent to the server? > >> > >> exactly, that’s the way our sessions work: If you want to run queries > >> in parallel, you’ll have to create multiple sessions. In the use cases > >> we are familiar with, this works out fine, because creating a new > >> session is a pretty light-weight operation. > >> > >> Hope this helps, > >> Christian > >> _______________________________________________ > >> BaseX-Talk mailing list > >> BaseX-Talk at mailman.uni-konstanz.de > >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > >
Written for BaseX 7.6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery