Big recursive XQueries

> I have a BaseX backed website which stores all the pieces of the page in the DB. There is a significant amount of processing to be done. Pretty much any piece of the page can have effective or expiration dates (so that page parts can come and go on schedule). Some pages have dynamic elements, such as a Twitter feed. The page is build from objects which are wrapped in frames (for adding temporal and styling parameters), which are inserted into columns, which go into rows, which make up a page (which itself is wrapped in a template). So there is a lot of recursion. > > I do the recursion using local functions. > > It works well, but the page-building query is getting enormous, most of which is all these functions to insert parts of the page, often recursively. > > I am wondering what the best practice would be for this. I'm working in Scala, and I could break the process up into repeated passes hitting the db with a different query for each, but it seems to me one big query will almost certainly be faster. But including all the local functions in the query makes it huge. > > Is there some way to compile and load the functions once on startup, and then just run a small XQuery that calls functions which call functions, etc.? > > Any ideas (or resources) for ways to optimize the query? Everything gets reused, so there is very little nesting -- mostly references are passed. For example: > > <pages> > <page> > <id>1</id> > <rows> > <row id="2"/> > <row id="3"/> > </rows> > </page> > </pages> > > <rows> > <row> > <id>2</id> > <columns> > <column width="18"> > <frames> > <frame id="4"/> > <frame id="5"/> > </frames> > </column> > <column width="6"> > <frames> > <frame id="6/> > </frames> > </column> > </columns> > </row> > </rows> > > <frames> > <frame> > <id>4</id> > <contents> > <content id="7"/> > </contents> > </frame> > </frames> > > <contents> > <content type="TEXT"/> > <id>7</id> > <body> > <p>Some text here.</p> > </body> > </content> > <content type="WIDGET"/> > <id>7</id> > <!-- Widget parameters here --> > </content> > </contents> > > Thanks! > Chas. > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

On 12/29/2010 2:17 AM, Huib Verweij wrote: > Hi Chas, > > using the following documents in the collection "Chas": > > <pages> > <pageid="1"> > <rows> > <rowref="2"/> > <rowref="3"/> > </rows> > </page> > </pages> > > <rows> > <rowid="2"> > <columns> > <columnwidth="18"> > <frames> > <frameref="4"/> > <frameref="5"/> > </frames> > </column> > <columnwidth="6"> > <frames> > <frameref="6"/> > </frames> > </column> > </columns> > </row> > <rowid="3"> > <columns> > <columnwidth="18"> > <frames> > <frameref="4"/> > <frameref="5"/> > </frames> > </column> > </columns> > </row> > </rows> > > <frames> > <frameid="4"> > <contents> > <contentref="7"/> > </contents> > </frame> > <frameid="5"> > <contents> > <contentref="8"/> > </contents> > </frame> > <frameid="6"> > <contents> > <contentref="7"/> > <contentref="8"/> > </contents> > </frame> > </frames> > > > and > > > <contents> > <contenttype="TEXT"id="7"> > <body> > <p>Some text here.</p> > </body> > </content> > <contenttype="WIDGET"id="8"> > <!-- Widget parameters here --> > </content> > </contents> > > > > the following query: > > declare function local:getIt($n as node()) as node() { > if ($n/@ref) > then local:getIt(collection('Chas')//*[node-name() eq node-name($n) > and @id eq $n/@ref]) > else element {node-name($n)} { $n/@*, for $child in $n/* return if > ($child instance of element()) then local:getIt($child) else $child} > }; > > local:getIt(<page ref="1"/>) > > gives me this result: > > <pageid="1"> > <rows> > <rowid="2"> > <columns> > <columnwidth="18"> > <frames> > <frameid="4"> > <contents> > <contenttype="TEXT"id="7"> > <body> > <p/> > </body> > </content> > </contents> > </frame> > <frameid="5"> > <contents> > <contenttype="WIDGET"id="8"/> > </contents> > </frame> > </frames> > </column> > <columnwidth="6"> > <frames> > <frameid="6"> > <contents> > <contenttype="TEXT"id="7"> > <body> > <p/> > </body> > </content> > <contenttype="WIDGET"id="8"/> > </contents> > </frame> > </frames> > </column> > </columns> > </row> > <rowid="3"> > <columns> > <columnwidth="18"> > <frames> > <frameid="4"> > <contents> > <contenttype="TEXT"id="7"> > <body> > <p/> > </body> > </content> > </contents> > </frame> > <frameid="5"> > <contents> > <contenttype="WIDGET"id="8"/> > </contents> > </frame> > </frames> > </column> > </columns> > </row> > </rows> > </page> > > > Is that what you are looking for? (I've replaced the @id attributes > with @ref attributes and the <id/> elements with @id attributes and > fixed up the <content/> elements, but otherwise it's roughly the > same.) It doesn't have all the bells and whistles yet and doesn't even > check if an element can be found in the database, but these things can > be added later. > > Regards, > > Huib Verweij. > > > > > > > > Op 29 dec 2010, om 04:47 heeft Charles F. Munat het volgende geschreven: > >> I have a BaseX backed website which stores all the pieces of the page >> in the DB. There is a significant amount of processing to be done. >> Pretty much any piece of the page can have effective or expiration >> dates (so that page parts can come and go on schedule). Some pages >> have dynamic elements, such as a Twitter feed. The page is build from >> objects which are wrapped in frames (for adding temporal and styling >> parameters), which are inserted into columns, which go into rows, >> which make up a page (which itself is wrapped in a template). So >> there is a lot of recursion. >> >> I do the recursion using local functions. >> >> It works well, but the page-building query is getting enormous, most >> of which is all these functions to insert parts of the page, often >> recursively. >> >> I am wondering what the best practice would be for this. I'm working >> in Scala, and I could break the process up into repeated passes >> hitting the db with a different query for each, but it seems to me >> one big query will almost certainly be faster. But including all the >> local functions in the query makes it huge. >> >> Is there some way to compile and load the functions once on startup, >> and then just run a small XQuery that calls functions which call >> functions, etc.? >> >> Any ideas (or resources) for ways to optimize the query? Everything >> gets reused, so there is very little nesting -- mostly references are >> passed. For example: >> >> <pages> >> <page> >> <id>1</id> >> <rows> >> <row id="2"/> >> <row id="3"/> >> </rows> >> </page> >> </pages> >> >> <rows> >> <row> >> <id>2</id> >> <columns> >> <column width="18"> >> <frames> >> <frame id="4"/> >> <frame id="5"/> >> </frames> >> </column> >> <column width="6"> >> <frames> >> <frame id="6/> >> </frames> >> </column> >> </columns> >> </row> >> </rows> >> >> <frames> >> <frame> >> <id>4</id> >> <contents> >> <content id="7"/> >> </contents> >> </frame> >> </frames> >> >> <contents> >> <content type="TEXT"/> >> <id>7</id> >> <body> >> <p>Some text here.</p> >> </body> >> </content> >> <content type="WIDGET"/> >> <id>7</id> >> <!-- Widget parameters here --> >> </content> >> </contents> >> >> Thanks! >> Chas. >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> <mailto:BaseX-Talk at mailman.uni-konstanz.de> >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

On 12/29/2010 2:17 AM, Huib Verweij wrote: > Hi Chas, > > using the following documents in the collection "Chas": > > <pages> > <pageid="1"> > <rows> > <rowref="2"/> > <rowref="3"/> > </rows> > </page> > </pages> > > <rows> > <rowid="2"> > <columns> > <columnwidth="18"> > <frames> > <frameref="4"/> > <frameref="5"/> > </frames> > </column> > <columnwidth="6"> > <frames> > <frameref="6"/> > </frames> > </column> > </columns> > </row> > <rowid="3"> > <columns> > <columnwidth="18"> > <frames> > <frameref="4"/> > <frameref="5"/> > </frames> > </column> > </columns> > </row> > </rows> > > <frames> > <frameid="4"> > <contents> > <contentref="7"/> > </contents> > </frame> > <frameid="5"> > <contents> > <contentref="8"/> > </contents> > </frame> > <frameid="6"> > <contents> > <contentref="7"/> > <contentref="8"/> > </contents> > </frame> > </frames> > > > and > > > <contents> > <contenttype="TEXT"id="7"> > <body> > <p>Some text here.</p> > </body> > </content> > <contenttype="WIDGET"id="8"> > <!-- Widget parameters here --> > </content> > </contents> > > > > the following query: > > declare function local:getIt($n as node()) as node() { > if ($n/@ref) > then local:getIt(collection('Chas')//*[node-name() eq node-name($n) > and @id eq $n/@ref]) > else element {node-name($n)} { $n/@*, for $child in $n/* return if > ($child instance of element()) then local:getIt($child) else $child} > }; > > local:getIt(<page ref="1"/>) > > gives me this result: > > <pageid="1"> > <rows> > <rowid="2"> > <columns> > <columnwidth="18"> > <frames> > <frameid="4"> > <contents> > <contenttype="TEXT"id="7"> > <body> > <p/> > </body> > </content> > </contents> > </frame> > <frameid="5"> > <contents> > <contenttype="WIDGET"id="8"/> > </contents> > </frame> > </frames> > </column> > <columnwidth="6"> > <frames> > <frameid="6"> > <contents> > <contenttype="TEXT"id="7"> > <body> > <p/> > </body> > </content> > <contenttype="WIDGET"id="8"/> > </contents> > </frame> > </frames> > </column> > </columns> > </row> > <rowid="3"> > <columns> > <columnwidth="18"> > <frames> > <frameid="4"> > <contents> > <contenttype="TEXT"id="7"> > <body> > <p/> > </body> > </content> > </contents> > </frame> > <frameid="5"> > <contents> > <contenttype="WIDGET"id="8"/> > </contents> > </frame> > </frames> > </column> > </columns> > </row> > </rows> > </page> > > > Is that what you are looking for? (I've replaced the @id attributes > with @ref attributes and the <id/> elements with @id attributes and > fixed up the <content/> elements, but otherwise it's roughly the > same.) It doesn't have all the bells and whistles yet and doesn't even > check if an element can be found in the database, but these things can > be added later. > > Regards, > > Huib Verweij. > > > > > > > > Op 29 dec 2010, om 04:47 heeft Charles F. Munat het volgende geschreven: > >> I have a BaseX backed website which stores all the pieces of the page >> in the DB. There is a significant amount of processing to be done. >> Pretty much any piece of the page can have effective or expiration >> dates (so that page parts can come and go on schedule). Some pages >> have dynamic elements, such as a Twitter feed. The page is build from >> objects which are wrapped in frames (for adding temporal and styling >> parameters), which are inserted into columns, which go into rows, >> which make up a page (which itself is wrapped in a template). So >> there is a lot of recursion. >> >> I do the recursion using local functions. >> >> It works well, but the page-building query is getting enormous, most >> of which is all these functions to insert parts of the page, often >> recursively. >> >> I am wondering what the best practice would be for this. I'm working >> in Scala, and I could break the process up into repeated passes >> hitting the db with a different query for each, but it seems to me >> one big query will almost certainly be faster. But including all the >> local functions in the query makes it huge. >> >> Is there some way to compile and load the functions once on startup, >> and then just run a small XQuery that calls functions which call >> functions, etc.? >> >> Any ideas (or resources) for ways to optimize the query? Everything >> gets reused, so there is very little nesting -- mostly references are >> passed. For example: >> >> <pages> >> <page> >> <id>1</id> >> <rows> >> <row id="2"/> >> <row id="3"/> >> </rows> >> </page> >> </pages> >> >> <rows> >> <row> >> <id>2</id> >> <columns> >> <column width="18"> >> <frames> >> <frame id="4"/> >> <frame id="5"/> >> </frames> >> </column> >> <column width="6"> >> <frames> >> <frame id="6/> >> </frames> >> </column> >> </columns> >> </row> >> </rows> >> >> <frames> >> <frame> >> <id>4</id> >> <contents> >> <content id="7"/> >> </contents> >> </frame> >> </frames> >> >> <contents> >> <content type="TEXT"/> >> <id>7</id> >> <body> >> <p>Some text here.</p> >> </body> >> </content> >> <content type="WIDGET"/> >> <id>7</id> >> <!-- Widget parameters here --> >> </content> >> </contents> >> >> Thanks! >> Chas. >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> <mailto:BaseX-Talk at mailman.uni-konstanz.de> >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

> SET RUNS N

Am 29.12.2010 um 07:40 schrieb Charles F. Munat: > The other question I had was if it was possible to pre-load the embedded database with the functions, rather than having to load them all at the time the xquery is run. Sort of the way functions or triggers are preloaded into an RDBMS. Do you happen to know if that's possible?

Am 29.12.10 11:08, schrieb Michael Seiferle: > Hi Chas, > (and hi Huib), > > thanks for your input! > > At the moment preloading& compiling queries ahead of time is not possible. > The main reason for this is that, from our experience, queries are usually rather short and cheap to compile. > > Providing an infrastructure that takes care of precompiled queries might present quite a challenge, yet it should not be impossible. > > You two probably knew this, but it might be of interest to the list as well, there's a very basic benchmark mechanism: > >> SET RUNS N > Now every query you issue is run N-times, including parsing& compiling. You will get average duration values for each step, if you enable the query info view. > > This is not representative, but on my machine the 40KB functx library [1] takes an avg of ~13ms to parse and ~7ms to compile, > so the overhead introduced by reparsing/recompiling should be rather low in general. > > In case you experience particular problems with specific queries feel free to contact us. > Often tweaking the query a little, such that possible index optimizations are recognized correctly by our compiler will speed up queries considerably. > > I hope this helps, feel free to ask for more! Feedback is very welcome :-) > > Kind regards > Michael > > > Am 29.12.2010 um 07:40 schrieb Charles F. Munat: > >> The other question I had was if it was possible to pre-load the embedded database with the functions, rather than having to load them all at the time the xquery is run. Sort of the way functions or triggers are preloaded into an RDBMS. Do you happen to know if that's possible? > [1] http://www.xqueryfunctions.com/xq/download.html > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

> Good to know. I'm not worried about performance that much. I'm sure I can > optimize the query(s) once I get them all figured out. What would have made > the ability to store functions nice is that there are many that are reused > in different queries, so currently I have to have multiple copies. Then if I > make a change, I have to remember to make it to all of them. Not a big deal, > just a matter of efficiency. But I can work around this on my end by storing > the functions in one spot and then just inserting them programatically as > necessary into the query. > > When I get to optimizing, I will gladly take advantage of your offer. Huib > has already given me some good ideas. > > Chas. > > > On 12/29/2010 7:08 AM, Michael Seiferle wrote: > >> Hi Chas, >> (and hi Huib), >> >> thanks for your input! >> >> At the moment preloading& compiling queries ahead of time is not >> possible. >> The main reason for this is that, from our experience, queries are usually >> rather short and cheap to compile. >> >> Providing an infrastructure that takes care of precompiled queries might >> present quite a challenge, yet it should not be impossible. >> >> You two probably knew this, but it might be of interest to the list as >> well, there's a very basic benchmark mechanism: >> >> SET RUNS N >>> >> Now every query you issue is run N-times, including parsing& compiling. >> You will get average duration values for each step, if you enable the query >> info view. >> >> This is not representative, but on my machine the 40KB functx library [1] >> takes an avg of ~13ms to parse and ~7ms to compile, >> so the overhead introduced by reparsing/recompiling should be rather low >> in general. >> >> In case you experience particular problems with specific queries feel free >> to contact us. >> Often tweaking the query a little, such that possible index optimizations >> are recognized correctly by our compiler will speed up queries considerably. >> >> I hope this helps, feel free to ask for more! Feedback is very welcome :-) >> >> Kind regards >> Michael >> >> >> Am 29.12.2010 um 07:40 schrieb Charles F. Munat: >> >> The other question I had was if it was possible to pre-load the embedded >>> database with the functions, rather than having to load them all at the time >>> the xquery is run. Sort of the way functions or triggers are preloaded into >>> an RDBMS. Do you happen to know if that's possible? >>> >> [1] http://www.xqueryfunctions.com/xq/download.html >> > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

On 12/29/2010 7:08 AM, Michael Seiferle wrote: > Hi Chas, > (and hi Huib), > > thanks for your input! > > At the moment preloading& compiling queries ahead of time is not possible. > The main reason for this is that, from our experience, queries are usually rather short and cheap to compile. > > Providing an infrastructure that takes care of precompiled queries might present quite a challenge, yet it should not be impossible. > > You two probably knew this, but it might be of interest to the list as well, there's a very basic benchmark mechanism: > >> SET RUNS N > Now every query you issue is run N-times, including parsing& compiling. You will get average duration values for each step, if you enable the query info view. > > This is not representative, but on my machine the 40KB functx library [1] takes an avg of ~13ms to parse and ~7ms to compile, > so the overhead introduced by reparsing/recompiling should be rather low in general. > > In case you experience particular problems with specific queries feel free to contact us. > Often tweaking the query a little, such that possible index optimizations are recognized correctly by our compiler will speed up queries considerably. > > I hope this helps, feel free to ask for more! Feedback is very welcome :-) > > Kind regards > Michael > > > Am 29.12.2010 um 07:40 schrieb Charles F. Munat: > >> The other question I had was if it was possible to pre-load the embedded database with the functions, rather than having to load them all at the time the xquery is run. Sort of the way functions or triggers are preloaded into an RDBMS. Do you happen to know if that's possible? > [1] http://www.xqueryfunctions.com/xq/download.html

On 12/29/2010 1:48 PM, Jan Vlčinský (CAD) wrote: > Hi > It seems like client side library manager would be sort of option. > Possibly with components like: > > 1. library storage (central? some sort of source control system, > e.g. Mercurial?) > 2. merger - would take "live" XQuery and merge in any function from > library, which seems appropriate > 1. real time one - having libraries somewhere at hand and > inserting them there > 1. If you would use REST access to BaseX, it could be > implemented at web server as filter > 2. Or as I use Python API, I would modify the library > to intercept any XQuery command, and enhance the > XQuery content at this place. > 2. preprocessing way - building final XQuery once at the > moment source code is written and before it is run > multiple times > > Just some ideas. > > Jan > > 2010/12/29 Charles F. Munat <charles.munat at gmail.com > <mailto:charles.munat at gmail.com>> > > Good to know. I'm not worried about performance that much. I'm > sure I can optimize the query(s) once I get them all figured out. > What would have made the ability to store functions nice is that > there are many that are reused in different queries, so currently > I have to have multiple copies. Then if I make a change, I have to > remember to make it to all of them. Not a big deal, just a matter > of efficiency. But I can work around this on my end by storing the > functions in one spot and then just inserting them programatically > as necessary into the query. > > When I get to optimizing, I will gladly take advantage of your > offer. Huib has already given me some good ideas. > > Chas. > > > On 12/29/2010 7:08 AM, Michael Seiferle wrote: > > Hi Chas, > (and hi Huib), > > thanks for your input! > > At the moment preloading& compiling queries ahead of time is > not possible. > The main reason for this is that, from our experience, queries > are usually rather short and cheap to compile. > > Providing an infrastructure that takes care of precompiled > queries might present quite a challenge, yet it should not be > impossible. > > You two probably knew this, but it might be of interest to the > list as well, there's a very basic benchmark mechanism: > > SET RUNS N > > Now every query you issue is run N-times, including parsing& > compiling. You will get average duration values for each > step, if you enable the query info view. > > This is not representative, but on my machine the 40KB functx > library [1] takes an avg of ~13ms to parse and ~7ms to compile, > so the overhead introduced by reparsing/recompiling should be > rather low in general. > > In case you experience particular problems with specific > queries feel free to contact us. > Often tweaking the query a little, such that possible index > optimizations are recognized correctly by our compiler will > speed up queries considerably. > > I hope this helps, feel free to ask for more! Feedback is very > welcome :-) > > Kind regards > Michael > > > Am 29.12.2010 um 07:40 schrieb Charles F. Munat: > > The other question I had was if it was possible to > pre-load the embedded database with the functions, rather > than having to load them all at the time the xquery is > run. Sort of the way functions or triggers are preloaded > into an RDBMS. Do you happen to know if that's possible? > > [1] http://www.xqueryfunctions.com/xq/download.html > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > <mailto:BaseX-Talk at mailman.uni-konstanz.de> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > -- > *Ing. Jan Vlčinský* > CAD programy > Slunečnicová 338/3, 734 01 Karviná Ráj, Czech Republic > tel: +420-597 602 024; mob: +420-608 979 040 > skype: janvlcinsky; GoogleTalk: jan.vlcinsky at gmail.com > <mailto:jan.vlcinsky at gmail.com> > http://cz.linkedin.com/in/vlcinsky >

On Wed, Dec 29, 2010 at 6:32 AM, Charles F. Munat <charles.munat at gmail.com> wrote: > Good to know. I'm not worried about performance that much. I'm sure I can > optimize the query(s) once I get them all figured out. What would have made > the ability to store functions nice is that there are many that are reused > in different queries, so currently I have to have multiple copies. Then if I > make a change, I have to remember to make it to all of them. Not a big deal, > just a matter of efficiency. But I can work around this on my end by storing > the functions in one spot and then just inserting them programatically as > necessary into the query. > > When I get to optimizing, I will gladly take advantage of your offer. Huib > has already given me some good ideas. > > Chas. > > On 12/29/2010 7:08 AM, Michael Seiferle wrote: >> >> Hi Chas, >> (and hi Huib), >> >> thanks for your input! >> >> At the moment preloading& compiling queries ahead of time is not >> possible. >> The main reason for this is that, from our experience, queries are usually >> rather short and cheap to compile. >> >> Providing an infrastructure that takes care of precompiled queries might >> present quite a challenge, yet it should not be impossible. >> >> You two probably knew this, but it might be of interest to the list as >> well, there's a very basic benchmark mechanism: >> >>> SET RUNS N >> >> Now every query you issue is run N-times, including parsing& compiling. >> You will get average duration values for each step, if you enable the query >> info view. >> >> This is not representative, but on my machine the 40KB functx library [1] >> takes an avg of ~13ms to parse and ~7ms to compile, >> so the overhead introduced by reparsing/recompiling should be rather low >> in general. >> >> In case you experience particular problems with specific queries feel free >> to contact us. >> Often tweaking the query a little, such that possible index optimizations >> are recognized correctly by our compiler will speed up queries considerably. >> >> I hope this helps, feel free to ask for more! Feedback is very welcome :-) >> >> Kind regards >> Michael >> >> >> Am 29.12.2010 um 07:40 schrieb Charles F. Munat: >> >>> The other question I had was if it was possible to pre-load the embedded >>> database with the functions, rather than having to load them all at the time >>> the xquery is run. Sort of the way functions or triggers are preloaded into >>> an RDBMS. Do you happen to know if that's possible? >> >> [1] http://www.xqueryfunctions.com/xq/download.html > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >

On 12/29/2010 10:48 PM, Mark Boon wrote: > A little while ago I asked something similar. Even when it's not a > performance concern, then at least it very quickly gets to be a > developer's concern when each query is lead by a huge amount of unused > functions. > > What I did was the following, functions are stored in a function element: > > <function name='ar:A'> > declare function ar:A ( $arg as xs:string ) as xs:string > { > let $result := collection()//variable[@name=$arg]/@value > return ( if ($result) then $result/string() else > fn:concat('"',fn:concat($arg,' not found"')) ) > } > </function> > <function name='ar:B> > <use name="ar:A"/> > declare function ar:B ( $arg as xs:string ) as xs:string > { > ar:A($arg) > } > </function> > > and then when I call it: > > <query> > <use name="ar:B"/> > ar:B('test') > </query> > > and then I have an XSLT script that transforms the above by replacing > the<use> element with the text-node of the<function> element with > the same name at the time the XML file is read in. This is just an > unrealistically simple example of course. It works for me because my > queries are stored in an XML file. Otherwise you may have to invent a > similar method for your particular need. > > You may also want to check out the improved error-reporting I posted a > few threads back. > > Mark Boon > > (I hope GMail didn't mangle the formatting too much) > > On Wed, Dec 29, 2010 at 6:32 AM, Charles F. Munat > <charles.munat at gmail.com> wrote: >> Good to know. I'm not worried about performance that much. I'm sure I can >> optimize the query(s) once I get them all figured out. What would have made >> the ability to store functions nice is that there are many that are reused >> in different queries, so currently I have to have multiple copies. Then if I >> make a change, I have to remember to make it to all of them. Not a big deal, >> just a matter of efficiency. But I can work around this on my end by storing >> the functions in one spot and then just inserting them programatically as >> necessary into the query. >> >> When I get to optimizing, I will gladly take advantage of your offer. Huib >> has already given me some good ideas. >> >> Chas. >> >> On 12/29/2010 7:08 AM, Michael Seiferle wrote: >>> Hi Chas, >>> (and hi Huib), >>> >>> thanks for your input! >>> >>> At the moment preloading& compiling queries ahead of time is not >>> possible. >>> The main reason for this is that, from our experience, queries are usually >>> rather short and cheap to compile. >>> >>> Providing an infrastructure that takes care of precompiled queries might >>> present quite a challenge, yet it should not be impossible. >>> >>> You two probably knew this, but it might be of interest to the list as >>> well, there's a very basic benchmark mechanism: >>> >>>> SET RUNS N >>> Now every query you issue is run N-times, including parsing& compiling. >>> You will get average duration values for each step, if you enable the query >>> info view. >>> >>> This is not representative, but on my machine the 40KB functx library [1] >>> takes an avg of ~13ms to parse and ~7ms to compile, >>> so the overhead introduced by reparsing/recompiling should be rather low >>> in general. >>> >>> In case you experience particular problems with specific queries feel free >>> to contact us. >>> Often tweaking the query a little, such that possible index optimizations >>> are recognized correctly by our compiler will speed up queries considerably. >>> >>> I hope this helps, feel free to ask for more! Feedback is very welcome :-) >>> >>> Kind regards >>> Michael >>> >>> >>> Am 29.12.2010 um 07:40 schrieb Charles F. Munat: >>> >>>> The other question I had was if it was possible to pre-load the embedded >>>> database with the functions, rather than having to load them all at the time >>>> the xquery is run. Sort of the way functions or triggers are preloaded into >>>> an RDBMS. Do you happen to know if that's possible? >>> [1] http://www.xqueryfunctions.com/xq/download.html >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>

> This is an interesting idea, if a bit complex.