Performance (index?) on large intermediate structures
Wednesday, 29 August 2018 · 2 messages
SA
Sebastian Albert29 Aug 2018 09:02 #
Dear BaseX community
I am currently working with BaseX to write a rather long XQuery statement with quite large intermediate structures (XML documents / fragments) bound to variables (in FLWOR expressions), which I query by XPath using attributes etc. further down. Having nested loops, this is becoming very slow.
Are those intermediate structures automatically indexed? Or how can I achieve indexing on them?
Or do I need to write them into a database for this indexing? (Can I specify a datbase as "in-memory"?) Then I guess I need a "BaseX script" rather than a single XQuery? Of course, I am looking for the most elegant/idiomatic way.
Or is it best to build index-like intermediate structures myself, probably using maps rather than XML documents?
Furthermore, are there profiling tools inside BaseX, or would I need to fire up a Java IDE to see where most time is being spent?
Thanks in advance for any suggestions!
Sebastian
CG
Christian Grün29 Aug 2018 09:14 #
Dear Sebastian,
In fact XQuery maps are very fast, and they are a great data structure for speeding up repeated lookups if you encounter cases in which this is not done automatically by the optimizer.
If you manage to send us a little example query that demonstrates the pattern you are working on, we may be able to give some more specific help.
Best,
Christian
Quoted text · Sebastian Albert · 28 lines
On Wed, Aug 29, 2018 at 9:02 AM Sebastian Albert <albert at math.uni-goettingen.de> wrote: > > Dear BaseX community > > I am currently working with BaseX to write a rather long XQuery > statement with quite large intermediate structures (XML documents / > fragments) bound to variables (in FLWOR expressions), which I query by > XPath using attributes etc. further down. Having nested loops, this is > becoming very slow. > > Are those intermediate structures automatically indexed? Or how can I > achieve indexing on them? > > Or do I need to write them into a database for this indexing? (Can I > specify a datbase as "in-memory"?) Then I guess I need a "BaseX script" > rather than a single XQuery? Of course, I am looking for the most > elegant/idiomatic way. > > Or is it best to build index-like intermediate structures myself, > probably using maps rather than XML documents? > > Furthermore, are there profiling tools inside BaseX, or would I need to > fire up a Java IDE to see where most time is being spent? > > Thanks in advance for any suggestions! > Sebastian >
Written for BaseX 9.0. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery