Time query takes looking for element with text() ID
Hi,
Sorry for all the questions of late first of all...
I've been using basex for a while now and wrote some nice xquery that gather metrics from datasets ~7000 files querying the whole corpus to create statistics and it quite fast at that but were thinking this query below feels slow at 34 seconds so we thought we should ask for you thoughts on it's duration.
Query:
let $content := db:open('F-DDEX')//MessageHeader/MessageThreadId[text() eq
'8937478']
return $contentContent: So every file has a message header with a MessageThreadId.
<MessageHeader xmlns:ern="http://ddex.net/xml/2011/ern-main/33" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance">
<MessageThreadId>8937478</MessageThreadId>
<MessageId>C2C977FDFDHF98DHF9D8FHEURYX</MessageId>
<MessageSender>
<PartyId>PA47F93H54HU93HJSFDINF</PartyId>
<PartyName>
<FullName>Warner Music Group</FullName>
</PartyName>
</MessageSender>
<MessageRecipient>
<PartyId>3G3E</PartyId>
<PartyName>
<FullName>3G3E-YADS</FullName>
</PartyName>
</MessageRecipient>
<MessageCreatedDateTime>2012-06-18T05:35:54Z</MessageCreatedDateTime>
</MessageHeader>Given the following database with text, attribute and fulltext indexes on.
Database Properties Name: F-DDEX Size: 5251 MB Nodes: 239945615 Documents: 7954
Specs Server:
$ cat /proc/meminfo | grep MemTotal MemTotal: 7633876 kB
$ cat /proc/cpuinfo | grep name model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz
Thoughts?
Thanks
Hi Alex,
Quoted text · Alex Muir · 3 lines
> let $content := db:open('F-DDEX')//MessageHeader/MessageThreadId[text() eq > '8937478'] > return $content
due to some specifics of the XQuery and XML semantics, the "eq" operator cannot also be rewritten for index access. The following query should be processed much faster:
let $content := db:open('F-DDEX')//MessageHeader/
MessageThreadId[text() = '8937478']
return $contentIf that's not the case, you could try to remove the double slash before MessageHeader (usually, it should be optimized by the compiler anyway). If that doesn't help either, you could prefix all element names with namespace prefixes..
let $content := db:open('F-DDEX')/*:MessageHeader/
*:MessageThreadId[text() = '8937478']
return $contentIf processing is still too slow, please provide us with a dump of the InfoView (or the verbose command line output triggered with -V).
Christian
Quoted text · Alex Muir · 59 lines
> <MessageHeader xmlns:ern="http://ddex.net/xml/2011/ern-main/33" > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> > <MessageThreadId>8937478</MessageThreadId> > <MessageId>C2C977FDFDHF98DHF9D8FHEURYX</MessageId> > <MessageSender> > <PartyId>PA47F93H54HU93HJSFDINF</PartyId> > <PartyName> > <FullName>Warner Music Group</FullName> > </PartyName> > </MessageSender> > <MessageRecipient> > <PartyId>3G3E</PartyId> > <PartyName> > <FullName>3G3E-YADS</FullName> > </PartyName> > </MessageRecipient> > <MessageCreatedDateTime>2012-06-18T05:35:54Z</MessageCreatedDateTime> > </MessageHeader> > > > Given the following database with text, attribute and fulltext indexes on. > > Database Properties > Name: F-DDEX > Size: 5251 MB > Nodes: 239945615 > Documents: 7954 > > > Specs Server: > > $ cat /proc/meminfo | grep MemTotal > MemTotal: 7633876 kB > > $ cat /proc/cpuinfo | grep name > model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz > model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz > > > Thoughts? > > Thanks > > -- > - > > Alex G. Muir > Software Engineering Consultant > Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125 > Love African Kora Music? Take a moment to listen to Gambia's - Amadu > Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com Your support keeps > Africa's griot tradition alive... Cheers! > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Hi Christian,
Thanks that = version runs in 0.4 seconds
This one returns an empty sequence however..
let $content := db:open('F-DDEX')/*:
Quoted text · Christian Grün · 4 lines
> > MessageHeader/ > *:MessageThreadId[text() = '8937478'] > return $content
Thanks Alex
On Thu, Nov 29, 2012 at 4:12 PM, Christian Grün <christian.gruen at gmail.com>wrote:
Quoted text · Christian Grün · 91 lines
> Hi Alex, > > > let $content := db:open('F-DDEX')//MessageHeader/MessageThreadId[text() > eq > > '8937478'] > > return $content > > due to some specifics of the XQuery and XML semantics, the "eq" > operator cannot also be rewritten for index access. The following > query should be processed much faster: > > let $content := db:open('F-DDEX')//MessageHeader/ > MessageThreadId[text() = '8937478'] > return $content > > If that's not the case, you could try to remove the double slash > before MessageHeader (usually, it should be optimized by the compiler > anyway). If that doesn't help either, you could prefix all element > names with namespace prefixes.. > > let $content := db:open('F-DDEX')/*:MessageHeader/ > *:MessageThreadId[text() = '8937478'] > return $content > > If processing is still too slow, please provide us with a dump of the > InfoView (or the verbose command line output triggered with -V). > > Christian > _____________________________________ > > > <MessageHeader xmlns:ern="http://ddex.net/xml/2011/ern-main/33" > > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"> > > <MessageThreadId>8937478</MessageThreadId> > > <MessageId>C2C977FDFDHF98DHF9D8FHEURYX</MessageId> > > <MessageSender> > > <PartyId>PA47F93H54HU93HJSFDINF</PartyId> > > <PartyName> > > <FullName>Warner Music Group</FullName> > > </PartyName> > > </MessageSender> > > <MessageRecipient> > > <PartyId>3G3E</PartyId> > > <PartyName> > > <FullName>3G3E-YADS</FullName> > > </PartyName> > > </MessageRecipient> > > <MessageCreatedDateTime>2012-06-18T05:35:54Z</MessageCreatedDateTime> > > </MessageHeader> > > > > > > Given the following database with text, attribute and fulltext indexes > on. > > > > Database Properties > > Name: F-DDEX > > Size: 5251 MB > > Nodes: 239945615 > > Documents: 7954 > > > > > > Specs Server: > > > > $ cat /proc/meminfo | grep MemTotal > > MemTotal: 7633876 kB > > > > $ cat /proc/cpuinfo | grep name > > model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz > > model name : Intel(R) Xeon(R) CPU E5430 @ 2.66GHz > > > > > > Thoughts? > > > > Thanks > > > > -- > > - > > > > Alex G. Muir > > Software Engineering Consultant > > Linkedin Profile : http://ca.linkedin.com/pub/alex-muir/36/ab7/125 > > Love African Kora Music? Take a moment to listen to Gambia's - Amadu > > Diabarte & Jali Bakary Konteh www.bafila.bandcamp.com Your support keeps > > Africa's griot tradition alive... Cheers! > > > > > > _______________________________________________ > > BaseX-Talk mailing list > > BaseX-Talk at mailman.uni-konstanz.de > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > >
Written for BaseX 7.3. BaseX may have changed since: please check the current documentation.