Query Execution Taking Too Long...
Monday, 23 March 2015 · 3 messages · last reply 24 Mar 2015
AK
ankit kumar23 Mar 2015 16:23 #
Hi,
I am trying to get all distinct start element '/products/p:category/start' nodes of a big file. I have written a query which is given below. It is taking to long to get the result. I am attaching the query info and the xml file.
After Running couple of minutes, I stopped the execution.
Query is trying to get all the distinct start elements
There are 3 lac category elements.
declare namespace functx = "http://www.functx.com";
declare namespace p="a:b:c";
declare function functx:is-node-in-sequence( $node as node()? , $seq as
node()* ) as xs:boolean {some $nodeInSeq in $seq satisfies
deep-equal($nodeInSeq,$node)};
declare function functx:distinct-nodes( $nodes as node()*
large.xml
<https://docs.google.com/file/d/0B_pB7l14skhVLWxvZGNyMnFwc0k/edit?usp=drive_web>
) as node()* {for $seq in (1 to count($nodes))return
$nodes[$seq][not(functx:is-node-in-sequence(.,$nodes[position() < $seq]))]}
;
let $diff_starts := functx:distinct-nodes(/products/p:category/start)
return $diff_startsPlease let me know if you require further details.
LW
Leonard Wörteler24 Mar 2015 01:33 #
Hi Ankit,
Quoted text · ankit kumar · 11 lines
Am 23.03.2015 um 16:23 schrieb ankit kumar: > I am trying to get all distinct start element > '/products/p:category/start' nodes of a big file. I have written a query > which is given below. It is taking to long to get the result. I am > attaching the query info and the xml file. > > After Running couple of minutes, I stopped the execution. > > Query is trying to get all the distinct start elements > > There are 3 lac category elements.
the function
functx:distinct-nodes($nodes) distinguishes the elements of its input sequence by their *node identity*, not the string contents. Since all of the nodes you iterate over are distinct, this will not do what you want. It also has very bad performance because each node is compared to all preceding ones.
If you want to get all distinct text contents, you can just use fn:distince-values($seq): declare namespace p="a:b:c";
fn:distinct-values(/products/p:category/start)Result:
Quoted text · ankit kumar · 3 lines
> 10-2-2012 > 10-5-2011 > 10-2-2009
If you also want one of the XML elements with those text contents you can use
group by: declare namespace p="a:b:c";
for $start in /products/p:category/start
group by $date := $start/string()
return <unique n="{count($start)}">{ $start[1] }</unique>Result:
Quoted text · ankit kumar · 9 lines
> <unique n="100000"> > <start xmlns:p="a:b:c">10-2-2012</start> > </unique> > <unique n="100000"> > <start xmlns:p="a:b:c">10-5-2011</start> > </unique> > <unique n="100000"> > <start xmlns:p="a:b:c">10-2-2009</start> > </unique>
Hope that helps,
Leo
LW
Leonard Wörteler24 Mar 2015 12:02 #
Hi Ankit (CC mailing list),
Quoted text · Leonard Wörteler · 6 lines
Am 24.03.2015 um 06:57 schrieb ankit kumar: > We could have used distinct-values() But, in our case we would like to > distinguish on the basis of attributes values. also distinct-values() > will be treating values of <a>1</a> and <a>01</a> as different. . > For the later case will be using custom node-correspond function instead > of deep-equal which will be doing comparison between two nodes.
the general problem is that grouping a set of elements where all you have is an equivalence predicate is always expensive to compute (i.e. quadratic in the size of the set). In order to speed up your use case you should therefore try to find some way to expose more information about the nodes to compare.
If it is possible to generate an identifier for each node that can be used for comparisons, then efficient hash-based algorithms can be used (e.g. those in
group by or distinct-values(...)). You could for example have a function my:id($node) so that my:id($n1) eq my:id($n2) holds iff node-correspond($n1, $n2) does. Then you can rewrite your query to run in linear time: declare namespace p="a:b:c";
declare namespace my="my://namespace";
declare function my:id($n) { (: your code here :) $n/string() };
for $start in /products/p:category/start
group by $id := my:id($start)
return $start[1]If you cannot generate such an ID, you can also use sorting to reduce running time. If you can define a comparison function
my:less-than($n1, $n2) that sorts the nodes into an order where equivalent items are next to each other, you can first sort the sequence in O(n log n) time (e.g. using hof:sort-with($seq, my:less-than#2) in BaseX) and then merge the runs of equivalent nodes in linear time.
Hope that helps,
LeoWritten for BaseX 8.0. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery