retrieve a sequence of all values within an attribute index

> Is there a way to retrieve a sequence of all values within an attribute > index?

On Mon, Jul 11, 2016 at 5:42 PM, Christian Grün <christian.gruen at gmail.com> wrote: > > Is there a way to retrieve a sequence of all values within an attribute > > index? > > Yep: http://docs.basex.org/wiki/Index_Module#index:attributes >

> any way to retrieve the index for a specific attribute name?

On Mon, Jul 11, 2016 at 6:21 PM, Christian Grün <christian.gruen at gmail.com> wrote: > > any way to retrieve the index for a specific attribute name? > > Nope, sorry. The index itself has no information on the location of > the text and attribute values. You’ll have to use distinct-values: > > distinct-values(//periodOfReport) > > If the number of distinct values is smaller than MAXCATS [1], the path > index will be utilized to speed up your query [2]. You can set MAXCATS > to a much larger value, but this might slow down the time required for > opening a database. > > Hope this helps > Christian > > [1] http://docs.basex.org/wiki/Options#MAXCATS > [2] http://docs.basex.org/wiki/Indexes#Path_Index >

> I wrote the following query which returns 59 distinct periods from a 8gb > db.. It's quite slow but it works

> let $periods := distinct-values(db:open("13F")//data/@periodOfReport) > let $transform := > <periods> > { > for $period in $periods > return <period>{$period}</period> > } > </periods> > > return file:write('/var/www/appusec3.jahtoe.com/xml/periods.xml', > $transform) > > > Regards > Alex > tech.jahtoe.com > bafila.jahtoe.com > > On Mon, Jul 11, 2016 at 6:21 PM, Christian Grün <christian.gruen at gmail.com> > wrote: >> >> > any way to retrieve the index for a specific attribute name? >> >> Nope, sorry. The index itself has no information on the location of >> the text and attribute values. You’ll have to use distinct-values: >> >> distinct-values(//periodOfReport) >> >> If the number of distinct values is smaller than MAXCATS [1], the path >> index will be utilized to speed up your query [2]. You can set MAXCATS >> to a much larger value, but this might slow down the time required for >> opening a database. >> >> Hope this helps >> Christian >> >> [1] http://docs.basex.org/wiki/Options#MAXCATS >> [2] http://docs.basex.org/wiki/Indexes#Path_Index > >

From: basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] On Behalf Of Christian Grün Sent: Monday, July 11, 2016 2:46 PM To: Alex Muir <alex.g.muir at gmail.com> Cc: BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] retrieve a sequence of all values within an attribute index > I wrote the following query which returns 59 distinct periods from a 8gb > db.. It's quite slow but it works Ah, well… I guess that all the values are numeric? In that case, only the min and max value will be stored in the statistics (and that won’t help you in fact). Bad luck. You can call index:facets("13F") to get more insight… Maybe we can fix that in future and store distinct numbers as well. > let $periods := distinct-values(db:open("13F")//data/@periodOfReport) > let $transform := > <periods> > { > for $period in $periods > return <period>{$period}</period> > } > </periods> > > return file:write('/var/www/appusec3.jahtoe.com/xml/periods.xml', > $transform) > > > Regards > Alex > tech.jahtoe.com > bafila.jahtoe.com > > On Mon, Jul 11, 2016 at 6:21 PM, Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> > wrote: >> >> > any way to retrieve the index for a specific attribute name? >> >> Nope, sorry. The index itself has no information on the location of >> the text and attribute values. You’ll have to use distinct-values: >> >> distinct-values(//periodOfReport) >> >> If the number of distinct values is smaller than MAXCATS [1], the path >> index will be utilized to speed up your query [2]. You can set MAXCATS >> to a much larger value, but this might slow down the time required for >> opening a database. >> >> Hope this helps >> Christian >> >> [1] http://docs.basex.org/wiki/Options#MAXCATS<http://docs.basex.org/wiki/Options#MAXCATS> >> [2] http://docs.basex.org/wiki/Indexes#Path_Index<http://docs.basex.org/wiki/Indexes#Path_Index> > >

On Mon, Jul 11, 2016 at 7:17 PM, Lizzi, Vincent < Vincent.Lizzi at taylorandfrancis.com> wrote: > I have a similar situation in which I want to get all distinct values of a > specific attribute. I’ve tried using 2 different approaches: group and > distinct-values. On small or medium size databases group tends to be > faster. When trying to get distinct values of a specific attribute from > large databases however both approaches are timing out for me. I’m looking > for a way to optimize this query: > > > > distinct-values(for $db in db:list() return > distinct-values(db:open($db)//@sec-type)) > > >

On Mon, Jul 11, 2016 at 10:56 PM, Alex Muir <alex.g.muir at gmail.com> wrote: > > On Mon, Jul 11, 2016 at 7:17 PM, Lizzi, Vincent > <Vincent.Lizzi at taylorandfrancis.com> wrote: >> >> I have a similar situation in which I want to get all distinct values of a >> specific attribute. I’ve tried using 2 different approaches: group and >> distinct-values. On small or medium size databases group tends to be faster. >> When trying to get distinct values of a specific attribute from large >> databases however both approaches are timing out for me. I’m looking for a >> way to optimize this query: >> >> >> >> distinct-values(for $db in db:list() return >> distinct-values(db:open($db)//@sec-type)) >> >> > > > > With the current logic available it look possible given an attribute index > on sec-type to associate a prefix onto the attribute value prior to > insertion into the database like sec-type="type:13F" with the prefix type: > and then use index:texts("dbname","type:") to get a distinct list of types > all be it with a prefix that would need adjusting logic in using that data > or querying. > > > > Regards > Alex > tech.jahtoe.com > bafila.jahtoe.com

From: Christian Grün [mailto:christian.gruen at gmail.com] Sent: Sunday, October 09, 2016 4:49 PM To: Alex Muir <alex.g.muir at gmail.com> Cc: Lizzi, Vincent <Vincent.Lizzi at taylorandfrancis.com>; BaseX <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] retrieve a sequence of all values within an attribute index Hi Vincent, hi Alex, I am glad to report that with BaseX 8.6 the distinct values of numeric elements and attributes will also be stored in the index. You are invited to check out the latest stable snapshot [1]. Cheers, Christian [1] http://files.basex.org/releases/latest/<http://files.basex.org/releases/latest/> On Mon, Jul 11, 2016 at 10:56 PM, Alex Muir <alex.g.muir at gmail.com<mailto:alex.g.muir at gmail.com>> wrote: > > On Mon, Jul 11, 2016 at 7:17 PM, Lizzi, Vincent > <Vincent.Lizzi at taylorandfrancis.com<mailto:Vincent.Lizzi at taylorandfrancis.com>> wrote: >> >> I have a similar situation in which I want to get all distinct values of a >> specific attribute. I’ve tried using 2 different approaches: group and >> distinct-values. On small or medium size databases group tends to be faster. >> When trying to get distinct values of a specific attribute from large >> databases however both approaches are timing out for me. I’m looking for a >> way to optimize this query: >> >> >> >> distinct-values(for $db in db:list() return >> distinct-values(db:open($db)//@sec-type)) >> >> > > > > With the current logic available it look possible given an attribute index > on sec-type to associate a prefix onto the attribute value prior to > insertion into the database like sec-type="type:13F" with the prefix type: > and then use index:texts("dbname","type:") to get a distinct list of types > all be it with a prefix that would need adjusting logic in using that data > or querying. > > > > Regards > Alex > tech.jahtoe.com > bafila.jahtoe.com