climbing around in maps

Thursday, 16 June 2016 · 2 messages
GS
Graydon Saunders16 Jun 2016 00:43 #
Hi -- I'm using 8.4.4. So I'm got a 15 MiB XML file . It consists of a bunch of definitions, which reference other definitions in the same file. It's not guaranteed to not be circular in the general case but I expect it isn't in my particular use case. (And I've stuck a "don't get too deep" check in anyway, which is currently sitting at 1 in the hopes that wouldn't blow up.) The idea is to find the paths to some particular values by started at those values and climbing up the references until a magic "this is the top of a chain of references" point is reached. (Which can be detected via a pattern in the name of the definition.) I stuffed the things I expect to care about into maps, since that way I process it once, and can hopefully reference it thereafter. The downside is that the stack dies instantly when I attempt to recurse up the (virtual, rather than XML-document-actual) tree. (I've never seen BaseX run the query plan along the bottom bar in lieu of an error message before.) I've attached the full query info. I can't attach the data because it's somebody else's. (entry has relationship children; an entry describes an XSD schema, and a relationship element describes (possibly through reference to another entry element than its ancestor) either a complex type or a simple type that goes in that schema. There's only one entry with a given @name but many relationship elements may reference it. As a result, the scope of what's being climbed expands rapidly; this first test "do one name" turns into 20 names on the first pass. ) The query looks like:
declare function local:climb($relByType as map(*), $built as xs:string*,
$name as xs:string)
{
  for $path in $relByType($name)
  return if (matches($path,'^\p{Lu}{4}_IN') or count($path) gt 1)
       then ($name,$built)
       else local:climb($relByType,($name,$built),$path)
};

let $relationships as element(relationship)+ := //relationship[@type]
let $relByType as map(*) := map:merge(for $x in $relationships
                            let $key := $x/@type
                            group by $key
                            return
map:entry($key,$x/ancestor::entry[1]/@name))

let $fixedRelationships as element(relationship)+ :=
//relationship[@type][@fixedValue]
let $fixed := map:merge(for $x in $fixedRelationships
                          let $key :=
$x/string-join((@name,@type,@fixedValue),'|')
                          group by $key
                          return map:entry($key,$x/ancestor::entry[1]))

(:trying a single fixed value as the start of the climb for testing:)
let $eventNames := map:get($fixed,'moodCode|CS|EVN.CRT')/@name

(: so we can get the relationship element, but we want its event ancestor's
name :)
return for $x in $eventNames return local:climb($relByType,'',$x)
My suspicion is that passing a big map to the function is causing the problem. But I don't know, and I don't think there's a way to declare a function with an external variable. ("take this value from the function declaration's context and put it in scope for the function"). Any suggestions? -- Graydon
GS
Graydon Saunders16 Jun 2016 00:59 #
I am such a cabbage: return if (matches($path,'^\p{Lu}{4}_IN') or count($path) gt 1) should be return if (matches($path,'^\p{Lu}{4}_IN') or count($built) gt 1) and happily goes up to path lengths of 50 without complain in <250 ms. So I guess the interesting thing is "dumping the query plan as the error message". Sorry! Graydon
Quoted text · Graydon Saunders · 76 lines
On Wed, Jun 15, 2016 at 4:43 PM, Graydon Saunders <graydonish at gmail.com> wrote: > Hi -- > > I'm using 8.4.4. > > So I'm got a 15 MiB XML file . It consists of a bunch of definitions, > which reference other definitions in the same file. It's not guaranteed to > not be circular in the general case but I expect it isn't in my particular > use case. (And I've stuck a "don't get too deep" check in anyway, which is > currently sitting at 1 in the hopes that wouldn't blow up.) > > The idea is to find the paths to some particular values by started at > those values and climbing up the references until a magic "this is the top > of a chain of references" point is reached. (Which can be detected via a > pattern in the name of the definition.) > > I stuffed the things I expect to care about into maps, since that way I > process it once, and can hopefully reference it thereafter. > > The downside is that the stack dies instantly when I attempt to recurse up > the (virtual, rather than XML-document-actual) tree. (I've never seen > BaseX run the query plan along the bottom bar in lieu of an error message > before.) > > I've attached the full query info. I can't attach the data because it's > somebody else's. (entry has relationship children; an entry describes an > XSD schema, and a relationship element describes (possibly through > reference to another entry element than its ancestor) either a complex type > or a simple type that goes in that schema. There's only one entry with a > given @name but many relationship elements may reference it. As a result, > the scope of what's being climbed expands rapidly; this first test "do one > name" turns into 20 names on the first pass. ) > > The query looks like: > > declare function local:climb($relByType as map(*), $built as xs:string*, > $name as xs:string) > { > for $path in $relByType($name) > return if (matches($path,'^\p{Lu}{4}_IN') or count($path) gt 1) > then ($name,$built) > else local:climb($relByType,($name,$built),$path) > }; > > let $relationships as element(relationship)+ := //relationship[@type] > let $relByType as map(*) := map:merge(for $x in $relationships > let $key := $x/@type > group by $key > return > map:entry($key,$x/ancestor::entry[1]/@name)) > > let $fixedRelationships as element(relationship)+ := > //relationship[@type][@fixedValue] > let $fixed := map:merge(for $x in $fixedRelationships > let $key := > $x/string-join((@name,@type,@fixedValue),'|') > group by $key > return map:entry($key,$x/ancestor::entry[1])) > > (:trying a single fixed value as the start of the climb for testing:) > let $eventNames := map:get($fixed,'moodCode|CS|EVN.CRT')/@name > > (: so we can get the relationship element, but we want its event > ancestor's name :) > return for $x in $eventNames return local:climb($relByType,'',$x) > > My suspicion is that passing a big map to the function is causing the > problem. But I don't know, and I don't think there's a way to declare a > function with an external variable. ("take this value from the function > declaration's context and put it in scope for the function"). > > Any suggestions? > > -- Graydon >

⚡Generated with XQuery