Main Page » Mailing List » May 2022

Constructing and querying map structures in a dynamic way

Wednesday, 25 May 2022 · 13 messages · last reply 28 May 2022
ME
Markus Elfring25 May 2022 17:00 #
Hello, I would like to construct a map structure based on a dynamic selection by the means of a FLWOR expression. https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-maps The keys and associated values should be presented as a CSV-like file by a subsequent data processing step. Can it become easier to find existing XQuery script examples for such an use case? Regards, Markus
ME
Markus Elfringin reply to Graydon, 25 May 2022, 19:0725 May 2022 17:35 #
Quoted text · Graydon · 1 line
> Maps are a derivative of the function type and highly general.
I got informed that the data structure “map” got a clear meaning according to XQuery programming interfaces.
Quoted text · Graydon · 2 lines
> You're in effect asking for examples of using a function with no other > information provided.
I came along another clarification desire according to the mentioned standard data structure.
Quoted text · Graydon · 1 line
> It's just about impossible to answer usefully.
The standard specification just shows examples for topics like “days” and “books” with map constructors based on literal data. How would the XQuery code look like if a detailed FLWOR expression would be applied instead? The subsequent CSV result output should show finally that keys and associated values were properly determined before. Regards, Markus
MH
Martin Honnen25 May 2022 17:49 #
Quoted text · Markus Elfring · 4 lines
Am 25.05.2022 um 17:35 schrieb Markus Elfring: > > The standard specification just shows examples for topics like “days” > and “books” with map constructors based on literal data.
Perhaps https://www.w3.org/TR/xquery-31-requirements/ shows some more complex queries.
MS
Michael Seiferlein reply to Markus Elfring, 25 May 2022, 17:3525 May 2022 17:57 #
Hi Markus, The standard specification just shows examples for topics like “days” and “books” with map constructors based on literal data. If you are looking for functions to construct maps dynamically, maybe map:merge & map:entry are what you are after?
map:merge(
  for $i at $pos in 1 to 10
  return map:entry($i, $i*$i)
)
Best Michael
Quoted text · BaseX-Talk · 21 lines
Von: BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> im Auftrag von Markus Elfring <Markus.Elfring at web.de> Datum: Mittwoch, 25. Mai 2022 um 17:35 An: Graydon Saunders <graydonish at gmail.com> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] Constructing and querying map structures in a dynamic way > Maps are a derivative of the function type and highly general. I got informed that the data structure “map” got a clear meaning according to XQuery programming interfaces. > You're in effect asking for examples of using a function with no other > information provided. I came along another clarification desire according to the mentioned standard data structure. > It's just about impossible to answer usefully. The standard specification just shows examples for topics like “days” and “books” with map constructors based on literal data. How would the XQuery code look like if a detailed FLWOR expression would be applied instead? The subsequent CSV result output should show finally that keys and associated values were properly determined before. Regards, Markus
HR
Hans-Juergen Rennauin reply to Markus Elfring, 25 May 2022, 17:0025 May 2022 18:13 #
Hi Markus, although I do not quite understand your description (why should I care for the representation of the map I construct?), it may be that one piece of information is useful to you, as it is not obvious from the spec: it's about how to construct a map dynamically, that is, (1) when the keys are not known beforehand, (2) and/or when the entry values are assigned in "iterations" of a FLWOR expression (Especially (2) might be what puzzles you.) (1) and (2) can't be accomplished when using the map constructor expression, like this: basex "map{'x':1, 'y': false()}" The trick consists of using a combination of the functions map:merge() [1] and map:entry() [2]. So summarize, the code constructing the map entries is the argument of function map:merge().This code may for example be (most often is) a FLWOR expression. In this code, each entry is constructed by a call of function map:entry(). For a tiny example see below. Kind regards,Hans-Jürgen PS: Tiny example demonstrating the construction of maps via map:merge() & map:entry(). (: Creates a map with one entry for each element name encountered in the input document. The entry value is the number of elements with that name. As the map keys are not known in advance, map:merge() is used. :)map:merge( for $elem in //* group by $name := $elem/local-name(.) return map:entry($name, count($elem))) [1] https://www.w3.org/TR/xpath-functions-31/#func-map-merge[2] https://www.w3.org/TR/xpath-functions-31/#func-map-entry Am Mittwoch, 25. Mai 2022, 16:59:48 MESZ hat Markus Elfring <markus.elfring at web.de> Folgendes geschrieben: Hello, I would like to construct a map structure based on a dynamic selection by the means of a FLWOR expression. https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-maps The keys and associated values should be presented as a CSV-like file by a subsequent data processing step. Can it become easier to find existing XQuery script examples for such an use case? Regards, Markus
G
Graydonin reply to Markus Elfring, 25 May 2022, 17:0025 May 2022 19:07 #
Hi Markus --
Quoted text · Markus Elfring · 9 lines
On Wed, May 25, 2022 at 05:00:22PM +0200, Markus Elfring scripsit: > I would like to construct a map structure based on a dynamic selection by > the means of a FLWOR expression. > https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-maps > > The keys and associated values should be presented as a CSV-like file by > a subsequent data processing step. > > Can it become easier to find existing XQuery script examples for such an use case?
Maps are a derivative of the function type and highly general. You're in effect asking for examples of using a function with no other information provided. It's just about impossible to answer usefully. Can you provide something tangible and specific? Some actual XML input, or some XML with the same structure as your intended input, and some indication of how it should work in the form of a line or three of the expected CSV result corresponding to that input? Thanks!
-- Graydon Saunders | graydonish at gmail.com Þæs oferéode, ðisses swá mæg. -- Deor ("That passed, so may this.")
MS
Majewski, Steven Dennis (sdm7g)in reply to Hans-Juergen Rennau, 25 May 2022, 18:1325 May 2022 19:19 #
Note that the optional options argument in map:merge can change the way it handles duplicate keys. I’ve used this to create maps of normalized document ids from two different collections to de-dup, Where the ids SHOULD be uniq within a collection, but actually aren’t, or are missing. https://www.w3.org/TR/xpath-functions-31/#func-map-merge ( Next step was to split the uniq and non-uniq ones to handle separately. )
declare variable $OAI := map:merge( for $d in $ORG_OAI
  return map:entry( local:normalize( local:unitid($d )), $d ), map{'duplicates':'combine'});
declare variable $VH := map:merge( for $d in $ORG_VH
  return map:entry( local:normalize( local:unitid($d )), $d ), map{'duplicates':'combine'});
However, after discussion on this list about SQL join equivalents, I discovered I could do something almost equivalent with a for/join, and that full text search tokenization gives me a similar way of normalizing IDs:
for $VH in collection( $R($ORG)('vh'))/ead/archdesc/did/unitid,
                $OAI in collection( $R($ORG)('oai'))/ead/archdesc/did/unitid[ . contains text  {string($VH)}]
      		order by $VH
I think the map version may be a bit faster, but I haven’t actually measured And they are doing things a bit differently, as I’m not removing the internal duplicates, But using the order by to group multiple matches together to make it visible in the report. I would also note that dealing with messy data, often with hidden encoding errors in documents, Is the reason I was asking in a previous thread about using the result of one script as context for other queries. Sometimes this ends up being a rather exploratory process, so that has come in very handy. — Steve M.
Quoted text · Hans-Juergen Rennau · 112 lines
> On May 25, 2022, at 12:13 PM, Hans-Juergen Rennau <hrennau at yahoo.de> wrote: > > > > > Hi Markus, > > > although I do not quite understand your description (why should I care for the representation of the map I construct?), it may be that one piece of information is useful to you, as it is not obvious from the spec: it's about how to construct a map dynamically, that is, > > > (1) when the keys are not known beforehand, > (2) and/or when the entry values are assigned in "iterations" of a FLWOR expression > > > (Especially (2) might be what puzzles you.) > > > (1) and (2) can't be accomplished when using the map constructor expression, like this: basex "map{'x':1, 'y': false()}" > > > The trick consists of using a combination of the functions map:merge() [1] and map:entry() [2]. > > > So summarize, the code constructing the map entries is the argument of function map:merge().This code may for example be (most often is) a FLWOR expression. In this code, each entry is constructed by a call of function map:entry(). For a tiny example see below. > > > Kind regards, > Hans-Jürgen > > > PS: Tiny example demonstrating the construction of maps via map:merge() & map:entry(). > > > (: Creates a map with one entry for each element name > encountered in the input document. The entry value > is the number of elements with that name. As the > map keys are not known in advance, map:merge() is > used. > :) > map:merge( > for $elem in //* > group by $name := $elem/local-name(.) > return map:entry($name, count($elem)) > ) > > > > > > [1] https://www.w3.org/TR/xpath-functions-31/#func-map-merge > [2] https://www.w3.org/TR/xpath-functions-31/#func-map-entry > > > > > > > > > > > > > > > > > > > > > Am Mittwoch, 25. Mai 2022, 16:59:48 MESZ hat Markus Elfring <markus.elfring at web.de> Folgendes geschrieben: > > > > > > > > > Hello, > > > > I would like to construct a map structure based on a dynamic selection by > > the means of a FLWOR expression. > > https://www.w3.org/TR/2017/REC-xquery-31-20170321/#id-maps > > > > The keys and associated values should be presented as a CSV-like file by > > a subsequent data processing step. > > > > Can it become easier to find existing XQuery script examples for such an use case? > > > > Regards, > > Markus > > > > > >
ME
Markus Elfringin reply to Hans-Juergen Rennau, 25 May 2022, 18:1327 May 2022 10:50 #
Quoted text · Hans-Juergen Rennau · 1 line
> although I do not quite understand your description
Which wording variant would you have found clearer for the mentioned use case?
Quoted text · Hans-Juergen Rennau · 1 line
> (why should I care for the representation of the map I construct?),
* Selection of a general data structure for the handling of key/value pairs * I guess that you would like to put customised data into keys and associated values.
Quoted text · Hans-Juergen Rennau · 1 line
> it may be that one piece of information is useful to you,
Thanks for your constructive feedback.
Quoted text · Hans-Juergen Rennau · 1 line
> as it is not obvious from the spec: it's about how to construct a map dynamically,
Can such a view trigger any further consequences?
Quoted text · Hans-Juergen Rennau · 6 lines
> that is, > > (1) when the keys are not known beforehand, > (2) and/or when the entry values are assigned in "iterations" of a FLWOR expression > > (Especially (2) might be what puzzles you.)
I thought also about this data processing area in special ways for a moment.
Quoted text · Hans-Juergen Rennau · 1 line
> (1) and (2) can't be accomplished when using the map constructor expression, like this: basex "map{'x':1, 'y': false()}"
Will any information sources help to understand this technical detail better (besides the XPath function library)?
Quoted text · Hans-Juergen Rennau · 1 line
> The trick consists of using a combination of the functions map:merge() [1] and map:entry() [2].
I became curious how application considerations will evolve further.
Quoted text · Hans-Juergen Rennau · 2 lines
> This code may for example be (most often is) a FLWOR expression. > In this code, each entry is constructed by a call of function map:entry().
Now I stumble on the error message “[XPTY0004] Item expected, empty sequence found.” from my software test according to a special data source. How would you determine where unexpected gaps occur? Regards, Markus
HR
Hans-Juergen Rennau27 May 2022 15:07 #
Hi Markus, you wrote "How would you determine where unexpected gaps occur?" Gaps? First, to avoid a misunderstanding: whereas map keys must not be the empty sequence, map values may. An elegant way to construct map entries safely - taking care that the key is not empty - is to use the map operator (!) or a path step, in both cases with the map:entry() on the right-hand side. Example, map operator:. $thingy/@name/normalize-space(.)[string()] ! map:entry(., $tingy/string()) Example, path step: map:merge(doc('things.xml')//thingy/@name[normalize-space(.)]/map:entry(., ../string())) Kind regards, Hans-Jürgen Am Freitag, 27. Mai 2022, 10:51:00 MESZ hat Markus Elfring <markus.elfring at web.de> Folgendes geschrieben:
Quoted text · Markus Elfring · 1 line
> although I do not quite understand your description
Which wording variant would you have found clearer for the mentioned use case?
Quoted text · Markus Elfring · 1 line
> (why should I care for the representation of the map I construct?),
* Selection of a general data structure for the handling of key/value pairs * I guess that you would like to put customised data into keys and associated values.
Quoted text · Markus Elfring · 1 line
> it may be that one piece of information is useful to you,
Thanks for your constructive feedback.
Quoted text · Markus Elfring · 1 line
> as it is not obvious from the spec: it's about how to construct a map dynamically,
Can such a view trigger any further consequences?
Quoted text · Markus Elfring · 6 lines
> that is, > > (1) when the keys are not known beforehand, > (2) and/or when the entry values are assigned in "iterations" of a FLWOR expression > > (Especially (2) might be what puzzles you.)
I thought also about this data processing area in special ways for a moment.
Quoted text · Markus Elfring · 1 line
> (1) and (2) can't be accomplished when using the map constructor expression, like this: basex "map{'x':1, 'y': false()}"
Will any information sources help to understand this technical detail better (besides the XPath function library)?
Quoted text · Markus Elfring · 1 line
> The trick consists of using a combination of the functions map:merge() [1] and map:entry() [2].
I became curious how application considerations will evolve further.
Quoted text · Markus Elfring · 2 lines
> This code may for example be (most often is) a FLWOR expression. > In this code, each entry is constructed by a call of function map:entry().
Now I stumble on the error message “[XPTY0004] Item expected, empty sequence found.” from my software test according to a special data source. How would you determine where unexpected gaps occur? Regards, Markus
ME
Markus Elfring27 May 2022 15:37 #
Quoted text · Hans-Juergen Rennau · 1 line
> Gaps? First, to avoid a misunderstanding: whereas map keys must not be the empty sequence, map values may.
Special data sources might contain challenges for further clarification of the desired referential integrity. Regards, Markus
G
Graydonin reply to Markus Elfring, 27 May 2022, 10:5027 May 2022 15:38 #
Quoted text · Markus Elfring · 8 lines
On Fri, May 27, 2022 at 10:50:57AM +0200, Markus Elfring scripsit: > > This code may for example be (most often is) a FLWOR expression. In > > this code, each entry is constructed by a call of function > > map:entry(). > > Now I stumble on the error message “[XPTY0004] Item expected, empty > sequence found.” from my software test according to a special data > source.
So either the key or the value isn't in the source data.
Quoted text · Markus Elfring · 1 line
> How would you determine where unexpected gaps occur?
let $simple as element(bucket) :=
  <bucket>
    <thingy name="oleander"/>
    <thingy name="primrose">likes well-drained partial shade</thingy>
    <thingy/>
    <thingy>zone 7</thingy>
    <thingy name="tulip">marsh plant, not a swamp plant</thingy>
  </bucket>

let $thingMap as map(xs:string,xs:string) := map:merge(
  for $thing in $simple/descendant::thingy
    let $key as xs:string? := $thing/@name/string()
    let $value as xs:string? := $thing/string()
    where normalize-space($key) and normalize-space($value)
    return map:entry($key,$value)
)
return $thingMap The map entry can be constructed only when it has meaning. Which means the code has to allow for finding neither the key nor the value, which is why the ? (0 or 1) quantifier on the xs:string types. (The map-constructing FLOWR expression should also consider that there might be duplicate keys, but one thing at a time.) If you really want to know where in the data something is missing
let $simple as element(bucket) :=
  <bucket>
    <thingy name="oleander"/>
    <thingy name="primrose">likes well-drained partial shade</thingy>
    <thingy/>
    <thingy>zone 7</thingy>
    <thingy name="tulip">marsh plant, not a swamp plant</thingy>
  </bucket>

let $thingProblems as xs:string* :=
  for $thing in $simple/descendant::thingy
    let $key as xs:string? := $thing/@name/string()
    let $value as xs:string? := $thing/string()
    where not(normalize-space($key)) or not(normalize-space($value))
    return $thing ! concat(base-uri(.),path(.))
return $thingProblems will tell you which nodes in the input can't create map entries. This approach can be made more robust but this is the general idea.
-- Graydon Saunders | graydonish at gmail.com Þæs oferéode, ðisses swá mæg. -- Deor ("That passed, so may this.")
G
Graydonin reply to Markus Elfring, 27 May 2022, 15:3727 May 2022 20:29 #
Quoted text · Markus Elfring · 6 lines
On Fri, May 27, 2022 at 03:37:57PM +0200, Markus Elfring scripsit: > > Gaps? First, to avoid a misunderstanding: whereas map keys must not > > be the empty sequence, map values may. > > Special data sources might contain challenges for further > clarification of the desired referential integrity.
Well, sure! But if you want a useful answer you've got to ask a specific question with reference to actual code and actual XML. Otherwise the answer is always the entire language. Understanding the whole comes from understanding an accumulation of parts and understanding the individual parts comes from making specific mistakes. An unwillingness to make mistakes in public is a hindrance to learning.
-- Graydon Saunders | graydonish at gmail.com Þæs oferéode, ðisses swá mæg. -- Deor ("That passed, so may this.")
HR
Hans-Juergen Rennauin reply to Markus Elfring, 27 May 2022, 15:3728 May 2022 00:13 #
Well - but are such concerns related to using maps as the output? I do not think so, and therefore I suggest to avoid mixing specific how-to's with general challenges. By now you know the idiom how to construct maps dynamically, as you had already known how to construct node trees dynamically. I agree with Graydon - best to stick to specific questions. General questions are also welcome, but they should be phrased very carefully, in order to enable meaningful answers. A yardstick worth trying is to pose the question in the form "How can I ..."? or "How should one ..."? If you try, for example, to construct a question starting with "How can I" and then introducing referential integrity, you may find it difficult to come up with a question making real sense. Graydon wrote: "Understanding the whole comes from understanding an accumulation of parts and understanding the individual parts comes from making specific mistakes" Yes, but I would like to add something: PRINCIPLES. XQuery is a passionately principled language, and it is of prime importance to grasp the principles - a process that takes time, but one should be aware of it and welcome it. Most important is to understand the "everything-is-an-expression principle", which makes the language composable - expressions are defined in terms of operands which are expressions. Another key principle is the Effectvie Boolean Value, defined by a little mantra mapping value to boolean value. (With a couple of nasty pitfalls!) And, of course, the underlying data model - XDM - with its beautiful stream of consciousness "Information is a value. A value is a sequence of items. An item is a node, or an atom, or a mapping. A node has one of six kinds. An atom is a string plus a schema type. A mapping is a mapping of values to value, can be a map, or an array, or a function item." The firmer your understanding of the principles, the less your need to "try out" - you know. Kind regards,Hans-Jürgen Am Freitag, 27. Mai 2022, 15:38:12 MESZ hat Markus Elfring <markus.elfring at web.de> Folgendes geschrieben:
Quoted text · Markus Elfring · 1 line
> Gaps? First, to avoid a misunderstanding: whereas map keys must not be the empty sequence, map values may.
Special data sources might contain challenges for further clarification of the desired referential integrity. Regards, Markus

Written for BaseX 9.7. BaseX may have changed since: please check the current documentation.

« Lock avoidance using inspect:functions?Encoding hassle .... »
⚡Generated with XQuery