How to apply array:for-each on an array of arrays?
Hi,
In textmining, the 'idf' or inverse document frequency is defined as idf(term)=ln(ndocuments / ndocuments containing term). I am working on a function that should return this idf.
This function:
declare function local:wordFreq_idf($nodes as node()*) as array(*) {
let $count := count($nodes)
let $text := for $node in $nodes
return $node/text() => tokenize() => distinct-values()
let $idf := $text => tidyTM:wordCount_arr()
return $idf
};returns:
["probleem", 703]
["opgelost.", 248]
["dictu", 235]
["opgelost", 217]
["medewerker", 193]
...
For "probleem", the idf should be calculated as ln($count/703). Since
there are 1780 nodes this would result in 0.929011751.
I tried to exten the 'let $idf' line with:
=> array:for-each(function($idf) {array:append($idf,
math:log($count div $idf[2]) )})
which should result in ["probleem", 703, 0.929011751]but no mather what I do, every time I get this error: [XPTY0004] Cannot promote (array(xs:anyAtomicType))+ to array(*): ([ "probleem", 703 ], [ "opgelost.", 248 ], ...).
Is it possible to apply array:for-each on an array of arrays?
Ben
Hi Ben - I'm on mobile, please excuse any typos.
Maybe
return array { $idf }
is closer?
Untested, apologies! Best, Bridger
Quoted text · Ben Engbers · 42 lines
On Mon, Mar 30, 2020, 5:16 PM Ben Engbers <Ben.Engbers at be-logical.nl> wrote: > Hi, > > In textmining, the 'idf' or inverse document frequency is defined as > idf(term)=ln(ndocuments / ndocuments containing term). I am working on a > function that should return this idf. > > This function: > > declare function local:wordFreq_idf($nodes as node()*) as array(*) { > let $count := count($nodes) > let $text := for $node in $nodes > return $node/text() => tokenize() => distinct-values() > let $idf := $text => tidyTM:wordCount_arr() > return $idf > }; > > returns: > > ["probleem", 703] > ["opgelost.", 248] > ["dictu", 235] > ["opgelost", 217] > ["medewerker", 193] > ... > > For "probleem", the idf should be calculated as ln($count/703). Since > there are 1780 nodes this would result in 0.929011751. > I tried to exten the 'let $idf' line with: > => array:for-each(function($idf) {array:append($idf, > math:log($count div $idf[2]) )}) > which should result in ["probleem", 703, 0.929011751] > > but no mather what I do, every time I get this error: > [XPTY0004] Cannot promote (array(xs:anyAtomicType))+ to array(*): ([ > "probleem", 703 ], [ "opgelost.", 248 ], ...). > > Is it possible to apply array:for-each on an array of arrays? > > Ben > >
Quoted text · Ben Engbers · 1 line
On Mon, Mar 30, 2020 at 11:16:23PM +0200, Ben Engbers scripsit:
[snip]
Quoted text · Ben Engbers · 10 lines
> For "probleem", the idf should be calculated as ln($count/703). Since > there are 1780 nodes this would result in 0.929011751. > I tried to exten the 'let $idf' line with: > => array:for-each(function($idf) {array:append($idf, > math:log($count div $idf[2]) )}) > which should result in ["probleem", 703, 0.929011751] > > but no mather what I do, every time I get this error: > [XPTY0004] Cannot promote (array(xs:anyAtomicType))+ to array(*): ([ > "probleem", 703 ], [ "opgelost.", 248 ], ...).
The errors says you're trying to feed a sequence of arrays to an array function; maybe you want ! where you have => ?
-- Graydon
Quoted text · Graydon · 19 lines
Op 31-03-2020 om 01:18 schreef Graydon: > On Mon, Mar 30, 2020 at 11:16:23PM +0200, Ben Engbers scripsit: > [snip] >> For "probleem", the idf should be calculated as ln($count/703). Since >> there are 1780 nodes this would result in 0.929011751. >> I tried to exten the 'let $idf' line with: >> => array:for-each(function($idf) {array:append($idf, >> math:log($count div $idf[2]) )}) >> which should result in ["probleem", 703, 0.929011751] >> >> but no mather what I do, every time I get this error: >> [XPTY0004] Cannot promote (array(xs:anyAtomicType))+ to array(*): ([ >> "probleem", 703 ], [ "opgelost.", 248 ], ...). > > The errors says you're trying to feed a sequence of arrays to an array > function; maybe you want ! where you have => ? > > -- Graydon >
Hi,
Upon your remark about feeding a sequence of arrays, I first tried to apply 'for-each' instead of 'array:for-each'. Alas, that didn't help ;-(, the error was still the same.
I then tried to understand what you mean with the '!'.
In the book from Priscilla Walmsley, the ! is mentioned as a simple map operator. How is that related to this problem?
Cheers, Ben
Quoted text · Ben Engbers · 25 lines
Am 30.03.2020 um 23:16 schrieb Ben Engbers: > Hi, > > In textmining, the 'idf' or inverse document frequency is defined as > idf(term)=ln(ndocuments / ndocuments containing term). I am working on a > function that should return this idf. > > This function: > > declare function local:wordFreq_idf($nodes as node()*) as array(*) { > let $count := count($nodes) > let $text := for $node in $nodes > return $node/text() => tokenize() => distinct-values() > let $idf := $text => tidyTM:wordCount_arr() > return $idf > }; > > returns: > > ["probleem", 703] > ["opgelost.", 248] > ["dictu", 235] > ["opgelost", 217] > ["medewerker", 193] > ...
So does the working function return a sequence of arrays? That doesn't match the as array(*) return type declaration, it seems.
What does tidyTM:wordCount_arr() return, a single array (of atomic items)?
Hi,
Quoted text · Graydon · 2 lines
> => means "take the thing on the left and substitute it for the first > parameter of the function on the right, so
I thought it meant "The first parameter on the right will be subsituted with the thing on the left"?
Quoted text · Graydon · 8 lines
> ('weasels') => replace('weasels','mustelids') works > > ('weasels','badgers') => replace('weasels','mustelids') DOES NOT work > > This is because a one-item sequence can be treated as the single string > value the first parameter of replace() requires, but a > greater-then-one-item sequence can't be. (This one gives you "item > expected, sequence found" if you try it from the GUI.)
The following is quite similar to the 'piping' mechanism in R. I'll start experimenting with it.
Thanx, Ben
Quoted text · Graydon · 21 lines
> ! means "take each item of the sequence on the left and pass it to the > thing on the right in turn", so > > ('weasels','badgers') ! replace(.,'weasels','mustelids') works. > > (note that replace() got its first parameter back as the context item > dot.) > > so if you take > > => array:for-each(function($idf) {array:append($idf,math:log($count div $idf[2]) )}) > > and replace it with > ! array:for-each(.,function($idf) {array:append($idf,math:log($count div $idf[2]) )}) > > (note the context-item dot!) > > you should at least get a different error message. > > -- Graydon >
Quoted text · Ben Engbers · 21 lines
On Tue, Mar 31, 2020 at 04:21:52PM +0200, Ben Engbers scripsit: > Op 31-03-2020 om 01:18 schreef Graydon: > > On Mon, Mar 30, 2020 at 11:16:23PM +0200, Ben Engbers scripsit: > > [snip] > >> For "probleem", the idf should be calculated as ln($count/703). Since > >> there are 1780 nodes this would result in 0.929011751. > >> I tried to exten the 'let $idf' line with: > >> => array:for-each(function($idf) {array:append($idf, > >> math:log($count div $idf[2]) )}) > >> which should result in ["probleem", 703, 0.929011751] > >> > >> but no mather what I do, every time I get this error: > >> [XPTY0004] Cannot promote (array(xs:anyAtomicType))+ to array(*): ([ > >> "probleem", 703 ], [ "opgelost.", 248 ], ...). > > > > The errors says you're trying to feed a sequence of arrays to an array > > function; maybe you want ! where you have => ? > > Upon your remark about feeding a sequence of arrays, I first tried to > apply 'for-each' instead of 'array:for-each'. Alas, that didn't help > ;-(, the error was still the same.
array:for-each takes a single array and gives you back a new array based on what the anonymous function passed as the second parameter does to each member of the original array.
So you have to make sure you're feeding a single array to it. (and you're not; that's what the error message is telling you, you've got a sequence of arrays on the left of the => operator.)
Quoted text · Ben Engbers · 3 lines
> I then tried to understand what you mean with the '!'. > In the book from Priscilla Walmsley, the ! is mentioned as a simple map > operator. How is that related to this problem?
=> means "take the thing on the left and substitute it for the first parameter of the function on the right, so
('weasels') => replace('weasels','mustelids') works
('weasels','badgers') => replace('weasels','mustelids') DOES NOT work
This is because a one-item sequence can be treated as the single string value the first parameter of replace() requires, but a greater-then-one-item sequence can't be. (This one gives you "item expected, sequence found" if you try it from the GUI.)
! means "take each item of the sequence on the left and pass it to the thing on the right in turn", so
('weasels','badgers') ! replace(.,'weasels','mustelids') works.
(note that replace() got its first parameter back as the context item dot.)
so if you take
=> array:for-each(function($idf) {array:append($idf,math:log($count div $idf[2]) )})
and replace it with
! array:for-each(.,function($idf) {array:append($idf,math:log($count div $idf[2]) )})
(note the context-item dot!)
you should at least get a different error message.
-- Graydon
Hi,
For (my personal) clarity, I have split up the original function in two parts:
declare function local:step_one($nodes as node()*) as array(*)*
{
let $text := for $node in $nodes
return $node/text() =>
tokenize() => distinct-values()
let $idf := $text =>
tidyTM:wordCount_arr()
return $idf
};
In local:step_one(), I first create a sequence with the distinct tokens
for each $node. All the sequences are joined in $text.
I then call wordCount_arr to count the occurences of each word in $text:
declare function tidyTM:wordCount_arr(
$Words as xs:string*)
as array(*) {
for $w in $Words
let $f := $w
group by $f
order by count($w) descending
return ([$f, count($w)])
} ;I would say that tidyTM:wordCount_arr returns a sequence of arrays but I am not certain if I have specified the correct return-type?
Calling local:step_one(tidyTM:remove_Stopwords($nodes, "Stp", $Stoppers))
returns:
["probleem", 703]
["opgelost.", 248]
....I had hoped that calling the following local:wordFreq, would add the idf to each element but instead I get an error
declare function local:wordFreq_idf($nodes as node()*) as array(*)
{
let $count := count($nodes)
let $idf := local:step_one($nodes)
let $result := for-each( $idf,
function($z) {array:append ($z, math:log($count div $z(2) ) ) } )
return $result
};
[XPTY0004] Cannot promote (array(xs:anyAtomicType))+ to array(*): $idf
:= ([ "probleem", 703 ], [ "opgelost.", 248 ], ...).Cheers, Ben
Quoted text · Martin Honnen · 7 lines
Op 31-03-2020 om 16:29 schreef Martin Honnen: > So does the working function return a sequence of arrays? That doesn't > match the > as array(*) > return type declaration, it seems. > > What does tidyTM:wordCount_arr() return, a single array (of atomic items)?
Quoted text · Ben Engbers · 32 lines
On 31.03.2020 18:32, Ben Engbers wrote: > Hi, > > For (my personal) clarity, I have split up the original function in two > parts: > > declare function local:step_one($nodes as node()*) as array(*)* > { > let $text := for $node in $nodes > return $node/text() => > tokenize() => distinct-values() > let $idf := $text => > tidyTM:wordCount_arr() > return $idf > }; > > In local:step_one(), I first create a sequence with the distinct tokens > for each $node. All the sequences are joined in $text. > I then call wordCount_arr to count the occurences of each word in $text: > > declare function tidyTM:wordCount_arr( > $Words as xs:string*) > as array(*) { > for $w in $Words > let $f := $w > group by $f > order by count($w) descending > return ([$f, count($w)]) > } ; > > I would say that tidyTM:wordCount_arr returns a sequence of arrays but I > am not certain if I have specified the correct return-type?
Reading the code I agree that the return type seems to be a sequence of arrays but therefore I wonder why you don't get a similar error as later on with declaring array(*) and not array(*)*
Quoted text · Ben Engbers · 19 lines
> Calling local:step_one(tidyTM:remove_Stopwords($nodes, "Stp", $Stoppers)) > returns: > ["probleem", 703] > ["opgelost.", 248] > .... > > I had hoped that calling the following local:wordFreq, would add the > idf to each element but instead I get an error > > declare function local:wordFreq_idf($nodes as node()*) as array(*) > { > let $count := count($nodes) > let $idf := local:step_one($nodes) > let $result := for-each( $idf, > function($z) {array:append ($z, math:log($count div $z(2) ) ) } ) > return $result > }; > [XPTY0004] Cannot promote (array(xs:anyAtomicType))+ to array(*): $idf > := ([ "probleem", 703 ], [ "opgelost.", 248 ], ...).
The message tries to tell you that the declared return type array(*) is a single array while the function returns a (non-empty) sequence of arrays so using declare function local:wordFreq_idf($nodes as node()*) as array(*)* would remove that error.
To insert the third value into each array I think you want
let $result := $idf ! array:append(., math:log($count div .(2) ))Hi,
Quoted text · Martin Honnen · 3 lines
> To insert the third value into each array I think you want > > let $result := $idf ! array:append(., math:log($count div .(2) ))
This works!
Martin and Graydon, thanks for the help and the explanation.
Ben
import module namespace tidyTM = 'http://www.be-logical.nl';
declare function local:step_one($nodes as node()*) as array(*)*
{
let $text := for $node in $nodes
return $node/text() =>
tokenize() => distinct-values()
let $idf := $text =>
tidyTM:wordCount_arr()
return $idf
};
declare function local:wordFreq_idf($nodes as node()*) as array(*)
{
let $count := count($nodes)
let $idf := local:step_one($nodes)
let $result := $idf ! array:append(., math:log($count div .(2) ))
return $result
};
let $nodes :=
collection('IncidentRemarks/Incidenten-180101-190630.csv')/csv/record/INC_RM
let $Stoppers := doc('TextMining/Stopwoorden.txt')/text/line/text()
return local:wordFreq_idf(
tidyTM:remove_Stopwords($nodes, "Stp", $Stoppers))--------------
declare function tidyTM:wordCount_arr(
$Words as xs:string*)
as array(*)* {
for $w in $Words
let $f := $w
group by $f
order by count($w) descending
return ([$f, count($w)])
} ;-----------
["probleem", 703, 9.362885817944681e-1]
["opgelost.", 248, 1.9782167274401508e0]
...Written for BaseX 9.3. BaseX may have changed since: please check the current documentation.