Should it be possible to declare a function in the client?

> Session$Execute(fn_get_words_txt) returns:

> fn_get_words_Query <- Session$Query(fn_get_words_txt)

Op 27-02-2020 om 16:41 schreef Christian Grün: > Hi Ben, > > …create a query object, and attach the actual function call to your > query string.

> If you want to make XQuery code persistent for future invocations, you > can include your function in an XQuery library module and install this > module in the repository [1].

> print(proc.time() - ptm)

> print(proc.time() - ptm)

> > While experimenting (I try to speed up the querys), I compared the > results from these 2 querys: >

> Word_Inc_Rm_Stop_txt <- "import module namespace functx = > 'http://www.functx.com'; > let $txt := > collection('IncidentRemarks/Incidents')/csv/record/INC_RM/text() > let $INC_RM := tokenize(lower-case(string-join($txt)), > '(\\s|[,.!:;]|[n][b][s][p][;])+') > let $Stoppers := doc('TextMining/Stopwoorden.txt')/text/line/text() > let $Stop := functx:value-except($INC_RM, $Stoppers) > return $Stop" > Word_Inc_Rm_Stop <- Session$Execute(as.character(glue("xquery > {Word_Inc_Rm_Stop_txt}")))$result[[1]] > Word_Inc_Rm_Stop_Count <- length(Word_Inc_Rm_Stop) > > Word_Inc_Rm_Stop_txt_2 <- "import module namespace functx = > 'http://www.functx.com'; > let $txt := > collection('IncidentRemarks/Incidents')/csv/record/INC_RM/text() > let $INC_RM := tokenize(lower-case(string-join($txt)), > '(\\s|[,.!:;]|[n][b][s][p][;])+') > let $Stoppers := doc('TextMining/Stopwoorden.txt')/text/line/text() > let $Stop := functx:value-except($INC_RM, $Stoppers) > return count($Stop)" > Word_Inc_Rm_Stop_Count_2 <- Session$Execute(as.character(glue("xquery > {Word_Inc_Rm_Stop_txt_2}")))$result[[1]] > > These are the processing-times: > > Version 1: > > print(proc.time() - ptm) > user system elapsed > 2.903 0.022 3.160 > Version 2: > > print(proc.time() - ptm) > user system elapsed > 0.041 0.004 1.089 > > I guess it makes sense to put effort in speeding up my code. But what > bothers me is the following. > > The first query computes the length from the vector that is returned, > The result is 5842. > The second query returns the length as computed by basex. This result is > 5843. The GUI also returns 5843 as result. > > I copied the output from > .. > return $Stop > > to a new LibreOffice-document. That document counts 5842 words. > > Who is right? > > Cheers, > Ben > > > > >

Op 27-02-2020 om 19:19 schreef Christian Grün: > It’s difficult to understand what’s going on here. Could you please > provide us self-contained queries without the R wrapper code?

> On Feb 27, 2020, at 2:48 PM, Ben Engbers <Ben.Engbers at Be-Logical.nl> wrote: > > Op 27-02-2020 om 19:19 schreef Christian Grün: >> It’s difficult to understand what’s going on here. Could you please >> provide us self-contained queries without the R wrapper code? > > Version 1: > > import module namespace functx = 'http://www.functx.com'; > (: Extract the text :) > let $txt := collection('IncidentRemarks/Incidents')/csv/record/INC_RM/text() > (: Convert to lower-case and tokenize :) > let $INC_RM := tokenize(lower-case(string-join($txt)), > '(\\s|[,.!:;]|[n][b][s][p][;])+') > (: Read Stopwords :) > let $Stoppers := doc('TextMining/Stopwoorden.txt')/text/line/text() > (: Remove Stopwords :) > let $Stop := functx:value-except($INC_RM, $Stoppers) > return $Stop" > > My R-code first executes this as XQUERY and then calculates the length > of the returned list (=5842). > > Version 2: > > import module namespace functx = 'http://www.functx.com'; > let $txt := collection('IncidentRemarks/Incidents')/csv/record/INC_RM/text() > let $INC_RM := tokenize(lower-case(string-join($txt)), > '(\\s|[,.!:;]|[n][b][s][p][;])+') > let $Stoppers := doc('TextMining/Stopwoorden.txt')/text/line/text() > let $Stop := functx:value-except($INC_RM, $Stoppers) > return count($Stop) > > Returns the length of the sequence (counts 5843 words). > > The '\\' in the regular expression is intentional (R-specific). With a > single '\' the query can be executed in BaseXGUI. > > Does this help? > > Ben >

> On Feb 27, 2020, at 3:43 PM, Majewski, Steven Dennis (sdm7g) <sdm7g at virginia.edu> wrote: > > So, if the counts are different depending on who is counting ( R or BaseX ), > The first question is : who is correct ? > ( And the 2nd question is probably: what do you mean by correct ? as the semantics of XQuery sequences and whatever destination R datatype is being counted may be slightly different. I don’t know R that well, but semantics of XQuery sequences and arrays are rather different, for example. ) > > — Steve M. > > >> On Feb 27, 2020, at 2:48 PM, Ben Engbers <Ben.Engbers at Be-Logical.nl> wrote: >> >> Op 27-02-2020 om 19:19 schreef Christian Grün: >>> It’s difficult to understand what’s going on here. Could you please >>> provide us self-contained queries without the R wrapper code? >> >> Version 1: >> >> import module namespace functx = 'http://www.functx.com'; >> (: Extract the text :) >> let $txt := collection('IncidentRemarks/Incidents')/csv/record/INC_RM/text() >> (: Convert to lower-case and tokenize :) >> let $INC_RM := tokenize(lower-case(string-join($txt)), >> '(\\s|[,.!:;]|[n][b][s][p][;])+') >> (: Read Stopwords :) >> let $Stoppers := doc('TextMining/Stopwoorden.txt')/text/line/text() >> (: Remove Stopwords :) >> let $Stop := functx:value-except($INC_RM, $Stoppers) >> return $Stop" >> >> My R-code first executes this as XQUERY and then calculates the length >> of the returned list (=5842). >> >> Version 2: >> >> import module namespace functx = 'http://www.functx.com'; >> let $txt := collection('IncidentRemarks/Incidents')/csv/record/INC_RM/text() >> let $INC_RM := tokenize(lower-case(string-join($txt)), >> '(\\s|[,.!:;]|[n][b][s][p][;])+') >> let $Stoppers := doc('TextMining/Stopwoorden.txt')/text/line/text() >> let $Stop := functx:value-except($INC_RM, $Stoppers) >> return count($Stop) >> >> Returns the length of the sequence (counts 5843 words). >> >> The '\\' in the regular expression is intentional (R-specific). With a >> single '\' the query can be executed in BaseXGUI. >> >> Does this help? >> >> Ben >> >

Op 27-02-2020 om 22:03 schreef Majewski, Steven Dennis (sdm7g): > Also: is ‘(&amp;nbsp;)’ what you want as part of you regex to also catch the ampersand ? > I’m just guessing your intent here. > You could also try ‘(\W|&amp;nbsp;)+’ - i.e. non-word, but I’m kind of assuming that it handles non-normalized unicode accented characters correctly and reads them as word chars and not delimiters. That would be, of course, the right thing, but I’ld probably test it first. > > — Steve.

On Fri, Feb 28, 2020 at 11:45 AM Ben Engbers <Ben.Engbers at be-logical.nl> wrote: > > Op 27-02-2020 om 22:03 schreef Majewski, Steven Dennis (sdm7g): > > Also: is ‘(&amp;nbsp;)’ what you want as part of you regex to also catch the ampersand ? > > I’m just guessing your intent here. > > You could also try ‘(\W|&amp;nbsp;)+’ - i.e. non-word, but I’m kind of assuming that it handles non-normalized unicode accented characters correctly and reads them as word chars and not delimiters. That would be, of course, the right thing, but I’ld probably test it first. > > > > — Steve. > > I just copied the regex-expression from this page > "https://en.wikibooks.org/wiki/XQuery/Tag_Cloud" (using regex always > gives me headaches ;-( ). But even after removing the "|[n][b][s][p][;]" > from the regex, basexgui still returns 5843. > > Ben

> On Feb 28, 2020, at 5:45 AM, Ben Engbers <Ben.Engbers at be-logical.nl> wrote: > > Op 27-02-2020 om 22:03 schreef Majewski, Steven Dennis (sdm7g): >> Also: is ‘(&amp;nbsp;)’ what you want as part of you regex to also catch the ampersand ? >> I’m just guessing your intent here. >> You could also try ‘(\W|&amp;nbsp;)+’ - i.e. non-word, but I’m kind of assuming that it handles non-normalized unicode accented characters correctly and reads them as word chars and not delimiters. That would be, of course, the right thing, but I’ld probably test it first. >> >> — Steve. > > I just copied the regex-expression from this page > "https://en.wikibooks.org/wiki/XQuery/Tag_Cloud" (using regex always > gives me headaches ;-( ). But even after removing the "|[n][b][s][p][;]" > from the regex, basexgui still returns 5843. > > Ben > > >

Op 28-02-2020 om 14:39 schreef Christian Grün: > I was wondering about nbsp as well. Maybe you don’t need it at all, > but we’d need to have a look at your files. > > Could you additionally provide us with minimized instances of your > Incidents and Stopwoorden.txt XML documents? They should have the same > structure, but contain only a few lines of contents.

> Op 28-02-2020 om 14:39 schreef Christian Grün: > > I was wondering about nbsp as well. Maybe you don’t need it at all, > > but we’d need to have a look at your files. > > > > Could you additionally provide us with minimized instances of your > > Incidents and Stopwoorden.txt XML documents? They should have the same > > structure, but contain only a few lines of contents. > > It should be relatively easy to create a database with the > (approximately 500) stopwords and another database with with the Incidents. > Shall I send you a backup of those two databases? > > Ben > >

On Fri, Feb 28, 2020 at 4:57 PM Christian Grün <christian.gruen at gmail.com> wrote: > > A few incident and stories entries should be sufficient. Just attach the two XML documents to your next reply. > > > > Ben Engbers <Ben.Engbers at be-logical.nl> schrieb am Fr., 28. Feb. 2020, 16:20: >> >> Op 28-02-2020 om 14:39 schreef Christian Grün: >> > I was wondering about nbsp as well. Maybe you don’t need it at all, >> > but we’d need to have a look at your files. >> > >> > Could you additionally provide us with minimized instances of your >> > Incidents and Stopwoorden.txt XML documents? They should have the same >> > structure, but contain only a few lines of contents. >> >> It should be relatively easy to create a database with the >> (approximately 500) stopwords and another database with with the Incidents. >> Shall I send you a backup of those two databases? >> >> Ben >>

Op 02-03-2020 om 13:27 schreef Christian Grün: > Hi Ben, > > Here is an alternative version that, as I believe, should match your > requirements better: > > let $words := distinct-values( > for $text in db:open('Incidents')/csv/record/INC_RM > return ft:tokenize($text) > ) > let $stopwords := db:open('Stopwords')/text/line > let $result := $words[not(. = $stopwords)] > return sort($result) > > There is no need to remove nbsp substrings as they’ll never occur in > your input, and the ft:tokenize function will ensure that your input > (case, special characters, diacritics) will be normalized (see [1,2] > for more details). Using functx is perfectly valid; I only removed the > reference to make the code a bit shorter. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/Full-Text_Module#ft:tokenize > [2] http://docs.basex.org/wiki/Full-Text

Op 02-03-2020 om 13:27 schreef Christian Grün: > Hi Ben, > > Here is an alternative version that, as I believe, should match your > requirements better: > > let $words := distinct-values( > for $text in db:open('Incidents')/csv/record/INC_RM > return ft:tokenize($text) > ) > let $stopwords := db:open('Stopwords')/text/line > let $result := $words[not(. = $stopwords)] > return sort($result) >

> What I would like to see in BaseX, is that similar as in xquery, > 'Stopwords' could be used as if it were a separate resource in the > 'Incidents'-database and that it could be used as follows in the query:

> As far as I understand, 'stemming' has alrady been made available in the > ft:module. > Would it also be possible to use STOPWORDS in a similar way?