rest vs. restxq - strange difference

On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> wrote: > > I am running a web script in two identical versions (identical as in "cut > and paste"), one via RESTXQ and one vi REST. The response is different, and > I wondered what may be the trouble. > > For example the output (the URLs only works locally) for > http://ljohnsen:8984/hyphens/mellom > is the same as > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom > > which is a set of hyphenation data: > mellom > mel - lom 17005 > Mel - lom 144 > mel - lom. 50 > > but if "mellom" is exchanged with "nasjonalbiblioteket" only the REST > version shows any result, which then is the same as I get experimenting in > the GUI. > > The actual script is added below, and which runs in both versions > (identical apart form the rest and restxq interfaces), it uses full text > search, but results differ when run under the REST-regime. > > All the best > Lars G Johnsen > National Library of Norway > > module namespace page = 'http://basex.org/modules/web-page'; > > declare > %rest:path("/hyphens/{$word}") > %output:method("html") > > function page:show-hyphens($word) { > let $db := db:open('hyphen-data') > let $hyphens := for $hyp in $db/hyphens/hyphens[full contains text > {$word}] > group by $first := $hyp/first, $second := $hyp/second > let $count := count($hyp) > order by xs:int($count) descending > return element p { > attribute freq {$count}, > $first, " - ", $second, $count > } > > let $total := sum($hyphens//@freq) > let $div := element div { > element p {$word}, > for $hyp in $hyphens > return element div { > attribute class {"hyph"}, > attribute style {"font-size:", 1 +round(xs:int($hyp//@freq/data()) > div $total,1) || "em"}, > $hyp > > } > } > return > <html encoding="UTF-8"> > <head> > <meta http-equiv="Content-Type" content="text/html" charset="UTF-8" > /> > <title>Orddelinger</title> > </head> > <body>{$div} > </body> > </html> > > };

> Hi Lars, > > hm, that's difficult to tell. All I can say is that this sounds > unusual, so I'm coming up with my standard questions: Do you think you > could build us a little example that allows us to reproduce the > problem? Have you tried the latest version of BaseX? > > Best, > Christian > > > On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> wrote: > > > > I am running a web script in two identical versions (identical as in "cut > > and paste"), one via RESTXQ and one vi REST. The response is different, > and > > I wondered what may be the trouble. > > > > For example the output (the URLs only works locally) for > > http://ljohnsen:8984/hyphens/mellom > > is the same as > > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom > > > > which is a set of hyphenation data: > > mellom > > mel - lom 17005 > > Mel - lom 144 > > mel - lom. 50 > > > > but if "mellom" is exchanged with "nasjonalbiblioteket" only the REST > > version shows any result, which then is the same as I get experimenting > in > > the GUI. > > > > The actual script is added below, and which runs in both versions > > (identical apart form the rest and restxq interfaces), it uses full text > > search, but results differ when run under the REST-regime. > > > > All the best > > Lars G Johnsen > > National Library of Norway > > > > module namespace page = 'http://basex.org/modules/web-page'; > > > > declare > > %rest:path("/hyphens/{$word}") > > %output:method("html") > > > > function page:show-hyphens($word) { > > let $db := db:open('hyphen-data') > > let $hyphens := for $hyp in $db/hyphens/hyphens[full contains text > > {$word}] > > group by $first := $hyp/first, $second := $hyp/second > > let $count := count($hyp) > > order by xs:int($count) descending > > return element p { > > attribute freq {$count}, > > $first, " - ", $second, $count > > } > > > > let $total := sum($hyphens//@freq) > > let $div := element div { > > element p {$word}, > > for $hyp in $hyphens > > return element div { > > attribute class {"hyph"}, > > attribute style {"font-size:", 1 > +round(xs:int($hyp//@freq/data()) > > div $total,1) || "em"}, > > $hyp > > > > } > > } > > return > > <html encoding="UTF-8"> > > <head> > > <meta http-equiv="Content-Type" content="text/html" > charset="UTF-8" > > /> > > <title>Orddelinger</title> > > </head> > > <body>{$div} > > </body> > > </html> > > > > }; >

> Hi Christian - and thanks for fast response. Latest version 8.11 is in use > (same behaviour as previous). Let me see if I can make a self contained > example. > > best, > Lars > > 2015-05-18 13:40 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: > >> Hi Lars, >> >> hm, that's difficult to tell. All I can say is that this sounds >> unusual, so I'm coming up with my standard questions: Do you think you >> could build us a little example that allows us to reproduce the >> problem? Have you tried the latest version of BaseX? >> >> Best, >> Christian >> >> >> On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> wrote: >> > >> > I am running a web script in two identical versions (identical as in >> "cut >> > and paste"), one via RESTXQ and one vi REST. The response is different, >> and >> > I wondered what may be the trouble. >> > >> > For example the output (the URLs only works locally) for >> > http://ljohnsen:8984/hyphens/mellom >> > is the same as >> > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom >> > >> > which is a set of hyphenation data: >> > mellom >> > mel - lom 17005 >> > Mel - lom 144 >> > mel - lom. 50 >> > >> > but if "mellom" is exchanged with "nasjonalbiblioteket" only the REST >> > version shows any result, which then is the same as I get experimenting >> in >> > the GUI. >> > >> > The actual script is added below, and which runs in both versions >> > (identical apart form the rest and restxq interfaces), it uses full text >> > search, but results differ when run under the REST-regime. >> > >> > All the best >> > Lars G Johnsen >> > National Library of Norway >> > >> > module namespace page = 'http://basex.org/modules/web-page'; >> > >> > declare >> > %rest:path("/hyphens/{$word}") >> > %output:method("html") >> > >> > function page:show-hyphens($word) { >> > let $db := db:open('hyphen-data') >> > let $hyphens := for $hyp in $db/hyphens/hyphens[full contains text >> > {$word}] >> > group by $first := $hyp/first, $second := $hyp/second >> > let $count := count($hyp) >> > order by xs:int($count) descending >> > return element p { >> > attribute freq {$count}, >> > $first, " - ", $second, $count >> > } >> > >> > let $total := sum($hyphens//@freq) >> > let $div := element div { >> > element p {$word}, >> > for $hyp in $hyphens >> > return element div { >> > attribute class {"hyph"}, >> > attribute style {"font-size:", 1 >> +round(xs:int($hyp//@freq/data()) >> > div $total,1) || "em"}, >> > $hyp >> > >> > } >> > } >> > return >> > <html encoding="UTF-8"> >> > <head> >> > <meta http-equiv="Content-Type" content="text/html" >> charset="UTF-8" >> > /> >> > <title>Orddelinger</title> >> > </head> >> > <body>{$div} >> > </body> >> > </html> >> > >> > }; >> > >

> REST output (2 first lines): > føre > fø - re 219 > > RESTXQ > føre > fo - re 123 > > The first word quoted is "føre" in both cases and is what the scripts see, > so the full text is given the same in both cases. Could it be that within > RESTXQ the full text index is treated differently? > > I will work closer on a self contained example, but thought this might > point to something. > > Cheers > Lars > > > 2015-05-18 13:44 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: >> >> Hi Christian - and thanks for fast response. Latest version 8.11 is in use >> (same behaviour as previous). Let me see if I can make a self contained >> example. >> >> best, >> Lars >> >> 2015-05-18 13:40 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: >>> >>> Hi Lars, >>> >>> hm, that's difficult to tell. All I can say is that this sounds >>> unusual, so I'm coming up with my standard questions: Do you think you >>> could build us a little example that allows us to reproduce the >>> problem? Have you tried the latest version of BaseX? >>> >>> Best, >>> Christian >>> >>> >>> On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> wrote: >>> > >>> > I am running a web script in two identical versions (identical as in >>> > "cut >>> > and paste"), one via RESTXQ and one vi REST. The response is different, >>> > and >>> > I wondered what may be the trouble. >>> > >>> > For example the output (the URLs only works locally) for >>> > http://ljohnsen:8984/hyphens/mellom >>> > is the same as >>> > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom >>> > >>> > which is a set of hyphenation data: >>> > mellom >>> > mel - lom 17005 >>> > Mel - lom 144 >>> > mel - lom. 50 >>> > >>> > but if "mellom" is exchanged with "nasjonalbiblioteket" only the REST >>> > version shows any result, which then is the same as I get experimenting >>> > in >>> > the GUI. >>> > >>> > The actual script is added below, and which runs in both versions >>> > (identical apart form the rest and restxq interfaces), it uses full >>> > text >>> > search, but results differ when run under the REST-regime. >>> > >>> > All the best >>> > Lars G Johnsen >>> > National Library of Norway >>> > >>> > module namespace page = 'http://basex.org/modules/web-page'; >>> > >>> > declare >>> > %rest:path("/hyphens/{$word}") >>> > %output:method("html") >>> > >>> > function page:show-hyphens($word) { >>> > let $db := db:open('hyphen-data') >>> > let $hyphens := for $hyp in $db/hyphens/hyphens[full contains >>> > text >>> > {$word}] >>> > group by $first := $hyp/first, $second := $hyp/second >>> > let $count := count($hyp) >>> > order by xs:int($count) descending >>> > return element p { >>> > attribute freq {$count}, >>> > $first, " - ", $second, $count >>> > } >>> > >>> > let $total := sum($hyphens//@freq) >>> > let $div := element div { >>> > element p {$word}, >>> > for $hyp in $hyphens >>> > return element div { >>> > attribute class {"hyph"}, >>> > attribute style {"font-size:", 1 >>> > +round(xs:int($hyp//@freq/data()) >>> > div $total,1) || "em"}, >>> > $hyp >>> > >>> > } >>> > } >>> > return >>> > <html encoding="UTF-8"> >>> > <head> >>> > <meta http-equiv="Content-Type" content="text/html" >>> > charset="UTF-8" >>> > /> >>> > <title>Orddelinger</title> >>> > </head> >>> > <body>{$div} >>> > </body> >>> > </html> >>> > >>> > }; >> >> >

> It could be that your URL is decoded in a wrong way.. What happens if > you run the following function with REST and RESTXQ and "føre" as > word? > > declare > %rest:path("/test/encoding/{$word}") > function page:test-encoding($word) { > string-to-codepoints($word) > }; > > Thanks, > Christian > > > string-to-codepoints() > > REST output (2 first lines): > > føre > > fø - re 219 > > > > RESTXQ > > føre > > fo - re 123 > > > > The first word quoted is "føre" in both cases and is what the scripts > see, > > so the full text is given the same in both cases. Could it be that within > > RESTXQ the full text index is treated differently? > > > > I will work closer on a self contained example, but thought this might > > point to something. > > > > Cheers > > Lars > > > > > > 2015-05-18 13:44 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: > >> > >> Hi Christian - and thanks for fast response. Latest version 8.11 is in > use > >> (same behaviour as previous). Let me see if I can make a self contained > >> example. > >> > >> best, > >> Lars > >> > >> 2015-05-18 13:40 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: > >>> > >>> Hi Lars, > >>> > >>> hm, that's difficult to tell. All I can say is that this sounds > >>> unusual, so I'm coming up with my standard questions: Do you think you > >>> could build us a little example that allows us to reproduce the > >>> problem? Have you tried the latest version of BaseX? > >>> > >>> Best, > >>> Christian > >>> > >>> > >>> On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> > wrote: > >>> > > >>> > I am running a web script in two identical versions (identical as in > >>> > "cut > >>> > and paste"), one via RESTXQ and one vi REST. The response is > different, > >>> > and > >>> > I wondered what may be the trouble. > >>> > > >>> > For example the output (the URLs only works locally) for > >>> > http://ljohnsen:8984/hyphens/mellom > >>> > is the same as > >>> > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom > >>> > > >>> > which is a set of hyphenation data: > >>> > mellom > >>> > mel - lom 17005 > >>> > Mel - lom 144 > >>> > mel - lom. 50 > >>> > > >>> > but if "mellom" is exchanged with "nasjonalbiblioteket" only the > REST > >>> > version shows any result, which then is the same as I get > experimenting > >>> > in > >>> > the GUI. > >>> > > >>> > The actual script is added below, and which runs in both versions > >>> > (identical apart form the rest and restxq interfaces), it uses full > >>> > text > >>> > search, but results differ when run under the REST-regime. > >>> > > >>> > All the best > >>> > Lars G Johnsen > >>> > National Library of Norway > >>> > > >>> > module namespace page = 'http://basex.org/modules/web-page'; > >>> > > >>> > declare > >>> > %rest:path("/hyphens/{$word}") > >>> > %output:method("html") > >>> > > >>> > function page:show-hyphens($word) { > >>> > let $db := db:open('hyphen-data') > >>> > let $hyphens := for $hyp in $db/hyphens/hyphens[full contains > >>> > text > >>> > {$word}] > >>> > group by $first := $hyp/first, $second := $hyp/second > >>> > let $count := count($hyp) > >>> > order by xs:int($count) descending > >>> > return element p { > >>> > attribute freq {$count}, > >>> > $first, " - ", $second, $count > >>> > } > >>> > > >>> > let $total := sum($hyphens//@freq) > >>> > let $div := element div { > >>> > element p {$word}, > >>> > for $hyp in $hyphens > >>> > return element div { > >>> > attribute class {"hyph"}, > >>> > attribute style {"font-size:", 1 > >>> > +round(xs:int($hyp//@freq/data()) > >>> > div $total,1) || "em"}, > >>> > $hyp > >>> > > >>> > } > >>> > } > >>> > return > >>> > <html encoding="UTF-8"> > >>> > <head> > >>> > <meta http-equiv="Content-Type" content="text/html" > >>> > charset="UTF-8" > >>> > /> > >>> > <title>Orddelinger</title> > >>> > </head> > >>> > <body>{$div} > >>> > </body> > >>> > </html> > >>> > > >>> > }; > >> > >> > > >

> However, when using text index instead of full text the results are the same > for both, except that RESTXQ takes almost forever

> 2015-05-18 14:28 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: >> >> It could be that your URL is decoded in a wrong way.. What happens if >> you run the following function with REST and RESTXQ and "føre" as >> word? >> >> declare >> %rest:path("/test/encoding/{$word}") >> function page:test-encoding($word) { >> string-to-codepoints($word) >> }; >> >> Thanks, >> Christian >> >> >> string-to-codepoints() >> > REST output (2 first lines): >> > føre >> > fø - re 219 >> > >> > RESTXQ >> > føre >> > fo - re 123 >> > >> > The first word quoted is "føre" in both cases and is what the scripts >> > see, >> > so the full text is given the same in both cases. Could it be that >> > within >> > RESTXQ the full text index is treated differently? >> > >> > I will work closer on a self contained example, but thought this might >> > point to something. >> > >> > Cheers >> > Lars >> > >> > >> > 2015-05-18 13:44 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: >> >> >> >> Hi Christian - and thanks for fast response. Latest version 8.11 is in >> >> use >> >> (same behaviour as previous). Let me see if I can make a self contained >> >> example. >> >> >> >> best, >> >> Lars >> >> >> >> 2015-05-18 13:40 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: >> >>> >> >>> Hi Lars, >> >>> >> >>> hm, that's difficult to tell. All I can say is that this sounds >> >>> unusual, so I'm coming up with my standard questions: Do you think you >> >>> could build us a little example that allows us to reproduce the >> >>> problem? Have you tried the latest version of BaseX? >> >>> >> >>> Best, >> >>> Christian >> >>> >> >>> >> >>> On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> >> >>> wrote: >> >>> > >> >>> > I am running a web script in two identical versions (identical as in >> >>> > "cut >> >>> > and paste"), one via RESTXQ and one vi REST. The response is >> >>> > different, >> >>> > and >> >>> > I wondered what may be the trouble. >> >>> > >> >>> > For example the output (the URLs only works locally) for >> >>> > http://ljohnsen:8984/hyphens/mellom >> >>> > is the same as >> >>> > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom >> >>> > >> >>> > which is a set of hyphenation data: >> >>> > mellom >> >>> > mel - lom 17005 >> >>> > Mel - lom 144 >> >>> > mel - lom. 50 >> >>> > >> >>> > but if "mellom" is exchanged with "nasjonalbiblioteket" only the >> >>> > REST >> >>> > version shows any result, which then is the same as I get >> >>> > experimenting >> >>> > in >> >>> > the GUI. >> >>> > >> >>> > The actual script is added below, and which runs in both versions >> >>> > (identical apart form the rest and restxq interfaces), it uses full >> >>> > text >> >>> > search, but results differ when run under the REST-regime. >> >>> > >> >>> > All the best >> >>> > Lars G Johnsen >> >>> > National Library of Norway >> >>> > >> >>> > module namespace page = 'http://basex.org/modules/web-page'; >> >>> > >> >>> > declare >> >>> > %rest:path("/hyphens/{$word}") >> >>> > %output:method("html") >> >>> > >> >>> > function page:show-hyphens($word) { >> >>> > let $db := db:open('hyphen-data') >> >>> > let $hyphens := for $hyp in $db/hyphens/hyphens[full contains >> >>> > text >> >>> > {$word}] >> >>> > group by $first := $hyp/first, $second := $hyp/second >> >>> > let $count := count($hyp) >> >>> > order by xs:int($count) descending >> >>> > return element p { >> >>> > attribute freq {$count}, >> >>> > $first, " - ", $second, $count >> >>> > } >> >>> > >> >>> > let $total := sum($hyphens//@freq) >> >>> > let $div := element div { >> >>> > element p {$word}, >> >>> > for $hyp in $hyphens >> >>> > return element div { >> >>> > attribute class {"hyph"}, >> >>> > attribute style {"font-size:", 1 >> >>> > +round(xs:int($hyp//@freq/data()) >> >>> > div $total,1) || "em"}, >> >>> > $hyp >> >>> > >> >>> > } >> >>> > } >> >>> > return >> >>> > <html encoding="UTF-8"> >> >>> > <head> >> >>> > <meta http-equiv="Content-Type" content="text/html" >> >>> > charset="UTF-8" >> >>> > /> >> >>> > <title>Orddelinger</title> >> >>> > </head> >> >>> > <body>{$div} >> >>> > </body> >> >>> > </html> >> >>> > >> >>> > }; >> >> >> >> >> > > >

> > However, when using text index instead of full text the results are the > same > > for both, except that RESTXQ takes almost forever > > What about the original query: Has it been slow as well, or do you > think this is a new problem? > > > > 2015-05-18 14:28 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: > >> > >> It could be that your URL is decoded in a wrong way.. What happens if > >> you run the following function with REST and RESTXQ and "føre" as > >> word? > >> > >> declare > >> %rest:path("/test/encoding/{$word}") > >> function page:test-encoding($word) { > >> string-to-codepoints($word) > >> }; > >> > >> Thanks, > >> Christian > >> > >> > >> string-to-codepoints() > >> > REST output (2 first lines): > >> > føre > >> > fø - re 219 > >> > > >> > RESTXQ > >> > føre > >> > fo - re 123 > >> > > >> > The first word quoted is "føre" in both cases and is what the scripts > >> > see, > >> > so the full text is given the same in both cases. Could it be that > >> > within > >> > RESTXQ the full text index is treated differently? > >> > > >> > I will work closer on a self contained example, but thought this > might > >> > point to something. > >> > > >> > Cheers > >> > Lars > >> > > >> > > >> > 2015-05-18 13:44 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: > >> >> > >> >> Hi Christian - and thanks for fast response. Latest version 8.11 is > in > >> >> use > >> >> (same behaviour as previous). Let me see if I can make a self > contained > >> >> example. > >> >> > >> >> best, > >> >> Lars > >> >> > >> >> 2015-05-18 13:40 GMT+02:00 Christian Grün <christian.gruen at gmail.com > >: > >> >>> > >> >>> Hi Lars, > >> >>> > >> >>> hm, that's difficult to tell. All I can say is that this sounds > >> >>> unusual, so I'm coming up with my standard questions: Do you think > you > >> >>> could build us a little example that allows us to reproduce the > >> >>> problem? Have you tried the latest version of BaseX? > >> >>> > >> >>> Best, > >> >>> Christian > >> >>> > >> >>> > >> >>> On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> > >> >>> wrote: > >> >>> > > >> >>> > I am running a web script in two identical versions (identical as > in > >> >>> > "cut > >> >>> > and paste"), one via RESTXQ and one vi REST. The response is > >> >>> > different, > >> >>> > and > >> >>> > I wondered what may be the trouble. > >> >>> > > >> >>> > For example the output (the URLs only works locally) for > >> >>> > http://ljohnsen:8984/hyphens/mellom > >> >>> > is the same as > >> >>> > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom > >> >>> > > >> >>> > which is a set of hyphenation data: > >> >>> > mellom > >> >>> > mel - lom 17005 > >> >>> > Mel - lom 144 > >> >>> > mel - lom. 50 > >> >>> > > >> >>> > but if "mellom" is exchanged with "nasjonalbiblioteket" only the > >> >>> > REST > >> >>> > version shows any result, which then is the same as I get > >> >>> > experimenting > >> >>> > in > >> >>> > the GUI. > >> >>> > > >> >>> > The actual script is added below, and which runs in both versions > >> >>> > (identical apart form the rest and restxq interfaces), it uses > full > >> >>> > text > >> >>> > search, but results differ when run under the REST-regime. > >> >>> > > >> >>> > All the best > >> >>> > Lars G Johnsen > >> >>> > National Library of Norway > >> >>> > > >> >>> > module namespace page = 'http://basex.org/modules/web-page'; > >> >>> > > >> >>> > declare > >> >>> > %rest:path("/hyphens/{$word}") > >> >>> > %output:method("html") > >> >>> > > >> >>> > function page:show-hyphens($word) { > >> >>> > let $db := db:open('hyphen-data') > >> >>> > let $hyphens := for $hyp in $db/hyphens/hyphens[full > contains > >> >>> > text > >> >>> > {$word}] > >> >>> > group by $first := $hyp/first, $second := $hyp/second > >> >>> > let $count := count($hyp) > >> >>> > order by xs:int($count) descending > >> >>> > return element p { > >> >>> > attribute freq {$count}, > >> >>> > $first, " - ", $second, $count > >> >>> > } > >> >>> > > >> >>> > let $total := sum($hyphens//@freq) > >> >>> > let $div := element div { > >> >>> > element p {$word}, > >> >>> > for $hyp in $hyphens > >> >>> > return element div { > >> >>> > attribute class {"hyph"}, > >> >>> > attribute style {"font-size:", 1 > >> >>> > +round(xs:int($hyp//@freq/data()) > >> >>> > div $total,1) || "em"}, > >> >>> > $hyp > >> >>> > > >> >>> > } > >> >>> > } > >> >>> > return > >> >>> > <html encoding="UTF-8"> > >> >>> > <head> > >> >>> > <meta http-equiv="Content-Type" content="text/html" > >> >>> > charset="UTF-8" > >> >>> > /> > >> >>> > <title>Orddelinger</title> > >> >>> > </head> > >> >>> > <body>{$div} > >> >>> > </body> > >> >>> > </html> > >> >>> > > >> >>> > }; > >> >> > >> >> > >> > > > > > >

> The full text query is blisteringly fast for both, the text index query is > fast only for REST queries and seems not to be used with queries in RESTXQ. > I am rebuilding the whole database now to see how it goes, and will restart > everything for a new assessment. > > > > 2015-05-18 15:00 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: > >> > However, when using text index instead of full text the results are the >> same >> > for both, except that RESTXQ takes almost forever >> >> What about the original query: Has it been slow as well, or do you >> think this is a new problem? >> >> >> > 2015-05-18 14:28 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: >> >> >> >> It could be that your URL is decoded in a wrong way.. What happens if >> >> you run the following function with REST and RESTXQ and "føre" as >> >> word? >> >> >> >> declare >> >> %rest:path("/test/encoding/{$word}") >> >> function page:test-encoding($word) { >> >> string-to-codepoints($word) >> >> }; >> >> >> >> Thanks, >> >> Christian >> >> >> >> >> >> string-to-codepoints() >> >> > REST output (2 first lines): >> >> > føre >> >> > fø - re 219 >> >> > >> >> > RESTXQ >> >> > føre >> >> > fo - re 123 >> >> > >> >> > The first word quoted is "føre" in both cases and is what the scripts >> >> > see, >> >> > so the full text is given the same in both cases. Could it be that >> >> > within >> >> > RESTXQ the full text index is treated differently? >> >> > >> >> > I will work closer on a self contained example, but thought this >> might >> >> > point to something. >> >> > >> >> > Cheers >> >> > Lars >> >> > >> >> > >> >> > 2015-05-18 13:44 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: >> >> >> >> >> >> Hi Christian - and thanks for fast response. Latest version 8.11 is >> in >> >> >> use >> >> >> (same behaviour as previous). Let me see if I can make a self >> contained >> >> >> example. >> >> >> >> >> >> best, >> >> >> Lars >> >> >> >> >> >> 2015-05-18 13:40 GMT+02:00 Christian Grün < >> christian.gruen at gmail.com>: >> >> >>> >> >> >>> Hi Lars, >> >> >>> >> >> >>> hm, that's difficult to tell. All I can say is that this sounds >> >> >>> unusual, so I'm coming up with my standard questions: Do you think >> you >> >> >>> could build us a little example that allows us to reproduce the >> >> >>> problem? Have you tried the latest version of BaseX? >> >> >>> >> >> >>> Best, >> >> >>> Christian >> >> >>> >> >> >>> >> >> >>> On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> >> >> >>> wrote: >> >> >>> > >> >> >>> > I am running a web script in two identical versions (identical >> as in >> >> >>> > "cut >> >> >>> > and paste"), one via RESTXQ and one vi REST. The response is >> >> >>> > different, >> >> >>> > and >> >> >>> > I wondered what may be the trouble. >> >> >>> > >> >> >>> > For example the output (the URLs only works locally) for >> >> >>> > http://ljohnsen:8984/hyphens/mellom >> >> >>> > is the same as >> >> >>> > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom >> >> >>> > >> >> >>> > which is a set of hyphenation data: >> >> >>> > mellom >> >> >>> > mel - lom 17005 >> >> >>> > Mel - lom 144 >> >> >>> > mel - lom. 50 >> >> >>> > >> >> >>> > but if "mellom" is exchanged with "nasjonalbiblioteket" only the >> >> >>> > REST >> >> >>> > version shows any result, which then is the same as I get >> >> >>> > experimenting >> >> >>> > in >> >> >>> > the GUI. >> >> >>> > >> >> >>> > The actual script is added below, and which runs in both versions >> >> >>> > (identical apart form the rest and restxq interfaces), it uses >> full >> >> >>> > text >> >> >>> > search, but results differ when run under the REST-regime. >> >> >>> > >> >> >>> > All the best >> >> >>> > Lars G Johnsen >> >> >>> > National Library of Norway >> >> >>> > >> >> >>> > module namespace page = 'http://basex.org/modules/web-page'; >> >> >>> > >> >> >>> > declare >> >> >>> > %rest:path("/hyphens/{$word}") >> >> >>> > %output:method("html") >> >> >>> > >> >> >>> > function page:show-hyphens($word) { >> >> >>> > let $db := db:open('hyphen-data') >> >> >>> > let $hyphens := for $hyp in $db/hyphens/hyphens[full >> contains >> >> >>> > text >> >> >>> > {$word}] >> >> >>> > group by $first := $hyp/first, $second := $hyp/second >> >> >>> > let $count := count($hyp) >> >> >>> > order by xs:int($count) descending >> >> >>> > return element p { >> >> >>> > attribute freq {$count}, >> >> >>> > $first, " - ", $second, $count >> >> >>> > } >> >> >>> > >> >> >>> > let $total := sum($hyphens//@freq) >> >> >>> > let $div := element div { >> >> >>> > element p {$word}, >> >> >>> > for $hyp in $hyphens >> >> >>> > return element div { >> >> >>> > attribute class {"hyph"}, >> >> >>> > attribute style {"font-size:", 1 >> >> >>> > +round(xs:int($hyp//@freq/data()) >> >> >>> > div $total,1) || "em"}, >> >> >>> > $hyp >> >> >>> > >> >> >>> > } >> >> >>> > } >> >> >>> > return >> >> >>> > <html encoding="UTF-8"> >> >> >>> > <head> >> >> >>> > <meta http-equiv="Content-Type" content="text/html" >> >> >>> > charset="UTF-8" >> >> >>> > /> >> >> >>> > <title>Orddelinger</title> >> >> >>> > </head> >> >> >>> > <body>{$div} >> >> >>> > </body> >> >> >>> > </html> >> >> >>> > >> >> >>> > }; >> >> >> >> >> >> >> >> > >> > >> > >> > >

> As an update, after rebuilding database with > > text index, > full text index (no language, no stemming, keep diacritics) > > restarting server: > BaseX 8.1.1 [Server] > Server was started (port: 29084) > [main] INFO org.eclipse.jetty.server.AbstractConnector - Started > SelectChannelConnector at 0.0.0.0:8984 > HTTP Server was started (port: 8984) > > RESTXQ: Norwegian characters are converted using full text index, changing > to text index takes forever. > REST: Full-text works as expected, and text index works as expected (same > as runing in GUI for both). > > It looks as if the index structure is treated differently. > > > 2015-05-18 15:07 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: > >> The full text query is blisteringly fast for both, the text index query >> is fast only for REST queries and seems not to be used with queries in >> RESTXQ. I am rebuilding the whole database now to see how it goes, and will >> restart everything for a new assessment. >> >> >> >> 2015-05-18 15:00 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: >> >>> > However, when using text index instead of full text the results are >>> the same >>> > for both, except that RESTXQ takes almost forever >>> >>> What about the original query: Has it been slow as well, or do you >>> think this is a new problem? >>> >>> >>> > 2015-05-18 14:28 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: >>> >> >>> >> It could be that your URL is decoded in a wrong way.. What happens if >>> >> you run the following function with REST and RESTXQ and "føre" as >>> >> word? >>> >> >>> >> declare >>> >> %rest:path("/test/encoding/{$word}") >>> >> function page:test-encoding($word) { >>> >> string-to-codepoints($word) >>> >> }; >>> >> >>> >> Thanks, >>> >> Christian >>> >> >>> >> >>> >> string-to-codepoints() >>> >> > REST output (2 first lines): >>> >> > føre >>> >> > fø - re 219 >>> >> > >>> >> > RESTXQ >>> >> > føre >>> >> > fo - re 123 >>> >> > >>> >> > The first word quoted is "føre" in both cases and is what the >>> scripts >>> >> > see, >>> >> > so the full text is given the same in both cases. Could it be that >>> >> > within >>> >> > RESTXQ the full text index is treated differently? >>> >> > >>> >> > I will work closer on a self contained example, but thought this >>> might >>> >> > point to something. >>> >> > >>> >> > Cheers >>> >> > Lars >>> >> > >>> >> > >>> >> > 2015-05-18 13:44 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: >>> >> >> >>> >> >> Hi Christian - and thanks for fast response. Latest version 8.11 >>> is in >>> >> >> use >>> >> >> (same behaviour as previous). Let me see if I can make a self >>> contained >>> >> >> example. >>> >> >> >>> >> >> best, >>> >> >> Lars >>> >> >> >>> >> >> 2015-05-18 13:40 GMT+02:00 Christian Grün < >>> christian.gruen at gmail.com>: >>> >> >>> >>> >> >>> Hi Lars, >>> >> >>> >>> >> >>> hm, that's difficult to tell. All I can say is that this sounds >>> >> >>> unusual, so I'm coming up with my standard questions: Do you >>> think you >>> >> >>> could build us a little example that allows us to reproduce the >>> >> >>> problem? Have you tried the latest version of BaseX? >>> >> >>> >>> >> >>> Best, >>> >> >>> Christian >>> >> >>> >>> >> >>> >>> >> >>> On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> >>> >> >>> wrote: >>> >> >>> > >>> >> >>> > I am running a web script in two identical versions (identical >>> as in >>> >> >>> > "cut >>> >> >>> > and paste"), one via RESTXQ and one vi REST. The response is >>> >> >>> > different, >>> >> >>> > and >>> >> >>> > I wondered what may be the trouble. >>> >> >>> > >>> >> >>> > For example the output (the URLs only works locally) for >>> >> >>> > http://ljohnsen:8984/hyphens/mellom >>> >> >>> > is the same as >>> >> >>> > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom >>> >> >>> > >>> >> >>> > which is a set of hyphenation data: >>> >> >>> > mellom >>> >> >>> > mel - lom 17005 >>> >> >>> > Mel - lom 144 >>> >> >>> > mel - lom. 50 >>> >> >>> > >>> >> >>> > but if "mellom" is exchanged with "nasjonalbiblioteket" only >>> the >>> >> >>> > REST >>> >> >>> > version shows any result, which then is the same as I get >>> >> >>> > experimenting >>> >> >>> > in >>> >> >>> > the GUI. >>> >> >>> > >>> >> >>> > The actual script is added below, and which runs in both >>> versions >>> >> >>> > (identical apart form the rest and restxq interfaces), it uses >>> full >>> >> >>> > text >>> >> >>> > search, but results differ when run under the REST-regime. >>> >> >>> > >>> >> >>> > All the best >>> >> >>> > Lars G Johnsen >>> >> >>> > National Library of Norway >>> >> >>> > >>> >> >>> > module namespace page = 'http://basex.org/modules/web-page'; >>> >> >>> > >>> >> >>> > declare >>> >> >>> > %rest:path("/hyphens/{$word}") >>> >> >>> > %output:method("html") >>> >> >>> > >>> >> >>> > function page:show-hyphens($word) { >>> >> >>> > let $db := db:open('hyphen-data') >>> >> >>> > let $hyphens := for $hyp in $db/hyphens/hyphens[full >>> contains >>> >> >>> > text >>> >> >>> > {$word}] >>> >> >>> > group by $first := $hyp/first, $second := $hyp/second >>> >> >>> > let $count := count($hyp) >>> >> >>> > order by xs:int($count) descending >>> >> >>> > return element p { >>> >> >>> > attribute freq {$count}, >>> >> >>> > $first, " - ", $second, $count >>> >> >>> > } >>> >> >>> > >>> >> >>> > let $total := sum($hyphens//@freq) >>> >> >>> > let $div := element div { >>> >> >>> > element p {$word}, >>> >> >>> > for $hyp in $hyphens >>> >> >>> > return element div { >>> >> >>> > attribute class {"hyph"}, >>> >> >>> > attribute style {"font-size:", 1 >>> >> >>> > +round(xs:int($hyp//@freq/data()) >>> >> >>> > div $total,1) || "em"}, >>> >> >>> > $hyp >>> >> >>> > >>> >> >>> > } >>> >> >>> > } >>> >> >>> > return >>> >> >>> > <html encoding="UTF-8"> >>> >> >>> > <head> >>> >> >>> > <meta http-equiv="Content-Type" content="text/html" >>> >> >>> > charset="UTF-8" >>> >> >>> > /> >>> >> >>> > <title>Orddelinger</title> >>> >> >>> > </head> >>> >> >>> > <body>{$div} >>> >> >>> > </body> >>> >> >>> > </html> >>> >> >>> > >>> >> >>> > }; >>> >> >> >>> >> >> >>> >> > >>> > >>> > >>> >> >> >

On Mon, May 18, 2015 at 6:46 PM, Lars Johnsen <yoonsen at gmail.com> wrote: > A last update, which may illuminate a little. After reindexing the database > using Norwegian (snowball), stemming, and keeping diacritis, RESTXQ > processes neither the special characters (treats them as closest ascii), nor > inflected forms. > > The words "mannen" (=the man, definite) and "spaserer" (=walks, present > tense), result in no output, while using the naked stems "mann" and "spaser" > the full result is displayed. In contrast to REST which behaves as expected. > > > Cheers > Lars > > 2015-05-18 15:28 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: >> >> As an update, after rebuilding database with >> >> text index, >> full text index (no language, no stemming, keep diacritics) >> >> restarting server: >> BaseX 8.1.1 [Server] >> Server was started (port: 29084) >> [main] INFO org.eclipse.jetty.server.AbstractConnector - Started >> SelectChannelConnector at 0.0.0.0:8984 >> HTTP Server was started (port: 8984) >> >> RESTXQ: Norwegian characters are converted using full text index, changing >> to text index takes forever. >> REST: Full-text works as expected, and text index works as expected (same >> as runing in GUI for both). >> >> It looks as if the index structure is treated differently. >> >> >> 2015-05-18 15:07 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: >>> >>> The full text query is blisteringly fast for both, the text index query >>> is fast only for REST queries and seems not to be used with queries in >>> RESTXQ. I am rebuilding the whole database now to see how it goes, and will >>> restart everything for a new assessment. >>> >>> >>> >>> 2015-05-18 15:00 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: >>>> >>>> > However, when using text index instead of full text the results are >>>> > the same >>>> > for both, except that RESTXQ takes almost forever >>>> >>>> What about the original query: Has it been slow as well, or do you >>>> think this is a new problem? >>>> >>>> >>>> > 2015-05-18 14:28 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: >>>> >> >>>> >> It could be that your URL is decoded in a wrong way.. What happens if >>>> >> you run the following function with REST and RESTXQ and "føre" as >>>> >> word? >>>> >> >>>> >> declare >>>> >> %rest:path("/test/encoding/{$word}") >>>> >> function page:test-encoding($word) { >>>> >> string-to-codepoints($word) >>>> >> }; >>>> >> >>>> >> Thanks, >>>> >> Christian >>>> >> >>>> >> >>>> >> string-to-codepoints() >>>> >> > REST output (2 first lines): >>>> >> > føre >>>> >> > fø - re 219 >>>> >> > >>>> >> > RESTXQ >>>> >> > føre >>>> >> > fo - re 123 >>>> >> > >>>> >> > The first word quoted is "føre" in both cases and is what the >>>> >> > scripts >>>> >> > see, >>>> >> > so the full text is given the same in both cases. Could it be that >>>> >> > within >>>> >> > RESTXQ the full text index is treated differently? >>>> >> > >>>> >> > I will work closer on a self contained example, but thought this >>>> >> > might >>>> >> > point to something. >>>> >> > >>>> >> > Cheers >>>> >> > Lars >>>> >> > >>>> >> > >>>> >> > 2015-05-18 13:44 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: >>>> >> >> >>>> >> >> Hi Christian - and thanks for fast response. Latest version 8.11 >>>> >> >> is in >>>> >> >> use >>>> >> >> (same behaviour as previous). Let me see if I can make a self >>>> >> >> contained >>>> >> >> example. >>>> >> >> >>>> >> >> best, >>>> >> >> Lars >>>> >> >> >>>> >> >> 2015-05-18 13:40 GMT+02:00 Christian Grün >>>> >> >> <christian.gruen at gmail.com>: >>>> >> >>> >>>> >> >>> Hi Lars, >>>> >> >>> >>>> >> >>> hm, that's difficult to tell. All I can say is that this sounds >>>> >> >>> unusual, so I'm coming up with my standard questions: Do you >>>> >> >>> think you >>>> >> >>> could build us a little example that allows us to reproduce the >>>> >> >>> problem? Have you tried the latest version of BaseX? >>>> >> >>> >>>> >> >>> Best, >>>> >> >>> Christian >>>> >> >>> >>>> >> >>> >>>> >> >>> On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> >>>> >> >>> wrote: >>>> >> >>> > >>>> >> >>> > I am running a web script in two identical versions (identical >>>> >> >>> > as in >>>> >> >>> > "cut >>>> >> >>> > and paste"), one via RESTXQ and one vi REST. The response is >>>> >> >>> > different, >>>> >> >>> > and >>>> >> >>> > I wondered what may be the trouble. >>>> >> >>> > >>>> >> >>> > For example the output (the URLs only works locally) for >>>> >> >>> > http://ljohnsen:8984/hyphens/mellom >>>> >> >>> > is the same as >>>> >> >>> > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom >>>> >> >>> > >>>> >> >>> > which is a set of hyphenation data: >>>> >> >>> > mellom >>>> >> >>> > mel - lom 17005 >>>> >> >>> > Mel - lom 144 >>>> >> >>> > mel - lom. 50 >>>> >> >>> > >>>> >> >>> > but if "mellom" is exchanged with "nasjonalbiblioteket" only >>>> >> >>> > the >>>> >> >>> > REST >>>> >> >>> > version shows any result, which then is the same as I get >>>> >> >>> > experimenting >>>> >> >>> > in >>>> >> >>> > the GUI. >>>> >> >>> > >>>> >> >>> > The actual script is added below, and which runs in both >>>> >> >>> > versions >>>> >> >>> > (identical apart form the rest and restxq interfaces), it uses >>>> >> >>> > full >>>> >> >>> > text >>>> >> >>> > search, but results differ when run under the REST-regime. >>>> >> >>> > >>>> >> >>> > All the best >>>> >> >>> > Lars G Johnsen >>>> >> >>> > National Library of Norway >>>> >> >>> > >>>> >> >>> > module namespace page = 'http://basex.org/modules/web-page'; >>>> >> >>> > >>>> >> >>> > declare >>>> >> >>> > %rest:path("/hyphens/{$word}") >>>> >> >>> > %output:method("html") >>>> >> >>> > >>>> >> >>> > function page:show-hyphens($word) { >>>> >> >>> > let $db := db:open('hyphen-data') >>>> >> >>> > let $hyphens := for $hyp in $db/hyphens/hyphens[full >>>> >> >>> > contains >>>> >> >>> > text >>>> >> >>> > {$word}] >>>> >> >>> > group by $first := $hyp/first, $second := $hyp/second >>>> >> >>> > let $count := count($hyp) >>>> >> >>> > order by xs:int($count) descending >>>> >> >>> > return element p { >>>> >> >>> > attribute freq {$count}, >>>> >> >>> > $first, " - ", $second, $count >>>> >> >>> > } >>>> >> >>> > >>>> >> >>> > let $total := sum($hyphens//@freq) >>>> >> >>> > let $div := element div { >>>> >> >>> > element p {$word}, >>>> >> >>> > for $hyp in $hyphens >>>> >> >>> > return element div { >>>> >> >>> > attribute class {"hyph"}, >>>> >> >>> > attribute style {"font-size:", 1 >>>> >> >>> > +round(xs:int($hyp//@freq/data()) >>>> >> >>> > div $total,1) || "em"}, >>>> >> >>> > $hyp >>>> >> >>> > >>>> >> >>> > } >>>> >> >>> > } >>>> >> >>> > return >>>> >> >>> > <html encoding="UTF-8"> >>>> >> >>> > <head> >>>> >> >>> > <meta http-equiv="Content-Type" content="text/html" >>>> >> >>> > charset="UTF-8" >>>> >> >>> > /> >>>> >> >>> > <title>Orddelinger</title> >>>> >> >>> > </head> >>>> >> >>> > <body>{$div} >>>> >> >>> > </body> >>>> >> >>> > </html> >>>> >> >>> > >>>> >> >>> > }; >>>> >> >> >>>> >> >> >>>> >> > >>>> > >>>> > >>> >>> >> >

> I'll check out how this can be fixed.

> On Mon, May 18, 2015 at 6:46 PM, Lars Johnsen <yoonsen at gmail.com> wrote: >> A last update, which may illuminate a little. After reindexing the database >> using Norwegian (snowball), stemming, and keeping diacritis, RESTXQ >> processes neither the special characters (treats them as closest ascii), nor >> inflected forms. >> >> The words "mannen" (=the man, definite) and "spaserer" (=walks, present >> tense), result in no output, while using the naked stems "mann" and "spaser" >> the full result is displayed. In contrast to REST which behaves as expected. >> >> >> Cheers >> Lars >> >> 2015-05-18 15:28 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: >>> >>> As an update, after rebuilding database with >>> >>> text index, >>> full text index (no language, no stemming, keep diacritics) >>> >>> restarting server: >>> BaseX 8.1.1 [Server] >>> Server was started (port: 29084) >>> [main] INFO org.eclipse.jetty.server.AbstractConnector - Started >>> SelectChannelConnector at 0.0.0.0:8984 >>> HTTP Server was started (port: 8984) >>> >>> RESTXQ: Norwegian characters are converted using full text index, changing >>> to text index takes forever. >>> REST: Full-text works as expected, and text index works as expected (same >>> as runing in GUI for both). >>> >>> It looks as if the index structure is treated differently. >>> >>> >>> 2015-05-18 15:07 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: >>>> >>>> The full text query is blisteringly fast for both, the text index query >>>> is fast only for REST queries and seems not to be used with queries in >>>> RESTXQ. I am rebuilding the whole database now to see how it goes, and will >>>> restart everything for a new assessment. >>>> >>>> >>>> >>>> 2015-05-18 15:00 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: >>>>> >>>>> > However, when using text index instead of full text the results are >>>>> > the same >>>>> > for both, except that RESTXQ takes almost forever >>>>> >>>>> What about the original query: Has it been slow as well, or do you >>>>> think this is a new problem? >>>>> >>>>> >>>>> > 2015-05-18 14:28 GMT+02:00 Christian Grün <christian.gruen at gmail.com>: >>>>> >> >>>>> >> It could be that your URL is decoded in a wrong way.. What happens if >>>>> >> you run the following function with REST and RESTXQ and "føre" as >>>>> >> word? >>>>> >> >>>>> >> declare >>>>> >> %rest:path("/test/encoding/{$word}") >>>>> >> function page:test-encoding($word) { >>>>> >> string-to-codepoints($word) >>>>> >> }; >>>>> >> >>>>> >> Thanks, >>>>> >> Christian >>>>> >> >>>>> >> >>>>> >> string-to-codepoints() >>>>> >> > REST output (2 first lines): >>>>> >> > føre >>>>> >> > fø - re 219 >>>>> >> > >>>>> >> > RESTXQ >>>>> >> > føre >>>>> >> > fo - re 123 >>>>> >> > >>>>> >> > The first word quoted is "føre" in both cases and is what the >>>>> >> > scripts >>>>> >> > see, >>>>> >> > so the full text is given the same in both cases. Could it be that >>>>> >> > within >>>>> >> > RESTXQ the full text index is treated differently? >>>>> >> > >>>>> >> > I will work closer on a self contained example, but thought this >>>>> >> > might >>>>> >> > point to something. >>>>> >> > >>>>> >> > Cheers >>>>> >> > Lars >>>>> >> > >>>>> >> > >>>>> >> > 2015-05-18 13:44 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: >>>>> >> >> >>>>> >> >> Hi Christian - and thanks for fast response. Latest version 8.11 >>>>> >> >> is in >>>>> >> >> use >>>>> >> >> (same behaviour as previous). Let me see if I can make a self >>>>> >> >> contained >>>>> >> >> example. >>>>> >> >> >>>>> >> >> best, >>>>> >> >> Lars >>>>> >> >> >>>>> >> >> 2015-05-18 13:40 GMT+02:00 Christian Grün >>>>> >> >> <christian.gruen at gmail.com>: >>>>> >> >>> >>>>> >> >>> Hi Lars, >>>>> >> >>> >>>>> >> >>> hm, that's difficult to tell. All I can say is that this sounds >>>>> >> >>> unusual, so I'm coming up with my standard questions: Do you >>>>> >> >>> think you >>>>> >> >>> could build us a little example that allows us to reproduce the >>>>> >> >>> problem? Have you tried the latest version of BaseX? >>>>> >> >>> >>>>> >> >>> Best, >>>>> >> >>> Christian >>>>> >> >>> >>>>> >> >>> >>>>> >> >>> On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen <yoonsen at gmail.com> >>>>> >> >>> wrote: >>>>> >> >>> > >>>>> >> >>> > I am running a web script in two identical versions (identical >>>>> >> >>> > as in >>>>> >> >>> > "cut >>>>> >> >>> > and paste"), one via RESTXQ and one vi REST. The response is >>>>> >> >>> > different, >>>>> >> >>> > and >>>>> >> >>> > I wondered what may be the trouble. >>>>> >> >>> > >>>>> >> >>> > For example the output (the URLs only works locally) for >>>>> >> >>> > http://ljohnsen:8984/hyphens/mellom >>>>> >> >>> > is the same as >>>>> >> >>> > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom >>>>> >> >>> > >>>>> >> >>> > which is a set of hyphenation data: >>>>> >> >>> > mellom >>>>> >> >>> > mel - lom 17005 >>>>> >> >>> > Mel - lom 144 >>>>> >> >>> > mel - lom. 50 >>>>> >> >>> > >>>>> >> >>> > but if "mellom" is exchanged with "nasjonalbiblioteket" only >>>>> >> >>> > the >>>>> >> >>> > REST >>>>> >> >>> > version shows any result, which then is the same as I get >>>>> >> >>> > experimenting >>>>> >> >>> > in >>>>> >> >>> > the GUI. >>>>> >> >>> > >>>>> >> >>> > The actual script is added below, and which runs in both >>>>> >> >>> > versions >>>>> >> >>> > (identical apart form the rest and restxq interfaces), it uses >>>>> >> >>> > full >>>>> >> >>> > text >>>>> >> >>> > search, but results differ when run under the REST-regime. >>>>> >> >>> > >>>>> >> >>> > All the best >>>>> >> >>> > Lars G Johnsen >>>>> >> >>> > National Library of Norway >>>>> >> >>> > >>>>> >> >>> > module namespace page = 'http://basex.org/modules/web-page'; >>>>> >> >>> > >>>>> >> >>> > declare >>>>> >> >>> > %rest:path("/hyphens/{$word}") >>>>> >> >>> > %output:method("html") >>>>> >> >>> > >>>>> >> >>> > function page:show-hyphens($word) { >>>>> >> >>> > let $db := db:open('hyphen-data') >>>>> >> >>> > let $hyphens := for $hyp in $db/hyphens/hyphens[full >>>>> >> >>> > contains >>>>> >> >>> > text >>>>> >> >>> > {$word}] >>>>> >> >>> > group by $first := $hyp/first, $second := $hyp/second >>>>> >> >>> > let $count := count($hyp) >>>>> >> >>> > order by xs:int($count) descending >>>>> >> >>> > return element p { >>>>> >> >>> > attribute freq {$count}, >>>>> >> >>> > $first, " - ", $second, $count >>>>> >> >>> > } >>>>> >> >>> > >>>>> >> >>> > let $total := sum($hyphens//@freq) >>>>> >> >>> > let $div := element div { >>>>> >> >>> > element p {$word}, >>>>> >> >>> > for $hyp in $hyphens >>>>> >> >>> > return element div { >>>>> >> >>> > attribute class {"hyph"}, >>>>> >> >>> > attribute style {"font-size:", 1 >>>>> >> >>> > +round(xs:int($hyp//@freq/data()) >>>>> >> >>> > div $total,1) || "em"}, >>>>> >> >>> > $hyp >>>>> >> >>> > >>>>> >> >>> > } >>>>> >> >>> > } >>>>> >> >>> > return >>>>> >> >>> > <html encoding="UTF-8"> >>>>> >> >>> > <head> >>>>> >> >>> > <meta http-equiv="Content-Type" content="text/html" >>>>> >> >>> > charset="UTF-8" >>>>> >> >>> > /> >>>>> >> >>> > <title>Orddelinger</title> >>>>> >> >>> > </head> >>>>> >> >>> > <body>{$div} >>>>> >> >>> > </body> >>>>> >> >>> > </html> >>>>> >> >>> > >>>>> >> >>> > }; >>>>> >> >> >>>>> >> >> >>>>> >> > >>>>> > >>>>> > >>>> >>>> >>> >>

> > I'll check out how this can be fixed. > > So I checked out how to fix it, and I fixed it [1]. Feel free to try > the latest snapshot [2]! > Christian > > [1] https://github.com/BaseXdb/basex/issues/1144 > [2] http://files.basex.org/releases/latest > > > > On Mon, May 18, 2015 at 6:46 PM, Lars Johnsen <yoonsen at gmail.com> wrote: > >> A last update, which may illuminate a little. After reindexing the > database > >> using Norwegian (snowball), stemming, and keeping diacritis, RESTXQ > >> processes neither the special characters (treats them as closest > ascii), nor > >> inflected forms. > >> > >> The words "mannen" (=the man, definite) and "spaserer" (=walks, present > >> tense), result in no output, while using the naked stems "mann" and > "spaser" > >> the full result is displayed. In contrast to REST which behaves as > expected. > >> > >> > >> Cheers > >> Lars > >> > >> 2015-05-18 15:28 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: > >>> > >>> As an update, after rebuilding database with > >>> > >>> text index, > >>> full text index (no language, no stemming, keep diacritics) > >>> > >>> restarting server: > >>> BaseX 8.1.1 [Server] > >>> Server was started (port: 29084) > >>> [main] INFO org.eclipse.jetty.server.AbstractConnector - Started > >>> SelectChannelConnector at 0.0.0.0:8984 > >>> HTTP Server was started (port: 8984) > >>> > >>> RESTXQ: Norwegian characters are converted using full text index, > changing > >>> to text index takes forever. > >>> REST: Full-text works as expected, and text index works as expected > (same > >>> as runing in GUI for both). > >>> > >>> It looks as if the index structure is treated differently. > >>> > >>> > >>> 2015-05-18 15:07 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: > >>>> > >>>> The full text query is blisteringly fast for both, the text index > query > >>>> is fast only for REST queries and seems not to be used with queries in > >>>> RESTXQ. I am rebuilding the whole database now to see how it goes, > and will > >>>> restart everything for a new assessment. > >>>> > >>>> > >>>> > >>>> 2015-05-18 15:00 GMT+02:00 Christian Grün <christian.gruen at gmail.com > >: > >>>>> > >>>>> > However, when using text index instead of full text the results are > >>>>> > the same > >>>>> > for both, except that RESTXQ takes almost forever > >>>>> > >>>>> What about the original query: Has it been slow as well, or do you > >>>>> think this is a new problem? > >>>>> > >>>>> > >>>>> > 2015-05-18 14:28 GMT+02:00 Christian Grün < > christian.gruen at gmail.com>: > >>>>> >> > >>>>> >> It could be that your URL is decoded in a wrong way.. What > happens if > >>>>> >> you run the following function with REST and RESTXQ and "føre" as > >>>>> >> word? > >>>>> >> > >>>>> >> declare > >>>>> >> %rest:path("/test/encoding/{$word}") > >>>>> >> function page:test-encoding($word) { > >>>>> >> string-to-codepoints($word) > >>>>> >> }; > >>>>> >> > >>>>> >> Thanks, > >>>>> >> Christian > >>>>> >> > >>>>> >> > >>>>> >> string-to-codepoints() > >>>>> >> > REST output (2 first lines): > >>>>> >> > føre > >>>>> >> > fø - re 219 > >>>>> >> > > >>>>> >> > RESTXQ > >>>>> >> > føre > >>>>> >> > fo - re 123 > >>>>> >> > > >>>>> >> > The first word quoted is "føre" in both cases and is what the > >>>>> >> > scripts > >>>>> >> > see, > >>>>> >> > so the full text is given the same in both cases. Could it be > that > >>>>> >> > within > >>>>> >> > RESTXQ the full text index is treated differently? > >>>>> >> > > >>>>> >> > I will work closer on a self contained example, but thought > this > >>>>> >> > might > >>>>> >> > point to something. > >>>>> >> > > >>>>> >> > Cheers > >>>>> >> > Lars > >>>>> >> > > >>>>> >> > > >>>>> >> > 2015-05-18 13:44 GMT+02:00 Lars Johnsen <yoonsen at gmail.com>: > >>>>> >> >> > >>>>> >> >> Hi Christian - and thanks for fast response. Latest version > 8.11 > >>>>> >> >> is in > >>>>> >> >> use > >>>>> >> >> (same behaviour as previous). Let me see if I can make a self > >>>>> >> >> contained > >>>>> >> >> example. > >>>>> >> >> > >>>>> >> >> best, > >>>>> >> >> Lars > >>>>> >> >> > >>>>> >> >> 2015-05-18 13:40 GMT+02:00 Christian Grün > >>>>> >> >> <christian.gruen at gmail.com>: > >>>>> >> >>> > >>>>> >> >>> Hi Lars, > >>>>> >> >>> > >>>>> >> >>> hm, that's difficult to tell. All I can say is that this > sounds > >>>>> >> >>> unusual, so I'm coming up with my standard questions: Do you > >>>>> >> >>> think you > >>>>> >> >>> could build us a little example that allows us to reproduce > the > >>>>> >> >>> problem? Have you tried the latest version of BaseX? > >>>>> >> >>> > >>>>> >> >>> Best, > >>>>> >> >>> Christian > >>>>> >> >>> > >>>>> >> >>> > >>>>> >> >>> On Mon, May 18, 2015 at 1:35 PM, Lars Johnsen < > yoonsen at gmail.com> > >>>>> >> >>> wrote: > >>>>> >> >>> > > >>>>> >> >>> > I am running a web script in two identical versions > (identical > >>>>> >> >>> > as in > >>>>> >> >>> > "cut > >>>>> >> >>> > and paste"), one via RESTXQ and one vi REST. The response is > >>>>> >> >>> > different, > >>>>> >> >>> > and > >>>>> >> >>> > I wondered what may be the trouble. > >>>>> >> >>> > > >>>>> >> >>> > For example the output (the URLs only works locally) for > >>>>> >> >>> > http://ljohnsen:8984/hyphens/mellom > >>>>> >> >>> > is the same as > >>>>> >> >>> > > http://ljohnsen:8984/rest?run=hyphen-show.xq&word=mellom > >>>>> >> >>> > > >>>>> >> >>> > which is a set of hyphenation data: > >>>>> >> >>> > mellom > >>>>> >> >>> > mel - lom 17005 > >>>>> >> >>> > Mel - lom 144 > >>>>> >> >>> > mel - lom. 50 > >>>>> >> >>> > > >>>>> >> >>> > but if "mellom" is exchanged with "nasjonalbiblioteket" only > >>>>> >> >>> > the > >>>>> >> >>> > REST > >>>>> >> >>> > version shows any result, which then is the same as I get > >>>>> >> >>> > experimenting > >>>>> >> >>> > in > >>>>> >> >>> > the GUI. > >>>>> >> >>> > > >>>>> >> >>> > The actual script is added below, and which runs in both > >>>>> >> >>> > versions > >>>>> >> >>> > (identical apart form the rest and restxq interfaces), it > uses > >>>>> >> >>> > full > >>>>> >> >>> > text > >>>>> >> >>> > search, but results differ when run under the REST-regime. > >>>>> >> >>> > > >>>>> >> >>> > All the best > >>>>> >> >>> > Lars G Johnsen > >>>>> >> >>> > National Library of Norway > >>>>> >> >>> > > >>>>> >> >>> > module namespace page = 'http://basex.org/modules/web-page > '; > >>>>> >> >>> > > >>>>> >> >>> > declare > >>>>> >> >>> > %rest:path("/hyphens/{$word}") > >>>>> >> >>> > %output:method("html") > >>>>> >> >>> > > >>>>> >> >>> > function page:show-hyphens($word) { > >>>>> >> >>> > let $db := db:open('hyphen-data') > >>>>> >> >>> > let $hyphens := for $hyp in $db/hyphens/hyphens[full > >>>>> >> >>> > contains > >>>>> >> >>> > text > >>>>> >> >>> > {$word}] > >>>>> >> >>> > group by $first := $hyp/first, $second := $hyp/second > >>>>> >> >>> > let $count := count($hyp) > >>>>> >> >>> > order by xs:int($count) descending > >>>>> >> >>> > return element p { > >>>>> >> >>> > attribute freq {$count}, > >>>>> >> >>> > $first, " - ", $second, $count > >>>>> >> >>> > } > >>>>> >> >>> > > >>>>> >> >>> > let $total := sum($hyphens//@freq) > >>>>> >> >>> > let $div := element div { > >>>>> >> >>> > element p {$word}, > >>>>> >> >>> > for $hyp in $hyphens > >>>>> >> >>> > return element div { > >>>>> >> >>> > attribute class {"hyph"}, > >>>>> >> >>> > attribute style {"font-size:", 1 > >>>>> >> >>> > +round(xs:int($hyp//@freq/data()) > >>>>> >> >>> > div $total,1) || "em"}, > >>>>> >> >>> > $hyp > >>>>> >> >>> > > >>>>> >> >>> > } > >>>>> >> >>> > } > >>>>> >> >>> > return > >>>>> >> >>> > <html encoding="UTF-8"> > >>>>> >> >>> > <head> > >>>>> >> >>> > <meta http-equiv="Content-Type" content="text/html" > >>>>> >> >>> > charset="UTF-8" > >>>>> >> >>> > /> > >>>>> >> >>> > <title>Orddelinger</title> > >>>>> >> >>> > </head> > >>>>> >> >>> > <body>{$div} > >>>>> >> >>> > </body> > >>>>> >> >>> > </html> > >>>>> >> >>> > > >>>>> >> >>> > }; > >>>>> >> >> > >>>>> >> >> > >>>>> >> > > >>>>> > > >>>>> > > >>>> > >>>> > >>> > >> >