Main Page » Mailing List » August 2018

diacritics sensitive not working

Wednesday, 1 August 2018 · 16 messages · last reply 3 Sep 2018
RK
Ron Katriel1 Aug 2018 07:08 #
Hi, The following from your website (docs.basex.org/wiki/Full-Text) appears to be syntactically incorrect "'Äpfel' will not be found..." contains text "Apfel" diacritics sensitive In the BaseX GUI the keyword diacritics is underlined in red and the following error is reported Unexpected end of query: 'diacritic sens...'. This happens in version 8.6.4 and also the latest (9.0.2). Thanks, Ron *Ron Katriel, Ph.D. *| Principal Data Scientist | Medidata Solutions <http://www.mdsol.com/> 350 Hudson Street, 7th Floor, New York, NY 10014
rkatriel at mdsol.com <tbrophy at mdsol.com> | direct: +1 201 337 3622
<//201%20337%203622> | mobile: +1 201 675 5598
<//mobile:%20+1%20201%20675%205598> | main: +1 212 918 1800
<//+1%20212%20918%201800>
CG
Christian Grün1 Aug 2018 09:57 #
I have fixed the example in the doc. Best, Christian
Quoted text · Ron Katriel · 25 lines
On Wed, Aug 1, 2018 at 5:08 AM Ron Katriel <rkatriel at mdsol.com> wrote: > > Hi, > > The following from your website (docs.basex.org/wiki/Full-Text) appears to be syntactically incorrect > > "'Äpfel' will not be found..." contains text "Apfel" diacritics sensitive > > In the BaseX GUI the keyword diacritics is underlined in red and the following error is reported > > Unexpected end of query: 'diacritic sens...'. > > This happens in version 8.6.4 and also the latest (9.0.2). > > Thanks, > Ron > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > > 350 Hudson Street, 7th Floor, New York, NY 10014 > > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800 > >
RK
Ron Katriel1 Aug 2018 20:41 #
Thanks, Christian. Strange, prior to contacting you and on a hunch, I tried adding the missing “using” keyword but still got the syntax error. Anyway, everything is good now! Best, Ron
Quoted text · Christian Grün (christian.gruen at gmail.com) · 2 lines
On August 1, 2018 at 3:57:51 AM, Christian Grün (christian.gruen at gmail.com) wrote:
I have fixed the example in the doc. Best, Christian
Quoted text · Ron Katriel · 5 lines
On Wed, Aug 1, 2018 at 5:08 AM Ron Katriel <rkatriel at mdsol.com> wrote: > > Hi, > > The following from your website (docs.basex.org/wiki/Full-Text) appears
to be syntactically incorrect
Quoted text · Christian Grün · 4 lines
> > "'Äpfel' will not be found..." contains text "Apfel" diacritics sensitive > > In the BaseX GUI the keyword diacritics is underlined in red and the
following error is reported
Quoted text · Christian Grün · 14 lines
> > Unexpected end of query: 'diacritic sens...'. > > This happens in version 8.6.4 and also the latest (9.0.2). > > Thanks, > Ron > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > > 350 Hudson Street, 7th Floor, New York, NY 10014 > > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 |
main: +1 212 918 1800
Quoted text · Christian Grün · 2 lines
> >
IL
Imsieke, Gerrit, le-texin reply to Ron Katriel, 3 Aug 2018, 16:393 Aug 2018 15:08 #
Hi Ron, You can add an extra element (or attribute) to the content when importing or modifying it. (Or another document in another database if you like – you can create and later find such an index document by giving it the same db:path as the original document.) In this extra database, document, element and/or attribute, you can recreate the original text, except that you normalize the characters with diacritical marks to a canonical decomposition form and then strip away the diacritical marks like this: replace(normalize-unicode($input, 'NFKD'), '\p{Mn}', '') The full updating statement is beyond my cursory XQuery capabilities – I’d probably do it in XSLT. Also I don’t know how to trigger an event that would cause an update of the auxiliary fields when the underlying data changes. Gerrit
Quoted text · Ron Katriel · 81 lines
On 03.08.2018 14:39, Ron Katriel wrote: > Christian, > > Adding diacritics sensitive slows execution by a factor of 3. My script > (fragment below), which joins two large databases, namely CT.gov > <http://clinicaltrials.gov> and DrugBank, takes 2 hours without the > diacritics sensitive constraint but 6 hours with it. Given the > combinatorics involved, I am wondering if there is a better way to do > this in BaseX. > > Thanks, > Ron > > > for $drug in db:open('DrugBank')/drugbank/drug > let $drug_name := $drug/name/text() > let $drug_synonyms := > functx:value-union(normalize-space(lower-case($drug/name)), > local:drug-synonyms($drug_name)) > for $synonym_name in $drug_synonyms > ... > for $study in > db:open('CTGov')/clinical_study[intervention/intervention_name contains > text { $synonym_name } using case insensitive using diacritics sensitive] > ... > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > <http://www.mdsol.com/> > 350 Hudson Street, 7th Floor, New York, NY 10014 > rkatriel at mdsol.com <mailto:tbrophy at mdsol.com> | direct: +1 201 337 3622 > <tel://201%20337%203622> | mobile: +1 201 675 5598 > <tel://+1%20201%20675%205598> | main: +1 212 918 1800 > <tel://+1%20212%20918%201800> > > On August 1, 2018 at 12:41:26 PM, Ron Katriel (rkatriel at mdsol.com > <mailto:rkatriel at mdsol.com>) wrote: > >> Thanks, Christian. Strange, prior to contacting you and on a hunch, I >> tried adding the missing “using” keyword but still got the syntax >> error. Anyway, everything is good now! >> >> Best, >> Ron >> >> On August 1, 2018 at 3:57:51 AM, Christian Grün >> (christian.gruen at gmail.com <mailto:christian.gruen at gmail.com>) wrote: >> >>> I have fixed the example in the doc. >>> Best, Christian >>> >>> >>> On Wed, Aug 1, 2018 at 5:08 AM Ron Katriel <rkatriel at mdsol.com >>> <mailto:rkatriel at mdsol.com>> wrote: >>> > >>> > Hi, >>> > >>> > The following from your website (docs.basex.org/wiki/Full-Text >>> <http://docs.basex.org/wiki/Full-Text>) appears to be syntactically >>> incorrect >>> > >>> > "'Äpfel' will not be found..." contains text "Apfel" diacritics sensitive >>> > >>> > In the BaseX GUI the keyword diacritics is underlined in red and the following error is reported >>> > >>> > Unexpected end of query: 'diacritic sens...'. >>> > >>> > This happens in version 8.6.4 and also the latest (9.0.2). >>> > >>> > Thanks, >>> > Ron >>> > >>> > >>> > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions >>> > >>> > 350 Hudson Street, 7th Floor, New York, NY 10014 >>> > >>> > rkatriel at mdsol.com <mailto:rkatriel at mdsol.com> | direct: +1 201 337 >>> 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800 >>> > >>> >
RK
Ron Katrielin reply to Ron Katriel, 1 Aug 2018, 20:413 Aug 2018 16:39 #
Christian, Adding diacritics sensitive slows execution by a factor of 3. My script (fragment below), which joins two large databases, namely CT.gov <http://clinicaltrials.gov> and DrugBank, takes 2 hours without the diacritics sensitive constraint but 6 hours with it. Given the combinatorics involved, I am wondering if there is a better way to do this in BaseX. Thanks, Ron
  for $drug in db:open('DrugBank')/drugbank/drug
 let $drug_name := $drug/name/text()
 let $drug_synonyms :=
functx:value-union(normalize-space(lower-case($drug/name)),
local:drug-synonyms($drug_name))
 for $synonym_name in $drug_synonyms
 ...
 for $study in
db:open('CTGov')/clinical_study[intervention/intervention_name contains
text { $synonym_name } using case insensitive using diacritics sensitive]
 ...
Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions <http://www.mdsol.com/> 350 Hudson Street, 7th Floor, New York, NY 10014 rkatriel at mdsol.com <tbrophy at mdsol.com> | direct: +1 201 337 3622 <//201 337 3622> | mobile: +1 201 675 5598 <//+1 201 675 5598> | main: +1 212 918 1800 <//+1 212 918 1800>
Quoted text · Ron Katriel (rkatriel at mdsol.com) · 1 line
On August 1, 2018 at 12:41:26 PM, Ron Katriel (rkatriel at mdsol.com) wrote:
Thanks, Christian. Strange, prior to contacting you and on a hunch, I tried adding the missing “using” keyword but still got the syntax error. Anyway, everything is good now! Best, Ron
Quoted text · Christian Grün (christian.gruen at gmail.com) · 2 lines
On August 1, 2018 at 3:57:51 AM, Christian Grün (christian.gruen at gmail.com) wrote:
I have fixed the example in the doc. Best, Christian
Quoted text · Ron Katriel · 5 lines
On Wed, Aug 1, 2018 at 5:08 AM Ron Katriel <rkatriel at mdsol.com> wrote: > > Hi, > > The following from your website (docs.basex.org/wiki/Full-Text) appears
to be syntactically incorrect
Quoted text · Ron Katriel · 4 lines
> > "'Äpfel' will not be found..." contains text "Apfel" diacritics sensitive > > In the BaseX GUI the keyword diacritics is underlined in red and the
following error is reported
Quoted text · Ron Katriel · 14 lines
> > Unexpected end of query: 'diacritic sens...'. > > This happens in version 8.6.4 and also the latest (9.0.2). > > Thanks, > Ron > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > > 350 Hudson Street, 7th Floor, New York, NY 10014 > > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 |
main: +1 212 918 1800
Quoted text · Ron Katriel · 2 lines
> >
CG
Christian Grün3 Aug 2018 22:12 #
Hi Ron, Did you a) create a full-text index for your data and b) ensure that your query is rewritten for index access? Best, Christian
Quoted text · Ron Katriel · 61 lines
On Fri, Aug 3, 2018 at 2:39 PM Ron Katriel <rkatriel at mdsol.com> wrote: > > Christian, > > Adding diacritics sensitive slows execution by a factor of 3. My script (fragment below), which joins two large databases, namely CT.gov and DrugBank, takes 2 hours without the diacritics sensitive constraint but 6 hours with it. Given the combinatorics involved, I am wondering if there is a better way to do this in BaseX. > > Thanks, > Ron > > > for $drug in db:open('DrugBank')/drugbank/drug > let $drug_name := $drug/name/text() > let $drug_synonyms := functx:value-union(normalize-space(lower-case($drug/name)), local:drug-synonyms($drug_name)) > for $synonym_name in $drug_synonyms > ... > for $study in db:open('CTGov')/clinical_study[intervention/intervention_name contains text { $synonym_name } using case insensitive using diacritics sensitive] > ... > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > 350 Hudson Street, 7th Floor, New York, NY 10014 > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800 > > On August 1, 2018 at 12:41:26 PM, Ron Katriel (rkatriel at mdsol.com) wrote: > > Thanks, Christian. Strange, prior to contacting you and on a hunch, I tried adding the missing “using” keyword but still got the syntax error. Anyway, everything is good now! > > Best, > Ron > > On August 1, 2018 at 3:57:51 AM, Christian Grün (christian.gruen at gmail.com) wrote: > > I have fixed the example in the doc. > Best, Christian > > > On Wed, Aug 1, 2018 at 5:08 AM Ron Katriel <rkatriel at mdsol.com> wrote: > > > > Hi, > > > > The following from your website (docs.basex.org/wiki/Full-Text) appears to be syntactically incorrect > > > > "'Äpfel' will not be found..." contains text "Apfel" diacritics sensitive > > > > In the BaseX GUI the keyword diacritics is underlined in red and the following error is reported > > > > Unexpected end of query: 'diacritic sens...'. > > > > This happens in version 8.6.4 and also the latest (9.0.2). > > > > Thanks, > > Ron > > > > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > > > > 350 Hudson Street, 7th Floor, New York, NY 10014 > > > > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800 > > > >
CG
Christian Grünin reply to Ron Katriel, 4 Aug 2018, 01:203 Aug 2018 23:49 #
Our documentation should help you here: http://docs.basex.org/wiki/Indexes
Quoted text · Ron Katriel · 110 lines
Ron Katriel <rkatriel at mdsol.com> schrieb am Fr., 3. Aug. 2018, 23:20: > Hi Christian, > > Yes, I created a full-text index when the databases where loaded (see the > commands below). I also verified that FTINDEX is true for both databases > (in the GUI under Database > Open & Manage). > > How do I ensure that my query is rewritten for index access? > > Thanks, > Ron > > > SET FTINDEX true; SET TOKENINDEX true; CREATE DB CTGov "/Data Sets/ > ct.gov/xml" > SET FTINDEX true; SET TOKENINDEX true; SET STRIPNS true; CREATE DB > DrugBank “/Data Sets/DrugBank/drugbank.xml" > > On August 3, 2018 at 4:12:43 PM, Christian Grün (christian.gruen at gmail.com) > wrote: > > Hi Ron, > > Did you a) create a full-text index for your data and b) ensure that > your query is rewritten for index access? > > Best, > Christian > > > On Fri, Aug 3, 2018 at 2:39 PM Ron Katriel <rkatriel at mdsol.com> wrote: > > > > Christian, > > > > Adding diacritics sensitive slows execution by a factor of 3. My script > (fragment below), which joins two large databases, namely CT.gov and > DrugBank, takes 2 hours without the diacritics sensitive constraint but 6 > hours with it. Given the combinatorics involved, I am wondering if there is > a better way to do this in BaseX. > > > > Thanks, > > Ron > > > > > > for $drug in db:open('DrugBank')/drugbank/drug > > let $drug_name := $drug/name/text() > > let $drug_synonyms := > functx:value-union(normalize-space(lower-case($drug/name)), > local:drug-synonyms($drug_name)) > > for $synonym_name in $drug_synonyms > > ... > > for $study in > db:open('CTGov')/clinical_study[intervention/intervention_name contains > text { $synonym_name } using case insensitive using diacritics sensitive] > > ... > > > > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > > 350 Hudson Street, 7th Floor, New York, NY 10014 > > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 > | main: +1 212 918 1800 > > > > On August 1, 2018 at 12:41:26 PM, Ron Katriel (rkatriel at mdsol.com) > wrote: > > > > Thanks, Christian. Strange, prior to contacting you and on a hunch, I > tried adding the missing “using” keyword but still got the syntax error. > Anyway, everything is good now! > > > > Best, > > Ron > > > > On August 1, 2018 at 3:57:51 AM, Christian Grün ( > christian.gruen at gmail.com) wrote: > > > > I have fixed the example in the doc. > > Best, Christian > > > > > > On Wed, Aug 1, 2018 at 5:08 AM Ron Katriel <rkatriel at mdsol.com> wrote: > > > > > > Hi, > > > > > > The following from your website (docs.basex.org/wiki/Full-Text) > appears to be syntactically incorrect > > > > > > "'Äpfel' will not be found..." contains text "Apfel" diacritics > sensitive > > > > > > In the BaseX GUI the keyword diacritics is underlined in red and the > following error is reported > > > > > > Unexpected end of query: 'diacritic sens...'. > > > > > > This happens in version 8.6.4 and also the latest (9.0.2). > > > > > > Thanks, > > > Ron > > > > > > > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > > > > > > 350 Hudson Street, 7th Floor, New York, NY 10014 > > > > > > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 > 5598 | main: +1 212 918 1800 > > > > > > > >
RK
Ron Katrielin reply to Christian Grün, 3 Aug 2018, 22:124 Aug 2018 01:20 #
Hi Christian, Yes, I created a full-text index when the databases where loaded (see the commands below). I also verified that FTINDEX is true for both databases (in the GUI under Database > Open & Manage). How do I ensure that my query is rewritten for index access? Thanks, Ron SET FTINDEX true; SET TOKENINDEX true; CREATE DB CTGov "/Data Sets/ ct.gov/xml" SET FTINDEX true; SET TOKENINDEX true; SET STRIPNS true; CREATE DB DrugBank “/Data Sets/DrugBank/drugbank.xml"
Quoted text · Christian Grün (christian.gruen at gmail.com) · 2 lines
On August 3, 2018 at 4:12:43 PM, Christian Grün (christian.gruen at gmail.com) wrote:
Hi Ron, Did you a) create a full-text index for your data and b) ensure that your query is rewritten for index access? Best, Christian
Quoted text · Ron Katriel · 5 lines
On Fri, Aug 3, 2018 at 2:39 PM Ron Katriel <rkatriel at mdsol.com> wrote: > > Christian, > > Adding diacritics sensitive slows execution by a factor of 3. My script
(fragment below), which joins two large databases, namely CT.gov and DrugBank, takes 2 hours without the diacritics sensitive constraint but 6 hours with it. Given the combinatorics involved, I am wondering if there is a better way to do this in BaseX.
Quoted text · Christian Grün · 8 lines
> > Thanks, > Ron > > > for $drug in db:open('DrugBank')/drugbank/drug > let $drug_name := $drug/name/text() > let $drug_synonyms :=
functx:value-union(normalize-space(lower-case($drug/name)),
local:drug-synonyms($drug_name))
Quoted text · Christian Grün · 3 lines
> for $synonym_name in $drug_synonyms > ... > for $study in
db:open('CTGov')/clinical_study[intervention/intervention_name contains
text { $synonym_name } using case insensitive using diacritics sensitive]
Quoted text · Christian Grün · 6 lines
> ... > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > 350 Hudson Street, 7th Floor, New York, NY 10014 > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 |
main: +1 212 918 1800
Quoted text · Ron Katriel (rkatriel at mdsol.com) · 4 lines
> > On August 1, 2018 at 12:41:26 PM, Ron Katriel (rkatriel at mdsol.com) wrote: > > Thanks, Christian. Strange, prior to contacting you and on a hunch, I
tried adding the missing “using” keyword but still got the syntax error. Anyway, everything is good now!
Quoted text · Christian Grün · 16 lines
> > Best, > Ron > > On August 1, 2018 at 3:57:51 AM, Christian Grün (christian.gruen at gmail.com) wrote: > > I have fixed the example in the doc. > Best, Christian > > > On Wed, Aug 1, 2018 at 5:08 AM Ron Katriel <rkatriel at mdsol.com> wrote: > > > > Hi, > > > > The following from your website (docs.basex.org/wiki/Full-Text) appears
to be syntactically incorrect
Quoted text · Christian Grün · 2 lines
> > > > "'Äpfel' will not be found..." contains text "Apfel" diacritics
sensitive
Quoted text · Christian Grün · 2 lines
> > > > In the BaseX GUI the keyword diacritics is underlined in red and the
following error is reported
Quoted text · Christian Grün · 14 lines
> > > > Unexpected end of query: 'diacritic sens...'. > > > > This happens in version 8.6.4 and also the latest (9.0.2). > > > > Thanks, > > Ron > > > > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > > > > 350 Hudson Street, 7th Floor, New York, NY 10014 > > > > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598
| main: +1 212 918 1800
Quoted text · Christian Grün · 2 lines
> > > >
RK
Ron Katrielin reply to Imsieke, Gerrit, le-tex, 3 Aug 2018, 15:084 Aug 2018 01:26 #
Hi Gerrit, Thanks for the suggestions. I would like to retain the original diacritics (for output purposes) but only match them when warranted (e.g., match acétazolamide to acétazolamide, but not acétazolamide to acetazolamide). I am looking for a simple solution that does not involve modifying the database or maintaining multiple copies (both for processing simplicity and storage efficiency reasons). Thanks, Ron
Quoted text · le-tex ( gerrit.imsieke at le-tex.de) · 2 lines
On August 3, 2018 at 9:08:19 AM, Imsieke, Gerrit, le-tex ( gerrit.imsieke at le-tex.de) wrote:
Hi Ron, You can add an extra element (or attribute) to the content when importing or modifying it. (Or another document in another database if you like – you can create and later find such an index document by giving it the same db:path as the original document.) In this extra database, document, element and/or attribute, you can recreate the original text, except that you normalize the characters with diacritical marks to a canonical decomposition form and then strip away the diacritical marks like this: replace(normalize-unicode($input, 'NFKD'), '\p{Mn}', '') The full updating statement is beyond my cursory XQuery capabilities – I’d probably do it in XSLT. Also I don’t know how to trigger an event that would cause an update of the auxiliary fields when the underlying data changes. Gerrit
Quoted text · Ron Katriel · 6 lines
On 03.08.2018 14:39, Ron Katriel wrote: > Christian, > > Adding diacritics sensitive slows execution by a factor of 3. My script > (fragment below), which joins two large databases, namely CT.gov > <
https://urldefense.proofpoint.com/v2/url?u=http-3A__clinicaltrials.gov&d=DwIDaQ&c=fi2D4-9xMzmjyjREwHYlAw&r=44jDQvzmnB_-ovfO6Iusj0ItciJrcWMOQQwd2peEBBE&m=Ey4jDDhPLggInP39ySeaE3VfSTCYVNh_9_fJGgZfoMQ&s=koceIUV9xm7YkAEx4zHuVLM00ueSFrJPydvVoqoa_JE&e=> and
Quoted text · Imsieke, Gerrit, le-tex · 54 lines
DrugBank, takes 2 hours without the > diacritics sensitive constraint but 6 hours with it. Given the > combinatorics involved, I am wondering if there is a better way to do > this in BaseX. > > Thanks, > Ron > > > for $drug in db:open('DrugBank')/drugbank/drug > let $drug_name := $drug/name/text() > let $drug_synonyms := > functx:value-union(normalize-space(lower-case($drug/name)), > local:drug-synonyms($drug_name)) > for $synonym_name in $drug_synonyms > ... > for $study in > db:open('CTGov')/clinical_study[intervention/intervention_name contains > text { $synonym_name } using case insensitive using diacritics sensitive] > ... > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > <http://www.mdsol.com/> > 350 Hudson Street, 7th Floor, New York, NY 10014 > rkatriel at mdsol.com <mailto:tbrophy at mdsol.com> | direct: +1 201 337 3622 > <tel://201%20337%203622> | mobile: +1 201 675 5598 > <tel://+1%20201%20675%205598> | main: +1 212 918 1800 > <tel://+1%20212%20918%201800> > > On August 1, 2018 at 12:41:26 PM, Ron Katriel (rkatriel at mdsol.com > <mailto:rkatriel at mdsol.com>) wrote: > >> Thanks, Christian. Strange, prior to contacting you and on a hunch, I >> tried adding the missing “using” keyword but still got the syntax >> error. Anyway, everything is good now! >> >> Best, >> Ron >> >> On August 1, 2018 at 3:57:51 AM, Christian Grün >> (christian.gruen at gmail.com <mailto:christian.gruen at gmail.com>) wrote: >> >>> I have fixed the example in the doc. >>> Best, Christian >>> >>> >>> On Wed, Aug 1, 2018 at 5:08 AM Ron Katriel <rkatriel at mdsol.com >>> <mailto:rkatriel at mdsol.com>> wrote: >>> > >>> > Hi, >>> > >>> > The following from your website (docs.basex.org/wiki/Full-Text >>> <
https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Full-2DText&d=DwIDaQ&c=fi2D4-9xMzmjyjREwHYlAw&r=44jDQvzmnB_-ovfO6Iusj0ItciJrcWMOQQwd2peEBBE&m=Ey4jDDhPLggInP39ySeaE3VfSTCYVNh_9_fJGgZfoMQ&s=SiWAa4ymPcj4HabGlA411Rp03-IG4l2krSrbu2-WJSs&e=>)
Quoted text · Imsieke, Gerrit, le-tex · 4 lines
appears to be syntactically >>> incorrect >>> > >>> > "'Äpfel' will not be found..." contains text "Apfel" diacritics
sensitive
Quoted text · Imsieke, Gerrit, le-tex · 2 lines
>>> > >>> > In the BaseX GUI the keyword diacritics is underlined in red and the
following error is reported
Quoted text · Imsieke, Gerrit, le-tex · 17 lines
>>> > >>> > Unexpected end of query: 'diacritic sens...'. >>> > >>> > This happens in version 8.6.4 and also the latest (9.0.2). >>> > >>> > Thanks, >>> > Ron >>> > >>> > >>> > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions >>> > >>> > 350 Hudson Street, 7th Floor, New York, NY 10014 >>> > >>> > rkatriel at mdsol.com <mailto:rkatriel at mdsol.com> | direct: +1 201 337 >>> 3622 | mobile: +1 201 675 5598 | main: +1 212 918 1800 >>> > >>> >
RK
Ron Katrielin reply to Christian Grün, 3 Aug 2018, 23:494 Aug 2018 02:44 #
Christian, Thanks for sharing that. I assumed all along that this happens automatically. Anyway, I ran my query (for one drug, to save time) and see the following in the Info view - apply text index for "Lenalidomide" I believe the slow execution may be due to a combinatorial issue: the cross product of 280,000 clinical trials and ~10,000 drugs in DrugBank (not counting synonyms). I am considering an algorithmic solution that involves storing the DrugBank information in a hash table (map) and looking it up while iterating through the CT.gov <http://clinicaltrials.gov> trials. Best, Ron
Quoted text · Christian Grün (christian.gruen at gmail.com) · 2 lines
On August 3, 2018 at 5:49:30 PM, Christian Grün (christian.gruen at gmail.com) wrote:
Our documentation should help you here: http://docs.basex.org/wiki/Indexes <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Indexes&d=DwMFaQ&c=fi2D4-9xMzmjyjREwHYlAw&r=44jDQvzmnB_-ovfO6Iusj0ItciJrcWMOQQwd2peEBBE&m=mk1COTV1sAZu82fBqU9P70ZPQXi-d6NrV1-5QYTPHOo&s=Esza6Q3FyaDERIFJTWBAjifLIDVFW3bWKMLS4hbqv_A&e=>
Quoted text · Christian Grün · 113 lines
Ron Katriel <rkatriel at mdsol.com> schrieb am Fr., 3. Aug. 2018, 23:20: > Hi Christian, > > Yes, I created a full-text index when the databases where loaded (see the > commands below). I also verified that FTINDEX is true for both databases > (in the GUI under Database > Open & Manage). > > How do I ensure that my query is rewritten for index access? > > Thanks, > Ron > > > SET FTINDEX true; SET TOKENINDEX true; CREATE DB CTGov "/Data Sets/ > ct.gov/xml > <https://urldefense.proofpoint.com/v2/url?u=http-3A__ct.gov_xml&d=DwMFaQ&c=fi2D4-9xMzmjyjREwHYlAw&r=44jDQvzmnB_-ovfO6Iusj0ItciJrcWMOQQwd2peEBBE&m=mk1COTV1sAZu82fBqU9P70ZPQXi-d6NrV1-5QYTPHOo&s=nDUqSutsQr7QyD8E6-XysRp1qudWO6I05tJaWjkCUI4&e=> > " > SET FTINDEX true; SET TOKENINDEX true; SET STRIPNS true; CREATE DB > DrugBank “/Data Sets/DrugBank/drugbank.xml" > > On August 3, 2018 at 4:12:43 PM, Christian Grün (christian.gruen at gmail.com) > wrote: > > Hi Ron, > > Did you a) create a full-text index for your data and b) ensure that > your query is rewritten for index access? > > Best, > Christian > > > On Fri, Aug 3, 2018 at 2:39 PM Ron Katriel <rkatriel at mdsol.com> wrote: > > > > Christian, > > > > Adding diacritics sensitive slows execution by a factor of 3. My script > (fragment below), which joins two large databases, namely CT.gov and > DrugBank, takes 2 hours without the diacritics sensitive constraint but 6 > hours with it. Given the combinatorics involved, I am wondering if there is > a better way to do this in BaseX. > > > > Thanks, > > Ron > > > > > > for $drug in db:open('DrugBank')/drugbank/drug > > let $drug_name := $drug/name/text() > > let $drug_synonyms := > functx:value-union(normalize-space(lower-case($drug/name)), > local:drug-synonyms($drug_name)) > > for $synonym_name in $drug_synonyms > > ... > > for $study in > db:open('CTGov')/clinical_study[intervention/intervention_name contains > text { $synonym_name } using case insensitive using diacritics sensitive] > > ... > > > > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > > 350 Hudson Street, 7th Floor, New York, NY 10014 > > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 5598 > | main: +1 212 918 1800 > > > > On August 1, 2018 at 12:41:26 PM, Ron Katriel (rkatriel at mdsol.com) > wrote: > > > > Thanks, Christian. Strange, prior to contacting you and on a hunch, I > tried adding the missing “using” keyword but still got the syntax error. > Anyway, everything is good now! > > > > Best, > > Ron > > > > On August 1, 2018 at 3:57:51 AM, Christian Grün ( > christian.gruen at gmail.com) wrote: > > > > I have fixed the example in the doc. > > Best, Christian > > > > > > On Wed, Aug 1, 2018 at 5:08 AM Ron Katriel <rkatriel at mdsol.com> wrote: > > > > > > Hi, > > > > > > The following from your website (docs.basex.org/wiki/Full-Text > <https://urldefense.proofpoint.com/v2/url?u=http-3A__docs.basex.org_wiki_Full-2DText&d=DwMFaQ&c=fi2D4-9xMzmjyjREwHYlAw&r=44jDQvzmnB_-ovfO6Iusj0ItciJrcWMOQQwd2peEBBE&m=mk1COTV1sAZu82fBqU9P70ZPQXi-d6NrV1-5QYTPHOo&s=fzrCGjX9wfPKGZuwd7u4KJ4_AyzK0ZQtU9_PRyCam3U&e=>) > appears to be syntactically incorrect > > > > > > "'Äpfel' will not be found..." contains text "Apfel" diacritics > sensitive > > > > > > In the BaseX GUI the keyword diacritics is underlined in red and the > following error is reported > > > > > > Unexpected end of query: 'diacritic sens...'. > > > > > > This happens in version 8.6.4 and also the latest (9.0.2). > > > > > > Thanks, > > > Ron > > > > > > > > > Ron Katriel, Ph.D. | Principal Data Scientist | Medidata Solutions > > > > > > 350 Hudson Street, 7th Floor, New York, NY 10014 > > > > > > rkatriel at mdsol.com | direct: +1 201 337 3622 | mobile: +1 201 675 > 5598 | main: +1 212 918 1800 > > > > > > > >
CG
Christian Grün4 Aug 2018 12:00 #
Hi Ron,
Quoted text · Ron Katriel · 1 line
> I believe the slow execution may be due to a combinatorial issue: the cross product of 280,000 clinical trials and ~10,000 drugs in DrugBank (not counting synonyms).
Yes, this sounds like a pretty expensive operation. Having maps (XQuery, Java) will be much faster indeed. As Gerrit suggested, and if you will run your query more than once, it would definitely be another interesting option to build an auxiliary, custom "index database" that allows you to do exact searches (this database may still have references to your original data sets). Since version 9 of BaseX, volatile hash maps will be created for looped string comparisons. See the following example:
  let $values1 := (1 to 500000) ! string()
  let $values2 := (500001 to 1000000) ! string()
  return $values1[. = $values2]
Algorithmically, 500'000 * 500'000 string comparisons will need to be performed, resulting in a total of 250 billion operations (and no results). The runtime is much faster as you might expect (and, as far as I can judge, much faster than in any other XQuery processor). Best, Christian
RK
Ron Katriel5 Aug 2018 04:48 #
Hi Christian, Thanks for the advise. The BaseX engine is phenomenal so I realized quickly that the problem was performing a naive cross product. Since this query is run only once a month (to serialize XML to CSV) and applied to new data (DB) each time, a BaseX map will likely be the most straightforward solution (I used the same idea for another project with good results). I will not be able to implement and test this for another couple of weeks but will summarize my findings to the group as soon as possible. Best, Ron
Quoted text · Christian Grün · 27 lines
> On Aug 4, 2018, at 6:00 AM, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi Ron, > >> I believe the slow execution may be due to a combinatorial issue: the cross product of 280,000 clinical trials and ~10,000 drugs in DrugBank (not counting synonyms). > > Yes, this sounds like a pretty expensive operation. Having maps > (XQuery, Java) will be much faster indeed. > > As Gerrit suggested, and if you will run your query more than once, it > would definitely be another interesting option to build an auxiliary, > custom "index database" that allows you to do exact searches (this > database may still have references to your original data sets). Since > version 9 of BaseX, volatile hash maps will be created for looped > string comparisons. See the following example: > > let $values1 := (1 to 500000) ! string() > let $values2 := (500001 to 1000000) ! string() > return $values1[. = $values2] > > Algorithmically, 500'000 * 500'000 string comparisons will need to be > performed, resulting in a total of 250 billion operations (and no > results). The runtime is much faster as you might expect (and, as far > as I can judge, much faster than in any other XQuery processor). > > Best, > Christian
RK
Ron Katriel3 Sep 2018 00:52 #
Hi Christian, As promised here is a summary of my experimentation. I replaced the expensive join with a map lookup and the program finished in 4 minutes vs. 1 hour using a naive loop over the two databases (the original 6 hours reported were due to overly aggressive virus scanning software, which I turned off for this benchmarking). The downside of not using “contains text” inside the double loop (due to its slowness) is that I had to tokenize the CT.gov interventions and remove stopwords prior to looking them up in the DrugBank map. This is a subpar solution as some drugs are missed (looking up all the possible word combinations would be expensive). It would be nice if there was a way to combine the matching flexibility of the “contains text” construct (with its myriad of options) and the efficiency of a map lookup but that may require a finite-state automaton such as the Aho–Corasick algorithm. If you are aware of any existing solutions I would appreciate your sharing them. Thanks, Ron
Quoted text · Ron Katriel (rkatriel at mdsol.com) · 1 line
On August 4, 2018 at 8:47:49 PM, Ron Katriel (rkatriel at mdsol.com) wrote:
Hi Christian, Thanks for the advise. The BaseX engine is phenomenal so I realized quickly that the problem was performing a naive cross product. Since this query is run only once a month (to serialize XML to CSV) and applied to new data (DB) each time, a BaseX map will likely be the most straightforward solution (I used the same idea for another project with good results). I will not be able to implement and test this for another couple of weeks but will summarize my findings to the group as soon as possible. Best, Ron
Quoted text · Christian Grün · 6 lines
> On Aug 4, 2018, at 6:00 AM, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi Ron, > >> I believe the slow execution may be due to a combinatorial issue: the
cross product of 280,000 clinical trials and ~10,000 drugs in DrugBank (not counting synonyms).
Quoted text · Ron Katriel · 22 lines
> > Yes, this sounds like a pretty expensive operation. Having maps > (XQuery, Java) will be much faster indeed. > > As Gerrit suggested, and if you will run your query more than once, it > would definitely be another interesting option to build an auxiliary, > custom "index database" that allows you to do exact searches (this > database may still have references to your original data sets). Since > version 9 of BaseX, volatile hash maps will be created for looped > string comparisons. See the following example: > > let $values1 := (1 to 500000) ! string() > let $values2 := (500001 to 1000000) ! string() > return $values1[. = $values2] > > Algorithmically, 500'000 * 500'000 string comparisons will need to be > performed, resulting in a total of 250 billion operations (and no > results). The runtime is much faster as you might expect (and, as far > as I can judge, much faster than in any other XQuery processor). > > Best, > Christian
GS
Graydon Saunders3 Sep 2018 05:16 #
Maps that reference nodes are pointers, rather than copies. It sounds like you could map every drug name to every "interesting" XML node that contains it using grouping during map creation and then just iterate on the keys to process the nodes.
Quoted text · Ron Katriel · 75 lines
On Sun, Sep 2, 2018 at 4:52 PM Ron Katriel <rkatriel at mdsol.com> wrote: > Hi Christian, > > As promised here is a summary of my experimentation. I replaced the > expensive join with a map lookup and the program finished in 4 minutes vs. > 1 hour using a naive loop over the two databases (the original 6 hours > reported were due to overly aggressive virus scanning software, which I > turned off for this benchmarking). > > The downside of not using “contains text” inside the double loop (due to > its slowness) is that I had to tokenize the CT.gov interventions and > remove stopwords prior to looking them up in the DrugBank map. This is a > subpar solution as some drugs are missed (looking up all the possible word > combinations would be expensive). > > It would be nice if there was a way to combine the matching flexibility of > the “contains text” construct (with its myriad of options) and the > efficiency of a map lookup but that may require a finite-state automaton > such as the Aho–Corasick algorithm. If you are aware of any existing > solutions I would appreciate your sharing them. > > Thanks, > Ron > > On August 4, 2018 at 8:47:49 PM, Ron Katriel (rkatriel at mdsol.com) wrote: > > Hi Christian, > > Thanks for the advise. The BaseX engine is phenomenal so I realized > quickly that the problem was performing a naive cross product. > > Since this query is run only once a month (to serialize XML to CSV) and > applied to new data (DB) each time, a BaseX map will likely be the most > straightforward solution (I used the same idea for another project with > good results). > > I will not be able to implement and test this for another couple of weeks > but will summarize my findings to the group as soon as possible. > > Best, > Ron > > > > On Aug 4, 2018, at 6:00 AM, Christian Grün <christian.gruen at gmail.com> > wrote: > > > > Hi Ron, > > > >> I believe the slow execution may be due to a combinatorial issue: the > cross product of 280,000 clinical trials and ~10,000 drugs in DrugBank (not > counting synonyms). > > > > Yes, this sounds like a pretty expensive operation. Having maps > > (XQuery, Java) will be much faster indeed. > > > > As Gerrit suggested, and if you will run your query more than once, it > > would definitely be another interesting option to build an auxiliary, > > custom "index database" that allows you to do exact searches (this > > database may still have references to your original data sets). Since > > version 9 of BaseX, volatile hash maps will be created for looped > > string comparisons. See the following example: > > > > let $values1 := (1 to 500000) ! string() > > let $values2 := (500001 to 1000000) ! string() > > return $values1[. = $values2] > > > > Algorithmically, 500'000 * 500'000 string comparisons will need to be > > performed, resulting in a total of 250 billion operations (and no > > results). The runtime is much faster as you might expect (and, as far > > as I can judge, much faster than in any other XQuery processor). > > > > Best, > > Christian > >
RK
Ron Katriel3 Sep 2018 05:25 #
Hi Graydon, Thanks for the suggestion. Could you provide sample code to help with this? If needed I can share the relevant BaseX snippet. Best, Ron
Quoted text · Graydon Saunders · 57 lines
> On Sep 2, 2018, at 9:16 PM, Graydon Saunders <graydonish at gmail.com> wrote: > > Maps that reference nodes are pointers, rather than copies. It sounds like you could map every drug name to every "interesting" XML node that contains it using grouping during map creation and then just iterate on the keys to process the nodes. > >> On Sun, Sep 2, 2018 at 4:52 PM Ron Katriel <rkatriel at mdsol.com> wrote: >> Hi Christian, >> >> As promised here is a summary of my experimentation. I replaced the expensive join with a map lookup and the program finished in 4 minutes vs. 1 hour using a naive loop over the two databases (the original 6 hours reported were due to overly aggressive virus scanning software, which I turned off for this benchmarking). >> >> The downside of not using “contains text” inside the double loop (due to its slowness) is that I had to tokenize the CT.gov interventions and remove stopwords prior to looking them up in the DrugBank map. This is a subpar solution as some drugs are missed (looking up all the possible word combinations would be expensive). >> >> It would be nice if there was a way to combine the matching flexibility of the “contains text” construct (with its myriad of options) and the efficiency of a map lookup but that may require a finite-state automaton such as the Aho–Corasick algorithm. If you are aware of any existing solutions I would appreciate your sharing them. >> >> Thanks, >> Ron >> >>> On August 4, 2018 at 8:47:49 PM, Ron Katriel (rkatriel at mdsol.com) wrote: >>> >>> Hi Christian, >>> >>> Thanks for the advise. The BaseX engine is phenomenal so I realized quickly that the problem was performing a naive cross product. >>> >>> Since this query is run only once a month (to serialize XML to CSV) and applied to new data (DB) each time, a BaseX map will likely be the most straightforward solution (I used the same idea for another project with good results). >>> >>> I will not be able to implement and test this for another couple of weeks but will summarize my findings to the group as soon as possible. >>> >>> Best, >>> Ron >>> >>> >>> > On Aug 4, 2018, at 6:00 AM, Christian Grün <christian.gruen at gmail.com> wrote: >>> > >>> > Hi Ron, >>> > >>> >> I believe the slow execution may be due to a combinatorial issue: the cross product of 280,000 clinical trials and ~10,000 drugs in DrugBank (not counting synonyms). >>> > >>> > Yes, this sounds like a pretty expensive operation. Having maps >>> > (XQuery, Java) will be much faster indeed. >>> > >>> > As Gerrit suggested, and if you will run your query more than once, it >>> > would definitely be another interesting option to build an auxiliary, >>> > custom "index database" that allows you to do exact searches (this >>> > database may still have references to your original data sets). Since >>> > version 9 of BaseX, volatile hash maps will be created for looped >>> > string comparisons. See the following example: >>> > >>> > let $values1 := (1 to 500000) ! string() >>> > let $values2 := (500001 to 1000000) ! string() >>> > return $values1[. = $values2] >>> > >>> > Algorithmically, 500'000 * 500'000 string comparisons will need to be >>> > performed, resulting in a total of 250 billion operations (and no >>> > results). The runtime is much faster as you might expect (and, as far >>> > as I can judge, much faster than in any other XQuery processor). >>> > >>> > Best, >>> > Christian
GS
Graydon Saunders3 Sep 2018 14:41 #
Let's suppose you've got a map like: (and that by just typing this into the email I haven't left in any really horrible typos!)
let $drugInfo as map(xs:string,element()) := map:merge(
for $element in collection('newDrugInfo')/descendant::infoElement
let $name as xs:string := (: whatever you do to extract the official drug
name from the update data :)
return map:entry($name,$element))
then in the other docbase you've got:
let $updatePlaces as map(xs:string,element()+) := map:merge(
for $place in collection('updating-this-one')/descendant::couldBeInteresting
let $drugName as xs:string := (:whatever you're doing now to match the drug
name; there's an assumption you expect to find only one :)
where exists($drugName) (: because you might not have one! :)
group by $drugName (:baseX will magically make $place be a sequence of all
the $place values with this drug name, effectively a sequence of pointers
to those element nodes)
return map:entry($drugName,$place)
)
So now you can:
for $drug in map:keys($drugInfo) (: we're iterating through the official
list :)
let $needsUpdate as element()+ := $updatePlaces($drug)
for $place in $needsUpdate (: iterate through our sequence of pointers :)
(: do whatever you're doing to insert the information in $drugInfo($drug) :)
It looks like the same old n-squared inner-loop/outer-loop update process, but I have found that it doesn't perform like that. I am almost never updating the docbase so whatever magic is involved may go away when you do that, but I've found this "map both sides" pattern to be very useful when merging data. -- Graydon
Quoted text · Ron Katriel · 93 lines
On Sun, Sep 2, 2018 at 9:25 PM Ron Katriel <rkatriel at mdsol.com> wrote: > Hi Graydon, > > Thanks for the suggestion. Could you provide sample code to help with > this? If needed I can share the relevant BaseX snippet. > > Best, > Ron > > On Sep 2, 2018, at 9:16 PM, Graydon Saunders <graydonish at gmail.com> wrote: > > Maps that reference nodes are pointers, rather than copies. It sounds > like you could map every drug name to every "interesting" XML node that > contains it using grouping during map creation and then just iterate on the > keys to process the nodes. > > On Sun, Sep 2, 2018 at 4:52 PM Ron Katriel <rkatriel at mdsol.com> wrote: > >> Hi Christian, >> >> As promised here is a summary of my experimentation. I replaced the >> expensive join with a map lookup and the program finished in 4 minutes vs. >> 1 hour using a naive loop over the two databases (the original 6 hours >> reported were due to overly aggressive virus scanning software, which I >> turned off for this benchmarking). >> >> The downside of not using “contains text” inside the double loop (due to >> its slowness) is that I had to tokenize the CT.gov >> <https://urldefense.proofpoint.com/v2/url?u=http-3A__CT.gov&d=DwMFaQ&c=fi2D4-9xMzmjyjREwHYlAw&r=44jDQvzmnB_-ovfO6Iusj0ItciJrcWMOQQwd2peEBBE&m=BBeNdMyieZ0Pe47vEYrOWalNS4uDt0n0tRpqmay-0Ug&s=ThHquw8-y4wRz3ejr0mTEm-ImDJHZ_DKVjr8_laQAps&e=> >> interventions and remove stopwords prior to looking them up in the DrugBank >> map. This is a subpar solution as some drugs are missed (looking up all the >> possible word combinations would be expensive). >> >> It would be nice if there was a way to combine the matching flexibility >> of the “contains text” construct (with its myriad of options) and the >> efficiency of a map lookup but that may require a finite-state automaton >> such as the Aho–Corasick algorithm. If you are aware of any existing >> solutions I would appreciate your sharing them. >> >> Thanks, >> Ron >> >> On August 4, 2018 at 8:47:49 PM, Ron Katriel (rkatriel at mdsol.com) wrote: >> >> Hi Christian, >> >> Thanks for the advise. The BaseX engine is phenomenal so I realized >> quickly that the problem was performing a naive cross product. >> >> Since this query is run only once a month (to serialize XML to CSV) and >> applied to new data (DB) each time, a BaseX map will likely be the most >> straightforward solution (I used the same idea for another project with >> good results). >> >> I will not be able to implement and test this for another couple of weeks >> but will summarize my findings to the group as soon as possible. >> >> Best, >> Ron >> >> >> > On Aug 4, 2018, at 6:00 AM, Christian Grün <christian.gruen at gmail.com> >> wrote: >> > >> > Hi Ron, >> > >> >> I believe the slow execution may be due to a combinatorial issue: the >> cross product of 280,000 clinical trials and ~10,000 drugs in DrugBank (not >> counting synonyms). >> > >> > Yes, this sounds like a pretty expensive operation. Having maps >> > (XQuery, Java) will be much faster indeed. >> > >> > As Gerrit suggested, and if you will run your query more than once, it >> > would definitely be another interesting option to build an auxiliary, >> > custom "index database" that allows you to do exact searches (this >> > database may still have references to your original data sets). Since >> > version 9 of BaseX, volatile hash maps will be created for looped >> > string comparisons. See the following example: >> > >> > let $values1 := (1 to 500000) ! string() >> > let $values2 := (500001 to 1000000) ! string() >> > return $values1[. = $values2] >> > >> > Algorithmically, 500'000 * 500'000 string comparisons will need to be >> > performed, resulting in a total of 250 billion operations (and no >> > results). The runtime is much faster as you might expect (and, as far >> > as I can judge, much faster than in any other XQuery processor). >> > >> > Best, >> > Christian >> >>

Written for BaseX 9.0. BaseX may have changed since: please check the current documentation.

« Wasn't there a function, that would walk a website?Failure with comment in editor when commenting out a regex »
⚡Generated with XQuery