Main Page » Mailing List » June 2021

Error with full-text and fuzzy query

Friday, 4 June 2021 · 7 messages · last reply 7 Jun 2021
TT
Tim Thompson4 Jun 2021 17:53 #
Hello, I have a full-text index defined on a database with 21,936,670 entries. I'm hitting a stack overflow error when I try to query the following (normalized) string with a fuzzy query: "sajawandi, siraj al-din muhammad ibn muhammad, active 12th century. faraʼid al-sirajiyah" The index entries look like this:
<d a="http://id.loc.gov/authorities/names/n2008037847">
  <n>Sajāwandī, Sirāj al-Dīn Muḥammad ibn Muḥammad, active 12th century.
Farāʼiḍ al-Sirājīyah</n>
  <d>12th cent.</d>
</d>
If I create a new DB ("ftindex-test") with only that one entry and try the following lookup, it works: ft:search( "ftindex-test", "sajawandi, siraj al-din muhammad ibn muhammad, active 12th century. faraʼid al-sirajiyah" , map {"mode": "phrase", "content": "entire", "fuzzy": true()} )/.. However, if I try the same query against the full index (with ~22 million entries), it fails with a stack overflow error: [qtp1546693040-47] WARN org.eclipse.jetty.server.HttpChannel - /dba/query-eval java.lang.StackOverflowError at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:73) at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74) ... If I remove the "fuzzy" parameter, the query does work against the full index. Is this a bug, a known limitation here, or something that I'm missing? Thanks in advance, Tim
-- Tim A. Thompson Metadata Librarian Yale University Library
CG
Christian Grün4 Jun 2021 19:30 #
Hi Tim,
Quoted text · Tim Thompson · 4 lines
> [qtp1546693040-47] WARN org.eclipse.jetty.server.HttpChannel - /dba/query-eval > java.lang.StackOverflowError > at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:73) > at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74)
Could you share some more lines of the stack trace with us? Thanks in advance Christian
TT
Tim Thompson5 Jun 2021 01:03 #
Thanks, Christian. The rest of the stack trace was just many lines of "at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74)" until the end: [qtp1546693040-47] WARN org.eclipse.jetty.server.handler.ErrorHandler - Error page too large: 500 java.lang.StackOverflowError Request(POST // 10.5.157.229:10214/dba/query-eval)@7dcadc62 [qtp1546693040-47] INFO org.eclipse.jetty.server.handler.ErrorHandler - Disabling showsStacks for ErrorPageErrorHandler@53032c30{STARTED} Best regards, Tim
-- Tim A. Thompson Metadata Librarian Yale University Library
Quoted text · Christian Grün · 15 lines
On Fri, Jun 4, 2021 at 1:30 PM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Tim, > > > [qtp1546693040-47] WARN org.eclipse.jetty.server.HttpChannel - > /dba/query-eval > > java.lang.StackOverflowError > > at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:73) > > at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74) > > Could you share some more lines of the stack trace with us? > > Thanks in advance > Christian >
CG
Christian Grün7 Jun 2021 10:28 #
Hi Tim, I’m still trying to get this reproduced. The error is caused by excessive recursive functions call. Out of interest, could you try to increase the size of the Java Stack Trace and see if the error persists? This can be achieved by assigning a large value to the JVM via the Xss flag [1]. You could e.g. add -Xss64m to the BASEX_JVM variable in the BaseX start scripts [2]. Internal notes… If yes, we might be confronted with an infinite loop. If no, we should try to rewrite a recursive index lookup to an iterative one. If your data is not confidential, feel free to provide me with a download link. Cheers, Christian [1] https://stackoverflow.com/questions/3700459/how-to-increase-the-java-stack-size [2] https://docs.basex.org/wiki/Start_Scripts
Quoted text · Tim Thompson · 29 lines
On Fri, Jun 4, 2021 at 11:03 PM Tim Thompson <timathom at gmail.com> wrote: > > Thanks, Christian. The rest of the stack trace was just many lines of "at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74)" until the end: > > [qtp1546693040-47] WARN org.eclipse.jetty.server.handler.ErrorHandler - Error page too large: 500 java.lang.StackOverflowError Request(POST //10.5.157.229:10214/dba/query-eval)@7dcadc62 > [qtp1546693040-47] INFO org.eclipse.jetty.server.handler.ErrorHandler - Disabling showsStacks for ErrorPageErrorHandler@53032c30{STARTED} > > Best regards, > Tim > > > -- > Tim A. Thompson > Metadata Librarian > Yale University Library > > On Fri, Jun 4, 2021 at 1:30 PM Christian Grün <christian.gruen at gmail.com> wrote: >> >> Hi Tim, >> >> > [qtp1546693040-47] WARN org.eclipse.jetty.server.HttpChannel - /dba/query-eval >> > java.lang.StackOverflowError >> > at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:73) >> > at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74) >> >> Could you share some more lines of the stack trace with us? >> >> Thanks in advance >> Christian
CG
Christian Grün7 Jun 2021 12:19 #
I think I managed to build up a bug scenario that should resemble yours [1]: If thousands of similar terms are found, the results cannot be recursively joined anymore. [1] https://github.com/BaseXdb/basex/issues/2014
Quoted text · Christian Grün · 56 lines
On Mon, Jun 7, 2021 at 10:28 AM Christian Grün <christian.gruen at gmail.com> wrote: > > Hi Tim, > > I’m still trying to get this reproduced. > > The error is caused by excessive recursive functions call. Out of > interest, could you try to increase the size of the Java Stack Trace > and see if the error persists? This can be achieved by assigning a > large value to the JVM via the Xss flag [1]. You could e.g. add > -Xss64m to the BASEX_JVM variable in the BaseX start scripts [2]. > > Internal notes… If yes, we might be confronted with an infinite loop. > If no, we should try to rewrite a recursive index lookup to an > iterative one. > > If your data is not confidential, feel free to provide me with a download link. > > Cheers, > Christian > > [1] https://stackoverflow.com/questions/3700459/how-to-increase-the-java-stack-size > [2] https://docs.basex.org/wiki/Start_Scripts > > > > On Fri, Jun 4, 2021 at 11:03 PM Tim Thompson <timathom at gmail.com> wrote: > > > > Thanks, Christian. The rest of the stack trace was just many lines of "at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74)" until the end: > > > > [qtp1546693040-47] WARN org.eclipse.jetty.server.handler.ErrorHandler - Error page too large: 500 java.lang.StackOverflowError Request(POST //10.5.157.229:10214/dba/query-eval)@7dcadc62 > > [qtp1546693040-47] INFO org.eclipse.jetty.server.handler.ErrorHandler - Disabling showsStacks for ErrorPageErrorHandler@53032c30{STARTED} > > > > Best regards, > > Tim > > > > > > -- > > Tim A. Thompson > > Metadata Librarian > > Yale University Library > > > > On Fri, Jun 4, 2021 at 1:30 PM Christian Grün <christian.gruen at gmail.com> wrote: > >> > >> Hi Tim, > >> > >> > [qtp1546693040-47] WARN org.eclipse.jetty.server.HttpChannel - /dba/query-eval > >> > java.lang.StackOverflowError > >> > at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:73) > >> > at org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74) > >> > >> Could you share some more lines of the stack trace with us? > >> > >> Thanks in advance > >> Christian
CG
Christian Grün7 Jun 2021 18:58 #
cc to the list… Christian Grün <christian.gruen at gmail.com> schrieb am Mo., 7. Juni 2021, 13:50:
Quoted text · Christian Grün · 118 lines
> Thanks, Tim, that was helpful. It turned out that the index contains > an unusually high number of similar tokens for one search terms in > your query: > > faraʼid : 13709 similar terms > muhammad : 131 similar terms > siraj : 50 similar terms > ... > > A new snapshot with a bug fix is available, in which similar terms are > collected non-recursively [1]. I’d have wished that your search was > evaluated faster (it takes around 6 seconds on machine) but at least > it succeeds. I’ll have some more thoughts. > > Cheers, > Christian > > [1] https://files.basex.org/releases/latest/ > > > > On Mon, Jun 7, 2021 at 1:17 PM Tim Thompson <timathom at gmail.com> wrote: > > > > Thank you, Christian. If it's still helpful, here is a link to download > the document I am using as my full-text index. > > > > All best, > > Tim > > > > > > -- > > Tim A. Thompson > > Metadata Librarian > > Yale University Library > > > > On Mon, Jun 7, 2021 at 6:19 AM Christian Grün <christian.gruen at gmail.com> > wrote: > >> > >> I think I managed to build up a bug scenario that should resemble > >> yours [1]: If thousands of similar terms are found, the results cannot > >> be recursively joined anymore. > >> > >> [1] https://github.com/BaseXdb/basex/issues/2014 > >> > >> > >> > >> On Mon, Jun 7, 2021 at 10:28 AM Christian Grün > >> <christian.gruen at gmail.com> wrote: > >> > > >> > Hi Tim, > >> > > >> > I’m still trying to get this reproduced. > >> > > >> > The error is caused by excessive recursive functions call. Out of > >> > interest, could you try to increase the size of the Java Stack Trace > >> > and see if the error persists? This can be achieved by assigning a > >> > large value to the JVM via the Xss flag [1]. You could e.g. add > >> > -Xss64m to the BASEX_JVM variable in the BaseX start scripts [2]. > >> > > >> > Internal notes… If yes, we might be confronted with an infinite loop. > >> > If no, we should try to rewrite a recursive index lookup to an > >> > iterative one. > >> > > >> > If your data is not confidential, feel free to provide me with a > download link. > >> > > >> > Cheers, > >> > Christian > >> > > >> > [1] > https://stackoverflow.com/questions/3700459/how-to-increase-the-java-stack-size > >> > [2] https://docs.basex.org/wiki/Start_Scripts > >> > > >> > > >> > > >> > On Fri, Jun 4, 2021 at 11:03 PM Tim Thompson <timathom at gmail.com> > wrote: > >> > > > >> > > Thanks, Christian. The rest of the stack trace was just many lines > of "at > org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74)" until > the end: > >> > > > >> > > [qtp1546693040-47] WARN > org.eclipse.jetty.server.handler.ErrorHandler - Error page too large: 500 > java.lang.StackOverflowError Request(POST // > 10.5.157.229:10214/dba/query-eval)@7dcadc62 > >> > > [qtp1546693040-47] INFO > org.eclipse.jetty.server.handler.ErrorHandler - Disabling showsStacks for > ErrorPageErrorHandler@53032c30{STARTED} > >> > > > >> > > Best regards, > >> > > Tim > >> > > > >> > > > >> > > -- > >> > > Tim A. Thompson > >> > > Metadata Librarian > >> > > Yale University Library > >> > > > >> > > On Fri, Jun 4, 2021 at 1:30 PM Christian Grün < > christian.gruen at gmail.com> wrote: > >> > >> > >> > >> Hi Tim, > >> > >> > >> > >> > [qtp1546693040-47] WARN org.eclipse.jetty.server.HttpChannel - > /dba/query-eval > >> > >> > java.lang.StackOverflowError > >> > >> > at > org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:73) > >> > >> > at > org.basex.index.query.FTIndexIterator$2.pos(FTIndexIterator.java:74) > >> > >> > >> > >> Could you share some more lines of the stack trace with us? > >> > >> > >> > >> Thanks in advance > >> > >> Christian >
CG
Christian Grün7 Jun 2021 18:58 #
cc to the list (2)… Christian Grün <christian.gruen at gmail.com> schrieb am Mo., 7. Juni 2021, 15:47:
Quoted text · Christian Grün · 33 lines
> …and another snapshot for you: modifier letters [1] will now be > regarded as diacritical characters, and queries such as… > > tokenize('faraʼid faraid Birkan')[. contains text 'faraʼid' using fuzzy] > > …will only regard the first two terms as similar. As a consequence, > fuzzy queries for terms with modifiers should get a lot faster. > > You’ll need to recreate your full-text index to take full advantage of the > fix. > > [1] https://de.wikipedia.org/wiki/Unicodeblock_Spacing_Modifier_Letters > > > > On Mon, Jun 7, 2021 at 2:00 PM Christian Grün <christian.gruen at gmail.com> > wrote: > > > > > Out of curiosity, is there a way to access those individual term > similarity statistics via XQuery? > > > > I assume there is no straightforward way, but you can check the first > > results of a full-text query: > > > > ft:search('your-db', 'faraʼid', map { "fuzzy": true() })[position() < > 10] > > > > And I noticed that due to the modifier letter (ʼ), many non-similar > > results are returned as well. I’ve raised another issue to track this > > down [1]. > > > > [1] https://github.com/BaseXdb/basex/issues/2015 >

Written for BaseX 9.5. BaseX may have changed since: please check the current documentation.

« insert MS SQL server bigint from BaseX via sql:execute-preparedspeaking effectively to the optimizer »
⚡Generated with XQuery