Full text and stopwords

Wednesday, 12 February 2020 · 3 messages · last reply 13 Feb 2020
BE
Ben Engbers12 Feb 2020 10:21 #
Hi Christian, According to the docs, a stopword list can be used to decrease the size of the full text index. I had no problems when using this list while creating a database. Is it also possible to use this list for other purposes? 1 According to XQueryX 3.1.pdf it is possible to use a sequence of stopwords in a query: /books/book[@number="1"]//p contains text "propagating of errors" using stop words ("a", "the", "of"). How can I use this list in BaseX while building querys? 2 Is it possible to add words to the list, after that is has been loaded? Suppose that it shows that my text contains a lot of names that I want to exclude. How can I add those names to the stopwords list? 3 If I want to create a Wordcloud, I want to use all the words that remain after tokenization and removing all the words from the stopwords list. (I found this item 'https://en.wikibooks.org/wiki/XQuery/Tag_Cloud'. It might be a good starting point for creating a wordcloud) Cheers, Ben
BE
Ben Engbers12 Feb 2020 10:34 #
Quoted text · Ben Engbers · 3 lines
Op 12-02-2020 om 10:21 schreef Ben Engbers: > Hi Christian, >
Would it be a good approach to create a separate database for stop words and sentiments?
Quoted text · Ben Engbers · 4 lines
> > Cheers, > Ben >
CG
Christian Grünin reply to Ben Engbers, 12 Feb 2020, 10:2113 Feb 2020 18:10 #
Hi Ben,
Quoted text · Ben Engbers · 5 lines
> According to the docs, a stopword list can be used to decrease the size > of the full text index. I had no problems when using this list while > creating a database. > > Is it also possible to use this list for other purposes?
Yes. As it’s a simple word list, you can use do with it whatever you want.
Quoted text · Ben Engbers · 6 lines
> According to XQueryX 3.1.pdf it is possible to use a sequence of > stopwords in a query: > /books/book[@number="1"]//p contains text "propagating of errors" > using stop words ("a", "the", "of"). > > How can I use this list in BaseX while building querys?
The grammar requires the list to be a sequence of string literals. In practice, it’ll be always be a better move to supply a URI.
Quoted text · Ben Engbers · 3 lines
> Is it possible to add words to the list, after that is has been loaded? > Suppose that it shows that my text contains a lot of names that I want > to exclude. How can I add those names to the stopwords list?
You can add your words to the text file and recreate the index based on that file.
Quoted text · Ben Engbers · 1 line
> Would it be a good approach to create a separate database for stop words
and sentiments? It depends on your requirements. TIMTOWTDI ;) Christian

⚡Generated with XQuery