Full text and stopwords
Hi Christian,
According to the docs, a stopword list can be used to decrease the size of the full text index. I had no problems when using this list while creating a database.
Is it also possible to use this list for other purposes?
1 According to XQueryX 3.1.pdf it is possible to use a sequence of stopwords in a query: /books/book[@number="1"]//p contains text "propagating of errors" using stop words ("a", "the", "of").
How can I use this list in BaseX while building querys?
2 Is it possible to add words to the list, after that is has been loaded? Suppose that it shows that my text contains a lot of names that I want to exclude. How can I add those names to the stopwords list?
3 If I want to create a Wordcloud, I want to use all the words that remain after tokenization and removing all the words from the stopwords list.
(I found this item 'https://en.wikibooks.org/wiki/XQuery/Tag_Cloud'. It might be a good starting point for creating a wordcloud)
Cheers, Ben
Quoted text · Ben Engbers · 3 lines
Op 12-02-2020 om 10:21 schreef Ben Engbers: > Hi Christian, >
Would it be a good approach to create a separate database for stop words and sentiments?
Quoted text · Ben Engbers · 4 lines
> > Cheers, > Ben >
Hi Ben,
Quoted text · Ben Engbers · 5 lines
> According to the docs, a stopword list can be used to decrease the size > of the full text index. I had no problems when using this list while > creating a database. > > Is it also possible to use this list for other purposes?
Yes. As it’s a simple word list, you can use do with it whatever you want.
Quoted text · Ben Engbers · 6 lines
> According to XQueryX 3.1.pdf it is possible to use a sequence of > stopwords in a query: > /books/book[@number="1"]//p contains text "propagating of errors" > using stop words ("a", "the", "of"). > > How can I use this list in BaseX while building querys?
The grammar requires the list to be a sequence of string literals. In practice, it’ll be always be a better move to supply a URI.
Quoted text · Ben Engbers · 3 lines
> Is it possible to add words to the list, after that is has been loaded? > Suppose that it shows that my text contains a lot of names that I want > to exclude. How can I add those names to the stopwords list?
You can add your words to the text file and recreate the index based on that file.
Quoted text · Ben Engbers · 1 line
> Would it be a good approach to create a separate database for stop words
and sentiments?
It depends on your requirements. TIMTOWTDI ;)
Christian
Written for BaseX 9.3. BaseX may have changed since: please check the current documentation.