new to basex

Friday, 30 November 2012 · 5 messages
BB
Bill Beck30 Nov 2012 14:14 #
Hi all, Here that the Massachusetts Historical Society we're exploring using BaseX for some new transcriptions of newspapers from around the American Revolution, as collected by a fellow named Harbottle Dorr. I'm having trouble figuring out how exactly to enable fulltext XQuery searches. We have it working, but quite a few keywords are not being indexed. From the command line tool, we do:
Quoted text · 2 lines
>create database dorr_index >create index fulltext
Then we have a PHP tool (using the class found on the BaseX site) that calls $session->execute to open the collection and $session->add to place the files. But when I then run
Quoted text · 1 line
>info db
from the command line tool, it shows: Database Properties Name: dorr_index Size: 1390 KB Nodes: 32411 Resources: 24 Timestamp: 30.11.2012 08:00:11 Resource Properties Timestamp: 30.11.2012 08:03:21 Encoding: UTF-8 Whitespace Chopping: ON Indexes Up-to-date: false Path Summary: ON Text Index: OFF Attribute Index: OFF Full-Text Index: OFF if I then run the command
Quoted text · 1 line
>create index fulltext
again, info db shows full-text index on, but up-to-date is still false, and our search results still have many missing hits. our XQuery is:
for $i in collection(PATH)//item
let $s := $i/mhs:sort
where $i contains text ("QUERY" using stemming)
return ...
Are there any steps missing? Also, any tutorial out there to follow? Thanks very much in advance! -Bill B
-- Bill Beck, Web Development Specialist Massachusetts Historical Society 1154 Boylston Street, Boston, MA 02215 Tel: 617-646-0505, Fax: 617-859-0074 www.masshist.org - America's Oldest Historical Society - Founded 1791 In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/.
CG
Christian Grün30 Nov 2012 14:50 #
Hi Bill,
Quoted text · Bill Beck · 1 line
> I'm having trouble figuring out how exactly to enable fulltext XQuery searches. We have it working, but quite a few keywords are not being indexed. [...]
as you already indicated, the fulltext index gets lost when performing updates. If you set FTNDEX to TRUE before creating your database, your fulltext index will always be recreated when calling OPTIMIZE: set ftindex true create db dorr_index add ... optimize
Quoted text · Bill Beck · 4 lines
> for $i in collection(PATH)//item > let $s := $i/mhs:sort > where $i contains text ("QUERY" using stemming) > return ...
I know too less about your xml structure, so it's difficult to tell what/if sth. goes wrong. Does the behavior change when following the steps suggested above? Cheers, Christian
Quoted text · Bill Beck · 23 lines
> for $i in collection(PATH)//item > let $s := $i/mhs:sort > where $i contains text ("QUERY" using stemming) > return ... > > Are there any steps missing? Also, any tutorial out there to follow? > > Thanks very much in advance! > > -Bill B > -- > Bill Beck, Web Development Specialist > Massachusetts Historical Society > 1154 Boylston Street, Boston, MA 02215 > Tel: 617-646-0505, Fax: 617-859-0074 > www.masshist.org - America's Oldest Historical Society - Founded 1791 > > > In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/. > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
J
Johannes.Lichtenbergerin reply to Bill Beck, 30 Nov 2012, 14:1430 Nov 2012 14:53 #
Quoted text · Bill Beck · 4 lines
On 11/30/2012 02:14 PM, Bill Beck wrote: > [Activating Fulltext index] > > Are there any steps missing? Also, any tutorial out there to follow?
Usually it's up to the UPDINDEX[1] option (for text/attribute values), but currently the Fulltext index can not be updated efficiently (at least I think so). It is planned[2] however to rewrite the index structure. However, I'm not sure about the real problem, as "QUERY" shouldn't be a stopword which is not indexed. But shouldn't it be "where $i/text() contains text 'QUERY' using stemming"? At least $i is bound to an item I guess, and you have to use text values (or probably attribute values), but not an element. [1] http://docs.basex.org/wiki/Options#UPDINDEX [2] https://github.com/BaseXdb/basex/issues/169
J
Johannes.Lichtenberger30 Nov 2012 14:56 #
Quoted text · Bill Beck · 16 lines
On 11/30/2012 02:53 PM, Johannes.Lichtenberger wrote: > On 11/30/2012 02:14 PM, Bill Beck wrote: >> [Activating Fulltext index] > > >> Are there any steps missing? Also, any tutorial out there to follow? > > Usually it's up to the UPDINDEX[1] option (for text/attribute values), > but currently the Fulltext index can not be updated efficiently (at > least I think so). It is planned[2] however to rewrite the index structure. > > However, I'm not sure about the real problem, as "QUERY" shouldn't be a > stopword which is not indexed. > > But shouldn't it be "where $i/text() contains text 'QUERY' using > stemming"? At least $i is bound to an item I guess, and you have to use > text values (or probably attribute values), but not an element.
Ok, the last sentence is wrong ;-)
CG
Christian Grün30 Nov 2012 15:14 #
Hi Bill (cc to the list), the current fulltext global indexing options (specified via SET) are applied and stored in the target database whenever an index is created via CREATE INDEX. If OPTIMIZE is used, these database options will be used for recreating the index. Both the database options and index structures should not get lost if you restart your server. The following (non-server) example may help to demonstrate that fulltext options such as stemming will be stored along with the database: basex -c "set stemming on;set ftindex on; create db db <a>A</a>; optimize; close; set stemming off; open db;info index fulltext" Btw, which BaseX version do you use? It would be interesting to see what happens with our latest stable snapshot [1]; the final version is long overdue, but it will finally be released in December. Christian [1] http://files.basex.org/releases/latest/
Quoted text · Johannes.Lichtenberger · 82 lines
> HI Christian, > > Thanks for the fast reply. > > I tried your steps, with no change. I am curious about a couple of things I notice: > > Using these command I enabled fulltext with stemming: > >>set stemming true >>create index fulltext > > And info index shows these to be true. But if I stop the server and restart, these are all off, and the fulltext index is gone (the collection and it's files are still there). And yet, the same flawed search still works. > > So my questions are: > Do we need to reset globals and reindex each time the server is restart? It our implimentation it seems so (running standalone mode). > > Thanks. > > -bb > > > On Nov 30, 2012, at 8:50 AM, Christian Grün wrote: > >> Hi Bill, >> >>> I'm having trouble figuring out how exactly to enable fulltext XQuery searches. We have it working, but quite a few keywords are not being indexed. [...] >> >> as you already indicated, the fulltext index gets lost when performing >> updates. If you set FTNDEX to TRUE before creating your database, your >> fulltext index will always be recreated when calling OPTIMIZE: >> >> set ftindex true >> create db dorr_index >> add ... >> optimize >> >>> for $i in collection(PATH)//item >>> let $s := $i/mhs:sort >>> where $i contains text ("QUERY" using stemming) >>> return ... >> >> I know too less about your xml structure, so it's difficult to tell >> what/if sth. goes wrong. Does the behavior change when following the >> steps suggested above? >> >> Cheers, >> Christian >> ___________________________ >> >>> for $i in collection(PATH)//item >>> let $s := $i/mhs:sort >>> where $i contains text ("QUERY" using stemming) >>> return ... >>> >>> Are there any steps missing? Also, any tutorial out there to follow? >>> >>> Thanks very much in advance! >>> >>> -Bill B >>> -- >>> Bill Beck, Web Development Specialist >>> Massachusetts Historical Society >>> 1154 Boylston Street, Boston, MA 02215 >>> Tel: 617-646-0505, Fax: 617-859-0074 >>> www.masshist.org - America's Oldest Historical Society - Founded 1791 >>> >>> >>> In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/. >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > -- > Bill Beck, Web Development Specialist > Massachusetts Historical Society > 1154 Boylston Street, Boston, MA 02215 > Tel: 617-646-0505, Fax: 617-859-0074 > www.masshist.org - America's Oldest Historical Society - Founded 1791 > > > In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/.

⚡Generated with XQuery