new to basex
Friday, 30 November 2012 · 5 messages
BB
Bill Beck30 Nov 2012 14:14 #
Hi all,
Here that the Massachusetts Historical Society we're exploring using BaseX for some new transcriptions of newspapers from around the American Revolution, as collected by a fellow named Harbottle Dorr.
I'm having trouble figuring out how exactly to enable fulltext XQuery searches. We have it working, but quite a few keywords are not being indexed. From the command line tool, we do:
Quoted text · 2 lines
>create database dorr_index >create index fulltext
Then we have a PHP tool (using the class found on the BaseX site) that calls $session->execute to open the collection and $session->add to place the files. But when I then run
Quoted text · 1 line
>info db
from the command line tool, it shows:
Database Properties
Name: dorr_index
Size: 1390 KB
Nodes: 32411
Resources: 24
Timestamp: 30.11.2012 08:00:11
Resource Properties
Timestamp: 30.11.2012 08:03:21
Encoding: UTF-8
Whitespace Chopping: ON
Indexes
Up-to-date: false
Path Summary: ON
Text Index: OFF
Attribute Index: OFF
Full-Text Index: OFF
if I then run the command
Quoted text · 1 line
>create index fulltext
again, info db shows full-text index on, but up-to-date is still false, and our search results still have many missing hits. our XQuery is:
for $i in collection(PATH)//item
let $s := $i/mhs:sort
where $i contains text ("QUERY" using stemming)
return ...Are there any steps missing? Also, any tutorial out there to follow?
Thanks very much in advance!
-Bill B
--
Bill Beck, Web Development Specialist
Massachusetts Historical Society
1154 Boylston Street, Boston, MA 02215
Tel: 617-646-0505, Fax: 617-859-0074
www.masshist.org - America's Oldest Historical Society - Founded 1791
In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/.
CG
Christian Grün30 Nov 2012 14:50 #
Hi Bill,
Quoted text · Bill Beck · 1 line
> I'm having trouble figuring out how exactly to enable fulltext XQuery searches. We have it working, but quite a few keywords are not being indexed. [...]
as you already indicated, the fulltext index gets lost when performing updates. If you set FTNDEX to TRUE before creating your database, your fulltext index will always be recreated when calling OPTIMIZE:
set ftindex true
create db dorr_index
add ...
optimize
Quoted text · Bill Beck · 4 lines
> for $i in collection(PATH)//item > let $s := $i/mhs:sort > where $i contains text ("QUERY" using stemming) > return ...
I know too less about your xml structure, so it's difficult to tell what/if sth. goes wrong. Does the behavior change when following the steps suggested above?
Cheers,
Christian
Quoted text · Bill Beck · 23 lines
> for $i in collection(PATH)//item > let $s := $i/mhs:sort > where $i contains text ("QUERY" using stemming) > return ... > > Are there any steps missing? Also, any tutorial out there to follow? > > Thanks very much in advance! > > -Bill B > -- > Bill Beck, Web Development Specialist > Massachusetts Historical Society > 1154 Boylston Street, Boston, MA 02215 > Tel: 617-646-0505, Fax: 617-859-0074 > www.masshist.org - America's Oldest Historical Society - Founded 1791 > > > In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/. > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk
J
Quoted text · Bill Beck · 4 lines
On 11/30/2012 02:14 PM, Bill Beck wrote: > [Activating Fulltext index] > > Are there any steps missing? Also, any tutorial out there to follow?
Usually it's up to the
UPDINDEX[1] option (for text/attribute values), but currently the Fulltext index can not be updated efficiently (at least I think so). It is planned[2] however to rewrite the index structure.
However, I'm not sure about the real problem, as "QUERY" shouldn't be a stopword which is not indexed.
But shouldn't it be "where $i/text() contains text 'QUERY' using stemming"? At least $i is bound to an item I guess, and you have to use text values (or probably attribute values), but not an element.
[1] http://docs.basex.org/wiki/Options#UPDINDEX
[2] https://github.com/BaseXdb/basex/issues/169J
Johannes.Lichtenberger30 Nov 2012 14:56 #
Quoted text · Bill Beck · 16 lines
On 11/30/2012 02:53 PM, Johannes.Lichtenberger wrote: > On 11/30/2012 02:14 PM, Bill Beck wrote: >> [Activating Fulltext index] > > >> Are there any steps missing? Also, any tutorial out there to follow? > > Usually it's up to the UPDINDEX[1] option (for text/attribute values), > but currently the Fulltext index can not be updated efficiently (at > least I think so). It is planned[2] however to rewrite the index structure. > > However, I'm not sure about the real problem, as "QUERY" shouldn't be a > stopword which is not indexed. > > But shouldn't it be "where $i/text() contains text 'QUERY' using > stemming"? At least $i is bound to an item I guess, and you have to use > text values (or probably attribute values), but not an element.
Ok, the last sentence is wrong ;-)
CG
Christian Grün30 Nov 2012 15:14 #
Hi Bill (cc to the list),
the current fulltext global indexing options (specified via SET) are applied and stored in the target database whenever an index is created via CREATE INDEX. If OPTIMIZE is used, these database options will be used for recreating the index. Both the database options and index structures should not get lost if you restart your server.
The following (non-server) example may help to demonstrate that fulltext options such as stemming will be stored along with the database:
basex -c "set stemming on;set ftindex on; create db db <a>A</a>; optimize; close; set stemming off; open db;info index fulltext"
Btw, which BaseX version do you use? It would be interesting to see what happens with our latest stable snapshot [1]; the final version is long overdue, but it will finally be released in December.
Christian
[1] http://files.basex.org/releases/latest/
Quoted text · Johannes.Lichtenberger · 82 lines
> HI Christian, > > Thanks for the fast reply. > > I tried your steps, with no change. I am curious about a couple of things I notice: > > Using these command I enabled fulltext with stemming: > >>set stemming true >>create index fulltext > > And info index shows these to be true. But if I stop the server and restart, these are all off, and the fulltext index is gone (the collection and it's files are still there). And yet, the same flawed search still works. > > So my questions are: > Do we need to reset globals and reindex each time the server is restart? It our implimentation it seems so (running standalone mode). > > Thanks. > > -bb > > > On Nov 30, 2012, at 8:50 AM, Christian Grün wrote: > >> Hi Bill, >> >>> I'm having trouble figuring out how exactly to enable fulltext XQuery searches. We have it working, but quite a few keywords are not being indexed. [...] >> >> as you already indicated, the fulltext index gets lost when performing >> updates. If you set FTNDEX to TRUE before creating your database, your >> fulltext index will always be recreated when calling OPTIMIZE: >> >> set ftindex true >> create db dorr_index >> add ... >> optimize >> >>> for $i in collection(PATH)//item >>> let $s := $i/mhs:sort >>> where $i contains text ("QUERY" using stemming) >>> return ... >> >> I know too less about your xml structure, so it's difficult to tell >> what/if sth. goes wrong. Does the behavior change when following the >> steps suggested above? >> >> Cheers, >> Christian >> ___________________________ >> >>> for $i in collection(PATH)//item >>> let $s := $i/mhs:sort >>> where $i contains text ("QUERY" using stemming) >>> return ... >>> >>> Are there any steps missing? Also, any tutorial out there to follow? >>> >>> Thanks very much in advance! >>> >>> -Bill B >>> -- >>> Bill Beck, Web Development Specialist >>> Massachusetts Historical Society >>> 1154 Boylston Street, Boston, MA 02215 >>> Tel: 617-646-0505, Fax: 617-859-0074 >>> www.masshist.org - America's Oldest Historical Society - Founded 1791 >>> >>> >>> In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/. >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > -- > Bill Beck, Web Development Specialist > Massachusetts Historical Society > 1154 Boylston Street, Boston, MA 02215 > Tel: 617-646-0505, Fax: 617-859-0074 > www.masshist.org - America's Oldest Historical Society - Founded 1791 > > > In Death Lamented: The Tradition of Anglo-American Mourning Jewelry opens on 28 September 2012 and runs through 31 January 2013. The exhibition will be on display Monday through Saturday from 10 AM to 4 PM. More information is available at www.masshist.org/events/.
Written for BaseX 7.3. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery