Question regards "add path"; does it break existing indexes?
Wednesday, 25 August 2010 · 2 messages
SS
Sandra Maria Silcot25 Aug 2010 06:12 #
Hi all,
Using Basex v6.2.3 jar under linux, sun jdk1.6.0_18. I have a database named 'linkdb' with a few documents and a full set of indexes built. I would like to be able to add/delete docs and simply optimise the indexes without needing to rebuild them from scratch. But I am having problems doing so.
Before adding a document, the "info" command reports:
Database Properties
Name: linkdb
Size: 29338 KB
Nodes: 1085033
Height: 6
Database Creation
...
Documents: 3
Whitespace Chopping: ON
Entity Parsing: OFF
Indexes
Path Summary: ON
Text Index: ON
Attribute Index: ON
Full-Text Index: ON
All is well. I then add another document and I can access the new document via xquery, but now, even after an "optimize", "info db" reports:
Database Properties
Name: linkdb
Size: 29414 KB
Nodes: 1088588
Height: 6
Database Creation
...
Documents: 4
Whitespace Chopping: ON
Entity Parsing: OFF
Indexes
Path Summary: ON
Text Index: OFF
Attribute Index: OFF
Full-Text Index: OFF
So what has just happened to the text, attribute and fulltext indexes?
The only process which seems to work is to completely rebuild them, like so:
LOGIN admin OK
SET pathindex on OK 0.19 ms
SET textindex on OK 0.13 ms
SET attrindex on OK 0.14 ms
SET ftindex on OK 0.13 ms
SET wildcards on OK 0.14 ms
SET casesens off OK 0.13 ms
SET scoring 2 OK 0.13 ms
OPEN linkdb OK 0.69 ms
CREATE INDEX TEXT OK 424.54 ms
CREATE INDEX PATH OK 90.74 ms
CREATE INDEX ATTRIBUTE OK 344.46 ms
CREATE INDEX FULLTEXT OK 996.65 ms
OPTIMIZE OK 1784.94 ms
CLOSE OK 0.18 ms
And now:
Indexes
Path Summary: ON
Text Index: ON
Attribute Index: ON
Full-Text Index: ON (wildcards)
In the .basex file in the home dir of the user the basex server is run as, I attempt to set global options like so:
TEXTINDEX = ON
ATTRINDEX = ON
FTINDEX = ON
WILDCARDS = ON
CASESENS = OFF
SCORING = 2
but this seems to be always ignored.
What am I doing wrong, or is this the intended behaviour? Where does the ".basex" file need to be to take effect as a set of defaults.
The behaviour is repeatable irrespective of whether I use the java client or run the commands via a perl script.
Many thanks for any advice,
Cheers,
Sandra.CG
Christian Grün25 Aug 2010 18:58 #
Dear Sandra,
thanks for your e-mail. It's true, indexes get lost after updates. As many database system we know are pretty slow when it comes to update existing indexes, we decided to go for very slim index structures, which can be built much faster than indexes from competing systems. As the general behavior of dropping indexes could as well be provoked by the user, if desired (by calling DROP INDEX, doing an update and calling CREATE INDEX), we're currently working on two steps to add incremental updates:
– allowing (out-of-dated) index results after updates
– incremental updates aligned with document updates
Note that an OPTIMIZE call is enough to recreate the currently activated indexes; calling "create index" will do the job twice:
set ftindex on
create db ..
add ...
optimize
Index updates are one of the todos on our request list:
https://mailman.uni-konstanz.de/pipermail/basex-talk/2010-July/000482.html
Hope this helps,
Christian
Quoted text · Sandra Maria Silcot · 111 lines
On Wed, Aug 25, 2010 at 6:12 AM, Sandra Maria Silcot <ssilcot at unimelb.edu.au> wrote: > Hi all, > > Using Basex v6.2.3 jar under linux, sun jdk1.6.0_18. I have a database > named 'linkdb' with a few documents and a full set of indexes built. I > would like to be able to add/delete docs and simply optimise the indexes > without needing to rebuild them from scratch. But I am having problems > doing so. > > Before adding a document, the "info" command reports: > > Database Properties > Name: linkdb > Size: 29338 KB > Nodes: 1085033 > Height: 6 > > Database Creation > ... > Documents: 3 > Whitespace Chopping: ON > Entity Parsing: OFF > > Indexes > Path Summary: ON > Text Index: ON > Attribute Index: ON > Full-Text Index: ON > > All is well. I then add another document and I can access the new document > via xquery, but now, even after an "optimize", "info db" reports: > > Database Properties > Name: linkdb > Size: 29414 KB > Nodes: 1088588 > Height: 6 > > Database Creation > ... > Documents: 4 > Whitespace Chopping: ON > Entity Parsing: OFF > > Indexes > Path Summary: ON > Text Index: OFF > Attribute Index: OFF > Full-Text Index: OFF > > So what has just happened to the text, attribute and fulltext indexes? > > The only process which seems to work is to completely rebuild them, like so: > > LOGIN admin OK > SET pathindex on OK 0.19 ms > SET textindex on OK 0.13 ms > SET attrindex on OK 0.14 ms > SET ftindex on OK 0.13 ms > SET wildcards on OK 0.14 ms > SET casesens off OK 0.13 ms > SET scoring 2 OK 0.13 ms > OPEN linkdb OK 0.69 ms > CREATE INDEX TEXT OK 424.54 ms > CREATE INDEX PATH OK 90.74 ms > CREATE INDEX ATTRIBUTE OK 344.46 ms > CREATE INDEX FULLTEXT OK 996.65 ms > OPTIMIZE OK 1784.94 ms > CLOSE OK 0.18 ms > > And now: > > Indexes > Path Summary: ON > Text Index: ON > Attribute Index: ON > Full-Text Index: ON (wildcards) > > In the .basex file in the home dir of the user the basex server is run as, > I attempt to set global options like so: > > TEXTINDEX = ON > ATTRINDEX = ON > FTINDEX = ON > WILDCARDS = ON > CASESENS = OFF > SCORING = 2 > > but this seems to be always ignored. > > What am I doing wrong, or is this the intended behaviour? Where does the > ".basex" file need to be to take effect as a set of defaults. > > The behaviour is repeatable irrespective of whether I use the java client > or run the commands via a perl script. > > Many thanks for any advice, > > Cheers, > > Sandra. > > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
Written for BaseX 6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery