BaseX: How to create database with attribute indexes using REST
Thursday, 28 March 2013 · 4 messages
WF
Willi Firulais28 Mar 2013 12:37 #
Hallo,
Some hints and help on how to create database with attribute indexes using REST are needed.
Im using the BaseX77-20130321.085136.war on JBoss7.1.1 and
want to create a database with attibute indexes using the REST interfaces. Fresh install and Options haven't been changed.
Create of database over REST works fine, but there are not attribute indexes (not in the info - see blow, and there are no atvl, atvr files neither)
http://127.0.0.1:8080/BaseX77-20130321.085136/rest?command=INFO General Information
Version: 7.7 beta
...
Options
...
ADDCACHE: false
ATTRINDEX: true
AUTOFLUSH: true
FTINDEX: false
INDEXSPLITSIZE: 0
PARSER: xml
STEMMING: false
TEXTINDEX: true
UPDINDEX: false
WRITEBACK: false
...
http://127.0.0.1:8080/BaseX77-20130321.085136/rest/austria20130328103351?command=INFO+DB ...
Database Properties
Name: osmdb
Size: 7407 MB
Nodes: 355247222
Documents: 588
...
Indexes
Up-to-date: false
Text Index: OFF
Attribute Index: OFF
Full-Text Index: OFF
UPDINDEX: OFF
...
Why in INFO+DB the "Attribute Index: OFF" is displayed - i have never changed any option. My assumtion was that with this options the attribute indixes will be created during upload.curl -i -X PUT http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb
curl -i -X PUT -T "part1.osm" http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb/part1.osm
curl -i -X PUT -T "part2.osm" http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb/part2.osm
...
curl -i -X GET http://127.0.0.1:8080/BaseX77-20130321.085136/rest/osm?command=FLUSHHave tried to create the attribute index afterwards usint this but the creation runs into an "Out of Main Memory." error. curl -i -X GET http://127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb?command=CREATE+INDEX+ATTRIBUTE
Some hints are welcome.
Thx for all help in advance,
Willi
CG
Christian Grün28 Mar 2013 12:51 #
Hi Willi,
Quoted text · Willi Firulais · 6 lines
> curl -i -X PUT > http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb > curl -i -X PUT -T "part1.osm" > http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb/part1.osm > curl -i -X PUT -T "part2.osm" > http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb/part2.osm
the attribute index will be discarded when running updates. You can either run "optimize" after having performed all updates, or set the "updindex" flag to "true". Please check out [1] for more details
Quoted text · Willi Firulais · 2 lines
> curl -i -X GET > http://127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb?command=CREATE+INDEX+ATTRIBUTE
How much main memory has been assigned to the BaseX JVM, and how large is your data?
[1] http://docs.basex.org/wiki/Index#Updates
WF
Hallo Chirs,
Thx for your hints - without that would have never come up to that!
Based on that i've decided to add the following line to the creation script. This copies over the whole data into a new directory on your application server and in addition adds the index files (so check if you have enough space and time ;-) for this operation.
curl -i -X GET http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb?command=OPTIMIZE+ALLCurrently im gettion an "Out of Main Memory" with the OPTIMAZE+ALL command when it comes to build the index. During the OPTIMIZE+ALL the .atv,.tbl,.tbli files have been copied over into the reorganization directory successful and then the process dies with the "Out of Main Memory".
Size of db (as mentioned in thread):
http://127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb?command=INFO+DB ...
Database Properties
Name: osmdb
Size: 7407 MB
Nodes: 355247222
Documents: 588
...
Memory of JVM (JBoss 7.1.1 with only baseX deployed to it):
JAVA_OPTS: -XX:+TieredCompilation -Dprogram.name=standalone.bat -Xms512m -Xmx 1024m -XX:MaxPermSize=512m
best Willi
P.S.: I realy like this REST interfaces of baseX (simple and powerful) - great work - keep on going!
Quoted text · christian.gruen at gmail.com · 1 line
Mar 28, 2013 07:52:14 AM, christian.gruen at gmail.com wrote:
Hi Willi,
Quoted text · Willi Firulais · 20 lines
> >> curl -i -X PUT >> http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb >> curl -i -X PUT -T "part1.osm" >> http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb/part1.osm >> curl -i -X PUT -T "part2.osm" >> http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb/part2.osm > >the attribute index will be discarded when running updates. You can >either run "optimize" after having performed all updates, or set the >"updindex" flag to "true". Please check out [1] for more details > >> curl -i -X GET >> http://127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb?command=CREATE+INDEX+ATTRIBUTE > >How much main memory has been assigned to the BaseX JVM, and how large >is your data? > >[1] http://docs.basex.org/wiki/Index#Updates >
CG
Christian Grün28 Mar 2013 17:45 #
Hi Willi,
a general note: when working with large XML instances, Add will always be slower than Create. If you use the command line, you can specify a directory including all input files, e.g.:
basex create osmdb /path/to/osmdb/files
Using REST, you’ll either have to merge the files before running the request, or use multiple add requests (as you are already doing).
Quoted text · Willi Firulais · 2 lines
> JAVA_OPTS: -XX:+TieredCompilation -Dprogram.name=standalone.bat -Xms512m > -Xmx 1024m -XX:MaxPermSize=512m
I’m not sure what’s the particular reason for the failure, but you could try to change -Xmx to a higher or (yes!), smaller value. If that doesn’t help, you may be successful with setting some explicit split values (see [1]). In this case, however, I would recommend you to try the command-line first and set debugging to true. In debugging mode, you will get some hints what values might be promising.
Hope this helps,
Christian
[1] http://docs.basex.org/wiki/Indexes#Index_Construction
Quoted text · Willi Firulais · 65 lines
On Thu, Mar 28, 2013 at 5:29 PM, Willi Firulais <willi.firulais at lycos.com> wrote: > Hallo Chirs, > > Thx for your hints - without that would have never come up to that! > > Based on that i've decided to add the following line to the creation script. > This copies over the whole data into a new directory on your application > server and in addition adds the index files > (so check if you have enough space and time ;-) for this operation. > > curl -i -X GET > http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb?command=OPTIMIZE+ALL > > Currently im gettion an "Out of Main Memory" with the OPTIMAZE+ALL command > when it comes to build the index. > During the OPTIMIZE+ALL the .atv,.tbl,.tbli files have been copied over into > the reorganization directory > successful and then the process dies with the "Out of Main Memory". > > Size of db (as mentioned in thread): > http://127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb?command=INFO+DB > > ... > Database Properties > Name: osmdb > Size: 7407 MB > Nodes: 355247222 > Documents: 588 > ... > Memory of JVM (JBoss 7.1.1 with only baseX deployed to it): > JAVA_OPTS: -XX:+TieredCompilation -Dprogram.name=standalone.bat -Xms512m > -Xmx 1024m -XX:MaxPermSize=512m > > best Willi > > P.S.: I realy like this REST interfaces of baseX (simple and powerful) - > great work - keep on going! > > > Mar 28, 2013 07:52:14 AM, christian.gruen at gmail.com wrote: > Hi Willi, >> >>> curl -i -X PUT >>> http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb >>> curl -i -X PUT -T "part1.osm" >>> >>> http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb/part1.osm >>> curl -i -X PUT -T "part2.osm" >>> >>> http://admin:admin at 127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb/part2.osm >> >>the attribute index will be discarded when running updates. You can >>either run "optimize" after having performed all updates, or set the >>"updindex" flag to "true". Please check out [1] for more details >> >>> curl -i -X GET >>> >>> http://127.0.0.1:8080/BaseX77-20130321.085136/rest/osmdb?command=CREATE+INDEX+ATTRIBUTE >> >>How much main memory has been assigned to the BaseX JVM, and how large >>is your data? >> >>[1] http://docs.basex.org/wiki/Index#Updates >>
Written for BaseX 7.6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery