Bug/question: large collection

> info db

On Mon, Oct 22, 2018 at 4:40 AM Marko Niinimaki <manzikki at gmail.com> wrote: > > Hi, > it looks like "nodes" exceeds some integer range if I add 2 million documents (below). > > Another, unrelated question: our server has 24 cores. What would be the best way to utilize that kind of parallel power in queries? > > > info db > Database Properties > NAME: tmp > SIZE: 47 GB > NODES: -2147476286 > DOCUMENTS: 705708 > BINARIES: 0 > TIMESTAMP: 2018-08-29T02:27:58.000Z > UPTODATE: false > > > Improper use? Potential bug? Your feedback is welcome: > Contact: basex-talk at mailman.uni-konstanz.de > Version: BaseX 9.0.2 > Java: Oracle Corporation, 1.8.0_66 > OS: Linux, amd64 > Stack Trace: > java.lang.ArrayIndexOutOfBoundsException >

> However, I would have expected BaseX to raise an error message. Could you

On Mon, Oct 22, 2018 at 3:40 PM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Marko, > > Databases are restricted to 2^31 nodes. If the limit is exceeded, > you’ll need to distribute your documents across multiple database > instances (see [1] for more details). > > However, I would have expected BaseX to raise an error message. Could > you give us more detail how you imported the documents? > > By default, 8 parallel queries are allowed. The number can be changed > by assigning a different value to the PARALLEL option [2]. In most > cases, you’ll get best results if you ensure that your queries are > rewritten for index access (provided that your queries allows such > rewritings), as multiple concurrent databases access may have negative > effects, in particular if sequential scans are required. Obviously, > things looks slightly better for SSDs. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/Databases > [2] http://docs.basex.org/wiki/Options#PARALLEL > > > > On Mon, Oct 22, 2018 at 4:40 AM Marko Niinimaki <manzikki at gmail.com> > wrote: > > > > Hi, > > it looks like "nodes" exceeds some integer range if I add 2 million > documents (below). > > > > Another, unrelated question: our server has 24 cores. What would be the > best way to utilize that kind of parallel power in queries? > > > > > info db > > Database Properties > > NAME: tmp > > SIZE: 47 GB > > NODES: -2147476286 > > DOCUMENTS: 705708 > > BINARIES: 0 > > TIMESTAMP: 2018-08-29T02:27:58.000Z > > UPTODATE: false > > > > > > Improper use? Potential bug? Your feedback is welcome: > > Contact: basex-talk at mailman.uni-konstanz.de > > Version: BaseX 9.0.2 > > Java: Oracle Corporation, 1.8.0_66 > > OS: Linux, amd64 > > Stack Trace: > > java.lang.ArrayIndexOutOfBoundsException > > >

> Yes it did. I had a file with about 1 million add filename statements and when the max nodes limit was exceeded, each statement gave an error message.

On Mon, Oct 22, 2018 at 4:21 PM Christian Grün <christian.gruen at gmail.com> wrote: > > Yes it did. I had a file with about 1 million add filename statements > and when the max nodes limit was exceeded, each statement gave an error > message. > > Was this a “command script” [1], and did you call the ADD command or > the db:add function? > > Thanks in advance, > Christian > > [1] http://docs.basex.org/wiki/Commands#Command_Scripts >