Potential Bug - message from fulltext

Sat, Jun 27, 2015 at 12:50 PM, Lars Johnsen <yoonsen at gmail.com> wrote: > When trying to to a full text index on a collection of texts, the process > runs for a couple of hours with the exit message below - I think it is near > completed. From the GUI, I have at least seen the progress bar get to around > 80 %, so I think it is safe to assume that the error is connectedt the final > stages. > > The texts are unstructured and represented as one line pr. book. Here is the > result from the index process. Parameters set in GUI are: Norwegian > Snowball, lemmatization, diacritics. There is set aside 30GB for the GUI. > > Path summary: > doc(): 317259x, strings > text: 317259x, leaf > text(): 317259x, strings, leaf > > Here is the error message: > > Improper use? Potential bug? Your feedback is welcome: > Contact: basex-talk at mailman.uni-konstanz.de > Version: BaseX 8.2 beta 7d38949 > Java: Oracle Corporation, 1.7.0_79 > OS: Linux, amd64 > Stack Trace: > java.lang.NegativeArraySizeException > at java.util.Arrays.copyOf(Arrays.java:2271) > at org.basex.util.TokenBuilder.add(TokenBuilder.java:303) > at org.basex.util.TokenBuilder.add(TokenBuilder.java:290) > at org.basex.index.ft.FTBuilder.merge(FTBuilder.java:248) > at org.basex.index.ft.FTBuilder.write(FTBuilder.java:155) > at org.basex.index.ft.FTBuilder.index(FTBuilder.java:94) > at org.basex.index.ft.FTBuilder.build(FTBuilder.java:102) > at org.basex.index.ft.FTBuilder.build(FTBuilder.java:1) > at org.basex.data.DiskData.createIndex(DiskData.java:195) > at org.basex.core.cmd.ACreate.create(ACreate.java:117) > at org.basex.core.cmd.CreateIndex.run(CreateIndex.java:62) > at org.basex.core.Command.run(Command.java:398) > at org.basex.core.Command.execute(Command.java:100) > at org.basex.core.Command.execute(Command.java:123) > at org.basex.gui.dialog.DialogProgress$1.run(DialogProgress.java:178) > > Regards > Lars G Johnsen > National Library of Norway

> Hi Lars, > > It looks as if the input data is indeed too large to be indexed (the > internal id lists seem to exceed the maximum array size in main > memory). The usual alternative to make it work is to distribute your > document(s) into multiple databases. > > If you want, you can also provide us with the input data, but I assume > it will take pretty much space? > > Best, > Christian > > > Sat, Jun 27, 2015 at 12:50 PM, Lars Johnsen <yoonsen at gmail.com> wrote: > > When trying to to a full text index on a collection of texts, the process > > runs for a couple of hours with the exit message below - I think it is > near > > completed. From the GUI, I have at least seen the progress bar get to > around > > 80 %, so I think it is safe to assume that the error is connectedt the > final > > stages. > > > > The texts are unstructured and represented as one line pr. book. Here is > the > > result from the index process. Parameters set in GUI are: Norwegian > > Snowball, lemmatization, diacritics. There is set aside 30GB for the GUI. > > > > Path summary: > > doc(): 317259x, strings > > text: 317259x, leaf > > text(): 317259x, strings, leaf > > > > Here is the error message: > > > > Improper use? Potential bug? Your feedback is welcome: > > Contact: basex-talk at mailman.uni-konstanz.de > > Version: BaseX 8.2 beta 7d38949 > > Java: Oracle Corporation, 1.7.0_79 > > OS: Linux, amd64 > > Stack Trace: > > java.lang.NegativeArraySizeException > > at java.util.Arrays.copyOf(Arrays.java:2271) > > at org.basex.util.TokenBuilder.add(TokenBuilder.java:303) > > at org.basex.util.TokenBuilder.add(TokenBuilder.java:290) > > at org.basex.index.ft.FTBuilder.merge(FTBuilder.java:248) > > at org.basex.index.ft.FTBuilder.write(FTBuilder.java:155) > > at org.basex.index.ft.FTBuilder.index(FTBuilder.java:94) > > at org.basex.index.ft.FTBuilder.build(FTBuilder.java:102) > > at org.basex.index.ft.FTBuilder.build(FTBuilder.java:1) > > at org.basex.data.DiskData.createIndex(DiskData.java:195) > > at org.basex.core.cmd.ACreate.create(ACreate.java:117) > > at org.basex.core.cmd.CreateIndex.run(CreateIndex.java:62) > > at org.basex.core.Command.run(Command.java:398) > > at org.basex.core.Command.execute(Command.java:100) > > at org.basex.core.Command.execute(Command.java:123) > > at org.basex.gui.dialog.DialogProgress$1.run(DialogProgress.java:178) > > > > Regards > > Lars G Johnsen > > National Library of Norway >