Index issue for a large collection

-----Ursprüngliche Nachricht----- Von: basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von Ivan Lagunov Gesendet: Mittwoch, 20. Oktober 2010 16:00 An: Christian Grün Cc: BaseX Talk Betreff: [basex-talk] Index issue for a large collection Hi, I have a database collection with over 17000 documents (about 190MB in total). Everything worked fine until I created text and attribute indexes to improve performance. Now I cannot open database and perform any operation with it because I get 'Error: Out of Main Memory'. Setting "intparse on" doesn't help. I've increased Xmx value up to 2560m (maximum value for my system), it doesn't help too. Is there any way to fix it? Could you provide an approximate size of memory that is required for text and attribute indexes depending on the size of the database? Is there any possibility for me to use indexes on current system? Thanks in advance! Best regards, Ivan BaseX-Talk mailing list BaseX-Talk at mailman.uni-konstanz.de https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

20.10.2010 18:21, Oliver Egli wrote: > Hi Ivan, > > I used to have the same problem. If you are working with the GUI mode, try > to close the Map Visualization. (Open another smaller DB, close the > Visualization, and try to open your Database again). > That worked fine for me. > > Regards, Oli > > -----Ursprüngliche Nachricht----- > Von: basex-talk-bounces at mailman.uni-konstanz.de > [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von Ivan > Lagunov > Gesendet: Mittwoch, 20. Oktober 2010 16:00 > An: Christian Grün > Cc: BaseX Talk > Betreff: [basex-talk] Index issue for a large collection > > Hi, > > I have a database collection with over 17000 documents (about 190MB in > total). Everything worked fine until I created text and attribute > indexes to improve performance. Now I cannot open database and perform > any operation with it because I get 'Error: Out of Main Memory'. Setting > "intparse on" doesn't help. I've increased Xmx value up to 2560m > (maximum value for my system), it doesn't help too. Is there any way to > fix it? Could you provide an approximate size of memory that is required > for text and attribute indexes depending on the size of the database? Is > there any possibility for me to use indexes on current system? Thanks in > advance! > > Best regards, > Ivan > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >

On Wed, Oct 20, 2010 at 10:42 PM, Ivan Lagunov <ilagunov at evelopers.com> wrote: > Hi Oliver, > > I tried both GUI mode and terminal client. I get the same error there. And > also Map Visualization worked fine for the same collection before I created > indexes. Don't you have the same problem in terminal mode? How big is your > collection then (number of files and size in MB)? > > Best regards, > Ivan > > 20.10.2010 18:21, Oliver Egli wrote: >> >> Hi Ivan, >> >> I used to have the same problem. If you are working with the GUI mode, try >> to close the Map Visualization. (Open another smaller DB, close the >> Visualization, and try to open your Database again). >> That worked fine for me. >> >> Regards, Oli >> >> -----Ursprüngliche Nachricht----- >> Von: basex-talk-bounces at mailman.uni-konstanz.de >> [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von Ivan >> Lagunov >> Gesendet: Mittwoch, 20. Oktober 2010 16:00 >> An: Christian Grün >> Cc: BaseX Talk >> Betreff: [basex-talk] Index issue for a large collection >> >> Hi, >> >> I have a database collection with over 17000 documents (about 190MB in >> total). Everything worked fine until I created text and attribute >> indexes to improve performance. Now I cannot open database and perform >> any operation with it because I get 'Error: Out of Main Memory'. Setting >> "intparse on" doesn't help. I've increased Xmx value up to 2560m >> (maximum value for my system), it doesn't help too. Is there any way to >> fix it? Could you provide an approximate size of memory that is required >> for text and attribute indexes depending on the size of the database? Is >> there any possibility for me to use indexes on current system? Thanks in >> advance! >> >> Best regards, >> Ivan >> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> > >

Am 21.10.2010 16:09, schrieb Christian Grün: > Ivan, > > hm, 190 MB sounds like a pretty small collection; 2.5 GB RAM should > suffice to process 10 times the data and more. Have you tried version > 6.2.9 of BaseX? Could you please check out the debugging flag on > command line? Try sth. like: > > java -cp basex.jar org.basex.BaseX -d -c "create db ...." > > Thanks, > Christian > > > > On Wed, Oct 20, 2010 at 10:42 PM, Ivan Lagunov<ilagunov at evelopers.com> wrote: >> Hi Oliver, >> >> I tried both GUI mode and terminal client. I get the same error there. And >> also Map Visualization worked fine for the same collection before I created >> indexes. Don't you have the same problem in terminal mode? How big is your >> collection then (number of files and size in MB)? >> >> Best regards, >> Ivan >> >> 20.10.2010 18:21, Oliver Egli wrote: >>> Hi Ivan, >>> >>> I used to have the same problem. If you are working with the GUI mode, try >>> to close the Map Visualization. (Open another smaller DB, close the >>> Visualization, and try to open your Database again). >>> That worked fine for me. >>> >>> Regards, Oli >>> >>> -----Ursprüngliche Nachricht----- >>> Von: basex-talk-bounces at mailman.uni-konstanz.de >>> [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von Ivan >>> Lagunov >>> Gesendet: Mittwoch, 20. Oktober 2010 16:00 >>> An: Christian Grün >>> Cc: BaseX Talk >>> Betreff: [basex-talk] Index issue for a large collection >>> >>> Hi, >>> >>> I have a database collection with over 17000 documents (about 190MB in >>> total). Everything worked fine until I created text and attribute >>> indexes to improve performance. Now I cannot open database and perform >>> any operation with it because I get 'Error: Out of Main Memory'. Setting >>> "intparse on" doesn't help. I've increased Xmx value up to 2560m >>> (maximum value for my system), it doesn't help too. Is there any way to >>> fix it? Could you provide an approximate size of memory that is required >>> for text and attribute indexes depending on the size of the database? Is >>> there any possibility for me to use indexes on current system? Thanks in >>> advance! >>> >>> Best regards, >>> Ivan >>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> >>> >> > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk

On 10/25/2010 02:25 PM, Andreas Weiler wrote: > Dear Ivan, > > there are good news...Christian has fixed the index problem. > Get the newest version here... > > http://basex.org/maven/org/basex/basex/6.3 > > Kind regards, > Andreas > > Am 21.10.2010 16:09, schrieb Christian Grün: >> Ivan, >> >> hm, 190 MB sounds like a pretty small collection; 2.5 GB RAM should >> suffice to process 10 times the data and more. Have you tried version >> 6.2.9 of BaseX? Could you please check out the debugging flag on >> command line? Try sth. like: >> >> java -cp basex.jar org.basex.BaseX -d -c "create db ...." >> >> Thanks, >> Christian >> >> >> >> On Wed, Oct 20, 2010 at 10:42 PM, Ivan >> Lagunov<ilagunov at evelopers.com> wrote: >>> Hi Oliver, >>> >>> I tried both GUI mode and terminal client. I get the same error >>> there. And >>> also Map Visualization worked fine for the same collection before I >>> created >>> indexes. Don't you have the same problem in terminal mode? How big >>> is your >>> collection then (number of files and size in MB)? >>> >>> Best regards, >>> Ivan >>> >>> 20.10.2010 18:21, Oliver Egli wrote: >>>> Hi Ivan, >>>> >>>> I used to have the same problem. If you are working with the GUI >>>> mode, try >>>> to close the Map Visualization. (Open another smaller DB, close the >>>> Visualization, and try to open your Database again). >>>> That worked fine for me. >>>> >>>> Regards, Oli >>>> >>>> -----Ursprüngliche Nachricht----- >>>> Von: basex-talk-bounces at mailman.uni-konstanz.de >>>> [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von >>>> Ivan >>>> Lagunov >>>> Gesendet: Mittwoch, 20. Oktober 2010 16:00 >>>> An: Christian Grün >>>> Cc: BaseX Talk >>>> Betreff: [basex-talk] Index issue for a large collection >>>> >>>> Hi, >>>> >>>> I have a database collection with over 17000 documents (about 190MB in >>>> total). Everything worked fine until I created text and attribute >>>> indexes to improve performance. Now I cannot open database and perform >>>> any operation with it because I get 'Error: Out of Main Memory'. >>>> Setting >>>> "intparse on" doesn't help. I've increased Xmx value up to 2560m >>>> (maximum value for my system), it doesn't help too. Is there any >>>> way to >>>> fix it? Could you provide an approximate size of memory that is >>>> required >>>> for text and attribute indexes depending on the size of the >>>> database? Is >>>> there any possibility for me to use indexes on current system? >>>> Thanks in >>>> advance! >>>> >>>> Best regards, >>>> Ivan >>>> >>>> _______________________________________________ >>>> BaseX-Talk mailing list >>>> BaseX-Talk at mailman.uni-konstanz.de >>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>>> >>>> >>> >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >

Am 25.10.2010 12:51, schrieb Ivan Lagunov: > Dear Andreas, > > I've started the server with the following command: > java -Xms256m -Xmx2560m -XX:MaxPermSize=256m -cp basex.jar > org.basex.BaseXServer -d > > Then I've tried to 'open database' in basex client. > > Finally I've got the following exception: > BaseX 6.3 [Server] > Server was started. > java.util.Arrays.copyOf(Arrays.java:2834) > org.basex.util.IntList.add(IntList.java:53) > org.basex.data.Data.doc(Data.java:195) > org.basex.core.Context.update(Context.java:126) > org.basex.core.Context.openDB(Context.java:108) > org.basex.core.cmd.Open.run(Open.java:35) > org.basex.core.Command.run(Command.java:236) > org.basex.core.Command.exec(Command.java:218) > org.basex.core.Command.execute(Command.java:66) > org.basex.server.ServerProcess.run(ServerProcess.java:161) > > Memory usage of the server java process reaches 1.6GB (I was > monitoring it visually using 'top' linux command so real usage might > have been even higher). > > Best regards, > Ivan > > On 10/25/2010 02:25 PM, Andreas Weiler wrote: >> Dear Ivan, >> >> there are good news...Christian has fixed the index problem. >> Get the newest version here... >> >> http://basex.org/maven/org/basex/basex/6.3 >> >> Kind regards, >> Andreas >> >> Am 21.10.2010 16:09, schrieb Christian Grün: >>> Ivan, >>> >>> hm, 190 MB sounds like a pretty small collection; 2.5 GB RAM should >>> suffice to process 10 times the data and more. Have you tried version >>> 6.2.9 of BaseX? Could you please check out the debugging flag on >>> command line? Try sth. like: >>> >>> java -cp basex.jar org.basex.BaseX -d -c "create db ...." >>> >>> Thanks, >>> Christian >>> >>> >>> >>> On Wed, Oct 20, 2010 at 10:42 PM, Ivan >>> Lagunov<ilagunov at evelopers.com> wrote: >>>> Hi Oliver, >>>> >>>> I tried both GUI mode and terminal client. I get the same error >>>> there. And >>>> also Map Visualization worked fine for the same collection before I >>>> created >>>> indexes. Don't you have the same problem in terminal mode? How big >>>> is your >>>> collection then (number of files and size in MB)? >>>> >>>> Best regards, >>>> Ivan >>>> >>>> 20.10.2010 18:21, Oliver Egli wrote: >>>>> Hi Ivan, >>>>> >>>>> I used to have the same problem. If you are working with the GUI >>>>> mode, try >>>>> to close the Map Visualization. (Open another smaller DB, close the >>>>> Visualization, and try to open your Database again). >>>>> That worked fine for me. >>>>> >>>>> Regards, Oli >>>>> >>>>> -----Ursprüngliche Nachricht----- >>>>> Von: basex-talk-bounces at mailman.uni-konstanz.de >>>>> [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von >>>>> Ivan >>>>> Lagunov >>>>> Gesendet: Mittwoch, 20. Oktober 2010 16:00 >>>>> An: Christian Grün >>>>> Cc: BaseX Talk >>>>> Betreff: [basex-talk] Index issue for a large collection >>>>> >>>>> Hi, >>>>> >>>>> I have a database collection with over 17000 documents (about >>>>> 190MB in >>>>> total). Everything worked fine until I created text and attribute >>>>> indexes to improve performance. Now I cannot open database and >>>>> perform >>>>> any operation with it because I get 'Error: Out of Main Memory'. >>>>> Setting >>>>> "intparse on" doesn't help. I've increased Xmx value up to 2560m >>>>> (maximum value for my system), it doesn't help too. Is there any >>>>> way to >>>>> fix it? Could you provide an approximate size of memory that is >>>>> required >>>>> for text and attribute indexes depending on the size of the >>>>> database? Is >>>>> there any possibility for me to use indexes on current system? >>>>> Thanks in >>>>> advance! >>>>> >>>>> Best regards, >>>>> Ivan >>>>> >>>>> _______________________________________________ >>>>> BaseX-Talk mailing list >>>>> BaseX-Talk at mailman.uni-konstanz.de >>>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>>>> >>>>> >>>> >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> > >

On 10/25/2010 03:21 PM, Andreas Weiler wrote: > Dear Ivan, > > did u create the database with the newest version? > > Kind regards, > Andreas > > Am 25.10.2010 12:51, schrieb Ivan Lagunov: >> Dear Andreas, >> >> I've started the server with the following command: >> java -Xms256m -Xmx2560m -XX:MaxPermSize=256m -cp basex.jar >> org.basex.BaseXServer -d >> >> Then I've tried to 'open database' in basex client. >> >> Finally I've got the following exception: >> BaseX 6.3 [Server] >> Server was started. >> java.util.Arrays.copyOf(Arrays.java:2834) >> org.basex.util.IntList.add(IntList.java:53) >> org.basex.data.Data.doc(Data.java:195) >> org.basex.core.Context.update(Context.java:126) >> org.basex.core.Context.openDB(Context.java:108) >> org.basex.core.cmd.Open.run(Open.java:35) >> org.basex.core.Command.run(Command.java:236) >> org.basex.core.Command.exec(Command.java:218) >> org.basex.core.Command.execute(Command.java:66) >> org.basex.server.ServerProcess.run(ServerProcess.java:161) >> >> Memory usage of the server java process reaches 1.6GB (I was >> monitoring it visually using 'top' linux command so real usage might >> have been even higher). >> >> Best regards, >> Ivan >> >> On 10/25/2010 02:25 PM, Andreas Weiler wrote: >>> Dear Ivan, >>> >>> there are good news...Christian has fixed the index problem. >>> Get the newest version here... >>> >>> http://basex.org/maven/org/basex/basex/6.3 >>> >>> Kind regards, >>> Andreas >>> >>> Am 21.10.2010 16:09, schrieb Christian Grün: >>>> Ivan, >>>> >>>> hm, 190 MB sounds like a pretty small collection; 2.5 GB RAM should >>>> suffice to process 10 times the data and more. Have you tried version >>>> 6.2.9 of BaseX? Could you please check out the debugging flag on >>>> command line? Try sth. like: >>>> >>>> java -cp basex.jar org.basex.BaseX -d -c "create db ...." >>>> >>>> Thanks, >>>> Christian >>>> >>>> >>>> >>>> On Wed, Oct 20, 2010 at 10:42 PM, Ivan >>>> Lagunov<ilagunov at evelopers.com> wrote: >>>>> Hi Oliver, >>>>> >>>>> I tried both GUI mode and terminal client. I get the same error >>>>> there. And >>>>> also Map Visualization worked fine for the same collection before >>>>> I created >>>>> indexes. Don't you have the same problem in terminal mode? How big >>>>> is your >>>>> collection then (number of files and size in MB)? >>>>> >>>>> Best regards, >>>>> Ivan >>>>> >>>>> 20.10.2010 18:21, Oliver Egli wrote: >>>>>> Hi Ivan, >>>>>> >>>>>> I used to have the same problem. If you are working with the GUI >>>>>> mode, try >>>>>> to close the Map Visualization. (Open another smaller DB, close the >>>>>> Visualization, and try to open your Database again). >>>>>> That worked fine for me. >>>>>> >>>>>> Regards, Oli >>>>>> >>>>>> -----Ursprüngliche Nachricht----- >>>>>> Von: basex-talk-bounces at mailman.uni-konstanz.de >>>>>> [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag >>>>>> von Ivan >>>>>> Lagunov >>>>>> Gesendet: Mittwoch, 20. Oktober 2010 16:00 >>>>>> An: Christian Grün >>>>>> Cc: BaseX Talk >>>>>> Betreff: [basex-talk] Index issue for a large collection >>>>>> >>>>>> Hi, >>>>>> >>>>>> I have a database collection with over 17000 documents (about >>>>>> 190MB in >>>>>> total). Everything worked fine until I created text and attribute >>>>>> indexes to improve performance. Now I cannot open database and >>>>>> perform >>>>>> any operation with it because I get 'Error: Out of Main Memory'. >>>>>> Setting >>>>>> "intparse on" doesn't help. I've increased Xmx value up to 2560m >>>>>> (maximum value for my system), it doesn't help too. Is there any >>>>>> way to >>>>>> fix it? Could you provide an approximate size of memory that is >>>>>> required >>>>>> for text and attribute indexes depending on the size of the >>>>>> database? Is >>>>>> there any possibility for me to use indexes on current system? >>>>>> Thanks in >>>>>> advance! >>>>>> >>>>>> Best regards, >>>>>> Ivan >>>>>> >>>>>> _______________________________________________ >>>>>> BaseX-Talk mailing list >>>>>> BaseX-Talk at mailman.uni-konstanz.de >>>>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>>>>> >>>>>> >>>>> >>>> _______________________________________________ >>>> BaseX-Talk mailing list >>>> BaseX-Talk at mailman.uni-konstanz.de >>>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> >>> >> >> > >

Am 25.10.2010 14:47, schrieb Ivan Lagunov: > It worked fine as soon as I created the database with the newest > version. But I'd like to be confident of BaseX that I won't lose my data > as it happened for the version 6.1.