Could not reserve enough space for object heap

-----Ursprüngliche Nachricht----- Von: basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von Bram Vanroy Gesendet: Dienstag, 15. August 2017 22:22 An: basex-talk at mailman.uni-konstanz.de Betreff: [basex-talk] Could not reserve enough space for object heap Hi all I'm running into an issue with many databases. I.e. one server instance with millions of databases. When creating all of these, I found that the more databases are included on the instance, the slower further database generation got. For instance, I could see in the logs that in the first < 10.000 databases the creation happened smoothly with around 50ms per file of 1-4kB. However, when having more and more databases for this server instance, things got very slow: for an XML file of 1-4kB the logs show ~600ms. This is terribly slow, as you can imagine. At first I thought something was wrong with my hardware, but I checked on another system and the same issues arises. Then I thought maybe Java is doing something strange, so I figured I'd reboot and see if that cleared some stuff up. But now when I try to launch 'basex' or 'basexserver', I get the following message: Could not reserve enough space for 1433600KB object heap I googled the issue, and it was suggested that I added a JAVA option to my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the memory it could use. I set that to 2048MB. But still the same issue persists. I have contacted the list before, with issues of generating millions of database with the same server instance, and this seems another one related to the problem. I am no expert AT ALL, but isn't it possible there is some sort of micro memory leak that only becomes apparent when creating an amount of databases of this magnitude? If not, other ideas are welcome as well. At least on how to get rid of the Java error mentioned above. Kind regards Bram Vanroy

On 16.08.2017 07:53, Kirsten, Dirk wrote: > Hi Bram > > How did you set the java heap space? Usual way would be by using the > command line and setting e.g. > > java -Xmx2g > > However, the object which throws the error is already nearly 1,5GB in > size, so if other stuff has to be allocates as well it could simply be > that it is still not enough. Try increasing it to at least 4GB. > > Indicating the memory leak: Well, it could certainly be, but I > personally doubt it. First of all, having millions of databases is a > lot, so some performance impact is expected. And given the scale of your > databases I wouldn't call 600ms "terribly slow", you just seem to have a > lot of data. When you create a new database there is also stuff which > has to be checked (I would guess for example that BaseX checks that the > database not already exists, which runtime obviously depends on the > number of databases you already have), so I would guess the performance > is somewhat expected. Of course, it might be possible to optimize this > and try to make BaseX more performant for your given scenario. > > Cheers > Dirk > > ** > Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - > Amtsgericht Frankfurt am Main - Reg.-Nr.: HRB 105546 > Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender: > Daniel Grözinger > > > -----Ursprüngliche Nachricht----- > Von: basex-talk-bounces at mailman.uni-konstanz.de > [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von Bram > Vanroy > Gesendet: Dienstag, 15. August 2017 22:22 > An: basex-talk at mailman.uni-konstanz.de > Betreff: [basex-talk] Could not reserve enough space for object heap > > Hi all > > I'm running into an issue with many databases. I.e. one server instance > with millions of databases. When creating all of these, I found that the > more databases are included on the instance, the slower further database > generation got. For instance, I could see in the logs that in the first > < 10.000 databases the creation happened smoothly with around 50ms per > file of 1-4kB. However, when having more and more databases for this > server instance, things got very slow: for an XML file of 1-4kB the logs > show ~600ms. This is terribly slow, as you can imagine. > > At first I thought something was wrong with my hardware, but I checked > on another system and the same issues arises. Then I thought maybe Java > is doing something strange, so I figured I'd reboot and see if that > cleared some stuff up. But now when I try to launch 'basex' or > 'basexserver', I get the following message: > > Could not reserve enough space for 1433600KB object heap > > I googled the issue, and it was suggested that I added a JAVA option to > my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating > the memory it could use. I set that to 2048MB. But still the same issue > persists. > > I have contacted the list before, with issues of generating millions of > database with the same server instance, and this seems another one > related to the problem. I am no expert AT ALL, but isn't it possible > there is some sort of micro memory leak that only becomes apparent when > creating an amount of databases of this magnitude? If not, other ideas > are welcome as well. At least on how to get rid of the Java error > mentioned above. > > > Kind regards > > Bram Vanroy >

Von: basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von Imsieke, Gerrit, le-tex Gesendet: Mittwoch, 16. August 2017 08:04 An: basex-talk at mailman.uni-konstanz.de Betreff: Re: [basex-talk] Could not reserve enough space for object heap It wasn't clear to me from the OP whether the issue of *slowness* persisted after raising max heap to 2 GB, or whether the issue of *not being able to allocate this amount of space at all* persisted. If Bram set max heap to more than 1.5 GB and immediately received the message "Could not reserve enough space ...", this could be an indication that he is running a 32-bit Java on his 64-bit machine. Bram, what is the output of 'java -version'? Mine is: java version "1.8.0_144" Java(TM) SE Runtime Environment (build 1.8.0_144-b01) Java HotSpot(TM) 64-Bit Server VM (build 25.144-b01, mixed mode) If it does not contain '64-Bit', it's 32 bit, and you need to install a different binary from Oracle's site. Gerrit On 16.08.2017 07:53, Kirsten, Dirk wrote: > Hi Bram > > How did you set the java heap space? Usual way would be by using the > command line and setting e.g. > > java -Xmx2g > > However, the object which throws the error is already nearly 1,5GB in > size, so if other stuff has to be allocates as well it could simply be > that it is still not enough. Try increasing it to at least 4GB. > > Indicating the memory leak: Well, it could certainly be, but I > personally doubt it. First of all, having millions of databases is a > lot, so some performance impact is expected. And given the scale of > your databases I wouldn't call 600ms "terribly slow", you just seem to > have a lot of data. When you create a new database there is also stuff > which has to be checked (I would guess for example that BaseX checks > that the database not already exists, which runtime obviously depends > on the number of databases you already have), so I would guess the > performance is somewhat expected. Of course, it might be possible to > optimize this and try to make BaseX more performant for your given scenario. > > Cheers > Dirk > > ** > Senacor Technologies Aktiengesellschaft - Sitz: Eschborn - Amtsgericht > Frankfurt am Main - Reg.-Nr.: HRB 105546 > Vorstand: Matthias Tomann, Marcus Purzer - Aufsichtsratsvorsitzender: > Daniel Gr?zinger > > > -----Urspr?ngliche Nachricht----- > Von: basex-talk-bounces at mailman.uni-konstanz.de > [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von > Bram Vanroy > Gesendet: Dienstag, 15. August 2017 22:22 > An: basex-talk at mailman.uni-konstanz.de > Betreff: [basex-talk] Could not reserve enough space for object heap > > Hi all > > I'm running into an issue with many databases. I.e. one server > instance with millions of databases. When creating all of these, I > found that the more databases are included on the instance, the slower > further database generation got. For instance, I could see in the logs > that in the first < 10.000 databases the creation happened smoothly > with around 50ms per file of 1-4kB. However, when having more and more > databases for this server instance, things got very slow: for an XML > file of 1-4kB the logs show ~600ms. This is terribly slow, as you can imagine. > > At first I thought something was wrong with my hardware, but I checked > on another system and the same issues arises. Then I thought maybe > Java is doing something strange, so I figured I'd reboot and see if > that cleared some stuff up. But now when I try to launch 'basex' or > 'basexserver', I get the following message: > > Could not reserve enough space for 1433600KB object heap > > I googled the issue, and it was suggested that I added a JAVA option > to my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) > indicating the memory it could use. I set that to 2048MB. But still > the same issue persists. > > I have contacted the list before, with issues of generating millions > of database with the same server instance, and this seems another one > related to the problem. I am no expert AT ALL, but isn't it possible > there is some sort of micro memory leak that only becomes apparent > when creating an amount of databases of this magnitude? If not, other > ideas are welcome as well. At least on how to get rid of the Java > error mentioned above. > > > Kind regards > > Bram Vanroy > -- Gerrit Imsieke Gesch?ftsf?hrer / Managing Director le-tex publishing services GmbH Weissenfelser Str. 84, 04229 Leipzig, Germany Phone +49 341 355356 110, Fax +49 341 355356 510 gerrit.imsieke at le-tex.de, http://www.le-tex.de Registergericht / Commercial Register: Amtsgericht Leipzig Registernummer / Registration Number: HRB 24930 Gesch?ftsf?hrer: Gerrit Imsieke, Svea Jelonek, Thomas Schmidt, Dr. Reinhard V?ckler

On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy <Bram.Vanroy at ugent.be> wrote: > Hi all > > I'm running into an issue with many databases. I.e. one server instance with > millions of databases. When creating all of these, I found that the more > databases are included on the instance, the slower further database > generation got. For instance, I could see in the logs that in the first < > 10.000 databases the creation happened smoothly with around 50ms per file of > 1-4kB. However, when having more and more databases for this server > instance, things got very slow: for an XML file of 1-4kB the logs show > ~600ms. This is terribly slow, as you can imagine. > > At first I thought something was wrong with my hardware, but I checked on > another system and the same issues arises. Then I thought maybe Java is > doing something strange, so I figured I'd reboot and see if that cleared > some stuff up. But now when I try to launch 'basex' or 'basexserver', I get > the following message: > > Could not reserve enough space for 1433600KB object heap > > I googled the issue, and it was suggested that I added a JAVA option to my > system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the > memory it could use. I set that to 2048MB. But still the same issue > persists. > > I have contacted the list before, with issues of generating millions of > database with the same server instance, and this seems another one related > to the problem. I am no expert AT ALL, but isn't it possible there is some > sort of micro memory leak that only becomes apparent when creating an amount > of databases of this magnitude? If not, other ideas are welcome as well. At > least on how to get rid of the Java error mentioned above. > > > Kind regards > > Bram Vanroy >

> Hi Bram, > > As was indicated before, the JVM is restricted to appr. 1.4 -1.5, > sometimes 1.6 GB on 32 bit Windows systems. > > Generally (and as you have impressively proven), it is indeed possible > to have millions of databases in a single BaseX instance. In practice, > it is recommendable to find a tradeoff between number of databases and > number of documents/XML nodes per database. If there is no obvious way > how to distribute your documents, an additional mapping database might > be reasonable in order to look up where a document has gone. > > Hope this helps, > Christian > > > On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy <Bram.Vanroy at ugent.be> wrote: >> Hi all >> >> I'm running into an issue with many databases. I.e. one server instance with >> millions of databases. When creating all of these, I found that the more >> databases are included on the instance, the slower further database >> generation got. For instance, I could see in the logs that in the first < >> 10.000 databases the creation happened smoothly with around 50ms per file of >> 1-4kB. However, when having more and more databases for this server >> instance, things got very slow: for an XML file of 1-4kB the logs show >> ~600ms. This is terribly slow, as you can imagine. >> >> At first I thought something was wrong with my hardware, but I checked on >> another system and the same issues arises. Then I thought maybe Java is >> doing something strange, so I figured I'd reboot and see if that cleared >> some stuff up. But now when I try to launch 'basex' or 'basexserver', I get >> the following message: >> >> Could not reserve enough space for 1433600KB object heap >> >> I googled the issue, and it was suggested that I added a JAVA option to my >> system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the >> memory it could use. I set that to 2048MB. But still the same issue >> persists. >> >> I have contacted the list before, with issues of generating millions of >> database with the same server instance, and this seems another one related >> to the problem. I am no expert AT ALL, but isn't it possible there is some >> sort of micro memory leak that only becomes apparent when creating an amount >> of databases of this magnitude? If not, other ideas are welcome as well. At >> least on how to get rid of the Java error mentioned above. >> >> >> Kind regards >> >> Bram Vanroy >>

> I am running 64 bit, a x86-x64 JVM so I'd thought I could assign a larger > amount of memory to it.

> Why does it need that much memory to just launch? Is it > trying to load some thing of all the existing databases in memory?

> Citeren Christian Grün <christian.gruen at gmail.com>: > > >> Hi Bram, >> >> As was indicated before, the JVM is restricted to appr. 1.4 -1.5, >> sometimes 1.6 GB on 32 bit Windows systems. >> >> Generally (and as you have impressively proven), it is indeed possible >> to have millions of databases in a single BaseX instance. In practice, >> it is recommendable to find a tradeoff between number of databases and >> number of documents/XML nodes per database. If there is no obvious way >> how to distribute your documents, an additional mapping database might >> be reasonable in order to look up where a document has gone. >> >> Hope this helps, >> Christian >> >> >> On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy <Bram.Vanroy at ugent.be> >> wrote: >>> >>> Hi all >>> >>> I'm running into an issue with many databases. I.e. one server instance >>> with >>> millions of databases. When creating all of these, I found that the more >>> databases are included on the instance, the slower further database >>> generation got. For instance, I could see in the logs that in the first < >>> 10.000 databases the creation happened smoothly with around 50ms per file >>> of >>> 1-4kB. However, when having more and more databases for this server >>> instance, things got very slow: for an XML file of 1-4kB the logs show >>> ~600ms. This is terribly slow, as you can imagine. >>> >>> At first I thought something was wrong with my hardware, but I checked on >>> another system and the same issues arises. Then I thought maybe Java is >>> doing something strange, so I figured I'd reboot and see if that cleared >>> some stuff up. But now when I try to launch 'basex' or 'basexserver', I >>> get >>> the following message: >>> >>> Could not reserve enough space for 1433600KB object heap >>> >>> I googled the issue, and it was suggested that I added a JAVA option to >>> my >>> system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) indicating the >>> memory it could use. I set that to 2048MB. But still the same issue >>> persists. >>> >>> I have contacted the list before, with issues of generating millions of >>> database with the same server instance, and this seems another one >>> related >>> to the problem. I am no expert AT ALL, but isn't it possible there is >>> some >>> sort of micro memory leak that only becomes apparent when creating an >>> amount >>> of databases of this magnitude? If not, other ideas are welcome as well. >>> At >>> least on how to get rid of the Java error mentioned above. >>> >>> >>> Kind regards >>> >>> Bram Vanroy >>> > > >

Van: Christian Grün [mailto:christian.gruen at gmail.com] Verzonden: woensdag 16 augustus 2017 11:31 Aan: Bram Vanroy <Bram.Vanroy at ugent.be> CC: BaseX <basex-talk at mailman.uni-konstanz.de> Onderwerp: Re: [basex-talk] Could not reserve enough space for object heap > I am running 64 bit, a x86-x64 JVM so I'd thought I could assign a > larger amount of memory to it. Hm. You could try to specify -d64 to ensure that the 64bit JVM is used. > Why does it need that much memory to just launch? Is it trying to load > some thing of all the existing databases in memory? As the error is returned by Java, not by BaseX, it’s most probably returnend before any databases are touched. Maybe it helps to consult some other Java forum pages and compare the configuration with yours? Hope this helps, Christian > Citeren Christian Grün <christian.gruen at gmail.com>: > > >> Hi Bram, >> >> As was indicated before, the JVM is restricted to appr. 1.4 -1.5, >> sometimes 1.6 GB on 32 bit Windows systems. >> >> Generally (and as you have impressively proven), it is indeed >> possible to have millions of databases in a single BaseX instance. In >> practice, it is recommendable to find a tradeoff between number of >> databases and number of documents/XML nodes per database. If there is >> no obvious way how to distribute your documents, an additional >> mapping database might be reasonable in order to look up where a document has gone. >> >> Hope this helps, >> Christian >> >> >> On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy <Bram.Vanroy at ugent.be> >> wrote: >>> >>> Hi all >>> >>> I'm running into an issue with many databases. I.e. one server >>> instance with millions of databases. When creating all of these, I >>> found that the more databases are included on the instance, the >>> slower further database generation got. For instance, I could see in >>> the logs that in the first < >>> 10.000 databases the creation happened smoothly with around 50ms per >>> file of 1-4kB. However, when having more and more databases for this >>> server instance, things got very slow: for an XML file of 1-4kB the >>> logs show ~600ms. This is terribly slow, as you can imagine. >>> >>> At first I thought something was wrong with my hardware, but I >>> checked on another system and the same issues arises. Then I thought >>> maybe Java is doing something strange, so I figured I'd reboot and >>> see if that cleared some stuff up. But now when I try to launch >>> 'basex' or 'basexserver', I get the following message: >>> >>> Could not reserve enough space for 1433600KB object heap >>> >>> I googled the issue, and it was suggested that I added a JAVA option >>> to my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) >>> indicating the memory it could use. I set that to 2048MB. But still >>> the same issue persists. >>> >>> I have contacted the list before, with issues of generating millions >>> of database with the same server instance, and this seems another >>> one related to the problem. I am no expert AT ALL, but isn't it >>> possible there is some sort of micro memory leak that only becomes >>> apparent when creating an amount of databases of this magnitude? If >>> not, other ideas are welcome as well. >>> At >>> least on how to get rid of the Java error mentioned above. >>> >>> >>> Kind regards >>> >>> Bram Vanroy >>> > > >

On Thu, Aug 24, 2017 at 8:46 AM, <bram.vanroy at ugent.be> wrote: > I can confirm that explicitly using a 64 bit JVM does not solve the issue I mentioned in my first email. > > The problem persists; when working with a single server instance the creation of very small databases (~1-5kb) goes smoothly at the beginning but turns into a nightmare when reaching a certain threshold. Let's say around ~1M databases, the time to create subsequent databases increases dramatically. We're talking about a 10x duration increase. > > At first I thought I was doing something wrong, but for the life of me I can't figure out what! The java process doesn't utilize any more or less resources than it does when just starting, so that's not it either. Do you have any information that might help here? Does a server do anything to the databases when creating new ones? E.g. when one creates a new database, is a list of all databases re-indexed or something like that? This could explain the issue. But other inspiration is welcome as well, because I don't know where to look. > > Thanks in advance > > -----Oorspronkelijk bericht----- > Van: Christian Grün [mailto:christian.gruen at gmail.com] > Verzonden: woensdag 16 augustus 2017 11:31 > Aan: Bram Vanroy <Bram.Vanroy at ugent.be> > CC: BaseX <basex-talk at mailman.uni-konstanz.de> > Onderwerp: Re: [basex-talk] Could not reserve enough space for object heap > >> I am running 64 bit, a x86-x64 JVM so I'd thought I could assign a >> larger amount of memory to it. > > Hm. You could try to specify -d64 to ensure that the 64bit JVM is used. > >> Why does it need that much memory to just launch? Is it trying to load >> some thing of all the existing databases in memory? > > As the error is returned by Java, not by BaseX, it’s most probably returnend before any databases are touched. Maybe it helps to consult some other Java forum pages and compare the configuration with yours? > > Hope this helps, > Christian > > > >> Citeren Christian Grün <christian.gruen at gmail.com>: >> >> >>> Hi Bram, >>> >>> As was indicated before, the JVM is restricted to appr. 1.4 -1.5, >>> sometimes 1.6 GB on 32 bit Windows systems. >>> >>> Generally (and as you have impressively proven), it is indeed >>> possible to have millions of databases in a single BaseX instance. In >>> practice, it is recommendable to find a tradeoff between number of >>> databases and number of documents/XML nodes per database. If there is >>> no obvious way how to distribute your documents, an additional >>> mapping database might be reasonable in order to look up where a document has gone. >>> >>> Hope this helps, >>> Christian >>> >>> >>> On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy <Bram.Vanroy at ugent.be> >>> wrote: >>>> >>>> Hi all >>>> >>>> I'm running into an issue with many databases. I.e. one server >>>> instance with millions of databases. When creating all of these, I >>>> found that the more databases are included on the instance, the >>>> slower further database generation got. For instance, I could see in >>>> the logs that in the first < >>>> 10.000 databases the creation happened smoothly with around 50ms per >>>> file of 1-4kB. However, when having more and more databases for this >>>> server instance, things got very slow: for an XML file of 1-4kB the >>>> logs show ~600ms. This is terribly slow, as you can imagine. >>>> >>>> At first I thought something was wrong with my hardware, but I >>>> checked on another system and the same issues arises. Then I thought >>>> maybe Java is doing something strange, so I figured I'd reboot and >>>> see if that cleared some stuff up. But now when I try to launch >>>> 'basex' or 'basexserver', I get the following message: >>>> >>>> Could not reserve enough space for 1433600KB object heap >>>> >>>> I googled the issue, and it was suggested that I added a JAVA option >>>> to my system's variable (I'm on Windows 10 64 bit, BaseX 8.6.4) >>>> indicating the memory it could use. I set that to 2048MB. But still >>>> the same issue persists. >>>> >>>> I have contacted the list before, with issues of generating millions >>>> of database with the same server instance, and this seems another >>>> one related to the problem. I am no expert AT ALL, but isn't it >>>> possible there is some sort of micro memory leak that only becomes >>>> apparent when creating an amount of databases of this magnitude? If >>>> not, other ideas are welcome as well. >>>> At >>>> least on how to get rid of the Java error mentioned above. >>>> >>>> >>>> Kind regards >>>> >>>> Bram Vanroy >>>> >> >> >> >

Van: Christian Grün [mailto:christian.gruen at gmail.com] Verzonden: donderdag 24 augustus 2017 10:19 Aan: Bram Vanroy <bram.vanroy at ugent.be> CC: BaseX <basex-talk at mailman.uni-konstanz.de> Onderwerp: Re: [basex-talk] Could not reserve enough space for object heap Hi Bram, As indicated in a previous reply, BaseX was not optimized to organize such a great number of databases. The best solution will be to distribute your documents to a smaller number of databases. How many documents are stored in average in your databases, and how large are the documents? Cheers, C. On Thu, Aug 24, 2017 at 8:46 AM, <bram.vanroy at ugent.be> wrote: > I can confirm that explicitly using a 64 bit JVM does not solve the issue I mentioned in my first email. > > The problem persists; when working with a single server instance the creation of very small databases (~1-5kb) goes smoothly at the beginning but turns into a nightmare when reaching a certain threshold. Let's say around ~1M databases, the time to create subsequent databases increases dramatically. We're talking about a 10x duration increase. > > At first I thought I was doing something wrong, but for the life of me I can't figure out what! The java process doesn't utilize any more or less resources than it does when just starting, so that's not it either. Do you have any information that might help here? Does a server do anything to the databases when creating new ones? E.g. when one creates a new database, is a list of all databases re-indexed or something like that? This could explain the issue. But other inspiration is welcome as well, because I don't know where to look. > > Thanks in advance > > -----Oorspronkelijk bericht----- > Van: Christian Grün [mailto:christian.gruen at gmail.com] > Verzonden: woensdag 16 augustus 2017 11:31 > Aan: Bram Vanroy <Bram.Vanroy at ugent.be> > CC: BaseX <basex-talk at mailman.uni-konstanz.de> > Onderwerp: Re: [basex-talk] Could not reserve enough space for object > heap > >> I am running 64 bit, a x86-x64 JVM so I'd thought I could assign a >> larger amount of memory to it. > > Hm. You could try to specify -d64 to ensure that the 64bit JVM is used. > >> Why does it need that much memory to just launch? Is it trying to >> load some thing of all the existing databases in memory? > > As the error is returned by Java, not by BaseX, it’s most probably returnend before any databases are touched. Maybe it helps to consult some other Java forum pages and compare the configuration with yours? > > Hope this helps, > Christian > > > >> Citeren Christian Grün <christian.gruen at gmail.com>: >> >> >>> Hi Bram, >>> >>> As was indicated before, the JVM is restricted to appr. 1.4 -1.5, >>> sometimes 1.6 GB on 32 bit Windows systems. >>> >>> Generally (and as you have impressively proven), it is indeed >>> possible to have millions of databases in a single BaseX instance. >>> In practice, it is recommendable to find a tradeoff between number >>> of databases and number of documents/XML nodes per database. If >>> there is no obvious way how to distribute your documents, an >>> additional mapping database might be reasonable in order to look up where a document has gone. >>> >>> Hope this helps, >>> Christian >>> >>> >>> On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy <Bram.Vanroy at ugent.be> >>> wrote: >>>> >>>> Hi all >>>> >>>> I'm running into an issue with many databases. I.e. one server >>>> instance with millions of databases. When creating all of these, I >>>> found that the more databases are included on the instance, the >>>> slower further database generation got. For instance, I could see >>>> in the logs that in the first < >>>> 10.000 databases the creation happened smoothly with around 50ms >>>> per file of 1-4kB. However, when having more and more databases for >>>> this server instance, things got very slow: for an XML file of >>>> 1-4kB the logs show ~600ms. This is terribly slow, as you can imagine. >>>> >>>> At first I thought something was wrong with my hardware, but I >>>> checked on another system and the same issues arises. Then I >>>> thought maybe Java is doing something strange, so I figured I'd >>>> reboot and see if that cleared some stuff up. But now when I try to >>>> launch 'basex' or 'basexserver', I get the following message: >>>> >>>> Could not reserve enough space for 1433600KB object heap >>>> >>>> I googled the issue, and it was suggested that I added a JAVA >>>> option to my system's variable (I'm on Windows 10 64 bit, BaseX >>>> 8.6.4) indicating the memory it could use. I set that to 2048MB. >>>> But still the same issue persists. >>>> >>>> I have contacted the list before, with issues of generating >>>> millions of database with the same server instance, and this seems >>>> another one related to the problem. I am no expert AT ALL, but >>>> isn't it possible there is some sort of micro memory leak that only >>>> becomes apparent when creating an amount of databases of this >>>> magnitude? If not, other ideas are welcome as well. >>>> At >>>> least on how to get rid of the Java error mentioned above. >>>> >>>> >>>> Kind regards >>>> >>>> Bram Vanroy >>>> >> >> >> >

-----Ursprüngliche Nachricht----- Von: basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] Im Auftrag von bram.vanroy at ugent.be Gesendet: Donnerstag, 24. August 2017 23:45 An: 'Christian Grün' <christian.gruen at gmail.com> Cc: 'BaseX' <basex-talk at mailman.uni-konstanz.de> Betreff: Re: [basex-talk] Could not reserve enough space for object heap Hi Christian I am aware that you indicated that BaseX was not intended nor optimised for having so many databases, but (out of curiosity and interest) I am asking why a server instance has difficulty with so many databases. What is it that the server does when creating a new database, that causes it to be so slow when dealing with many databases? As I said before, it sounds as if it does some sort of sorting or indexing on EACH creation of a new database, which makes the import slow? Or am I wrong? If so, what is it exactly that a BaseX server instance does to its existing databases when it creates a new database? In our case, each database only consists of a single document, a very small XML file of < 1 MB. There are millions of these databases. Don't get me wrong, I definitely agree that we should look for a better way to organize our data. Just to get an idea, is there a 'limit' of the amount of documents you'd add to a database to allow for good performance? In other words, would it be fruitful if we could split our millions of databases in a much smaller amount of databases with many documents per database? E.g. let's say we now have one million databases (one tiny XML file per db), would it seem a better idea to you to instead have only 25 databases, with 40.000 tiny XML documents in each? Or is that question hard to answer without real data? Thanks for your patience Bram -----Oorspronkelijk bericht----- Van: Christian Grün [mailto:christian.gruen at gmail.com] Verzonden: donderdag 24 augustus 2017 10:19 Aan: Bram Vanroy <bram.vanroy at ugent.be> CC: BaseX <basex-talk at mailman.uni-konstanz.de> Onderwerp: Re: [basex-talk] Could not reserve enough space for object heap Hi Bram, As indicated in a previous reply, BaseX was not optimized to organize such a great number of databases. The best solution will be to distribute your documents to a smaller number of databases. How many documents are stored in average in your databases, and how large are the documents? Cheers, C. On Thu, Aug 24, 2017 at 8:46 AM, <bram.vanroy at ugent.be> wrote: > I can confirm that explicitly using a 64 bit JVM does not solve the issue I mentioned in my first email. > > The problem persists; when working with a single server instance the creation of very small databases (~1-5kb) goes smoothly at the beginning but turns into a nightmare when reaching a certain threshold. Let's say around ~1M databases, the time to create subsequent databases increases dramatically. We're talking about a 10x duration increase. > > At first I thought I was doing something wrong, but for the life of me I can't figure out what! The java process doesn't utilize any more or less resources than it does when just starting, so that's not it either. Do you have any information that might help here? Does a server do anything to the databases when creating new ones? E.g. when one creates a new database, is a list of all databases re-indexed or something like that? This could explain the issue. But other inspiration is welcome as well, because I don't know where to look. > > Thanks in advance > > -----Oorspronkelijk bericht----- > Van: Christian Grün [mailto:christian.gruen at gmail.com] > Verzonden: woensdag 16 augustus 2017 11:31 > Aan: Bram Vanroy <Bram.Vanroy at ugent.be> > CC: BaseX <basex-talk at mailman.uni-konstanz.de> > Onderwerp: Re: [basex-talk] Could not reserve enough space for object > heap > >> I am running 64 bit, a x86-x64 JVM so I'd thought I could assign a >> larger amount of memory to it. > > Hm. You could try to specify -d64 to ensure that the 64bit JVM is used. > >> Why does it need that much memory to just launch? Is it trying to >> load some thing of all the existing databases in memory? > > As the error is returned by Java, not by BaseX, it's most probably returnend before any databases are touched. Maybe it helps to consult some other Java forum pages and compare the configuration with yours? > > Hope this helps, > Christian > > > >> Citeren Christian Grün <christian.gruen at gmail.com>: >> >> >>> Hi Bram, >>> >>> As was indicated before, the JVM is restricted to appr. 1.4 -1.5, >>> sometimes 1.6 GB on 32 bit Windows systems. >>> >>> Generally (and as you have impressively proven), it is indeed >>> possible to have millions of databases in a single BaseX instance. >>> In practice, it is recommendable to find a tradeoff between number >>> of databases and number of documents/XML nodes per database. If >>> there is no obvious way how to distribute your documents, an >>> additional mapping database might be reasonable in order to look up where a document has gone. >>> >>> Hope this helps, >>> Christian >>> >>> >>> On Tue, Aug 15, 2017 at 10:22 PM, Bram Vanroy <Bram.Vanroy at ugent.be> >>> wrote: >>>> >>>> Hi all >>>> >>>> I'm running into an issue with many databases. I.e. one server >>>> instance with millions of databases. When creating all of these, I >>>> found that the more databases are included on the instance, the >>>> slower further database generation got. For instance, I could see >>>> in the logs that in the first < >>>> 10.000 databases the creation happened smoothly with around 50ms >>>> per file of 1-4kB. However, when having more and more databases for >>>> this server instance, things got very slow: for an XML file of >>>> 1-4kB the logs show ~600ms. This is terribly slow, as you can imagine. >>>> >>>> At first I thought something was wrong with my hardware, but I >>>> checked on another system and the same issues arises. Then I >>>> thought maybe Java is doing something strange, so I figured I'd >>>> reboot and see if that cleared some stuff up. But now when I try to >>>> launch 'basex' or 'basexserver', I get the following message: >>>> >>>> Could not reserve enough space for 1433600KB object heap >>>> >>>> I googled the issue, and it was suggested that I added a JAVA >>>> option to my system's variable (I'm on Windows 10 64 bit, BaseX >>>> 8.6.4) indicating the memory it could use. I set that to 2048MB. >>>> But still the same issue persists. >>>> >>>> I have contacted the list before, with issues of generating >>>> millions of database with the same server instance, and this seems >>>> another one related to the problem. I am no expert AT ALL, but >>>> isn't it possible there is some sort of micro memory leak that only >>>> becomes apparent when creating an amount of databases of this >>>> magnitude? If not, other ideas are welcome as well. >>>> At >>>> least on how to get rid of the Java error mentioned above. >>>> >>>> >>>> Kind regards >>>> >>>> Bram Vanroy >>>> >> >> >> >