Performance tuning

On Tue, Feb 25, 2025 at 2:10 PM Csaba Fekete <feketecsaba at gmail.com> wrote: > Hi > I have a web server that runs Basex 11.1. The server is a VPS with 18G of > RAM. > I have a directory of documents in various sizes, ranging from a few > kilobytes up to 2G. > I am trying to import these documents with the command > CREATE DB mydb /path/to/docs > With the default jvm max heap size (2GB) I get the error: Out of main > memory > If I raise the max heap size to 4GB, I get the same error. > If I raise it to 8GB, the system becomes unresponsive. > How can I determine how much system memory I need to be able to carry out > this task? > Thanks >

Csaba Fekete <feketecsaba at gmail.com> schrieb am Do., 27. Feb. 2025, 17:36: > Hi Christian > Sorry, I thought I was sending this to the mailing list. Thanks for > answering anyway! > Now I'm trying with a smaller dataset and I am adding the documents one by > one. I also upgraded BaseX to the latest version. > The largest document is 1151M in size and it can't be imported, even if I > use attrindex and textindex. > The file is actually publicly available: > http://taurusreisen.hu/partner/v2/SPANYOLORSZAG.zip > Here is my command and the output: > /opt/basex/bin/basex -Oattrindex=true -Otextindex=true -v -V -c"OPEN > taurus; ADD ./SPANYOLORSZÁG.xml" > Database 'taurus' was opened in 18.21 ms. > Out of Main Memory. > I am thinking of solving the problem by splitting the file to several > chunks, which will be CPU-demanding but could make it work. > Any ideas are welcome. > Thank you again, and a million thanks for BaseX! It is a fantastic tool. > Regards, > Csaba > > On Thu, 27 Feb 2025 at 15:52, Christian Grün <christian.gruen at gmail.com> > wrote: > >> Hi Csaba, >> >> It’s difficult to give a general advice; XML documents are just too >> different. In principle, a few GB or even MB can be sufficient to create >> databases for very large collections (10 GB and more), but sometimes >> namespaces are a showstopper. See [1] for some statistics. >> >> What’s the total size of your XML documents? Can you create the database >> if you enable the text and attribute index? >> >> Best, >> Christian >> >> [1] https://docs.basex.org/main/Statistics >> >> >> >> >> On Tue, Feb 25, 2025 at 2:10 PM Csaba Fekete <feketecsaba at gmail.com> >> wrote: >> >>> Hi >>> I have a web server that runs Basex 11.1. The server is a VPS with 18G >>> of RAM. >>> I have a directory of documents in various sizes, ranging from a few >>> kilobytes up to 2G. >>> I am trying to import these documents with the command >>> CREATE DB mydb /path/to/docs >>> With the default jvm max heap size (2GB) I get the error: Out of main >>> memory >>> If I raise the max heap size to 4GB, I get the same error. >>> If I raise it to 8GB, the system becomes unresponsive. >>> How can I determine how much system memory I need to be able to carry >>> out this task? >>> Thanks >>> >>

On Thu, 27 Feb 2025 at 17:43, Christian Grün <christian.gruen at gmail.com> wrote: > Just some quick feedback: Does it work if you specify the input along with > CREATE DB? > > basex -c"CREATE DB taurus SPANYOLORSZÁG.xml" > > You can also specify a directory as input. > > Thanks, > Christian > > > > Csaba Fekete <feketecsaba at gmail.com> schrieb am Do., 27. Feb. 2025, 17:36: > >> Hi Christian >> Sorry, I thought I was sending this to the mailing list. Thanks for >> answering anyway! >> Now I'm trying with a smaller dataset and I am adding the documents one >> by one. I also upgraded BaseX to the latest version. >> The largest document is 1151M in size and it can't be imported, even if I >> use attrindex and textindex. >> The file is actually publicly available: >> http://taurusreisen.hu/partner/v2/SPANYOLORSZAG.zip >> Here is my command and the output: >> /opt/basex/bin/basex -Oattrindex=true -Otextindex=true -v -V -c"OPEN >> taurus; ADD ./SPANYOLORSZÁG.xml" >> Database 'taurus' was opened in 18.21 ms. >> Out of Main Memory. >> I am thinking of solving the problem by splitting the file to several >> chunks, which will be CPU-demanding but could make it work. >> Any ideas are welcome. >> Thank you again, and a million thanks for BaseX! It is a fantastic tool. >> Regards, >> Csaba >> >> On Thu, 27 Feb 2025 at 15:52, Christian Grün <christian.gruen at gmail.com> >> wrote: >> >>> Hi Csaba, >>> >>> It’s difficult to give a general advice; XML documents are just too >>> different. In principle, a few GB or even MB can be sufficient to create >>> databases for very large collections (10 GB and more), but sometimes >>> namespaces are a showstopper. See [1] for some statistics. >>> >>> What’s the total size of your XML documents? Can you create the database >>> if you enable the text and attribute index? >>> >>> Best, >>> Christian >>> >>> [1] https://docs.basex.org/main/Statistics >>> >>> >>> >>> >>> On Tue, Feb 25, 2025 at 2:10 PM Csaba Fekete <feketecsaba at gmail.com> >>> wrote: >>> >>>> Hi >>>> I have a web server that runs Basex 11.1. The server is a VPS with 18G >>>> of RAM. >>>> I have a directory of documents in various sizes, ranging from a few >>>> kilobytes up to 2G. >>>> I am trying to import these documents with the command >>>> CREATE DB mydb /path/to/docs >>>> With the default jvm max heap size (2GB) I get the error: Out of main >>>> memory >>>> If I raise the max heap size to 4GB, I get the same error. >>>> If I raise it to 8GB, the system becomes unresponsive. >>>> How can I determine how much system memory I need to be able to carry >>>> out this task? >>>> Thanks >>>> >>>

On Thu, 27 Feb 2025 at 18:25, Csaba Fekete <feketecsaba at gmail.com> wrote: > Yeah, I get the same error using this command, too. > Thanks > > On Thu, 27 Feb 2025 at 17:43, Christian Grün <christian.gruen at gmail.com> > wrote: > >> Just some quick feedback: Does it work if you specify the input along >> with CREATE DB? >> >> basex -c"CREATE DB taurus SPANYOLORSZÁG.xml" >> >> You can also specify a directory as input. >> >> Thanks, >> Christian >> >> >> >> Csaba Fekete <feketecsaba at gmail.com> schrieb am Do., 27. Feb. 2025, >> 17:36: >> >>> Hi Christian >>> Sorry, I thought I was sending this to the mailing list. Thanks for >>> answering anyway! >>> Now I'm trying with a smaller dataset and I am adding the documents one >>> by one. I also upgraded BaseX to the latest version. >>> The largest document is 1151M in size and it can't be imported, even if >>> I use attrindex and textindex. >>> The file is actually publicly available: >>> http://taurusreisen.hu/partner/v2/SPANYOLORSZAG.zip >>> Here is my command and the output: >>> /opt/basex/bin/basex -Oattrindex=true -Otextindex=true -v -V -c"OPEN >>> taurus; ADD ./SPANYOLORSZÁG.xml" >>> Database 'taurus' was opened in 18.21 ms. >>> Out of Main Memory. >>> I am thinking of solving the problem by splitting the file to several >>> chunks, which will be CPU-demanding but could make it work. >>> Any ideas are welcome. >>> Thank you again, and a million thanks for BaseX! It is a fantastic tool. >>> Regards, >>> Csaba >>> >>> On Thu, 27 Feb 2025 at 15:52, Christian Grün <christian.gruen at gmail.com> >>> wrote: >>> >>>> Hi Csaba, >>>> >>>> It’s difficult to give a general advice; XML documents are just too >>>> different. In principle, a few GB or even MB can be sufficient to create >>>> databases for very large collections (10 GB and more), but sometimes >>>> namespaces are a showstopper. See [1] for some statistics. >>>> >>>> What’s the total size of your XML documents? Can you create the >>>> database if you enable the text and attribute index? >>>> >>>> Best, >>>> Christian >>>> >>>> [1] https://docs.basex.org/main/Statistics >>>> >>>> >>>> >>>> >>>> On Tue, Feb 25, 2025 at 2:10 PM Csaba Fekete <feketecsaba at gmail.com> >>>> wrote: >>>> >>>>> Hi >>>>> I have a web server that runs Basex 11.1. The server is a VPS with 18G >>>>> of RAM. >>>>> I have a directory of documents in various sizes, ranging from a few >>>>> kilobytes up to 2G. >>>>> I am trying to import these documents with the command >>>>> CREATE DB mydb /path/to/docs >>>>> With the default jvm max heap size (2GB) I get the error: Out of main >>>>> memory >>>>> If I raise the max heap size to 4GB, I get the same error. >>>>> If I raise it to 8GB, the system becomes unresponsive. >>>>> How can I determine how much system memory I need to be able to carry >>>>> out this task? >>>>> Thanks >>>>> >>>>

On Sun, 2 Mar 2025 at 15:33, Christian Grün <christian.gruen at gmail.com> wrote: > Hi Csaba (cc to the list), > > You already found the reason for the excessive memory consumption by > yourself. If you don’t need to rely on the pathological element names, you > can replace them with a simple <i/> element, e.g. as follows: > > let $file := file:base-dir() || 'SPANYOLORSZÁG.xml' > return file:write-text-lines( > 'normalized-' || $file, > file:read-text-lines($file) ! replace(., '<(/?)i\d+>', '<$1i>') > ) > > This will also reduce memory consumption quite a lot. > > It would be a considerable effort to change the limit for element and > attribute names, and it would also increase the database size for ordinary > XML input, which is it’s improbable we will touch this. However, a > relatively easy exercise would be to output error messages once the limits > are exceeded, and not at the very end. > > Best, > Christian > >