Out of Main Memory

On Tue, Mar 12, 2024 at 8:34 PM Murray, Gregory <gregory.murray at ptsem.edu> wrote: > Hello, > > > > I’m working with a database that has a full-text index. I have found that > if I iteratively add XML documents, then optimize, add more documents, > optimize again, and so on, eventually the “optimize” command will fail with > “Out of Main Memory.” I edited the basex startup script to change the > memory allocation from -Xmx2g to -Xmx12g. My computer has 16 GB of memory, > but of course the OS uses up some of it. I have found that if I exit > memory-hungry programs (web browser, Oxygen), start basex, and then run the > “optimize” command, I still get “Out of Main Memory.” I’m wondering if > there are any known workarounds or strategies for this situation. If I > understand the documentation about indexes correctly, index data is > periodically written to disk during optimization. Does this mean that > running optimize again will pick up where the previous attempt left off, > such that running optimize repeatedly will eventually succeed? > > > > Thanks, > > Greg > > > > > > Gregory Murray > > Director of Digital Initiatives > > Wright Library > > Princeton Theological Seminary > > > > >

From: Christian Grün <christian.gruen at gmail.com> Date: Thursday, March 14, 2024 at 8:48 AM To: Murray, Gregory <gregory.murray at ptsem.edu> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Out of Main Memory Hi Greg, A quick reply: If only parts of your documents are relevant for full-text queries, you can restrict the selection with the FTINDEX option (see [1] for more information). How large is the total size of your input documents? Best, Christian [1] https://docs.basex.org/wiki/Indexes#Selective_Indexing On Tue, Mar 12, 2024 at 8:34 PM Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> wrote: Hello, I’m working with a database that has a full-text index. I have found that if I iteratively add XML documents, then optimize, add more documents, optimize again, and so on, eventually the “optimize” command will fail with “Out of Main Memory.” I edited the basex startup script to change the memory allocation from -Xmx2g to -Xmx12g. My computer has 16 GB of memory, but of course the OS uses up some of it. I have found that if I exit memory-hungry programs (web browser, Oxygen), start basex, and then run the “optimize” command, I still get “Out of Main Memory.” I’m wondering if there are any known workarounds or strategies for this situation. If I understand the documentation about indexes correctly, index data is periodically written to disk during optimization. Does this mean that running optimize again will pick up where the previous attempt left off, such that running optimize repeatedly will eventually succeed? Thanks, Greg Gregory Murray Director of Digital Initiatives Wright Library Princeton Theological Seminary

On Thu, Mar 14, 2024 at 9:55 PM Murray, Gregory <gregory.murray at ptsem.edu> wrote: > Thanks, Christian. I don’t think selective indexing is applicable in my > use case, because I need to perform full-text searches on the entirety of > each document. Each XML document represents a physical book that was > digitized, and the structure of each document is essentially a header with > metadata and a body with the OCR text of the book. The OCR text is split > into pages, where one <page> element contains all the words from one > corresponding printed page from the physical book. Obviously the number of > words in each <page> varies widely based on the physical dimensions of the > book and the typeface. > > > > So far, I have loaded 12,331 documents, containing a total of 2,196,771 > pages. The total size of those XML documents on disk is 4.7GB. But that is > only a fraction of the total number of documents I want to load into BaseX. > The total number is more like 160,000 documents. Assuming that the > documents I’ve loaded so far are a representative sample, and I believe > that’s true, then the total size of the XML documents on disk, prior to > loading them into BaseX, would be about 4.7GB * 13 = 61.1GB. > > > > Normally the OCR text, once loaded, almost never changes. But the metadata > fields do change as corrections are made. Also we add more XML documents > routinely as we digitize more books over time. Therefore updates and > additions are commonplace, such that keeping indexes up to date is > important, to allow full-text searches to stay performant. I’m wondering if > there are techniques for optimizing such quantities of text. > > > > Thanks, > > Greg > > > > *From: *Christian Grün <christian.gruen at gmail.com> > *Date: *Thursday, March 14, 2024 at 8:48 AM > *To: *Murray, Gregory <gregory.murray at ptsem.edu> > *Cc: *basex-talk at mailman.uni-konstanz.de < > basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Out of Main Memory > > Hi Greg, > > > > A quick reply: If only parts of your documents are relevant for full-text > queries, you can restrict the selection with the FTINDEX option (see [1] > for more information). > > > > How large is the total size of your input documents? > > > > Best, > > Christian > > > > [1] https://docs.basex.org/wiki/Indexes#Selective_Indexing > > > > > > > > On Tue, Mar 12, 2024 at 8:34 PM Murray, Gregory <gregory.murray at ptsem.edu> > wrote: > > Hello, > > > > I’m working with a database that has a full-text index. I have found that > if I iteratively add XML documents, then optimize, add more documents, > optimize again, and so on, eventually the “optimize” command will fail with > “Out of Main Memory.” I edited the basex startup script to change the > memory allocation from -Xmx2g to -Xmx12g. My computer has 16 GB of memory, > but of course the OS uses up some of it. I have found that if I exit > memory-hungry programs (web browser, Oxygen), start basex, and then run the > “optimize” command, I still get “Out of Main Memory.” I’m wondering if > there are any known workarounds or strategies for this situation. If I > understand the documentation about indexes correctly, index data is > periodically written to disk during optimization. Does this mean that > running optimize again will pick up where the previous attempt left off, > such that running optimize repeatedly will eventually succeed? > > > > Thanks, > > Greg > > > > > > Gregory Murray > > Director of Digital Initiatives > > Wright Library > > Princeton Theological Seminary > > > > > >

From: Bridger Dyson-Smith <bdysonsmith at gmail.com> Date: Thursday, March 14, 2024 at 6:43 PM To: Murray, Gregory <gregory.murray at ptsem.edu> Cc: Christian Grün <christian.gruen at gmail.com>, basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Out of Main Memory You don't often get email from bdysonsmith at gmail.com. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hi Greg, Have you tried experimenting with the ADDCACHE[1] option when building your database? While it's been a bit, I recall having good results with, especially in a RAM-constrained environment. Hope that's helpful! Best, Bridger [1] https://docs.basex.org/wiki/Options#ADDCACHE On Thu, Mar 14, 2024 at 9:55 PM Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> wrote: Thanks, Christian. I don’t think selective indexing is applicable in my use case, because I need to perform full-text searches on the entirety of each document. Each XML document represents a physical book that was digitized, and the structure of each document is essentially a header with metadata and a body with the OCR text of the book. The OCR text is split into pages, where one <page> element contains all the words from one corresponding printed page from the physical book. Obviously the number of words in each <page> varies widely based on the physical dimensions of the book and the typeface. So far, I have loaded 12,331 documents, containing a total of 2,196,771 pages. The total size of those XML documents on disk is 4.7GB. But that is only a fraction of the total number of documents I want to load into BaseX. The total number is more like 160,000 documents. Assuming that the documents I’ve loaded so far are a representative sample, and I believe that’s true, then the total size of the XML documents on disk, prior to loading them into BaseX, would be about 4.7GB * 13 = 61.1GB. Normally the OCR text, once loaded, almost never changes. But the metadata fields do change as corrections are made. Also we add more XML documents routinely as we digitize more books over time. Therefore updates and additions are commonplace, such that keeping indexes up to date is important, to allow full-text searches to stay performant. I’m wondering if there are techniques for optimizing such quantities of text. Thanks, Greg From: Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> Date: Thursday, March 14, 2024 at 8:48 AM To: Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> Cc: basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Out of Main Memory Hi Greg, A quick reply: If only parts of your documents are relevant for full-text queries, you can restrict the selection with the FTINDEX option (see [1] for more information). How large is the total size of your input documents? Best, Christian [1] https://docs.basex.org/wiki/Indexes#Selective_Indexing On Tue, Mar 12, 2024 at 8:34 PM Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> wrote: Hello, I’m working with a database that has a full-text index. I have found that if I iteratively add XML documents, then optimize, add more documents, optimize again, and so on, eventually the “optimize” command will fail with “Out of Main Memory.” I edited the basex startup script to change the memory allocation from -Xmx2g to -Xmx12g. My computer has 16 GB of memory, but of course the OS uses up some of it. I have found that if I exit memory-hungry programs (web browser, Oxygen), start basex, and then run the “optimize” command, I still get “Out of Main Memory.” I’m wondering if there are any known workarounds or strategies for this situation. If I understand the documentation about indexes correctly, index data is periodically written to disk during optimization. Does this mean that running optimize again will pick up where the previous attempt left off, such that running optimize repeatedly will eventually succeed? Thanks, Greg Gregory Murray Director of Digital Initiatives Wright Library Princeton Theological Seminary

From: BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on behalf of Murray, Gregory <gregory.murray at ptsem.edu> Date: Thursday, March 14, 2024 at 5:55 PM To: Christian Grün <christian.gruen at gmail.com> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Out of Main Memory Thanks, Christian. I don’t think selective indexing is applicable in my use case, because I need to perform full-text searches on the entirety of each document. Each XML document represents a physical book that was digitized, and the structure of each document is essentially a header with metadata and a body with the OCR text of the book. The OCR text is split into pages, where one <page> element contains all the words from one corresponding printed page from the physical book. Obviously the number of words in each <page> varies widely based on the physical dimensions of the book and the typeface. So far, I have loaded 12,331 documents, containing a total of 2,196,771 pages. The total size of those XML documents on disk is 4.7GB. But that is only a fraction of the total number of documents I want to load into BaseX. The total number is more like 160,000 documents. Assuming that the documents I’ve loaded so far are a representative sample, and I believe that’s true, then the total size of the XML documents on disk, prior to loading them into BaseX, would be about 4.7GB * 13 = 61.1GB. Normally the OCR text, once loaded, almost never changes. But the metadata fields do change as corrections are made. Also we add more XML documents routinely as we digitize more books over time. Therefore updates and additions are commonplace, such that keeping indexes up to date is important, to allow full-text searches to stay performant. I’m wondering if there are techniques for optimizing such quantities of text. Thanks, Greg From: Christian Grün <christian.gruen at gmail.com> Date: Thursday, March 14, 2024 at 8:48 AM To: Murray, Gregory <gregory.murray at ptsem.edu> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Out of Main Memory Hi Greg, A quick reply: If only parts of your documents are relevant for full-text queries, you can restrict the selection with the FTINDEX option (see [1] for more information). How large is the total size of your input documents? Best, Christian [1] https://docs.basex.org/wiki/Indexes#Selective_Indexing On Tue, Mar 12, 2024 at 8:34 PM Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> wrote: Hello, I’m working with a database that has a full-text index. I have found that if I iteratively add XML documents, then optimize, add more documents, optimize again, and so on, eventually the “optimize” command will fail with “Out of Main Memory.” I edited the basex startup script to change the memory allocation from -Xmx2g to -Xmx12g. My computer has 16 GB of memory, but of course the OS uses up some of it. I have found that if I exit memory-hungry programs (web browser, Oxygen), start basex, and then run the “optimize” command, I still get “Out of Main Memory.” I’m wondering if there are any known workarounds or strategies for this situation. If I understand the documentation about indexes correctly, index data is periodically written to disk during optimization. Does this mean that running optimize again will pick up where the previous attempt left off, such that running optimize repeatedly will eventually succeed? Thanks, Greg Gregory Murray Director of Digital Initiatives Wright Library Princeton Theological Seminary

On Thu, Mar 14, 2024 at 10:58 PM Murray, Gregory <gregory.murray at ptsem.edu> wrote: > Thanks, Christian. I don’t think selective indexing is applicable in my > use case, because I need to perform full-text searches on the entirety of > each document. Each XML document represents a physical book that was > digitized, and the structure of each document is essentially a header with > metadata and a body with the OCR text of the book. The OCR text is split > into pages, where one <page> element contains all the words from one > corresponding printed page from the physical book. Obviously the number of > words in each <page> varies widely based on the physical dimensions of the > book and the typeface. > > > > So far, I have loaded 12,331 documents, containing a total of 2,196,771 > pages. The total size of those XML documents on disk is 4.7GB. But that is > only a fraction of the total number of documents I want to load into BaseX. > The total number is more like 160,000 documents. Assuming that the > documents I’ve loaded so far are a representative sample, and I believe > that’s true, then the total size of the XML documents on disk, prior to > loading them into BaseX, would be about 4.7GB * 13 = 61.1GB. > > > > Normally the OCR text, once loaded, almost never changes. But the metadata > fields do change as corrections are made. Also we add more XML documents > routinely as we digitize more books over time. Therefore updates and > additions are commonplace, such that keeping indexes up to date is > important, to allow full-text searches to stay performant. I’m wondering if > there are techniques for optimizing such quantities of text. > > > > Thanks, > > Greg > > > > *From: *Christian Grün <christian.gruen at gmail.com> > *Date: *Thursday, March 14, 2024 at 8:48 AM > *To: *Murray, Gregory <gregory.murray at ptsem.edu> > *Cc: *basex-talk at mailman.uni-konstanz.de < > basex-talk at mailman.uni-konstanz.de> > *Subject: *Re: [basex-talk] Out of Main Memory > > Hi Greg, > > > > A quick reply: If only parts of your documents are relevant for full-text > queries, you can restrict the selection with the FTINDEX option (see [1] > for more information). > > > > How large is the total size of your input documents? > > > > Best, > > Christian > > > > [1] https://docs.basex.org/wiki/Indexes#Selective_Indexing > > > > > > > > On Tue, Mar 12, 2024 at 8:34 PM Murray, Gregory <gregory.murray at ptsem.edu> > wrote: > > Hello, > > > > I’m working with a database that has a full-text index. I have found that > if I iteratively add XML documents, then optimize, add more documents, > optimize again, and so on, eventually the “optimize” command will fail with > “Out of Main Memory.” I edited the basex startup script to change the > memory allocation from -Xmx2g to -Xmx12g. My computer has 16 GB of memory, > but of course the OS uses up some of it. I have found that if I exit > memory-hungry programs (web browser, Oxygen), start basex, and then run the > “optimize” command, I still get “Out of Main Memory.” I’m wondering if > there are any known workarounds or strategies for this situation. If I > understand the documentation about indexes correctly, index data is > periodically written to disk during optimization. Does this mean that > running optimize again will pick up where the previous attempt left off, > such that running optimize repeatedly will eventually succeed? > > > > Thanks, > > Greg > > > > > > Gregory Murray > > Director of Digital Initiatives > > Wright Library > > Princeton Theological Seminary > > > > > >

From: Christian Grün <christian.gruen at gmail.com> Date: Friday, March 15, 2024 at 11:51 AM To: Murray, Gregory <gregory.murray at ptsem.edu> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Out of Main Memory Hi Greg, I would have guessed that 12 GB is enough for 4.7 GB; but it sometimes depends on the input. If you like, you can share a single typical document with us, and we can have a look at it. 61 GB will be too large for a complete full-text index, though. However, it’s always possible to distribute documets across multiple databases and access them with a single query [1]. The full-text index is not incremental (in opposition to the other index structures), which means it must be re-created it after updates. However, it’s possible to re-index updated database instances and query fully indexed databases at the same time. Hope this helps, Christian [1] https://docs.basex.org/wiki/Databases On Thu, Mar 14, 2024 at 10:58 PM Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> wrote: Thanks, Christian. I don’t think selective indexing is applicable in my use case, because I need to perform full-text searches on the entirety of each document. Each XML document represents a physical book that was digitized, and the structure of each document is essentially a header with metadata and a body with the OCR text of the book. The OCR text is split into pages, where one <page> element contains all the words from one corresponding printed page from the physical book. Obviously the number of words in each <page> varies widely based on the physical dimensions of the book and the typeface. So far, I have loaded 12,331 documents, containing a total of 2,196,771 pages. The total size of those XML documents on disk is 4.7GB. But that is only a fraction of the total number of documents I want to load into BaseX. The total number is more like 160,000 documents. Assuming that the documents I’ve loaded so far are a representative sample, and I believe that’s true, then the total size of the XML documents on disk, prior to loading them into BaseX, would be about 4.7GB * 13 = 61.1GB. Normally the OCR text, once loaded, almost never changes. But the metadata fields do change as corrections are made. Also we add more XML documents routinely as we digitize more books over time. Therefore updates and additions are commonplace, such that keeping indexes up to date is important, to allow full-text searches to stay performant. I’m wondering if there are techniques for optimizing such quantities of text. Thanks, Greg From: Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> Date: Thursday, March 14, 2024 at 8:48 AM To: Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> Cc: basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Out of Main Memory Hi Greg, A quick reply: If only parts of your documents are relevant for full-text queries, you can restrict the selection with the FTINDEX option (see [1] for more information). How large is the total size of your input documents? Best, Christian [1] https://docs.basex.org/wiki/Indexes#Selective_Indexing On Tue, Mar 12, 2024 at 8:34 PM Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> wrote: Hello, I’m working with a database that has a full-text index. I have found that if I iteratively add XML documents, then optimize, add more documents, optimize again, and so on, eventually the “optimize” command will fail with “Out of Main Memory.” I edited the basex startup script to change the memory allocation from -Xmx2g to -Xmx12g. My computer has 16 GB of memory, but of course the OS uses up some of it. I have found that if I exit memory-hungry programs (web browser, Oxygen), start basex, and then run the “optimize” command, I still get “Out of Main Memory.” I’m wondering if there are any known workarounds or strategies for this situation. If I understand the documentation about indexes correctly, index data is periodically written to disk during optimization. Does this mean that running optimize again will pick up where the previous attempt left off, such that running optimize repeatedly will eventually succeed? Thanks, Greg Gregory Murray Director of Digital Initiatives Wright Library Princeton Theological Seminary

From: BaseX-Talk <basex-talk-bounces at mailman.uni-konstanz.de> on behalf of Murray, Gregory <gregory.murray at ptsem.edu> Date: Friday, March 15, 2024 at 12:12 PM To: Christian Grün <christian.gruen at gmail.com> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Out of Main Memory Thanks, Christian. Distributing documents across many databases sounds fine, as long as XPath expressions and full-text searching remain reasonably efficient. In the documentation, the example of addressing multiple databases uses a loop: for $i in 1 to 100 return db:get('books' || $i)//book/title Is that the preferred technique? Also, is it possible to perform searches in the same manner without interfering with relevance scores? Thanks, Greg From: Christian Grün <christian.gruen at gmail.com> Date: Friday, March 15, 2024 at 11:51 AM To: Murray, Gregory <gregory.murray at ptsem.edu> Cc: basex-talk at mailman.uni-konstanz.de <basex-talk at mailman.uni-konstanz.de> Subject: Re: [basex-talk] Out of Main Memory Hi Greg, I would have guessed that 12 GB is enough for 4.7 GB; but it sometimes depends on the input. If you like, you can share a single typical document with us, and we can have a look at it. 61 GB will be too large for a complete full-text index, though. However, it’s always possible to distribute documets across multiple databases and access them with a single query [1]. The full-text index is not incremental (in opposition to the other index structures), which means it must be re-created it after updates. However, it’s possible to re-index updated database instances and query fully indexed databases at the same time. Hope this helps, Christian [1] https://docs.basex.org/wiki/Databases On Thu, Mar 14, 2024 at 10:58 PM Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> wrote: Thanks, Christian. I don’t think selective indexing is applicable in my use case, because I need to perform full-text searches on the entirety of each document. Each XML document represents a physical book that was digitized, and the structure of each document is essentially a header with metadata and a body with the OCR text of the book. The OCR text is split into pages, where one <page> element contains all the words from one corresponding printed page from the physical book. Obviously the number of words in each <page> varies widely based on the physical dimensions of the book and the typeface. So far, I have loaded 12,331 documents, containing a total of 2,196,771 pages. The total size of those XML documents on disk is 4.7GB. But that is only a fraction of the total number of documents I want to load into BaseX. The total number is more like 160,000 documents. Assuming that the documents I’ve loaded so far are a representative sample, and I believe that’s true, then the total size of the XML documents on disk, prior to loading them into BaseX, would be about 4.7GB * 13 = 61.1GB. Normally the OCR text, once loaded, almost never changes. But the metadata fields do change as corrections are made. Also we add more XML documents routinely as we digitize more books over time. Therefore updates and additions are commonplace, such that keeping indexes up to date is important, to allow full-text searches to stay performant. I’m wondering if there are techniques for optimizing such quantities of text. Thanks, Greg From: Christian Grün <christian.gruen at gmail.com<mailto:christian.gruen at gmail.com>> Date: Thursday, March 14, 2024 at 8:48 AM To: Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> Cc: basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de> <basex-talk at mailman.uni-konstanz.de<mailto:basex-talk at mailman.uni-konstanz.de>> Subject: Re: [basex-talk] Out of Main Memory Hi Greg, A quick reply: If only parts of your documents are relevant for full-text queries, you can restrict the selection with the FTINDEX option (see [1] for more information). How large is the total size of your input documents? Best, Christian [1] https://docs.basex.org/wiki/Indexes#Selective_Indexing On Tue, Mar 12, 2024 at 8:34 PM Murray, Gregory <gregory.murray at ptsem.edu<mailto:gregory.murray at ptsem.edu>> wrote: Hello, I’m working with a database that has a full-text index. I have found that if I iteratively add XML documents, then optimize, add more documents, optimize again, and so on, eventually the “optimize” command will fail with “Out of Main Memory.” I edited the basex startup script to change the memory allocation from -Xmx2g to -Xmx12g. My computer has 16 GB of memory, but of course the OS uses up some of it. I have found that if I exit memory-hungry programs (web browser, Oxygen), start basex, and then run the “optimize” command, I still get “Out of Main Memory.” I’m wondering if there are any known workarounds or strategies for this situation. If I understand the documentation about indexes correctly, index data is periodically written to disk during optimization. Does this mean that running optimize again will pick up where the previous attempt left off, such that running optimize repeatedly will eventually succeed? Thanks, Greg Gregory Murray Director of Digital Initiatives Wright Library Princeton Theological Seminary