Query regarding delete documents
Monday, 5 February 2024 · 4 messages
DD
Deepak Dinakara5 Feb 2024 13:09 #
Hi,
I wanted to understand more on how deleting documents impact the storage layout.
I did go through https://docs.basex.org/wiki/Storage_Layout but didn't find all the required information.
I observed that size of files like "txt.basex" and "atv.basex" doesn't shrink on deleting documents.
Is that expected?
PS: Thanks for the great product.
Thank you,
Deepak
CG
Christian Grün5 Feb 2024 13:12 #
Hi Deepak,
Some more time would have to be spent to explain this in full detail.
In a nutshell, deleted space is not immediately reduced to a minimum size because inserts are more regular. If you want to minimize your database structure after extensive updates, you can call OPTIMIZE (ALL).
Hope this helps,
Christian
Quoted text · Deepak Dinakara · 17 lines
On Mon, Feb 5, 2024 at 1:09 PM Deepak Dinakara <deepukalmane at gmail.com> wrote: > Hi, > I wanted to understand more on how deleting documents impact the storage > layout. > I did go through https://docs.basex.org/wiki/Storage_Layout but didn't > find all the required information. > I observed that size of files like "txt.basex" and "atv.basex" doesn't > shrink on deleting documents. > Is that expected? > > PS: Thanks for the great product. > > Thank you, > Deepak > >
DD
Deepak Dinakara5 Feb 2024 13:25 #
Hi Christian,
Thank you for such a quick turnaround. It's indeed well documented but would be great to have these aspects also covered in detail. I avoided "OPTIMIZE" since it caused OOM issues and I was fine without the index which looks like a double edged sword : ).
QQ, Is there some stat on how much RAM is needed for maintaining basex DB of size 'X'GB (with regular inserts and delete) so that "optimize" could be called without worrying about OOM?
-Deepak
Quoted text · Christian Grün · 33 lines
On Mon, Feb 5, 2024 at 5:42 PM Christian Grün <christian.gruen at gmail.com> wrote: > Hi Deepak, > > Some more time would have to be spent to explain this in full detail. > > In a nutshell, deleted space is not immediately reduced to a minimum size > because inserts are more regular. If you want to minimize your database > structure after extensive updates, you can call OPTIMIZE (ALL). > > Hope this helps, > Christian > > > On Mon, Feb 5, 2024 at 1:09 PM Deepak Dinakara <deepukalmane at gmail.com> > wrote: > >> Hi, >> I wanted to understand more on how deleting documents impact the storage >> layout. >> I did go through https://docs.basex.org/wiki/Storage_Layout but didn't >> find all the required information. >> I observed that size of files like "txt.basex" and "atv.basex" doesn't >> shrink on deleting documents. >> Is that expected? >> >> PS: Thanks for the great product. >> >> Thank you, >> Deepak >> >> >
CG
Christian Grün5 Feb 2024 13:33 #
Quoted text · Deepak Dinakara · 4 lines
> > I avoided "OPTIMIZE" since it caused OOM issues and I was fine without the > index which looks like a double edged sword : ). >
If you don’t need the indexes, you can disable them when running
db:optimize, and setting textindex and attrindex to false.Quoted text · Deepak Dinakara · 4 lines
QQ, Is there some stat on how much RAM is needed for maintaining basex DB > of size 'X'GB (with regular inserts and delete) so that "optimize" could > be called without worrying about OOM? >
Hardly possible to say in general; it depends a lot on the “regular inserts and deletes” ;) If you cannot solve the optimization problem, feel free to share the OOM stack trace with us.
Written for BaseX 10.7. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery