size on collection in the time
Wednesday, 2 September 2015 · 10 messages · last reply 14 Sep 2015
M
Marc2 Sep 2015 17:36 #
Hi,
I have a remark about the size of the files of a collection. We do a lot of updates (REPLACE) with an attributes index but without a text index.
I see that the size of the files grows up in the time we arrive until 30Go until the limit of the filessytems quota.
When I drop the collection and recreate it from the last version of the files the size was only 6Go.
We will redo the test with the time, but I just want to know if the program clears the disk use when he does a REPLACE?
Marc
DK
Dirk Kirsten3 Sep 2015 09:52 #
Hello Marc,
sure, a replace should lead to a true replace, i.e. the old data will be dropped. So this sounds like a bug, but it is difficult to replicate from what you describe.
Do you think it would be possible to make the issue reproducable, maybe also with a smaller data set? Also, how exactly do you add documents or do you also do any queries and maybe other updating queries?
Cheers
Dirk
Quoted text · Marc · 14 lines
On 09/02/2015 05:36 PM, Marc wrote: > Hi, > I have a remark about the size of the files of a collection. > We do a lot of updates (REPLACE) with an attributes index but without > a text index. > I see that the size of the files grows up in the time we arrive until > 30Go until the limit of the filessytems quota. > When I drop the collection and recreate it from the last version of > the files the size was only 6Go. > > We will redo the test with the time, but I just want to know if the > program clears the disk use when he does a REPLACE? > > Marc
--
Dirk Kirsten, BaseX GmbH, http://basexgmbh.de
|-- Firmensitz: Blarerstrasse 56, 78462 Konstanz
|-- Registergericht Freiburg, HRB: 708285, Geschäftsführer:
| Dr. Christian Grün, Dr. Alexander Holupirek, Michael Seiferle `-- Phone: 0049 7531 28 28 676, Fax: 0049 7531 20 05 22
M
Marc3 Sep 2015 09:56 #
Hello Dirk,
We use a BaseX script (.xbs) with a list of REPLACE.
We try to investigate this point if this is reproducable or not. If I find this problem again, I take a copy of the collection before to delete her.
Thank you for answer.
Marc
Quoted text · Dirk Kirsten · 29 lines
Le 03/09/2015 09:52, Dirk Kirsten a écrit : > Hello Marc, > > sure, a replace should lead to a true replace, i.e. the old data will be > dropped. So this sounds like a bug, but it is difficult to replicate > from what you describe. > > Do you think it would be possible to make the issue reproducable, maybe > also with a smaller data set? Also, how exactly do you add documents or > do you also do any queries and maybe other updating queries? > > Cheers > Dirk > > On 09/02/2015 05:36 PM, Marc wrote: >> Hi, >> I have a remark about the size of the files of a collection. >> We do a lot of updates (REPLACE) with an attributes index but without >> a text index. >> I see that the size of the files grows up in the time we arrive until >> 30Go until the limit of the filessytems quota. >> When I drop the collection and recreate it from the last version of >> the files the size was only 6Go. >> >> We will redo the test with the time, but I just want to know if the >> program clears the disk use when he does a REPLACE? >> >> Marc >
M
Marc3 Sep 2015 10:01 #
I forgot to give this precision, we use the 7.9 version on this server.
Marc
Quoted text · Marc · 39 lines
Le 01/01/1970 01:00, a écrit : > Hello Dirk, > We use a BaseX script (.xbs) with a list of REPLACE. > We try to investigate this point if this is reproducable or not. > If I find this problem again, I take a copy of the collection before to > delete her. > > Thank you for answer. > > Marc > Le 03/09/2015 09:52, Dirk Kirsten a écrit : >> Hello Marc, >> >> sure, a replace should lead to a true replace, i.e. the old data will be >> dropped. So this sounds like a bug, but it is difficult to replicate >> from what you describe. >> >> Do you think it would be possible to make the issue reproducable, maybe >> also with a smaller data set? Also, how exactly do you add documents or >> do you also do any queries and maybe other updating queries? >> >> Cheers >> Dirk >> >> On 09/02/2015 05:36 PM, Marc wrote: >>> Hi, >>> I have a remark about the size of the files of a collection. >>> We do a lot of updates (REPLACE) with an attributes index but without >>> a text index. >>> I see that the size of the files grows up in the time we arrive until >>> 30Go until the limit of the filessytems quota. >>> When I drop the collection and recreate it from the last version of >>> the files the size was only 6Go. >>> >>> We will redo the test with the time, but I just want to know if the >>> program clears the disk use when he does a REPLACE? >>> >>> Marc >>
JB
Marc,
Are you using the updatable index?
UPDINDEX true.
If so, if you look back through the list Christian and I had an exchange on this as I was experiencing an ever growing index - a replace always appended to the end of the index. IIRC the logic is a bit better now in that if on a replace there is enough room in the index structure to reuse slots it will, but if not it will be appended. And the db will reuse empty slots while the database remains open. I can’t remember which is the first version where this change was made, sorry.
I had a DB the other day that had got to 14GB but was only around 600MB once I did a full optimise. My suggestion is next time to try a full optimise instead of dropping the collection and see if that helps.
Regards, JamesQuoted text · Marc · 22 lines
> > Message: 1 > Date: Wed, 2 Sep 2015 17:36:19 +0200 > From: Marc <Marc.Liste at free.fr> > To: BaseX <basex-talk at mailman.uni-konstanz.de> > Subject: [basex-talk] size on collection in the time > Message-ID: <55E71773.7050102 at free.fr> > Content-Type: text/plain; charset=iso-8859-15; format=flowed > > Hi, > I have a remark about the size of the files of a collection. > We do a lot of updates (REPLACE) with an attributes index but without a > text index. > I see that the size of the files grows up in the time we arrive until > 30Go until the limit of the filessytems quota. > When I drop the collection and recreate it from the last version of the > files the size was only 6Go. > > We will redo the test with the time, but I just want to know if the > program clears the disk use when he does a REPLACE? > > Marc
M
Marc3 Sep 2015 19:06 #
James,
We have the
UPDINDEX to FALSE.
I'll try to do an OPTIMIZE command, but we do it at the end of all the REPLACE so I think that it's not enough.
After that I prepare a study case with public data.
Regards
MarcQuoted text · James Ball · 33 lines
Le 03/09/2015 12:17, James Ball a écrit : > Marc, > > Are you using the updatable index? UPDINDEX true. > > If so, if you look back through the list Christian and I had an exchange on this as I was experiencing an ever growing index - a replace always appended to the end of the index. IIRC the logic is a bit better now in that if on a replace there is enough room in the index structure to reuse slots it will, but if not it will be appended. And the db will reuse empty slots while the database remains open. I can’t remember which is the first version where this change was made, sorry. > > I had a DB the other day that had got to 14GB but was only around 600MB once I did a full optimise. My suggestion is next time to try a full optimise instead of dropping the collection and see if that helps. > > Regards, James > >> >> Message: 1 >> Date: Wed, 2 Sep 2015 17:36:19 +0200 >> From: Marc <Marc.Liste at free.fr> >> To: BaseX <basex-talk at mailman.uni-konstanz.de> >> Subject: [basex-talk] size on collection in the time >> Message-ID: <55E71773.7050102 at free.fr> >> Content-Type: text/plain; charset=iso-8859-15; format=flowed >> >> Hi, >> I have a remark about the size of the files of a collection. >> We do a lot of updates (REPLACE) with an attributes index but without a >> text index. >> I see that the size of the files grows up in the time we arrive until >> 30Go until the limit of the filessytems quota. >> When I drop the collection and recreate it from the last version of the >> files the size was only 6Go. >> >> We will redo the test with the time, but I just want to know if the >> program clears the disk use when he does a REPLACE? >> >> Marc
CM
Christophe Marchand3 Sep 2015 21:20 #
Hi Dirk,
attached a simple test-case to reproduce the problem.
data.xml is a file that contains attributes and text. Each time you run run.sh, data.xml is re-created, with current-dateTime() in. So each time, the file content is changed.
The file size does not change (date format ensures it).
reload.bxs deletes the file from the collection and add it back to the collection.
run.sh displays the size of files in the collection.
Each time you start run.sh, file size grows.
You need to modify run.sh to set the correct directories locations. You have to add saxon9he.jar to run it.
I hope this will help,
Christophe
Quoted text · basex-talk-request at mailman.uni-konstanz.de · 138 lines
Le 03/09/2015 15:27, basex-talk-request at mailman.uni-konstanz.de a écrit : > Send BaseX-Talk mailing list submissions to > basex-talk at mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request at mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner at mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. Re: size on collection in the time (Marc) (cmarchand at oxiane.com) > 2. Re: Unexpected error: Improper use? Potential bug? > (Pierre-Yves JALLUD) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 03 Sep 2015 14:59:53 +0200 > From: cmarchand at oxiane.com > To: basex-talk at mailman.uni-konstanz.de > Subject: Re: [basex-talk] size on collection in the time (Marc) > Message-ID: <2fe47eb3ff5f77da91888afed19d2d72 at oxiane.com> > Content-Type: text/plain; charset="us-ascii" > > > > I confirm the problem : on another database (same data, same script.xbs, > different server), we have exactly the same problem : the database grows > each time the script is run. > > I'm going to prepare a test-case with small public data tonight. > > Best regards, > > Christophe > > Hello Dirk, > We use a BaseX script (.xbs) with a list of REPLACE. > We try to investigate this point if this is reproducable or not. > If I find this problem again, I take a copy of the collection before to > delete her. > > Thank you for answer. > > Marc > Le 03/09/2015 09:52, Dirk Kirsten a ?crit : > >> Hello Marc, >> >> sure, a replace should lead to a true replace, i.e. the old data will be >> dropped. So this sounds like a bug, but it is difficult to replicate >> from what you describe. >> >> Do you think it would be possible to make the issue reproducable, maybe >> also with a smaller data set? Also, how exactly do you add documents or >> do you also do any queries and maybe other updating queries? >> >> Cheers >> Dirk >> >> On 09/02/2015 05:36 PM, Marc wrote: >> >>> Hi, >>> I have a remark about the size of the files of a collection. >>> We do a lot of updates (REPLACE) with an attributes index but without >>> a text index. >>> I see that the size of the files grows up in the time we arrive until >>> 30Go until the limit of the filessytems quota. >>> When I drop the collection and recreate it from the last version of >>> the files the size was only 6Go. >>> >>> We will redo the test with the time, but I just want to know if the >>> program clears the disk use when he does a REPLACE? >>> >>> Marc > ------------------------------ > > Message: 4 > Date: Thu, 3 Sep 2015 10:01:05 +0200 > From: Marc <Marc.Liste at free.fr> > To: Dirk Kirsten <dk at basex.org> > Cc: BaseX <basex-talk at mailman.uni-konstanz.de> > Subject: Re: [basex-talk] (no subject) > Message-ID: <55E7FE41.5080205 at free.fr> > Content-Type: text/plain; charset=iso-8859-15; format=flowed > > I forgot to give this precision, we use the 7.9 version on this server. > > Marc > Le 01/01/1970 01:00, a ?crit : > >> Hello Dirk, >> We use a BaseX script (.xbs) with a list of REPLACE. >> We try to investigate this point if this is reproducable or not. >> If I find this problem again, I take a copy of the collection before to >> delete her. >> >> Thank you for answer. >> >> Marc >> Le 03/09/2015 09:52, Dirk Kirsten a ?crit : Hello Marc, >> >> sure, a replace should lead to a true replace, i.e. the old data will be >> dropped. So this sounds like a bug, but it is difficult to replicate >> from what you describe. >> >> Do you think it would be possible to make the issue reproducable, maybe >> also with a smaller data set? Also, how exactly do you add documents or >> do you also do any queries and maybe other updating queries? >> >> Cheers >> Dirk >> >> On 09/02/2015 05:36 PM, Marc wrote: Hi, >> I have a remark about the size of the files of a collection. >> We do a lot of updates (REPLACE) with an attributes index but without >> a text index. >> I see that the size of the files grows up in the time we arrive until >> 30Go until the limit of the filessytems quota. >> When I drop the collection and recreate it from the last version of >> the files the size was only 6Go. >> >> We will redo the test with the time, but I just want to know if the >> program clears the disk use when he does a REPLACE? >> >> Marc > End of BaseX-Talk Digest, Vol 69, Issue 5 > ***************************************** >
CM
Christophe Marchand3 Sep 2015 21:38 #
James, you are right, OPTIMIZE ALL solves the problem.
Thanks a lot !
Best regards,
Christophe
Quoted text · basex-talk-request at mailman.uni-konstanz.de · 59 lines
Le 03/09/2015 21:20, basex-talk-request at mailman.uni-konstanz.de a écrit : > Send BaseX-Talk mailing list submissions to > basex-talk at mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request at mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner at mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. Xquery collections (michele.greco2 at email.it) > 2. Re: Xquery collections (Dirk Kirsten) > 3. Re: size on collection in the time (Marc) > 4. Re: size on collection in the time (Christophe Marchand) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 3 Sep 2015 17:48:23 +0200 > From: michele.greco2 at email.it > To: basex-talk at mailman.uni-konstanz.de > Subject: [basex-talk] Xquery collections > Message-ID: <22e8f2f70298a35093e933ab65f26489 at wm10.email.it> > Content-Type: text/plain; charset="iso-8859-1" > > > Hello I Michele i'm new to basex, > i created a collection with different documents .xml. > I want to know how to query the entire collection e its documents at one > xquery. > I'm trying,but i can't. > > Thanks. > MG > > > > > > > > > -- > ZE-Light e ZE-Pro: servizi zimbra per caselle con dominio email.it, per tutti i dettagli > Clicca qui http://posta.email.it/caselle-di-posta-z-email-it/?utm_campaign=email_Zimbra_102014=main_footer/f > > Sponsor: > Caselle con tuo dominio su piattaforma Zimbra, fino a 30 GB di spazio, sincronizzazione dati e backup > Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=13324&d=3-9 >
CM
Christophe Marchand3 Sep 2015 21:52 #
James, you are right, OPTIMIZE ALL solves the problem.
The txt.basex was growing at each replace, other files staying almost identical.
Thanks a lot !
Best regards,
Christophe
Quoted text · basex-talk-request at mailman.uni-konstanz.de · 59 lines
Le 03/09/2015 21:20, basex-talk-request at mailman.uni-konstanz.de a écrit : > Send BaseX-Talk mailing list submissions to > basex-talk at mailman.uni-konstanz.de > > To subscribe or unsubscribe via the World Wide Web, visit > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > or, via email, send a message with subject or body 'help' to > basex-talk-request at mailman.uni-konstanz.de > > You can reach the person managing the list at > basex-talk-owner at mailman.uni-konstanz.de > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of BaseX-Talk digest..." > > > Today's Topics: > > 1. Xquery collections (michele.greco2 at email.it) > 2. Re: Xquery collections (Dirk Kirsten) > 3. Re: size on collection in the time (Marc) > 4. Re: size on collection in the time (Christophe Marchand) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 3 Sep 2015 17:48:23 +0200 > From: michele.greco2 at email.it > To: basex-talk at mailman.uni-konstanz.de > Subject: [basex-talk] Xquery collections > Message-ID: <22e8f2f70298a35093e933ab65f26489 at wm10.email.it> > Content-Type: text/plain; charset="iso-8859-1" > > > Hello I Michele i'm new to basex, > i created a collection with different documents .xml. > I want to know how to query the entire collection e its documents at one > xquery. > I'm trying,but i can't. > > Thanks. > MG > > > > > > > > > -- > ZE-Light e ZE-Pro: servizi zimbra per caselle con dominio email.it, per tutti i dettagli > Clicca qui http://posta.email.it/caselle-di-posta-z-email-it/?utm_campaign=email_Zimbra_102014=main_footer/f > > Sponsor: > Caselle con tuo dominio su piattaforma Zimbra, fino a 30 GB di spazio, sincronizzazione dati e backup > Clicca qui: http://adv.email.it/cgi-bin/foclick.cgi?mid=13324&d=3-9 >
CG
Hi Marc,
I think you've already got some helpful feedback on the list. I would just like to add that you should absolutely try BaseX 8.x, as we have spend quite some time in speeding up our replace operations.
Hope this helps,
Christian
Quoted text · Marc · 49 lines
On Thu, Sep 3, 2015 at 10:01 AM, Marc <Marc.Liste at free.fr> wrote: > > I forgot to give this precision, we use the 7.9 version on this server. > > Marc > Le 01/01/1970 01:00, a écrit : >> >> Hello Dirk, >> We use a BaseX script (.xbs) with a list of REPLACE. >> We try to investigate this point if this is reproducable or not. >> If I find this problem again, I take a copy of the collection before to >> delete her. >> >> Thank you for answer. >> >> Marc >> Le 03/09/2015 09:52, Dirk Kirsten a écrit : >>> >>> Hello Marc, >>> >>> sure, a replace should lead to a true replace, i.e. the old data will be >>> dropped. So this sounds like a bug, but it is difficult to replicate >>> from what you describe. >>> >>> Do you think it would be possible to make the issue reproducable, maybe >>> also with a smaller data set? Also, how exactly do you add documents or >>> do you also do any queries and maybe other updating queries? >>> >>> Cheers >>> Dirk >>> >>> On 09/02/2015 05:36 PM, Marc wrote: >>>> >>>> Hi, >>>> I have a remark about the size of the files of a collection. >>>> We do a lot of updates (REPLACE) with an attributes index but without >>>> a text index. >>>> I see that the size of the files grows up in the time we arrive until >>>> 30Go until the limit of the filessytems quota. >>>> When I drop the collection and recreate it from the last version of >>>> the files the size was only 6Go. >>>> >>>> We will redo the test with the time, but I just want to know if the >>>> program clears the disk use when he does a REPLACE? >>>> >>>> Marc >>> >>> >
Written for BaseX 8.2. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery