Database folders with numbers after
Friday, 8 January 2016 · 6 messages · last reply 9 Jan 2016
JB
James Ball8 Jan 2016 12:25 #
Hello,
I’m seeing regular occurrences where I end up with extra folders in my database location with numbers after, for example:
ODF-PUSH-T-JU
ODF-PUSH-T-JU_1079560404
These have some but not all of the files from the original database with the same name, crucially don’t have the inf.basex file and so prevent the /dba from listing databases.
What creates these folders?
* I have a job that runs each night to optimise all the databases that have changed that day, I think these are coming about due to a failure to optimise but I’m not sure, I’m trying to understand what might creates them to be created so I can look deeper into the cause.
Running 8.4 beta 5687c30 on Linux.
Many thanks for your help, James
CG
Christian Grün8 Jan 2016 12:47 #
Hi James,
As you already guessed, these folders are relics from OPTIMIZE ALL calls – or
db:optimize($db, true) –, and they are removed after the operation if everything went alright.
Hope this helps,
ChristianQuoted text · James Ball · 17 lines
On Fri, Jan 8, 2016 at 12:25 PM, James Ball <basex-talk at jamesball.co.uk> wrote: > Hello, > > I’m seeing regular occurrences where I end up with extra folders in my database location with numbers after, for example: > > ODF-PUSH-T-JU > ODF-PUSH-T-JU_1079560404 > > These have some but not all of the files from the original database with the same name, crucially don’t have the inf.basex file and so prevent the /dba from listing databases. > > What creates these folders? > > * I have a job that runs each night to optimise all the databases that have changed that day, I think these are coming about due to a failure to optimise but I’m not sure, I’m trying to understand what might creates them to be created so I can look deeper into the cause. > > Running 8.4 beta 5687c30 on Linux. > > Many thanks for your help, James
JB
James Ball8 Jan 2016 14:16 #
Hi Christian,
Thank you for that - so I’m looking in the right area.
Quoted text · Christian Grün · 3 lines
> As you already guessed, these folders are relics from OPTIMIZE ALL > calls – or db:optimize($db, true) –, and they are removed after the > operation if everything went alright.
So another question - these seem to come about from running OPTIMIZE ALL on databases with
UPDINDEX TRUE. Is there any reason that OPTIMIZE ALL can’t be run on such databases?
(I’m running it because these databases become huge after lots of files are added)
Many thanks, JamesCG
Christian Grün8 Jan 2016 20:00 #
Quoted text · James Ball · 1 line
> So another question - these seem to come about from running OPTIMIZE ALL on databases with UPDINDEX TRUE. Is there any reason that OPTIMIZE ALL can’t be run on such databases?
It shouldn’t cause any problems. Here is a little command scripts that demonstrates that it should work alright:
<commands>
<set option='updindex'>true</set>
<create-db name='db'/>
<add path='a.xml'><a/></add>
<add path='b.xml'><b/></add>
<optimize-all/>
<info-db/>
<close/>
</commands>Can you also reproduce the behavior with smaller data sets?
Thanks,
Christian
JB
James Ball8 Jan 2016 20:11 #
Christian,
I can’t reliably reproduce the behaviour at all at the moment but I’m losing a fair number of databases to the problem. I first thought it was related to running out of disk space mid-operation but I don’t think that’s the cause. I just wanted to check that I wasn’t doing anything I shouldn’t before I try to find a reproducible error.
UPDINDEX is really useful for my use case but seems to leave me with database in GB range until I OPTIMIZE them. I shall continue monitoring.
Many thanks, JamesQuoted text · Christian Grün · 21 lines
> On 8 Jan 2016, at 19:00, Christian Grün <christian.gruen at gmail.com> wrote: > >> So another question - these seem to come about from running OPTIMIZE ALL on databases with UPDINDEX TRUE. Is there any reason that OPTIMIZE ALL can’t be run on such databases? > > It shouldn’t cause any problems. Here is a little command scripts that > demonstrates that it should work alright: > > <commands> > <set option='updindex'>true</set> > <create-db name='db'/> > <add path='a.xml'><a/></add> > <add path='b.xml'><b/></add> > <optimize-all/> > <info-db/> > <close/> > </commands> > > Can you also reproduce the behavior with smaller data sets? > > Thanks, > Christian
JB
James Ball9 Jan 2016 14:03 #
So this happened again overnight but I have more information now.
1) I had switched off
UPDINDEX on some of the databases and one of these was affected so this is not an UPDINDEX issue
2) I found that there was a hard crash of the JVM - which would account for why many databases were left locked and one had the folder with the numbers after
3) My Upstart script restarts the server quite happily which disguises 2) if you’re not looking closely!
The hard crash was due to the JVM not being able to allocate memory. What I’m not sure is why my script to OPTIMIZE all the databases seems to run quite happily manually but is failing when run by cron overnight.
I will continue monitoring.
Regards, JamesQuoted text · James Ball · 32 lines
> On 8 Jan 2016, at 19:11, James Ball <basex-talk at jamesball.co.uk> wrote: > > Christian, > > I can’t reliably reproduce the behaviour at all at the moment but I’m losing a fair number of databases to the problem. I first thought it was related to running out of disk space mid-operation but I don’t think that’s the cause. I just wanted to check that I wasn’t doing anything I shouldn’t before I try to find a reproducible error. > > UPDINDEX is really useful for my use case but seems to leave me with database in GB range until I OPTIMIZE them. I shall continue monitoring. > > Many thanks, James > >> On 8 Jan 2016, at 19:00, Christian Grün <christian.gruen at gmail.com> wrote: >> >>> So another question - these seem to come about from running OPTIMIZE ALL on databases with UPDINDEX TRUE. Is there any reason that OPTIMIZE ALL can’t be run on such databases? >> >> It shouldn’t cause any problems. Here is a little command scripts that >> demonstrates that it should work alright: >> >> <commands> >> <set option='updindex'>true</set> >> <create-db name='db'/> >> <add path='a.xml'><a/></add> >> <add path='b.xml'><b/></add> >> <optimize-all/> >> <info-db/> >> <close/> >> </commands> >> >> Can you also reproduce the behavior with smaller data sets? >> >> Thanks, >> Christian >
Written for BaseX 8.3. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery