Expected behaviour with full disk
Friday, 1 April 2016 · 6 messages · last reply 3 Apr 2016
JB
James Ball1 Apr 2016 16:48 #
Hello,
I’m having some problems with BaseX in situations where there it runs out of disk space.
I’m delivering XML files to BaseX 8.4 using RestXQ.
A request is made and the XML is added to a database depending on some criteria. I receive many hundreds of thousands of requests and everything works well unless I run out of disk space.
What I would expect to happen is that BaseX would try to add the document, find it couldn’t due to the disk being full and return a server error (e.g. 500) and the transaction would be aborted leaving the database as it was before the request. Any further requests would all get errors returned.
What I’m seeing is that when a request is made that causes the disk to become full BaseX gets part way through adding the document, fails and returns an error but also leaves the database corrupted. Usually the server has to be restarted and then any database that has been affected has to be deleted - it’s rare to be able to save anything from it.
The errors vary but it is usually a NullPointer exception and then java.nio.channels.OverlappingFileLockException. for further requests.
I’ve tried turning off logging and setting parallel to 1 to see if that made any difference but it doesn’t.
Is what I’m seeing unique to my setup or do others see this?
Is there anything I can do to make BaseX just stop accepting requests when disk space runs out to protect my databases?
Many thanks, James
SP
Strasser Pablo1 Apr 2016 17:10 #
Hi,
Quoted text · James Ball · 4 lines
On 01. 04. 16 16:48, James Ball wrote: > > Is what I’m seeing unique to my setup or do others see this? >
I have seen exactly the same behaviour in case of network hard drive loosing connection causing the Kernel to make the filesystem read only. However, I don't really expect BaseX to manage this kind of case contrary to disk full situation.
Pablo
EF
Hello James,
Did you try to set
ADDCACHE ?
This way you might run out of disk space during disk caching, before database corruption occurs. Be careful, you may have to serialize the XML in order to use this option with db:add/db:replace (I read you are in the REST-XQ layer).
Otherwise, the only way to prevent database corruption might be to check disk size regularly and reject new data by program…
http://docs.basex.org/wiki/Options#ADDCACHE
http://docs.basex.org/wiki/Database_Module#db:add
Best regards,
Fabrice Etanchaud
Horanet Smart Objects
FR-85200 Fontenay le ComteQuoted text · James Ball · 16 lines
De : basex-talk-bounces at mailman.uni-konstanz.de [mailto:basex-talk-bounces at mailman.uni-konstanz.de] De la part de James Ball Envoyé : vendredi 1 avril 2016 16:48 À : BaseX <basex-talk at mailman.uni-konstanz.de> Objet : [basex-talk] Expected behaviour with full disk Hello, I’m having some problems with BaseX in situations where there it runs out of disk space. I’m delivering XML files to BaseX 8.4 using RestXQ. A request is made and the XML is added to a database depending on some criteria. I receive many hundreds of thousands of requests and everything works well unless I run out of disk space. J What I would expect to happen is that BaseX would try to add the document, find it couldn’t due to the disk being full and return a server error (e.g. 500) and the transaction would be aborted leaving the database as it was before the request. Any further requests would all get errors returned. What I’m seeing is that when a request is made that causes the disk to become full BaseX gets part way through adding the document, fails and returns an error but also leaves the database corrupted. Usually the server has to be restarted and then any database that has been affected has to be deleted - it’s rare to be able to save anything from it. The errors vary but it is usually a NullPointer exception and then java.nio.channels.OverlappingFileLockException. for further requests. I’ve tried turning off logging and setting parallel to 1 to see if that made any difference but it doesn’t. Is what I’m seeing unique to my setup or do others see this? Is there anything I can do to make BaseX just stop accepting requests when disk space runs out to protect my databases? Many thanks, James
CG
Hi James,
I’m sorry, there is general solution for that on database level; if a disk is full, basically anything can be happen.
However, as you are using RESTXQ, you could check memory via XQuery:
declare namespace file = 'java:java.io.File';
let $free := file:getFreeSpace(file:new('/'))
return if($free < 100000000) then
error((), "Less than 100 MB left on disk") else
( (: continue :) )This way, you can also decide what is going to happen if the disk runs out of space (raise an error? send an e-mail? etc.). By the way, raising an error will probably be the most elegant solution, as you can catch the error via %rest:error and do some more things [1].
Hope this helps,
Christian
[1] http://docs.basex.org/wiki/RESTXQ#Error_Handling
Quoted text · James Ball · 35 lines
On Fri, Apr 1, 2016 at 4:48 PM, James Ball <basex-talk at jamesball.co.uk> wrote: > Hello, > > I’m having some problems with BaseX in situations where there it runs out of > disk space. > > I’m delivering XML files to BaseX 8.4 using RestXQ. > > A request is made and the XML is added to a database depending on some > criteria. I receive many hundreds of thousands of requests and everything > works well unless I run out of disk space. > > What I would expect to happen is that BaseX would try to add the document, > find it couldn’t due to the disk being full and return a server error (e.g. > 500) and the transaction would be aborted leaving the database as it was > before the request. Any further requests would all get errors returned. > > What I’m seeing is that when a request is made that causes the disk to > become full BaseX gets part way through adding the document, fails and > returns an error but also leaves the database corrupted. Usually the server > has to be restarted and then any database that has been affected has to be > deleted - it’s rare to be able to save anything from it. > > The errors vary but it is usually a NullPointer exception and then > java.nio.channels.OverlappingFileLockException. for further requests. > > I’ve tried turning off logging and setting parallel to 1 to see if that made > any difference but it doesn’t. > > Is what I’m seeing unique to my setup or do others see this? > > Is there anything I can do to make BaseX just stop accepting requests when > disk space runs out to protect my databases? > > Many thanks, James
JB
James Ball2 Apr 2016 19:00 #
Hi Christian,
Thank you for update.
I had assumed that BaseX would make a request for more disk space before trying to write to it and if it couldn’t be allocated would abort the transaction.
Luckily in my case there is only one function that ever writes to the databases so your suggested solution is ideal. Basically one copy and paste has got it working!
Thank you so much.
James
Quoted text · Christian Grün · 62 lines
> On 1 Apr 2016, at 17:23, Christian Grün <christian.gruen at gmail.com> wrote: > > Hi James, > > I’m sorry, there is general solution for that on database level; if a > disk is full, basically anything can be happen. > > However, as you are using RESTXQ, you could check memory via XQuery: > > declare namespace file = 'java:java.io.File'; > let $free := file:getFreeSpace(file:new('/')) > return if($free < 100000000) then > error((), "Less than 100 MB left on disk") else > ( (: continue :) ) > > This way, you can also decide what is going to happen if the disk runs > out of space (raise an error? send an e-mail? etc.). By the way, > raising an error will probably be the most elegant solution, as you > can catch the error via %rest:error and do some more things [1]. > > Hope this helps, > Christian > > [1] http://docs.basex.org/wiki/RESTXQ#Error_Handling > > > > On Fri, Apr 1, 2016 at 4:48 PM, James Ball <basex-talk at jamesball.co.uk> wrote: >> Hello, >> >> I’m having some problems with BaseX in situations where there it runs out of >> disk space. >> >> I’m delivering XML files to BaseX 8.4 using RestXQ. >> >> A request is made and the XML is added to a database depending on some >> criteria. I receive many hundreds of thousands of requests and everything >> works well unless I run out of disk space. >> >> What I would expect to happen is that BaseX would try to add the document, >> find it couldn’t due to the disk being full and return a server error (e.g. >> 500) and the transaction would be aborted leaving the database as it was >> before the request. Any further requests would all get errors returned. >> >> What I’m seeing is that when a request is made that causes the disk to >> become full BaseX gets part way through adding the document, fails and >> returns an error but also leaves the database corrupted. Usually the server >> has to be restarted and then any database that has been affected has to be >> deleted - it’s rare to be able to save anything from it. >> >> The errors vary but it is usually a NullPointer exception and then >> java.nio.channels.OverlappingFileLockException. for further requests. >> >> I’ve tried turning off logging and setting parallel to 1 to see if that made >> any difference but it doesn’t. >> >> Is what I’m seeing unique to my setup or do others see this? >> >> Is there anything I can do to make BaseX just stop accepting requests when >> disk space runs out to protect my databases? >> >> Many thanks, James
CG
Christian Grün3 Apr 2016 09:41 #
Hi James,
I'm glad to hear that the XQuery check solves your problems.
Quoted text · James Ball · 1 line
> I had assumed that BaseX would make a request for more disk space before
trying to write to it and if it couldn’t be allocated would abort the transaction.
True, that'd be nice. As the memory consumption per update varies a lot, depending on the chosen database options, it turned out it's pretty hard to guess how much disk space will eventually be required.
Quoted text · James Ball · 1 line
> Thank you so much.
You're always welcome,
Christian
Quoted text · James Ball · 36 lines
> > James > > > On 1 Apr 2016, at 17:23, Christian Grün <christian.gruen at gmail.com> wrote: > > > > Hi James, > > > > I’m sorry, there is general solution for that on database level; if a > > disk is full, basically anything can be happen. > > > > However, as you are using RESTXQ, you could check memory via XQuery: > > > > declare namespace file = 'java:java.io.File'; > > let $free := file:getFreeSpace(file:new('/')) > > return if($free < 100000000) then > > error((), "Less than 100 MB left on disk") else > > ( (: continue :) ) > > > > This way, you can also decide what is going to happen if the disk runs > > out of space (raise an error? send an e-mail? etc.). By the way, > > raising an error will probably be the most elegant solution, as you > > can catch the error via %rest:error and do some more things [1]. > > > > Hope this helps, > > Christian > > > > [1] http://docs.basex.org/wiki/RESTXQ#Error_Handling > > > > > > > > On Fri, Apr 1, 2016 at 4:48 PM, James Ball <basex-talk at jamesball.co.uk> wrote: > >> Hello, > >> > >> I’m having some problems with BaseX in situations where there it runs
out of
Quoted text · James Ball · 6 lines
> >> disk space. > >> > >> I’m delivering XML files to BaseX 8.4 using RestXQ. > >> > >> A request is made and the XML is added to a database depending on some > >> criteria. I receive many hundreds of thousands of requests and
everything
Quoted text · James Ball · 3 lines
> >> works well unless I run out of disk space. > >> > >> What I would expect to happen is that BaseX would try to add the
document,
Quoted text · James Ball · 1 line
> >> find it couldn’t due to the disk being full and return a server error
(e.g.
Quoted text · James Ball · 1 line
> >> 500) and the transaction would be aborted leaving the database as it
was
Quoted text · James Ball · 5 lines
> >> before the request. Any further requests would all get errors returned. > >> > >> What I’m seeing is that when a request is made that causes the disk to > >> become full BaseX gets part way through adding the document, fails and > >> returns an error but also leaves the database corrupted. Usually the
server
Quoted text · James Ball · 1 line
> >> has to be restarted and then any database that has been affected has
to be
Quoted text · James Ball · 6 lines
> >> deleted - it’s rare to be able to save anything from it. > >> > >> The errors vary but it is usually a NullPointer exception and then > >> java.nio.channels.OverlappingFileLockException. for further requests. > >> > >> I’ve tried turning off logging and setting parallel to 1 to see if
that made
Quoted text · James Ball · 5 lines
> >> any difference but it doesn’t. > >> > >> Is what I’m seeing unique to my setup or do others see this? > >> > >> Is there anything I can do to make BaseX just stop accepting requests
when
Quoted text · James Ball · 4 lines
> >> disk space runs out to protect my databases? > >> > >> Many thanks, James >
Written for BaseX 8.4. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery