Too many open files? Ack!
Tuesday, 4 January 2011 · 6 messages
LK
Hey Chas,
I'm not quite sure if I understand your scenario so this is just a quick guess. The maximum number of opened files is depending on your operating system. Are you sure you close all files, sockets, streams that are no longer needed?
Otherwise you could try to raise the file limit depending on your system ... but this might be a no-go for you as the problem is eventually postponed, but not fixed.
Cheers,
Lukas
On Tue, Jan 4, 2011 at 8:57 PM, Charles F. Munat <charles.munat at gmail.com>wrote:
Quoted text · Charles F. Munat · 21 lines
> Got this error: > > org.basex.core.BaseXException: /var/www/xxx/db/data/atv.basex (Too many > open files) > > Oh, noes! What does this mean? I was just hitting the page quickly. I'm > sunk if people can't hit the page repeatedly. Is there a setting to adjust > this? Is it a heap space issue? How can I make sure this never happens? > > I have an embedded database. I load it through the application (no file). I > don't know what you call that. It uses one big XQuery to generate the page. > Any ideas? > > Thanks very much! > > Chas. > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >
CM
Charles F. Munat4 Jan 2011 21:57 #
Got this error:
org.basex.core.BaseXException: /var/www/xxx/db/data/atv.basex (Too many open files)
Oh, noes! What does this mean? I was just hitting the page quickly. I'm sunk if people can't hit the page repeatedly. Is there a setting to adjust this? Is it a heap space issue? How can I make sure this never happens?
I have an embedded database. I load it through the application (no file). I don't know what you call that. It uses one big XQuery to generate the page. Any ideas?
Thanks very much!
Chas.
LK
As the hard file limit is around 2^16, depending on your system, it's very unlikely to burst the roof in a real world application. Somebody might correct me if I'm wrong ...
Glad to hear that it works now! (fingers crossed ... )
On Tue, Jan 4, 2011 at 10:14 PM, Charles F. Munat
<charles.munat at gmail.com>wrote:
Quoted text · Charles F. Munat · 64 lines
> I think that the problem might have been that I had the ClientGUI open and > was accessing the embedded database at the same time (same database) and had > numerous other databases possibly "open" (plus a couple hundred files). If > it doesn't happen on the server, I'm happy. > > I'm not very clear what the scenario is either... > > If no one else has a problem with this, it was probably just an extremely > unusual situation. > > Thanks! > > Chas. > > > On 01/04/2011 5:48 PM, Lukas Kircher wrote: > >> Hey Chas, >> >> >> I'm not quite sure if I understand your scenario so this is just a quick >> guess. The maximum number of opened files is depending on your operating >> system. Are you sure you close all files, sockets, streams that are no >> longer needed? >> >> Otherwise you could try to raise the file limit depending on your system >> ... but this might be a no-go for you as the problem is eventually >> postponed, but not fixed. >> >> >> Cheers, >> Lukas >> >> >> >> On Tue, Jan 4, 2011 at 8:57 PM, Charles F. Munat >> <charles.munat at gmail.com <mailto:charles.munat at gmail.com>> wrote: >> >> Got this error: >> >> org.basex.core.BaseXException: /var/www/xxx/db/data/atv.basex (Too >> many open files) >> >> Oh, noes! What does this mean? I was just hitting the page quickly. >> I'm sunk if people can't hit the page repeatedly. Is there a setting >> to adjust this? Is it a heap space issue? How can I make sure this >> never happens? >> >> I have an embedded database. I load it through the application (no >> file). I don't know what you call that. It uses one big XQuery to >> generate the page. Any ideas? >> >> Thanks very much! >> >> Chas. >> _______________________________________________ >> BaseX-Talk mailing list >> BaseX-Talk at mailman.uni-konstanz.de >> <mailto:BaseX-Talk at mailman.uni-konstanz.de> >> >> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >> >> >>
CM
I think that the problem might have been that I had the ClientGUI open and was accessing the embedded database at the same time (same database) and had numerous other databases possibly "open" (plus a couple hundred files). If it doesn't happen on the server, I'm happy.
I'm not very clear what the scenario is either...
If no one else has a problem with this, it was probably just an extremely unusual situation.
Thanks!
Chas.
Quoted text · Lukas Kircher · 46 lines
On 01/04/2011 5:48 PM, Lukas Kircher wrote: > Hey Chas, > > > I'm not quite sure if I understand your scenario so this is just a quick > guess. The maximum number of opened files is depending on your operating > system. Are you sure you close all files, sockets, streams that are no > longer needed? > > Otherwise you could try to raise the file limit depending on your system > ... but this might be a no-go for you as the problem is eventually > postponed, but not fixed. > > > Cheers, > Lukas > > > > On Tue, Jan 4, 2011 at 8:57 PM, Charles F. Munat > <charles.munat at gmail.com <mailto:charles.munat at gmail.com>> wrote: > > Got this error: > > org.basex.core.BaseXException: /var/www/xxx/db/data/atv.basex (Too > many open files) > > Oh, noes! What does this mean? I was just hitting the page quickly. > I'm sunk if people can't hit the page repeatedly. Is there a setting > to adjust this? Is it a heap space issue? How can I make sure this > never happens? > > I have an embedded database. I load it through the application (no > file). I don't know what you call that. It uses one big XQuery to > generate the page. Any ideas? > > Thanks very much! > > Chas. > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > <mailto:BaseX-Talk at mailman.uni-konstanz.de> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
GB
On Tue, Jan 4, 2011 at 4:24 PM, Lukas Kircher
<lukaskircher1 at googlemail.com>wrote:
Quoted text · Lukas Kircher · 5 lines
> As the hard file limit is around 2^16, depending on your system, it's very > unlikely to burst the roof in a real world application. Somebody might > correct me if I'm wrong ... > >
On typical Linux installations, die open filedescriptor limit is 1024:
$ grep 'open files' /proc/self/limits
Max open files 1024 1024 files
In Java, if a file-based object (FileWriter, FileReader, etc.) is not closed, the underlying file descriptor is not closed. Bug detectors (FindBugs) check for that intraprocedurally. I recommend you run both BaseX as well as this application through it.
If an open file-based object becomes unreachable, the finalizer will eventually close it - but it's possible to run out of open file descriptors simply due to unreachable, but not yet finalized objects. (Of course, if the object is leaked, it won't be closed ever.)
In Linux, use 'ps' to find out the pid of the Java JVM process, then do a ls -l /proc/<pid>/fds to see which file descriptors the process in question has open; or use the 'lsof' command.
- Godmar
Quoted text · Lukas Kircher · 79 lines
> Glad to hear that it works now! (fingers crossed ... ) > > > > > On Tue, Jan 4, 2011 at 10:14 PM, Charles F. Munat < > charles.munat at gmail.com> wrote: > >> I think that the problem might have been that I had the ClientGUI open and >> was accessing the embedded database at the same time (same database) and had >> numerous other databases possibly "open" (plus a couple hundred files). If >> it doesn't happen on the server, I'm happy. >> >> I'm not very clear what the scenario is either... >> >> If no one else has a problem with this, it was probably just an extremely >> unusual situation. >> >> Thanks! >> >> Chas. >> >> >> On 01/04/2011 5:48 PM, Lukas Kircher wrote: >> >>> Hey Chas, >>> >>> >>> I'm not quite sure if I understand your scenario so this is just a quick >>> guess. The maximum number of opened files is depending on your operating >>> system. Are you sure you close all files, sockets, streams that are no >>> longer needed? >>> >>> Otherwise you could try to raise the file limit depending on your system >>> ... but this might be a no-go for you as the problem is eventually >>> postponed, but not fixed. >>> >>> >>> Cheers, >>> Lukas >>> >>> >>> >>> On Tue, Jan 4, 2011 at 8:57 PM, Charles F. Munat >>> <charles.munat at gmail.com <mailto:charles.munat at gmail.com>> wrote: >>> >>> Got this error: >>> >>> org.basex.core.BaseXException: /var/www/xxx/db/data/atv.basex (Too >>> many open files) >>> >>> Oh, noes! What does this mean? I was just hitting the page quickly. >>> I'm sunk if people can't hit the page repeatedly. Is there a setting >>> to adjust this? Is it a heap space issue? How can I make sure this >>> never happens? >>> >>> I have an embedded database. I load it through the application (no >>> file). I don't know what you call that. It uses one big XQuery to >>> generate the page. Any ideas? >>> >>> Thanks very much! >>> >>> Chas. >>> _______________________________________________ >>> BaseX-Talk mailing list >>> BaseX-Talk at mailman.uni-konstanz.de >>> <mailto:BaseX-Talk at mailman.uni-konstanz.de> >>> >>> https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk >>> >>> >>> > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
CM
Charles F. Munat4 Jan 2011 23:41 #
Very useful, thanks. I'm pretty sure all these will work on the Mac -- it's BSD-base. But I think it was a dev machine problem, and I am happy as long as it never happens on the server.
I have learned a bit about the Linux file system here.
Thanks!
Chas.
Quoted text · Godmar Back · 126 lines
On 01/04/2011 6:31 PM, Godmar Back wrote: > On Tue, Jan 4, 2011 at 4:24 PM, Lukas Kircher > <lukaskircher1 at googlemail.com <mailto:lukaskircher1 at googlemail.com>> wrote: > > As the hard file limit is around 2^16, depending on your system, > it's very unlikely to burst the roof in a real world application. > Somebody might correct me if I'm wrong ... > > > On typical Linux installations, die open filedescriptor limit is 1024: > > $ grep 'open files' /proc/self/limits > Max open files 1024 1024 files > > In Java, if a file-based object (FileWriter, FileReader, etc.) is not > closed, the underlying file descriptor is not closed. Bug detectors > (FindBugs) check for that intraprocedurally. I recommend you run both > BaseX as well as this application through it. > > If an open file-based object becomes unreachable, the finalizer will > eventually close it - but it's possible to run out of open file > descriptors simply due to unreachable, but not yet finalized objects. > (Of course, if the object is leaked, it won't be closed ever.) > > In Linux, use 'ps' to find out the pid of the Java JVM process, then do > a ls -l /proc/<pid>/fds to see which file descriptors the process in > question has open; or use the 'lsof' command. > > - Godmar > > Glad to hear that it works now! (fingers crossed ... ) > > > > > On Tue, Jan 4, 2011 at 10:14 PM, Charles F. Munat > <charles.munat at gmail.com <mailto:charles.munat at gmail.com>> wrote: > > I think that the problem might have been that I had the > ClientGUI open and was accessing the embedded database at the > same time (same database) and had numerous other databases > possibly "open" (plus a couple hundred files). If it doesn't > happen on the server, I'm happy. > > I'm not very clear what the scenario is either... > > If no one else has a problem with this, it was probably just an > extremely unusual situation. > > Thanks! > > Chas. > > > On 01/04/2011 5:48 PM, Lukas Kircher wrote: > > Hey Chas, > > > I'm not quite sure if I understand your scenario so this is > just a quick > guess. The maximum number of opened files is depending on > your operating > system. Are you sure you close all files, sockets, streams > that are no > longer needed? > > Otherwise you could try to raise the file limit depending on > your system > ... but this might be a no-go for you as the problem is > eventually > postponed, but not fixed. > > > Cheers, > Lukas > > > > On Tue, Jan 4, 2011 at 8:57 PM, Charles F. Munat > <charles.munat at gmail.com <mailto:charles.munat at gmail.com> > <mailto:charles.munat at gmail.com > <mailto:charles.munat at gmail.com>>> wrote: > > Got this error: > > org.basex.core.BaseXException: > /var/www/xxx/db/data/atv.basex (Too > many open files) > > Oh, noes! What does this mean? I was just hitting the > page quickly. > I'm sunk if people can't hit the page repeatedly. Is > there a setting > to adjust this? Is it a heap space issue? How can I make > sure this > never happens? > > I have an embedded database. I load it through the > application (no > file). I don't know what you call that. It uses one big > XQuery to > generate the page. Any ideas? > > Thanks very much! > > Chas. > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > <mailto:BaseX-Talk at mailman.uni-konstanz.de> > <mailto:BaseX-Talk at mailman.uni-konstanz.de > <mailto:BaseX-Talk at mailman.uni-konstanz.de>> > > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > > > > > _______________________________________________ > BaseX-Talk mailing list > BaseX-Talk at mailman.uni-konstanz.de > <mailto:BaseX-Talk at mailman.uni-konstanz.de> > https://mailman.uni-konstanz.de/mailman/listinfo/basex-talk > >
Written for BaseX 6. BaseX may have changed since: please check the current documentation.
⚡Generated with XQuery