basex with ext3 vs ext4, [kjournald]
Hi all,
I'm been running basex on two environments, one is my laptop with an SSD drive and ext4, the other is a server with raid-1 HDD and ext3.
Using iotop, I am watching the disk access of the server and laptop, and comparing the two.
After some update queries, I notice that the server (ext3) has a process, [kjournald], which writes for about 20 seconds at 600-1000KB/sec. This only begins after the update is complete and the java process actually stops writing. I am not experiencing such a thing on my laptop. If I do too many of these queries on the server, it will eventually hang for 20-30 seconds.
I've copied the exact data on both the laptop and the server.
Are there any recommended filesystem types for basex? Does it sound like I've configured something incorrectly, or should I add some more drives to my server and have them using ext4? Or is the difference SSD vs HDD? I never saw my SSD write speed go above 200KB/sec or so during the same queries.
Any help would be greatly appreciated!
- Adrian
Just an update,
I have installed a new ext4 volume on the server and moved basex data over to it. Updates are now fast, so there was definitely an improvement. I will wait to see it running for a whole day before I draw conclusions, but I wasn't able to reproduce my high-io kjournald problem (or any other similar problem) with the data on ext4.
- Adrian
Quoted text · Adrian Hossu · 28 lines
On Sat, Jan 7, 2012 at 1:33 AM, Adrian Hossu <adibadi at gmail.com> wrote: > Hi all, > > I'm been running basex on two environments, one is my laptop with an SSD > drive and ext4, the other is a server with raid-1 HDD and ext3. > > Using iotop, I am watching the disk access of the server and laptop, and > comparing the two. > > After some update queries, I notice that the server (ext3) has a process, > [kjournald], which writes for about 20 seconds at 600-1000KB/sec. This > only begins after the update is complete and the java process actually > stops writing. I am not experiencing such a thing on my laptop. If I do > too many of these queries on the server, it will eventually hang for 20-30 > seconds. > > I've copied the exact data on both the laptop and the server. > > Are there any recommended filesystem types for basex? Does it sound like > I've configured something incorrectly, or should I add some more drives to > my server and have them using ext4? Or is the difference SSD vs HDD? I > never saw my SSD write speed go above 200KB/sec or so during the same > queries. > > Any help would be greatly appreciated! > > - Adrian >
Hi Adrian,
Quoted text · 01:33:24 · 11 lines
Am Samstag, 7. Januar 2012, 01:33:24 schrieb Adrian Hossu: > Hi all, > > I'm been running basex on two environments, one is my laptop with an SSD > drive and ext4, the other is a server with raid-1 HDD and ext3. > > Using iotop, I am watching the disk access of the server and laptop, and > comparing the two. > > After some update queries, I notice that the server (ext3) has a process, > [kjournald], which writes for about 20 seconds at 600-1000KB/sec.
kjournald is a kernel thread which updates the journal [1] of the file system.
Quoted text · Adrian Hossu · 4 lines
> This only begins after the update is complete and the java process actually > stops writing. I am not experiencing such a thing on my laptop. If I do > too many of these queries on the server, it will eventually hang for 20-30 > seconds.
Ext3 (and ext4) have different levels of journaling [2], defining what and when gets written to the journal and the main file system. It sounds to me, as if your file system is journaling both meta-data and file contents, since kjournald write a lot of data. I suggest, that you check with what arguments your file systems are mounted. Generally, you can see that in /etc/mtab; the corresponding line should contain something like "data=ordered". You could also check the documentation of your linux distribution.
Quoted text · Adrian Hossu · 3 lines
> I've copied the exact data on both the laptop and the server. > > Are there any recommended filesystem types for basex?
Personally, I'm not aware if there is a recommended file system, I think no, but generally it Should Just Work(tm). However, if someone has experimented with different types of file systems and/or types of disks may share results.
Quoted text · Adrian Hossu · 9 lines
> Does it sound like > I've configured something incorrectly, or should I add some more drives to > my server and have them using ext4? Or is the difference SSD vs HDD? I > never saw my SSD write speed go above 200KB/sec or so during the same > queries. > > Any help would be greatly appreciated! > > - Adrian
Regards, Dimitar
P.S. it's recommended to use Btrfs with SSDs, since the file system is designed to improve the wear leveling of the disk [3].
[1] http://en.wikipedia.org/wiki/Journaling_file_system [2] http://en.wikipedia.org/wiki/Ext3#Journaling_levels [3] http://en.wikipedia.org/wiki/Btrfs
Written for BaseX 7.0. BaseX may have changed since: please check the current documentation.