[Gluster-users] GlusterFS with FUSE slow vs ZFS volume

[ML mail]

Hello,

I am testing GlusterFS for the first time and have installed the latest GlusterFS 3.5 stable version on Debian 7 on brand new SuperMicro hardware with ZFS instead of hardware RAID. My ZFS pool is a RAIDZ-2 with 6 SATA disks of 2 TB each.

After setting up a first and single test brick on my currently single test node I wanted first to see how much slower will GlusterFS be compared to writting directly to the ZFS volume. For that purpose I have mounted my GlusterFS volume locally on the same server using FUSE.

For my tests I have used bonnie++ with the command "bonnie++ -n16 -b" and I must say I am quite shocked to see that with this current setup GlusterFS slows down the whole file system with a factor of approximately 6 to 8. For example:

ZFS volume

Sequential output by block (read): 936 MB/sec
Sequential input by block (write): 1520 MB/sec


GlusterFS on top of same ZFS volume mounted with FUSE
Sequential output by block (read): 114 MB/sec
Sequential input by block (write): 312 MB/sec


Now I was wondering if such a performance drop on a single GlusterFS node is expected? If not is it maybe ZFS which is messing up things?

bonnie++ took 3 minutes to rune on the ZFS volume and 18 minutes on the GlusterFS mount. I have copied the bonnie++ results below just in case in CVS format:

1.96,1.96,ZFS,1,1422907597,31960M,,170,99,936956,94,484417,74,463,99,1520120,98,815.4,41,16,,,,,3376,26,+++++,+++,3109,22,3261,21,+++++,+++,3305,20,66881us,15214us,84887us,23648us,53641us,93322us,39607us,363us,298ms,136ms,18us,176ms
1.96,1.96,GFS,1,1422897979,31960M,,16,17,114223,20,92610,20,+++++,+++,312557,14,444.5,6,16,,,,,385,3,5724,5,916,4,357,3,2044,4,750,4,550ms,9715us,23094us,3334us,125ms,90070us,154ms,8609us,17570us,67180us,4116us,7879us

Maybe they are a few performance tuning trick that I am not aware of?

Let me know if I should provide any more information. In advance thanks for your comments.

Best regards
ML

[Pranith Kumar Karampuri]

+Kiran Patil may know about this.

Pranith

Post by ML mail
Hello,
I am testing GlusterFS for the first time and have installed the latest GlusterFS 3.5 stable version on Debian 7 on brand new SuperMicro hardware with ZFS instead of hardware RAID. My ZFS pool is a RAIDZ-2 with 6 SATA disks of 2 TB each.
After setting up a first and single test brick on my currently single test node I wanted first to see how much slower will GlusterFS be compared to writting directly to the ZFS volume. For that purpose I have mounted my GlusterFS volume locally on the same server using FUSE.
ZFS volume
Sequential output by block (read): 936 MB/sec
Sequential input by block (write): 1520 MB/sec
GlusterFS on top of same ZFS volume mounted with FUSE
Sequential output by block (read): 114 MB/sec
Sequential input by block (write): 312 MB/sec
Now I was wondering if such a performance drop on a single GlusterFS node is expected? If not is it maybe ZFS which is messing up things?
1.96,1.96,ZFS,1,1422907597,31960M,,170,99,936956,94,484417,74,463,99,1520120,98,815.4,41,16,,,,,3376,26,+++++,+++,3109,22,3261,21,+++++,+++,3305,20,66881us,15214us,84887us,23648us,53641us,93322us,39607us,363us,298ms,136ms,18us,176ms
1.96,1.96,GFS,1,1422897979,31960M,,16,17,114223,20,92610,20,+++++,+++,312557,14,444.5,6,16,,,,,385,3,5724,5,916,4,357,3,2044,4,750,4,550ms,9715us,23094us,3334us,125ms,90070us,154ms,8609us,17570us,67180us,4116us,7879us
Maybe they are a few performance tuning trick that I am not aware of?
Let me know if I should provide any more information. In advance thanks for your comments.
Best regards
ML
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[Kiran Patil]

Hi,

We are using fio(https://github.com/axboe/fio) for load/stress testing.

We have not done performance check on a single node.

I will try to verify it.

Thanks,
Kiran.

On Thu, Feb 5, 2015 at 4:47 PM, Pranith Kumar Karampuri

Post by Pranith Kumar Karampuri
+Kiran Patil may know about this.
Pranith

Post by ML mail
Hello,
I am testing GlusterFS for the first time and have installed the latest
GlusterFS 3.5 stable version on Debian 7 on brand new SuperMicro hardware
with ZFS instead of hardware RAID. My ZFS pool is a RAIDZ-2 with 6 SATA
disks of 2 TB each.
After setting up a first and single test brick on my currently single test
node I wanted first to see how much slower will GlusterFS be compared to
writting directly to the ZFS volume. For that purpose I have mounted my
GlusterFS volume locally on the same server using FUSE.
For my tests I have used bonnie++ with the command "bonnie++ -n16 -b" and
I must say I am quite shocked to see that with this current setup GlusterFS
slows down the whole file system with a factor of approximately 6 to 8. For
ZFS volume
Sequential output by block (read): 936 MB/sec
Sequential input by block (write): 1520 MB/sec
GlusterFS on top of same ZFS volume mounted with FUSE
Sequential output by block (read): 114 MB/sec
Sequential input by block (write): 312 MB/sec
Now I was wondering if such a performance drop on a single GlusterFS node
is expected? If not is it maybe ZFS which is messing up things?
bonnie++ took 3 minutes to rune on the ZFS volume and 18 minutes on the
GlusterFS mount. I have copied the bonnie++ results below just in case in
1.96,1.96,ZFS,1,1422907597,31960M,,170,99,936956,94,484417,74,463,99,1520120,98,815.4,41,16,,,,,3376,26,+++++,+++,3109,22,3261,21,+++++,+++,3305,20,66881us,15214us,84887us,23648us,53641us,93322us,39607us,363us,298ms,136ms,18us,176ms
1.96,1.96,GFS,1,1422897979,31960M,,16,17,114223,20,92610,20,+++++,+++,312557,14,444.5,6,16,,,,,385,3,5724,5,916,4,357,3,2044,4,750,4,550ms,9715us,23094us,3334us,125ms,90070us,154ms,8609us,17570us,67180us,4116us,7879us
Maybe they are a few performance tuning trick that I am not aware of?
Let me know if I should provide any more information. In advance thanks
for your comments.
Best regards
ML
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users

[Vijay Bellur]

Post by ML mail
Hello,
I am testing GlusterFS for the first time and have installed the latest GlusterFS 3.5 stable version on Debian 7 on brand new SuperMicro hardware with ZFS instead of hardware RAID. My ZFS pool is a RAIDZ-2 with 6 SATA disks of 2 TB each.
After setting up a first and single test brick on my currently single test node I wanted first to see how much slower will GlusterFS be compared to writting directly to the ZFS volume. For that purpose I have mounted my GlusterFS volume locally on the same server using FUSE.
ZFS volume
Sequential output by block (read): 936 MB/sec
Sequential input by block (write): 1520 MB/sec
GlusterFS on top of same ZFS volume mounted with FUSE
Sequential output by block (read): 114 MB/sec
Sequential input by block (write): 312 MB/sec
Now I was wondering if such a performance drop on a single GlusterFS node is expected? If not is it maybe ZFS which is messing up things?
1.96,1.96,ZFS,1,1422907597,31960M,,170,99,936956,94,484417,74,463,99,1520120,98,815.4,41,16,,,,,3376,26,+++++,+++,3109,22,3261,21,+++++,+++,3305,20,66881us,15214us,84887us,23648us,53641us,93322us,39607us,363us,298ms,136ms,18us,176ms
1.96,1.96,GFS,1,1422897979,31960M,,16,17,114223,20,92610,20,+++++,+++,312557,14,444.5,6,16,,,,,385,3,5724,5,916,4,357,3,2044,4,750,4,550ms,9715us,23094us,3334us,125ms,90070us,154ms,8609us,17570us,67180us,4116us,7879us
Maybe they are a few performance tuning trick that I am not aware of?
Let me know if I should provide any more information. In advance thanks for your comments.

Is ZFS using SA based extended attributes here? Since GlusterFS makes
use of extended attributes for storing state, having SA support in ZFS
would be necessary for good performance.

Regards,
Vijay

[ML mail]

Yes, I have activated the SA xattr for my ZFS volume that I use for GlusterFS.




On Thursday, February 5, 2015 12:22 PM, Vijay Bellur <***@redhat.com> wrote:
On 02/02/2015 08:26 PM, ML mail wrote:

Is ZFS using SA based extended attributes here? Since GlusterFS makes
use of extended attributes for storing state, having SA support in ZFS
would be necessary for good performance.

Regards,
Vijay

[Charles Williams]

Vijay,

I first reported this problem on this list and in IRC somewhere around
the beginning to middle of last year and have since put all Gluster
integration projects on hold until I can free someone up to actually
start a more in-depth investigation as the situation was pretty much
chalked up to ineptitude when first reported.

If you do find a solution PLEASE let us know. Thanks.

Chuck