Discussion:
[Gluster-users] ganesha.nfsd process dies when copying files
Karli Sjöberg
2018-08-10 12:08:57 UTC
Permalink
Hey all!

I am playing around on my computer with setting up a virtual mini-
cluster of five VM's:

1x router
1x client
3x Gluster/NFS-Ganesha servers

The router is pfSense, the client is Xubuntu 18.04 and the servers are
CentOS 7.5.

I set up the cluster using 'gdeploy' with configuration snippets taken
from oVirt/Cockpit HCI setup and another snippet for setting up the
NFS-Ganesha part of it. The configuration is successful apart from some
minor details I debugged but I'm fairly sure I haven't made any obvious
misses.

All of the VM's are registered in pfSense's DNS, as well as the VIP's
for the NFS-Ganesha nodes, which works great and the client have no
issues with resolving any of the names.

hv01.localdomain 192.168.1.101
hv02.localdomain 192.168.1.102
hv03.localdomain 192.168.1.103
hv01v.localdomain 192.168.1.110
hv02v.localdomain 192.168.1.111
hv03v.localdomain 192.168.1.112

The cluster status is HEALTHY accoring to
'/usr/libexec/ganesha/ganesha-ha.sh' before I start my tests:

client# mount -t nfs -o vers=4.1 hv01v.localdomain:/data /mnt
client# dd if=/dev/urandom of=/var/tmp/test.bin bs=1M count=1024
client# while true; do rsync /var/tmp/test.bin /mnt/; rm -f
/mnt/test.bin; done

Then after a while, the 'nfs-ganesha' service unexpectedly dies and
doesn't restart by itself. The copy loop gets picked up after a while
on 'hv02' until history repeats itself until all of the nodes' 'nfs-
ganesha' services are dead.

With normal logs activated, the dead node says nothing before dying;
sudden heart attack syndrome- so no clues there, and ones remaining
only says they've taken over...

Right now I'm running with FULL_DEBUG which makes testing very
difficult since the throughput is down to a crawl. Nothing strange
about that, just takes a lot more time to provoke.

Please don't hesitate to ask for more information in case there's
something else you'd like me to share!

I'm hoping someone recognizes this behaviour and knows what I'm doing
wrong:)

glusterfs-client-xlators-3.10.12-1.el7.x86_64
glusterfs-api-3.10.12-1.el7.x86_64
nfs-ganesha-2.4.5-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-3.10.12-1.el7.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
nfs-ganesha-gluster-2.4.5-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-ganesha-3.10.12-1.el7.x86_64

Thanks in advance!

/K
Kaleb S. KEITHLEY
2018-08-10 12:39:46 UTC
Permalink
Post by Karli Sjöberg
Hey all!
...
glusterfs-client-xlators-3.10.12-1.el7.x86_64
glusterfs-api-3.10.12-1.el7.x86_64
nfs-ganesha-2.4.5-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-3.10.12-1.el7.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
nfs-ganesha-gluster-2.4.5-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-ganesha-3.10.12-1.el7.x86_64
For nfs-ganesha problems you'd really be better served by posting to
support@ or ***@lists.nfs-ganesha.org.

Both glusterfs-3.10 and nfs-ganesha-2.4 are really old. glusterfs-3.10
is even officially EOL. Ganesha isn't really organized enough to have
done anything as bold as officially declaring 2.4 as having reached EOL.

The nfs-ganesha devs are currently working on 2.7; maintaining and
supporting 2.6, and less so 2.5, is pretty much at the limit of what
they might be willing to help debug.

I strongly encourage you to update to a more recent version of both
glusterfs and nfs-ganesha. glusterfs-4.1 and nfs-ganesha-2.6 would be
ideal. Then if you still have problems you're much more likely to get help.
--
Kaleb
Karli Sjöberg
2018-08-10 13:08:39 UTC
Permalink
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Hey all!
...
glusterfs-client-xlators-3.10.12-1.el7.x86_64
glusterfs-api-3.10.12-1.el7.x86_64
nfs-ganesha-2.4.5-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-3.10.12-1.el7.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
nfs-ganesha-gluster-2.4.5-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-ganesha-3.10.12-1.el7.x86_64
For nfs-ganesha problems you'd really be better served by posting to
Both glusterfs-3.10 and nfs-ganesha-2.4 are really old. glusterfs-
3.10
is even officially EOL. Ganesha isn't really organized enough to have
done anything as bold as officially declaring 2.4 as having reached EOL.
The nfs-ganesha devs are currently working on 2.7; maintaining and
supporting 2.6, and less so 2.5, is pretty much at the limit of what
they might be willing to help debug.
I strongly encourage you to update to a more recent version of both
glusterfs and nfs-ganesha. glusterfs-4.1 and nfs-ganesha-2.6 would be
ideal. Then if you still have problems you're much more likely to get help.
Hi, thank you for your answer, but it raises even more questions about
any potential production deployment.

Actually, I knew that the versions are old, but it seems to me that you
are contradicting yourself:

https://lists.gluster.org/pipermail/gluster-users/2017-July/031753.html

"After 3.10 you'd need to use storhaug.... Which.... doesn't work
(yet).

You need to use 3.10 for now."

So how is that supposed to work?

Is there documentation for how to get there?

Thanks in advance!

/K
Pui Edylie
2018-08-10 13:23:39 UTC
Permalink
Hi Karli,

Storhaug works with glusterfs 4.1.2 and latest nfs-ganesha.


I just installed them last weekend ... they are working very well :)

Cheers,
Edy
Post by Karli Sjöberg
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Hey all!
...
glusterfs-client-xlators-3.10.12-1.el7.x86_64
glusterfs-api-3.10.12-1.el7.x86_64
nfs-ganesha-2.4.5-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-3.10.12-1.el7.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
nfs-ganesha-gluster-2.4.5-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-ganesha-3.10.12-1.el7.x86_64
For nfs-ganesha problems you'd really be better served by posting to
Both glusterfs-3.10 and nfs-ganesha-2.4 are really old. glusterfs-
3.10
is even officially EOL. Ganesha isn't really organized enough to have
done anything as bold as officially declaring 2.4 as having reached EOL.
The nfs-ganesha devs are currently working on 2.7; maintaining and
supporting 2.6, and less so 2.5, is pretty much at the limit of what
they might be willing to help debug.
I strongly encourage you to update to a more recent version of both
glusterfs and nfs-ganesha. glusterfs-4.1 and nfs-ganesha-2.6 would be
ideal. Then if you still have problems you're much more likely to get help.
Hi, thank you for your answer, but it raises even more questions about
any potential production deployment.
Actually, I knew that the versions are old, but it seems to me that you
https://lists.gluster.org/pipermail/gluster-users/2017-July/031753.html
"After 3.10 you'd need to use storhaug.... Which.... doesn't work
(yet).
You need to use 3.10 for now."
So how is that supposed to work?
Is there documentation for how to get there?
Thanks in advance!
/K
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Karli Sjöberg
2018-08-10 13:23:37 UTC
Permalink
Post by Pui Edylie
Hi Karli,
Storhaug works with glusterfs 4.1.2 and latest nfs-ganesha.
I just installed them last weekend ... they are working very well :)
Okay, awesome!

Is there any documentation on how to do that?

/K
Post by Pui Edylie
Cheers,
Edy
Post by Karli Sjöberg
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Hey all!
...
glusterfs-client-xlators-3.10.12-1.el7.x86_64
glusterfs-api-3.10.12-1.el7.x86_64
nfs-ganesha-2.4.5-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-3.10.12-1.el7.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
nfs-ganesha-gluster-2.4.5-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-ganesha-3.10.12-1.el7.x86_64
For nfs-ganesha problems you'd really be better served by posting to
Both glusterfs-3.10 and nfs-ganesha-2.4 are really old.
glusterfs-
3.10
is even officially EOL. Ganesha isn't really organized enough to have
done anything as bold as officially declaring 2.4 as having
reached
EOL.
The nfs-ganesha devs are currently working on 2.7; maintaining and
supporting 2.6, and less so 2.5, is pretty much at the limit of what
they might be willing to help debug.
I strongly encourage you to update to a more recent version of both
glusterfs and nfs-ganesha. glusterfs-4.1 and nfs-ganesha-2.6
would
be
ideal. Then if you still have problems you're much more likely to
get
help.
Hi, thank you for your answer, but it raises even more questions about
any potential production deployment.
Actually, I knew that the versions are old, but it seems to me that you
https://lists.gluster.org/pipermail/gluster-users/2017-July/031753.
html
"After 3.10 you'd need to use storhaug.... Which.... doesn't work
(yet).
You need to use 3.10 for now."
So how is that supposed to work?
Is there documentation for how to get there?
Thanks in advance!
/K
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Kaleb S. KEITHLEY
2018-08-10 13:39:55 UTC
Permalink
Post by Karli Sjöberg
Post by Pui Edylie
Hi Karli,
Storhaug works with glusterfs 4.1.2 and latest nfs-ganesha.
I just installed them last weekend ... they are working very well :)
Okay, awesome!
Is there any documentation on how to do that?
https://github.com/gluster/storhaug/wiki
--
Kaleb
Karli Sjöberg
2018-08-14 13:10:55 UTC
Permalink
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Post by Pui Edylie
Hi Karli,
Storhaug works with glusterfs 4.1.2 and latest nfs-ganesha.
I just installed them last weekend ... they are working very well :)
Okay, awesome!
Is there any documentation on how to do that?
https://github.com/gluster/storhaug/wiki
Thanks Kaleb and Edy!

I have now redone the cluster using the latest and greatest following
the above guide and repeated the same test I was doing before (the
rsync while loop) with success. I let (forgot) it run for about a day
and it was still chugging along nicely when I aborted it, so success
there!

On to the next test; the catastrophic failure test- where one of the
servers dies, I'm having a more difficult time with.

1) I start with mounting the share over NFS 4.1 and then proceed with
writing a 8 GiB large random data file with 'dd', while "hard-cutting"
the power to the server I'm writing to, the transfer just stops
indefinitely, until the server comes back again. Is that supposed to
happen? Like this:

# dd if=/dev/urandom of=/var/tmp/test.bin bs=1M count=8192
# mount -o vers=4.1 hv03v.localdomain:/data /mnt/
# dd if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress
2434793472 bytes (2,4 GB, 2,3 GiB) copied, 42 s, 57,9 MB/s

(here I cut the power and let it be for almost two hours before turning
it on again)

dd: error writing '/mnt/test.bin': Remote I/O error
2325+0 records in
2324+0 records out
2436890624 bytes (2,4 GB, 2,3 GiB) copied, 6944,84 s, 351 kB/s
# umount /mnt

Here the unmount command hung and I had to hard reset the client.

2) Another question I have is why some files "change" as you copy them
out to the Gluster storage? Is that the way it should be? This time, I
deleted eveything in the destination directory to start over:

# mount -o vers=4.1 hv03v.localdomain:/data /mnt/
# rm -f /mnt/test.bin
# dd if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress
8557428736 bytes (8,6 GB, 8,0 GiB) copied, 122 s, 70,1 MB/s
8192+0 records in
8192+0 records out
8589934592 bytes (8,6 GB, 8,0 GiB) copied, 123,039 s, 69,8 MB/s
# md5sum /var/tmp/test.bin
073867b68fa8eaa382ffe05adb90b583 /var/tmp/test.bin
# md5sum /mnt/test.bin
634187d367f856f3f5fb31846f796397 /mnt/test.bin
# umount /mnt

Thanks in advance!

/K
Alex Chekholko
2018-08-14 16:25:41 UTC
Permalink
Hi Karli,

I'm not 100% sure this is related, but when I set up my ZFS NFS HA per
https://github.com/ewwhite/zfs-ha/wiki I was not able to get the failover
to work with NFS v4 but only with NFS v3.

From the client point of view, it really looked like with NFS v4 there is
an open file handle and that just goes stale and hangs, or something like
that, whereas with NFSv3 the client retries and recovers and continues. I
did not investigate further, I just use v3. I think it has something to do
with NFSv4 being "stateful" and NFSv3 being "stateless".

Can you re-run your test but using NFSv3 on the client mount? Or do you
need to use v4.x?

Regards,
Alex
Post by Karli Sjöberg
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Post by Pui Edylie
Hi Karli,
Storhaug works with glusterfs 4.1.2 and latest nfs-ganesha.
I just installed them last weekend ... they are working very well :)
Okay, awesome!
Is there any documentation on how to do that?
https://github.com/gluster/storhaug/wiki
Thanks Kaleb and Edy!
I have now redone the cluster using the latest and greatest following
the above guide and repeated the same test I was doing before (the
rsync while loop) with success. I let (forgot) it run for about a day
and it was still chugging along nicely when I aborted it, so success
there!
On to the next test; the catastrophic failure test- where one of the
servers dies, I'm having a more difficult time with.
1) I start with mounting the share over NFS 4.1 and then proceed with
writing a 8 GiB large random data file with 'dd', while "hard-cutting"
the power to the server I'm writing to, the transfer just stops
indefinitely, until the server comes back again. Is that supposed to
# dd if=/dev/urandom of=/var/tmp/test.bin bs=1M count=8192
# mount -o vers=4.1 hv03v.localdomain:/data /mnt/
# dd if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress
2434793472 bytes (2,4 GB, 2,3 GiB) copied, 42 s, 57,9 MB/s
(here I cut the power and let it be for almost two hours before turning
it on again)
dd: error writing '/mnt/test.bin': Remote I/O error
2325+0 records in
2324+0 records out
2436890624 bytes (2,4 GB, 2,3 GiB) copied, 6944,84 s, 351 kB/s
# umount /mnt
Here the unmount command hung and I had to hard reset the client.
2) Another question I have is why some files "change" as you copy them
out to the Gluster storage? Is that the way it should be? This time, I
# mount -o vers=4.1 hv03v.localdomain:/data /mnt/
# rm -f /mnt/test.bin
# dd if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress
8557428736 bytes (8,6 GB, 8,0 GiB) copied, 122 s, 70,1 MB/s
8192+0 records in
8192+0 records out
8589934592 bytes (8,6 GB, 8,0 GiB) copied, 123,039 s, 69,8 MB/s
# md5sum /var/tmp/test.bin
073867b68fa8eaa382ffe05adb90b583 /var/tmp/test.bin
# md5sum /mnt/test.bin
634187d367f856f3f5fb31846f796397 /mnt/test.bin
# umount /mnt
Thanks in advance!
/K
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Pui Edylie
2018-08-15 05:42:40 UTC
Permalink
Hi Karli,

I think Alex is right in regards with the NFS version and state.

I am only using NFSv3 and the failover is working per expectation.

In my use case, I have 3 nodes with ESXI 6.7 as OS and setup 1x gluster
VM on each of the ESXI host using its local datastore.

Once I have formed the replicate 3, I use the CTDB VIP to present the
NFS3 back to the Vcenter and uses it as a shared storage.

Everything works great other than performance is not very good ... I am
still looking for ways to improve it.

Cheers,
Edy
Post by Pui Edylie
Hi Karli,
I'm not 100% sure this is related, but when I set up my ZFS NFS HA per
https://github.com/ewwhite/zfs-ha/wiki I was not able to get the
failover to work with NFS v4 but only with NFS v3.
From the client point of view, it really looked like with NFS v4 there
is an open file handle and that just goes stale and hangs, or
something like that, whereas with NFSv3 the client retries and
recovers and continues.  I did not investigate further, I just use
v3.  I think it has something to do with NFSv4 being "stateful" and
NFSv3 being "stateless".
Can you re-run your test but using NFSv3 on the client mount?  Or do
you need to use v4.x?
Regards,
Alex
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Post by Pui Edylie
Hi Karli,
Storhaug works with glusterfs 4.1.2 and latest nfs-ganesha.
I just installed them last weekend ... they are working very
well
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Post by Pui Edylie
:)
Okay, awesome!
Is there any documentation on how to do that?
https://github.com/gluster/storhaug/wiki
Thanks Kaleb and Edy!
I have now redone the cluster using the latest and greatest following
the above guide and repeated the same test I was doing before (the
rsync while loop) with success. I let (forgot) it run for about a day
and it was still chugging along nicely when I aborted it, so success
there!
On to the next test; the catastrophic failure test- where one of the
servers dies, I'm having a more difficult time with.
1) I start with mounting the share over NFS 4.1 and then proceed with
writing a 8 GiB large random data file with 'dd', while "hard-cutting"
the power to the server I'm writing to, the transfer just stops
indefinitely, until the server comes back again. Is that supposed to
# dd if=/dev/urandom of=/var/tmp/test.bin bs=1M count=8192
# mount -o vers=4.1 hv03v.localdomain:/data /mnt/
# dd if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress
2434793472 bytes (2,4 GB, 2,3 GiB) copied, 42 s, 57,9 MB/s
(here I cut the power and let it be for almost two hours before turning
it on again)
dd: error writing '/mnt/test.bin': Remote I/O error
2325+0 records in
2324+0 records out
2436890624 bytes (2,4 GB, 2,3 GiB) copied, 6944,84 s, 351 kB/s
# umount /mnt
Here the unmount command hung and I had to hard reset the client.
2) Another question I have is why some files "change" as you copy them
out to the Gluster storage? Is that the way it should be? This time, I
# mount -o vers=4.1 hv03v.localdomain:/data /mnt/
# rm -f /mnt/test.bin
# dd if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress
8557428736 bytes (8,6 GB, 8,0 GiB) copied, 122 s, 70,1 MB/s
8192+0 records in
8192+0 records out
8589934592 bytes (8,6 GB, 8,0 GiB) copied, 123,039 s, 69,8 MB/s
# md5sum /var/tmp/test.bin
073867b68fa8eaa382ffe05adb90b583  /var/tmp/test.bin
# md5sum /mnt/test.bin
634187d367f856f3f5fb31846f796397  /mnt/test.bin
# umount /mnt
Thanks in advance!
/K
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Karli Sjöberg
2018-08-15 11:14:08 UTC
Permalink
Post by Pui Edylie
Hi Karli,
I think Alex is right in regards with the NFS version and state.
I am only using NFSv3 and the failover is working per expectation.
OK, so I've remade the test again and it goes like this:

1) Start copy loop[*]
2) Power off hv02
3) Copy loop stalls indefinitely

I have attached a snippet of the ctdb log that looks interesting but
doesn't say much to me execpt that something's wrong:)

[*]: while true; do mount -o vers=3 hv03v.localdomain:/data /mnt/; dd
if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress; rm -fv
/mnt/test.bin; umount /mnt; done

Thanks in advance!

/K
Post by Pui Edylie
In my use case, I have 3 nodes with ESXI 6.7 as OS and setup 1x
gluster VM on each of the ESXI host using its local datastore.
Once I have formed the replicate 3, I use the CTDB VIP to present the
NFS3 back to the Vcenter and uses it as a shared storage.
Everything works great other than performance is not very good ... I
am still looking for ways to improve it.
Cheers,
Edy
Post by Pui Edylie
Hi Karli,
I'm not 100% sure this is related, but when I set up my ZFS NFS HA
per https://github.com/ewwhite/zfs-ha/wiki I was not able to get
the failover to work with NFS v4 but only with NFS v3.
From the client point of view, it really looked like with NFS v4
there is an open file handle and that just goes stale and hangs, or
something like that, whereas with NFSv3 the client retries and
recovers and continues. I did not investigate further, I just use
v3. I think it has something to do with NFSv4 being "stateful" and
NFSv3 being "stateless".
Can you re-run your test but using NFSv3 on the client mount? Or
do you need to use v4.x?
Regards,
Alex
Post by Pui Edylie
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Post by Pui Edylie
Hi Karli,
Storhaug works with glusterfs 4.1.2 and latest nfs-ganesha.
I just installed them last weekend ... they are working
very well
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Post by Pui Edylie
:)
Okay, awesome!
Is there any documentation on how to do that?
https://github.com/gluster/storhaug/wiki
Thanks Kaleb and Edy!
I have now redone the cluster using the latest and greatest following
the above guide and repeated the same test I was doing before (the
rsync while loop) with success. I let (forgot) it run for about a day
and it was still chugging along nicely when I aborted it, so success
there!
On to the next test; the catastrophic failure test- where one of the
servers dies, I'm having a more difficult time with.
1) I start with mounting the share over NFS 4.1 and then proceed with
writing a 8 GiB large random data file with 'dd', while "hard-
cutting"
the power to the server I'm writing to, the transfer just stops
indefinitely, until the server comes back again. Is that supposed to
# dd if=/dev/urandom of=/var/tmp/test.bin bs=1M count=8192
# mount -o vers=4.1 hv03v.localdomain:/data /mnt/
# dd if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress
2434793472 bytes (2,4 GB, 2,3 GiB) copied, 42 s, 57,9 MB/s
(here I cut the power and let it be for almost two hours before turning
it on again)
dd: error writing '/mnt/test.bin': Remote I/O error
2325+0 records in
2324+0 records out
2436890624 bytes (2,4 GB, 2,3 GiB) copied, 6944,84 s, 351 kB/s
# umount /mnt
Here the unmount command hung and I had to hard reset the client.
2) Another question I have is why some files "change" as you copy them
out to the Gluster storage? Is that the way it should be? This time, I
# mount -o vers=4.1 hv03v.localdomain:/data /mnt/
# rm -f /mnt/test.bin
# dd if=/var/tmp/test.bin of=/mnt/test.bin bs=1M status=progress
8557428736 bytes (8,6 GB, 8,0 GiB) copied, 122 s, 70,1 MB/s
8192+0 records in
8192+0 records out
8589934592 bytes (8,6 GB, 8,0 GiB) copied, 123,039 s, 69,8 MB/s
# md5sum /var/tmp/test.bin
073867b68fa8eaa382ffe05adb90b583 /var/tmp/test.bin
# md5sum /mnt/test.bin
634187d367f856f3f5fb31846f796397 /mnt/test.bin
# umount /mnt
Thanks in advance!
/K
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Pui Edylie
2018-08-10 13:40:29 UTC
Permalink
Hi Karli,

The following is my note which i gathered from google searches, storhaug
wiki and more google searches ... i might have missed certain steps and
this is based on Centos 7

install centos 7.x
yum update -y

i have disabled both firewalld and selinux

In our setup we are using LSI raid card RAID10 and present the virtual
drive partition as /dev/sdb

Create LVM so that we could utilise the the snapshot feature of gluster

pvcreate --dataalignment 256k /dev/sdb
vgcreate --physicalextentsize 256K gfs_vg /dev/sdb

set the volume to use all the space with -l 100%FREE
lvcreate --thinpool gfs_vg/thin_pool -l 100%FREE  --chunksize 256K
--poolmetadatasize 15G --zero n

we use XFS file system for our glusterfs
mkfs.xfs -i size=512 /dev/gfs_vg/thin_pool

Adding the following into /etc/fstab with mount point /bring1683 (you
could change the name accordingly)
/dev/gfs_vg/thin_pool                 /brick1683 xfs    defaults 1 2

Enable gluster 4.1 repro

vi /etc/yum.repos.d/Gluster.repo

[gluster41]
name=Gluster 4.1
baseurl=http://mirror.centos.org/centos/7/storage/$basearch/gluster-4.1/
gpgcheck=0
enabled=1

install gluster 4.1

yum install -y centos-release-gluster

Once we have done the above steps on our 3 nodes, login to 1 of the node
and issue the following

gluster volume create gv0 replica 3 192.168.0.1:/brick1683/gv0
192.168.0.2:/brick1684/gv0 192.168.0.3:/brick1685/gv0


Setting up HA for NFS-Ganesha using CTDB

install the storhaug package on all participating nodes
Install the storhaug package on all nodes using the appropriate command
for your system:

yum -y install storhaug-nfs

Note: this will install all the dependencies, e.g. ctdb,
nfs-ganesha-gluster, glusterfs, and their related dependencies.

Create a passwordless ssh key and copy it to all participating nodes
On one of the participating nodes (Fedora, RHEL, CentOS):
node1% ssh-keygen -f /etc/sysconfig/storhaug.d/secret.pem
or (Debian, Ubuntu):
node1% ssh-keygen -f /etc/default/storhaug.d/secret.pem
When prompted for a password, press the Enter key.

Copy the public key to all the nodes nodes (Fedora, RHEL, CentOS):
node1% ssh-copy-id -i /etc/sysconfig/storhaug.d/secret.pem.pub ***@node1
node1% ssh-copy-id -i /etc/sysconfig/storhaug.d/secret.pem.pub ***@node2
node1% ssh-copy-id -i /etc/sysconfig/storhaug.d/secret.pem.pub ***@node3

...

You can confirm that it works with (Fedora, RHEL, CentOS):
node1% ssh -oPasswordAuthentication=no -oStrictHostKeyChecking=no -i
/etc/sysconfig/storhaug.d/secret.pem ***@node1


populate /etc/ctdb/nodes and /etc/ctdb/public_addresses
Select one node as your lead node, e.g. node1. On the lead node,
create/edit /etc/ctdb/nodes and populate it with the (fixed) IP
addresses of the participating nodes. It should look like this:
192.168.122.81
192.168.122.82
192.168.122.83
192.168.122.84

On the lead node, create/edit /etc/ctdb/public_addresses and populate it
with the floating IP addresses (a.k.a. VIPs) for the participating
nodes. These must be different than the IP addresses in /etc/ctdb/nodes.
It should look like this:
192.168.122.85 eth0
192.168.122.86 eth0
192.168.122.87 eth0
192.168.122.88 eth0

edit /etc/ctdb/ctdbd.conf
Ensure that the line CTDB_MANAGES_NFS=yes exists. If not, add it or
change it from no to yes. Add or change the following lines:
CTDB_RECOVERY_LOCK=/run/gluster/shared_storage/.ctdb/reclock
CTDB_NFS_CALLOUT=/etc/ctdb/nfs-ganesha-callout
CTDB_NFS_STATE_FS_TYPE=glusterfs
CTDB_NFS_STATE_MNT=/run/gluster/shared_storage
CTDB_NFS_SKIP_SHARE_CHECK=yes
NFS_HOSTNAME=localhost

create a bare minimum /etc/ganesha/ganesha.conf file
On the lead node:
node1% touch /etc/ganesha/ganesha.conf
or
node1% echo "### NFS-Ganesha.config" > /etc/ganesha/ganesha.conf

Note: you can edit this later to set global configuration options.

create a trusted storage pool and start the gluster shared-storage volume
On all the participating nodes:
node1% systemctl start glusterd
node2% systemctl start glusterd
node3% systemctl start glusterd
...

On the lead node, peer probe the other nodes:
node1% gluster peer probe node2
node1% gluster peer probe node3
...

Optional: on one of the other nodes, peer probe node1:
node2% gluster peer probe node1

Enable the gluster shared-storage volume:
node1% gluster volume set all cluster.enable-shared-storage enable
This takes a few moments. When done check that the
gluster_shared_storage volume is mounted at /run/gluster/shared_storage
on all the nodes.

start the ctdbd and ganesha.nfsd daemons
On the lead node:
node1% storhaug setup
You can watch the ctdb (/var/log/ctdb.log) and ganesha log
(/var/log/ganesha/ganesha.log) to monitor their progress. From this
point on you may enter storhaug commands from any of the participating
nodes.

export a gluster volume
Create a gluster volume
node1% gluster volume create replica 2 myvol node1:/bricks/vol/myvol
node2:/bricks/vol/myvol node3:/bricks/vol/myvol node4:/bricks/vol/myvol ...

Start the gluster volume you just created
node1% gluster volume start myvol

Export the gluster volume from ganesha
node1% storhaug export myvol

Regards,
Edy
Post by Karli Sjöberg
Post by Pui Edylie
Hi Karli,
Storhaug works with glusterfs 4.1.2 and latest nfs-ganesha.
I just installed them last weekend ... they are working very well :)
Okay, awesome!
Is there any documentation on how to do that?
/K
Post by Pui Edylie
Cheers,
Edy
Post by Karli Sjöberg
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Hey all!
...
glusterfs-client-xlators-3.10.12-1.el7.x86_64
glusterfs-api-3.10.12-1.el7.x86_64
nfs-ganesha-2.4.5-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-3.10.12-1.el7.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
nfs-ganesha-gluster-2.4.5-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-ganesha-3.10.12-1.el7.x86_64
For nfs-ganesha problems you'd really be better served by posting to
Both glusterfs-3.10 and nfs-ganesha-2.4 are really old.
glusterfs-
3.10
is even officially EOL. Ganesha isn't really organized enough to have
done anything as bold as officially declaring 2.4 as having
reached
EOL.
The nfs-ganesha devs are currently working on 2.7; maintaining and
supporting 2.6, and less so 2.5, is pretty much at the limit of what
they might be willing to help debug.
I strongly encourage you to update to a more recent version of both
glusterfs and nfs-ganesha. glusterfs-4.1 and nfs-ganesha-2.6
would
be
ideal. Then if you still have problems you're much more likely to
get
help.
Hi, thank you for your answer, but it raises even more questions about
any potential production deployment.
Actually, I knew that the versions are old, but it seems to me that you
https://lists.gluster.org/pipermail/gluster-users/2017-July/031753.
html
"After 3.10 you'd need to use storhaug.... Which.... doesn't work
(yet).
You need to use 3.10 for now."
So how is that supposed to work?
Is there documentation for how to get there?
Thanks in advance!
/K
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Kaleb S. KEITHLEY
2018-08-10 13:36:37 UTC
Permalink
Post by Karli Sjöberg
Post by Kaleb S. KEITHLEY
Post by Karli Sjöberg
Hey all!
...
glusterfs-client-xlators-3.10.12-1.el7.x86_64
glusterfs-api-3.10.12-1.el7.x86_64
nfs-ganesha-2.4.5-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-3.10.12-1.el7.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
nfs-ganesha-gluster-2.4.5-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-ganesha-3.10.12-1.el7.x86_64
For nfs-ganesha problems you'd really be better served by posting to
Both glusterfs-3.10 and nfs-ganesha-2.4 are really old. glusterfs-
3.10
is even officially EOL. Ganesha isn't really organized enough to have
done anything as bold as officially declaring 2.4 as having reached EOL.
The nfs-ganesha devs are currently working on 2.7; maintaining and
supporting 2.6, and less so 2.5, is pretty much at the limit of what
they might be willing to help debug.
I strongly encourage you to update to a more recent version of both
glusterfs and nfs-ganesha. glusterfs-4.1 and nfs-ganesha-2.6 would be
ideal. Then if you still have problems you're much more likely to get help.
Hi, thank you for your answer, but it raises even more questions about
any potential production deployment.
Actually, I knew that the versions are old, but it seems to me that you
https://lists.gluster.org/pipermail/gluster-users/2017-July/031753.html
"After 3.10 you'd need to use storhaug.... Which.... doesn't work
(yet).
I don't recall the context of that email.

But I did also send this email
https://lists.gluster.org/pipermail/gluster-devel/2018-June/054896.html
announcing the availability of (working) storhaug-1.0.

There are packages in Fedora, CentOS Storage SIG, gluster's Launchpad
PPA, gluster's OBS, and for Debian on download.gluster.org.
--
Kaleb
Karli Sjöberg
2018-08-10 14:26:41 UTC
Permalink
_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
Karli Sjöberg
2018-08-15 05:48:17 UTC
Permalink
_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users
Karli Sjöberg
2018-08-16 17:29:50 UTC
Permalink
_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

Loading...