[Gluster-users] How to set up a 4 way gluster file system

[Thing]

Hi,

I have 4 servers each with 1TB of storage set as /dev/sdb1, I would like to
set these up in a raid 10 which will? give me 2TB useable. So Mirrored and
concatenated?

The command I am running is as per documents but I get a warning error,
how do I get this to proceed please as the documents do not say.

gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
glusterp4:/bricks/brick1/gv0
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
avoid this. See:
http://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/
.
Do you still want to continue?
(y/n) n

Usage:
volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter
<COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>]
[transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force]

[***@glustep1 ~]#

thanks

[Milind Changire]

Post by Thing
Hi,
I have 4 servers each with 1TB of storage set as /dev/sdb1, I would like
to set these up in a raid 10 which will? give me 2TB useable. So Mirrored
and concatenated?
The command I am running is as per documents but I get a warning error,
how do I get this to proceed please as the documents do not say.
gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
glusterp4:/bricks/brick1/gv0
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/
Split%20brain%20and%20ways%20to%20deal%20with%20it/.
Do you still want to continue?
(y/n) n

This is not an error. It is a recommendation and an explicit cautionary
step to remind users of the problems using a replica 2.
It is advisable to use a replica 3 instead of a replica 2 to avoid
split-brain conditions.

If you still insist on using a replica 2, you can go ahead and answer a 'y'
to a the "Do you still want to continue ?" question.

Post by Thing
volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter
<COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>]
[transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force]
thanks
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

[Karthik Subrahmanya]

Hi,

With replica 2 volumes one can easily end up in split-brains if there are
frequent disconnects and high IOs going on.
If you use replica 3 or arbiter volumes, it will guard you by using the
quorum mechanism giving you both consistency and availability.
But in replica 2 volumes, quorum does not make sense since it needs both
the nodes up to guarantee consistency, which costs availability.

If you can consider having a replica 3 or arbiter volumes it would be
great. Otherwise you can anyway go ahead and continue with the replica 2
volume
by selecting *y* for the warning message. It will create the replica 2
configuration as you wanted.

HTH,
Karthik

Post by Thing
Hi,
I have 4 servers each with 1TB of storage set as /dev/sdb1, I would like
to set these up in a raid 10 which will? give me 2TB useable. So Mirrored
and concatenated?
The command I am running is as per documents but I get a warning error,
how do I get this to proceed please as the documents do not say.
gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
glusterp4:/bricks/brick1/gv0
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/
Split%20brain%20and%20ways%20to%20deal%20with%20it/.
Do you still want to continue?
(y/n) n
volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter
<COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>]
[transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force]
thanks
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

[Thing]

Hi,

I have 4 nodes, so a quorum would be 3 of 4. The Q is I suppose why does
the documentation give this command as an example without qualifying it?

SO I am running the wrong command? I want a "raid10"

Post by Karthik Subrahmanya
Hi,
With replica 2 volumes one can easily end up in split-brains if there are
frequent disconnects and high IOs going on.
If you use replica 3 or arbiter volumes, it will guard you by using the
quorum mechanism giving you both consistency and availability.
But in replica 2 volumes, quorum does not make sense since it needs both
the nodes up to guarantee consistency, which costs availability.
If you can consider having a replica 3 or arbiter volumes it would be
great. Otherwise you can anyway go ahead and continue with the replica 2
volume
by selecting *y* for the warning message. It will create the replica 2
configuration as you wanted.
HTH,
Karthik

Post by Thing
Hi,
I have 4 servers each with 1TB of storage set as /dev/sdb1, I would like
to set these up in a raid 10 which will? give me 2TB useable. So Mirrored
and concatenated?
The command I am running is as per documents but I get a warning error,
how do I get this to proceed please as the documents do not say.
gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
glusterp4:/bricks/brick1/gv0
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
avoid this. See: http://docs.gluster.org/en/lat
est/Administrator%20Guide/Split%20brain%20and%20ways%20to%
20deal%20with%20it/.
Do you still want to continue?
(y/n) n
volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter
<COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>]
[transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force]
thanks
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

[Sunil Kumar Heggodu Gopala Acharya]

Hi,

Post by Thing

Post by Thing
gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0

glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
glusterp4:/bricks/brick1/gv0

This command will create a distributed-replicate volume(yes you have to opt
'y' at the warning message to get it created). We will have two
distribution legs each containing a replica pair(made of two bricks). When
a file is placed on the volume by a user, it will be placed in one of the
distribution legs which as mentioned earlier will have only two copies.
With replica 2 volumes(volume type: replicate /distributed-replicate) we
might hit split-brain situation. So we recommend replica 3 or arbiter
volume to provide consistency and availability.

Regards,

Sunil kumar Acharya

Senior Software Engineer

Red Hat

<https://www.redhat.com>

T: [phone] <http://redhatemailsignature-marketing.itos.redhat.com/>

<https://red.ht/sig>
TRIED. TESTED. TRUSTED. <https://redhat.com/trusted>

Post by Thing
Hi,
I have 4 nodes, so a quorum would be 3 of 4. The Q is I suppose why does
the documentation give this command as an example without qualifying it?
SO I am running the wrong command? I want a "raid10"

Post by Thing
Hi,
With replica 2 volumes one can easily end up in split-brains if there are
frequent disconnects and high IOs going on.
If you use replica 3 or arbiter volumes, it will guard you by using the
quorum mechanism giving you both consistency and availability.
But in replica 2 volumes, quorum does not make sense since it needs both
the nodes up to guarantee consistency, which costs availability.
If you can consider having a replica 3 or arbiter volumes it would be
great. Otherwise you can anyway go ahead and continue with the replica 2
volume
by selecting *y* for the warning message. It will create the replica 2
configuration as you wanted.
HTH,
Karthik

Post by Thing
Hi,
I have 4 servers each with 1TB of storage set as /dev/sdb1, I would like
to set these up in a raid 10 which will? give me 2TB useable. So Mirrored
and concatenated?
The command I am running is as per documents but I get a warning error,
how do I get this to proceed please as the documents do not say.
gluster volume create gv0 replica 2 glusterp1:/bricks/brick1/gv0
glusterp2:/bricks/brick1/gv0 glusterp3:/bricks/brick1/gv0
glusterp4:/bricks/brick1/gv0
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to
avoid this. See: http://docs.gluster.org/en/lat
est/Administrator%20Guide/Split%20brain%20and%20ways%20to%20
deal%20with%20it/.
Do you still want to continue?
(y/n) n
volume create <NEW-VOLNAME> [stripe <COUNT>] [replica <COUNT> [arbiter
<COUNT>]] [disperse [<COUNT>]] [disperse-data <COUNT>] [redundancy <COUNT>]
[transport <tcp|rdma|tcp,rdma>] <NEW-BRICK>?<vg_name>... [force]
thanks
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

[Dave Sherohman]

Post by Thing
I have 4 nodes, so a quorum would be 3 of 4.

Nope, gluster doesn't work quite the way you're looking at it.
(Incidentally, I started off with the same expectation that you have.)

When you create a 4-brick replica 2 volume, you don't get a single
cluster with a quorum of 3 out of 4 bricks. You get two subvolumes,
each of which consists of two mirrored bricks.

Each individual subvolume is susceptible to split-brain if one of the
two bricks in that pair goes down, regardless of how many bricks in
other subvolumes are still up. Thus, quorum has to be handled on the
subvolume level rather than only being a consideration for the overall
volume as a whole.

One small wrinkle here is that, for calculating quorum, gluster treats
the brick in each pair which was listed first when you created the
volume as "one plus epsilon", so the subvolume will continue to operate
normally if the second brick goes down, but not if the first brick is
missing.

The easy solution to this is to switch from replica 2 to replica 2 +
arbiter. Arbiter bricks don't need to be nearly as large as data bricks
because they store only file metadata, not file contents, so you can
just scrape up a little spare disk space on two of your boxes, call that
space an arbiter, and run with it. In my case, I have 10T data bricks
and 100G arbiter bricks; I'm using a total of under 1G across all
arbiter bricks for 3T of data in the volume.