Discussion:
Clarification about remove-brick
(too old to reply)
Richard Klein (RSI)
2016-07-28 19:47:21 UTC
Permalink
We are using Gluster 3.7.6 in a replica 2 distributed-replicate configuration. I am wondering when we do a remove-brick with just one brick pair will the data be moved off the bricks once the status show complete and then you do the commit? Also, if you start a remove-brick process can you stop it? Is there an abort or stop command or do you just don't do the commit?

Any help would be appreciated.

Richard Klein
RSI
Lenovo Lastname
2016-07-28 19:55:02 UTC
Permalink
I'm using 3.7.11, this command works with me,
!remove-brick
[***@node2 ~]# gluster volume remove-brick v1 replica 2 192.168.3.73:/gfs/b1/v1 force
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit force: success
Don't know about the commit thingy...

On Thursday, July 28, 2016 3:47 PM, Richard Klein (RSI) <***@rsitex.com> wrote:


<!--#yiv5393753197 _filtered #yiv5393753197 {font-family:Calibri;panose-1:2 15 5 2 2 2 4 3 2 4;}#yiv5393753197 #yiv5393753197 p.yiv5393753197MsoNormal, #yiv5393753197 li.yiv5393753197MsoNormal, #yiv5393753197 div.yiv5393753197MsoNormal {margin:0in;margin-bottom:.0001pt;font-size:11.0pt;font-family:"Calibri", "sans-serif";}#yiv5393753197 a:link, #yiv5393753197 span.yiv5393753197MsoHyperlink {color:blue;text-decoration:underline;}#yiv5393753197 a:visited, #yiv5393753197 span.yiv5393753197MsoHyperlinkFollowed {color:purple;text-decoration:underline;}#yiv5393753197 span.yiv5393753197EmailStyle17 {font-family:"Calibri", "sans-serif";color:windowtext;}#yiv5393753197 .yiv5393753197MsoChpDefault {font-family:"Calibri", "sans-serif";} _filtered #yiv5393753197 {margin:1.0in 1.0in 1.0in 1.0in;}#yiv5393753197 div.yiv5393753197WordSection1 {}-->We are using Gluster 3.7.6 in a replica 2 distributed-replicate configuration.  I am wondering when we do a remove-brick with just one brick pair will the data be moved off the bricks once the status show complete and then you do the commit?    Also, if you start a remove-brick process can you stop it?  Is there an abort or stop command or do you just don’t do the commit?  Any help would be appreciated.  Richard KleinRSI      
Mohammed Rafi K C
2016-07-29 06:09:32 UTC
Permalink
I will summarize the procedure for removing a brick with description.


1) start an add brick operation using gluster volume remov-brick
command. This command will mark the mentioned brick as a decommissioned
brick. Also, this will kick a process that will start migrating data
from the decommissioned brick to the other bricks.

2) Once the migration is finished you can safely do a remove-brick commit.

3) Or if you wish to stop the process and reset the decommissioned
brick, you can do remove-brick stop. This will not migrate the data back
to the decommissioned brick. It will stay in the other bricks and the
data will be still accessible, if you want to have proper load balancing
after this, you can start rebalance process.

4) If you wish to do an instant remove brick you can use force option,
which will not migrate data, hence your whole data in the removed brick
will be lost from mount point.
Post by Lenovo Lastname
I'm using 3.7.11, this command works with me,
!remove-brick
192.168.3.73:/gfs/b1/v1 force
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit force: success
Don't know about the commit thingy...
On Thursday, July 28, 2016 3:47 PM, Richard Klein (RSI)
We are using Gluster 3.7.6 in a replica 2 distributed-replicate
configuration. I am wondering when we do a remove-brick with just one
brick pair will the data be moved off the bricks once the status show
complete and then you do the commit? Also, if you start a
remove-brick process can you stop it? Is there an abort or stop
command or do you just don’t do the commit?
Any help would be appreciated.
Richard Klein
RSI
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
Mohammed Rafi K C
2016-07-29 06:37:32 UTC
Permalink
You can also see the documentation here
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Managing%20Volumes/#shrinking-volumes


Rafi KC
Post by Mohammed Rafi K C
I will summarize the procedure for removing a brick with description.
1) start an add brick operation using gluster volume remov-brick
command. This command will mark the mentioned brick as a
decommissioned brick. Also, this will kick a process that will start
migrating data from the decommissioned brick to the other bricks.
2) Once the migration is finished you can safely do a remove-brick commit.
3) Or if you wish to stop the process and reset the decommissioned
brick, you can do remove-brick stop. This will not migrate the data
back to the decommissioned brick. It will stay in the other bricks and
the data will be still accessible, if you want to have proper load
balancing after this, you can start rebalance process.
4) If you wish to do an instant remove brick you can use force option,
which will not migrate data, hence your whole data in the removed
brick will be lost from mount point.
Post by Lenovo Lastname
I'm using 3.7.11, this command works with me,
!remove-brick
192.168.3.73:/gfs/b1/v1 force
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit force: success
Don't know about the commit thingy...
On Thursday, July 28, 2016 3:47 PM, Richard Klein (RSI)
We are using Gluster 3.7.6 in a replica 2 distributed-replicate
configuration. I am wondering when we do a remove-brick with just
one brick pair will the data be moved off the bricks once the status
show complete and then you do the commit? Also, if you start a
remove-brick process can you stop it? Is there an abort or stop
command or do you just don’t do the commit?
Any help would be appreciated.
Richard Klein
RSI
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
Richard Klein (RSI)
2016-07-29 17:38:33 UTC
Permalink
Thank you very much for the information. I have read the documentation link you have listed below and just wanted confirmation about the remove-brick process. I did not see any documentation about the ability to use the remove-brick stop command so that is good to know.

I have a couple of related follow up questions as well:


1) Let's say a brick fails (hard drive goes bad). I want to replace the bad drive with a new one and then get it back operational in the cluster. In this case I don't have spare brick available so I can't do the replace-brick. The only procedure I have found is a blog and bug report for 3.4 about this issue with a work around. Here is the link to the blog: https://joejulian.name/blog/replacing-a-brick-on-glusterfs-340/. Here is the link to the bug report: https://bugzilla.redhat.com/show_bug.cgi?id=991084. My apologies if this has been addressed before but I have searched and can't find a solution where you just replace the bad drive with a good one and recover. Any instructions about this process would be appreciated.

2) Does the rebalance process lock data files in Gluster? We are using Gluster as primary storage in Cloudstack 4.7. If we shrink or expand the Gluster volume we do a rebalance of the layout and data. However, after this process we have several VMs which have disk volumes in a read only state like there were disk problems. Once we power cycle the VM all is well and no data loss occurred but there seems to be a correlation between the rebalance and the errors. I am wondering if the rebalance process locks the data somehow and makes it unavailable to the VM.

Thanks again for the response.

Richard Klein
RSI


From: Mohammed Rafi K C [mailto:***@redhat.com]
Sent: Friday, July 29, 2016 1:38 AM
To: Lenovo Lastname; Richard Klein (RSI); gluster-***@gluster.org
Subject: Re: [Gluster-users] Clarification about remove-brick


You can also see the documentation here https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Managing%20Volumes/#shrinking-volumes


Rafi KC
On 07/29/2016 11:39 AM, Mohammed Rafi K C wrote:



I will summarize the procedure for removing a brick with description.



1) start an add brick operation using gluster volume remov-brick command. This command will mark the mentioned brick as a decommissioned brick. Also, this will kick a process that will start migrating data from the decommissioned brick to the other bricks.

2) Once the migration is finished you can safely do a remove-brick commit.

3) Or if you wish to stop the process and reset the decommissioned brick, you can do remove-brick stop. This will not migrate the data back to the decommissioned brick. It will stay in the other bricks and the data will be still accessible, if you want to have proper load balancing after this, you can start rebalance process.
4) If you wish to do an instant remove brick you can use force option, which will not migrate data, hence your whole data in the removed brick will be lost from mount point.

On 07/29/2016 01:25 AM, Lenovo Lastname wrote:
I'm using 3.7.11, this command works with me,

!remove-brick
[***@node2 ~]# gluster volume remove-brick v1 replica 2 192.168.3.73:/gfs/b1/v1 force
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit force: success

Don't know about the commit thingy...

On Thursday, July 28, 2016 3:47 PM, Richard Klein (RSI) <***@rsitex.com><mailto:***@rsitex.com> wrote:

We are using Gluster 3.7.6 in a replica 2 distributed-replicate configuration. I am wondering when we do a remove-brick with just one brick pair will the data be moved off the bricks once the status show complete and then you do the commit? Also, if you start a remove-brick process can you stop it? Is there an abort or stop command or do you just don't do the commit?

Any help would be appreciated.

Richard Klein
RSI




_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org<mailto:Gluster-***@gluster.org>
http://www.gluster.org/mailman/listinfo/gluster-users




_______________________________________________

Gluster-users mailing list

Gluster-***@gluster.org<mailto:Gluster-***@gluster.org>

http://www.gluster.org/mailman/listinfo/gluster-users
Mohammed Rafi K C
2016-08-01 05:07:58 UTC
Permalink
Post by Richard Klein (RSI)
Thank you very much for the information. I have read the
documentation link you have listed below and just wanted confirmation
about the remove-brick process. I did not see any documentation about
the ability to use the remove-brick stop command so that is good to know.
I'm not sure, how much testing has been done in that area. But we do
have the remove stop command, and theoretically it should work. If you
do rebalance after remove-brick stop, then again the cluster will be
load balanced if files are already migrated as part of the remove brick
operation.
Post by Richard Klein (RSI)
1) Let’s say a brick fails (hard drive goes bad). I want to
replace the bad drive with a new one and then get it back operational
in the cluster. In this case I don’t have spare brick available so I
can’t do the replace-brick. The only procedure I have found is a blog
and bug report for 3.4 about this issue with a work around. Here is
https://joejulian.name/blog/replacing-a-brick-on-glusterfs-340/. Here
https://bugzilla.redhat.com/show_bug.cgi?id=991084. My apologies if
this has been addressed before but I have searched and can’t find a
solution where you just replace the bad drive with a good one and
recover. Any instructions about this process would be appreciated.
If I understand you correctly, you want to replace a drive and create
the brick with same name. If that is the case, patch
http://review.gluster.org/#/c/12250/ will solve the. In fact the patch
is under review. CCing the replace brick experts to give you more info
regarding your replace queries.
Post by Richard Klein (RSI)
2) Does the rebalance process lock data files in Gluster? We are
using Gluster as primary storage in Cloudstack 4.7. If we shrink or
expand the Gluster volume we do a rebalance of the layout and data.
However, after this process we have several VMs which have disk
volumes in a read only state like there were disk problems. Once we
power cycle the VM all is well and no data loss occurred but there
seems to be a correlation between the rebalance and the errors. I am
wondering if the rebalance process locks the data somehow and makes it
unavailable to the VM.
Rebalance process will take locks as part of the healing. But I'm not
sure whether it cause VM's to go into read-only state. The locks will be
took on directory for healing and for files during migration, but only
for a specific offset. So both should not cause an entire file to be
read-only for a long time.

Can you describe your problem more ? I will try to see if there is
anything else causing the issue.


Regards
Rafi KC
Post by Richard Klein (RSI)
Thanks again for the response.
Richard Klein
RSI
*Sent:* Friday, July 29, 2016 1:38 AM
*Subject:* Re: [Gluster-users] Clarification about remove-brick
You can also see the documentation here
https://gluster.readthedocs.io/en/latest/Administrator%20Guide/Managing%20Volumes/#shrinking-volumes
Rafi KC
I will summarize the procedure for removing a brick with description.
1) start an add brick operation using gluster volume remov-brick
command. This command will mark the mentioned brick as a
decommissioned brick. Also, this will kick a process that will
start migrating data from the decommissioned brick to the other
bricks.
2) Once the migration is finished you can safely do a remove-brick commit.
3) Or if you wish to stop the process and reset the decommissioned
brick, you can do remove-brick stop. This will not migrate the
data back to the decommissioned brick. It will stay in the other
bricks and the data will be still accessible, if you want to have
proper load balancing after this, you can start rebalance process.
4) If you wish to do an instant remove brick you can use force
option, which will not migrate data, hence your whole data in the
removed brick will be lost from mount point.
I'm using 3.7.11, this command works with me,
!remove-brick
192.168.3.73:/gfs/b1/v1 force
Removing brick(s) can result in data loss. Do you want to Continue? (y/n) y
volume remove-brick commit force: success
Don't know about the commit thingy...
On Thursday, July 28, 2016 3:47 PM, Richard Klein (RSI)
We are using Gluster 3.7.6 in a replica 2
distributed-replicate configuration. I am wondering when we
do a remove-brick with just one brick pair will the data be
moved off the bricks once the status show complete and then
you do the commit? Also, if you start a remove-brick
process can you stop it? Is there an abort or stop command or
do you just don’t do the commit?
Any help would be appreciated.
Richard Klein
RSI
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
Continue reading on narkive:
Loading...