Discussion:
Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption
(too old to reply)
Pavel Szalbot
2017-07-10 16:52:10 UTC
Permalink
Raw Message
Hi,

is there a recommended way to upgrade Gluster cluster when upgrading
to newer revision? I experienced filesystem corruption on several but
not all VMs (KVM, FUSE) stored on Gluster during Gluster upgrade.

After upgrading one of two nodes, I checked peer status and volume
heal info, everything seemed fine so I upgraded second node and then
two VMs remounted root as read-only and dmesg contained I/O errors.

This did not happen in the past while following the same upgrade
procedure. Documentation mentions only upgrades to higher minor
version. Is there a recommended way to upgrade or did I do something
wrong that should be avoided?

Thanks for any suggestion
Pavel Szalbot
Mahdi Adnan
2017-07-10 17:03:20 UTC
Permalink
Raw Message
I upgraded from 3.8.12 to 3.8.13 without issues.

Two replicated volumes with online update, upgraded clients first and followed by servers upgrade, "stop glusterd, pkill gluster*, update gluster*, start glusterd, monitor healing process and logs, after completion proceed to the other node"

check gluster logs for more information.


--

Respectfully
Mahdi A. Mahdi

________________________________
From: gluster-users-***@gluster.org <gluster-users-***@gluster.org> on behalf of Pavel Szalbot <***@gmail.com>
Sent: Monday, July 10, 2017 7:52:10 PM
To: gluster-users
Subject: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

Hi,

is there a recommended way to upgrade Gluster cluster when upgrading
to newer revision? I experienced filesystem corruption on several but
not all VMs (KVM, FUSE) stored on Gluster during Gluster upgrade.

After upgrading one of two nodes, I checked peer status and volume
heal info, everything seemed fine so I upgraded second node and then
two VMs remounted root as read-only and dmesg contained I/O errors.

This did not happen in the past while following the same upgrade
procedure. Documentation mentions only upgrades to higher minor
version. Is there a recommended way to upgrade or did I do something
wrong that should be avoided?

Thanks for any suggestion
Pavel Szalbot
Pranith Kumar Karampuri
2017-07-11 03:41:28 UTC
Permalink
Raw Message
On Mon, Jul 10, 2017 at 10:33 PM, Mahdi Adnan <***@outlook.com>
wrote:

> I upgraded from 3.8.12 to 3.8.13 without issues.
>
> Two replicated volumes with online update, upgraded clients first and
> followed by servers upgrade, "stop glusterd, pkill gluster*, update
> gluster*, start glusterd, monitor healing process and logs, after
> completion proceed to the other node"
>
You should first upgrade servers and then clients. New servers can
understand old clients, but it is not easy for old servers to understand
new clients in case it started doing something new.


> check gluster logs for more information.
>
>
>
> --
>
> Respectfully
> *Mahdi A. Mahdi*
>
> ------------------------------
> *From:* gluster-users-***@gluster.org <gluster-users-bounces@
> gluster.org> on behalf of Pavel Szalbot <***@gmail.com>
> *Sent:* Monday, July 10, 2017 7:52:10 PM
> *To:* gluster-users
> *Subject:* [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13)
> caused underlying VM fs corruption
>
> Hi,
>
> is there a recommended way to upgrade Gluster cluster when upgrading
> to newer revision? I experienced filesystem corruption on several but
> not all VMs (KVM, FUSE) stored on Gluster during Gluster upgrade.
>
> After upgrading one of two nodes, I checked peer status and volume
> heal info, everything seemed fine so I upgraded second node and then
> two VMs remounted root as read-only and dmesg contained I/O errors.
>
> This did not happen in the past while following the same upgrade
> procedure. Documentation mentions only upgrades to higher minor
> version. Is there a recommended way to upgrade or did I do something
> wrong that should be avoided?
>
> Thanks for any suggestion
> Pavel Szalbot
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



--
Pranith
Diego Remolina
2017-07-11 11:42:37 UTC
Permalink
Raw Message
>
> You should first upgrade servers and then clients. New servers can
> understand old clients, but it is not easy for old servers to understand new
> clients in case it started doing something new.

But isn't that the reason op-version exists? So that regardless of
client/server mix, nobody tries to do "new" things above the current
op-version?

He is not changing mayor versions, just a small step from 3.8.12 to
3.8.13. Corruption should not be happening.

Diego
Pranith Kumar Karampuri
2017-07-11 11:55:47 UTC
Permalink
Raw Message
On Tue, Jul 11, 2017 at 5:12 PM, Diego Remolina <***@gmail.com> wrote:

> >
> > You should first upgrade servers and then clients. New servers can
> > understand old clients, but it is not easy for old servers to understand
> new
> > clients in case it started doing something new.
>
> But isn't that the reason op-version exists? So that regardless of
> client/server mix, nobody tries to do "new" things above the current
> op-version?
>

> He is not changing mayor versions, just a small step from 3.8.12 to
> 3.8.13. Corruption should not be happening.
>

For some reason 3.8 upgrade guide is not where it is supposed to be.
We highly recommend upgrading servers ahead of clients.
https://github.com/nixpanic/glusterdocs/commit/f6d48dc17f2cb6ee4680e372520ec3358641b2bc

I think at some point it is better to make this mandatory. Without a
predefined way of upgrading, it is very difficult to fix bugs in backward
compatible manner.

I am not sure why the corruption happened either :-(. Pavel, could you give
log files may be?


> Diego
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
>



--
Pranith
Pavel Szalbot
2017-07-11 12:29:48 UTC
Permalink
Raw Message
Well it was probably caused by running replica 2 and doing online
upgrade. However I added brick, turned volume to replica 3 with
arbiter and got very strange issue I will mail to this list in a
moment...

Thanks.
-ps


On Tue, Jul 11, 2017 at 1:55 PM, Pranith Kumar Karampuri
<***@redhat.com> wrote:
>
>
> On Tue, Jul 11, 2017 at 5:12 PM, Diego Remolina <***@gmail.com> wrote:
>>
>> >
>> > You should first upgrade servers and then clients. New servers can
>> > understand old clients, but it is not easy for old servers to understand
>> > new
>> > clients in case it started doing something new.
>>
>> But isn't that the reason op-version exists? So that regardless of
>> client/server mix, nobody tries to do "new" things above the current
>> op-version?
>>
>>
>> He is not changing mayor versions, just a small step from 3.8.12 to
>> 3.8.13. Corruption should not be happening.
>
>
> For some reason 3.8 upgrade guide is not where it is supposed to be.
> We highly recommend upgrading servers ahead of clients.
> https://github.com/nixpanic/glusterdocs/commit/f6d48dc17f2cb6ee4680e372520ec3358641b2bc
>
> I think at some point it is better to make this mandatory. Without a
> predefined way of upgrading, it is very difficult to fix bugs in backward
> compatible manner.
>
> I am not sure why the corruption happened either :-(. Pavel, could you give
> log files may be?
>
>>
>> Diego
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-***@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
> Pranith
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
Mahdi Adnan
2017-07-13 07:19:05 UTC
Permalink
Raw Message
Hi,


Doing online upgrade with replica 2 should be fine, i think there might be something else causing the corruption.


--

Respectfully
Mahdi A. Mahdi

________________________________
From: gluster-users-***@gluster.org <gluster-users-***@gluster.org> on behalf of Pavel Szalbot <***@gmail.com>
Sent: Tuesday, July 11, 2017 3:29:48 PM
To: Pranith Kumar Karampuri
Cc: gluster-users
Subject: Re: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

Well it was probably caused by running replica 2 and doing online
upgrade. However I added brick, turned volume to replica 3 with
arbiter and got very strange issue I will mail to this list in a
moment...

Thanks.
-ps


On Tue, Jul 11, 2017 at 1:55 PM, Pranith Kumar Karampuri
<***@redhat.com> wrote:
>
>
> On Tue, Jul 11, 2017 at 5:12 PM, Diego Remolina <***@gmail.com> wrote:
>>
>> >
>> > You should first upgrade servers and then clients. New servers can
>> > understand old clients, but it is not easy for old servers to understand
>> > new
>> > clients in case it started doing something new.
>>
>> But isn't that the reason op-version exists? So that regardless of
>> client/server mix, nobody tries to do "new" things above the current
>> op-version?
>>
>>
>> He is not changing mayor versions, just a small step from 3.8.12 to
>> 3.8.13. Corruption should not be happening.
>
>
> For some reason 3.8 upgrade guide is not where it is supposed to be.
> We highly recommend upgrading servers ahead of clients.
> https://github.com/nixpanic/glusterdocs/commit/f6d48dc17f2cb6ee4680e372520ec3358641b2bc
>
> I think at some point it is better to make this mandatory. Without a
> predefined way of upgrading, it is very difficult to fix bugs in backward
> compatible manner.
>
> I am not sure why the corruption happened either :-(. Pavel, could you give
> log files may be?
>
>>
>> Diego
>> _______________________________________________
>> Gluster-users mailing list
>> Gluster-***@gluster.org
>> http://lists.gluster.org/mailman/listinfo/gluster-users
>
>
>
>
> --
> Pranith
>
> _______________________________________________
> Gluster-users mailing list
> Gluster-***@gluster.org
> http://lists.gluster.org/mailman/listinfo/gluster-users
Mahdi Adnan
2017-07-11 06:08:34 UTC
Permalink
Raw Message
Noted, many thanks


--

Respectfully
Mahdi A. Mahdi

________________________________
From: Pranith Kumar Karampuri <***@redhat.com>
Sent: Tuesday, July 11, 2017 6:41:28 AM
To: Mahdi Adnan
Cc: Pavel Szalbot; gluster-users
Subject: Re: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption



On Mon, Jul 10, 2017 at 10:33 PM, Mahdi Adnan <***@outlook.com<mailto:***@outlook.com>> wrote:

I upgraded from 3.8.12 to 3.8.13 without issues.

Two replicated volumes with online update, upgraded clients first and followed by servers upgrade, "stop glusterd, pkill gluster*, update gluster*, start glusterd, monitor healing process and logs, after completion proceed to the other node"

You should first upgrade servers and then clients. New servers can understand old clients, but it is not easy for old servers to understand new clients in case it started doing something new.


check gluster logs for more information.


--

Respectfully
Mahdi A. Mahdi

________________________________
From: gluster-users-***@gluster.org<mailto:gluster-users-***@gluster.org> <gluster-users-***@gluster.org<mailto:gluster-users-***@gluster.org>> on behalf of Pavel Szalbot <***@gmail.com<mailto:***@gmail.com>>
Sent: Monday, July 10, 2017 7:52:10 PM
To: gluster-users
Subject: [Gluster-users] Upgrading Gluster revision (3.8.12 to 3.8.13) caused underlying VM fs corruption

Hi,

is there a recommended way to upgrade Gluster cluster when upgrading
to newer revision? I experienced filesystem corruption on several but
not all VMs (KVM, FUSE) stored on Gluster during Gluster upgrade.

After upgrading one of two nodes, I checked peer status and volume
heal info, everything seemed fine so I upgraded second node and then
two VMs remounted root as read-only and dmesg contained I/O errors.

This did not happen in the past while following the same upgrade
procedure. Documentation mentions only upgrades to higher minor
version. Is there a recommended way to upgrade or did I do something
wrong that should be avoided?

Thanks for any suggestion
Pavel Szalbot
_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org<mailto:Gluster-***@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org<mailto:Gluster-***@gluster.org>
http://lists.gluster.org/mailman/listinfo/gluster-users



--
Pranith
Loading...