Discussion:
Strange file corruption
(too old to reply)
Udo Giacomozzi
2015-12-07 11:03:21 UTC
Permalink
Hi all,


yesterday I had a strange situation where Gluster healing corrupted
*all* my VM images.


In detail:
I had about 15 VMs running (in Proxmox 4.0) totaling about 600 GB of
qcow2 images. Gluster is used as storage for those images in replicate 3
setup (ie. 3 physical servers replicating all data).
All VMs were running on machine #1 - the two other machines (#2 and #3)
were *idle*.
Gluster was fully operating (no healing) when I rebooted machine #2.
For other reasons I had to reboot machines #2 and #3 a few times, but
since all VMs were running on machine #1 and nothing on the other
machines was accessing Gluster files, I was confident that this wouldn't
disturb Gluster.
But anyway this means that I rebootet Gluster nodes during a healing
process.

After a few minutes, Gluster files began showing corruption - up to the
point that the qcow2 files became unreadable and all VMs stopped working.

I was forced to restore VM backups (loosing a few hours of data), which
means that the corrupt files were left as-is by Proxmox and new qcow2
files were created to be used for the VMs.

This means that Gluster could continue healing it's files all night
long. Today, many files seem to be intact, but I think they have been
replaced with older versions (it's a bit difficult to tell exactly)

Please note that node #1 was up at all times. It seems to me that the
healing process corrupted the files. How can that be?
Anybody has an explanation? Any way to avoid such a situation (except
checking if Gluster is healing before rebooting)?

Setup details:
- Proxmox 4.0 cluster (not yet in HA mode) = Debian 8 Jessie
- redundant Gbit LAN (bonding)
- Gluster 3.5.2 (most current Proxmox package)
- two volumes, both "replicate" type, 1 x 3 = 3 bricks
- cluster.server-quorum-ratio: 51%

Thanks,
Udo
Lindsay Mathieson
2015-12-07 14:01:08 UTC
Permalink
Post by Udo Giacomozzi
- Proxmox 4.0 cluster (not yet in HA mode) = Debian 8 Jessie
- redundant Gbit LAN (bonding)
- Gluster 3.5.2 (most current Proxmox package)
- two volumes, both "replicate" type, 1 x 3 = 3 bricks
- cluster.server-quorum-ratio: 51%
Probably post the output from "gluster volume info" as well.
Udo Giacomozzi
2015-12-07 14:41:34 UTC
Permalink
Post by Lindsay Mathieson
Post by Udo Giacomozzi
- Proxmox 4.0 cluster (not yet in HA mode) = Debian 8 Jessie
- redundant Gbit LAN (bonding)
- Gluster 3.5.2 (most current Proxmox package)
- two volumes, both "replicate" type, 1 x 3 = 3 bricks
- cluster.server-quorum-ratio: 51%
Probably post the output from "gluster volume info" as well.
Hi Lindsay,

as requested:

# gluster volume info

Volume Name: docker-repo
Type: Replicate
Volume ID: d762efae-d558-466b-bb4b-8c698ff90569
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: metal1:/data/gluster/docker-repo
Brick2: metal2:/data/gluster/docker-repo
Brick3: metal3:/data/gluster/docker-repo
Options Reconfigured:
cluster.server-quorum-ratio: 51%

Volume Name: systems
Type: Replicate
Volume ID: b2d72784-4b0e-4f7b-b858-4ec59979a064
Status: Started
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: metal1:/data/gluster/systems
Brick2: metal2:/data/gluster/systems
Brick3: metal3:/data/gluster/systems
Options Reconfigured:
cluster.server-quorum-ratio: 51%


...and also:

# gluster volume heal "systems" info
Brick metal1:/data/gluster/systems/
Number of entries: 0

Brick metal2:/data/gluster/systems/
Number of entries: 0

Brick metal3:/data/gluster/systems/
Number of entries: 0


Simular output for volume "docker-repo".

Udo
Lindsay Mathieson
2015-12-08 01:59:00 UTC
Permalink
Hi Udo, thanks for posting your volume info settings. Please note for
the following, I am not one of the devs, just a user, so unfortunately I
have no authoritative answers :(

I am running a very similar setup - Proxmox 4.0, three nodes, but using
ceph for our production storage. Am heavily testing gluster 3.7 on the
side. We find the performance of ceph slow on these small setups and
management of it a PITA.


Some more questions

- how are your VM images being accessed by Proxmox? gfapi? (Proxmox
Gluster storage type) or by using the fuse mount?

- whats your underlying filesystem (ext4, zfs etc)

- Are you using the HA/Watchdog system in Proxmox?
Post by Udo Giacomozzi
esterday I had a strange situation where Gluster healing corrupted
*all* my VM images.
I had about 15 VMs running (in Proxmox 4.0) totaling about 600 GB of
qcow2 images. Gluster is used as storage for those images in replicate
3 setup (ie. 3 physical servers replicating all data).
All VMs were running on machine #1 - the two other machines (#2 and
#3) were *idle*.
Gluster was fully operating (no healing) when I rebooted machine #2.
For other reasons I had to reboot machines #2 and #3 a few times, but
since all VMs were running on machine #1 and nothing on the other
machines was accessing Gluster files, I was confident that this
wouldn't disturb Gluster.
But anyway this means that I rebootet Gluster nodes during a healing
process.
After a few minutes, Gluster files began showing corruption - up to
the point that the qcow2 files became unreadable and all VMs stopped
working.
:( sounds painful - my sympathies.

You're running 3.5.2 - thats getting rather old. I use the gluster
debian repos:

3.6.7 :
http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/Debian/
3.7.6 :
http://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/jessie/

3.6.x is the latest stable, 3.7 is close to stable(?) 3.7 has some nice
new features such as sharding, which is very useful for VM hosting - it
enables much faster heal times.

Regards what happened with your VM's, I'm not sure. Having two servers
down should have disabled the entire store making it not readable or
writable. I note that you are missing some settings that need to be set
for VM stores - there will be corruption problems if you live migrate
without them.

quick-read=off
read-ahead=off
io-cache=off
stat-prefetch=off
eager-lock=enable
remote-dio=enable
quorum-type=auto
server-quorum-type=server


"stat-prefetch=off" is particularly important.
Krutika Dhananjay
2015-12-08 06:57:08 UTC
Permalink
----- Original Message -----
Sent: Tuesday, December 8, 2015 7:29:00 AM
Subject: Re: [Gluster-users] Strange file corruption
Hi Udo, thanks for posting your volume info settings. Please note for the
following, I am not one of the devs, just a user, so unfortunately I have no
authoritative answers :(
I am running a very similar setup - Proxmox 4.0, three nodes, but using ceph
for our production storage. Am heavily testing gluster 3.7 on the side. We
find the performance of ceph slow on these small setups and management of it
a PITA.
Some more questions
- how are your VM images being accessed by Proxmox? gfapi? (Proxmox Gluster
storage type) or by using the fuse mount?
- whats your underlying filesystem (ext4, zfs etc)
- Are you using the HA/Watchdog system in Proxmox?
esterday I had a strange situation where Gluster healing corrupted * all *
my
VM images.
I had about 15 VMs running (in Proxmox 4.0) totaling about 600 GB of qcow2
images. Gluster is used as storage for those images in replicate 3 setup
(ie. 3 physical servers replicating all data).
All VMs were running on machine #1 - the two other machines (#2 and #3)
were
* idle * .
Gluster was fully operating (no healing) when I rebooted machine #2.
For other reasons I had to reboot machines #2 and #3 a few times, but since
all VMs were running on machine #1 and nothing on the other machines was
accessing Gluster files, I was confident that this wouldn't disturb
Gluster.
But anyway this means that I rebootet Gluster nodes during a healing
process.
After a few minutes, Gluster files began showing corruption - up to the
point
that the qcow2 files became unreadable and all VMs stopped working.
:( sounds painful - my sympathies.
You're running 3.5.2 - thats getting rather old. I use the gluster debian
3.6.7 : http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/Debian/
http://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/jessie/
3.6.x is the latest stable, 3.7 is close to stable(?) 3.7 has some nice new
features such as sharding, which is very useful for VM hosting - it enables
much faster heal times.
Regards what happened with your VM's, I'm not sure. Having two servers down
should have disabled the entire store making it not readable or writable. I
note that you are missing some settings that need to be set for VM stores -
there will be corruption problems if you live migrate without them.
quick-read=off
read-ahead=off
io-cache=off
stat-prefetch=off
eager-lock=enable
remote-dio=enable
quorum-type=auto
server-quorum-type=server
Perfectly put. I am one of the devs who work on replicate module. You can alternatively enable this configuration in one shot using the following command for VM workloads:
# gluster volume set <VOLNAME> group virt

Like Lindsay mentioned, it is not clear why the brick on the first node did not go down when the other two nodes were rebooted, given that server quorum is enabled on your volume.
Client-quorum option seems to be missing in your volume. Once you enable it, it is necessary for at least two bricks per replica set to be online for the writes to happen on the volume.
If two nodes are down, the volume becomes read-only, causing the VMs to possibly go into a paused state.

-Krutika
"stat-prefetch=off" is particularly important.
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
Lindsay Mathieson
2015-12-08 11:28:37 UTC
Permalink
Post by Krutika Dhananjay
You can alternatively enable this configuration in one shot using the
# gluster volume set <VOLNAME> group virt
Alas, it is not packaged in the debian packages:)

vi /var/lib/glusterd/groups/virt

And add the settings, then "gluster volume set docker-repo group virt"
will do the trick.
Udo Giacomozzi
2015-12-09 15:18:07 UTC
Permalink
Post by Lindsay Mathieson
quick-read=off
read-ahead=off
io-cache=off
stat-prefetch=off
eager-lock=enable
remote-dio=enable
quorum-type=auto
server-quorum-type=server
Perfectly put. I am one of the devs who work on replicate module. You
can alternatively enable this configuration in one shot using the
# gluster volume set <VOLNAME> group virt
The RedHat guide
<https://access.redhat.com/documentation/en-US/Red_Hat_Storage/2.0/html/Quick_Start_Guide/chap-Quick_Start_Guide-Virtual_Preparation.html>
Post by Lindsay Mathieson
After the volume is tagged using |group virt| command, you must not
use the volume for any other storage purpose, other than to store
virtual machine images. Also, ensure to access the volume only through
gluster native client.
Okay, I'll give it a try.
I've created a new volume, set the options mentioned above, accessing it
via native GlusterFS (using 127.0.0.1 address) and moved some
non-critical VMs to that storage.
We'll see it that works fine... :-)

Udo
Udo Giacomozzi
2015-12-09 14:41:33 UTC
Permalink
Post by Lindsay Mathieson
Hi Udo, thanks for posting your volume info settings. Please note for
the following, I am not one of the devs, just a user, so unfortunately
I have no authoritative answers :(
I am running a very similar setup - Proxmox 4.0, three nodes, but
using ceph for our production storage. Am heavily testing gluster 3.7
on the side. We find the performance of ceph slow on these small
setups and management of it a PITA.
Some more questions
- how are your VM images being accessed by Proxmox? gfapi? (Proxmox
Gluster storage type) or by using the fuse mount?
Sorry, forgot to say that: I'm accessing the Gluster Storage via NFS
since (at least in version 3.4 of Proxmox) the gfapi method has some
problems with sockets.
Post by Lindsay Mathieson
- whats your underlying filesystem (ext4, zfs etc)
a dedicated ext4 partition
Post by Lindsay Mathieson
- Are you using the HA/Watchdog system in Proxmox?
I am now (watchdog HA), but Proxmox was running in non-HA mode at the
time of failure.
Post by Lindsay Mathieson
Post by Udo Giacomozzi
esterday I had a strange situation where Gluster healing corrupted
*all* my VM images.
I had about 15 VMs running (in Proxmox 4.0) totaling about 600 GB of
qcow2 images. Gluster is used as storage for those images in
replicate 3 setup (ie. 3 physical servers replicating all data).
All VMs were running on machine #1 - the two other machines (#2 and
#3) were *idle*.
Gluster was fully operating (no healing) when I rebooted machine #2.
For other reasons I had to reboot machines #2 and #3 a few times, but
since all VMs were running on machine #1 and nothing on the other
machines was accessing Gluster files, I was confident that this
wouldn't disturb Gluster.
But anyway this means that I rebootet Gluster nodes during a healing
process.
After a few minutes, Gluster files began showing corruption - up to
the point that the qcow2 files became unreadable and all VMs stopped
working.
:( sounds painful - my sympathies.
You're running 3.5.2 - thats getting rather old. I use the gluster
http://download.gluster.org/pub/gluster/glusterfs/3.6/LATEST/Debian/
http://download.gluster.org/pub/gluster/glusterfs/LATEST/Debian/jessie/
3.6.x is the latest stable, 3.7 is close to stable(?) 3.7 has some
nice new features such as sharding, which is very useful for VM
hosting - it enables much faster heal times.
I'm using the most current version provided by Proxmox APT sources.

Can 3.6 or even 3.7 Gluster nodes work together with 3.5 node? If not
I'm wondering how I could upgrade...

You might understand that I hesitate a bit to upgrade Gluster without
having some certainty that it won't make things even worse. I mean, this
is a production system..
Post by Lindsay Mathieson
Regards what happened with your VM's, I'm not sure. Having two servers
down should have disabled the entire store making it not readable or
writable.
I'm not sure if both servers were down at the same time (could be,
though). I'm just sure that I rebootet them rather quickly in sequence.

Right now my credo is "/never ever reboot/shutdown more than 1 node at a
time and most importantly, always make sure that no Gluster healing is
in progress/". For sure, I did not respect that when I crashed my storage.
Post by Lindsay Mathieson
I note that you are missing some settings that need to be set for VM
stores - there will be corruption problems if you live migrate without
them.
quick-read=off
read-ahead=off
io-cache=off
stat-prefetch=off
eager-lock=enable
remote-dio=enable
quorum-type=auto
server-quorum-type=server
"stat-prefetch=off" is particularly important.
Thanks. Is there a document that explains the reasoning behind this config?

Does this apply to volumes for virtual HDD images only? My "docker-repo"
is still "replicate 3" type but is used by the VMs themselves, not by
the hypervisor - I guess other settings apply there..

Thanks a lot,
Udo
Lindsay Mathieson
2015-12-09 13:39:08 UTC
Permalink
Post by Udo Giacomozzi
All VMs were running on machine #1 - the two other machines (#2 and
#3) were *idle*.
Gluster was fully operating (no healing) when I rebooted machine #2.
For other reasons I had to reboot machines #2 and #3 a few times, but
since all VMs were running on machine #1 and nothing on the other
machines was accessing Gluster files, I was confident that this
wouldn't disturb Gluster.
But anyway this means that I rebootet Gluster nodes during a healing
process.
After a few minutes, Gluster files began showing corruption - up to
the point that the qcow2 files became unreadable and all VMs stopped
working.
Udo, it occurs to me that if your VM's were running on #2 & #3 and you
live migrated them to #1 prior to rebooting #2/3, then you would indeed
rapidly get progressive VM corruption.

However it wouldn't be due to the heal process, but rather the live
migration with "performance.stat-prefetch" on. This always leads to
qcow2 files becoming corrupted and unusable.
--
Lindsay Mathieson
Udo Giacomozzi
2015-12-09 14:53:11 UTC
Permalink
Post by Lindsay Mathieson
Udo, it occurs to me that if your VM's were running on #2 & #3 and you
live migrated them to #1 prior to rebooting #2/3, then you would
indeed rapidly get progressive VM corruption.
However it wouldn't be due to the heal process, but rather the live
migration with "performance.stat-prefetch" on. This always leads to
qcow2 files becoming corrupted and unusable.
Nope. All VMs were running on #1, no exception.
Nodes #2 and #3 never had a VM running on them, so they were pratically
idle since their installation.

Basically I set up node #1, including all VMs.
Then I've installed nodes #2 and #3, configured Proxmox and Gluster
cluster and then waited quite some time until Gluster had synced up
nodes #2 and #3 (healing).
From then on, I've rebooted nodes 2 & 3, but in theory these nodes
never had to do any writes to the Gluster volume at all.

If you're interested, you can read about my upgrade strategy in this
Proxmox forum post:
http://forum.proxmox.com/threads/24990-Upgrade-3-4-HA-cluster-to-4-0-via-reinstallation-with-minimal-downtime?p=125040#post125040

Also, It seems rather strange to me that pratically all ~15 VMs (!)
suffered from data corruption. It's like if Gluster considered node #2
or #3 to be ahead and it "healed" in the wrong direction. I don't know..

BTW, once I understood what was going on, /with the problematic
"healing" still in progress/, I was able to overwrite the bad images
(still active on #1) by using standard Proxmox backup-restore and
Gluster handled it correctly.


Anway, I really love the simplicity of Gluster (setting up and
maintaining a cluster is extremely easy), but these healing issues are
causing some headache to me... ;-)

Udo
Joe Julian
2015-12-09 16:17:48 UTC
Permalink
A-1) shut down node #1 (the first that is about to be upgraded)
A-2) remove node #1 from the Proxmox cluster (/pvevm delnode "metal1"/)
A-3) remove node #1 from the Gluster volume/cluster (/gluster volume
remove-brick ... && gluster peer detach "metal1"/)
A-4) install Debian Jessie on node #1, overwriting all data on the HDD
-*with same Network settings and hostname as before*
A-5)install Proxmox 4.0
<https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie>on
node #1
A-6) install Gluster on node #1 and add it back to the Gluster volume
(/gluster volume add-brick .../) => shared storage will be complete
again (spanning 3.4 and 4.0 nodes)
A-7) configure the Gluster volume as shared storage in Proxmox 4 (node #1)
A-8) configure the external Backup storage on node #1 (Proxmox 4)
Was the data on the gluster brick deleted as part of step 4? When you
remove the brick, gluster will no longer track pending changes for that
brick. If you add it back in with stale data but matching gfids, you
would have two clean bricks with mismatching data. Did you have to use
"add-brick...force"?
Post by Lindsay Mathieson
Udo, it occurs to me that if your VM's were running on #2 & #3 and
you live migrated them to #1 prior to rebooting #2/3, then you would
indeed rapidly get progressive VM corruption.
However it wouldn't be due to the heal process, but rather the live
migration with "performance.stat-prefetch" on. This always leads to
qcow2 files becoming corrupted and unusable.
Nope. All VMs were running on #1, no exception.
Nodes #2 and #3 never had a VM running on them, so they were
pratically idle since their installation.
Basically I set up node #1, including all VMs.
Then I've installed nodes #2 and #3, configured Proxmox and Gluster
cluster and then waited quite some time until Gluster had synced up
nodes #2 and #3 (healing).
From then on, I've rebooted nodes 2 & 3, but in theory these nodes
never had to do any writes to the Gluster volume at all.
If you're interested, you can read about my upgrade strategy in this
http://forum.proxmox.com/threads/24990-Upgrade-3-4-HA-cluster-to-4-0-via-reinstallation-with-minimal-downtime?p=125040#post125040
Also, It seems rather strange to me that pratically all ~15 VMs (!)
suffered from data corruption. It's like if Gluster considered node #2
or #3 to be ahead and it "healed" in the wrong direction. I don't know..
BTW, once I understood what was going on, /with the problematic
"healing" still in progress/, I was able to overwrite the bad images
(still active on #1) by using standard Proxmox backup-restore and
Gluster handled it correctly.
Anway, I really love the simplicity of Gluster (setting up and
maintaining a cluster is extremely easy), but these healing issues are
causing some headache to me... ;-)
Udo
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
Udo Giacomozzi
2015-12-09 17:15:35 UTC
Permalink
Post by Joe Julian
A-1) shut down node #1 (the first that is about to be upgraded)
A-2) remove node #1 from the Proxmox cluster (/pvevm delnode "metal1"/)
A-3) remove node #1 from the Gluster volume/cluster (/gluster volume
remove-brick ... && gluster peer detach "metal1"/)
A-4) install Debian Jessie on node #1, overwriting all data on the
HDD -*with same Network settings and hostname as before*
A-5)install Proxmox 4.0
<https://pve.proxmox.com/wiki/Install_Proxmox_VE_on_Debian_Jessie>on
node #1
A-6) install Gluster on node #1 and add it back to the Gluster volume
(/gluster volume add-brick .../) => shared storage will be complete
again (spanning 3.4 and 4.0 nodes)
A-7) configure the Gluster volume as shared storage in Proxmox 4 (node #1)
A-8) configure the external Backup storage on node #1 (Proxmox 4)
Was the data on the gluster brick deleted as part of step 4?
Yes, all data on physical HDD was deleted (reformatted / repartitioned).
Post by Joe Julian
When you remove the brick, gluster will no longer track pending
changes for that brick. If you add it back in with stale data but
matching gfids, you would have two clean bricks with mismatching data.
Did you have to use "add-brick...force"?
No, "force" was not necessary and the added directory
"/data/gluster/systems" did not exist.

This were the commands executed on node #2 during step 6:

gluster volume add-brick "systems" replica 3
metal1:/data/gluster/systems
gluster volume heal "systems" full # to trigger sync


Then I waited for replication to finish before doing anything else
(about 1 hour or maybe more), checking _gluster volume heal "systems" info_


Udo
Lindsay Mathieson
2015-12-09 21:33:26 UTC
Permalink
Post by Udo Giacomozzi
gluster volume add-brick "systems" replica 3
metal1:/data/gluster/systems
gluster volume heal "systems" full # to trigger sync
Then I waited for replication to finish before doing anything else
(about 1 hour or maybe more), checking _gluster volume heal "systems" info_
Did you execute the heal command from host #2? Might be related to a
possible issue I encountered during testing adding bricks recently,
still in the process of recreating and testing the issue.
--
Lindsay Mathieson
Udo Giacomozzi
2015-12-10 15:12:56 UTC
Permalink
Post by Lindsay Mathieson
Post by Udo Giacomozzi
gluster volume add-brick "systems" replica 3
metal1:/data/gluster/systems
gluster volume heal "systems" full # to trigger sync
Then I waited for replication to finish before doing anything else
(about 1 hour or maybe more), checking _gluster volume heal "systems" info_
Did you execute the heal command from host #2? Might be related to a
possible issue I encountered during testing adding bricks recently,
still in the process of recreating and testing the issue.
I'm afraid I can't tell anymore. Could be, I'm not sure, sorry...


Udo
Udo Giacomozzi
2015-12-14 11:14:54 UTC
Permalink
Hi,

it happened again:

today I've upgraded some packages on node #3. Since the Kernel had a
minor update, I was asked to reboot the server, and did so.

At that time only one (non-critical) VM was running on that node. I've
checked twice and Gluster was *not* healing when I've rebooted.

After rebooting, and while *automatic* healing was in progress, one VM
started to get HDD corruption again, up to the point that it wasn't able
to boot anymore(!).

That poor VM was one of the only two VMs that were still using NFS for
accessing the Gluster storage - if that matters.
The second VM survived the healing, even if it has rather large disks
(~380 GB) and is rather busy.

All other ~13 VMs had been moved to native glusterfs mount days before
and had no problem with the reboot. The Gluster access type may be
related or not - I don't know...

All Gluster packages are at version "3.5.2-2+deb8u1" on all three
servers - so Gluster has *not* been upgraded this time.
Kernel on node #3: Linux metal3 4.2.6-1-pve #1 SMP Wed Dec 9 10:49:55
CET 2015 x86_64 GNU/Linux
Kenrle node #1&#2: Linux metal1 4.2.3-2-pve #1 SMP Sun Nov 15 16:08:19
CET 2015 x86_64 GNU/Linux


Any idea??

Udo
Post by Udo Giacomozzi
Post by Lindsay Mathieson
Post by Udo Giacomozzi
gluster volume add-brick "systems" replica 3
metal1:/data/gluster/systems
gluster volume heal "systems" full # to trigger sync
Then I waited for replication to finish before doing anything else
(about 1 hour or maybe more), checking _gluster volume heal
"systems" info_
Did you execute the heal command from host #2? Might be related to a
possible issue I encountered during testing adding bricks recently,
still in the process of recreating and testing the issue.
I'm afraid I can't tell anymore. Could be, I'm not sure, sorry...
Udo
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
Pranith Kumar Karampuri
2015-12-16 10:25:37 UTC
Permalink
Post by Udo Giacomozzi
Hi,
today I've upgraded some packages on node #3. Since the Kernel had a
minor update, I was asked to reboot the server, and did so.
At that time only one (non-critical) VM was running on that node. I've
checked twice and Gluster was *not* healing when I've rebooted.
After rebooting, and while *automatic* healing was in progress, one VM
started to get HDD corruption again, up to the point that it wasn't
able to boot anymore(!).
That poor VM was one of the only two VMs that were still using NFS for
accessing the Gluster storage - if that matters.
The second VM survived the healing, even if it has rather large disks
(~380 GB) and is rather busy.
All other ~13 VMs had been moved to native glusterfs mount days before
and had no problem with the reboot. The Gluster access type may be
related or not - I don't know...
All Gluster packages are at version "3.5.2-2+deb8u1" on all three
servers - so Gluster has *not* been upgraded this time.
Kernel on node #3: Linux metal3 4.2.6-1-pve #1 SMP Wed Dec 9 10:49:55
CET 2015 x86_64 GNU/Linux
Kenrle node #1&#2: Linux metal1 4.2.3-2-pve #1 SMP Sun Nov 15 16:08:19
CET 2015 x86_64 GNU/Linux
Could you give us the logs of all the nodes for your setup and the
name/gfid of the file that was corrupted?

Pranith
Post by Udo Giacomozzi
Any idea??
Udo
Post by Udo Giacomozzi
Post by Lindsay Mathieson
Post by Udo Giacomozzi
gluster volume add-brick "systems" replica 3
metal1:/data/gluster/systems
gluster volume heal "systems" full # to trigger sync
Then I waited for replication to finish before doing anything else
(about 1 hour or maybe more), checking _gluster volume heal
"systems" info_
Did you execute the heal command from host #2? Might be related to a
possible issue I encountered during testing adding bricks recently,
still in the process of recreating and testing the issue.
I'm afraid I can't tell anymore. Could be, I'm not sure, sorry...
Udo
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://www.gluster.org/mailman/listinfo/gluster-users
Continue reading on narkive:
Loading...