Kevin Lemonnier
2016-10-10 13:30:11 UTC
Hi,
We have a few Proxmox clusters using GlusterFS as storage.
The nodes are both running the gluster brick and proxmox,
and one of the problems we have often is that when a server
crashes for some reason, the InnoDB of the VM that were
running on the VMs it hosted are dead. Most of the time
setting innodb_force_recovery is enough to fix it, but sometime
it isn't and we have to restart from scratch and import backups.
So my question is this : Does anyone had that problem, and if
yes how did you solve it ? Am I missing some critical piece of
configuration in MySQL or Gluster to prevent innodb from exploding
at the first crash ? I've forced reboot of plenty of physical machines
without any problems, so there is clearly something with gluster there.
I might be wrong, but I imagine that the problem is that MySQL writes,
but when the server crashes the two other nodes are still on an older
version of the files. I tried adding cache=directsync to the VMs,
hoping that would mean a write is successfull only when the 3 replicas
have completed it, but that doesn't really seem better.
We are considering having separate physical MySQL servers shared between
the different VMs, but that doesn't seem like an ideal sale speech :
"You can do whatever you want on the VM, but try to avoid MySQL, it crashes".
We have a few Proxmox clusters using GlusterFS as storage.
The nodes are both running the gluster brick and proxmox,
and one of the problems we have often is that when a server
crashes for some reason, the InnoDB of the VM that were
running on the VMs it hosted are dead. Most of the time
setting innodb_force_recovery is enough to fix it, but sometime
it isn't and we have to restart from scratch and import backups.
So my question is this : Does anyone had that problem, and if
yes how did you solve it ? Am I missing some critical piece of
configuration in MySQL or Gluster to prevent innodb from exploding
at the first crash ? I've forced reboot of plenty of physical machines
without any problems, so there is clearly something with gluster there.
I might be wrong, but I imagine that the problem is that MySQL writes,
but when the server crashes the two other nodes are still on an older
version of the files. I tried adding cache=directsync to the VMs,
hoping that would mean a write is successfull only when the 3 replicas
have completed it, but that doesn't really seem better.
We are considering having separate physical MySQL servers shared between
the different VMs, but that doesn't seem like an ideal sale speech :
"You can do whatever you want on the VM, but try to avoid MySQL, it crashes".
--
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111
Kevin Lemonnier
PGP Fingerprint : 89A5 2283 04A0 E6E9 0111