Discussion:
VM Images Over GlusterFS
(too old to reply)
Alexander Atticus
2009-01-17 23:15:20 UTC
Permalink
Hello,

I am wondering if there is anyone on list that is successfully serving
virtual machine images (Xen, KVM, VMWare, etc.) over GlusterFS. I am
concerned about difficulties the interaction with GlusterFS and large
files that are held open for large periods of time (except when the VM
is rebooted).

My main concern is with replication and failover. Is it currently
possible to setup relatively transparent failover of a brick or will
the VMs all need to be rebooted to "reopen" their disk images when a
brick goes down. The setup we were thinking of going with here is
client striping + server side AFR. I don't know whether HA will need
to be involved in order to make this happen. Also, for the current
self-healing mechanism to work my understanding is that the image
(file) will need to be closed and reopened, i.e. the VM will need to
be rebooted but this may be fixed with the new healing tools I think
are coming in 2.0?

Thanks,

Alexander
Matthias Teege
2009-01-18 08:27:51 UTC
Permalink
Post by Alexander Atticus
I am wondering if there is anyone on list that is successfully serving
virtual machine images (Xen, KVM, VMWare, etc.) over GlusterFS. I am
I using it with Xen and vserver. With Xen and large disk images it
works. Live migration and failover works. Healing is a major problem. It
blocks the image for sometime so you have downtime anyway. I have to
write a heartbeat ressource to manage the healing before restart the
VM. Linux Vserver works to but there are performance issues im working
on. It is no problem to "dd" a file to gluster with reasonable speed but
in "realtime" vserver is much faster on drbd then on gluster. Maybe I hit
a "lot of small files" Problem here or there is a problem with vservers
COW mechanism. I use 1.3.12 only.

Matthias
Enrico Valsecchi
2009-01-19 14:03:15 UTC
Permalink
Post by Matthias Teege
Post by Alexander Atticus
I am wondering if there is anyone on list that is successfully serving
virtual machine images (Xen, KVM, VMWare, etc.) over GlusterFS. I am
I using it with Xen and vserver. With Xen and large disk images it
works. Live migration and failover works. Healing is a major problem. It
blocks the image for sometime so you have downtime anyway. I have to
write a heartbeat ressource to manage the healing before restart the
VM. Linux Vserver works to but there are performance issues im working
on. It is no problem to "dd" a file to gluster with reasonable speed but
in "realtime" vserver is much faster on drbd then on gluster. Maybe I hit
a "lot of small files" Problem here or there is a problem with vservers
COW mechanism. I use 1.3.12 only.
Hello!

I have tested with success Xen with GlusterFs.
I does not have a cleaned idea if my scenario is same at your.
I have a xen server attached with Infiniband and RDMA access on 2
GlusterFS servers, used to store xen images files, and configured in AFR
by Client.
GlusterFS version are 1.3.12, xen tested are 3.2.0, 3.2.1, 3.3.0.
Images file are mounted in xen with tap:aio disk option, because with
this system have more performance that "file" option and,
in order to more post found in the net, it's more safe.
Only way to use tap:aio on xen are to mount GlusterFS with "-d" option.

Now, where problems you have found?


Enrico
Alexander Atticus
2009-01-19 16:18:23 UTC
Permalink
I'm mostly concerned about the auto-healing feature. I'm assuming you
have some kind of redundancy setup in your GlusterFS configuration.
What happens when you have a VM running on an image served from
Gluster, and you simulate a crash on one of your bricks? Does the VM
stay up?

My impression is that it may but if you reboot the VM the VM after you
bring the crashed brick back up it will block until the VM disk image
has a chance to heal (sync).

What I wanted to setup here is Striping over all bricks with each
brick setup to AFR to another for redundancy. I don't know whether
this will deliver the availability I'm looking for or whether I need
HA or something else as well. Essentially I don't want VMs to notice
at all when a brick goes down and I'm not sure this is possible?

Alexander
Post by Enrico Valsecchi
Post by Matthias Teege
Post by Alexander Atticus
I am wondering if there is anyone on list that is successfully serving
virtual machine images (Xen, KVM, VMWare, etc.) over GlusterFS. I am
I using it with Xen and vserver. With Xen and large disk images it
works. Live migration and failover works. Healing is a major problem. It
blocks the image for sometime so you have downtime anyway. I have to
write a heartbeat ressource to manage the healing before restart the
VM. Linux Vserver works to but there are performance issues im working
on. It is no problem to "dd" a file to gluster with reasonable speed but
in "realtime" vserver is much faster on drbd then on gluster. Maybe I hit
a "lot of small files" Problem here or there is a problem with vservers
COW mechanism. I use 1.3.12 only.
Hello!
I have tested with success Xen with GlusterFs.
I does not have a cleaned idea if my scenario is same at your.
I have a xen server attached with Infiniband and RDMA access on 2
GlusterFS servers, used to store xen images files, and configured in AFR
by Client.
GlusterFS version are 1.3.12, xen tested are 3.2.0, 3.2.1, 3.3.0.
Images file are mounted in xen with tap:aio disk option, because with
this system have more performance that "file" option and,
in order to more post found in the net, it's more safe.
Only way to use tap:aio on xen are to mount GlusterFS with "-d" option.
Now, where problems you have found?
Enrico
_______________________________________________
Gluster-users mailing list
http://zresearch.com/cgi-bin/mailman/listinfo/gluster-users
Kirby Zhou
2009-01-19 16:18:52 UTC
Permalink
I have 2 questions please.
Which io mode of xen did you take ? tap:aio, file, or phy?
Does XEN 3 within RHEL-5.2 works with glusterfs?

-----Original Message-----
From: gluster-users-***@gluster.org
[mailto:gluster-users-***@gluster.org] On Behalf Of Enrico Valsecchi
Sent: Monday, January 19, 2009 10:03 PM
To: gluster-***@gluster.org
Subject: Re: [Gluster-users] VM Images Over GlusterFS
Post by Matthias Teege
Post by Alexander Atticus
I am wondering if there is anyone on list that is successfully serving
virtual machine images (Xen, KVM, VMWare, etc.) over GlusterFS. I am
I using it with Xen and vserver. With Xen and large disk images it
works. Live migration and failover works. Healing is a major problem. It
blocks the image for sometime so you have downtime anyway. I have to
write a heartbeat ressource to manage the healing before restart the
VM. Linux Vserver works to but there are performance issues im working
on. It is no problem to "dd" a file to gluster with reasonable speed but
in "realtime" vserver is much faster on drbd then on gluster. Maybe I hit
a "lot of small files" Problem here or there is a problem with vservers
COW mechanism. I use 1.3.12 only.
Hello!

I have tested with success Xen with GlusterFs.
I does not have a cleaned idea if my scenario is same at your.
I have a xen server attached with Infiniband and RDMA access on 2
GlusterFS servers, used to store xen images files, and configured in AFR
by Client.
GlusterFS version are 1.3.12, xen tested are 3.2.0, 3.2.1, 3.3.0.
Images file are mounted in xen with tap:aio disk option, because with
this system have more performance that "file" option and,
in order to more post found in the net, it's more safe.
Only way to use tap:aio on xen are to mount GlusterFS with "-d" option.

Now, where problems you have found?


Enrico
Daniel Maher
2009-01-21 09:27:28 UTC
Permalink
Post by Alexander Atticus
I am wondering if there is anyone on list that is successfully serving
virtual machine images (Xen, KVM, VMWare, etc.) over GlusterFS. I am
concerned about difficulties the interaction with GlusterFS and large
files that are held open for large periods of time (except when the VM
is rebooted).
Afaik, VMWare won't recognize a Gluster mountpoint as a valid datastore.
This isn't an issue with Gluster specifically, but rather with VMWare
generally ; Samba and NFS are the only network filesystems which work in
this capacity, with Samba being a relatively new addition at that.
--
Daniel Maher <dma+gluster AT witbe DOT net>
Continue reading on narkive:
Loading...