Discussion:
[Gluster-users] glustefs as vmware datastore in production
j***@7lan.net
2018-05-25 17:27:31 UTC
Permalink
Hi,

Does anyone have glusterfs as vmware datastore working in production in a real world case? How to serve the glusterfs cluster? As iscsi, NFS?

Thanks in advance
김경표
2018-05-26 17:56:48 UTC
Permalink
Post by j***@7lan.net
Hi,
Does anyone have glusterfs as vmware datastore working in production in a
real world case? How to serve the glusterfs cluster? As iscsi, NFS?
Hi,

I am using glusterfs 3.10.x for VMware ESXi 5.5 NFS DataStore.

Our Environment is
- 4 node supermicro server (each 50TB, NL SAS 4TB used, LSI 9260-8i)
- Totally 100TB service volume
- 10G Storage Network and Service network (for NFS)
- VMware / Linux / IBM AIX Clients

Currently, VM images on GlusterFS is used for Data(Backup) not OS Image.

There is no glusterfs issue in 10 months.

Best Regards
João Reis
2018-05-28 16:45:33 UTC
Permalink
Nice to read this. Any particular reason to *not* run the OS image in
glusterfs cluster?

Thanks
Post by j***@7lan.net
Hi,
Does anyone have glusterfs as vmware datastore working in
production in a real world case? How to serve the glusterfs
cluster? As iscsi, NFS?
Hi,
I am using glusterfs 3.10.x for VMware ESXi 5.5 NFS DataStore.
Our Environment is
- 4 node supermicro server (each 50TB, NL SAS 4TB used, LSI 9260-8i)
- Totally 100TB service volume
- 10G Storage Network and Service network (for NFS)
- VMware / Linux / IBM AIX Clients
Currently, VM images on GlusterFS is used for Data(Backup) not OS Image.
There is no glusterfs issue in 10 months.
Best Regards
김경표
2018-05-29 00:03:04 UTC
Permalink
Sometimes os disk hang occured and re-mounted with ro in vm guest(centos6)
when storage was busy.

After install vmware plugin, increased block response timeout to 30 sec.
But os workload reponse time was not good.

I guess my system composed with 5400 rpm disks with raid6.
Overall storage performance is not good for multiple os images.

best regards.


After tha
Post by João Reis
Nice to read this. Any particular reason to *not* run the OS image in
glusterfs cluster?
Thanks
Hi,
Post by j***@7lan.net
Does anyone have glusterfs as vmware datastore working in production in a
real world case? How to serve the glusterfs cluster? As iscsi, NFS?
Hi,
I am using glusterfs 3.10.x for VMware ESXi 5.5 NFS DataStore.
Our Environment is
- 4 node supermicro server (each 50TB, NL SAS 4TB used, LSI 9260-8i)
- Totally 100TB service volume
- 10G Storage Network and Service network (for NFS)
- VMware / Linux / IBM AIX Clients
Currently, VM images on GlusterFS is used for Data(Backup) not OS Image.
There is no glusterfs issue in 10 months.
Best Regards
Dave Sherohman
2018-05-29 13:20:03 UTC
Permalink
Post by 김경표
Sometimes os disk hang occured and re-mounted with ro in vm guest(centos6)
when storage was busy.
I had similar problems in the early days of running my gluster volume,
then I switched the gluster mounts from fuse to libgfapi and haven't had
a problem since, even when running the volume harder than I had been
previously.

But, then, I'm running kvm/qemu virtualization rather than vmware and I
don't know whether vmware supports libgfapi or not. (I noticed that it
wasn't on the list of options you mentioned for how to access the
volume.)
--
Dave Sherohman
Benjamin Kingston
2018-06-06 01:38:16 UTC
Permalink
You're better off exporting LUNs via iSCSI. I spent a long time trying to
get NFS to work via NFS-Ganesha as a datastore and the performance is not
there, especially since HA NFS isn't an official feature of NFS-Ganesha.

Also keep in mind your write speed is cut in half/thirds/etc... with
gluster as a VM datastore if you use replication since all writes are
multiplied.
Post by 김경표
Post by 김경표
Sometimes os disk hang occured and re-mounted with ro in vm
guest(centos6)
Post by 김경표
when storage was busy.
I had similar problems in the early days of running my gluster volume,
then I switched the gluster mounts from fuse to libgfapi and haven't had
a problem since, even when running the volume harder than I had been
previously.
But, then, I'm running kvm/qemu virtualization rather than vmware and I
don't know whether vmware supports libgfapi or not. (I noticed that it
wasn't on the list of options you mentioned for how to access the
volume.)
--
Dave Sherohman
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Jonathan Archer
2018-06-06 09:53:23 UTC
Permalink
Dave,
What are you using for your presentation if not NFS? Are you using VMWare as the hypervisor?
Are you using a cluster vip across your nodes or using a single entry point via one node?
CheersJon
Post by Benjamin Kingston
You're better off exporting LUNs via iSCSI.
Speak for yourself.  I'm running the VMs on multiple physical systems
and migrating between them.  We were using LVM on top of iSCSI LUNs
before setting up gluster and it was a constant PITA having to propagate
filesystem metadata between the host systems, with the occasional
filesystem corruption when one host expected an lv to be a certain size
(or whatever) and a different host expected something else.

Turning the disk images into files on a remote filesystem removed all of
those issues.

clvm probably would have also resolved those problems, but gluster
looked easier to set up, and it worked.  I had one minor problem with
FUSE (which was resolved by switching to libgfapi) and one less-minor
problem because I misunderstood how gluster handles quorum (which was
resolved by switching from replica 2 to replica 2+A).  Other than that,
gluster has worked perfectly for me in my use case since day one.
Post by Benjamin Kingston
I spent a long time trying to get NFS to work via NFS-Ganesha as a
datastore and the performance is not there, especially since HA NFS
isn't an official feature of NFS-Ganesha.
Perhaps your issue was in the NFS layer, which I'm not using.  Even when
I was using FUSE mounts instead of libgfapi, I was mounting them as GFS,
not NFS.
Post by Benjamin Kingston
Also keep in mind your write speed is cut in half/thirds/etc... with
gluster as a VM datastore if you use replication since all writes are
multiplied.
Yep, that's the price you pay for HA.

Also, although the writes are multiplied, they're also (at least
partially) concurrent, so performance isn't as bad as "divide by the
number of replicas".
--
Dave Sherohman
Jonathan Archer
2018-05-29 08:02:32 UTC
Permalink
Hi,I've gone through a bit of testing around using Gluster as a VMWare datastore, here are my findings:
running VMWare vSphere 6.5 with ESXi nodesGluster running on Supermicro kit, 6 SAS disks with 2 SSD's for caching all carved up using LVM on to of CentOS 7.
I set up a 4 node cluster, ultimately to scale to 12 should this become usable, initially just using 3 nodes to satisfy quorum requirements without playing too much with default values.The idea was to preset via NFS, have a cluster VIP and heartbeat. Obviously NFS is no longer a viable option direct from Gluster so the route I took was down the NFS-Ganesha route. Had no end of troubles using the Ganesha tools with gluster 3.10 so ended up doing the whole thing manually which seems to stand up ok.For this I used Gluster, NFS-Ganesha with VFS connections, Pacemaker and Corrosync (still not 100% happy with the config and it needs tweaking but it does what it needs so far).
Next job was connecting from VMWare, I had to ensure that relevant read/write permissions were set both on the filesystem and in the Ganesha config but connection succeeded. From here was where I'm suffering a few issues.
If you migrate a VM into the NFS volume and then away again it doesn't clear the parent directory, it removes everything else but that, SSHing onto the ESXi server and manually trying to remove it results in 2 errors:
rm directory - cannot remove as it is a directoryrm -rf directory - cannot remove as it isn't a directory
second issue is to do with failover, The VIP fails over extremely fast, but VMWare always loses connection to the volume and a weird oddity of VMware seems to be that it will not remount an NFS volume so stays unavailable.
I have not got over these hurdles as yet, but will persevere as performance wise and cost wise its brilliant.
Hope this helps, or if anyone has any clues to the above issues I'd be most grateful.
Regards
Jon
On Friday, 25 May 2018, 18:33:09 BST, <***@7lan.net> wrote:


Hi,

Does anyone have glusterfs as vmware datastore working in production in a real world case? How to serve the glusterfs cluster? As iscsi, NFS?

Thanks in advance _______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org
http://lists.gluster.org/mailman/listinfo/gluster-users
Loading...