Discussion:
[Gluster-users] does your samba work with 4.1.x (centos 7.5)
lejeczek
2018-11-09 13:12:01 UTC
Permalink
hi guys

I presume because 4.1.x has been in EPEL repo it is confirmed and
validated to work 100% with default samba installation.

But, I'd prefer to here you guys say you ACTUALLY have your samba work
100% with 4.1.x. Anybody?

many thanks, L.
Kaleb S. KEITHLEY
2018-11-09 15:08:58 UTC
Permalink
Post by lejeczek
hi guys
I presume because 4.1.x has been in EPEL repo it is confirmed and
validated to work 100% with default samba installation.
GlusterFS — any version — is _not_ in EPEL.

However it is in the CentOS Storage SIG.
Post by lejeczek
But, I'd prefer to here you guys say you ACTUALLY have your samba work
100% with 4.1.x. Anybody?
Nobody has built Samba in the CentOS Storage SIG with GlusterFS support.
Not for a long time anyway. The last time it was built was over four
years ago — for el6.

It would be great if someone in one of the Gluster, Samba, or CentOS
communities would start building it on a regular basis.

Any volunteers?
--
Kaleb
Diego Remolina
2018-11-09 15:19:53 UTC
Permalink
Hmmm yes and no...

Have had problems with samba 4.7.x from Centos repos and Gluster
3.10.x when using plugin vfs gluster to access shares so I had to use
fuse mounts and point samba to fuse mounts.

I tried an upgrade to glusterfs 4.1.5 a couple days ago and samba
worked with it but still using fuse mounts, not vfs gluster.

A while back a combination of samba vfs plugin ans gluster broke the
use of Autodesk Revit files stored in gluster. However since I am
still running 3.10.12, I am not able to open any bug reports or
anything given it is no longer supported. I was hopping to do that
with gluster 4.1.5, but had to roll back to 3.10.12.

These are my current rpms (one version behind in samba) from Centos:

rpm -qa | grep samba
samba-client-libs-4.7.1-6.el7.x86_64
samba-common-4.7.1-6.el7.noarch
samba-common-libs-4.7.1-6.el7.x86_64
samba-4.7.1-6.el7.x86_64
samba-vfs-glusterfs-4.7.1-6.el7.x86_64
samba-libs-4.7.1-6.el7.x86_64
samba-common-tools-4.7.1-6.el7.x86_64

rpm -qa | grep gluster
glusterfs-api-3.10.12-1.el7.x86_64
glusterfs-3.10.12-1.el7.x86_64
glusterfs-client-xlators-3.10.12-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
samba-vfs-glusterfs-4.7.1-6.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-rdma-3.10.12-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64


Diego
Post by Kaleb S. KEITHLEY
Post by lejeczek
hi guys
I presume because 4.1.x has been in EPEL repo it is confirmed and
validated to work 100% with default samba installation.
GlusterFS — any version — is _not_ in EPEL.
However it is in the CentOS Storage SIG.
Post by lejeczek
But, I'd prefer to here you guys say you ACTUALLY have your samba work
100% with 4.1.x. Anybody?
Nobody has built Samba in the CentOS Storage SIG with GlusterFS support.
Not for a long time anyway. The last time it was built was over four
years ago — for el6.
It would be great if someone in one of the Gluster, Samba, or CentOS
communities would start building it on a regular basis.
Any volunteers?
--
Kaleb
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Kaleb S. KEITHLEY
2018-11-09 19:57:49 UTC
Permalink
Post by Diego Remolina
Hmmm yes and no...
Yes and no what?

That samba comes from the CentOS-Base/updates repo.

If you don't use the Storage SIG you will get glusterfs 3.8.4,
client-side only; also from the CentOS-Base/updates repo.

That samba's rpm (.spec file) says it Requires: GlusterFS > 3.4, but in
all likelihood it was built with GlusterFS 3.8.4 from the Base/updates repo.

But the original question was about EPEL, and there is no version of
Samba or GlusterFS in EPEL. Enable the EPEL repo and disable the
Base/updates repo and try `yumdownloader samba\*` or `yumdownloader
glusterfs\*`. You won't get anything.

And if we extrapolate to the CentOS Storage SIG, there's still no
version of Samba there either. It would be nice if there was, but there
isn't.
Post by Diego Remolina
Have had problems with samba 4.7.x from Centos repos and Gluster
3.10.x when using plugin vfs gluster to access shares so I had to use
fuse mounts and point samba to fuse mounts.
I tried an upgrade to glusterfs 4.1.5 a couple days ago and samba
worked with it but still using fuse mounts, not vfs gluster.
A while back a combination of samba vfs plugin ans gluster broke the
use of Autodesk Revit files stored in gluster. However since I am
still running 3.10.12, I am not able to open any bug reports or
anything given it is no longer supported. I was hopping to do that
with gluster 4.1.5, but had to roll back to 3.10.12.
rpm -qa | grep samba
samba-client-libs-4.7.1-6.el7.x86_64
samba-common-4.7.1-6.el7.noarch
samba-common-libs-4.7.1-6.el7.x86_64
samba-4.7.1-6.el7.x86_64
samba-vfs-glusterfs-4.7.1-6.el7.x86_64
samba-libs-4.7.1-6.el7.x86_64
samba-common-tools-4.7.1-6.el7.x86_64
rpm -qa | grep gluster
glusterfs-api-3.10.12-1.el7.x86_64
glusterfs-3.10.12-1.el7.x86_64
glusterfs-client-xlators-3.10.12-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
samba-vfs-glusterfs-4.7.1-6.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-rdma-3.10.12-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
Diego
Post by Kaleb S. KEITHLEY
Post by lejeczek
hi guys
I presume because 4.1.x has been in EPEL repo it is confirmed and
validated to work 100% with default samba installation.
GlusterFS — any version — is _not_ in EPEL.
However it is in the CentOS Storage SIG.
Post by lejeczek
But, I'd prefer to here you guys say you ACTUALLY have your samba work
100% with 4.1.x. Anybody?
Nobody has built Samba in the CentOS Storage SIG with GlusterFS support.
Not for a long time anyway. The last time it was built was over four
years ago — for el6.
It would be great if someone in one of the Gluster, Samba, or CentOS
communities would start building it on a regular basis.
Any volunteers?
--
Kaleb
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Kaleb
Diego Remolina
2018-11-09 20:03:35 UTC
Permalink
Yes it works if you use fuse mount

No it does not work well if you use: vfs objects = glusterfs

The samba comes directly from CentOS repository.

Gluster comes from SIG 3.10.12

# rpm -qa | grep centos-release-gluster
centos-release-gluster310-1.0-1.el7.centos.noarch

# yum info samba
Loaded plugins: fastestmirror, verify
Loading mirror speeds from cached hostfile
* base: reflector.westga.edu
* extras: reflector.westga.edu
* updates: reflector.westga.edu
Installed Packages
Name : samba
Arch : x86_64
Version : 4.7.1
Release : 6.el7
Size : 1.9 M
Repo : installed
From repo : base
Summary : Server and Client software to interoperate with Windows machines
URL : http://www.samba.org/
License : GPLv3+ and LGPLv3+
Description : Samba is the standard Windows interoperability suite of
programs for Linux and
: Unix.

# yum info glusterfs-server
Loaded plugins: fastestmirror, verify
Loading mirror speeds from cached hostfile
* base: reflector.westga.edu
* extras: reflector.westga.edu
* updates: reflector.westga.edu
Installed Packages
Name : glusterfs-server
Arch : x86_64
Version : 3.10.12
Release : 1.el7
Size : 4.3 M
Repo : installed
From repo : centos-gluster310
Summary : Distributed file-system server
URL : http://gluster.readthedocs.io/en/latest/
License : GPLv2 or LGPLv3+
Description : GlusterFS is a distributed file-system capable of
scaling to several
: petabytes. It aggregates various storage bricks over
Infiniband RDMA
: or TCP/IP interconnect into one large parallel network file
: system. GlusterFS is one of the most sophisticated file systems in
: terms of features and extensibility. It borrows a powerful concept
: called Translators from GNU Hurd kernel. Much of the code
in GlusterFS
: is in user space and easily manageable.
:
: This package provides the glusterfs server daemon.


Diego
Post by Kaleb S. KEITHLEY
Post by Diego Remolina
Hmmm yes and no...
Yes and no what?
That samba comes from the CentOS-Base/updates repo.
If you don't use the Storage SIG you will get glusterfs 3.8.4,
client-side only; also from the CentOS-Base/updates repo.
That samba's rpm (.spec file) says it Requires: GlusterFS > 3.4, but in
all likelihood it was built with GlusterFS 3.8.4 from the Base/updates repo.
But the original question was about EPEL, and there is no version of
Samba or GlusterFS in EPEL. Enable the EPEL repo and disable the
Base/updates repo and try `yumdownloader samba\*` or `yumdownloader
glusterfs\*`. You won't get anything.
And if we extrapolate to the CentOS Storage SIG, there's still no
version of Samba there either. It would be nice if there was, but there
isn't.
Post by Diego Remolina
Have had problems with samba 4.7.x from Centos repos and Gluster
3.10.x when using plugin vfs gluster to access shares so I had to use
fuse mounts and point samba to fuse mounts.
I tried an upgrade to glusterfs 4.1.5 a couple days ago and samba
worked with it but still using fuse mounts, not vfs gluster.
A while back a combination of samba vfs plugin ans gluster broke the
use of Autodesk Revit files stored in gluster. However since I am
still running 3.10.12, I am not able to open any bug reports or
anything given it is no longer supported. I was hopping to do that
with gluster 4.1.5, but had to roll back to 3.10.12.
rpm -qa | grep samba
samba-client-libs-4.7.1-6.el7.x86_64
samba-common-4.7.1-6.el7.noarch
samba-common-libs-4.7.1-6.el7.x86_64
samba-4.7.1-6.el7.x86_64
samba-vfs-glusterfs-4.7.1-6.el7.x86_64
samba-libs-4.7.1-6.el7.x86_64
samba-common-tools-4.7.1-6.el7.x86_64
rpm -qa | grep gluster
glusterfs-api-3.10.12-1.el7.x86_64
glusterfs-3.10.12-1.el7.x86_64
glusterfs-client-xlators-3.10.12-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
samba-vfs-glusterfs-4.7.1-6.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-rdma-3.10.12-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
Diego
Post by Kaleb S. KEITHLEY
Post by lejeczek
hi guys
I presume because 4.1.x has been in EPEL repo it is confirmed and
validated to work 100% with default samba installation.
GlusterFS — any version — is _not_ in EPEL.
However it is in the CentOS Storage SIG.
Post by lejeczek
But, I'd prefer to here you guys say you ACTUALLY have your samba work
100% with 4.1.x. Anybody?
Nobody has built Samba in the CentOS Storage SIG with GlusterFS support.
Not for a long time anyway. The last time it was built was over four
years ago — for el6.
It would be great if someone in one of the Gluster, Samba, or CentOS
communities would start building it on a regular basis.
Any volunteers?
--
Kaleb
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Kaleb
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Anoop C S
2018-11-12 04:41:18 UTC
Permalink
Post by Diego Remolina
Yes it works if you use fuse mount
No it does not work well if you use: vfs objects = glusterfs
Would you mind explaining the issue in detail? Apologies if you have already raised it here before
and could not get to resolution.

Please attach the output of `testparm -s` and any relevant error messages from logs under
/var/log/samba/ when glusterfs vfs module is being used.
Post by Diego Remolina
The samba comes directly from CentOS repository.
Gluster comes from SIG 3.10.12
# rpm -qa | grep centos-release-gluster
centos-release-gluster310-1.0-1.el7.centos.noarch
# yum info samba
Loaded plugins: fastestmirror, verify
Loading mirror speeds from cached hostfile
* base: reflector.westga.edu
* extras: reflector.westga.edu
* updates: reflector.westga.edu
Installed Packages
Name : samba
Arch : x86_64
Version : 4.7.1
Release : 6.el7
Size : 1.9 M
Repo : installed
From repo : base
Summary : Server and Client software to interoperate with Windows machines
URL : http://www.samba.org/
License : GPLv3+ and LGPLv3+
Description : Samba is the standard Windows interoperability suite of
programs for Linux and
: Unix.
# yum info glusterfs-server
Loaded plugins: fastestmirror, verify
Loading mirror speeds from cached hostfile
* base: reflector.westga.edu
* extras: reflector.westga.edu
* updates: reflector.westga.edu
Installed Packages
Name : glusterfs-server
Arch : x86_64
Version : 3.10.12
Release : 1.el7
Size : 4.3 M
Repo : installed
From repo : centos-gluster310
Summary : Distributed file-system server
URL : http://gluster.readthedocs.io/en/latest/
License : GPLv2 or LGPLv3+
Description : GlusterFS is a distributed file-system capable of
scaling to several
: petabytes. It aggregates various storage bricks over
Infiniband RDMA
: or TCP/IP interconnect into one large parallel network file
: system. GlusterFS is one of the most sophisticated file systems in
: terms of features and extensibility. It borrows a powerful concept
: called Translators from GNU Hurd kernel. Much of the code
in GlusterFS
: is in user space and easily manageable.
: This package provides the glusterfs server daemon.
Diego
Post by Kaleb S. KEITHLEY
Post by Diego Remolina
Hmmm yes and no...
Yes and no what?
That samba comes from the CentOS-Base/updates repo.
If you don't use the Storage SIG you will get glusterfs 3.8.4,
client-side only; also from the CentOS-Base/updates repo.
That samba's rpm (.spec file) says it Requires: GlusterFS > 3.4, but in
all likelihood it was built with GlusterFS 3.8.4 from the Base/updates repo.
But the original question was about EPEL, and there is no version of
Samba or GlusterFS in EPEL. Enable the EPEL repo and disable the
Base/updates repo and try `yumdownloader samba\*` or `yumdownloader
glusterfs\*`. You won't get anything.
And if we extrapolate to the CentOS Storage SIG, there's still no
version of Samba there either. It would be nice if there was, but there
isn't.
Post by Diego Remolina
Have had problems with samba 4.7.x from Centos repos and Gluster
3.10.x when using plugin vfs gluster to access shares so I had to use
fuse mounts and point samba to fuse mounts.
I tried an upgrade to glusterfs 4.1.5 a couple days ago and samba
worked with it but still using fuse mounts, not vfs gluster.
A while back a combination of samba vfs plugin ans gluster broke the
use of Autodesk Revit files stored in gluster. However since I am
still running 3.10.12, I am not able to open any bug reports or
anything given it is no longer supported. I was hopping to do that
with gluster 4.1.5, but had to roll back to 3.10.12.
rpm -qa | grep samba
samba-client-libs-4.7.1-6.el7.x86_64
samba-common-4.7.1-6.el7.noarch
samba-common-libs-4.7.1-6.el7.x86_64
samba-4.7.1-6.el7.x86_64
samba-vfs-glusterfs-4.7.1-6.el7.x86_64
samba-libs-4.7.1-6.el7.x86_64
samba-common-tools-4.7.1-6.el7.x86_64
rpm -qa | grep gluster
glusterfs-api-3.10.12-1.el7.x86_64
glusterfs-3.10.12-1.el7.x86_64
glusterfs-client-xlators-3.10.12-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
samba-vfs-glusterfs-4.7.1-6.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-rdma-3.10.12-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
Diego
Post by Kaleb S. KEITHLEY
Post by lejeczek
hi guys
I presume because 4.1.x has been in EPEL repo it is confirmed and
validated to work 100% with default samba installation.
GlusterFS — any version — is _not_ in EPEL.
However it is in the CentOS Storage SIG.
Post by lejeczek
But, I'd prefer to here you guys say you ACTUALLY have your samba work
100% with 4.1.x. Anybody?
Nobody has built Samba in the CentOS Storage SIG with GlusterFS support.
Not for a long time anyway. The last time it was built was over four
years ago — for el6.
It would be great if someone in one of the Gluster, Samba, or CentOS
communities would start building it on a regular basis.
Any volunteers?
--
Kaleb
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Kaleb
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Diego Remolina
2018-11-13 01:04:26 UTC
Permalink
Hi Anoop,

This is an overview of how to use Central files in Revit:

https://revitpure.com/blog/how-to-use-central-and-local-files-in-revit

Once a central file is created, other folders are also created in the
location of the file, which contains many other files.

[***@ysmha02 vfsgluster]# ls -la
total 385588
drwxrws---. 4 dijuremo Staff 4096 Nov 12 19:03 .
drwxr-xr-x. 21 root root 4096 Nov 7 19:51 ..
drwxrws---. 2 dijuremo Staff 4096 Nov 12 19:05 2017-07-06 CAPE CORAL
CJDR_CENTRAL_R2017_backup
-rw-rw----. 1 dijuremo Staff 394825728 Jul 23 2017 2017-07-06 CAPE CORAL
CJDR_CENTRAL_R2017.rvt
drwxrws---. 2 dijuremo Staff 4096 Nov 12 19:03 Revit_temp

So I copied the file 2017 2017-07-06 CAPE CORAL CJDR_CENTRAL_R2017.rvt to
the network share:
\\ysmserver\vfsgluster

When I attempted to create a central file, it failed giving this error
message:

[image: revit-central-creation-error.JPG]

A simple ls -l of the _backup folder shows there is an existing file there
called incrementtable.2108.dat:

[***@ysmha02 vfsgluster]# ls -l 2017-07-06\ CAPE\ CORAL\
CJDR_CENTRAL_R2017_backup/incrementtable.2108.dat
-rw-rw----. 1 dijuremo Staff 2357 Nov 12 19:13 2017-07-06 CAPE CORAL
CJDR_CENTRAL_R2017_backup/incrementtable.2108.dat

However, at this point, things are not OK. The file is *not* a central
file. If I hit close and open the file again, then Revit will hang, usually
just go into the usual windows "Not Responding" state. This can last for
several minutes. I just closed the application via End Task after 5 minutes
of waiting. Rather than double clicking on the file from the share, I also
tried opening Revit first, then opening the file from Revit using the Open
dialog. This also hangs the program.

In one occasion, I tried to manually delete the folders (long_name_backup
and Revit_temp) from windows using File Explorer to try and recreate the
central again and then the delete process hung in one file,
preview.1957.dat for almost a minute, but it finally succeeded. This is not
normal behavior.

On the server, I can see this is a rather small file:

[***@ysmha02 2017-07-06 CAPE CORAL CJDR_CENTRAL_R2017_backup]# ls -la
total 11
drwxrws---. 2 dijuremo Staff 4096 Nov 12 19:39 .
drwxrws---. 4 dijuremo Staff 4096 Nov 12 19:03 ..
-rw-rw----. 1 dijuremo Staff 2753 Nov 12 19:36 preview.1957.dat

This is the test samba share exported using vfs object = glusterfs:

[vfsgluster]
path = /vfsgluster
browseable = yes
create mask = 660
directory mask = 770
write list = @Staff
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 7
glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
glusterfs:volume = export

Full smb.conf
http://termbin.com/y4j0

/var/log/samba/glusterfs-vfsgluster.log
http://termbin.com/5hdr

Please let me know if there is any other information I can provide.

Diego
Post by Anoop C S
Post by Diego Remolina
Yes it works if you use fuse mount
No it does not work well if you use: vfs objects = glusterfs
Would you mind explaining the issue in detail? Apologies if you have
already raised it here before
and could not get to resolution.
Please attach the output of `testparm -s` and any relevant error messages from logs under
/var/log/samba/ when glusterfs vfs module is being used.
Post by Diego Remolina
The samba comes directly from CentOS repository.
Gluster comes from SIG 3.10.12
# rpm -qa | grep centos-release-gluster
centos-release-gluster310-1.0-1.el7.centos.noarch
# yum info samba
Loaded plugins: fastestmirror, verify
Loading mirror speeds from cached hostfile
* base: reflector.westga.edu
* extras: reflector.westga.edu
* updates: reflector.westga.edu
Installed Packages
Name : samba
Arch : x86_64
Version : 4.7.1
Release : 6.el7
Size : 1.9 M
Repo : installed
From repo : base
Summary : Server and Client software to interoperate with Windows
machines
Post by Diego Remolina
URL : http://www.samba.org/
License : GPLv3+ and LGPLv3+
Description : Samba is the standard Windows interoperability suite of
programs for Linux and
: Unix.
# yum info glusterfs-server
Loaded plugins: fastestmirror, verify
Loading mirror speeds from cached hostfile
* base: reflector.westga.edu
* extras: reflector.westga.edu
* updates: reflector.westga.edu
Installed Packages
Name : glusterfs-server
Arch : x86_64
Version : 3.10.12
Release : 1.el7
Size : 4.3 M
Repo : installed
From repo : centos-gluster310
Summary : Distributed file-system server
URL : http://gluster.readthedocs.io/en/latest/
License : GPLv2 or LGPLv3+
Description : GlusterFS is a distributed file-system capable of
scaling to several
: petabytes. It aggregates various storage bricks over
Infiniband RDMA
: or TCP/IP interconnect into one large parallel network file
: system. GlusterFS is one of the most sophisticated file
systems in
Post by Diego Remolina
: terms of features and extensibility. It borrows a powerful
concept
Post by Diego Remolina
: called Translators from GNU Hurd kernel. Much of the code
in GlusterFS
: is in user space and easily manageable.
: This package provides the glusterfs server daemon.
Diego
Post by Kaleb S. KEITHLEY
Post by Diego Remolina
Hmmm yes and no...
Yes and no what?
That samba comes from the CentOS-Base/updates repo.
If you don't use the Storage SIG you will get glusterfs 3.8.4,
client-side only; also from the CentOS-Base/updates repo.
That samba's rpm (.spec file) says it Requires: GlusterFS > 3.4, but in
all likelihood it was built with GlusterFS 3.8.4 from the Base/updates
repo.
Post by Diego Remolina
Post by Kaleb S. KEITHLEY
But the original question was about EPEL, and there is no version of
Samba or GlusterFS in EPEL. Enable the EPEL repo and disable the
Base/updates repo and try `yumdownloader samba\*` or `yumdownloader
glusterfs\*`. You won't get anything.
And if we extrapolate to the CentOS Storage SIG, there's still no
version of Samba there either. It would be nice if there was, but there
isn't.
Post by Diego Remolina
Have had problems with samba 4.7.x from Centos repos and Gluster
3.10.x when using plugin vfs gluster to access shares so I had to use
fuse mounts and point samba to fuse mounts.
I tried an upgrade to glusterfs 4.1.5 a couple days ago and samba
worked with it but still using fuse mounts, not vfs gluster.
A while back a combination of samba vfs plugin ans gluster broke the
use of Autodesk Revit files stored in gluster. However since I am
still running 3.10.12, I am not able to open any bug reports or
anything given it is no longer supported. I was hopping to do that
with gluster 4.1.5, but had to roll back to 3.10.12.
rpm -qa | grep samba
samba-client-libs-4.7.1-6.el7.x86_64
samba-common-4.7.1-6.el7.noarch
samba-common-libs-4.7.1-6.el7.x86_64
samba-4.7.1-6.el7.x86_64
samba-vfs-glusterfs-4.7.1-6.el7.x86_64
samba-libs-4.7.1-6.el7.x86_64
samba-common-tools-4.7.1-6.el7.x86_64
rpm -qa | grep gluster
glusterfs-api-3.10.12-1.el7.x86_64
glusterfs-3.10.12-1.el7.x86_64
glusterfs-client-xlators-3.10.12-1.el7.x86_64
centos-release-gluster310-1.0-1.el7.centos.noarch
glusterfs-fuse-3.10.12-1.el7.x86_64
glusterfs-server-3.10.12-1.el7.x86_64
samba-vfs-glusterfs-4.7.1-6.el7.x86_64
glusterfs-libs-3.10.12-1.el7.x86_64
glusterfs-rdma-3.10.12-1.el7.x86_64
libvirt-daemon-driver-storage-gluster-3.9.0-14.el7_5.8.x86_64
glusterfs-cli-3.10.12-1.el7.x86_64
Diego
On Fri, Nov 9, 2018 at 10:09 AM Kaleb S. KEITHLEY <
Post by lejeczek
hi guys
I presume because 4.1.x has been in EPEL repo it is confirmed and
validated to work 100% with default samba installation.
GlusterFS — any version — is _not_ in EPEL.
However it is in the CentOS Storage SIG.
Post by lejeczek
But, I'd prefer to here you guys say you ACTUALLY have your
samba work
Post by Diego Remolina
Post by Kaleb S. KEITHLEY
Post by Diego Remolina
Post by lejeczek
100% with 4.1.x. Anybody?
Nobody has built Samba in the CentOS Storage SIG with GlusterFS
support.
Post by Diego Remolina
Post by Kaleb S. KEITHLEY
Post by Diego Remolina
Not for a long time anyway. The last time it was built was over
four
Post by Diego Remolina
Post by Kaleb S. KEITHLEY
Post by Diego Remolina
years ago — for el6.
It would be great if someone in one of the Gluster, Samba, or
CentOS
Post by Diego Remolina
Post by Kaleb S. KEITHLEY
Post by Diego Remolina
communities would start building it on a regular basis.
Any volunteers?
--
Kaleb
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Kaleb
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Anoop C S
2018-11-13 08:02:44 UTC
Permalink
Post by Diego Remolina
Hi Anoop,
https://revitpure.com/blog/how-to-use-central-and-local-files-in-revit
Once a central file is created, other folders are also created in the location of the file, which
contains many other files.
total 385588
drwxrws---. 4 dijuremo Staff 4096 Nov 12 19:03 .
drwxr-xr-x. 21 root root 4096 Nov 7 19:51 ..
drwxrws---. 2 dijuremo Staff 4096 Nov 12 19:05 2017-07-06 CAPE CORAL
CJDR_CENTRAL_R2017_backup
-rw-rw----. 1 dijuremo Staff 394825728 Jul 23 2017 2017-07-06 CAPE CORAL CJDR_CENTRAL_R2017.rvt
drwxrws---. 2 dijuremo Staff 4096 Nov 12 19:03 Revit_temp
\\ysmserver\vfsgluster
A simple ls -l of the _backup folder shows there is an existing file there called
CJDR_CENTRAL_R2017_backup/incrementtable.2108.dat
-rw-rw----. 1 dijuremo Staff 2357 Nov 12 19:13 2017-07-06 CAPE CORAL
CJDR_CENTRAL_R2017_backup/incrementtable.2108.dat
However, at this point, things are not OK. The file is *not* a central file. If I hit close and
open the file again, then Revit will hang, usually just go into the usual windows "Not Responding"
state. This can last for several minutes. I just closed the application via End Task after 5
minutes of waiting. Rather than double clicking on the file from the share, I also tried opening
Revit first, then opening the file from Revit using the Open dialog. This also hangs the program.
In one occasion, I tried to manually delete the folders (long_name_backup and Revit_temp) from
windows using File Explorer to try and recreate the central again and then the delete process hung
in one file, preview.1957.dat for almost a minute, but it finally succeeded. This is not normal
behavior.
total 11
drwxrws---. 2 dijuremo Staff 4096 Nov 12 19:39 .
drwxrws---. 4 dijuremo Staff 4096 Nov 12 19:03 ..
-rw-rw----. 1 dijuremo Staff 2753 Nov 12 19:36 preview.1957.dat
Thanks for explaining the issue.

I understand that you are experiencing hang while doing some operations on files/directories in a
GlusterFS volume share from a Windows client. For simplicity can you attach the output of following
command:

# gluster volume info <volume>
# testparm -s --section-name global
Post by Diego Remolina
[vfsgluster]
path = /vfsgluster
browseable = yes
create mask = 660
directory mask = 770
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 7
glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
glusterfs:volume = export
Since you have mentioned path as /vfsgluster I hope you are sharing a subdirectory under root of the
volume.
Post by Diego Remolina
Full smb.conf
http://termbin.com/y4j0
I see the "clustering" parameter set to 'yes'. How many nodes are there in the cluster? Out of those
how many are running as samba and/or gluster nodes?
Post by Diego Remolina
/var/log/samba/glusterfs-vfsgluster.log
http://termbin.com/5hdr
Please let me know if there is any other information I can provide.
Are there any errors in /var/log/samba/log.<IP/hostname>? IP/hostname = Windows client machine
Diego Remolina
2018-11-13 12:50:33 UTC
Permalink
Post by Anoop C S
Thanks for explaining the issue.
I understand that you are experiencing hang while doing some operations on files/directories in a
GlusterFS volume share from a Windows client. For simplicity can you attach the output of following
# gluster volume info <volume>
# testparm -s --section-name global
gluster v status export
Status of volume: export
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.0.1.7:/bricks/hdds/brick 49153 0 Y 2540
Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y 2800
Self-heal Daemon on localhost N/A N/A Y 2912
Self-heal Daemon on 10.0.1.6 N/A N/A Y 3107
Self-heal Daemon on 10.0.1.5 N/A N/A Y 5877

Task Status of Volume export
------------------------------------------------------------------------------
There are no active volume tasks

# gluster volume info export

Volume Name: export
Type: Replicate
Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: 10.0.1.7:/bricks/hdds/brick
Brick2: 10.0.1.6:/bricks/hdds/brick
Options Reconfigured:
diagnostics.brick-log-level: INFO
diagnostics.client-log-level: INFO
performance.cache-max-file-size: 256MB
client.event-threads: 5
server.event-threads: 5
cluster.readdir-optimize: on
cluster.lookup-optimize: on
performance.io-cache: on
performance.io-thread-count: 64
nfs.disable: on
cluster.server-quorum-type: server
performance.cache-size: 10GB
server.allow-insecure: on
transport.address-family: inet
performance.cache-samba-metadata: on
features.cache-invalidation-timeout: 600
performance.md-cache-timeout: 600
features.cache-invalidation: on
performance.cache-invalidation: on
network.inode-lru-limit: 65536
performance.cache-min-file-size: 0
performance.stat-prefetch: on
cluster.server-quorum-ratio: 51%

I had sent you the full smb.conf, so no need to run testparm -s
--section-name global, please reference:
http://termbin.com/y4j0
Post by Anoop C S
Post by Diego Remolina
[vfsgluster]
path = /vfsgluster
browseable = yes
create mask = 660
directory mask = 770
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 7
glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
glusterfs:volume = export
Since you have mentioned path as /vfsgluster I hope you are sharing a subdirectory under root of the
volume.
Yes, vfsgluster is a directory at the root of the export volume. It is
also currently mounted in /export so that the rest of the files can be
exported via samba with fuse mounts:

# mount | grep export
10.0.1.7:/export on /export type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)

# ls -ld /export/vfsgluster
drwxrws---. 4 dijuremo Staff 4096 Nov 12 20:24 /export/vfsgluster
Post by Anoop C S
Post by Diego Remolina
Full smb.conf
http://termbin.com/y4j0
I see the "clustering" parameter set to 'yes'. How many nodes are there in the cluster? Out of those
how many are running as samba and/or gluster nodes?
There are a total of 3 gluster peers, but only two have bricks. The
other is just present, but not even configured as an arbiter. Two of
the nodes with bricks run ctdb and samba.
Post by Anoop C S
Post by Diego Remolina
/var/log/samba/glusterfs-vfsgluster.log
http://termbin.com/5hdr
Please let me know if there is any other information I can provide.
Are there any errors in /var/log/samba/log.<IP/hostname>? IP/hostname = Windows client machine
I do not currently have the log file directive enabled in smb.conf, I
would have to enable it. Do you need me to repeat the process with it?

Diego
Anoop C S
2018-11-13 13:46:22 UTC
Permalink
Post by Diego Remolina
Post by Anoop C S
Thanks for explaining the issue.
I understand that you are experiencing hang while doing some operations on files/directories in
a
GlusterFS volume share from a Windows client. For simplicity can you attach the output of
following
# gluster volume info <volume>
# testparm -s --section-name global
gluster v status export
Status of volume: export
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.0.1.7:/bricks/hdds/brick 49153 0 Y 2540
Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y 2800
Self-heal Daemon on localhost N/A N/A Y 2912
Self-heal Daemon on 10.0.1.6 N/A N/A Y 3107
Self-heal Daemon on 10.0.1.5 N/A N/A Y 5877
Task Status of Volume export
------------------------------------------------------------------------------
There are no active volume tasks
# gluster volume info export
Volume Name: export
Type: Replicate
Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: 10.0.1.7:/bricks/hdds/brick
Brick2: 10.0.1.6:/bricks/hdds/brick
diagnostics.brick-log-level: INFO
diagnostics.client-log-level: INFO
performance.cache-max-file-size: 256MB
client.event-threads: 5
server.event-threads: 5
cluster.readdir-optimize: on
cluster.lookup-optimize: on
performance.io-cache: on
performance.io-thread-count: 64
nfs.disable: on
cluster.server-quorum-type: server
performance.cache-size: 10GB
server.allow-insecure: on
transport.address-family: inet
performance.cache-samba-metadata: on
features.cache-invalidation-timeout: 600
performance.md-cache-timeout: 600
features.cache-invalidation: on
performance.cache-invalidation: on
network.inode-lru-limit: 65536
performance.cache-min-file-size: 0
performance.stat-prefetch: on
cluster.server-quorum-ratio: 51%
I had sent you the full smb.conf, so no need to run testparm -s
http://termbin.com/y4j0
Fine.
Post by Diego Remolina
Post by Anoop C S
Post by Diego Remolina
[vfsgluster]
path = /vfsgluster
browseable = yes
create mask = 660
directory mask = 770
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 7
glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
glusterfs:volume = export
Since you have mentioned path as /vfsgluster I hope you are sharing a subdirectory under root of
the
volume.
Yes, vfsgluster is a directory at the root of the export volume.
Thanks for the confirmation.
Post by Diego Remolina
It is also currently mounted in /export so that the rest of the files can be
# mount | grep export
10.0.1.7:/export on /export type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
# ls -ld /export/vfsgluster
drwxrws---. 4 dijuremo Staff 4096 Nov 12 20:24 /export/vfsgluster
Post by Anoop C S
Post by Diego Remolina
Full smb.conf
http://termbin.com/y4j0
I see the "clustering" parameter set to 'yes'. How many nodes are there in the cluster? Out of
those
how many are running as samba and/or gluster nodes?
There are a total of 3 gluster peers, but only two have bricks. The
other is just present, but not even configured as an arbiter. Two of
the nodes with bricks run ctdb and samba.
OK. So basically a two node Samba-CTDB cluster.
Post by Diego Remolina
Post by Anoop C S
Post by Diego Remolina
/var/log/samba/glusterfs-vfsgluster.log
http://termbin.com/5hdr
Please let me know if there is any other information I can provide.
Are there any errors in /var/log/samba/log.<IP/hostname>? IP/hostname = Windows client machine
I do not currently have the log file directive enabled in smb.conf, I
would have to enable it. Do you need me to repeat the process with it?
Yes, preferably after adding the following parameters to [vfsgluster] share section(and of course a
restart):

kernel change notify = no
kernel oplocks = no
posix locking = no
Diego Remolina
2018-11-15 03:19:29 UTC
Permalink
Hi,

Please download the logs from:

https://www.dropbox.com/s/4k0zvmn4izhjtg7/samba-logs.tar.bz2?dl=0

These options had to be set in the [global] section:
kernel change notify = no
kernel oplocks = no

I also set log level = 10

I renamed the file as Test-Project.rvt for simplicity I opened revit
and from revit I opened this file. I then started attempting to save
it as a central at around 22:01:30s The file save then got stuck and
at around 22:05 it finally failed saying that two dat files did not
exist.

Diego
Post by Anoop C S
Post by Diego Remolina
Post by Anoop C S
Thanks for explaining the issue.
I understand that you are experiencing hang while doing some operations on files/directories in
a
GlusterFS volume share from a Windows client. For simplicity can you attach the output of
following
# gluster volume info <volume>
# testparm -s --section-name global
gluster v status export
Status of volume: export
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.0.1.7:/bricks/hdds/brick 49153 0 Y 2540
Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y 2800
Self-heal Daemon on localhost N/A N/A Y 2912
Self-heal Daemon on 10.0.1.6 N/A N/A Y 3107
Self-heal Daemon on 10.0.1.5 N/A N/A Y 5877
Task Status of Volume export
------------------------------------------------------------------------------
There are no active volume tasks
# gluster volume info export
Volume Name: export
Type: Replicate
Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: 10.0.1.7:/bricks/hdds/brick
Brick2: 10.0.1.6:/bricks/hdds/brick
diagnostics.brick-log-level: INFO
diagnostics.client-log-level: INFO
performance.cache-max-file-size: 256MB
client.event-threads: 5
server.event-threads: 5
cluster.readdir-optimize: on
cluster.lookup-optimize: on
performance.io-cache: on
performance.io-thread-count: 64
nfs.disable: on
cluster.server-quorum-type: server
performance.cache-size: 10GB
server.allow-insecure: on
transport.address-family: inet
performance.cache-samba-metadata: on
features.cache-invalidation-timeout: 600
performance.md-cache-timeout: 600
features.cache-invalidation: on
performance.cache-invalidation: on
network.inode-lru-limit: 65536
performance.cache-min-file-size: 0
performance.stat-prefetch: on
cluster.server-quorum-ratio: 51%
I had sent you the full smb.conf, so no need to run testparm -s
http://termbin.com/y4j0
Fine.
Post by Diego Remolina
Post by Anoop C S
Post by Diego Remolina
[vfsgluster]
path = /vfsgluster
browseable = yes
create mask = 660
directory mask = 770
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 7
glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
glusterfs:volume = export
Since you have mentioned path as /vfsgluster I hope you are sharing a subdirectory under root of
the
volume.
Yes, vfsgluster is a directory at the root of the export volume.
Thanks for the confirmation.
Post by Diego Remolina
It is also currently mounted in /export so that the rest of the files can be
# mount | grep export
10.0.1.7:/export on /export type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
# ls -ld /export/vfsgluster
drwxrws---. 4 dijuremo Staff 4096 Nov 12 20:24 /export/vfsgluster
Post by Anoop C S
Post by Diego Remolina
Full smb.conf
http://termbin.com/y4j0
I see the "clustering" parameter set to 'yes'. How many nodes are there in the cluster? Out of
those
how many are running as samba and/or gluster nodes?
There are a total of 3 gluster peers, but only two have bricks. The
other is just present, but not even configured as an arbiter. Two of
the nodes with bricks run ctdb and samba.
OK. So basically a two node Samba-CTDB cluster.
Post by Diego Remolina
Post by Anoop C S
Post by Diego Remolina
/var/log/samba/glusterfs-vfsgluster.log
http://termbin.com/5hdr
Please let me know if there is any other information I can provide.
Are there any errors in /var/log/samba/log.<IP/hostname>? IP/hostname = Windows client machine
I do not currently have the log file directive enabled in smb.conf, I
would have to enable it. Do you need me to repeat the process with it?
Yes, preferably after adding the following parameters to [vfsgluster] share section(and of course a
kernel change notify = no
kernel oplocks = no
posix locking = no
Anoop C S
2018-11-15 13:03:56 UTC
Permalink
Post by Diego Remolina
Hi,
https://www.dropbox.com/s/4k0zvmn4izhjtg7/samba-logs.tar.bz2?dl=0
[2018/11/14 22:01:31.974084, 10, pid=7577, effective(1009, 513), real(1009, 0)]
../source3/smbd/smb2_server.c:2279(smbd_smb2_request_dis
patch)
smbd_smb2_request_dispatch: opcode[SMB2_OP_FLUSH] mid = 9303
[2018/11/14 22:01:31.974123, 4, pid=7577, effective(1009, 513), real(1009, 0)]
../source3/smbd/uid.c:384(change_to_user)
Skipping user change - already user
[2018/11/14 22:01:31.974163, 10, pid=7577, effective(1009, 513), real(1009, 0)]
../source3/smbd/smb2_flush.c:134(smbd_smb2_flush_send)
smbd_smb2_flush: Test-Project_backup/preview.1957.dat - fnum 2399596398

I see the above flush request(SMB2_OP_FLUSH) without a response being logged compared to other
request-response pairs for SMB2_OP_QUERY_DIRECTORY, SMB2_OP_IOCTL, SMB2_OP_WRITE, SMB2_OP_READ which
brings me to the following bug:

https://bugzilla.samba.org/show_bug.cgi?id=13297

Is it also possible for you to collect network traces using wireshark from Windows client to samba
server along with corresponding samba logs?
Post by Diego Remolina
kernel change notify = no
kernel oplocks = no
Well, 'kernel oplocks' and 'posix locking' are listed (S) in man smb.conf(5) which can also be used
in [global] section meaning it affects all services. Anyway this should be fine.
Post by Diego Remolina
I also set log level = 10
Good one.
Post by Diego Remolina
I renamed the file as Test-Project.rvt for simplicity I opened revit
and from revit I opened this file. I then started attempting to save
it as a central at around 22:01:30s The file save then got stuck and
at around 22:05 it finally failed saying that two dat files did not
exist.
Do you experience problems with other applications and/or with other file types? Is this specific to
Revit?
Post by Diego Remolina
Diego
Post by Anoop C S
Post by Diego Remolina
Post by Anoop C S
Thanks for explaining the issue.
I understand that you are experiencing hang while doing some operations on files/directories
in
a
GlusterFS volume share from a Windows client. For simplicity can you attach the output of
following
# gluster volume info <volume>
# testparm -s --section-name global
gluster v status export
Status of volume: export
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.0.1.7:/bricks/hdds/brick 49153 0 Y 2540
Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y 2800
Self-heal Daemon on localhost N/A N/A Y 2912
Self-heal Daemon on 10.0.1.6 N/A N/A Y 3107
Self-heal Daemon on 10.0.1.5 N/A N/A Y 5877
Task Status of Volume export
------------------------------------------------------------------------------
There are no active volume tasks
# gluster volume info export
Volume Name: export
Type: Replicate
Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: 10.0.1.7:/bricks/hdds/brick
Brick2: 10.0.1.6:/bricks/hdds/brick
diagnostics.brick-log-level: INFO
diagnostics.client-log-level: INFO
performance.cache-max-file-size: 256MB
client.event-threads: 5
server.event-threads: 5
cluster.readdir-optimize: on
cluster.lookup-optimize: on
performance.io-cache: on
performance.io-thread-count: 64
nfs.disable: on
cluster.server-quorum-type: server
performance.cache-size: 10GB
server.allow-insecure: on
transport.address-family: inet
performance.cache-samba-metadata: on
features.cache-invalidation-timeout: 600
performance.md-cache-timeout: 600
features.cache-invalidation: on
performance.cache-invalidation: on
network.inode-lru-limit: 65536
performance.cache-min-file-size: 0
performance.stat-prefetch: on
cluster.server-quorum-ratio: 51%
I had sent you the full smb.conf, so no need to run testparm -s
http://termbin.com/y4j0
Fine.
Post by Diego Remolina
Post by Anoop C S
Post by Diego Remolina
[vfsgluster]
path = /vfsgluster
browseable = yes
create mask = 660
directory mask = 770
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 7
glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
glusterfs:volume = export
Since you have mentioned path as /vfsgluster I hope you are sharing a subdirectory under
root of
the
volume.
Yes, vfsgluster is a directory at the root of the export volume.
Thanks for the confirmation.
Post by Diego Remolina
It is also currently mounted in /export so that the rest of the files can be
# mount | grep export
10.0.1.7:/export on /export type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
# ls -ld /export/vfsgluster
drwxrws---. 4 dijuremo Staff 4096 Nov 12 20:24 /export/vfsgluster
Post by Anoop C S
Post by Diego Remolina
Full smb.conf
http://termbin.com/y4j0
I see the "clustering" parameter set to 'yes'. How many nodes are there in the cluster? Out
of
those
how many are running as samba and/or gluster nodes?
There are a total of 3 gluster peers, but only two have bricks. The
other is just present, but not even configured as an arbiter. Two of
the nodes with bricks run ctdb and samba.
OK. So basically a two node Samba-CTDB cluster.
Post by Diego Remolina
Post by Anoop C S
Post by Diego Remolina
/var/log/samba/glusterfs-vfsgluster.log
http://termbin.com/5hdr
Please let me know if there is any other information I can provide.
Are there any errors in /var/log/samba/log.<IP/hostname>? IP/hostname = Windows client
machine
I do not currently have the log file directive enabled in smb.conf, I
would have to enable it. Do you need me to repeat the process with it?
Yes, preferably after adding the following parameters to [vfsgluster] share section(and of
course a
kernel change notify = no
kernel oplocks = no
posix locking = no
Diego Remolina
2018-11-15 14:34:10 UTC
Permalink
I will try to do the wireshark captures within a week from now.

Up until recently, problems were only with Revit central files, but we
lost a server and ran on one (out of two) for a few days until the
motherboard arrived and we started seeing problems with other files
while writing them. At that point I decided to switch everything to
FUSE mounts to get rid of vfs objects = glusterfs in all shares. At
that point I also downgraded one samba version (running 4.7.1-6.el7_5)
since I had recently also upgraded to the latest CentOS had published
(4.7.1-9.el7_5) at the time before seeing more errors.

We are also working on getting ovirt and self-hosted engine out of
these servers, so once that happens, I can even upgrade to the latest
glusterfs 4.1.5 and do tests on a more current glusterfs version.

Someone at the office who has contacts with another IT person who
worked at a large Architectural firm said that IT person also had
mentioned that gluster did not work with Revit as they tested it an
failed. I have no other details than this.

Could you also try to reproduce it yourself? May help to allow you to
capture all the information you need. A free trial of Revit is
available at:

https://www.autodesk.com/products/revit/free-trial

You can just use one of the demo files that Revit comes with and
convert it to a central while stored in a gluster volume when using
vfs objects = glusterfs in samba and that should trigger the problems.

Diego
Post by Anoop C S
Post by Diego Remolina
Hi,
https://www.dropbox.com/s/4k0zvmn4izhjtg7/samba-logs.tar.bz2?dl=0
[2018/11/14 22:01:31.974084, 10, pid=7577, effective(1009, 513), real(1009, 0)]
../source3/smbd/smb2_server.c:2279(smbd_smb2_request_dis
patch)
smbd_smb2_request_dispatch: opcode[SMB2_OP_FLUSH] mid = 9303
[2018/11/14 22:01:31.974123, 4, pid=7577, effective(1009, 513), real(1009, 0)]
../source3/smbd/uid.c:384(change_to_user)
Skipping user change - already user
[2018/11/14 22:01:31.974163, 10, pid=7577, effective(1009, 513), real(1009, 0)]
../source3/smbd/smb2_flush.c:134(smbd_smb2_flush_send)
smbd_smb2_flush: Test-Project_backup/preview.1957.dat - fnum 2399596398
I see the above flush request(SMB2_OP_FLUSH) without a response being logged compared to other
request-response pairs for SMB2_OP_QUERY_DIRECTORY, SMB2_OP_IOCTL, SMB2_OP_WRITE, SMB2_OP_READ which
https://bugzilla.samba.org/show_bug.cgi?id=13297
Is it also possible for you to collect network traces using wireshark from Windows client to samba
server along with corresponding samba logs?
Post by Diego Remolina
kernel change notify = no
kernel oplocks = no
Well, 'kernel oplocks' and 'posix locking' are listed (S) in man smb.conf(5) which can also be used
in [global] section meaning it affects all services. Anyway this should be fine.
Post by Diego Remolina
I also set log level = 10
Good one.
Post by Diego Remolina
I renamed the file as Test-Project.rvt for simplicity I opened revit
and from revit I opened this file. I then started attempting to save
it as a central at around 22:01:30s The file save then got stuck and
at around 22:05 it finally failed saying that two dat files did not
exist.
Do you experience problems with other applications and/or with other file types? Is this specific to
Revit?
Post by Diego Remolina
Diego
Post by Anoop C S
Post by Diego Remolina
Post by Anoop C S
Thanks for explaining the issue.
I understand that you are experiencing hang while doing some operations on files/directories
in
a
GlusterFS volume share from a Windows client. For simplicity can you attach the output of
following
# gluster volume info <volume>
# testparm -s --section-name global
gluster v status export
Status of volume: export
Gluster process TCP Port RDMA Port Online Pid
------------------------------------------------------------------------------
Brick 10.0.1.7:/bricks/hdds/brick 49153 0 Y 2540
Brick 10.0.1.6:/bricks/hdds/brick 49153 0 Y 2800
Self-heal Daemon on localhost N/A N/A Y 2912
Self-heal Daemon on 10.0.1.6 N/A N/A Y 3107
Self-heal Daemon on 10.0.1.5 N/A N/A Y 5877
Task Status of Volume export
------------------------------------------------------------------------------
There are no active volume tasks
# gluster volume info export
Volume Name: export
Type: Replicate
Volume ID: b4353b3f-6ef6-4813-819a-8e85e5a95cff
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: 10.0.1.7:/bricks/hdds/brick
Brick2: 10.0.1.6:/bricks/hdds/brick
diagnostics.brick-log-level: INFO
diagnostics.client-log-level: INFO
performance.cache-max-file-size: 256MB
client.event-threads: 5
server.event-threads: 5
cluster.readdir-optimize: on
cluster.lookup-optimize: on
performance.io-cache: on
performance.io-thread-count: 64
nfs.disable: on
cluster.server-quorum-type: server
performance.cache-size: 10GB
server.allow-insecure: on
transport.address-family: inet
performance.cache-samba-metadata: on
features.cache-invalidation-timeout: 600
performance.md-cache-timeout: 600
features.cache-invalidation: on
performance.cache-invalidation: on
network.inode-lru-limit: 65536
performance.cache-min-file-size: 0
performance.stat-prefetch: on
cluster.server-quorum-ratio: 51%
I had sent you the full smb.conf, so no need to run testparm -s
http://termbin.com/y4j0
Fine.
Post by Diego Remolina
Post by Anoop C S
Post by Diego Remolina
[vfsgluster]
path = /vfsgluster
browseable = yes
create mask = 660
directory mask = 770
kernel share modes = No
vfs objects = glusterfs
glusterfs:loglevel = 7
glusterfs:logfile = /var/log/samba/glusterfs-vfsgluster.log
glusterfs:volume = export
Since you have mentioned path as /vfsgluster I hope you are sharing a subdirectory under
root of
the
volume.
Yes, vfsgluster is a directory at the root of the export volume.
Thanks for the confirmation.
Post by Diego Remolina
It is also currently mounted in /export so that the rest of the files can be
# mount | grep export
10.0.1.7:/export on /export type fuse.glusterfs
(rw,relatime,user_id=0,group_id=0,allow_other,max_read=131072)
# ls -ld /export/vfsgluster
drwxrws---. 4 dijuremo Staff 4096 Nov 12 20:24 /export/vfsgluster
Post by Anoop C S
Post by Diego Remolina
Full smb.conf
http://termbin.com/y4j0
I see the "clustering" parameter set to 'yes'. How many nodes are there in the cluster? Out
of
those
how many are running as samba and/or gluster nodes?
There are a total of 3 gluster peers, but only two have bricks. The
other is just present, but not even configured as an arbiter. Two of
the nodes with bricks run ctdb and samba.
OK. So basically a two node Samba-CTDB cluster.
Post by Diego Remolina
Post by Anoop C S
Post by Diego Remolina
/var/log/samba/glusterfs-vfsgluster.log
http://termbin.com/5hdr
Please let me know if there is any other information I can provide.
Are there any errors in /var/log/samba/log.<IP/hostname>? IP/hostname = Windows client
machine
I do not currently have the log file directive enabled in smb.conf, I
would have to enable it. Do you need me to repeat the process with it?
Yes, preferably after adding the following parameters to [vfsgluster] share section(and of
course a
kernel change notify = no
kernel oplocks = no
posix locking = no
lejeczek
2018-11-12 20:11:21 UTC
Permalink
Post by Kaleb S. KEITHLEY
Post by lejeczek
hi guys
I presume because 4.1.x has been in EPEL repo it is confirmed and
validated to work 100% with default samba installation.
GlusterFS — any version — is _not_ in EPEL.
However it is in the CentOS Storage SIG.
all glusterfs I see & get is simply from/by installing packages, eg

centos-release-gluster312-1.0-2.el7.centos.noarch

centos-release-gluster41-1.0-3.el7.centos.noarch

These are simply repo packages but still these repos points to centos'
own mirrors.

To me it seems like all gluster versions are there in Centos, so is Samba.

So I do not tamper with packages, nor I get/use extra third party repos.

Thus my question is valid and correct - anybody would assume that these
should be tested, specifically Samba against glusterfs, especially
against 4.1.x

No?
Post by Kaleb S. KEITHLEY
Post by lejeczek
But, I'd prefer to here you guys say you ACTUALLY have your samba work
100% with 4.1.x. Anybody?
Nobody has built Samba in the CentOS Storage SIG with GlusterFS support.
Not for a long time anyway. The last time it was built was over four
years ago — for el6.
It would be great if someone in one of the Gluster, Samba, or CentOS
communities would start building it on a regular basis.
Any volunteers?
Loading...