[Gluster-users] Structure needs cleaning on some files

[Johan Huysmans]

Hi All,

When reading some files we get this error:
md5sum: /path/to/file.xml: Structure needs cleaning

in /var/log/glusterfs/mnt-sharedfs.log we see these errors:
[2013-12-10 08:07:32.256910] W
[client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-0: remote
operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
[client-rpc-fops.c:526:client3_3_stat_cbk] 1-testvolume-client-1: remote
operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure needs
cleaning)

We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)

This is my gluster configuration:
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Bricks:
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
Options Reconfigured:
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5

And this is how the applications work:
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these files.
On the node where the files are written we don't see any problem, and
can read that file without problems.
On the other node we have problems (error messages above) reading that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.


How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and mount of
the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.


I hope someone can help me with this problems.

Thanks,
Johan Huysmans

[Johan Huysmans]

I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.

gr.
Johan Huysmans

Post by Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these files.
On the node where the files are written we don't see any problem, and
can read that file without problems.
On the other node we have problems (error messages above) reading that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and mount
of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Johan Huysmans]

Hi All,

It seems I can easily reproduce the problem.

* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum

This is really strange behaviour.
Is this normal, can this be altered with a a setting?

Thanks for any info,
gr.
Johan

Post by Johan Huysmans
I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans

Post by Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these files.
On the node where the files are written we don't see any problem, and
can read that file without problems.
On the other node we have problems (error messages above) reading that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and mount
of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Johan Huysmans]

I created a bug for this issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1041109

gr.
Johan

Post by Johan Huysmans
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan

Post by Johan Huysmans
I could reproduce this problem with while my mount point is running
in debug mode.
logfile is attached.
gr.
Johan Huysmans

Post by Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Johan Huysmans]

I created a bug for this issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1041109

gr.
Johan

Post by Johan Huysmans
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan

Post by Johan Huysmans
I could reproduce this problem with while my mount point is running
in debug mode.
logfile is attached.
gr.
Johan Huysmans

Post by Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Johan Huysmans]

987555 -?is that fixed in 3.5?Or did it even make it into 3.4.2couldn't=20
find a note on that.Show stopper for moving from?3.2.x to anywhere for me!
cheersb?

=20
Cheers,
Vijay
=20
=20
=20
=20
=20
=20
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F

Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

--=20


=20
=20
=20
=20
=20
=20
=20
Bernhard Glomm

IT Administration


=20
=20
Phone:
=20
=20
[phone]
=20
=20
Fax:
=20
=20
[phone]
=20
=20
Skype:
=20
=20
bernhard.glomm.ecologic
=20
=20
=20
=20
=20
=20
=20
=20
=20
=20
=20
=20
=20
=20
=20
=20
Ecologic Institut gemeinn?tzige GmbH | Pfalzburger Str. 43/44 |=20
10717 Berlin | Germany

GF: R. Andreas Kraemer | AG: Charlottenburg HRB 57947 |=20
USt/VAT-IdNr.: DE811963464

Ecologic? is a Trade Mark (TM) of Ecologic Institut=20
gemeinn?tzige GmbH
=20
=20
=20
=20

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131210=
/475454d4/attachment-0001.html
------------------------------

Message: 36
Date: Tue, 10 Dec 2013 10:02:14 +0100
From: Johan Huysmans <***@inuits.be>
To: "gluster-***@gluster.org" <gluster-***@gluster.org>
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Message-ID: <***@inuits.be>
Content-Type: text/plain; charset=3D"iso-8859-1"; Format=3D"flowed"

I could reproduce this problem with while my mount point is running in=20
debug mode.
logfile is attached.

gr.
Johan Huysmans

Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W=20
[client-rpc-fops.c:526:client3=5F3=5Fstat=5Fcbk] 1-testvolume-client-0:=20
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W=20
[client-rpc-fops.c:526:client3=5F3=5Fstat=5Fcbk] 1-testvolume-client-1:=20
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse=5Fattr=5Fcbk]=20
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml =3D> -1 (Structure=20
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using=20
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 =3D 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these files.
On the node where the files are written we don't see any problem, and=20
can read that file without problems.
On the other node we have problems (error messages above) reading that=20
file.
The problem occurs when we perform a md5sum on the exact file, when=20
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and mount=20
of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F

Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
A non-text attachment was scrubbed...
Name: gluster=5Fdebug.log
Type: text/x-log
Size: 16600 bytes
Desc: not available
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131210=
/bdf626dc/attachment-0001.bin
------------------------------

Message: 37
Date: Tue, 10 Dec 2013 10:08:43 +0100
From: Heiko Kr?mer <***@anynines.com>
To: gluster-***@gluster.org
Subject: Re: [Gluster-users] Gluster infrastructure question
Message-ID: <***@anynines.com>
Content-Type: text/plain; charset=3D"iso-8859-1"

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi guys,

thanks for all these reports. Well, I think I'll change my Raid level
to 6 and let the Raid controller build and rebuild all Raid members
and replicate again with glusterFS. I get more capacity but I need to
check if the write throughput acceptable.

I think, I can't take advantage of using glusterFS with a lot of
Bricks because I've found more cons as pros in my case.

@Ben thx for this very detailed document!


Cheers and Thanks
Heiko

Replicas are defined in the order bricks are listed in the volume
create command. So gluster volume create myvol replica 2
server1:/data/brick1 server2:/data/brick1 server3:/data/brick1
server4:/data/brick1 will replicate between server1 and server2
and replicate between server3 and server4.
=20
Bricks added to a replica 2 volume after it's been created will
require pairs of bricks,
=20
The best way to "force" replication to happen on another server
is to just define it that way.

=20
Yup, that's understood. The problem is when (for argument's sake)
=20
* We've defined 4 hosts with 10 disks each * Each individual disk
is a brick * Replication is defined correctly when creating the
volume initially * I'm on holidays, my employer buys a single node,
configures it brick-per-disk, and the IT junior adds it to the
cluster
=20
All good up until that final point, and then I've got that fifth
node at the end replicating to itself. Node goes down some months
later, chaos ensues.
=20
Not a GlusterFS/technology problem, but a problem with what
frequently happens at a human level. As a sysadmin, these are also
things I need to work around, even if it means deviating from best
practices. :)
=20
-Dan =5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=
Gluster-users

http://supercolony.gluster.org/mailman/listinfo/gluster-users
=20

- --=20
Anynines.com

Avarteq GmbH
B.Sc. Informatik
Heiko Kr?mer
CIO
Twitter: @anynines

- ----
Gesch?ftsf?hrer: Alexander Fai?t, Dipl.-Inf.(FH) Julian Fischer
Handelsregister: AG Saarbr?cken HRB 17413, Ust-IdNr.: DE262633168
Sitz: Saarbr?cken
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.14 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJSptoTAAoJELxFogM4ixOFJTsIAJBWed3AGiiI+PDC2ubfboKc
UPkMc+zuirRh2+QJBAoZ4CsAv9eIZ5NowclSSby9PTq2XRjjLvMdKuI+IbXCRT4j
AbMLYfP3g4Q+agXnY6N6WJ6ZIqXQ8pbCK3shYp9nBfVYkiDUT1bGk0WcgQmEWTCw
ta1h17LYkworIDRtqWQAl4jr4JR4P3x4cmwOZiHCVCtlyOP02x/fN4dji6nyOtuB
kQPBVsND5guQNU8Blg5cQoES5nthtuwJdkWXB+neaCZd/u3sexVSNe5m15iWbyYg
mAoVvlBJ473IKATlxM5nVqcUhmjFwNcc8MMwczXxTkwniYzth53BSoltPn7kIx4=3D
=3Depys
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: hkraemer.vcf
Type: text/x-vcard
Size: 277 bytes
Desc: not available
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131210=
/f663943d/attachment-0001.vcf
------------------------------

Message: 38
Date: Tue, 10 Dec 2013 10:42:43 +0100
From: Johan Huysmans <***@inuits.be>
To: gluster-***@gluster.org, ***@web.de
Subject: Re: [Gluster-users] Errors from PHP stat() on files and
directories in a glusterfs mount
Message-ID: <***@inuits.be>
Content-Type: text/plain; charset=3D"iso-8859-1"; Format=3D"flowed"

Hi,

It seems I have a related problem (just posted this on the mailing list).
Do you already have a solution for this problem?

gr.
Johan Huysmans

Hi,
I'm trying to use glusterfs to mirror the ownCloud "data" area between=20
2 servers.
They are using debian jessie due to some dependancies that I have for=20
other components.
This is where my issue rears it's ugly head. This is failing because I=20
can't stat the files and directories on my glusterfs mount.
/var/www/owncloud/data is where I am mounting the volume and I can=20
reproduce the error using a simple php test application, so I don't=20
think that it is apache or owncloud related.
I'd be grateful for any pointers on how to resolve this problem.
Thanks,
Bill
Attached is "simple.php" test and the results of executing "strace=20
php5 simple.php" twice, once with the glusterfs mounted=20
(simple.php.strace-glusterfs) and once against the file system when=20
unmounted (simple.php.strace-unmounted).
------------------------------------------------------------------------
/var/log/glusterfs/var-www-owncloud-data.log
[2013-12-05 18:33:50.802250] D=20
[client-handshake.c:185:client=5Fstart=5Fping] 0-gv-ocdata-client-0:=20
returning as transport is already disconnected OR there are no frames=20
(0 || 0)
[2013-12-05 18:33:50.825132] D=20
[afr-self-heal-common.c:138:afr=5Fsh=5Fprint=5Fpending=5Fmatrix]=20
0-gv-ocdata-replicate-0: pending=5Fmatrix: [ 0 0 ]
[2013-12-05 18:33:50.825322] D=20
[afr-self-heal-common.c:138:afr=5Fsh=5Fprint=5Fpending=5Fmatrix]=20
0-gv-ocdata-replicate-0: pending=5Fmatrix: [ 0 0 ]
[2013-12-05 18:33:50.825393] D=20
[afr-self-heal-common.c:887:afr=5Fmark=5Fsources] 0-gv-ocdata-replicate-0=

:=20

Number of sources: 0
[2013-12-05 18:33:50.825456] D=20
[afr-self-heal-data.c:825:afr=5Flookup=5Fselect=5Fread=5Fchild=5Fby=5Ftxn=

=5Ftype]=20

0-gv-ocdata-replicate-0: returning read=5Fchild: 0
[2013-12-05 18:33:50.825511] D=20
[afr-common.c:1380:afr=5Flookup=5Fselect=5Fread=5Fchild]=20
0-gv-ocdata-replicate-0: Source selected as 0 for /
[2013-12-05 18:33:50.825579] D=20
[afr-common.c:1117:afr=5Flookup=5Fbuild=5Fresponse=5Fparams]=20
0-gv-ocdata-replicate-0: Building lookup response from 0
[2013-12-05 18:33:50.827069] D=20
[afr-common.c:131:afr=5Flookup=5Fxattr=5Freq=5Fprepare]=20
0-gv-ocdata-replicate-0: /check.txt: failed to get the gfid from dict
[2013-12-05 18:33:50.829409] D=20
[client-handshake.c:185:client=5Fstart=5Fping] 0-gv-ocdata-client-0:=20
returning as transport is already disconnected OR there are no frames=20
(0 || 0)
[2013-12-05 18:33:50.836719] D=20
[afr-self-heal-common.c:138:afr=5Fsh=5Fprint=5Fpending=5Fmatrix]=20
0-gv-ocdata-replicate-0: pending=5Fmatrix: [ 0 0 ]
[2013-12-05 18:33:50.836870] D=20
[afr-self-heal-common.c:138:afr=5Fsh=5Fprint=5Fpending=5Fmatrix]=20
0-gv-ocdata-replicate-0: pending=5Fmatrix: [ 0 0 ]
[2013-12-05 18:33:50.836941] D=20
[afr-self-heal-common.c:887:afr=5Fmark=5Fsources] 0-gv-ocdata-replicate-0=

:=20

Number of sources: 0
[2013-12-05 18:33:50.837002] D=20
[afr-self-heal-data.c:825:afr=5Flookup=5Fselect=5Fread=5Fchild=5Fby=5Ftxn=

=5Ftype]=20

0-gv-ocdata-replicate-0: returning read=5Fchild: 0
[2013-12-05 18:33:50.837058] D=20
[afr-common.c:1380:afr=5Flookup=5Fselect=5Fread=5Fchild]=20
0-gv-ocdata-replicate-0: Source selected as 0 for /check.txt
[2013-12-05 18:33:50.837129] D=20
[afr-common.c:1117:afr=5Flookup=5Fbuild=5Fresponse=5Fparams]=20
0-gv-ocdata-replicate-0: Building lookup response from 0
Other bits of information
Linux bbb-1 3.8.13-bone30 #1 SMP Thu Nov 14 02:59:07 UTC 2013 armv7l=20
GNU/Linux
Desired=3DUnknown/Install/Remove/Purge/Hold
|=20
=20

Status=3DNot/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-p=
end

|/ Err?=3D(none)/Reinst-required (Status,Err: uppercase=3Dbad)
||/ Name Version Architecture Description
=20

+++-=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D-=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D-=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D-=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

ii glusterfs-client 3.4.1-1 armhf clustered=20
file-system (client package)
ii glusterfs-common 3.4.1-1 armhf GlusterFS=20
common libraries and translator modules
ii glusterfs-server 3.4.1-1 armhf clustered=20
file-system (server package)
mount
bbb-1:gv-ocdata on /var/www/owncloud/data type fuse.glusterfs=20
=20

(rw,relatime,user=5Fid=3D0,group=5Fid=3D0,default=5Fpermissions,allow=5Foth=
er,max=5Fread=3D131072)

/etc/fstab
UUID=3Dxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx /sdhc ext4 defaults 0 0
bbb-1:gv-ocdata /var/www/owncloud/data glusterfs=20
defaults,=5Fnetdev,log-level=3DDEBUG 0 0
ls -al on the various paths
drwxrwxr-x 7 root root 4096 Nov 28 19:15 /sdhc/
drwxrwx--- 5 www-data www-data 4096 Dec 5 00:50 /sdhc/gv-ocdata/
-rw-r--r-- 2 root root 10 Dec 5 00:50 /sdhc/gv-ocdata/check.txt
drwxrwx--- 5 www-data www-data 4096 Dec 5 00:50 /var/www/owncloud/data/
-rw-r--r-- 1 root root 10 Dec 5 00:50 /var/www/owncloud/data/check.txt
Attribute "glusterfs.volume-id" has a 16 byte value for=20
/var/www/owncloud/data
Attribute "glusterfs.volume-id" has a 16 byte value for /sdhc/gv-ocdata/
Attribute "gfid" has a 16 byte value for /sdhc/gv-ocdata/
Attribute "glusterfs.dht" has a 16 byte value for /sdhc/gv-ocdata/
Attribute "afr.gv-ocdata-client-0" has a 12 byte value for=20
/sdhc/gv-ocdata/
Attribute "afr.gv-ocdata-client-1" has a 12 byte value for=20
/sdhc/gv-ocdata/
Attribute "gfid" has a 16 byte value for /sdhc/gv-ocdata/check.txt
Attribute "afr.gv-ocdata-client-0" has a 12 byte value for=20
/sdhc/gv-ocdata/check.txt
Attribute "afr.gv-ocdata-client-1" has a 12 byte value for=20
/sdhc/gv-ocdata/check.txt
=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=

=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F=5F

Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131210=
/d77e25bb/attachment-0001.html
------------------------------

Message: 39
Date: Tue, 10 Dec 2013 21:03:36 +1100
From: Andrew Lau <***@andrewklau.com>
To: Ben Turner <***@redhat.com>
Cc: "gluster-***@gluster.org List" <gluster-***@gluster.org>
Subject: Re: [Gluster-users] Gluster infrastructure question
Message-ID:
<CAD7dF9c3uexEG++1YEHwh3zw7a1Xy+=3DCo=5FxO+***@mail.gmail.com>
Content-Type: text/plain; charset=3D"utf-8"

Hi Ben,

For glusterfs would you recommend the enterprise-storage
or throughput-performance tuned profile?

Thanks,
Andrew

----- Original Message -----

Sent: Monday, December 9, 2013 2:26:45 PM
Subject: Re: [Gluster-users] Gluster infrastructure question
----- Original Message -----

Sent: Monday, December 9, 2013 8:18:28 AM
Subject: [Gluster-users] Gluster infrastructure question
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Heyho guys,
I'm running since years glusterfs in a small environment without big
problems.
Now I'm going to use glusterFS for a bigger cluster but I've some
questions :)
* 4 Servers
* 20 x 2TB HDD, each
* Raidcontroller
* Raid 10
* 4x bricks =3D> Replicated, Distributed volume
* Gluster 3.4
1)
I'm asking me, if I can delete the raid10 on each server and create
for each HDD a separate brick.
In this case have a volume 80 Bricks so 4 Server x 20 HDD's. Is=20

there

any experience about the write throughput in a production system=20

with

many of bricks like in this case? In addition i'll get double of HDD
capacity.

=20

http://rhsummit.files.wordpress.com/2012/03/england-rhs-performance.pdf

=20

http://rhsummit.files.wordpress.com/2013/07/england=5Fth=5F0450=5Frhs=5Fper=
f=5Fpractices-4=5Fneependra.pdf

-b

? RAID arrays
? More RAID LUNs for better concurrency
? For RAID6, 256-KB stripe size
I use a single RAID 6 that is divided into several LUNs for my bricks.

For

example, on my Dell servers(with PERC6 RAID controllers) each server=20

has

12

disks that I put into raid 6. Then I break the RAID 6 into 6 LUNs and
create a new PV/VG/LV for each brick.

[Maik Kulbe]

How do you mount your Client? FUSE? I had similar problems when playing around with the timeout options for the FUSE mount. If they are too high they cache the metadata for too long. When you move the file the inode should stay the same and on the second node the path should stay in cache for a while so it still knows the inode for that moved files old path thus can act on the file without knowing it's path.

The problems kick in when you delete a file and recreate it - the cache tries to access the old inode, which was deleted, thus throwing errors. If I recall correctly the "structure needs cleaning" is one of two error messages I got, depending on which of the timeout mount options was set to a higher value.

-----Original Mail-----
From: Johan Huysmans [***@inuits.be]
Sent: 12.12.13 - 14:51:35
To: gluster-***@gluster.org [gluster-***@gluster.org]

Subject: Re: [Gluster-users] Structure needs cleaning on some files

Post by Johan Huysmans
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan
I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these
files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Anand Avati]

I have the same question. Do you have excessively high --entry-timeout
parameter to your FUSE mount? In any case, "Structure needs cleaning" error
should not surface up to FUSE and that is still a bug.


On Thu, Dec 12, 2013 at 12:46 PM, Maik Kulbe

Post by Maik Kulbe
How do you mount your Client? FUSE? I had similar problems when playing
around with the timeout options for the FUSE mount. If they are too high
they cache the metadata for too long. When you move the file the inode
should stay the same and on the second node the path should stay in cache
for a while so it still knows the inode for that moved files old path thus
can act on the file without knowing it's path.
The problems kick in when you delete a file and recreate it - the cache
tries to access the old inode, which was deleted, thus throwing errors. If
I recall correctly the "structure needs cleaning" is one of two error
messages I got, depending on which of the timeout mount options was set to
a higher value.
-----Original Mail-----
Sent: 12.12.13 - 14:51:35
Subject: Re: [Gluster-users] Structure needs cleaning on some files

Post by Johan Huysmans
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan
I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these
files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Anand Avati]

Looks like your issue was fixed by patch http://review.gluster.org/4989/ in
master branch. Backporting this to release-3.4 now.

Thanks!
Avati

Post by Anand Avati
I have the same question. Do you have excessively high --entry-timeout
parameter to your FUSE mount? In any case, "Structure needs cleaning" error
should not surface up to FUSE and that is still a bug.
On Thu, Dec 12, 2013 at 12:46 PM, Maik Kulbe <

Post by Maik Kulbe
How do you mount your Client? FUSE? I had similar problems when playing
around with the timeout options for the FUSE mount. If they are too high
they cache the metadata for too long. When you move the file the inode
should stay the same and on the second node the path should stay in cache
for a while so it still knows the inode for that moved files old path thus
can act on the file without knowing it's path.
The problems kick in when you delete a file and recreate it - the cache
tries to access the old inode, which was deleted, thus throwing errors. If
I recall correctly the "structure needs cleaning" is one of two error
messages I got, depending on which of the timeout mount options was set to
a higher value.
-----Original Mail-----
Sent: 12.12.13 - 14:51:35
Subject: Re: [Gluster-users] Structure needs cleaning on some files

Post by Johan Huysmans
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan
I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these
files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Johan Huysmans]

Hi All,

I retested this on Gluster 3.4.2. The problem reoccured.
It doesn't show the "Structure needs cleaning" message, but just a
message that the file doesn't exists.

I updated the bug with the usecase I used to reproduce it.
https://bugzilla.redhat.com/show_bug.cgi?id=1041109

Thanks for taking a look at it and hopefully fixing it.

Greetings
Johan Huysmans

Post by Anand Avati
Looks like your issue was fixed by patch
http://review.gluster.org/4989/ in master branch. Backporting this to
release-3.4 now.
Thanks!
Avati
I have the same question. Do you have excessively high
--entry-timeout parameter to your FUSE mount? In any case,
"Structure needs cleaning" error should not surface up to FUSE and
that is still a bug.
On Thu, Dec 12, 2013 at 12:46 PM, Maik Kulbe
How do you mount your Client? FUSE? I had similar problems
when playing around with the timeout options for the FUSE
mount. If they are too high they cache the metadata for too
long. When you move the file the inode should stay the same
and on the second node the path should stay in cache for a
while so it still knows the inode for that moved files old
path thus can act on the file without knowing it's path.
The problems kick in when you delete a file and recreate it -
the cache tries to access the old inode, which was deleted,
thus throwing errors. If I recall correctly the "structure
needs cleaning" is one of two error messages I got, depending
on which of the timeout mount options was set to a higher value.
-----Original Mail-----
Sent: 12.12.13 - 14:51:35
Subject: Re: [Gluster-users] Structure needs cleaning on some files
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum
/path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum
/path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan
I could reproduce this problem with while my mount point
is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
[client-rpc-fops.c:526:client3_3_stat_cbk]
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
[client-rpc-fops.c:526:client3_3_stat_cbk]
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W
[fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these
files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is
no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

[Khoi Mai]

John,

We've experienced the same problem too it appears. I've ran an strace on
the file when it returns "structure needs cleaning" after trying to do an
'ls' . I get wind of this from my developers when they make modifcations
to their content and want to see it served up from apache.

It will stay like that for infinite it seems like. The band-aid I found
was if I did a 'ls -l' of the directory the file lives in, it will clear
up and then I am able to do an 'ls -l' on the file that had the "structure
needs cleaning" message. My setup is a 2x2 distri/rep bricks.....

[Khoi Mai]

Avati,

Do you know how long that will take to have that packaged into the
glusterfs-fuse3.4.1-x.rpm on gluster.org/downloads for rhel6

Thanks,
Khoi



Message: 8
Date: Thu, 12 Dec 2013 13:38:18 -0800
From: Anand Avati <***@gluster.org>
To: Maik Kulbe <***@linux-web-development.de>
Cc: "gluster-***@gluster.org" <gluster-***@gluster.org>
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Message-ID:
<CAFboF2zNOFFbuM9_ayrw6Wv+DdXVOr+D=9Az0cxNx+***@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Looks like your issue was fixed by patch http://review.gluster.org/4989/
in
master branch. Backporting this to release-3.4 now.

Thanks!
Avati




From: gluster-users-***@gluster.org
To: gluster-***@gluster.org
Date: 12/13/2013 05:58 AM
Subject: Gluster-users Digest, Vol 68, Issue 14
Sent by: gluster-users-***@gluster.org



Send Gluster-users mailing list submissions to
gluster-***@gluster.org

To subscribe or unsubscribe via the World Wide Web, visit

http://supercolony.gluster.org/mailman/listinfo/gluster-users
or, via email, send a message with subject or body 'help' to
gluster-users-***@gluster.org

You can reach the person managing the list at
gluster-users-***@gluster.org

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Gluster-users digest..."


Today's Topics:

1. Re: Structure needs cleaning on some files (Johan Huysmans)
2. Re: Structure needs cleaning on some files (Johan Huysmans)
3. Re: Gluster Community Weekly Meeting (Vijay Bellur)
4. Re: Gluster Community Weekly Meeting (James)
5. Re: Gluster Community Weekly Meeting (Vijay Bellur)
6. Re: Structure needs cleaning on some files (Maik Kulbe)
7. Re: Structure needs cleaning on some files (Anand Avati)
8. Re: Structure needs cleaning on some files (Anand Avati)
9. Gerrit doesn't use HTTPS (James)
10. gluster fails under heavy array job load load (harry mangalam)
11. qemu remote insecure connections (Joe Topjian)
12. Documentation hackathon for 3.5 (Vijay Bellur)
13. Re: gluster fails under heavy array job load load (Anand Avati)
14. Re: Gluster Community Weekly Meeting (Niels de Vos)


----------------------------------------------------------------------

Message: 1
Date: Thu, 12 Dec 2013 14:40:37 +0100
From: Johan Huysmans <***@inuits.be>
To: "gluster-***@gluster.org" <gluster-***@gluster.org>
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Message-ID: <***@inuits.be>
Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"

I created a bug for this issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1041109

gr.
Johan

Post by Johan Huysmans
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan

Post by Johan Huysmans
I could reproduce this problem with while my mount point is running
in debug mode.
logfile is attached.
gr.
Johan Huysmans

Post by Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these

files.

Post by Johan Huysmans

Post by Johan Huysmans

Post by Johan Huysmans
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/4152cb13/attachment-0001.html
------------------------------

Message: 2
Date: Thu, 12 Dec 2013 14:51:35 +0100
From: Johan Huysmans <***@inuits.be>
To: "gluster-***@gluster.org" <gluster-***@gluster.org>
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Message-ID: <***@inuits.be>
Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"

I created a bug for this issue:

https://bugzilla.redhat.com/show_bug.cgi?id=1041109

gr.
Johan

Post by Johan Huysmans
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan

Post by Johan Huysmans
I could reproduce this problem with while my mount point is running
in debug mode.
logfile is attached.
gr.
Johan Huysmans

Post by Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these

files.

Post by Johan Huysmans

Post by Johan Huysmans

Post by Johan Huysmans
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/97a7843c/attachment-0001.html
------------------------------

Message: 3
Date: Fri, 13 Dec 2013 00:13:30 +0530
From: Vijay Bellur <***@redhat.com>
To: James <***@gmail.com>
Cc: "gluster-***@gluster.org" <gluster-***@gluster.org>, Gluster
Devel <gluster-***@nongnu.org>, Niels de Vos
<***@redhat.com>
Subject: Re: [Gluster-users] Gluster Community Weekly Meeting
Message-ID: <***@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed
No problem. It would be really good to have everybody in the meeting,
but if you cannot comments are definitely welcome :).

Post by Johan Huysmans
1) About the pre-packaged VM comment's. I've gotten Vagrant working on
Fedora. I'm using this to rapidly spin up and test GlusterFS.
https://ttboj.wordpress.com/2013/12/09/vagrant-on-fedora-with-libvirt/
In the coming week or so, I'll be publishing the Vagrant file for my
GlusterFS setup, but if you really want it now I can send you an early
version. This obviously integrates with Puppet-Gluster, but whether
you use that or not is optional. I think this is the best way to test
GlusterFS. If someone gives me hosting, I could publish "pre-built"
images very easily. Let me know what you think.

Niels - do you have any thoughts here?

Post by Johan Huysmans
2) I never heard back from any action items from 2 weeks ago. I think
someone was going to connect me with a way to get access to some VM's
for testing stuff !

I see that there is an ongoing offline thread now. I think that should
result in you getting those VMs.

Post by Johan Huysmans
3) Hagarth: RE: typos, I have at least one spell check patch against
3.4.1 I sent it to list before, but someone told me to enroll in the
jenkins thing, which wasn't worth it for a small patch. Let me know if
you want it.

There are more typos now. I ran a cursory check with misspell-check [1]
and found quite a few. Having that cleaned up on master and release-3.5
would be great. Since the number is more, I am sure the patch would be
non-trivial and having that routed through gerrit would be great. If you
need a how to on getting to gerrit, it is available at [2].

Post by Johan Huysmans
4a) Someone mentioned documentation. Please feel free to merge in

https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md

Post by Johan Huysmans
(markdown format). I have gone to great lengths to format this so that
it displays properly in github markdown, and standard (pandoc)
markdown. This way it works on github, and can also be rendered to a

https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf

Post by Johan Huysmans
You can use the file as a template!

Again having this in gerrit would be useful for merging the puppet
documentation.

Post by Johan Huysmans
4b) I think the documentation should be kept in the same repo as
GlusterFS. This way, when you submit a feature branch, it can also
come with documentation. Lots of people work this way. It helps you
get minimal docs there, and/or at least some example code or a few
sentences. Also, looking at the docs, you can see what commits came
with this

I am with you on this one. After we are done with the planned
documentation hackathon, let us open a new thread on this to get more
opinions.

-Vijay

[1] https://github.com/lyda/misspell-check

[2]
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow

Post by Johan Huysmans
Thanks!
James

------------------------------

Message: 4
Date: Thu, 12 Dec 2013 13:48:31 -0500
From: James <***@gmail.com>
To: Vijay Bellur <***@redhat.com>
Cc: "gluster-***@gluster.org" <gluster-***@gluster.org>, Gluster
Devel <gluster-***@nongnu.org>
Subject: Re: [Gluster-users] Gluster Community Weekly Meeting
Message-ID:
<CADCaTgqJrJ6uTyGiti+q0SpXxMjE+m-***@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Post by Johan Huysmans

Post by Johan Huysmans
4a) Someone mentioned documentation. Please feel free to merge in

https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md

Post by Johan Huysmans

Post by Johan Huysmans
(markdown format). I have gone to great lengths to format this so that
it displays properly in github markdown, and standard (pandoc)
markdown. This way it works on github, and can also be rendered to a

https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf

Post by Johan Huysmans

Post by Johan Huysmans
You can use the file as a template!

Again having this in gerrit would be useful for merging the puppet
documentation.

Okay, I'll try to look into Gerrit and maybe submit a fake patch for
testing.
When and where (in the tree) would be a good time to submit a doc
patch? It's probably best to wait until after your docs hackathon,
right?


------------------------------

Message: 5
Date: Fri, 13 Dec 2013 00:30:09 +0530
From: Vijay Bellur <***@redhat.com>
To: James <***@gmail.com>
Cc: "gluster-***@gluster.org" <gluster-***@gluster.org>, Gluster
Devel <gluster-***@nongnu.org>
Subject: Re: [Gluster-users] Gluster Community Weekly Meeting
Message-ID: <***@redhat.com>
Content-Type: text/plain; charset=UTF-8; format=flowed

Post by Johan Huysmans

Post by Johan Huysmans

Post by Johan Huysmans
4a) Someone mentioned documentation. Please feel free to merge in

https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md

Post by Johan Huysmans

Post by Johan Huysmans

Post by Johan Huysmans
(markdown format). I have gone to great lengths to format this so that
it displays properly in github markdown, and standard (pandoc)
markdown. This way it works on github, and can also be rendered to a

https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf

Post by Johan Huysmans

Post by Johan Huysmans

Post by Johan Huysmans
You can use the file as a template!

Again having this in gerrit would be useful for merging the puppet
documentation.

Okay, I'll try to look into Gerrit and maybe submit a fake patch for

testing.

Post by Johan Huysmans
When and where (in the tree) would be a good time to submit a doc
patch? It's probably best to wait until after your docs hackathon,
right?

Just added a page in preparation for the documentation hackathon:

http://www.gluster.org/community/documentation/index.php/Submitting_Documentation_Patches


I think the puppet guide can be under a new hierarchy located at
doc/deploy-guide/markdown/en-US/. You can certainly submit the puppet
doc patch as part of the hackathon.

-Vijay



------------------------------

Message: 6
Date: Thu, 12 Dec 2013 21:46:12 +0100
From: "Maik Kulbe" <***@linux-web-development.de>
To: "Johan Huysmans" <***@inuits.be>,
"gluster-***@gluster.org" <gluster-***@gluster.org>
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Message-ID:
<***@linux-web-development.de>
Content-Type: text/plain; charset="utf-8"; Format="flowed"

How do you mount your Client? FUSE? I had similar problems when playing
around with the timeout options for the FUSE mount. If they are too high
they cache the metadata for too long. When you move the file the inode
should stay the same and on the second node the path should stay in cache
for a while so it still knows the inode for that moved files old path thus
can act on the file without knowing it's path.

The problems kick in when you delete a file and recreate it - the cache
tries to access the old inode, which was deleted, thus throwing errors. If
I recall correctly the "structure needs cleaning" is one of two error
messages I got, depending on which of the timeout mount options was set to
a higher value.

-----Original Mail-----
From: Johan Huysmans [***@inuits.be]
Sent: 12.12.13 - 14:51:35
To: gluster-***@gluster.org [gluster-***@gluster.org]

Subject: Re: [Gluster-users] Structure needs cleaning on some files

Post by Johan Huysmans
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan
I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these
files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 2332 bytes
Desc: not available
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/99d59ead/attachment-0001.bin
------------------------------

Message: 7
Date: Thu, 12 Dec 2013 13:26:56 -0800
From: Anand Avati <***@gluster.org>
To: Maik Kulbe <***@linux-web-development.de>
Cc: "gluster-***@gluster.org" <gluster-***@gluster.org>
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Message-ID:
<CAFboF2x1CraXbYSokGt1jhOhBCny+9LRPzASt-***@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

I have the same question. Do you have excessively high --entry-timeout
parameter to your FUSE mount? In any case, "Structure needs cleaning"
error
should not surface up to FUSE and that is still a bug.


On Thu, Dec 12, 2013 at 12:46 PM, Maik Kulbe

Post by Johan Huysmans
How do you mount your Client? FUSE? I had similar problems when playing
around with the timeout options for the FUSE mount. If they are too high
they cache the metadata for too long. When you move the file the inode
should stay the same and on the second node the path should stay in

cache

Post by Johan Huysmans
for a while so it still knows the inode for that moved files old path

thus

Post by Johan Huysmans
can act on the file without knowing it's path.
The problems kick in when you delete a file and recreate it - the cache
tries to access the old inode, which was deleted, thus throwing errors.

If

Post by Johan Huysmans
I recall correctly the "structure needs cleaning" is one of two error
messages I got, depending on which of the timeout mount options was set

to

Post by Johan Huysmans
a higher value.
-----Original Mail-----
Sent: 12.12.13 - 14:51:35
Subject: Re: [Gluster-users] Structure needs cleaning on some files

Post by Johan Huysmans
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan
I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these
files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/730b11a2/attachment-0001.html
------------------------------

Message: 8
Date: Thu, 12 Dec 2013 13:38:18 -0800
From: Anand Avati <***@gluster.org>
To: Maik Kulbe <***@linux-web-development.de>
Cc: "gluster-***@gluster.org" <gluster-***@gluster.org>
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Message-ID:
<CAFboF2zNOFFbuM9_ayrw6Wv+DdXVOr+D=9Az0cxNx+***@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Looks like your issue was fixed by patch http://review.gluster.org/4989/
in
master branch. Backporting this to release-3.4 now.

Thanks!
Avati

Post by Johan Huysmans
I have the same question. Do you have excessively high --entry-timeout
parameter to your FUSE mount? In any case, "Structure needs cleaning"

error

Post by Johan Huysmans
should not surface up to FUSE and that is still a bug.
On Thu, Dec 12, 2013 at 12:46 PM, Maik Kulbe <

Post by Johan Huysmans
How do you mount your Client? FUSE? I had similar problems when playing
around with the timeout options for the FUSE mount. If they are too

high

Post by Johan Huysmans

Post by Johan Huysmans
they cache the metadata for too long. When you move the file the inode
should stay the same and on the second node the path should stay in

cache

Post by Johan Huysmans

Post by Johan Huysmans
for a while so it still knows the inode for that moved files old path

thus

Post by Johan Huysmans

Post by Johan Huysmans
can act on the file without knowing it's path.
The problems kick in when you delete a file and recreate it - the cache
tries to access the old inode, which was deleted, thus throwing errors.

If

Post by Johan Huysmans

Post by Johan Huysmans
I recall correctly the "structure needs cleaning" is one of two error
messages I got, depending on which of the timeout mount options was set

to

Post by Johan Huysmans

Post by Johan Huysmans
a higher value.
-----Original Mail-----
Sent: 12.12.13 - 14:51:35
Subject: Re: [Gluster-users] Structure needs cleaning on some files

Post by Johan Huysmans
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan
I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these
files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <
http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/f3934ac1/attachment-0001.html
------------------------------

Message: 9
Date: Thu, 12 Dec 2013 17:35:02 -0500
From: James <***@gmail.com>
To: "gluster-***@gluster.org" <gluster-***@gluster.org>, Gluster
Devel <gluster-***@nongnu.org>
Subject: [Gluster-users] Gerrit doesn't use HTTPS
Message-ID:
<CADCaTgrmcoJuNL4=***@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

I just noticed that the Gluster Gerrit [1] doesn't use HTTPS!

Can this be fixed ASAP?

Cheers,
James

[1] http://review.gluster.org/


------------------------------

Message: 10
Date: Thu, 12 Dec 2013 17:03:12 -0800
From: harry mangalam <***@uci.edu>
To: "gluster-***@gluster.org" <gluster-***@gluster.org>
Subject: [Gluster-users] gluster fails under heavy array job load load
Message-ID: <***@stunted>
Content-Type: text/plain; charset="us-ascii"

Hi All,
(Gluster Volume Details at bottom)

I've posted some of this previously, but even after various upgrades,
attempted fixes, etc, it remains a problem.


Short version: Our gluster fs (~340TB) provides scratch space for a
~5000core
academic compute cluster.
Much of our load is streaming IO, doing a lot of genomics work, and that
is
the load under which we saw this latest failure.
Under heavy batch load, especially array jobs, where there might be
several
64core nodes doing I/O on the 4servers/8bricks, we often get job failures
that
have the following profile:

Client POV:
Here is a sampling of the client logs (/var/log/glusterfs/gl.log) for all
compute nodes that indicated interaction with the user's files
<http://pastie.org/8548781>

Here are some client Info logs that seem fairly serious:
<http://pastie.org/8548785>

The errors that referenced this user were gathered from all the nodes that

were running his code (in compute*) and agglomerated with:

cut -f2,3 -d']' compute* |cut -f1 -dP | sort | uniq -c | sort -gr

and placed here to show the profile of errors that his run generated.
<http://pastie.org/8548796>

so 71 of them were:
W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-gl-client-7: remote
operation failed: Transport endpoint is not connected.
etc

We've seen this before and previously discounted it bc it seems to have
been
related to the problem of spurious NFS-related bugs, but now I'm wondering

whether it's a real problem.
Also the 'remote operation failed: Stale file handle. ' warnings.

There were no Errors logged per se, tho some of the W's looked fairly
nasty,
like the 'dht_layout_dir_mismatch'

[Vijay Bellur]

Post by Khoi Mai
Avati,
Do you know how long that will take to have that packaged into the
glusterfs-fuse3.4.1-x.rpm on gluster.org/downloads for rhel6

This has been merged to release-3.4 and will be part of 3.4.2 which is
scheduled for sometime next week. The next qa release for 3.4.2 will
also contain this fix. If you can test out the fix and let us know the
outcome, that would be great.

Thanks,
Vijay

Post by Khoi Mai
Thanks,
Khoi
Message: 8
Date: Thu, 12 Dec 2013 13:38:18 -0800
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Content-Type: text/plain; charset="iso-8859-1"
Looks like your issue was fixed by patch http://review.gluster.org/4989/in
master branch. Backporting this to release-3.4 now.
Thanks!
Avati
Date: 12/13/2013 05:58 AM
Subject: Gluster-users Digest, Vol 68, Issue 14
------------------------------------------------------------------------
Send Gluster-users mailing list submissions to
To subscribe or unsubscribe via the World Wide Web, visit
http://supercolony.gluster.org/mailman/listinfo/gluster-users
or, via email, send a message with subject or body 'help' to
You can reach the person managing the list at
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Gluster-users digest..."
1. Re: Structure needs cleaning on some files (Johan Huysmans)
2. Re: Structure needs cleaning on some files (Johan Huysmans)
3. Re: Gluster Community Weekly Meeting (Vijay Bellur)
4. Re: Gluster Community Weekly Meeting (James)
5. Re: Gluster Community Weekly Meeting (Vijay Bellur)
6. Re: Structure needs cleaning on some files (Maik Kulbe)
7. Re: Structure needs cleaning on some files (Anand Avati)
8. Re: Structure needs cleaning on some files (Anand Avati)
9. Gerrit doesn't use HTTPS (James)
10. gluster fails under heavy array job load load (harry mangalam)
11. qemu remote insecure connections (Joe Topjian)
12. Documentation hackathon for 3.5 (Vijay Bellur)
13. Re: gluster fails under heavy array job load load (Anand Avati)
14. Re: Gluster Community Weekly Meeting (Niels de Vos)
----------------------------------------------------------------------
Message: 1
Date: Thu, 12 Dec 2013 14:40:37 +0100
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan

Post by Johan Huysmans
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan

Post by Johan Huysmans
I could reproduce this problem with while my mount point is running
in debug mode.
logfile is attached.
gr.
Johan Huysmans

Post by Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/4152cb13/attachment-0001.html>
------------------------------
Message: 2
Date: Thu, 12 Dec 2013 14:51:35 +0100
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan

Post by Johan Huysmans
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan

Post by Johan Huysmans
I could reproduce this problem with while my mount point is running
in debug mode.
logfile is attached.
gr.
Johan Huysmans

Post by Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/97a7843c/attachment-0001.html>
------------------------------
Message: 3
Date: Fri, 13 Dec 2013 00:13:30 +0530
Subject: Re: [Gluster-users] Gluster Community Weekly Meeting
Content-Type: text/plain; charset=UTF-8; format=flowed
No problem. It would be really good to have everybody in the meeting,
but if you cannot comments are definitely welcome :).

Post by Johan Huysmans
1) About the pre-packaged VM comment's. I've gotten Vagrant working on
Fedora. I'm using this to rapidly spin up and test GlusterFS.
https://ttboj.wordpress.com/2013/12/09/vagrant-on-fedora-with-libvirt/
In the coming week or so, I'll be publishing the Vagrant file for my
GlusterFS setup, but if you really want it now I can send you an early
version. This obviously integrates with Puppet-Gluster, but whether
you use that or not is optional. I think this is the best way to test
GlusterFS. If someone gives me hosting, I could publish "pre-built"
images very easily. Let me know what you think.

Niels - do you have any thoughts here?

Post by Johan Huysmans
2) I never heard back from any action items from 2 weeks ago. I think
someone was going to connect me with a way to get access to some VM's
for testing stuff !

I see that there is an ongoing offline thread now. I think that should
result in you getting those VMs.

Post by Johan Huysmans
3) Hagarth: RE: typos, I have at least one spell check patch against
3.4.1 I sent it to list before, but someone told me to enroll in the
jenkins thing, which wasn't worth it for a small patch. Let me know if
you want it.

There are more typos now. I ran a cursory check with misspell-check [1]
and found quite a few. Having that cleaned up on master and release-3.5
would be great. Since the number is more, I am sure the patch would be
non-trivial and having that routed through gerrit would be great. If you
need a how to on getting to gerrit, it is available at [2].

Post by Johan Huysmans
4a) Someone mentioned documentation. Please feel free to merge in
https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md
(markdown format). I have gone to great lengths to format this so that
it displays properly in github markdown, and standard (pandoc)
markdown. This way it works on github, and can also be rendered to a

https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf

Post by Johan Huysmans
You can use the file as a template!

Again having this in gerrit would be useful for merging the puppet
documentation.

Post by Johan Huysmans
4b) I think the documentation should be kept in the same repo as
GlusterFS. This way, when you submit a feature branch, it can also
come with documentation. Lots of people work this way. It helps you
get minimal docs there, and/or at least some example code or a few
sentences. Also, looking at the docs, you can see what commits came
with this

I am with you on this one. After we are done with the planned
documentation hackathon, let us open a new thread on this to get more
opinions.
-Vijay
[1] https://github.com/lyda/misspell-check
[2]
http://www.gluster.org/community/documentation/index.php/Development_Work_Flow

Post by Johan Huysmans
Thanks!
James

------------------------------
Message: 4
Date: Thu, 12 Dec 2013 13:48:31 -0500
Subject: Re: [Gluster-users] Gluster Community Weekly Meeting
Content-Type: text/plain; charset=UTF-8

Post by Johan Huysmans

Post by Johan Huysmans
4a) Someone mentioned documentation. Please feel free to merge in

https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md

Post by Johan Huysmans

Post by Johan Huysmans
(markdown format). I have gone to great lengths to format this so that
it displays properly in github markdown, and standard (pandoc)
markdown. This way it works on github, and can also be rendered to a

https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf

Post by Johan Huysmans

Post by Johan Huysmans
You can use the file as a template!

Again having this in gerrit would be useful for merging the puppet
documentation.

Okay, I'll try to look into Gerrit and maybe submit a fake patch for
testing.
When and where (in the tree) would be a good time to submit a doc
patch? It's probably best to wait until after your docs hackathon,
right?
------------------------------
Message: 5
Date: Fri, 13 Dec 2013 00:30:09 +0530
Subject: Re: [Gluster-users] Gluster Community Weekly Meeting
Content-Type: text/plain; charset=UTF-8; format=flowed

Post by Johan Huysmans

Post by Johan Huysmans

Post by Johan Huysmans
4a) Someone mentioned documentation. Please feel free to merge in

https://github.com/purpleidea/puppet-gluster/blob/master/DOCUMENTATION.md

Post by Johan Huysmans

Post by Johan Huysmans

Post by Johan Huysmans
(markdown format). I have gone to great lengths to format this so that
it displays properly in github markdown, and standard (pandoc)
markdown. This way it works on github, and can also be rendered to a

https://github.com/purpleidea/puppet-gluster/raw/master/puppet-gluster-documentation.pdf

Post by Johan Huysmans

Post by Johan Huysmans

Post by Johan Huysmans
You can use the file as a template!

Again having this in gerrit would be useful for merging the puppet
documentation.

Okay, I'll try to look into Gerrit and maybe submit a fake patch for

testing.

Post by Johan Huysmans
When and where (in the tree) would be a good time to submit a doc
patch? It's probably best to wait until after your docs hackathon,
right?

http://www.gluster.org/community/documentation/index.php/Submitting_Documentation_Patches
I think the puppet guide can be under a new hierarchy located at
doc/deploy-guide/markdown/en-US/. You can certainly submit the puppet
doc patch as part of the hackathon.
-Vijay
------------------------------
Message: 6
Date: Thu, 12 Dec 2013 21:46:12 +0100
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Content-Type: text/plain; charset="utf-8"; Format="flowed"
How do you mount your Client? FUSE? I had similar problems when playing
around with the timeout options for the FUSE mount. If they are too high
they cache the metadata for too long. When you move the file the inode
should stay the same and on the second node the path should stay in
cache for a while so it still knows the inode for that moved files old
path thus can act on the file without knowing it's path.
The problems kick in when you delete a file and recreate it - the cache
tries to access the old inode, which was deleted, thus throwing errors.
If I recall correctly the "structure needs cleaning" is one of two error
messages I got, depending on which of the timeout mount options was set
to a higher value.
-----Original Mail-----
Sent: 12.12.13 - 14:51:35
Subject: Re: [Gluster-users] Structure needs cleaning on some files

Post by Johan Huysmans
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan
I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these
files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 2332 bytes
Desc: not available
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/99d59ead/attachment-0001.bin>
------------------------------
Message: 7
Date: Thu, 12 Dec 2013 13:26:56 -0800
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Content-Type: text/plain; charset="iso-8859-1"
I have the same question. Do you have excessively high --entry-timeout
parameter to your FUSE mount? In any case, "Structure needs cleaning" error
should not surface up to FUSE and that is still a bug.
On Thu, Dec 12, 2013 at 12:46 PM, Maik Kulbe

Post by Johan Huysmans
How do you mount your Client? FUSE? I had similar problems when playing
around with the timeout options for the FUSE mount. If they are too high
they cache the metadata for too long. When you move the file the inode
should stay the same and on the second node the path should stay in cache
for a while so it still knows the inode for that moved files old path

thus

Post by Johan Huysmans
can act on the file without knowing it's path.
The problems kick in when you delete a file and recreate it - the cache
tries to access the old inode, which was deleted, thus throwing

errors. If

Post by Johan Huysmans
I recall correctly the "structure needs cleaning" is one of two error
messages I got, depending on which of the timeout mount options was

set to

Post by Johan Huysmans
a higher value.
-----Original Mail-----
Sent: 12.12.13 - 14:51:35
Subject: Re: [Gluster-users] Structure needs cleaning on some files

Post by Johan Huysmans
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan
I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these
files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/730b11a2/attachment-0001.html>
------------------------------
Message: 8
Date: Thu, 12 Dec 2013 13:38:18 -0800
Subject: Re: [Gluster-users] Structure needs cleaning on some files
Content-Type: text/plain; charset="iso-8859-1"
Looks like your issue was fixed by patch http://review.gluster.org/4989/in
master branch. Backporting this to release-3.4 now.
Thanks!
Avati

Post by Johan Huysmans
I have the same question. Do you have excessively high --entry-timeout
parameter to your FUSE mount? In any case, "Structure needs cleaning"

error

Post by Johan Huysmans
should not surface up to FUSE and that is still a bug.
On Thu, Dec 12, 2013 at 12:46 PM, Maik Kulbe <

Post by Johan Huysmans
How do you mount your Client? FUSE? I had similar problems when playing
around with the timeout options for the FUSE mount. If they are too high
they cache the metadata for too long. When you move the file the inode
should stay the same and on the second node the path should stay in

cache

Post by Johan Huysmans

Post by Johan Huysmans
for a while so it still knows the inode for that moved files old

path thus

Post by Johan Huysmans

Post by Johan Huysmans
can act on the file without knowing it's path.
The problems kick in when you delete a file and recreate it - the cache
tries to access the old inode, which was deleted, thus throwing

errors. If

Post by Johan Huysmans

Post by Johan Huysmans
I recall correctly the "structure needs cleaning" is one of two error
messages I got, depending on which of the timeout mount options was

set to

Post by Johan Huysmans

Post by Johan Huysmans
a higher value.
-----Original Mail-----
Sent: 12.12.13 - 14:51:35
Subject: Re: [Gluster-users] Structure needs cleaning on some files

Post by Johan Huysmans
https://bugzilla.redhat.com/show_bug.cgi?id=1041109
gr.
Johan
Hi All,
It seems I can easily reproduce the problem.
* on node 1 create a file (touch , cat , ...).
* on node 2 take md5sum of direct file (md5sum /path/to/file)
* on node 1 move file to other name (mv file file1)
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working although the file is not really there
* on node 1 change file content
* on node 2 take md5sum of direct file (md5sum /path/to/file), this is
still working and has a changed md5sum
This is really strange behaviour.
Is this normal, can this be altered with a a setting?
Thanks for any info,
gr.
Johan
I could reproduce this problem with while my mount point is running in
debug mode.
logfile is attached.
gr.
Johan Huysmans
Hi All,
md5sum: /path/to/file.xml: Structure needs cleaning
[2013-12-10 08:07:32.256910] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.257436] W
remote operation failed: No such file or directory
[2013-12-10 08:07:32.259356] W [fuse-bridge.c:705:fuse_attr_cbk]
0-glusterfs-fuse: 8230: STAT() /path/to/file.xml => -1 (Structure
needs cleaning)
We are using gluster 3.4.1-3 on CentOS6.
Our servers are 64-bit, our clients 32-bit (we are already using
--enable-ino32 on the mountpoint)
Volume Name: testvolume
Type: Replicate
Volume ID: ca9c2f87-5d5b-4439-ac32-b7c138916df7
Status: Started
Number of Bricks: 1 x 2 = 2
Transport-type: tcp
Brick1: SRV-1:/gluster/brick1
Brick2: SRV-2:/gluster/brick2
performance.force-readdirp: on
performance.stat-prefetch: off
network.ping-timeout: 5
We have 2 client nodes who both have a fuse.glusterfs mountpoint.
On 1 client node we have a application which writes files.
On the other client node we have a application which reads these
files.
On the node where the files are written we don't see any problem,
and can read that file without problems.
On the other node we have problems (error messages above) reading
that file.
The problem occurs when we perform a md5sum on the exact file, when
perform a md5sum on all files in that directory there is no problem.
How can we solve this problem as this is annoying.
The problem occurs after some time (can be days), an umount and
mount of the mountpoint solves it for some days.
Once it occurs (and we don't remount) it occurs every time.
I hope someone can help me with this problems.
Thanks,
Johan Huysmans
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

_______________________________________________
Gluster-users mailing list
http://supercolony.gluster.org/mailman/listinfo/gluster-users

-------------- next part --------------
An HTML attachment was scrubbed...
<http://supercolony.gluster.org/pipermail/gluster-users/attachments/20131212/f3934ac1/attachment-0001.html>
------------------------------
Message: 9
Date: Thu, 12 Dec 2013 17:35:02 -0500
Subject: [Gluster-users] Gerrit doesn't use HTTPS
Content-Type: text/plain; charset=UTF-8
I just noticed that the Gluster Gerrit [1] doesn't use HTTPS!
Can this be fixed ASAP?
Cheers,
James
[1] http://review.gluster.org/
------------------------------
Message: 10
Date: Thu, 12 Dec 2013 17:03:12 -0800
Subject: [Gluster-users] gluster fails under heavy array job load load
Content-Type: text/plain; charset="us-ascii"
Hi All,
(Gluster Volume Details at bottom)
I've posted some of this previously, but even after various upgrades,
attempted fixes, etc, it remains a problem.
Short version: Our gluster fs (~340TB) provides scratch space for a
~5000core
academic compute cluster.
Much of our load is streaming IO, doing a lot of genomics work, and that is
the load under which we saw this latest failure.
Under heavy batch load, especially array jobs, where there might be several
64core nodes doing I/O on the 4servers/8bricks, we often get job
failures that
Here is a sampling of the client logs (/var/log/glusterfs/gl.log) for all
compute nodes that indicated interaction with the user's files
<http://pastie.org/8548781>
<http://pastie.org/8548785>
The errors that referenced this user were gathered from all the nodes that
cut -f2,3 -d']' compute* |cut -f1 -dP | sort | uniq -c | sort -gr
and placed here to show the profile of errors that his run generated.
<http://pastie.org/8548796>
W [client-rpc-fops.c:2624:client3_3_lookup_cbk] 0-gl-client-7: remote
operation failed: Transport endpoint is not connected.
etc
We've seen this before and previously discounted it bc it seems to have
been
related to the problem of spurious NFS-related bugs, but now I'm wondering
whether it's a real problem.
Also the 'remote operation failed: Stale file handle. ' warnings.
There were no Errors logged per se, tho some of the W's looked fairly
nasty,
like the 'dht_layout_dir_mismatch'