[Gluster-users] sometimes entry remains in "gluster v heal vol-name info" until visit it from mnt

[Ravishankar N]

+ gluster-users.

Adding Karthik to see if he has some cycles to look into this.

-Ravi

Hi, glusterfs expert
When I test with glusterfs version 3.12.3 I find it quite often that
sometimes, there are entry remains in gluster volume heal
<vol-name>info output for long time, *it does not disappear until you
visit it from the mount point, is this normal*?
# gluster v heal services info
Brick sn-0.local:/mnt/bricks/services/brick
Status: Connected
Number of entries: 0
Brick sn-1.local:/mnt/bricks/services/brick
Status: Connected
Number of entries: 0
Brick sn-2.local:/mnt/bricks/services/brick
/fstest_88402c989256d6e39e50208c90c1e85d          //this entry remains
in the output until you touch /mnt/services/
fstest_88402c989256d6e39e50208c90c1e85d
Status: Connected
Number of entries: 1
# ssh sn-2.local
Warning: Permanently added 'sn-2.local' (RSA) to the list of known hosts.
USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR
LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL
(fsclish) AND/OR LIMITED BASH SHELL.
Read /opt/nokia/share/security/readme_root.txt for more details.
# cd /mnt/bricks/services/brick/.glusterfs/indices/xattrop/
# ls
9138e315-efd6-46e0-8a3a-db535078c781
xattrop-dfcd7e67-8c2d-4ef1-93e2-c180073c8d87
# getfattr -m . -d -e hex
/mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/
getfattr: Removing leading '/' from absolute path names
# file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/
trusted.afr.services-client-1=0x000000000000000100000001
trusted.gfid=0x9138e315efd646e08a3adb535078c781
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
# getfattr -m . -d -e hex
/mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/
getfattr: Removing leading '/' from absolute path names
mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/
trusted.gfid=0x0ccb5c1f96064e699f62fdc72cf036f5
“fstest_88402c989256d6e39e50208c90c1e85d” is only seen from sn-2 mount
point and sn-2 service brick, there is no such entry if you ls
/mnt/services on sn-0 or sn-1.
# cd /mnt/services/
# ls
backup db fstest_88402c989256d6e39e50208c90c1e85d  LCM NE3SAgent 
_nokrcpautoremoteuser  PM9  RCP_Backup SS_AlLightProcessor  SymptomDataUpl
commoncollector EventCorrelationEngine  hypertracer          Log 
netserv    ODS                    ptp rcpha       SWM

[Karthik Subrahmanya]

Hey,

Please provide the glustershd log from all the nodes and client logs on the
node from where you did the lookup on the file to resolve this issue.

Regards,
Karthik

Post by Ravishankar N
+ gluster-users.
Adding Karthik to see if he has some cycles to look into this.
-Ravi
Hi, glusterfs expert
When I test with glusterfs version 3.12.3 I find it quite often that
sometimes, there are entry remains in gluster volume heal <vol-name>info
output for long time, *it does not disappear until you visit it from the
mount point, is this normal*?
# gluster v heal services info
Brick sn-0.local:/mnt/bricks/services/brick
Status: Connected
Number of entries: 0
Brick sn-1.local:/mnt/bricks/services/brick
Status: Connected
Number of entries: 0
Brick sn-2.local:/mnt/bricks/services/brick
/fstest_88402c989256d6e39e50208c90c1e85d //this entry remains in
the output until you touch /mnt/services/
fstest_88402c989256d6e39e50208c90c1e85d
Status: Connected
Number of entries: 1
# ssh sn-2.local
Warning: Permanently added 'sn-2.local' (RSA) to the list of known hosts.
USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR
LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL (fsclish)
AND/OR LIMITED BASH SHELL.
Read /opt/nokia/share/security/readme_root.txt for more details.
# cd /mnt/bricks/services/brick/.glusterfs/indices/xattrop/
# ls
9138e315-efd6-46e0-8a3a-db535078c781
xattrop-dfcd7e67-8c2d-4ef1-93e2-c180073c8d87
# getfattr -m . -d -e hex
/mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/
getfattr: Removing leading '/' from absolute path names
# file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/
trusted.afr.services-client-1=0x000000000000000100000001
trusted.gfid=0x9138e315efd646e08a3adb535078c781
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
# getfattr -m . -d -e hex
/mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/
getfattr: Removing leading '/' from absolute path names
mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/
trusted.gfid=0x0ccb5c1f96064e699f62fdc72cf036f5
“fstest_88402c989256d6e39e50208c90c1e85d” is only seen from sn-2 mount
point and sn-2 service brick, there is no such entry if you ls
/mnt/services on sn-0 or sn-1.
# cd /mnt/services/
# ls
backup db
fstest_88402c989256d6e39e50208c90c1e85d LCM NE3SAgent
_nokrcpautoremoteuser PM9 RCP_Backup SS_AlLightProcessor SymptomDataUpl
commoncollector EventCorrelationEngine hypertracer
Log netserv ODS ptp rcpha SWM

[Zhou, Cynthia (NSB - CN/Hangzhou)]

Hi, I meet this issue again, from my env on sn-1 mstate volume brick dir there is additional fstest_49e9d4767e95a3eda0a599437ef6ce8f, but on other sn nodes(sn-0, sn-2) there is no such directory.


[***@sn-1:/mnt/bricks/mstate/brick]
# stat fstest_49e9d4767e95a3eda0a599437ef6ce8f
File: fstest_49e9d4767e95a3eda0a599437ef6ce8f
Size: 4096 Blocks: 16 IO Block: 4096 directory
Device: fd61h/64865d Inode: 45450 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-10-10 06:51:09.045660991 +0300
Modify: 2018-10-10 06:51:09.045660991 +0300
Change: 2018-10-10 07:06:05.060278140 +0300
Birth: -
[***@sn-1:/mnt/bricks/mstate/brick]
# gluster v heal mstate info
Brick sn-0.local:/mnt/bricks/mstate/brick
Status: Connected
Number of entries: 0

Brick sn-2.local:/mnt/bricks/mstate/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/mstate/brick
/fstest_49e9d4767e95a3eda0a599437ef6ce8f
Status: Connected
Number of entries: 1

[***@sn-1:/mnt/bricks/mstate/brick]
# cd .glusterfs/indices/xattrop/
[***@sn-1:/mnt/bricks/mstate/brick/.glusterfs/indices/xattrop]
# ls
0c40b296-a4d6-46c9-b7a5-037105de84aa xattrop-90daa317-04db-4704-9f34-6f571eb3691c
[***@sn-1:/mnt/bricks/mstate/brick/.glusterfs/indices/xattrop]
# cd ../../..
[***@sn-1:/mnt/bricks/mstate/brick]
# getfattr -m . -d -e hex fstest_49e9d4767e95a3eda0a599437ef6ce8f
# file: fstest_49e9d4767e95a3eda0a599437ef6ce8f
trusted.afr.mstate-client-1=0x000000000000000100000001
trusted.gfid=0x0c40b296a4d646c9b7a5037105de84aa
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

[***@sn-1:/mnt/bricks/mstate/brick]
#
Enclosed is the glustershd log on sn-1,from which I see following log related to fstest_49e9d4767e95a3eda0a599437ef6ce8f

[2018-10-10 06:22:12.976487] I [MSGID: 108026] [afr-self-heald.c:341:afr_shd_selfheal] 0-mstate-replicate-0: entry: path /fstest_49e9d4767e95a3eda0a599437ef6ce8f, gfid: 0c40b296-a4d6-46c9-b7a5-037105de84aa //this is added in the entrance of function afr_shd_selfheal.




From: Karthik Subrahmanya <***@redhat.com>
Sent: Saturday, September 29, 2018 12:33 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <***@nokia-sbell.com>
Cc: gluster-users <gluster-***@gluster.org>; Ravishankar N <***@redhat.com>
Subject: Re: sometimes entry remains in "gluster v heal vol-name info" until visit it from mnt

Hey,

Please provide the glustershd log from all the nodes and client logs on the node from where you did the lookup on the file to resolve this issue.

Regards,
Karthik

On Fri, Sep 28, 2018 at 5:27 PM Ravishankar N <***@redhat.com<mailto:***@redhat.com>> wrote:

+ gluster-users.

Adding Karthik to see if he has some cycles to look into this.

-Ravi

On 09/28/2018 12:07 PM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:
Hi, glusterfs expert
When I test with glusterfs version 3.12.3 I find it quite often that sometimes, there are entry remains in gluster volume heal <vol-name>info output for long time, it does not disappear until you visit it from the mount point, is this normal?


[***@sn-0:/root]
# gluster v heal services info
Brick sn-0.local:/mnt/bricks/services/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/services/brick
Status: Connected
Number of entries: 0

Brick sn-2.local:/mnt/bricks/services/brick
/fstest_88402c989256d6e39e50208c90c1e85d //this entry remains in the output until you touch /mnt/services/ fstest_88402c989256d6e39e50208c90c1e85d
Status: Connected
Number of entries: 1

[***@sn-0:/root]
# ssh sn-2.local
Warning: Permanently added 'sn-2.local' (RSA) to the list of known hosts.

USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL (fsclish) AND/OR LIMITED BASH SHELL.

Read /opt/nokia/share/security/readme_root.txt for more details.

[***@sn-2:/root]
# cd /mnt/bricks/services/brick/.glusterfs/indices/xattrop/
[***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>]
# ls
9138e315-efd6-46e0-8a3a-db535078c781 xattrop-dfcd7e67-8c2d-4ef1-93e2-c180073c8d87
[***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>]
# getfattr -m . -d -e hex /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/
getfattr: Removing leading '/' from absolute path names
# file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/
trusted.afr.services-client-1=0x000000000000000100000001
trusted.gfid=0x9138e315efd646e08a3adb535078c781
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

[***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>]
# getfattr -m . -d -e hex /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/
getfattr: Removing leading '/' from absolute path names
# file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/
trusted.gfid=0x0ccb5c1f96064e699f62fdc72cf036f5



“fstest_88402c989256d6e39e50208c90c1e85d” is only seen from sn-2 mount point and sn-2 service brick, there is no such entry if you ls /mnt/services on sn-0 or sn-1.
[***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>]
# cd /mnt/services/
[***@sn-2:/mnt/services]
# ls
backup db fstest_88402c989256d6e39e50208c90c1e85d LCM NE3SAgent _nokrcpautoremoteuser PM9 RCP_Backup SS_AlLightProcessor SymptomDataUpl
commoncollector EventCorrelationEngine hypertracer Log netserv ODS ptp rcpha SWM
[***@sn-2:/mnt/services]

[Zhou, Cynthia (NSB - CN/Hangzhou)]

On sn-0
[***@sn-0:/mnt/mstate]
# ls
as-0 as-2 cp-1 db-1 fstest_a3470921fb4be651d29221817856e4b5 fstest_f7e6762c626da53fbe55fd9a688ded0a mn-0 sn-0 sn-2
as-1 cp-0 db-0 fstest_3f3cea30d1f281ba68f2798873e8af50 fstest_c2f4ee8d4bfb939536b8e3c00b21212e _global mn-1 sn-1
[***@sn-0:/mnt/mstate]

On sn-1:
[***@sn-1:/mnt/mstate]
# ls
as-0 cp-0 db-1 fstest_49e9d4767e95a3eda0a599437ef6ce8f fstest_c2f4ee8d4bfb939536b8e3c00b21212e _global sn-0
as-1 cp-1 fstest_1c196bd015911c9f1a0042e433bcf7cc fstest_4fd4d74cb48a532926fba26db3d0d0d2 fstest_cf3482748d0ba4b7a2c669a9b5359ddc mn-0 sn-1
as-2 db-0 fstest_3f3cea30d1f281ba68f2798873e8af50 fstest_a3470921fb4be651d29221817856e4b5 fstest_f7e6762c626da53fbe55fd9a688ded0a mn-1 sn-2


you will not see fstest_4fd4d74cb48a532926fba26db3d0d0d2 until you visit /mnt/mstate/ fstest_4fd4d74cb48a532926fba26db3d0d0d2


From: Zhou, Cynthia (NSB - CN/Hangzhou)
Sent: Wednesday, October 10, 2018 2:33 PM
To: Karthik Subrahmanya <***@redhat.com>
Cc: gluster-users <gluster-***@gluster.org>; Ravishankar N <***@redhat.com>
Subject: RE: sometimes entry remains in "gluster v heal vol-name info" until visit it from mnt

Hi, I meet this issue again, from my env on sn-1 mstate volume brick dir there is additional fstest_49e9d4767e95a3eda0a599437ef6ce8f, but on other sn nodes(sn-0, sn-2) there is no such directory.


[***@sn-1:/mnt/bricks/mstate/brick]
# stat fstest_49e9d4767e95a3eda0a599437ef6ce8f
File: fstest_49e9d4767e95a3eda0a599437ef6ce8f
Size: 4096 Blocks: 16 IO Block: 4096 directory
Device: fd61h/64865d Inode: 45450 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 0/ root) Gid: ( 0/ root)
Access: 2018-10-10 06:51:09.045660991 +0300
Modify: 2018-10-10 06:51:09.045660991 +0300
Change: 2018-10-10 07:06:05.060278140 +0300
Birth: -
[***@sn-1:/mnt/bricks/mstate/brick]
# gluster v heal mstate info
Brick sn-0.local:/mnt/bricks/mstate/brick
Status: Connected
Number of entries: 0

Brick sn-2.local:/mnt/bricks/mstate/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/mstate/brick
/fstest_49e9d4767e95a3eda0a599437ef6ce8f
Status: Connected
Number of entries: 1

[***@sn-1:/mnt/bricks/mstate/brick]
# cd .glusterfs/indices/xattrop/
[***@sn-1:/mnt/bricks/mstate/brick/.glusterfs/indices/xattrop]
# ls
0c40b296-a4d6-46c9-b7a5-037105de84aa xattrop-90daa317-04db-4704-9f34-6f571eb3691c
[***@sn-1:/mnt/bricks/mstate/brick/.glusterfs/indices/xattrop]
# cd ../../..
[***@sn-1:/mnt/bricks/mstate/brick]
# getfattr -m . -d -e hex fstest_49e9d4767e95a3eda0a599437ef6ce8f
# file: fstest_49e9d4767e95a3eda0a599437ef6ce8f
trusted.afr.mstate-client-1=0x000000000000000100000001
trusted.gfid=0x0c40b296a4d646c9b7a5037105de84aa
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

[***@sn-1:/mnt/bricks/mstate/brick]
#
Enclosed is the glustershd log on sn-1,from which I see following log related to fstest_49e9d4767e95a3eda0a599437ef6ce8f

[2018-10-10 06:22:12.976487] I [MSGID: 108026] [afr-self-heald.c:341:afr_shd_selfheal] 0-mstate-replicate-0: entry: path /fstest_49e9d4767e95a3eda0a599437ef6ce8f, gfid: 0c40b296-a4d6-46c9-b7a5-037105de84aa //this is added in the entrance of function afr_shd_selfheal.




From: Karthik Subrahmanya <***@redhat.com<mailto:***@redhat.com>>
Sent: Saturday, September 29, 2018 12:33 AM
To: Zhou, Cynthia (NSB - CN/Hangzhou) <***@nokia-sbell.com<mailto:***@nokia-sbell.com>>
Cc: gluster-users <gluster-***@gluster.org<mailto:gluster-***@gluster.org>>; Ravishankar N <***@redhat.com<mailto:***@redhat.com>>
Subject: Re: sometimes entry remains in "gluster v heal vol-name info" until visit it from mnt

Hey,

Please provide the glustershd log from all the nodes and client logs on the node from where you did the lookup on the file to resolve this issue.

Regards,
Karthik

On Fri, Sep 28, 2018 at 5:27 PM Ravishankar N <***@redhat.com<mailto:***@redhat.com>> wrote:

+ gluster-users.

Adding Karthik to see if he has some cycles to look into this.

-Ravi

On 09/28/2018 12:07 PM, Zhou, Cynthia (NSB - CN/Hangzhou) wrote:
Hi, glusterfs expert
When I test with glusterfs version 3.12.3 I find it quite often that sometimes, there are entry remains in gluster volume heal <vol-name>info output for long time, it does not disappear until you visit it from the mount point, is this normal?


[***@sn-0:/root]
# gluster v heal services info
Brick sn-0.local:/mnt/bricks/services/brick
Status: Connected
Number of entries: 0

Brick sn-1.local:/mnt/bricks/services/brick
Status: Connected
Number of entries: 0

Brick sn-2.local:/mnt/bricks/services/brick
/fstest_88402c989256d6e39e50208c90c1e85d //this entry remains in the output until you touch /mnt/services/ fstest_88402c989256d6e39e50208c90c1e85d
Status: Connected
Number of entries: 1

[***@sn-0:/root]
# ssh sn-2.local
Warning: Permanently added 'sn-2.local' (RSA) to the list of known hosts.

USAGE OF THE ROOT ACCOUNT AND THE FULL BASH IS RECOMMENDED ONLY FOR LIMITED USE. PLEASE USE A NON-ROOT ACCOUNT AND THE SCLI SHELL (fsclish) AND/OR LIMITED BASH SHELL.

Read /opt/nokia/share/security/readme_root.txt for more details.

[***@sn-2:/root]
# cd /mnt/bricks/services/brick/.glusterfs/indices/xattrop/
[***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>]
# ls
9138e315-efd6-46e0-8a3a-db535078c781 xattrop-dfcd7e67-8c2d-4ef1-93e2-c180073c8d87
[***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>]
# getfattr -m . -d -e hex /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/
getfattr: Removing leading '/' from absolute path names
# file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/
trusted.afr.services-client-1=0x000000000000000100000001
trusted.gfid=0x9138e315efd646e08a3adb535078c781
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

[***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>]
# getfattr -m . -d -e hex /mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/
getfattr: Removing leading '/' from absolute path names
# file: mnt/bricks/services/brick/fstest_88402c989256d6e39e50208c90c1e85d/fstest_4cf1be62e0b12d3d65fac8eacb523ef3/
trusted.gfid=0x0ccb5c1f96064e699f62fdc72cf036f5



“fstest_88402c989256d6e39e50208c90c1e85d” is only seen from sn-2 mount point and sn-2 service brick, there is no such entry if you ls /mnt/services on sn-0 or sn-1.
[***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop<mailto:***@sn-2:/mnt/bricks/services/brick/.glusterfs/indices/xattrop>]
# cd /mnt/services/
[***@sn-2:/mnt/services]
# ls
backup db fstest_88402c989256d6e39e50208c90c1e85d LCM NE3SAgent _nokrcpautoremoteuser PM9 RCP_Backup SS_AlLightProcessor SymptomDataUpl
commoncollector EventCorrelationEngine hypertracer Log netserv ODS ptp rcpha SWM
[***@sn-2:/mnt/services]