[Gluster-users] Self-healing not healing 27k files on GlusterFS 4.1.5 3 nodes replica

[mabi]

Hello,

I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).

I already tried running a "volume heal" but none of the files got healed.

In the glfsheal log file for that particular volume the only error I see is a few of these entries:

[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152

and then a few of these warnings:

[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]

the glustershd.log file shows the following:

[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]

any idea what could be wrong here?

Regards,
Mabi

[mabi]

Is it possible the problem I encounter described below is caused by the following bug, introduced in 4.1.5:

Bug 1637953 - data-self-heal in arbiter volume results in stale locks.
https://bugzilla.redhat.com/show_bug.cgi?id=1637953

Regards,
M.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi

[mabi]

I tried again to manually run a heal by using the "gluster volume heal" command because still not files have been healed and noticed the following warning in the glusterd.log file:

[2018-11-02 18:04:19.454702] I [MSGID: 106533] [glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume myvol-private
[2018-11-02 18:04:19.457311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0

It looks like the glustershd can't connect to "host:(null)", could that be the reason why there is no healing taking place? if yes why do I see here "host:(null)"? and what needs fixing?

This seeem to have happened since I upgraded from 3.12.14 to 4.1.5.

I really would appreciate some help here, I suspect being an issue with GlusterFS 4.1.5.

Thank you in advance for any feedback.


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi

[Ravishankar N]

Mabi,

If bug 1637953 is what you are experiencing, then you need to follow the
workarounds mentioned in
https://lists.gluster.org/pipermail/gluster-users/2018-October/035178.html.
Can you see if this works?

-Ravi

Post by mabi
[2018-11-02 18:04:19.454702] I [MSGID: 106533] [glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume myvol-private
[2018-11-02 18:04:19.457311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0
It looks like the glustershd can't connect to "host:(null)", could that be the reason why there is no healing taking place? if yes why do I see here "host:(null)"? and what needs fixing?
This seeem to have happened since I upgraded from 3.12.14 to 4.1.5.
I really would appreciate some help here, I suspect being an issue with GlusterFS 4.1.5.
Thank you in advance for any feedback.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi

_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

[mabi]

Ravi, I actually restarted glustershd by unmounting my volume on the clients, stopping and starting the volume on the cluster and re-mounting it on the clients yesterday evening and it managed to get around 1500~ files cleared from the "volume heal info" output. So I am down now to around ~25k more files to heal. While restarting the volume I saw the following log entries in the brick log file:

[2018-11-02 17:51:07.078738] W [inodelk.c:610:pl_inodelk_log_cleanup] 0-myvol-private-server: releasing lock on da4f31fb-ac53-4d78-a633-f0046ac3ebcc held by {client=0x7fd48400c160, pid=-6 lk-owner=b0d405e0167f0000}


What also bothers me is that if I run a manual "volume heal" nothing happens except the following log entry in glusterd log:

[2018-11-03 06:32:16.033214] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0

That does not seem normal... what do you think?


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Mabi,
If bug 1637953 is what you are experiencing, then you need to follow the
workarounds mentioned in
https://lists.gluster.org/pipermail/gluster-users/2018-October/035178.html.
Can you see if this works?
-Ravi

Post by mabi
[2018-11-02 18:04:19.454702] I [MSGID: 106533] [glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume myvol-private
[2018-11-02 18:04:19.457311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0
It looks like the glustershd can't connect to "host:(null)", could that be the reason why there is no healing taking place? if yes why do I see here "host:(null)"? and what needs fixing?
This seeem to have happened since I upgraded from 3.12.14 to 4.1.5.
I really would appreciate some help here, I suspect being an issue with GlusterFS 4.1.5.
Thank you in advance for any feedback.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi

Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

[mabi]

Ravi (or anyone else who can help), I now have even more files which are pending for healing. Here is the output of a "volume heal info summary":

Brick node1:/data/myvol-private/brick
Status: Connected
Total Number of entries: 49845
Number of entries in heal pending: 49845
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick node2:/data/myvol-private/brick
Status: Connected
Total Number of entries: 26644
Number of entries in heal pending: 26644
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Brick node3:/srv/glusterfs/myvol-private/brick
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0

Should I try to set the "cluster.data-self-heal" parameter of that volume to "off" as mentioned in the bug?

And by doing that, does it mean that my files pending heal are in danger of being lost?

Also is it dangerous to leave "cluster.data-self-heal" to off?



‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Mabi,
If bug 1637953 is what you are experiencing, then you need to follow the
workarounds mentioned in
https://lists.gluster.org/pipermail/gluster-users/2018-October/035178.html.
Can you see if this works?
-Ravi

Post by mabi
[2018-11-02 18:04:19.454702] I [MSGID: 106533] [glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume myvol-private
[2018-11-02 18:04:19.457311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0
It looks like the glustershd can't connect to "host:(null)", could that be the reason why there is no healing taking place? if yes why do I see here "host:(null)"? and what needs fixing?
This seeem to have happened since I upgraded from 3.12.14 to 4.1.5.
I really would appreciate some help here, I suspect being an issue with GlusterFS 4.1.5.
Thank you in advance for any feedback.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi

Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

[Ravishankar N]

Post by mabi
Ravi (or anyone else who can help), I now have even more files which are pending for healing.

If the count is increasing, there is likely a network (disconnect)
problem between the gluster clients and the bricks that needs fixing.

Post by mabi
Brick node1:/data/myvol-private/brick
Status: Connected
Total Number of entries: 49845
Number of entries in heal pending: 49845
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node2:/data/myvol-private/brick
Status: Connected
Total Number of entries: 26644
Number of entries in heal pending: 26644
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node3:/srv/glusterfs/myvol-private/brick
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Should I try to set the "cluster.data-self-heal" parameter of that volume to "off" as mentioned in the bug?

Yes, as  mentioned in the workaround in the thread that I shared.

Post by mabi
And by doing that, does it mean that my files pending heal are in danger of being lost?

No.

Post by mabi
Also is it dangerous to leave "cluster.data-self-heal" to off?

No. This is only disabling client side data healing. Self-heal daemon
would still heal the files.
-Ravi

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Mabi,
If bug 1637953 is what you are experiencing, then you need to follow the
workarounds mentioned in
https://lists.gluster.org/pipermail/gluster-users/2018-October/035178.html.
Can you see if this works?
-Ravi

Post by mabi
[2018-11-02 18:04:19.454702] I [MSGID: 106533] [glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume myvol-private
[2018-11-02 18:04:19.457311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0
It looks like the glustershd can't connect to "host:(null)", could that be the reason why there is no healing taking place? if yes why do I see here "host:(null)"? and what needs fixing?
This seeem to have happened since I upgraded from 3.12.14 to 4.1.5.
I really would appreciate some help here, I suspect being an issue with GlusterFS 4.1.5.
Thank you in advance for any feedback.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi

Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

[mabi]

Ravi, I did not yet modify the cluster.data-self-heal parameter to off because in the mean time node2 of my cluster had a memory shortage (this node has 32 GB of RAM) and as such I had to reboot it. After that reboot all locks got released and there are no more files left to heal on that volume. So the reboot of node2 did the trick (but this still seems to be a bug).

Now on another volume of this same cluster I have a total of 8 files (4 of them being directories) unsynced from node1 and node3 (arbiter) as you can see below:

Brick node1:/data/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
<gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360>
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/le_dir
<gfid:aae4098a-1a71-4155-9cc9-e564b89957cf>
Status: Connected
Number of entries: 4

Brick node2:/data/myvol-pro/brick
Status: Connected
Number of entries: 0

Brick node3:/srv/glusterfs/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/le_dir
<gfid:aae4098a-1a71-4155-9cc9-e564b89957cf>
<gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360>
Status: Connected
Number of entries: 4

If I check the "/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/" with an "ls -l" directory on the client (gluster fuse mount) I get the following garbage:

drwxr-xr-x 4 www-data www-data 4096 Nov 5 14:19 .
drwxr-xr-x 31 www-data www-data 4096 Nov 5 14:23 ..
d????????? ? ? ? ? ? le_dir

I checked on the nodes and indeed node1 and node3 have the same directory from the time 14:19 but node2 has a directory from the time 14:12.

Again here the self-heal daemon doesn't seem to be doing anything... What do you recommend me to do in order to heal these unsynced files?



‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N

Post by mabi
Ravi (or anyone else who can help), I now have even more files which are pending for healing.

If the count is increasing, there is likely a network (disconnect)
problem between the gluster clients and the bricks that needs fixing.

Post by mabi
Brick node1:/data/myvol-private/brick
Status: Connected
Total Number of entries: 49845
Number of entries in heal pending: 49845
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node2:/data/myvol-private/brick
Status: Connected
Total Number of entries: 26644
Number of entries in heal pending: 26644
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node3:/srv/glusterfs/myvol-private/brick
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Should I try to set the "cluster.data-self-heal" parameter of that volume to "off" as mentioned in the bug?

Yes, as  mentioned in the workaround in the thread that I shared.

Post by mabi
And by doing that, does it mean that my files pending heal are in danger of being lost?

No.

Post by mabi
Also is it dangerous to leave "cluster.data-self-heal" to off?

No. This is only disabling client side data healing. Self-heal daemon
would still heal the files.
-Ravi

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Mabi,
If bug 1637953 is what you are experiencing, then you need to follow the
workarounds mentioned in
https://lists.gluster.org/pipermail/gluster-users/2018-October/035178.html.
Can you see if this works?
-Ravi

Post by mabi
[2018-11-02 18:04:19.454702] I [MSGID: 106533] [glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume myvol-private
[2018-11-02 18:04:19.457311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0
It looks like the glustershd can't connect to "host:(null)", could that be the reason why there is no healing taking place? if yes why do I see here "host:(null)"? and what needs fixing?
This seeem to have happened since I upgraded from 3.12.14 to 4.1.5.
I really would appreciate some help here, I suspect being an issue with GlusterFS 4.1.5.
Thank you in advance for any feedback.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

[mabi]

To my eyes this specific case looks like a split-brain scenario but the output of "volume info split-brain" does not show any files. Should I still use the process for split-brain files as documented in the glusterfs documentation? or what do you recommend here?


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Ravi, I did not yet modify the cluster.data-self-heal parameter to off because in the mean time node2 of my cluster had a memory shortage (this node has 32 GB of RAM) and as such I had to reboot it. After that reboot all locks got released and there are no more files left to heal on that volume. So the reboot of node2 did the trick (but this still seems to be a bug).
Brick node1:/data/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/le_dir
gfid:aae4098a-1a71-4155-9cc9-e564b89957cf
Status: Connected
Number of entries: 4
Brick node2:/data/myvol-pro/brick
Status: Connected
Number of entries: 0
Brick node3:/srv/glusterfs/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/le_dir
gfid:aae4098a-1a71-4155-9cc9-e564b89957cf
gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360
Status: Connected
Number of entries: 4
drwxr-xr-x 4 www-data www-data 4096 Nov 5 14:19 .
drwxr-xr-x 31 www-data www-data 4096 Nov 5 14:23 ..
d????????? ? ? ? ? ? le_dir
I checked on the nodes and indeed node1 and node3 have the same directory from the time 14:19 but node2 has a directory from the time 14:12.
Again here the self-heal daemon doesn't seem to be doing anything... What do you recommend me to do in order to heal these unsynced files?
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N

Post by mabi
Ravi (or anyone else who can help), I now have even more files which are pending for healing.

If the count is increasing, there is likely a network (disconnect)
problem between the gluster clients and the bricks that needs fixing.

Post by mabi
Brick node1:/data/myvol-private/brick
Status: Connected
Total Number of entries: 49845
Number of entries in heal pending: 49845
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node2:/data/myvol-private/brick
Status: Connected
Total Number of entries: 26644
Number of entries in heal pending: 26644
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node3:/srv/glusterfs/myvol-private/brick
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Should I try to set the "cluster.data-self-heal" parameter of that volume to "off" as mentioned in the bug?

Yes, as  mentioned in the workaround in the thread that I shared.

Post by mabi
And by doing that, does it mean that my files pending heal are in danger of being lost?

No.

Post by mabi
Also is it dangerous to leave "cluster.data-self-heal" to off?

No. This is only disabling client side data healing. Self-heal daemon
would still heal the files.
-Ravi

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Mabi,
If bug 1637953 is what you are experiencing, then you need to follow the
workarounds mentioned in
https://lists.gluster.org/pipermail/gluster-users/2018-October/035178.html.
Can you see if this works?
-Ravi

Post by mabi
[2018-11-02 18:04:19.454702] I [MSGID: 106533] [glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume myvol-private
[2018-11-02 18:04:19.457311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0
It looks like the glustershd can't connect to "host:(null)", could that be the reason why there is no healing taking place? if yes why do I see here "host:(null)"? and what needs fixing?
This seeem to have happened since I upgraded from 3.12.14 to 4.1.5.
I really would appreciate some help here, I suspect being an issue with GlusterFS 4.1.5.
Thank you in advance for any feedback.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

[Ravishankar N]

Can you share the getfattr output of all 4 entries from all 3 bricks?

Also, can you tailf glustershd.log on all nodes and see if anything is
logged for these entries when you run 'gluster volume heal $volname'?

Regards,

Ravi

Post by mabi
To my eyes this specific case looks like a split-brain scenario but the output of "volume info split-brain" does not show any files. Should I still use the process for split-brain files as documented in the glusterfs documentation? or what do you recommend here?
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Ravi, I did not yet modify the cluster.data-self-heal parameter to off because in the mean time node2 of my cluster had a memory shortage (this node has 32 GB of RAM) and as such I had to reboot it. After that reboot all locks got released and there are no more files left to heal on that volume. So the reboot of node2 did the trick (but this still seems to be a bug).
Brick node1:/data/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/le_dir
gfid:aae4098a-1a71-4155-9cc9-e564b89957cf
Status: Connected
Number of entries: 4
Brick node2:/data/myvol-pro/brick
Status: Connected
Number of entries: 0
Brick node3:/srv/glusterfs/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/le_dir
gfid:aae4098a-1a71-4155-9cc9-e564b89957cf
gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360
Status: Connected
Number of entries: 4
drwxr-xr-x 4 www-data www-data 4096 Nov 5 14:19 .
drwxr-xr-x 31 www-data www-data 4096 Nov 5 14:23 ..
d????????? ? ? ? ? ? le_dir
I checked on the nodes and indeed node1 and node3 have the same directory from the time 14:19 but node2 has a directory from the time 14:12.
Again here the self-heal daemon doesn't seem to be doing anything... What do you recommend me to do in order to heal these unsynced files?
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N

Post by mabi
Ravi (or anyone else who can help), I now have even more files which are pending for healing.

If the count is increasing, there is likely a network (disconnect)
problem between the gluster clients and the bricks that needs fixing.

Post by mabi
Brick node1:/data/myvol-private/brick
Status: Connected
Total Number of entries: 49845
Number of entries in heal pending: 49845
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node2:/data/myvol-private/brick
Status: Connected
Total Number of entries: 26644
Number of entries in heal pending: 26644
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node3:/srv/glusterfs/myvol-private/brick
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Should I try to set the "cluster.data-self-heal" parameter of that volume to "off" as mentioned in the bug?

Yes, as  mentioned in the workaround in the thread that I shared.

Post by mabi
And by doing that, does it mean that my files pending heal are in danger of being lost?

No.

Post by mabi
Also is it dangerous to leave "cluster.data-self-heal" to off?

No. This is only disabling client side data healing. Self-heal daemon
would still heal the files.
-Ravi

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Mabi,
If bug 1637953 is what you are experiencing, then you need to follow the
workarounds mentioned in
https://lists.gluster.org/pipermail/gluster-users/2018-October/035178.html.
Can you see if this works?
-Ravi

Post by mabi
[2018-11-02 18:04:19.454702] I [MSGID: 106533] [glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume myvol-private
[2018-11-02 18:04:19.457311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0
It looks like the glustershd can't connect to "host:(null)", could that be the reason why there is no healing taking place? if yes why do I see here "host:(null)"? and what needs fixing?
This seeem to have happened since I upgraded from 3.12.14 to 4.1.5.
I really would appreciate some help here, I suspect being an issue with GlusterFS 4.1.5.
Thank you in advance for any feedback.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

[mabi]

Dear Ravi,

Thank you for your answer. I will start first by sending you below the getfattr from the first entry which does not get healed (it is in fact a directory). It is the following path/dir from the output of one of my previous mails: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir

# NODE 1
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000000000000300000003
trusted.gfid=0x25e2616b4fb64b2a89451afc956fff19
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

# NODE 2
trusted.gfid=0xd9ac192ce85e4402af105551f587ed9a
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

# NODE 3 (arbiter)
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000000000000300000003
trusted.gfid=0x25e2616b4fb64b2a89451afc956fff19
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

Notice here that node 2 does not seem to have any AFR attributes which must be problematic. Also that specific directory on my node 2 has the oldest timestamp (14:12) where as that same directory on node 1 and 3 have 14:19 as timestamp.

I did run "volume heal myvol-pro" and on the console it shows:

Launching heal operation to perform index self heal on volume myvol-pro has been successful
Use heal info commands to check status.

but then in the glustershd.log file of both 3 nodes there has been nothing new logged.

The log file cmd_history.log shows:
[2018-11-08 07:20:24.481603] : volume heal myvol-pro : SUCCESS

and glusterd.log:
[2018-11-08 07:20:24.474032] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0

That's it... To me it looks like a split-brain but GlusterFS does not report it as split-brain and neither does any self-heal on it.

What do you think?

Regards,
M.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Can you share the getfattr output of all 4 entries from all 3 bricks?
Also, can you tailf glustershd.log on all nodes and see if anything is
logged for these entries when you run 'gluster volume heal $volname'?
Regards,
Ravi

Post by mabi
To my eyes this specific case looks like a split-brain scenario but the output of "volume info split-brain" does not show any files. Should I still use the process for split-brain files as documented in the glusterfs documentation? or what do you recommend here?
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Ravi, I did not yet modify the cluster.data-self-heal parameter to off because in the mean time node2 of my cluster had a memory shortage (this node has 32 GB of RAM) and as such I had to reboot it. After that reboot all locks got released and there are no more files left to heal on that volume. So the reboot of node2 did the trick (but this still seems to be a bug).
Brick node1:/data/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/le_dir
gfid:aae4098a-1a71-4155-9cc9-e564b89957cf
Status: Connected
Number of entries: 4
Brick node2:/data/myvol-pro/brick
Status: Connected
Number of entries: 0
Brick node3:/srv/glusterfs/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/le_dir
gfid:aae4098a-1a71-4155-9cc9-e564b89957cf
gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360
Status: Connected
Number of entries: 4
drwxr-xr-x 4 www-data www-data 4096 Nov 5 14:19 .
drwxr-xr-x 31 www-data www-data 4096 Nov 5 14:23 ..
d????????? ? ? ? ? ? le_dir
I checked on the nodes and indeed node1 and node3 have the same directory from the time 14:19 but node2 has a directory from the time 14:12.
Again here the self-heal daemon doesn't seem to be doing anything... What do you recommend me to do in order to heal these unsynced files?
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Ravi (or anyone else who can help), I now have even more files which are pending for healing.
If the count is increasing, there is likely a network (disconnect)
problem between the gluster clients and the bricks that needs fixing.
Brick node1:/data/myvol-private/brick
Status: Connected
Total Number of entries: 49845
Number of entries in heal pending: 49845
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node2:/data/myvol-private/brick
Status: Connected
Total Number of entries: 26644
Number of entries in heal pending: 26644
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node3:/srv/glusterfs/myvol-private/brick
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Should I try to set the "cluster.data-self-heal" parameter of that volume to "off" as mentioned in the bug?
Yes, as  mentioned in the workaround in the thread that I shared.
And by doing that, does it mean that my files pending heal are in danger of being lost?
No.
Also is it dangerous to leave "cluster.data-self-heal" to off?
No. This is only disabling client side data healing. Self-heal daemon
would still heal the files.
-Ravi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Mabi,
If bug 1637953 is what you are experiencing, then you need to follow the
workarounds mentioned in
https://lists.gluster.org/pipermail/gluster-users/2018-October/035178.html.
Can you see if this works?
-Ravi

Post by mabi
[2018-11-02 18:04:19.454702] I [MSGID: 106533] [glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume myvol-private
[2018-11-02 18:04:19.457311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0
It looks like the glustershd can't connect to "host:(null)", could that be the reason why there is no healing taking place? if yes why do I see here "host:(null)"? and what needs fixing?
This seeem to have happened since I upgraded from 3.12.14 to 4.1.5.
I really would appreciate some help here, I suspect being an issue with GlusterFS 4.1.5.
Thank you in advance for any feedback.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

[Ravishankar N]

Post by mabi
Dear Ravi,
Thank you for your answer. I will start first by sending you below the getfattr from the first entry which does not get healed (it is in fact a directory). It is the following path/dir from the output of one of my previous mails: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
# NODE 1
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000000000000300000003
trusted.gfid=0x25e2616b4fb64b2a89451afc956fff19
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
# NODE 2
trusted.gfid=0xd9ac192ce85e4402af105551f587ed9a
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
# NODE 3 (arbiter)
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000000000000300000003
trusted.gfid=0x25e2616b4fb64b2a89451afc956fff19
trusted.glusterfs.dht=0x000000010000000000000000ffffffff
Notice here that node 2 does not seem to have any AFR attributes which must be problematic. Also that specific directory on my node 2 has the oldest timestamp (14:12) where as that same directory on node 1 and 3 have 14:19 as timestamp.
Launching heal operation to perform index self heal on volume myvol-pro has been successful
Use heal info commands to check status.
but then in the glustershd.log file of both 3 nodes there has been nothing new logged.
[2018-11-08 07:20:24.481603] : volume heal myvol-pro : SUCCESS
[2018-11-08 07:20:24.474032] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0

Not sure of this part.

Post by mabi
That's it... To me it looks like a split-brain but GlusterFS does not report it as split-brain and neither does any self-heal on it.

It is not a split-brain. Nodes 1 and 3 have xattrs indicating a pending
entry heal on node2 , so heal must have happened ideally. Can you check
a few things?
- Is there any disconnects between each of the shds and the brick
processes (check via statedump or look for disconnect messages in
glustershd.log). Does restarting shd via a `volume start force` solve
the problem?
- Is the symlink pointing to oc_dir present inside .glusterfs/25/e2 in
all 3 bricks?

Regards,
Ravi

Post by mabi
What do you think?
Regards,
M.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Can you share the getfattr output of all 4 entries from all 3 bricks?
Also, can you tailf glustershd.log on all nodes and see if anything is
logged for these entries when you run 'gluster volume heal $volname'?
Regards,
Ravi

Post by mabi
To my eyes this specific case looks like a split-brain scenario but the output of "volume info split-brain" does not show any files. Should I still use the process for split-brain files as documented in the glusterfs documentation? or what do you recommend here?
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Ravi, I did not yet modify the cluster.data-self-heal parameter to off because in the mean time node2 of my cluster had a memory shortage (this node has 32 GB of RAM) and as such I had to reboot it. After that reboot all locks got released and there are no more files left to heal on that volume. So the reboot of node2 did the trick (but this still seems to be a bug).
Brick node1:/data/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/le_dir
gfid:aae4098a-1a71-4155-9cc9-e564b89957cf
Status: Connected
Number of entries: 4
Brick node2:/data/myvol-pro/brick
Status: Connected
Number of entries: 0
Brick node3:/srv/glusterfs/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/le_dir
gfid:aae4098a-1a71-4155-9cc9-e564b89957cf
gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360
Status: Connected
Number of entries: 4
drwxr-xr-x 4 www-data www-data 4096 Nov 5 14:19 .
drwxr-xr-x 31 www-data www-data 4096 Nov 5 14:23 ..
d????????? ? ? ? ? ? le_dir
I checked on the nodes and indeed node1 and node3 have the same directory from the time 14:19 but node2 has a directory from the time 14:12.
Again here the self-heal daemon doesn't seem to be doing anything... What do you recommend me to do in order to heal these unsynced files?
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Ravi (or anyone else who can help), I now have even more files which are pending for healing.
If the count is increasing, there is likely a network (disconnect)
problem between the gluster clients and the bricks that needs fixing.
Brick node1:/data/myvol-private/brick
Status: Connected
Total Number of entries: 49845
Number of entries in heal pending: 49845
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node2:/data/myvol-private/brick
Status: Connected
Total Number of entries: 26644
Number of entries in heal pending: 26644
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Brick node3:/srv/glusterfs/myvol-private/brick
Status: Connected
Total Number of entries: 0
Number of entries in heal pending: 0
Number of entries in split-brain: 0
Number of entries possibly healing: 0
Should I try to set the "cluster.data-self-heal" parameter of that volume to "off" as mentioned in the bug?
Yes, as  mentioned in the workaround in the thread that I shared.
And by doing that, does it mean that my files pending heal are in danger of being lost?
No.
Also is it dangerous to leave "cluster.data-self-heal" to off?
No. This is only disabling client side data healing. Self-heal daemon
would still heal the files.
-Ravi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Mabi,
If bug 1637953 is what you are experiencing, then you need to follow the
workarounds mentioned in
https://lists.gluster.org/pipermail/gluster-users/2018-October/035178.html.
Can you see if this works?
-Ravi

Post by mabi
[2018-11-02 18:04:19.454702] I [MSGID: 106533] [glusterd-volume-ops.c:938:__glusterd_handle_cli_heal_volume] 0-management: Received heal vol req for volume myvol-private
[2018-11-02 18:04:19.457311] W [rpc-clnt.c:1753:rpc_clnt_submit] 0-glustershd: error returned while attempting to connect to host:(null), port:0
It looks like the glustershd can't connect to "host:(null)", could that be the reason why there is no healing taking place? if yes why do I see here "host:(null)"? and what needs fixing?
This seeem to have happened since I upgraded from 3.12.14 to 4.1.5.
I really would appreciate some help here, I suspect being an issue with GlusterFS 4.1.5.
Thank you in advance for any feedback.
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by mabi
Hello,
I have a GlusterFS 4.1.5 cluster with 3 nodes (including 1 arbiter) and currently have a volume with around 27174 files which are not being healed. The "volume heal info" command shows the same 27k files under the first node and the second node but there is nothing under the 3rd node (arbiter).
I already tried running a "volume heal" but none of the files got healed.
[2018-10-31 10:06:41.524300] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0x108b sent = 2018-10-31 09:36:41.314203. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:08:12.161498] W [dict.c:671:dict_ref] (-->/usr/lib/x86_64-linux-gnu/glusterfs/4.1.5/xlator/cluster/replicate.so(+0x6734a) [0x7f2a6dff434a] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(+0x5da84) [0x7f2a798e8a84] -->/usr/lib/x86_64-linux-gnu/libglusterfs.so.0(dict_ref+0x58) [0x7f2a798a37f8] ) 0-dict: dict is NULL [Invalid argument]
[2018-10-31 10:10:52.502453] E [rpc-clnt.c:184:call_bail] 0-myvol-private-client-0: bailing out frame type(GlusterFS 4.x v1) op(INODELK(29)) xid = 0xaa398 sent = 2018-10-31 09:40:50.927816. timeout = 1800 for 127.0.1.1:49152
[2018-10-31 10:10:52.502502] E [MSGID: 114031] [client-rpc-fops_v2.c:1306:client4_0_inodelk_cbk] 0-myvol-private-client-0: remote operation failed [Transport endpoint is not connected]
any idea what could be wrong here?
Regards,
Mabi
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

[mabi]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
It is not a split-brain. Nodes 1 and 3 have xattrs indicating a pending
entry heal on node2 , so heal must have happened ideally. Can you check
a few things?
- Is there any disconnects between each of the shds and the brick
processes (check via statedump or look for disconnect messages in
glustershd.log). Does restarting shd via a `volume start force` solve
the problem?

Yes there is one disconnect at 14:21 (UTC 13:21) because node2 ran out of memory (although it has 32 GB of RAM) and I had to reboot it. Here are the relevant log entries taken from glustershd.log on node1:

[2018-11-05 13:21:16.284239] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-myvol-pro-client-1: server 192.168.10.33:49154 has not responded in the last 42 seconds, disconnecting.
[2018-11-05 13:21:16.284385] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-myvol-pro-client-1: disconnected from myvol-pro-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-11-05 13:21:16.284889] W [rpc-clnt-ping.c:222:rpc_clnt_ping_cbk] 0-myvol-pro-client-1: socket disconnected

I also just ran a "volume start force" and saw that the glustershd processes got restarted on all 3 nodes but that did not trigger any healing. There are still the same amount of files/dirs pending heal...

Post by Ravishankar N
- Is the symlink pointing to oc_dir present inside .glusterfs/25/e2 in
all 3 bricks?

They are yes for node1 and node3 but node2 there is no such symlink...

I hope that helps to debug the issue further, else please let me know if you need more info

[Ravishankar N]

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
It is not a split-brain. Nodes 1 and 3 have xattrs indicating a pending
entry heal on node2 , so heal must have happened ideally. Can you check
a few things?
- Is there any disconnects between each of the shds and the brick
processes (check via statedump or look for disconnect messages in
glustershd.log). Does restarting shd via a `volume start force` solve
the problem?

[2018-11-05 13:21:16.284239] C [rpc-clnt-ping.c:166:rpc_clnt_ping_timer_expired] 0-myvol-pro-client-1: server 192.168.10.33:49154 has not responded in the last 42 seconds, disconnecting.
[2018-11-05 13:21:16.284385] I [MSGID: 114018] [client.c:2254:client_rpc_notify] 0-myvol-pro-client-1: disconnected from myvol-pro-client-1. Client process will keep trying to connect to glusterd until brick's port is available
[2018-11-05 13:21:16.284889] W [rpc-clnt-ping.c:222:rpc_clnt_ping_cbk] 0-myvol-pro-client-1: socket disconnected
I also just ran a "volume start force" and saw that the glustershd processes got restarted on all 3 nodes but that did not trigger any healing. There are still the same amount of files/dirs pending heal...

Post by Ravishankar N
- Is the symlink pointing to oc_dir present inside .glusterfs/25/e2 in
all 3 bricks?

They are yes for node1 and node3 but node2 there is no such symlink...
I hope that helps to debug the issue further, else please let me know if you need more info

Please re-create the symlink on node 2 to match how it is in the other
nodes and launch heal again. Check if this is the case for other entries
too.
-Ravi

[mabi]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Please re-create the symlink on node 2 to match how it is in the other
nodes and launch heal again. Check if this is the case for other entries
too.
-Ravi

I can't create the missing symlink on node2 because the target (../../70/c8/70c894ca-422b-4bce-acf1-5cfb4669abbd/oc_dir) of that link does not exist. So basically the symlink and the target of that symlink are missing.

Or shall I create a symlink to a non-existing target?

[mabi]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Please re-create the symlink on node 2 to match how it is in the other
nodes and launch heal again. Check if this is the case for other entries
too.
-Ravi

Please ignore my previous mail, I was looking for a symlink with the GFID of node1 or node 3 on my node2 whereas I should have been looking with the GFID of node2 of course. I have now found the symlink on node2 pointing to that problematic directory and it looks like this:

node2# cd /data/myvol-pro/brick/.glusterfs/d9/ac
node2# ls -la | grep d9ac19
lrwxrwxrwx 1 root root 66 Nov 5 14:12 d9ac192c-e85e-4402-af10-5551f587ed9a -> ../../10/ec/10ec1eb1-c854-4ff2-a36c-325681713093/oc_dir

When you say "re-create the symlink", do you mean I should delete the current symlink on node2 (d9ac192c-e85e-4402-af10-5551f587ed9a) and re-create it with the GFID which is used on my node 1 and node 3 like this?

node2# cd /data/myvol-pro/brick/.glusterfs/d9/ac
node2# rm d9ac192c-e85e-4402-af10-5551f587ed9a
node2# cd /data/myvol-pro/brick/.glusterfs/25/e2
node2# ln -s ../../10/ec/10ec1eb1-c854-4ff2-a36c-325681713093/oc_dir 25e2616b-4fb6-4b2a-8945-1afc956fff19

Just want to make sure I understood you correctly before doing that. Could you please let me know if this is correct?

Thanks

[Ravishankar N]

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Please re-create the symlink on node 2 to match how it is in the other
nodes and launch heal again. Check if this is the case for other entries
too.
-Ravi

node2# cd /data/myvol-pro/brick/.glusterfs/d9/ac
node2# ls -la | grep d9ac19
lrwxrwxrwx 1 root root 66 Nov 5 14:12 d9ac192c-e85e-4402-af10-5551f587ed9a -> ../../10/ec/10ec1eb1-c854-4ff2-a36c-325681713093/oc_dir
When you say "re-create the symlink", do you mean I should delete the current symlink on node2 (d9ac192c-e85e-4402-af10-5551f587ed9a) and re-create it with the GFID which is used on my node 1 and node 3 like this?

I thought it was missing which is why I asked you to create it.  The
trusted.gfid xattr for any given file or directory must be same in all 3
bricks.  But it looks like that isn't the case. Are the gfids and the
symlinks for all the dirs leading to the parent dir of oc_dir same on
all nodes? (i.e evey directory in
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/)?

Post by mabi
node2# cd /data/myvol-pro/brick/.glusterfs/d9/ac
node2# rm d9ac192c-e85e-4402-af10-5551f587ed9a
node2# cd /data/myvol-pro/brick/.glusterfs/25/e2
node2# ln -s ../../10/ec/10ec1eb1-c854-4ff2-a36c-325681713093/oc_dir 25e2616b-4fb6-4b2a-8945-1afc956fff19
Just want to make sure I understood you correctly before doing that. Could you please let me know if this is correct?

Let us see if the parents' gfids are the same before deleting anything.
Is the heal info still showing 4 entries? Please also share the getfattr
output of the the parent directory (i.e. dir11) .
Thanks,
Ravi

Post by mabi
Thanks

[mabi]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
I thought it was missing which is why I asked you to create it.  The
trusted.gfid xattr for any given file or directory must be same in all 3
bricks.  But it looks like that isn't the case. Are the gfids and the
symlinks for all the dirs leading to the parent dir of oc_dir same on
all nodes? (i.e evey directory in
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/)?

I now checked the GFIDs of all directories leading back down to the parent dir (13 directories in total) and for node 1 and node 3 the GIFDs of all underlying directories match each other. On node 2 they are also all the same except for the two highest directories (".../dir11" and and ".../dir11/oc_dir"). It's exactly these two directories which are also listed in the "volume heal info" output under node 1 and node 2 and which do not get healed.

For your reference I have pasted below the GFIDs for all underlying directories up to the parent directory and for all 3 nodes. I start at the top with the highest directory and at the bottom of the list is the parent directory (/data).

# NODE 1

trusted.gfid=0x25e2616b4fb64b2a89451afc956fff19 # /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
trusted.gfid=0x70c894ca422b4bceacf15cfb4669abbd # /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11
trusted.gfid=0x7d7d2165f4804edf8c93de01c8768269 # ...
trusted.gfid=0xdbc0bfa0a052405ca3fad2d1ca137f82
trusted.gfid=0xbb75051c24ba4c119351bef938c55ad4
trusted.gfid=0x0002ad0c3fbe4806a75f8e68304f5b94
trusted.gfid=0xf120657977274247900db4e9cc8129dd
trusted.gfid=0x8afeb00bb1e74cbab932acea705b7dd9
trusted.gfid=0x2174086880fc4fd19b187d1384300add
trusted.gfid=0x2057e87cf4cc43f9bbad160cbec43d01 # ...
trusted.gfid=0xa7d78519db61459399e01fad2badf3fb # /data/dir1/dir2
trusted.gfid=0xfaa0ed7ccaf84f6c8bdb20a7f657c4b4 # /data/dir1
trusted.gfid=0x2683990126724adbb6416b911180e62b # /data

# NODE 2

trusted.gfid=0xd9ac192ce85e4402af105551f587ed9a
trusted.gfid=0x10ec1eb1c8544ff2a36c325681713093
trusted.gfid=0x7d7d2165f4804edf8c93de01c8768269
trusted.gfid=0xdbc0bfa0a052405ca3fad2d1ca137f82
trusted.gfid=0xbb75051c24ba4c119351bef938c55ad4
trusted.gfid=0x0002ad0c3fbe4806a75f8e68304f5b94
trusted.gfid=0xf120657977274247900db4e9cc8129dd
trusted.gfid=0x8afeb00bb1e74cbab932acea705b7dd9
trusted.gfid=0x2174086880fc4fd19b187d1384300add
trusted.gfid=0x2057e87cf4cc43f9bbad160cbec43d01
trusted.gfid=0xa7d78519db61459399e01fad2badf3fb
trusted.gfid=0xfaa0ed7ccaf84f6c8bdb20a7f657c4b4
trusted.gfid=0x2683990126724adbb6416b911180e62b

# NODE 3

trusted.gfid=0x25e2616b4fb64b2a89451afc956fff19
trusted.gfid=0x70c894ca422b4bceacf15cfb4669abbd
trusted.gfid=0x7d7d2165f4804edf8c93de01c8768269
trusted.gfid=0xdbc0bfa0a052405ca3fad2d1ca137f82
trusted.gfid=0xbb75051c24ba4c119351bef938c55ad4
trusted.gfid=0x0002ad0c3fbe4806a75f8e68304f5b94
trusted.gfid=0xf120657977274247900db4e9cc8129dd
trusted.gfid=0x8afeb00bb1e74cbab932acea705b7dd9
trusted.gfid=0x2174086880fc4fd19b187d1384300add
trusted.gfid=0x2057e87cf4cc43f9bbad160cbec43d01
trusted.gfid=0xa7d78519db61459399e01fad2badf3fb
trusted.gfid=0xfaa0ed7ccaf84f6c8bdb20a7f657c4b4
trusted.gfid=0x2683990126724adbb6416b911180e62b

Post by Ravishankar N
Let us see if the parents' gfids are the same before deleting anything.
Is the heal info still showing 4 entries? Please also share the getfattr
output of the the parent directory (i.e. dir11) .

Yes, the heal info still shows the 4 entries but on node 1 the directory name is not shown anymore but just the GFID. This is the actual output of a "volume heal info":

Brick node1:/data/myvol-pro/brick
<gfid:25e2616b-4fb6-4b2a-8945-1afc956fff19>
<gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360>
<gfid:70c894ca-422b-4bce-acf1-5cfb4669abbd>
<gfid:aae4098a-1a71-4155-9cc9-e564b89957cf>
Status: Connected
Number of entries: 4

Brick node2:/data/myvol-pro/brick
Status: Connected
Number of entries: 0

Brick node3:/srv/glusterfs/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
<gfid:aae4098a-1a71-4155-9cc9-e564b89957cf>
<gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360>
Status: Connected
Number of entries: 4

What are the next steps in order to fix that?

[Ravishankar N]

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
I thought it was missing which is why I asked you to create it.  The
trusted.gfid xattr for any given file or directory must be same in all 3
bricks.  But it looks like that isn't the case. Are the gfids and the
symlinks for all the dirs leading to the parent dir of oc_dir same on
all nodes? (i.e evey directory in
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/)?

I now checked the GFIDs of all directories leading back down to the parent dir (13 directories in total) and for node 1 and node 3 the GIFDs of all underlying directories match each other. On node 2 they are also all the same except for the two highest directories (".../dir11" and and ".../dir11/oc_dir"). It's exactly these two directories which are also listed in the "volume heal info" output under node 1 and node 2 and which do not get healed.
For your reference I have pasted below the GFIDs for all underlying directories up to the parent directory and for all 3 nodes. I start at the top with the highest directory and at the bottom of the list is the parent directory (/data).
# NODE 1
trusted.gfid=0x25e2616b4fb64b2a89451afc956fff19 # /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
trusted.gfid=0x70c894ca422b4bceacf15cfb4669abbd # /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11
trusted.gfid=0x7d7d2165f4804edf8c93de01c8768269 # ...
trusted.gfid=0xdbc0bfa0a052405ca3fad2d1ca137f82
trusted.gfid=0xbb75051c24ba4c119351bef938c55ad4
trusted.gfid=0x0002ad0c3fbe4806a75f8e68304f5b94
trusted.gfid=0xf120657977274247900db4e9cc8129dd
trusted.gfid=0x8afeb00bb1e74cbab932acea705b7dd9
trusted.gfid=0x2174086880fc4fd19b187d1384300add
trusted.gfid=0x2057e87cf4cc43f9bbad160cbec43d01 # ...
trusted.gfid=0xa7d78519db61459399e01fad2badf3fb # /data/dir1/dir2
trusted.gfid=0xfaa0ed7ccaf84f6c8bdb20a7f657c4b4 # /data/dir1
trusted.gfid=0x2683990126724adbb6416b911180e62b # /data
# NODE 2
trusted.gfid=0xd9ac192ce85e4402af105551f587ed9a
trusted.gfid=0x10ec1eb1c8544ff2a36c325681713093
trusted.gfid=0x7d7d2165f4804edf8c93de01c8768269
trusted.gfid=0xdbc0bfa0a052405ca3fad2d1ca137f82
trusted.gfid=0xbb75051c24ba4c119351bef938c55ad4
trusted.gfid=0x0002ad0c3fbe4806a75f8e68304f5b94
trusted.gfid=0xf120657977274247900db4e9cc8129dd
trusted.gfid=0x8afeb00bb1e74cbab932acea705b7dd9
trusted.gfid=0x2174086880fc4fd19b187d1384300add
trusted.gfid=0x2057e87cf4cc43f9bbad160cbec43d01
trusted.gfid=0xa7d78519db61459399e01fad2badf3fb
trusted.gfid=0xfaa0ed7ccaf84f6c8bdb20a7f657c4b4
trusted.gfid=0x2683990126724adbb6416b911180e62b
# NODE 3
trusted.gfid=0x25e2616b4fb64b2a89451afc956fff19
trusted.gfid=0x70c894ca422b4bceacf15cfb4669abbd
trusted.gfid=0x7d7d2165f4804edf8c93de01c8768269
trusted.gfid=0xdbc0bfa0a052405ca3fad2d1ca137f82
trusted.gfid=0xbb75051c24ba4c119351bef938c55ad4
trusted.gfid=0x0002ad0c3fbe4806a75f8e68304f5b94
trusted.gfid=0xf120657977274247900db4e9cc8129dd
trusted.gfid=0x8afeb00bb1e74cbab932acea705b7dd9
trusted.gfid=0x2174086880fc4fd19b187d1384300add
trusted.gfid=0x2057e87cf4cc43f9bbad160cbec43d01
trusted.gfid=0xa7d78519db61459399e01fad2badf3fb
trusted.gfid=0xfaa0ed7ccaf84f6c8bdb20a7f657c4b4
trusted.gfid=0x2683990126724adbb6416b911180e62b

Post by Ravishankar N
Let us see if the parents' gfids are the same before deleting anything.
Is the heal info still showing 4 entries? Please also share the getfattr
output of the the parent directory (i.e. dir11) .

Brick node1:/data/myvol-pro/brick
<gfid:25e2616b-4fb6-4b2a-8945-1afc956fff19>
<gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360>
<gfid:70c894ca-422b-4bce-acf1-5cfb4669abbd>
<gfid:aae4098a-1a71-4155-9cc9-e564b89957cf>
Status: Connected
Number of entries: 4
Brick node2:/data/myvol-pro/brick
Status: Connected
Number of entries: 0
Brick node3:/srv/glusterfs/myvol-pro/brick
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11
/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
<gfid:aae4098a-1a71-4155-9cc9-e564b89957cf>
<gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360>
Status: Connected
Number of entries: 4
What are the next steps in order to fix that?

1.Could you provide the getfattr output of the following 3 dirs from all
3 nodes?
i)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10
ii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/
iii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir

2. Do you know the file (or directory) names corresponding to the other
2 gfids  in heal info output, i.e
<gfid:aae4098a-1a71-4155-9cc9-e564b89957cf>
<gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360>
Please share the getfattr output of them as well.

Regards,
Ravi

[mabi]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
1.Could you provide the getfattr output of the following 3 dirs from all
3 nodes?
i)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10
ii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/
iii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir

Sure, you will find below the getfattr output of all 3 directories from all 3 nodes.

i)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10

# NODE 1
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000000000000000000000
trusted.gfid=0x7d7d2165f4804edf8c93de01c8768269
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

# NODE 2
trusted.gfid=0x7d7d2165f4804edf8c93de01c8768269
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

# NODE 3
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000000000000000000000
trusted.gfid=0x7d7d2165f4804edf8c93de01c8768269
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

ii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/

# NODE 1
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000000000000400000003
trusted.gfid=0x70c894ca422b4bceacf15cfb4669abbd
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

# NODE 2
trusted.gfid=0x10ec1eb1c8544ff2a36c325681713093
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

# NODE 3
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000000000000400000003
trusted.gfid=0x70c894ca422b4bceacf15cfb4669abbd
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

iii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir

# NODE 1
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000000000000300000003
trusted.gfid=0x25e2616b4fb64b2a89451afc956fff19
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

# NODE 2
trusted.gfid=0xd9ac192ce85e4402af105551f587ed9a
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

# NODE 3
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000000000000300000003
trusted.gfid=0x25e2616b4fb64b2a89451afc956fff19
trusted.glusterfs.dht=0x000000010000000000000000ffffffff

Post by Ravishankar N
2. Do you know the file (or directory) names corresponding to the other
2 gfids  in heal info output, i.e
gfid:aae4098a-1a71-4155-9cc9-e564b89957cf
gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360
Please share the getfattr output of them as well.

Unfortunately no. I tried the trick of mounting the volume with the mount option "aux-gfid-mount" in order to find the filename corresponding to the GFID and then using the following getfattr command:

getfattr -n trusted.glusterfs.pathinfo -e text /mnt/g/.gfid/aae4098a-1a71-4155-9cc9-e564b89957cf

this gave me the following output:

trusted.glusterfs.pathinfo="(<DISTRIBUTE:myvol-pro-dht> (<REPLICATE:myvol-pro-replicate-0> <POSIX(/srv/glusterfs/myvol-pro/brick):node3:/srv/glusterfs/myvol-pro/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf> <POSIX(/data/myvol-pro/brick):node1:/data/myvol-pro/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf>))"

then if I check the ".../brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf" on node 1 or node 3 it does not have any symlink to a file. Or am I looking at the wrong place maybe or there is another trick in order to find the GFID->filename?

Regards,
Mabi

[Ravishankar N]

Post by mabi
Sure, you will find below the getfattr output of all 3 directories from all 3 nodes.

Thanks, noted. One more query. Are there files inside each of these
directories? Or is it just empty directories?

Post by mabi

Post by Ravishankar N
2. Do you know the file (or directory) names corresponding to the other
2 gfids  in heal info output, i.e
gfid:aae4098a-1a71-4155-9cc9-e564b89957cf
gfid:3c92459b-8fa1-4669-9a3d-b38b8d41c360
Please share the getfattr output of them as well.

getfattr -n trusted.glusterfs.pathinfo -e text /mnt/g/.gfid/aae4098a-1a71-4155-9cc9-e564b89957cf
trusted.glusterfs.pathinfo="(<DISTRIBUTE:myvol-pro-dht> (<REPLICATE:myvol-pro-replicate-0> <POSIX(/srv/glusterfs/myvol-pro/brick):node3:/srv/glusterfs/myvol-pro/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf> <POSIX(/data/myvol-pro/brick):node1:/data/myvol-pro/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf>))"
then if I check the ".../brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf" on node 1 or node 3 it does not have any symlink to a file. Or am I looking at the wrong place maybe or there is another trick in order to find the GFID->filename?

symlinks are only for dirs. For files, they would be hard links to the
actual files. So if stat
../brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf gives you
a file, then you can use find -samefile to get the other hardlinks like so:
#cd /brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf
#find /brick -samefile aae4098a-1a71-4155-9cc9-e564b89957cf

If it is a hardlink, then you can do a getfattr on
/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf itself.
-Ravi

Post by mabi
Regards,
Mabi

[mabi]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Thanks, noted. One more query. Are there files inside each of these
directories? Or is it just empty directories?

You will find below the content of each of these 3 directories taken the brick on node 1:

i)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10

drwxr-xr-x 4 www-data www-data 4 Nov 5 14:19 .
drwxr-xr-x 31 www-data www-data 31 Nov 5 14:23 ..
drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 dir11
drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 another_dir

ii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/
drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 .
drwxr-xr-x 4 www-data www-data 4 Nov 5 14:19 ..
drwxr-xr-x 2 www-data www-data 4 Nov 5 14:19 oc_dir

iii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir

drwxr-xr-x 2 www-data www-data 4 Nov 5 14:19 .
drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 ..
-rw-r--r-- 2 www-data www-data 32 Nov 5 14:19 fileKey
-rw-r--r-- 2 www-data www-data 512 Nov 5 14:19 username.shareKey

so as you see from the output above only the "oc_dir" directory has two files inside.

Post by Ravishankar N
symlinks are only for dirs. For files, they would be hard links to the
actual files. So if stat
../brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf gives you
#cd /brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf
#find /brick -samefile aae4098a-1a71-4155-9cc9-e564b89957cf
If it is a hardlink, then you can do a getfattr on
/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf itself.
-Ravi

Thank you for explaining this important part. So yes with your help I could find the filenames associated to these 2 GFIDs and guess what? they are the 2 files which listed in the output above of the "oc_dir" directory. Have a look at this:

# find /data/myvol-pro/brick -samefile aae4098a-1a71-4155-9cc9-e564b89957cf
/data/myvol-pro/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf
/data/myvol-pro/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey

# find /data/myvol-pro/brick -samefile 3c92459b-8fa1-4669-9a3d-b38b8d41c360
/data/myvol-pro/brick/.glusterfs/3c/92/3c92459b-8fa1-4669-9a3d-b38b8d41c360
/data/myvol-pro/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKey

I hope that helps the debug further else let me know if you need anything else.

[Ravishankar N]

Post by mabi
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Thanks, noted. One more query. Are there files inside each of these
directories? Or is it just empty directories?

i)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10
drwxr-xr-x 4 www-data www-data 4 Nov 5 14:19 .
drwxr-xr-x 31 www-data www-data 31 Nov 5 14:23 ..
drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 dir11
drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 another_dir
ii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/
drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 .
drwxr-xr-x 4 www-data www-data 4 Nov 5 14:19 ..
drwxr-xr-x 2 www-data www-data 4 Nov 5 14:19 oc_dir
iii)/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir
drwxr-xr-x 2 www-data www-data 4 Nov 5 14:19 .
drwxr-xr-x 3 www-data www-data 3 Nov 5 14:19 ..
-rw-r--r-- 2 www-data www-data 32 Nov 5 14:19 fileKey
-rw-r--r-- 2 www-data www-data 512 Nov 5 14:19 username.shareKey
so as you see from the output above only the "oc_dir" directory has two files inside.

Okay, I'm assuming the list of files+dirs are the same on nodes 2 and 3
as well. Correct me if that isn't the case.

Post by mabi

Post by Ravishankar N
symlinks are only for dirs. For files, they would be hard links to the
actual files. So if stat
../brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf gives you
#cd /brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf
#find /brick -samefile aae4098a-1a71-4155-9cc9-e564b89957cf
If it is a hardlink, then you can do a getfattr on
/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf itself.
-Ravi

# find /data/myvol-pro/brick -samefile aae4098a-1a71-4155-9cc9-e564b89957cf
/data/myvol-pro/brick/.glusterfs/aa/e4/aae4098a-1a71-4155-9cc9-e564b89957cf
/data/myvol-pro/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey
# find /data/myvol-pro/brick -samefile 3c92459b-8fa1-4669-9a3d-b38b8d41c360
/data/myvol-pro/brick/.glusterfs/3c/92/3c92459b-8fa1-4669-9a3d-b38b8d41c360
/data/myvol-pro/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKey

Okay, as asked in the previous mail, please share the getfattr output
from all bricks for these 2 files. I think once we have this, we can try
either 'adjusting' the the gfid and symlinks on node 2 for dir11 and
oc_dir or see if we can set afr xattrs on dir10 for self-heal to purge
everything under it on node 2 and recreate it using the other 2 nodes.
-Ravi

Post by mabi
I hope that helps the debug further else let me know if you need anything else.

[mabi]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Okay, as asked in the previous mail, please share the getfattr output
from all bricks for these 2 files. I think once we have this, we can try
either 'adjusting' the the gfid and symlinks on node 2 for dir11 and
oc_dir or see if we can set afr xattrs on dir10 for self-heal to purge
everything under it on node 2 and recreate it using the other 2 nodes.

And finally here is the output of a getfattr from both files from the 3 nodes:

FILE 1: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey

NODE 1:
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579

NODE 2:
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x48ccb52b788f4361b33fad43157b8ea8
trusted.gfid2path.32a8dc56983f7b8f=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f66696c654b6579

NODE 3:
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579


FILE 2: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKey

NODE 1:
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579

NODE 2:
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xae880a4f19824bc6a3baabe2e3c62ace
trusted.gfid2path.0c0f97b97351b4af=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f6a6d406d616765726c2e63682e73686172654b6579

NODE 3:
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579

Thanks again in advance for your answer.

[Ravishankar N]

Okay so for all files and dirs, node 2 seems to be the bad copy. Try the
following:

1. On both node 1 and node3, set theafr xattr for dir10:
setfattr -n trusted.afr.myvol-pro-client-1 -v 0x000000000000000100000001
/data/myvol-private/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10

2. Fuse mount the volume temporarily in some location and from that
mount point, do a `find .|xargs stat >/dev/null`

3. Run `gluster volume heal $volname`

HTH,
Ravi

Post by mabi
FILE 1: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x48ccb52b788f4361b33fad43157b8ea8
trusted.gfid2path.32a8dc56983f7b8f=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f66696c654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579
FILE 2: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKey
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xae880a4f19824bc6a3baabe2e3c62ace
trusted.gfid2path.0c0f97b97351b4af=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f6a6d406d616765726c2e63682e73686172654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579

[mabi]

Thank you Ravi for your answer. I have now set the afr xattr as you suggested and I am running the "find . | xargs -d '\n' stat" on my gluster fuse mount for this volume.

This volume has around 3 million of files and directories so it will take a long time to finish I suppose. Do I really need to run this find over the whole volume starting from its root?

Note that I added the "-d '\n'" option in xargs in order to deal with filenames which have spaces inside.


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Okay so for all files and dirs, node 2 seems to be the bad copy. Try the
setfattr -n trusted.afr.myvol-pro-client-1 -v 0x000000000000000100000001
/data/myvol-private/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10
2. Fuse mount the volume temporarily in some location and from that
mount point, do a `find .|xargs stat >/dev/null`
3. Run`gluster volume heal $volname`
HTH,
Ravi

Post by mabi
FILE 1: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x48ccb52b788f4361b33fad43157b8ea8
trusted.gfid2path.32a8dc56983f7b8f=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f66696c654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579
FILE 2: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKey
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xae880a4f19824bc6a3baabe2e3c62ace
trusted.gfid2path.0c0f97b97351b4af=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f6a6d406d616765726c2e63682e73686172654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579

[mabi]

Good news, the stat on all files of my volume finished after running for over 6 hours and the 4 files (actually 2 directories and 2 files) are now finally all healed. I checked the 3 bricks and all have the correct data. On node 1 I also saw 4 healing log entries in glustershd.log log file. I did not even need to manually run a "volume heal" as it healed automatically.

Now, I would really like to avoid this situation in the future, it's a pain for me and maybe also for you guys helping me ;-) Is this a bug or am I doing something wrong? How can I avoid this type of manual fixing in the future?

Again a big thank you Ravi for your patience helping me out with this issue.

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

Post by Ravishankar N
Okay so for all files and dirs, node 2 seems to be the bad copy. Try the
setfattr -n trusted.afr.myvol-pro-client-1 -v 0x000000000000000100000001
/data/myvol-private/brick/data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10
2. Fuse mount the volume temporarily in some location and from that
mount point, do a `find .|xargs stat >/dev/null`
3. Run`gluster volume heal $volname`
HTH,
Ravi

Post by mabi
FILE 1: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/fileKey
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0x48ccb52b788f4361b33fad43157b8ea8
trusted.gfid2path.32a8dc56983f7b8f=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f66696c654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0xaae4098a1a7141559cc9e564b89957cf
trusted.gfid2path.9a863b050c1975ed=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f66696c654b6579
FILE 2: /data/dir1/dir2/dir3/dir4/dir5/dir6/dir7/dir8/dir9/dir10/dir11/oc_dir/username.shareKey
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.gfid=0xae880a4f19824bc6a3baabe2e3c62ace
trusted.gfid2path.0c0f97b97351b4af=0x64396163313932632d653835652d343430322d616631302d3535353166353837656439612f6a6d406d616765726c2e63682e73686172654b6579
trusted.afr.dirty=0x000000000000000000000000
trusted.afr.myvol-pro-client-1=0x000000020000000100000000
trusted.gfid=0x3c92459b8fa146699a3db38b8d41c360
trusted.gfid2path.510dd4750ef350f9=0x32356532363136622d346662362d346232612d383934352d3161666339353666666631392f6a6d406d616765726c2e63682e73686172654b6579