lejeczek
2018-06-30 19:00:51 UTC
hi guys
something wrong with my gluster, it saysthere are files
healing but it does not seem like it actually heals anything.
Here is, apologies for biggish snippet, a bit of log from
one volume. I cannot decode it but have a felling that can
expert/devel spot something is not completely okey there.
(gluster does not show vol is in split-brain)
many thanks, L.
log:
...
[2018-06-30 18:55:56.420785] W [MSGID: 101174]
[graph.c:363:_log_if_unknown_option]
0-GROUP-WORK-readdir-ahead: option 'parallel-readdir' is not
recognized
[2018-06-30 18:55:56.421105] I [MSGID: 104045]
[glfs-master.c:91:notify] 0-gfapi: New graph
7768616c-652e-7072-6976-6174652e6363 (0) coming up
[2018-06-30 18:55:56.421144] I [MSGID: 114020]
[client.c:2360:notify] 0-GROUP-WORK-client-7: parent
translators are ready, attempting connect on transport
[2018-06-30 18:55:56.433472] I [MSGID: 114020]
[client.c:2360:notify] 0-GROUP-WORK-client-8: parent
translators are ready, attempting connect on transport
[2018-06-30 18:55:56.437464] I
[rpc-clnt.c:1986:rpc_clnt_reconfig] 0-GROUP-WORK-client-7:
changing port to 49154 (from 0)
[2018-06-30 18:55:56.438162] I [MSGID: 114020]
[client.c:2360:notify] 0-GROUP-WORK-client-9: parent
translators are ready, attempting connect on transport
[2018-06-30 18:55:56.446455] I [MSGID: 114057]
[client-handshake.c:1478:select_server_supported_programs]
0-GROUP-WORK-client-7: Using Program GlusterFS 3.3, Num
(1298437), Version (330)
Final graph:
+------------------------------------------------------------------------------+
1: volume GROUP-WORK-client-7
2: type protocol/client
3: option opversion 31202
4: option clnt-lk-version 1
5: option volfile-checksum 0
6: option volfile-key GROUP-WORK
7: option client-version 3.12.9
8: option process-uuid
whale.private-3684460-2018/06/30-18:55:56:400699-GROUP-WORK-client-7-0-0
9: option fops-version 1298437
10: option ping-timeout 42
11: option remote-host 10.5.6.49
12: option remote-subvolume
/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK
13: option transport-type socket
14: option transport.address-family inet
15: option username 7e319da6-d30c-4885-bfac-aaa3ddbe2725
16: option password ca697271-8219-4b58-b03b-698ff1901d0e
17: option transport.tcp-user-timeout 0
18: option transport.socket.keepalive-time 20
19: option transport.socket.keepalive-interval 2
20: option transport.socket.keepalive-count 9
21: option send-gids true
22: end-volume
23:
24: volume GROUP-WORK-client-8
25: type protocol/client
26: option ping-timeout 42
27: option remote-host 10.5.6.100
28: option remote-subvolume
/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK
29: option transport-type socket
30: option transport.address-family inet
31: option username 7e319da6-d30c-4885-bfac-aaa3ddbe2725
32: option password ca697271-8219-4b58-b03b-698ff1901d0e
33: option transport.tcp-user-timeout 0
34: option transport.socket.keepalive-time 20
35: option transport.socket.keepalive-interval 2
[2018-06-30 18:55:56.446995] I
[rpc-clnt.c:1986:rpc_clnt_reconfig] 0-GROUP-WORK-client-8:
changing port to 49154 (from 0)
36: option transport.socket.keepalive-count 9
37: option send-gids true
38: end-volume
39:
40: volume GROUP-WORK-client-9
41: type protocol/client
42: option ping-timeout 42
43: option remote-host 10.5.6.81
44: option remote-subvolume
/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.GROUP-WORK
45: option transport-type socket
46: option transport.address-family inet
47: option username 7e319da6-d30c-4885-bfac-aaa3ddbe2725
48: option password ca697271-8219-4b58-b03b-698ff1901d0e
49: option transport.tcp-user-timeout 0
50: option transport.socket.keepalive-time 20
51: option transport.socket.keepalive-interval 2
52: option transport.socket.keepalive-count 9
53: option send-gids true
54: end-volume
55:
56: volume GROUP-WORK-replicate-0
57: type cluster/replicate
58: option background-self-heal-count 0
59: option afr-pending-xattr
GROUP-WORK-client-7,GROUP-WORK-client-8,GROUP-WORK-client-9
60: option use-compound-fops off
61: subvolumes GROUP-WORK-client-7 GROUP-WORK-client-8
GROUP-WORK-client-9
62: end-volume
63:
64: volume GROUP-WORK-dht
65: type cluster/distribute
66: option lock-migration off
67: subvolumes GROUP-WORK-replicate-0
68: end-volume
69:
70: volume GROUP-WORK-write-behind
71: type performance/write-behind
72: subvolumes GROUP-WORK-dht
73: end-volume
74:
75: volume GROUP-WORK-read-ahead
76: type performance/read-ahead
77: subvolumes GROUP-WORK-write-behind
78: end-volume
79:
80: volume GROUP-WORK-readdir-ahead
81: type performance/readdir-ahead
82: option parallel-readdir off
83: option rda-request-size 131072
84: option rda-cache-limit 10MB
85: subvolumes GROUP-WORK-read-ahead
86: end-volume
87:
88: volume GROUP-WORK-io-cache
89: type performance/io-cache
90: option cache-size 128MB
91: subvolumes GROUP-WORK-readdir-ahead
92: end-volume
93:
94: volume GROUP-WORK-quick-read
95: type performance/quick-read
96: option cache-size 128MB
97: subvolumes GROUP-WORK-io-cache
98: end-volume
99:
100: volume GROUP-WORK-open-behind
101: type performance/open-behind
102: subvolumes GROUP-WORK-quick-read
103: end-volume
104:
105: volume GROUP-WORK-md-cache
106: type performance/md-cache
107: option md-cache-timeout 600
108: option cache-samba-metadata on
109: option cache-invalidation on
110: subvolumes GROUP-WORK-open-behind
111: end-volume
112:
113: volume GROUP-WORK
114: type debug/io-stats
115: option log-level INFO
116: option latency-measurement off
117: option count-fop-hits off
118: subvolumes GROUP-WORK-md-cache
119: end-volume
120:
121: volume meta-autoload
122: type meta
123: subvolumes GROUP-WORK
124: end-volume
125:
+------------------------------------------------------------------------------+
[2018-06-30 18:55:56.448438] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk]
0-GROUP-WORK-client-7: Connected to GROUP-WORK-client-7,
attached to remote volume
'/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK'.
[2018-06-30 18:55:56.448473] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk]
0-GROUP-WORK-client-7: Server and Client lk-version numbers
are not same, reopening the fds
[2018-06-30 18:55:56.448625] I [MSGID: 108005]
[afr-common.c:5015:__afr_handle_child_up_event]
0-GROUP-WORK-replicate-0: Subvolume 'GROUP-WORK-client-7'
came back up; going online.
[2018-06-30 18:55:56.452916] I
[rpc-clnt.c:1986:rpc_clnt_reconfig] 0-GROUP-WORK-client-9:
changing port to 49156 (from 0)
[2018-06-30 18:55:56.453000] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-GROUP-WORK-client-7: Server lk version = 1
[2018-06-30 18:55:56.456971] I [MSGID: 114057]
[client-handshake.c:1478:select_server_supported_programs]
0-GROUP-WORK-client-8: Using Program GlusterFS 3.3, Num
(1298437), Version (330)
[2018-06-30 18:55:56.458254] I [MSGID: 114057]
[client-handshake.c:1478:select_server_supported_programs]
0-GROUP-WORK-client-9: Using Program GlusterFS 3.3, Num
(1298437), Version (330)
[2018-06-30 18:55:56.459241] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk]
0-GROUP-WORK-client-9: Connected to GROUP-WORK-client-9,
attached to remote volume
'/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.GROUP-WORK'.
[2018-06-30 18:55:56.459282] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk]
0-GROUP-WORK-client-9: Server and Client lk-version numbers
are not same, reopening the fds
[2018-06-30 18:55:56.459353] I [MSGID: 108002]
[afr-common.c:5312:afr_notify] 0-GROUP-WORK-replicate-0:
Client-quorum is met
[2018-06-30 18:55:56.459535] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-GROUP-WORK-client-9: Server lk version = 1
[2018-06-30 18:55:56.459860] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk]
0-GROUP-WORK-client-8: Connected to GROUP-WORK-client-8,
attached to remote volume
'/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK'.
[2018-06-30 18:55:56.459888] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk]
0-GROUP-WORK-client-8: Server and Client lk-version numbers
are not same, reopening the fds
[2018-06-30 18:55:56.461806] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-GROUP-WORK-client-8: Server lk version = 1
[2018-06-30 18:55:56.481552] I [MSGID: 108031]
[afr-common.c:2458:afr_local_discovery_cbk]
0-GROUP-WORK-replicate-0: selecting local read_child
GROUP-WORK-client-7
[2018-06-30 18:55:56.482490] I [MSGID: 104041]
[glfs-resolve.c:971:__glfs_active_subvol] 0-GROUP-WORK:
switched to graph 7768616c-652e-7072-6976-6174652e6363 (0)
[2018-06-30 18:55:56.582941] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]" repeated 2 times between [2018-06-30
18:55:56.582941] and [2018-06-30 18:55:56.586687]
[2018-06-30 18:55:56.773663] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]" repeated 2 times between [2018-06-30
18:55:56.773663] and [2018-06-30 18:55:56.780197]
[2018-06-30 18:55:58.475889] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]" repeated 2 times between [2018-06-30
18:55:58.475889] and [2018-06-30 18:55:58.479828]
[2018-06-30 18:55:58.685911] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]" repeated 2 times between [2018-06-30
18:55:58.685911] and [2018-06-30 18:55:58.691170]
[2018-06-30 18:56:00.317702] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]" repeated 2 times between [2018-06-30
18:56:00.317702] and [2018-06-30 18:56:00.322284]
[2018-06-30 18:56:00.324742] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]" repeated 2 times between [2018-06-30
18:56:00.324742] and [2018-06-30 18:56:00.329691]
[2018-06-30 18:56:00.334014] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]" repeated 2 times between [2018-06-30
18:56:00.334014] and [2018-06-30 18:56:00.339246]
[2018-06-30 18:56:00.341721] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]
something wrong with my gluster, it saysthere are files
healing but it does not seem like it actually heals anything.
Here is, apologies for biggish snippet, a bit of log from
one volume. I cannot decode it but have a felling that can
expert/devel spot something is not completely okey there.
(gluster does not show vol is in split-brain)
many thanks, L.
log:
...
[2018-06-30 18:55:56.420785] W [MSGID: 101174]
[graph.c:363:_log_if_unknown_option]
0-GROUP-WORK-readdir-ahead: option 'parallel-readdir' is not
recognized
[2018-06-30 18:55:56.421105] I [MSGID: 104045]
[glfs-master.c:91:notify] 0-gfapi: New graph
7768616c-652e-7072-6976-6174652e6363 (0) coming up
[2018-06-30 18:55:56.421144] I [MSGID: 114020]
[client.c:2360:notify] 0-GROUP-WORK-client-7: parent
translators are ready, attempting connect on transport
[2018-06-30 18:55:56.433472] I [MSGID: 114020]
[client.c:2360:notify] 0-GROUP-WORK-client-8: parent
translators are ready, attempting connect on transport
[2018-06-30 18:55:56.437464] I
[rpc-clnt.c:1986:rpc_clnt_reconfig] 0-GROUP-WORK-client-7:
changing port to 49154 (from 0)
[2018-06-30 18:55:56.438162] I [MSGID: 114020]
[client.c:2360:notify] 0-GROUP-WORK-client-9: parent
translators are ready, attempting connect on transport
[2018-06-30 18:55:56.446455] I [MSGID: 114057]
[client-handshake.c:1478:select_server_supported_programs]
0-GROUP-WORK-client-7: Using Program GlusterFS 3.3, Num
(1298437), Version (330)
Final graph:
+------------------------------------------------------------------------------+
1: volume GROUP-WORK-client-7
2: type protocol/client
3: option opversion 31202
4: option clnt-lk-version 1
5: option volfile-checksum 0
6: option volfile-key GROUP-WORK
7: option client-version 3.12.9
8: option process-uuid
whale.private-3684460-2018/06/30-18:55:56:400699-GROUP-WORK-client-7-0-0
9: option fops-version 1298437
10: option ping-timeout 42
11: option remote-host 10.5.6.49
12: option remote-subvolume
/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK
13: option transport-type socket
14: option transport.address-family inet
15: option username 7e319da6-d30c-4885-bfac-aaa3ddbe2725
16: option password ca697271-8219-4b58-b03b-698ff1901d0e
17: option transport.tcp-user-timeout 0
18: option transport.socket.keepalive-time 20
19: option transport.socket.keepalive-interval 2
20: option transport.socket.keepalive-count 9
21: option send-gids true
22: end-volume
23:
24: volume GROUP-WORK-client-8
25: type protocol/client
26: option ping-timeout 42
27: option remote-host 10.5.6.100
28: option remote-subvolume
/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK
29: option transport-type socket
30: option transport.address-family inet
31: option username 7e319da6-d30c-4885-bfac-aaa3ddbe2725
32: option password ca697271-8219-4b58-b03b-698ff1901d0e
33: option transport.tcp-user-timeout 0
34: option transport.socket.keepalive-time 20
35: option transport.socket.keepalive-interval 2
[2018-06-30 18:55:56.446995] I
[rpc-clnt.c:1986:rpc_clnt_reconfig] 0-GROUP-WORK-client-8:
changing port to 49154 (from 0)
36: option transport.socket.keepalive-count 9
37: option send-gids true
38: end-volume
39:
40: volume GROUP-WORK-client-9
41: type protocol/client
42: option ping-timeout 42
43: option remote-host 10.5.6.81
44: option remote-subvolume
/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.GROUP-WORK
45: option transport-type socket
46: option transport.address-family inet
47: option username 7e319da6-d30c-4885-bfac-aaa3ddbe2725
48: option password ca697271-8219-4b58-b03b-698ff1901d0e
49: option transport.tcp-user-timeout 0
50: option transport.socket.keepalive-time 20
51: option transport.socket.keepalive-interval 2
52: option transport.socket.keepalive-count 9
53: option send-gids true
54: end-volume
55:
56: volume GROUP-WORK-replicate-0
57: type cluster/replicate
58: option background-self-heal-count 0
59: option afr-pending-xattr
GROUP-WORK-client-7,GROUP-WORK-client-8,GROUP-WORK-client-9
60: option use-compound-fops off
61: subvolumes GROUP-WORK-client-7 GROUP-WORK-client-8
GROUP-WORK-client-9
62: end-volume
63:
64: volume GROUP-WORK-dht
65: type cluster/distribute
66: option lock-migration off
67: subvolumes GROUP-WORK-replicate-0
68: end-volume
69:
70: volume GROUP-WORK-write-behind
71: type performance/write-behind
72: subvolumes GROUP-WORK-dht
73: end-volume
74:
75: volume GROUP-WORK-read-ahead
76: type performance/read-ahead
77: subvolumes GROUP-WORK-write-behind
78: end-volume
79:
80: volume GROUP-WORK-readdir-ahead
81: type performance/readdir-ahead
82: option parallel-readdir off
83: option rda-request-size 131072
84: option rda-cache-limit 10MB
85: subvolumes GROUP-WORK-read-ahead
86: end-volume
87:
88: volume GROUP-WORK-io-cache
89: type performance/io-cache
90: option cache-size 128MB
91: subvolumes GROUP-WORK-readdir-ahead
92: end-volume
93:
94: volume GROUP-WORK-quick-read
95: type performance/quick-read
96: option cache-size 128MB
97: subvolumes GROUP-WORK-io-cache
98: end-volume
99:
100: volume GROUP-WORK-open-behind
101: type performance/open-behind
102: subvolumes GROUP-WORK-quick-read
103: end-volume
104:
105: volume GROUP-WORK-md-cache
106: type performance/md-cache
107: option md-cache-timeout 600
108: option cache-samba-metadata on
109: option cache-invalidation on
110: subvolumes GROUP-WORK-open-behind
111: end-volume
112:
113: volume GROUP-WORK
114: type debug/io-stats
115: option log-level INFO
116: option latency-measurement off
117: option count-fop-hits off
118: subvolumes GROUP-WORK-md-cache
119: end-volume
120:
121: volume meta-autoload
122: type meta
123: subvolumes GROUP-WORK
124: end-volume
125:
+------------------------------------------------------------------------------+
[2018-06-30 18:55:56.448438] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk]
0-GROUP-WORK-client-7: Connected to GROUP-WORK-client-7,
attached to remote volume
'/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK'.
[2018-06-30 18:55:56.448473] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk]
0-GROUP-WORK-client-7: Server and Client lk-version numbers
are not same, reopening the fds
[2018-06-30 18:55:56.448625] I [MSGID: 108005]
[afr-common.c:5015:__afr_handle_child_up_event]
0-GROUP-WORK-replicate-0: Subvolume 'GROUP-WORK-client-7'
came back up; going online.
[2018-06-30 18:55:56.452916] I
[rpc-clnt.c:1986:rpc_clnt_reconfig] 0-GROUP-WORK-client-9:
changing port to 49156 (from 0)
[2018-06-30 18:55:56.453000] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-GROUP-WORK-client-7: Server lk version = 1
[2018-06-30 18:55:56.456971] I [MSGID: 114057]
[client-handshake.c:1478:select_server_supported_programs]
0-GROUP-WORK-client-8: Using Program GlusterFS 3.3, Num
(1298437), Version (330)
[2018-06-30 18:55:56.458254] I [MSGID: 114057]
[client-handshake.c:1478:select_server_supported_programs]
0-GROUP-WORK-client-9: Using Program GlusterFS 3.3, Num
(1298437), Version (330)
[2018-06-30 18:55:56.459241] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk]
0-GROUP-WORK-client-9: Connected to GROUP-WORK-client-9,
attached to remote volume
'/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER.GROUP-WORK'.
[2018-06-30 18:55:56.459282] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk]
0-GROUP-WORK-client-9: Server and Client lk-version numbers
are not same, reopening the fds
[2018-06-30 18:55:56.459353] I [MSGID: 108002]
[afr-common.c:5312:afr_notify] 0-GROUP-WORK-replicate-0:
Client-quorum is met
[2018-06-30 18:55:56.459535] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-GROUP-WORK-client-9: Server lk version = 1
[2018-06-30 18:55:56.459860] I [MSGID: 114046]
[client-handshake.c:1231:client_setvolume_cbk]
0-GROUP-WORK-client-8: Connected to GROUP-WORK-client-8,
attached to remote volume
'/__.aLocalStorages/0/0-GLUSTERs/0GLUSTER-GROUP-WORK'.
[2018-06-30 18:55:56.459888] I [MSGID: 114047]
[client-handshake.c:1242:client_setvolume_cbk]
0-GROUP-WORK-client-8: Server and Client lk-version numbers
are not same, reopening the fds
[2018-06-30 18:55:56.461806] I [MSGID: 114035]
[client-handshake.c:202:client_set_lk_version_cbk]
0-GROUP-WORK-client-8: Server lk version = 1
[2018-06-30 18:55:56.481552] I [MSGID: 108031]
[afr-common.c:2458:afr_local_discovery_cbk]
0-GROUP-WORK-replicate-0: selecting local read_child
GROUP-WORK-client-7
[2018-06-30 18:55:56.482490] I [MSGID: 104041]
[glfs-resolve.c:971:__glfs_active_subvol] 0-GROUP-WORK:
switched to graph 7768616c-652e-7072-6976-6174652e6363 (0)
[2018-06-30 18:55:56.582941] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]" repeated 2 times between [2018-06-30
18:55:56.582941] and [2018-06-30 18:55:56.586687]
[2018-06-30 18:55:56.773663] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]" repeated 2 times between [2018-06-30
18:55:56.773663] and [2018-06-30 18:55:56.780197]
[2018-06-30 18:55:58.475889] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]" repeated 2 times between [2018-06-30
18:55:58.475889] and [2018-06-30 18:55:58.479828]
[2018-06-30 18:55:58.685911] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]" repeated 2 times between [2018-06-30
18:55:58.685911] and [2018-06-30 18:55:58.691170]
[2018-06-30 18:56:00.317702] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]" repeated 2 times between [2018-06-30
18:56:00.317702] and [2018-06-30 18:56:00.322284]
[2018-06-30 18:56:00.324742] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]" repeated 2 times between [2018-06-30
18:56:00.324742] and [2018-06-30 18:56:00.329691]
[2018-06-30 18:56:00.334014] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]
The message "W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2>
(ab3a34f8-8bab-4c0e-ad8e-b21aa1e23cb2) [No such file or
directory]" repeated 2 times between [2018-06-30
18:56:00.334014] and [2018-06-30 18:56:00.339246]
[2018-06-30 18:56:00.341721] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk]
0-GROUP-WORK-client-9: remote operation failed. Path:
<gfid:3e550eb9-33df-42b4-8fd2-29dce852e371>
(3e550eb9-33df-42b4-8fd2-29dce852e371) [No such file or
directory]