Hello Rafi,
a few days ago I had the scenario described below. Here are the log
entries. As you can see glusterd was stopped at 15:22:48. The same time
FUSE of shared_storage was stopped. It seems to be nearly the same error
message. You can also see that I restarted glusterd manually but FUSE of
shared_storage was not restarted
*#-> /var/log/glusterfs/glusterd.log[2018-11-21 15:22:37.406032] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:21:45.377310]
and [2018-11-21 15:22:39.382300][2018-11-21 15:22:48.623951] W
[glusterfsd.c:1514:cleanup_and_exit] (-->/lib64/libpthread.so.0(+0x7559)
[0x7fb1e5f11559] -->/usr/sbin/glusterd(glusterfs_sigwaiter+0xfd)
[0x561c3626340d] -->/usr/sbin/glusterd(cleanup_and_exit+0x5a)
[0x561c362631fa] ) 0-: received signum (15), shutting down[2018-11-21
15:22:53.491120] I [MSGID: 100030] [glusterfsd.c:2741:main]
0-/usr/sbin/glusterd: Started running /usr/sbin/glusterd version 4.1.5
(args: /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level
INFO)[2018-11-21 15:22:53.494326] I [MSGID: 106478] [glusterd.c:1423:init]
0-management: Maximum allowed open file descriptors set to 65536[2018-11-21
15:22:53.494372] I [MSGID: 106479] [glusterd.c:1481:init] 0-management:
Using /var/lib/glusterd as working directory[2018-11-21 15:22:53.494393] I
[MSGID: 106479] [glusterd.c:1486:init] 0-management: Using /var/run/gluster
as pid file working directory[2018-11-21 15:22:53.496564] I
[socket.c:4477:socket_init] 0-socket.management: SSL support on the I/O
path is ENABLED[2018-11-21 15:22:53.496598] I [socket.c:4480:socket_init]
0-socket.management: SSL support for glusterd is ENABLED[2018-11-21
15:22:53.496607] I [socket.c:4497:socket_init] 0-socket.management: using
private polling thread[2018-11-21 15:22:53.496615] I
[socket.c:4507:socket_init] 0-socket.management: using certificate depth
1[2018-11-21 15:22:53.496891] E [socket.c:4554:socket_init]
0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are
disabled[2018-11-21 15:22:53.499933] W [MSGID: 103071]
[rdma.c:4629:__gf_rdma_ctx_create] 0-rpc-transport/rdma: rdma_cm event
channel creation failed [No such device][2018-11-21 15:22:53.499970] W
[MSGID: 103055] [rdma.c:4938:init] 0-rdma.management: Failed to initialize
IB Device[2018-11-21 15:22:53.499983] W
[rpc-transport.c:351:rpc_transport_load] 0-rpc-transport: 'rdma'
initialization failed[2018-11-21 15:22:53.500148] W
[rpcsvc.c:1781:rpcsvc_create_listener] 0-rpc-service: cannot create
listener, initing the transport failed[2018-11-21 15:22:53.500174] E
[MSGID: 106244] [glusterd.c:1764:init] 0-management: creation of 1
listeners failed, continuing with succeeded transport[2018-11-21
15:22:53.500361] I [socket.c:4480:socket_init] 0-socket.management: SSL
support for glusterd is ENABLED[2018-11-21 15:22:53.500373] I
[socket.c:4507:socket_init] 0-socket.management: using certificate depth
1[2018-11-21 15:22:53.500615] E [socket.c:4554:socket_init]
0-socket.management: failed to open /etc/ssl/dhparam.pem, DH ciphers are
disabled[2018-11-21 15:22:55.353929] I [MSGID: 106513]
[glusterd-store.c:2240:glusterd_restore_op_version] 0-glusterd: retrieved
op-version: 40100[2018-11-21 15:22:55.423433] I [MSGID: 106544]
[glusterd.c:158:glusterd_uuid_init] 0-management: retrieved UUID:
7e7fd30a-94c9-4c21-9c4b-3817508ef121[2018-11-21 15:22:55.429214] I [MSGID:
106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo]
0-management: connect returned 0[2018-11-21 15:22:55.429317] I [MSGID:
106498] [glusterd-handler.c:3614:glusterd_friend_add_from_peerinfo]
0-management: connect returned 0[2018-11-21 15:22:55.429379] W [MSGID:
106061] [glusterd-handler.c:3408:glusterd_transport_inet_options_build]
0-glusterd: Failed to get tcp-user-timeout[2018-11-21 15:22:55.429443] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting
frame-timeout to 600[2018-11-21 15:22:55.429623] I
[socket.c:4477:socket_init] 0-management: SSL support on the I/O path is
ENABLED[2018-11-21 15:22:55.429635] I [socket.c:4480:socket_init]
0-management: SSL support for glusterd is ENABLED[2018-11-21
15:22:55.429643] I [socket.c:4497:socket_init] 0-management: using private
polling thread[2018-11-21 15:22:55.429650] I [socket.c:4507:socket_init]
0-management: using certificate depth 1[2018-11-21 15:22:55.429914] E
[socket.c:4554:socket_init] 0-management: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:55.431676] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management: setting
frame-timeout to 600[2018-11-21 15:22:55.431846] I
[socket.c:4477:socket_init] 0-management: SSL support on the I/O path is
ENABLED[2018-11-21 15:22:55.431858] I [socket.c:4480:socket_init]
0-management: SSL support for glusterd is ENABLED[2018-11-21
15:22:55.431866] I [socket.c:4497:socket_init] 0-management: using private
polling thread[2018-11-21 15:22:55.431873] I [socket.c:4507:socket_init]
0-management: using certificate depth 1[2018-11-21 15:22:55.432136] E
[socket.c:4554:socket_init] 0-management: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabledFinal
graph:+------------------------------------------------------------------------------+
1: volume management 2: type mgmt/glusterd 3: option
transport.socket.ssl-enabled on 4: option rpc-auth.auth-glusterfs on
5: option rpc-auth.auth-unix on 6: option rpc-auth.auth-null on
7: option rpc-auth-allow-insecure on 8: option
transport.listen-backlog 10 9: option event-threads 1 10: option
ping-timeout 0 11: option transport.socket.read-fail-log off 12:
option transport.socket.keepalive-interval 2 13: option
transport.socket.keepalive-time 10 14: option transport-type
rdma 15: option working-directory /var/lib/glusterd 16: end-volume 17:
+------------------------------------------------------------------------------+[2018-11-21
15:22:55.431634] W [MSGID: 106061]
[glusterd-handler.c:3408:glusterd_transport_inet_options_build] 0-glusterd:
Failed to get tcp-user-timeout[2018-11-21 15:22:55.433711] I [MSGID:
101190] [event-epoll.c:617:event_dispatch_epoll_worker] 0-epoll: Started
thread with index 1[2018-11-21 15:22:58.975934] I [MSGID: 106142]
[glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding brick
/gluster/brick2/glusterbrick on port 49153[2018-11-21 15:22:58.979840] I
[MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding
brick /gluster/brick1/glusterbrick on port 49152[2018-11-21
15:22:59.462323] I [MSGID: 106493]
[glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk] 0-glusterd: Received ACC
from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b, host: fs-apollo-c1-n1,
port: 0[2018-11-21 15:22:59.463676] I [MSGID: 106492]
[glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid:
4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:22:59.463728] I [MSGID:
106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update]
0-management: Received my uuid as Friend[2018-11-21 15:22:59.464948] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-nfs: setting frame-timeout to
600[2018-11-21 15:22:59.465116] I [socket.c:4480:socket_init] 0-nfs: SSL
support for glusterd is ENABLED[2018-11-21 15:22:59.465130] I
[socket.c:4507:socket_init] 0-nfs: using certificate depth 1[2018-11-21
15:22:59.465387] E [socket.c:4554:socket_init] 0-nfs: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:59.466154] I
[MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management:
nfs already stopped[2018-11-21 15:22:59.466195] I [MSGID: 106568]
[glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is
stopped[2018-11-21 15:22:59.467179] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-glustershd: setting
frame-timeout to 600[2018-11-21 15:22:59.467309] I
[socket.c:4480:socket_init] 0-glustershd: SSL support for glusterd is
ENABLED[2018-11-21 15:22:59.467321] I [socket.c:4507:socket_init]
0-glustershd: using certificate depth 1[2018-11-21 15:22:59.467553] E
[socket.c:4554:socket_init] 0-glustershd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:22:59.473619] I
[MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management:
Stopping glustershd daemon running in pid: 4143[2018-11-21 15:22:59.478144]
E [socket.c:2792:socket_poller] 0-socket.management: socket_poller
127.0.0.1:49149 <http://127.0.0.1:49149> failed (Success)[2018-11-21
15:23:00.473882] I [MSGID: 106568]
[glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd
service is stopped[2018-11-21 15:23:00.474011] I [MSGID: 106567]
[glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting
glustershd service[2018-11-21 15:23:01.478838] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-quotad: setting frame-timeout
to 600[2018-11-21 15:23:01.479039] I [socket.c:4480:socket_init] 0-quotad:
SSL support for glusterd is ENABLED[2018-11-21 15:23:01.479054] I
[socket.c:4507:socket_init] 0-quotad: using certificate depth 1[2018-11-21
15:23:01.479336] E [socket.c:4554:socket_init] 0-quotad: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:01.480664] I
[MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management:
quotad already stopped[2018-11-21 15:23:01.480711] I [MSGID: 106568]
[glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: quotad service is
stopped[2018-11-21 15:23:01.480777] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-bitd: setting frame-timeout to
600[2018-11-21 15:23:01.480902] I [socket.c:4480:socket_init] 0-bitd: SSL
support for glusterd is ENABLED[2018-11-21 15:23:01.480912] I
[socket.c:4507:socket_init] 0-bitd: using certificate depth 1[2018-11-21
15:23:01.481145] E [socket.c:4554:socket_init] 0-bitd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:01.486112] I
[MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management:
Stopping bitd daemon running in pid: 4343[2018-11-21 15:23:02.486437] I
[MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management:
bitd service is stopped[2018-11-21 15:23:02.486531] I [MSGID: 106567]
[glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting bitd
service[2018-11-21 15:23:03.492955] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-scrub: setting frame-timeout
to 600[2018-11-21 15:23:03.493131] I [socket.c:4480:socket_init] 0-scrub:
SSL support for glusterd is ENABLED[2018-11-21 15:23:03.493147] I
[socket.c:4507:socket_init] 0-scrub: using certificate depth 1[2018-11-21
15:23:03.493416] E [socket.c:4554:socket_init] 0-scrub: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:03.498042] I
[MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management:
Stopping scrub daemon running in pid: 4354[2018-11-21 15:23:04.498341] I
[MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management:
scrub service is stopped[2018-11-21 15:23:04.498426] I [MSGID: 106567]
[glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting scrub
service[2018-11-21 15:23:05.503972] I
[glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered
already-running brick /gluster/brick1/glusterbrick[2018-11-21
15:23:05.504033] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind]
0-pmap: adding brick /gluster/brick1/glusterbrick on port 49152[2018-11-21
15:23:05.504103] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600[2018-11-21 15:23:05.504229] I
[socket.c:4480:socket_init] 0-management: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.504241] I [socket.c:4507:socket_init]
0-management: using certificate depth 1[2018-11-21 15:23:05.504505] E
[socket.c:4554:socket_init] 0-management: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.510177] I
[glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered
already-running brick /gluster/brick2/glusterbrick[2018-11-21
15:23:05.510210] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind]
0-pmap: adding brick /gluster/brick2/glusterbrick on port 49153[2018-11-21
15:23:05.510259] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600[2018-11-21 15:23:05.510368] I
[socket.c:4480:socket_init] 0-management: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.510380] I [socket.c:4507:socket_init]
0-management: using certificate depth 1[2018-11-21 15:23:05.510635] E
[socket.c:4554:socket_init] 0-management: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.516357] I
[glusterd-utils.c:5994:glusterd_brick_start] 0-management: discovered
already-running brick /var/lib/glusterd/ss_brick[2018-11-21
15:23:05.516387] I [MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind]
0-pmap: adding brick /var/lib/glusterd/ss_brick on port 49154[2018-11-21
15:23:05.516433] I [rpc-clnt.c:1059:rpc_clnt_connection_init] 0-management:
setting frame-timeout to 600[2018-11-21 15:23:05.516538] I
[socket.c:4480:socket_init] 0-management: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.516550] I [socket.c:4507:socket_init]
0-management: using certificate depth 1[2018-11-21 15:23:05.516795] E
[socket.c:4554:socket_init] 0-management: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.522223] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout
to 600[2018-11-21 15:23:05.522301] I [socket.c:4480:socket_init] 0-snapd:
SSL support for glusterd is ENABLED[2018-11-21 15:23:05.522308] I
[socket.c:4507:socket_init] 0-snapd: using certificate depth 1[2018-11-21
15:23:05.522709] E [socket.c:4554:socket_init] 0-snapd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.523245] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout
to 600[2018-11-21 15:23:05.523364] I [socket.c:4480:socket_init] 0-snapd:
SSL support for glusterd is ENABLED[2018-11-21 15:23:05.523379] I
[socket.c:4507:socket_init] 0-snapd: using certificate depth 1[2018-11-21
15:23:05.523544] E [socket.c:4554:socket_init] 0-snapd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.523993] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-snapd: setting frame-timeout
to 600[2018-11-21 15:23:05.524564] I [socket.c:4480:socket_init] 0-snapd:
SSL support for glusterd is ENABLED[2018-11-21 15:23:05.524585] I
[socket.c:4507:socket_init] 0-snapd: using certificate depth 1[2018-11-21
15:23:05.524804] E [socket.c:4554:socket_init] 0-snapd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.525379] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting
frame-timeout to 600[2018-11-21 15:23:05.525451] I
[socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.525458] I [socket.c:4507:socket_init]
0-gfproxyd: using certificate depth 1[2018-11-21 15:23:05.525649] E
[socket.c:4554:socket_init] 0-gfproxyd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.526277] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting
frame-timeout to 600[2018-11-21 15:23:05.526348] I
[socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.526355] I [socket.c:4507:socket_init]
0-gfproxyd: using certificate depth 1[2018-11-21 15:23:05.526507] E
[socket.c:4554:socket_init] 0-gfproxyd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.527193] I
[rpc-clnt.c:1059:rpc_clnt_connection_init] 0-gfproxyd: setting
frame-timeout to 600[2018-11-21 15:23:05.527265] I
[socket.c:4480:socket_init] 0-gfproxyd: SSL support for glusterd is
ENABLED[2018-11-21 15:23:05.527274] I [socket.c:4507:socket_init]
0-gfproxyd: using certificate depth 1[2018-11-21 15:23:05.527437] E
[socket.c:4554:socket_init] 0-gfproxyd: failed to open
/etc/ssl/dhparam.pem, DH ciphers are disabled[2018-11-21 15:23:05.529501] I
[MSGID: 106493] [glusterd-rpc-ops.c:486:__glusterd_friend_add_cbk]
0-glusterd: Received ACC from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7,
host: fs-apollo-c1-n3, port: 0[2018-11-21 15:23:05.530466] I [MSGID:
106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update]
0-glusterd: Received friend update from uuid:
94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:05.531325] I [MSGID:
106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update]
0-management: Received my uuid as Friend[2018-11-21 15:23:05.531934] I
[MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk]
0-management: Received ACC from uuid:
4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:23:05.532036] I [MSGID:
106163] [glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack]
0-management: using the op-version 40100[2018-11-21 15:23:05.532812] I
[MSGID: 106142] [glusterd-pmap.c:297:pmap_registry_bind] 0-pmap: adding
brick /var/lib/glusterd/ss_brick on port 49154[2018-11-21 15:23:05.533478]
I [MSGID: 106163]
[glusterd-handshake.c:1356:__glusterd_mgmt_hndsk_versions_ack]
0-management: using the op-version 40100[2018-11-21 15:23:05.534519] I
[MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk]
0-management: Received ACC from uuid:
94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:05.535549] I [MSGID:
106490] [glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req]
0-glusterd: Received probe from uuid:
4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:23:05.538671] I [MSGID:
106493] [glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to fs-apollo-c1-n1 (0), ret: 0, op_ret: 0[2018-11-21
15:23:05.541086] I [MSGID: 106490]
[glusterd-handler.c:2548:__glusterd_handle_incoming_friend_req] 0-glusterd:
Received probe from uuid: 94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21
15:23:05.543763] I [MSGID: 106493]
[glusterd-handler.c:3811:glusterd_xfer_friend_add_resp] 0-glusterd:
Responded to fs-apollo-c1-n3 (0), ret: 0, op_ret: 0[2018-11-21
15:23:05.546026] I [MSGID: 106493]
[glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk] 0-management:
Received ACC from uuid: 4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21
15:23:05.547097] I [MSGID: 106492]
[glusterd-handler.c:2726:__glusterd_handle_friend_update] 0-glusterd:
Received friend update from uuid:
4d5226a7-bb9c-4432-ae6b-2f2a8446d41b[2018-11-21 15:23:05.547154] I [MSGID:
106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update]
0-management: Received my uuid as Friend[2018-11-21 15:23:05.547765] I
[MSGID: 106492] [glusterd-handler.c:2726:__glusterd_handle_friend_update]
0-glusterd: Received friend update from uuid:
94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:05.548774] I [MSGID:
106502] [glusterd-handler.c:2771:__glusterd_handle_friend_update]
0-management: Received my uuid as Friend[2018-11-21 15:23:05.548958] I
[MSGID: 106493] [glusterd-rpc-ops.c:702:__glusterd_friend_update_cbk]
0-management: Received ACC from uuid:
94d352c3-4fac-4926-bbde-e90af52bcad7[2018-11-21 15:23:31.411509] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:23:33.372957] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 3 times between [2018-11-21 15:23:33.372957]
and [2018-11-21 15:23:33.376516][2018-11-21 15:24:25.407528] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:24:27.370775] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 3 times between [2018-11-21 15:24:27.370775]
and [2018-11-21 15:24:27.373796][2018-11-21 15:25:19.407885] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:25:21.371482] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:26:13.415390] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:25:21.371482]
and [2018-11-21 15:26:15.378777][2018-11-21 15:27:07.403685] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:27:09.371804] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:28:02.728185] E
[socket.c:2792:socket_poller] 0-socket.management: socket_poller
10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output
error)[2018-11-21 15:28:01.415055] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:27:09.371804]
and [2018-11-21 15:28:03.370301][2018-11-21 15:28:55.414118] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:28:57.375508] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol reqThe message "I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req" repeated 2 times between [2018-11-21
15:28:55.414118] and [2018-11-21 15:30:43.419479]The message "I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req" repeated 11 times between [2018-11-21
15:28:57.375508] and [2018-11-21 15:30:45.383263][2018-11-21
15:31:37.415788] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req[2018-11-21 15:31:39.375304] I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req[2018-11-21 15:32:31.415510] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:31:39.375304]
and [2018-11-21 15:32:33.375024][2018-11-21 15:33:25.407192] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:33:27.379564] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:34:20.746832] E
[socket.c:2792:socket_poller] 0-socket.management: socket_poller
10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output
error)[2018-11-21 15:34:19.411130] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:33:27.379564]
and [2018-11-21 15:34:21.368809][2018-11-21 15:35:13.415187] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:35:15.371647] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:36:00.958833] E [MSGID:
106115] [glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management:
Locking failed on fs-apollo-c1-n3. Please check log file for
details.[2018-11-21 15:36:00.960197] W
[glusterd-locks.c:622:glusterd_mgmt_v3_lock]
(-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43)
[0x7f206876cc43]
-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x30c73)
[0x7f206875dc73]
-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdb5d3)
[0x7f20688085d3] ) 0-management: Lock for archive1 held by
7e7fd30a-94c9-4c21-9c4b-3817508ef121[2018-11-21 15:36:00.960234] E [MSGID:
106118] [glusterd-op-sm.c:4173:glusterd_op_ac_lock] 0-management: Unable to
acquire lock for archive1[2018-11-21 15:36:00.960311] E [MSGID: 106376]
[glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler returned:
-1[2018-11-21 15:36:00.961494] E [MSGID: 106115]
[glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Locking
failed on fs-apollo-c1-n1. Please check log file for details.[2018-11-21
15:36:00.961603] E [MSGID: 106150]
[glusterd-syncop.c:1957:gd_sync_task_begin] 0-management: Locking Peers
Failed.[2018-11-21 15:36:00.962117] W
[glusterd-locks.c:856:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3fc43)
[0x7f206876cc43]
-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0x3099f)
[0x7f206875d99f]
-->/usr/lib64/glusterfs/4.1.5/xlator/mgmt/glusterd.so(+0xdba74)
[0x7f2068808a74] ) 0-management: Lock owner mismatch. Lock for vol archive1
held by 7e7fd30a-94c9-4c21-9c4b-3817508ef121[2018-11-21 15:36:00.962144] E
[MSGID: 106117] [glusterd-op-sm.c:4236:glusterd_op_ac_unlock] 0-management:
Unable to release lock for archive1[2018-11-21 15:36:00.962222] E [MSGID:
106376] [glusterd-op-sm.c:8305:glusterd_op_sm] 0-management: handler
returned: 1[2018-11-21 15:36:00.962406] E [MSGID: 106115]
[glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking
failed on fs-apollo-c1-n1. Please check log file for details.[2018-11-21
15:36:00.962973] E [MSGID: 106115]
[glusterd-mgmt.c:124:gd_mgmt_v3_collate_errors] 0-management: Unlocking
failed on fs-apollo-c1-n3. Please check log file for details.[2018-11-21
15:36:00.963033] E [MSGID: 106151]
[glusterd-syncop.c:1640:gd_unlock_op_phase] 0-management: Failed to unlock
on some peer(s)[2018-11-21 15:36:00.991734] W
[rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting
to connect to host:(null), port:0[2018-11-21 15:36:01.186043] W
[rpc-clnt.c:1753:rpc_clnt_submit] 0-scrub: error returned while attempting
to connect to host:(null), port:0[2018-11-21 15:36:07.416984] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 11 times between [2018-11-21
15:35:15.371647] and [2018-11-21 15:36:09.383236][2018-11-21
15:37:01.410979] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req[2018-11-21 15:37:03.383408] I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol reqThe message "I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req" repeated 2 times between [2018-11-21
15:37:01.410979] and [2018-11-21 15:38:49.410710]The message "I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req" repeated 11 times between [2018-11-21
15:37:03.383408] and [2018-11-21 15:38:51.378524][2018-11-21
15:39:43.421483] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list req[2018-11-21 15:39:44.786383] E
[socket.c:2792:socket_poller] 0-socket.management: socket_poller
10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output
error)[2018-11-21 15:39:45.373093] I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req[2018-11-21 15:40:37.412592] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:39:45.373093]
and [2018-11-21 15:40:39.378904][2018-11-21 15:41:31.415866] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:41:33.374993] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:42:25.415585] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:41:33.374993]
and [2018-11-21 15:42:27.379013][2018-11-21 15:43:19.411237] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:43:21.376718] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:44:13.425206] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:43:21.376718]
and [2018-11-21 15:44:15.375796][2018-11-21 15:45:07.424344] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:45:09.382299] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req[2018-11-21 15:46:02.769673] E
[socket.c:2792:socket_poller] 0-socket.management: socket_poller
10.10.12.64:49133 <http://10.10.12.64:49133> failed (Input/output
error)[2018-11-21 15:46:01.422742] I [MSGID: 106487]
[glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd:
Received cli list reqThe message "I [MSGID: 106488]
[glusterd-handler.c:1549:__glusterd_handle_cli_get_volume] 0-management:
Received get vol req" repeated 7 times between [2018-11-21 15:45:09.382299]
and [2018-11-21 15:46:03.370463][2018-11-21 15:46:55.426992] I [MSGID:
106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends]
0-glusterd: Received cli list req[2018-11-21 15:46:57.379488] I [MSGID:
106488] [glusterd-handler.c:1549:__glusterd_handle_cli_get_volume]
0-management: Received get vol req#->
/var/log/glusterfs/run-gluster-shared_storage.log[2018-11-21
15:22:48.623969] W [glusterfsd.c:1514:cleanup_and_exit]
(-->/lib64/libpthread.so.0(+0x7559) [0x7f9e25569559]
-->/usr/sbin/glusterfs(glusterfs_sigwaiter+0xfd) [0x561fd19c840d]
-->/usr/sbin/glusterfs(cleanup_and_exit+0x5a) [0x561fd19c81fa] ) 0-:
received signum (15), shutting down*
*[2018-11-21 15:22:48.624042] I [fuse-bridge.c:5981:fini] 0-fuse:
Unmounting '/run/gluster/shared_storage'.[2018-11-21 15:22:48.624185] E
[fuse-bridge.c:212:check_and_dump_fuse_W] 0-glusterfs-fuse: writing to fuse
device failed: Bad file descriptor[2018-11-21 15:22:48.624246] I
[fuse-bridge.c:5199:fuse_thread_proc] 0-fuse: initating unmount of
/run/gluster/shared_storage[2018-11-21 15:22:48.624679] I
[fuse-bridge.c:5986:fini] 0-fuse: Closing fuse connection to
'/run/gluster/shared_storage'.#->
/var/log/glusterfs/bricks/var-lib-glusterd-ss_brick.log[2018-11-21
15:22:48.626860] E [socket.c:2792:socket_poller]
0-tcp.gluster_shared_storage-server: socket_poller 10.10.12.62:49151
<http://10.10.12.62:49151> failed (Success)[2018-11-21 15:22:48.627048] I
[MSGID: 115036] [server.c:483:server_rpc_notify]
0-gluster_shared_storage-server: disconnecting connection from
CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21
15:22:48.627214] I [MSGID: 101055] [client_t.c:444:gf_client_unref]
0-gluster_shared_storage-server: Shutting down connection
CTX_ID:797e6f66-b69a-4045-bc43-c38d26a2c413-GRAPH_ID:0-PID:4447-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21
15:22:48.632303] W [socket.c:599:__socket_rwv] 0-glusterfs: readv on
10.10.12.62:24007 <http://10.10.12.62:24007> failed (Success)[2018-11-21
15:22:48.632531] I [glusterfsd-mgmt.c:2348:mgmt_rpc_notify]
0-glusterfsd-mgmt: disconnected from remote-host:
fs-apollo-c1-n2[2018-11-21 15:22:48.632558] I
[glusterfsd-mgmt.c:2369:mgmt_rpc_notify] 0-glusterfsd-mgmt: Exhausted all
volfile servers[2018-11-21 15:22:59.478134] I [MSGID: 115036]
[server.c:483:server_rpc_notify] 0-gluster_shared_storage-server:
disconnecting connection from
CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21
15:22:59.478287] I [MSGID: 101055] [client_t.c:444:gf_client_unref]
0-gluster_shared_storage-server: Shutting down connection
CTX_ID:87d36408-3140-457a-b844-bf3ee87f5595-GRAPH_ID:0-PID:4143-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0[2018-11-21
15:23:05.529596] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs:
No change in volfile,continuing[2018-11-21 15:23:05.625165] I
[addr.c:55:compare_addr_and_update] 0-/var/lib/glusterd/ss_brick: allowed =
"*", received addr = "10.10.12.62"[2018-11-21 15:23:05.625203] I
[login.c:45:gf_auth] 0-auth/login: connecting user name:
fs-apollo-c1-n2[2018-11-21 15:23:05.625238] I [login.c:111:gf_auth]
0-auth/login: allowed user names: *[2018-11-21 15:23:05.625254] I [MSGID:
115029] [server-handshake.c:763:server_setvolume]
0-gluster_shared_storage-server: accepted client from
CTX_ID:1c203d59-02da-46ec-b836-371ebb47e3dc-GRAPH_ID:0-PID:20295-HOST:fs-apollo-c1-n2-PC_NAME:gluster_shared_storage-client-0-RECON_NO:-0
(version: 4.1.5)[2018-11-21 15:36:01.009586] I
[glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed[2018-11-21
15:36:01.017077] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs:
No change in volfile,continuing[2018-11-21 15:36:01.206445] I
[glusterfsd-mgmt.c:53:mgmt_cbk_spec] 0-mgmt: Volume file changed[2018-11-21
15:36:01.212266] I [glusterfsd-mgmt.c:1953:mgmt_getspec_cbk] 0-glusterfs:
No change in volfile,continuing*
Post by David SpislaSure, I will do that the next time shared_storage is unmount after
stopping glusterd
Am Fr., 9. Nov. 2018 um 10:23 Uhr schrieb Rafi Kavungal Chundattu Parambil
Post by Rafi Kavungal Chundattu ParambilHi David Spisla,
gluster_shared_storage works just like yet another volume(There are some
special handling related to authentication). So bringing down a glusterd
node shouldn't have lead to unmount of the volume.
Can you please share the mount log, brick logs at time when you stopped
glusterd. Also make sure your shared storage bricks up and running.
Regards
Rafi KC
----- Original Message -----
Sent: Tuesday, November 6, 2018 3:25:18 PM
Subject: [Gluster-users] Shared Storage is unmount after stopping glusterd
Dear Gluster Community,
I have a v4.1.5 4-node-cluster with SLES15 machines. The last time I
observe that after stopping glusterd with
$ sudo systemctl stop glusterd
sometimes also the FUSE mount of the shared_storage on the same node is
unmount but shared_storage ist still in the volume list. I mean really
"sometimes" because it is not reproducible in a deterministic way and I am
not sure if there is really a dependency between stopping glusterd and
shared_storage.
What is the expected behaviour if stopping glusterd? In my opinion all
volumes included shared_storage should still have FUSE mounts.
Regards
David Spisla
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users