Discussion:
[Gluster-users] Healing : No space left on device
Hoggins!
2018-05-02 09:49:44 UTC
Permalink
Hello list,

I have an issue on my Gluster cluster. It is composed of two data nodes
and an arbiter for all my volumes.

After having upgraded my bricks to gluster 3.12.9 (Fedora 27), this is
what I get :

    - on node 1, volumes won't start, and glusterd.log shows a lot of :
        [2018-05-02 09:46:06.267817] W
[glusterd-locks.c:843:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x22549)
[0x7f0047ae2549]
-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x2bdf0)
[0x7f0047aebdf0]
-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0xd8371)
[0x7f0047b98371] ) 0-management: Lock for vol thedude not held
        The message "W [MSGID: 106118]
[glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
not released for rom" repeated 3 times between [2018-05-02
09:45:57.262321] and [2018-05-02 09:46:06.267804]
        [2018-05-02 09:46:06.267826] W [MSGID: 106118]
[glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
not released for thedude


    - on node 2, volume are up but don't seem to be willing to correctly
heal. The logs show a lot of :
        [2018-05-02 09:23:01.054196] I [MSGID: 108026]
[afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-thedude-replicate-0:
performing entry selfheal on 4dc0ae36-c365-4fc7-b44c-d717392c7bd3
        [2018-05-02 09:23:01.222596] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-thedude-client-2: remote
operation failed. Path: <gfid:74ea4c57-61e5-4674-96e4-51356dd710db> [No
space left on device]


    - on arbiter, glustershd.log shows a lot of :
        [2018-05-02 09:44:54.619476] I [MSGID: 108026]
[afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0:
performing entry selfheal on 146a9a84-3db1-42ef-828e-0e4131af3667
        [2018-05-02 09:44:54.640276] E [MSGID: 114031]
[client-rpc-fops.c:295:client3_3_mkdir_cbk] 0-web-client-2: remote
operation failed. Path: <gfid:47b16567-9acc-454b-b20f-9821e6f1d420> [No
space left on device]
        [2018-05-02 09:44:54.657045] I [MSGID: 108026]
[afr-self-heal-entry.c:887:afr_selfheal_entry_do] 0-web-replicate-0:
performing entry selfheal on 9f9122ed-2794-4ed1-91db-be0c7fe89389
        [2018-05-02 09:47:09.121060] W [MSGID: 101088]
[common-utils.c:4166:gf_backtrace_save] 0-mailer-replicate-0: Failed to
save the backtrace.


The clients connecting to the cluster experience problems, such as
Gluster refusing to create files, etc.

I'm lost here, where should I start ?

    Thanks for your help !

        Hoggins!
Hoggins!
2018-05-02 10:39:58 UTC
Permalink
Oh, and *there is* space on the device where the brick's data is located.

    /dev/mapper/fedora-home   942G    868G   74G  93% /export
Post by Hoggins!
Hello list,
I have an issue on my Gluster cluster. It is composed of two data nodes
and an arbiter for all my volumes.
After having upgraded my bricks to gluster 3.12.9 (Fedora 27), this is
        [2018-05-02 09:46:06.267817] W
[glusterd-locks.c:843:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x22549)
[0x7f0047ae2549]
-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x2bdf0)
[0x7f0047aebdf0]
-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0xd8371)
[0x7f0047b98371] ) 0-management: Lock for vol thedude not held
        The message "W [MSGID: 106118]
[glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
not released for rom" repeated 3 times between [2018-05-02
09:45:57.262321] and [2018-05-02 09:46:06.267804]
        [2018-05-02 09:46:06.267826] W [MSGID: 106118]
[glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
not released for thedude
    - on node 2, volume are up but don't seem to be willing to correctly
        [2018-05-02 09:23:01.054196] I [MSGID: 108026]
performing entry selfheal on 4dc0ae36-c365-4fc7-b44c-d717392c7bd3
        [2018-05-02 09:23:01.222596] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-thedude-client-2: remote
operation failed. Path: <gfid:74ea4c57-61e5-4674-96e4-51356dd710db> [No
space left on device]
        [2018-05-02 09:44:54.619476] I [MSGID: 108026]
performing entry selfheal on 146a9a84-3db1-42ef-828e-0e4131af3667
        [2018-05-02 09:44:54.640276] E [MSGID: 114031]
[client-rpc-fops.c:295:client3_3_mkdir_cbk] 0-web-client-2: remote
operation failed. Path: <gfid:47b16567-9acc-454b-b20f-9821e6f1d420> [No
space left on device]
        [2018-05-02 09:44:54.657045] I [MSGID: 108026]
performing entry selfheal on 9f9122ed-2794-4ed1-91db-be0c7fe89389
        [2018-05-02 09:47:09.121060] W [MSGID: 101088]
[common-utils.c:4166:gf_backtrace_save] 0-mailer-replicate-0: Failed to
save the backtrace.
The clients connecting to the cluster experience problems, such as
Gluster refusing to create files, etc.
I'm lost here, where should I start ?
    Thanks for your help !
        Hoggins!
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Hoggins!
2018-05-03 09:58:42 UTC
Permalink
There are also free inodes on the disks of all the machines... don't
where to look to solve this. Any idea ?
Post by Hoggins!
Oh, and *there is* space on the device where the brick's data is located.
    /dev/mapper/fedora-home   942G    868G   74G  93% /export
Post by Hoggins!
Hello list,
I have an issue on my Gluster cluster. It is composed of two data nodes
and an arbiter for all my volumes.
After having upgraded my bricks to gluster 3.12.9 (Fedora 27), this is
        [2018-05-02 09:46:06.267817] W
[glusterd-locks.c:843:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x22549)
[0x7f0047ae2549]
-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x2bdf0)
[0x7f0047aebdf0]
-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0xd8371)
[0x7f0047b98371] ) 0-management: Lock for vol thedude not held
        The message "W [MSGID: 106118]
[glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
not released for rom" repeated 3 times between [2018-05-02
09:45:57.262321] and [2018-05-02 09:46:06.267804]
        [2018-05-02 09:46:06.267826] W [MSGID: 106118]
[glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
not released for thedude
    - on node 2, volume are up but don't seem to be willing to correctly
        [2018-05-02 09:23:01.054196] I [MSGID: 108026]
performing entry selfheal on 4dc0ae36-c365-4fc7-b44c-d717392c7bd3
        [2018-05-02 09:23:01.222596] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-thedude-client-2: remote
operation failed. Path: <gfid:74ea4c57-61e5-4674-96e4-51356dd710db> [No
space left on device]
        [2018-05-02 09:44:54.619476] I [MSGID: 108026]
performing entry selfheal on 146a9a84-3db1-42ef-828e-0e4131af3667
        [2018-05-02 09:44:54.640276] E [MSGID: 114031]
[client-rpc-fops.c:295:client3_3_mkdir_cbk] 0-web-client-2: remote
operation failed. Path: <gfid:47b16567-9acc-454b-b20f-9821e6f1d420> [No
space left on device]
        [2018-05-02 09:44:54.657045] I [MSGID: 108026]
performing entry selfheal on 9f9122ed-2794-4ed1-91db-be0c7fe89389
        [2018-05-02 09:47:09.121060] W [MSGID: 101088]
[common-utils.c:4166:gf_backtrace_save] 0-mailer-replicate-0: Failed to
save the backtrace.
The clients connecting to the cluster experience problems, such as
Gluster refusing to create files, etc.
I'm lost here, where should I start ?
    Thanks for your help !
        Hoggins!
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Nithya Balachandran
2018-05-03 10:50:26 UTC
Permalink
Hi,

We need some more information in order to debug this.
The version of Gluster you were running before the upgrade
The output of gluster volume info <volname>
The brick logs for the volume when the operation is performed.

Regards,
Nithya
Post by Hoggins!
Hello list,
I have an issue on my Gluster cluster. It is composed of two data nodes
and an arbiter for all my volumes.
After having upgraded my bricks to gluster 3.12.9 (Fedora 27), this is
[2018-05-02 09:46:06.267817] W
[glusterd-locks.c:843:glusterd_mgmt_v3_unlock]
(-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x22549)
[0x7f0047ae2549]
-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0x2bdf0)
[0x7f0047aebdf0]
-->/usr/lib64/glusterfs/3.12.9/xlator/mgmt/glusterd.so(+0xd8371)
[0x7f0047b98371] ) 0-management: Lock for vol thedude not held
The message "W [MSGID: 106118]
[glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
not released for rom" repeated 3 times between [2018-05-02
09:45:57.262321] and [2018-05-02 09:46:06.267804]
[2018-05-02 09:46:06.267826] W [MSGID: 106118]
[glusterd-handler.c:6342:__glusterd_peer_rpc_notify] 0-management: Lock
not released for thedude
- on node 2, volume are up but don't seem to be willing to correctly
[2018-05-02 09:23:01.054196] I [MSGID: 108026]
performing entry selfheal on 4dc0ae36-c365-4fc7-b44c-d717392c7bd3
[2018-05-02 09:23:01.222596] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-thedude-client-2: remote
operation failed. Path: <gfid:74ea4c57-61e5-4674-96e4-51356dd710db> [No
space left on device]
[2018-05-02 09:44:54.619476] I [MSGID: 108026]
performing entry selfheal on 146a9a84-3db1-42ef-828e-0e4131af3667
[2018-05-02 09:44:54.640276] E [MSGID: 114031]
[client-rpc-fops.c:295:client3_3_mkdir_cbk] 0-web-client-2: remote
operation failed. Path: <gfid:47b16567-9acc-454b-b20f-9821e6f1d420> [No
space left on device]
[2018-05-02 09:44:54.657045] I [MSGID: 108026]
performing entry selfheal on 9f9122ed-2794-4ed1-91db-be0c7fe89389
[2018-05-02 09:47:09.121060] W [MSGID: 101088]
[common-utils.c:4166:gf_backtrace_save] 0-mailer-replicate-0: Failed to
save the backtrace.
The clients connecting to the cluster experience problems, such as
Gluster refusing to create files, etc.
I'm lost here, where should I start ?
Thanks for your help !
Hoggins!
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Loading...