Discussion:
Snapshot auto-delete unmount problem
(too old to reply)
Gary Lloyd
2017-05-31 09:30:28 UTC
Permalink
Raw Message
Hi I am having a problem deleting snapshots, gluster is failing to unmount
them. I am running centos 7.3 with gluster-3.10.2-1

here is some log output:

[2017-05-31 09:21:39.961371] W [MSGID: 106057]
[glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find] 0-management:
Snap volume
331ec972f90d494d8a86dd4f69d718b7.glust01-li.run-gluster-snaps-331ec972f90d494d8a86dd4f69d718b7-brick1-b
not found [Invalid argument]
[2017-05-31 09:21:51.520811] W [MSGID: 106112]
[glusterd-snapshot.c:8128:glusterd_handle_snap_limit] 0-management:
Soft-limit (value = 27) of volume shares1 is reached. Deleting snapshot
Snap_GMT-2017.05.31-09.20.04.
[2017-05-31 09:21:51.531729] E [MSGID: 106095]
[glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 failed (No such
file or directory) [No such file or directory]
[2017-05-31 09:22:00.540373] E [MSGID: 106038]
[glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove] 0-management:
umount failed for path
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 (brick:
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b): No such file
or directory.
[2017-05-31 09:22:02.442048] W [MSGID: 106033]
[glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove] 0-management:
Failed to rmdir: /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/, err:
Directory not empty. More than one glusterd running on this node.
[Directory not empty]
[2017-05-31 09:22:02.443336] W [MSGID: 106039]
[glusterd-snapshot-utils.c:55:glusterd_snapobject_delete] 0-management:
Failed destroying lockof snap Snap_GMT-2017.05.31-09.20.04
[2017-05-31 09:22:02.444038] I [MSGID: 106144]
[glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b on port 49157



Can anyone help ?

Thanks


*Gary Lloyd*
________________________________________________
I.T. Systems:Keele University
Finance & IT Directorate
Keele:Staffs:IC1 Building:ST5 5NB:UK
________________________________________________
Mohammed Rafi K C
2017-05-31 11:38:44 UTC
Permalink
Raw Message
Can you give us more logs on this issue. Also by any chance did somebody
unmount the lv's in any cases ?


Regards

Rafi KC
Post by Gary Lloyd
Hi I am having a problem deleting snapshots, gluster is failing to
unmount them. I am running centos 7.3 with gluster-3.10.2-1
[2017-05-31 09:21:39.961371] W [MSGID: 106057]
[glusterd-snapshot-utils.c:410:glusterd_snap_volinfo_find]
0-management: Snap volume
331ec972f90d494d8a86dd4f69d718b7.glust01-li.run-gluster-snaps-331ec972f90d494d8a86dd4f69d718b7-brick1-b
not found [Invalid argument]
[2017-05-31 09:21:51.520811] W [MSGID: 106112]
Soft-limit (value = 27) of volume shares1 is reached. Deleting
snapshot Snap_GMT-2017.05.31-09.20.04.
[2017-05-31 09:21:51.531729] E [MSGID: 106095]
umounting /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1
failed (No such file or directory) [No such file or directory]
[2017-05-31 09:22:00.540373] E [MSGID: 106038]
[glusterd-snapshot.c:2895:glusterd_do_lvm_snapshot_remove]
0-management: umount failed for path
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b): No such
file or directory.
[2017-05-31 09:22:02.442048] W [MSGID: 106033]
Failed to rmdir: /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/,
err: Directory not empty. More than one glusterd running on this node.
[Directory not empty]
[2017-05-31 09:22:02.443336] W [MSGID: 106039]
[glusterd-snapshot-utils.c:55:glusterd_snapobject_delete]
0-management: Failed destroying lockof snap Snap_GMT-2017.05.31-09.20.04
[2017-05-31 09:22:02.444038] I [MSGID: 106144]
[glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b on port 49157
Can anyone help ?
Thanks
/Gary Lloyd/
________________________________________________
I.T. Systems:Keele University
Finance & IT Directorate
Keele:Staffs:IC1 Building:ST5 5NB:UK
________________________________________________
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Gary Lloyd
2017-05-31 13:46:39 UTC
Permalink
Raw Message
Here is the glusterd.log file as it has just happened with a manual removal
of a snapshot.
The lvs exist even after the snapshot gets removed. I'm also running samba
on these with shadowcopy.

Thanks

*Gary Lloyd*
________________________________________________
I.T. Systems:Keele University
Finance & IT Directorate
Keele:Staffs:IC1 Building:ST5 5NB:UK
+44 1782 733063 <%2B44%201782%20733073>
________________________________________________
Post by Mohammed Rafi K C
Can you give us more logs on this issue. Also by any chance did somebody
unmount the lv's in any cases ?
Regards
Rafi KC
Hi I am having a problem deleting snapshots, gluster is failing to unmount
them. I am running centos 7.3 with gluster-3.10.2-1
410:glusterd_snap_volinfo_find] 0-management: Snap volume
331ec972f90d494d8a86dd4f69d718b7.glust01-li.run-gluster-snaps-
331ec972f90d494d8a86dd4f69d718b7-brick1-b not found [Invalid argument]
[2017-05-31 09:21:51.520811] W [MSGID: 106112] [glusterd-snapshot.c:8128:glusterd_handle_snap_limit]
0-management: Soft-limit (value = 27) of volume shares1 is reached.
Deleting snapshot Snap_GMT-2017.05.31-09.20.04.
[2017-05-31 09:21:51.531729] E [MSGID: 106095] [glusterd-snapshot-utils.c:3359:glusterd_umount]
0-management: umounting /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1
failed (No such file or directory) [No such file or directory]
glusterd_do_lvm_snapshot_remove] 0-management: umount failed for path
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b): No such
file or directory.
[2017-05-31 09:22:02.442048] W [MSGID: 106033] [glusterd-snapshot.c:3094:glusterd_lvm_snapshot_remove]
0-management: Failed to rmdir: /run/gluster/snaps/
4f980da64dec424ba0b48d6d36c4c54e/, err: Directory not empty. More than
one glusterd running on this node. [Directory not empty]
[2017-05-31 09:22:02.443336] W [MSGID: 106039]
Failed destroying lockof snap Snap_GMT-2017.05.31-09.20.04
[2017-05-31 09:22:02.444038] I [MSGID: 106144] [glusterd-pmap.c:377:pmap_registry_remove]
0-pmap: removing brick /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b
on port 49157
Can anyone help ?
Thanks
*Gary Lloyd*
________________________________________________
I.T. Systems:Keele University
Finance & IT Directorate
Keele:Staffs:IC1 Building:ST5 5NB:UK
________________________________________________
_______________________________________________
Gary Lloyd
2017-07-13 10:03:37 UTC
Permalink
Raw Message
Incase anyone is interested this issue was caused by turning on brick
multiplexing.

Switching it off made the problem go away...

*Gary Lloyd*
________________________________________________
I.T. Systems:Keele University
Finance & IT Directorate
Keele:Staffs:IC1 Building:ST5 5NB:UK
+44 1782 733063 <%2B44%201782%20733073>
________________________________________________
Post by Gary Lloyd
Here is the glusterd.log file as it has just happened with a manual
removal of a snapshot.
The lvs exist even after the snapshot gets removed. I'm also running samba
on these with shadowcopy.
Thanks
*Gary Lloyd*
________________________________________________
I.T. Systems:Keele University
Finance & IT Directorate
Keele:Staffs:IC1 Building:ST5 5NB:UK
+44 1782 733063 <%2B44%201782%20733073>
________________________________________________
Post by Mohammed Rafi K C
Can you give us more logs on this issue. Also by any chance did somebody
unmount the lv's in any cases ?
Regards
Rafi KC
Hi I am having a problem deleting snapshots, gluster is failing to
unmount them. I am running centos 7.3 with gluster-3.10.2-1
[2017-05-31 09:21:39.961371] W [MSGID: 106057]
Snap volume 331ec972f90d494d8a86dd4f69d718b7.glust01-li.run-gluster-snap
s-331ec972f90d494d8a86dd4f69d718b7-brick1-b not found [Invalid argument]
[2017-05-31 09:21:51.520811] W [MSGID: 106112]
Soft-limit (value = 27) of volume shares1 is reached. Deleting snapshot
Snap_GMT-2017.05.31-09.20.04.
[2017-05-31 09:21:51.531729] E [MSGID: 106095]
[glusterd-snapshot-utils.c:3359:glusterd_umount] 0-management: umounting
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1 failed (No
such file or directory) [No such file or directory]
[2017-05-31 09:22:00.540373] E [MSGID: 106038]
umount failed for path /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1
No such file or directory.
[2017-05-31 09:22:02.442048] W [MSGID: 106033]
Failed to rmdir: /run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/,
err: Directory not empty. More than one glusterd running on this node.
[Directory not empty]
[2017-05-31 09:22:02.443336] W [MSGID: 106039]
Failed destroying lockof snap Snap_GMT-2017.05.31-09.20.04
[2017-05-31 09:22:02.444038] I [MSGID: 106144]
[glusterd-pmap.c:377:pmap_registry_remove] 0-pmap: removing brick
/run/gluster/snaps/4f980da64dec424ba0b48d6d36c4c54e/brick1/b on port 49157
Can anyone help ?
Thanks
*Gary Lloyd*
________________________________________________
I.T. Systems:Keele University
Finance & IT Directorate
Keele:Staffs:IC1 Building:ST5 5NB:UK
________________________________________________
_______________________________________________
Loading...