[Gluster-users] GlusterFS 4.1.3, Geo replication unable to setup

Discussion:

Nico van Royen

2018-09-06 09:24:14 UTC

Hello,

On our dev environment we want to test GeoReplication with GlusterFS 4.1 and every attempt so far fails.
For now, we don't care (yet) about running it as a non-root user (not using the mountbroker etc).

Installed packages, both on master and slaves:
[***@clrv0000110367 geo-replication]# rpm -qa | grep gluster
glusterfs-client-xlators-4.1.3-1.el7.x86_64
glusterfs-events-4.1.3-1.el7.x86_64
glusterfs-geo-replication-4.1.3-1.el7.x86_64
glusterfs-4.1.3-1.el7.x86_64
glusterfs-api-4.1.3-1.el7.x86_64
glusterfs-fuse-4.1.3-1.el7.x86_64
glusterfs-server-4.1.3-1.el7.x86_64
glusterfs-rdma-4.1.3-1.el7.x86_64
glusterfs-extra-xlators-4.1.3-1.el7.x86_64
glusterfs-libs-4.1.3-1.el7.x86_64
glusterfs-cli-4.1.3-1.el7.x86_64
python2-gluster-4.1.3-1.el7.x86_64
glusterfs-coreutils-0.2.0-1.el7.x86_64

Master volume setup:
# gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110367:/gluster/VOLUME2/export clrv0000110371:/gluster/VOLUME2/export clrv0000110389:/gluster/VOLUME2/export
# gluster v start VOLUME2
# gluster volume set all cluster.enable-shared-storage enable

Slave volume setup
# gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110605:/gluster/VOLUME2/export clrv0000110608:/gluster/VOLUME2/export clrv0000110606:/gluster/VOLUME2/export
# gluster v start VOLUME2
# gluster volume set all cluster.enable-shared-storage enable

On master server:
# ssh-keygen (accepting all defaults)
# ssh-copy-id clrv0000110605 (one of the slave servers)
# gluster-georep-sshkey generate
# gluster volume geo-replication VOLUME2 clrv0000110605.ic.ing.net::VOLUME2 create push-pem

Several seconds later, all of the glusterd instances on the master side crash, with /var/log/glusterfs/glusterd.log such as:

[2018-09-06 08:50:20.663584] W [MSGID: 106028] [glusterd-geo-rep.c:2568:glusterd_get_statefile_name] 0-management: Config file (/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf) missing. Looking for template config file (/var/lib/glusterd/geo-replication/gsyncd_template.conf) [No such file or directory]
[2018-09-06 08:50:20.663724] I [MSGID: 106294] [glusterd-geo-rep.c:2577:glusterd_get_statefile_name] 0-management: Using default config template(/var/lib/glusterd/geo-replication/gsyncd_template.conf).
[2018-09-06 08:50:24.072321] I [MSGID: 106494] [glusterd-handler.c:3024:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume VOLUME1
[2018-09-06 08:50:24.074876] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-09-06 08:50:24.744276] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2018-09-06 08:50:24.744443] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is stopped
[2018-09-06 08:50:24.744497] I [MSGID: 106599] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed
[2018-09-06 08:50:24.749139] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 40886
[2018-09-06 08:50:25.749748] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd service is stopped
[2018-09-06 08:50:25.750047] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting glustershd service
[2018-09-06 08:50:25.757036] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2018-09-06 08:50:25.757100] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: bitd service is stopped
[2018-09-06 08:50:25.757288] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2018-09-06 08:50:25.757330] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: scrub service is stopped
[2018-09-06 08:50:28.391332] I [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe2b1a) [0x7fae33053b1a] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe25e5) [0x7fae330535e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fae3e55f0c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh --volname=VOLUME2 is_push_pem=1,pub_file=/var/lib/glusterd/geo-replication/common_secret.pem.pub,slave_user=root,slave_ip=clrv0000110605,slave_vol=VOLUME2,ssh_port=22
[2018-09-06 08:50:31.901854] I [MSGID: 106327] [glusterd-geo-rep.c:2562:glusterd_get_statefile_name] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).
[2018-09-06 08:50:32.252426] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on clrv0000110389.ic.ing.net. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.
[2018-09-06 08:50:32.271707] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on clrv0000110371.ic.ing.net. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.
[2018-09-06 08:50:32.332909] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on clrv0000110372.ic.ing.net. Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.
[2018-09-06 08:50:35.690440] E [mem-pool.c:335:__gf_free] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e) [0x7fae32f9536e] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x20e10) [0x7fae32f91e10] -->/lib64/libglusterfs.so.0(__gf_free+0x104) [0x7fae3e5344f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t *)((char *)free_ptr + header->size)
The message "I [MSGID: 106584] [glusterd-handler.c:5904:__glusterd_handle_get_state] 0-management: Received request to get state for glusterd"
repeated 6 times between [2018-09-06 08:49:02.500390] and [2018-09-06 08:50:35.690241]
[2018-09-06 08:50:35.846286] I [MSGID: 106327] [glusterd-geo-rep.c:4482:glusterd_read_status_file] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).
[2018-09-06 08:50:36.144575] E [mem-pool.c:326:__gf_free] (-->/lib64/libglusterfs.so.0(+0x1a2c0) [0x7fae3e4fe2c0] -->/lib64/libglusterfs.so.0(data_destroy+0x5d) [0x7fae3e4fd92d] -->/lib64/libglusterfs.so.0(__gf_free+0xa4) [0x7fae3e534494] ) 0-: Assertion failed: GF_MEM_HEADER_MAGIC ==
header->magic
[2018-09-06 08:50:11.549220] E [MSGID: 106332] [glusterd-utils.c:12886:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for xattr-user-namespace-mode key
pending frames:
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 6
time of crash:
2018-09-06 08:50:36
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 4.1.3
/lib64/libglusterfs.so.0(+0x25920)[0x7fae3e509920]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fae3e513874]
/lib64/libc.so.6(+0x36280)[0x7fae3cb6e280]
/lib64/libc.so.6(gsignal+0x37)[0x7fae3cb6e207]
/lib64/libc.so.6(abort+0x148)[0x7fae3cb6f8f8]
/lib64/libc.so.6(+0x78cc7)[0x7fae3cbb0cc7]
/lib64/libc.so.6(+0x81429)[0x7fae3cbb9429]
/lib64/libglusterfs.so.0(data_destroy+0x5d)[0x7fae3e4fd92d]
/lib64/libglusterfs.so.0(+0x1a2c0)[0x7fae3e4fe2c0]
/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x22254)[0x7fae32f93254]
/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e)[0x7fae32f9536e]
/lib64/libglusterfs.so.0(+0x622b0)[0x7fae3e5462b0]
/lib64/libc.so.6(+0x47fc0)[0x7fae3cb7ffc0]
---------

Note that the gsyncd_template.conf file (as found in /var/lib/glusterd/geo-replication) is empty, and was (re)created when the gluster volume geo-replication command was issued. (is it supposed to be empty ?)

Anyone any idea's ?

Regards, Nico van Roijen

Kotresh Hiremath Ravishankar

2018-09-06 10:58:37 UTC

Permalink

Hi Nico,

The glusterd has crashed on this node. Please raise a bug with core file?

Please use the following tool [1] to setup geo-rep by bringing back the
glusterd
if you are finding it difficult with geo-rep setup steps and let us know if
if it still crashes?

[1] http://aravindavk.in/blog/introducing-georepsetup/

Post by Nico van Royen
Hello,
On our dev environment we want to test GeoReplication with GlusterFS 4.1
and every attempt so far fails.
For now, we don't care (yet) about running it as a non-root user (not
using the mountbroker etc).
glusterfs-client-xlators-4.1.3-1.el7.x86_64
glusterfs-events-4.1.3-1.el7.x86_64
glusterfs-geo-replication-4.1.3-1.el7.x86_64
glusterfs-4.1.3-1.el7.x86_64
glusterfs-api-4.1.3-1.el7.x86_64
glusterfs-fuse-4.1.3-1.el7.x86_64
glusterfs-server-4.1.3-1.el7.x86_64
glusterfs-rdma-4.1.3-1.el7.x86_64
glusterfs-extra-xlators-4.1.3-1.el7.x86_64
glusterfs-libs-4.1.3-1.el7.x86_64
glusterfs-cli-4.1.3-1.el7.x86_64
python2-gluster-4.1.3-1.el7.x86_64
glusterfs-coreutils-0.2.0-1.el7.x86_64
# gluster v create VOLUME2 replica 3 arbiter 1 transport tcp
clrv0000110367:/gluster/VOLUME2/export clrv0000110371:/gluster/VOLUME2/export
clrv0000110389:/gluster/VOLUME2/export
# gluster v start VOLUME2
# gluster volume set all cluster.enable-shared-storage enable
Slave volume setup
# gluster v create VOLUME2 replica 3 arbiter 1 transport tcp
clrv0000110605:/gluster/VOLUME2/export clrv0000110608:/gluster/VOLUME2/export
clrv0000110606:/gluster/VOLUME2/export
# gluster v start VOLUME2
# gluster volume set all cluster.enable-shared-storage enable
# ssh-keygen (accepting all defaults)
# ssh-copy-id clrv0000110605 (one of the slave servers)
# gluster-georep-sshkey generate
# gluster volume geo-replication VOLUME2 clrv0000110605.ic.ing.net::VOLUME2 create push-pem
Several seconds later, all of the glusterd instances on the master side
[2018-09-06 08:50:20.663584] W [MSGID: 106028] [glusterd-geo-rep.c:2568:glusterd_get_statefile_name]
0-management: Config file (/var/lib/glusterd/geo-replication/VOLUME2_
clrv0000110605_VOLUME2/gsyncd.conf) missing. Looking for template config
file (/var/lib/glusterd/geo-replication/gsyncd_template.conf) [No such
file or directory]
[2018-09-06 08:50:20.663724] I [MSGID: 106294] [glusterd-geo-rep.c:2577:glusterd_get_statefile_name]
0-management: Using default config template(/var/lib/glusterd/
geo-replication/gsyncd_template.conf).
[2018-09-06 08:50:24.072321] I [MSGID: 106494] [glusterd-handler.c:3024:__
glusterd_handle_cli_profile_volume] 0-management: Received volume profile
req for volume VOLUME1
[2018-09-06 08:50:24.074876] I [MSGID: 106487] [glusterd-handler.c:1486:__
glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-09-06 08:50:24.744276] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop]
0-management: nfs already stopped
[2018-09-06 08:50:24.744443] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop]
0-management: nfs service is stopped
[2018-09-06 08:50:24.744497] I [MSGID: 106599] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager]
0-management: nfs/server.so xlator is not installed
[2018-09-06 08:50:24.749139] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop]
0-management: Stopping glustershd daemon running in pid: 40886
[2018-09-06 08:50:25.749748] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop]
0-management: glustershd service is stopped
[2018-09-06 08:50:25.750047] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start]
0-management: Starting glustershd service
[2018-09-06 08:50:25.757036] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop]
0-management: bitd already stopped
[2018-09-06 08:50:25.757100] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop]
0-management: bitd service is stopped
[2018-09-06 08:50:25.757288] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop]
0-management: scrub already stopped
[2018-09-06 08:50:25.757330] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop]
0-management: scrub service is stopped
[2018-09-06 08:50:28.391332] I [run.c:241:runner_log]
(-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe2b1a)
[0x7fae33053b1a] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe25e5)
[0x7fae330535e5] -->/lib64/libglusterfs.so.0(runner_log+0x115)
[0x7fae3e55f0c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/
gsync-create/post/S56glusterd-geo-rep-create-post.sh --volname=VOLUME2
is_push_pem=1,pub_file=/var/lib/glusterd/geo-replication/
common_secret.pem.pub,slave_user=root,slave_ip=clrv0000110605,slave_vol=
VOLUME2,ssh_port=22
[2018-09-06 08:50:31.901854] I [MSGID: 106327] [glusterd-geo-rep.c:2562:glusterd_get_statefile_name]
0-management: Using passed config template(/var/lib/glusterd/
geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).
[2018-09-06 08:50:32.252426] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors]
Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does
not exist.
[2018-09-06 08:50:32.271707] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors]
Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does
not exist.
[2018-09-06 08:50:32.332909] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors]
Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does
not exist.
[2018-09-06 08:50:35.690440] E [mem-pool.c:335:__gf_free]
(-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e)
[0x7fae32f9536e] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x20e10)
[0x7fae32f91e10] -->/lib64/libglusterfs.so.0(__gf_free+0x104)
[0x7fae3e5344f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t
*)((char *)free_ptr + header->size)
The message "I [MSGID: 106584] [glusterd-handler.c:5904:__glusterd_handle_get_state]
0-management: Received request to get state for glusterd"
repeated 6 times between [2018-09-06 08:49:02.500390] and [2018-09-06 08:50:35.690241]
[2018-09-06 08:50:35.846286] I [MSGID: 106327] [glusterd-geo-rep.c:4482:glusterd_read_status_file]
0-management: Using passed config template(/var/lib/glusterd/
geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).
[2018-09-06 08:50:36.144575] E [mem-pool.c:326:__gf_free]
(-->/lib64/libglusterfs.so.0(+0x1a2c0) [0x7fae3e4fe2c0]
-->/lib64/libglusterfs.so.0(data_destroy+0x5d) [0x7fae3e4fd92d]
Assertion failed: GF_MEM_HEADER_MAGIC ==
header->magic
glusterd_get_value_for_vme_entry] 0-management: Failed to get option for
xattr-user-namespace-mode key
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 6
2018-09-06 08:50:36
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 4.1.3
/lib64/libglusterfs.so.0(+0x25920)[0x7fae3e509920]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fae3e513874]
/lib64/libc.so.6(+0x36280)[0x7fae3cb6e280]
/lib64/libc.so.6(gsignal+0x37)[0x7fae3cb6e207]
/lib64/libc.so.6(abort+0x148)[0x7fae3cb6f8f8]
/lib64/libc.so.6(+0x78cc7)[0x7fae3cbb0cc7]
/lib64/libc.so.6(+0x81429)[0x7fae3cbb9429]
/lib64/libglusterfs.so.0(data_destroy+0x5d)[0x7fae3e4fd92d]
/lib64/libglusterfs.so.0(+0x1a2c0)[0x7fae3e4fe2c0]
/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+
0x22254)[0x7fae32f93254]
/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+
0x2436e)[0x7fae32f9536e]
/lib64/libglusterfs.so.0(+0x622b0)[0x7fae3e5462b0]
/lib64/libc.so.6(+0x47fc0)[0x7fae3cb7ffc0]
---------
Note that the gsyncd_template.conf file (as found in /var/lib/glusterd/geo-replication)
is empty, and was (re)created when the gluster volume geo-replication
command was issued. (is it supposed to be empty ?)
Anyone any idea's ?
Regards, Nico van Roijen
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

--
Thanks and Regards,
Kotresh H R

Nico van Royen

2018-09-06 11:46:30 UTC

Permalink

Hi Kotresh,

Same result using the georepsetup tool.

We removed the volume (on the master side) and recreated it. (its empty anyway).

# gluster-georep-setup VOLUME2 clrv0000110605.ic.ing.net::VOLUME2
Geo-replication session will be established between VOLUME2 and clrv0000110605::VOLUME2
Root password of clrv0000110605 is required to complete the setup. NOTE: Password will not be stored.

***@clrv0000110605.ic.ing.net's password:
[ OK] clrv0000110605.ic.ing.net is Reachable(Port 22)
[ OK] SSH Connection established ***@clrv0000110605.ic.ing.net
[ OK] Master Volume and Slave Volume are compatible (Version: 4.1.3)
[ OK] Common secret pub file present at /var/lib/glusterd/geo-replication/common_secret.pem.pub
[ OK] common_secret.pem.pub file copied to clrv0000110605
[ OK] Master SSH Keys copied to all Up Slave nodes
[ OK] Updated Master SSH Keys to all Up Slave nodes authorized_keys file
[NOT OK] Failed to Establish Geo-replication Session
Connection failed. Please check if gluster daemon is operational.

Gluster deamon on all master servers crashed/stopped with similair entries in the logfile:

[2018-09-06 11:33:09.552529] I [MSGID: 106584] [glusterd-handler.c:5904:__glusterd_handle_get_state] 0-management: Received request to get state for glusterd
[2018-09-06 11:33:09.552667] E [mem-pool.c:335:__gf_free] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e) [0x7fb27220636e] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x20e10) [0x7fb272202e10] -->/lib64/libglusterfs.so.0(__gf_free+0x104) [0x7fb27d7a54f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t *)((char *)free_ptr + header->size)
[2018-09-06 11:33:09.583766] I [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe2b1a) [0x7fb2722c4b1a] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe25e5) [0x7fb2722c45e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fb27d7d00c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh --volname=VOLUME2 is_push_pem=0,pub_file=/var/lib/glusterd/geo-replication/common_secret.pem.pub,slave_user=root,slave_ip=clrv0000110605,slave_vol=VOLUME2,ssh_port=22
[2018-09-06 11:33:09.725093] I [MSGID: 106327] [glusterd-geo-rep.c:4482:glusterd_read_status_file] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).
[2018-09-06 11:33:10.020642] E [mem-pool.c:326:__gf_free] (-->/lib64/libglusterfs.so.0(+0x1a2c0) [0x7fb27d76f2c0] -->/lib64/libglusterfs.so.0(data_destroy+0x5d) [0x7fb27d76e92d] -->/lib64/libglusterfs.so.0(__gf_free+0xa4) [0x7fb27d7a5494] ) 0-: Assertion failed: GF_MEM_HEADER_MAGIC ==
header->magic
pending frames:
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
frame : type(0) op(0)
patchset: git://git.gluster.org/glusterfs.git
signal received: 6
time of crash:
2018-09-06 11:33:10
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 4.1.3
/lib64/libglusterfs.so.0(+0x25920)[0x7fb27d77a920]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fb27d784874]
/lib64/libc.so.6(+0x36280)[0x7fb27bddf280]
/lib64/libc.so.6(gsignal+0x37)[0x7fb27bddf207]
/lib64/libc.so.6(abort+0x148)[0x7fb27bde08f8]
/lib64/libc.so.6(+0x78cc7)[0x7fb27be21cc7]
/lib64/libc.so.6(+0x81429)[0x7fb27be2a429]
/lib64/libglusterfs.so.0(data_destroy+0x5d)[0x7fb27d76e92d]
/lib64/libglusterfs.so.0(+0x1a2c0)[0x7fb27d76f2c0]
/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x22254)[0x7fb272204254]
/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e)[0x7fb27220636e]
/lib64/libglusterfs.so.0(+0x622b0)[0x7fb27d7b72b0]
/lib64/libc.so.6(+0x47fc0)[0x7fb27bdf0fc0]
---------

Will try to see if we can enable coredumps (none have been created so far).

Thanks already, Nico van Roijen

Van: "Kotresh Hiremath Ravishankar" <***@redhat.com>
Aan: "Nico van Royen" <***@van-royen.nl>
Cc: "gluster-users" <gluster-***@gluster.org>
Verzonden: Donderdag 6 september 2018 12:58:37
Onderwerp: Re: [Gluster-users] GlusterFS 4.1.3, Geo replication unable to setup

Hi Nico,

The glusterd has crashed on this node. Please raise a bug with core file?

Please use the following tool [1] to setup geo-rep by bringing back the glusterd
if you are finding it difficult with geo-rep setup steps and let us know if if it still crashes?

[1] [ http://aravindavk.in/blog/introducing-georepsetup/ | http://aravindavk.in/blog/introducing-georepsetup/ ]

On Thu, Sep 6, 2018 at 2:54 PM, Nico van Royen < [ mailto:***@van-royen.nl | ***@van-royen.nl ] > wrote:

Hello,

On our dev environment we want to test GeoReplication with GlusterFS 4.1 and every attempt so far fails.
For now, we don't care (yet) about running it as a non-root user (not using the mountbroker etc).

Installed packages, both on master and slaves:
[***@clrv0000110367 geo-replication]# rpm -qa | grep gluster
glusterfs-client-xlators-4.1.3-1.el7.x86_64
glusterfs-events-4.1.3-1.el7.x86_64
glusterfs-geo-replication-4.1.3-1.el7.x86_64
glusterfs-4.1.3-1.el7.x86_64
glusterfs-api-4.1.3-1.el7.x86_64
glusterfs-fuse-4.1.3-1.el7.x86_64
glusterfs-server-4.1.3-1.el7.x86_64
glusterfs-rdma-4.1.3-1.el7.x86_64
glusterfs-extra-xlators-4.1.3-1.el7.x86_64
glusterfs-libs-4.1.3-1.el7.x86_64
glusterfs-cli-4.1.3-1.el7.x86_64
python2-gluster-4.1.3-1.el7.x86_64
glusterfs-coreutils-0.2.0-1.el7.x86_64

Master volume setup:
# gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110367:/gluster/VOLUME2/export clrv0000110371:/gluster/VOLUME2/export clrv0000110389:/gluster/VOLUME2/export
# gluster v start VOLUME2
# gluster volume set all cluster.enable-shared-storage enable

Slave volume setup
# gluster v create VOLUME2 replica 3 arbiter 1 transport tcp clrv0000110605:/gluster/VOLUME2/export clrv0000110608:/gluster/VOLUME2/export clrv0000110606:/gluster/VOLUME2/export
# gluster v start VOLUME2
# gluster volume set all cluster.enable-shared-storage enable

On master server:
# ssh-keygen (accepting all defaults)
# ssh-copy-id clrv0000110605 (one of the slave servers)
# gluster-georep-sshkey generate
# gluster volume geo-replication VOLUME2 [ http://clrv0000110605.ic.ing.net/ | clrv0000110605.ic.ing.net ] ::VOLUME2 create push-pem

Several seconds later, all of the glusterd instances on the master side crash, with /var/log/glusterfs/glusterd.log such as:

[2018-09-06 08:50:20.663584] W [MSGID: 106028] [glusterd-geo-rep.c:2568:glusterd_get_statefile_name] 0-management: Config file (/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf) missing. Looking for template config file (/var/lib/glusterd/geo-replication/gsyncd_template.conf) [No such file or directory]
[2018-09-06 08:50:20.663724] I [MSGID: 106294] [glusterd-geo-rep.c:2577:glusterd_get_statefile_name] 0-management: Using default config template(/var/lib/glusterd/geo-replication/gsyncd_template.conf).
[2018-09-06 08:50:24.072321] I [MSGID: 106494] [glusterd-handler.c:3024:__glusterd_handle_cli_profile_volume] 0-management: Received volume profile req for volume VOLUME1
[2018-09-06 08:50:24.074876] I [MSGID: 106487] [glusterd-handler.c:1486:__glusterd_handle_cli_list_friends] 0-glusterd: Received cli list req
[2018-09-06 08:50:24.744276] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: nfs already stopped
[2018-09-06 08:50:24.744443] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: nfs service is stopped
[2018-09-06 08:50:24.744497] I [MSGID: 106599] [glusterd-nfs-svc.c:82:glusterd_nfssvc_manager] 0-management: nfs/server.so xlator is not installed
[2018-09-06 08:50:24.749139] I [MSGID: 106568] [glusterd-proc-mgmt.c:87:glusterd_proc_stop] 0-management: Stopping glustershd daemon running in pid: 40886
[2018-09-06 08:50:25.749748] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: glustershd service is stopped
[2018-09-06 08:50:25.750047] I [MSGID: 106567] [glusterd-svc-mgmt.c:203:glusterd_svc_start] 0-management: Starting glustershd service
[2018-09-06 08:50:25.757036] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: bitd already stopped
[2018-09-06 08:50:25.757100] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: bitd service is stopped
[2018-09-06 08:50:25.757288] I [MSGID: 106131] [glusterd-proc-mgmt.c:83:glusterd_proc_stop] 0-management: scrub already stopped
[2018-09-06 08:50:25.757330] I [MSGID: 106568] [glusterd-svc-mgmt.c:235:glusterd_svc_stop] 0-management: scrub service is stopped
[2018-09-06 08:50:28.391332] I [run.c:241:runner_log] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe2b1a) [0x7fae33053b1a] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0xe25e5) [0x7fae330535e5] -->/lib64/libglusterfs.so.0(runner_log+0x115) [0x7fae3e55f0c5] ) 0-management: Ran script: /var/lib/glusterd/hooks/1/gsync-create/post/S56glusterd-geo-rep-create-post.sh --volname=VOLUME2 is_push_pem=1,pub_file=/var/lib/glusterd/geo-replication/common_secret.pem.pub,slave_user=root,slave_ip=clrv0000110605,slave_vol=VOLUME2,ssh_port=22
[2018-09-06 08:50:31.901854] I [MSGID: 106327] [glusterd-geo-rep.c:2562:glusterd_get_statefile_name] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).
[2018-09-06 08:50:32.252426] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on [ http://clrv0000110389.ic.ing.net/ | clrv0000110389.ic.ing.net ] . Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.
[2018-09-06 08:50:32.271707] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on [ http://clrv0000110371.ic.ing.net/ | clrv0000110371.ic.ing.net ] . Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.
[2018-09-06 08:50:32.332909] E [MSGID: 106152] [glusterd-syncop.c:113:gd_collate_errors] 0-glusterd: Staging failed on [ http://clrv0000110372.ic.ing.net/ | clrv0000110372.ic.ing.net ] . Error: Geo-replication session between VOLUME2 and clrv0000110605::VOLUME2 does not exist.
[2018-09-06 08:50:35.690440] E [mem-pool.c:335:__gf_free] (-->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e) [0x7fae32f9536e] -->/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x20e10) [0x7fae32f91e10] -->/lib64/libglusterfs.so.0(__gf_free+0x104) [0x7fae3e5344f4] ) 0-: Assertion failed: GF_MEM_TRAILER_MAGIC == *(uint32_t *)((char *)free_ptr + header->size)
The message "I [MSGID: 106584] [glusterd-handler.c:5904:__glusterd_handle_get_state] 0-management: Received request to get state for glusterd"
repeated 6 times between [2018-09-06 08:49:02.500390] and [2018-09-06 08:50:35.690241]
[2018-09-06 08:50:35.846286] I [MSGID: 106327] [glusterd-geo-rep.c:4482:glusterd_read_status_file] 0-management: Using passed config template(/var/lib/glusterd/geo-replication/VOLUME2_clrv0000110605_VOLUME2/gsyncd.conf).
[2018-09-06 08:50:36.144575] E [mem-pool.c:326:__gf_free] (-->/lib64/libglusterfs.so.0(+0x1a2c0) [0x7fae3e4fe2c0] -->/lib64/libglusterfs.so.0(data_destroy+0x5d) [0x7fae3e4fd92d] -->/lib64/libglusterfs.so.0(__gf_free+0xa4) [0x7fae3e534494] ) 0-: Assertion failed: GF_MEM_HEADER_MAGIC ==
header->magic
[2018-09-06 08:50:11.549220] E [MSGID: 106332] [glusterd-utils.c:12886:glusterd_get_value_for_vme_entry] 0-management: Failed to get option for xattr-user-namespace-mode key
pending frames:
frame : type(0) op(0)
patchset: git:// [ http://git.gluster.org/glusterfs.git | git.gluster.org/glusterfs.git ]
signal received: 6
time of crash:
2018-09-06 08:50:36
configuration details:
argp 1
backtrace 1
dlfcn 1
libpthread 1
llistxattr 1
setfsid 1
spinlock 1
epoll.h 1
xattr.h 1
st_atim.tv_nsec 1
package-string: glusterfs 4.1.3
/lib64/libglusterfs.so.0(+0x25920)[0x7fae3e509920]
/lib64/libglusterfs.so.0(gf_print_trace+0x334)[0x7fae3e513874]
/lib64/libc.so.6(+0x36280)[0x7fae3cb6e280]
/lib64/libc.so.6(gsignal+0x37)[0x7fae3cb6e207]
/lib64/libc.so.6(abort+0x148)[0x7fae3cb6f8f8]
/lib64/libc.so.6(+0x78cc7)[0x7fae3cbb0cc7]
/lib64/libc.so.6(+0x81429)[0x7fae3cbb9429]
/lib64/libglusterfs.so.0(data_destroy+0x5d)[0x7fae3e4fd92d]
/lib64/libglusterfs.so.0(+0x1a2c0)[0x7fae3e4fe2c0]
/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x22254)[0x7fae32f93254]
/usr/lib64/glusterfs/4.1.3/xlator/mgmt/glusterd.so(+0x2436e)[0x7fae32f9536e]
/lib64/libglusterfs.so.0(+0x622b0)[0x7fae3e5462b0]
/lib64/libc.so.6(+0x47fc0)[0x7fae3cb7ffc0]
---------

Note that the gsyncd_template.conf file (as found in /var/lib/glusterd/geo-replication) is empty, and was (re)created when the gluster volume geo-replication command was issued. (is it supposed to be empty ?)

Anyone any idea's ?

Regards, Nico van Roijen

_______________________________________________
Gluster-users mailing list
[ mailto:Gluster-***@gluster.org | Gluster-***@gluster.org ]
[ https://lists.gluster.org/mailman/listinfo/gluster-users | https://lists.gluster.org/mailman/listinfo/gluster-users ]

--
Thanks and Regards,
Kotresh H R