[Gluster-users] geo-replication failure.

[Alvin Starr]

The brick contains  about 8T of live data using up about 61M inodes so
we have LOTS of directories with small files.

It took something like 45 days for the systems to sync the first time
and I believe it completed but some time over the last month or so it
has failed.

both sides are running glusterfs-3.8.9-1.el7.x86_64

What can I do to clear this error?


[2018-07-30 11:09:59.112396] I [monitor(monitor):267:monitor] Monitor:
------------------------------------------------------------
[2018-07-30 11:09:59.112936] I [monitor(monitor):268:monitor] Monitor:
starting gsyncd worker
[2018-07-30 11:09:59.291553] I [changelogagent(agent):73:__init__]
ChangelogAgent: Agent listining...
[2018-07-30 11:09:59.308311] I [gsyncd(/bricks/cc_us/data):736:main_i]
<top>: syncing: gluster://localhost:CC-US-EDOCS ->
ssh://***@archive2.vpn.domain.net:gluster://localhost:arch-CC-US-EDOCS
[2018-07-30 11:10:10.118316] I
[master(/bricks/cc_us/data):83:gmaster_builder] <top>: setting up xsync
change detection mode
[2018-07-30 11:10:10.119020] I [master(/bricks/cc_us/data):367:__init__]
_GMaster: using 'rsync' as the sync engine
[2018-07-30 11:10:10.120713] I
[master(/bricks/cc_us/data):83:gmaster_builder] <top>: setting up
changelog change detection mode
[2018-07-30 11:10:10.121114] I [master(/bricks/cc_us/data):367:__init__]
_GMaster: using 'rsync' as the sync engine
[2018-07-30 11:10:10.123591] I
[master(/bricks/cc_us/data):83:gmaster_builder] <top>: setting up
changeloghistory change detection mode
[2018-07-30 11:10:10.123972] I [master(/bricks/cc_us/data):367:__init__]
_GMaster: using 'rsync' as the sync engine
[2018-07-30 11:10:12.165112] I
[master(/bricks/cc_us/data):1251:register] _GMaster: xsync temp
directory:
/var/lib/misc/glusterfsd/CC-US-EDOCS/ssh%3A%2F%2Froot%4065.39.151.110%3Agluster%3A%2F%2F127.0.0.1%3Aarch-CC-US-EDOCS/0a70d065ebfb511403fa881adc1073e6/xsync
[2018-07-30 11:10:12.165685] I
[resource(/bricks/cc_us/data):1533:service_loop] GLUSTER: Register time:
1532949012
[2018-07-30 11:10:12.171063] I
[master(/bricks/cc_us/data):510:crawlwrap] _GMaster: primary master with
volume id 900656fd-3f13-4ba2-bf04-90832508566e ...
[2018-07-30 11:10:12.176691] I
[master(/bricks/cc_us/data):519:crawlwrap] _GMaster: crawl interval: 1
seconds
[2018-07-30 11:10:12.183186] I [master(/bricks/cc_us/data):1165:crawl]
_GMaster: starting history crawl... turns: 1, stime: (1531938320, 0),
etime: 1532949012
[2018-07-30 11:10:12.184734] E [repce(agent):117:worker] <top>: call
failed:
Traceback (most recent call last):
  File "/usr/libexec/glusterfs/python/syncdaemon/repce.py", line 113,
in worker
    res = getattr(self.obj, rmeth)(*in_data[2:])
  File "/usr/libexec/glusterfs/python/syncdaemon/changelogagent.py",
line 54, in history
    num_parallel)
  File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py",
line 100, in cl_history_changelog
    cls.raise_changelog_err()
  File "/usr/libexec/glusterfs/python/syncdaemon/libgfchangelog.py",
line 27, in raise_changelog_err
    raise ChangelogException(errn, os.strerror(errn))
ChangelogException: [Errno 2] No such file or directory
[2018-07-30 11:10:12.186373] E [repce(/bricks/cc_us/data):207:__call__]
RepceClient: call 12294:140093078464320:1532949012.18 (history) failed
on peer with ChangelogException
[2018-07-30 11:10:12.186680] E
[resource(/bricks/cc_us/data):1551:service_loop] GLUSTER: Changelog
History Crawl failed, [Errno 2] No such file or directory
[2018-07-30 11:10:12.187215] I
[syncdutils(/bricks/cc_us/data):220:finalize] <top>: exiting.
[2018-07-30 11:10:12.192742] I [repce(agent):92:service_loop]
RepceServer: terminating on reaching EOF.
[2018-07-30 11:10:12.193228] I [syncdutils(agent):220:finalize] <top>:
exiting.
[2018-07-30 11:10:13.142468] I [monitor(monitor):344:monitor] Monitor:
worker(/bricks/cc_us/data) died in startup phase