[Gluster-users] Problem adding replicated bricks on FreeBSD

Discussion:

Mark Staudinger

2018-04-25 18:02:15 UTC

Hi Folks,

I'm trying to debug an issue that I've found while attempting to qualify
GlusterFS for potential distributed storage projects on the FreeBSD-11.1
server platform - using the existing package of GlusterFS v3.11.1_4

The main issue I've encountered is that I cannot add new bricks while
setting/increasing the replica count.

If I create a replicated volume "poc" on two hosts, say s1:/gluster/1/poc
and s2:/gluster/1/poc, the volume is created properly and shows replicated
status, files are written to both volumes.

If I create a single volume: s1:/gluster/1/poc as a single / distributed
brick, and then try to run

gluster volume add-brick poc replica 2 s2:/gluster/1/poc

it will always fail (sometimes after a pause, sometimes not.) The only
error I'm seeing on the server hosting the new brick, aside from the
generic "Unable to add bricks" message, is like so:

I [MSGID: 106578]
[glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks] 0-management:
replica-count is set 2
I [MSGID: 106578]
[glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks] 0-management:
type is set 2, need to change it
E [MSGID: 106054]
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops] 0-management:
Failed to set extended attribute trusted.add-brick : Operation not
supported [Operation not supported]
E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]
0-glusterd: Unable to add bricks
E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn] 0-management:
Add-brick commit failed.

I was initially using ZFS and noted that ZFS on FreeBSD does not support
xattr, so I reverted to using UFS as the storage type for the brick, and
still encounter this behavior.

I also recompiled the port (again, GlusterFS v3.11.1) with the patch from
https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals
specifically with xattr handling in FreeBSD.

To recap - I'm able to create any type of volume (2 or 3-way replicated or
distributed), but I'm unable to add replicated bricks to a volume.

I was, however, able to add a second distributed brick ( gluster volume
add-brick poc s2:/gluster/1/poc ) - so the issue seems specific to adding
and/or changing the replica count while adding a new brick.

Please let me know if there are any other issues in addition to bug
#1452961 I should be aware of, or additional log or debug info I can
provide.

Best Regards,
Mark

Kaushal M

2018-04-26 16:00:30 UTC

Permalink

The log here seems to indicate the filesystem on the new brick being added
doesn't seem to support setting xattrs.
Maybe check the new brick again?

Post by Mark Staudinger
E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]
0-glusterd: Unable to add bricks
Add-brick commit failed.
I was initially using ZFS and noted that ZFS on FreeBSD does not support
xattr, so I reverted to using UFS as the storage type for the brick, and
still encounter this behavior.
I also recompiled the port (again, GlusterFS v3.11.1) with the patch from
https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals
specifically with xattr handling in FreeBSD.
To recap - I'm able to create any type of volume (2 or 3-way replicated or
distributed), but I'm unable to add replicated bricks to a volume.
I was, however, able to add a second distributed brick ( gluster volume
add-brick poc s2:/gluster/1/poc ) - so the issue seems specific to adding
and/or changing the replica count while adding a new brick.
Please let me know if there are any other issues in addition to bug
#1452961 I should be aware of, or additional log or debug info I can
provide.
Best Regards,
Mark
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Mark Staudinger

2018-04-27 19:40:09 UTC

Permalink

Post by Kaushal M

The log here seems to indicate the filesystem on the new brick being added
doesn't seem to support setting xattrs.
Maybe check the new brick again?

Hi Kaushal,

That's what the error message would indicate, but UFS2 does support
xattr so that's a bit unexpected. Since I'm able to add a distributed
brick, I can rule out basic setup issues.

All the same, I can see via 'truss'

extattr_set_link(0x8081ff890,0x1,0x803b27886,0x805258c40,0x400) ERR#45
'Operation not supported’

Is anyone else using a similar setup (FreeBSD >= 11.1-RELEASE with the
official pkg) that has tested adding bricks in this way? Are there any
changes to the newfs or mount settings required for this to work that
aren't obvious?

Cheers,
-=Mark

Mark Staudinger

2018-04-28 11:35:24 UTC

Permalink

Post by Mark Staudinger
Hi Folks,
I'm trying to debug an issue that I've found while attempting to
qualify GlusterFS for potential distributed storage projects on the
FreeBSD-11.1 server platform - using the existing package of GlusterFS
v3.11.1_4
The main issue I've encountered is that I cannot add new bricks while
setting/increasing the replica count.
If I create a replicated volume "poc" on two hosts, say
s1:/gluster/1/poc and s2:/gluster/1/poc, the volume is created
properly and shows replicated status, files are written to both volumes.
If I create a single volume: s1:/gluster/1/poc as a single /
distributed brick, and then try to run
gluster volume add-brick poc replica 2 s2:/gluster/1/poc
it will always fail (sometimes after a pause, sometimes not.) The
only error I'm seeing on the server hosting the new brick, aside from
I [MSGID: 106578]
[glusterd-brick-ops.c:1352:glusterd_op_perform_add_bricks]
0-management: replica-count is set 2
I [MSGID: 106578]
[glusterd-brick-ops.c:1362:glusterd_op_perform_add_bricks]
0-management: type is set 2, need to change it
E [MSGID: 106054]
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops]
Operation not supported [Operation not supported]
E [MSGID: 106074] [glusterd-brick-ops.c:2565:glusterd_op_add_brick]
0-glusterd: Unable to add bricks
E [MSGID: 106123] [glusterd-mgmt.c:311:gd_mgmt_v3_commit_fn]
0-management: Add-brick commit failed.
I was initially using ZFS and noted that ZFS on FreeBSD does not
support xattr, so I reverted to using UFS as the storage type for the
brick, and still encounter this behavior.
I also recompiled the port (again, GlusterFS v3.11.1) with the patch
from https://bugzilla.redhat.com/show_bug.cgi?id=1484246 as this deals
specifically with xattr handling in FreeBSD.
To recap - I'm able to create any type of volume (2 or 3-way
replicated or distributed), but I'm unable to add replicated bricks to
a volume.
I was, however, able to add a second distributed brick ( gluster
volume add-brick poc s2:/gluster/1/poc ) - so the issue seems specific
to adding and/or changing the replica count while adding a new brick.
Please let me know if there are any other issues in addition to bug
#1452961 I should be aware of, or additional log or debug info I can
provide.

Hi Folks,

As a follow-up to this, and it not making sense that xattrs would not be
supported on UFS2, I am attaching the 'truss' output of the extattr
syscalls from two operations. In both situations, the command was
initiated on server1, and traced on server2.

The first is a working "create-brick" issued from server1, traced from
server2: ( gluster volume create poc transport tcp replica 2
s1:/gluster/1/poc s2:/gluster/1/poc )

extattr_set_link(0x807a55440,0x1,0x803b24b9c,0x803b24bb3,0x8) = 8 (0x8)
extattr_delete_link(0x807a55440,0x1,0x803b24b9c) = 0 (0x0)
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b28065,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807dfcbc0,0x1,0x803b24c17,0x807dfbb40,0x10) ERR#87
'Attribute not found'
extattr_set_link(0x807a55440,0x1,0x803b24c17,0x807dfe250,0x10) = 16 (0x10)

And the second is from the not-working situation where the brick from s2
is being added to the volume ( gluster volume add-brick poc replica 2
s2:/gluster/1/poc commit )

extattr_set_link(0x80545c440,0x1,0x803b24b9c,0x803b24bb3,0x8) = 8 (0x8)
extattr_delete_link(0x80545c440,0x1,0x803b24b9c) = 0 (0x0)
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b28065,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_get_link(0x807bfdb60,0x1,0x803b24c17,0x807bfcae0,0x10) ERR#87
'Attribute not found'
extattr_set_link(0x80545c440,0x1,0x803b24c17,0x8051e9c20,0x10) = 16 (0x10)
extattr_set_link(0x80545c440,0x1,0x803b2774c,0x807bff068,0xc) = 12 (0xc)
write(6," 31: option afr-pending-xattr
poc-client-0,poc-client-1\n",64) = 64 (0x40)
extattr_set_link(0x807bff890,0x1,0x803b27886,0x80545d440,0x400) ERR#45
'Operation not supported'
write(6,"[2018-04-28 15:21:51.271551] E [MSGID: 106054]
[glusterd-utils.c:12974:glusterd_handle_replicate_brick_ops]
0-management: Failed to set extended attribute trusted.add-brick :
Operation not supported [Operation not supported]\n",225) = 225 (0xe1)

Cheers,
-=Mark