[Gluster-users] rdma.management: could not create QP [Permission denied]

[Thomas Simmons]

Hello All,

I'm having trouble mounting gluster volumes over RDMA. Everything seems to
work fine until I try to mount the gluster volume. At this point it seems
to crash glusterd and give me an error "could not create QP [Permission
Denied]. If I change change the transport to tcp, everything works as
expected. I am pretty new to Infiniband and RDMA, so this may very well be
something I'm doing wrong. That said, I was able to get NFSoRDMA (and
qperf) working on these same hosts, so I'm confident my hardware is
working. This is on CentOS 7.5 using gluster 3.12.15 from the CentOS
repository. I can't find anything regarding this specific error and any
help is much appreciated.

# rdma works...
[***@gluster1 ~]# qperf --use_bits_per_sec gluster2-ib rc_bi_bw
rc_bi_bw:
bw = 35.3 Gb/sec

# NFSoRDMA works....
[***@gluster1 data]# mount -t nfs gluster2-ib:/test /mnt/test -o
rdma,port=20049
[***@gluster1 ~]#
[***@gluster1 data]# df -h|grep root
/dev/mapper/centos_gluster1-root 33G 7.8G 26G 24% /
gluster2-ib:/test 33G 29G 4.1G 88% /mnt/test

[***@gluster1 ~]# gluster volume create data replica 3 transport rdma
gluster1-ib:/zp1/data vmh2-ib:/zp1/data vmh3-ib:/zp1/data force
volume create: data: success: please start the volume to access data
[***@gluster1 ~]#
[***@gluster1 ~]# gluster volume start data
volume start: data: success
[***@gluster1 ~]#
[***@gluster1 ~]# gluster volume info data

Volume Name: data
Type: Replicate
Volume ID: 6f32999c-2d20-4fc2-838b-82972171df7e
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: rdma
Bricks:
Brick1: gluster1-ib:/zp1/data
Brick2: gluster2-ib:/zp1/data
Brick3: gluster3-ib:/zp1/data
Options Reconfigured:
nfs.disable: on
performance.client-io-threads: off
[***@gluster1 ~]#
[***@gluster1 ~]# mount -t glusterfs -o transport=rdma gluster1:/data
/mnt/data
Mount failed. Please check the log file for more details.
[***@gluster1 ~]#
[***@gluster1 ~]# gluster volume info data
Connection failed. Please check if gluster daemon is operational.
[***@gluster1 ~]#
[***@gluster1 ~]# tail -n 30 /var/log/messages|grep glusterd
Nov 10 11:18:07 gluster1 glusterd[1881]: [2018-11-10 16:18:07.457818] C
[MSGID: 103021] [rdma.c:3263:gf_rdma_create_qp] 0-rdma.management:
rdma.management: could not create QP [Permission denied]
Nov 10 11:18:07 gluster1 glusterd[1881]: pending frames:
Nov 10 11:18:07 gluster1 glusterd[1881]: patchset: git://
git.gluster.org/glusterfs.git
Nov 10 11:18:07 gluster1 glusterd[1881]: signal received: 11
Nov 10 11:18:07 gluster1 glusterd[1881]: time of crash:
Nov 10 11:18:07 gluster1 glusterd[1881]: 2018-11-10 16:18:07
Nov 10 11:18:07 gluster1 glusterd[1881]: configuration details:
Nov 10 11:18:07 gluster1 glusterd[1881]: argp 1
Nov 10 11:18:07 gluster1 glusterd[1881]: backtrace 1
Nov 10 11:18:07 gluster1 glusterd[1881]: dlfcn 1
Nov 10 11:18:07 gluster1 glusterd[1881]: libpthread 1
Nov 10 11:18:07 gluster1 glusterd[1881]: llistxattr 1
Nov 10 11:18:07 gluster1 glusterd[1881]: setfsid 1
Nov 10 11:18:07 gluster1 glusterd[1881]: spinlock 1
Nov 10 11:18:07 gluster1 glusterd[1881]: epoll.h 1
Nov 10 11:18:07 gluster1 glusterd[1881]: xattr.h 1
Nov 10 11:18:07 gluster1 glusterd[1881]: st_atim.tv_nsec 1
Nov 10 11:18:07 gluster1 glusterd[1881]: package-string: glusterfs 3.12.15
Nov 10 11:18:07 gluster1 glusterd[1881]: ---------

[Mike Lykov]

Post by Thomas Simmons
Hello All,
I'm having trouble mounting gluster volumes over RDMA. Everything seems
to work fine until I try to mount the gluster volume. At this point it
seems to crash glusterd and give me an error "could not create QP
[Permission Denied].

Please see my post with subject "glusterd SIGSEGV crash when create
volume with transport=rdma" at 07.11.2018

I'm getting this errors&crashes too.

But I'm resolve it via creating SELinux module with audit2allow... and I
get worse situation %)

Now, when SELinux allow rule "Infiniband pkeys" and I start a RDMA
volume, server (peer) silently REBOOT with no traces..

I enable persistent journal storage, but after reboot I get NO records
about reboots : journalctl --last-boots show EMPTY list, no boots at all.
Usually it show 1 last boot... I think XFS lose some files when accident
reboot happens.

And more, XFS lose vol files (made it zero-size) and one of my peers
cannot start ALL bricks (from all volumes, not only RDMA, but also tcp).

[Mike Lykov]

Post by Thomas Simmons
Hello All,
I'm having trouble mounting gluster volumes over RDMA. Everything seems
to work fine until I try to mount the gluster volume. At this point it
seems to crash glusterd and give me an error "could not create QP

Are you tried to permit it in SELinux? after permitting are glusterfs
works as expected ?