[Gluster-users] Glusterfs fuse uses system ports to mount remote bricks

[Canh Ngo]

Hi all,

We run a storage cluster using GlusterFS v3.10.12 on CentOS7. Clients
(CentOS) are using glusterfs 3.8.4.

We notice when clients mounts bricks of a volume, sometimes glusterfs uses
system ports (i.e. in port range 0-1024) to connect to remote glusterfsd
port. e.g:

Server:
tcp 0 0 10.165.210.169:49161 10.165.210.51:850
ESTABLISHED 32760/glusterfsd

Client:
tcp 0 0 10.165.210.51:850 10.165.210.169:49161
ESTABLISHED 25483/glusterfs

Thus, sometimes glusterfs occupies our system ports (e.g. 995, 179, 443,
etc), that causes other services cannot start.

Is is a bug or an expected behavior? I expect glusterfs should use IANA
private ports rather than system ports. Do you know if we can configure
glusterfs to use a specific port range?

​Thanks,
Canh Ngo.

[Milind Changire]

Post by Canh Ngo
Hi all,
We run a storage cluster using GlusterFS v3.10.12 on CentOS7. Clients
(CentOS) are using glusterfs 3.8.4.
We notice when clients mounts bricks of a volume, sometimes glusterfs uses
system ports (i.e. in port range 0-1024) to connect to remote glusterfsd
tcp 0 0 10.165.210.169:49161 10.165.210.51:850
ESTABLISHED 32760/glusterfsd
tcp 0 0 10.165.210.51:850 10.165.210.169:49161
ESTABLISHED 25483/glusterfs
Thus, sometimes glusterfs occupies our system ports (e.g. 995, 179, 443,
etc), that causes other services cannot start.
Is is a bug or an expected behavior? I expect glusterfs should use IANA
private ports rather than system ports. Do you know if we can configure
glusterfs to use a specific port range?
​Thanks,
Canh Ngo.
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Here's some info on the kernel (sysctl) tunables that you could tweak:
/proc/sys/net/ipv4/ip_local_port_range
/proc/sys/net/ipv4/ip_local_reserved_ports

Here's what networking/ip-sysctl.txt from the kernel documentation
directory says:
ip_local_port_range - 2 INTEGERS
Defines the local port range that is used by TCP and UDP to
choose the local port. The first number is the first, the
second the last local port number.
If possible, it is better these numbers have different parity.
(one even and one odd values)
The default values are 32768 and 60999 respectively.

ip_local_reserved_ports - list of comma separated ranges
Specify the ports which are reserved for known third-party
applications. These ports will not be used by automatic port
assignments (e.g. when calling connect() or bind() with port
number 0). Explicit port allocation behavior is unchanged.

The format used for both input and output is a comma separated
list of ranges (e.g. "1,2-4,10-10" for ports 1, 2, 3, 4 and
10). Writing to the file will clear all previously reserved
ports and update the current list with the one given in the
input.

Note that ip_local_port_range and ip_local_reserved_ports
settings are independent and both are considered by the kernel
when determining which ports are available for automatic port
assignments.

You can reserve ports which are not in the current
ip_local_port_range, e.g.:

$ cat /proc/sys/net/ipv4/ip_local_port_range
32000 60999
$ cat /proc/sys/net/ipv4/ip_local_reserved_ports
8080,9148

although this is redundant. However such a setting is useful
if later the port range is changed to a value that will
include the reserved ports.

Default: Empty

You could check the values of these files on your system and configure them
accordingly. Gluster specifically looks at
/proc/sys/net/ipv4/ip_local_reserved_ports to avoid assigning values from
the reserved port range. Alternatively, you could configure the system via
/etc/sysctl.conf to persist the settings across reboots:

net.ipv4.ip_local_reserved_ports
net.ipv4.ip_local_port_range

Hope this helps.

--
Milind

[Raghavendra Talur]

Try

gluster volume set VOLNAME client.bind-insecure on

and remount clients. If servers refuse connection, you might also have to
set server.allow-insecure to on.

Post by Milind Changire

Post by Canh Ngo
Hi all,
We run a storage cluster using GlusterFS v3.10.12 on CentOS7. Clients
(CentOS) are using glusterfs 3.8.4.
We notice when clients mounts bricks of a volume, sometimes glusterfs
uses system ports (i.e. in port range 0-1024) to connect to remote
tcp 0 0 10.165.210.169:49161 10.165.210.51:850
ESTABLISHED 32760/glusterfsd
tcp 0 0 10.165.210.51:850 10.165.210.169:49161
ESTABLISHED 25483/glusterfs
Thus, sometimes glusterfs occupies our system ports (e.g. 995, 179, 443,
etc), that causes other services cannot start.
Is is a bug or an expected behavior? I expect glusterfs should use IANA
private ports rather than system ports. Do you know if we can configure
glusterfs to use a specific port range?
​Thanks,
Canh Ngo.
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

/proc/sys/net/ipv4/ip_local_port_range
/proc/sys/net/ipv4/ip_local_reserved_ports
Here's what networking/ip-sysctl.txt from the kernel documentation
ip_local_port_range - 2 INTEGERS
Defines the local port range that is used by TCP and UDP to
choose the local port. The first number is the first, the
second the last local port number.
If possible, it is better these numbers have different parity.
(one even and one odd values)
The default values are 32768 and 60999 respectively.
ip_local_reserved_ports - list of comma separated ranges
Specify the ports which are reserved for known third-party
applications. These ports will not be used by automatic port
assignments (e.g. when calling connect() or bind() with port
number 0). Explicit port allocation behavior is unchanged.
The format used for both input and output is a comma separated
list of ranges (e.g. "1,2-4,10-10" for ports 1, 2, 3, 4 and
10). Writing to the file will clear all previously reserved
ports and update the current list with the one given in the
input.
Note that ip_local_port_range and ip_local_reserved_ports
settings are independent and both are considered by the kernel
when determining which ports are available for automatic port
assignments.
You can reserve ports which are not in the current
$ cat /proc/sys/net/ipv4/ip_local_port_range
32000 60999
$ cat /proc/sys/net/ipv4/ip_local_reserved_ports
8080,9148
although this is redundant. However such a setting is useful
if later the port range is changed to a value that will
include the reserved ports.
Default: Empty
You could check the values of these files on your system and configure
them accordingly. Gluster specifically looks at /proc/sys/net/ipv4/ip_local_reserved_ports
to avoid assigning values from the reserved port range. Alternatively, you
could configure the system via /etc/sysctl.conf to persist the settings
net.ipv4.ip_local_reserved_ports
net.ipv4.ip_local_port_range
Hope this helps.
--
Milind
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

[Canh Ngo]

Thank you for all your suggestions!

I choose to reserve some system ports for our services as it has less work
to do comparing to change Gluster cluster config: need to restart Gluster
node one by one and remount volumes of all clients.

Kind regards,
Canh Ngo.

Post by Raghavendra Talur
Try
gluster volume set VOLNAME client.bind-insecure on
and remount clients. If servers refuse connection, you might also have to
set server.allow-insecure to on.

Post by Milind Changire

Post by Canh Ngo
Hi all,
We run a storage cluster using GlusterFS v3.10.12 on CentOS7. Clients
(CentOS) are using glusterfs 3.8.4.
We notice when clients mounts bricks of a volume, sometimes glusterfs
uses system ports (i.e. in port range 0-1024) to connect to remote
tcp 0 0 10.165.210.169:49161 10.165.210.51:850
ESTABLISHED 32760/glusterfsd
tcp 0 0 10.165.210.51:850 10.165.210.169:49161
ESTABLISHED 25483/glusterfs
Thus, sometimes glusterfs occupies our system ports (e.g. 995, 179, 443,
etc), that causes other services cannot start.
Is is a bug or an expected behavior? I expect glusterfs should use IANA
private ports rather than system ports. Do you know if we can configure
glusterfs to use a specific port range?
​Thanks,
Canh Ngo.
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

/proc/sys/net/ipv4/ip_local_port_range
/proc/sys/net/ipv4/ip_local_reserved_ports
Here's what networking/ip-sysctl.txt from the kernel documentation
ip_local_port_range - 2 INTEGERS
Defines the local port range that is used by TCP and UDP to
choose the local port. The first number is the first, the
second the last local port number.
If possible, it is better these numbers have different parity.
(one even and one odd values)
The default values are 32768 and 60999 respectively.
ip_local_reserved_ports - list of comma separated ranges
Specify the ports which are reserved for known third-party
applications. These ports will not be used by automatic port
assignments (e.g. when calling connect() or bind() with port
number 0). Explicit port allocation behavior is unchanged.
The format used for both input and output is a comma separated
list of ranges (e.g. "1,2-4,10-10" for ports 1, 2, 3, 4 and
10). Writing to the file will clear all previously reserved
ports and update the current list with the one given in the
input.
Note that ip_local_port_range and ip_local_reserved_ports
settings are independent and both are considered by the kernel
when determining which ports are available for automatic port
assignments.
You can reserve ports which are not in the current
$ cat /proc/sys/net/ipv4/ip_local_port_range
32000 60999
$ cat /proc/sys/net/ipv4/ip_local_reserved_ports
8080,9148
although this is redundant. However such a setting is useful
if later the port range is changed to a value that will
include the reserved ports.
Default: Empty
You could check the values of these files on your system and configure
them accordingly. Gluster specifically looks at
/proc/sys/net/ipv4/ip_local_reserved_ports to avoid assigning values
from the reserved port range. Alternatively, you could configure the system
net.ipv4.ip_local_reserved_ports
net.ipv4.ip_local_port_range
Hope this helps.
--
Milind
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users