[Gluster-users] Can glusterd be restarted running on all nodes at once while clients are mounted?

[Jeevan Patnaik]

Hi,

I have different issues:

I have restarted glusterd service on my 72 nodes almost parallelly with
ansible while the gluster NFS clients are in mounted state

After that many of the gluster peers went to rejected state. In logs, I see
msg id 106010 stating that checksum doesn't match.

I'm confused which checksum is that and how is it changed after I restart.

I restarted because gluster volume status commands gives timeout. I have
tiering enabled on the volume and was trying to detach. And that too never
completed. The status shows only in progress even the tiered volume
contains only a few 100 8MB filese I created for testing.

my overall experience with gluster tiering is really bad :(

Besides, what's the best way to restore old state if something goes wrong?
Till now, I have been using no volfile at all.. I only use volume status
commands to configure my cluster. Do I need to use a volfile inorder to
restore something?

Gluster version is 3.12.15
I have checked the op Version on all nodes and they all are same.


Regards
Jeevanी

[Jeevan Patnaik]

Hi,

I understand something now:
I think glusterd should not be restarted on all nodes at once. And if this
true, can anyone provide technical explanation of how it effects the
checksum?
And it seems to fix the rejected hosts, I need to clear the
/var/lib/glusterd except gluster.info and start glusterd and peer probe
again.

Regards,
Jeevanी

Post by Jeevan Patnaik
Hi,
I have restarted glusterd service on my 72 nodes almost parallelly with
ansible while the gluster NFS clients are in mounted state
After that many of the gluster peers went to rejected state. In logs, I
see msg id 106010 stating that checksum doesn't match.
I'm confused which checksum is that and how is it changed after I restart.
I restarted because gluster volume status commands gives timeout. I have
tiering enabled on the volume and was trying to detach. And that too never
completed. The status shows only in progress even the tiered volume
contains only a few 100 8MB filese I created for testing.
my overall experience with gluster tiering is really bad :(
Besides, what's the best way to restore old state if something goes wrong?
Till now, I have been using no volfile at all.. I only use volume status
commands to configure my cluster. Do I need to use a volfile inorder to
restore something?
Gluster version is 3.12.15
I have checked the op Version on all nodes and they all are same.
Regards
Jeevanी

[Jeevan Patnaik]

Ah..I am able to differentiate the hosts which are commonly rejected.

It's the hosts that aren't serving any bricks. Is it a bad idea to keep a
host that's not serving any bricks in pool? Don't they in sync with the
other hosts? Regarding my previous assumption that all nodes shoudo be
restarted ate once, I guess it's okay.

Now, I just have to understand the issue with the tiering. Logs are not
helping.


Regards,
Jeevan.

Post by Jeevan Patnaik
Hi,
I have restarted glusterd service on my 72 nodes almost parallelly with
ansible while the gluster NFS clients are in mounted state
After that many of the gluster peers went to rejected state. In logs, I
see msg id 106010 stating that checksum doesn't match.
I'm confused which checksum is that and how is it changed after I restart.
I restarted because gluster volume status commands gives timeout. I have
tiering enabled on the volume and was trying to detach. And that too never
completed. The status shows only in progress even the tiered volume
contains only a few 100 8MB filese I created for testing.
my overall experience with gluster tiering is really bad :(
Besides, what's the best way to restore old state if something goes wrong?
Till now, I have been using no volfile at all.. I only use volume status
commands to configure my cluster. Do I need to use a volfile inorder to
restore something?
Gluster version is 3.12.15
I have checked the op Version on all nodes and they all are same.
Regards
Jeevanी

[Andreas Davour]

72 nodes!!???

Can the common wisdom come to the rescue here. Does this even work? Wont
the translator overhead make so many nodes scale terribly?

Are people building clusters that big, and getting any performance at
all?

/andreas

Post by Jeevan Patnaik
Hi,
I have restarted glusterd service on my 72 nodes almost parallelly with
ansible while the gluster NFS clients are in mounted state
After that many of the gluster peers went to rejected state. In logs, I see
msg id 106010 stating that checksum doesn't match.
I'm confused which checksum is that and how is it changed after I restart.
I restarted because gluster volume status commands gives timeout. I have
tiering enabled on the volume and was trying to detach. And that too never
completed. The status shows only in progress even the tiered volume
contains only a few 100 8MB filese I created for testing.
my overall experience with gluster tiering is really bad :(
Besides, what's the best way to restore old state if something goes wrong?
Till now, I have been using no volfile at all.. I only use volume status
commands to configure my cluster. Do I need to use a volfile inorder to
restore something?
Gluster version is 3.12.15
I have checked the op Version on all nodes and they all are same.
Regards
Jeevanी

--
"economics is a pseudoscience; the astrology of our time"
Kim Stanley Robinson

[Jeevan Patnaik]

Hi Andreas,

Before rebooting, I have tried some performance tuning inorder to prevent
timeout errors. As we have sufficient RAM and cpu power, I have
increased transport.listen-backlog in Kernel and syn_backlog and
max-connections in Kernel. So, I expected that it won't cause a problem.
Also the NFS clients are mounted but not being used. And all the nodes are
in same network.

My assumption was that some slowness in the beginning can be seen, which
will be resolved automatically.

Is it still a base idea to have 72 nodes and starting them at once?

Regards,
Jeevan.

Post by Andreas Davour
72 nodes!!???
Can the common wisdom come to the rescue here. Does this even work? Wont
the translator overhead make so many nodes scale terribly?
Are people building clusters that big, and getting any performance at
all?
/andreas

Post by Jeevan Patnaik
Hi,
I have restarted glusterd service on my 72 nodes almost parallelly with
ansible while the gluster NFS clients are in mounted state
After that many of the gluster peers went to rejected state. In logs, I

see

Post by Jeevan Patnaik
msg id 106010 stating that checksum doesn't match.
I'm confused which checksum is that and how is it changed after I

restart.

Post by Jeevan Patnaik
I restarted because gluster volume status commands gives timeout. I have
tiering enabled on the volume and was trying to detach. And that too

never

Post by Jeevan Patnaik
completed. The status shows only in progress even the tiered volume
contains only a few 100 8MB filese I created for testing.
my overall experience with gluster tiering is really bad :(
Besides, what's the best way to restore old state if something goes

wrong?

Post by Jeevan Patnaik
Till now, I have been using no volfile at all.. I only use volume status
commands to configure my cluster. Do I need to use a volfile inorder to
restore something?
Gluster version is 3.12.15
I have checked the op Version on all nodes and they all are same.
Regards
Jeevanी

--
"economics is a pseudoscience; the astrology of our time"
Kim Stanley Robinson

[Andreas Davour]

Post by Jeevan Patnaik
Hi Andreas,
Before rebooting, I have tried some performance tuning inorder to prevent
timeout errors. As we have sufficient RAM and cpu power, I have
increased transport.listen-backlog in Kernel and syn_backlog and
max-connections in Kernel. So, I expected that it won't cause a problem.
Also the NFS clients are mounted but not being used. And all the nodes are
in same network.
My assumption was that some slowness in the beginning can be seen, which
will be resolved automatically.
Is it still a base idea to have 72 nodes and starting them at once?

The reason for my question is that we have a cluster of 16 nodes, and
see excessive metadata ops slowdowns, and it seems to be at least
partially because of the size of the cluster.

/andreas

--
"economics is a pseudoscience; the astrology of our time"
Kim Stanley Robinson

[Serkan Çoban]

2500-3000 disks per cluster is maximum usable limit, after that almost
nothing works.
We are using 2700 disk cluster for cold storage with ec.
Be careful on heal operations, i see 1 week /8T heal throughput...

Post by Andreas Davour

Post by Jeevan Patnaik
Hi Andreas,
Before rebooting, I have tried some performance tuning inorder to prevent
timeout errors. As we have sufficient RAM and cpu power, I have
increased transport.listen-backlog in Kernel and syn_backlog and
max-connections in Kernel. So, I expected that it won't cause a problem.
Also the NFS clients are mounted but not being used. And all the nodes are
in same network.
My assumption was that some slowness in the beginning can be seen, which
will be resolved automatically.
Is it still a base idea to have 72 nodes and starting them at once?

The reason for my question is that we have a cluster of 16 nodes, and
see excessive metadata ops slowdowns, and it seems to be at least
partially because of the size of the cluster.
/andreas
--
"economics is a pseudoscience; the astrology of our time"
Kim Stanley Robinson
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users