[Gluster-users] Gluster 3.12.14: slow ls -l command + "Another transaction is in progress. Please try again after sometime" message

[Mauro Tridici]

Dear All,

during last two days, our distributed dispersed gluster volume is showing some strange behaviors:

1) "df -h" and "ls -l" linux commands are very slow
2) “TAB” key usage causes the hang of the shell
3) nagios monitoring system is showing some alerts related to QUOTA and VOLUME UTILIZATION similar to the following one:

Notification Type: PROBLEM

Service: Volume Quota - tier2
Host: tier2-gluster
Address: tier2-gluster
State: WARNING

Date/Time: Wed Oct 24 16:27:13 CEST 2018

Additional Info:

QUOTA: Quota status could not be determined.

4) on some gluster servers, “df -h” command shows a new mount “/run/gluster/tier2_quota_list”
5) gluster vol status returns this message: “Another transaction is in progress. Please try again after sometime”.

I just restarted the “glusterd” service on each server, one by one.
Also, I executed “umount -l /run/gluster/tier2_quota_list” on each server.
Now, everything seems to be ok, but I would like to know if there is a way to avoid this kind of problem.

Last question: do you think that it is a good practice executing “service glusterd restart” command on each server when the problem appears?
In attachment you can find the glusterd.log file

Thank you in advance,
Mauro

[Ashish Pandey]

Hi,

Slow "ls -l" command which is also the reason behind TAB and df -h issue.
I would suggest to check "disperse.other-eager-lock" option and see if it is "ON" or "OFF"

gluster v get <volname> all | grep other

If it is ON, change it to OFF by following command
gluster v set <volname> disperse.other-eager-lock off

It should solve the problem.

---
Ashish




----- Original Message -----

From: "Mauro Tridici" <***@cmcc.it>
To: "gluster-users" <gluster-***@gluster.org>
Sent: Wednesday, October 24, 2018 8:20:48 PM
Subject: [Gluster-users] Gluster 3.12.14: slow ls -l command + "Another transaction is in progress. Please try again after sometime" message

Dear All,

during last two days, our distributed dispersed gluster volume is showing some strange behaviors:

1) "df -h" and "ls -l" linux commands are very slow
2) “TAB” key usage causes the hang of the shell
3) nagios monitoring system is showing some alerts related to QUOTA and VOLUME UTILIZATION similar to the following one:

Notification Type: PROBLEM

Service: Volume Quota - tier2
Host: tier2-gluster
Address: tier2-gluster
State: WARNING

Date/Time: Wed Oct 24 16:27:13 CEST 2018

Additional Info:

QUOTA: Quota status could not be determined.

4) on some gluster servers, “df -h” command shows a new mount “/run/gluster/tier2_quota_list”
5) gluster vol status returns this message: “Another transaction is in progress. Please try again after sometime”.

I just restarted the “glusterd” service on each server, one by one.
Also, I executed “umount -l /run/gluster/tier2_quota_list” on each server.
Now, everything seems to be ok, but I would like to know if there is a way to avoid this kind of problem.

Last question: do you think that it is a good practice executing “service glusterd restart” command on each server when the problem appears?
In attachment you can find the glusterd.log file

Thank you in advance,
Mauro







_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Mauro Tridici]

Hi Ashish,

"gluster v get tier2 all | grep other” command returned empty output.

In attachment you can find the output of “gluster v get tier2 all” command.

Thank you very much,
Mauro

Post by Ashish Pandey
Hi,
Slow "ls -l" command which is also the reason behind TAB and df -h issue.
I would suggest to check "disperse.other-eager-lock" option and see if it is "ON" or "OFF"
gluster v get <volname> all | grep other
If it is ON, change it to OFF by following command
gluster v set <volname> disperse.other-eager-lock off
It should solve the problem.
---
Ashish
Sent: Wednesday, October 24, 2018 8:20:48 PM
Subject: [Gluster-users] Gluster 3.12.14: slow ls -l command + "Another transaction is in progress. Please try again after sometime" message
Dear All,
1) "df -h" and "ls -l" linux commands are very slow
2) “TAB” key usage causes the hang of the shell
Notification Type: PROBLEM
Service: Volume Quota - tier2
Host: tier2-gluster
Address: tier2-gluster
State: WARNING
Date/Time: Wed Oct 24 16:27:13 CEST 2018
QUOTA: Quota status could not be determined.
4) on some gluster servers, “df -h” command shows a new mount “/run/gluster/tier2_quota_list”
5) gluster vol status returns this message: “Another transaction is in progress. Please try again after sometime”.
I just restarted the “glusterd” service on each server, one by one.
Also, I executed “umount -l /run/gluster/tier2_quota_list” on each server.
Now, everything seems to be ok, but I would like to know if there is a way to avoid this kind of problem.
Last question: do you think that it is a good practice executing “service glusterd restart” command on each server when the problem appears?
In attachment you can find the glusterd.log file
Thank you in advance,
Mauro
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users

[Ashish Pandey]

I think you are using an old version, release-3.12 which does not have his option.
This option is present in release-3.13, so you have to upgrade your gluster .

If you want to try with the current version then you have to disable eager-lock option
gluster v set <volname> disperse.eager-lock off

This will solve the issue but will impact performance of data operation on files.
Give it a try and see if it works and if that is the real problem.

---
Ashish

----- Original Message -----

From: "Mauro Tridici" <***@cmcc.it>
To: "Ashish Pandey" <***@redhat.com>
Cc: "gluster-users" <gluster-***@gluster.org>
Sent: Thursday, October 25, 2018 1:28:28 PM
Subject: Re: [Gluster-users] Gluster 3.12.14: slow ls -l command + "Another transaction is in progress. Please try again after sometime" message


Hi Ashish,

"gluster v get tier2 all | grep other” command returned empty output.

In attachment you can find the output of “gluster v get tier2 all” command.

Thank you very much,
Mauro







Il giorno 25 ott 2018, alle ore 09:11, Ashish Pandey < ***@redhat.com > ha scritto:


Hi,

Slow "ls -l" command which is also the reason behind TAB and df -h issue.
I would suggest to check "disperse.other-eager-lock" option and see if it is "ON" or "OFF"

gluster v get <volname> all | grep other

If it is ON, change it to OFF by following command
gluster v set <volname> disperse.other-eager-lock off

It should solve the problem.

---
Ashish




----- Original Message -----

From: "Mauro Tridici" < ***@cmcc.it >
To: "gluster-users" < gluster-***@gluster.org >
Sent: Wednesday, October 24, 2018 8:20:48 PM
Subject: [Gluster-users] Gluster 3.12.14: slow ls -l command + "Another transaction is in progress. Please try again after sometime" message

Dear All,

during last two days, our distributed dispersed gluster volume is showing some strange behaviors:

1) "df -h" and "ls -l" linux commands are very slow
2) “TAB” key usage causes the hang of the shell
3) nagios monitoring system is showing some alerts related to QUOTA and VOLUME UTILIZATION similar to the following one:

Notification Type: PROBLEM

Service: Volume Quota - tier2
Host: tier2-gluster
Address: tier2-gluster
State: WARNING

Date/Time: Wed Oct 24 16:27:13 CEST 2018

Additional Info:

QUOTA: Quota status could not be determined.

4) on some gluster servers, “df -h” command shows a new mount “/run/gluster/tier2_quota_list”
5) gluster vol status returns this message: “Another transaction is in progress. Please try again after sometime”.

I just restarted the “glusterd” service on each server, one by one.
Also, I executed “umount -l /run/gluster/tier2_quota_list” on each server.
Now, everything seems to be ok, but I would like to know if there is a way to avoid this kind of problem.

Last question: do you think that it is a good practice executing “service glusterd restart” command on each server when the problem appears?
In attachment you can find the glusterd.log file

Thank you in advance,
Mauro







_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users








_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org
https://lists.gluster.org/mailman/listinfo/gluster-users

[Mauro Tridici]

Thank you, Ashish.
Yes, you are right, I’m using the 3.12.14 release.

I’ll try to disable eager-lock option and I’ll check if it solves the issue.

Regards,
Mauro

Post by Ashish Pandey
I think you are using an old version, release-3.12 which does not have his option.
This option is present in release-3.13, so you have to upgrade your gluster .
If you want to try with the current version then you have to disable eager-lock option
gluster v set <volname> disperse.eager-lock off
This will solve the issue but will impact performance of data operation on files.
Give it a try and see if it works and if that is the real problem.
---
Ashish
Sent: Thursday, October 25, 2018 1:28:28 PM
Subject: Re: [Gluster-users] Gluster 3.12.14: slow ls -l command + "Another transaction is in progress. Please try again after sometime" message
Hi Ashish,
"gluster v get tier2 all | grep other” command returned empty output.
In attachment you can find the output of “gluster v get tier2 all” command.
Thank you very much,
Mauro
Hi,
Slow "ls -l" command which is also the reason behind TAB and df -h issue.
I would suggest to check "disperse.other-eager-lock" option and see if it is "ON" or "OFF"
gluster v get <volname> all | grep other
If it is ON, change it to OFF by following command
gluster v set <volname> disperse.other-eager-lock off
It should solve the problem.
---
Ashish
Sent: Wednesday, October 24, 2018 8:20:48 PM
Subject: [Gluster-users] Gluster 3.12.14: slow ls -l command + "Another transaction is in progress. Please try again after sometime" message
Dear All,
1) "df -h" and "ls -l" linux commands are very slow
2) “TAB” key usage causes the hang of the shell
Notification Type: PROBLEM
Service: Volume Quota - tier2
Host: tier2-gluster
Address: tier2-gluster
State: WARNING
Date/Time: Wed Oct 24 16:27:13 CEST 2018
QUOTA: Quota status could not be determined.
4) on some gluster servers, “df -h” command shows a new mount “/run/gluster/tier2_quota_list”
5) gluster vol status returns this message: “Another transaction is in progress. Please try again after sometime”.
I just restarted the “glusterd” service on each server, one by one.
Also, I executed “umount -l /run/gluster/tier2_quota_list” on each server.
Now, everything seems to be ok, but I would like to know if there is a way to avoid this kind of problem.
Last question: do you think that it is a good practice executing “service glusterd restart” command on each server when the problem appears?
In attachment you can find the glusterd.log file
Thank you in advance,
Mauro
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users