Discussion:
[Gluster-users] Slow write times to gluster disk
Pat Haley
2018-06-20 23:30:07 UTC
Permalink
Hi,

We were recently revisiting our problems with the slowness of gluster
writes
(http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html).
Specifically we were testing the suggestions in a recent post
(http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html).
The first two suggestions (specifying a negative-timeout in the mount
settings or adding rpc-auth-allow-insecure to glusterd.vol) did not
improve our performance, while setting "disperse.eager-lock off"
provided a tiny (5%) speed-up.

Some of the various tests we have tried earlier can be seen in the links
below.  Do any of the above observations suggest what we could try next
to either improve the speed or debug the issue?  Thanks

http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html

Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: ***@mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Raghavendra Gowdappa
2018-06-21 03:02:21 UTC
Permalink
For the case of reading from Glusterfs mount, read-ahead should help.
However, we've known issues with read-ahead[1][2]. To work around these,
can you try with,

1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off

2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of gluster
writes (http://lists.gluster.org/pipermail/gluster-users/2017-April
/030529.html). Specifically we were testing the suggestions in a recent
post (http://lists.gluster.org/pipermail/gluster-users/2018-March
/033699.html). The first two suggestions (specifying a negative-timeout
in the mount settings or adding rpc-auth-allow-insecure to glusterd.vol)
did not improve our performance, while setting "disperse.eager-lock off"
provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the links
below. Do any of the above observations suggest what we could try next to
either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Raghavendra Gowdappa
2018-06-21 03:03:42 UTC
Permalink
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should help.
However, we've known issues with read-ahead[1][2]. To work around these,
can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of gluster
writes (http://lists.gluster.org/pipermail/gluster-users/2017-April
/030529.html). Specifically we were testing the suggestions in a recent
post (http://lists.gluster.org/pipermail/gluster-users/2018-March
/033699.html). The first two suggestions (specifying a negative-timeout
in the mount settings or adding rpc-auth-allow-insecure to glusterd.vol)
did not improve our performance, while setting "disperse.eager-lock off"
provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the links
below. Do any of the above observations suggest what we could try next to
either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Raghavendra Gowdappa
2018-06-21 04:54:13 UTC
Permalink
For the case of writes to glusterfs mount,

I saw in earlier conversations that there are too many lookups, but small
number of writes. Since writes cached in write-behind would invalidate
metadata cache, lookups won't be absorbed by md-cache. I am wondering what
would results look like if we turn off performance.write-behind.

@Pat,

Can you set,

# gluster volume set <volname> performance.write-behind off

and redo the tests writing to glusterfs mount? Let us know about the
results you see.

regards,
Raghavendra
Post by Raghavendra Gowdappa
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should help.
However, we've known issues with read-ahead[1][2]. To work around these,
can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of gluster
writes (http://lists.gluster.org/pipermail/gluster-users/2017-April
/030529.html). Specifically we were testing the suggestions in a recent
post (http://lists.gluster.org/pipermail/gluster-users/2018-March
/033699.html). The first two suggestions (specifying a negative-timeout
in the mount settings or adding rpc-auth-allow-insecure to glusterd.vol)
did not improve our performance, while setting "disperse.eager-lock off"
provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the links
below. Do any of the above observations suggest what we could try next to
either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Raghavendra Gowdappa
2018-06-21 04:55:23 UTC
Permalink
Please note that these suggestions are for native fuse mount.
Post by Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups, but small
number of writes. Since writes cached in write-behind would invalidate
metadata cache, lookups won't be absorbed by md-cache. I am wondering what
would results look like if we turn off performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
and redo the tests writing to glusterfs mount? Let us know about the
results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should help.
However, we've known issues with read-ahead[1][2]. To work around these,
can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of gluster
writes (http://lists.gluster.org/pipermail/gluster-users/2017-April
/030529.html). Specifically we were testing the suggestions in a
recent post (http://lists.gluster.org/pipe
rmail/gluster-users/2018-March/033699.html). The first two suggestions
(specifying a negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our performance,
while setting "disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the
links below. Do any of the above observations suggest what we could try
next to either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Raghavendra Gowdappa
2018-06-21 05:01:35 UTC
Permalink
Post by Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups, but small
number of writes. Since writes cached in write-behind would invalidate
metadata cache, lookups won't be absorbed by md-cache. I am wondering what
would results look like if we turn off performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
Post by Raghavendra Gowdappa
and redo the tests writing to glusterfs mount? Let us know about the
results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should help.
However, we've known issues with read-ahead[1][2]. To work around these,
can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of gluster
writes (http://lists.gluster.org/pipermail/gluster-users/2017-April
/030529.html). Specifically we were testing the suggestions in a
recent post (http://lists.gluster.org/pipe
rmail/gluster-users/2018-March/033699.html). The first two suggestions
(specifying a negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our performance,
while setting "disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the
links below. Do any of the above observations suggest what we could try
next to either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Pat Haley
2018-06-21 15:11:17 UTC
Permalink
Hi Raghavendra,

Thanks for the suggestions.  Our technician will be in on Monday. We'll
test then and let you know the results.

One question I have, is the "group metadata-cache" option supposed to
directly impact the performance or is it to help collect data? If the
latter, where will the data be located?

Thanks again.

Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups,
but small number of writes. Since writes cached in write-behind
would invalidate metadata cache, lookups won't be absorbed by
md-cache. I am wondering what would results look like if we turn
off performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
and redo the tests writing to glusterfs mount? Let us know about
the results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead
should help. However, we've known issues with
read-ahead[1][2]. To work around these, can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
<https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
<https://bugzilla.redhat.com/show_bug.cgi?id=1214489>
Hi,
We were recently revisiting our problems with the
slowness of gluster writes
(http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
<http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>).
Specifically we were testing the suggestions in a
recent post
(http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html
<http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>).
The first two suggestions (specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not
improve our performance, while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be
seen in the links below.  Do any of the above
observations suggest what we could try next to either
improve the speed or debug the issue?  Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
<http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html>
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
<http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html>
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering  Phone:  (617) 253-6824
Dept. of Mechanical Engineering Fax:    (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: ***@mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Raghavendra Gowdappa
2018-06-22 11:51:26 UTC
Permalink
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions. Our technician will be in on Monday. We'll
test then and let you know the results.
One question I have, is the "group metadata-cache" option supposed to
directly impact the performance or is it to help collect data? If the
latter, where will the data be located?
It impacts performance.
Post by Pat Haley
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups, but small
number of writes. Since writes cached in write-behind would invalidate
metadata cache, lookups won't be absorbed by md-cache. I am wondering what
would results look like if we turn off performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
Post by Raghavendra Gowdappa
and redo the tests writing to glusterfs mount? Let us know about the
results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should help.
However, we've known issues with read-ahead[1][2]. To work around these,
can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of gluster
writes (http://lists.gluster.org/pipermail/gluster-users/2017-April
/030529.html). Specifically we were testing the suggestions in a
recent post (http://lists.gluster.org/pipe
rmail/gluster-users/2018-March/033699.html). The first two
suggestions (specifying a negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our performance,
while setting "disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the
links below. Do any of the above observations suggest what we could try
next to either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
Pat Haley
2018-06-25 21:51:43 UTC
Permalink
Hi Raghavendra,

Setting the performance.write-behind off had a small improvement on the
write speed (~3%),

We were unable to turn on "group metadata-cache".  When we try get
errors like

# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format not valid.

Was metadata-cache available for gluster 3.7.11? We ask because the
release notes for 3.11 mentions “Feature for metadata-caching/small file
performance is production ready.”
(https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/).

Do any of these results suggest anything?  If not, what further tests
would be useful?

Thanks

Pat
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions.  Our technician will be in on Monday. 
We'll test then and let you know the results.
One question I have, is the "group metadata-cache" option supposed
to directly impact the performance or is it to help collect data? 
If the latter, where will the data be located?
It impacts performance.
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many
lookups, but small number of writes. Since writes cached in
write-behind would invalidate metadata cache, lookups won't
be absorbed by md-cache. I am wondering what would results
look like if we turn off performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
and redo the tests writing to glusterfs mount? Let us know
about the results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa
For the case of reading from Glusterfs mount,
read-ahead should help. However, we've known issues
with read-ahead[1][2]. To work around these, can you
try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
<https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
<https://bugzilla.redhat.com/show_bug.cgi?id=1214489>
On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley
Hi,
We were recently revisiting our problems with the
slowness of gluster writes
(http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
<http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>).
Specifically we were testing the suggestions in a
recent post
(http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html
<http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>).
The first two suggestions (specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not
improve our performance, while setting
"disperse.eager-lock off" provided a tiny (5%)
speed-up.
Some of the various tests we have tried earlier
can be seen in the links below. Do any of the
above observations suggest what we could try next
to either improve the speed or debug the issue?
Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
<http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html>
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
<http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html>
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering    Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax:    (617)
253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: ***@mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Raghavendra Gowdappa
2018-06-26 01:39:15 UTC
Permalink
Post by Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a small improvement on the
write speed (~3%),
We were unable to turn on "group metadata-cache". When we try get errors
like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format not valid.
Was metadata-cache available for gluster 3.7.11? We ask because the
release notes for 3.11 mentions “Feature for metadata-caching/small file
performance is production ready.” (https://gluster.readthedocs.
io/en/latest/release-notes/3.11.0/).
Do any of these results suggest anything? If not, what further tests
would be useful?
Group metadata-cache is just a bunch of options one sets on a volume. So,
You can set them manually using gluster cli. Following are the options and
their values:

performance.md-cache-timeout=600
network.inode-lru-limit=50000
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions. Our technician will be in on Monday. We'll
test then and let you know the results.
One question I have, is the "group metadata-cache" option supposed to
directly impact the performance or is it to help collect data? If the
latter, where will the data be located?
It impacts performance.
Post by Pat Haley
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups, but
small number of writes. Since writes cached in write-behind would
invalidate metadata cache, lookups won't be absorbed by md-cache. I am
wondering what would results look like if we turn off
performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
Post by Raghavendra Gowdappa
and redo the tests writing to glusterfs mount? Let us know about the
results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should help.
However, we've known issues with read-ahead[1][2]. To work around these,
can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of gluster
writes (http://lists.gluster.org/pipermail/gluster-users/2017-April
/030529.html). Specifically we were testing the suggestions in a
recent post (http://lists.gluster.org/pipe
rmail/gluster-users/2018-March/033699.html). The first two
suggestions (specifying a negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to glusterd.vol) did not improve our performance,
while setting "disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the
links below. Do any of the above observations suggest what we could try
next to either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/0
31565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Pat Haley
2018-06-29 14:54:46 UTC
Permalink
Hi Raghavendra,

Our technician was able to try the manual setting today.  He found that
our upper limit for performance.md-cache-timeout was 60 not 600, so he
used that value, along with the network.inode-lru-limit=50000.

The result was another small (~1%) increase in speed.  Does this suggest
some addition tests/changes we could try?

Thanks

Pat
Post by Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a small improvement
on the write speed (~3%),
We were unable to turn on "group metadata-cache".  When we try get
errors like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format not valid.
Was metadata-cache available for gluster 3.7.11? We ask because
the release notes for 3.11 mentions “Feature for
metadata-caching/small file performance is production ready.”
(https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/
<https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/>).
Do any of these results suggest anything?  If not, what further
tests would be useful?
Group metadata-cache is just a bunch of options one sets on a volume.
So, You can set them manually using gluster cli. Following are the
performance.md-cache-timeout=600
network.inode-lru-limit=50000
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions.  Our technician will be in on
Monday.  We'll test then and let you know the results.
One question I have, is the "group metadata-cache" option
supposed to directly impact the performance or is it to help
collect data?  If the latter, where will the data be located?
It impacts performance.
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many
lookups, but small number of writes. Since writes cached
in write-behind would invalidate metadata cache, lookups
won't be absorbed by md-cache. I am wondering what would
results look like if we turn off performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
and redo the tests writing to glusterfs mount? Let us
know about the results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra
For the case of reading from Glusterfs mount,
read-ahead should help. However, we've known
issues with read-ahead[1][2]. To work around
these, can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname>
performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1]
https://bugzilla.redhat.com/show_bug.cgi?id=1084508
<https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
[2]
https://bugzilla.redhat.com/show_bug.cgi?id=1214489
<https://bugzilla.redhat.com/show_bug.cgi?id=1214489>
On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley
Hi,
We were recently revisiting our problems
with the slowness of gluster writes
(http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
<http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>).
Specifically we were testing the suggestions
in a recent post
(http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html
<http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>).
The first two suggestions (specifying a
negative-timeout in the mount settings or
adding rpc-auth-allow-insecure to
glusterd.vol) did not improve our
performance, while setting
"disperse.eager-lock off" provided a tiny
(5%) speed-up.
Some of the various tests we have tried
earlier can be seen in the links below. Do
any of the above observations suggest what
we could try next to either improve the
speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
<http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html>
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
<http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html>
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering    Phone: (617)
253-6824
Dept. of Mechanical Engineering Fax:   
(617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: ***@mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Raghavendra Gowdappa
2018-06-29 16:28:35 UTC
Permalink
Post by Pat Haley
Hi Raghavendra,
Our technician was able to try the manual setting today. He found that
our upper limit for performance.md-cache-timeout was 60 not 600, so he
used that value, along with the network.inode-lru-limit=50000.
The result was another small (~1%) increase in speed. Does this suggest
some addition tests/changes we could try?
Can you set gluster option diagnostics.client-log-level to TRACE and run
sequential read tests again (with md-cache-timeout value of 60)?

#gluster volume set <volname> diagnostics.client-log-level TRACE

Also are you sure that open-behind was turned off? Can you give the output
of,

# gluster volume info <volname>
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a small improvement on the
write speed (~3%),
We were unable to turn on "group metadata-cache". When we try get errors
like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format not valid.
Was metadata-cache available for gluster 3.7.11? We ask because the
release notes for 3.11 mentions “Feature for metadata-caching/small file
performance is production ready.” (https://gluster.readthedocs.i
o/en/latest/release-notes/3.11.0/).
Do any of these results suggest anything? If not, what further tests
would be useful?
Group metadata-cache is just a bunch of options one sets on a volume. So,
You can set them manually using gluster cli. Following are the options and
performance.md-cache-timeout=600
network.inode-lru-limit=50000
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions. Our technician will be in on Monday. We'll
test then and let you know the results.
One question I have, is the "group metadata-cache" option supposed to
directly impact the performance or is it to help collect data? If the
latter, where will the data be located?
It impacts performance.
Post by Pat Haley
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups, but
small number of writes. Since writes cached in write-behind would
invalidate metadata cache, lookups won't be absorbed by md-cache. I am
wondering what would results look like if we turn off
performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
Post by Raghavendra Gowdappa
and redo the tests writing to glusterfs mount? Let us know about the
results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should help.
However, we've known issues with read-ahead[1][2]. To work around these,
can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of
gluster writes (http://lists.gluster.org/pipe
rmail/gluster-users/2017-April/030529.html). Specifically we were
testing the suggestions in a recent post (
http://lists.gluster.org/pipermail/gluster-users/2018-March
/033699.html). The first two suggestions (specifying a
negative-timeout in the mount settings or adding rpc-auth-allow-insecure to
glusterd.vol) did not improve our performance, while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the
links below. Do any of the above observations suggest what we could try
next to either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/0
31565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/03
0937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Pat Haley
2018-06-29 17:08:41 UTC
Permalink
Hi Raghavendra,

We ran the tests (write tests) and I copied the log files for both the
server and the client to
http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/ .  Is
there any additional trace information you need?  (If so, where should I
look for it?)

Also the volume information you requested

[***@mseas-data2 ~]# gluster volume info data-volume

Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Bricks:
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
Options Reconfigured:
diagnostics.client-log-level: TRACE
network.inode-lru-limit: 50000
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
Post by Pat Haley
Hi Raghavendra,
Our technician was able to try the manual setting today.  He found
that our upper limit for performance.md-cache-timeout was 60 not
600, so he used that value, along with the
network.inode-lru-limit=50000.
The result was another small (~1%) increase in speed. Does this
suggest some addition tests/changes we could try?
Can you set gluster option diagnostics.client-log-level to TRACE  and
run sequential read tests again (with md-cache-timeout value of 60)?
#gluster volume set <volname> diagnostics.client-log-level TRACE
Also are you sure that open-behind was turned off? Can you give the
output of,
# gluster volume info <volname>
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a small
improvement on the write speed (~3%),
We were unable to turn on "group metadata-cache".  When we
try get errors like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format not valid.
Was metadata-cache available for gluster 3.7.11? We ask
because the release notes for 3.11 mentions “Feature for
metadata-caching/small file performance is production ready.”
(https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/
<https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/>).
Do any of these results suggest anything?  If not, what
further tests would be useful?
Group metadata-cache is just a bunch of options one sets on a
volume. So, You can set them manually using gluster cli.
performance.md-cache-timeout=600
network.inode-lru-limit=50000
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions.  Our technician will be in
on Monday.  We'll test then and let you know the results.
One question I have, is the "group metadata-cache"
option supposed to directly impact the performance or is
it to help collect data?  If the latter, where will the
data be located?
It impacts performance.
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too
many lookups, but small number of writes. Since
writes cached in write-behind would invalidate
metadata cache, lookups won't be absorbed by
md-cache. I am wondering what would results look
like if we turn off performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname>
performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
and redo the tests writing to glusterfs mount? Let
us know about the results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra
For the case of reading from Glusterfs
mount, read-ahead should help. However,
we've known issues with read-ahead[1][2].
To work around these, can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname>
performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group
metadata-cache
[1]
https://bugzilla.redhat.com/show_bug.cgi?id=1084508
<https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
[2]
https://bugzilla.redhat.com/show_bug.cgi?id=1214489
<https://bugzilla.redhat.com/show_bug.cgi?id=1214489>
On Thu, Jun 21, 2018 at 5:00 AM, Pat Haley
Hi,
We were recently revisiting our
problems with the slowness of gluster
writes
(http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
<http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>).
Specifically we were testing the
suggestions in a recent post
(http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html
<http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>).
The first two suggestions (specifying a
negative-timeout in the mount settings
or adding rpc-auth-allow-insecure to
glusterd.vol) did not improve our
performance, while setting
"disperse.eager-lock off" provided a
tiny (5%) speed-up.
Some of the various tests we have tried
earlier can be seen in the links below.
Do any of the above observations
suggest what we could try next to
either improve the speed or debug the
issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
<http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html>
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
<http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html>
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
(617) 253-6824
Dept. of Mechanical Engineering Fax:   
(617) 253-8125
MIT, Room 5-213
http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: ***@mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Raghavendra Gowdappa
2018-06-30 03:25:22 UTC
Permalink
Post by Pat Haley
Hi Raghavendra,
We ran the tests (write tests) and I copied the log files for both the
server and the client to http://mseas.mit.edu/download/
phaley/GlusterUsers/2018/Jun29/ . Is there any additional trace
information you need? (If so, where should I look for it?)
Nothing for now. I can see from logs that workaround is not helping. fstat
requests are not absorbed by md-cache and read-ahead is witnessing them and
flushing its read-ahead cache. I am investigating more on md-cache (It also
seems to be invalidating inodes quite frequently which actually might be
the root cause of seeing so many fstat requests from kernel). Will post
when I find anything relevant.
Post by Pat Haley
Also the volume information you requested
Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
diagnostics.client-log-level: TRACE
network.inode-lru-limit: 50000
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
Post by Pat Haley
Hi Raghavendra,
Our technician was able to try the manual setting today. He found that
our upper limit for performance.md-cache-timeout was 60 not 600, so he
used that value, along with the network.inode-lru-limit=50000.
The result was another small (~1%) increase in speed. Does this suggest
some addition tests/changes we could try?
Can you set gluster option diagnostics.client-log-level to TRACE and run
sequential read tests again (with md-cache-timeout value of 60)?
#gluster volume set <volname> diagnostics.client-log-level TRACE
Also are you sure that open-behind was turned off? Can you give the output
of,
# gluster volume info <volname>
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a small improvement on the
write speed (~3%),
We were unable to turn on "group metadata-cache". When we try get
errors like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format not valid.
Was metadata-cache available for gluster 3.7.11? We ask because the
release notes for 3.11 mentions “Feature for metadata-caching/small file
performance is production ready.” (https://gluster.readthedocs.i
o/en/latest/release-notes/3.11.0/).
Do any of these results suggest anything? If not, what further tests
would be useful?
Group metadata-cache is just a bunch of options one sets on a volume. So,
You can set them manually using gluster cli. Following are the options and
performance.md-cache-timeout=600
network.inode-lru-limit=50000
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions. Our technician will be in on Monday.
We'll test then and let you know the results.
One question I have, is the "group metadata-cache" option supposed to
directly impact the performance or is it to help collect data? If the
latter, where will the data be located?
It impacts performance.
Post by Pat Haley
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups, but
small number of writes. Since writes cached in write-behind would
invalidate metadata cache, lookups won't be absorbed by md-cache. I am
wondering what would results look like if we turn off
performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
Post by Raghavendra Gowdappa
and redo the tests writing to glusterfs mount? Let us know about the
results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should
help. However, we've known issues with read-ahead[1][2]. To work around
these, can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of
gluster writes (http://lists.gluster.org/pipe
rmail/gluster-users/2017-April/030529.html). Specifically we were
testing the suggestions in a recent post (
http://lists.gluster.org/pipermail/gluster-users/2018-March
/033699.html). The first two suggestions (specifying a
negative-timeout in the mount settings or adding rpc-auth-allow-insecure to
glusterd.vol) did not improve our performance, while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the
links below. Do any of the above observations suggest what we could try
next to either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/0
31565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/03
0937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Pat Haley
2018-07-05 23:59:58 UTC
Permalink
Hi Raghavendra,

Our technician may have some time to look at this issue tomorrow. Are
there any tests that you'd like to see?

Thanks

Pat
Post by Pat Haley
Hi Raghavendra,
We ran the tests (write tests) and I copied the log files for both
the server and the client to
http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/
<http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/> . 
Is there any additional trace information you need? (If so, where
should I look for it?)
Nothing for now. I can see from logs that workaround is not helping.
fstat requests are not absorbed by md-cache and read-ahead is
witnessing them and flushing its read-ahead cache. I am investigating
more on md-cache (It also seems to be invalidating inodes quite
frequently which actually might be the root cause of seeing so many
fstat requests from kernel). Will post when I find anything relevant.
Also the volume information you requested
Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
diagnostics.client-log-level: TRACE
network.inode-lru-limit: 50000
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
Post by Pat Haley
Hi Raghavendra,
Our technician was able to try the manual setting today.  He
found that our upper limit for performance.md-cache-timeout
was 60 not 600, so he used that value, along with the
network.inode-lru-limit=50000.
The result was another small (~1%) increase in speed.  Does
this suggest some addition tests/changes we could try?
Can you set gluster option diagnostics.client-log-level to TRACE 
and run sequential read tests again (with md-cache-timeout value
of 60)?
#gluster volume set <volname> diagnostics.client-log-level TRACE
Also are you sure that open-behind was turned off? Can you give
the output of,
# gluster volume info <volname>
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a small
improvement on the write speed (~3%),
We were unable to turn on "group metadata-cache".  When
we try get errors like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format
not valid.
Was metadata-cache available for gluster 3.7.11? We ask
because the release notes for 3.11 mentions “Feature for
metadata-caching/small file performance is production
ready.”
(https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/
<https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/>).
Do any of these results suggest anything?  If not, what
further tests would be useful?
Group metadata-cache is just a bunch of options one sets on
a volume. So, You can set them manually using gluster cli.
performance.md-cache-timeout=600
network.inode-lru-limit=50000
Thanks
Pat
On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley
Hi Raghavendra,
Thanks for the suggestions. Our technician will be
in on Monday.  We'll test then and let you know the
results.
One question I have, is the "group metadata-cache"
option supposed to directly impact the performance
or is it to help collect data? If the latter, where
will the data be located?
It impacts performance.
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are
too many lookups, but small number of writes.
Since writes cached in write-behind would
invalidate metadata cache, lookups won't be
absorbed by md-cache. I am wondering what
would results look like if we turn off
performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname>
performance.write-behind off
Please turn on "group metadata-cache" for write
tests too.
and redo the tests writing to glusterfs mount?
Let us know about the results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra
On Thu, Jun 21, 2018 at 8:32 AM,
For the case of reading from Glusterfs
mount, read-ahead should help.
However, we've known issues with
read-ahead[1][2]. To work around
these, can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname>
performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group
metadata-cache
[1]
https://bugzilla.redhat.com/show_bug.cgi?id=1084508
<https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
[2]
https://bugzilla.redhat.com/show_bug.cgi?id=1214489
<https://bugzilla.redhat.com/show_bug.cgi?id=1214489>
On Thu, Jun 21, 2018 at 5:00 AM, Pat
Hi,
We were recently revisiting our
problems with the slowness of
gluster writes
(http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
<http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>).
Specifically we were testing the
suggestions in a recent post
(http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html
<http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>).
The first two suggestions
(specifying a negative-timeout in
the mount settings or adding
rpc-auth-allow-insecure to
glusterd.vol) did not improve our
performance, while setting
"disperse.eager-lock off" provided
a tiny (5%) speed-up.
Some of the various tests we have
tried earlier can be seen in the
links below. Do any of the above
observations suggest what we could
try next to either improve the
speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
<http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html>
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
<http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html>
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering  
 Phone: (617) 253-6824
Dept. of Mechanical Engineering
Fax:    (617) 253-8125
MIT, Room 5-213
http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: ***@mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Raghavendra Gowdappa
2018-07-06 05:27:42 UTC
Permalink
Post by Pat Haley
Hi Raghavendra,
Our technician may have some time to look at this issue tomorrow. Are
there any tests that you'd like to see?
Sorry. I've been busy with other things and was away from work for couple
of days. It'll take me another 2 days to work on this issue again. So, most
likely you'll have an update on this next week.
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
We ran the tests (write tests) and I copied the log files for both the
server and the client to http://mseas.mit.edu/download/
phaley/GlusterUsers/2018/Jun29/ . Is there any additional trace
information you need? (If so, where should I look for it?)
Nothing for now. I can see from logs that workaround is not helping. fstat
requests are not absorbed by md-cache and read-ahead is witnessing them and
flushing its read-ahead cache. I am investigating more on md-cache (It also
seems to be invalidating inodes quite frequently which actually might be
the root cause of seeing so many fstat requests from kernel). Will post
when I find anything relevant.
Post by Pat Haley
Also the volume information you requested
Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
diagnostics.client-log-level: TRACE
network.inode-lru-limit: 50000
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
Post by Pat Haley
Hi Raghavendra,
Our technician was able to try the manual setting today. He found that
our upper limit for performance.md-cache-timeout was 60 not 600, so he
used that value, along with the network.inode-lru-limit=50000.
The result was another small (~1%) increase in speed. Does this suggest
some addition tests/changes we could try?
Can you set gluster option diagnostics.client-log-level to TRACE and run
sequential read tests again (with md-cache-timeout value of 60)?
#gluster volume set <volname> diagnostics.client-log-level TRACE
Also are you sure that open-behind was turned off? Can you give the
output of,
# gluster volume info <volname>
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a small improvement on the
write speed (~3%),
We were unable to turn on "group metadata-cache". When we try get
errors like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format not valid.
Was metadata-cache available for gluster 3.7.11? We ask because the
release notes for 3.11 mentions “Feature for metadata-caching/small file
performance is production ready.” (https://gluster.readthedocs.i
o/en/latest/release-notes/3.11.0/).
Do any of these results suggest anything? If not, what further tests
would be useful?
Group metadata-cache is just a bunch of options one sets on a volume.
So, You can set them manually using gluster cli. Following are the options
performance.md-cache-timeout=600
network.inode-lru-limit=50000
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions. Our technician will be in on Monday.
We'll test then and let you know the results.
One question I have, is the "group metadata-cache" option supposed to
directly impact the performance or is it to help collect data? If the
latter, where will the data be located?
It impacts performance.
Post by Pat Haley
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups, but
small number of writes. Since writes cached in write-behind would
invalidate metadata cache, lookups won't be absorbed by md-cache. I am
wondering what would results look like if we turn off
performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
Post by Raghavendra Gowdappa
and redo the tests writing to glusterfs mount? Let us know about the
results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should
help. However, we've known issues with read-ahead[1][2]. To work around
these, can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of
gluster writes (http://lists.gluster.org/pipe
rmail/gluster-users/2017-April/030529.html). Specifically we were
testing the suggestions in a recent post (
http://lists.gluster.org/pipermail/gluster-users/2018-March
/033699.html). The first two suggestions (specifying a
negative-timeout in the mount settings or adding rpc-auth-allow-insecure to
glusterd.vol) did not improve our performance, while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the
links below. Do any of the above observations suggest what we could try
next to either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/0
31565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/03
0937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Pat Haley
2018-07-12 23:30:58 UTC
Permalink
Hi Raghavendra,

We were wondering if you have had a chance to look at this again, and if
so, did you have any further suggestions?

Thanks

Pat
Post by Pat Haley
Hi Raghavendra,
Our technician may have some time to look at this issue tomorrow. 
Are there any tests that you'd like to see?
Sorry. I've been busy with other things and was away from work for
couple of days. It'll take me another 2 days to work on this issue
again. So, most likely you'll have an update on this next week.
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
We ran the tests (write tests) and I copied the log files for
both the server and the client to
http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/
<http://mseas.mit.edu/download/phaley/GlusterUsers/2018/Jun29/>
.  Is there any additional trace information you need?  (If
so, where should I look for it?)
Nothing for now. I can see from logs that workaround is not
helping. fstat requests are not absorbed by md-cache and
read-ahead is witnessing them and flushing its read-ahead cache.
I am investigating more on md-cache (It also seems to be
invalidating inodes quite frequently which actually might be the
root cause of seeing so many fstat requests from kernel). Will
post when I find anything relevant.
Also the volume information you requested
Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
diagnostics.client-log-level: TRACE
network.inode-lru-limit: 50000
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
Post by Pat Haley
Hi Raghavendra,
Our technician was able to try the manual setting
today.  He found that our upper limit for
performance.md-cache-timeout was 60 not 600, so he used
that value, along with the network.inode-lru-limit=50000.
The result was another small (~1%) increase in speed. 
Does this suggest some addition tests/changes we could try?
Can you set gluster option diagnostics.client-log-level to
TRACE  and run sequential read tests again (with
md-cache-timeout value of 60)?
#gluster volume set <volname> diagnostics.client-log-level TRACE
Also are you sure that open-behind was turned off? Can you
give the output of,
# gluster volume info <volname>
Thanks
Pat
On Tue, Jun 26, 2018 at 3:21 AM, Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a
small improvement on the write speed (~3%),
We were unable to turn on "group metadata-cache".
When we try get errors like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file
format not valid.
Was metadata-cache available for gluster 3.7.11? We
ask because the release notes for 3.11 mentions
“Feature for metadata-caching/small file
performance is production ready.”
(https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/
<https://gluster.readthedocs.io/en/latest/release-notes/3.11.0/>).
Do any of these results suggest anything?  If not,
what further tests would be useful?
Group metadata-cache is just a bunch of options one
sets on a volume. So, You can set them manually using
performance.md-cache-timeout=600
network.inode-lru-limit=50000
Thanks
Pat
On Thu, Jun 21, 2018 at 8:41 PM, Pat Haley
Hi Raghavendra,
Thanks for the suggestions. Our technician
will be in on Monday.  We'll test then and let
you know the results.
One question I have, is the "group
metadata-cache" option supposed to directly
impact the performance or is it to help
collect data? If the latter, where will the
data be located?
It impacts performance.
Thanks again.
Pat
On 06/21/2018 01:01 AM, Raghavendra Gowdappa
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra
For the case of writes to glusterfs mount,
I saw in earlier conversations that there
are too many lookups, but small number of
writes. Since writes cached in
write-behind would invalidate metadata
cache, lookups won't be absorbed by
md-cache. I am wondering what would
results look like if we turn off
performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname>
performance.write-behind off
Please turn on "group metadata-cache" for
write tests too.
and redo the tests writing to glusterfs
mount? Let us know about the results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM,
On Thu, Jun 21, 2018 at 8:32 AM,
Raghavendra Gowdappa
For the case of reading from
Glusterfs mount, read-ahead
should help. However, we've known
issues with read-ahead[1][2]. To
work around these, can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname>
performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname>
group metadata-cache
[1]
https://bugzilla.redhat.com/show_bug.cgi?id=1084508
<https://bugzilla.redhat.com/show_bug.cgi?id=1084508>
[2]
https://bugzilla.redhat.com/show_bug.cgi?id=1214489
<https://bugzilla.redhat.com/show_bug.cgi?id=1214489>
On Thu, Jun 21, 2018 at 5:00 AM,
Hi,
We were recently revisiting
our problems with the
slowness of gluster writes
(http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html
<http://lists.gluster.org/pipermail/gluster-users/2017-April/030529.html>).
Specifically we were testing
the suggestions in a recent
post
(http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html
<http://lists.gluster.org/pipermail/gluster-users/2018-March/033699.html>).
The first two suggestions
(specifying a
negative-timeout in the mount
settings or adding
rpc-auth-allow-insecure to
glusterd.vol) did not improve
our performance, while
setting "disperse.eager-lock
off" provided a tiny (5%)
speed-up.
Some of the various tests we
have tried earlier can be
seen in the links below. Do
any of the above observations
suggest what we could try
next to either improve the
speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html
<http://lists.gluster.org/pipermail/gluster-users/2017-June/031565.html>
http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html
<http://lists.gluster.org/pipermail/gluster-users/2017-May/030937.html>
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering
   Phone: (617) 253-6824
Dept. of Mechanical
Engineering Fax:    (617)
253-8125
MIT, Room 5-213
http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
<http://lists.gluster.org/mailman/listinfo/gluster-users>
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Pat Haley Email: ***@mit.edu
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Raghavendra Gowdappa
2018-07-14 02:07:20 UTC
Permalink
Post by Pat Haley
Hi Raghavendra,
We were wondering if you have had a chance to look at this again, and if
so, did you have any further suggestions?
Sorry Pat. Too much of work :). I'll be working on a patch today to make
sure read-ahead doesn't flush its cache due to fstats making this behavior
optional. You can try this patch and let us know about results.
Will let you know when patch is ready.
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Our technician may have some time to look at this issue tomorrow. Are
there any tests that you'd like to see?
Sorry. I've been busy with other things and was away from work for couple
of days. It'll take me another 2 days to work on this issue again. So, most
likely you'll have an update on this next week.
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
We ran the tests (write tests) and I copied the log files for both the
server and the client to http://mseas.mit.edu/download/
phaley/GlusterUsers/2018/Jun29/ . Is there any additional trace
information you need? (If so, where should I look for it?)
Nothing for now. I can see from logs that workaround is not helping.
fstat requests are not absorbed by md-cache and read-ahead is witnessing
them and flushing its read-ahead cache. I am investigating more on md-cache
(It also seems to be invalidating inodes quite frequently which actually
might be the root cause of seeing so many fstat requests from kernel). Will
post when I find anything relevant.
Post by Pat Haley
Also the volume information you requested
Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
diagnostics.client-log-level: TRACE
network.inode-lru-limit: 50000
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
Post by Pat Haley
Hi Raghavendra,
Our technician was able to try the manual setting today. He found that
our upper limit for performance.md-cache-timeout was 60 not 600, so he
used that value, along with the network.inode-lru-limit=50000.
The result was another small (~1%) increase in speed. Does this
suggest some addition tests/changes we could try?
Can you set gluster option diagnostics.client-log-level to TRACE and
run sequential read tests again (with md-cache-timeout value of 60)?
#gluster volume set <volname> diagnostics.client-log-level TRACE
Also are you sure that open-behind was turned off? Can you give the
output of,
# gluster volume info <volname>
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a small improvement on
the write speed (~3%),
We were unable to turn on "group metadata-cache". When we try get
errors like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format not valid.
Was metadata-cache available for gluster 3.7.11? We ask because the
release notes for 3.11 mentions “Feature for metadata-caching/small file
performance is production ready.” (https://gluster.readthedocs.i
o/en/latest/release-notes/3.11.0/).
Do any of these results suggest anything? If not, what further tests
would be useful?
Group metadata-cache is just a bunch of options one sets on a volume.
So, You can set them manually using gluster cli. Following are the options
performance.md-cache-timeout=600
network.inode-lru-limit=50000
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions. Our technician will be in on Monday.
We'll test then and let you know the results.
One question I have, is the "group metadata-cache" option supposed to
directly impact the performance or is it to help collect data? If the
latter, where will the data be located?
It impacts performance.
Post by Pat Haley
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups, but
small number of writes. Since writes cached in write-behind would
invalidate metadata cache, lookups won't be absorbed by md-cache. I am
wondering what would results look like if we turn off
performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
Post by Raghavendra Gowdappa
and redo the tests writing to glusterfs mount? Let us know about the
results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should
help. However, we've known issues with read-ahead[1][2]. To work around
these, can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of
gluster writes (http://lists.gluster.org/pipe
rmail/gluster-users/2017-April/030529.html). Specifically we
were testing the suggestions in a recent post (
http://lists.gluster.org/pipermail/gluster-users/2018-March
/033699.html). The first two suggestions (specifying a
negative-timeout in the mount settings or adding rpc-auth-allow-insecure to
glusterd.vol) did not improve our performance, while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in
the links below. Do any of the above observations suggest what we could
try next to either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/0
31565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/03
0937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Raghavendra Gowdappa
2018-07-14 13:04:01 UTC
Permalink
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi Raghavendra,
We were wondering if you have had a chance to look at this again, and if
so, did you have any further suggestions?
Sorry Pat. Too much of work :). I'll be working on a patch today to make
sure read-ahead doesn't flush its cache due to fstats making this behavior
optional. You can try this patch and let us know about results.
Will let you know when patch is ready.
Can you apply following two patches, build and remount your client and
redo sequential read tests?
https://review.gluster.org/#/q/status:open+project:glusterfs+branch:master+topic:ref-1455872

You can keep performance.open-behind on during your tests.

Let us know the results.
Post by Raghavendra Gowdappa
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Our technician may have some time to look at this issue tomorrow. Are
there any tests that you'd like to see?
Sorry. I've been busy with other things and was away from work for couple
of days. It'll take me another 2 days to work on this issue again. So, most
likely you'll have an update on this next week.
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
We ran the tests (write tests) and I copied the log files for both the
server and the client to http://mseas.mit.edu/download/
phaley/GlusterUsers/2018/Jun29/ . Is there any additional trace
information you need? (If so, where should I look for it?)
Nothing for now. I can see from logs that workaround is not helping.
fstat requests are not absorbed by md-cache and read-ahead is witnessing
them and flushing its read-ahead cache. I am investigating more on md-cache
(It also seems to be invalidating inodes quite frequently which actually
might be the root cause of seeing so many fstat requests from kernel). Will
post when I find anything relevant.
Post by Pat Haley
Also the volume information you requested
Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
diagnostics.client-log-level: TRACE
network.inode-lru-limit: 50000
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
Post by Pat Haley
Hi Raghavendra,
Our technician was able to try the manual setting today. He found that
our upper limit for performance.md-cache-timeout was 60 not 600, so he
used that value, along with the network.inode-lru-limit=50000.
The result was another small (~1%) increase in speed. Does this
suggest some addition tests/changes we could try?
Can you set gluster option diagnostics.client-log-level to TRACE and
run sequential read tests again (with md-cache-timeout value of 60)?
#gluster volume set <volname> diagnostics.client-log-level TRACE
Also are you sure that open-behind was turned off? Can you give the
output of,
# gluster volume info <volname>
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a small improvement on
the write speed (~3%),
We were unable to turn on "group metadata-cache". When we try get
errors like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format not valid.
Was metadata-cache available for gluster 3.7.11? We ask because the
release notes for 3.11 mentions “Feature for metadata-caching/small
file
performance is production ready.” (https://gluster.readthedocs.i
o/en/latest/release-notes/3.11.0/).
Do any of these results suggest anything? If not, what further tests
would be useful?
Group metadata-cache is just a bunch of options one sets on a volume.
So, You can set them manually using gluster cli. Following are the options
performance.md-cache-timeout=600
network.inode-lru-limit=50000
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions. Our technician will be in on Monday.
We'll test then and let you know the results.
One question I have, is the "group metadata-cache" option supposed to
directly impact the performance or is it to help collect data? If the
latter, where will the data be located?
It impacts performance.
Post by Pat Haley
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups, but
small number of writes. Since writes cached in write-behind would
invalidate metadata cache, lookups won't be absorbed by md-cache. I am
wondering what would results look like if we turn off
performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
Post by Raghavendra Gowdappa
and redo the tests writing to glusterfs mount? Let us know about the
results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should
help. However, we've known issues with read-ahead[1][2]. To work around
these, can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of
gluster writes (http://lists.gluster.org/pipe
rmail/gluster-users/2017-April/030529.html). Specifically we
were testing the suggestions in a recent post (
http://lists.gluster.org/pipermail/gluster-users/2018-March
/033699.html). The first two suggestions (specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to
glusterd.vol) did not improve our performance, while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in
the links below. Do any of the above observations suggest what
we could
try next to either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/0
31565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/03
0937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213
http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Raghavendra Gowdappa
2018-07-15 01:59:42 UTC
Permalink
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi Raghavendra,
We ran the tests (write tests) and I copied the log files for both the
server and the client to http://mseas.mit.edu/download/
phaley/GlusterUsers/2018/Jun29/ . Is there any additional trace
information you need? (If so, where should I look for it?)
Nothing for now. I can see from logs that workaround is not helping. fstat
requests are not absorbed by md-cache and read-ahead is witnessing them and
flushing its read-ahead cache. I am investigating more on md-cache (It also
seems to be invalidating inodes quite frequently which actually might be
the root cause of seeing so many fstat requests from kernel). Will post
when I find anything relevant.
+Poornima.

@Poornima,

Can you investigate why fstats sent by kernel are not absorbed by
md-cache in sequential read tests? Note that md-cache doesn't flush
its metadata cache on reads (which can be a bug for applications
requiring strict atime consistency). So, I am expecting fstats
should've absorbed by it.

regards,
Raghavendra
Post by Raghavendra Gowdappa
Post by Pat Haley
Also the volume information you requested
Volume Name: data-volume
Type: Distribute
Volume ID: c162161e-2a2d-4dac-b015-f31fd89ceb18
Status: Started
Number of Bricks: 2
Transport-type: tcp
Brick1: mseas-data2:/mnt/brick1
Brick2: mseas-data2:/mnt/brick2
diagnostics.client-log-level: TRACE
network.inode-lru-limit: 50000
performance.md-cache-timeout: 60
performance.open-behind: off
disperse.eager-lock: off
auth.allow: *
server.allow-insecure: on
nfs.exports-auth-enable: on
diagnostics.brick-sys-log-level: WARNING
performance.readdir-ahead: on
nfs.disable: on
nfs.export-volumes: off
Post by Pat Haley
Hi Raghavendra,
Our technician was able to try the manual setting today. He found that
our upper limit for performance.md-cache-timeout was 60 not 600, so he
used that value, along with the network.inode-lru-limit=50000.
The result was another small (~1%) increase in speed. Does this suggest
some addition tests/changes we could try?
Can you set gluster option diagnostics.client-log-level to TRACE and run
sequential read tests again (with md-cache-timeout value of 60)?
#gluster volume set <volname> diagnostics.client-log-level TRACE
Also are you sure that open-behind was turned off? Can you give the output
of,
# gluster volume info <volname>
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Setting the performance.write-behind off had a small improvement on the
write speed (~3%),
We were unable to turn on "group metadata-cache". When we try get
errors like
# gluster volume set data-volume group metadata-cache
'/var/lib/glusterd/groups/metadata-cache' file format not valid.
Was metadata-cache available for gluster 3.7.11? We ask because the
release notes for 3.11 mentions “Feature for metadata-caching/small
file
performance is production ready.” (https://gluster.readthedocs.i
o/en/latest/release-notes/3.11.0/).
Do any of these results suggest anything? If not, what further tests
would be useful?
Group metadata-cache is just a bunch of options one sets on a volume. So,
You can set them manually using gluster cli. Following are the options and
performance.md-cache-timeout=600
network.inode-lru-limit=50000
Post by Pat Haley
Thanks
Pat
Post by Pat Haley
Hi Raghavendra,
Thanks for the suggestions. Our technician will be in on Monday.
We'll test then and let you know the results.
One question I have, is the "group metadata-cache" option supposed to
directly impact the performance or is it to help collect data? If the
latter, where will the data be located?
It impacts performance.
Post by Pat Haley
Thanks again.
Pat
On Thu, Jun 21, 2018 at 10:24 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of writes to glusterfs mount,
I saw in earlier conversations that there are too many lookups, but
small number of writes. Since writes cached in write-behind would
invalidate metadata cache, lookups won't be absorbed by md-cache. I am
wondering what would results look like if we turn off
performance.write-behind.
@Pat,
Can you set,
# gluster volume set <volname> performance.write-behind off
Please turn on "group metadata-cache" for write tests too.
Post by Raghavendra Gowdappa
and redo the tests writing to glusterfs mount? Let us know about the
results you see.
regards,
Raghavendra
On Thu, Jun 21, 2018 at 8:33 AM, Raghavendra Gowdappa <
On Thu, Jun 21, 2018 at 8:32 AM, Raghavendra Gowdappa <
Post by Raghavendra Gowdappa
For the case of reading from Glusterfs mount, read-ahead should
help. However, we've known issues with read-ahead[1][2]. To work around
these, can you try with,
1. Turn off performance.open-behind
#gluster volume set <volname> performance.open-behind off
2. enable group meta metadata-cache
# gluster volume set <volname> group metadata-cache
[1] https://bugzilla.redhat.com/show_bug.cgi?id=1084508
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1214489
Post by Raghavendra Gowdappa
Post by Pat Haley
Hi,
We were recently revisiting our problems with the slowness of
gluster writes (http://lists.gluster.org/pipe
rmail/gluster-users/2017-April/030529.html). Specifically we were
testing the suggestions in a recent post (
http://lists.gluster.org/pipermail/gluster-users/2018-March
/033699.html). The first two suggestions (specifying a
negative-timeout in the mount settings or adding
rpc-auth-allow-insecure to
glusterd.vol) did not improve our performance, while setting
"disperse.eager-lock off" provided a tiny (5%) speed-up.
Some of the various tests we have tried earlier can be seen in the
links below. Do any of the above observations suggest what we could try
next to either improve the speed or debug the issue? Thanks
http://lists.gluster.org/pipermail/gluster-users/2017-June/0
31565.html
http://lists.gluster.org/pipermail/gluster-users/2017-May/03
0937.html
Pat
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Center for Ocean Engineering Phone: (617) 253-6824
Dept. of Mechanical Engineering Fax: (617) 253-8125
MIT, Room 5-213 http://web.mit.edu/phaley/www/
77 Massachusetts Avenue
Cambridge, MA 02139-4301
Loading...