[Gluster-users] On making ctime generator enabled by default in stack

[Raghavendra Gowdappa]

All,

There is a patch [1] from Kotresh, which makes ctime generator as default
in stack. Currently ctime generator is being recommended only for usecases
where ctime is important (like for Elasticsearch). However, a reliable
(c)(m)time can fix many consistency issues within glusterfs stack too.
These are issues with caching layers having stale (meta)data [2][3][4].
Basically just like applications, components within glusterfs stack too
need a time to find out which among racing ops (like write, stat, etc) has
latest (meta)data.

Also note that a consistent (c)(m)time is not an optional feature, but
instead forms the core of the infrastructure. So, I am proposing to merge
this patch. If you've any objections, please voice out before Nov 13, 2018
(a week from today).

As to the existing known issues/limitations with ctime generator, my
conversations with Kotresh, revealed following:
* Potential performance degradation (we don't yet have data to conclusively
prove it, preliminary basic tests from Kotresh didn't indicate a
significant perf drop).
* atime consistency. ctime generator offers atime consistency equivalent to
noatime mounts. But, with my limited experience I've not seen too many
usecases that require atime consistency. If you've a usecase please point
it out and we'll think how we can meet that requirement.

[1] https://review.gluster.org/#/c/glusterfs/+/21060/
[2] https://bugzilla.redhat.com/show_bug.cgi?id=1600923
[3] https://bugzilla.redhat.com/show_bug.cgi?id=1617972
[4] https://bugzilla.redhat.com/show_bug.cgi?id=1393743

regards,
Raghavendra

[Vijay Bellur]

Post by Raghavendra Gowdappa
All,
There is a patch [1] from Kotresh, which makes ctime generator as default
in stack. Currently ctime generator is being recommended only for usecases
where ctime is important (like for Elasticsearch). However, a reliable
(c)(m)time can fix many consistency issues within glusterfs stack too.
These are issues with caching layers having stale (meta)data [2][3][4].
Basically just like applications, components within glusterfs stack too
need a time to find out which among racing ops (like write, stat, etc) has
latest (meta)data.
Also note that a consistent (c)(m)time is not an optional feature, but
instead forms the core of the infrastructure. So, I am proposing to merge
this patch. If you've any objections, please voice out before Nov 13, 2018
(a week from today).
As to the existing known issues/limitations with ctime generator, my
* Potential performance degradation (we don't yet have data to
conclusively prove it, preliminary basic tests from Kotresh didn't indicate
a significant perf drop).

Do we have this data captured somewhere? If not, would it be possible to
share that data here?

Thanks,
Vijay

[Raghavendra Gowdappa]

Post by Vijay Bellur

Post by Raghavendra Gowdappa
All,
There is a patch [1] from Kotresh, which makes ctime generator as default
in stack. Currently ctime generator is being recommended only for usecases
where ctime is important (like for Elasticsearch). However, a reliable
(c)(m)time can fix many consistency issues within glusterfs stack too.
These are issues with caching layers having stale (meta)data [2][3][4].
Basically just like applications, components within glusterfs stack too
need a time to find out which among racing ops (like write, stat, etc) has
latest (meta)data.
Also note that a consistent (c)(m)time is not an optional feature, but
instead forms the core of the infrastructure. So, I am proposing to merge
this patch. If you've any objections, please voice out before Nov 13, 2018
(a week from today).
As to the existing known issues/limitations with ctime generator, my
* Potential performance degradation (we don't yet have data to
conclusively prove it, preliminary basic tests from Kotresh didn't indicate
a significant perf drop).

Do we have this data captured somewhere? If not, would it be possible to
share that data here?

I misquoted Kotresh. He had measured impact of gfid2path and said both
features might've similar impact as major perf cost is related to storing
xattrs on backend fs. I am in the process of getting a fresh set of
numbers. Will post those numbers when available.

Post by Vijay Bellur
Thanks,
Vijay

[Vijay Bellur]

Post by Raghavendra Gowdappa

Post by Vijay Bellur

Post by Raghavendra Gowdappa
All,
There is a patch [1] from Kotresh, which makes ctime generator as
default in stack. Currently ctime generator is being recommended only for
usecases where ctime is important (like for Elasticsearch). However, a
reliable (c)(m)time can fix many consistency issues within glusterfs stack
too. These are issues with caching layers having stale (meta)data
[2][3][4]. Basically just like applications, components within glusterfs
stack too need a time to find out which among racing ops (like write, stat,
etc) has latest (meta)data.
Also note that a consistent (c)(m)time is not an optional feature, but
instead forms the core of the infrastructure. So, I am proposing to merge
this patch. If you've any objections, please voice out before Nov 13, 2018
(a week from today).
As to the existing known issues/limitations with ctime generator, my
* Potential performance degradation (we don't yet have data to
conclusively prove it, preliminary basic tests from Kotresh didn't indicate
a significant perf drop).

Do we have this data captured somewhere? If not, would it be possible to
share that data here?

I misquoted Kotresh. He had measured impact of gfid2path and said both
features might've similar impact as major perf cost is related to storing
xattrs on backend fs. I am in the process of getting a fresh set of
numbers. Will post those numbers when available.

I observe that the patch under discussion has been merged now [1]. A quick
search did not yield me any performance data. Do we have the performance
numbers posted somewhere?

Thanks,
Vijay

[1] https://review.gluster.org/#/c/glusterfs/+/21060/

[Raghavendra Gowdappa]

Post by Vijay Bellur

Post by Raghavendra Gowdappa

Post by Vijay Bellur

Post by Raghavendra Gowdappa
All,
There is a patch [1] from Kotresh, which makes ctime generator as
default in stack. Currently ctime generator is being recommended only for
usecases where ctime is important (like for Elasticsearch). However, a
reliable (c)(m)time can fix many consistency issues within glusterfs stack
too. These are issues with caching layers having stale (meta)data
[2][3][4]. Basically just like applications, components within glusterfs
stack too need a time to find out which among racing ops (like write, stat,
etc) has latest (meta)data.
Also note that a consistent (c)(m)time is not an optional feature, but
instead forms the core of the infrastructure. So, I am proposing to merge
this patch. If you've any objections, please voice out before Nov 13, 2018
(a week from today).
As to the existing known issues/limitations with ctime generator, my
* Potential performance degradation (we don't yet have data to
conclusively prove it, preliminary basic tests from Kotresh didn't indicate
a significant perf drop).

Do we have this data captured somewhere? If not, would it be possible to
share that data here?

I misquoted Kotresh. He had measured impact of gfid2path and said both
features might've similar impact as major perf cost is related to storing
xattrs on backend fs. I am in the process of getting a fresh set of
numbers. Will post those numbers when available.

I observe that the patch under discussion has been merged now [1]. A quick
search did not yield me any performance data. Do we have the performance
numbers posted somewhere?

No. Perf benchmarking is a task pending on me.

Post by Vijay Bellur
Thanks,
Vijay
[1] https://review.gluster.org/#/c/glusterfs/+/21060/

[Vijay Bellur]

Post by Raghavendra Gowdappa

Post by Vijay Bellur

Post by Raghavendra Gowdappa

On Mon, Nov 5, 2018 at 7:56 PM Raghavendra Gowdappa <

Post by Raghavendra Gowdappa
All,
There is a patch [1] from Kotresh, which makes ctime generator as
default in stack. Currently ctime generator is being recommended only for
usecases where ctime is important (like for Elasticsearch). However, a
reliable (c)(m)time can fix many consistency issues within glusterfs stack
too. These are issues with caching layers having stale (meta)data
[2][3][4]. Basically just like applications, components within glusterfs
stack too need a time to find out which among racing ops (like write, stat,
etc) has latest (meta)data.
Also note that a consistent (c)(m)time is not an optional feature, but
instead forms the core of the infrastructure. So, I am proposing to merge
this patch. If you've any objections, please voice out before Nov 13, 2018
(a week from today).
As to the existing known issues/limitations with ctime generator, my
* Potential performance degradation (we don't yet have data to
conclusively prove it, preliminary basic tests from Kotresh didn't indicate
a significant perf drop).

Do we have this data captured somewhere? If not, would it be possible
to share that data here?

I misquoted Kotresh. He had measured impact of gfid2path and said both
features might've similar impact as major perf cost is related to storing
xattrs on backend fs. I am in the process of getting a fresh set of
numbers. Will post those numbers when available.

I observe that the patch under discussion has been merged now [1]. A
quick search did not yield me any performance data. Do we have the
performance numbers posted somewhere?

No. Perf benchmarking is a task pending on me.

When can we expect this task to be complete?

In any case, I don't think it is ideal for us to merge a patch without
completing our due diligence on it. How do we want to handle this scenario
since the patch is already merged?

We could:

1. Revert the patch now
2. Review the performance data and revert the patch if performance
characterization indicates a significant dip. It would be preferable to
complete this activity before we branch off for the next release.
3. Think of some other option?

Thanks,
Vijay

[Amar Tumballi]

Post by Vijay Bellur

Post by Raghavendra Gowdappa

Post by Vijay Bellur

Post by Raghavendra Gowdappa

On Mon, Nov 5, 2018 at 7:56 PM Raghavendra Gowdappa <

Post by Raghavendra Gowdappa
All,
There is a patch [1] from Kotresh, which makes ctime generator as
default in stack. Currently ctime generator is being recommended only for
usecases where ctime is important (like for Elasticsearch). However, a
reliable (c)(m)time can fix many consistency issues within glusterfs stack
too. These are issues with caching layers having stale (meta)data
[2][3][4]. Basically just like applications, components within glusterfs
stack too need a time to find out which among racing ops (like write, stat,
etc) has latest (meta)data.
Also note that a consistent (c)(m)time is not an optional feature,
but instead forms the core of the infrastructure. So, I am proposing to
merge this patch. If you've any objections, please voice out before Nov 13,
2018 (a week from today).
As to the existing known issues/limitations with ctime generator, my
* Potential performance degradation (we don't yet have data to
conclusively prove it, preliminary basic tests from Kotresh didn't indicate
a significant perf drop).

Do we have this data captured somewhere? If not, would it be possible
to share that data here?

I misquoted Kotresh. He had measured impact of gfid2path and said both
features might've similar impact as major perf cost is related to storing
xattrs on backend fs. I am in the process of getting a fresh set of
numbers. Will post those numbers when available.

I observe that the patch under discussion has been merged now [1]. A
quick search did not yield me any performance data. Do we have the
performance numbers posted somewhere?

No. Perf benchmarking is a task pending on me.

When can we expect this task to be complete?
In any case, I don't think it is ideal for us to merge a patch without
completing our due diligence on it. How do we want to handle this scenario
since the patch is already merged?
1. Revert the patch now
2. Review the performance data and revert the patch if performance
characterization indicates a significant dip. It would be preferable to
complete this activity before we branch off for the next release.

I am for option 2. Considering the branch out for next release is another 2
months, and no one is expected to use the 'release' off a master branch
yet, it makes sense to give that buffer time to get this activity completed.

Regards,
Amar

3. Think of some other option?

Post by Vijay Bellur
Thanks,
Vijay

Post by Raghavendra Gowdappa
_______________________________________________

Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users