Discussion:
[Gluster-users] Gluster tier in progress after attach
Jeevan Patnaik
2018-09-24 12:47:49 UTC
Permalink
Hi,

I have created a 18 disk replica 3 volume and created 1000 random files
each of 10M.

Later I have attached 15 disk replica 3 tier with each hot tier disk coming
from cold tier host, except on 3 (as we don't have hot tier disk)

After attach, I see no files are promoted at even after 15 minutes. Status
is showing as In progress.

Is it always this much slow? For 10G data, 15 minutes is too much
Also will the data try to be promoted to a hot tier coming from the same
host to reduce the time taken to move the files?

Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.


Regards,
Jeevan.
Jeevan Patnaik
2018-09-24 13:02:07 UTC
Permalink
I see it still not promoting any files. Do we need to run any command to
force the movement?

This would be an issue if we need to expand disk as then we also need to
detach existing tier and attach again and expect the data to be promoted to
the hot tier.

Regards,
Jeevan.
Post by Jeevan Patnaik
Hi,
I have created a 18 disk replica 3 volume and created 1000 random files
each of 10M.
Later I have attached 15 disk replica 3 tier with each hot tier disk
coming from cold tier host, except on 3 (as we don't have hot tier disk)
After attach, I see no files are promoted at even after 15 minutes. Status
is showing as In progress.
Is it always this much slow? For 10G data, 15 minutes is too much
Also will the data try to be promoted to a hot tier coming from the same
host to reduce the time taken to move the files?
Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.
Regards,
Jeevan.
Nithya Balachandran
2018-09-24 13:30:14 UTC
Permalink
Are those files being accessed? Tiering will only promote those that have
been accessed recently.

Regards,
Nithya
Post by Jeevan Patnaik
I see it still not promoting any files. Do we need to run any command to
force the movement?
This would be an issue if we need to expand disk as then we also need to
detach existing tier and attach again and expect the data to be promoted to
the hot tier.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi,
I have created a 18 disk replica 3 volume and created 1000 random files
each of 10M.
Later I have attached 15 disk replica 3 tier with each hot tier disk
coming from cold tier host, except on 3 (as we don't have hot tier disk)
After attach, I see no files are promoted at even after 15 minutes.
Status is showing as In progress.
Is it always this much slow? For 10G data, 15 minutes is too much
Also will the data try to be promoted to a hot tier coming from the same
host to reduce the time taken to move the files?
Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.
Regards,
Jeevan.
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Jeevan Patnaik
2018-09-24 13:45:29 UTC
Permalink
Hi Nithya,

No..the files are not being accessed. And tiering mode is in cache mode and
what I understood is in cache mode every file is moved to hot tier until it
reaches low watermark, then it only promotes highly accessed files.

And do you mean everytime we expand volume, we are virtually flushing the
tier and it regenerates only when a files are accessed?

Also I see one more issue now, I hav generated another 1000 files after
tiering is started and I see all 1000 files are now created on hot tier.
Now, I detached again..in detach status it shows completed..but when I
checked, around 260 files are still located on hot tier..those files are
not being used anywhere..ca you tell why are they still not moved to cold
tier and why the detach status is showing a wrong status.

I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2 hosts
also.

Regards,
Jeevan.
Post by Nithya Balachandran
Are those files being accessed? Tiering will only promote those that have
been accessed recently.
Regards,
Nithya
Post by Jeevan Patnaik
I see it still not promoting any files. Do we need to run any command to
force the movement?
This would be an issue if we need to expand disk as then we also need to
detach existing tier and attach again and expect the data to be promoted to
the hot tier.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi,
I have created a 18 disk replica 3 volume and created 1000 random files
each of 10M.
Later I have attached 15 disk replica 3 tier with each hot tier disk
coming from cold tier host, except on 3 (as we don't have hot tier disk)
After attach, I see no files are promoted at even after 15 minutes.
Status is showing as In progress.
Is it always this much slow? For 10G data, 15 minutes is too much
Also will the data try to be promoted to a hot tier coming from the same
host to reduce the time taken to move the files?
Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.
Regards,
Jeevan.
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Nithya Balachandran
2018-09-24 15:14:30 UTC
Permalink
Post by Jeevan Patnaik
Hi Nithya,
No..the files are not being accessed. And tiering mode is in cache mode
and what I understood is in cache mode every file is moved to hot tier
until it reaches low watermark, then it only promotes highly accessed files.
Not quite . All files will initially exist on the cold tier. Any new files
will be created on the hot tier. Any old files that are accessed will be
promoted ot the hot tier.


And do you mean everytime we expand volume, we are virtually flushing the
Post by Jeevan Patnaik
tier and it regenerates only when a files are accessed?
What do you mean by expanding the volume?
Also I see one more issue now, I hav generated another 1000 files after
tiering is started and I see all 1000 files are now created on hot tier.
Now, I detached again..in detach status it shows completed..but when I
checked, around 260 files are still located on hot tier..those files are
not being used anywhere..ca you tell why are they still not moved to cold
tier and why the detach status is showing a wrong status.
What command did you use to detach the tier? And how did you check that the
files still exist on the hot tier?

Regards,
Nithya
Post by Jeevan Patnaik
I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2 hosts
also.
Regards,
Jeevan.
Post by Nithya Balachandran
Are those files being accessed? Tiering will only promote those that have
been accessed recently.
Regards,
Nithya
Post by Jeevan Patnaik
I see it still not promoting any files. Do we need to run any command to
force the movement?
This would be an issue if we need to expand disk as then we also need to
detach existing tier and attach again and expect the data to be promoted to
the hot tier.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi,
I have created a 18 disk replica 3 volume and created 1000 random files
each of 10M.
Later I have attached 15 disk replica 3 tier with each hot tier disk
coming from cold tier host, except on 3 (as we don't have hot tier disk)
After attach, I see no files are promoted at even after 15 minutes.
Status is showing as In progress.
Is it always this much slow? For 10G data, 15 minutes is too much
Also will the data try to be promoted to a hot tier coming from the
same host to reduce the time taken to move the files?
Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.
Regards,
Jeevan.
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Jeevan Patnaik
2018-09-24 15:31:58 UTC
Permalink
Hi,

What do you mean by expanding volume?

I mean adding new bricks to extend storage. It is mentioned that detaching
tier is must for adding or removing bricks.

What command did you use to detach the tier? And how did you check that the
files still exist on the hot tier?

gluster volume tier volume_name detach start

I check the files directly from backend filesystem on disk and there are
still left over files on all hot tier disks.

Anyhow I proceeded with detach commit to see what would happen..so after
commit, I can't access those files from client filesystems..so they're gone
with the hot tier and were not moved to cold tier.

And even if I try to attach the hot tier bricks again to bring back the
files, attach won't work with existing files.

So, it seems we shouldn't detach hot tier without making sure all files are
moved to cold tier..but in my case, some files just won't move, even though
they aren't being accessed anywhere.

There's a statement saying POSIX locks can cause files to be still left in
hot tier and in that case, either the application causing posix lock has to
be closed or files should be moved manually (but where?)... however, I
don't think in my case there's any posix locks as the files are not
accessed by any application except gluster itself.

Regards,
Jeevan.
Post by Jeevan Patnaik
Hi Nithya,
No..the files are not being accessed. And tiering mode is in cache mode
and what I understood is in cache mode every file is moved to hot tier
until it reaches low watermark, then it only promotes highly accessed files.
Not quite . All files will initially exist on the cold tier. Any new
files will be created on the hot tier. Any old files that are accessed will
be promoted ot the hot tier.
And do you mean everytime we expand volume, we are virtually flushing the
Post by Jeevan Patnaik
tier and it regenerates only when a files are accessed?
What do you mean by expanding the volume?
Also I see one more issue now, I hav generated another 1000 files after
tiering is started and I see all 1000 files are now created on hot tier.
Now, I detached again..in detach status it shows completed..but when I
checked, around 260 files are still located on hot tier..those files are
not being used anywhere..ca you tell why are they still not moved to cold
tier and why the detach status is showing a wrong status.
What command did you use to detach the tier? And how did you check that
the files still exist on the hot tier?
Regards,
Nithya
Post by Jeevan Patnaik
I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2 hosts
also.
Regards,
Jeevan.
Post by Nithya Balachandran
Are those files being accessed? Tiering will only promote those that
have been accessed recently.
Regards,
Nithya
Post by Jeevan Patnaik
I see it still not promoting any files. Do we need to run any command
to force the movement?
This would be an issue if we need to expand disk as then we also need
to detach existing tier and attach again and expect the data to be promoted
to the hot tier.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi,
I have created a 18 disk replica 3 volume and created 1000 random
files each of 10M.
Later I have attached 15 disk replica 3 tier with each hot tier disk
coming from cold tier host, except on 3 (as we don't have hot tier disk)
After attach, I see no files are promoted at even after 15 minutes.
Status is showing as In progress.
Is it always this much slow? For 10G data, 15 minutes is too much
Also will the data try to be promoted to a hot tier coming from the
same host to reduce the time taken to move the files?
Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.
Regards,
Jeevan.
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Nithya Balachandran
2018-09-24 15:50:01 UTC
Permalink
Post by Jeevan Patnaik
Hi,
What do you mean by expanding volume?
I mean adding new bricks to extend storage. It is mentioned that detaching
tier is must for adding or removing bricks.
What command did you use to detach the tier? And how did you check that
the files still exist on the hot tier?
gluster volume tier volume_name detach start
I check the files directly from backend filesystem on disk and there are
still left over files on all hot tier disks.
Anyhow I proceeded with detach commit to see what would happen..so after
commit, I can't access those files from client filesystems..so they're gone
with the hot tier and were not moved to cold tier.
Did you check that the detach tier operation had completed before running
detach commit?
Post by Jeevan Patnaik
And even if I try to attach the hot tier bricks again to bring back the
files, attach won't work with existing files.
So, it seems we shouldn't detach hot tier without making sure all files
are moved to cold tier..but in my case, some files just won't move, even
though they aren't being accessed anywhere.
There's a statement saying POSIX locks can cause files to be still left in
hot tier and in that case, either the application causing posix lock has to
be closed or files should be moved manually (but where?)... however, I
don't think in my case there's any posix locks as the files are not
accessed by any application except gluster itself.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi Nithya,
No..the files are not being accessed. And tiering mode is in cache mode
and what I understood is in cache mode every file is moved to hot tier
until it reaches low watermark, then it only promotes highly accessed files.
Not quite . All files will initially exist on the cold tier. Any new
files will be created on the hot tier. Any old files that are accessed will
be promoted ot the hot tier.
And do you mean everytime we expand volume, we are virtually flushing
Post by Jeevan Patnaik
the tier and it regenerates only when a files are accessed?
What do you mean by expanding the volume?
Also I see one more issue now, I hav generated another 1000 files after
tiering is started and I see all 1000 files are now created on hot tier.
Now, I detached again..in detach status it shows completed..but when I
checked, around 260 files are still located on hot tier..those files are
not being used anywhere..ca you tell why are they still not moved to cold
tier and why the detach status is showing a wrong status.
What command did you use to detach the tier? And how did you check that
the files still exist on the hot tier?
Regards,
Nithya
Post by Jeevan Patnaik
I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2
hosts also.
Regards,
Jeevan.
Post by Nithya Balachandran
Are those files being accessed? Tiering will only promote those that
have been accessed recently.
Regards,
Nithya
Post by Jeevan Patnaik
I see it still not promoting any files. Do we need to run any command
to force the movement?
This would be an issue if we need to expand disk as then we also need
to detach existing tier and attach again and expect the data to be promoted
to the hot tier.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi,
I have created a 18 disk replica 3 volume and created 1000 random
files each of 10M.
Later I have attached 15 disk replica 3 tier with each hot tier disk
coming from cold tier host, except on 3 (as we don't have hot tier disk)
After attach, I see no files are promoted at even after 15 minutes.
Status is showing as In progress.
Is it always this much slow? For 10G data, 15 minutes is too much
Also will the data try to be promoted to a hot tier coming from the
same host to reduce the time taken to move the files?
Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.
Regards,
Jeevan.
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Jeevan Patnaik
2018-09-24 15:51:24 UTC
Permalink
Hi,

Yes, I did. The status shows all as completed.

Regards,
Jeevan.
Post by Nithya Balachandran
Post by Jeevan Patnaik
Hi,
What do you mean by expanding volume?
I mean adding new bricks to extend storage. It is mentioned that
detaching tier is must for adding or removing bricks.
What command did you use to detach the tier? And how did you check that
the files still exist on the hot tier?
gluster volume tier volume_name detach start
I check the files directly from backend filesystem on disk and there are
still left over files on all hot tier disks.
Anyhow I proceeded with detach commit to see what would happen..so after
commit, I can't access those files from client filesystems..so they're gone
with the hot tier and were not moved to cold tier.
Did you check that the detach tier operation had completed before running
detach commit?
Post by Jeevan Patnaik
And even if I try to attach the hot tier bricks again to bring back the
files, attach won't work with existing files.
So, it seems we shouldn't detach hot tier without making sure all files
are moved to cold tier..but in my case, some files just won't move, even
though they aren't being accessed anywhere.
There's a statement saying POSIX locks can cause files to be still left
in hot tier and in that case, either the application causing posix lock has
to be closed or files should be moved manually (but where?)... however, I
don't think in my case there's any posix locks as the files are not
accessed by any application except gluster itself.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi Nithya,
No..the files are not being accessed. And tiering mode is in cache mode
and what I understood is in cache mode every file is moved to hot tier
until it reaches low watermark, then it only promotes highly accessed files.
Not quite . All files will initially exist on the cold tier. Any new
files will be created on the hot tier. Any old files that are accessed will
be promoted ot the hot tier.
And do you mean everytime we expand volume, we are virtually flushing
Post by Jeevan Patnaik
the tier and it regenerates only when a files are accessed?
What do you mean by expanding the volume?
Also I see one more issue now, I hav generated another 1000 files after
tiering is started and I see all 1000 files are now created on hot tier.
Now, I detached again..in detach status it shows completed..but when I
checked, around 260 files are still located on hot tier..those files are
not being used anywhere..ca you tell why are they still not moved to cold
tier and why the detach status is showing a wrong status.
What command did you use to detach the tier? And how did you check that
the files still exist on the hot tier?
Regards,
Nithya
Post by Jeevan Patnaik
I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2
hosts also.
Regards,
Jeevan.
Post by Nithya Balachandran
Are those files being accessed? Tiering will only promote those that
have been accessed recently.
Regards,
Nithya
Post by Jeevan Patnaik
I see it still not promoting any files. Do we need to run any command
to force the movement?
This would be an issue if we need to expand disk as then we also need
to detach existing tier and attach again and expect the data to be promoted
to the hot tier.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi,
I have created a 18 disk replica 3 volume and created 1000 random
files each of 10M.
Later I have attached 15 disk replica 3 tier with each hot tier disk
coming from cold tier host, except on 3 (as we don't have hot tier disk)
After attach, I see no files are promoted at even after 15 minutes.
Status is showing as In progress.
Is it always this much slow? For 10G data, 15 minutes is too much
Also will the data try to be promoted to a hot tier coming from the
same host to reduce the time taken to move the files?
Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.
Regards,
Jeevan.
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Nithya Balachandran
2018-09-24 16:06:35 UTC
Permalink
Please check the rebalance log to see why those files were not migrated.

Regards,
Nithya
Post by Jeevan Patnaik
Hi,
Yes, I did. The status shows all as completed.
Regards,
Jeevan.
Post by Nithya Balachandran
Post by Jeevan Patnaik
Hi,
What do you mean by expanding volume?
I mean adding new bricks to extend storage. It is mentioned that
detaching tier is must for adding or removing bricks.
What command did you use to detach the tier? And how did you check that
the files still exist on the hot tier?
gluster volume tier volume_name detach start
I check the files directly from backend filesystem on disk and there are
still left over files on all hot tier disks.
Anyhow I proceeded with detach commit to see what would happen..so after
commit, I can't access those files from client filesystems..so they're gone
with the hot tier and were not moved to cold tier.
Did you check that the detach tier operation had completed before running
detach commit?
Post by Jeevan Patnaik
And even if I try to attach the hot tier bricks again to bring back the
files, attach won't work with existing files.
So, it seems we shouldn't detach hot tier without making sure all files
are moved to cold tier..but in my case, some files just won't move, even
though they aren't being accessed anywhere.
There's a statement saying POSIX locks can cause files to be still left
in hot tier and in that case, either the application causing posix lock has
to be closed or files should be moved manually (but where?)... however, I
don't think in my case there's any posix locks as the files are not
accessed by any application except gluster itself.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi Nithya,
No..the files are not being accessed. And tiering mode is in cache
mode and what I understood is in cache mode every file is moved to hot tier
until it reaches low watermark, then it only promotes highly accessed files.
Not quite . All files will initially exist on the cold tier. Any new
files will be created on the hot tier. Any old files that are accessed will
be promoted ot the hot tier.
And do you mean everytime we expand volume, we are virtually flushing
Post by Jeevan Patnaik
the tier and it regenerates only when a files are accessed?
What do you mean by expanding the volume?
Also I see one more issue now, I hav generated another 1000 files
after tiering is started and I see all 1000 files are now created on hot
tier. Now, I detached again..in detach status it shows completed..but when
I checked, around 260 files are still located on hot tier..those files are
not being used anywhere..ca you tell why are they still not moved to cold
tier and why the detach status is showing a wrong status.
What command did you use to detach the tier? And how did you check that
the files still exist on the hot tier?
Regards,
Nithya
Post by Jeevan Patnaik
I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2
hosts also.
Regards,
Jeevan.
Post by Nithya Balachandran
Are those files being accessed? Tiering will only promote those that
have been accessed recently.
Regards,
Nithya
Post by Jeevan Patnaik
I see it still not promoting any files. Do we need to run any
command to force the movement?
This would be an issue if we need to expand disk as then we also
need to detach existing tier and attach again and expect the data to be
promoted to the hot tier.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi,
I have created a 18 disk replica 3 volume and created 1000 random
files each of 10M.
Later I have attached 15 disk replica 3 tier with each hot tier
disk coming from cold tier host, except on 3 (as we don't have hot tier
disk)
After attach, I see no files are promoted at even after 15 minutes.
Status is showing as In progress.
Is it always this much slow? For 10G data, 15 minutes is too much
Also will the data try to be promoted to a hot tier coming from the
same host to reduce the time taken to move the files?
Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.
Regards,
Jeevan.
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Jeevan Patnaik
2018-09-24 16:08:06 UTC
Permalink
Ok, I will check.

Thank you.
Post by Nithya Balachandran
Please check the rebalance log to see why those files were not migrated.
Regards,
Nithya
Post by Jeevan Patnaik
Hi,
Yes, I did. The status shows all as completed.
Regards,
Jeevan.
Post by Nithya Balachandran
Post by Jeevan Patnaik
Hi,
What do you mean by expanding volume?
I mean adding new bricks to extend storage. It is mentioned that
detaching tier is must for adding or removing bricks.
What command did you use to detach the tier? And how did you check that
the files still exist on the hot tier?
gluster volume tier volume_name detach start
I check the files directly from backend filesystem on disk and there
are still left over files on all hot tier disks.
Anyhow I proceeded with detach commit to see what would happen..so
after commit, I can't access those files from client filesystems..so
they're gone with the hot tier and were not moved to cold tier.
Did you check that the detach tier operation had completed before
running detach commit?
Post by Jeevan Patnaik
And even if I try to attach the hot tier bricks again to bring back the
files, attach won't work with existing files.
So, it seems we shouldn't detach hot tier without making sure all files
are moved to cold tier..but in my case, some files just won't move, even
though they aren't being accessed anywhere.
There's a statement saying POSIX locks can cause files to be still left
in hot tier and in that case, either the application causing posix lock has
to be closed or files should be moved manually (but where?)... however, I
don't think in my case there's any posix locks as the files are not
accessed by any application except gluster itself.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi Nithya,
No..the files are not being accessed. And tiering mode is in cache
mode and what I understood is in cache mode every file is moved to hot tier
until it reaches low watermark, then it only promotes highly accessed files.
Not quite . All files will initially exist on the cold tier. Any new
files will be created on the hot tier. Any old files that are accessed will
be promoted ot the hot tier.
And do you mean everytime we expand volume, we are virtually flushing
Post by Jeevan Patnaik
the tier and it regenerates only when a files are accessed?
What do you mean by expanding the volume?
Also I see one more issue now, I hav generated another 1000 files
after tiering is started and I see all 1000 files are now created on hot
tier. Now, I detached again..in detach status it shows completed..but when
I checked, around 260 files are still located on hot tier..those files are
not being used anywhere..ca you tell why are they still not moved to cold
tier and why the detach status is showing a wrong status.
What command did you use to detach the tier? And how did you check
that the files still exist on the hot tier?
Regards,
Nithya
Post by Jeevan Patnaik
I am using gluster 3.12.3 and server hosts include RHEL 6.7 and 7.2
hosts also.
Regards,
Jeevan.
On Mon 24 Sep, 2018, 7:00 PM Nithya Balachandran, <
Post by Nithya Balachandran
Are those files being accessed? Tiering will only promote those that
have been accessed recently.
Regards,
Nithya
Post by Jeevan Patnaik
I see it still not promoting any files. Do we need to run any
command to force the movement?
This would be an issue if we need to expand disk as then we also
need to detach existing tier and attach again and expect the data to be
promoted to the hot tier.
Regards,
Jeevan.
Post by Jeevan Patnaik
Hi,
I have created a 18 disk replica 3 volume and created 1000 random
files each of 10M.
Later I have attached 15 disk replica 3 tier with each hot tier
disk coming from cold tier host, except on 3 (as we don't have hot tier
disk)
After attach, I see no files are promoted at even after 15
minutes. Status is showing as In progress.
Is it always this much slow? For 10G data, 15 minutes is too much
Also will the data try to be promoted to a hot tier coming from
the same host to reduce the time taken to move the files?
Cold tier is made of RAID supported SSD and hot tier is made of NVMe SSDs.
Regards,
Jeevan.
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
Loading...