Discussion:
[Gluster-users] Hot Tier exceeding watermark-hi
David Brown
2018-09-29 12:08:07 UTC
Permalink
Hey Everyone,

I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD cold
tier.
I recently ran into some problems when the hot tier became full with df-h
showing 100%.

I did not have a watermark-hi set, but it is my understanding that 90% is
the default. In an attempt to get the cluster to demote some files, I set
cluster.watermark-hi: 80 but it is still not demoting.


[***@Glus1 ~]# gluster volume info

Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
Options Reconfigured:
cluster.tier-promote-frequency: 120
cluster.tier-demote-frequency: 1800
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[***@Glus1 ~]# gluster volume tier FFPrimary status
Node Promoted files Demoted files Status
run time in h:m:s
--------- --------- --------- ---------
---------
localhost 49 0 in progress
5151:30:45
Glus2 0 0 in progress
5151:30:45
Glus3 0 2075 in progress
5151:30:47
Tiering Migration Functionality: FFPrimary: success
[***@Glus1 ~]#

What can cause GlusterFS to stop demoting files and allow it to completely
fill the Hot Tier?

Thank you!
Hari Gowtham
2018-09-29 15:08:12 UTC
Permalink
Hi,

I can see that the demotion is happening from the status provided by you.
Do verify it.
I would recommend you to change the cluster.tier-demote-frequency to 120
and cluster.tier-promote-frequency to 1800 to increase the demotions until the
hot tier is emptied to a certain extent. Later you can use the values
existing now.
Post by David Brown
Hey Everyone,
I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD cold tier.
I recently ran into some problems when the hot tier became full with df-h showing 100%.
I did not have a watermark-hi set, but it is my understanding that 90% is the default. In an attempt to get the cluster to demote some files, I set cluster.watermark-hi: 80 but it is still not demoting.
Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
cluster.tier-promote-frequency: 120
cluster.tier-demote-frequency: 1800
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Node Promoted files Demoted files Status run time in h:m:s
--------- --------- --------- --------- ---------
localhost 49 0 in progress 5151:30:45
Glus2 0 0 in progress 5151:30:45
Glus3 0 2075 in progress 5151:30:47
Tiering Migration Functionality: FFPrimary: success
What can cause GlusterFS to stop demoting files and allow it to completely fill the Hot Tier?
Thank you!
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Regards,
Hari Gowtham.
David Brown
2018-09-29 15:56:10 UTC
Permalink
Thank you Hari,

I have set:
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120

I will let you know if it makes a difference after some time. So far (10
minutes), nothing has changed.
I would agree with you, that by looking at the result of 'gluster volume
tier FFPrimary status' it would seem that demoting is happening. However,
for the last 24hrs, nothing has changed in the tier status report except
the time. Could it be stuck? How would I know? Is there a way to restart it
without restarting the cluster?
Post by Hari Gowtham
Hi,
I can see that the demotion is happening from the status provided by you.
Do verify it.
I would recommend you to change the cluster.tier-demote-frequency to 120
and cluster.tier-promote-frequency to 1800 to increase the demotions until the
hot tier is emptied to a certain extent. Later you can use the values
existing now.
Post by David Brown
Hey Everyone,
I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD cold
tier.
Post by David Brown
I recently ran into some problems when the hot tier became full with
df-h showing 100%.
Post by David Brown
I did not have a watermark-hi set, but it is my understanding that 90%
is the default. In an attempt to get the cluster to demote some files, I
set cluster.watermark-hi: 80 but it is still not demoting.
Post by David Brown
Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
cluster.tier-promote-frequency: 120
cluster.tier-demote-frequency: 1800
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Node Promoted files Demoted files Status
run time in h:m:s
Post by David Brown
--------- --------- ---------
--------- ---------
Post by David Brown
localhost 49 0 in
progress 5151:30:45
Post by David Brown
Glus2 0 0 in progress
5151:30:45
Post by David Brown
Glus3 0 2075 in progress
5151:30:47
Post by David Brown
Tiering Migration Functionality: FFPrimary: success
What can cause GlusterFS to stop demoting files and allow it to
completely fill the Hot Tier?
Post by David Brown
Thank you!
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Regards,
Hari Gowtham.
David Brown
2018-09-29 19:54:09 UTC
Permalink
4 hours later, no files have been demoted

[***@Glus1 ~]# gluster volume status FFPrimary detail
Status of volume: FFPrimary
Hot Bricks:
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick3
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 24177
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 24.3GB
Total Disk Space : 476.7GB
Inode Count : 50877088
Free Inodes : 50874696
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick2
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 17994
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 15.5GB
Total Disk Space : 476.7GB
Inode Count : 32560288
Free Inodes : 32557896
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick1
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 23573
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 13.8GB
Total Disk Space : 476.7GB
Inode Count : 29027000
Free Inodes : 29024515
Cold Bricks:
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick5
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 23442
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969488
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick6
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 17856
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969489
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick7
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 24018
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969488
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick8
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 23518
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969607
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick9
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 17943
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969607
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick10
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 24108
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969604

[***@Glus1 ~]# gluster volume info

Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier :
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier:
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
Options Reconfigured:
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off

[***@Glus1 ~]# gluster volume tier FFPrimary status
Node Promoted files Demoted files Status
run time in h:m:s
--------- --------- --------- ---------
---------
localhost 49 0 in progress
5159:15:50
Glus2 0 0 in progress
5159:15:50
Glus3 0 2075 in progress
5159:15:52
Tiering Migration Functionality: FFPrimary: success
Post by David Brown
Thank you Hari,
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120
I will let you know if it makes a difference after some time. So far (10
minutes), nothing has changed.
I would agree with you, that by looking at the result of 'gluster volume
tier FFPrimary status' it would seem that demoting is happening. However,
for the last 24hrs, nothing has changed in the tier status report except
the time. Could it be stuck? How would I know? Is there a way to restart it
without restarting the cluster?
Post by Hari Gowtham
Hi,
I can see that the demotion is happening from the status provided by you.
Do verify it.
I would recommend you to change the cluster.tier-demote-frequency to 120
and cluster.tier-promote-frequency to 1800 to increase the demotions until the
hot tier is emptied to a certain extent. Later you can use the values
existing now.
Post by David Brown
Hey Everyone,
I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD
cold tier.
Post by David Brown
I recently ran into some problems when the hot tier became full with
df-h showing 100%.
Post by David Brown
I did not have a watermark-hi set, but it is my understanding that 90%
is the default. In an attempt to get the cluster to demote some files, I
set cluster.watermark-hi: 80 but it is still not demoting.
Post by David Brown
Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
cluster.tier-promote-frequency: 120
cluster.tier-demote-frequency: 1800
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Node Promoted files Demoted files Status
run time in h:m:s
Post by David Brown
--------- --------- ---------
--------- ---------
Post by David Brown
localhost 49 0 in
progress 5151:30:45
Post by David Brown
Glus2 0 0 in progress
5151:30:45
Post by David Brown
Glus3 0 2075 in progress
5151:30:47
Post by David Brown
Tiering Migration Functionality: FFPrimary: success
What can cause GlusterFS to stop demoting files and allow it to
completely fill the Hot Tier?
Post by David Brown
Thank you!
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Regards,
Hari Gowtham.
David Brown
2018-09-30 12:33:30 UTC
Permalink
Just found this in the tierd.log Not sure what it means or how to fix it
tho, but I assume it may be the cause of my problem with files not being
demoted from the hot tier..


[2018-09-30 12:25:56.438821] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
[Transport endpoint is not connected]
[2018-09-30 12:25:56.440940] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
(10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.444633] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
(32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.448347] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
(389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.451919] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //test-march-3-bc-file-501
(41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.456198] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //krishna
(49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.459702] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //test1
(5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.463164] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
(5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.466601] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
(6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.470129] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
(7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.473758] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
(8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.477237] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
(91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.480696] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
(972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.484354] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
(a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.487884] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:56.490973] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
(ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.494618] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
(ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.498059] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
(c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.501428] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //DBTest
(c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.504736] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //testCliff
(cac8d0c8-2a10-4fa1-8b94-bf47981c01ff). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
The message "I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file]
0-FFPrimary-tier-dht: Demotion failed for
6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)"
repeated 4 times between [2018-09-30 12:25:53.164873] and [2018-09-30
12:25:57.515127]
[2018-09-30 12:25:57.520933] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
[2018-09-30 12:25:57.521545] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
[2018-09-30 12:25:57.521854] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
[Transport endpoint is not connected]
[2018-09-30 12:25:57.524079] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
(10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.527993] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
(32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.531829] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
(389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.535529] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //test-march-3-bc-file-501
(41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.539852] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //krishna
(49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.543468] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //test1
(5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.547051] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
(5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.550694] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
(6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.554498] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
(7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.558086] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
(8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.561568] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
(91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.565094] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
(972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.568562] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
(a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.571974] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:57.574925] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
(ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.578639] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
(ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.582118] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
(c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.585547] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //DBTest
(c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
The message "E [MSGID: 109037] [tier.c:2532:tier_run] 0-FFPrimary-tier-dht:
Demotion failed" repeated 3 times between [2018-09-30 12:25:54.323716] and
[2018-09-30 12:25:57.592361]
[2018-09-30 12:25:58.002724] I [MSGID: 0]
[dht-rebalance.c:4194:gf_defrag_total_file_size] 0-FFPrimary-tier-dht:
local subvol: FFPrimary-cold-dht,cnt = 272288612352
[2018-09-30 12:25:58.018840] I [MSGID: 0]
[dht-rebalance.c:4198:gf_defrag_total_file_size] 0-FFPrimary-tier-dht:
Total size files = 272288612352
[2018-09-30 12:25:58.018854] I
[dht-rebalance.c:4325:dht_file_counter_thread] 0-dht: tmp data size
=272288612352
The message "E [MSGID: 109037] [tier.c:720:tier_lookup_file]
0-FFPrimary-tier-dht: Failed to lookup file
6bed3d1e-17b8-11e8-a5dc-902b3450f388
[No such file or directory]" repeated 4 times between [2018-09-30
12:25:54.247352] and [2018-09-30 12:25:58.599151]
[2018-09-30 12:25:58.599162] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for
6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)
[2018-09-30 12:25:58.604808] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
[2018-09-30 12:25:58.605428] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
[2018-09-30 12:25:58.605750] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
[Transport endpoint is not connected]
[2018-09-30 12:25:58.607865] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
(10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.611433] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
(32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.615059] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
(389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.618669] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //test-march-3-bc-file-501
(41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.622667] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //krishna
(49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.626029] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //test1
(5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.629459] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
(5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.632994] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
(6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.636669] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
(7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.640155] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
(8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.643516] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
(91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.648787] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
(972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.652106] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
(a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.655577] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:58.658482] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
(ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.661703] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
(ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.665051] W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
(c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
The message "W [MSGID: 114031]
[client-rpc-fops.c:1080:client3_3_getxattr_cbk] 0-FFPrimary-client-8:
remote operation failed. Path: //e8a6aba1-e5ce-4ded-b474-1c5bf49b1285
(edffea55-296b-4d1b-8114-b4f8dd10920a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]" repeated 3 times between [2018-09-30
12:25:55.406782] and [2018-09-30 12:25:58.674842]
[2018-09-30 12:25:58.675351] E [MSGID: 109037] [tier.c:2532:tier_run]
0-FFPrimary-tier-dht: Demotion failed
Post by David Brown
4 hours later, no files have been demoted
Status of volume: FFPrimary
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick3
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 24177
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 24.3GB
Total Disk Space : 476.7GB
Inode Count : 50877088
Free Inodes : 50874696
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick2
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 17994
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 15.5GB
Total Disk Space : 476.7GB
Inode Count : 32560288
Free Inodes : 32557896
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick1
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 23573
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 13.8GB
Total Disk Space : 476.7GB
Inode Count : 29027000
Free Inodes : 29024515
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick5
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 23442
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969488
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick6
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 17856
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969489
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick7
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 24018
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969488
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick8
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 23518
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969607
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick9
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 17943
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969607
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick10
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 24108
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969604
Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Node Promoted files Demoted files Status
run time in h:m:s
--------- --------- --------- ---------
---------
localhost 49 0 in
progress 5159:15:50
Glus2 0 0 in progress
5159:15:50
Glus3 0 2075 in progress
5159:15:52
Tiering Migration Functionality: FFPrimary: success
Post by David Brown
Thank you Hari,
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120
I will let you know if it makes a difference after some time. So far (10
minutes), nothing has changed.
I would agree with you, that by looking at the result of 'gluster volume
tier FFPrimary status' it would seem that demoting is happening. However,
for the last 24hrs, nothing has changed in the tier status report except
the time. Could it be stuck? How would I know? Is there a way to restart it
without restarting the cluster?
Post by Hari Gowtham
Hi,
I can see that the demotion is happening from the status provided by you.
Do verify it.
I would recommend you to change the cluster.tier-demote-frequency to 120
and cluster.tier-promote-frequency to 1800 to increase the demotions until the
hot tier is emptied to a certain extent. Later you can use the values
existing now.
Post by David Brown
Hey Everyone,
I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD
cold tier.
Post by David Brown
I recently ran into some problems when the hot tier became full with
df-h showing 100%.
Post by David Brown
I did not have a watermark-hi set, but it is my understanding that 90%
is the default. In an attempt to get the cluster to demote some files, I
set cluster.watermark-hi: 80 but it is still not demoting.
Post by David Brown
Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
cluster.tier-promote-frequency: 120
cluster.tier-demote-frequency: 1800
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Node Promoted files Demoted files Status
run time in h:m:s
Post by David Brown
--------- --------- ---------
--------- ---------
Post by David Brown
localhost 49 0 in
progress 5151:30:45
Post by David Brown
Glus2 0 0 in progress
5151:30:45
Post by David Brown
Glus3 0 2075 in progress
5151:30:47
Post by David Brown
Tiering Migration Functionality: FFPrimary: success
What can cause GlusterFS to stop demoting files and allow it to
completely fill the Hot Tier?
Post by David Brown
Thank you!
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Regards,
Hari Gowtham.
Hari Gowtham
2018-09-30 18:35:15 UTC
Permalink
Hi,

The demotion has failed, so it's not moving files to cold tier any more.
Transport endpoint usually refers to the bricks or servers. Need to see if
the brick processes are up. The demotion failed because of the bricks being
unavailable.
Can you send the "gluster volume status" output again, the log files,
version of gluster and the operation performed on the volume.

From the status output you sent earlier, it looks like all the processes
are running. But the log file says the other way. The log files and the
operations performed on the volume will help to identify what happened here.

If the bricks are down then the tier process might have gone down too. then
you can use " gluster volume tier <volname> start" to start the tier
process.
Post by David Brown
Just found this in the tierd.log Not sure what it means or how to fix it
tho, but I assume it may be the cause of my problem with files not being
demoted from the hot tier..
[2018-09-30 12:25:56.438821] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
[Transport endpoint is not connected]
[2018-09-30 12:25:56.440940] W [MSGID: 114031]
remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
(10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.444633] W [MSGID: 114031]
remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
(32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.448347] W [MSGID: 114031]
remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
(389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.451919] W [MSGID: 114031]
remote operation failed. Path: //test-march-3-bc-file-501
(41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.456198] W [MSGID: 114031]
remote operation failed. Path: //krishna
(49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.459702] W [MSGID: 114031]
remote operation failed. Path: //test1
(5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.463164] W [MSGID: 114031]
remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
(5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.466601] W [MSGID: 114031]
remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
(6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.470129] W [MSGID: 114031]
remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
(7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.473758] W [MSGID: 114031]
remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
(8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.477237] W [MSGID: 114031]
remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
(91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.480696] W [MSGID: 114031]
remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
(972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.484354] W [MSGID: 114031]
remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
(a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.487884] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:56.490973] W [MSGID: 114031]
remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
(ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.494618] W [MSGID: 114031]
remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
(ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.498059] W [MSGID: 114031]
remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
(c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.501428] W [MSGID: 114031]
remote operation failed. Path: //DBTest
(c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.504736] W [MSGID: 114031]
remote operation failed. Path: //testCliff
(cac8d0c8-2a10-4fa1-8b94-bf47981c01ff). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
The message "I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file]
0-FFPrimary-tier-dht: Demotion failed for
6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)"
repeated 4 times between [2018-09-30 12:25:53.164873] and [2018-09-30
12:25:57.515127]
[2018-09-30 12:25:57.520933] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
[2018-09-30 12:25:57.521545] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
[2018-09-30 12:25:57.521854] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
[Transport endpoint is not connected]
[2018-09-30 12:25:57.524079] W [MSGID: 114031]
remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
(10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.527993] W [MSGID: 114031]
remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
(32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.531829] W [MSGID: 114031]
remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
(389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.535529] W [MSGID: 114031]
remote operation failed. Path: //test-march-3-bc-file-501
(41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.539852] W [MSGID: 114031]
remote operation failed. Path: //krishna
(49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.543468] W [MSGID: 114031]
remote operation failed. Path: //test1
(5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.547051] W [MSGID: 114031]
remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
(5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.550694] W [MSGID: 114031]
remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
(6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.554498] W [MSGID: 114031]
remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
(7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.558086] W [MSGID: 114031]
remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
(8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.561568] W [MSGID: 114031]
remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
(91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.565094] W [MSGID: 114031]
remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
(972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.568562] W [MSGID: 114031]
remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
(a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.571974] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:57.574925] W [MSGID: 114031]
remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
(ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.578639] W [MSGID: 114031]
remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
(ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.582118] W [MSGID: 114031]
remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
(c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.585547] W [MSGID: 114031]
remote operation failed. Path: //DBTest
(c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
The message "E [MSGID: 109037] [tier.c:2532:tier_run]
0-FFPrimary-tier-dht: Demotion failed" repeated 3 times between [2018-09-30
12:25:54.323716] and [2018-09-30 12:25:57.592361]
[2018-09-30 12:25:58.002724] I [MSGID: 0]
local subvol: FFPrimary-cold-dht,cnt = 272288612352
[2018-09-30 12:25:58.018840] I [MSGID: 0]
Total size files = 272288612352
[2018-09-30 12:25:58.018854] I
[dht-rebalance.c:4325:dht_file_counter_thread] 0-dht: tmp data size
=272288612352
The message "E [MSGID: 109037] [tier.c:720:tier_lookup_file]
0-FFPrimary-tier-dht: Failed to lookup file
6bed3d1e-17b8-11e8-a5dc-902b3450f388
[No such file or directory]" repeated 4 times between [2018-09-30
12:25:54.247352] and [2018-09-30 12:25:58.599151]
[2018-09-30 12:25:58.599162] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for
6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)
[2018-09-30 12:25:58.604808] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
[2018-09-30 12:25:58.605428] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
[2018-09-30 12:25:58.605750] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
[Transport endpoint is not connected]
[2018-09-30 12:25:58.607865] W [MSGID: 114031]
remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
(10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.611433] W [MSGID: 114031]
remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
(32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.615059] W [MSGID: 114031]
remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
(389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.618669] W [MSGID: 114031]
remote operation failed. Path: //test-march-3-bc-file-501
(41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.622667] W [MSGID: 114031]
remote operation failed. Path: //krishna
(49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.626029] W [MSGID: 114031]
remote operation failed. Path: //test1
(5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.629459] W [MSGID: 114031]
remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
(5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.632994] W [MSGID: 114031]
remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
(6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.636669] W [MSGID: 114031]
remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
(7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.640155] W [MSGID: 114031]
remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
(8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.643516] W [MSGID: 114031]
remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
(91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.648787] W [MSGID: 114031]
remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
(972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.652106] W [MSGID: 114031]
remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
(a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.655577] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:58.658482] W [MSGID: 114031]
remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
(ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.661703] W [MSGID: 114031]
remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
(ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.665051] W [MSGID: 114031]
remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
(c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
The message "W [MSGID: 114031]
remote operation failed. Path: //e8a6aba1-e5ce-4ded-b474-1c5bf49b1285
(edffea55-296b-4d1b-8114-b4f8dd10920a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]" repeated 3 times between [2018-09-30
12:25:55.406782] and [2018-09-30 12:25:58.674842]
[2018-09-30 12:25:58.675351] E [MSGID: 109037] [tier.c:2532:tier_run]
0-FFPrimary-tier-dht: Demotion failed
Post by David Brown
4 hours later, no files have been demoted
Status of volume: FFPrimary
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick3
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 24177
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 24.3GB
Total Disk Space : 476.7GB
Inode Count : 50877088
Free Inodes : 50874696
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick2
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 17994
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 15.5GB
Total Disk Space : 476.7GB
Inode Count : 32560288
Free Inodes : 32557896
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick1
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 23573
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 13.8GB
Total Disk Space : 476.7GB
Inode Count : 29027000
Free Inodes : 29024515
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick5
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 23442
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969488
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick6
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 17856
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969489
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick7
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 24018
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969488
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick8
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 23518
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969607
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick9
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 17943
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969607
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick10
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 24108
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969604
Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Node Promoted files Demoted files Status
run time in h:m:s
--------- --------- --------- ---------
---------
localhost 49 0 in
progress 5159:15:50
Glus2 0 0 in progress
5159:15:50
Glus3 0 2075 in progress
5159:15:52
Tiering Migration Functionality: FFPrimary: success
Post by David Brown
Thank you Hari,
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120
I will let you know if it makes a difference after some time. So far (10
minutes), nothing has changed.
I would agree with you, that by looking at the result of 'gluster volume
tier FFPrimary status' it would seem that demoting is happening. However,
for the last 24hrs, nothing has changed in the tier status report except
the time. Could it be stuck? How would I know? Is there a way to restart it
without restarting the cluster?
Post by Hari Gowtham
Hi,
I can see that the demotion is happening from the status provided by you.
Do verify it.
I would recommend you to change the cluster.tier-demote-frequency to 120
and cluster.tier-promote-frequency to 1800 to increase the demotions until the
hot tier is emptied to a certain extent. Later you can use the values
existing now.
Post by David Brown
Hey Everyone,
I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD
cold tier.
Post by David Brown
I recently ran into some problems when the hot tier became full with
df-h showing 100%.
Post by David Brown
I did not have a watermark-hi set, but it is my understanding that
90% is the default. In an attempt to get the cluster to demote some files,
I set cluster.watermark-hi: 80 but it is still not demoting.
Post by David Brown
Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
cluster.tier-promote-frequency: 120
cluster.tier-demote-frequency: 1800
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Node Promoted files Demoted files
Status run time in h:m:s
Post by David Brown
--------- --------- ---------
--------- ---------
Post by David Brown
localhost 49 0 in
progress 5151:30:45
Post by David Brown
Glus2 0 0 in progress
5151:30:45
Post by David Brown
Glus3 0 2075 in progress
5151:30:47
Post by David Brown
Tiering Migration Functionality: FFPrimary: success
What can cause GlusterFS to stop demoting files and allow it to
completely fill the Hot Tier?
Post by David Brown
Thank you!
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Regards,
Hari Gowtham.
Amar Tumballi
2018-10-09 12:44:34 UTC
Permalink
Hi David,

Just for your information, as a project, we are not currently taking any
development focus on Tiering feature!

Please refer to email thread @
https://lists.gluster.org/pipermail//gluster-devel/2018-July/055017.html

It is recommended to use 'dmcache' on your disks to get best performance
out of your backend instead! Also note that this may get retired in next
upcoming release: https://review.gluster.org/21331

Hope this email will save you lot of time!

Regards,
Amar
Post by David Brown
Just found this in the tierd.log Not sure what it means or how to fix it
tho, but I assume it may be the cause of my problem with files not being
demoted from the hot tier..
[2018-09-30 12:25:56.438821] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
[Transport endpoint is not connected]
[2018-09-30 12:25:56.440940] W [MSGID: 114031]
remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
(10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.444633] W [MSGID: 114031]
remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
(32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.448347] W [MSGID: 114031]
remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
(389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.451919] W [MSGID: 114031]
remote operation failed. Path: //test-march-3-bc-file-501
(41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.456198] W [MSGID: 114031]
remote operation failed. Path: //krishna
(49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.459702] W [MSGID: 114031]
remote operation failed. Path: //test1
(5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.463164] W [MSGID: 114031]
remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
(5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.466601] W [MSGID: 114031]
remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
(6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.470129] W [MSGID: 114031]
remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
(7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.473758] W [MSGID: 114031]
remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
(8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.477237] W [MSGID: 114031]
remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
(91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.480696] W [MSGID: 114031]
remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
(972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.484354] W [MSGID: 114031]
remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
(a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.487884] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:56.490973] W [MSGID: 114031]
remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
(ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.494618] W [MSGID: 114031]
remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
(ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.498059] W [MSGID: 114031]
remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
(c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.501428] W [MSGID: 114031]
remote operation failed. Path: //DBTest
(c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:56.504736] W [MSGID: 114031]
remote operation failed. Path: //testCliff
(cac8d0c8-2a10-4fa1-8b94-bf47981c01ff). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
The message "I [MSGID: 109038] [tier.c:1122:tier_migrate_using_query_file]
0-FFPrimary-tier-dht: Demotion failed for
6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)"
repeated 4 times between [2018-09-30 12:25:53.164873] and [2018-09-30
12:25:57.515127]
[2018-09-30 12:25:57.520933] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
[2018-09-30 12:25:57.521545] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
[2018-09-30 12:25:57.521854] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
[Transport endpoint is not connected]
[2018-09-30 12:25:57.524079] W [MSGID: 114031]
remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
(10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.527993] W [MSGID: 114031]
remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
(32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.531829] W [MSGID: 114031]
remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
(389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.535529] W [MSGID: 114031]
remote operation failed. Path: //test-march-3-bc-file-501
(41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.539852] W [MSGID: 114031]
remote operation failed. Path: //krishna
(49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.543468] W [MSGID: 114031]
remote operation failed. Path: //test1
(5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.547051] W [MSGID: 114031]
remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
(5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.550694] W [MSGID: 114031]
remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
(6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.554498] W [MSGID: 114031]
remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
(7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.558086] W [MSGID: 114031]
remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
(8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.561568] W [MSGID: 114031]
remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
(91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.565094] W [MSGID: 114031]
remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
(972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.568562] W [MSGID: 114031]
remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
(a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.571974] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:57.574925] W [MSGID: 114031]
remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
(ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.578639] W [MSGID: 114031]
remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
(ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.582118] W [MSGID: 114031]
remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
(c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:57.585547] W [MSGID: 114031]
remote operation failed. Path: //DBTest
(c831be42-c63b-4664-b62e-1c3f9c24e201). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
The message "E [MSGID: 109037] [tier.c:2532:tier_run]
0-FFPrimary-tier-dht: Demotion failed" repeated 3 times between [2018-09-30
12:25:54.323716] and [2018-09-30 12:25:57.592361]
[2018-09-30 12:25:58.002724] I [MSGID: 0]
local subvol: FFPrimary-cold-dht,cnt = 272288612352
[2018-09-30 12:25:58.018840] I [MSGID: 0]
Total size files = 272288612352
[2018-09-30 12:25:58.018854] I
[dht-rebalance.c:4325:dht_file_counter_thread] 0-dht: tmp data size
=272288612352
The message "E [MSGID: 109037] [tier.c:720:tier_lookup_file]
0-FFPrimary-tier-dht: Failed to lookup file
6bed3d1e-17b8-11e8-a5dc-902b3450f388
[No such file or directory]" repeated 4 times between [2018-09-30
12:25:54.247352] and [2018-09-30 12:25:58.599151]
[2018-09-30 12:25:58.599162] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for
6bed3d1e-17b8-11e8-a5dc-902b3450f388(gfid:0249ff10-18aa-4727-b873-d67b42047f2d)
[2018-09-30 12:25:58.604808] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-3: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [No such file or directory]
[2018-09-30 12:25:58.605428] W [MSGID: 114031]
[client-rpc-fops.c:2860:client3_3_lookup_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
(10fb1bd9-b962-415c-8751-f0ef8bf06473) [Transport endpoint is not connected]
[2018-09-30 12:25:58.605750] E [MSGID: 114031]
[client-rpc-fops.c:233:client3_3_mknod_cbk] 0-FFPrimary-client-5: remote
operation failed. Path: <gfid:10fb1bd9-b962-415c-8751-f0ef8bf06473>
[Transport endpoint is not connected]
[2018-09-30 12:25:58.607865] W [MSGID: 114031]
remote operation failed. Path: //720ee8d5-1667-11e8-a5dc-902b3450f388
(10fb1bd9-b962-415c-8751-f0ef8bf06473). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.611433] W [MSGID: 114031]
remote operation failed. Path: //e7df9e17-b62f-4668-a4c1-dc5d86dcae6e
(32eff7dc-dcda-4488-8464-9eace06e1b69). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.615059] W [MSGID: 114031]
remote operation failed. Path: //9758be45-c466-45a2-9cd6-572f80c54da9
(389acc57-d205-4022-acea-d0f400c2ad89). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.618669] W [MSGID: 114031]
remote operation failed. Path: //test-march-3-bc-file-501
(41ee3e27-40be-4f64-af01-e18cc63065e3). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.622667] W [MSGID: 114031]
remote operation failed. Path: //krishna
(49657a82-8c64-43c0-94df-e1e78840aa1d). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.626029] W [MSGID: 114031]
remote operation failed. Path: //test1
(5ac7caba-f2c3-4bf1-bb38-cf6ed940dac0). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.629459] W [MSGID: 114031]
remote operation failed. Path: //c7947fa1-a496-400c-b6a4-b4e084b8f316
(5e909f4e-6263-4091-8378-26479496e715). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.632994] W [MSGID: 114031]
remote operation failed. Path: //ea37891d-1ab8-40f8-95a3-eee822c7040a
(6dfe1d97-34f4-440b-9502-5eab172de58a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.636669] W [MSGID: 114031]
remote operation failed. Path: //a42d1e12-fc11-4a51-a744-8e6c3b11be0a
(7a081218-3cc1-442c-be4b-43bd7dd01724). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.640155] W [MSGID: 114031]
remote operation failed. Path: //b0acb442-fe60-4022-bee2-d11d49422f20
(8788d650-9800-47ab-bf07-87f9dcd0392c). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.643516] W [MSGID: 114031]
remote operation failed. Path: //f7357147-c2ea-4abe-9c59-136f049bfccb
(91ecb7b5-84fb-48d2-af2b-440ab6f25cfa). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.648787] W [MSGID: 114031]
remote operation failed. Path: //35949c80-5496-445d-b2d6-e7d2061e9135
(972256c3-8eb8-49d5-a4ab-cca34abc7b0a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.652106] W [MSGID: 114031]
remote operation failed. Path: //13660ae8-4138-47f2-a858-8880d97b4e8d
(a6027333-b269-4810-a188-3af51c04fdcb). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.655577] I [MSGID: 109038]
[tier.c:1122:tier_migrate_using_query_file] 0-FFPrimary-tier-dht: Demotion
failed for melvin(gfid:a9b49996-ba84-4b88-b182-7a0e677749aa)
[2018-09-30 12:25:58.658482] W [MSGID: 114031]
remote operation failed. Path: //d1cb5906-e2d9-444a-9622-599509a73e3b
(ac3cd35b-d766-46c1-ae4b-a8373e35a77b). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.661703] W [MSGID: 114031]
remote operation failed. Path: //74ad729b-17dd-44d1-8b86-1db0dce862d8
(ac54d8b4-f826-48b2-ab18-75b561526689). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
[2018-09-30 12:25:58.665051] W [MSGID: 114031]
remote operation failed. Path: //0368ebe6-9ac3-4d72-9795-0a46800aa90b
(c627ff61-1819-4a0b-81ac-d53b8ab872d4). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]
The message "W [MSGID: 114031]
remote operation failed. Path: //e8a6aba1-e5ce-4ded-b474-1c5bf49b1285
(edffea55-296b-4d1b-8114-b4f8dd10920a). Key: trusted.glusterfs.node-uuid
[Transport endpoint is not connected]" repeated 3 times between [2018-09-30
12:25:55.406782] and [2018-09-30 12:25:58.674842]
[2018-09-30 12:25:58.675351] E [MSGID: 109037] [tier.c:2532:tier_run]
0-FFPrimary-tier-dht: Demotion failed
Post by David Brown
4 hours later, no files have been demoted
Status of volume: FFPrimary
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick3
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 24177
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 24.3GB
Total Disk Space : 476.7GB
Inode Count : 50877088
Free Inodes : 50874696
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick2
TCP Port : 49155
RDMA Port : 0
Online : Y
Pid : 17994
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 15.5GB
Total Disk Space : 476.7GB
Inode Count : 32560288
Free Inodes : 32557896
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick1
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 23573
File System : xfs
Device : /dev/nvme0n1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 13.8GB
Total Disk Space : 476.7GB
Inode Count : 29027000
Free Inodes : 29024515
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick5
TCP Port : 49152
RDMA Port : 0
Online : Y
Pid : 23442
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969488
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick6
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 17856
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969489
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick7
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 24018
File System : xfs
Device : /dev/sdb1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.5TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969488
------------------------------------------------------------------------------
Brick : Brick Glus1:/data/glusterfs/FFPrimary/brick8
TCP Port : 49153
RDMA Port : 0
Online : Y
Pid : 23518
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969607
------------------------------------------------------------------------------
Brick : Brick Glus2:/data/glusterfs/FFPrimary/brick9
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 17943
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969607
------------------------------------------------------------------------------
Brick : Brick Glus3:/data/glusterfs/FFPrimary/brick10
TCP Port : 49154
RDMA Port : 0
Online : Y
Pid : 24108
File System : xfs
Device : /dev/sdc1
Mount Options : rw,seclabel,relatime,attr2,inode64,noquota
Inode Size : 512
Disk Space Free : 2.7TB
Total Disk Space : 2.7TB
Inode Count : 292971904
Free Inodes : 292969604
Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Node Promoted files Demoted files Status
run time in h:m:s
--------- --------- --------- ---------
---------
localhost 49 0 in
progress 5159:15:50
Glus2 0 0 in progress
5159:15:50
Glus3 0 2075 in progress
5159:15:52
Tiering Migration Functionality: FFPrimary: success
Post by David Brown
Thank you Hari,
cluster.tier-promote-frequency: 1800
cluster.tier-demote-frequency: 120
I will let you know if it makes a difference after some time. So far (10
minutes), nothing has changed.
I would agree with you, that by looking at the result of 'gluster volume
tier FFPrimary status' it would seem that demoting is happening. However,
for the last 24hrs, nothing has changed in the tier status report except
the time. Could it be stuck? How would I know? Is there a way to restart it
without restarting the cluster?
Post by Hari Gowtham
Hi,
I can see that the demotion is happening from the status provided by you.
Do verify it.
I would recommend you to change the cluster.tier-demote-frequency to 120
and cluster.tier-promote-frequency to 1800 to increase the demotions until the
hot tier is emptied to a certain extent. Later you can use the values
existing now.
Post by David Brown
Hey Everyone,
I have a 3 node GlusterFS cluster that uses NVMe hot tier and a HDD
cold tier.
Post by David Brown
I recently ran into some problems when the hot tier became full with
df-h showing 100%.
Post by David Brown
I did not have a watermark-hi set, but it is my understanding that
90% is the default. In an attempt to get the cluster to demote some files,
I set cluster.watermark-hi: 80 but it is still not demoting.
Post by David Brown
Volume Name: FFPrimary
Type: Tier
Volume ID: 466ec53c-d1ef-4ebc-8414-d7d070dfe61e
Status: Started
Snapshot Count: 0
Number of Bricks: 9
Transport-type: tcp
Hot Tier Type : Replicate
Number of Bricks: 1 x 3 = 3
Brick1: Glus3:/data/glusterfs/FFPrimary/brick3
Brick2: Glus2:/data/glusterfs/FFPrimary/brick2
Brick3: Glus1:/data/glusterfs/FFPrimary/brick1
Cold Tier Type : Distributed-Replicate
Number of Bricks: 2 x 3 = 6
Brick4: Glus1:/data/glusterfs/FFPrimary/brick5
Brick5: Glus2:/data/glusterfs/FFPrimary/brick6
Brick6: Glus3:/data/glusterfs/FFPrimary/brick7
Brick7: Glus1:/data/glusterfs/FFPrimary/brick8
Brick8: Glus2:/data/glusterfs/FFPrimary/brick9
Brick9: Glus3:/data/glusterfs/FFPrimary/brick10
cluster.tier-promote-frequency: 120
cluster.tier-demote-frequency: 1800
cluster.watermark-low: 60
cluster.watermark-hi: 80
performance.flush-behind: on
performance.cache-max-file-size: 128MB
performance.cache-size: 25GB
diagnostics.count-fop-hits: off
diagnostics.latency-measurement: off
cluster.tier-mode: cache
features.ctr-enabled: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
Node Promoted files Demoted files
Status run time in h:m:s
Post by David Brown
--------- --------- ---------
--------- ---------
Post by David Brown
localhost 49 0 in
progress 5151:30:45
Post by David Brown
Glus2 0 0 in progress
5151:30:45
Post by David Brown
Glus3 0 2075 in progress
5151:30:47
Post by David Brown
Tiering Migration Functionality: FFPrimary: success
What can cause GlusterFS to stop demoting files and allow it to
completely fill the Hot Tier?
Post by David Brown
Thank you!
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Regards,
Hari Gowtham.
_______________________________________________
Gluster-users mailing list
https://lists.gluster.org/mailman/listinfo/gluster-users
--
Amar Tumballi (amarts)
Loading...