[Gluster-users] Is the size of bricks limiting the size of files I can store?

Post by Andreas Davour
Hi
I've found something that works so weird I'm certain I have missed how
gluster is supposed to be used, but I can not figure out how. This is my
scenario.
I have a volume, created from 16 nodes, each with a brick of the same
size. The total of that volume thus is in the Terabyte scale. It's a
distributed volume with a replica count of 2.
The filesystem when mounted on the clients is not even close to getting
full, as displayed by 'df'.
But, when one of my users try to copy a file from another network storage
to the gluster volume, he gets a 'filesystem full' error. What happened? I
looked at the bricks and figured out that one big file had ended up on a
brick that was half full or so, and the big file did not fit in the space
that was left on that brick.

Hi,

This is working as expected. As files are not split up (unless you are
using shards) the size of the file is restricted by the size of the
individual bricks.

Post by Andreas Davour
This resulted in the absurd situation that the user could see a filesystem
with massive amount of free space, but still got a filesystem full error.
This is misleading, yes. Do you have any numbers for the size of the file

vs the size of the brick?

Post by Andreas Davour
The only workaround I've found is to set some values for max free size,
but this is a very wonky solution, as those values will fluctuate as the
filesystem is used.
Surely I'm missing something obvious? A filesystem that has much free
space should not randomly give an error like that?
Interested to hear some best practices for this kind of stuff.
/andreas
--
"economics is a pseudoscience; the astrology of our time"
Kim Stanley Robinson
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

Andreas Davour

2018-04-02 18:07:33 UTC

Hi,
This is working as expected. As files are not split up (unless you are
using shards) the size of the file is restricted by the size of the
individual bricks.

Thanks a lot for that definitive answer. Is there a way to manage this?
Can you shard just those files, making them replicated in the process?

I just can't have users see 15TB free and fail copying a 15GB file. They
will show me the bill they paid for those "disks" and flay me.

-andreas

--
"economics is a pseudoscience; the astrology of our time"
Kim Stanley Robinson

Jim Kinney

2018-04-02 20:54:35 UTC

Hi,
This is working as expected. As files are not split up (unless you are
using shards) the size of the file is restricted by the size of the
individual bricks.

Thanks a lot for that definitive answer. Is there a way to manage this?
Can you shard just those files, making them replicated in the
process?

I manage this by using thin pool, thin lvm and add new drives to the
lvm across all gluster nodes and expand the user space. My thinking on
this is a RAID 10 with the RAID 0 in the lvm and the RAID1 handled by
gluster replica 2+ :-)

Post by Andreas Davour
I just can't have users see 15TB free and fail copying a 15GB file. They
will show me the bill they paid for those "disks" and flay me.
-andreas
--
"economics is a pseudoscience; the astrology of our time"
Kim Stanley Robinson
_______________________________________________
Gluster-users mailing list
http://lists.gluster.org/mailman/listinfo/gluster-users

--
James P. Kinney III

Every time you stop a school, you will have to build a jail. What you
gain at one end you lose at the other. It's like feeding a dog on his
own tail. It won't fatten the dog.
- Speech 11/23/1900 Mark Twain

http://heretothereideas.blogspot.com/

Andreas Davour

2018-04-12 19:48:32 UTC

Post by Jim Kinney

Post by Andreas Davour
Hi
I've found something that works so weird I'm certain I have
missed how
gluster is supposed to be used, but I can not figure out how. This is my
scenario.
I have a volume, created from 16 nodes, each with a brick of the same
size. The total of that volume thus is in the Terabyte scale. It's a
distributed volume with a replica count of 2.
The filesystem when mounted on the clients is not even close to getting
full, as displayed by 'df'.
But, when one of my users try to copy a file from another network storage
to the gluster volume, he gets a 'filesystem full' error. What happened? I
looked at the bricks and figured out that one big file had ended up on a
brick that was half full or so, and the big file did not fit in the space
that was left on that brick.

Hi,
This is working as expected. As files are not split up (unless you are
using shards) the size of the file is restricted by the size of the
individual bricks.

Thanks a lot for that definitive answer. Is there a way to manage this?
Can you shard just those files, making them replicated in the
process?

I'm not sure I see how that solved the problem, but as you have though
it through I think you are trying to say something I should understand.

/andreas

--
"economics is a pseudoscience; the astrology of our time"
Kim Stanley Robinson

Jim Kinney

2018-04-13 12:02:16 UTC

Post by Jim Kinney

Hi,
This is working as expected. As files are not split up (unless you are
using shards) the size of the file is restricted by the size of the
individual bricks.

Thanks a lot for that definitive answer. Is there a way to manage this?
Can you shard just those files, making them replicated in the process?

I manage this by using thin pool, thin lvm and add new drives to the
lvm across all gluster nodes and expand the user space. My thinking

Post by Jim Kinney
this is a RAID 10 with the RAID 0 in the lvm and the RAID1 handled by
gluster replica 2+ :-)

I'm not sure I see how that solved the problem, but as you have though
it through I think you are trying to say something I should understand.
/andreas

By adding space to a logical volume, effectively below the control of gluster, the entire space is available for users. Gluster manages replication across hosts and lvm provides absolute space allocation on each host.

So I have 3 hosts, replica 3, and 12 bricks on each host, 1 brick for each mount point the clients see. Some bricks are a single drive while others are 2 and 1 is 5 drives. That same lvm setup is replicated on all 3 hosts. Now a client wants more storage. They buy 3 new drives, 1 for each host. Each host gets the lvm command queued up to add the new drive to the volume for that client. Then in parallel, all 3 hosts expand the volume along with a filesystem resize. In about 2 seconds gluster picks up the change in size. Since this size change is at the host filesystem level, a file larger than the remaining space on the original drive can be written as lvm will simply span the physical volumes. Gluster would choke and not span bricks.

Post by Andreas Davour
--
"economics is a pseudoscience; the astrology of our time"
Kim Stanley Robinson

--
Sent from my Android device with K-9 Mail. All tyopes are thumb related and reflect authenticity.

Raghavendra Gowdappa

2018-04-03 02:19:30 UTC

Hi,
This is working as expected. As files are not split up (unless you are
using shards) the size of the file is restricted by the size of the
individual bricks.

Thanks a lot for that definitive answer. Is there a way to manage this?
Can you shard just those files, making them replicated in the process?

+Krutika, xlator/shard maintainer for the answer.

Andreas Davour

2018-04-12 19:49:20 UTC

Post by Raghavendra Gowdappa

Hi,
This is working as expected. As files are not split up (unless you are
using shards) the size of the file is restricted by the size of the
individual bricks.

Thanks a lot for that definitive answer. Is there a way to manage this?
Can you shard just those files, making them replicated in the process?

+Krutika, xlator/shard maintainer for the answer.

Post by Andreas Davour
I just can't have users see 15TB free and fail copying a 15GB file. They
will show me the bill they paid for those "disks" and flay me.

Any input on that Krutika?

/andreas
--
"economics is a pseudoscience; the astrology of our time"
Kim Stanley Robinson

Krutika Dhananjay

2018-04-13 09:51:26 UTC

Sorry about the late reply, I missed seeing your mail.

To begin with, what is your use-case? Sharding is currently supported only
for virtual machine image storage use-case.
It *could* work in other single-writer use-cases but it's only tested
thoroughly for the vm use-case.
If yours is not a vm store use-case, you might want to do some tests first
to see if it works fine.
If you find any issues, you can raise a bug. I'll be more than happy to fix
them.

Post by Raghavendra Gowdappa

Post by Nithya Balachandran
Hi

Post by Andreas Davour
I've found something that works so weird I'm certain I have missed how
gluster is supposed to be used, but I can not figure out how. This is my
scenario.
I have a volume, created from 16 nodes, each with a brick of the same
size. The total of that volume thus is in the Terabyte scale. It's a
distributed volume with a replica count of 2.
The filesystem when mounted on the clients is not even close to getting
full, as displayed by 'df'.
But, when one of my users try to copy a file from another network storage
to the gluster volume, he gets a 'filesystem full' error. What
happened?
I
looked at the bricks and figured out that one big file had ended up on a
brick that was half full or so, and the big file did not fit in the space
that was left on that brick.
Hi,

This is working as expected. As files are not split up (unless you are
using shards) the size of the file is restricted by the size of the
individual bricks.

Thanks a lot for that definitive answer. Is there a way to manage this?
Can you shard just those files, making them replicated in the process?

Is your question about whether you can shard just that big file that caused
space to run out and keep the rest of the files unsharded?
This is a bit tricky. From the time you enable sharding on your volume, all
newly created shards will get sharded once their size
exceeds features.shard-block-size value (which is configurable) because
it's a volume-wide option.

As for volumes which have pre-existing data even before shard is enabled,
for you to shard them, you'll need to perform either of the two steps below:

1. move the existing file to a local fs from your glusterfs volume and then
move it back into the volume.
2. copy the existing file into a temporary file on the same volume and
rename the file back to its original name.

-Krutika