Discussion:
Fwd: Glusterfs over wan?
(too old to reply)
s***@shayn.ch
2010-05-27 13:31:10 UTC
Permalink
Hello !

sorry for english, but, just don't understand why glusterfs is not
destinated to be used over the wan ?
I think that if the files are tiny, there's no problem to use this
solution over wan.. ?

But okay, if file are large.. (don't know .. 100MB - 200MB and more.) i
understand that it depend of the internet connection and that's the
replication can be so soo soo slow..

Please, give me some renseignments about this solution over wan

Shayn
Count Zero
2010-05-27 13:36:54 UTC
Permalink
Hi Shayn,

The problem is not replication speed. Actually that part is just fine, and it's quite fast.

The problem with WAN is that when the gluster client receives a request to read the file, it first checks with all the nodes in the cluster, to make sure there are no discrepancies. Only after all nodes have answered, it will read the local file (if it's replicated locally).

Also take into account that international links are not as reliable as local LAN based links. If a node is suddenly inaccessible, it can slow down everything (again because READ operations are dependent on all nodes in the cluster answering, synchronously).

I know that the gluster team are working on an async solution, down the road, which would make glusterfs more suitable for WAN scenarios.

My suggestion is NOT to try it, until the gluster team officially announce WAN support.
Post by s***@shayn.ch
Hello !
sorry for english, but, just don't understand why glusterfs is not
destinated to be used over the wan ?
I think that if the files are tiny, there's no problem to use this
solution over wan.. ?
But okay, if file are large.. (don't know .. 100MB - 200MB and more.) i
understand that it depend of the internet connection and that's the
replication can be so soo soo slow..
Please, give me some renseignments about this solution over wan
Shayn_______________________________________________
Gluster-users mailing list
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
s***@shayn.ch
2010-05-27 13:05:28 UTC
Permalink
Hello !

sorry for english, but, just don't understand why glusterfs is not
destinated to be used over the wan ?
I think that if the files are tiny, there's no problem to use this
solution over wan.. ?

But okay, if file are large.. (don't know .. 100MB - 200MB and more.) i
understand that it depend of the internet connection and that's the
replication can be so soo soo slow..

Please, give me some renseignments about this solution over wan

Shayn
m***@5-cent.us
2010-05-27 14:42:42 UTC
Permalink
Excuse me, can you say me how I answer to a thread on the maillist ? ..
I think that normally I musn't reply to the mail that I received..
ARGH!!! I *hate* the configuration of this list - I intended it to be
posted that way. You have to either hit <reply all>, or cut and paste the
list address in, where *none* of the mailing lists I've been on for 10-15
years are that way - reply is to the list.
First, thank you for your answer..
And then, imagine this scenario..
We have just two gluster serveur with a replicated volume betweem those
server..> The connexion between the two site (geographically sepate) is
one fiber
connection (1gb symetric)
What do you think..? A gluster implementation is now possible with this
conditions ?
How many km apart? And again, do the test I suggested: ping on your LAN,
then ping on your WAN, and see the difference.

mark
Thanks a lot for the time you take for me.. !
Post by s***@shayn.ch
Hello !
sorry for english, but, just don't understand why glusterfs is not
destinated to be used over the wan ?
I think that if the files are tiny, there's no problem to use this
solution over wan.. ?
But okay, if file are large.. (don't know .. 100MB - 200MB and more.) i
understand that it depend of the internet connection and that's the
replication can be so soo soo slow..
Please, give me some renseignments about this solution over wan
Ok, think of this: ping a server over your LAN. Now ping the server on
the
WAN you want to replicate to. What's the difference in milliseconds?
Now
multiply that difference by the number of small files, and make a wild
stab at how many packets each will need, and multiply by *that*.
How many seconds, not milliseconds, per transaction?
Just because it's moving at close to the speed of light does *not*
mean
there is no measurable time, and it *certainly* does not mean that it
doesn't multiply fast.
mark
Count Zero
2010-05-27 14:48:01 UTC
Permalink
I agree with you, something is wrong with the configuration of this list, specifically.
I have to click "Reply All", then remove the specific person's email address.
If I click just "Reply", it will reply directly to the person, not to the list.

Works fine for me on other lists...
Post by m***@5-cent.us
Excuse me, can you say me how I answer to a thread on the maillist ? ..
I think that normally I musn't reply to the mail that I received..
ARGH!!! I *hate* the configuration of this list - I intended it to be
posted that way. You have to either hit <reply all>, or cut and paste the
list address in, where *none* of the mailing lists I've been on for 10-15
years are that way - reply is to the list.
First, thank you for your answer..
And then, imagine this scenario..
We have just two gluster serveur with a replicated volume betweem those
server..> The connexion between the two site (geographically sepate) is
one fiber
connection (1gb symetric)
What do you think..? A gluster implementation is now possible with this
conditions ?
How many km apart? And again, do the test I suggested: ping on your LAN,
then ping on your WAN, and see the difference.
mark
Thanks a lot for the time you take for me.. !
Post by s***@shayn.ch
Hello !
sorry for english, but, just don't understand why glusterfs is not
destinated to be used over the wan ?
I think that if the files are tiny, there's no problem to use this
solution over wan.. ?
But okay, if file are large.. (don't know .. 100MB - 200MB and more.) i
understand that it depend of the internet connection and that's the
replication can be so soo soo slow..
Please, give me some renseignments about this solution over wan
Ok, think of this: ping a server over your LAN. Now ping the server on
the
WAN you want to replicate to. What's the difference in milliseconds?
Now
multiply that difference by the number of small files, and make a wild
stab at how many packets each will need, and multiply by *that*.
How many seconds, not milliseconds, per transaction?
Just because it's moving at close to the speed of light does *not*
mean
there is no measurable time, and it *certainly* does not mean that it
doesn't multiply fast.
mark
_______________________________________________
Gluster-users mailing list
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Patricio A. Bruna
2010-05-27 16:30:12 UTC
Permalink
Hi,
just for information, what file system are available to use over WAN?


------------------------------------
Patricio Bruna V.
IT Linux Ltda.
www.it-linux.cl
Fono : (+56-2) 333 0578
Móvil: (+56-9) 7776 2899








Hi Shayn,

The problem is not replication speed. Actually that part is just fine, and it's quite fast.

The problem with WAN is that when the gluster client receives a request to read the file, it first checks with all the nodes in the cluster, to make sure there are no discrepancies. Only after all nodes have answered, it will read the local file (if it's replicated locally).

Also take into account that international links are not as reliable as local LAN based links. If a node is suddenly inaccessible, it can slow down everything (again because READ operations are dependent on all nodes in the cluster answering, synchronously).

I know that the gluster team are working on an async solution, down the road, which would make glusterfs more suitable for WAN scenarios.

My suggestion is NOT to try it, until the gluster team officially announce WAN support.
Post by s***@shayn.ch
Hello !
sorry for english, but, just don't understand why glusterfs is not
destinated to be used over the wan ?
I think that if the files are tiny, there's no problem to use this
solution over wan.. ?
But okay, if file are large.. (don't know .. 100MB - 200MB and more.) i
understand that it depend of the internet connection and that's the
replication can be so soo soo slow..
Please, give me some renseignments about this solution over wan
Shayn_______________________________________________
Gluster-users mailing list
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Tejas N. Bhise
2010-05-27 18:07:39 UTC
Permalink
Patricio,

All networked filesystems like nfs, cifs/samba and even GlusterFS can be used over WAN. The problem arises for the more complex functionality, like replication across a WAN.

Replication over WAN has different meanings for different users - some just want a good way to have a remote backup, others want a hot-cold kind of a backup so namespace issue come into the picture, but its relatively easier to solve as there is only one side active at a time.

The most difficult one is to keep both sites active ( and R/W ) with their own full namespace and security functionality with a designated unit of the filesystem replicated. Access from each side looks like it just pulled data from its own data cache and the currency of data could move around ( potentially ) multiple locations across the globe.

I don't think anyone does that last kind of replication. AFS and DFS from the old days did go upto a certain point by providing R/O replication with remote snapshots by providing integrated multisite security and namespace, but no R/W replication.

GlusterFS plan is to start with async WAN replication for remote backup and slowly moving into the multisite cluster architecture and active active replicated data copies with integrated namespace, which we hope to bring in with a new unified ( write ) caching.

Async WAN replication for remote backup might come in as early as end of the year with the other functionality following sometime after that.

I would like to hear more about how community users will put these different levels of WAN replication to use, how much data would change on a daily basis and hence get moved around, what kind of latency would be acceptable to your applications ? I would also like to hear if anyone is using other solutions/edge caching appliances etc to do this in other ways.

Regards,
Tejas.

----- Original Message -----
From: "Patricio A. Bruna" <***@it-linux.cl>
To: gluster-***@gluster.org
Sent: Thursday, May 27, 2010 10:00:12 PM
Subject: Re: [Gluster-users] Fwd: Glusterfs over wan?

Hi,
just for information, what file system are available to use over WAN?


------------------------------------
Patricio Bruna V.
IT Linux Ltda.
www.it-linux.cl
Fono : (+56-2) 333 0578
Móvil: (+56-9) 7776 2899








Hi Shayn,

The problem is not replication speed. Actually that part is just fine, and it's quite fast.

The problem with WAN is that when the gluster client receives a request to read the file, it first checks with all the nodes in the cluster, to make sure there are no discrepancies. Only after all nodes have answered, it will read the local file (if it's replicated locally).

Also take into account that international links are not as reliable as local LAN based links. If a node is suddenly inaccessible, it can slow down everything (again because READ operations are dependent on all nodes in the cluster answering, synchronously).

I know that the gluster team are working on an async solution, down the road, which would make glusterfs more suitable for WAN scenarios.

My suggestion is NOT to try it, until the gluster team officially announce WAN support.
Post by s***@shayn.ch
Hello !
sorry for english, but, just don't understand why glusterfs is not
destinated to be used over the wan ?
I think that if the files are tiny, there's no problem to use this
solution over wan.. ?
But okay, if file are large.. (don't know .. 100MB - 200MB and more.) i
understand that it depend of the internet connection and that's the
replication can be so soo soo slow..
Please, give me some renseignments about this solution over wan
Shayn_______________________________________________
Gluster-users mailing list
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
Gluster-***@gluster.org
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Jeffrey Negro
2010-05-27 18:59:44 UTC
Permalink
I'm looking forward to this feature. I'm planning on having 2 clusters - a
primary and a DR. I would like to be able to replicate data from the
production cluster to the DR, but would also like to see two way replication
so that when I eventually implement BGP in a dual homed scenario I could
have two independent systems replicating back and forth.


Jeffrey Negro, Network Engineer
Billtrust - Improving Your Billing, Improving Your Business
www.billtrust.com
609.235.1010 x137
Post by Tejas N. Bhise
Patricio,
All networked filesystems like nfs, cifs/samba and even GlusterFS can be
used over WAN. The problem arises for the more complex functionality, like
replication across a WAN.
Replication over WAN has different meanings for different users - some just
want a good way to have a remote backup, others want a hot-cold kind of a
backup so namespace issue come into the picture, but its relatively easier
to solve as there is only one side active at a time.
The most difficult one is to keep both sites active ( and R/W ) with their
own full namespace and security functionality with a designated unit of the
filesystem replicated. Access from each side looks like it just pulled data
from its own data cache and the currency of data could move around (
potentially ) multiple locations across the globe.
I don't think anyone does that last kind of replication. AFS and DFS from
the old days did go upto a certain point by providing R/O replication with
remote snapshots by providing integrated multisite security and namespace,
but no R/W replication.
GlusterFS plan is to start with async WAN replication for remote backup and
slowly moving into the multisite cluster architecture and active active
replicated data copies with integrated namespace, which we hope to bring in
with a new unified ( write ) caching.
Async WAN replication for remote backup might come in as early as end of
the year with the other functionality following sometime after that.
I would like to hear more about how community users will put these
different levels of WAN replication to use, how much data would change on a
daily basis and hence get moved around, what kind of latency would be
acceptable to your applications ? I would also like to hear if anyone is
using other solutions/edge caching appliances etc to do this in other ways.
Regards,
Tejas.
----- Original Message -----
Sent: Thursday, May 27, 2010 10:00:12 PM
Subject: Re: [Gluster-users] Fwd: Glusterfs over wan?
Hi,
just for information, what file system are available to use over WAN?
------------------------------------
Patricio Bruna V.
IT Linux Ltda.
www.it-linux.cl
Fono : (+56-2) 333 0578
Móvil: (+56-9) 7776 2899
Hi Shayn,
The problem is not replication speed. Actually that part is just fine, and it's quite fast.
The problem with WAN is that when the gluster client receives a request to
read the file, it first checks with all the nodes in the cluster, to make
sure there are no discrepancies. Only after all nodes have answered, it will
read the local file (if it's replicated locally).
Also take into account that international links are not as reliable as
local LAN based links. If a node is suddenly inaccessible, it can slow down
everything (again because READ operations are dependent on all nodes in the
cluster answering, synchronously).
I know that the gluster team are working on an async solution, down the
road, which would make glusterfs more suitable for WAN scenarios.
My suggestion is NOT to try it, until the gluster team officially announce WAN support.
Post by s***@shayn.ch
Hello !
sorry for english, but, just don't understand why glusterfs is not
destinated to be used over the wan ?
I think that if the files are tiny, there's no problem to use this
solution over wan.. ?
But okay, if file are large.. (don't know .. 100MB - 200MB and more.) i
understand that it depend of the internet connection and that's the
replication can be so soo soo slow..
Please, give me some renseignments about this solution over wan
Shayn_______________________________________________
Gluster-users mailing list
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
_______________________________________________
Gluster-users mailing list
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
s***@shayn.ch
2010-05-28 09:13:32 UTC
Permalink
Thanks for explain..

So, you think, that with glusterfs, we can't have a replication between
two (or more) different site over the wan, when both site are active
(read and write functions) ?
Post by Tejas N. Bhise
Patricio,
All networked filesystems like nfs, cifs/samba and even GlusterFS can
be used over WAN. The problem arises for the more complex
functionality, like replication across a WAN.
Replication over WAN has different meanings for different users - some
just want a good way to have a remote backup, others want a hot-cold
kind of a backup so namespace issue come into the picture, but its
relatively easier to solve as there is only one side active at a time.
The most difficult one is to keep both sites active ( and R/W ) with
their own full namespace and security functionality with a designated
unit of the filesystem replicated. Access from each side looks like it
just pulled data from its own data cache and the currency of data
could move around ( potentially ) multiple locations across the globe.
I don't think anyone does that last kind of replication. AFS and DFS
from the old days did go upto a certain point by providing R/O
replication with remote snapshots by providing integrated multisite
security and namespace, but no R/W replication.
GlusterFS plan is to start with async WAN replication for remote
backup and slowly moving into the multisite cluster architecture and
active active replicated data copies with integrated namespace, which
we hope to bring in with a new unified ( write ) caching.
Async WAN replication for remote backup might come in as early as end
of the year with the other functionality following sometime after
that.
I would like to hear more about how community users will put these
different levels of WAN replication to use, how much data would change
on a daily basis and hence get moved around, what kind of latency
would be acceptable to your applications ? I would also like to hear
if anyone is using other solutions/edge caching appliances etc to do
this in other ways.
Regards,
Tejas.
----- Original Message -----
Sent: Thursday, May 27, 2010 10:00:12 PM
Subject: Re: [Gluster-users] Fwd: Glusterfs over wan?
Hi,
just for information, what file system are available to use over WAN?
------------------------------------
Patricio Bruna V.
IT Linux Ltda.
<http://www.it-linux.cl>
Fono : (+56-2) 333 0578
Móvil: (+56-9) 7776 2899
Hi Shayn,
The problem is not replication speed. Actually that part is just fine, and it's quite fast.
The problem with WAN is that when the gluster client receives a
request to read the file, it first checks with all the nodes in the
cluster, to make sure there are no discrepancies. Only after all nodes
have answered, it will read the local file (if it's replicated
locally).
Also take into account that international links are not as reliable as
local LAN based links. If a node is suddenly inaccessible, it can slow
down everything (again because READ operations are dependent on all
nodes in the cluster answering, synchronously).
I know that the gluster team are working on an async solution, down
the road, which would make glusterfs more suitable for WAN scenarios.
My suggestion is NOT to try it, until the gluster team officially announce WAN support.
Post by s***@shayn.ch
Hello !
sorry for english, but, just don't understand why glusterfs is not
destinated to be used over the wan ?
I think that if the files are tiny, there's no problem to use this
solution over wan.. ?
But okay, if file are large.. (don't know .. 100MB - 200MB and more.) i
understand that it depend of the internet connection and that's the
replication can be so soo soo slow..
Please, give me some renseignments about this solution over wan
Shayn_______________________________________________
Gluster-users mailing list
<http://gluster.org/cgi-bin/mailman/listinfo/gluster-users>
_______________________________________________
Gluster-users mailing list
<http://gluster.org/cgi-bin/mailman/listinfo/gluster-users>
_______________________________________________
Gluster-users mailing list
<http://gluster.org/cgi-bin/mailman/listinfo/gluster-users>
_______________________________________________
Gluster-users mailing list
<http://gluster.org/cgi-bin/mailman/listinfo/gluster-users>
Michiel van Es
2010-05-28 09:25:18 UTC
Permalink
You can do it but performance wise I would wait for the newer versions..

Michiel

-------- Original Message --------
Subject: Re: [Gluster-users] Fwd: Glusterfs over wan?
From: ***@shayn.ch
To: Patricio A. Bruna <***@it-linux.cl>, Tejas N. Bhise
<***@gluster.com>
Date: 05/28/2010 11:13 AM
Post by s***@shayn.ch
Thanks for explain..
So, you think, that with glusterfs, we can't have a replication between
two (or more) different site over the wan, when both site are active
(read and write functions) ?
Post by Tejas N. Bhise
Patricio,
All networked filesystems like nfs, cifs/samba and even GlusterFS can
be used over WAN. The problem arises for the more complex
functionality, like replication across a WAN.
Replication over WAN has different meanings for different users - some
just want a good way to have a remote backup, others want a hot-cold
kind of a backup so namespace issue come into the picture, but its
relatively easier to solve as there is only one side active at a time.
The most difficult one is to keep both sites active ( and R/W ) with
their own full namespace and security functionality with a designated
unit of the filesystem replicated. Access from each side looks like it
just pulled data from its own data cache and the currency of data
could move around ( potentially ) multiple locations across the globe.
I don't think anyone does that last kind of replication. AFS and DFS
from the old days did go upto a certain point by providing R/O
replication with remote snapshots by providing integrated multisite
security and namespace, but no R/W replication.
GlusterFS plan is to start with async WAN replication for remote
backup and slowly moving into the multisite cluster architecture and
active active replicated data copies with integrated namespace, which
we hope to bring in with a new unified ( write ) caching.
Async WAN replication for remote backup might come in as early as end
of the year with the other functionality following sometime after
that.
I would like to hear more about how community users will put these
different levels of WAN replication to use, how much data would change
on a daily basis and hence get moved around, what kind of latency
would be acceptable to your applications ? I would also like to hear
if anyone is using other solutions/edge caching appliances etc to do
this in other ways.
Regards,
Tejas.
----- Original Message -----
Sent: Thursday, May 27, 2010 10:00:12 PM
Subject: Re: [Gluster-users] Fwd: Glusterfs over wan?
Hi,
just for information, what file system are available to use over WAN?
------------------------------------
Patricio Bruna V.
IT Linux Ltda.
<http://www.it-linux.cl>
Fono : (+56-2) 333 0578
Móvil: (+56-9) 7776 2899
Hi Shayn,
The problem is not replication speed. Actually that part is just fine,
and it's quite fast.
The problem with WAN is that when the gluster client receives a
request to read the file, it first checks with all the nodes in the
cluster, to make sure there are no discrepancies. Only after all nodes
have answered, it will read the local file (if it's replicated
locally).
Also take into account that international links are not as reliable as
local LAN based links. If a node is suddenly inaccessible, it can slow
down everything (again because READ operations are dependent on all
nodes in the cluster answering, synchronously).
I know that the gluster team are working on an async solution, down
the road, which would make glusterfs more suitable for WAN scenarios.
My suggestion is NOT to try it, until the gluster team officially announce WAN support.
Post by s***@shayn.ch
Hello !
sorry for english, but, just don't understand why glusterfs is not
destinated to be used over the wan ?
I think that if the files are tiny, there's no problem to use this
solution over wan.. ?
But okay, if file are large.. (don't know .. 100MB - 200MB and more.) i
understand that it depend of the internet connection and that's the
replication can be so soo soo slow..
Please, give me some renseignments about this solution over wan
Shayn_______________________________________________
Gluster-users mailing list
<http://gluster.org/cgi-bin/mailman/listinfo/gluster-users>
_______________________________________________
Gluster-users mailing list
<http://gluster.org/cgi-bin/mailman/listinfo/gluster-users>
_______________________________________________
Gluster-users mailing list
<http://gluster.org/cgi-bin/mailman/listinfo/gluster-users>
_______________________________________________
Gluster-users mailing list
<http://gluster.org/cgi-bin/mailman/listinfo/gluster-users>
_______________________________________________
Gluster-users mailing list
http://gluster.org/cgi-bin/mailman/listinfo/gluster-users
Loading...