Brad Fitzpatrick wrote:
- Clustering LiveJournal
I wrote up my plans for scaling LiveJournal into infinity and beyond. Read it: http://www.livejournal.com/misc/clusterlj/
Hmmm...
[at 1356h today]
- Q: When are the big plans scheduled to take place?
Brad: They'll take place slowly over the next few weeks.
Excuse me while I go back up my entries now... and make sure I have the Paypal Confirmation of Payment for both my account and Esmerelda's... check.
Comments
Of course, I can see other troubles, but eh, I'm not too concerned... maybe I'll go re-read and see where the QA Bride has problems with. I mean, the "ViSiO" is superb quality....
I don't think we'll be saturating our 100mbit line anytime soon... and if we do, we have some far more serious problems (like, how can we pay for that!@#). :)
Has it been decided what the basis for the clusters will be?
Gigabit hardware is mad expensive, and we have very little money. We do our best, even though we barely know anything. :(
But yeah, planning for the long-term future... who knows. As I vaguely referred to above, once we can saturate a 100mbit line, we'll probably need to have some sort of nonlocal clustering of servers, like hosting servers in other locations. But first, it'd be nice to be able to pay Brad a living wage... :\
What do you mean by basis? (Feel free to use baby terms; I know nothing about clustering.)
Haha... okay, I'll stop ribbing you about it then =)
What do you mean by basis?
Well, there was talk about splitting the users up by geographical location, time zone... neither of which, I think is a good idea because then, my question would be: where do Communities live (geographically and cluster-wise)? And then there was one suggestion about splitting alphabetically by the first letter of the username, which makes a little more sense to me because it makes the clusters a little more random. I was just wondering if this has been decided yet...
The trick is to reduce cross-cluster communication. Ideally, the clusters would never have to intercommunicate, but that's impossible-- as you said, communities will probably have to be arbitrarily assigned to a cluster, and they have members everywhere. The major load on the site (the major *use* of the site) is the friends view, which pulls a bunch of data from each of your friends, so the best way to split people will be whichever way puts everyone with their friends.
Of course, it's impossible to do it well, but splitting based on username (for example) will do a worse job than splitting by geography (because I'm more likely to know people in my area than people in other places).
But who knows... none of the code has been written yet.
That's exactly why I was wondering about the soundness of the proposed architecture.
because I'm more likely to know people in my area than people in other places
Heh... I'm not. =) Most of the users in my Fiends List are not within a few hours' drive from me.
You have your choice of XML or CSV as an export format. =)