?

Log in

No account? Create an account

Previous Entry | Next Entry

LiveJournal Re-Architecturing

Brad Fitzpatrick wrote:


Hmmm...

[at 1356h today]

    Q: When are the big plans scheduled to take place?
    Brad: They'll take place slowly over the next few weeks.

Excuse me while I go back up my entries now... and make sure I have the Paypal Confirmation of Payment for both my account and Esmerelda's... check.

Comments

( 11 comments — Leave a comment )
ntang
Dec. 13th, 2001 12:05 pm (UTC)
Why hmmmm....? :)
bride
Dec. 13th, 2001 12:17 pm (UTC)
I'm hmm-ing, I guess, wondering just how sound that idea is. Yes, it makes sense in general.
xinit
Dec. 13th, 2001 01:47 pm (UTC)
I'm worried just based on the existence of the single pair of Big/IPs at the head end... they don't scale past two, so there's a real network bandwidth limitation there...

Of course, I can see other troubles, but eh, I'm not too concerned... maybe I'll go re-read and see where the QA Bride has problems with. I mean, the "ViSiO" is superb quality....
evan
Dec. 13th, 2001 06:46 pm (UTC)
LJ currently uses a sustained 4mbit of traffic, and peaks around 14mbit.
I don't think we'll be saturating our 100mbit line anytime soon... and if we do, we have some far more serious problems (like, how can we pay for that!@#). :)
bride
Dec. 13th, 2001 06:56 pm (UTC)
"No one will ever need more than 64K of memory." -- W. H. Gates III

Has it been decided what the basis for the clusters will be?
evan
Dec. 13th, 2001 07:07 pm (UTC)
...blah. Whenever I talk about real-life limits, someone always brings up that quote. I've seen it at least five times in LiveJournal-related discussions. :(
Gigabit hardware is mad expensive, and we have very little money. We do our best, even though we barely know anything. :(

But yeah, planning for the long-term future... who knows. As I vaguely referred to above, once we can saturate a 100mbit line, we'll probably need to have some sort of nonlocal clustering of servers, like hosting servers in other locations. But first, it'd be nice to be able to pay Brad a living wage... :\


What do you mean by basis? (Feel free to use baby terms; I know nothing about clustering.)
bride
Dec. 13th, 2001 07:16 pm (UTC)
someone always brings up that quote.

Haha... okay, I'll stop ribbing you about it then =)

What do you mean by basis?

Well, there was talk about splitting the users up by geographical location, time zone... neither of which, I think is a good idea because then, my question would be: where do Communities live (geographically and cluster-wise)? And then there was one suggestion about splitting alphabetically by the first letter of the username, which makes a little more sense to me because it makes the clusters a little more random. I was just wondering if this has been decided yet...
evan
Dec. 13th, 2001 07:34 pm (UTC)
Hm... theoretically, we can always tweak the clustering after it's up.

The trick is to reduce cross-cluster communication. Ideally, the clusters would never have to intercommunicate, but that's impossible-- as you said, communities will probably have to be arbitrarily assigned to a cluster, and they have members everywhere. The major load on the site (the major *use* of the site) is the friends view, which pulls a bunch of data from each of your friends, so the best way to split people will be whichever way puts everyone with their friends.

Of course, it's impossible to do it well, but splitting based on username (for example) will do a worse job than splitting by geography (because I'm more likely to know people in my area than people in other places).

But who knows... none of the code has been written yet.
bride
Dec. 13th, 2001 07:45 pm (UTC)
The trick is to reduce cross-cluster communication. Ideally, the clusters would never have to intercommunicate, but that's impossible-- as you said, communities will probably have to be arbitrarily assigned to a cluster, and they have members everywhere. The major load on the site (the major *use* of the site) is the friends view, which pulls a bunch of data from each of your friends, so the best way to split people will be whichever way puts everyone with their friends.

That's exactly why I was wondering about the soundness of the proposed architecture.

because I'm more likely to know people in my area than people in other places

Heh... I'm not. =) Most of the users in my Fiends List are not within a few hours' drive from me.
bellacrow
Dec. 13th, 2001 04:07 pm (UTC)
sounds good actually
how are you going to do the backup? Save as text files or?
bride
Dec. 13th, 2001 04:09 pm (UTC)
Re: sounds good actually
http://www.livejournal.com/export.bml

You have your choice of XML or CSV as an export format. =)
( 11 comments — Leave a comment )

Profile

eLouai
bride
The Bride of the First House

Latest Month

March 2015
S M T W T F S
1234567
891011121314
15161718192021
22232425262728
293031