Slashdot ran its first story about the terrorist attacks on New York City and Washington D.C. at 9:12 a.m. EDT, 23 minutes after the first hijacked airliner slammed into the World Trade Center. Slashdot founder Rob 'CmdrTaco' Malda thought briefly about linking to mainstream news sites but, he says, 'I couldn't get to CNN. MSBNC loaded but very slowly. Far too slow to bother linking. I posted whatever facts we had. At the time just the first two impacts. Over the next hour I updated the story as events happened.' During that hour Slashdot's traffic climbed to double its usual rate and things started to come apart in the server room.

Normally Slashdot serves pages to users at a maximum rate of about 20 per second, but by 10 a.m. Tuesday the server load was spiking as high as 40 pages per second, approximately double normal capacity. But the real problem came from the number of comments readers submitted, not the raw number of people trying to simply read the site.

Slashdot has a famously talkative audience, one that regularly proves the old adage, 'opinions are like email addresses; everybody has one.' Tuesday, it seemed like everyone who had ever heard of Slashdot wanted to join the discussion, and since Slashdot comments are automatically displayed only a few seconds or minutes after they are submitted, the database behind the Web servers had to not only send out pages but add data at the same time, and adding data to a database takes more computer time than taking data out.

Worse (from a technical perspective), each time a comment is added to a Slashdot story page, that page grows and takes more server time and bandwidth to deliver, and Slashdot stories with comments rapidly being added to them tend to draw even more readers, who then add yet more comments, creating a nearly exponential server and database load cycle. This usually takes place over many hours, and our programmers and network administrators have things set up to handle loads on MySQL, Slash, and Apache that would crush most database and server software, at least under normal conditions. But Tuesday's conditions weren't normal.

Everything Has a Limit Shortly after 10 a.m. Slashdot overloaded. Slash programmer (and Ask Slashdot section editor) Cliff Wood has details: We were getting slammed. Slashdot was handling three times its usual peak traffic. ... First, the database went down. We had already been experiencing problems with MySQL due to the sheer scale of what Slashdot has become. ... Slashdot went from keeping no more than 50 - 70 kilobytes of comments in its database to holding over 2 million comments. ...[D]ue to the thousands of comments from concerned readers that were now flooding in.

One of the main things that saved us, and I assume where many other sites failed, is that Slashdot dropped its dynamic traffic and converted to serving static pages while we fixed things in the background.

This is how we shouldered the traffic for the remainder of the day: we kept watch on memory usage, and if one server was close to exhausting its RAM we killed it and restarted it, first as static, and then as dynamic when hits allowed.

Through the remainder of the day, most of us were around to handle what bugs arose as the software shouldered more load than it ever had before. During all of that, we had disabled as much of the process intensive code [like the 'search Slashdot function] as we could while still maintaining enough functionality that our readers could communicate their thoughts on the horrors they had witnessed during the day.

By 10 p.m. EDT, Slashdot was down to just normal peak hour traffic.... The Humans Worked as Hard as the Machines Slashdot depends almost entirely on reader submissions for stories and story leads. Usually the site sees 300 or 400 submissions per day, out of which 12 to 15 are posted directly and one or two trigger additional research that may or may not lead to staff-written feature stories. Tuesday alone, Slashdot editors were hit with close to 1,000 story submissions.

We were also flooded with email. By Slashdot standards, 500 messages per day is a normal -- even light -- email load. We don't keep an exact email count, since Slashdot readers tend to write to individual editors instead of to the site in general, but we estimate that Slashdot editors received at least five times as much email as Foxnews.com between Tuesday morning and Wednesday evening.

Here's Slashdot editor Timothy Lord's story: Submissions would flow in, I would open them, cut to a text document on my laptop to edit, and delete them from the submissions bin as fast as possible. I don't know how many submissions there were Tuesday and Wednesday, or how many I cleared from the bin.

Email never let up either, and neither did IRC -- people wrote with links to survivor pages (we tried to incorporate all these into our wrap-ups), links to pictures, questions about what else had happened, poetry, essays (and links to essays), conjectures about what the FBI knew. I've gotten a dozen or so notes of thanks from readers who said ours was the only site that was up enough for them to get news from on Tuesday. Within half an hour after I suggested that a searchable meta site of the 'I'm OK' sites would be a good idea, I got email from pete@elbnet.com, who set up just such a site, sent us the URL, and had it running within a few hours at www.elbnet.com/wtc. Slashdot editors Rob 'CmdrTaco' Malda, Jeff 'Hemos' Bates and Michael Sims worked as hard as Timothy Lord, in shifts, nearly around the clock. Malda was also deeply involved in the server and database problems. Cliff Wood and Jamie McCarthy, who would usually take some of the story posting and editing duty, worked on nothing but software and only did minor bits of story editing. By Thursday, when the load let up a bit, Slashdot staffers' primary desire was, as one put it, 'SLEEP!!!'

The Scoreboard In the 24 hours immediately following the first report of the attack on the World Trade Center, Slashdot delivered over 3 million page views, allowing for static pages delivered but not recorded by our monitoring system. Wednesday's total was close to 2.5 million. A 'normal' Slashdot weekday is between 1.3 and 1.5 million.

More telling (and harder on the database and servers) was the number of comments posted. Slashdot usually publishes an average of 4000 reader comments per day. In the 24 hours after the first 'World Trade Towers and Pentagon Attacked' story went live, we had over 8460 reader comments. The next 24 hours saw another 5978.

But Slashdot does not exist in a vacuum. It is part of OSDN [the Open Source Development Network], and other OSDN sites saw traffic decreases on Tuesday. Our primary Open Source software development site, SourceForge.net, reported over 586,000 pageviews Monday, but only 422,000 Tuesday. Our ecommerce site, ThinkGeek.com, displayed about 242,000 pages Monday but barely made it over 153,000 Tuesday. Other OSDN software and information sites showed similar drops. Indeed, the only OSDN site other than Slashdot to show an increase in traffic was our dedicated Open Source 'hard news' site, NewsForge.com, which got about 30% more traffic than usual between Tuesday and Wednesday.

This week, at least on OSDN, news has been the undisputed traffic king.

Does Any of This Mean Anything? When media pundits talk about 'news on the Internet' Slashdot is almost never mentioned, even though it has more regular readers than all but a few newspapers. It is apparently viewed, out there in mainstream-land, as some sort of freakish geek discussion board that lives in a different world from publications and broadcast outlets run by dignified suit-wearers who have journalism degrees, years of experience or both.

We don't really blame the 'suits' for overlooking us, though. From their perspective, there is no reason a Web site started (and still overseen) by someone who calls himself CmdrTaco needs to be taken seriously. And surely they ask themselves, 'How can a news operation (or whatever this Slashdot thing is) be run purely on the Internet, with staff scattered all over the United States and contributors all over the world?

The answer is that Slashdot's wide-open comment posting and story submission policy allows every reader to have a hand in creating the site's content, and to act as copy editor and fact-checker, too. If Slashdot's staff author/editors make a mistake, a dozen readers will usually catch it in minutes. If Slashdot posts a reader-submitted eyewitness account of an event, other readers will almost immediately add more eyewitness account to the original one, which will be followed by comments about the eyewitness stories, and all the stories and commentary will then be moderated by readers who did not post in that thread but, instead have chosen to rate the quality of their fellow Slashdot users' writing on a scale of -1 to +5. (The Slashdot comment posting and moderation system, is not easy to understand until you've read Slashdot for a few weeks, and a lot of people say it is more complicated than it needs to be. But it works pretty well.)

Slashdot is not perfect. Far from it. We have too few people, working with too small a budget, to compete effectively with CNN, MSNBC, and other general interest news sites in covering every piece of national and international news that deserves wide exposure. And, to tell you the truth, we are technology junkies who love Linux and enjoy writing about nothing but leading edge software, computer hardware, science, and Internet-related topics, along with talk about toys, games, and (mostly sci-fi) movies to keep things from getting too serious.

But this week, demand from readers who could not get through to overcrwoded 'major online news sources' forced Slashdot to step out of its niche and become an ad hoc portal for news, background information, rumors, speculation, discussion, and political invective about Tuesday's tragedy and its aftermath. We did this almost instinctively, without any special preparation or training, and without formal staff or editorial meetings. We just plain went ahead and did it, and it all worked out.

Want to Know Our Secret? Come close. I'm going to whisper in your ear, okay?

Slashdot's secret (sshhhh... don't tell anyone. It's a secret, remember) is that our whole crew -- programmers, systems adminstrators, editorial personnel, and readers -- don't use the Internet as a one-way, broadcast-style or newspaper-like information distribution medium, but as a collaborative, fully interactive network that has the power to bring many voices together and weave them into a single web. A World Wide Web, you might almost say.