Since this was a table with close to 60 million records, it took a while to execute on the slave, so we were informed in time and could prevent the query from propagating to the slave by killing it, forcing a rollback and stopping the slave replication. Luckily, this saved us the hassle of restoring from backups. We successfully dumped the table on the slave and updated the master again with the correct entries still on the slave using “SET SESSION SQL_LOG_BIN=0″. The master was saved and in good shape.

However, this left the slave without replication, and our worst nightmare came true. The Query Of Death was executed more than once on the master and thus was all over the binlogs that would be replicated. This meant that it was not as simple as skipping the next query on the slave and start replication again from there. The only thing I could think of was replaying the binlogs on the slave with the –safe-updates argument passed to MySQL, but that would take hours. We consulted MySQL to see if there would be an easier way, unfortunately it turns out there was not. So there I was, painfully executing log by log, anxiously monitoring every query that was executed. Finally at 4 o’clock everything was back up and running. The best thing? The website’s users didn’t notice a thing, apart from seeing ”This is a test” in some places for a while. Since the slave is used for full-text indexing with Sphinx as well, the searches were a bit off too, but nothing major.

I just wish there would have been an easier way… A query blacklist? Something?
If anyone has tips on how this could have been solved in an elegant way, please do tell.

For most of the staff, FOSDEM will begin tomorrow. We’ll start preparations at the ULB around 11AM, and hope to be finished around 10PM.

See you this weekend!

The fosdem.be domain name was registered by Wouter Coekaerts, known for his contributions to the irssi IRC client. Jan VB contacted him, and he immediately agreed to transfer it to us. Also a big thank you to you, Wouter!

FOSDEM was very lucky that these domains were registered by people with good intentions. They could have been registered by a domain squatter, and we would probably have lost those domains forever. Such things teach you a valuable lesson. Please make sure that, if you start a project, you immediately keep the most important TLD’s from being registered by squatters.

“Spelletjesgarnaal” was owned by Planet Internet at that time. It had a forum and a website with tons of reviews of all the latest games, all in true Web 1.0 style. A few LAN parties were organized by and for the forum visitors, as were online gaming events and more. It was all great fun. Then came the dark ages of the interweb. Planet Internet decided that they no longer had any interest in supporting the Spelletjesgarnaal website. The “leader”, known as Garnaal, stood up as the savior. He would continue on his own, and try to make the best out of it. The “Pro” membership status was created, and lots of people from the community gave money simply to support the website. People grew up, so the interest in gaming kind of decreased for most people, but it was for the good cause.
Later on, Spelletjesgarnaal was acquired by Skynet, and re-branded to Arena51. Arena51 drastically changed the target audience, from approx. age 16-25 to approx. age 11-16. The website was no longer all about PC/console gaming. Flashgames, user photo galleries and other non-interesting stuff was added, and it got flooded by the new audience. The forums, however, were the hideout for the Old Community. Granted, newbies were sometimes bashed for using bad language, but what do you want? It’s a completely different age group; it was written in the sky that those would clash a little. Some two years passed, the occasional clash passed by, but everyone was relatively happy, and the Old Community still had lots of fun. Great success!

Then, on Friday 5th of January 2007, totally unannounced, the forums were closed. Skynet, and our savior Garnaal, had turned against us. The Old Community, which paid lots of money to keep Spelletjesgarnaal alive in between the PI and Skynet period, got dumped.
Everyone was lost, as the forum was the main communication channel. Questions rose, people talked to each other through alternative channels. One member set up a new forum and we chose to go on, independently from Arena51/Skynet, where we were clearly unwanted. The Arena51 community forums can now be found here. I feel it has never been as active as the last couple of days. It really shows the commitment of the people involved. I wish the new initiative all the luck in the world to keep on going the way it’s going now. One thing is certain; my respect for Skynet and our “savior” has gone deep down the drain.

Basically we had two options. We could call someone to break open the lock, but that would mean destroying the front door, since its secured by extra locks at both the top and the bottom. The other possibility was to break a window. That would certainly be the cheapest option, but since it’s all double glazed, would make a hell of a noise.
Considering almost the entire street had joined us by now, we went for the second option.
While we were hammering on the glass, a police car passed by. The ladder was still there, against our frontage, so you would expect them to stop and have a look at what’s going on, right? Guess again. They just drove by, and about 10m further down the street they stopped. Not to have a look at our situation, oh heavens no. They pulled over a cyclist who had no lights on. That’s far more important and suspect than 3 people with a huge hammer and a ladder against a frontage at 23.30 PM.

I’m seriously considering writing a letter to them. I do not feel safe at all right now. If our actions weren’t suspect to them, what is?

Anyway, I’m off to bed. Enough hard labor for today

Oh yeah; happy birthday me. Guess this will be one I won’t forget soon

Initially, our idea was to go for ISPConfig, a web-based control panel, to easily manage all of our users, websites, domains etc. This turned out to be quite some overkill, so we got rid of that and made the setup by hand. I am really happy with the way things are being done. Those who want more information about their account settings or have some general questions should take a look at the staff wiki.

Now that the most important mailinglists have been set up too, it might be time to set a date for a first meeting to get FOSDEM 2007 on the road. I am really looking forward to this edition, and I have a feeling this one will be The Best One Ever (c)(tm)(r).

The machine is really fast, and the BENESOL network is of an outstanding quality. Thanks again guys for sponsoring us!

Notez-bien: eth0 was manually configured.

rush:~# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:10:A7:0F:AB:43
          inet addr:192.168.92.7  Bcast:192.168.92.255  Mask:255.255.255.0

eth1      Link encap:Ethernet  HWaddr 00:10:DC:DD:75:DE
          inet addr:192.168.92.3  Bcast:192.168.92.255  Mask:255.255.255.0
rush:~# vi /etc/network/interfaces
 -- change eth1's IP from DHCP to static .7
 -- create configuration for eth0 with DHCP
 -- make sure your DHCP server hands out the old address of eth1 to eth0
rush:/etc/network# ifdown eth1; ifconfig eth0 down; ifup eth0; ifup eth1;
Internet Software Consortium DHCP Client 2.0pl5
Copyright 1995, 1996, 1997, 1998, 1999 The Internet Software Consortium.
... yaddieyaddieya
bound to 192.168.92.3 -- renewal in 2000 seconds.
Interface "eth0" is already enabled.
rush:/etc/network# ifconfig
eth0      Link encap:Ethernet  HWaddr 00:10:A7:0F:AB:43
          inet addr:192.168.92.3  Bcast:192.168.92.255  Mask:255.255.255.0

eth1      Link encap:Ethernet  HWaddr 00:10:DC:DD:75:DE
          inet addr:192.168.92.6  Bcast:192.168.92.255  Mask:255.255.255.0

How cool is that?

Let’s hope this project really hits off!