About 2 hours ago the bulk of the network was packed up and loaded onto the trucks. I only know about this because I got a message from a team member saying they were going to the pool, So I have to assume that they finally have some idle time.
Me, well I’m happily at home, sitting on the couch with my feet up taking a moment or two to dump out the last few interesting things in my brain.
I flew back yesterday using “meetings” as a thinly veiled excuse to get out of tear-down. This will actually be the first show where is missed most of Hot Stage, the First day of the show and the Last day of the show.
However it did give me the required quiet time to write all this down. As SSH pointed out in the report from 5/1 (or is that 1/5 ?) its hard to take notes when you have work to do. RJ had the advantage that he mostly worked vampire hours. Me, well I just got on a plane and got the hell out of Dodge.
So back to the eNet for a moment.
Sarcastic comment of the week actually has to go to SSH for saying today that
“I don’t think the eNet really had very much going on.”
(Well, I /hope/ it was sarcastic). From where I was sitting there were 60+ people who were not only working very hard on over-engineering a network but still had enough energy to develop and devise creative ways of killing their colleagues when someone fat fingered a config.
Now, I will concede that the huge arse TV in the middle of the NOC designed for video conferencing got usurped for a pair of X-Boxes, but other than that there was a lot of networking going on.
When you look at it on paper, any one component’s design makes a lot of sense. But trying to glue together a meshed array of firewalls and routers, a cluster of L3 switches and wireless access points with multiple overlapping VLANs, VoIP phones delivered to every booth being driven by in-line POE switches, a cluster of NMS servers pinging the be-Jesus out of everyone, goodness knows how much test equipment and some sort of crazy clustered DNS infrastructure, and you have, well one big FusterCluck.
So that meant we spent a small amount of time configuring things, and a great deal of time trying to work out what what we broke by doing it. But to our credit, we fixed some things in an hour that some IT departments would take months to work out. Oh and it always gave us someone to blame. I think the voice guys got the worst of it but they took it very well. When, for the Nth time, a routing problem stopped the phones booting, I couldn’t resist and I handed them a ‘VoIP for Dummies” book that was published by their competition.
We even felt the courage to enable 802.1X in the NOC. After the people building it (I won’t name names [Alice, Myself]) stopped trying to test it with their own broken laptops. We got it working for everyone except for Neal, who ironically didn’t have a laptop that was even close to dot1X capable. His response was :
“Most of my test gear isn’t either.”
Cowboy style also kicked in while we were publishing the addressing. The magical mystery transparent proxy got the impression that the feedback from the webserver was a big download, and not just syslog output. To our horror we then discovered that even though we had killed all our browser sessions, the proxy was holding the connection open for us and wouldn’t let the publish process die.
“how long will the publish take ?”
“Given the overloaded web server, the 5 firewalls, and the
amount of data, I’d guess at about Two Hours.”
We quickly pinged Glenn and asked him to kill the process on the server side. Glenn was, of course, too busy with other problems and just re-booted the whole server.
“That should fix the problem.” He said.
This sense of faux courage meant that we all secretly upgraded, or swapped out equipment entirely, in the middle of the night and mostly got away with it. But not always. There were a few morning surprises for some people (and more finger pointing) But you had to admire the defense of “It would have been worse if we didn’t do something”. I stopped at rebooting the NTP servers (again) when I noticed that they were out by 19 seconds and drifting at about 3 seconds a day. Alice was quick to set me straight :
“So what if all our clocks are flashing ’12:00″,
at least they are ALL showing the same time”
I think the clock will have finally caught up by tomorrow.
Anyway, the show is over for another year.
I’m sure that show floor is littered with the detritus of bits of Cat5, random metal parts, snack food wrappers and all the left over white papers. Some poor folk will come through tomorrow to clean up after us and try and make sense of what the hell was going on
“It’s like a geek exploded in here”
( That may also explain the incident a few years ago when, on the Friday, as they tore down the NOC around us, someone snuck in and stole all our beer from the NOC fridge. Talk about a hint to go home )
Meanwhile the rest of us will face another 24 hours of looking for bars with nerd friendly dress codes. As we walk the strip we will quietly deal with the annual separation anxiety that overwhelms us when we realise that we won’t be spending another night drinking too much and telling bad war stories to our friends until it is way past our bedtime.
Ssh, someone is sleeping.