Saturday, September 14 2019

outages



Dear Journal,

Good morning everyone, and happy Saturday. I'm struggling this morning, it's been a really tough weekend thus far, and I don't even know where to begin. I'll tell you this, the theme of this weekend is "broken software".

Yesterday at work, we had a pretty severe outage. It was nobody's fault, and everyone was just doing their job, but one of the systems that my team manages failed and took down our production services for most of our customers. I'm on call this week, so I was paged in the late morning. My whole team would shortly jump in, so I hardly felt alone, but incidents suck. Trying to fix things during an incident is nothing short of a horrible out of body experience. Suddenly, you feel like a stranger at your own computer. You feel every painful passing second in slow motion, and yet somehow the whole day disappears. The incident lasted about three hours, I think. I was supposed to take a half day yesterday, but by the time we called all clear it was into the afternoon. I packed up my stuff, gave a few friendly waves to my weary, shell-shocked team, then left the building and walked toward my house down East Wash for about an hour. Marissa and Rodney scooped me up somewhere by the Yahara bridge with a bag of Portillo's.

It actually took a bit of time for the negative thoughts to creep in. At first I was still high from the adrenaline, and quite proud that we managed to restore the service at all, but the feeling wore off and I was pretty down in the dumps for the rest of the night. We tell new people at work a number of platitudes, like "we're a blameless culture", "breaking things is part of the job", and "shit happens, let's do better next time" - all very true and healthy perspectives, but it's hard to practice what you preach. What always gets me is reading angry tweets from customers. It bothers me that something I was responsible for kept people from getting their work done.

After Rodney's nap, I took him to the grocery store. I was tempted to wallow around in my sweatpants for the rest of the day, but I knew that getting out of the house and doing something that didn't involve technology would feel good - and it did. Rodney and I had a great time walking around the grocery store. The Jenny Street Market had pumpkins lined up along every aisle, and Rodney was jazzed. You should have seen his face when I told him we could get one, he just about crapped his pants from excitement. He carried the tiny pumpkin around like it was a sacred relic.

When we got home, I cooked a pizza while Marissa painted and Rodney watched TV. We normally make sfincione on Friday, and we have to kind of rush it into the oven so we don't eat late, but having a head start allowed to me to take my time, listen to music, and do some thinking. After the pizza was out of the oven, we took it over to the Malthouse and ate on the back patio. We stayed there for about an hour until it was Rodney's bed time.

Marissa put Rodney to bed, and I decided to sit on the deck and continue to enjoy the cool fall air. I started a little fire in my grill and put on some music. I was so comfortable, that I actually nodded off for about a half hour, waking up when Marissa joined me outside.

My night started to get better. Marissa and I talked for a bit out by the fire, then made our way inside to watch Rambo 2. We were in the mood for a stupid action movie, and Rambo is one of those movies I like to periodically check in on to see how poorly it's aging. And it's true, Stallone looks so uncomfortable clambering through the jungle.

Just as the weekend started to take an upturn, I woke up this morning to a fried file server. It appears at some point, my home file server's boot drive failed - the kind of error that would go unnoticed for weeks until you finally decided to reboot it - like I did this morning. I'm not going to lie to you, reader - I lost my temper, and I had a new one for the anger journal. I'm just about fed up with software breaking. Sure, home projects like this aren't critically depended on. The only consequence of this outage is a wife and son that will have to just stick with Netflix for the next few days, but for people like me, home tech projects are supposed to be kind of a refuge from the software problems at work, and it still sucks when they break.

I was tempted to skip journaling today, but just like when I forced myself to take Rod to the grocery store, I knew that deep down it would be good for me to take pause before losing myself in fixing things. It felt good to share how I'm feeling, and I'm already in a better mood. Thanks for reading. I hope you all have a great weekend. Now if you'll excuse me, I have a file server to fix. I had better brew some more coffee.