I began working at SoftLayer in May of 2010 as a customer support administrator. When I signed on, I was issued a BlackBerry to help me follow tickets and answer questions from my coworkers when I was out of the office. In August of 2011, that sparingly used BlackBerry started getting a lot more use. I became a systems engineer in development support, and I was tasked to provide first-tier support for development-related escalations, and I joined the on-call rotation.
In the Dev Support group, each systems engineer works a seven-day period each month as the on-call engineer to monitor and respond to off-hours issues. I enjoy tackling challenging problems, and my Blackberry became an integral tool in keeping me connected and alerting me to new escalations. To give you an idea of what kinds of issues get escalated to development support, let me walk you through one particularly busy on-call night:
I leave the office and get home just in time to receive a call about an escalation. An automated transaction is throwing an error, and I need to check it out. I unload my things, VPN into the SoftLayer network and begin investigating. I find the fix and I get it implemented. I go about my evening, and before I get in bed, I make sure my BlackBerry is set to alert me if a call comes in the middle of the night. Escalations to development support typically slow down after around 11 p.m., but with international presences in Amsterdam and Singapore, it’s always good to be ready for a call 2:30 a.m. to make sure their issues are resolved with the same speed as issues found in the middle of the day in one of our US facilities.
Little did I know, my SoftLayer experience was actually preparing me for a different kind of “on-call” rotation … One that’s 24x7x365.
In June 2012, my wife and I adopted an infant from El Paso, Texas. We’d been trying to adopt for almost two years, and through lots of patience and persistence, we were finally selected to be the parents of a brand new baby boy. When we brought him home, he woke up every 3 hours for his feeding, and my on-call work experience paid off. I didn’t have a problem waking up when it was my turn to feed him, and once he was fed, I hopped back in bed to get back to sleep. After taking a little time off to spend with the new baby, I returned to my job, and that first week back was also my turn on the on-call rotation.
The first night of that week, I got a 1 a.m. call from Amsterdam to check out a cloud template transfer that was stuck, and I got that resolved quickly. About 30 minutes later, our son cried because he was hungry, so I volunteered to get up and feed him. After 45 minutes, he’d eaten and fallen asleep again, so I went back to bed. An hour later, I got a call from our San Jose to investigate a cloud reload transaction that was stalling with an error. I worked that escalation and made it back to bed. An hour and a half later, the little baby was hungry again. My wife graciously took the feeding responsibilities this time, and I tried to get back to sleep after waking up to the baby’s cries. About an hour later, another data center had an issue for me to investigate. At this point, I was red-eyed and very sleepy. When my teammates got up the next morning, they generously took the on-call phone number so I could try to get some rest.
This pattern continued for the next six days. By the end of that first week, I got a call from work at about 3 a.m., and I picked up the Baby Monitor from the night stand and answered, “Dev support, this is Greg.” My wife just laughed at me.
I’ve come to realize that being on-call for a baby is a lot more difficult than being on-call for development support. In dev support, I can usually documentation on how to resolve a given issue. I can search my email for the same error or behavior, and my coworkers are faithful to document how they resolve any unique issues they come across. If I get to a point where I need help, I can enlist the assistance of an SME/Developer that commonly works on a given piece of code. When you’re on-call with a baby, all the documentation in the world won’t help you get your newborn to stop crying faster, you don’t get any clear “error messages” to guide you to the most effective response, and you can’t pass the baby off to another person if you can’t figure out what’s wrong.
And when you’re on-call for development support, you get some much-needed rest and relaxation after your seven days of work. When you’re on-call for a new baby, you’ve got at least a few months of duty before you’re sleeping through the night.
As I look back at those long nights early on, I laugh and appreciate important things in my life: My wife, my son, my job and my coworkers.
– Greg
