This week at Work

One would think that being a short work week (we only had to work 3 days), things would be fairly laid back and easy. After all half the people were out.

Boy, were we in for a big surprise. Monday was a "normal" work day. Pretty much uneventful. Same crap from the customers, same crap that had to be taken care of.
Tuesday morning I logged on early since we had to see why a client didn't get their file and yup it fell into my lap. While checking that out my boss calls me before 6:30 a.m.
Now, my boss and I never talk before 7:30 a.m. He tells me we have problems. I proceed to let him know that I'm looking into this file. He says we need to check that out but we have bigger problems. That nothing ran Monday night, there were Mainframe problems and he was on conference calls with the other managers and the tech guys. They had decided that they were not going to run the batch for Monday night and double up for Tuesday. That works nicely for most of the systems but not for the billing system. We have never doubled up on the billing system before.

We talked when everyone got in because some of the jobs ran before the system decided to take a hit and then some of the jobs ran in the morning after they got the system to limp along.
Our plan was to try to get them to let us run day time, complete Monday's cycle and be on track for Tuesday night. HA! Everyone said no. The system was up and running but they didn't have it completely up and running and they couldn't allow us to chew up all the CPU cycles.

Shift gears, who can work that night to babysit the Billing System and the 2 other systems that we own. It turns out that anyone not on vacation, ended up working that night. There were 1 or 2 people not involved in the recovery so there were a few people in during the day as those of us who stayed all night slowly left and went home to sleep.
I only slept a couple of hours, did a couple of errands and logged back on to relieve the person who was staying to someone else came back. Of course there were a number of jobs that had failed because data was available to send to outside clients or the connections weren't working right. So, I kept getting hit with those errors. We were lucky because the Tech guy we were working with, was very flexible and picked up a lot of the slack on this for us. He didn't have to.

In between trouble shooting, we have our customers asking us to do things they can do themselves (nothing new), trying to figure out how to run Tuesday nights processing which would actually be Tues/Wed processing.

People working Wednesday night started to float in around 5 p.m. I stayed on for awhile on Wednesday but there wasn't much I could do except watch and check out a few jobs. I told them if they needed me to call, wake me if they had to. I just can't work those hours like I used to!
When I woke up on Turkey day - they were still working, trying to get some critical pieces of the processing done. They eventually called someone who had retired to help them out so they could get through the processing.

I logged off yesterday around 11 and they were still on. I checked around 12:30 and thankfully they were all logged off. One of the guys gets on last night and sends me an IM to see if I know about some outside Client. Now he is one of the guys that worked both Tuesday and Wednesday night. Luckily the problem turned out not to be ours and the other area figured out what they needed to do but I had already logged onto work. I also switch the on call list and put myself on call a couple of days early.
I just need a ton of prayers right now that everything will run smoothly next week. I don't know if I can handle being up all night most nights next week.

That was our short holiday week. That week that should be fairly easy and quiet at work. There is some clean up and followup to do and I know I'm going to try and get my stuff at least started today or tomorrow.

Life in the FAST lane!