Monday, June 25, 2007

Big Company = Butt Munch

My first task at my new job (remember, I went from an 18 person gig to a 1,500 person nightmare) was to load test the new application. Stress it out using tools until it blows sky high. Fun! I love to break stuff. I developed the plan, submitted it, got approval....and waited....and waited....and waited. I finally got a go date a few weeks ago of June 25th. Sweet. They were flying in reps from all interested parties and a Softie Specialist and it was my show. Then the date was moved to June 18th and I was going to be away. Nothing changed - just another technical guy would be playing the part of me for the week. When I got back, he was leaving for a week - but he would brief me on all happenings. The region we were using for the load test is called Pre Prod. Go figure. It was to be ready on June 8th, which turned into June 12th, which turned into the 13th, 14th, 15th.....it wasn't even ready before I left! Now the dude playing me actually had to script the stuff himself as well! Great way to make new friends at work! When I rolled in today I was expecting to get filled in on all the results. What I got filled in on was the joke. The region wasn't ready to script in until late Thursday the 21st. Everyone flew in to see we weren't quite ready. Egads! What can take so long to set up an environment to mimic production? Oh, that's right - how can you mimic something that
doesn't exist yet? All the other regions are simply virtual environments which can't be used to load test - they can, but the results are as useless as a diet book on Rosie O'Donnel's bookshelf. Once the region came live, it was discovered that the tool I trained on for 2 weeks couldn't be used on the software in question due to unexpected dynamic session variables tied into each UI. Nice. We didn't know it was going to be a problem until the software vendor turned his trace on which we couldn't do by ourselves. Nice again. Alas, the Softie had a tool that we have a license for that worked and allowed us to code around the session ID variables. He gave the other guy a crash course Friday and he gave a me 25% of that knowledge today. We hit a brick wall when we found out a patch went in over the weekend that changed the first UI on the Home screen rendering his work to that point pretty useless since this tool doesn't support mid script changes very well. i guess it would be ironic to point out here that the first tool handles mid script changes just wonderfully! So, due to hidden page names that we need the software vendor for, I can't start something that was to begin on June 18th until tomorrow the 26th. Did I mention this is going live in less than 3 weeks. Who waits that long before they stress test something? Ohm that's right - you can't stress it until it's done and a production region is ready and available. Sigh. At a small company, 4 people would shut a door and not come out until this is ready with all 4 doing the majority of the work. Now we need a meeting to see what meetings we need to see who we need to talk to and who should be filled in on the situation and what direction we should take. Sigh (again). Dilbert is so real to life it's scary. Art imitating life indeed. In the land of cubicles, the half brain twit is king. If that's true, I must be an Emperor or something. I can't get to 52 fast enough.

2 comments:

Anonymous said...

I would think that you had seen enough at other shops to know this kinda stuff still happens (and will continue to happen). I have production jobs that run every day (including weekends) - I still get clowns shutting down regions or re-booting servers over the weekend while my jobs are running. One of these days I know they're going to cause a job to corrupt the database.

Just a reminder - be careful what you post about your current employer on the internet. You never know who might find it.

Anonymous said...

Welcome to my world! Granted I am not installing the system upgrades BUT I have been on the "post prduction" side of BAD installs so many times it is a wonder I have any hair left at all! (the wife and kid cause the gray's) Just think of the meetings you have to attend about the meetings to discussing how the test enviroment did not proprer simulate the production evirmonment because GOD forbid you are actually allowed to complete a true stress test!

Have I mentioned how much I enjoy team meetings! Just about as much as Still A Fan!