Wednesday, July 03, 2013

The Morning After The Deploy Before

Now, I know you all have long wondered what a typical deploy cycle looks like. Who are the players in these mysterious circles? What sorts of science create the magic of a (mostly) functioning web application? How do the various parties work together to create these glorious examples of intelligent application of information systems technology?
WONDER. NO. MORE.
It pretty much goes down like this.
Our Players:
PrincessThis is Princess. She is usually some combination of business-type person, analyst-type-person, and project manager.
Ghost: This is $DEITY. Which you don’t really need to know ahead of time because $DEITY will tell you so every time he speaks. He is usually Princess’s boss (or her boss’s boss), and will occupy a fairly high rung on the food chain.
Punch: This is Coffee Cup Guy. He probably has a title like Database Developer V. Probably also had a new nameplate drilled for himself that says “Coffee Cup Guy: DEV V” on it within hours of the promotion. Tends to be dragged into things whenever Princess isn’t hearing what she wants to hear, agrees with her that whatever it is would be “very simple” to implement and then proceeds to actually do…nothing.
Rolling on the floor laughing: This is the new kid, Ms. QA. Wait. Ms. Analyst. Wait. She also does development work. Even though she has absolutely zero dev experience. But that’s ok, because Coffee Cup Guy will tell her is it very simple, point her at the first site he found on Google with at least some matching keywords and then claims he ‘taught her everything she knows.’ VERY enthusiastic, somewhat light on experience and commonsense.
Devil: Here we have Tech Lead. He might be a for-real manager, he might just be The Guy Formerly Known As Coffee Cup Guy, before he titled-out of the system and had to be promoted to a management position due to there being no ‘Developer VI’ in the HR book. Cool guy, but tends to not have time to get into the trenches with you.
Alien: And this is our developer. Or, as we like to call him for the story being told here, the only sane person in the whole @^*&@ing organization. (Note that who has this ‘only sane person’ title varies from story to story…sometimes it is the developer being a complete idiot, but for this particular vignette, he is our one sane crew member.)

The Morning After The Last Disaster Deploy:
Princess: OK, so, Developer, here are your tasks for this release. There are nine of them total, but I want to bring your attention to this item in particular as it is a Priority 1 task. I have attached the finalized requirements documentation for your review; I don’t think there are any surprises there, as we discussed this earlier. Keep in mind that as a priority 1 task, it is a must go, so, the whole thing must be 100% finished before 7/15. You sized it during our pre-sizing meeting last month at 15 hours, so this will not be a problem.
 Alien: Hmm. OK. Let’s see here. Oh yeah, I remember this one…land new data from blah blah blah…whoa-whoa-whoa, time out! Princess, what we discussed and I estimated 15 hours for was one (1) new package to load three (3) tables from one source. This item has twenty-seven (27!!!!) new tables from FIVE (5!!!!!) sources?!
Princess: Ahem. We discussed this. Oh. Wait. You weren’t invited to that meeting. Because you did not need to be. Plus I think you were trying to catch all those snakes somebody released in the server room…did we ever figure out who did that?! Anyway, don’t worry, we did adjust your sizing to account for the additional tables.
 Alien: Oh, I see, OK, well…heh heh…yeah, because, the last time you did this, heh-heh…
Princess: We’re not idiots, you know!
 Alien: Heh-heh, well, I wasn’t trying to imply…
Princess: So, you have 20 hours in the plan for this.
 Alien: …
Princess: What?
 Alien: …
Princess: WHAT?!
 Alien: That’s a big old negative, there, Princess. I’m gonna need more like 150 hours total. This is four brand new sources. I have to set up application user ids, get connections strings, update Oracle linkages, get data review and governance, build four additional packages, not to mention creating all the objects themselves…
Princess: Pffft. It’s cookie cutter. Once you’ve done the first one you can just copy/paste the others. 25 hours.
Punch: It really is very simple, you know. In my experience, Cntl+C and then Cntl+V will work for that.
Princess: Exactly. It is very simple so it really shouldn’t take that much longer. You’re being ridiculous.
 Alien: Uh-huh. Very simple. Copy/paste. Gotcha. Tell you what: How about you show me how simple it is, Princess…?
Ghost: I am $DEITY. Let’s keep the conversation constructive.
Alien: Sigh. 70 hours.
Princess: No.
 Alien: Do you want this to fail?
Princess: No. It cannot fail. It is a must go item.
 Alien: 70 hours.
Princess: No. I can give you 30. Plus Coffee Cup Guy.
Punch: No problem. Always glad to help the juniors out with stuff they don’t understand.
 Alien: …
Alien: 600 hours.
Princess: Fine. No Coffee Cup Guy, and I’ll give you 35 hours.
 Alien: 60.
Princess: 40
 Alien: 55
Princess: 45
 Alien: 55
Princess: No. You’re supposed to say 50.
 Alien: On this planet, we have an assortment of things called laws. There are the laws of time and also of space, and…
Princess: DADDDDDDYYYYYYY!
Ghost: I am $DEITY. Princess is right. You will have 45 hours. 
Alien: {thinks} @^*&@ you, $DEITY…
Ghost: I AM $DEITY. I HEARD THAT.
 Alien: {thinks harder} CAN YOU ALSO HEAR THI-
Devil: {texts} Time outta, bro.
Devil: {texts} Doughnut comment suey side on this.
Devil: {texts} Dammit. Don’t. Comment. Suicide. LOL. Autocorrect.
Devil: {texts} *commit
Devil: {texts} I got ur back. Will dizcus w/Big gay l8r.
Devil: {texts} Dammit.
Devil: {texts} Big Guy. LOL. Autocorrect!
Devil: {texts} agree w/Pins for now, will fix l8r, gtg letz bee done hair!
Devil: {texts} *here. Argh. Autocorrect.
Alien: {texts} OK, thanks. {says} OK, fine, 45 hours for initial rough-out, but my lead is going to circle back with you…
Devil: {texts} np, btw, am OOO for 6 wks lol but this 1st thing when back, kk?
Alien: {texts} WUT?! out of office, WTF man?!?!
Devil: {texts} in hq mtgs then PTO, sent email?
Alien: {texts} @^&@!!
Devil: {texts} all gud, I got this 4 u
Alien: So, actually, my lead is out, maybe we should revisit…
Princess: You said you were fine with 45 hours! It is carved in stone on these tablets! WE ALL HEARD YOU!
Ghost: I am $DEITY. It was spoken. It is now written. So it shall be. 45 shall be the counting of the hours. I am $DEITY.
Alien: …I hate my life…so much right now…oh well, maybe a miracle will happen…let’s just…moving on, where was I in this unending nightmare…blah blah blah tables blah blah blah schedules blah bla-…wait…you want to land all this garbage directly into…um…hang on…you realize that’s the application database, right? That’s our transactional tier, we can’t start landing large tables there, they have to drop into staging first and then…
Princess: YOU ARE BORING ME. I NEED THEM IN THE APP TIER SO THAT I CAN PLAY WITH THEM AS THEY ARE MY NEW SHINY TOYS. MAKE IT HAPPEN.
Alien: Um, I really have to push back on this one, because see, if we do that, we will be using up resources that are needed for even super-basic functions within…
Princess: PUH-PUH-PUH! Just stop right there. Someone wants to say something to you about this!
Ghost: I am $DEITY. Do what Princess says. My orders. It is necessary.
Alien: …
Alien: …
Alien: Ooookayyyyy, this has Bad Idea written all over it, but, you’re the boss…lessee…you want to land etc. etc. etc…lemme just take a look at these source tables HOOOOOOOOOLY CRAP-APPLES OF DIVINE RETRIBUTION +10 OF THE GODS!!!!
Alien: {does bunch of calculations on scratch paper} {does them again} {pales} {re-reads requirements} {throws requirements across room} {picks requirements up again} {does more calculations} {picks up phone, has terse exchange with DBA team in re: storage space, buffer memory, transaction log size and so forth} {discusses what kind of idiot would decide this was a good idea with now equally irritable DBA} {rinses mouth out with soap} {returns to discussion}
Alien: OK. Um. Let’s…just take a quick sanity check here…what are you going to be using this data for again?
Princess: Stuff!
Alien: L’il more detail?
Princess: Lotsa stuff!
Alien: Not exactly what I need to know…maybe if I explain the problem…seeeeeeeee, it’s going to take, 5-6 hours to pull just this one 300+ column by 800 million row table, and, you’ve got eleven others just like it listed here. That you want pulled daily. Each table will eat up around 50 gig of storage space, plus of course using tables with that big a footprint in update statements is, just, wow…HOW ABOUT if we just pull over the eight or so columns we actually use, and only the source types we need…that’s 20 minutes start to finish and under 3 gig of data, which…
Princess: BLAH BLAH BLAH, just stop talking! SIIIIIIIIIIGH…look, I do not understand any of that mumbo-jumbo. And I don’t know what I need. Or might need someday. Just pull all of it. Besides! We might want to do analysis and some junk!
Alien: …
Alien: …you’re going to do “analysis”…directly on our application tier… which runs our web services…against tables that size…lemme guess, you’re going to be joining them to every other table in the tier?
Princess: Probably! We will need to see all the stuff!
Alien: …I am…it’s just…there are no words for this feeling…
Princess: Pffft! don’t worry! Coffee Cup Guy will handle that analysis portion!
Punch: SELECT * really is the fastest way to get all the information from a table, in my experience.
Princess: Tee-hee! Oh, Coffee Cup Guy, you’re so smart!
Punch: Yes. Yes I am. Note how my coffee cup says “#1 Developer” on it. Because it is true.
Alien: Uh-HUH. Well, I guess I’ll start working on the…
Princess: Oh. Real quick, just a couple little things…we kind of need it all done by Thursday afternoon at the latest. Because I hired a new QA person, well, actually, she’s more of a BA but she has super-crazy-MAD database skills!, and since she has no idea what’s going on pretty much at all yet, she’ll need the extra time for testing sooooooo we need it done-done super-early so she can test it while it is still in the development environment.
Rolling on the floor laughing: HI! LOL! I’M ENTHUSIASTIC! LOL! DO WE GET A KEYBOARD? ROFL!! HOW DO I CONNECT TO THE DATABASE DO I NEED A PASSWORD OR SOMETHING? ROFLMAO! I USE SELECT * EXCLUSIVELY BECAUSE SOMEONE SAID IT SHOWS ME EVERYTHING! I ONLY USE LEFT JOINS BECAUSE THAT ALSO SHOWS ME EVERYTHING! AND I DON’T KNOW WHAT A WHERE CLAUSE EVEN IS, LOL!! ALSO I KEEP TYPING ‘DATABASE’ INTO THE RUN COMMAND WHICH IS ALL I HAD TO DO AT MY LAST JOB WHERE I WAS THE SUPER HIGH RANKED LEAD DATABASE PERSON EXCEPT THAT MY JOB TITLE WAS ‘GOPHER’ BUT THAT WAS EVERYBODY’S TITLE PRETTY MUCH AND GOSH, THIS NEW BLACK SQUARE OBJECT THEY GAVE ME COMES WITH A CUPHOLDER AND EVERYTHING! SUPER FANCY! WAIT! WHAT ARE WE DOING AGAIN?! I LOVE IT! EXCEPT THAT YOUR STUFF IS WRONG, BECAUSE I COMPARED IT TO THE TABLE OF CONTENTS IN THIS BOOK I FOUND UNDER MY DESK AND I THINK THE DATA TYPES ARE DIFFERENT!
Princess: OMG IT’S WRONG?!?! WTF, DEVELOPER?!?!
Alien: …I haven’t even started it yet, how can it be…?
Punch: An experienced developer would have known not to make that kind of rookie mistake…
Princess: Does this mean the task is red? It’s red, isn’t it? We’d better have a meeting about this! The task is clearly red if the data types are wrong!
Alien: THE DATA TYPES AREN’T WRONG! THERE ARE NO DATA TYPES YET! Geez. We haven’t even started the deploy cycle itself officially yet, so -
Rolling on the floor laughing: YEAH LOL BECAUSE YOUR DATA TYPES ARE ALL WRONG IN THE TABLES I FOUND ROFL ABSOLUTELY NONE OF THEM ARE ‘PICKLE’ THEY’RE ALL THINGS LIKE ‘VARCHAR’ AND ‘NUMERIC’ ROFL I MEAN LIKE WHAT DOES THAT EVEN MEAN?! RIGHT?!?! PLUS I’D PREFER IF THE QUERY WINDOW WERE PURPLE. CAN WE MAKE IT PURPLE? PURPLE IS GOOD. MORE GOOD THAN BLUE. AND DATETIME ISN’T EVEN A WORD, YOU KNOW, LOL? SO YOU TOTALLY NEED TO USE DATA TYPE = PICKLE THERE. IT ONLY MAKES SENSE. ROFL.
Alien: …
Alien: …
Alien: …Kill it. Kill it with fire
Princess: I do not appreciate your tone. If you had just done your coding right in the first place, you have nothing to fear from our new QA person.
Alien: I’M NOT AFRAID OF HER! She’s just…wrong. She can’t be looking at the tables and finding issues, because they do not exist yet. I don’t know what she is looking at, but, I’ll come in again!, whatever she just QA’d? Was not THESE tables!
Ghost: I am $DEITY. The QA person is right. Your code must be wrong. The item will be pulled if you do not fix it per QA’s findings.
Alien: With all due respect, on this one, QA is wrong. Among many other things…“pickle” is not a valid data type.
Ghost: I am $DEITY. You will crate a new custom datatype called “pickle” and assign it to those columns.
Princess: BRILLIANT PROBLEM SOLVING, SIR!
Punch: That’s why they pay him the big bucks, for sure!
Rolling on the floor laughing: JUST LIKE WOW ROFL!
Ghost: I am $DEITY. Have a benevolent smile! {beams}
Alien: I…work with insane people
Rolling on the floor laughing: OO! MY ACCESS REQUEST CAME IN! WHAT DOES ‘SYSADMIN’ MEAN? IS THAT GOOD? DOES THAT MEAN I QUERY STUFF NOW?
Alien: …dear mother of god…
Ghost: I am $DEITY. Watch it.
Rolling on the floor laughing: I LIKE PICKLES!
Punch: I will now quote an extremely long beat poem regarding pickles…a-chem! Pickle…pickle…pickle-pickle-pickle…pickle on hamburger, pickle on rye…
Alien: I’m just…gonna go over here and…see if I can’t somehow get a hundred hours of work done in the next two days…
Princess: Is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet is it done yet
Rolling on the floor laughing: IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET IS IT DONE YET
Ghost: I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet. I am $DEITY. Is it done yet.
Punch: …pickles on my eggs, pickles on my cheese…
Devil: RU thr bro? lol, bahamas r gr8, hope L iz gud tlk l8r!
Alien: Screw you, guys. Seriously. OK. These are done and will import daily. These five are too big to be pulled in daily. We don’t have enough disk space, plus it is causing some dangerous locking / blocking on the system in the test environment. Given the overall loads in production right now, trying to pull all of this in this way at that time of day might well crash the box. We need to rethink these and do them in a future release. 
Princess: Nooooooo, it is a MUST GO. ALL the tables must go!
Ghost: I am $DEITY. Make all the tables go.
Alien: Oy! Read my lips: Laws! Time! Space! Cannot be done. PERIOD.
Punch: It’s actually very simple, you know…
Alien: Cool. You do it.
Punch: Ohhhhhh, yeahhhhhhhhh, would love to take up the slack and get this done for you since you’re struggling with it, buuuuuuuuuut, I’m up to ‘here’ in pickle-related testing right now. Sorry.
Rolling on the floor laughing: HEY! HEY! HEY! I CAN DO IT! I CAN TOTALLY DO IT! I HAVE MAD SKILZ! WATCH!
Alien: Um, no, if you do that you’re going to lock down the whole app fo-
Punch: I’ll mentor her. She can totally handle it. It really is very simple. And mentoring is a major part of my job function. Developer V stuff.
Ghost: I am $DEITY. She will do this work. This is a must go item.
Alien: You’re going to break the…!
Rolling on the floor laughing: I FIXED IT! COFFEE CUP GUY SENT ME A WEBSITE LINK AND I MADE A PACKAGE LOL! AND IT WORKED AND NOW THE TABLES ARE LOADING YAY ME! ROFLMAO!
Princess: YAY! You’ve saved the day! Gosh, I guess you learned something today, huh, Developer?!
Punch: My pleasure, Princess. After all, I taught her everything she knows. Mentoring is a major part of my day-to-day.
Rolling on the floor laughing: I IZ A NATURAL TALENT ROFL!
Princess: It’s so great working with such smart, positive people! Everyone should try to be more like you guys! {significant glance at Developer}
Alien: {glances at code, winces, shakes head, closes program, sits back and waaaaaaits for it}
(Fast forward to the morning after go-live)
(!!!!!!BOOM!!!!!!)
Princess: OH MY GAHD THE SERVICE IS DOWN! NOBODY CAN DO ANYTHING! EVERYONE IS LOCKED OUT! WHAT HAS HAPPENED! IT’S THE END OF LIFE AS WE KNOW IT! FIRE ALL THE PEOPLE! FIRE ALL THE PEOPLE!!
Ghost: I am $DEITY! Someone will fix this!
Rolling on the floor laughing: WOW THERE’S LIKE NO DATA ANYWHERE LOL IT’S LIKE THE DATABASE ISN’T EVEN THERE ROFL IS THAT BAD ROFLMAO!
Punch: Ya, I told you guys this would happen…
Alien: No you didn’t. You recited poems about pickles and said it was very simple.
Alien: And that you taught Ms. QA everything she used to build the package that blew out the transaction logs, locked all the resources, corrupted half the tables and then threw the entire app tier into recovery mode.
Alien: At 2:15 in the morning.
Alien: Which was what time the REST of us got paged to come in and deal with it, by the way, just sayin’.
Rolling on the floor laughing: WHAT’S RECOVERY MODE LOL? IS THAT LIKE FOR DRUGS OR SOMETHING ROFL?!
Ghost: I am $DEITY. You will take the database out of recovery mode immediately.
Alien: Can’t. That level of access is restricted to the DBA group. As I’m sure you recall, they don’t report to You, sir, they report to Herself. I’m sure you’ll be hearing from Her soon. Oh hey, is that your pager going off?
Ghost: …mother…of…Me…
Alien: Yeah, pretty much. Have fun with that, sir.
Ghost: …whimper…
Devil: Aaaaaaaaaand, I’m BACK! What’d I miss?!
Alien: Nothin’ much. The usual.
Devil: Hokay then! If you need me, I’ll be unavailable due to meetings!
Alien: Cool. Later man.
Princess: So. While we’re working on fixing the problems your bad code caused…
Alien: …wait, wut…?!
Rolling on the floor laughing: YEAH LOL B/C THAT’S LIKE REALLY BAD CODE ROFL YOU SHOULD HAVE KNOWN BETTER THAN TO DO THAT ROFLMAO!
Alien: …the @^*&@?!?!?!
Punch: No worries, man. Everybody screws up big time now and then. Except me. I’ll clean up your mess for you. You’re welcome, bro.
Alien: Don’t touch it. Seriously. Look at my eyes: I will literally, with malice aforethought and quite thoroughly, kill you dead if you even open the package and look at it.
Punch: No, no, hey: it’s cool. I’m here for you buddy. Gotta mentor the juniors, kinda part of the gig for the #1 Developer. {points at cup}
Alien: YOU ARE NOT THE…
Princess: …aaaaaaaanyway, I’m gonna need your sizing on these items for the next release…we don’t exactly have any requirements documentation just yet, but I think what we’re trying to accomplish is plenty clear without them, we just want to put the thing back in the other thing if a bunch of stuff is or isn’t a certain way. So! Now that you understand the whole vision there…whaddya think? Three hours? Four?
Alien: I quit. I mean it this time. I quit. Wait. You want to put what things in which place? Hang on, because, where are you going to get the…
Alien{becomes absorbed with interesting problem to solve, forgets is quitting, quotes time estimate based on phone conversation with Princess about project because never, EVER learns, like, EVER}
{go back to top and start over, because this is an infinite loop that repeats itself over and over and over and over and over and over and…}

6 comments:

Jan said...

Um, yeah, this is pretty much my favorite post ever in the history of the entire interwebz.

Unknown said...

Oh dear heavens, you work at my company..... :-)

Yes, totally the best post ever.

ZT! said...

You're lurking in our back room aren't you? Except in my case, I'm the Data Cruncher... I don't code the system, I wrest the numbers from the system using my Superpowers of Excel and Stubbornness. And I drive the client and oure managers nuts when I say "the database doesn't work that way, and if you're not careful you're gonna break and IT is gonna hate us again, and then we'll all lose hours while they fix it."

And then I play WoW or Knit and watch the West Wing while IT fixes our database.

Jenn said...

I am laughing so hard I am CRYING over here. Thank you for this!

Ewe-niss said...

I am having an anxiety attack. This is too true.

Michael Irwin said...

Shared this with some of the team here (I also work behind a web site now). General opinion was that it's all too true-to-life.