Firefighter    ■    Power Dispatcher    ■    Husband    ■    Daddy    ■    Grandpa    ■    Crazy Man
Showing posts with label grid. Show all posts
Showing posts with label grid. Show all posts

Thursday, November 10, 2022

Hell Day

Hey faithful readers. If you are still here, know that I appreciate you still checking back once in a while, I know it is a long dry spell between posts.

Been here at my current power company gig now for about 18 months.  Really liking it, really glad I made the move, and fully intend to retire from here.

When I wrote this post, I talked about how I was one of three hires.  They actually only hired two at first, myself and an internal guy I'll call Eric.  Then, one of the other companies I had applied for called, offering me a job that I turned down because I had accepted this one. Turns out, one of the existing operators here also put in for that other company, and after I declined, he got the offer and subsequently left. So my bosses went back to the list that Eric and I got chosen from, and grabbed the #3 guy, Dave.

Eric started first because he was already an employee and didn't need to relocate.  I had to give two weeks and then move across the country, so my start date was almost a month later.  Dave was an internal employee like Eric, and came in just a week or so after I did

We all went through training more or less together, but it was understood by everyone that I was the fast track guy because I was the only one coming with experience and there was a hole in the rotation that needed to be filled. I was released to operate first as expected, before the year was out.

What wasn't expected is that Eric would struggle as much as he did.  This job just isn't for everyone.  There are so many moving parts, so many things that you need to understand, and it requires substantial capacity to maintain situational awareness of everything going on.  That and the confidence to do your job, be decisive, and take actions as needed without delay.  Then there's loads of mundane abilities you have to master, mostly having to do with navigating procedures and software and such.

Still, Eric was making progress, but despite starting a little later, Dave was progressing more.  About four months after I was cleared to operate, Dave got the blessing and went on shift as well.  Eric and Dave actually were friends before coming in here, they worked together in the field for years before this. Dave being released first was hard on Eric, for sure.

Today was finally Eric's big day to make the final hurdle. Hell Day.  After completing myriad task checkoffs and required courses, getting all of your documentation completed and certifications filed, and after passing a final written exam, you have an all-day intensive review in the training room on the system simulator, getting grilled by four or five other operators and supervisors. Various operating scenarios and system events are loaded into the simulator so the trainee can demonstrate recognition of what is wrong and what to do about it.  Lots of questions, and maybe a few curveballs.  You don't have to get everything exactly right, but you do have to demonstrate that you can function under pressure and are teachable, and that you at least don't do anything to wreck the system.  Better to be decisive even if imperfect, than to do something seriously wrong, or to freeze up and do nothing. It's high pressure, no doubt.

So about three hours into Eric's Hell Day review, during a break, he came out, went to the restroom, and then... left the building without telling anyone, and never came back.

I mean, not everyone is cut out for this line of work, but usually this is discovered in the course of training well before Hell Day or whatever various companies do, and the operator candidate is allowed to coast along and help out as able until they are able to bid back out to a job in another department. Violently flaming out like this is, well... I know it happens, but I've never personally seen it happen before.

It left a dark cloud hanging over the place. Everyone feels bad for Eric. He's a nice enough kid, but maybe he never really recovered his confidence after Dave moved ahead of him. I do fully expect the company will work with him to find him a home elsewhere, probably in his previous department. But man, it's just a horrible thing to have happen.

I'll try to get back to you guys to let you know how it all ends up working out for him. My hopes and prayers are that this ends up ultimately being a good thing, that he finds a job he likes and can thrive at, instead of suffering in here over his head.

Keep the faith, my friends.

Monday, January 3, 2022

Culture

There have been a few challenges in this great reset.  But it seems to be working out so far.

Let me just say, though, that in many ways it is a hell of a lot easier to be a new guy who knows nothing, than a new guy who knows things. How much that helps or hurts, as it turns out, is greatly dependent on the culture of where you land.  Between the cultures of the new power company and the new (old) fire department, the differences have been stark.

It turned out that my new power company historically is known for basically never hiring external power dispatcher candidates. If I had known that going in, there's a fair chance that I would not have even put my name in the bin due to the long odds. But as luck would have it, out of a staff of fourteen dispatchers, they had three openings in short order due to retirements and other factors. Normally they could have handled this because they had enough notice, but COVID gummed up the hiring processes and they ended up behind the ball. As a result, when the logjam was broken, they needed one of the three new hires to be experienced so they could get up to speed within like six months or so instead of the normal 18-24 month training period. And that person ended up being me.

Now it is no secret that I think that I'm really good at what I do, but it still felt pretty good to be the #1 pick for this rare external hire out of a nationwide search. Yeah.

Also, for the record, I put my name in the bin for jobs at other power companies in my plan to relocate. I was offered a job at another one just two days after I was offered this one, before I made it a point to call around and withdraw myself from those other processes.

#1 pick for two nationwide searches in the same week? Yeah, that feels nice.

Anyway... culture.....

At the new power company, open discussion and questioning is encouraged.  My new supervisors and coworkers have all been very open to me talking about my previous jobs. They know very well that this company tends to hire from within, there's not a lot of external influence and therefore not a lot of diversity in ways of thinking about how to do things. When I see something that I think could be done better, they have encouraged me to talk about it, which is pretty amazing really. To be fair, after I say "this is how we did it at this other place", more often than not the response is "that's nice, and interesting, you make some very good points, but we're probably not going to change that right away".  And I'm fine with that.  It's just wonderful being invited to contribute ideas that get a genuine listen. What an amazing culture. Open to discussion, challenges to the norm, respectful discourse. Glorious.

At the fire department, not so much.

I resolved from the very beginning to try to not be "that guy", particularly since in this case I was returning to an agency where there was still some familiarity, a handful of guys I worked with in my earliest Fire days are still here. Still.... mouth shut. Smile and wave. Yeah.... I'm not always very good at that.

Fell in with the old guys well enough for the most part, as I had found and then stayed in touch with many of them via social media, so I was not a total unknown to them.  But there are also of course a lot of newer guys here.  "New" being relative, of course, a lot can happen in 20+ years of absence.  Namely, the agency went through a couple of painful Chiefs, the second of which was so bad that as far as I can tell nearly the entire staff threatened a collective resignation to the City unless something was done. The City got rid of that guy, but the culture damage done over almost two decades of poor leadership had taken root. 

Cases in point: One of the first things I helped with upon rejoining was hose testing. Quite a few recruits and fellow probies there. When three of them grabbed LDH to drag them out, I tried to tell them it was easier work if they'd space themselves out. When multiple people grab hose and they're just a few feet apart, the last person ends up doing the actual pull while the others are carrying just the few pounds of hose in their hands. Also, almost no one was wearing a lid when the LDH and attack lines were pressurized. I've personally seen hoses fail during tests and had couplings launch. That crap can kill you. Tried to suggest people get those helmets on. At least one of the (relatively) older members was greatly affronted by me offering advice (some of it potentially life saving), and he talked crap to a few other members, and that greatly set back my reintroduction to the agency. Maybe I didn't need to give advice about the hose pull and should have just smiled and waved, but I was a trainer for too long, it is in my nature. I won't apologize for suggesting lids be worn, though.

People are just showing up and doing their jobs. People are easily butthurt. People are not open to discussion or mutual improvement to the team. Basically, unless you see something so dangerous that injury is probable, it seems that you mind your own business and look the other way.  And sometimes even in those cases....

This is a horrible culture that is going to get someone hurt or killed.

This has also led to a culture of insecurities and jealousy for some. We don't have room for that in this business. Check your feelings at the door. If I do something wrong or could do it better, I want you to tell me so we can have a rational conversation about it, and if necessary we can do some mutual investigation and fact finding to find the best answers. I might have been right all along, or I might be wrong, and there's also probably more information to inject into the conversation that properly sets context, instead of  just operating with the base assumptions that started the discussion, and until we know, we.... don't. That's how we all improve ourselves professionally. It's not personal. That attitude is lacking amongst several members here, and those members cannot process how much of a barrier that is to their professional development. And as such, in their positions of relative experience and authority, they are passing those errors downstream to new recruits who don't know better.

If I could just bide my time and wait until I am re-established here, then I would be able to gradually try to bend the culture back in the right direction. But there's a wrinkle.... my lovely new bride also decided to join the department.  She is totally new to Fire.  You think you worry about your crew getting home safely, and your trainees being able to absorb enough to keep themselves alive? And now that person is your spouse. Pressure? Worry? Yeah.

So in like our second month back, the wife and I were going over SCBA, just the two of us, with the blessing of the on duty captain whom I've known for a long time. Sean comes into the station, sees us on the floor with a pack.  Being a new (and insecure) Captain who wants to assert his superiority and position, he asks if we need help. Now, I know he is insecure and I want to help him feel better about himself, and I don't want him to feel challenged by me, so I gratefully defer and let him take over.

In my best Dave Barry voice, I swear I am not making this up, he starts going over the various components of the SCBA, and when he gets to the emergency bypass valve, he says "you can use this thing to defog your mask if you need to", and then moves on.  Never identifies it by its actual name. Never mentions what it is actually for, never explains how to operate it, never discusses why you might need to use it.

Dude, you're a Captain.  Really?

Mouth shut. Smile and wave. I can fix this with her later. Right?

Then he moves on and shows her the ICM.... the module with the analog and digital pressure gauges, and the motion sensing unit, that hangs in front of your chest, and says "here's the regulator..."

I can't. I just can't. But still trying to defer to his insecurities and not be "that guy", I try leave him an escape. I know people have different names for the ICM... or just don't know what to call it in the first place.

"What's that? We called it the ICM where I came from, but what is it called here, so I know?", and I casually pick up the actual regulator and make like I'm verifying the bypass is closed.

"We call it the regulator. It's the regulator."

"I thought this was the regulator." I hold it up.

Dave Barry voice. I swear I am not making this up. He doubles down. "No, that just goes on the front of your mask, this is the regulator."

I simply couldn't find the strength to keep my mouth shut. "I've never heard that called the regulator. I'm pretty sure this is the regulator..." (holding it up) "... because this is the emergency bypass which allows higher pressure into your facepiece in case the regulator fails. If that was the regulator, there would not be higher pressure available here at this valve."

I honestly don't remember what all happened after that, but I do know that I basically said we would have to agree to disagree and that he abbreviated the rest of the session and bugged out, but let's just say our relationship has not been the same since. I've watched him bluster through a few other situations, and felt bad for him and his insecurities.... to a point. He's had a couple of times to condescend towards me about "you said you have all this experience when you interviewed, but I haven't really seen it". If you knew how much I wasn't saying, you'd know a thing or two about my experience, Sean.

A part of me dies inside every time we run a call and the chauffeur and officer are more concerned with stomping the Q to death and using the airhorn - through green lights at 3AM - than driving with due regard. Dude, let go of the airhorn chain and put both hands on the wheel.  If something goes wrong, yanking on that chain is not going to be the thing that saves us. Did I mention that our rig, the third out of the house on one of those calls, was halfway out of the bay when the unit that responded before us was canceled by Command, yet we responded anyway by carefully not announcing that we were en route? In order for Sean to sit in the front right seat and make noise and feel important for a few minutes?

Smile and wave.

A few months have gone by since these episodes, and the waters are finally beginning to settle. Thankfully the number of guys with butthurt and insecurity are not in the majority, though their influence is still significant.  There are years of poor formation that will take more years to undo, because so many people just turn the other way because what should be a minor behavioral correction with minimal coaching has turned into hills no one wants to die on.

Next task: Convince certain influential people here that 100psi at the tip is absolutely not appropriate for smoothbore nozzles. They train here to operate all lines for 100psi at the tip no matter what hardware is on the end of the line. Getting zero traction on this so far, but I'm a new guy so I can only push so hard without making those waves. Help.

I'm not trying to bag heavily on my new (old) fire agency.  There are a lot of great people here, and it is a great organization overall, but I remember what it was and know where it can be again. There's work to do here.

If you have a good culture, be ever so thankful for it, and cultivate it constantly.  Recovering it when it is lost is an enormous task. I sure wish some certain new fire compadres could know what it feels like to work in a cooperative atmosphere, one where everyone is on the same team and has the same goals unhindered by sensitive feelings, where a conflict is seen as a growth opportunity, not an attack, without the fear that someone is going to undermine them and pin them with gotchas. Bad culture is toxic. It's not entirely their fault, but they have now become the obstacle to progress... the trick is making some of them realize it. If they were to read this, would they even recognize themselves as such.... or just be offended some more and allow the cycle to continue? I want to bring them to the power company job and let them see what it is like to, you know.... actually selflessly work together.

Or, maybe.... maybe I'm just an insufferable know it all pain in the ass.  Can't rule that out I guess.

All that said, overall it is GREAT to be back.

Stay safe out there, and hug your loved ones.

Wednesday, March 3, 2021

Echoes

My mind comes back around to thinking about this blog from time to time. But most days it feels like I've already told the best stories, and then of course it also seems like the golden age of blogging came to a close in like 2014, you know?

A big part of why I stepped back was that, frankly, my new power company job that I started in 2011 just turned out to not be that interesting. Don't get me wrong, it was a great job that I enjoyed very much, but it didn't provide much the way of blog fodder. And then, after a heavy dose of personal burnout combined with incompetent leadership and growing toxic culture at my last fire job, I had to step away from that. So I really didn't have a source for fire blog fodder any more, either.

So, Three Mouse Clicks went pretty much idle.

In the intervening years, other stuff broke down. In 2018, my wife of 26 years, after too much personal stress not entirely under her control, decided she didn't want to be who she was any more. She walked away from us, her professional life, and nearly all of her family and friends, to start a new life. Three years on, it hasn't worked out so well for her, but you know, choices....consequences. I wish her well anyway.

Then, 2020.

W T F

Don't get me started.

I have spent the last 15 years in a land of predominantly liberal leadership, and closer to one of the cities renowned for 2020's senseless nonsensical violence and rioting than I'm comfortable with. I never took this blog to political places much, but suffice to say I was done living in a region lacking leadership that also couldn't seem to allow people to make choices for themselves. Between that and losing interest in my power company job there, I hit my limit.

Time to hit the reset button.

I took a new job as a dispatcher for a very large midwestern utility. This job is very similar to my first one, but with a better company. Nice, spacious control room. Chill personnel. Big map board. Lots of consoles. High security all the way around. An interesting system with lots of complexity. I like it.

I also have moved back to the small town where I got my first fire gig. The FD there runs 10 pieces out of a large central station, including three paramedic ambulances. It's been 24 years since I left there, but there's still a handful of guys there I know. Dropped in with pie and ice cream to visit the duty shift a few days ago, and all signs point to me ending my five-year hiatus from Fire as soon as things get settled in. I'm still got a few years left in me, and this time I want to go out on my own terms.

Anyway, this place has been filled with nothing but echoes of old tales for a long while. But now I'm picking up the pieces of those earliest echoes, when I was happiest. The reset button worked beautifully.

Oh and yeah.... I'm getting married again. Found a great partner, making it permanent, and no longer going to have to fight the single-parenting struggle with my youngest kids who are still at home.

Life is what you make it, right?

And maybe my new adventures will lead to new stories for this place.

If anyone is still out there reading this, drop a comment so I know this didn't just get broadcast into a black hole.

God bless and stay safe out there.

Monday, December 9, 2013

Emails from the Awesome

Counterpoint to our previous post, this gem arrived recently from one of our end users.
---

I am unsure if this response is automated or not, either way I am going to leave you with a slew of compliments and varying levels of awesome sentences.

Let me begin by saying that this company is the shit (metaphorically speaking of course). If I ever decide to have offspring of my own I am going to name him after your company. I say this because I want him to be as professional and as efficient as you all are. I say I will have a boy because I hate women and they steal all your money (much like my tweaker neighbor).

Next compliment; Pat, your response was both timely and to the point. You deserve a raise. I am not talking about those bullshit 50 cent an hour raises, I am talking about a $30,000 raise. Talk to your boss about this in your review (which will be next week, as I have contacted them for you). 30K may seem a bit extreme but you are the best at your job and you DESERVE THIS RAISE. You could be the king of Amurica' one day. I'd vote for you if I could read.

Next Compliment; You guys are like 7/11, you aren't always doing business but you're always open. If you can tell me where that quote is from I will buy you a pound of coffee and deliver it to your skyscraper headquarters up at the international space station. Sorry, I got off track. Your compliment is as follows: Your facial hair is awesome. If you are a chick named Pat than your facial hair is probably still better than mine. Hats off to you.

---
The customer service rep was not afraid to send this follow up.
---

I want to thank you for taking the time to send the slew of compliments and awesome sentences (I felt they were consistently awesome). We agree that Pat (female, no facial hair) is fantastic. Your kind words have been forwarded to her and I’m sure she will appreciate your email. I do have to disagree slightly with the quote from “The Boondock Saints,” in that we are always open and always doing business. No need to waste your postage money on a package to any skyscraper at the international space station, we are located here in (town). Coffee can be delivered to either location:

(Headquarters location)
(Satellite Office location)

Thank you again for your feedback and best of luck with our future namesake.

---
That was fun.


Thursday, December 5, 2013

Emails from the Crazies

At the power company, we responded to a routine outage, about 20 people were out due to the failure of underground cable.

I've always been a little annoyed at the physics involved in faulted cable.  It's like my hot water heater when it sprung a leak.  It was about 15 years old, and it was fine yesterday, but today it is peeing a thin stream onto the basement floor.  What exactly happened to it, physically, (molecularly?), that was the final straw so that it now leaks?  It is like that with distribution cable if no one dug into it.  Yesterday it was fine, today it blows?  What gives?

Anyway, pretty straightforward problem.  About eight padmount transformers are attached to it.  The fault current was enough to blow the cutout at the tap fuse.  My crews located the fault spot, exposed it, and spliced in a new section.  Very routine.

Later, we get this email in dispatch, forwarded by an amused customer service rep.

I live at 1345 S.E. 49 AVE i have a power outage this morning i want you to che3ck if the power grid got hacked by a computer hacker. I have been fighting computer hacker in my computer for a month or so if it was hacked a hit it was (named removed) of (local medium-sized city) check your system over the outage was on 49th Ave.

Unless computer hackers can access remote strands of underground cable and cause their insulation to fail on demand, I think we're safe from the hacking menace.  This is the reply I sent him, flooding him with so much info he hopefully wouldn't ask any more questions.  He didn't.

Hello _____,

A customer service representative forwarded your concern to me.  I am the dispatch supervisor on duty, and will explain what caused your power outage this morning.

Your home is served by the Pleasant Grove Substation, located on the northwest corner of SE 62nd Avenue and SE 29th Street.  You are on Pleasant Grove #2, which proceeds east from the station on 29th/Boxville before running south on SE 49th Avenue.

At the intersection of 18th and 49th there is what we call a riser pole, where the overhead power lines have a tap with a protective fuse that then runs underground to serve the homes north of that intersection, including yours, and it runs on up to the end of Overlook Drive.  Any faults (short circuits) anywhere in this run of cable from the riser pole to the end of Overlook Drive will cause the fuse at the riser pole to open.

What happened this morning that caused your power outage is that a section of underground power cable went bad near 1109 SE Overlook Drive.  When this occurred and the fuse opened, crews responded and began checking the various transformers and cable sections in your neighborhood before locating the bad cable section.  Underground cable cannot be impacted by computer or hacking activities, it only fails for physical reasons (digging, rocks shifting, etc), or deterioration from age.  Once the failed section was identified, it was isolated from the rest of the area cable, and nearly everyone in your neighborhood was restored before the crews began work to repair the bad section.

This entire incident involved physically-activated equipment and devices, no electronic or computer resources were associated with any part of the event.  I hope this alleviates your concerns, thanks for letting us know.  Let me know if we can be of further assistance.


Still, I have to admit wondering how much it must suck to be convinced an evil hacker is out to get you.  Then again, just because you're paranoid does not automatically mean no one is out to get you, right?



Tuesday, April 26, 2011

This video explains my job in 30 seconds

I swear Sprint must have interviewed some actual power dispatchers before making this commercial, because this is EXACTLY how we roll. Arguing about the dumbest petty crap while simultaneously saving the world as an aside. And then continuing the argument uninterrupted.

Our priorities can be so screwed up sometimes. This is so perfect, I can't believe it. This was aired a few years ago and I just stumbled onto it. Watch this, and you will know everything you need to know about power dispatchers. AWESOME!




Tuesday, April 12, 2011

Don't Be Owned

Originally posted 9/12/2009, removed during a spate of paranoid content cleanup 5/26/2010, and now re-posted after some review and editing.

My company isn't unlike most electric utilities in North America, belt-tightening and cutting back everywhere in the 15 or so years following deregulation. Maintenance deferred, projects dropped, upgrades delayed, and my personal pet peeve... strict adherence to lowest bidder crap instead of paying a bit more for something of exponentially more value.

C'mon... in real life, if you have a choice between a truly great product at a certain cost, or a lousy crappy one for a little less, you put out a little extra and get the job done right the first time, else you pay for the savings later. Usually you pay more than you saved.

Some of us know that.

The company does not. Lowest bidder. Least cost. Period. And we continually pay for it through all available orifices as well as some new ones created for the situation. Overtime, emergency outages, repairs, lost time from dealing with crap. You get the idea.

That wasn't really the rant I meant to share when I started, though.

So, due to this lowest bidder - least cost - maintenance deferred frame of mind, One of my guys gets yet another routine alarm on a high voltage circuit breaker that is low on its arc-extinguishing internal gas (SF6).

He refers to the callout list. It's after-hours, you see. And these breakers can't be allowed to run out of SF6 gas while in service. They will either automatically trip before the gas gets too low, degrading the grid slightly, or they will "block closed", meaning if something happens that would normally trip this breaker, the trip signal will instead be sent to every adjacent breaker. It puts a big dent in your grid when you get a breaker fail operation like that.

So, (once again) due to this lowest bidder - least cost - maintenance deferred frame of mind, there are not very many guys assigned to this area any more. They're not being replaced as they retire or quit or move on. Three guys (including the designated on-call guy) don't answer calls to their home or cell numbers. The one guy that does answer is with his wife at the hospital. Bless his heart, he volunteers to go if we really need it. To hell with that, I'll see a substation burn violently to the ground on my watch to teach this company a lesson before I pull a guy from his sick wife's hospital bedside.

And let me not harp too much on the good old days, when we didn't have an "on-call" guy, we had an on-duty guy assigned exclusively to a trouble truck and waiting for action, fire department-style.

Grrrr.

So, anyway, I call the field supervisor for that area.

Field Supervisor: Hello?

Grumpy Dispatcher: Hey, this is Grumpy, Shift Supervisor at the ECC, We've got a critical low gas alarm at Outback Sub, and my dispatcher can't raise any of your area guys. You have anyone else available or know how we can reach your crews?

FS: Really, you couldn't get Tom or Dick?

GD: No answer on home or cell. we got Harry, but he's with his wife at the hospital and can't go.

I didn't reveal that Harry offered to go anyway, lest this guy glom onto that idea.

FS: Low gas, huh? Well, I don't have anyone around here, I'd have to call someone on OT from Far Away Service Center, that's three hours away. It'll probably be fine until morning.

An aside: I love that our phones are recorded.

GD: All right then, I understand you're assuming responsibility for this incident, and are choosing to defer work on the critical gas alarm until tomorrow. If anything worsens, can I call you?

Silence ensues.... field boss just remembered that our calls are recorded.

FS: Well. ........ You know ..... you know, let me make some calls and see if I can reach anyone.

Uh-huh. Go team.

GD: Sounds good, I'll wait to hear back from you when when you've assigned a crew.

Don't be owned.

If I had let this guy put off the work and something blew up or got someone killed, you know where responsibility lies? The Dispatcher is In Charge. And as the Dispatcher's Supervisor, I am In Charge-In Charge.

When stuff has to get done, make it clear where responsibility lies. I took that burden off and placed in unequivocally in his lap. He knew it, and he knew why. He didn't want to bear that liability. Guess what Captain Cavalier, I don't either. They pay you field supervisor pay. Go earn it. Find your guys, and get it fixed.

I'm sorry about the cutbacks and all, but no part of the cutbacks includes the requirement that I retain the liability for getting people killed or blowing stuff up simply because you couldn't/wouldn't do your job. You see, despite the clear transfer of the burden to the field boss, I would still be held at least partially accountable for not pushing harder and finding someone somehow to get it done. Frankly, spending 45 minutes on the phone and playing voice mail tag is not my job. We are running critical national infrastructure. When I want help, I want it right now. If I can't get it, I will make the boss of that area assume that burden. If he doesn't like that, maybe he'll ensure his guys aren't impossible to reach in the future, huh?

Captain Cavalier shouldn't tolerate being owned by me, either, He should be using the same approach to his bosses to put responsibility where it belongs: Sorry boss, I can't help the OT costs, because you won't hire help and we're behind on scheduled maintenance, so stuff is breaking all around our ears. If we fall behind, and especially if something big breaks because of it, we get fined big bucks.

Push responsibility where it belongs. And deal with it if it is truly yours.

Don't be owned.



Thursday, March 31, 2011

Two-Hatting

Originally posted 7/28/2009, removed during a spate of paranoid content cleanup 5/26/2010, and now re-posted after some review and editing.

Got home from the night shift at the power company at about 0620. Wasn't asleep long when the call came in. Unknown injury accident, vehicle on fire.

It is in 51's first-due, which means me. Meh. I like sleep, and could really use it right now.

As a general rule, due to my location and lack of members nearby, I usually roll alone. Help is coming, though, from the next station over which is staffed full time.

Oops.... forgot, the engine was out for maintenance and no reserve in its place. All I've got is the brush/EMS squad. So be it.

Brush51 has arrived and is Command. One car upside down off the road, fire coming from the engine compartment, There is a damaged pole with wires down on the road. Unknown patient status, investigating. Need the power company, and law enforcement for traffic.

I hate running alone sometimes. There is just too much crap to do by yourself.

I was able to quickly determine that the car was empty, and was told that the driver had walked away somewhere, telling bystanders she was 'fine'. With no one trapped and no exposures to speak of, I was not nearly as concerned about the fire as I was about the lines and adjacent poles holding the weight up. But, the sheared-off pole ended up being a guy wire support pole, the draped wires are just the three guy wires attached to it. None of the 3-phase primary on the other side of the road is damaged, and the fallen guy wires are nowhere near the primary or neutral, so there is no immediate danger from them.

I was two-hatting... power company guy in turnouts, studying the poles and wires while the car burned merrily away.... the fire at that point was of no serious consequence to me beyond keeping people away from it.

I assigned the first-due engine to deal with the fire while I finished surveying the scene and looked for the driver. They arrived a minute or so later and took care of that business. I tracked down the driver, a pleasant young lady who claimed falling asleep at the wheel, refusing treatment.

I returned most of the troops, transforming the scene mostly into a law enforcement event. Besides, I was more than happy to turn control over to them. Without an ongoing fire or a patient, there wasn't much reason for us to be around, but we were happy to provide big BRTs to block traffic.

The young lady presented to all of us (brothers in blue included) as coherent and lucid, not to mention lucky. Then, one of the officers on scene, in spite of the early morning hour, picked up something on his cop spidey sense, and decided maybe a FST was in order.

She failed. Apparently from the residuals of beverages consumed the night prior. At the BAC level she blew, most of us would show the effects. This young lady was not of legal age, either. Practice a lot? And seriously... it's 0730. What was that BAC at the end of the party?

But wait, that's not all. Young lady then admitted that this is the second time she has walked away from a totaled vehicle.

Young lady, you are out of chances. If you want to get dead, please do so in a manner that does not put the rest of us at risk. And preferably in a manner that does not require me to wake up when I am trying to sleep between power company night shifts. A little courtesy, please!

Anyway, two-hatting worked out OK this time, but generally is to be avoided. I know better.



Friday, February 18, 2011

Creative Time Bomb

A long time ago....

Some of the large, stage theater-style lights mounted way above the static board had been acting up. The building was over 30 years old, and the warranties on these lights just seemed to be up. Some burned out bulbs over and over, some just wouldn't work. It was time to replace them.

They picked a quiet Saturday night to start the project, and brought some scaffolding in to assemble over and around the consoles. The new lighting system to be installed required some sort of structural or cosmetic work up there to shield the lights from shining peripherally straight down on us, so they had to completely remove all of the old lights, build whatever it was they were building, and then install the new system. It was going to be an unusually noisy and dusty time in the control center for a while.

As soon as they were ready to start removing the lights, they had to kill the circuit. We had backup lighting set up already so we could continue to work around the project. Construction-style halogen floodlights on tripods.

Several hours later, for whatever reason any particular time is chosen for random things to happen, the panel circuit breaker carrying the backup lights had had enough and tripped. Darkness settled in, except for the slight ambient light from our computer monitors, desk lamps, and the fluorescent lights that were still on behind the static board, bleeding around the sides.

Well, crap. Small-scale outage, eh? The two guys on the scaffolding pulled out their flashlights and starting working their way down. Rich unplugged the tripods and then went to reset the panel, left the rest of us to disconnect some of the tripod lights so we could try it again. The construction grunts basically took that over, so Shawn and I just got to stand there and absorb the unusual situation of a darkened control room.

I looked at the static board to see how much I could make out without walking up closer to it. Not much. Then I noticed lots of points of light on the board. No, I had not been smoking anything.

The static board, you see, is a magnet-based board. However, once upon a time it was a pin-based board, so the entire surface area of the painted sheet metal was covered with a grid of pinholes spaced at 1/4". But the back of the board had some sort of protective gunk slathered on it, so you couldn't see the holes from the back.

Except I could see some. A lot. In a pattern. Of words. Someone had meticulously and carefully punched discreetly through the board to pierce the back cover material so light could come through certain pinholes, not obvious from the back (who ever looked there anyway?), and visible from the front only in this very unusual lighting configuration. It was two lines of words, and several feet wide. You had to move around to get the backlight lined up to see all of it.

"What the hell.... who is Fletch?" I asked Shawn, pointing to the board. It took him a second before he saw it. Shawn snickered as he read the board, but "Fletch" was a mystery.

Rich came back in, and we pointed out the extremely rudely-phrased claim made about Fletch's sexual preferences. Despite his constant joyless grumpy exterior, Rich just started to laugh and laugh and laugh. I thought he was going to get to tears.

Rich told us that a guy who somehow got nicknamed "Fletch", after the character portrayed by Chevy Chase, was frequently a buffoonish pain in the ass. He had retired from operations about five years prior, before Shawn and I arrived. A lot of people didn't like Fletch, he said.

Yeah, what was revealed was pretty strong evidence of that.

Rich was going to take a picture and mail it anonymously to Fletch, but we couldn't get the digital camera to see the pinholes, even when we tried for closeups so we could piece a collage together.

No one ever took credit for it among the crusties or the retirees.

Creative time bomb. FTW.


Saturday, February 5, 2011

Horseplay

Usually a problem in the fire station, right?

Many years ago, when I was just a power dispatch grunt....

Shawn was dispatching central division, and I was on the western division desk, with supervisor Rich on the east side and merely tolerating our presence.

Shawn and I had a goofy game for long slow night shifts where we would lightly toss a racquetball up about 12' to try to get it to settle on a narrow ledge formed by a support beam in the control room. If the ball bounced or rolled at all, it came back down. The trick was for the ball to make it just high enough, and land with no bounce. The competition was to see who could get the ball to stay up there in as few throws as possible.

Once it got parked up there, the contest continued to see who could get it down first by using another racquetball to bump it.

So I am futzing around on nothing in particular, and Shawn decides its time for a match. I stand up and he tosses the ball at me, calling for me to catch it. I am slow to react, and turn with perfect timing as the ball hits me perfectly square in the face, on the bridge of my nose right between the eyes. No harm done, it was rather hilarious, actually.

My brilliant revenge? Pick it up and chuck it back at him. With quite a bit more force. This turned out to be a situational awareness fail, as I did not consider what was behind Shawn when I threw the ball.

No, it was not Rich, the crusty supervisor. He was not remotely in the line of fire. It was the static display board, the large, painted sheet metal wall schematic of the transmission and distribution system, with colored magnets identifying opened switches, closed switches that are normally open, and switches locked and tagged out for jobs.

Things moved in slow motion as Shawn ducked, and the ball blasted into the static board, knocking every magnet and tag within four or five feet of the impact point off the board. They fell like Monopoly game pieces dropped when someone knocks the board off the table. A big mess.

Oh. Crap.

Shawn gave me the 'sucks to be you' grin.

Rich looked up. Sighed. Went back to his book.

It took Shawn and I just a little over half an hour to account for the proper locations of all the fallen magnets and tags from the midnight system conditions, log, and clearance ledger, and then re-check our work. No room for errors.

Rich never said anything about it, because he knew it wouldn't happen again. It didn't.



Monday, December 27, 2010

How Not to do Distribution

If you show up at the wreck or lines down call, and see a distribution pole like this, hopefully it is merely a bad dream. Roll over and go back to sleep. Hopefully it will go away.



I don't like to post pics too close together, but I am having a hard time getting up to write anything right now. Still circling around some of the issues that required some time off from the blog this past summer. Thanks for your understanding.

Hope you all had a good holiday.


Wednesday, December 1, 2010

Bureaucracy

The power company has mandatory safety training annually.

This is good, no? Of course it's good.

It's just that, in regard to their requirements for my job in the control center, I am already trained well in excess of power company standards in all the areas they are worried about through my fire department gig. Now if I was a field grunt, the story would be different. But, I'm not.

I've ducked the mandatory power company training for years, leaving them with a fresh photocopy of my most current EMS cert, professional caregiver CPR cert, and Haz Mat Ops cert every December.

Until this year. The bureaucracy decided that it was not going to be flexible this year. It was do the company-provided training, or forfeit the end-of-year 10% performance bonus.

OK, fine. Whatever. Money talks.

Oh, just so you know, there's only one day left this year that you can make it. Guess what. It's smack in between two 12-hour night shifts, from 0800-1500.

Freaking terrific. I'm not even sure that making me attend this between two night shifts, making for 31 working hours inside of 36 clock hours (with a couple of parking lot catnaps wedged in) is legal or not, but when you're talking about the year-end 10% performance check, I can try to stay awake for a few hours.

Erg.

First off, it had been forever since I had seen first aid training videos not intended for emergency service professionals. The acting was worse, but it was the background music .... now that was so very realistic, just like the music we hear when our tones drop, when we drive to the calls, when we arrive. Very dramatic, just the way we like it piped in wherever we go with our big red trucks. Gripping. Perfect. I told my closest compadre in the class that we have a looping audio cassette in many of our rigs that plays that sort of music when we're running hot, just to get is in the hero mood groove.

I like how the video showed compressions-only CPR, with two people taking turns. Except that the person taking a break is doing nothing to maintain an airway. Airways are overrated, I guess.

Or how the instructor said that someone who is not breathing will always be in cardiac arrest. Don't assess for a pulse, just start compressions. ???

The review of how to read haz mat labels and MSDS sheets, something a shade beneath the annual haz mat ops training we go through every year at the FD.

Scene safety? What's that? The first-aid-trained actors arrived at a traffic accident at night and just left their car in the road without even hazard flashers on, never looked for scene safety problems, just waltzed straight into the thick of it.

I like how a rescuer actor carefully donned their PPE gloves before administering the Heimlich. That choking guy can just hang on while you go get your gloves, you know. Where did I put them anyway?

When doing the hands-on CPR run through, not even one person (except for me) managed to get chest rise from the practice doll. No matter. Next!

(Sorry this is so disjointed and choppy, my bad form comes from about three total hours of sleep over the previous 40 hours or so.)

A particular favorite scene: a couple of actors are talking, one asks the other, "So Kenny, how is that new project you're working on going?" Kenny opens his mouth, draws up a pained expression, and collapses. My compadre behind me wasted not even a second before piping up with "Oh my God! They killed Kenny!" FTW!!


Then we did Halon awareness training. Zzzzz.

And fire extinguisher inspections and discharge practice. I've taught this class myself many times. Zzzzzzzzzz.

Let us not forget I was up most of the day before, did 12 hours awake on shift overnight, and then had to sit through seven hours of this for my 10% check before rolling back into another 12 hours of babysitting power lines overnight. ZZZZZZZZZZZZZZZZZZZ.

I didn't want to come off as a smartass, and was determined to keep my mouth shut. Still, a couple of times the instructor said some well-meaning but half-baked stuff that I felt obligated to do damage control on. It didn't go over with her very well despite my overly deferential approach, so I stopped trying. Oh well.


I did learn something new though. Our Halon tanks have symbol signage which clearly endorses pranking your friends with portable marine air horns. Note that there is no circle with a slash through the pranking action to imply prohibition. How cool is that?

Nap time. Please hold all 911 calls for about ten hours or so. Sorry my attitude sucks today, it's only temporary. Thanks.


Thursday, November 18, 2010

Classic Kaboom!

This clip has been around for a long time, so you may have seen it elsewhere.

This is a 138/23kV substation, and a capacitor bank on the low side (used to boost voltage) is for whatever reason experiencing a relatively low-current ground fault. For some reason, the station's protective equipment is not detecting and then clearing the fault.

The sustained fault arc is cooking and welding and destroying everything it can reach, like a death ray blaster stuck on maximum. How hot does an electrical arc get, you ask? Oh you didn't ask? Well I'll tell you anyway. Depending on the amperage and voltage of the arc, the temperature can vary widely between 9,000°F and 36,000°F. Hot. For reference, the outer visible surface of the sun, the photosphere, is only about 10,000°F. Hot.

So, where were we? Oh yes, this hotter-than-the-sun arc is blazing away next to this unlucky transformer. It doesn't take long for the transformer's flammable mineral oil to overheat, boil and expand, and finally cause the transformer's overpressure safety valves to release the oil as a high-pressure spray.

Incidentally, transformer banks are typically equipped with pressure relays to detect and de-energize the bank within a second or less when something like this goes down. Since there are two independent show-stopping problems going on (arc fault current and transformer overpressure) that are not causing any kind of shutdown, I have to conclude that the station's entire protective relaying system was inoperative.

Back to the long boring story. The high-pressure flammable oil spray meets the hotter-then-the-sun arc, and the result is a foregone conclusion. The spray and associated fireball with its conductive smoke particulate byproducts also seems to somehow finally cause a good hard high-side fault to occur (watch and listen for the flash/bang right as the fireball goes up), and this fault is at last detected, by whatever station is feeding this one. The other station says "Whoa, something's going down out there!" and opens up the feed. Alas, a tad late to save the day.

The transformer alone will cost perhaps about $750,000 to replace, the other destroyed station components at least that much again, and don't forget to add the cost of the environmental cleanup, and other peripherals.

Repeat message from an earlier post. Stay away from the pretty sparkly show:




Friday, October 22, 2010

Transmission Towers as Art

Although I am a power company guy, and I have a first-hand understanding of the dire need for transmission system upgrades in North America, I am not a proponent of stringing power lines everywhere willy-nilly. They're not pretty, they degrade property values, intrude on scenery, and the jury is still out on health effects (though power companies deny that with the same gusto that tobacco companies denied nicotine addiction).

I got a note from Mack505 of Notes From Mosquito Hill with a link to some interesting stuff. It seems that a company has engineered a transmission tower structure with configurable components, built to resemble the human form. These towers can be installed in a variety of positions and arrangements while still maintaining their required structural strength and integrity and maintaining adequate conductor clearances.

I'll just leave you with the pictures, two links, and this comment:

These are freaking cool.




See more, read more:

"The Land of Giants" by Choi+Shine Architects

Who said pylons have to be boring? (Bayou Renaissance Man)

Thanks, Mack505. Good stuff.


Monday, October 18, 2010

Boom, Baby!

Check this fault current arcing over the cutout fuse. This is just line current serving load, not grounding as a fault. At least not at first.....



There's a big, huge, violent difference between arcing line current and fault current to ground.

Stay away from the pretty, sparkly show.


Friday, October 8, 2010

Erasing the Name

I came in for a night shift at the power company a few days ago. The guy I was replacing was not at his post. I waited for him to reappear, but he was nowhere to be found. I finally asked the other dispatchers. Where's Connor? In the upstairs conference room? Oh, OK.

Went to the conference room, and opened the door. Lots of suits and ties in there, most of whom I recognize. A Director, a VP, a few HR people, my direct boss, and Connor. He wasn't in a "hot seat" configuration, so I knew he wasn't the subject of the meeting. The talk stopped instantly as eyes turned to me. Uh, Connor, can I log you out? Yeah. Closed the door and escaped.

------------

Gerald was enjoying a quiet night. Until of course a storm tore through and started causing him some fits. Trips and recloses. A few lockouts. He was sending crews out here and there for the things he couldn't fix remotely by SCADA. That's what we do, after all. At the end of his shift, he handed off to Matt with the required hand-off debriefing, and went home.

Matt knew that a particular 115kV line was still out from the storm, and that crews were already at work on repairs. A few hours later, the crew called back to return their Clearance. Matt grabbed the Clearance ledger and the "in-progress" work documents. There was a small problem, however. The Clearance ledger had no record of the Clearance on that line. And there was no documentation of the switching orders issued to clear it, let alone the orders prepared to return it.

So, Matt pulled me into it. Why me? Because I'm a Transmission Operations Duty Supervisor now. Yeah, I had not shared that bit with you yet on the blog, but I've been a shift work boss for almost a year now, directly or indirectly supervising about ten dispatchers at a time. Why in the world they think I am responsible enough to do that is beyond me.

Matt and I dug around some more, and found nothing. Matt had to find out from the field crew exactly what had been switched out, and how the Clearance had been given, in order to take it back and put it together. Needless to say, this is irritating and scary for a dispatcher, to have to re-fabricate a "go-back" with no record of the "take-out" to base it on. Matt is good though, and handled it.

But what happened to get here? Gerald didn't answer his phone, as no doubt he was now asleep. So I got to do my favorite boss job: Listening to taped calls.

And I was not pleased with what I heard. Gerald's quality of work was somewhere between very unprofessional to deadly dangerous, and I could tell the field guy was not liking how it was going by the tone in his recorded voice. But the field guy pushed back just enough that things were done at least to his satisfaction, though he had no control to make sure Gerald documented what had happened.

No documentation. Crappy phone work. Shoddy switching. Invalid Clearances.

Connor was Gerald's shift boss, so he and I got together to compare notes. Connor and I later held a little internal meeting with Gerald in the conference room. Gerald got copies of our switching and communications policies, and copies of the scary transcripts I had prepared from his calls. Gerald was warned. Stay on operating policy and follow the rules.

------------

Gerald was bummed. He felt that the world was out to get him. Yeah, if you're risking lives through your incompetence, then that is probably a fair statement. All dispatchers whine too much, but Gerald was doing a lot of it in the weeks since his brush with Connor and I.

Then Gerald issued a routine Clearance. Logged it in the ledger. Put the in-progress document away. Not long after, he got a call from a crew that had just arrived to switch out one end of the job.

Say what?  A crew has just arrived to switch out something that just had a Clearance issued on it??

Yeah, Gerald managed to issue a Clearance on a section that, while de-energized by opening circuit breakers via SCADA, was not actually lockout/tagout cleared. Crews had hung hard grounds on 345kV wires that were just three mouse clicks from being closed in on.

Gerald failed to follow policy and issued his orders... well, out of order.

So Gerald pulled the in-progress document back out, issued the order to clear that end, got it all back, and quietly put it back away. Generally speaking, no one critically listens to phone calls or reviews the logs without cause, so it will all go away as long as no one has reason to look.

Wrong answer. The switchman and the work crew somehow got together for lunch, and somehow ended up comparing notes on the times they were working on stuff. Needless to say, the work crew foreman was out for freaking blood.

Connor took that call.

------------

So I logged Connor out and settled in for my shift. He eventually came out and briefed me on the day's operating events, but was not at liberty to discuss what had happened in the conference room. Need to know and all that. Whatever, I have my own problems. Office gossip holds no appeal for me.

------------

Came in tonight for another 12 hours. Checked my email. There's one from my boss to the troops. "Gerald has announced that he is retiring effective today".

Next email was a management-only note from the boss with the details of the incident.

So that's what that was about. Retire or get fired. You don't get many chances in this line of work, any more than you would get in, say, air traffic control. Realistically, if you commit a switching or clearance error more than once over a two or three year span, you're pretty much gone, no matter your tenure or length of service. The last guy I had to participate in "helping out the door", like Connor had just done with Gerald, had over 20 years in. I felt bad, but he seriously nearly killed linemen three times within a year. There is absolutely no tolerance for that.

Looking up on the wall of the Duty Supervisor's office at the shift assignment board, I saw Gerald's name, on 'A' shift, where it had been for years.

I erased it, and then logged in for the night and looked over the next day's scheduled work.



Friday, September 24, 2010

Unplanned Kitchen Update

I actually thought the kitchen saga was more or less over. The work was done, illogically and haphazardly, and we had moved on. But no, someone had to kick the beehive again.

If you're new to this blog, there is a long, long history to this. I dare you to put on your favorite Dilbert pocket protector, refill your coffee, and read all the posts tagged Kitchen Remodel, from the oldest post forward.

Came in for the day shift a few days ago to find a slight modification. Specifically, a new counter.


No one asked for it. We sure didn't need it. There's no plumbing, no electric, and no nearby anything else you want in that corner. And, so far, nothing to put on it.

We just added clutter and cleaning space, usage pending. Important stuff.

I suspect we could have used this money to move the fridge as previously intended, but as usual I am not privy to the master plan.



Thursday, September 16, 2010

Incomprehensibility Follow-Up

More laugh-out-loud fun with numbers. I promise to be brief.

Review: Grasping Incomprehensibility

I realized that I did not try to quantify how big of a computer monitor we would need to show the trend line of our ACE without changing the scale. Our display shows the ACE up to 300MW high or low before it goes off the edge. We can still see the +/- 300MW numeric value when it goes out that far, but the trend line falls off the monitor.

Anyway, today I did the math.

When the ACE jumped to 1,428,382,474,567,680, how high would that theoretical trend line have gone? Try 3 trillion miles. 3,005,855,375,774 miles, to be exact.

To put that into perspective, that is 819 times the distance from the Sun to Pluto.

It is 8.25% of the distance to the next closest star, Proxima Centauri.

If you wanted to see the top of that trend when it happened, you would have to travel at the speed of light for just over six months to get to the high end of your theoretical cosmic computer monitor.

See, math can be fun if you let it.



Wednesday, September 15, 2010

Owning the Incompetent with Humor

In another life, I spent some time in the software industry. I had been in the electric utility business for a few years when I was offered a job with one of our vendors. At the time, it seemed like a good idea. OK, I'll admit that in retrospect it actually was a good idea, as my three years there looks really good on the ol' resume now that I am back in the system operations and dispatch circle of the industry.

I became sympathetic to the lives of software engineers, as I worked with them at length while dealing with dispatchers and power marketers using the software. I became a liaison of sorts, telling the software wonks about life in the control center, and telling dispatchers about software wonk life trying to make lots of people with often conflicting opinions all happy at the same time.

Sympathetic to a point, that is. I've never been able to tolerate blatant incompetence. My own included, I must add.

Unrelated to that, as yet, is that I am also a wise-ass at work. Many many years ago the senior dispatcher on my shift would always react the same way when a line tripped. The first thing he would do was quote Robot (voiced by Dick Tufeld) from Lost in Space: "Danger! Danger, Will Robinson!" Then he'd get up, refill his coffee, and saunter back to the console to fix the broken stuff.

The quote inspired me, and it was very easy to find a WAV file of that quote online. I added a toolbar shortcut to it, and could play it loudly with a single click every time a line tripped. Hilarious. You had to be there.

I soon expanded my WAV collection and built an interface to instantly play back any of the 75 or so witty one-liners I collected over time. Some examples:

"I've felt a great disturbance in the force." (Obi-Wan Kenobi/Alec Guinness, Star Wars: A New Hope)

"Scotty, I need warp speed in three minutes or we're all dead!" (James T Kirk/William Shatner, Star Trek II: The Wrath of Khan)

"Look, I'm gonna tell you about an accident and I don't wanna hear about an 'act of God', OK?" (Jack Burton/Kurt Russell, Big Trouble in Little China)

"No sense worrying about it now." (Raymond Stantz/Dan Akroyd, Ghostbusters)

"I think it is time we demonstrated the full power of this station." (Grand Moff Tarkin/Peter Cushing, Star Wars: A New Hope),

"Finally, we shall bring order out of chaos." (Egg Shen/Victor Wong, Big Trouble in Little China)

"I'm sorry, but this is a highly sophistamicated doowacky, you don't treat it responsibly, ka-BLAM-o!" (Homer Simpson/Dan Castellaneta, The Simpsons).

You get the idea. So where the heck am I going with this?

Do you remember how reserve sharing works? If not, go refill your coffee now, then come back and review Tutorial 4: Basic ACE, and Reserve Sharing before proceeding here.

So one day we're testing a new reserve sharing application built by my former vendor employer. It's not tied to anything yet so it can't actually make things happen, but you can tell it to do stuff and see how it would theoretically react. I have a sneaking suspicion (aka insider info) that it isn't quite ready for prime time, and evilly relish the thought of breaking it. I am suspicious that they are testing it while it has known deficiencies, hence is a waste of everyone's time. Test it when you really think it is ready, instead of pretending you're making good progress and hoping no one exposes you by breaking it. Their bad luck I was on duty.

Every utility in the entire reserve sharing pool had someone in on the conference call. Our turn came up, and we were to simulate requesting replacement energy for loss of a power plant in our system. I selected a large, 550MW plant that was shared contractually with three utilities. The software threw down an ugly error message to everyone's screens. Exposed. Software guys fumble mumbled apologetically about how the application was not configured to handle shared units yet.

So why are we testing it? Grrr.

OK, I say, let me try something else. When picking off the big 550MW unit, I noticed that their drop-down list of power plants included every single dinky unit we had. So this time I picked a tiny 0.85MW (850kW) hydro plant. Submit.

Everyone's alarms went off, as you could hear over the phone. So the application alarm was working fine. Then the comments started. "I got the alarm, but no megawatt contribution". "Same here". "Yeah, we see the event but are not being asked to provide assistance" Then the largest utility, with the biggest prorated share of the pool, said "I got one megawatt on my display". Mind you, they are so big that they can swing 100MW every few minutes as a matter of routine. 1MW to them is rather less than a day-old fart in the wind.

Software guys were feeling pretty burned now. But I knew these guys, and was not exactly a big fan of their work to begin with. Finally I played along and picked a "safe" 220MW sole-owned plant, and the application worked OK and let everyone see numbers like they wanted to.

As we continued through the round-robin testing, a couple of other dispatchers followed my lead and started trying to break the thing on purpose with new tricks. A few were successful.

As the call wound down and a sort of debrief was being held, I grabbed my PC speaker and pulled up the WAV interface. Holding the speaker by the phone, the entire conference call was treated to:

"So far, this is not blowing my skirt up, gentlemen." (Spencer Trilby/Charleton Heston, True Lies).

Several rather rewarding snickers were heard from other participants. Conference call anonymity is nice.  The software guys were silenced for a few seconds, and then went on like it hadn't happened.

Next time we tested the application a few weeks later, shared units worked, and no plants below 25MW were in the pick list.


Sunday, August 15, 2010

Grasping Incomprehensibility

It's a geek post. You've been warned.

If you are a regular reader, you have some basic familiarity with how bulk power systems are run. The topic of this post involves generation and load balancing. That is, matching power output to power consumption, adjusted for the expected power flows to other systems based on purchase and sale agreements. If you're meeting your obligations, your instantaneous ACE (Area Control Error) is more or less zero.

So, once in a while, events transpire such as those documented in Of Yardsticks and Manly Measurements, but without the drama. Having a meter go bonkers and sending you out by a few hundred or even a few thousand is just something that happens once in a while, and part of why we're here. Someone has to fix the broken stuff, you know.

But this event was different.

The visual ACE alarm activated, meaning we were at least a few hundred megawatts out of nominal, so we stopped jawing and took a look. But instead of showing us a number that we could comprehend on the big monitor, we saw 1.42838E+15.  (Not that I memorized the number....I took a screen shot.)

What the huh?

I didn't even know that our software knew how to show gigantic numbers in that scientific notation format, let alone what to do with them.

Although I didn't dig deep enough to get the exact value until later, I can tell you now that it was 1,428,382,474,567,680.

That's 1.4 quadrillion plus.

Every application and system that was was paying attention to this number summarily disabled itself, having just had a clear view into infinity and needing to sit down with a cup of tea to think things over.

It wasn't long before we found the source of this particular problem. For typical meter oddities that crop up, the 100 here, 500 there, we sometimes have to poke around to find it, but 1.4 quadrillion is hard to hide in a closet. We quickly tracked it down to a faulty tie line value that represented a remotely-located power plant, and flipped it to its backup data source, problem resolved.

That was cool.

So, how big is 1.4 quadrillion, exactly?  Stay with me through this, because I promise this will get slightly amusing again after we trudge through the technical stuff.

First, let's translate what a megawatt is again, as a refresher.  Megawatts (MW) are a representation of rate, or flow, while a Megawatt Hour (MWh) is a representation of delivered totals.  Think of it in fire pump terms.  A pump flowing 500gpm will deliver 500 gallons every minute.  If allowed to run for only 30 seconds, it will deliver only 250 gallons.  Likewise, a power plant running at 100MW but online for only 15 minutes will deliver only 25MWh.

To allow you to get your hands around what a MW is, exactly, look at your power bill.  1 megawatt is equal to 1,000 kilowatts, and your monthly household electricity consumption will be shown in kilowatt hours (KWh).  A typical home might use roughly 1,000 KWh (or 1 MWh) per month.  So, a 100MW power plant produces, roughly, enough power every hour to supply one hundred homes for a month.

Back to the big number: 1,428,382,474,567,680

It isn't really accurate to call it 1.4 quadrillion megawatts, it's like saying a million billions or something.  Quantity steps of watts use the same system as computer memory storage measurements: kilobyte, megabyte, gigabyte, etc.  1,000 kilowatts is a megawatt, 1,000 megawatts is a gigawatt (already moving into irrationality in power company terms). 1,000 gigawatts is a terawatt, then.... petawatt, exawatt, .... what we have here is 1.4 zettawatts.  Zettawatts?  L O L

How much theoretical power was this, really?

According to Wikipedia (the oracle of truth, I know), global energy usage is approximately 15 terawatts annually.  If this power plant had actually been producing 1.4 zettawatts and kept it up for just a single hour, it would have produced enough energy to power the globe for 95,225,498 years.

That is still beyond comprehension.

OK, it produced enough power every second to serve the Earth's energy needs for 26,452 years. Or to power Doc Brown's DeLorean DMC-12 time machine through nearly 328 million time-travel jumps.

Still hard to grasp, but we'll leave it at that.

For what it was worth, it was still only equivalent to 4 trillionths of 1% of the sun's energy output rate.

In real life it would probably have left a crater at that power plant's location a few thousand miles wide, and vaporized every power line on the continent in one trillisecond of glorious multicolored flame.

Good thing it was just a meter error. Feel free to allow a banking error of that magnitude on my account, as I am sure it will meet my needs for a several thousand years, too.

That was cool.  Thanks for letting me geek out a little bit.  My two jobs make me so serious on the clock, that I have to get a little goofy sometimes.