Just checking in. Busy week.
Training new dispatchers is a drain on the soul. Some more than others. There's more I would like to say about it, but I'll leave it at that. I'm beat.
I need to get motivated to put up the next installment of the tutorial series, which will talk more about system protection (or rather, line tripping and fault clearing, that makes it sound more interesting). It really is among the most fascinating things in the job for grid geeks like me.
Translation for everyone else: Boring. But, many of you are polite enough to sometimes say something about these being 'insightful' or some such anyway. Y'all are so sweet to say such things, you know.
Speaking of grid geeks, one of my old mentors had a very amusing power grid-related cartoon on his office wall before he retired. Thankfully, he still had it when I asked him about it last week, with this blog in mind, and he passed it along to me for your amusement, now posted over there on the right. Click it to see it full size.
Stay safe, and I'll try to keep the lights on at the fire station.
Friday, November 20, 2009
Monday, November 16, 2009
Budget Cuts Fail
Is it still an Engine Company without a pump? Nope. I guess we can go back to calling it a Hose Company.
It's OK, though. In case of a big fire, just call for more companies.
It's OK, though. In case of a big fire, just call for more companies.
Saturday, November 14, 2009
The Grumpy Dispatcher Gets Owned
So one night at drill, I stroll into the classroom a little late and find that the duty officer is leaving shortly and that I was put on the first-out engine in his place as the company officer (cringe).
I look at the assignments, and see a name I do not recognize. Turns out one of our newer members has brought his dad along, who is reportedly looking for an FD to join when he retires and moves to our area, and he has been assigned a ride-along spot on my Big Red Truck. That's cool. No problem.
After the brief classroom portion is complete, I find the guy and introduce myself. A very likeable guy, he's attentive and courteous in every way. I ask him if he's seen the engine yet, as I don't want any confusion if we get dispatched. We walk over and I show him the front-facing rear seat he'll be sitting in, and explain that the rear-facing seat has the SCBA and needs to be kept free for my crew. I tell him we won't leave until everyone reports being buckled in, including him. I tell him to go ahead and hang with us if we go out, follow along but try to stay out of the way, and we'll tell him if he needs to back off. I ask him if he's comfortable with that and if he has any questions for me before we start drill.
He's been listening patiently and showing some interest, and finally nods and says "I'm OK, I'm comfortable around this stuff".
Really? I say, What do you do?
"I've been a firefighter for Santa Barbara County for thirty years" he says.
Just jam that big ol' turnout boot down my throat, please.
He and his kid and several others had a great laugh at my expense - taught me to not walk into class late (after those guest introductions).
It was a perfect job, I'll have to remember that one for use later on.
I look at the assignments, and see a name I do not recognize. Turns out one of our newer members has brought his dad along, who is reportedly looking for an FD to join when he retires and moves to our area, and he has been assigned a ride-along spot on my Big Red Truck. That's cool. No problem.
After the brief classroom portion is complete, I find the guy and introduce myself. A very likeable guy, he's attentive and courteous in every way. I ask him if he's seen the engine yet, as I don't want any confusion if we get dispatched. We walk over and I show him the front-facing rear seat he'll be sitting in, and explain that the rear-facing seat has the SCBA and needs to be kept free for my crew. I tell him we won't leave until everyone reports being buckled in, including him. I tell him to go ahead and hang with us if we go out, follow along but try to stay out of the way, and we'll tell him if he needs to back off. I ask him if he's comfortable with that and if he has any questions for me before we start drill.
He's been listening patiently and showing some interest, and finally nods and says "I'm OK, I'm comfortable around this stuff".
Really? I say, What do you do?
"I've been a firefighter for Santa Barbara County for thirty years" he says.
Just jam that big ol' turnout boot down my throat, please.
He and his kid and several others had a great laugh at my expense - taught me to not walk into class late (after those guest introductions).
It was a perfect job, I'll have to remember that one for use later on.
Wednesday, November 11, 2009
Switching Error: The Verdict
Review: The Switching Error.
The review took three days to complete.
I had to write a lengthy narrative of the events as I remembered them. The station crewman and relay tech involved at the other side also got interviewed. All of my phone calls were listened to, more than once. Some of them were transcribed. My written switching and notes and electronic log were photocopied and sent places.
The 'short' version of events......
The station in question runs a 'dual-bus' configuration. That means each side of the switchgear is used to serve different loads. For protection purposes, the flows are measured across each transformer and then again out each bus, independently. If the "in" does not equal the "out", the logical conclusion is a fault or abnormal condition, and everything related to the faulted equipment is tripped.
We needed to flop one of the loads to another bus, and disabled this special tripping so that everything was counted as one combined sum instead of the two-bus method.
The relay tech convinced me, at 4AM or so, that it was a good idea to go back to the 2-bus protection while we had something flopped over. His motivation: They wanted to protect the thing being flopped over from getting knocked off by their work, and in the total-sum mode any fault anywhere in the bubble would crash the whole thing. Being the relay tech, responsible for installing and maintaining these protection systems, I disregarded the feeling of bad juju based on his "expertise" and being right there with the equipment, and let him do it. Stupid, stupid, stupid error. With one item on the 'wrong' bus, as soon as he toggled back to the dual-bus protection, both buses had in/out mismatches and both tripped.
Still with me? Heh.
FAIL. We dumped what we were trying to protect in the first place, plus everything else off those buses. EPIC FAIL.
Firmly back in charge and angry, I had them toggle back to the 'all together' mode and directed them to start closing stuff up again.
Problem was, one of the breakers that opened also lost its control power, and they couldn't close it back up. I'm still not clear on what the problem was with that, but those guys certainly knew the scrutiny that was now coming, so I don't doubt their motivation to get the breaker closed ASAP. It took an hour, being a Murphy's Law event and all. A preventable momentary interruption is bad, but an hour? Very Bad.
So.... today I learned that I am more or less off the hook. No letter in the file. It was characterized as split responsibility, and I am responsible for perhaps 10%, or something like that. Not free from all blame entirely but won't get pinned to the wall by any means.
I still feel fully responsible, as I know that this could have been avoided.
I got good marks for questioning the change of plan from the relay tech, demerits for folding on his advice against my judgement, but more good marks for taking command and responsibility, and for the extensive documentation. Also, I have a record rather free of blemishes. It didn't hurt that in the background on the tape, one of the guys at the station could be heard, right after it happened, saying "We really screwed up!". I didn't hear that at the time, though. I was preoccupied.
Anyway, they may have let me off the hook, but I still know I could have prevented it. This was the first significant switching error in my entire dispatch career. My best attempt to relate it to you fellow fire guys is the feeling you have when you miss a victim in your search, whether they were viable or not.
I'm unhappy. Grumpy, in fact.
The review took three days to complete.
I had to write a lengthy narrative of the events as I remembered them. The station crewman and relay tech involved at the other side also got interviewed. All of my phone calls were listened to, more than once. Some of them were transcribed. My written switching and notes and electronic log were photocopied and sent places.
The 'short' version of events......
The station in question runs a 'dual-bus' configuration. That means each side of the switchgear is used to serve different loads. For protection purposes, the flows are measured across each transformer and then again out each bus, independently. If the "in" does not equal the "out", the logical conclusion is a fault or abnormal condition, and everything related to the faulted equipment is tripped.
We needed to flop one of the loads to another bus, and disabled this special tripping so that everything was counted as one combined sum instead of the two-bus method.
The relay tech convinced me, at 4AM or so, that it was a good idea to go back to the 2-bus protection while we had something flopped over. His motivation: They wanted to protect the thing being flopped over from getting knocked off by their work, and in the total-sum mode any fault anywhere in the bubble would crash the whole thing. Being the relay tech, responsible for installing and maintaining these protection systems, I disregarded the feeling of bad juju based on his "expertise" and being right there with the equipment, and let him do it. Stupid, stupid, stupid error. With one item on the 'wrong' bus, as soon as he toggled back to the dual-bus protection, both buses had in/out mismatches and both tripped.
Still with me? Heh.
FAIL. We dumped what we were trying to protect in the first place, plus everything else off those buses. EPIC FAIL.
Firmly back in charge and angry, I had them toggle back to the 'all together' mode and directed them to start closing stuff up again.
Problem was, one of the breakers that opened also lost its control power, and they couldn't close it back up. I'm still not clear on what the problem was with that, but those guys certainly knew the scrutiny that was now coming, so I don't doubt their motivation to get the breaker closed ASAP. It took an hour, being a Murphy's Law event and all. A preventable momentary interruption is bad, but an hour? Very Bad.
So.... today I learned that I am more or less off the hook. No letter in the file. It was characterized as split responsibility, and I am responsible for perhaps 10%, or something like that. Not free from all blame entirely but won't get pinned to the wall by any means.
I still feel fully responsible, as I know that this could have been avoided.
I got good marks for questioning the change of plan from the relay tech, demerits for folding on his advice against my judgement, but more good marks for taking command and responsibility, and for the extensive documentation. Also, I have a record rather free of blemishes. It didn't hurt that in the background on the tape, one of the guys at the station could be heard, right after it happened, saying "We really screwed up!". I didn't hear that at the time, though. I was preoccupied.
Anyway, they may have let me off the hook, but I still know I could have prevented it. This was the first significant switching error in my entire dispatch career. My best attempt to relate it to you fellow fire guys is the feeling you have when you miss a victim in your search, whether they were viable or not.
I'm unhappy. Grumpy, in fact.
Tuesday, November 10, 2009
MedEvac + Power Lines = Danger X2
Brought to my attention by Dave Statter, here.
The item of most importance: No one was hurt in this incident and the helo landed safely.
What I want to point out as the fire AND power guy, is that probably most people there did what the camera guy did: follow the helicopter. It's louder, more interesting, and presents itself as more threatening. Equally deadly, though, is the power line, which the camera guy eventually figured out was still live and not drawing enough fault current to trip. It might have initially tripped and then got popped on later by the dispatcher. It could be flopping around. Imagine how your attention would have been split if two lightly damaged helicopters were landing. That double-threat mindset is how this should have been treated: Two separate immediately lethal hazards.
Situational awareness keeps you alive. Tunnel vision kills.
What I want to point out as the fire AND power guy, is that probably most people there did what the camera guy did: follow the helicopter. It's louder, more interesting, and presents itself as more threatening. Equally deadly, though, is the power line, which the camera guy eventually figured out was still live and not drawing enough fault current to trip. It might have initially tripped and then got popped on later by the dispatcher. It could be flopping around. Imagine how your attention would have been split if two lightly damaged helicopters were landing. That double-threat mindset is how this should have been treated: Two separate immediately lethal hazards.
Situational awareness keeps you alive. Tunnel vision kills.
Idiots With Stop Paddles
What happens when someone gives a stop paddle to one of the goofballs?
So I am on my way to work at the power company, still frosted about the switching error, otherwise minding my own business. I have arrived at a cloverleaf, passing underneath and getting ready to do the 270 and get on the other highway, when...
We're all stopping.
An aside: I know not many people do this, but Dad taught me to always look as far ahead as possible when driving. It's easy to spot the people who don't do that by noticing who gets surprised by being caught behind a bus at a stop that you yourself saw three blocks ago and changed lanes to avoid. They're all like "Where did this BUS come from??!" Duh.
Looking ahead during the approach, I noticed no backup on the upper highway, no backup on the ramp, so whatever is going on has just happened. I am about ten cars back and I still see nothing on the ramp. The cars stopping behind me are already blocking cars coming off the other 270 to pass under the overpass, backing them up on the curve. A collision back there feels imminent. I first assume that a wreck has just occurred, so, first checking behind me for other goofballs about to try the same stunt that I am, I then pull onto the right shoulder to approach, mentally remembering where my POV med kit is.
What do I see? A small pilot car for an oversize load convoy. He's just stopped in the middle of the lane, amber lights on, holding a stop paddle out the window. There is no oversize load truck in sight. He's watching a ramp for the truck to approach, but it is nowhere to be seen.
You have got to be kidding me, right? Daily tip: Close traffic right when you need to, but not until then. That's what radios and phones are for, to coordinate the timing. And do it in a position that doesn't cause cross-merging traffic to back up convergent paths. This is the dumbest possible place to pull the stunt he's pulling.
In some states and locales, fire and rescue folks have been arrested for closing the highway in order to protect fire and EMS crews. I know this isn't typical and don't mean to poke at my brothers in blue. I am just illustrating that even we fire guys don't have the authority to close the highway all the time, and better have a good reason when we do. And here's this goofball, sitting in his car, closing the highway in a very dangerous way and without immediate requirement, from the comfort of his front seat.
Oh, no you don't.
So I hesitate a bit to see if the big truck is going to appear. Waiting. Waiting. Waiting. OK, no truck, so I continue on up to where he's at. I can feel the evil stares of people I pass, who assume I am just going to zip by everybody. But I stop behind and to the right of the idiot, and get out, ready to give him a piece of Grumpy Dispatcher mind. Remember, I am still frosted about the switching error, and that was days ago.
No sooner do I get out and take about three steps to engage the idiot, and I hear a siren blurp at me.
Hello, Highway Patrol! The cavalry has arrived. I grin widely at him and give him the thumbs up, but I am in his way. He nods. We understand each other. I move my vehicle, and he moves up to engage the idiot.
I poke my nose out behind the Crown Vic. I roll down my window for a listen. I can hear his tone, he is not happy. Idiot quickly turns off his 'I think I am in charge but I am not' amber lights and pulls to the side. All those people back there who delighted in glee that I got followed to the front by a State Trooper are awaiting my turn to get tagged by the law.
Imagine their disappointment as he disengages to open the lane, me behind him, and off we go. End of incident.
Wish I had caught the name of the company with that flagger. That really warranted a follow-up call before he gets someone killed.
Idiots with stop paddles. Who knew?
So I am on my way to work at the power company, still frosted about the switching error, otherwise minding my own business. I have arrived at a cloverleaf, passing underneath and getting ready to do the 270 and get on the other highway, when...
We're all stopping.
An aside: I know not many people do this, but Dad taught me to always look as far ahead as possible when driving. It's easy to spot the people who don't do that by noticing who gets surprised by being caught behind a bus at a stop that you yourself saw three blocks ago and changed lanes to avoid. They're all like "Where did this BUS come from??!" Duh.
Looking ahead during the approach, I noticed no backup on the upper highway, no backup on the ramp, so whatever is going on has just happened. I am about ten cars back and I still see nothing on the ramp. The cars stopping behind me are already blocking cars coming off the other 270 to pass under the overpass, backing them up on the curve. A collision back there feels imminent. I first assume that a wreck has just occurred, so, first checking behind me for other goofballs about to try the same stunt that I am, I then pull onto the right shoulder to approach, mentally remembering where my POV med kit is.What do I see? A small pilot car for an oversize load convoy. He's just stopped in the middle of the lane, amber lights on, holding a stop paddle out the window. There is no oversize load truck in sight. He's watching a ramp for the truck to approach, but it is nowhere to be seen.
You have got to be kidding me, right? Daily tip: Close traffic right when you need to, but not until then. That's what radios and phones are for, to coordinate the timing. And do it in a position that doesn't cause cross-merging traffic to back up convergent paths. This is the dumbest possible place to pull the stunt he's pulling.
In some states and locales, fire and rescue folks have been arrested for closing the highway in order to protect fire and EMS crews. I know this isn't typical and don't mean to poke at my brothers in blue. I am just illustrating that even we fire guys don't have the authority to close the highway all the time, and better have a good reason when we do. And here's this goofball, sitting in his car, closing the highway in a very dangerous way and without immediate requirement, from the comfort of his front seat.
Oh, no you don't.
So I hesitate a bit to see if the big truck is going to appear. Waiting. Waiting. Waiting. OK, no truck, so I continue on up to where he's at. I can feel the evil stares of people I pass, who assume I am just going to zip by everybody. But I stop behind and to the right of the idiot, and get out, ready to give him a piece of Grumpy Dispatcher mind. Remember, I am still frosted about the switching error, and that was days ago.
No sooner do I get out and take about three steps to engage the idiot, and I hear a siren blurp at me.
Hello, Highway Patrol! The cavalry has arrived. I grin widely at him and give him the thumbs up, but I am in his way. He nods. We understand each other. I move my vehicle, and he moves up to engage the idiot.
I poke my nose out behind the Crown Vic. I roll down my window for a listen. I can hear his tone, he is not happy. Idiot quickly turns off his 'I think I am in charge but I am not' amber lights and pulls to the side. All those people back there who delighted in glee that I got followed to the front by a State Trooper are awaiting my turn to get tagged by the law.
Imagine their disappointment as he disengages to open the lane, me behind him, and off we go. End of incident.
Wish I had caught the name of the company with that flagger. That really warranted a follow-up call before he gets someone killed.
Idiots with stop paddles. Who knew?
Saturday, November 7, 2009
Three Mouse Clicks From...... Damn it
I don't have time to get into great detail right now, but suffice to say that sometimes it isn't a good idea to trust your protective relaying technician in the field at oh-dark-thirty when he says it is OK to go to a strange substation configuration that you'd really rather not do if given the choice.As the Dispatcher, it is my job to verify everything. I trusted the guy who builds this stuff, and disregarded my gut feeling. His expertise notwithstanding, I am in The Chair, the one ultimately held accountable for what happens.
Click, click...click. Yeah, that third click created some interesting times. 2,610 customers were out of power for an hour this morning because I failed to listen to my spidey sense.
Hello, paperwork.
Hello, extended log entries. An aside: Always write so much in the log that no one calls you at home (waking you up) so you can fill in the details.
Just like police/fire/EMS incident reports, these things are legal documents, too. Be detailed.
And someone, actually multiple someones, will be reviewing all my phone calls for that time period. Good thing I kept the shared complaining about the company with my field crews to a minimum tonight. Kind of.
It is also a good thing that it is Saturday morning. Would have been a lot more folks late to work if this had been on a weekday morning.
Damn it.
More later. Have to clean the mess up before shift change.
Friday, November 6, 2009
Your Thoughts and Opinions, Please
OK, I am humbly requesting feedback from my small pool of regular readers who aren't afraid to admit coming back here more than once to read my drivel.The comments attached to the last few power grid tutorials are, well... they aren't. I am out of touch with how they are being received.
Are they working for you?
Are you learning anything?
Am I going too fast?
Are they too long?
Using too much jargon?
Do you care?
Is there something specific on the grid you want me to talk about?
Should I give up on it all and just make this another fire blog in blogosphere already getting crowded with better fire blogs than this one?
I just need to know what you want, so I can deliver.
Thanks for indulging me with your thoughts.....
Wednesday, November 4, 2009
Subscribe to:
Posts (Atom)