Switching Error: The Verdict

Review: The Switching Error.

The review took three days to complete.

I had to write a lengthy narrative of the events as I remembered them. The station crewman and relay tech involved at the other side also got interviewed. All of my phone calls were listened to, more than once. Some of them were transcribed. My written switching and notes and electronic log were photocopied and sent places.

The 'short' version of events......

The station in question runs a 'dual-bus' configuration. That means each side of the switchgear is used to serve different loads. For protection purposes, the flows are measured across each transformer and then again out each bus, independently. If the "in" does not equal the "out", the logical conclusion is a fault or abnormal condition, and everything related to the faulted equipment is tripped.

We needed to flop one of the loads to another bus, and disabled this special tripping so that everything was counted as one combined sum instead of the two-bus method.

The relay tech convinced me, at 4AM or so, that it was a good idea to go back to the 2-bus protection while we had something flopped over. His motivation: They wanted to protect the thing being flopped over from getting knocked off by their work, and in the total-sum mode any fault anywhere in the bubble would crash the whole thing. Being the relay tech, responsible for installing and maintaining these protection systems, I disregarded the feeling of bad juju based on his "expertise" and being right there with the equipment, and let him do it. Stupid, stupid, stupid error. With one item on the 'wrong' bus, as soon as he toggled back to the dual-bus protection, both buses had in/out mismatches and both tripped.

Still with me? Heh.

FAIL. We dumped what we were trying to protect in the first place, plus everything else off those buses. EPIC FAIL.

Firmly back in charge and angry, I had them toggle back to the 'all together' mode and directed them to start closing stuff up again.

Problem was, one of the breakers that opened also lost its control power, and they couldn't close it back up. I'm still not clear on what the problem was with that, but those guys certainly knew the scrutiny that was now coming, so I don't doubt their motivation to get the breaker closed ASAP. It took an hour, being a Murphy's Law event and all. A preventable momentary interruption is bad, but an hour? Very Bad.

So.... today I learned that I am more or less off the hook. No letter in the file. It was characterized as split responsibility, and I am responsible for perhaps 10%, or something like that. Not free from all blame entirely but won't get pinned to the wall by any means.

I still feel fully responsible, as I know that this could have been avoided.

I got good marks for questioning the change of plan from the relay tech, demerits for folding on his advice against my judgement, but more good marks for taking command and responsibility, and for the extensive documentation. Also, I have a record rather free of blemishes. It didn't hurt that in the background on the tape, one of the guys at the station could be heard, right after it happened, saying "We really screwed up!". I didn't hear that at the time, though. I was preoccupied.

Anyway, they may have let me off the hook, but I still know I could have prevented it. This was the first significant switching error in my entire dispatch career. My best attempt to relate it to you fellow fire guys is the feeling you have when you miss a victim in your search, whether they were viable or not.

I'm unhappy. Grumpy, in fact.