The Smart Grid, Part 1 - First Tell Me About LMP

I already know this won't be brief. No matter how much I say I'll try to keep it short, I can't. I talk too much and over-explain everything. Having accepted that, I sleep OK at night at peace with who I am.

With that out of the way....

Why do we need a "smart grid"?

If you've been following me for a while and have read the tutorials, you know the basic truth that AC power is generated exactly as it is consumed, with no practical storage applications. As demand goes up, frequency drops, and power plants increase output to match demand, and vice-versa. It is fluctuating every second. Thankfully, the smoothed out impacts of millions and millions of people turning lights and toasters and Blu Ray players on and off don't amount to much instantaneous fluctuation relative to the size of the interconnected grid. If you were trying to run a power grid of only five houses with a plant just barely big enough to do it, and they all happened to turn on a bunch of stuff at once, everyone would see a big dip and maybe low voltage damage - assuming it didn't trip the generator outright.

Following demand is not an efficient game. During high demand periods, power plants with expensive fuel sources have to run to keep supply up, raising the aggregate cost of power. During low periods, only the cheapest units run, lowering the aggregate cost, but when coal plants run below efficiency (usually under 80% or so), they create more pollution relative to power produced. Running thermal plants up and down every day for years wears them out, as the boilers and systems fatigue and break down from being heated and cooled over and over and over again. Overall energy costs go up not just from fuel costs, but repair and maintenance costs.

Despite real-time hourly changes in energy prices based on what power plants are running at that time of day, almost everyone at the consumer level pays a flat per-kilowatt hour charge for electricity. This is tricky for power companies, as they need to make their rate case with magic math that figures out how much revenue will be coming in based on which hours their systems are used and what energy costs actually are, with hopes that they get enough money to run the thing, but not so much that the consumers scream foul. It is frankly very easy to screw this up, and a couple of disastrous power market days with real-time energy prices increased by 1000% or more can take years for a company to recover from with their rate case locked-in usage fees.

The first step towards efficiency is to get all power companies in a large area to cooperatively control their power plants instead of playing the hourly market game. The Midwest Independent System Operator (MISO), a quasi-governmental reliability agency with authority over a huge footprint of power companies in the midwest, was the first to do this on a really large scale. Their coordinated control concept was based on a model called Locational Marginal Pricing (LMP).

Don't let these acronyms scare you, I'll try to keep it simple.

Let's say you have a self-contained power system with four power plants and no ties to the outside world. Your transmission lines are overbuilt and there are no restrictions on power flows anywhere in your system. Your customers pay for their electricity based on hourly usage instead of a flat round-the-clock average cost. Your power plants use different fuels and therefore have different costs. You run the cheapest unit until it reaches capacity, and then start the next cheapest unit. When #2 peaks, you start the third unit, etc. Figuring out your hourly energy costs are easy, it's basic math based on how much was produced from each plant for any particular hour, and everyone in your system is charged the same thing for any particular hour.

In this model, location means nothing. Pricing is flat across all areas by the hour. The wrinkle that screws everything up is constraints on the electric transmission system.

Back to the same model. Self-contained system with four plants, one in each quadrant. But this time, the transmission lines between the quadrants do not have unlimited capacity. As long as no lines are at their limits, the flat hourly pricing works fine. Let's say the cheapest unit is in quadrant 1. Now, as long as demand is low, that cheap unit can supply everyone fine, but as the day picks up, demand in quadrant 2 rises to the point that if the cheap unit in quadrant 1 keeps going up to cover, the power line between them will overload. Since the cheap plant cannot supply the demand in quadrant 2, the next-cheapest plant somewhere else will have to come online to make up the shortage.

At this point, you have locational marginal pricing, based on transmission (transportation) constraints. Back in quadrant 1, energy stays cheap, and only the people in quadrant 2 who did not have the foresight to increase their transmission line's capacity have to pay the averaged costs of what they could get from the cheap plant and the remainder they had to get from somewhere else. It's no different than living next to a bakery and getting cheap donuts, but over the hills where they have a small bakery that can't keep up and they can't ship enough donuts in from elsewhere on the only dirt road into town, the value of a donut logically goes up.

Now, the MISO did exactly this, back in about 2003 or so, but on a GRAND scale. Covering all or parts of 11 U.S. States and the Canadian Province of Manitoba, with 35 power companies owning transmission assets and another 98 companies owning generation and/or serving load, they coordinate over 130,000 megawatts of energy generation from hundreds of power plants. In real time, pulsing power plants to move up and down to live within the cheapest-power-possible-to-where-you-are LMP model every few seconds.

How do they set prices? The generation owners bid the units into the MISO system, with costs based on output. They even have allowances for efficiency bandwidths, so you can say your power plant costs 'x' from 200-400MW, 'y' from 401-475Mw, 'z' 476-510MW, etc. The MISO super AGC computer sees real-time demand across their entire footprint, determines when generation needs to increase, and finds the next cheapest unit anywhere in their entire area to go up. Except that the MISO super AGC computer also knows what the real-time flows on all of the transmission lines are, and knows how moving any generator up or down will impact every transmission line. So it doesn't just grab the next cheap unit in the stack, it grabs the next cheap unit that can increase without overloading any power lines anywhere else.

This creates hundreds of pricing bubbles in the MISO footprint based on LMP restrictions. The MISO super AGC computer measures consumption at thousands of points on the system and knows which power company is the consumer at every given point. Knowing how much energy you used at any given consumption point every hour, and knowing the energy cost in that bubble for that hour, it is no great leap to figure out what to charge you for the energy based on your usage at all of your consumption points. So, in effect, the MISO "buys" the energy using cheapest-possible source from the bidding process and influenced by the LMP model, and then sells the energy back to the members with pricing based on their usage and location, keeping a cut to operate their massive bureaucracy.

This effectively killed the hourly power market in MISO, as it completely ended the hourly guessing game. Your units ran if you priced them low enough to get picked up and had transmission availability out of your system, and you know you always got the cheapest energy possible that was offered. If your costs seem too high, build more power lines into your system or help your neighbors build lines that will help you get more cheap stuff. The day of the energy trader making hourly deals like working Wall Street ended in MISO.

It was gigantic gain in efficiency, as you no longer run your own units up and down to meet only your local demand. Cheap units run pretty much full power all day, and the load increases and decreases at scales large enough that it is unlikely that the stack will land on any one unit and run it up and down for hours. Increased demand flies through the stack quickly enough that your unit running at minimum waiting get picked up gets the signal to go to full and then stays there for hours.

MISO is not the only one playing the LMP game in the U.S., but they are the largest and did it best first (though not without some noteworthy hiccups worth another post another day). And while efficiency was vastly improved, it did not address any of the problems of meeting demand when things really go bad on the system, or how to manage the somewhat uncontrollable "green power" resources such as wind and solar. They still flex the system to work around those, and stress the system when a big contingency hits somewhere. MISO's LMP manages generation pretty well, but what it doesn't really do at all is manage the other side.... it doesn't manage load. Imagine how much nicer it would be if you could prevent donut riots on the other side of the hills by magically influencing how badly those people wanted donuts on any particular day? That is the foot in the door for the Smart Grid.

Another post, another day.

So..... did any of this make sense?

Don't Be Owned

Originally posted 9/12/2009, removed during a spate of paranoid content cleanup 5/26/2010, and now re-posted after some review and editing.

My company isn't unlike most electric utilities in North America, belt-tightening and cutting back everywhere in the 15 or so years following deregulation. Maintenance deferred, projects dropped, upgrades delayed, and my personal pet peeve... strict adherence to lowest bidder crap instead of paying a bit more for something of exponentially more value.

C'mon... in real life, if you have a choice between a truly great product at a certain cost, or a lousy crappy one for a little less, you put out a little extra and get the job done right the first time, else you pay for the savings later. Usually you pay more than you saved.

Some of us know that.

The company does not. Lowest bidder. Least cost. Period. And we continually pay for it through all available orifices as well as some new ones created for the situation. Overtime, emergency outages, repairs, lost time from dealing with crap. You get the idea.

That wasn't really the rant I meant to share when I started, though.

So, due to this lowest bidder - least cost - maintenance deferred frame of mind, One of my guys gets yet another routine alarm on a high voltage circuit breaker that is low on its arc-extinguishing internal gas (SF6).

He refers to the callout list. It's after-hours, you see. And these breakers can't be allowed to run out of SF6 gas while in service. They will either automatically trip before the gas gets too low, degrading the grid slightly, or they will "block closed", meaning if something happens that would normally trip this breaker, the trip signal will instead be sent to every adjacent breaker. It puts a big dent in your grid when you get a breaker fail operation like that.

So, (once again) due to this lowest bidder - least cost - maintenance deferred frame of mind, there are not very many guys assigned to this area any more. They're not being replaced as they retire or quit or move on. Three guys (including the designated on-call guy) don't answer calls to their home or cell numbers. The one guy that does answer is with his wife at the hospital. Bless his heart, he volunteers to go if we really need it. To hell with that, I'll see a substation burn violently to the ground on my watch to teach this company a lesson before I pull a guy from his sick wife's hospital bedside.

And let me not harp too much on the good old days, when we didn't have an "on-call" guy, we had an on-duty guy assigned exclusively to a trouble truck and waiting for action, fire department-style.

Grrrr.

So, anyway, I call the field supervisor for that area.

Field Supervisor: Hello?

Grumpy Dispatcher: Hey, this is Grumpy, Shift Supervisor at the ECC, We've got a critical low gas alarm at Outback Sub, and my dispatcher can't raise any of your area guys. You have anyone else available or know how we can reach your crews?

FS: Really, you couldn't get Tom or Dick?

GD: No answer on home or cell. we got Harry, but he's with his wife at the hospital and can't go.

I didn't reveal that Harry offered to go anyway, lest this guy glom onto that idea.

FS: Low gas, huh? Well, I don't have anyone around here, I'd have to call someone on OT from Far Away Service Center, that's three hours away. It'll probably be fine until morning.

An aside: I love that our phones are recorded.

GD: All right then, I understand you're assuming responsibility for this incident, and are choosing to defer work on the critical gas alarm until tomorrow. If anything worsens, can I call you?

Silence ensues.... field boss just remembered that our calls are recorded.

FS: Well. ........ You know ..... you know, let me make some calls and see if I can reach anyone.

Uh-huh. Go team.

GD: Sounds good, I'll wait to hear back from you when when you've assigned a crew.

Don't be owned.

If I had let this guy put off the work and something blew up or got someone killed, you know where responsibility lies? The Dispatcher is In Charge. And as the Dispatcher's Supervisor, I am In Charge-In Charge.

When stuff has to get done, make it clear where responsibility lies. I took that burden off and placed in unequivocally in his lap. He knew it, and he knew why. He didn't want to bear that liability. Guess what Captain Cavalier, I don't either. They pay you field supervisor pay. Go earn it. Find your guys, and get it fixed.

I'm sorry about the cutbacks and all, but no part of the cutbacks includes the requirement that I retain the liability for getting people killed or blowing stuff up simply because you couldn't/wouldn't do your job. You see, despite the clear transfer of the burden to the field boss, I would still be held at least partially accountable for not pushing harder and finding someone somehow to get it done. Frankly, spending 45 minutes on the phone and playing voice mail tag is not my job. We are running critical national infrastructure. When I want help, I want it right now. If I can't get it, I will make the boss of that area assume that burden. If he doesn't like that, maybe he'll ensure his guys aren't impossible to reach in the future, huh?

Captain Cavalier shouldn't tolerate being owned by me, either, He should be using the same approach to his bosses to put responsibility where it belongs: Sorry boss, I can't help the OT costs, because you won't hire help and we're behind on scheduled maintenance, so stuff is breaking all around our ears. If we fall behind, and especially if something big breaks because of it, we get fined big bucks.

Push responsibility where it belongs. And deal with it if it is truly yours.

Don't be owned.

Back in the Control Center

I have returned from a weeklong training session. I had a lot of fun, picked up a handful of CEH's, and met a lot of folks I often only hear on the phone or radio.

The world of power dispatchers is an exclusive one. There just aren't that many of us, you see. Last I heard, there are only about 3,000 NERC-certified dispatchers in all of North America, and something less than half of those are actually in dispatching. The rest are supervisory-types, power marketers, engineers, and other peripherally-involved utility employees who are not required to have it, but feel that the NERC cert helps them do their jobs better or looks good on a resume. Really, the NERC cert is all but useless even to us actual dispatchers, as very very little of what is learned in the certification process is ever used on the job anyway. The CEH stuff brings a little value, though, once you're in the club.

I have two NERC certificates (a rarity in the industry, since one of mine superseded the other anyway). Nonetheless, owning two NERC cert serial numbers does look pretty good on the ol' resume.

This small dispatcher world is amusing sometimes. One of the Reliability Coordinators (persons who oversee multiple dispatch centers) I was sitting next to works in a facility about 1,400 miles away from my control center. He knew a dispatcher I worked with at my last dispatch gig, which was over 1,000 miles farther away from his office. Not just 'knew' him, either, but knew his nickname and had played golf with him. This RC also works with another RC-in-training who I helped train as a dispatcher in my previous job years ago and miles away.

It gets better. This same RC also works with another guy at that RC center who happens to be the very guy who trained my Dad (the Smooth Substation Operator) when he joined his small municipal utility decades ago. Yeah, Dad retired a few years ago, so you know this dude is crusty! Still going, though. Amazing.

It gets better. This same RC knows, from his previous power company job in the midwest, a dispatcher with whom I served as a firefighter in my first fire agency many many years ago. From before either he or I became dispatchers ourselves. How cool is that? Here's two guys working a midsized midwestern fire department, who end up going separate ways, and through turns of fate both end up as dispatchers at companies far away from each other. Really, it should come as no surprise that we would end up knowing the same people. So here, after all these years, I get to reconnect with a firefighter friend who is now also a power dispatcher.

Our small power dispatcher world is a pretty neat club to be a part of.

Anyway, enough of my rambling. I see it is high time for another tutorial, and I've got some other things on my mind from my time off. I'm back now, so you'll be hearing more of my mindless drivel in the next few days.

Compliance Follies

After the 2003 blackout, the balance of authority shifted dramatically. For the past several decades, even through the deregulation disaster of the late 90's, the control areas and transmission providers existed in a sort of self-policing arrangement where the North American Reliability Council (NERC) maintained non-binding Operating Policies that the members sort of agreed to follow even though there was no real enforcement or punishment for noncompliance.

After the 2003 incident and the (non)revelation that not everyone treated the Policies as the gospel, the Federal Energy Regulatory Commission (FERC) decided to flex some federal muscle and lay down the law. FERC (government) deputized NERC (industry nanny) to stop being a helpless nanny and learn to be a mean, justice-delivering pit bull (apologies to pit bull lovers).

Sorry, this is getting boring, let me get back on track. We'll have to come back to this stuff in bits and pieces eventually, young padawans... there is much to learn.

So, FERC's new NERC dog chucked the vaguely all-pleasing Policies and replaced them with strictly-defined Standards. These Standards now include definitions for different degrees of noncompliance and the associated financial penalties that can be assessed for violations. Huge penalties. HUGE. In fact, Grumpy's company got whacked with an 8-figure NERC/FERC fine relatively recently.

Anyway, drilling in to one specific Policy.... there is a requirement that each control area be able to monitor the system frequency (normally 60.00Hz +/-0.20... what produces the hum you hear coming from power substations and lines). It isn't enough to monitor it in one place, however. Each dispatcher must be capable of seeing the system frequency as measured in multiple locations. This is vital in case your system is broken into pieces in a collapse so that, for example, you do not direct a power station to change its output to correct a problem in an area it is no longer connected to. In addition, the frequency monitors must be independent of the software application used to control the grid, so that in theory you can still see the different frequencies even while you are helpless to do anything because your Energy Management System (EMS) app is dead. It's nice to see the ground coming up to meet you through the window when the stick is dead. Yeah.

In essence, the Policy actually is a good idea. In practice as written, you get what happened here at Grumpy's office.... just another thing that makes him Grumpy.

First off, according to the guy tasked to set this up, the company could not find any digital displays capable of talking to our non-EMS frequency app smaller than the huge LED boxes with retina-burning redness aimed at our faces from just six feet away, now hanging on the wall a few feet from our desk. Hanging. By zip ties. Real professional looking. We would like to reprogram them to scroll game scores or something. They are that big. I cannot fathom that a company owning $billions$ in assets is unable to find a way to convert data in some magical way to be viewed on an appropriately-sized LED number display. OK, whatever.

Secondly, as it turns out, of the four frequencies displayed, the two on one side are measured from stations about three miles apart, while the two on the other side are perhaps ten miles apart. This will be very useful if the system is split neatly in half down the middle of these specific cities, so that each side is attached to isolated islands. The reality is, splits occur at major substations where one or a handful of major lines ties one region to another. Cities do not get split into independently-surviving halves in collapses.

Yip-de-flipping-yay. We're compliant. With ugly hanging oversized scoreboards of no value except the once-a-decade blackout, except that they are of no value unless we split this or that city in half, which happens once every never. But we're compliant!