The “Flawless Game”: The Candlestick Park Incident

I chose this title after reading an article in the San Francisco Chronicle:  “PG&E: New power line will pass Candlestick test“.  The article points out some of the issues that many utilities are facing with their smart grid initiatives.  According to the article, PG&E spent about $980,000 to ensure that the upcoming playoff game at Candlestick Park would not embarrass them again.  After I read the article, the choice of words used by the “company representative”, quoted in the article this should be a “Flawless Game”.  It got me thinking.

As an overview, let’s look at the resources and actions PG&E put forth to ensure that this would be a flawless game.  This effort, according to the article, took about 60 workers, the replacement of 8,700 feet of 3-phase line and moving 750 homes to a different power line circuit.  My guess is that the outages actually cost PG&E well over $1 million and I arrive at that number thinking they probably did not include the emergency crews and expenses from the actual outages.  Either number you use shows PG&E was willing to put these funds and efforts on the line to insure a “Flawless Game”.

The interesting thing about the two outages is they were caused by issues on the PG&E side as well as issues on the customer’s side, the city of San Francisco’s Recreation and Park Department.

On the PG&E side, the use of “bump sleeves” is a very common industry practice and used throughout the electric line industry.  When a line breaks, having the luxury of replacing the whole line as they did here is not the typical option.  Wire breaks are routinely repaired by applying these “bump sleeves” and the crews move on.  After further investigation of this line, PG&E found three other “bump sleeves” on the lines that were “warmer” than normal.  From my past experience, this situation shows that the line was probably old and in dire need of replacement, which they did after the fact.  This very public issue along with the 2010 explosion of the natural gas pipeline in San Bruno, to me, indicates that PG&E has lowered the risk threshold it’s willing to take concerning maintenance and safety.  I make this observation because these very public incidents occurred on both sides of the utility’s operational arms, the electric and natural gas side.

On the city of San Francisco’s Recreation and Park Department side, poor maintenance of key equipment along with poor strategic decisions compounded the embarrassment of the Monday Night fiasco.  Failures will happen and a good routine maintenance plan for the generator switches could have caught the failure before it happened.  Most backup generators are exercised weekly to verify proper operation.  Only the maintenance records at the park department could confirm if a routine maintenance plan was in place.  Their strategic decision failure came with the use of lights which have a well-know property of long cool down and start up delays.  For a national stadium like Candlestick Park, you would think that someone should have pointed this out some time ago and they should have looked into having some kind of battery backup system to hold the lights on during short outages to prevent exactly what happened during the nationally televised game.  My guess is that when it was discussed, the park department more than likely due to funding cost chose to lower their risk threshold and decided against a backup for the lighting system.

Now back to the title, “The Flawless Game”.  I find it ironic that PG&E is facing all of this public humiliation at a time when it wanted to be recognized as a leader in smart grid implementation.  But again, there seems to be something in the statement made by Geisha Williams, PG&E’s executive vice president in charge of electric operations, “My team has really stepped up and made some incredible things happen, and over the holidays even.”  This was quite an effort and expense put forth by PG&E for a one-time football game while festering issues continue to eat at the real game, PG&E’s growing customer dissatisfaction.   The “Flawless Game” use to be played for all its customers, but due to poor decisions, the game seems to be quite flawed and very public.  Past cost cutting decisions are now being questioned and maintenance and safety practices seem to have been put at a lower risk threshold within PG&E and the effects of these decisions are showing up at a time when they wanted to be hailed as an industry leader.  I believe this is yet another look at the utility industry’s problem of implementing the smart grid and getting the public to buy in.  Increased outages and flawed service are high in the minds of the utility customers and is helping to escalate the pushback on smart grid implementations and raising questions as to who should pays for it.  I’m not sure yet, but my bet is that shareholders picked up the cost for the line upgrade to Candlestick Park.  It might get buried into other maintenance cost at PG&E since $1 million is small potatoes in their overall budget.  But, it would not surprise me to see someone in the public sector checking into the records to see who actually picked up the cost for the repairs at Candlestick Park.

Even more ironic, PG&E has not used the Candlestick Park incident to help further its need for a smarter grid.  As I stated earlier, “bump sleeves” are in common use throughout the electric utility industry and probably exist in numbers of at least tens of thousands if not hundreds of thousands on the PG&E system alone.  There were four that we know of, on the 8,700 feet on the old Candlestick Park line.  The PG&E website says they have 141,215 circuit miles of electric distribution lines which calculates to well over 745 million feet of electric distribution lines.  Could the smart grid have predicted or even better yet prevented the Candlestick outage, I believe yes.  These “bump sleeves” are a common electric service disruption point which usually has a predictable failure sequence before they fail.  Smart equipment along with smart analysis would be able to detect and predict many such failures if the equipment is deployed and implemented correctly.

The movement of 750 homes from the feeder circuit for Candlestick Park was a loading situation which should have been detected much sooner.  Yet, most utility engineering departments are undermanned and most likely technology starved and have not yet benefited from the smart grid efforts.  The movement of this many homes from the circuit says that the line was over loaded or close to it.  This issue seems to have not been caught until after the fact and brought to light after the two outages were analyzed.  Good data from smart equipment and smart analysis would have alerted PG&E to the fact of the overload circuit as well as detecting the increasing momentary outages on the circuit caused by the progressive failure of the “bump sleeve”.

Another issue was a customer service failure where PG&E could have, prior to the incident, worked with the park department’s team and offered recommendations and help with the lighting system.  If you have a good working relationship with customers then these opportunities should have been there when the lighting system was upgraded.  PG&E could have offered recommendations for a battery backup system which they have had deployed in other places on their system for years.  But, that effort would require having two-way conversations with the customer.  May be it was not seen as technology sexy as electric vehicle charging stations to the marketing departments.  Yet, it definitely got national attention on that Monday Night of the incident.

The big missed opportunity was to go on national TV and have a PG&E spokesperson step to the microphones of the frustrated sports announcers and make statements about the benefit of deploying the smart grid and its need in helping to detect and prevent events like what had just happened.  That requires a consistent message backed by good business metrics and employees trained to speak with one voice in order for this strategy to be carried out successfully.  This, as we know, is one of the main areas of failure by utilities in their efforts at deploying the smart grid.  This just shows as with everything, if you’re not ready, “Opportunity Passes”.

My intent here is not to point fingers and lay blame, but instead to use this opportunity to encourage new thinking.  Opportunities happen each and every day, but without strategic thinking or proper analysis you will continue to be stuck behind the eight ball on smart grid.