Lights Out: The Storm of the Century

Available Translations: es

Overview

The Need

Real time insight into the low voltage grid and accurate billing data after a CAT4/5 hurricane that caused complete devastation of Information Technology supporting systems,low voltage wiring, transformers, substations and supporting infrastructure.

The Solution

NES provided immediate on-site Information Technology system support and necessary technical services allowing the utility to focus on low voltage grid infrastructure recovery and restoration of customer services.

The Benefits

The NES solution offered insight and understanding into energized areas, and the ability to identify low voltage wire interruptions. The NES System provided accurate consumption data starting from the outage occurrence event to the recovery date without a need to estimate customer billing.

Solution Components

NES ANSI Smart Meters

MTR 5000 series ANSI smart grid sensor and revenue-grade residential customer multi-function smart meter based on OSGP and ANSI standard communications protocols.

NES Control Nodes

The NES Control Node (data concentrator) platform, which manages the low voltage grid devices, powers more grid sensors and smart meters than any other data concentrator platform in the world, collecting flexible data sets for Smart Grid applications.

NES System Software

NES's central data hub empowers smart grid applications. Powerful management, collection and integration software engines based in the utility's data center. Deploys, provisions, configures, audits, diagnoses, and retrieves data from low voltage grid sensors. Includes processing of grid events, collection of meter load profiles and power quality data.

Grid Operations Software

NES's browser-based user interface powered by System Software that provides management and analysis for millions of NES smart grid devices. Providing visual statistics, automated network functions, user rights management, auditing and logging, and time-saving diagnostics. (Formerly: Element Manager)

How Networked Energy Services (NES) innovation and technical support efforts during the worst blackout in U.S. history transitioned into humanitarian relief in a time of crisis.

PREPA is a vertically integrated utility that supplies power to 1.5 million customers in Puerto Rico along with the smaller islands of Vieques and Culebra. About half of the electricity comes from PREPA’s ten oil-fired power plants. The majority of the remaining portion of electricity is produced by a coal plant and a pair of natural-gas power plants. Renewables, which include seven solar farms, two wind farms, and seven hydropower sites, supply 2.4 percent of the generation. The electric grid includes 2,478 miles of transmission lines, 31,485 miles of distribution lines, and 334 substations across the service territory.

PREPA generates approximately two-thirds of the electricity and purchases the remaining amount from third parties. The pre-storm electric power generating capacity was 5,839 MW, which included 961 MW provided by two co-generators (EcoElectrica and AES-PR). The island's peak average demand is around 3,000 MW. Seventy percent (70%) of Puerto Rico’s power generation comes from the south side of the island, while seventy percent (70%) of power demand and usage is on the north side of the island.

The year 2017 produced 17 named storms; of which ten became hurricanes including six major hurricanes Category 3 or higher. Hurricane Irma struck Puerto Rico's northern coastline on September 6, 2017 as a Category 5 storm, knocking out power to more than one million residents and critical infrastructure.

In the following weeks, PREPA restored service for approximately 70% of the customers affected by Hurricane Irma, with the other 30% expecting to wait months for power to be restored.

Unfortunately, just two weeks later, on September 20, 2017, Hurricane Maria tore its way across the island as a Category 4 hurricane, bringing winds of 150-plus mph (280 kilometers per hour) and dumping 25 inches (635mm) of rain on Puerto Rico.

The storm took a diagonal path from the island’s southeast to its northwest, demolishing tens of thousands of homes, washing away roads and bridges, stripping the limbs from palm trees, and leaving in its wake a littered and decimated lifeless landscape. The force of the wind and rain toppled transmission towers, snapped concrete power poles, entangled low voltage lines in trees, and battered the island's power plants.

Hurricane Maria devastated the island, which was still in emergency response mode following Hurricane Irma. The combined impacts of the two hurricanes led to a complete failure of Puerto Rico's power grid, with little hope of a quick recovery.

Because of the extended and unprecedented damage, a significant portion of the generation, transmission, and distribution system had to be rebuilt, including high voltage transmission lines that often survive lower category hurricanes. The hurricanes decimated transmission and distribution lines across the island and caused widespread wind and flooding damage to substations, generation, and distribution facilities.

The storm damaged, destroyed, or otherwise compromised 80 percent of the island’s grid, according to the U.S. Army Corps of Engineers, which oversaw the reconstruction. Of 334 substations, nearly 40% suffered major damage. Surprisingly, a few of the power plants were not damaged; however, others were devastated. The largest generation facilities were operational within weeks of the storm but couldn’t export their energy into the grid until the low voltage grid and T&D network were repaired.

At the time of the storm, NES had a project deployment footprint of approximately 30,000 NES MTR5000 ANSI smart meters and more than 2,500 data concentrator control nodes deployed throughout the low voltage grid in various island cities.

The evening prior to the storm, NES Technical Account Manager, Roger Schreiner, was in-touch with PREPA's Utility and Information Technology team along with other colleagues who reside on the island. "I can remember the exact time the power went out" Schreiner recalls, "it was 2:02 AM EST, and I was on the phone with a good friend when he said ‘we've just lost power again’. I recall looking at my computer clock at that same moment. Just days before, this same friend, who resides in Guaynabo just outside the capital city of San Juan, had just had his power restored after Hurricane Irma.” So to him and many others, Hurricane Maria was a second major blow.”

“We continued to talk throughout the night via text messages, and at about 5:30AM, he had water coming in from the rain, damage to his front door and glass windows blown out from the wind. And the storm hadn't even made landfall yet. We lost total communication at 6:23 AM, which is about 8 minutes after the storm made landfall on the southeast coast. “

Over the next 72 hours, Roger watched the story of the devastation unfold on the island through the news media outlets. Unable to connect with PREPA employees or friends, "I felt helpless," Roger recalls. On the fourth day, finally he received word from Puerto Rico and learned of the devastation and severity of the situation first hand. Roger immediately took to mobilizing a fund raising campaign on GoFundme. He started ordering supplies to help: flashlights, D size batteries, AC/DC inverters, solar phone charges and solar light bulbs in large part due to the generosity of NES employees, his friends and family.

Trying to access and get the supplies to the island was just as much of a logistical challenge as the restoration process because Federal Emergency Management Agency (FEMA) had the airport tied up with aid relief, and there were very limited commercial flights in and out of Puerto Rico. Finally just ten days after the storm made landfall, with two huge suitcases of supplies, Roger landed in Puerto Rico.

On October 5, 2017, Roger had the first post storm meeting with PREPA employees, and it was clear that no disaster recovery plan could have been properly developed by the Utility for this much devastation all at once. The employees of PREPA had their hands tied with multiple system failures and of course all the extensive damage to the electric grid. It was apparent to anyone traveling around the island, who saw the downed and broken concrete poles, the non-existent traffic signals, and mass of low voltage wire piles on the ground, this was a crippling knock-out punch that the electrical grid and power system had suffered.

Communication on the island was very delicate, spotty, and still non-existent in many locations. There were voltage mismatches at certain delivery points, and as one example, a mismatch occurred at one of PREPA’s data centers during re-energization, which damaged equipment and systems.

Many places had been damaged by flooding and high water levels. Fresh drinking water, however, was sporadic and non-existent at times when the municipal water pump generators would run out of diesel fuel. Little to no bottled water was left in the grocery stores that managed to stay open by relying on local generators.

Over the next 16 day period, the NES and PREPA teams spent time recovering the NES Head End Server Systems to pre-storm capacity. “It was amazing working with the PREPA team,” Roger recalls, "Here are colleagues I had given solar light bulbs to light their homes, and the same people were charged with fixing this massive electrical grid outage and turning on the lights back on for the Island.”

“All of them doing so under the worst possible circumstances and yet most of these same people went home to sit in the dark and deal with no cellular service, no water, difficulty finding ice, locating fuel if they are lucky to have a generator, or finding propane for cooking. Things that seem trivial on a normal day, but now they were all dealing with their own personal life tragedy - just incredible human beings to witness and work with."

As a result of NES and PREPA teams’ collaborated efforts, the NES System outage was minimal and the NES AMI System was the first fully functioning remote AMI metering system to communicate on the island after the hurricane.

Over the subsequent months following the storm recovery, as the low voltage grid repairs commenced and lines were energized, the NES Power Line Communication (PLC) played a pivotal role enabling PREPA to have the immediate ability to read the AMI meters on demand and to gain power quality metrics for meters associated with the re-energized transformers.

The NES System enabled PREPA to understand where line faults were still present in the low voltage grid;directing man-power to those outage locations in order to restore customers’ service lost due to low voltage line and service line breaks.

The NES power quality metrics offered PREPA vision to see low voltage sags and swells and phase imbalances, as well as new sources of reverse energy (Net-Meters) due to the boom in solar power installations because people were struggling with no electric power for over two months and longer.

As the recovery process expanded and more meters were energized, PREPA experienced issues with their billing process because they were not able to read non-AMI meters due to the lack of employee manpower. It was “all hands on deck” at the utility in order to restore the power grid. So employees, who usually read non-AMI meters, were reassigned to other recovery related efforts, which of course took higher priority at the time.

In service areas where the NES AMI smart meters were previously deployed the NES System provided the ability for PREPA to accurately receive load profile customer data. This allowed PREPA to use real data and avoid having to estimate invoicing for electric service in the NES System
deployment areas.

NES Technical Manager, Roger Schreiner, made two trips to Puerto Rico, spending 60 days on the island after Hurricane Maria, and assisting PREPA with technical recovery on behalf of NES.

Roger, along with “two island friends”, organized a huge humanitarian relief effort across the island. The group of three managed to deliver seven pallets of water, 240 hot meals, 100 lbs. of rice, countless candles and batteries to remote areas of the island. They also provided the Puerto Rican residences from coast to coast with more than 3,300 solar light bulbs.

This lightbulbs proved to be pivotal for many families living in the dark, desolate mountain areas of Puerto Rico. Roger recalls one family in particular that had a young autistic child, who would bang his head every night due to being afraid of the dark. With the lack of batteries and fuel for lanterns, these areas would become pitch black over the night time hours. The following day, after receiving the solar lightbulbs, Roger got word from the parents that because of the solar lightbulbs, their child was able to handle the dark and enjoyed holding the lightbulb at night.

It was amazing and inspiring to see the spirit and devotion exhibited by both PREPA and NES employees. Hopefully Puerto Rico will never experience this type of storm and devastation again. However, it is satisfying to know that through the hard work of dedicated employees and the use of industry leading smart grid technology, NES was able to play a part in restoring the island’s electric grid.

Success Lights Out: The Storm of the Century