Gathering Data to Guide a Refurbishment Project

Having a full understanding of how generating units operate is vital when repair and/or refurbishment work is planned. Pacific Gas and Electric Co. details its process to gather data on – and replace – the three generators at its 1,212-MW Helms pumped storage facility.

The first indications of cracking were revealed via wet fluorescent magnetic particle testing of the dovetails.

The first indications of cracking were revealed via wet fluorescent magnetic particle testing of the dovetails.

In August 2010, the manufacturer of our three biggest pumped-storage units, operating at our 1,212-MW Helms facility on the North Fork Kings River in California, informed us a similar unit operating at a station in Austria had suffered a catastrophic pole attachment failure. The manufacturer recommended we inspect our machines for cracking of the dovetail attachments after 25 years of operation averaging two to three starts per day.

We undertook a detailed analysis of the start/stop history of the units and assembled a team of industry experts in both generator mechanics and failure analysis to assess the risk of cracking in the re-entrant dovetail radius. We determined that there was a possibility of cracking in the area and that it would be prudent to contract the European experts in wet fluorescent magnetic particle testing (WFMPT) to inspect the most likely rotor to experience damage during a special outage.

These inspections, beginning in November 2011, revealed widespread cracking of the pole attachment dovetails, eventually on all three rotors. Once cracks were discovered, Pacific Gas and Electric Co.’s in-house non-destructive examination (NDE) group was brought in and found three times as many cracks as were initially documented.

What now?

Historical usage of the units was two or three start/stop cycles per day, but requirements to provide load support complementing renewable generation were driving that up to as many as five. The poles were attached via two 60-degree dovetails, symmetric on poles and rim. The “re-entrant radius” was too small by design and the resulting stress concentration caused stress in excess of yield at every start/stop cycle of the unit. This implied that cracking had begun early in the life of the unit and had progressed regularly ever since. We needed to determine the remaining life of the units and determine and make any possible repairs that could allow us run time to procure and install new rotors.

All cracked laminations were ground to undisturbed material and the crack areas hand-radiused to a larger radius and polished to prevent re-initiation. One rotor with cracking in excess of the allowed limit was “re-indexed” to shift heaviest load from the damaged dovetail to a third symmetrical dovetail previously used for inter-pole wedge attachment.


We began with a risk assessment of possible mechanisms and scenarios of dovetail failure while a unit was operating. We also assembled an expert team that helped us assess, redesign, and shepherd us through the whole process of analyzing and managing the existing rotors, as well as procuring three new rotors.

We quantified the depth and length of more than 200 cracks in each rotor rim. We made molds of the exceptional crack repair areas and used finite element analyses (FEA) to define the crack growth characteristics. We performed local and global FEAs to characterize the loading, thermal, local geometric effects, etc, of various relevant parameters of the rotors in real-time as the NDE and repair work proceeded.

During this process, an unrelated failure on Unit 1 (the least-run unit) was caused by a generator breaker actuator rod failure. The breaker failed to open but indicated open, initiating the trip sequence and forcing the pumping rotor to run parallel without field for about 12 minutes, raising temperatures as high as 800 degrees Fahrenheit. This caused no additional issues with the field pole attachments.

Several concepts stand out in terms of general rotor design and inherent issues. Cracking was concentrated at the top of the rim, where wedges were driven, and was more prevalent in the two 180-degree “start areas” for pole attachment. While “driven load” of the wedges was assumed to be about 10% of synchronous load, localized effects of the impact loading of the upper wedge area during driving were significantly higher.

“Beam stiffness” of the poles has a significant effect on cyclic loading of the dovetails in the area of the pole body clamping plate and especially the cooling gaps in the rim stack. Load sharing in these areas is significantly affected by how tightly the through/clamping studs hold the pole laminations. The moment response of the clamping plate under load due to the dead weight of the pole end turns was particularly sensitive.

Structural impact tests were used to track the stiffness of poles repaired and to assess damage to the pole structure from the heating event on Unit 1.

Crack propagation was determined to be consistently about 1 micron per stop/start, largely independent of overspeed. Occasional overspeed events could actually reduce overall crack propagation rates via hardening of the crack tip area.

This photo shows a cracked re-entrant (inside) radius in the pole attachment, cracked beyond the plastic limit of immediate failure and so removed for analysis.

This photo shows a cracked re-entrant (inside) radius in the pole attachment, cracked beyond the plastic limit of immediate failure and so removed for analysis.

Definition of failure

We generally define “failure” in terms of an undesired effect, such as yield or cracking. This structure was stressed to local yield at every stop/start yet ran for 25-plus years. These units showed no signs of distress with significant cracking at most of the pole attachments.

We revised our failure definition as “reaching the stress level of unstable crack growth.” There is a point at which a crack no longer grows incrementally but rather “unzips” as the remaining load-carrying “filament” is no longer sufficient to “stabilize” the crack tip to limit growth. After that, the filament quickly reaches the point of “plastic collapse,” where the pole migrates into the air gap and contacts the stator.

A single safety factor of two was applied to this assessment, in terms of both cycles and stress, which is not overly conservative in the world of fatigue failure estimation.

Life estimation

The crack growth rate of about 1 micron per stop/start could be multiplied by the anticipated number of stop/starts to give the expected crack growth over a given period. A three-year remaining life was a good conservative estimate via this method.

To mitigate risk, we needed to determine a viable monitoring strategy. The pole face deflection at the point where a crack would become unstable was about 0.10 inch, thus we’d like to be able to detect face movement of the pole of 0.010 inch. There are 20 poles, at a radius of about 12 feet, rotating at 360 rpm.

Shaft vibration is not sensitive enough to see 0.010-inch migration of the top 20% of a pole.

Even at a pole weight of more than 17,000 lbm, that shift is well under a reasonable detection threshold via shaft vibration on a million-pound rotor.

Direct proximity detection at the pole back, relative to the rim, would work but would require 20 channels mounted and wired in the rotor and a 20-channel telemetry system. Possible, but not attractive.

We chose to look at the pole faces via air gap monitoring (AGM). A VibroSystM AGM system looks at the rotor faces from the stator bore with accuracy of a few mils. AGM was developed to address rotor/bore interference but has evolved to be remarkably effective for a host of rotor and stator deflection and vibration issues. It’s a differential measurement, so stator motion affects the sensor side. On the rotor side, rigid motion is included, as is deflection of the rotor as a beam, motion of the pole on the rotor, etc. Highlighting small quasi-static local relative pole deflections was not the norm.

We installed four sets of three sensors at the top, middle and bottom of the poles to detect motion of the top of a pole relative to the center and bottom as an indication of migration due to crack growth.

To help differentiate pole migration and growth of the rim with centrifugal force, we installed VibroSystM capacitive proximity sensors radially inside the rim to quantify rim growth from the spider interface using a Lord Microstrain telemetry system. Similarly, we monitored shaft and stator vibration to refine assessments of local pole face motion.

The new rotor installed at Helms sits next to an old one that was removed from the same unit.

The new rotor installed at Helms sits next to an old one that was removed from the same unit.


Trend data over the first few months showed the poles were originally a bit “loose.” After a few runs they began to find a comfortable smaller normal range of motion but never really stopped varying. Even at full overspeed of 468 rpm, the air gap never went under 1,000 mils, but motion of individual pole faces went up considerably. The numbers all returned to their pre-load-rejection values very repeatably.

The core saturation and breaker trip tests showed that the poles moved quite a bit more when unusual events drove them out of their normal “groove.” The overall range of radial motion of the pole faces throughout operation was about 100 to 110 mils.

We set an absolute action limit for the system at 1,000 mils minimum air gap on any sensor. If any sensor dropped below that gap level, we’d re-assess the behavior of the machine. This limit was never reached.

This is the repaired area of a dovetail.

This is the repaired area of a dovetail.

The new units

Procurement of the new rotors began concurrently with repairs by February 2012. Initial design errors and operating experience were considered in a functional specification for the new rotors, including interchangeability with the original rotors, ability to remove a pole in-place, and addressing dynamic rotor issues with the original machine.

The three new rotors were manufactured by Alstom (now GE). Components for the rotors were made in Brazil, Canada and Sweden and then assembled on site. The units were replaced during normal maintenance outages.

The revised rotors are interchangeable with the original Westinghouse rotors despite significant revision of the generator frame to address stiffness issues related to critical speed problems. The primary issue with the original rotors was insufficient stiffness in the spider endplates that resulted in low stiffness of the overall rotor. Length of the upper/mid bearing span was a limiting constraint for rotor dynamics, although thicker plates significantly improved critical speed response despite greater overall rotor weight.

The pole design is unique in our experience in that the coils are “chevron” shaped to eliminate inter-pole wedges. The lugs are now “hammerhead” style and there are three rather than two. Poles are not tightly wedged but are held in position via small steep wedges.

The machines at Tennessee Valley Authority’s 1,652-MW Raccoon Mountain powerhouse were several months ahead of us in assembly and construction, so we learned a lot from their experiences to improve our processes on several fronts. Rim stacking is not a common experience for most of us, especially at this size. The experience of Barry Sargeant, formerly of Westinghouse and responsible for rectifying the original design of the machines, was invaluable during this effort. The new rim was uniform in the entire height rather than stacked in layers with vents between. Still, many of the issues seen during the original construction reappeared during the refit, primarily in height “crown compensation,” and again, the experience of the original team was invaluable. Initial “shrink” of the rim onto the rotor spider is a much more complicated and sensitive process than it appears to be beforehand.

Unit 2 was the first to get a newly designed and built rotor, in 2014. The new Unit 3 rotor was installed and put into service in the spring of 2015. Unit 1 received its new rotor in November 2015. The final unit was released to service in December 2015.

The units have seen some issues with brake ring attachments and insulation migration and some other detail-oriented issues. Critical speed response was significantly improved and balance weight installation simplified. Electromagnetics have all been as advertised.

The total cost of this project came in well under the initial estimate of $100 million.

Jim Stone is principal mechanical engineer with Pacific Gas and Electric Co.


Stone, Jim, “Dealing with the Data: Quickly Assessing New Data Under Pressure,” Proceedings of HydroVision International 2016, PennWell Corp., Tulsa, Okla., U.S., 2016.

Further reading

“Proactive Repair of the Helms Pumped-Storage Plant,” Hydro Review, Volume 32, No. 9, November 2013,

Previous articleAsia and Oceania Market Brief
Next articleNext Steps for Moving Energy Recovery Hydropower Forward

No posts to display