Preventing Problems through Past Experience

Tasked with preparing its 100-year-old Nine Mile plant to operate for another hundred years, owner Avista Corp. faced the challenge of not only determining the most cost-effective way to operate and maintain it, but also dealing with sediment that had built up in the project’s forebay.

By Brian Hughes, Kara Hurtig, Rodney Pickett and David Schwall

As hydroelectric developments reach a century or more of operation, companies wrestle with what to do with these aging plants over the next 100 years. Past designs and equipment fail to make the best use of limited water resources to generate power, and old equipment requires significant spending to maintain and operate.

Avista faced one such dilemma with its 26-MW Nine Mile project and had to ask itself if it would replace the facility, react when something failed, or something in between. Using a series of models, the company was able to best determine Nine Mile’s fate, while also making improvements to its operation by dealing with a long-standing sedimentation problem.

The Nine Mile project

Located about 15 miles outside of Spokane, Wash., Nine Mile went into operations in 1908 to provide power for the Spokane and Eastern Inland Railway and Power Company’s electric interurban system. Excess power was sold to the surrounding area. Avista, then known as Washington Water Power, purchased the plant in 1925.

The project includes a 220-foot-long overflow spillway adjacent to the east bank of the river and a four-unit powerhouse on the west bank. Nine Mile operates in run-of-river mode with a small reservoir that extends about 4 miles upstream, which generally ranges between 400 and 600 feet wide.

Two of the powerhouse’s turbine original twin, double and horizontal turbines had been replaced by the mid-1990s on Units 3 and 4 with quad and horizontal Francis turbines manufactured by American Hydro. Units 1 and 2 were still the twin double-horizontal Francis turbines from Holyoke Machine when they were removed from the plant in 2013.

Building up to a problem

Nine Mile’s reservoir has been infilling slowly with sediment since the project began operating. The problem was accelerated in the 1970s, when highway improvements along Hangman Creek, a tributary to the Spokane River upstream of Nine Mile, straightened the creek’s prior meanders, creating more of a scour channel, which causes bank erosion during high flows contributing to increased sediment loading into the Spokane River and the Nine Mile reservoir. Sediment in the forebay became increasingly problematic through the 1980s as heavy bedload coarse sand began eroding the turbines and damaging bearings. The extent of the sediment accumulation in the reservoir was not fully realized until the mid-1990s, when heavy runoff periods increased water flow through the reservoir from a normal 25,000 cubic feet per second (cfs) to about 40,000 cfs. Inspections revealed that increased sediment passage through the powerhouse was causing the bearings in the new turbines to fail and severely damaging the turbine runners.

Avista determined that unless it could find a way to substantially reduce sediment passage through the turbines, a continual cycle of bearing replacement and turbine maintenance would be required. This work had already cost Avista considerable plant downtime and money to repair and replace equipment damaged by the sediment, including several replacement turbine runners that had been in operation fewer than 20 years before suffering damage.

Based on a series of field investigations and model studies conducted in 1995 and 1996 by Northwest Hydraulic Components (nhc), the proposed solution involved installing a sediment bypass tunnel just upstream of the powerhouse intakes. The sediment bypass intake tower structure and tunnel into the powerhouse control valve was built by Robert B. Goebel, Inc. The downstream portion of the sediment bypass pipe and thrust block was completed by Washington Water Power crews and placed into operation in January 1998. Reports in the same year indicated that it was effective at diverting sediment before it reached the powerhouse intakes.

In 2010, Avista completed spillway upgrades that involved replacing the spillway flashboard configuration with an adjustable Obermeyer spillway crest gate, with construction services by Robert B. Goebel. However, while personnel were lowering the reservoir level to prepare for installation of the new gate, the sediment bypass tunnel became blocked with large debris and sediment.

Coming to a head

Attempts to clear the debris and sediment by Knight Construction & Supply Inc. and Associated Underwater Services were initially successful, but it was only a few months until the first rain on snow event that resulted in the tunnel once again getting blocked by large woody debris. As a result, the tunnel stopped operating completely in March 2011. Subsequent investigations by Avista engineers indicated that the lack of a properly designed trashrack and trashrack cleaning system at the bypass tunnel inlet contributed to the debris blockage.

The original tunnel inlet trashrack consisted of a vertical bar rack, with bars set at 4-inch center-to-center spacing. The initial use of this trashrack led to frequent clogging with smaller debris, such as hay and weeds that wou ldbe scoured from the bank during high flows. To remedy this situation, the original trashrack was replaced with a 1-foot by 2-foot coarse grid bar rack that would allow smaller debris to pass through the tunnel open. Cleaning of the bar rack was achieved by periodically lifting the rack above the tunnel inlet, with the tunnel closed, and allowing the debris to fall off the rack and be flushed through the bypass pipe.

The problem with this arrangement was that as the rack was lowered back into position, the debris that had fallen from the rack was trapped beneath the bar rack, and with time a heavily compacted pile of debris developed within the tunnel inlet, leading to its eventual blockage in 2010 that coincided with the spillway gate project drawdown.

Additional complications

Another problem at Nine Mile was the fact that its waterbox design was outdated and less efficient than more modern turbine designs that include a penstock and scroll case. The plant’s design is unique because it uses a waterbox surrounding the turbine and components that runs the unit shaft through a watertight bulkhead to the generator. This design creates unique issues for maintaining the plant and is complicated by a sewage treatment plant, a smaller river and a highly erosive creek upstream of the facility.

Based on the existing plant infrastructure, serious consideration was given to replacing the waterbox by retrofitting the plant with vertical tube-type Kaplan units. But this would require extensive concrete removal and placement down to the dam’s foundation. The vertical turbine retrofit would also limit turbine efficiency because of the existing buttress structure of the powerhouse.

The most efficient design for a 12-MW vertical Kaplan unit at Nine Mile would have required excavation for the draft tubes below the existing concrete rock interface at the lower portion of the foundation of the powerhouse section of the dam, which was not acceptable. To limit the draft tube excavation within the existing concrete rock interface required the turbine runner centerline to raised by 1.5 feet, thus reducing efficiency and more importantly reducing capacity down to 10.5MW.

Planning for the future

After extensive research and management review, Avista formulated four solutions to address the issues at Nine Mile, calling them the “Base,” “Partial Replacement,” “Full Replacement” and “Plant Replacement” options.

Details of each option included:

  • Base Case: Take Unit 1 out of service, run Unit 2 at 50% capacity, and simply maintain Units 3 and 4;
  • Partial Replacement Case: Replace Units 1 and 2 with new 8-MW Seagull units from Weir American Hydro;
  • Full Replacement Case: Replace all four existing units with 8-MW Seagull units; and
  • Plant Replacement Case: Demolish the existing plant and construct a new plant within the footprint of the former powerhouse with five 12-MW units.

Analyzing the options

Because one of the biggest issues with Nine Mile revolved around the cost to maintain it, Avista assumed a new plant would reduce operations and maintenance spending by correcting several of the maintenance issues. However, the company questioned whether the O&M savings would justify the work.

Avista drew on past experiences to develop an AvSim+ reliability block diagram (RBD) model with ARMS Reliability, providing a tool to build a detailed picture of the lifecycle costs for the plant. This type of model also provided detailed resource management requirements, impacts of different maintenance strategies, effects of maintenance and failures on the plant’s operations and capacity, and an optimized maintenance strategy for the selected options. The RBDs were refined using mechanical and electrical drawings of the facility, along with input from Avista engineers, managers and foremen.

The model covered a period of 75 years to support the financial analysis, providing the following predictions for each option.

Table 1 provides a maintenance cost overview and Table 2 displays a total cost overview. Meanwhile, Table 3 shows the mean availabilities and mean capacities for each of the configurations.

Making a decision

After analyzing the data, Avista determined that continued maintenance on the existing plant would be a bad solution given the previous failures of Units 2 and 4. Meanwhile, the Plant Replacement option had a break-even point more than 60 years in the future, making a complete reconstruction unfeasible.

Thus, the best option proved to be the Full Replacement Case, in which the existing structures were kept and each turbine was replaced with a new turbine and auxiliary equipment.

Sediment solutions

Although Avista had determined a plan for the powerhouse, the problem with the sediment remained as the proposed replacements of Unit 1 and 2 would increase the turbines’ individual hydraulic capacities from 1,300 cfs to 1,800 cfs. Avista retained nhc to assist in evaluating alternative concepts for sediment bypass facilities at the project, which included both two-dimensional numerical modelingand three-dimensional physical modeling.

They first used 2D hydrodynamic and morphodynamic modeling to predict long-term morphologic conditions in the reservoir upstream of the dam, extending about 1.5 miles upstream of the spillway. This numerical model predicted the amount of infilling over a 10-year period, thereby providing the change in bed elevation upstream of the spillway and in the main and secondary channels under the new operating condition scenario with the Obermeyer spillway gate and new powerhouse units.

The results indicated that on average bed elevation immediately upstream of the spillway and powerhouse would increase by about 15 feet, while areas farther upstream in the main and secondary channels were predicted to increase by 3 feet and 1.5 feet, respectively.

Nine Mile’s sediment bypass tunnel has been plagued with blockage issues, leading owner Avista Corp. to study methods of improving flows.

Meanwhile, a 1:30 scale mobile-bed physical hydraulic model was used to evaluate the performance of alternative sediment bypass facilities under the predicted future conditions. The physical model represented an about 1,600-foot-long reach of the Nine Mile reservoir and included the entrance details of the existing sediment bypass tunnel, powerhouse and spillway.

Physical modeling was divided into four phases — calibration and validation, baseline, design modification and tunnel discharge optimization — and included six alternative design modifications selected for testing:

  • Sediment bypass tunnel operating at 400 cfs;
  • Sediment bypass tunnel operating at flows greater than 400 cfs;
  • Modified Obermeyer gate operations;
  • Powerhouse extension wall and tunnel operating at 400 cfs;
  • Powerhouse extension wall and skimming wall; and
  • Powerhouse extension wall, skimming wall and submerged vanes.

Based on the results of the design modification testing and a preliminary assessment of the constructability alternatives, the preferred alternative for reducing the volume of sediment passing through the units was determined to be the restoration of the existing sediment bypass tunnel. Testing showed that as tunnel discharge increased to 800 cfs, there was a reduction from approximately 23% to 4 % of the sediment in the volume of sediment that passed through the model powerhouse units.

For that reason, additional testing was conducted to investigate whether further increases in the bypass tunnel discharge would generate additional reductions in the sediment volume entering the powerhouse units. Testing showed an overall reduction in sediment passing through the powerhouse units as the tunnel discharge was increased, with the most significant improvement observed at the upstream 3 and 4 units. The results also showed diminishing return for increasing the tunnel capacity past about 800 cfs.

Given these results, current design efforts are focused on evaluating options for increasing the capacity of the sediment bypass tunnel and developing a robust trash removal system for the tunnel entrance. The site requires a trash clearing system that can deal with large waterlogged woody debris and can accommodate the oblique direction of flow approaching the trashracks. An increased tunnel capacity will result in increased flow velocities, which may have an impact on the trashrack and trash clearing system, tunnel interior, tunnel control gate, and area where discharge exiting the tunnel re-enters the downstream river channel. In addition to a trash clearing system, potential modifications may include installing a large debris barrier upstream of the bypass tunnel, installing a flared intake to reduce entrance velocities, installing a tunnel liner or specialized coating to minimize abrasion damage to the tunnel interior, and replacing the tunnel control gate with a mechanism capable of withstanding the increased flow velocities.

The nhc study also recommended that measures for reducing sediment coming into the system should be employed to help address the longer term issue of sediment supply to the forebay such as watershed restoration of Hangman Creek and that field monitoring (using turbidity meters) be used to assess the effectiveness of the proposed improvements.

The path forward

With the Full Replacement Case being the option selected, the Nine Mile Rehabilitation Program was established as a multi-year capital investment program to further restore the historic facility.

A number of projects have already been completed this year under the program, including the construction of a warehouse, a barge landing, crane pad, transformer foundation and communication upgrades.

In progress are upgrades to Units 1 and 2, which are the largest of the remaining projects and includes overhauls of the last two original turbines from 1908 with larger more efficient 8-MW horizontal dual runner Seagull-type Francis turbines from American Hydro Corp. (AHC), responsible for the water–to–wire equipment package. The remaining equipment supplied by AHC includes Hyundai-Ideal generators, Alstom governors and controls, Siemens switchgear, and Basler static exciters.

Other projects include the renovation of the historic powerhouse building in 2017 and overhauls of Units 3 and 4, which is scheduled for 2018 to 2019.

Brian Hughes is a principal and Kara Hurtig is a water resources engineer for Northwest Hydraulic Consultants. Rodney Pickett is an asset management engineering manager for Avista Utilities. David Schwall is an engineer for Avista Utilities.

Previous articleFinancing Long-term Hydropower Requires Mitigating Risks Prior to ROI
Next articleHydroVision International: How Times Have Changed

Preventing Problems through Past Experience

With the potential for disaster at hydropower projects around the world, the lack of organized training programs becomes a prominent consideration in how companies manage their assets. Using case studies of preventable accidents, a model to help organizations better train and prepare their employees has been created.

By Enes Zulovic

The keys to professional risk management and asset management are the identification of potential operational risks and the development of a plan to eliminate or minimize them in a cost-effective manner. For hydroelectric facilities, the scope of asset management plans is primarily focused on the management of physical assets and asset systems.

However, human assets are critical to the successful delivery of optimized and sustainable asset management and require due consideration. Risks should be identified and managed as part of all asset management activities, considering the risks throughout the life cycle of hydroelectric power assets.

A large number of incidents that have occurred at hydropower plants are a result of a lack of operators and owners understanding the risks, which often results in low confidence from stakeholders and the general public. Gaining data from incidents to better understand and prevent them can be difficult, however, due to the sensitivity of the information, insurance, litigation and company image.

Presented here are a series of case studies detailing situations when operational and maintenance problems occurred and preemptive steps that could have been taken to avoid them.

Figure 1 — Pelton turbine at Poatina  A diagram showing the “push out” (left) and “cut in” positions with a vertical Pelton shaft turbine.

Case Study 1: Poatina plant Pelton turbine in frequency control mode
When a vertical-shaft Pelton turbine at Hydro Tasmania’s 300 MW Poatina plant operated in frequency control mode with the “push out” deflector arrangement (see Figure 1, below left), it caused water to discharge through the turbine’s air admission pipes into the underground machine hall. The facility is located in the Great Lake and South Esk catchment area in Tasmania.

The incident caused 250 liters of oil to be sucked from the turbine guide bearing oil pot into the river, where it developed into an environmental problem. In addition, the unit’s frequency control function could have been cancelled due to a shut down of the machine by its protection system, potentially causing a system blackout that would have cost the company millions of dollars.

The owner eventually determined that the development of negative air pressure around the Pelton runner and oil loss from the turbine guide bearing were causing the problem. The problem had been observed since the plant was commissioned in 1968 in situations when machines were being used for frequency control. It had been assumed over the years that this was due to an inadequate air admission system instead of the design of the deflector push out arrangement.

The design review during the modernization project failed to address the deflector design arrangement problem for times when the machine was operated in frequency control mode, making the situation a knowledge deficiency transfer problem.

The problem was solved by changing the deflector arrangement to use a “cut in” system, which solved the water discharge problem.

Water floods into the Poatina plant after a turbine operating in frequency control mode caused water to discharge through the turbine air admission pipes into the machine hall.

Case Study 2: Kaplan turbine load rejection and runner lifting and low pressure in draft tubes
A 30 MW vertical shaft Kaplan turbine at Hydro Tasmania’s 30 MW Paloona plant, located in Tasmania’s Mersey-Forth catchment, generates low pressure in the space between the guide vanes and the top of the runner under certain operating positions. This, coupled with relatively high tailwater levels and low rotating mass elements, results in an upward force on the runner large enough to lift the rotating element clear of the thrust pads.

During commissioning of the unit in 1972, the phosphor bronze reverse thrust ring above the turbine guide bearing failed when the machine was first synchronized, due to runner skating. The upthrust bearing was replaced with a new leaded-bronze design, which failed when the machine skated for an extended period as it was motoring on the system.

The conditions that led to the phenomenon at Paloona and the resulting consequences include:

— Runner skating at low guide vane openings: When operating below 7% of the guide vane opening, or around 2 MW generator output, a negative pressure develops above the runner and the shaft system lifts off the thrust bearing; and
— High upthrust during load rejections causes lifting of the shaft system at low and high loads. The water pressure above the runner briefly drops, causing the shaft system to lift up and drop back down onto the thrust pads when the pressure above the runner increases.

Pelton nozzles and the deflector “push out” arrangement at the Poatina project in their original arrangement.

The problem was solved by increasing the upthrust bearing’s clearance and material. White metal was used as a contact bearing surface to prevent the development of high metal temperatures due to friction caused by the steel mating surfaces.

A deficiency in knowledge transfer is again to blame for the problem, as air admission to the runner and draft tube was not considered during the unit’s design stage. The weight of the machine’s rotating parts was higher than axial hydraulic forces, and the closing law of the wicket gates did not develop negative pressure under the turbine runner blades.

Case Study 3: Catastrophic failure at Sayano-Shushenskaya
At the time of an August 2009 accident at RusHydro’s 6,400-MW Sayano-Shushenskaya, on the Yenisei River in Russia, nine of the plant’s 10 total units were available for operation and one was undergoing the final stage of a refurbishment program. An explosion within Unit No. 2 killed more than 70 people and caused the turbine room to flood, prompting a rehabilitation and reconstruction effort.

Official reports and several technical discussions following the incident have drawn very general conclusions, attributing the failure to heavy vibration that might have been unchecked as vibration monitoring equipment on the unit was out of service at the time of the failure; and poor maintenance, associated with failed studs in the turbine head cover of Unit 2.

Another hypothesis is that the explosion was caused by water column separation in the draft tube. This condition can readily be caused by a rapid wicket gate closure during unit load rejection. Unit 2 experienced a load rejection the morning the turbine exploded, followed immediately by a loud bang heard in the administration and control building adjacent to the powerhouse.

The load rejection and development of the draft tube reverse water hammer precipitated a massive failure involving the lifting of the runner, shaft, head cover, turbine and generator bearings into the umbrella generator rotor spider, destroying it. The full penstock head was then released into the turbine pit, resulting in an enormous geyser and massive destruction.

This hydraulic transient phenomenon was probably caused by turbine governors that had been sped up — likely unknowingly — to an unsafe level in an attempt to improve frequency stability under changing electrical loads.

This reflects a problem in many electricity trading markets in that the supply of “ancillary services” — including frequency management, spinning reserve and synchronous condenser operation — is determined on an arbitrary basis or depending on the prices bid in by various generators at various times. This means it is no longer possible to ensure that the machines that are best-suited to provide frequency control management are selected to do so.

Hydraulic design of the runner and closing law of guide vanes (wicket gates) in relation to the Kaplan runner blades is to be set up to prevent occurrence of low pressure under the turbine runner, development of the reverse water hammer and uplift of the rotating parts. If large cavities can be detected at an early stage, full column separation can be avoided. The lower limit for the draft tube pressure is set to -3 m (water column) WC for the test to be further analyzed.

Poatina’s Pelton turbine deflector is pictured here in its new “cut in” arrangement.

Air admission system to the runner and draft tube (natural and or forced air admission) was not considered during the design stage, and design review did not recognise or raise this issue.

Identifying common problems in the knowledge transfer model
These case studies provide some interesting insights into the importance of hydro knowledge transfer deficiency when considering risk identification and risk severity and selecting risk management solutions.

The retirement of experienced hydro personnel leaves young engineers with fewer opportunities to gain experience, while the transfer of knowledge to this new generation also appears to be insufficient.

This has caused inexperienced manufacturers and plant operators attempting to design and run plants to take shortcuts in the process, giving the appearance that hydro operators are dismissing future problems such as deterioration risk and expenditure requirements to obtain short-term gains.

Many accidents at hydroelectric power plants can also be explained by a gradual drift toward unsafe conditions — many as a result of economic pressure, cost-cutting in maintenance, company reorganizations, or delayed retrofits and modernization efforts.

The turbine room at RusHydro’s Sayano-Shushenskaya plant, following an August 2009 accident that killed more than 70 people.

A lack of understanding of equipment by both equipment producers and operators can also be to blame, as designers and manufacturers have not yet succeeded in eliminating the possibility that the pulsation generated by a runner can create resonance within the complete plant hydraulic system.

Likewise, those responsible for the maintenance and operation of rotating machinery should be aware that the catastrophic failure of a critical machine can cause serious injury or death, the total loss of the machine, or extended shutdowns of an entire plant.

Utility trading plans committed to an unsustainable level of output are also to blame as they do not allow for necessary maintenance activities to be adequately carried out.

For these reasons, operators must take a proactive stance in not waiting until a machine fails – instead, taking a proactive stance toward knowledge transfer.

Knowledge-based mistakes occur when someone is confronted with a situation that has not occurred before and which has not been anticipated. It leads to errors and suggests that the most effective solution is to improve the knowledge of the people who have to make decisions.

Conducting risk analyses is often primarily associated with technological aspects. However, a risk analysis should integrate knowledge and organizational human factors to properly assess risks.

Solutions to solving the knowledge transfer model
There are new technologies, tools and benefits that have been proven in practice to be of measurable benefit to other hydro organizations. Active research is required to identify, investigate, try and evaluate such opportunities.

To transfer knowledge among workers, I have developed a proposed model that includes:

— Development of essential learning modules for knowledge about hydropower plants;
— Employing specialist organizations and personnel for design review and other critical activities;
— Improving graduate engineering development programs with mentor and technical coaches;
— Forums and seminars by professional bodies and trade associations to be tailored with a specific focus on knowledge transfer;
— Greater training involvement by original equipment manufacturers (OEMs) in the hydro industry;
— Research and development activities, with results of academic research to be published and readily available; and
— An institutional culture of life-long learning and cultivating on-the-job experience.

Hydro Tasmania has been developing an engineering development program and training courses that cover some of the specific topics previously described as indicative of the knowledge transfer deficiency.  

Enes Zulovic is a specialist mechanical engineer over major works, assets and infrastructure for Hydro Tasmania.

More HRW Current Issue Articles
More HRW Archives Issue Articles