From Alarm Floods to ‘Highly Protected’ Status

Summary

The Solvay Novecare chemical plant in Pasadena, Texas, suffered from nuisance alarms, “critical” alarms that were not critical, and production delays due to alarms being missed. It needed a stronger alarm management program to create a safer and more productive plant. So Solvay engineers sought help to create and execute a program that would relieve the sensory stress on their operators, eliminate production delays, and improve overall plant safety. Other goals of the alarm management program transformation were to achieve the highest level of underwriting and lowest premium for industrial insurance, and to establish baseline alarm management practices that could be rolled out to other sites.

The Pasadena plant, which opened in November 2015, operates a large-scale, “on pipe” alkoxylation unit as part of a batch process. Alkoxylates are used as emulsifiers, detergents, and wetting agents and are the chemical foundation for a wide range of Solvay Novecare specialty surfactants. Situated in the integrated industrial campus of LyondellBasell’s Equistar Chemicals affiliate, the plant is supplied with its key raw material, ethylene oxide, via pipeline, which enhances the sustainability and surety of supply.

Alarm management problems were affecting operations directly and indirectly. Operators were overloaded with nuisance alarms, oftentimes filling three to four pages of the alarm summary with stale alarms. This caused the operators to ignore alarms, which reduced their situation awareness and made it hard to notice new alarms.

But alarm management is more than just getting the alarm rate below one to two alarms every 10 minutes. It is not just quantity that is important. It is also quality. If you focus on quality—ensuring each individual alarm meets the definition criteria—then the quantity issue takes care of itself.

For example, because of the flood of nuisance alarms, the alarm horn had been left disabled after commissioning. That meant no audible notification was generated for alarms or for batch recipe prompts. This lack of audible notification made it more challenging for the operators to stay on top of the process and to know when new alarms came in. Occasionally, this led to production interruptions when key utility equipment shut down and went unnoticed.

Another problem was that a large percentage of alarms were given the highest priority, “critical,” by default. This made it hard for operators to distinguish importance. It was also misleading, because many of the alarms were not critical. If everything is critical, then nothing is critical. This puts extra burden and stress on the operator. There was also no training for engineers about basic alarm management principles, so even they overlooked the lurking alarm problem in favor of troubleshooting the day’s production problems.

Partners in transformation

Solvay engineers knew they could use some partners to help them with such a substantial alarm management overhaul. They chose to work with Emerson Automation Solutions, exida, and Puffer-Sweiven. The product certification and knowledge company exida specializes in automation system safety, alarm management, cybersecurity, and availability. Its SILAlarm tool and alarm management professionals brought the Solvay team step by step through the alarm rationalization process. Puffer-Sweiven is an Emerson Impact Partner, which means it is a local point of contact for sales, service, and applied engineering for Emerson Automation Solutions in Central Texas and the Texas Gulf Coast.

Solvay contracted with exida to implement a holistic alarm management approach, starting with the Pasadena plant, that would be intentionally designed as a baseline for other Solvay Novecare sites. Applying the ISA-18.2 standard, which is a recognized and generally accepted good engineering practice (RAGAGEP) by OSHA, was key to developing a successful alarm management program and is the basis of exida’s recommendations.

ISA-18.2 defines a recommended workflow, called the alarm management life cycle, which Solvay embraced as its foundation for success. Most companies jump into alarm remediation (bad actor knockdown) without first establishing guidelines or setting goals. For this project, the company wanted to discuss existing alarms, as well as find missing ones. The life-cycle approach defined in ISA-18.2 is the right way to achieve that. A step-by-step process brought Solvay the results it needed from its alarm management overall.

Step 1: Benchmark performance

One of the initial steps was to benchmark the performance of the alarm system. Benchmarking identifies systematic issues and sets the baseline for judging performance improvement. ISA-18.2 defines recommended key performance indicators (KPIs), such as the average number of alarms per day, alarm priority distribution, and number of stale alarms. So, with the help of Emerson’s DeltaV Analyze, the Pasadena site compared actual to recommended performance.

Step 2: Develop an alarm philosophy document

With the benchmarking step complete, the team moved on to developing an alarm philosophy document (APD) as the foundation of its alarm management program. The process started with exida leading a one-day training session for the site alarm management team, a cross-functional representation from operations, engineering, and maintenance. With training, the team members aligned their scattered ideas with the actual RAGAGEP practices found in ISA-18.2. The training also reviewed the alarm management functionality available directly in the DeltaV control system and how it could be used to improve performance of operator response to alarms.

Following training, exida led a two-day workshop with the team to develop the APD. The exida APD template is comprehensive; it addresses the entire life cycle to ensure that the necessary work practices are in place to drive performance improvement and to sustain it over time. Getting started is the hardest part, and operations groups at every site wanted a specific starting point, so that they did not have to reinvent the wheel. The APD was written with the understanding that it could be tweaked for each site with minor effort.

Establishing a consistent and objective methodology for alarm prioritization was an important goal for the alarm philosophy. Solvay aligned its alarm priority matrix with its existing corporate risk management documents, including consideration for risks to people, environment, assets, quality, and productivity. This ensures a common approach to consequence evaluation is used for process hazard analysis and alarm rationalization.

The alarm philosophy document is truly the core of Solvay’s program. Developing it forced the company to think about purposeful alarm management. What alarms are actually significant? What is in the way of recognizing them now? How can we provide clear information for the very human operator? How do we resist the next urge to add yet another brighter light or louder horn to bother the operator about the latest perceived imperative? Without the structure of the APD, the company would just be playing “Whac-A-Mole” forever.

Step 3: Identify and rationalize alarms

Having a solid alarm philosophy in place, the team was ready for rationalization. Solvay selected the SILAlarm tool, a DeltaV Alliance Program product for alarm management by exida, to be its master alarm database (MADB). SILAlarm was seeded with Solvay-specific details defined in the APD, such as the priority matrix, the DeltaV color scheme, and the urgency matrix. All DeltaV–configured alarms were also imported directly into SILAlarm.

With initial coaching from exida, Solvay used the SILAlarm MADB-guided tools to rationalize and document each alarm. The purpose of rationalization is to ensure that all alarms presented to the operator are (a) actionable, (b) represent an unexpected situation, and (c) are prioritized to indicate their relative importance and urgency. Rationalization is a systematic, bottoms-up approach that ensures alarm management fundamentals are met; “bad actor knockdown” is a reactive process.

The Solvay team knew that following the APD and implementing alarm rationalization represented a paradigm shift for operations. To sow the seeds of success, the team made sure that operations was heavily involved in rationalization with participation from all shifts. Direct involvement in the improvement process increased enthusiasm amongst all plant personnel. Operators gained a new perspective on their role and a better appreciation for engineering. Operators and engineering are collaborators now. As a result, operators are more likely to raise concerns and potential issues to the alarm team. This also enabled transition out of “project” mode and into ongoing continuous improvement mode, which is the essence of the life-cycle approach.

An important part of the rationalization process is to document in the MADB the likely cause, consequence, corrective action, and allowable time to respond for each alarm. The senior operators had a great deal of insight, effectively transforming rationalization into a knowledge capture process. They identified redundant alarms, which alarms would come first in a sequence, and causal events and patterns. Junior operators grew from the experience (what does it mean when I get this alarm?) and thought about things differently the next time they were back at the human-machine interface console.

Rationalization takes effort and requires patience. The Solvay team rationalized approximately 9,000 alarms over a six-month timeframe. To prevent burnout, meetings were held twice monthly for no more than 5.5 hours at a time. As a parallel benefit, the rationalization exercise even uncovered maintenance issues such as transmitter problems, equipment problems and process problems.

Step 4: Leverage DeltaV alarm system

SILAlarm allows all the rationalization results in the MADB to be imported directly into DeltaV, automatically updating alarm configuration and DeltaV Alarm Help. This saved significant time and prevented errors from manual entry. Now, when a new alarm occurs, the operators can quickly call up Alarm Help to review the cause, consequence, corrective action, and time-to-respond information for that alarm as it was captured during rationalization.

One of the outcomes of the APD was defining how to best leverage the alarm management functionality of the control system to better support operations. The team tailored the alarm priority system to differentiate and segregate alarms from batch prompts (with audible annunciation and without), bypasses (interlocks, equipment modules, and phases), indication of simulated/forced variables, and reports (situations that do not require a timely response or require no response at all).

The team at Puffer Sweiven created a dedicated summary display for the different categories with operator access as appropriate from the banner, main toolbar, dashboard, and overview display (figure 1). These displays are a way to improve the operator’s situation awareness while also improving their ability to focus on alarms and batch status (now that non-alarms are no longer mixed in). The APD defined a consistent approach to audible annunciation (alarm horns) for the control room and in the field, which was implemented accordingly in DeltaV.

The Puffer Sweiven team also was an instrumental resource in implementing conditional alarms and other advanced logic to make the alarms most useful for the operators. Conditional alarming allows an alarm to be dynamically enabled or disabled based on a status or indication. For example, a low discharge pressure alarm on the outlet of a pump can reference the pump RUN status, in addition to the pressure value, when determining whether to annunciate. That means what was once a common nuisance alarm—the low discharge pressure alarm being triggered just by the pump being turned off—can be eliminated.

Results: Better operator understanding

When critical alarms are rare, they really get the operator’s attention when they do occur. ISA-18.2 recommends that the distribution of alarm priorities be about 5% high priority, about 15% medium priority, and about 80% low priority. By following the standard’s alarm priority matrix during rationalization, Solvay’s alarms were reduced and reprioritized based on importance (figure 2). Previously, a majority of alarms were default categorized as critical, making it impossible for operators to distinguish their true importance. Now operators have a clearer understanding of what is important.

Alarm rationalization successfully eliminated nuisance alarms and reduced alarm load by 84%. The three to four pages of standing alarms were reduced to only 10 to 12 alarms active at any one time. The alarm horn was turned on. Now if utility upsets occur, the operator quickly notices and can investigate or call for maintenance to resolve the issue before there is a production delay.

Operator involvement was key to the success of implementing SILAlarm. Operators got a better understanding of the process and the initiative and gave alarm-help suggestions relevant to the work they do. They see the difference alarm rationalization has made, pay more attention to the alarms they do get, and are more likely to make suggestions for future alarm improvement.

One of the most notable benefits of the alarm management program was achieving Highly Protected Risk (HPR) status from Solvay’s insurance underwriter. The HPR designation reduced the plant’s insurance premium and means that the Pasadena facility meets the highest industry standards for property protection and risk management. Only 6% of the insurer’s customers reach this status. The Pasadena plant was only the eighth Solvay plant in the world to reach that status.

Solvay made substantial improvements to Pasadena’s alarm program, relieving sensory stress for the Pasadena operators and allowing them to focus on the important information. There is still work to do, but that is the nature of a life-cycle approach. The support of exida’s alarm management professionals and SILAlarm tool, along with the alarm tools in DeltaV, have facilitated the company’s success. Going forward Solvay wants to multiply the accomplishments from Pasadena by applying them at other facilities.
 

More Information on the ISA 18.2 Standard

Sovay’s alarm reduction and reprioritization followed recommendations found in ISA-18.2, Management of Alarm Systems for the Process Industries. First published in 2009, this standard is managed by the ISA18, Instrument Signals and Alarms committee, chaired by Donald G. Dunn and Nicholas P. Sands. The committee “establishes terminology and practices for alarm systems, including the definition, design, installation, operation, maintenance and modification, and work processes recommended to effectively maintain an alarm system over time.”

Current ISA18 standards and technical reports include:

• ISA-TR18.2.1-2018, Alarm Philosophy
• ISA-TR18.2.2-2016, Alarm Identification and Rationalization
• ISA-TR18.2.3-2015, Basic Alarm Design
• ISA-TR18.2.4-2012, Enhanced and Advanced Alarm Methods
• ISA-TR18.2.5-2012, Alarm System Monitoring, Assessment, and Auditing
• ISA-TR18.2.6-2012, Alarm Systems for Batch and Discrete Processes
• ISA-TR18.2.7-2017, Alarm Management When Utilizing Packaged Systems
• ISA-18.1-1979 (R2004), Annunciator Sequences and Specifications

More information is available online.

This article was originally published in InTech's July/August issue.