Why Many Control Systems Fail

See discussions, stats, and author profiles for this publication at: https://www.researchgate.net/publication/225298475

Why many control systems fail ARTICLE in JOURNAL OF WATER AND ENVIRONMENT TECHNOLOGY · JUNE 2012 DOI: 10.2175/193864711802764779

CITATIONS

READS

2

181

2 AUTHORS: Leiv Rieger

Gustaf Olsson

inCTRL Solutions Inc.

Lund University

94 PUBLICATIONS 939 CITATIONS

131 PUBLICATIONS 1,437 CITATIONS

SEE PROFILE

SEE PROFILE

Available from: Leiv Rieger Retrieved on: 13 January 2016

42

WE&T l JUNE 2012 l WWW.WEF.ORG/MAGAZINE

Why many control systems fail

Few utilities consider how the ‘human factor’ influences treatment performance Leiv Rieger and Gustaf Olsson

W

hen implementing real-time process control systems at wastewater treatment plants (WWTPs), several technical problems may have to be addressed. Sensors have to be reliable. Controllers should be properly designed and tuned. Fault detection should be implemented and safety nets designed. Actuators must have adequate control authority to deal with changing influent quality and quantity. But in their 1998 Water Science Technology (Vol. 37, No. 12, pp. 397–401) article “Reviewing, assessing, and speculating,” G. Olsson and B. Newell argue that it is the “human factor” that is often neglected even though it creates more problems than the technology. Lack of incentives or misleading incentives for plant stakeholders create poor WWTP efficiency and suboptimal effluent quality. Greatly varying interests and demands of the stakeholders involved at all levels of urban water management have to be taken into consideration in a successful incentive structure. Without effective incentives, the sustained application of a control concept or any other optimization measure might be jeopardized.

The stakeholders One of a utility’s greatest challenges when aiming for highly efficient plant operation is to define what should be optimized and provide clear performance criteria for different stakeholders. The lack of clear criteria at different levels of responsibility is one of the key reasons many systems fail. To better understand the motivations of the involved stakeholders — which include the public, federal agencies, state or provincial agencies, local political leadership, plant managers, chief operators, and operators — their interests, incentives, and pressures must be examined. The public. Through election of federal and provincial governments and local politicians, the public is indirectly involved in decision-making regarding wastewater treatment. Another perhaps more direct link to treatment is through public pressure concerning water and wastewater fees. The public wants a clean environment with access to clean water, convenient disposal of wastewater, and low fees for these services. This leads to an apparent incentive to ask for a highly efficient wastewater disposal and treatment system. Federal agencies. The federal agency is responsible for implementing federal laws and regulations that protect water quality. It can offer guidelines and feasibility studies to water utilities. A federal agency may subsidize special projects or efficiency measures of national and international significance; however, the general shortfalls in federal budgets mean that the importance of federal subsidies is decreasing.

State or provincial agencies. In most countries, federal laws have to be substantiated by state or provincial laws or regulations. Often, these agencies have the duty of controlling the observance of wastewater regulations and technical standards. Because of their limited budgets, it is in the interest of these agencies to minimize their monitoring efforts. State, provincial, or local agencies define specific treatment goals for WWTPs. They are typically the ones that set up incentives that favor better plant operation. Local political leadership. Local political leadership consists mainly of local politicians and the plant commission. They are responsible for defining the objectives of individual WWTPs, hiring plant managers, and participating in the budget management of WWTPs. They receive pressure from state or provincial agencies through laws and regulations, from their citizens through elections, and from the industry through potential jobs and tax revenue for their regions. Their duty is to guarantee WWTPs’ compliance with laws and regulations, and to ensure that plants operate according to budget. They also want to prevent any possible bad publicity. But local leadership often is overstretched by the complex processes of WWTPs. They often cannot be expected to give a reasonable and well-balanced response to technical and environmental problems that can occur. A solution might be to ask for external advice. The increasing demand for cost-covering wastewater fees represents an incentive for the political leadership to insist on plant optimization in order to obtain a local cost-of-living advantage. Plant manager. The plant manager is responsible for personnel management, financial aspects, and compliance with laws. He or she has an interest in reducing the operating costs of the WWTP. A major task of the plant manager is to organize a safe and healthy workplace and to motivate his or her co-workers. Clear and reachable goals have to be defined and re-evaluated periodically. Improving operational performance is a continuous process, and the incentive structures should be set up in a way that innovation is not stopped after reaching a certain efficiency level. Incentives may be financial, such as salary, or acknowledgments through evaluation ratings and honoring staff with awards. Managers can support innovation by ■ establishing systems to request and reward innovation, ■ implementing employee suggestions, ■ participating in benchmark initiatives, and ■ conducting external or internal audits. Chief operator. The chief operator has to communicate the incentives defined by the plant manager and formulate specific tasks for the other operators in the form of clear quantitative measures. The goals are to operate the plant within effluent limits, ensure treatment efficiency, and keep the plant running.

WWW.WEF.ORG/MAGAZINE l JUNE 2012 l WE&T

43

The extra work performed by operators in maintaining and monitoring plant sensors, as well as testing the control systems and reacting to any errors, should be acknowledged. Operators. There is no way to achieve highly efficient operation without motivated operators. The key to high performance is communicating clearly about goals, effluent limits, operational challenges, etc. Operators should be motivated to develop their own ideas to improve plant performance. Operators often are trained as electricians, mechanics, construction workers, etc.; therefore, in-depth wastewater treatment training is required. Because the application of advanced control concepts increases demands on the knowledge of the operators, their training should include control concepts, sensors, and actuators. The danger of violations of the effluent limits as a result of poor control measures should be addressed.

Misleading incentive structures Missing or even wrong incentives can have a serious impact on the sustainability and efficiency of treatment processes and effluent quality. The following is a fictitious example of poor incentive structure but is nevertheless based on various existing structures. The example: Federal legislation asks for increased total nitrogen and phosphorus removal. However, provincial law only enforces annual limits for total phosphorus and daily or monthly limits for ammonia, total suspended solids, and biochemical oxygen demand. The chief operator at the WWTP is responsible for the effluent limit, while the plant manager is responsible for the budget. The chief operator is evaluated annually on whether he complied with the provincial effluent limits. Failing these criteria means he does not get a pay raise. The operators at the WWTP have been working for the past 2 decades with no significant changes in operation practice. The level of automation at the plant is low, and only a few on-line probes are used. The plant violates the effluent limits a few times a year. Naturally, if limits are exceeded, the plant is required to take measures. The official explanation was that special, unpredictable circumstances caused the violations, so the provincial agency agreed not to enforce violation measures. The plant manager hires a consultant to design and implement new control systems for the aeration and phosphorus precipitation. He briefly discusses this with the chief operator, but the other operators are neither involved in the decision nor in the design process. The results: In this example of a poor incentive structure, there is a strong incentive for the chief operator to not optimize the plant with respect to resource utilization. There is also a clear difference in the interest of the chief operator and plant manager. The chief operator’s main interest is to keep the effluent concentration well below limits, while the plant manager’s interest is to reduce cost and maintain overall plant performance. To make things worse, at some utilities, a savings in one year can result in budget cuts the following year, so even the incentives for a plant manager to optimize the plant are missing. Furthermore, since the effluent concentrations for ammonia and phosphorus are taken into account, but not total nitrogen concentration or energy and chemical consumption, the operator is overaerating and overdosing precipitant chemicals to prevent violation of effluent limits. If the implementation of the new control system is successful, the plant manager will earn praise for reduced costs. But the operators will have more work to do regarding sensor maintenance and increased demands from the new and more complex actuators. The reduction of safety

44

WE&T l JUNE 2012 l WWW.WEF.ORG/MAGAZINE

margins requires more process understanding and increases the risk for the operators. There is a danger they will get blamed for effluent violations even if overall performance increases. This “blame culture” is a common reason that WWTP operation is kept below possible best performance.

Successful incentive structures In contrast, there are several examples of successful incentive structures that are currently in use. These structures help optimize WWTP performance. They include the following: ■ Evaluation of monitoring data. Essential for highly efficient plant operation is the availability of accurate data. Although unreliable data can lead to major follow-up costs (due to data reconciliation or the wrong designs), budgetary constraints often are given as a reason for the limited number of measurements and for limited data analysis. ■ Employee suggestion schemes. These provide a great way to involve operators and make the best use of their in-depth knowledge of the plant. ■ Feedback from agencies. Detailed annual feedback to the plants on performance and data quality helps improve plant operation and typically leads to increased acceptance of the measurements. However, providing this feedback often is neglected, or only consists of a compliance report and does not include an in-depth evaluation of the plant. Most plants appreciate an efficiency assessment, including indications of general optimization potentials at the plant that can provide incentives for upgrading plant control and operation. ■ Benchmark initiatives. These are an excellent measure to detect optimization potentials by comparing WWTP performances. Examples from Austria, Switzerland, Germany, and Sweden showed great acceptability and even introduced interplant competitions for better performance. Involving operators is a great incentive and increases their knowledge. ■ Effluent-load tax. A major challenge for state and provincial agencies is to develop measures that give continuous incentives to WWTPs to improve performance. A good incentive structure should require the use of existing technology and motivate the use of technological innovations. One successful incentive is the introduction of effluentload taxes, which can be payment of taxes per kilogram of pollutant per day, as opposed to fixed effluent limits. The principle of an effluentload tax appeals to the personal responsibility of the plant operators and is not based on a cost-intensive policy of bans, regulations, and sanctions. Effluent-load taxes often are combined with fixed effluent limits to guarantee compliance with environmental needs. Without an effluent-load tax, implementing control at WWTPs typically could be financed only by the savings achieved by reducing energy, chemical consumption, or disposal of sludge. If an effluent-load tax is imposed, the improvements to the environment are evaluated on a monetary basis and, therefore, are included in the cost calculations. ■ Carbon shares. The carbon share market enables WWTPs to sell carbon shares after increasing their efficiency. This means moving away from a policy of regulations to a market-driven incentive structure. It internalizes external pollution costs and provides continuous incentives to develop and implement more-efficient technologies. Additionally, using carbon as the main and single unit instead of concentrations, currency, or kilowatt-hours, hopefully should lead to a more holistic approach of evaluating wastewater treatment performance. There are a few examples of specific successful incentive structures. The Danish Action Plans for discharges from wastewater treatment plants provide excellent examples of how to set up the right incentives

to promote ecoefficient technologies. They are the concerted actions of national legislation, regional water boards, research institutes, engineers, and consultants. Another focus of the action plans is the education of operators to guarantee the more efficient and complex technical systems. Another example is a WWTP in Strass, Austria. In a 2009 report by the Water Environment Research Foundation (Alexandria, Va.), “Best Practices for Sustainable Water Treatment: Initial Case Study Incorporating European Experience and Evaluation Tool Concept,” (Report No. OWSO4R07a) the foundation analyzed the success of optimization efforts at the Strass WWTP. The main factors for the successful implementation were ■ a highly educated, well-paid workforce that was motivated, trained, and experienced; ■ a high level of automation that enabled the creation of smaller, specialized teams; ■ the use of advanced process analysis tools; ■ the tolerance of process risk and in-depth understanding of the processes deployed, including the use of novel treatment processes; and ■ the ability to quantify gains.

Creating sustainable process control systems Control and automation systems often are described as having three major objectives of varying importance: 1. keeping the plant running, 2. satisfying effluent requirements, and 3. maximizing efficiency. Nowadays, at most plants levels, flow rates, pressures, and temperatures are controlled automatically via pumps, compressors, and valves. At this level of automation, vendors usually have sufficient knowledge to create a successful control system. On the next level of automation, we find automatic control of dissolved oxygen, sludge age, return sludge, nutrients, etc. These are proven methodologies, but already, at the second level, many control systems fail. The reason is not that the systems are difficult or that the sensors are not sufficiently robust; rather, many implementers of these systems have insufficient knowledge of the process dynamics. Sensors may be located at wrong positions. Data analysis may be inadequate. Sampling may be too infrequent, or controller settings may be inadequate. To maximize efficiency means that unit processes should not be operated in isolation. A plantwide control strategy should be considered. For example, the sludge production in the liquid train of an activated sludge system should be related to the desired sludge characteristics in the anaerobic digester to produce biogas. Dedicated maintenance of measuring devices is essential and should be the responsibility of designated operators. Regular training on lab practices and instrument handling might improve awareness and, consequently, data quality. Data quality can become part of an incentive structure if data quality metrics are established, such as through mass balancing or comparative measurements. Outsourcing measurements could be a better solution at smaller WWTPs with a lack of trained personnel. It is important for a WWTP to translate data into usable information. New sensor development, particularly in nutrient removal systems, will provide even more data for WWTPs. Modern computer software is required to extract patterns in the enormous amounts of data measured. Process knowledge typically is built from the experience of operators and engineers, but this knowledge often leaves when they

leave. If process knowledge can be encapsulated in software, not only is it retained, but the computer can assist in the decision-making process in the plant operation. To ensure a sustained and adaptable application of the control loops, it is essential to integrate the operators from the outset and train them in the basics of process control. Problems arise if these systems are given low priority by the plant staff due to inadequate incentives, the overwhelming pressure of daily business, or lack of understanding of the processes involved. Larger plants often are able to maintain in-house experience and the knowledge to maintain control systems, but for smaller plants, regular external advice might be advantageous. A clear visualization of control actions in the supervisory control and data acquisition system is necessary to provide required information to the operators in charge. A hierarchical structure of controller parameters with different access levels will help daily operation and leaves critical parameters to the experts. The risks of a control strategy also should be clearly communicated to operators. Prevention or mitigation measures also must be discussed. Optimizing a plant comes with the inherent danger of increasing the risk of violating effluent quality permits as safety margins are reduced. For example, introducing ammonia control reduces the aeration intensity or time, consequently increasing the risk of violating the ammonia limit if part of the system fails or the disturbance is greater than anticipated. The responsibility for an increasing risk for effluent violations should be considered. Dynamic models can be excellent tools to train operators for these situations.

Setting a clear objective A well-functioning control system has to include not only various measuring devices to gather adequate variable information. It also has to include a monitoring system that can acquire data, detect and isolate abnormal situations, and assist with diagnosis and advice, as well as a control system to meet the goals of the operation. There has to be a qualified team of people who feel a deep sense of ownership of the system and the WWTP, and who are committed to its continuous improvement. It is important that all employees increase their competences through continual education. There must be a stringent plan in place to define the roles and responsibilities of all stakeholders involved. Regulatory requirements — which must include national and regional regulation — are crucial to give the various stakeholders a clear goal for the operation. One successful system is the effluent tax, as it gives continuous incentives for improvement. To work out the feasibility of stipulated targets and how targets should be met, regulatory agencies should involve research institutions. If the targets are formulated clearly, it forces WWTPs to find innovative ways to meet those targets. On a plant level, a clear incentive structure should be established that sets specific goals for treatment performance and resources spent. Goal-setting, together with detailed analysis and feedback from each operator, helps involve everyone in the overall goal, which is to get the best out of a treatment plant. Additional measures, such as benchmarking, reward systems, or audits, can help in setting improvement targets and further optimize operations. Leiv Rieger is a process modeling expert at EnviroSim Associates Ltd. (Hamilton, Ontario), and Gustaf Olsson is a professor of industrial engineering at Lund (Sweden) University.

WWW.WEF.ORG/MAGAZINE l JUNE 2012 l WE&T

45