When Safety Management Systems Fail

As aircraft accidents go, the crash of a Buffalo Airways Douglas DC-3C on Aug. 19, 2013, could have been a lot worse.

No one was injured in the accident, which was triggered by a right engine fire shortly after takeoff from Yellowknife Airport, N.W.T. The flight crew performed an emergency engine shutdown — a procedure that included feathering the right propeller (rotating the blades parallel to the airflow to reduce drag). Due to a pump failure, however, the propeller failed to feather properly, making the disabled aircraft that much more difficult to control.

As the crew made a low-altitude right turn back toward the airport, the plane clipped the tops of some 30-foot trees, then skidded onto the ground beyond them with its landing gear retracted. All 21 passengers were safely evacuated, followed by the captain, first officer, and flight attendant. In a CBC News report immediately following the crash, Buffalo Airways owner Joe McBryan — known to viewers of the TV series Ice Pilots NWT as “Buffalo Joe” — praised the crew for doing “a marvelous job under the circumstances. Mechanical problems do happen and it’s how you react to it. I think they reacted very well to it.”

The Transportation Safety Board (TSB) of Canada took a somewhat different perspective. In its report on the accident, released in April of this year, the TSB pointed out that the aircraft likely would have been more controllable if it hadn’t been an estimated 1,235 pounds (560 kilograms) overweight. The captain and first officer didn’t receive a cargo manifest prior to departure; neither did they complete weight-and-balance and performance calculations as required by regulations. According to the TSB report, this wasn’t unusual: “Investigation found that it was common to operate in this manner, and that weight and balance forms were normally completed en route without the benefit of accurate information.” Even if the crew had attempted to complete accurate calculations, it’s not clear that they would have succeeded, as the aircraft’s own weight and balance hadn’t been updated since 1990.

The TSB had harsh words for Buffalo Airways and its senior management, describing the company as an organization that met the basic requirements of regulations “only when pushed by the regulator.” But its harshest words were for the regulator, Transport Canada, which arguably stopped “pushing” Buffalo Airways when the agency transitioned from inspection-based oversight to an emphasis on safety management systems (SMS). Safety management systems evolved in response to the observation that strict regulatory compliance isn’t always enough to prevent accidents; when properly implemented, they can give companies valuable tools for proactively identifying hazards and managing risks. However, they can also fail — and in ways that aren’t always obvious on paper.

In the three years leading up to the accident, all of Transport Canada’s surveillance activities for Buffalo Airways were focused on various required elements of its SMS. The regulator never identified unsafe operating practices related to weight and balance and performance calculations, even though such practices had apparently become routine and were condoned by management. According to the TSB, that “current approach to regulatory oversight, which focuses on an operator’s SMS processes almost to the exclusion of verifying compliance with the regulations, is at risk of failing to address unsafe practices and conditions.” The board warned that the risk of accidents could increase if Transport Canada does not adopt a more “balanced approach” that shifts its focus back toward inspections for compliance.

Canada was the first country in the world to mandate safety management systems for commercial airlines, so it’s not surprising that Transport Canada has faced an especially steep learning curve. But while the regulator’s failings have brought SMS under critical scrutiny by the Canadian public, they’ve done remarkably little to slow the concept’s momentum within industry. If you’ve attended an aviation safety conference or read an aviation magazine in the past five years, you’ve been told you need a safety management system. Many contracts now require them, and an entire consulting industry has sprung up to help you implement them — even though there are few studies to indicate that they lead to lower accident rates.

That’s not to say that safety management systems don’t have benefits; plenty of testimonials indicate that they do. In our push to endorse and adopt them, however, we may have lost sight of their limitations. Not all safety management systems are effective. The worst of them are little more than corporate whitewash. Moreover, it can be difficult for outsiders — even the regulators and auditors charged with evaluating them — to distinguish between the ones that work and the ones that don’t.

Unless you’ve devoted considerable time and effort to learning about safety management systems, you may have only a vague idea of what they’re about. To line pilots in the field, “SMS” often represents a manual on the shelf; pre-flight risk assessments that may or may not be useful; or a safety reporting hotline that may or may not yield results. To a grudging senior executive, “SMS” might only mean a burdensome regulatory or contract requirement. Only in companies where SMS is particularly well implemented do managers and employees recognize it as greater than the sum of its parts — as a unified, proactive approach to safety that engages the entire organization in stopping accidents before they start.

“SMS is about helping companies find trouble before trouble finds them,” said TSB chair Kathy Fox, who is very much in favor of safety management systems, despite her criticism of Transport Canada’s oversight regime. In her own research, Fox has found that formal, structured approaches to searching for and documenting hazards — a cornerstone of SMS — can help managers recognize early warning signs of hazardous situations that might otherwise be overlooked. A properly implemented SMS will amplify such “weak signals,” encouraging managers to take corrective action before an accident occurs. “It’s about having formal documented processes for identifying hazards and managing risks,” Fox explained. “Those that have embraced SMS do really talk about the benefits.”

SMS isn’t unique to aviation; according to Fox, the concept originated in the chemical industry in the 1980s and has since been adopted in other safety-critical industries around the world. In Canada, SMS has been required in the federally regulated rail sector, and for Canadian marine vessels engaged in international voyages, since 2001. It has been progressively implemented in Canada’s commercial aviation sector since 2005. Under pressure from the International Civil Aviation Organization (ICAO), more countries are now following suit — for example, the United States Federal Aviation Administration (FAA) mandated SMS for Federal Aviation Regulations part 121 air carriers in January of this year.

Although there are some variations between safety management systems, most have common elements. The FAA defines SMS as a “decision-making system” with four key components: safety policy, safety risk management, safety assurance, and safety promotion. The first component, “safety policy,” involves establishing policies and procedures that clearly describe safety objectives, responsibilities, and expectations. According to the FAA, this is where management establishes its commitment to the safety performance of the organization, and to continuous safety improvement.

The next component, “safety risk management,” provides a decision-making process for identifying hazards (conditions that could foreseeably contribute to an accident) and assessing risk (the potential severity and likelihood of events resulting from those hazards). Based on that assessment, management will decide whether to tolerate risk or introduce risk controls — often in the form of new processes or equipment — that reduce risk to an acceptable level. Meanwhile, “safety assurance” involves monitoring and measuring safety performance to ensure that the system is working as it should. Data for safety assurance is obtained through normal operational monitoring, periodic auditing, investigation of accidents and incidents, and employee reporting (hence the existence of “safety hotlines” or online reporting tools in many companies with SMS).

Finally, there’s “safety promotion” — making sure that employees understand the company’s safety policy, their own safety responsibilities, reporting procedures, and risk controls. Safety promotion is accomplished through training and through regular communication, which might take the form of meetings, emails, or safety newsletters.

While a safety management system doesn’t have to be complicated, there’s a lot to be said on the topic: the FAA’s Advisory Circular on SMS (No. 120-92B) stretches to 80 pages. And you might assume that the sophisticated theory behind SMS — which has evolved over decades, is enshrined in international regulation, and has been the topic of countless books and articles — would be grounded in a body of objective, methodologically rigorous scientific evidence. You would be wrong. Although there are many studies testifying to the subjective benefits of SMS, practically none have linked SMS to quantifiable reductions in accident rates.

A 2012 report prepared for the Australian Transport Safety Bureau by Dr. Matthew J.W. Thomas of Westwood-Thomas Associates highlighted this lack of evidence. According to Thomas, a review of the relevant literature “found a lack of clarity in what might actually be driving safety performance,” especially in low-probability/high-consequence industries like aviation (that is, industries in which adverse events are unlikely, but extremely significant when they do occur). Much of this is due to the inherent difficulty of measuring events that don’t happen very often — and the fact that few researchers appear willing to tackle the challenge. Thomas found only one study of SMS that qualified as a randomized controlled trial, which is generally considered the “gold standard” for impact evaluation. “Overall, the scientific literature provides relatively low quality of evidence supporting SMS interventions,” he wrote, noting that most of the studies he analyzed were “methodologically compromised by utilizing self-assessment of SMS implementation or safety-related effort.”

And those were just the studies he deemed worthy of evaluation. “Of the studies excluded from analysis, there were a proliferation of ‘frameworks,’ ‘models,’ and ‘strategies’ that were argued for with respect to their potential contribution to safety performance through inclusion within an organization’s safety management system,” Thomas wrote. “These frameworks often lacked theoretical strength and it could be argued that most simply reflected an author’s common sense rather than any scientific rationale.” Such personal opinions — in lieu of scientific evidence — appear to have driven much of the development of SMS across a range of industry contexts.

Is there something to be said for this so-called common sense? Indeed there is. Companies that have successfully implemented SMS often provide compelling case studies of its benefits, even if they can’t necessarily prove what accidents SMS prevented. In the helicopter industry, one such company is Metro Aviation, the air medical operator and completions center based in Shreveport, La. Metro was an early participant in the FAA’s SMS Pilot Project, and in January 2014 transitioned to the highest level of the program (Level IV, “Continuous Improvement”) for its part 135 flight operations. A few months later, it also achieved Level IV status for its part 145 repair station, making it the first U.S. part 145 organization to achieve the milestone.

“For us, it was honestly a way to structure our safety system in the same way we do our business,” said Metro director of safety Tarek Loutfy. For Metro, SMS has been a critical tool for assuring safety as this private company has grown from a small family business to one with nearly 800 employees and 100 bases across the U.S. With the formal processes embedded in its safety management system, Metro is able to track safety as rigorously as any other key performance indicator, and respond immediately at the first signs of trouble. Underpinning the system, according to Loutfy, is the commitment to safety by president Mike Stanberry and managing director Milton Geltz, and their willingness to “put their money where their mouth is.” That, he explained, is essential: “If your senior management is not committed and backing [SMS], it’s not going to be a success.”

Metro’s three employee reporting systems, including one anonymous reporting system, are important elements of its SMS. Together, the systems generate around 2,000 reports per year (in SMS, high reporting volumes are generally seen as a good thing, as they reflect strong employee engagement). While the reports vary in significance, many of them call attention to important hazards or generate valuable “lessons learned” that can be shared throughout the company — and beyond.

“If there is a safety concern, we’re happy to take it outside the company,” Loutfy said, noting that Metro has used SMS data to suggest design changes to manufacturers of aircraft and components. Metro’s SMS also allows it to accurately track occurrences such as laser strikes and drone sightings, data that are potentially helpful to the entire industry. “Now hopefully we can trend things,” Loutfy said. “We can see, is there a problem specific to the base? Is there a problem specific to the industry?”

For large companies like Metro, where direct supervision of employees by senior management is often impossible, the benefits of SMS are especially obvious. But what about in small companies, where face-to-face conversations with managers are more natural than filling out hazard reporting forms? That’s where much of the controversy surrounding SMS is centered now. Although safety management systems developed in the context of large organizations, their champions want to see them adopted by small operators, too. But it’s not clear that SMS is always the most valuable use of resources for a small company.

Glen Sibbeston, former chief pilot of Trinity Helicopters in Yellowknife, became a believer in formalized safety systems during the eight years he spent flying Sea Kings in the Royal Canadian Air Force (RCAF). Although the RCAF’s safety program was not then identified as a “safety management system,” it was strongly focused on proactive hazard identification and risk management, “and it was really quite effective,” Sibbeston said. “The military has what are really inexperienced crews, in complex aircraft, on complex missions, with a very low accident rate.” That said, he pointed out that some aspects of safety that are important to bush operations simply aren’t well captured by SMS — such as the willingness of management to back up pilots who say no to customer pressure. Moreover, a lack of documentation doesn’t necessarily mean that sophisticated risk assessment isn’t taking place. For example, when a chief pilot weighs various factors in deciding which line pilot to assign to a particular job, the process wouldn’t automatically be improved by writing it down on paper.

In his report for the ATSB, Matthew Thomas noted that the components of safety management systems tend to be “ever-growing,” which can lead to a “dilution of effort” across the spectrum of safety management activities. He warned, “This dilution of effort may well result in poorer safety performance as the critical components receive less time and effort at the expense of yet another ‘good idea’ dressed up as a legitimate safety program.” Conceivably, in smaller companies with fewer resources, the consequences of this dilution of effort could be more severe.

Proponents of SMS say this doesn’t have to be the case. Don Baldwin, president and CEO of Baldwin Safety and Compliance, actually started his technology and consulting company specifically to help small operators reap the benefits of more structured approaches to safety (he now has larger customers as well). Baldwin said that the core principles of SMS are very applicable to small operators, although it’s also true that, “if a small operator tries to implement a safety management system like a large operator, it won’t work.” For small companies that focus on the basics, he said, “It doesn’t have to be complicated.” Many of his company’s software tools are focused on streamlining and simplifying SMS processes. More recently, Baldwin Safety and Compliance has established safety co-operatives for small operators to help them achieve benefits of scale.

Just as many large companies are eager to testify to the benefits of SMS, a growing number of small operators are also swearing by it. “It’s very simple if you keep it simple,” said John Kirk, who, as a pilot for Mercy Flight Central in New York, took the lead in developing its safety management system. According to Kirk, even small companies can benefit from the improved lines of communication associated with SMS — and simply documenting a safety concern can provide busy managers with a powerful incentive to address it. Kirk said there are also less tangible benefits associated with a well-implemented SMS, including a heightened sense of pride and ownership throughout the organization. “In aviation, there is a sense of pride in what we do,” he said. With SMS, “we can actually prove that we’re taking [safety] very seriously.”

For all of the testimonials in support of SMS, however, one troubling fact remains. Many companies with safety management systems are continuing to have accidents, many of which are preventable. In these instances, at least, SMS is failing. Why?

There are several broad reasons why safety management systems might fail. First, an SMS might be poorly implemented — but more out of ignorance than malice. “Some organizations have figured it out and some haven’t,” said Richard Komarniski of Grey Owl Aviation Consultants, who leads workshops in SMS familiarization and policy development. One particular problem he has noted is that many companies conflate aviation safety goals and occupational health and safety goals. It’s hard to make meaningful progress on the former when, as he put it, “you’re counting Band-Aids in a first aid kit.”

There’s also the fact that safety management systems are fundamentally evolutionary systems, not an out-of-the-box solution to everyone’s safety woes. SMS aims for continuous safety improvement, which takes for granted that some things will need improving. A safety management system takes time to mature, and along the way, some things will inevitably slip through the cracks. According to Tarek Loutfy, Metro Aviation is regularly reviewing and updating its SMS, revising policies and procedures as the need becomes apparent. “We’re not perfect,” he said, “but I think our heads and hearts are in the best interest of our employees and the company.”

Unfortunately, not everyone’s are. And that is the most problematic reason why some safety management systems fail: because they don’t represent a genuine commitment by management to keeping people and equipment safe. “You can collect a lot of reports, but if they’re not acted on, the system doesn’t work,” observed Don Baldwin. Employees who see managers consistently ignoring hazards and condoning rule violations will quickly become disillusioned and cynical about SMS — in direct proportion to how prominently SMS is featured in the company’s marketing material.

Many pilots who were interviewed off the record for this story were deeply skeptical of how safety management systems have been implemented in the industry. “I think SMS is a wonderful idea in principle,” said one U.S. air medical helicopter pilot. “In practice, though, it is a fancy façade for marketing, shareholders, and upper management to feel warm and fuzzy. Safety isn’t a manual or a risk assessment, it is a mindset and a personal characteristic, kind of like integrity, some people have it and some don’t. [SMS] is smoke and mirrors and down in the trenches most just find a way around it.”

Said a utility helicopter pilot: “If you work at a place where you believe an SMS will fix your place of work, it won’t, it can’t. Ethical decisions and professionalism aren’t something that is created on paper. They will never come out of an SMS. These attitudes are created from people that care about what they do and from owners that care about their employees.”

Even advocates of safety management systems acknowledge that the success of any SMS hinges upon an organization’s “safety culture” — something that’s tricky to define, but which, according to the FAA, “is embodied in the way that the organization and its members approach safety in their jobs.” In its advisory circular on SMS, the FAA notes that safety management systems can help improve safety cultures by setting up “the policies and processes that create a working environment that fosters safe behavior.” Improvement, however, is by no means assured. “Some companies may implement an SMS on paper that meets regulatory requirements but isn’t effectively used,” said TSB chair Kathy Fox. The Buffalo Airways accident, she pointed out, “really highlighted how ineffective an SMS can be if you don’t match it with a good safety culture.”

And the Buffalo Airways case is by no means the worst SMS failure witnessed by Canada in recent years. On July 6, 2013, an unattended Montreal, Maine & Atlantic (MMA) freight train parked for the night at Nantes, Que., started rolling toward the small town of Lac-Mégantic, seven miles (11 kilometers) away. It reached a speed of 65 miles per hour before derailing in Lac-Mégantic’s downtown, spilling about six million liters (1.6 million gallons) of petroleum crude oil that fed horrific fires and explosions. Forty-seven people were fatally injured in the derailment and ensuing firestorm. Forty buildings and 53 vehicles were destroyed. And while the resulting TSB investigation found the immediate cause of the accident to be the engineer’s failure to set sufficient handbrakes, it also identified severe deficiencies in MMA’s safety management system, and in Transport Canada’s oversight of the same. A key finding was that “[MMA’s] weak safety culture contributed to the continuation of unsafe conditions and unsafe practices, and compromised [MMA’s] ability to effectively manage safety.”

In its report on the Lac-Mégantic disaster, the TSB concluded, “If Transport Canada does not audit the safety management systems of railways in sufficient depth and frequency and confirm that corrective actions are effectively implemented, there is an increased risk that railways will not effectively manage safety.” In SMS, audits are an important element of safety assurance, and can be critical in uncovering safety deficiencies that might otherwise be overlooked. The problem, as Richard Komarniski observed, is that “an audit is only as good as the auditor, who needs to be able to determine if the company safety culture is actually changing.” Events such as the Lac-Mégantic and Buffalo Airways accidents have highlighted shortcomings in Transport Canada’s audit processes. Private auditing companies often employ talented specialists in the auditor role, but these companies may have their own limitations with respect to the frequency and relevance of their audits.

Here’s the essential problem with SMS. When implemented well by senior managers who genuinely prioritize safety, an SMS can accomplish what it’s supposed to. But when an SMS is not implemented well or sincerely — when it is adopted simply to satisfy a regulation or a contract requirement, or to provide a marketing advantage over one’s competitors — it will fail, possibly with tragic consequences. And outside audits of safety management systems do not consistently identify problems before those failures occur.

What is the solution? Is it more and better auditing? Is it more detailed and prescriptive regulation? Mark Winfield, a member of the Faculty of Environmental Studies at York University in Toronto, Ont., has suggested a radically different approach, arguing that the best way to encourage responsible adoption of SMS is to not make it regulatory at all. Instead, he said, the government should “pierce the corporate veil” by holding accountable executives in transportation companies personally liable for safety failings, in the way that Canadian environmental legislation has incorporated more active, civil, standards of liability regarding the duties of company officers and directors. According to Winfield, the introduction of such provisions has effectively driven the development of environmental management systems analogous to safety management systems — and, more importantly, has provided incentives for senior management to take them seriously.

“In the environmental case, [personal liability] has provided a very powerful incentive for senior officials to pay attention,” he said. The crucial point is that an executive’s primary defense in the event of prosecution is a defense of due diligence, and “the existence and implementation of these kind of internal management systems is very essential to mounting that kind of defense.”

Of course, there are strong arguments  for preserving the corporate veil, which is a central tenet of capitalism. Moreover, even the most safety-minded senior executive probably isn’t going to lobby for provisions that would heighten his or her personal liability for safety failings. According to Winfield, in the case of Canadian environmental legislation, company officers and directors were no more enthusiastic about the idea, but lost out to an overwhelming political consensus in favor of stricter standards. “They complained, but it didn’t really work very well,” he said. “They just had to live with it.”

Why hasn’t something similar already happened in the transportation industry? Winfield noted that the environmental movement benefited from strong public engagement and interest groups that were “in the business of pushing these things forward.” In transportation, he said, “You don’t have a civil society constituency that makes it its business to be a watchdog. . . . You need consistent voices that are well informed to press these issues.”

It is possible that the aviation industry is simply experiencing “growing pains” with SMS; that, as safety management systems become more widely implemented and better understood, they really will achieve the step change in safety we’re striving toward. That’s the hope of organizations like the TSB, which, according to Kathy Fox, does expect to see a continued decline in accident rates “if and when SMS is fully implemented and mature.”

But we’re not there yet — and it’s important to keep in mind how far we have to go. In pushing for a more proactive and predictive approach to safety, proponents of SMS have long admonished operators, “Just because you haven’t had an accident, doesn’t mean you’re safe.” Unfortunately, as the record shows, having a safety management system is no guarantee of safety, either.