The Safety Contrarian

A substitute for "causes" in accident investigations

2010-06-28T17:17:00.000-07:00

A source of continuing confusion, debate and complaints among investigators is the the determination of "cause" in its various forms as a central investigation output. A recent presentation to the International Society of Air Safety Investigators (ISASI forum April-June 2010, p5) raises the question "The Accident Cause Statement-Is It Beyond Its Time?" It is the latest in a long series of papers that challenge cause statements in investigation reports, mine among them.

One of the shortcomings of previous challenges has been the lack of an alternative to replace the cause statements. If not cause, then what? In a 2007 paper presented to the European Safety and Reliability Data Association, I offered the first glimmer of a substitute, commenting as an aside about an example of an alternative lessons learning system design that it made cause determination moot. It wasn't until later work that I recognized the full significance of that aside.

That recognition came to fruition as I was preparing a presentation for the 16th HPRCT conference in Baltimore this month, dealing with the shortest data pathway from the generation of source data during an incident to the demonstrated performance improvement in an activity.

In that presentation, the first step I described was how to transform source data generated during any experience into improved performance, by using a new standardized structure for building blocks documenting that data. The second was an analysis structure for organizing, coupling and analyzing those building blocks to show dynamic interactions during the experience. The third was a structure for reporting lessons to be learned from those analyses as input-output behavior sets that could be overlaid onto existing operational behavior patterns. The presentation also included the benefits that could be realized, and a plan for incrementally implementing the changes into an existing operational improvement processes.

The key change is to shift from an accident causation model to an input-output process model as the basis for investigating the accident/incident phenomena. The process which produced the outcomes precipitating the investigation can be described verifiably and explained in terms of behavioral inputs and outputs among people, objects and energies. Those interactions are represented as behavior sets, as explained in the paper and presentation. Anyone insisting on selection of causes can do so according to whatever criteria they desire from the interacting behaviors described.

The paper and presentation are posted in the lessons learned research section of the www.iprr.org web site, with a list noteworthy ideas introduced, abstracts and documents. They also can be downloaded directly from the web site home page.

Investigation process research quality

2010-02-20T13:59:00.000-08:00

I was hesitant about posting this message, but having witnessed the problem on other occasions, decided it is important enough to do so.

My interest in accident investigation process research dates back to the early 1970s, when anomalies, differences, disputes and contradictions among investigators and investigations, observed during my participation in NTSB investigations in each mode, led to my efforts to see if I could harmonize the processes. That prompted me to document them, question and understand why each existed, and try to learn what might be done to overcome the differences. I have seen and studied an increasing number of research reports comparing various investigation methodologies, processes and practices, and written extensively about the topic. What finally motivated this blog was publication of a recent report comparing analysis results using an established investigation process with a systemic process under development, and the claimed benefits of the developing systemic process for accident investigation and analysis. I was knowledgeable about both processes. My review of the research report generated concerns about the research bias evidenced, and the the effects widely read biased comparative research results can have on future investigations and analyses.

I attribute the problem to bias rather than poor scholarship. Evidence of authors' bias was readily discernible in the authors' limited search for and data source selection for information about one of the two processes; the difference in the effort invested in applying the two processes to a selected incident; differences in the nature and scope of the samples used to illustrate differences produced by each process; the serious misrepresentation of capabilities of one process because of a disregard for some of its rules during its application; unbalanced critical comments about the two processes; and the close personal involvement of the authors with the process they concluded was better.

Because of the exhibited bias favoring one of the processes, and the resultant under-analysis and misrepresentations of the "also ran" process, false impressions about the nature, application and results of the "also ran" process are reasonably predictable among uncritical or less knowledgeable readers. That can be expected to unfairly diminish the perceived value and potential use of the also-ran in the intellectual, social and economic marketplace. And that, in turn, could discourage use of a process that, fairly compared, might be superior to the favored process, to the detriment of safety.

Maximizing objectivity in comparative accident investigation and analysis research is a continuing challenge that apparently merits more attention than it has gotten, based on this reported research and previous documents. My criticisms offer a starter list for improvement. Other criteria for detecting or avoiding bias would be welcomed.

Safety Metrics

2010-01-06T17:19:00.000-08:00

Recent developments promising to make possible the objective assessment of accident prevention performance have stimulated my adrenaline. The development emerged from dialogues with my co-author of a paper presented at an ISASI Seminar in September 2009, during our development of our presentation for the Seminar. The concept is based on the use of "behavior sets" generated during accidents, incidents or other experiences with undesired outcomes, and the use of those behavior sets in future investigations, ongoing activities and new systems analyses.

No two accidents are identical. However, some interactions during an accident may be similar or identical to those experienced in previous accidents. The similarities can be found by analyzing the interactions as behavior sets - or interactions among the people, objects and energies involved. In future investigations, the presence of a previously experienced (and reported) behavior set indicates the accident is a "retrocursor" accident, or one with precedents which should not have been repeated - e.g., should have been prevented. Such presence offers a retrospective assessment of the lessons learned practices designed to prevent accident recurrence, or retrospective prevention performance measurement.

Behavior sets offer opportunities for examining ongoing operations to find the same or similar patterns of behavioral interactions in those operations, and if found, to change those behaviors and reduce the risk of future accidents. That's prospective accident prevention action, measured by the discovery and modification of risk raising behavior patterns. They also offer unambiguous operations audit check-off items, and guidance for change analysts and procedures developers and reviewers.

A third prevention opportunity is to review planned operations during design stages to determine whether they might contain planned or potential interactions identified in previous accidents. Their presence or absence in future operations then offers a measure of the effectiveness of the predictive analyses in preventing future accidents. I see the availability of behavior sets having a significant impact on future risk assessments, because the offer an unambiguous multi-dimensional definition of risk raisers at the lowest level of abstraction, as contrasted with the uni-dimensional nature of a factor, error, failure, hazard or cause of any kind.

When combined with the input-output data structure suggested in a 2007 paper, it seems to me that it's worth pursuing.

The ideas were first proposed in our ISASI paper and presentation that can be found at

http://www.iprr.org/research/llprojcontents.html, items 9-11.

Lessons learning system

2009-07-21T19:06:00.000-07:00

"Lessons learned" as a concept has been around a long long time, and has been examined often in the past. It is one of the underlying reasons for doing accident investigation and incident investigations, and other investigations of all kinds. Yet when it comes to using the lessons, the inquiry rates are very modest, and reasons for not using them are numerous.

To find what knowledge has been gained about lessons learned in accident investigations, and who may have studied the topic, a google search seemed like a good starting point. For a "lessons learned" search, Google produced around 20 million hits. Using "lessons learned" with accident or investigation produced 2,420,000 hits. That's a lot of lessons learned. Lessons learned process is another term used frequently in connection with these activities; an advanced Google search for "lessons learned process" produced around 300,00 hits in many diverse fields. When we narrow that search even further by looking for accident or investigation related lessons learned process, an advanced search produced 2810 hits. Now if we want to analyze those processes using a system analysis approach, a search for accident or investigation "lessons learned system" produced a slightly more manageable 645 hits, but that still included many hits not related to accident investigations. To try to narrow the search further, "lessons learning system" and accident or incident we entered, resulting in 5 hits with Google, and 5 with Yahoo. -{mostly my works.)

Using another tack, the 2810 and 645 hit lists were scanned to find references to organizations that had lessons learned process. Many do - some of which derive lessons from investigations. When the 645 hits were scanned, the tenor of the references listed observed to be focused on the lessons, rather that the full breadth and depth of the learning process, from the time data from which lessons are developed until changes based on the lessons have produced expected results. A few exceptions were noted: when major accident processes are examined thoroughly, as in a Challenger space shuttle accident investigation or in the Bunsfield tank farm explosions, calls for improvement in lessons learned processes sometimes occur.

Possibly more significantly goals, criteria, metrics, ouutput specifications, quality assurance and other properties of lessons were noteworthy by their ambiguity or absence, with most focusing on step by step actions to process the lessons that were inputed to the system.

How then can we reasonably expect lessons learning systems to be optimized, or even improved. This is what we are exploring. Some progress has been made and reported.

Contributions of criteria or suggestions for lessons learning system improvement - or critiques of some of the ideas are invited.

Accident statistics: valid?

2009-01-23T10:56:00.000-08:00

Statistical analyses of accidents has troubled me for many years. One of my main objections was that statistical correlation does not equate to "cause." Very recently an even more significant insight occurred to me during a discussion with a colleague, Ira Rimson, as we were discussing the system descriptions as inputs used by safety analysts. We were discussing the description requirements to help safety analysts understand the dynamics of the systems they were being asked to analyze, and how those dynamics should be described. Our experience suggested that the descriptions presently offered were fragmented elements at best. Extending that notion to the descriptions of accidents led to a new concern involving the idea of "sampling" accident dynamics during accident investigations. What should be sampled and how should the samples be documented?

Digitization of music provides an instructive analogy.

To digitally reproduce music, states of a musical work are sampled at various rates, typically ranging from 22k to 44k or more samples per second. The lower the sampling rate, the less faithfully the music is reproduced. Carried to its logical conclusion, a single sample of a song is useless if one wants to "hear" the data as music.

Think of the production of music or a song as a process, requiring the dynamic interactions of the people, and instruments, and the pitch, frequency and other constantly changing relationships necessary to produce the notes as the song progresses from beginning to end. To digitize the music the state of each of these attributes must be sampled frequently as the song progress. Digital video captures images sampling action in frames per second. Same idea: the greater the number of samples, the greater the fidelity of the reproduction.

Apply similar thinking to the descriptions of systems and their operation, and how such information is presented to system safety analysts who are expected to find hazards in a system and predict system aberrations. How well is it possible to meet those expectations if the descriptions of the processes they are to analyze has insufficient samples of the "music" the system operation might produce?

Now, think also of accidents as processes, involving dynamic interactions of the people, objects and energies needed to produce the accident as it progresses from its beginning to its outcome over time. What data do we sample to predict and capture a description of that process? That question has been answered after the fact in part in some activities, such as aircraft system operations captured with digital flight data recorders and analog cockpit voice recorders. But before the accidents happen, how can you predict them with insufficiently sampled process description?

The question seems worthy of more exploration.

Lessons Learning Systems

2008-06-07T08:13:00.000-07:00

Accident investigation practices that produce "lessons learned" are coming under increasing scrutiny. Issues surfacing as a result of this scrutiny include how to streamline and maximize lessons learned development, documentation, dissemination, accessibility, internalization and feedback. Serious challenges to many aspects of current lessons learned systems are emerging from this scrutiny, including aspects such as the

cause-based framework of present lessons learnedsystems
orientation of accident investigations relative to lessons learned
focus of lessons learned efforts,
derivation of lessons learned during accident investigations
maximization of the number of lessons learned
language and structure of lessons learned documentation,
latency in lessons learned cycles,
context available with lessons learned,
harmonizing of lessons learned data derived from accidents and other mishap investigation with lessons derived from other sources in an organization,
data density of lessons learned outputs,
breadth of the accessibility of lessons learned,
internalization of lessons learned when accessed,
monitoring of changes in activities attributable to lessons learned in mishaps,
lessons learned life span,
lessons learned obsolescence, and
strategies for improving lessons learning systems performance

See http://www.iprr.org/research/llset/SSTS08_tutorial.pdf for a presentation about this topic. And chip in your two cent's worth if you have some useful thoughts.

Drifting toward failure

2008-03-03T20:24:00.000-08:00

This expression is being recognized with growing frequency in major investigations like the Challenger loss. It reflects ideas about the responsibility of managers to direct their operations in a "safe" way, coming our of the human factors community. A recent book called 10 Questions about Human Error, by Sidney Dekker, discusses it at length.

This ideas behind the expression create some interesting challenges for mishap investigators. To name a few, what data should an investigator seek to illustrate this "drift" in a specific investigation? How does an investigator identify and acquire data that has to be coupled to the outcome of a mishap to show this "drift" in the context of when the mishap occurred? How should an investigator present the data to demonstrate its influence on the outcome in an objective, logical, persuasive and verifiable way? What risks are inherent in the investigating and reporting of this "drift" and who can be at risk of harm if the investigator does not do it properly? How can the quality of the reported drift be assessed or ensured? Are the answers primarily dependent on the investigator's experience or the methodology selected, or something else?

As one ponders this expression and the ideas behind it, it becomes evident that they have application to many kinds of activities other than mishaps.

Kickoff blog 1

2008-02-24T16:50:00.000-08:00

Here's a challenge.

I think anyone connected with safety or mishap investigations needs to rethink why we do investigations, rather than blindly accepting the folk wisdom about investigation purposes. I suggest that purposes like "to prevent recurrence!" or "find the cause or probable cause or root causes" or affix blame or find out what happened and why or find lessons learned or settle claims miss the ultimate purpose, and misdirect investigation practices.

I'd offer this simple candidate: TO CHANGE BEHAVIORS. The reason for investigating an occurrence is to find, understand and report the behaviors that produced an unwanted occurrence, and their context, in a way that others engaged in similar behaviors can learn about them and apply that information to their future activities.

I would argue that this purpose would change how investigators do investigations and what they produce, by focusing their inquiries on what people, objects or energies did during mishaps, and why they did that. I would argue further that this would improve future investigation efficiency, efficacy and value. I would also argue that this would produce better predictive understanding of mishap phenomena if it were applied to present system safety analysis practices.

Anyone want to defend the status quo?