As we close out this year, one thought has been dominating my days. We’ve all learned how to practice risk from different places (where I’ve worked is different from where you’ve worked, etc.). So much in the practice of risk is based on the notion of personality; we do risk one way because I’m leading it today. Tomorrow, a different person leading it would insist on doing it in a way that they are familiar with. Essentially, like all humans, we overemphasize our own experiences in risk as being more “true” than others. In other words, we are more likely to assume that what we’ve experienced before can be made to occur again with the same results. Unfortunately, this is not how science views the same set of experiences. In the scientific method, not everyone’s experience qualifies as valid scientific observation. In a given room full of risk people, the sheer fact that we’ve experienced different things doesn’t make them all valid, at least in a scientific way. Any attempt to include everyone’s experience as objectively valid is incorrect and potentially dangerous. I often refer to this as “risk relativism.”
An overriding characteristic of scientific observation is the ability for different practitioners to be able to recreate the results (reproducibility) given a similar set of environmental variables. In fact, a case study is probably the most accurate research method for what most of us call our work experience. Case study’s are very concerned with “validity:” construct validity, internal validity, external validity, and reliability. The combination of these four things contribute to the overall reproducibility (aka objectivity) of the research. In all cases, there is a need for using multiple sources of data and/or observations to ensure that each unit of analysis (workplace experience) are “coded” or analyzed accurately. Further, the applicability of results are carefully curtailed. For example, a single-unit case study has limited applicability outside of its own environment (one company’s risk experience is obviously most applicable to just that company), but multiple unit case study’s can be applied more broadly.
But what factors are the most critical for establishing broad applicability and reproducibility? In my opinion, that is the use of an accurate model; that is, the use of a model that can reliably used to predict outcomes. Put another way, across all your industry experience, which model are you applying that allows for meaningful measurements to be made that enables effective comparisons? Work in information theory tells us that all measurements are inexact (statistical if you will). This lends great credence to the use of statistical methods to reduce uncertainty in our measurements as we move from workplace to workplace.
What are you going to do in 2015 to increase your use of scientifically valid models of measurement?