Modelling clinical diagnostic errors: A system dynamics approach Shijing GUOa,1, Abdul ROUDSARIb and Artur d’Avila GARCEZa a Department of Computer Science, City University London, UK b School of Health Information Science, University of Victoria, Canada
Abstract. The diagnostic process involves a series of stages in the patient pathway. Any errors or misleading information from any stage could lead to errors in the final decision making. System dynamics modeling maps the diagnostic process as a whole and seeks to provide a quantitative way of analyzing different errors at each stage as well as relevant key factors. This paper provides a framework based on system dynamics for modeling the tracing of errors inside of the system from where errors initially occur, the routes of errors inside of the system, and how errors are delivered out of the system. Also, a detailed illustration of the phase history and physical examinations is provided as an example to explain how relevant factors can be interpreted and how this affects errors according to the framework. Keywords. System dynamics, diagnosis, diagnostic errors, modeling, stock and flow diagrams
1. Introduction Clinical diagnosis involves a series of phases from history and physical examinations, assessment tests to referral, where errors may occur in any phase and lead to diagnostic errors [1]. This study using system dynamics modeling maps an entire diagnosis process as a system, which not only illustrates the above phases where errors occur, but also covers the possible error detection stages. System dynamics modeling describes the number of errors occurring during different stages, tracks the routes of the errors, and provides simulations to show model behaviors while varying factors inside of the system. System dynamics is an approach that helps one to better understand problem situations and solve problems through finding different management policies and alternative organizational structures [2]. It was initially used to model corporate and managerial problems, and then generalized to non-corporate areas since the late 1960s[3]. Nowadays it has been used widely, including in healthcare. Its current applications in health and social care systems mainly cover illustrating the structure of system resources [4], as well as interpreting experimental results and understanding the dynamics of results in disease research, for example, the research of HIV and human immune system [5]. It is suggested that system dynamics could be implemented into dynamic systems, with time and spatial parameters to help one observe how system behaviors change. [6]
1
Corresponding Author: Shijing Guo. E-mail:
[email protected]
This study applied system dynamics to the modeling of errors during diagnosis, mainly because of two concerns. Firstly, errors during the diagnostic process change along with the change of other variables in the system. System dynamics could simulate the change and observe system behaviors over a period of time under different scenarios. Secondly, from a systemic view, new errors occur when doctors collect diagnostic information or diagnostic clues that could be detected and corrected in the subsequent stages, through the use of modeling. In other words, errors not only have multiple causes which increase error numbers, but also are delivered out of system via various routes. System dynamics has been implemented into two steps in this study: causal loop diagrams, and stock and flow diagrams. A Causal Loop Diagram has been designed as the first step by a previous study [7], which shows model variables and their interrelationships. This paper focuses on quantitative analysis using stock and flow diagrams to allow one to observe the information inputs and system behavior outputs.
2. Methods The basic theory of interpreting diagnostic errors into stock-flow diagrams is illustrated in Figure 1.
new errors
The number of existing diagnostic errors
detected errors
Figure 1. Interpreting diagnostic errors in a system dynamics model.
The above stock and flow diagram contains a rectangle with an input arrow and an output arrow. The rectangle, referred to as “Stock” in system dynamics, indicates a quantitative stock. The input arrow indicates the inflow of the stock, and it increases the stock level. The output arrow indicates the outflow of the stock, and it decreases the stock level. [8] The stock in Figure 1 denotes the quantitative level of diagnostic errors or “the number of existing diagnostic errors” in the system. The input arrow “new errors” indicate the new diagnostic errors per time unit, while the output arrow “detected errors” denote the detected errors after diagnosis per time unit. The stock could use the mathematical representation in Eq. (1) to explain its level changes over a period of time from initial time t0 to current time t. !ℎ! !"#$%& !" !"#$%#&' !"#$%&'(") !""#"$ ! =
! !!
!"# !""#"$ ! −
!"#"$#!" !""#"$ ! !" + !ℎ! !"#$%& !" !"#$#%#&' !"#$%&'(") !""#"$ !!
where s represents any time between the initial time t0 and the current time t.
(1)
1. 3. Results According to the patient pathway, the model in this study divides the diagnostic process into the following stages: pre-decision making, decision making and afterdecision making, which is shown in Figure 2. Specifically, pre-decision making is the period when new errors of diagnostic clues happen. Thus, this stage is further divided into 3 phases to explain where new errors come from, and it includes: “Phase1 history and physical examinations”, “Phase2 assessment tests” and “Phase3 referring to other healthcare providers”. [9] Phase1 history and physical examinations
Phase2 assessment tests
Phase3 referring to other healthcare providers
decision making
pre-decision making
after-decision making
Figure 2. Diagnostic process represented in the model.
Following this process, the final framework of modeling diagnostic errors was implemented using the software Vensim [10] and is shown in Figure 3.
Garcez, Artur 11/11/2014 13:37 Comment [1]: Add a sentence or two here to indicate how this was done!?
patient visit cases
cases with errors from Phase1 history and physical examinations per day
patient administration rate
cases without errors to be discharged after Phase1 per day
cases with errors from Phase1 history and physical examinations
Phase1 errors detected in Phase2 assessment tests per day
errors corrected after assessment tests
cases processed to next phase per day
Phase1 errors undetected in Phase2
cases without Phase1 errors require Phase2 per day
Phase1 errors to be detected in Phase3
Phase1 errors detected in Phase3 referral per day
errors corrected after referring to other healthcare providers
cases1 to be discharged without errors per day
cases without Phase1 errors require Phase2 assessment tests
cases1 with errors processed to decision making
errors from assessment tests per day cases without errors after Phase2 processed to next step per day
Phase2 errors corrected in re-tests
cases with errors from Phase2 assessment tests Phase2 errors without repeating tests
Phase2 errors to be detected in Phase3
Phase2 errors detected in Phase3 referral
errors corrected after referring to other healthcare provider
cases2 without errors to be discharged per day
cases with errors from Phase2 repeat Phase2
Phase2 errors receiving re-tests
cases with errors2 processed to decision making Pases2 errors uncorrected in re-tests
cases without errors after Phase2
cases requiring referral per day cases without errors to be discharged after Phase2
referral cases cases with errors from other healthcare providers per day
cases with errors from Phase3 referral
cases with errors3 processed to decision making
cases without errors to be discharged after Phase3
cases without errors to be discharged
diagnostic errors in index visits discharged with no harm
discharged per day
receiving treatments2
hospitalisations
detected errors during re-exam receiving treatments1
Re-visits (primary care visits followed by an unplanned hospitalisation during a period of 14 days)
detected errors during re-visit per day
Re-Visits (primary care visits followed by unscheduled primary care visits, an urgent care visit, or an ER visit
Re-visits
Figure 3. Framework of system dynamics modeling for errors during diagnosis.
undetected errors
Stocks are the number of patient visit cases. Arrows are the flows which deliver patient cases. The framework describes where errors initially happen, the routes of errors inside of the system, and how errors are delivered out of system. By adding relevant factors or variables into the framework, the model can demonstrate the behaviors of these variables inside of the system. It is easy to observe the change of the stocks given changes in the relevant variables. Phase1 history and physical examinations is used as an example in this section to explain how one can interpret variables into the framework for a full version of system dynamics modeling.
Garcez, Artur 11/11/2014 13:39 Comment [2]: Include brief examples here referring to Fig.3
Garcez, Artur 11/11/2014 13:41 patient medical history clartity of disease symptoms
missed symptoms in history evaluation
communications between doctors and patients
error percentage in history and physical examinations
continuity of care
patient administration rate
patient visit cases
patient index visits per day
missed signs in physical examinations history and physical examiniation sensitivity
cases with errors from Phase1 history and physical examinations per day
cases without errors to be discharged after Phase1 per day
cases with errors from Phase1 history and physical examinations pecentage1 processed to next step per day Phase1 errors undetected in Phase2
pecentage1" processed to next step per day
max administration rate
Comment [3]: ?
Phase1 error percentage with no effect
Phase1 errors to be detected in Phase3
cases without Phase1 errors require Phase2 per day
errors corrected after assessment tests
Phase1 errors detected in Phase2 assessment tests per day
percentage of Phase1 errors detected in Phase2
percentage1 processed to next phase per day cases processed to next phase per day
percentage1 of cases with missed assessment tests
errors detected after referring to other healthcare providers
Phase1 errors detected in Phase3 referral per day
percentage1 to be discharged per day cases1 to be discharged without errors per day
percentage1 of improper referrals
percentage of Phase1 errors detected in Phase3
pecentage1' processed to next step per day cases1 with errors processed to decision making
Cases in Phase2
cases without errors to be discharged after Phase2
cases requiring referral per day
cases with errors2 processed to decision making
cases2 without errors to be discharged per day
Cases in Phase3 cases with errors3 processed to decision making cases without errors to be discharged after Phase3 cases without errors to be discharged discharged with no harm
diagnostic errors in index visits re-visits
discharged per day hospitalisations ER visits receiving treatments1
detected errors during re-exams
Figure 4. System dynamics modeling of “Phase1 history and physical examinations”.
According to the previous causal loop diagram, relevant factors with their interrelationships were identified. These are added into the framework using blue arrows that represent relationships with other variables. Figure 4 shows the system dynamics modeling of Phase1 after adding relevant variables into the framework. Phase1 is the period of collecting patient history (symptoms) and physical examination (signs). Errors in this phase can be classified into three groups: missed diagnostic clues, history and physical examination result evaluation, and errors with no effect. Three main factors leading to missed diagnostic clues are: clarity of disease symptoms, communication quality between doctors and patients, and patient medical history retrieval and review. [7] The Phase1 results evaluation gives the diagnostic accuracy as measured by the sensitivity of results [11]. Presenting symptoms and signs sometimes does not account for the true positive or true negative for diseased or nondiseased subjects. False positives and false negatives arise in Phase1 as well. Thus, sensitivity is a key variable to represent the percentage of individuals with disease who have a positive symptom and sign.
Garcez, Artur 11/11/2014 13:41 Comment [4]: Where is it?
Garcez, Artur 11/11/2014 13:42 Comment [5]: How?
Garcez, Artur 11/11/2014 13:47 Comment [6]: This discussion needs to be connected with the diagnostic errors, ideally through an example that refers to Figure 4.
2. 4. Discussion System dynamics modeling offers a systematic mechanism for the explanation of errors during diagnosis, even though diagnostic errors involve multiple causes with complex relationships. It provides an opportunity to trace the error routes and quantitatively analyze different errors affected by other variables. The model outputs can be tested and compared under “what if” scenarios. Figure 5 shows the number of diagnostic errors over 100 days under different sensitivities of history and physical examination results, when Phase1 is simulated with the assumption of 100 index visits (first visits) per day. It is the output of a partial simulation of the entire system, which does not present strategy guidance, but explains how different scenarios are reflected into the variations of stocks. Currently, the framework is being extendedto add the rest of the relevant variables incrementaly which is expected to lead to a more complete view of system dynamics modeling. A full version of the model is expected to determine the dominant factors and suggest possible strategies of diagnostic error reduction by observing system response to the changes of variables, although it faces the challenge of a comprehensive input data to reflect system behaviors precisely. diagnostic errors in index visits 7
cases
5.25
3.5
1.75
0 0
20
40 60 Time (Day)
80
100
diagnostic errors in index visits : sensitivity=66% diagnostic errors in index visits : sensitivity=67% diagnostic errors in index visits : sensitivity=76%
Figure 5. Diagnostic errors of index visits under partial model simulation.
References [1] Schiff GD, Kim S, Abrams R, Cosby K, Lambert B, Elstein AS, et al. Diagnosing Diagnosis Errors : Lessons from a Multi-institutional Collaborative Project, Advances in Patient Safety, 2005. [2] Forrester JW, Industrial Dynamics, 1961. [3] Forrester JW, The Beginning of System Dynamics, International meeting of the System Dynamics Society, Germany, l989. [4] Wolstenholme EF, Mckelvie D, Smith G, Monk D, Using System Dynamics in Modelling Health and Social Care Commissioning in the UK, OML consulting. 2004. [5] Perelson AS, Modelling viral and immune System Dynamics. Nature Reviews, 2002. [6] Sterman JD. Business dynamics: systems thinking and modeling for a complex world, McGraw Hill; 2000. [7] Guo S, Roudsari A and Garcez A, A Causal Loop Approach to the Study of Diagnostic Errors, Studies in health technology and informatics. 2014; 205:73-7. [8] Wolstenholme EF, System enquiry: a system dynamics approach. Wiley, 1990. [9] Llewelyn H, Ang HA, Lewis KE, and Al-Abdullah A. Oxford Handbook of Clinical Diagnosis, Oxford University Press, 2009. [10] Vensim software, Available from: http://vensim.com/vensim-software/
Garcez, Artur 11/11/2014 13:50 Comment [7]: What does Figure 5 tell us?
[11] Simel DL, Rennie D, and Patrick MM, The STARD Statement for Reporting Diagnostic Accuracy Studies: Application to the History and Physical Examination, J Gen Intern Med. 2008; 23(6):768-74.
