Introduction

Bioterrorism and pandemics pose great risk to the health and safety of our modern world. Given the rarity in which they occur and the extent to which they have so far been mitigated, training opportunities are key to preparedness. Hypothetical scenario-based training and simulation training have been used in public health and epidemiology to train staff in active epidemics (1) and to assess capability and weaknesses (2). Modelling-based simulation training offers considerable advantages over a live functional exercise in both cost of training and number of participants required (2). Deliberate practice of skills relevant to the performance of expert tasks, such as decision making in pandemics and public health analysis and intervention, is associated with demonstrable improvements in performance (3–5). Workshops can be conducted in a low threat environment, unlike real events, and provide opportunities for practitioners to critically evaluate procedures and strategies without risk to personnel, at considerably lower financial cost and without significant disruption to normal services (6).

Scenario simulations commonly use diseases, such as influenza, that reflect the most concern regarding social and economic impact, but relatively modest mortality and morbidity compared to diseases such as security sensitive bioterrorism agents (7, 8). Despite the generalisable nature of most pandemic responses, these often fail to take account of the broader impacts of a pandemic scenario (7, 9) or the potential for an event involving a pathogen of greater transmissibility and severity. Additionally, the considerable logistical challenges likely to be faced in a pandemic, (10) such as impacts on civil infrastructure and society, are often considered superficially (11). Previous studies have focused on smaller scales, such as local public health responses (12), state preparations (13) and in one case from a transatlantic perspective (14). There have not been any recent workshops of this nature that take into account the unique factors of the Pacific region. Often, pandemic scenario activities are focused at the national level to meet national readiness or training goals, but assume away many of the complexities associated with infectious disease events that traverse jurisdictional, national, geographic or social boundaries (11 , 15, 16).

In August 2018, The Exercise Mataika pandemic workshop was conducted by the NHMRC Centre for Research Excellence Integrated Systems for Epidemic Response at the University of New South Wales, Australia. The scenario was designed to collect and examine the decision-making processes of participants sampled from a wide variety of occupations, seniority and nationalities in a hypothetical weaponised smallpox release in the Pacific. Representatives from ten Pacific nations attended, alongside representatives from academic, industry and Australian government agencies. By utilising a high risk, worst case scenario – the deliberate release of Smallpox in Fiji and a much larger Asian country – impacts not often considered in pandemic planning, such as the resiliency of the health system, absenteeism, social cohesion and broader impacts on society, were considered and compared across geographic and social groupings.

The scenario was divided into three phases; i) initial release and investigation, ii) national impact and emergency response, and iii) a pandemic scenario with international impacts on civil society, government, health care and the economy. Online supplementary material on smallpox was released to participants midway through Phase 1.

Aim

This study aimed to collect and analyse participant perceptions and evaluation of the Exercise Mataika workshop conducted at The University of New South Wales, Sydney, in August 2018.

Methods

The workshop utilised a novel design with extensive use of live electronic voting on decisions, inclusion of post-epidemic civil societal impacts as factors for the participants to consider, and was attended by international and cross sectorial participants. The workshop was an event of the NHMRC Centre for Research Excellence, Integrated Systems for Epidemic Response (ISER). Invitations to attend the workshop and the research activity consisted of direct selection of key stakeholders and short listed national, state and international organisations who were free to register any number of participants. Participants were required to be either affiliated or sponsored by an academic, government or industry organisation to attend the conference due to the potential sensitivities of the content. In this study the experiences of participants in the Exercise Mataika workshop were evaluated using a mixed quantitative and qualitative research methodology. This research was approved by the University of New South Wales Human Research Ethics Advisory Panel (HREAP) Executive (approval HC180501).

Data Collection

The data was collected from a paper-based survey distributed at the conclusion of the exercise (n=46 of 64 consenting participants). The survey consisted of 23 questions divided into three section: Pre-survey demographics, General feedback on the workshop, and About your role in the workshop. The survey used a mixture of categorical questions on demographics and ranking their thoughts on parts of the workshop, as well as text questions to expand on what particularly factored into their categorical responses.

Quantitative Data Analysis

Quantitative data were initially transcribed into Microsoft Excel and then analysed using descriptive statistics. Further statistical analysis was conducted using SPSS (IBM Corporation) version 25. Statistical comparisons of the Likert scale responses (17) between groups was conducted using the independent sample Mann Whitney U (18, 19), with significance set at p<0.05, and comparisons considered where group sizes were greater than 5 (detailed statistical results presented in Annex 1).

Qualitative Data Analysis

Qualitative data was imported into NVivo Pro Version 12 (20) and analysed for recurring concepts using an inductive approach. Each survey question was reviewed and answers were codified as primary codes searching for underlying concepts (themes) (21). Recurring concepts were grouped together as themes once saturation was achieved. All responses were cross-codified as being positive, negative or neutral. From this data matrix coding and cross coding functions were used to determine weightings of themes by their proportion of positivity.

Mixed methods analysis

Major themes were triangulated with predominant findings from the quantitative data collection and analysis (22). Resultant triangulated concepts were then consolidated into major concepts for discussion and reflection.

Findings

Participants

All workshop participants at the August 2018 Exercise Mataika were invited to complete a post activity survey at the completion of the exercise. From the 76 attendees at the workshop, 64 consented to be recorded for further research, 2 participants withdrew from the workshop, and 10 participants did not consent. All participants who did not consent were removed from transcripts. From the 64 attendees who gave their consent for further research, 46 completed the survey provided, giving a response rate of 58% of overall workshop attendees. Forty-five surveys were sufficiently completed, and one response stated only nationality of the participant and was not used in further analysis.

The survey respondents comprised of attendees from 13 nations (Australia, Fiji, French Polynesia, Guam, New Caledonia, New Zealand, Papua New Guinea, Tonga, Singapore, The United State of America, Denmark, Malaysia and the United Kingdom), the majority being from Australia (n=29). The respondents were comprised of a diverse range of employer type, representing governments, non-government organizations, industry and academia. Respondents were able to mark multiple occupations applicable to them, which resulted in a wide variety of occupations being recorded as present (11 Policy, 10 General public health, 13 Surveillance, monitoring and control of communicable diseases, 2 Environmental health, 6 Domestic emergency services, 10 International emergency response, 1 Acute care, 7 Defence/Military and 10 Other). The results from the survey showed that whilst all responses were either positive or neutral, there was variation in responses on basis of respondent demographic and this is outlined below. From the questionnaire, the most varied questions related to the themes of group work and discussion, relevance and workshop materials.

Workshop Supplementary Materials

Participants were asked to evaluate the usefulness of the online supplementary material provided to attendees during to the conduct of the exercise. These materials contained information relating to the modeling and simulation data presented during the activity and supporting references. Overall, the material was found to be useful by the majority of attendees, with response ranging from strongly agree (7/43) and agree (26/43) to neutral (10/43).

Internet Based Quiz Usefulness

Participants were asked to evaluate the usefulness of the real-time internet based quiz software. There was a difference in the responses between the National government attendee group and State/Territory government representatives. National government representatives were more positive (12/18 strongly agree) than State representatives who more often responded as agreed (7/9) (p= 0.033).

Group Work and Discussion

Participants were asked to evaluate the usefulness of the Group Work and Discussion activities conducted during the exercise. The majority of respondents either agreed or strongly agreed to this question (17/41), but there were some neutral responses (7/41), which were equally distributed across participant groups.

Workshop Relevance

Participants were asked to evaluate the relevance of the workshop in their respective professional roles. 27/45 respondents claimed to strongly agree that the workshop was relevant to their situation, with 13 agreeing and 5 respondents claiming neutral. There was a significant difference between those who self-identified as senior decision makers and those who considered themselves mid-career in their responses, with senior decision makers responding more positively (14/18 strongly agree) compared to mid-career (7/16 strongly agree, 6/16 agree, 3/16 neutral) (p= 0.027).

Role in the Scenario

Participants were asked to report on their role in the scenario, and their perceptions of the interactivity and engagement with the exercise. There was a lower response rate on the participant’s role in the scenario (35/45 participants). Overall, respondents were satisfied with their roles in the scenario (23/34), but a large number were unsure (10/34, one no response). There appeared to be clearly distinct groups who felt their contribution to discussion differed within the scenario; the two highest responses were for giving inputs only in important parts of discussion 12/29) and actively participated (11/29). No respondent stated that they led the discussion, but 4 respondents noted they did not participate in the discussion at all. All responses to the question of whether the scenario added to participants understanding of epidemic response were positive (35/35) and the three highest stated categories were in health systems capacity, logistics and supply chain, and vaccination strategy. 24/34 respondents stated they had participated in an epidemic response before, and of them, 7/26 stated that this scenario was similar to others, 14/26 stated it was different and 5/26 stated it was very different.

Thematic Analysis

Primary codes were generated from the survey data using an inductive approach (23). As saturation became apparent, recurrent concepts were recoded as themes (21). Each unique reference was also coded as being a positive, negative or neutral comment. From the resultant 12 themes, four overarching themes were identified; Value of the exercise, Practical and realistic nature of the exercise, the diversity of thought and attendees, and the format of the workshop itself. Themes were tabulated into a code matrix according to codes for positive, negative and neutral responses in Table 1, and the definition of each theme explored in the subsequent analysis.

Table 1

Frequency of thematic response grouped by positivity of comment.

  Results as coded by positivity
Themes A : Positive B : Neutral C : Negative Total
1 : Value 16 11 6 33
Priority areas raised 11 8 5 24
Networking opportunity 4 1 0 5
Expert opinions 3 2 1 6
2 : Practical and realistic 38 11 6 55
Novel Scenario 5 6 0 11
Focus of the Scenario 3 3 5 11
Educational value 27 10 1 38
3 : Diversity 20 6 4 30
Diverse attendees 6 2 1 9
Different perspectives and experiences 16 4 3 23
4 : Workshop Format 20 9 19 28
Use Of Technology 6 1 5 12
Workshop Logistics 3 3 7 13
Workshop Facilitation 4 0 0 4
Participant Interaction 12 7 12 31

Note: Some comments are coded into multiple values. Only unique references are counted for calculation of overall comments on a theme

Value of the exercise

Participants in the workshop made several comments regarding the value of the exercise that could be classified as positive, negative or neutral. There was importance placed on the ability of the workshop to generate areas for further consideration of research or capability development, highlighted by exploitation of current capability gaps and weaknesses in current disaster response strategies throughout the conduct of the exercise. Diversity of thought and approaches to issues were seen as a net benefit to discussions to such an extent that many of the comments made about the format of the workshop itself and level of interaction were linked to increasing their ability to hear these alternative viewpoints.

Identifying and raising priority areas was very important for attendees. Participants remarked that it “increased visibility on all of the various considerations of a pandemic scenario”, and there was “more thorough exploration of scenario and aftermath” when compared to similar workshops. Some neutral comments recognised that this workshop had “more focus on logistics and supply of PPE than the broader pandemic factors”, which made a point of difference with other workshops. Another issue that was criticized was a view that some responses to questions was either “led” or “a little artificial”.

Practical and Realistic

The participants commented positively about the realism and practical nature of the exercise itself. Having the scenario include discussion on logistics, such as stockpiling of vaccines, and insights into technical humanitarian, social, ethical and legal system insights was seen as a benefit to the exercise and provided comprehensive coverage of the scenario. This enabled critical thinking and further assessment of capability gaps between the various nations. Other attendees suggested that the specific focus on the Pacific was somewhat negative, as it reduced discussion of the global impacts of such a scenario and that a “Pacific focus was not directly useful and [the] opinion of individuals only”.

The focus on the Pacific and the conduct of the third phase of the exercise were the most controversial aspects. Some commenters found these areas to be irrelevant or not directly useful, whilst others found the inclusion of 2nd-4th order effects (effects on civil infrastructure and society) to be insightful and offered “increased visibility on all the various considerations of a pandemic scenario”. One commenter appeared unsure as to what purpose the post epidemic situation served, stating “Why focus on “post”-epidemic (sic) medicine?”. This highlights a recurring issue amongst some attendees that they either misunderstood the purpose of the workshop itself or expected the workshop to be similar to others they had attended before.

The use of a novel scenario, particularly one that several the participants were unfamiliar with, added a dimension to the exercise that 5/11 of participants commented as being positive with the remainder neutral. Responses focused on the agent used (Smallpox) (“This one is unique - both from the nature (small pox), scope and the details of the steps and challenges”), and generalisability of the workshop (“Perhaps highlighting the differences in outbreak situations in smaller Pacific Island Countries vs more developed ones and the differences in capacity to identify and response to outbreak situation”; “Knowing how the process was able to be broken down. For example - Media, how, military response, etc”; “Increased visibility on all of the various considerations of a pandemic scenario”), there were no negative comments raised regarding the novelty of the scenario used. Additionally, some commented that it was good to have an exercise that was not focused on influenza and was not seen as “US-centric”, implying that this is often the case.

Participants found the breaks and pauses after voting to discuss issues useful to gain a better understanding of said issues and learn more about the specific topics. This was noted as a particular benefit of assembling such a diverse range of experts and perspectives.

Diversity

Several participants (n=20) commented a number of times about how the exercise enabled them to hear different perspectives and different experiences from other attendees. The participants thought the diversity of attendees representing a broad range of occupations and nationalities enabled a better understanding of the issues particular to those fields and those nations. Multiple comments were made that the plenary format of the workshop didn’t sufficiently capitalise on the presence of these individuals and some attendees would have preferred more smaller group style discussions. This represents a potential missed opportunity for participants to hear more from the different perspectives the various roles and nationalities provided. Some commented that the diversity was perhaps not taken far enough and they would like to see an even wider representation at further workshops, specifically from “civil society organisations who have field experience in outbreak investigation/response”.

Many commented on the benefits of having multiple viewpoints and expertise present at the scenario (n=16) and three participants commented that their expertise was not utilized as much as it could have been. One commented that they remarked their “input was not used” and another noted “the topic did not cover my role due to a different focus of the discussion”, suggesting that further workshops could expand and further utilize the diverse range of expertise present. This is reflected in 10/34 responses feeling unsure about their role in the scenario.

Workshop Format

There were an equal number of positive and negative responses regarding the level of interaction (n=16). Some praised the use of electronic voting to provide immediate feedback on questions posed, whilst others stated that the plenary discussion format limited their ability to participate. A persistent theme that emerged was participants’ preference for small group settings or at least smaller group work to further engage and interact with the other participants. The use of technology was seen by several attendees to be useful as a means of making the workshop interactive and educational and was of high quality (n=6). There were issues noted with the internet connectivity at the venue used for the workshop. Given that the workshop relied heavily on use of online voting, the workshop comments related to the venue’s facilities including internet connectivity and AV technology had more negative comments (7/13) than positive or neutral. Whilst the format was the most criticized portion of the workshop, it was notable that there were no negative responses recorded in any section of the categorical rating of the survey. This suggests that despite feeling that the issue could be improved upon, the issue was not sufficient to negatively impact the conduct of the exercise.

Mixed Methods Analysis

The addition of factors such as logistics of PPE and vaccines added a level of complexity and educated the audience on issues they had not previously considered, which is supported by the quantitative data showing all respondents selecting that the scenario added to their understanding of epidemic response and that most of these responses were either in capacity of health systems (7/35), logistics (7/35) and/or vaccination strategy (6/35). The participants generally stated the workshop was relevant to their needs, particularly in the higher decision makers demographic. This is perhaps suggestive that the workshop, being a regional level scenario, was most applicable to the more senior attendees.

Positive overall, the largest number of negative free text comments related to the logistics of the exercise and workshop materials. Broadly, the reaction to workshop materials was positive but had the lowest number of the most strongly positive responses. The use of electronic voting and AV aids was beneficial and added to both the educational and realism of the exercise. There were a small number of problems with the internet connectivity and availability of online resources, accounted for in feedback.

Bringing the results together, while there were no directly negative responses relating to overall experience and utility of the workshop in the survey, participants were more forthcoming in their free text responses. The thematic analysis showed that while experiences were overall positive, specific constructive criticisms, but not negative concerns, were identified.

Discussion

This paper sought to evaluate the responses to a post-activity survey for a hypothetical pandemic scenario conducted at UNSW Sydney in August 2018. The most significant findings related to practicality and realism of the scenario, engagement and interaction of participants, and the value of a diverse range of attendees. Participants mentioned they gained the most from their interactions with other participants by hearing their experiences and different perspectives on issues, which helped them to consider the vulnerabilities of their current strategies in a response to a pandemic event. The inclusion of issues beyond the immediate epidemic event challenged participants to consider broader aspects of pandemic response beyond the initial public health response.

Whilst the quantitative data showed most respondents found the group discussions useful, some qualitative comments suggest that some participants would have preferred smaller group work. Some commenters (n=4) remarked that they found the discussions useful as a means to network and hear alternative viewpoints with regard to the scenario. A theme that emerged amongst some attendees was that they were unsure about the precise purpose of the workshop and the research aims and how the workshop worked towards those objectives. As the purpose of the exercise was to evaluate decision making processes of the individuals and not primarily targeted at either being a training or networking focus, the two most common focuses of research into disaster management table top exercises (11), participants may not have understood fully the underlying purpose of the exercise. This exercise utilised a novel approach to presenting an epidemic scenario and asked participants to explore dimensions to a response not often taken into consideration. An alternative interpretation of this feedback is that the exercise took some participants outside of their expectations of an epidemic scenario resulting in uncertainty regarding the approach.

Previous studies have shown that attendees at preparedness workshops utilize such exercises as a forum to share knowledge and experiences, suggesting an expectation that the purpose and main effort of such exercises is the sharing of experiences (24). Familiarity with previous workshop designs which are standardised (25, 26) may have led local participants to expect a particular format from the workshop that would be conducted along the lines of standardised processes such as the National Communicable Disease Plan (27) or Australian Health Management Plan for Pandemic Influenza (28). However, the exercise was not focused on Australia. Channelling thinking along the lines of standardised plans and procedures may lead to neglecting critically important broader societal issues that are not addressed in national plans. In this exercise, focus was placed on the intersectoral, international and interagency dynamics that are likely to emerge as a major epidemic unfolds. Rather than concentrate on the health system dimensions or business continuity dimensions of epidemic response, participants were challenged to consider dilemmas and problems that could only be solved through cooperative, synchronised and well-orchestrated response. Similar studies that focussed on outcomes from epidemic workshops have shown the importance of highlighting aspects of a response plan participants may not have considered before (6) and a requirement to focus on a single objective of the research (11), resulting in a practical benefits to not just the individual attendees, but also to the organisations they represent (24). By adding a critical infrastructure issue into a public health response scenario, indirect impacts to the resiliency of these vital areas can be assessed and response plans can be developed. Further research into this area is important to explore the links between organisational resilience and disaster resilience to ensure impacts to critical infrastructure are better understood (29).

This study has limitations, primarily on the sample size used in the study, as only 58% of attendees completed the survey, the majority of whom were Australians. Whilst most workshop attendees were from high income countries, it was nevertheless balanced by including views from low- and middle-income countries on account of support provided to participants to enable their attendance. This study was strengthened by the involvement of the Fiji Ministry of Health which provided assistance in the development of materials used in the exercise, cultural awareness and appropriateness of responses used in the scenario. While coding and thematic analysis were conducted by one investigator (JA) and validated by another (DH) – potentially introducing coding bias – there were no significant areas of difference of opinion between JA and DH during the process. Additionally, codes and themes were triangulated with the quantitative data which showed broad agreement across the study and both investigators assessed that thematic saturation was achieved approximately with around 20 surveys.

This exercise highlighted the value of including a wide range of attendees across both sectors and nationalities and addressing a far broader set of considerations across multiple sectors. While there were comments from participants about the use of a Pacific centred scenario over other options, this did not impact on the reported value of the exercise by the participants. The value of a scenario that drew on the backgrounds of a diverse range of participants was repeatedly highlighted by participants. Evaluation of exercises and scenarios can inform the planning of future exercises for maximal benefit to diverse groups of stakeholders. Participants found the most valuable aspects of exercises were the opportunities to hear from different perspectives, particularly from different nationalities and fields of occupation. Diversity of perspectives and experiences helped participants to challenge and critically assess their own capabilities in pandemic management and prepare for such high-impact events.

Acknowledgements

We would like to acknowledge Bavarian Nordic and Emergent Biosolutions for their generous support of this activity.

References

1. Delaney, M. H. , Lucero, F. P. , Maves, C. R. , Lawler, V. J. , Maddry, K. J. , Biever, A. K. , Womble, G. S. , Coffman, V. R. & Murray KC. Ebola Virus Disease Simulation Case Series: : Patient With Ebola Virus Disease in the Prodromal Phase of Illness (Scenario 1), the “Wet” Gastrointestinal Phase of Illness (Scenario 2), and the Late, Critically Ill Phase of Disease (Scenario 3). Simul Healthc J Soc Simul Healthc. 2016;11(2):106–16. DOI: https://doi.org/10.1097/SIH.0000000000000115

2. Uden-Holman, T. , Bedet, J. , Walkner, L. & Abd-Hamid HN. Adaptive Scenarios: A Training Model for Today’s Public Health Workforce. J Public Heal Manag Pract. 2014;20:4–48. DOI: https://doi.org/10.1097/PHH.0000000000000067

3. Ericsson KA. Deliberate practice and acquisition of expert performance: A general overview. Acad Emerg Med. 2008;15(11):988–94. DOI: https://doi.org/10.1111/j.1553-2712.2008.00227.x

4. Ericsson KA. Deliberate Practice and the Acquisition and Maintenance of Expert Performance in Medicine and Related Domains. Acad Med. 2004;79(10):S70–81. DOI: https://doi.org/10.1097/00001888-200410001-00022

5. Ericsson KA, Lehmann AC. Expert and Exceptional Performance: Evidence of Maximal Adaptation to Task Constraints. Annu Rev Psychol [Internet]. 1996;47(1):273–305. DOI: https://doi.org/10.1146/annurev.psych.47.1.273

6. High EH, Lovelace KA, Gansneder BM, Strack RW, Callahan B, Benson P. Promoting Community Preparedness: Lessons Learned From the Implementation of a Chemical Disaster Tabletop Exercise. Health Promot Pract. 2010;11(3):310–9. DOI: https://doi.org/10.1177/1524839908325063

7. Prieto DM, Das TK, Savachkin AA, Uribe A, Izurieta R, Malavade S. A systematic review to identify areas of enhancements of pandemic simulation models for operational use at provincial and local levels. BMC Public Health. 2012;12(1):251. DOI: https://doi.org/10.1186/1471-2458-12-251

8. MacIntyre CR, Seccull A, Lane JM, Plant A. Development of a Risk-Priority Score for Category A Bioterrorism Agents as an Aid for Public Health Policy. Mil Med. 2006;171(7):589–94. DOI: https://doi.org/10.7205/MILMED.171.7.589

9. Araz OM, Jehn M, Lant T, Fowler JW. A New Method of Exercising Pandemic Preparedness Through an Interactive Simulation and Visualization. J Med Syst. 2012 Jun;36(3):1475–83. DOI: https://doi.org/10.1007/s10916-010-9608-7

10. Oshitani H, Kamigaki T, Suzuki A. Major issues and challenges of influenza pandemic preparedness in developing countries. Emerg Infect Dis. 2008 Jun;14(6):875–80. DOI: https://doi.org/10.3201/eid1406.070839

11. Dausey DJ, Buehler JW, Lurie N. Designing and conducting tabletop exercises to assess public health preparedness for manmade and naturally occurring biological threats. BMC Public Health. 2007 May;7(1):92. DOI: https://doi.org/10.1186/1471-2458-7-92

12. Dausey, David J., Nicole Lurie, Alexis Diamond, Barbara Meade, Roger C. Molander, Karen A. Ricci, Michael A. Stoto WJ. Bioterrorism Preparedness Training and Assessment Exercises for Local Public Health Agencies. Bioterrorism Preparedness Training and Assessment Exercises for Local Public Health Agencies. Santa Monica; 2005.

13. Taylor JL, Roup BJ, Blythe D, Reed GK, Tate TA, Moore KA. Pandemic influenza preparedness in Maryland: improving readiness through a tabletop exercise. Biosecur Bioterror. 2005;3. DOI: https://doi.org/10.1089/bsp.2005.3.61

14. Smith BT, Inglesby T V, Brimmer E, Borio L, Franco C, Gronvall GK, et al. Navigating the storm: report and recommendations from the Atlantic Storm exercise. Biosecur Bioterror. 2005;3. DOI: https://doi.org/10.1089/bsp.2005.3.256

15. Sundelius B, Grönvall J. Strategic Dilemmas of Biosecurity in the European Union. Biosecurity Bioterrorism Biodefense Strateg Pract Sci. 2004;2(1):17–23. DOI: https://doi.org/10.1089/153871304322964309

16. Ostfield ML. Bioterrorism as a Foreign Policy Issue. SAIS Rev Int Aff. 2004;24(1):131–46. DOI: https://doi.org/10.1353/sais.2004.0017

17. Scales L, Sullivan GM. How do you analyze metagenomics data. 2013;(December):541–2.

18. Test MWU. Using SPSS for Ordinally Scaled Data : Mann-Whitney U , Sign Test , and Wilcoxon Tests. 2008;1–11.

19. Dewinter J, Dodou D. FivePoint Likert Items: t test versus MannWhitneyWilcoxon. 2010;(September).

20. NVivo Qualitative Data Analysis Software Professional. QRS international Pty Ltd; 2018.

21. Heaton JPW, Lording D, Liu SN, Litonjua AD, Guangwei L, Kim SC, et al. Using thematic analysis in psychology. Int J Impot Res. 2001;13(6):317–21. DOI: https://doi.org/10.1038/sj.ijir.3900760

22. Creswell JW, Clark VLP. Designing & Conducting Mixed Methods Research. 2011;1–417.

23. Anderson C. Presenting and evaluating qualitative research. Am J Pharm Educ. 2010;74(8). DOI: https://doi.org/10.5688/aj7408141

24. Skryabina E, Reedy G, Amlot R, Jaye P, Riley P. What is the value of health emergency preparedness exercises? A scoping review study. Int J DISASTER RISK Reduct. PO BOX 211, 1000 AE AMSTERDAM, NETHERLANDS: ELSEVIER SCIENCE BV; 2017 Mar;21:274–83.

25. FL Department of Health - Pasco County. Infection Management , Prevention, and Control Training Responding to Outbreaks of Healthcare - Associated Infections : A Table - Top Exercise for a Whole Community Approach. 2016.

26. Center for Health Policy. Public Health Emergency Exercise Toolkit - Planning, Designing, Conducting, and Evaluating Local Public Health Emergency Exercises. 2006.

27. Department of Health - Australian Government. Emergency Response Plan for Communicable Disease Incidents of National Significance : National Arrangements ( National CD Plan ). 2018.

28. Australian Government - Department of Health and Ageing. Australian Health Management Plan for Pandemic Influenza. 2014.

29. Commonwealth of Australia. Critical Infrastructure Resilience Strategy: Plan. 2015.