The Australian Institute of Health and Welfare (AIHW), as the Australian Collaborating Centre for the WHO Family of International Classifications, coordinated three phases of the Australian field testing of the ICD-11 MMS for morbidity purposes between 2016 and 2018. Phase 3 was the final phase of WHO-led field testing and was conducted in Australia in March 2018.

In the lead up to Phase 3 testing, the WHO had made significant changes and updates to both the ICD-11 tooling environment (the Coding Tool) and classification. Some of the improvements included: global word substitutions, flexible searching options (less strict searching rules) and postcoordination combinations added to search results for laterality, acute/chronic and anatomical site.

Purpose
The key objective of the Phase 3 testing was to determine if updates to the Coding Tool were deemed ‘improvements’, however the AIHW utilised this phase of testing to also measure the level of understanding Australian participants had of ICD-11, noting that this had been flagged as a potential issue in previous rounds of testing in Australia.

Participants
The Phase 3 testing cohort consisted of 22 participants who had demonstrated experience and exposure to ICD-11 due to their prior participation in Australian field tests. The participants were considered highly experienced clinical coders who represented both public and private hospitals as well as Australian state and territory health departments. 

Testing process
The testing process involved the reuse of diagnostic statements from the Phase 2 field testing where there had been low agreement between clinical coders in terms of code assignment. This reuse of diagnostic statements would allow comparative assessment of the differences in the tooling environment between Phase 2 and Phase 3. In total, Phase 3 consisted of 80 diagnostic statements.

In preparation for Phase 3 testing, participants attended a one-day workshop where their time was split between an education session on changes and improvements to the Coding Tool, followed by ‘time to code’ diagnostic statements. Participants were also asked to complete a survey before and after the workshop to gauge their learning experience and level of understanding.

The testing process consisted of two steps. Step one: participants used the Coding Tool to select the appropriate ICD-11 MMS code(s) for each diagnostic statement and entered the codes into the WHO’s field testing system, ICD-FiT. Step two: participants then evaluated the difficulty in finding and/or applying the codes and entered this information into ICD-FiT. Participants were also required to comment on whether the codes were the ‘best fit’ for the diagnostic statement in terms of specificity and ambiguity. 

Unlike in previous phases of testing, dual coding (i.e. coding in both ICD-10 and ICD-11) was not performed; participants were only required to code a diagnostic statement in ICD-11 MMS.

Results
Phase 3 participants were able to test, in aggregate, more than 1,100 diagnostic statements. 
Table 1 shows the difference in participant ratings of difficulty, ambiguity, specificity and time to code between Phase 2 and Phase 3 for the 80 cases that were identical between the phases. 

Table 1: Changes in the rating of different aspects of code assignment

The comparative results suggest that the diagnostic statements in Phase 3 were more difficult, however coders thought that the cases were less ambiguous and more often commented that ICD-11 had the correct level of specificity. The average coding time was also shorter.

Table 2 provides a summary of the key errors made by participants. Overall, there was a 62% reduction in identified errors when comparing Phase 2 and Phase 3 coded results.

Table 2: Comparison of errors in coding

The pre- and post-workshop survey results indicated that participants felt more confident in using the Coding Tool (81% before, 100% after) and using the postcoordination/clustering mechanism (56% before, 94% after) after receiving the education session. Participants also expressed a preference for face-to-face education rather than via webinar, which is what had been used in previous phases of testing.

Conclusion
The Phase 3 results highlighted that the improvements to the Coding Tool and to the classification had greatly assisted participants in navigating ICD-11 and resulted in reduced coding errors. It was also observed that through appropriate training and education, delivered at the right level and via the right medium, participant acceptance and competency in using the ICD-11 MMS tools was greatly improved and subsequently influenced the overall testing results. 

Acknowledgements
The AIHW would like to acknowledge the valuable support of WHO’s Lindy Best in facilitating the Phase 3 testing process.