Application Validation Methods in EHR Evaluation Models: A Systematic Literature Review

Ebnehoseini, Zahra; Meraji, Marziyhe; Tabesh, Hamed

doi:10.30483/rijm.2023.254461.1277

Application Validation Methods in EHR Evaluation Models: A Systematic Literature Review

Document Type : Review Article/ Systematic Review Article/ Meta Analysis

Authors

¹ Health Information Technology unit of Economic Health Department, Mashhad University of Medical Sciences, Mashhad, Iran

² Department of Health Information Technology, School of Paramedical Sciences, Mashhad University of Medical Sciences, Mashhad, Iran

³ Department of Medical Informatics, Faculty of Medicine, Mashhad University of Medical Sciences, Mashhad, Iran ‎

10.30483/rijm.2023.254461.1277

Abstract

Background: Over the last few decades, several theoretical frameworks have been proposed to evaluate electronic health records (EHRs). These frameworks provide a theoretical basis for assessing the impact and outcomes of technology adoption in healthcare settings. This can help identify areas for improvement and ensure that EHRs effectively support healthcare delivery and patient care.

Objectives: The purpose of this study is to present a comprehensive review of the use of validation methods in Electronic health records.

Methods: Out of a total of 62 EHR evaluation frameworks in our previous literature review, at the final stage, 34 relevant articles were included for analysis. Variables such as participants and study setting, analysis software, data gathering methods, missing data, and outlier handling, theoretical basis models to develop the EHR evaluation model, the relationship between variables of the EHR evaluation models with evaluation items, sampling technique and sample size reliability assessment methods and values, and statistical validation methods and criteria values were extracted.

Results: Among the 34 papers that disclosed the validation methods utilized, the most widely used technique was Structural Equation Modeling (SEM), employed in 26.5% of the studies. Other methods utilized were Confirmatory Factor Analysis (CFA), and Exploratory Factor Analysis (EFA). A reliability assessment was performed in 82% of the articles. Cronbach's alpha and composite reliability (CR) were popular reliability (internal validation) methods on identified papers.

Conclusion: It is our belief that the results of this study can assist researchers in examining and modifying EHR evaluation frameworks to suit their specific needs. Additionally, we believe that our findings serve as a solid foundation for the creation of new EHR evaluation frameworks. Furthermore, we recommend that researchers utilize the findings presented in this article to enhance the implementation and utilization of SEM, CFA, and EFA methods in EHR evaluation models.

Keywords

Open Access Policy: This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited. To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/

References

1. Nykänen P, Brender J, Talmon J, de Keizer N, Rigby M, Beuscart-Zephir MC, Ammenwerth E. Guideline for good evaluation practice in health informatics (GEP-HI). Int J Med Inform. 2011;80(12):815-27. doi: 10.1016/j.ijmedinf.2011.08.004. [PubMed: 21920809]. https://doi.org/10.1016/j.ijmedinf.2011.08.004 PMid:21920809

2. Sockolow PS, Crawford PR, Lehmann HP. Health services research evaluation principles. Methods Inf Med. 2012;51(02):122-30. doi:10.3414/me10-01-0066. [PubMed:22311125]. https://doi.org/10.3414/ME10-01-0066 PMid:22311125

3. Rahimi B, Nadri H, Afshar HL, Timpka T. A systematic review of the technology acceptance model in health informatics. Appl Clin Inform. 2018;9(03):604-34. doi: 10.1055/s-0038-1668091. [PubMed: 30112741]. https://doi.org/10.1055/s-0038-1668091 PMid:30112741 PMCid:PMC6094026

4. Ebnehoseini Z, Tabesh H, Jangi MJ, Deldar K, Mostafavi SM, Tara M. Investigating evaluation frameworks for electronic health record: a literature review. Open Access Maced J Med Sci. 2021;9(E):8-25. doi: 10.3889/oamjms.2021.3421. https://doi.org/10.3889/oamjms.2021.3421

5. Dash G, Paul J. CB-SEM vs PLS-SEM methods for research in social sciences and technology forecasting. Technol Forecast Soc Change. 2021;173:121092. doi: 10.1016/j.techfore.2021.121092. https://doi.org/10.1016/j.techfore.2021.121092

6. Deng, L., M. Yang and K.M. Marcoulides, Structural equation modeling with many variables: A systematic review of issues and developments. Front Psychol, 2018.9:580. doi: 10.3389/fpsyg.2018.00580. [PubMed: 29755388]. https://doi.org/10.3389/fpsyg.2018.00580 PMid:29755388 PMCid:PMC5932371

7. Ghanbar H, Rezvani R. Structural Equation Modeling in L2 Research: A Systematic Review. International Journal of Language Testing. 2023 Mar 1;13(Special Issue):79-108. doi: 10.22034/ijlt.2023.381619.1224.

8. Lambooij, M.S., H.W. Drewes and F. Koster, Use of electronic medical records and quality of patient data: different reaction patterns of doctors and nurses to the hospital organization. BMC Med Inform Decis Mak, 2017.17(1):1-11. doi:10.1186/s12911-017-0412-x. [PubMed: 28187729]. https://doi.org/10.1186/s12911-017-0412-x PMid:28187729 PMCid:PMC5303309

9. Mohd Salleh, M.I., N. Zakaria and R. Abdullah, The influence of system quality characteristics on health care providers' performance: Empirical evidence from Malaysia. Journal of infection and public health, 2016.9(6): 698-707. doi: 10.1016/j.jiph.2016.09.002. [PubMed:27659115]. https://doi.org/10.1016/j.jiph.2016.09.002 PMid:27659115

10. Kim, S., K.H. Lee, H. Hwang and S. Yoo, Analysis of the factors influencing healthcare professionals' adoption of mobile electronic medical record (EMR) using the unified theory of acceptance and use of technology (UTAUT) in a tertiary hospital. Med Inform Decis Mak, 2016.16(1). doi: 10.1186/s129117. [PubMed: 26831123]. https://doi.org/10.1186/s12911-016-0249-8 PMid:26831123 PMCid:PMC4736616

11. Liu, C.F. and T.J. Cheng, Exploring critical factors influencing physicians' acceptance of mobile electronic medical records based on the dual-factor model: a validation in Taiwan. BMC Med Inform Decis Mak, 2015. 15:4. doi:10.1186/s12911-014-0125-3. [PubMed: 25889506]. https://doi.org/10.1186/s12911-014-0125-3 PMid:25889506 PMCid:PMC4333263

12. Sintonen, S., K. Mäkelä and R. Miettinen, User acceptance of electronic health records: A post-implementation study. International Journal of Healthcare Technology and Management, 2015.15(2):162-75. doi:10.1504/IJHTM.2015.074556. https://doi.org/10.1504/IJHTM.2015.074556

13. Gan, Q., Is the adoption of electronic health record system "contagious"? Health Policy Technol, 2015. 4(2): 107-112. doi:10.1016/j.hlpt.2015.02.009. https://doi.org/10.1016/j.hlpt.2015.02.009

14. Kralj, D., J. Kern, S. Tonkovic and M. Koncar, Development of the quality assessment model of EHR software in family medicine practices: research based on user satisfaction. J Innov Health Inform, 2015.22(3): 340-58. doi:10.14236/jhi.v22i3.158. [PubMed:26577425]. https://doi.org/10.14236/jhi.v22i3.158 PMid:26577425

15. Wang, J.Y., H.Y. Ho, J.D. Chen, S. Chai, C.J. Tai and Y.F. Chen, Attitudes toward inter-hospital electronic patient record exchange: discrepancies among physicians, medical record staff, and patients. BMC Health Serv Res, 2015. 15: p. 264. doi:10.1186/s12913-015-0896-y. [PubMed: 26162748]. https://doi.org/10.1186/s12913-015-0896-y PMid:26162748 PMCid:PMC4499194

16. Michel-Verkerke, M.B., R.A. Stegwee and T.A.M. Spil, The six P's of the next step in electronic patient records in the Netherlands. Health Policy Technol, 2015;4(2): 137-43. doi: 10.1016/j.hlpt.2015.02.011. https://doi.org/10.1016/j.hlpt.2015.02.011

17. Hysong, S.J., C. Spitzmuller, D. Espadas, D.F. Sittig and H. Singh, Electronic alerts and clinician turnover: the influence of user acceptance. Am J Manag Care, 2014;20(17): p. SP520-SP530. [PubMed:25811826].

18. Gagnon, M.P., K. Ghandour el, P.K. Talla, D. Simonyan, G. Godin, M. Labrecque, et al., Electronic health record acceptance by physicians: testing an integrated theoretical model. J Biomed Inform, 2014;48:17-27. doi:10.1016/j.jbi.2013.10.010. [PubMed: 24184678]. https://doi.org/10.1016/j.jbi.2013.10.010 PMid:24184678

19. Iqbal, U., C.H. Ho, Y.C. Li, P.A. Nguyen, W.S. Jian and H.C. Wen, The relationship between usage intention and adoption of electronic health records at primary care clinics. Computer Methods and Programs in Biomedicine, 2013;112(3): 731-737. doi:10.1016/j.cmpb.2013.09.001. [PubMed:24091088]. https://doi.org/10.1016/j.cmpb.2013.09.001 PMid:24091088

20. Kuo, K.M., C.F. Liu and C.C. Ma, An investigation of the effect of nurses' technology readiness on the acceptance of mobile electronic medical record systems. Med Inform Decis Mak, 2013;13(1). doi:10.1186/1472-6947-13-88. [PubMed:23938040]. https://doi.org/10.1186/1472-6947-13-88 PMid:23938040 PMCid:PMC3750758

21. Lu, C.H., J.L. Hsiao and R.F. Chen, Factors determining nurse acceptance of hospital information systems. Comput Inform Nurs, 2012;30(5): 257-64. doi:10.1097/NCN.0b013e318224b4cf. [PubMed:22228251]. https://doi.org/10.1097/NCN.0b013e318224b4cf PMid:22228251

22. Aggelidis, V.P. and P.D. Chatzoglou, Hospital information systems: measuring end user computing satisfaction (EUCS). J Biomed Inform, 2012;45(3): 566-79. doi:10.1016/j.jbi.2012.02.009. [PubMed:22426283]. https://doi.org/10.1016/j.jbi.2012.02.009 PMid:22426283

23. Chen, R.F. and J.L. Hsiao, An investigation on physicians' acceptance of hospital information systems: a case study. Int J Med Inform, 2012;81(12): 810-20. doi:10.1016/j.ijmedinf.2012.05.003. [PubMed:22652011]. https://doi.org/10.1016/j.ijmedinf.2012.05.003 PMid:22652011

24. Lin, C., I.C. Lin and J. Roan, Barriers to physicians' adoption of healthcare information technology: An empirical study on multiple hospitals. J Med Syst, 2012;36(3): 1965-77. doi:10.1007/s10916-011-9656-7. [PubMed:21336605]. https://doi.org/10.1007/s10916-011-9656-7 PMid:21336605

25. Carayon, P., R. Cartmill, M.A. Blosky, R. Brown, M. Hackenberg, P. Hoonakker, et al., ICU nurses' acceptance of electronic health records. Journal of the American Medical Informatics Association : JAMIA, 2011;18(6): 812-9. doi:10.1136/amiajnl-2010-000018. [PubMed:21697291]. https://doi.org/10.1136/amiajnl-2010-000018 PMid:21697291 PMCid:PMC3197984

26. Tilahun, B. and F. Fritz, Modeling antecedents of electronic medical record system implementation success in low-resource setting hospitals. Med Inform Decis Mak, 2015;15: 61. doi:10.1186/s12911-015-0192-0. [PubMed:26231051]. https://doi.org/10.1186/s12911-015-0192-0 PMid:26231051 PMCid:PMC4522063

27. Steininger, K. and B. Stiglbauer, EHR acceptance among Austrian resident doctors. Health Policy Technol, 2015;4(2): p. 121-130. doi:10.1016/j.hlpt.2015.02.003. https://doi.org/10.1016/j.hlpt.2015.02.003

28. Kowitlawakul Y, S.W. Chan, J. Pulcini, W Wang, Factors influencing nursing students' acceptance of electronic health records for nursing education (EHRNE) software program. Nurse education today, 2015;35(1): 189-94. doi:10.1016/j.nedt.2014.05.010. [PubMed:24947068]. https://doi.org/10.1016/j.nedt.2014.05.010 PMid:24947068

29. Hsieh, P.J., Physicians' acceptance of electronic medical records exchange: An extension of the decomposed TPB model with institutional trust and perceived risk. Int J Med Inform, 2015;84(1): 1-14. doi:10.1016/j.ijmedinf.2014.08.008. [PubMed:25242228]. https://doi.org/10.1016/j.ijmedinf.2014.08.008 PMid:25242228

30. Abdekhoda, M., M. Ahmadi, M. Gohari and A. Noruzi, The effects of organizational contextual factors on physicians' attitude toward adoption of Electronic Medical Records. J Biomed Inform, 2015. 53: p. 174-179. doi:10.1016/j.jbi.2014.10.008. [PubMed: 25445481]. https://doi.org/10.1016/j.jbi.2014.10.008 PMid:25445481

31. Messeri, P., S. Khan, M. Millery, A. Campbell, J. Merrill, S. Shih, et al., An information systems model of the determinants of electronic health record use. Appl Clin Inform, 2013; 4(2): p. 185-200. doi:10.4338/aci-2013-01-ra-0005. [PubMed:23874357]. https://doi.org/10.4338/ACI-2013-01-RA-0005 PMid:23874357 PMCid:PMC3716425

32. Hsu, S.C., C.F. Liu, R.H. Weng and C.J. Chen, Factors influencing nurses' intentions toward the use of mobile electronic medical records. Comput Inform Nurs, 2013;31(3): 124-32. doi:10.1097/NXN.0b013e318270100b. [PubMed:23114391]. https://doi.org/10.1097/NXN.0b013e318270100b PMid:23114391

33. Leblanc, G., M.P. Gagnon and D. Sanderson, Determinants of primary care nurses' intention to adopt an electronic health record in their clinical practice. Comput Inform Nurs, 2012;30(9): 496-502. doi:10.1097/NXN.0b013e318257db17. [PubMed:22592453]. https://doi.org/10.1097/NXN.0b013e318257db17 PMid:22592453

34. Chen, R.F. and J.L. Hsiao, An empirical study of physicians' acceptance of hospital information systems in Taiwan. Telemedicine journal and e-health : Telemed J E Health, 2012;18(2): 120-5. doi:10.1089/tmj.2011.0081. [PubMed:22283362]. https://doi.org/10.1089/tmj.2011.0081 PMid:22283362

35. Hsiao, J.L., H.C. Chang and R.F. Chen, A study of factors affecting acceptance of hospital information systems: a nursing perspective. J Nurs Res. 2011;19(2): p. 150-60. doi:10.1097/JNR.0b013e31821cbb25. [PubMed:21586992]. https://doi.org/10.1097/JNR.0b013e31821cbb25 PMid:21586992

36. Sicotte, C., G. Pare, K.K. Bini, M.P. Moreault and G. Laverdure, Virtual organization of hospital medical imaging: a user satisfaction survey. J Digit Imaging, 2010;23(6): 689-700. doi:10.1007/s10278-009-9220-x. [PubMed:19588196]. https://doi.org/10.1007/s10278-009-9220-x PMid:19588196 PMCid:PMC3046687

37. Erlirianto LM, AHN Ali , A Herdiyanti. The Implementation of the Human, Organization, and Technology-Fit (HOT-Fit) Framework to Evaluate the Electronic Medical Record (EMR) System in a Hospital. Procedia Comput Sci, 2015;72: 580-7. doi:10.1016/j.procs.2015.12.166. https://doi.org/10.1016/j.procs.2015.12.166

38. Ho, C.H., H.C. Wene, C.M. Chu, Y.S. Wu and J.L. Wang, Importance-satisfaction analysis for primary care physicians' perspective on EHRs in Taiwan. Int J Environ Res Public Health, 2014;11(6): 6037-51. doi:10.3390/ijerph110606037. [PubMed:24914640]. https://doi.org/10.3390/ijerph110606037 PMid:24914640 PMCid:PMC4078564

39. Devine, E.B., R. Patel, D.R. Dixon and S.D. Sullivan, Assessing attitudes toward electronic prescribing adoption in primary care: a survey of prescribers and staff. Inform Prim Care, 2010;18(3): 177-87. [PubMed:21396241]. https://doi.org/10.14236/jhi.v18i3.770 PMid:21396241

40. Otieno GO, T Hinako. Motohiro K. Daisuke , N Keiko. Measuring effectiveness of electronic medical records systems: towards building a composite index for benchmarking hospitals. Int J Med Inform, 2008;77(10): 657-69. doi:10.1016/j.ijmedinf.2008.01.002. [PubMed: 18313352]. https://doi.org/10.1016/j.ijmedinf.2008.01.002 PMid:18313352

41. Otieno, O.G., H. Toyama, M. Asonuma, M. Kanai-Pak and K. Naitoh, Nurses' views on the use, quality and user satisfaction with electronic medical records: questionnaire development. J Adv Nurs, 2007;60(2): 209-19. doi:10.1111/j.1365-2648.2007.04384.x. [PubMed:17877568]. https://doi.org/10.1111/j.1365-2648.2007.04384.x PMid:17877568

42. Cho KW, SK Bae, JH. Ryu, KN Kim, CH An ,YM Chae, Performance evaluation of public hospital information systems by the information system success model. Healthc Inform Res, 2015;21(1): 43-8. doi:10.4258/hir.2015.21.1.43. [PubMed:25705557]. https://doi.org/10.4258/hir.2015.21.1.43 PMid:25705557 PMCid:PMC4330199

43. Bahadori MK, Alimohammadzadeh K. Abdolkarimi, RRavangard, Factors affecting physicians' attitudes towards the implementation of electronic health records using structural equation modeling modeling (SEM). Shiraz E Medical Journal, 2017; 18(11). doi: 10.5812/semj.13729. https://doi.org/10.5812/semj.13729

44. Beran, T.N. and C. Violato, Structural equation modeling in medical research: a primer. BMC Res Notes, 2010; 3(1): 1-10. doi: 10.1186/1756-0500-3-267. [PubMed:20969789]. https://doi.org/10.1186/1756-0500-3-267 PMid:20969789 PMCid:PMC2987867

45. Shaheen, F., N. Ahmad, M. Waqas, A. Waheed and O. Farooq, Structural equation modeling (SEM) in social sciences & medical research: a guide for improved analysis. Int. J Acad Res Bus Soc Sci. 2017;7(5): 132-43. doi: 10.6007/IJARBSS/v7-i5/2882. https://doi.org/10.6007/IJARBSS/v7-i5/2882

46. Nguyen, V.T. and C.T.H. Nguyen, A systematic review of structural equation modeling in augmented reality applications. arXiv preprint arXiv:2301.11811, 2023. doi:10.48550/arXiv.2301.11811.

47. Sharif, S.P., I. Mostafiz and V. Guptan, A systematic review of structural equation modelling in nursing research. Nurse Res, 2019;30(2). doi: 10.7748/nr.2018.e1577. [PubMed:30207432]. https://doi.org/10.7748/nr.2018.e1577 PMid:30207432

48. MacCallum, RC., JT. Austin. Applications of structural equation modeling in psychological research. Annu Rev Psychol, 2000;51(1): 201-26. doi:10.1146/annurev.psych.51.1.201. [PubMed:10751970]. https://doi.org/10.1146/annurev.psych.51.1.201 PMid:10751970

49. Ismael, K. and S. Duleba, A Systematic Review of the Latest Advancements on Structural Equation Modelling (SEM) Technique Focusing on Applications in Transportation Planning. Period. Polytech. Transp. Eng, 2022;50(4): 336-43. doi:10.3311/PPtr.17523. https://doi.org/10.3311/PPtr.17523

50. Dragan D, D Topolšek. Introduction to structural equation modeling: review, methodology and practical applications. The International Conference on Logistics & Sustainable Transport. 2014.

51. Venkatesh V, MG Morris, GB Davis , FD Davis. User acceptance of information technology: Toward a unified view. MIS quarterly. 2003: 425-78. doi: 10.2307/30036540. https://doi.org/10.2307/30036540

52. Kruse CS, K Kothman, K Anerobi, L Abanaka. Adoption Factors of the Electronic Health Record: A Systematic Review. JMIR medical informatics, 2016;4(2): e19. doi:10.2196/medinform.5525. [PubMed:27251559]. https://doi.org/10.2196/medinform.5525 PMid:27251559 PMCid:PMC4909978

Razavi International Journal of Medicine

Article View: 470
PDF Download: 244

Application Validation Methods in EHR Evaluation Models: A Systematic Literature Review

References

Volume 11, Issue 3
August 2023
Pages 1-11

Files

Share

How to cite

Statistics

Application Validation Methods in EHR Evaluation Models: A Systematic Literature Review

References

Volume 11, Issue 3August 2023Pages 1-11

Files

Share

How to cite

Statistics

Volume 11, Issue 3
August 2023
Pages 1-11