The The Barriers in Online Assessment for Professional Certification: A Case Study In Indonesia

Lena  Citra Manggalasari; Anne Jantos; Thomas Koehler; Yuliastuti Yuliastuti; Yureana Wijayanti; Oki Setyandito; Sutarto Sutarto; Serafin Wisni Septiarti

doi:10.23887/jere.v7i1.53839

Authors

Lena Citra Manggalasari Technische Universitat Dresden
Anne Jantos Technische Universität Dresden
Thomas Koehler Technische Universität Dresden
Yuliastuti Yuliastuti Bina Nusantara University
Yureana Wijayanti Bina Nusantara University https://orcid.org/0000-0002-9341-9681
Oki Setyandito Bina Nusantara University https://orcid.org/0000-0002-0147-8757
Sutarto Sutarto Yogyakarta State University
Serafin Wisni Septiarti Yogyakarta State University

DOI:

https://doi.org/10.23887/jere.v7i1.53839

Keywords:

Online Assessment, professional certification, assessor, MICMAC

Abstract

Certification exams conducted online must be of the same quality as those conducted on-site. To achieve this, it is necessary to conduct a study in which assessors need to identify obstacles in implementing online certification examinations which have been implemented during the Covid-19 pandemic. The study aims to analyze the perspective of the assessors to identify the barrier factors, understand the interrelation among them, and propose improvements to the online application process. The method applied was the MICMAC (Matrix of Cross-impact Multiplications Applied to Classification), a structural model, to analyze the variables based on their influence and dependence driving force. A case study was conducted on the Indonesia Hydraulic Engineer Association. The data was collected from a focus-group discussion with the assessors. The result revealed that the barrier factors consist of technical aspects, such as internet connection delay, hackers and viruses, and unverified e-document; and social aspects, such as questioning skills limitation, non-verbal communication skills limitation, and environment distraction. The study found that internet connection delay and questioning skill limitations are the barriers with the highest influence beyond other barriers in the online assessment process. Also, the questioning skills limitation factor has the highest dependency on other barriers. Hence, it can be proposed that the curricula of the assessors’ training and/or certification program, should be incorporated with the questioning skill in an online assessment environment.

References

Abdulghafor, R., Abdelmohsen, A., Turaev, S., Ali, M. A. H., & Wani, S. (2022). An Analysis of Body Language of Patients Using Artificial Intelligence. Healthcare, 10(12), 2504. https://doi.org/10.3390/healthcare10122504.

Ahmad, N., Quadri, N., Qureshi, M., & Alam, M. (2018). Relationship Modeling of Critical Success Factors for Enhancing Sustainability and Performance in E-Learning. Sustainability, 10(12), 4776. https://doi.org/10.3390/su10124776.

Akyazi, T., Alvarez, I., Alberdi, E., Oyarbide-Zubillaga, A., Goti, A., & Bayon, F. (2020). Skills needs of the civil engineering sector in the european union countries: Current situation and future trends. Applied Sciences (Switzerland), 10(20), 1–24. https://doi.org/10.3390/app10207226.

Al-Rawi, S. S., Alheeti, K. M. A., Abdul-Kader, S., & Al-Ani, M. S. (2022). Cheating monitoring and detection in examination from face movement recognition. AIP Conference Proceeding, 020004. https://doi.org/10.1063/5.0115539.

Alruwais, N., Wills, G., & Wald, M. (2018). Advantages and Challenges of Using e-Assessment. International Journal of Information and Education Technology, 8(1), 34–37. https://doi.org/10.18178/ijiet.2018.8.1.1008.

Arifa, F. N., & Prayitno, U. S. (2019). Peningkatan Kualitas Pendidikan: Program Pendidikan Profesi Guru Prajabatan dalam Pemenuhan Kebutuhan Guru Profesional di Indonesia. Aspirasi: Jurnal Masalah-Masalah Sosial, 10(1), 1–17. https://doi.org/10.22212/aspirasi.v7il.1084.

Barati, A. A., Azadi, H., Pour, M. D., Lebailly, P., & Qafori, M. (2019). Determining Key Agricultural Strategic Factors Using AHP-MICMAC. Sustainability, 11(14), 3947. https://doi.org/10.3390/su11143947.

Barrows, M., Clevenger, C. M., Abdallah, M., & Wu, W. (2020). Value of Certifications when Seeking Construction Employment. International Journal of Construction Education and Research, 16(1), 61–79. https://doi.org/10.1080/15578771.2019.1575936.

Beleulmi, S. (2022). Challenges of online assessment during Covid-19 Pandemic: An experience of Study Skills teachers. Afak for Science Journal, 7(2), 49–62. https://www.asjp.cerist.dz/en/downArticle/351/7/2/181.

Bhatt, S., Bhatt, A., & Thanki, S. (2021). Analysing the Key Enablers of Students’ Readiness for Online Learning: An Interpretive Structural Modeling Approach. In International Journal of Education and Development Using Information and Communication Technology (IJEDICT), 17. http://ijedict.dec.uwi.edu/viewarticle.php?id=2966.

Chen, H., Wang, S., & Li, Y. (2022). Aligning Engineering Education for Sustainable Development through Governance: The Case of the International Center for Engineering Education in China. Sustainability, 14(21), 14643. https://doi.org/10.3390/su142114643.

Crews, T. B., & Curtis, D. F. (2011). Online Course Evaluations: Faculty Perspective and Strategies for Improved Response Rates. Assessment and Evaluation in Higher Education, 36(7), 865–878. https://doi.org/10.1080/02602938.2010.493970.

Cruz, M. L., Saunders-Smits, G. N., & Groen, P. (2020). Evaluation of competency methods in engineering education: a systematic review. European Journal of Engineering Education, 45(5), 729–757. https://doi.org/10.1080/03043797.2019.1671810.

Damek, S., Söbke, H., Weise, F., & Reichelt, M. (2022). Teaching (Meta) Competences for Digital Practice Exemplified by Building Information Modeling Work Processes. Knowledge, 2(3), 452–464. https://doi.org/10.3390/knowledge2030027.

Doğan, N., Kibrislioğlu Uysal, N., Kelecioğlu, H., & Hambleton, R. K. (2020). An overview of e-assessment. In Hacettepe Egitim Dergisi. In Hacettepe Egitim Dergisi, 35, 1–5. https://doi.org/10.16986/HUJE.2020063669.

Dudzinski, M., Roszmann-Millican, M., & Sbank, K. (2000). Continuing Professional Development for Special Educators: Reforms and Implications for Unj"tversity Programs. Teacher Education and Special Education: The Journal of the Teacher Education Division of the Council for Exceptional Children, 23(2), 109–124. https://doi.org/10.1177/088840640002300205.

Ekaputri, D. A. (2020). Assessors’ Readiness in Implementing the Flexible Working Arrangement (FWA). Civil Service, 14(2), 1–10. https://jurnal.bkn.go.id/index.php/asn/article/view/245.

Farozin, M. (2019). Counselor professional identity of counselor profession education. Cakrawala Pendidikan, 38(1), 104–119. https://doi.org/10.21831/cp.v38i1.22515.

Gamage, K. A. A., Silva, E. K., & de Gunawardhana, N. (2020). Online Delivery and Assessment during COVID-19: Safeguarding Academic Integrity. Education Sciences, 10(11), 301. https://doi.org/10.3390/educsci10110301.

Hirudayaraj, M., Baker, R., Baker, F., & Eastman, M. (2021). Soft skills for entry-level engineers: What employers want. Education Sciences, 11(10). https://doi.org/10.3390/educsci11100641.

Hollister, K. K., & Berenson, M. L. (2009). Proctored Versus Unproctored Online Exams: Studying the Impact of Exam Environment on Student Performance. Decision Sciences Journal of Innovative Education, 7(1), 271–294. https://doi.org/10.1111/j.1540-4609.2008.00220.x.

Itani, M., & Srour, I. (2016). Engineering Students’ Perceptions of Soft Skills, Industry Expectations, and Career Aspirations. Journal of Professional Issues in Engineering Education and Practice, 142(1). https://doi.org/10.1061/(ASCE)EI.1943-5541.0000247.

Jiang, X., Lu, K., Xia, B., Liu, Y., & Cui, C. (2019). Identifying Significant Risks and Analyzing Risk Relationship for Construction PPP Projects in China Using Integrated FISM-MICMAC Approach. Sustainability, 11(19), 5206. https://doi.org/10.3390/su11195206.

Joshi, A., Virk, A., Saiyad, S., Mahajan, R., & Singh, T. (2020). Online assessment: Concept and applications. Journal of Research in Medical Education & Ethics, 10(2), 49. https://doi.org/10.5958/2231-6728.2020.00015.3.

Jung, S., Lee, S., & Yu, J. (2021). Identification and Prioritization of Critical Success Factors for Off-Site Construction Using ISM and MICMAC Analysis. Sustainability, 13(16), 8911. https://doi.org/10.3390/su13168911.

Kaddoura, S., & Gumaei, A. (2022). Towards effective and efficient online exam systems using deep learning-based cheating detection approach. Intelligent Systems with Applications, 16, 200153. https://doi.org/10.1016/j.iswa.2022.200153.

Karamat, J., Shurong, T., Ahmad, N., Waheed, A., & Khan, S. (2018). Barriers to Knowledge Management in the Health Sector of Pakistan. Sustainability, 10(11), 4155. https://doi.org/10.3390/su10114155.

Kharbat, F. F., & Abu Daabes, A. S. (2021). E-proctored exams during the COVID-19 pandemic: A close understanding. In Education and Information Technologies, 26(6), 6589–6605. https://doi.org/10.1007/s10639-021-10458-7.

Kusmiarto, K., Aditya, T., Djurdjani, D., & Subaryono, S. (2021). Digital transformation of land services in indonesia. A Readiness Assessment. Land, 10(2), 1–16. https://doi.org/10.3390/land10020120.

Leibur, T., Saks, K., & Chounta, I. A. (2021). Towards acquiring teachers’ professional qualification based on professional standards: Perceptions, expectations and needs on the application process. Education Sciences, 11(8). https://doi.org/10.3390/educsci11080391.

Linden, K., & Gonzalez, P. (2021). Zoom invigilated exams: A protocol for rapid adoption to remote examinations. British Journal of Educational Technology, 52(4), 1323–1337. https://doi.org/10.1111/bjet.13109.

Lobe, B., Morgan, D., & Hoffman, K. A. (2020). Qualitative Data Collection in an Era of Social Distancing. International Journal of Qualitative Methods, 19, 160940692093787. https://doi.org/10.1177/1609406920937875.

Luiz Neto, A., da Silva, L. F., & Penha, R. (2022). Sandbox of Competence: A Conceptual Model for Assessing Professional Competence. Administrative Sciences, 12(4), 182. https://doi.org/10.3390/admsci12040182.

Maruanaya, R. F., & Köhler, T. (2020). Student Self-Evaluation through German BLok Online Media in Teaching English for Tourism in Vocational High Schools. Proceeding The 3rd International Seminar on Language and Interdisciplinary Research on Language “Language and Its Prospect in the Future, 78–92. https://psb.unpatti.ac.id/wp-content/uploads/2021/03/Proceeding_ISELANG_2020_Final2.pdf.

Milone, A. S., Cortese, A. M., Balestrieri, R. L., & Pittenger, A. L. (2017). The impact of proctored online exams on the educational experience. Currents in Pharmacy Teaching and Learning, 9(1), 108–114. https://doi.org/10.1016/j.cptl.2016.08.037.

Montenegro-Rueda, M., Luque-de la Rosa, A., Sarasola, Sánchez-Serrano L., J., & Fernández-Cerero, J. (2021). Assessment in Higher Education during the COVID-19 Pandemic: A Systematic Review. Sustainability, 13(19), 10509. https://doi.org/10.3390/su131910509.

Nandal, V., Kumar, R., & Singh, S. K. (2019). Barriers identification and analysis of solar power implementation in Indian thermal power plants: An Interpretative Structural Modeling approach. Renewable and Sustainable Energy Reviews, 114, 109330. https://doi.org/10.1016/j.rser.2019.109330.

Naveed, Q. N., Qureshi, M. R. N., Alsayed, A. O., Ahmad, N., Sanober, S., & Shah, A. (2018). Assimilating E-Learning barriers using an interpretive structural modeling (ISM). 4th IEEE International Conference on Engineering Technologies and Applied Sciences, 1–7. https://doi.org/10.1109/ICETAS.2017.8277852.

Osuji, A. (2012). The Use of e-Assessments in The Nigerian Higher Education System. Turkish Online Journal of Distance Education, 13(4), 140–152. https://eric.ed.gov/?id=EJ1000419.

Pokhrel, S., & Chhetri, R. (2021). A Literature Review on Impact of COVID-19 Pandemic on Teaching and Learning. Higher Education for the Future, 8(1), 133–141. https://doi.org/10.1177/2347631120983481.

Rautela, S., Panackal, N., & Sharma, A. (2022). Modeling and analysis of barriers to ethics in online assessment by TISM and fuzzy MICMAC analysis. Asian Journal of Business Ethics, 1–28. https://doi.org/10.1007/s13520-022-00158-x.

Ribeiro, A. M., Arantes, A., & Cruz, C. O. (2022). Barriers to the Adoption of Modular Construction in Portugal: An Interpretive Structural Modeling Approach. Buildings, 12(10), 1509. https://doi.org/10.3390/buildings12101509.

Rusilowati, U., & Wahyudi, W. (2020). The Significance of Educator Certification in Developing Pedagogy, Personality, Social and Professional Competencies. 409(SoRes 2019), 446–451. https://doi.org/10.2991/assehr.k.200225.095.

Sultan, S., Morgan, R. L., Murad, M. H., Falck-Ytter, Y., Dahm, P., Schünemann, H. J., & Mustafa, R. A. (2020). A Theoretical Framework and Competency-Based Approach to Training in Guideline Development. Journal of General Internal Medicine, 35(2), 561–567. https://doi.org/10.1007/s11606-019-05502-9.

Tucker, B. P., & Parker, L. D. (2019). Researcher perceptions and choices of interview media: The case of accounting research. Accounting and Finance, 59(3), 1489–1517. https://doi.org/10.1111/acfi.12393.

Walters, J., Kaminsky, J., & Gottschamer, L. (2018). A systems analysis of factors influencing household solar PV adoption in Santiago, Chile. Sustainability (Switzerland), 10(4). https://doi.org/10.3390/su10041257.

Wijayanti, Y., Anda, M., Setyandito, O., Suangga, M., & Juliastuti. (2021). Analysis of Strategic Variables for Sustainable Infrastructure and Transportation in Rural Area of Serang Regency, Banten Province. IOP Conference Series: Earth and Environmental Science, 794(1). https://doi.org/10.1088/1755-1315/794/1/012053.