Formative Evaluation of Digital Learning Materials

Authors

  • Ni Made Sri Mertasari Universitas Pendidikan Ganesha, Singaraja, Indonesia
  • I Made Candiasa Universitas Pendidikan Ganesha, Singaraja, Indonesia

DOI:

https://doi.org/10.23887/jet.v6i3.44165

Keywords:

Evaluation, formative, learning materials, digital

Abstract

The use of digital learning materials through e-learning is expected to give students experience to be able to learn themselves in the industrial era 4.0. This study aims to formulate an evaluation method for digital learning materials that is relevant to the development model used. A qualitative approach with an evaluative design through in-depth analysis is applied to student research reports, specifically research on the development of digital learning materials. The formative evaluation method chosen and the reasons for the selection were analyzed and the results were discussed in a focus group discussion. The sample was taken from the results of research by prospective teacher. Expert review was conducted to test the validity of the content and test the design. Content validity test can be done qualitatively or quantitatively by applying Lawshe, Aiken, or Gregory techniques. The design test was carried out with a black-box test. White-box testing is very rarely implemented and is recommended when developing using a programming language. One-to-one and small group tests were conducted using think aloud, cognitive walk through, or heuristics techniques. The results of the analysis found that the formative evaluation methods of teaching materials needed were expert review, one-to-one, small group, and field evaluation. Furthermore, for field tests, inquiry methods can be applied with various techniques, such as learning object review instrument (LORI), user experience questionnaire (UEQ), or system usability scale (SUS).

References

Abrahams, I., & Reiss, M. J. (2012). Practical work: Its effectiveness in primary and secondary schools in England. Journal of Research in Science Teaching, 49(8). https://doi.org/10.1002/tea.21036.

Adzobu, N, Y. (2014). Design, Use and Evaluation of E-Learning Platforms: Experiences and Perspectives of a Practitioner from the Developing World Studying in the Developed World. Informatics, 1, 147–159. https://doi.org/10.3390/informatics1020147.

Aldoobie, N. (2015). ADDIE Model. American International Journal of Contemporary Research, 5(6). www.aijcrnet.com/journals/Vol_5_No_6_December_2015/10.pdf .

Avando Bastari, Adi Bandono, & Okol Sri Suharyo. (2021). The development strategy of smart campus for improving excellent navy human resources. Global Journal of Engineering and Technology Advances, 6(2), 033–043. https://doi.org/10.30574/gjeta.2021.6.2.0011.

Bhat, B. A., & Bhat, G. J. (2019). Formative and summative evaluation techniques for improvement of learning process. ,. European Journal of Business & Social Sciences, 7(5), 776–785. https://www.researchgate.net/profile/Bilal-Bhat-6/publication/333633265_Formative_and_Summative_Evaluation_Techniques_for_Improvement_of_Learning_Process/links/5e6b7e48a6fdccf321d9926e/Formative-and-Summative-Evaluation-Techniques-for-Improvement-of-Learning-Process.pdf.

Bojko, A. (2013). Eye Tracking The User Experience. Rosenfeld Media.

Brooke, J. (2013). SUS: a retrospective. Journal of Usability Studies, 8(2), 29–40. https://www.researchgate.net/profile/John-Brooke-6/publication/285811057_SUS_a_retrospective/links/5ee5c4a792851ce9e7e38a75/SUS-a-retrospective.pdf.

Candra, & Retnawati, H. (2020). A meta-analysis of constructivism learning implementation towards the learning outcomes on civic education lesson. International Journal of Instruction, 13(2), 835–846. https://doi.org/10.29333/iji.2020.13256a.

Carvalho, J. V., Pereira, R. H., & Rocha, Á. (2020). A systematic literature review on maturity models for information systems in higher education institutions. Innovations in Education and Teaching International, 57(4), 434–449. https://doi.org/10.1080/14703297.2019.1648219.

Coman, C., Țîru, L. G., Meseșan-Schmitz, L., Stanciu, C., & Bularca, M. C. (2020). Online teaching and learning in higher education during the coronavirus pandemic: Students’ perspective. Sustainability (Switzerland), 12(24), 1–22. https://doi.org/10.3390/su122410367.

Creswell, J. W. (2014). E Book Research Design Cressweell 2014 (4th Editio). SAGE Publications, Inc.

Davis, D., Chen, G., Hauff, C., & Houben, G. J. (2018). Activating learning at scale: A review of innovations in online learning strategies. Computers and Education, 125(June), 327–344. https://doi.org/10.1016/j.compedu.2018.05.019.

Dewi, E. R., & Alam, A. A. (2020). Transformation Model for Character Education of Students. Cypriot Journal of Educational Sciences, 15(5), 1228–1237. https://doi.org/10.18844/CJES.V15I5.5155.

Dudek, J., & Heiser, R. (2017). Elements, principles, and critical inquiry for identity-centered design of online environments. Journal of Distance Education, 32(2), 1–18. https://search.proquest.com/openview/36da03443bbbdec42efcb9d7d1ceb2d8/1?pq-origsite=gscholar&cbl=446313.

Elmahdi, I., Al-Hattami, A., & Fawzi, H. (2018). Using Technology for Formative Assessment to Improve Students’ Learning. Turkish Online Journal of Educational Technology-TOJET, 17(2), 182–188. https://eric.ed.gov/?id=EJ1176157.

Farquhar, J., & Michels, N. (2016). Triangulation without tears. In Marketing Challenges in a Turbulent Business Environment, 325–330. https://doi.org/10.1007/978-3-319-19428-8_86.

Ferdianto, F., & Dwiniasih. (2019). Learning Management System (LMS) schoology: Why it’s important and what it looks like. Journal of Physics: Conference Series, 1360(1). https://doi.org/10.1088/1742-6596/1360/1/012034.

García-Ceberino, J. M., Antúnez, A., Ibáñez, S. J., & Feu, S. (2020). Design and Validation of the Instrument for the Measurement of Learning and Performance in Football. International Journal of Environmental Research and Public Health, 17(13), 4629. https://doi.org/10.4135/9781506326139.n240.

Githua, B. N. (2013). Secondary school students’ perceptions of mathematics formative evaluation and the perceptions’ relationship to their motivation to learn the subject by gender in Nairobi and Rift Valley Provinces, Kenya. Asian Journal of Social Sciences and Humanities, 2(1), 174–183. http://www.ajssh.leena-luna.co.jp/AJSSHPDFs/Vol.2(1)/AJSSH2013(2.1-19).pdf.

Hoang, L. P., & Arch-Int, N. (2013). Assessment of open-ended questions using a multidimensional approach for the interaction and collaboration of learners in e-learning environments. Journal of Universal Computer Science, 19(7), 932–949. https://www.researchgate.net/profile/Loc-Hoang-Phuoc/publication/286190008_Assessment_of_Open-Ended_Questions_using_a_Multidimensional_Approach_for_the_Interaction_and_Collaboration_of_Learners_in_E-Learning_Environments/links/584e066808aed95c25032e12/Ass.

Hughes, M., Salamonson, Y., & Metcalfe, L. (2020). Student engagement using multiple-attempt ‘Weekly Participation Task’ quizzes with undergraduate nursing students. Nurse Education in Practice, 46(August 2018), 102803. https://doi.org/10.1016/j.nepr.2020.102803.

Istri Aryani, I. G. A., & Rahayuni, N. K. S. (2016). Innovation of Teaching and Learning English Applied to Animal Sciences’ Student with the Combination of Computer Media and Audio Visual. International Journal of Linguistics, Literature and Culture, 2(1). https://doi.org/10.21744/ijllc.v2i1.5.

Jacob, R. J., & Karn, K. S. (2003). Eye tracking in human-computer interaction and usability research: Ready to deliver the promises. In The Mind’s Eye, 573–605. https://doi.org/10.1016/B978-044451020-4/50031-1.

Kljun, M., Geroimenko, V., & Čopič Pucihar, K. (2020). Augmented Reality in Education: Current Status and Advancement of the Field. In Springer Series on Cultural Computing. https://doi.org/10.1007/978-3-030-42156-4_1.

Kundu, A., & Bej, T. (2021). Experiencing e-assessment during COVID-19: an analysis of Indian students’ perception. Higher Education Evaluation and Development, 15(2), 114–134. https://doi.org/10.1108/heed-03-2021-0032.

Lubis, A. H., & Dasopang, M. D. (2021). Online Learning During the Covid-19 Pandemic: How is it Implemented in Elementary Schools? Premiere Educandum : Jurnal Pendidikan Dasar Dan Pembelajaran, 11(1), 120. https://doi.org/10.25273/pe.v11i1.8618.

Lyon, A. R., Coifman, J., Cook, H., McRee, E., Liu, F. F., Ludwig, K., & McCauley, E. (2021). The Cognitive Walkthrough for Implementation Strategies (CWIS): a pragmatic method for assessing implementation strategy usability. Implementation Science Communications, 2(1), 1–16. https://doi.org/10.1186/s43058-021-00183-0.

Madaus, G. F., & Kellaghan, T. (2000). Models, metaphors, and definitions in evaluation. In Evaluation Models, 19–31. https://link.springer.com/content/pdf/10.1007/0-306-47559-6_2.pdf.

Maier, U., Wolf, N., & Randler, C. (2016). Effects of a computer-assisted formative assessment intervention based on multiple-tier diagnostic items and different feedback types. Computers and Education, 95, 85–98. https://doi.org/10.1016/j.compedu.2015.12.002.

Martin, F., Ritzhaupt, A., Kumar, S., & Budhrani, K. (2019). Award-winning faculty online teaching practices: Course design, assessment and evaluation, and facilitation. Internet and Higher Education, 42(March), 34–43. https://doi.org/10.1016/j.iheduc.2019.04.001.

Miles, B., Huberman, M., & Saldana, J. (2014). Qualitative Data Analysis: A Methods Sourcebook. SAGE Pub.

Morgan, H. (2022). Conducting a Qualitative Document Analysis. The Qualitative Report, 27(1), 64–77. https://doi.org/10.46743/2160-3715/2022.5044.

Morrar, R., Arman, H., & Mousa, S. (2017). The fourth industrial revolution (Industry 4.0): A social innovation perspective. Technology Innovation Management Review, 7(11), 12–20. https://timreview.ca/sites/default/files/Issue_PDF/TIMReview_November2017.pdf#page=12.

Noble, H., & Heale. (2019). Triangulation in research, with examples. Evidence-Based Nursing, 22(3). https://doi.org/10.1136/ebnurs-2019-103145

Nurtanto, M, Sofyan, H., & Pardjono, P. (2020). E-Learning Based AutoCAD 3D Interactive Multimedia on Vocational Education (VE) Learning. Journal of Engineering Education. https://doi.org/10.16920/jeet/2021/v34i4/155014.

Nurtanto, Muhammad, Pardjono, P., Widarto, W., & Ramdani, S. D. (2020). The effect of STEM-EDP in professional learning on automotive engineering competence in vocational high school. Journal for the Education of Gifted Young Scientists, 8(2), 633–649. https://doi.org/10.17478/JEGYS.645047.

Okoye, M. D. (2014). Authentic Assessment and Evaluation: Paramount Means for the Maximization of Teaching and Learning. Journal of Educational and Social Research, 4(7), 31–40. https://doi.org/10.5901/jesr.2014.v4n7p31.

Oktiningrum, W., & Wardhani, D. A. P. (2020). Developing HOT’s mathematics task with Indonesian heritage as context to assess mathematical literacy of students in primary school. International Journal for Educational and Vocational Studies, 1(8), 69. https://doi.org/10.29103/ijevs.v2i1.1997.

Ouyang, F., & Scharber, C. (2017). The influences of an experienced instructor’s discussion design and facilitation on an online learning community development: A social network analysis study. Internet High. Educ., 35, 34–47. https://doi.org/10.1016/j.iheduc.2017.07.002.

Öz, H. (2014). Pre-service English Teachers’ Perceptions of Web-based Assessment in a Pedagogical Content Knowledge Course. Procedia - Social and Behavioral Sciences, 141, 45–58. https://doi.org/https://doi.org/10.1016/j.sbspro.2014.05.010.

Quinõnes, D., & Rusu, C. (2017). How to develop usability heuristics: A systematic literature review. Computer Standards & Interfaces, V., 53, 89–122. https://doi.org/10.1016/j.csi.2017.03.009.

Rajendran, D., & Santhanam, P. (2021). Towards digital game-based learning content with multi-objective reinforcement learning. Materials Today: Proceedings, xxxx. https://doi.org/10.1016/j.matpr.2021.03.156.

Roblek, V., Meško, M., & Krapež., L. (2016). “A Complex View of Industry 4.0.” SAGE Open, 6(2), 2158244016653987. https://doi.org/10.1177/2158244016653987. sgo.sagepub.com.

Rodríguez-Gómez, G., Quesada-Serra, V., & Ibarra-Sáiz, M. S. (2014). Learning-Oriented E-Assessment: the Effects of A Training and Guidance Programme on Lecturers’ Perceptions. Assessment and Evaluation in Higher Education, 1–18. https://doi.org/http://dx.doi.org/10.1080/02602938.2014.979132.

Schneider, C., & Bodensohn, R. (2017). Student Teachers’ Appraisal of the Importance of Assessment in Teacher Education and Self-Reports on the Development of Assessment Competence. Assessment in Education: Principles, Policy and Practice, 24(2), 127–146. https://doi.org/http://dx.doi.org/10.1080/0969594X.2017.1293002.

Srivastava, T. K., Mishra, V., & Waghmare, L. S. (2018). Formative Assessment Classroom Techniques (FACTs) for better learning in pre-clinical medical education: A controlled trial. Journal of Clinical and Diagnostic Research, 12(9). https://www.researchgate.net/profile/Tripti-Srivastava/publication/327229061_Formative_Assessment_Classroom_Techniques_FACTs_for_Better_Learning_in_Pre-Clinical_Medical_Education_A_Controlled_Trial/links/5b8673f7299bf1d5a72eef8e/Formative-Assessment-Classroom-Techniques-FACTs-for-Better-Learning-in-Pre-Clinical-Medical-Education-A-Controlled-Trial.pdf.

Suryani, N. (2018). Utilization of digital media to improve the quality and attractiveness of the teaching of history. The 2nd International Conference On Teacher Training and Education, 2(1), 131–144. https://jurnal.uns.ac.id/ictte/article/view/8164.

Susilawati, W. O., Widodo, H., & Sumarno, S. (2019). Strategy of Teachers in Suporting Environmentally Sustainable Development. Journal of Education and Learning (EduLearn), 13(2), 247–254. https://doi.org/https://doi.org/10.11591/edulearn.v13i2.12167.

Suwartono, T., & Aniuranti, A. (2019). Digital Teaching Tools in 21st Century EFL Classroom: Are Our Teachers Ready? ELLITE: Journal of English Language, Literature, and Teaching, 3(2), 57. https://doi.org/10.32528/ellite.v3i2.1916.

Tessmer, M. (2013). Planning and conducting formative evaluations. Routledge.

Tjeerd Plomp. (2013). Educational Design Research: an Introduction. In T. Plomp & N. Nieveen (Eds.), An Introduction to Educational Design Research (pp. 9–36). Netzodruk, Enschede. https://www.academia.edu/download/48472657/educational-design-research-part-a.pdf#page=12.

Vagg, T., Balta, J. Y., Bolger, A., & Lone, M. (2020). Multimedia in Education: What do the Students Think? Health Professions Education, 6(3), 325–333. https://doi.org/doi.org/10.1016/j.hpe.2020.04.011.

Vargo, J., Nesbit, J. C., Belfer, K., & Archambault, A. (2003). Learning object evaluation: Computer mediated collaboration and inter-rater reliability. International Journal of Computers and Applications, 25(3), 1–8. https://doi.org/10.1080/1206212X.2003.11441703.

Verdinelli, S., & Scagnoli, N. I. (2013). Data display in qualitative research. International Journal of Qualitative Methods, 12(1), 359–381. https://doi.org/10.1177/160940691301200117.

Vieira, E. A. O., Silveira, A. C. D., & Martins, R. X. (2019). Heuristic evaluation on usability of educational games: A systematic review. Informatics in Education, 18(2), 427–442. https://doi.org/10.15388/infedu.2019.20.

Weninger, M., Grünbacher, P., Gander, E., & Schörgenhumer, A. (2020). Evaluating an interactive memory analysis tool: Findings from a cognitive walkthrough and a user study. Proceedings of the ACM on Human-Computer Interaction, 1–37. https://doi.org/10.1145/3394977.

Wongwatkit, C., Srisawasdi, N., Hwang, G. J., & Panjaburee, P. (2016). Influence of an Integrated Learning Diagnosis and Formative Assessment-based Personalized We Learning Approach on Students Learning Performances and Perceptions. Interactive Learning Environments, 1–15. https://doi.org/http://dx.doi.org/10.1080/10494820.2016.1224255.

Wu, J. Y., & Nian, M. W. (2021). The dynamics of an online learning community in a hybrid statistics classroom over time: Implications for the question-oriented problem-solving course design with the social network analysis approach. Computers & Education, 166, 104120. https://doi.org/10.1016/j.compedu.2020.104120.

Wu, X. ., He, Z. ., Li, M. ., Han, Z. ., & Huang, C. (2022). Identifying Learners’ Interaction Patterns in an Online Learning Community. Int. J. Environ. Res. Public Health, 19, 2245. https://doi.org/10.3390/ ijerph19042245.

Yeni, S., & Cagiltay., K. (2017). A heuristic evaluation to support the instructional and enjoyment aspects of a math game. Program, 51(4), 406–423. https://doi.org/10.1108/ PROG-07-2016-0050.

Yilmaz, R. (2017). Exploring the role of e-learning readiness on student satisfaction and motivation in flipped classroom. Computers in Human Behavior, 70, 251-260. https://doi.org/10.1016/j.chb.2016.12.085.

Young, T., Hazarika, D., Poria, S., & Cambria, E. (2018). Recent trends in deep learning based natural language processing. Ieee Computational IntelligenCe Magazine, 13(3), 55–75. https://doi.org/10.1109/MCI.2018.2840738.

Downloads

Published

2022-08-31

How to Cite

Sri Mertasari, N. M., & Candiasa, I. M. (2022). Formative Evaluation of Digital Learning Materials. Journal of Education Technology, 6(3), 507–514. https://doi.org/10.23887/jet.v6i3.44165

Issue

Section

Articles