Summative Evaluation of ICT-Based Learning Media
DOI:
https://doi.org/10.23887/jere.v6i4.54695Keywords:
Formative, Evaluation, ICT-based, Learning MediaAbstract
Prospective teacher students have been heavily involved in developing ICT-based learning media. Various forms of media have been produced, such as multimedia, hypermedia, web, video, animation, or presentation. Until now, not all media developed have undergone a summative evaluation but only arrived at a formative evaluation. One of the reasons is the lack of references. This study aims to formulate a summative evaluation model of ICT-based learning media developed by prospective teacher students using quantitative and qualitative descriptive approaches. The study began with observations of student research reports to examine the summative evaluation model used. The information is then discussed in a focus group discussion involving relevant experts and practitioners. The study found that the summative evaluation of ICT-based learning media could be done through a quasi-experimental survey or case study. Quasi-experimental methods that can be applied are nonequivalent control group design, factorial design, counterbalanced design, and time series design. The format was obtained mainly considering limited time, a place for testing, and opportunities to be involved in the class. Either quasi-experiments, surveys, or case studies can be carried out by students in collaboration with relevant teachers when students do teaching practicum at school and then mature when students do research.
References
Aldoobie, N. (2015). ADDIE model. American International Journal of Contemporary Research, 5(6), 68–72.
Allen, M. (2019). Counterbalancing. The SAGE Encyclopedia of Communication Research Methods.
Amin, Z., & Eng, K. H. (2003). Formative and Summative Assessment. Basics in Medical Education, 261–265. https://doi.org/10.1142/9789812795472_0028.
Bandura, A. (1977). Social Learning Theory. Prentice-Hall, Inc.
Basu, R. (2018). Instructional Design Models : Benefits and Challenges. UGC Approved Journal, Vol.-41 N(June), 31–36.
Brooke, J. (2013). SUS: A retrospective. Journal of Usability Studies, 8(2), 29–40.
Chang, S. L. (2006). The Systematic Design of Instruction. In Educational Technology Research and Development (Vol. 54, Issue 4). Harper Collins Publishers Inc. https://doi.org/10.1007/s11423-006-9606-0.
Creswell, J. W. (2015). A Concise Introduction to Mixed Methods Research. In SAGE Publications, Inc. (Vol. 1999, Issue December). Sage.
Deeley, S. J. (2018). Using technology to facilitate effective assessment for learning and feedback in higher education. Assessment and Evaluation in Higher Education, 43(3), 439–448. https://doi.org/10.1080/02602938.2017.1356906.
Drljača, D., Latinović, B., Stanković, Ž., & Cvetković, D. (2017). ADDIE Model for Development of E-Courses. In Documento Procedente de La International Scientific Conference on Information Technology and Data Related Research SINTEZA, 242–247. https://doi.org/10.15308/sinteza-2017-242-247.
Earl, L. M. (2003). Assessment As Learning: Using Classroom Assessment to Maximize Student Learning (Experts In Assessment Series). Corwin Press. http://www.amazon.com/Assessment-As-Learning-Classroom-Maximize/dp/0761946268.
Eraut, M. (1982). Evaluation a systematic approach. In International Journal of Educational Development (Vol. 2, Issue 3). SAGE. https://doi.org/10.1016/0738-0593(82)90010-4.
Farquhar, J., & Michels, N. (2016). Triangulation without Tears, Marketing Challenges in a Turbulent Business Environment (Issue 3). Heidelberg.
Fraenkel, Jack R., Wallen, N. E. (2009). How to Design and Evaluate Research in Education. In McGraw-Hill Higher Education (Issue 0). McGraw-Hill Inc.
Gall, M. D., Joyce P. Gall, & Borg., W. R. (2003). Educational Research: An Introduction. Pearson Education, Inc.
Gay, I. R., & Mills, E. G. (2012). Research method: Competencies for analysis and applications. In Pearson Education, Inc (Vol. 6, Issue 2). Pearson. http://library1.nida.ac.th/termpaper6/sd/2554/19755.pdf.
Glasow, P. A. (2005). Fundamentals of survey research methodology. MITRE, C3 Center. http://www.uky.edu/~kdbrad2/EPE619/Handouts/SurveyResearchReading.pdf.
Gravetter, F. J., Forzano, L. A. B. (2009). Research Methods Behavioral Sciences. WADSWORTH CENCAGE Learning.
Hattie, J. A., Brown, G. T. L., Ward, L., Irving, S. E., & Keegan, P. J. (2006). Formative Evaluation of an Educational Assessment Technology Innovation : Developers ’ Insights into Assessment Tools for Teaching and Learning ( asTTle ). Journal of Multidisciplinary Evaluation, 5(3), 1–54.
Heale, R., & Twycross., A. (2018). What is a case study? Roberta Heale. Evid Based Nurs, 21.
Indrayan, A. (2019). Sampling and Sample Size. Research Methods for Medical Graduates, 147–168. https://doi.org/10.1201/9780429435034-8.
Joyce, A. (2019). Formative vs. Summative Evaluation. https://www.nngroup.com/articles/formative-vs-summative-evaluations/.
Khodabandelou, R., & Samah, S. A. A. (2012). Instructional Design Models for Online Instruction: From the Perspective of Iranian Higher Education. Procedia - Social and Behavioral Sciences, 67(1), 545–552. https://doi.org/10.1016/j.sbspro.2012.11.359.
Laugwitz, B., Held, T., & Schrepp, M. (2008). Construction and evaluation of a user experience questionnaire. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 5298 LNCS, 63–76. https://doi.org/10.1007/978-3-540-89350-9_6.
Leacock, T. L., & Nesbit, J. C. (2007). A Framework for Evaluating the Quality of Multimedia Learning Resources. Educational Technology & Society, 10(2), 44–59.
Loeb, S., Dynarski, S., Mcfarland, D., Morris, P., Reardon, S., & Reber, S. (2017). Descriptive Analysis in Education: A Guide for Researchers. NCEE 2017-4023. In Eric. Department of Education, Institute of Education Sciences, National Center for Education Evaluation and Regional Assistance. https://eric.ed.gov/?id=ED573325.
LR, G., GE, M., & P, A. (2006). Educational Research: Competencies for Analysis and Applications 8thEdition. Pearson.
Mathiyazhagan, T., Nandan, D., & Khosla, M. (2010). Survey research method ABSTRACT. Media Mimansa, 4(September), 34–82. http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.464.5585&rep=rep1&type=pdf.
Mckenzie, B. K. (2006). Evaluating Educational Technologies : Technology Connoisseurs in the Campus *. Annual Meeting of Eastern Educational Research Association, 1998, 1–10.
Miles, M. B., Huerman, A. M., & Johny Saldana. (2014). Qualitatite Data Analysis: A Methods Sourcebook. SAGE Publication, Inc.
Nassaji, H. (2015). Qualitative and descriptive research: Data type versus data analysis. Language Teaching Research, 19(2), 129–132. https://doi.org/10.1177/1362168815572747.
Navarro-Rivera, J., & Kosmin, B. A. (2015). Surveys and Questionnaires. The Routledge Handbook of Research Methods in the Study of Religion. https://doi.org/10.4324/9780203154281.ch2_20.
Nieveen, N., & Folmer, E. (2013). Formative evaluation in educational design research. In Educational Design Research. Part A: An introduction. Institute for Curriculum Development (SLO). http://international.slo.nl/publications/edr/.
Patton, M. Q. (1994). The program evaluation standards: How to assess evaluations of educational programs. In Evaluation Practice (second, Vol. 15, Issue 2). SAGE. https://doi.org/10.1016/0886-1633(94)90009-4.
Plomp, T., & Nieveen, N. (2013). Educational Design Research Educational Design Research. In Netherlands Institute for Curriculum Development: SLO. Institute for Curriculum Development (SLO). http://www.eric.ed.gov/ERICWebPortal/recordDetail?accno=EJ815766.
Ponto, PhD, APRN, AGCNS-BC, AOCNS®, J. (2015). Understanding and Evaluating Survey Research. Journal of the Advanced Practitioner in Oncology, 6(2), 168. https://doi.org/10.6004/jadpro.2015.6.2.9.
Ponto. (2015). Understanding and Evaluating Survey Research. Journal of the Advanced Practitioner in Oncology, 6(2), 168–171. https://doi.org/10.6004/jadpro.2015.6.2.9.
Popham, W. J. (1999). Modern educational measurement. In Practical guidelines for the education leader. Michigan: Pearson (Issue Pde 105). Prentice Hall, Inc.
Rogaten, J., Rienties, B., Sharpe, R., Cross, S., Whitelock, D., Lygo-Baker, S., & Littlejohn, A. (2019). Reviewing affective, behavioural and cognitive learning gains in higher education. Assessment and Evaluation in Higher Education, 44(3), 321–337. https://doi.org/10.1080/02602938.2018.1504277.
Rowley, J. (2002). Using case studies in research. Management Research News, 25(1), 16–27. https://doi.org/10.1108/01409170210782990.
Sarkies, M. N., Skinner, E. H., Bowles, K. A., Morris, M. E., Williams, C., O’Brien, L., Bardoel, A., Martin, J., Holland, A. E., Carey, L., White, J., & Haines, T. P. (2019). A novel counterbalanced implementation study design: Methodological description and application to implementation research. Implementation Science, 14(1), 1–11. https://doi.org/10.1186/s13012-019-0896-0.
Scriven, M. (1992). Evaluation thesaurus. In Choice Reviews Online (Vol. 29, Issue 08). SAGE. https://doi.org/10.5860/choice.29-4272.
Spivey, G. (2007). A taxonomy for learning, teaching, and assessing digital logic design. In Proceedings - Frontiers in Education Conference, FIE. Addison Wesley Longman Inc. https://doi.org/10.1109/FIE.2007.4417846.
Springer, K. (2010). Educational Research: a contextual approach. Wiley.
Symonds, P., & Symonds, P. M. (2016a). The Case Study as a Research Method. Case Studies, 15–15. https://doi.org/10.4135/9781473915480.n2.
Symonds, P., & Symonds, P. M. (2016b). The Case Study as a Research Method. Case Studies, 9, 15–15. https://doi.org/10.4135/9781473915480.n2.
Vedung, E. (2017). Public policy and program evaluation. In Public Policy and Program Evaluation. Transaction Publishers. https://doi.org/10.4324/9781315127767.
Verdinelli, S., & Scagnoli, N. I. (2013). Data display in qualitative research. International Journal of Qualitative Methods, 12(1), 359–381. https://doi.org/10.1177/160940691301200117.
Wayne, A. J., Yoon, K. S., Zhu, P., Cronen, S., & Garet, M. S. (2008). Experimenting With Teacher Professional Development: Motives and Methods. Educational Researcher, 37(8), 469–479. https://doi.org/10.3102/0013189X08327154.
Weston, T. (2004). Formative Evaluation for Implementation: Evaluating Educational Technology Applications and Lessons. American Journal of Evaluation, 25(1), 51–64. https://doi.org/10.1177/109821400402500104.
Wüthrich, K. (2004). Triangulation in Social Research: Qualitative and Quantitative Methods Can Really Be Mixed. In Developments in Sociology. Causeway Press Ltd.
Zeelenberg, R., & Pecher, D. (2015). A method for simultaneously counterbalancing condition order and assignment of stimulus materials to conditions. Behavior Research Methods, 47(1), 127–133. https://doi.org/10.3758/s13428-014-0476-9.
Downloads
Published
Issue
Section
License
Copyright (c) 2022 Ni Made Sri Mertasari
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
Authors who publish with the Journal of Evaluation and Research in Education (JERE) agree to the following terms:
- Authors retain copyright and grant the journal the right of first publication with the work simultaneously licensed under a Creative Commons Attribution License (CC BY-SA 4.0) that allows others to share the work with an acknowledgment of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work. (See The Effect of Open Access)
