The Reflection of HOTS in EFL Teachers’ Summative Assessment

This study is aimed at analyzing how summative assessments made by eleventh-grade English teachers reflect Higher Order Thinking Skills (HOTS). This was qualitative research that involved 3 English subject teachers in Senior High Schools in Bali, Indonesia. Meanwhile, the object of this research was summative assessment in the form of teacher-made test items for class XI Semester I in the academic year of 2018/2019. The items were analyzed by the document analysis method to determine the extent to which HOTS based on Revised Bloom's Taxonomy was reflected in the teacher-made test. Meanwhile, a oneon-one interview was conducted to obtain data on teacher knowledge about HOTS. The results of this study found that the percentage of HOTS-based teacher-made test items was 5.4% and was still limited to the cognitive level of Analyze (C4). From the results of the analysis, the cognitive levels of students measured were still in Lower Order Thinking Skills (LOTS). This was due to the fact that teachers did not fully understand the concept of Higher Order Thinking Skills. Furthermore, it was obtained from the interview that there was a discrepancy between the knowledge and performance of the teacher in the assessment. Therefore, teachers especially eleventh-grade English teachers have to learn more about HOTS and join training in developing HOTS-based items. Copyright © Universitas Pendidikan Ganesha. All rights reserved. A R T I C L E I N F O

Thinking Skills (HOTS)?". Hence, this study is aimed to know how the eleventh-grade English teachers' summative assessment reflects Higher Order Thinking Skills (HOTS). This study will be important to be conducted for verifying the HOTS-based summative assessment instrument made by the eleventh-grade English teachers in Singaraja.

Methods
This study used a descriptive qualitative approach to describe the extent to which HOTS reflect on the summative assessment instrument made by the teacher. It involved three eleventh-grade English subject teachers from three state senior high schools in Singaraja. The data were in the form of nonnumerical data and collected from document analysis and interview. The analyzed documents were summative assessments in the form of the teacher-made test. The tests were in the form of multiplechoice items and short-answer items. The total items were 130 items. Document analysis was done in order to determine the extent to which HOTS based on Revised Bloom's Taxonomy was reflected in the teacher-made test. Furthermore, the one-on-one interview was conducted afterward to obtain data on teacher knowledge about HOTS. Two instruments were developed to collect the data namely analysis form and interview guide. The blueprint was adopted by the Revised Bloom's Taxonomy in the cognitive process dimension only. The theory used as reference was Revised Bloom's Taxonomy.

Result And Discussion
HOTS is important to be implemented in the teaching and learning process in this 21 st -century education. It could be trained through HOTS-based items made by the teachers. HOTS in Revised Bloom's Taxonomy involves the cognitive process of Analyze (C4), Evaluate (C5), and Create (C6). The findings of this study show that the most dominant items that appeared in the teacher-made test are in Lower Order Thinking Skills. HOTS-based items are rarely found. Even though there are some items categorized as HOTS-based items, yet those items only cover the cognitive process of Analyze (C4). The distribution of items from all teacher-made tests is presented in the table below: The item above belongs to the Analyze category (C4) of Revised Bloom's Taxonomy Furthermore, it indicates the differentiating cognitive process since it requires the students to distinguish between relevant and irrelevant information. According to Anderson et al. (2001), differentiating means distinguishing relevant from irrelevant parts or important from unimportant parts of the presented material. In the item, the students are asked to distinguish cause and effect in a sentence. After that, students are asked to choose which part of the sentence that became the effect in the sentence. So, this item belongs to the Analyze category (C4) especially differentiating.
Another example is presented below: Make a formal invitation letter by arranging the following sentences into a body of the formal invitation letter! a. We are looking forward to your attendance at the event b. We are enclosing the programme of events for the evening, so please be prompt on arrival. The item above belongs to the Analyze category (C4) of Revised Bloom's Taxonomy especially the indicator of organizing. Furthermore, according to Anderson et al. (2001), organizing means recognizing how the elements of communication or situation fit together into a coherent structure. In this item, the students are asked to arrange some sentences into a formal invitation letter. In this case, the students are given a series of sentences and must determine what will come first, second, and so on. Then, they have to determine how one sentence related to another within the formal invitation letter. Hence, these sentences would be a good paragraph. Therefore, this item belonged to C4.
Meanwhile, the result of the interview, which was conducted after the document analysis, represents that most of the teachers are indicated having different understandings about HOTS. It is proven by the statements of the teachers during the interview about HOTS. The statement from teacher 1 is as follow: In

my point of view, concluding the text belongs to HOTS. It is Higher Order Thinking Skills. And one of its characteristics, based on what I read, is critical thinking. It is like what I mentioned. It is very important to develop questions that lead students to think critically. Because we, sometimes, what is it called, want students to know, not only to master the theory, but how they can respond, analyze, or use reason if there is a problem with how to decide. That is why, in learning, it is important to develop questions that lead students to think critically. That is why in one set of tests, there are only, let us say 4-5 questions or 4, Ma'am. It only exists between 5 items
From this statement, it might indicate that the teacher is familiar with HOTS. The teacher has mentioned one of the characteristics of HOTS. He also stated that HOTS-based items are those which asked the students to conclude the text. It has been implemented in the items made by the teacher. However, this statement contrast with the result of the document analysis which was not found any HOTSbased items in the test made by Teacher 1. Another teacher has a different point of view about HOTS. The opinion of teacher 2 is below: Higher Order Thinking Skills, so the students should be able to determine something in a short time. Then, it is to increase the competitiveness of students. So, we train them to answer the question in a short time. The reference to develop HOTS-based item is the material that should be real.
The definition of HOTS by Teacher 2 shows that he has a misconception of HOTS. The teacher uttered that HOTS means students can determine an appropriate answer in the nick of time. This is used to improve student competitiveness. Therefore, in developing HOTS-based items, the material used as a reference must be contextual. It could be more challenging for students to compete in the learning process. Teacher 3 states another point of view about HOTS as stated below: In my mind, HOTS is a term of items used to measure higher-order thinking skills. And HOTS-based items are not just items, they must minimize their abilities to recall information yet they will be more measuring abilities such as the transfer of one concept to another. They also must process and apply information. They also have to look for links from different information. Then, they also use the information to solve a problem. And of course, they must examine the idea and information critically.
At this point, the teacher posed that HOTS is a term of the item to measure higher-order thinking skills of students. To measure HOTS, the teacher should develop the items which minimize the cognitive process of Remember (C1) yet it should maximize the cognitive process of Analyze (C4). She also expressed that HOTS-based item tends to ask the students to transfer one concept to another, process and apply information, be able to link various information, use the information to solve a problem and examine the idea and information critically With respect to the findings, it is found that only a few numbers of items belong to HOTS-based item. The items used in this assessment are in the form of multiple-choice. It goes along with the study conducted by Suah See Ling, Lan, and Osman (2014) in Mohamed & Lebar (2017) that pose school teachers use a lot of multiple-choice questions in the classroom assessment. This kind of item demands the students to choose the correct answer by guessing the answer among the options given. The students usually do not put a higher level of thinking when they guess the answer. It is in harmony with the statement of Mohamed & Lebar (2017) that multiple-choice item does not stimulate students' thinking when they guessed the answer without thinking. Therefore, Coombs et al. (1956) in Mohamed & Lebar, (2017) argue that the multiple-choice item is more concerned to emphasize on encouraging guesswork.
Multiple-choice item is the form of traditional assessment. This kind of assessment usually depends on a single correct answer of students per specific question (Wraga, 1994in Abosalem, 2016. The demonstration of overall knowledge and thinking process are omitted in this assessment (Abosalem, 2016). Therefore, Abosalem (2016) argues traditional tests are ineffective in measuring higher-order thinking skills or their abilities to deal with new and unusual problems since the students only give the impression that answers are always either right or wrong and encourage memorization rather than analyzing. This might become one of the reasons why HOTS-based items in the teacher-made tests in Singaraja are inadequate developed.
From the six levels of cognitive process dimension, the cognitive process of Understand (C2) was the most dominant level which appeared frequently with the percentage of 52.3%. It means the teachermade tests in Singaraja were assumed not to encourage the students to analyze, evaluate, and create new ideas. From the percentages, it implies that the eleventh-grade English teachers of this test did not include enough HOTS-based items which could generate and attract learners to use all their mental processes optimally. The few numbers of items requiring HOTS implies that the teachers did not take consideration in stimulating the learners to use HOTS. This is in line with the research conducted by Widana (2017a) that only about 7.58% of School examination items made by high school teachers which belong to the cognitive process of Analyze (C4), Evaluate (C5), and Create (C6).
As mentioned before, the teachers generally develop items dominantly in Remember and Understand level. This indicates that they do not have adequate comprehension of HOTS. It is in agreement with the research finding from Driana & Ernawati (2019). The research involved 10 primary school teachers and found out that in general, the teachers were only able to explain a partial understanding of HOTS. This is in line with the result of the interview in the present study. It suggests that the three teachers cannot define well what is meant by HOTS. One of the teachers pointed out that he still got confused about what HOTS is. Meanwhile, another teacher had a misconception about HOTS-based assessment. Meanwhile, another one defined HOTS as a term of test item which measures the ability of students in the realm of analysis. One of the indicators of HOTS is analyzing (C4). None of the teachers mentioned neither explicitly or implicitly about evaluate (C5) and create (C6). However, the result of the present study should not be generalized since the study was conducted to three eleventh-grade English teachers from three schools in Singaraja only. Schulz & FitzPatrick (2016) convey that paper and pencil assessments were sometimes based on what teachers thought their students could do. It suggests when teachers developed items in the Remember (C1) and Understand (C2) level, they believed that their students enable to reach those levels of the cognitive process. In other words, in developing the items, teachers will adjust it to the level of cognitive of their students as well. They will rarely put HOTS-based items in their test since they guess their students will not be able to answer it. It goes along with the statement by Schulz & FitzPatrick (2016) which point out teachers have the idea that if HOTS-based items were put on paper and pencil assessments and students could not answer them, then the teacher needed to take the time to re-teach and there was no time for this (Schulz & FitzPatrick, 2016). Consequently, LOTS-based items dominated in paper and pencil assessments, even though instructional activities sometimes allowed for higher order thinking.
Moreover, Marhaeni et al. (2018) argue that there are still many teachers think that the most important task is to teach in the class. It makes the assessment is ignored. Similar to this, in the result of the interview, the teachers claimed that they have a lack of time to handle their work for planning, teaching, and assessing the students. They spent most of their time to teach in class. This causes, to develop items for assessment, they usually adapt or adopt it from Buku Bank Soal, LKS (Worksheet Book) and sometimes the internet. Even though they have filtered the items that are following the subject matter, yet they cannot guarantee that the items in the test they made belong to LOTS or HOTS. No wonder, when they were asked about the items, they have uncertainty to choose which item belongs to LOTS or HOTS.
Furthermore, Tanujaya, Mumu, & Margono (2017) pose that HOTS are very necessary because they are believed to prepare students better for the challenges both in advanced academic life and adult's work and responsibility in a daily basis. It means that HOTS are very helpful to make the students ready to face problem in real life. It goes along with the statement by one of the teachers during the interview. She said that developing HOTS-based items is important in order to prepare the students to accomplish oral and written test in formative and summative assessment, and the other advanced tests. Besides, another teacher said that developing HOTS-based item is not only enable the students to master the subject matter theoretically in the class but also help the students to deal with the problem they face in daily life. Therefore, it is really important to develop HOTS-based items to prepare the students' readiness for anything in the future.
Even though the teachers acknowledge that developing HOTS-based items are important, but based on the result of document analysis and interview, it is found a discrepancy between the knowledge and performance of the teachers. In the result of document analysis, it is found that the teachers dominantly develop items in the level of Remember (C1) and Understand (C2). These two cognitive processes are categorized as LOTS. Meanwhile, in the result of the interview, one of the teachers argues that he already attached 4 to 5 items in each test. Even, another teacher states that almost all the time she attached HOTS in the test she made. However, it is rarely found the HOTS-based items in the teacher-made test in Singaraja.
It might have a relation to teacher assessment literacy. According to Stiggins (1994) in Marhaeni et al. (2018), Teacher Assessment Literacy (TAL) is the ability of teachers to conduct valid assessments and utilize the results of assessments to enhance learning. In other words, it relates to the knowledge possessed by the teacher about basic principles of assessment such as how to develop assessments, how to use assessments, and standards of assessment to improve the teaching and learning process. In the study conducted by Marhaeni et al. (2018), it was found that there was a discrepancy in the implementation of assessment by senior high school teachers in Bali. It indicates that the competencies of the teacher to develop valid and appropriate assessments are still deficient. It is in accord with the findings of the present study in which there is a discrepancy between the item that had been analyzed and the teacher's answer when they were asked to determine the HOTS-based items in their test. So, there is a discrepancy between the knowledge of the teacher and their actual performance on assessment.

Conclussion
This study analyzed the classification of Higher Order Thinking Skills (HOTS) in teacher-made test items in State Senior High Schools in Singaraja by correlating the test item and the six major indicators of cognitive process dimension in Revised Bloom's Taxonomy. Based on the result of document analysis, the percentage of HOTS-based items was 5.4% of the entire items. They were not distributed evenly in each school. After analyzing all items in all assessments, it was revealed that the items categorized as HOTS and made by the teachers were limited to the cognitive process of Analyze (C4). Whereas the items were mostly emphasized on the two low levels of Revised Bloom's taxonomy, namely Remember (C1) and Understand (C2). Moreover, the eleventh-grade English teachers involved in this research were aware of HOTS that is important to prepare the students to face the challenges of the 21st century. However, their understanding was not fully equipped yet. It was supported by the dominant number of the HOTS-based items they made were limited to analysis in the form of multiple-choice and short-answer items. In addition, there was a misunderstanding about the concept of HOTS in their minds. It indicated that there is a discrepancy between the teacher's knowledge and their actual performance on the assessment. Therefore, teachers especially eleventh-grade English teachers it is essential for the teacher to find out the trusted resource about HOTS. Besides, they can join a seminar or workshop about developing HOTS-based assessment or at least how to develop HOTS-based indicators. Whereas, the policymakers such as government and headmaster are suggested to make a policy that requires the teachers to make blueprint first before developing the test items. For future researchers, especially R&D researchers, it is expected that they can expand or enlarge research in developing HOTS-based items more often since the HOTS-based items are not adequately developed yet.