Advances in Social Science, Education and Humanities Research, volume 422

International Conference on Progressive Education (ICOPE 2019)

Development of Higher Order Thinking Skills

Problem on Statistics Senior High School

Yastri Nopalia Somakim

Magister Mathematics Education Sriwijaya University
Sriwijaya University Palembang, Indonesia
Palembang, Indonesia Somakim_math@yahoo.com
y.yastri@yahoo.co.id

Abstract. This study aims the development of valid and emphasize active learning, student-centered, and scientific
practical HOTS questions for senior high school and approaches. The assessment used in the 2013 curriculum
investigating the potential effects. The method is to design emphasizes authentic assessment. These three components,
research type development studies. This research was namely, competence, learning activities, and assessment, are
conducted on eleventh-grade students of SMA Negeri 1 OKU, a series of interrelated. One of them refers to the questions
which include walkthrough, documentation, interview, and test used in the 2013 curriculum that must make students think
for data collection. The validation was applied on the basis of critically so that the teacher is required to present high-level
the validator's assessment of the questions in terms of content, thinking questions.
constructs, and language as well as student
comments/suggestions at the one-to-one stage on clarity and High-level thinking should already exist in students in
readability. From the results of the students' work in the field Indonesia as their provision in facing the global era,
test, it can be concluded that the questions developed can lead advances in information technology, technological
to HOTS. The result of this study produce HOTS Statistics convergence, and technology as an impact of techno-science,
questions, each consisting of three questions that are analyzing and the rise of creative industries in the future [2]. According
(C4), Five questions are evaluated (C5), and one question is to [3], students who have good high-level thinking will have
create (C6) that is valid, practical, and has a potential effect a commitment to continue learning, grow, and develop and
based on the HOTS framework.
evolve to become more advanced. In addition, these students
Keywords: Higher Order Thinking Skill, mathematics will be better able to interpret and be able to review existing
information and be able to use that information to solve the
I. INTRODUCTION problem at hand.
In the present century, science has developed in According to [4] HOTS is defined therein, including
accordance with the demands of life, which have also critical thinking, logical, reflective, metacognitive, and
evolved based on technology and the progress of the times. creative. In the integrated thematic learning technical guide,
One effort in dealing with the demands of the 21st century is the ministry of education and culture explains that teachers
by developing skills and literacy skills that can be used in must train students in the form of higher-order thinking skills
facing challenges in today's life. Mathematical literacy can (HOTS), with the aim of increasing students' ability to think
be said as an individual's skill in using, explaining, and reasoning to answer more complicated questions and solve a
formulating mathematics into various real contexts. It is what more complicated problem case. This ability to think at a
guides individuals to be able to recognize the role of higher level requires someone to apply new information or
mathematics in everyday life and make correct judgments prior knowledge and manipulate information to reach
and can make constructive and reflective decisions [1] possible answers and new situations [5].
Mathematics learning, according to the 2013 curriculum The US-based Apollo Education Group identifies ten
[2], aims to emphasize the modern pedagogical dimension of (10) skills needed by students to work in the 21st century,
learning using a scientific approach. In mathematics, namely critical thinking skills, communication, leadership,
learning activities used during the learning process so that collaboration, adaptability, productivity and accountability,
meaningful learning by observing, asking, trying, reasoning, innovation, global citizenship, ability and spirit
presenting, and creating as contained in the 2013 curriculum. entrepreneurship, and the ability to access, analyze, and
synthesize information [6]. Based on the results of the
2013 curriculum requires the active learning of student- research conducted by OECD, three (3) descriptions of
centered, as more teachers as inspirators, generators in learning dimensions in the 21st century were obtained,
guiding students to be able to find concepts in the learning. namely information, communication, and ethics, and social
In evaluating learning activities, an evaluation of the learning influences [7].
must be conducted. In addition to competence, teaching and
learning activities and assessments also need to be observed. According to [8], HOTS is defined in it, including critical
Teaching and learning activities in the 2013 curriculum thinking, logical, reflective, metacognitive, and creative. In

Copyright © 2020 The Authors. Published by Atlantis Press SARL.

This is an open access article distributed under the CC BY-NC 4.0 license -http://creativecommons.org/licenses/by-nc/4.0/. 338
 Advances in Social Science, Education and Humanities Research, volume 422

the technical guide to integrated thematic learning, the rubric. The results of the preliminary are the initial
Ministry of Education and Culture explains that teachers prototypes, and the self-evaluation is carried out and then
must train students in the form of high-level thinking skills followed by expert reviews, one-to-one, small group, and
with the aim of increasing students' ability to think using field test. The eleventh-grade students of SMA Negeri 1
reasoning in answering more complicated questions or OKU in the academic year 2018/2019 were involved as the
solving a case of a more complicated problem. subject research. The document, walkthrough, test,
interview, and observation were used to gather the data.
In the technical guide to integrated thematic learning, the
Ministry of Education and Culture explains that teachers At the preliminary stage, researchers conducted student
must train students in the form of high-level thinking skills analysis, junior high school curriculum analysis, and analysis
or Higher Order Thinking Skill (HOTS), with the aim of of HOTS questions. Next, a problem device design is carried
increasing students' ability to think reasoning to answer out, which includes a grid of questions and questions about
questions that are more complicated and or solve a case of a HOTS high school mathematics. After producing the
more complicated problem. According to [8] Higher Order question grid and HOTS questions, continue in the formative
Thinking Skills are defined therein, including critical evaluation stage.
thinking, logical, reflective, metacognitive, and creative.
At the stage of formative evaluation, the first stage is a
Some relevant studies include [9], who developed the self-evaluation in which the researcher conducts his
Higher Order Thinking Skills (HOTS) questions on the assessment of the design results of the questions that have
volume and cube volume application type based on problem- been made. The result is referred to as prototype 1. A
based learning for eighth-grade students. The results of the prototyping phase is carried out consisting of expert review,
analysis produce high-level thinking skills with a percentage one-to-one, and small group.
value of 41%, where this value includes having a high level
of thinking ability in a good category. [10] with the title The expert review stage is the stage of testing validity
"Development of student worksheets in linear inequality one carried out by experienced experts or educators. The experts
PMRI-based variable level Higher Order Thinking Skill assessed and tested prototype 1 by examining, evaluating,
(HOTS) in class VII." The results of the analysis can make and evaluating using studies in terms of content, constructs,
the students easier to remember and study material in one and language. Expert suggestions are written on the
variable linear inequality and make learning more validation sheet and question card. In parallel (together), the
comfortable and easily understood so that students have no one-to-one stage is also performed. At this stage, the
problem in understanding the meaning of the sentences. prototype 1 was tested to three students as a tester who were
asked to work on the questions that had been developed and
Based on the explanations above, it needs to change the were also asked to give comments/responses to the questions
habits of students using routine questions by developing they had done. The results or findings obtained at the stage
questions that promote high-level thinking or HOTS to of expert review and one-to-one were taken into
develop a high-level mindset of students based on the consideration in revising the prototype 1. After revised, it
purpose of the train and be familiar with HOTS questions. In would produce a prototype 2.
this research, the writers investigated how the characteristics
of valid and practical high-level thinking questions (HOTS) This prototype II is then tested in the small group stage.
statistics materials for senior high school are and what the At this stage, six students were asked to solve the questions
potential effects of high-level thinking questions (HOTS) on on prototype II and also comments on the questions that had
high school statistics are. been done. Comments and findings in the small group stage
were taken into consideration when revising Prototype 2.
This study aims to produce HOTS questions that are The results of the prototype revision 2 are called prototype 3.
valid and practical and see potential effects on student work. Before proceeding to the field test stage, researchers
This research is expected to increase students' knowledge conducted a trial to see the level of difficulty of the
and insight so that they are trained in using HOTS questions. questions, distinguishing power, and also reliability. The
results were analyzed and discussed to produce suggestions
II. METHOD to improve the prototype III to produce the final prototype.
This final prototype was tested on the subjects of the study,
This research was conducted in the form of design namely students of class XII IPA 5, as many as 40 students
research as one type of development research as follows the from SMA N 1 OKU conducted in odd semester 2018/2019.
formative evaluation flow from [11-12]. Which are The results of the field test will be calculated on the scores of
preliminary step and formative evaluation step (self- each student and used as a basis for evaluating students'
evaluation, expert reviews, one-to-one, small group, and high-level thinking skills. The instruments of data collection
field test), at the preliminary step, the researchers determined used in this study are validation and test sheets. The
the place and subject of the study. The researchers also validation sheet is used to obtain information from experts as
contacted the mathematic teachers and curriculum expert judgment to provide input and suggestions about the
representatives to inquire about the procedure in conducting questions produced. Tests are used to obtain information
research. Then, researchers also analyzed the 2013 about the practicality and effectiveness of the questions
curriculum, the HOTS indicators for senior high school. developed and the students' reasoning abilities. The data
Also, the researchers designed a question instrument
consisting of a question grid, a question card, and a scoring

339
 Advances in Social Science, Education and Humanities Research, volume 422

analysis technique used is by assessing the level of complicated, the boundaries of the questions, and the
feasibility, answers were clear.
The quality and accuracy of the instruments produced. 2) Expert Reviews dan One-to-One
The question set developed into account three criteria are The validation process with Dr. Destiniar M.Pd is done
taken from the criteria proposed by [13], which are valid, face to face at PGRI University Palembang. Validation with
practical, and effective. Instruments are said to be valid if the the Dr. Bambang Suprihatin M.Si was done by giving the
results of validation from experts say that the questions device that was developed to his home after a while later, the
developed are valid both in terms of content, constructs, and new researcher received the validation result.
language. In addition, the instrument is said to be practical if
the questions developed can be used by all mathematical After a test to three students of class XII IPA 1 SMA
education practitioners, and experts who become problem Negeri 1 OKU with different abilities. The three students are
validators state that the questions developed can be applied. AZ, RS, and YL. The three students were asked to work on
While the instrument is said to be effective [14] if experts the questions amounting to twelve questions, then each
and practitioners based on their experience state that student was asked for their opinions, comments, and advice
instruments (questions) have a potential effect on students' on the questions.
abilities in this is reasoning ability. The comments and suggestions obtained from the
validator to the problem HOTS math prototype 1 can be seen
III. RESULTS AND DISCUSSION in Table 2 as follows:

A. Preliminary Step TABLE II. SUGGESTION AND COMMENTS FROM EXPERT REVIEWS
Students Analysis, the Curriculum Review, and HOTS AND STUDENT ON NUMBER 1 PROBLEM
Questions Analysis Validator Suggestion and Revision
Comments
Dr. Destiniar, Problems and drawings Capitalization in the
TABLE I. STUDENTS ANALYSIS RESULT
M.Pd are obvious where the world cup word was
Proses Student question of the improved to the "World
One to One Three students of XII IPA 8 with redeemed weight Cup."
high ability, moderate ability, and difference is known to Editorial sentences on
low ability the average, so "average weight of each"
Small-Group Six students of XII IPA 8 demanding the students ' in the added player and
two students with high ability reasoning to finish. country words to the
two students with moderate ability Dr. Bambang Improve capitalization in "average weight of each
two students with low ability Suprihatin, M.Si the word world cup player of the country."
Field Test One class: all of the students of XII should be the "World On the average number of
IPA 5 Cup." Editorial bodyweight, players have
sentences on "average a weight unit (Kg).
weight of each" in the In the sentence of the
added player and question, plus the word
Table 1 indicated that the students have different abilities player becomes "if one of
country words to the
in mathematics class. Furthermore, OKU Public High School "average weight of each the World Cup players of
1 tries to understand and present in the concrete domain player of the country." each country is redeemed
(parsing, stringing, modifying, and creating) and abstract On the average number out to get the average
domains (writing, reading, calculating, drawing, and of bodyweight, players weight of the player
compiling) the 2013 curriculum. And the HOTS questions have a weight unit (Kg). becomes the same."
In the sentence of the
analysis was on C4, C5, and C6 levels. question, plus the word
player becomes "if one
B. Formative Evaluation Step of the World Cup
players of each country
1) Self Evaluation is redeemed out to get
In terms of content (content), researchers evaluated the the average weight of
question that had been designed, whether it included the the player becomes the
HOTS questions. In addition, it also evaluated the context, same."
and what level prediction is in accordance with the HOTS Student Problem number 1 is
difficult because it does
indicator then evaluating the question whether the question not understand the intent
was in accordance with the basic competencies in the of his problem.
curriculum. In terms of constructs, researchers evaluate
whether the questions were designed like tables, graphs, Based on comments and suggestions from expert reviews
images on the question of whether it had been presented (six experts) and one-to-one, the questions on prototype 1
clearly, legibly, and functioned. In terms of language, with twice revisions so that they became valid prototype 2
researchers evaluated whether the question language was in and were tested in the small group stage. Decision revision
accordance with EBI (Indonesian Spelling), the question did questions developed based on expert reviews and one to one
not contain multiple interpretations, the questions were not suggestions and comments can be seen in table 3 :

340
 Advances in Social Science, Education and Humanities Research, volume 422

TABLE III. THE REVISED DECISION OF THE PROBLEM DEVELOPED TABLE IV. THE REVISION OF THE QUESTIONS AFTER SMALL GROUP
BASED ON SUGGESTIONS/COMMENTS EXPERT REVIEWS AND ONE TO ONE ON
NUMBER 1 PROBLEM No Before Revision After Revision
2 The median and average of The median and average of points
Before Revision points won by the World won by the World Cup players in
World Cup players from Spanish and Germany. Cup players in group B group B preliminary round are
preliminary round are sorted sorted starting from the smallest is
starting from the smallest is 4.5, and the range is 5. Does Elda
4.5. If the range is 5. Do you argue that the first point of the
think the result of the first fourth point is bigger than the
and fourth points is greater average? What do you think is the
than the average? Give us a opinion of Elda true? Give us a
reason to support your reason to support your answer!
answer!

If one of each country is redeemed, it turns out to get the average weight The results of the revised questions from the small group
to be the same. What do you think the difference in weight is exchanged?
stage are called prototypes 3, which are valid and practical
and can be seen in the booklet. Prototype 3 was then tested
on 40 students of class XII of OKU SMA 1 to find out the
potential effects of the questions developed.
After Revision
World Cup Player
C. Field Test
On the field test step, prototype 3 was tried out to the 40
students of XII IPA 5. After the students did the
questionnaire in order to know their responses to the
questions that had been answered, the researchers analyzed
the students` worksheets to know the potential effects of the
developed questions made by researchers. Picture 1
showed the activity on the field test.

If one of the World Cup players of each country is exchanged, apparently

it gets the average weight to be the same. What do you think the
difference in weight is exchanged

3) Small-Group
Questions that have been revised based on expert
reviews, one-to-one, are called prototypes 2. The prototype 2
is tested on small group students consisting of 6 students of
OKU 1 Public High School with different abilities, namely 2 Fig. 1. Students Activities on Field Test
students with high abilities, 2 students with moderate
abilities, and 2 students with low abilities. To see the Higher order thinking skills of students can
be known based on the results of tests on the questions
Based on the results of the small group, question 2 given to students. Then scoring of student answers and
was revised. The instruction of the question should be scores obtained by students were analyzed descriptively
changed so that the students understand it better. The qualitatively and grouped into categories according to the
sentence that is replaced is, "Do you think the results of the stipulated conditions. After scoring is done based on the
first and fourth points are bigger than the average? Give the higher-order thinking skills of students, the data obtained
reasons that support your answer! " It was changed into from the scoring are categorized based on the following
"Elda believes that the first point multiplied by the fourth table 5:
point is greater than the average? In your opinion, is Elda's
answer correct? Provide the reasons that support your
TABLE V. SCORING SYSTEM
answer!. The revision can be shown in table 4.
Score Criteria
4 Looks three descriptors
3 Looks two descriptors
2 Looks one descriptor
1 Looks zero descriptors

341
 Advances in Social Science, Education and Humanities Research, volume 422

The high order thinking ability score of each student is

the number of scores obtained according to the number of
descriptors that appear when completing a high-level
thinking ability test. The maximum score is the highest
score (score four) multiplied by the number of questions
(nine items), the maximum score is 9 x 4 = 36. While the
minimum score is 9 x 1 = 9, so the average interval score
of students' higher-order thinking skills is 36 - 9 = 27,
researchers divided the interval into four intervals with a
range of seven.
The test results data were then analyzed to determine
the average final score then converted into qualitative data
to determine the level of high order thinking categories of
students. The student-level category is determined as in
the following table 6 : Fig. 2. Answer Question 1

TABLE VI. LEVEL CATEGORY OF HIGHER ORDER THINKING

Interval Level category of HOTS
30 – 36 Very good
23 – 29 Good
16 – 22 Medium
9 – 15 Low

Based on the results of student tests at the time of the

field test obtained the average frequency of student test
distribution at the time of fields can be seen in the following
table 7:
Fig. 3. Answer Question 1
TABLE VII. AVERAGE STUDENT TEST FREQUENCY DISTRIBUTION AT
THE TIME OF THE FIELD TEST
Picture two and three described the students answer on
Interval Categories
Average question 1, the red picture showed the students ability in
Frequency % analyzing data of the world athlete weight and explaining the
30-36 Very Good 3 7,5 information to the smallest point to find out the difference of
23-29 Good 15 37,5
world athlete weight and also identifying the question
16-22 Medium 17 42,5
9-15 Low 5 12,5
correctly based on analysis ability (C4). Then, the green
picture presents about the students are not able to explain the
information so that they cannot answer correctly. From the
The average frequency distribution of student test results
field test analysis result, fifteen of forty students reach the
at field test in forty students, it is known that three students
indicator of analysis ability (C4).
(7.5%) have very good higher-order thinking skills, fifteen
students (37.5%) have a good category of Higher-order The next picture is the strategy of students 'answers to
thinking skills, seventeen students (42.5%) have a medium question number two can be seen in picture four as follows:
category of Higher-order thinking skills, and five (12.5%)
students have a low category Higher order thinking skills.
The following pictures are the explanation of the
questions and the student's answers on the field test step in
order to know the potential effect from the process of
answering the questions and the strategy used by the
students.

Fig. 4. The Answer of Question 2

342
 Advances in Social Science, Education and Humanities Research, volume 422

Picture 3 presents the student's answer on question 2, and questions well. After being valid and practical, the question
the students can give judgment on the given statement “Elda was tested in the field test to find out the potential effects of
argues that the first point times the fourth point is bigger than the problem. Finally, the researchers offered some
average? “In your opinion, is Elda`s statement, right?’. From suggestions: the students should be able to train themselves
the field test analysis, thirteen of forty students were in the to understand or solve math problems better, especially
indicator of evaluation ability, and the others were wrong in HOTS questions, the teachers can provide additional
judging the statement and incorrect in explaining the material for HOTS questions in learning, and for other
information. It can be concluded that the developed researchers, they can provide this result as resources of
questions have the potential effect that can allow the ability subsequent research. Advice for researchers can be used as
of analysis (C-4), evaluation (C-5), and Creating (C-6). reference material for further research. Problems designed
can be images, graphs, diagrams, and tables with little
From the results of the students' work in the field test, it information without having to use long sentences.
can be seen that the questions developed can lead to high-
level thinking skills. The high-level thinking ability that
emerges is analyzing (C4), which is divided into 3 first parts ACKNOWLEDGMENT
capable of recognizing and distinguishing the causes and Thank Dr. Destiniar, M.Pd, and Dr. Bambang Suprihatin,
consequences of a complex scenario, both being able to M.Si For his participation has helped validate the problem in
analyze the information that enters and divide or structure this study. We also thank M.s Yosi teacher mathematics and
information into smaller parts to recognize patterns or her students SMA N 1 OKU for participating in this
relationships. The three can identify/formulate questions, research.
evaluate, and create. Evaluating (C5) is divided into 3
abilities, namely being able to strengthen hypotheses,
REFERENCES
criticize, and test, accept or reject a statement based on
predetermined criteria and be able to provide an assessment [1] OECD. (2010). Draft PISA 2012 Assessment Framework.
http://www.oecd.org/dataoecd/61/15/46241909.pdf. Diakses pada 25
of solutions, ideas, and methodologies using suitable criteria Agustus 2018.
or existing standards for ensuring the value of effectiveness [2] Kemendikbud 2013 Kerangka Dasar dan Struktur Kurikulum
or benefits. Creating (C6) is being able to design a way to 2013.(Jakarta Kememdikbud).
solve a problem, generalize an idea or perspective on [3] Susanti E 2014 Pendidikan Matematika Realistik Berbantuan
something, and organize elements or parts into structures. Komputer Untuk Meningkatkan Higher Order Thingking Skill dan
Mathematical Habits of Mind Siswa SMP Disertasi, (Bandung:
From the results of the students' work in the field test, it Universitas Pendidikan Indonesia).
can be concluded that the questions developed can lead to [4] King, FJ., Ludwika, G., dan Faranak, R. (2011). Higher-Order
HOTS. The result of this study produce HOTS Statistics Thinking Skills. Center for Advancement of Learning and
questions, each consisting of three questions that are Assessment.
analyzing (C4), Five questions are evaluated (C5), and one (http://www.cala.fsu.edu/files/higher_order_thingking_skills.pdf).
Diakses pada 25 February 2019.
question is create (C6) that is valid, practical, and has a
[5] Heong, Y.M., Othman, W.D., Md Yunos, J., Kiong, T.T., Hassan. R.
potential effect based on the HOTS framework. & Mohamad, M.M. (2011) The Level of Marzano Higher-Order
The results of previous studies on HOTS are the research Thinking Skills Among Technical Education Student. International
Journal of Social and Humanity, Vol. 1, No. 2, July 2011, 121-125.
of Marlina, [15] with the title "Development of HOTS
[6] Barry, M., 2012. What skills will you need to succeed in the future?
questions on the type of application of volume cube and Phoenix Forward (online). Tempe, AZ, University of Phoenix.
beam-based problem based learning on class VIII students."
[7] Ananiadou K and Claro M 2009 21st Century Skills and Competences
The results of the analysis of this study produce HOTS for New Millennium Learners in OECD Countries (OECD Education
students with a percentage value of 41%, which means they Working Papers Paris, Publishing) p 41
have high-level thinking skills in both categories. What [8] King F J, Ludwika G dan Faranak R 2011 Higher-Order Thinking
distinguishes this study from previous research in terms of Skills. Center for Advancement of Learning and Assessment.
content, research subjects, and research results from this (http://www.cala.fsu.edu/files/higher_order_thingking_skills.pdf).
Diakses pada 25 Agustus 2018.
study resulted in 9 HOTS mathematics questions that are
valid and practical high school statistics mathematics. [9] Marlina E, Putri R I I & Darmawijoyo 2016 Pengembangan soal
Higher Order Thinking Skill (HOTS) tipe aplikasi volume kubus dan
balok berbasis problem basid learning pada siswa kelas VIII Tesis
IV. CONCLUSION (Palembang: Universitas Sriwijaya)
[10] Riyanti T Zulkardi & Susanti E 2016 Pengembangan lembar kerja
As conclusion, nine HOTS questions on statistical siswa pertidaksamaan linear satu variabel berbasis PMRI level Higher
material were valid and practical and had a potential effect Order Thinking Skill (HOTS) di kelas VII Tesis ( Palembang:
and feasible to be used. The validity of the question could be Universitas Sriwijaya)
shown from the results of the assessment validation from the [11] Tessmer M 1993 Planning and Conducting Formative Evaluations:
expert reviews stage, where the researcher got suggestions Improving the Quality of Education and Training. London: Kogan
and comments in terms of content, constructs, and question Page.
languages. Validity was also obtained from comments or [12] Zulkardi 2006 Formative Evaluation : What, why, when, and how.
(http://www.oocities.org/zulkardi/books.html) Diakses tanggal 1
suggestions from students at the one-to-one stage as a part of September 2018.
the readability of the questions. Practicality in the small [13] Nieveen T P ( November 23-26, 2007). An Introduction to
group stage, students could understand the HOTS math Educational Design Research. Proceeding of the seminar concucted

343
 Advances in Social Science, Education and Humanities Research, volume 422

at the East China Normal University (p. 1 -126). Shanghai (PR [15] Putri I I R, Zulkardi, dkk 2018 Soal HOTS Matematika Tipe PISA
China) : SLO. Netherlands Institute for Curriculum Development. Konteks Cabang Olahraga ASEAN GAMES( Universitas Sriwijaya)
[14] Akker J V D 1999 Principles and Methods of Development
Research dalam (Eds) Design Approches and Tools in Education and
Training (Dordrecht: Klower Academic Publisher)

344