Int. J. Res. Undergrad. Math. Ed.

https://doi.org/10.1007/s40753-018-0073-x

Classifying Combinations: Investigating Undergraduate

Students’ Responses to Different Categories
of Combination Problems

Elise Lockwood 1 & Nicholas H. Wasserman 2 &

William McGuffey 2

# Springer International Publishing AG, part of Springer Nature 2018

Abstract In this paper we report on a survey designed to test whether or not students
differentiated between two different types of problems involving combinations -
problems in which combinations are used to count unordered sets of distinct objects
(a natural, common way to use combinations), and problems in which combinations are
used to count ordered sequences of two (or more) indistinguishable objects (a less
obvious application of combinations). We hypothesized that novice students may
recognize combinations as appropriate for the first type but not for the second type,
and our results support this hypothesis. We briefly discuss the mathematics, share the
results, and offer implications and directions for future research.

Keywords Combinatorics . Binomial coefficients . Discrete mathematics . Counting

problems . Survey

Introduction and Motivation

Combinatorics is one component of discrete mathematics that fosters deep mathemat-

ical thinking, but it is also the source of much difficulty for students at a variety of

* Elise Lockwood
Elise.Lockwood@oregonstate.edu

Nicholas H. Wasserman
wasserman@tc.columbia.edu
William McGuffey
wcm2120@tc.columbia.edu

1
Department of Mathematics, Oregon State University, 064 Kidder Hall, Corvallis, OR 97331, USA
2
Teachers College, Columbia University, 525 West 120th St., New York, NY 10027, USA
 Int. J. Res. Undergrad. Math. Ed.

levels (e.g., Batanero et al. 1997; Eizenberg and Zaslavsky 2004). One fundamental
building block for understanding and solving combinatorial problems are combinations
(i.e., C(n,k)), which count k-element subsets of n-element sets. Combinations are
known as binomial coefficients because of their role in the Binomial Theorem, and
they are prominent in much of the counting and combinatorial activity in which
students engage. However, relatively little has been explicitly studied with regard to
student reasoning about combinations. This study contributes to our understanding of
students’ thinking about combinations, and, in particular, beginning undergraduate
students’ inclination to differentiate between combinatorics problems.

Background Literature

There is much documented evidence for the fact that students struggle with solving
counting problems correctly. Some reasons for difficulties are that counting problems
are hard to verify (Eizenberg and Zaslavsky 2004) and that counting involves a number
of different combinatorial operations and formulas that students struggle to keep
straight. Some researchers (e.g., Batanero et al. 1997; Dubois 1984; Fischbein and
Gazit 1988; Piaget and Inhelder 1975) have studied differences in students’ reasoning
about particular problem types such as permutations and combinations. Some of these
authors showed differences between students’ performances on different problem types
(demonstrating, for example, that students had more difficulty solving combination
problems than permutations problems, particularly after instruction on tree diagrams
(Fischbein and Gazit 1988)). Others suggested that student success was related to the
implicit combinatorial model of the problem (such as whether problems were framed in
terms of a selection, distribution, or partition model (Batanero et al. 1997)). However,
those researchers did not investigate any distinctions within a particular problem type,
such as whether students perceive differences within various combination problems or
permutation problems. We extend such existing work that focuses on students’ under-
standing of combinatorial operations by particularly focusing on students’ conceptions
of combinations.
Others have examined students as they explore the variety of settings in which binomial
coefficients arise. For example, Maher et al. (2011) and colleagues (specifically Speiser
(2011) and Tarlow (2011)) documented several episodes in which students in their
longitudinal study made meaningful connections between binomial coefficients, certain
counting problems, and Pascal’s Triangle. These studies highlight the many connections
that binomial coefficients afford in combinatorial settings, suggesting that it may be
beneficial for students to have a sophisticated understanding of binomial coefficients that
could facilitate these kinds of valuable connections. However, they also demonstrate that
while students were able to make such connections, this came about after considerable
time and effort, and such connections may not be natural or obvious to students.
Our work also builds on a recent study by Lockwood et al. (2015a, b) in which two
undergraduate students reinvented basic counting formulas, including the formula for
combinations. The students in that study could solve some combination problems but
not others. They had an especially difficult time with the Bits problem: BConsider
binary strings that are 256 bits long. How many 256-bit strings contain exactly 75
zeros?^ This was somewhat surprising (and concerning) given they had successfully
Int. J. Res. Undergrad. Math. Ed.

reinvented the combination formula and used it correctly on every other com-
bination problem. In our current study, we characterize a fundamental difference
between common combination problems based on a difference in their encoded
sets of outcomes, which stemmed from contrasting the Bits problem with the
other combination problems. In particular, we characterize outcomes of Catego-
ry I problems as an unordered selection of distinguishable objects and outcomes
of Category II problems as an ordered sequence of indistinguishable objects –
more detail is provided in the next section. We explore the hypothesis that, for
novice students, this distinction in how the set of outcomes is encoded appears
to pose a significant hurdle to their ability to recognize binomial coefficients as
useful in the solutions to both types of problems. We seek to examine quan-
titatively what we have previously observed in a paired teaching experiment
(Lockwood et al. 2015a, b) and anecdotally in our own experiences. Broadly,
we investigate how early undergraduate students answer these two different
categories of combination problems. We do so, in particular, by exploring the
following two research questions: 1) Is there consistency across student re-
sponses to Category I problems and to Category II problems; and 2) Do
individual students solve each type of problem in similar or different ways?

Theoretical Perspective

The Importance of Sets of Outcomes in Solving Counting Problems Lockwood

(2013, 2014) has argued for the importance of focusing on sets of outcomes in solving
counting problems. In an initial model of students’ combinatorial thinking, Lockwood
(2013) proposed that there are three inter-related components that students may draw
upon as they solve counting problems: formulas/expressions, counting processes, and
sets of outcomes. Formulas/expressions are terms involving numbers or variables that
reflect the answer to a counting problem. Counting processes are the series of proce-
dures, either mental or physical, in which one engages while solving a counting
problem. A set of outcomes is the complete set of objects that are being counted in a
problem, and the cardinality of the set of outcomes gives the answer to the counting
problem. Lockwood also pointed out that a given expression might reflect a counting
process, and that various counting processes may impose different respective structures
on the set of outcomes. Lockwood (2014) argued for a set-oriented perspective toward
counting, in which counting Binvolves attending to sets of outcomes as an intrinsic
component of solving counting problems^ (p. 31). In this paper, we conceptualize our
work from this perspective, namely that sets of outcomes are the key factor in
determining what counting situation one is in, and, thus, what counting process and
formula may be appropriate.
The nature of our data is such that we are, primarily, only able to examine a student’s
written expression. This information does not necessarily allow us to investigate
students’ counting processes or conceptualizations of sets of outcomes in depth.
However, Lockwood (2013) has suggested that counting processes and sets of out-
comes can and do underlie formulas and expressions; thus, we view differences in
formulas/expressions on combination problems as a likely indication of differences in
counting processes and conceptions of sets of outcomes.
 Int. J. Res. Undergrad. Math. Ed.

Mathematical Discussion – Classifying Two Categories of Combination

Problems From our observations, both in the paired teaching experiment (Lockwood
et al. 2015a, b) and anecdotally, we have some evidence that students did not respond to
particular combination problems (e.g., the Bits problem) in the same way they did to
others. In attempting to characterize why some combination problems may have been
different from others, we drew on our sets of outcomes perspective to generate a
hypothesis. Namely, we identified a meaningful difference between the sets of out-
comes in the combination problems they were responding to differently. This formed
the basis of investigation in our study, as we were then interested in exploring whether
this difference held up more broadly. We elaborate on this difference here.
We want to acknowledge that to some readers, the categories may seem superficial
or unnecessary, since both categories of problems can be solved using binomial
coefficients, and the outcomes can be modeled as sets of distinct objects. Indeed, we
are articulating this distinction so that we may study and explain a phenomenon we
have observed, not because we necessarily think this is how these different combination
problems are (or must be) conceived. Indeed, we have some indication (Lockwood
et al. 2015a, b) that students do not view such problems as similar, and we articulate the
categories to elaborate and test our hypothesis.
A combination is a set of distinct objects, as opposed to a permutation, which is an
arrangement of distinct objects. Combinations can also be described as the solution to
counting problems that count Bdistinguishable objects^ (without repetition), where
Border does not matter^ (in the sense that outcomes in which elements are ordered
differently should not be considered distinct outcomes). The total number of combina-
tions of size k from a set of n distinct objects is denoted C(n,k) and is verbalized as Bn
choose k.^1 So, the binomial coefficient C(n,k) represents the set of all combinations of
k objects from n distinct objects.
In this paper, we refer to combination problems as problems that can be solved using
binomial coefficients, in the sense that parts of their outcomes can be appropriately
encoded as sets of distinct objects. Sometimes this encoding is fairly straightforward, as
the outcomes are very apparently sets of distinct objects. For instance, consider the
Basketball Problem (stated in Table 1). The players could be encoded as the numbers 1
through 12, and the outcomes are fairly naturally modeled as 7-element sets taken from
the set of the numbers 1 through 12. Any such set is in direct correspondence with a
desired outcome; there are C(12,7) such sets. We call such problems, which involve an
unordered selection of distinguishable objects, Category I problems.
In other situations (such as the Bits problem), combination problems may still
appropriately be solved using a binomial coefficient, but there is a different encoding
of outcomes as sets of distinct objects. For example, in the Coin Flips problem (stated
in Table 1) a natural way to model an outcome in this problem is as an ordered
sequence of length 5 consisting of 3 (identical) Hs and 2 (identical) Ts, such as
HHTHT. Solving the problem is a matter of counting such sequences. One way to
count the number of such sequences is to arrange all of the five letters (in 5! ways), and
then divide out the repetitive outcomes based on the fact that there are three identical Hs
(3!) and two identical Ts (2!) – yielding a solution of 5!/(3!2!). However, we note that
the problem also can be solved using a binomial coefficient, and in order to do this, the

1
The derivation of the formula for C(n,k) as n!/((n – k)!k!) is not pertinent to the study.
Int. J. Res. Undergrad. Math. Ed.

Table 1 Characterizing two different categories of combination problems

Description Example problem Natural model for

outcomes

Category I An unordered selection of Basketball Problem. There are 12 athletes who {{1,2,3,4,5,6,7},
distinguishable objects try out for the basketball team – which can {1,3,5,7,9,11,
take exactly 7 players. How many different 12}, …}
basketball team rosters could there be?
Category II An ordered sequence of two Coin Flips Problem. Fred flipped a coin 5 {(HHTTH),
(or more) times, recording the result (Head or Tail) (HTHHT),
indistinguishable objects each time. In how many different ways (TTHHH), …}
could Fred get a sequence of 5 flips with
exactly 3 Heads?

outcomes must be encoded as a set of distinct positions in which the Hs are placed.
Given the five possible distinct positions (i.e., the set: {1, 2, 3, 4, 5}), the outcome
HHTHT would be encoded as the set {1, 2, 4}, for the positions of the Hs. In this way,
the answer to the Coin Flips problem is simply the number of 3-element subsets from 5
distinct objects (i.e., positions 1 to 5), which is C(5,3). This gives an identical formula
of 5!/(3!2!), and it is another way of solving the problem. We call these problems,
which can naturally be modeled as (ordered) sequences of indistinguishable objects,
but which can be encoded so as to be solved via a binomial coefficient, Category II
problems (see Table 1).
Combinations are applicable in both situations, because both can be encoded
as sets of distinct objects, but we argue that there could be a difference for
students in identifying both problems as counting combinations. Indeed, al-
though both categories can be, and frequently are, thought of in terms of
counting sets of objects, the use of a combination to solve Category II
problems would involve properly encoding the outcomes with a corresponding
set of distinct objects (e.g., {1, 2, 4}), rather than what one might consider to
be the natural model (e.g., HHTHT) for such outcomes. We thus posit that
Category I problems may be more natural problems on which novice students
might use binomial coefficients – i.e., more clearly representative of combina-
tion problems than Category II problems. This hypothesis was in accord with
the findings from Lockwood et al. (2015a, b).
In spite of the widespread applicability of combinations, we suggest that early
undergraduate students may not recognize both categories of problems as problems
involving combinations, despite the fact that both categories are common introductory
combinatorics problems. We also note that, despite the difference we have conceptu-
alized between their sets of outcomes, in textbooks both Category I and II combinations
problems tend to appear together, without explicit distinction. Indeed, in Rosen (2012)
both kinds of problems are included in exercises without further comment, and in Epp
(2004) and Tucker (2002) Category II problems are given as examples in the text but
are not treated as different than any other combination problems. Again, this is not
necessarily surprising, and we do not disagree with these authors. However, the key
point is that this may be a distinction that is typically not explicitly emphasized but that
is meaningful for students.
 Int. J. Res. Undergrad. Math. Ed.

Lastly, we note that it is also important and useful for students to be able to solve
these Category II problems, specifically using combinations to do so, because combi-
nations often arise as a stage in the counting process. For example, consider a problem
such as: Passwords consist of 8 upper-case letters. How many such passwords contain
exactly 3 Es? This problem can be solved by using combinations as a stage in the
counting process – first we select 3 of the 8 positions in which to place Es (there are
C(8,3) ways to do this), and then we fill in the remaining position with any of the 25
non-E letters (there are 255 ways to do this). Thus, the two-stage process yields an
answer of C(8,3)*255. Given that the ability to solve Category II combination problems
may allow students to solve a wider range of problems, and that it can reinforce a more
complete understanding of what binomial coefficients can do, we are motivated to
investigate whether or not students respond differently to the two different problem
categories.

Methodology

Survey We designed a survey instrument to answer the research questions, and we

briefly elaborate on relevant design features. First, the theoretical distinction we made
between combination problems allowed us to generate several problems of each for the
survey instrument. Doing so was important, as it enabled us to explore not just one but
several problems of each type in order to investigate whether students recognized our
hypothesized distinction more broadly. In addition, we also designed a layer of
complexity – simple or multistep2 – into the problems (in this report, however, we
do not focus on this distinction). The survey instrument consisted of 11 combinatorics
problems: four Category I (simple), three Category II (simple), one Category I (mul-
tistep), one Category II (multistep), and two dummy problems (which were included to
discourage students from assuming that every problem could be solved with a combi-
nation.) Second, in addition to generating several problems of each category, we created
two versions of our survey instrument. Structurally speaking, these two surveys were
identical in nature, containing the same number of problems of each category and
complexity. This use of two surveys, which allowed us to give even more problems for
each category, also reduced the chance that the observed phenomenon was limited to
the particular problems on one survey. Notably, there was no statistical difference in our
results between the surveys, further validating the reliability of the results, and so in this
report we join the results from the two surveys for the sake of simplicity. Third, we also
asked students to report what potentially relevant courses they had taken, including
Statistics or Discrete/Finite Mathematics in high school, and Statistics, Probability, or
Discrete Mathematics in college. Participants who had taken one of these courses were
considered to have combinatorics-related course experience.

Participants In this study, we targeted Calculus students. We did so because – even if

they only had typical course experience (i.e., had no combinatorics-related course

2
Simple combination problems refer to those that can be solved using a single binomial coefficient; multistep
combination problems would require multiple binomial coefficients in the solution (see Table 2 for problems
in Survey 1 and the Appendix for Survey 2).
Int. J. Res. Undergrad. Math. Ed.

Table 2 Survey 1

Question Description Problem

Q10 Category I Simple There are 12 athletes who try out for the basketball team –
which can take exactly 7 players. How many different
basketball team rosters could there be?
Q11 Category I Simple There are 8 children, and there are 3 identical lollipops
to give to the children. How many ways could we
distribute the lollipops if no child can have more
than one lollipop?
Q12 Category II Simple There are 3 green cubes and 4 red cubes. Sam is
making Btowers^ using all of the 7 blocks by stacking
the cubes on top of each other. How many different
Btowers^ could Sam make?
Q13 Category II Simple Computers store data using binary notation - an ordered
sequence of 0 s and 1 s. A particular piece of computer
data is 95 digits in length, and it has exactly 12 1s.
How many possible sequences fit this constraint?
Q14 Category II Multistep Stella is stacking ice cream scoops onto a cone. She
has 3 scoops of chocolate, 5 of vanilla, 2 of pistachio,
and 6 of strawberry. How many different ways can she
stack all of the ice cream scoops onto the cone?
Q15 Dummy In Montana, a license plate consists of a sequence
of 3 letters (A-Z), followed by 3 numbers (0–9).
How many different possible license plates are
there in Montana?
Q17 Category I Simple There are 12 points, all different colors, drawn on
a sheet of paper (and no three points are on a line).
How many different possible triangles can be
made from these 12 points?
Q18 Category I Simple Bob got a new job and is at a store looking for
new ties. The tie rack has 196 different ties to
choose from. In how many ways can Bob
select 10 ties to buy?
Q19 Category II Simple Fred flipped a coin 25 times, recording the result
(Head or Tail) each time. In how many different
ways could Fred get a sequence of 25 flips with
exactly 11 Heads?
Q20 Category I Multistep There are 8 females and 10 males who would like
to be on a committee. How many different committees
of 6 people could there be if there need to be exactly
2 females on the committee?
Q21 Dummy From an Olympic field of 15 athletes competing in the
100-m race, how many different possible results could there
be for gold, silver, and bronze medals?

experience) – we believed they would have been likely to have encountered combina-
tions at some point in their mathematical careers (likely in middle or high school, since
many such curricula and textbooks have a section on permutations and combinations)
yet still unlikely to have studied them in detail (since Calculus courses are typically
some of the first mathematics courses taken by undergraduates), making them informed
but novice counters. We sent the survey to students enrolled in calculus courses at two
universities during one particular term, and respondents could choose to be entered into
a drawing for one of five gift cards. Overall, 126 people (65 for Survey 1, 61 for Survey
2) responded to at least one of the combination questions. Of the 126 students, 48 had
 Int. J. Res. Undergrad. Math. Ed.

some combinatorics-related course experience, but, as we indicate in the results,

such prior experience did not appear to affect their attempts to solve Category
II problems. At the beginning of the survey, we also included a reminder that
offered a brief standard overview of combinations and permutations (see
Appendix Table 5 for this overview).
The prompt for the combination problems asked participants to use notation that
would suggest their approaches, rather than just numerical values. We gave this prompt
because we wanted to have a sense about the student’s counting process, as evident in
their expanded expression, and so we would not have to make inferences based only on
numerical responses. Specifically, the prompt was:

Read each problem and input your solution in the text box. Please write a
solution to the problem that indicates your approach. If you're not sure, input
your best guess. NOTE: Appropriate notation includes: 9+20, C(5,2), 5C2,
21*9*3, 5*5*5*5=5^4, 8!, 8!/5! = 8*7*6 = P(8,3), C(10,2)*3, Sum(i,i,1,10),
12!/(5!*7!), etc. Only if you individually count all of the outcomes should you
input a numerical answer, such as 35.

In addition, for three of the problems, participants were prompted to expand

particularly on how they understood the set of outcomes they were counting (see
Appendix for this prompt.) The intent was to elicit further evidence from the
students about their counting process and the corresponding set of outcomes.
Although many students had difficulties with the overall instructions, frequently
providing numerical answers in their initial responses, taken together the various
prompts yielded useful information.

Analysis The first phase of our analysis involved coding responses, by which we mean
their answer to the problem. Many of the participants initially only included numerical
answers, for which we could only confirm correctness, but not process. For those
participants who wrote a response indicating their approach, we coded the Bdefinite^
method that characterized their response. We coded whether they used a combination
(C), further indicating correctness (CC) or incorrectness (CI), and, if not, we coded if
their answer involved: a permutation (P); multiplying numbers (M); only involved
factorials not in a permutation or combination formula (F); exponents (E); only a sum
(S); a single number stated in the problem (N). We coded (O) if it did not fall into any of
the previous categories. Although we do not share or analyze the methods beyond (C),
we coded the methods so as to clarify for ourselves some of the different approaches
students attempted. We also coded whether the response was Bcorrect^ or Bincorrect.^
As noted, for three of the problems, participants had an additional opportunity to
explain more about their response in relation to the sets of outcomes. For some
participants who wrote numerical expressions, it was here that they explained their
process, which frequently included an expression for the solution indicative of their
approach (e.g., 8*7*6). Therefore, in these cases, when the numerical answer and their
process description aligned, we included a Bdefinite^ method based on their explana-
tion. From some responses, however, the method would not be completely clear.
Particularly considering combinations, we had to determine the meaning of a response
such as: B8!/(5!3!).^ Thankfully, there were very few such instances. For participants who
Int. J. Res. Undergrad. Math. Ed.

Table 3 Coding of select problems for one participant

ID# Q10 Q11 Q17 Q18

55 ANS: 225,792,840 I chose a combination ANS: 336 ANS: ANS: 6

CODE: CC for this problem so I CODE: P 479,001,600 CODE: n/a
(The explanation plugged into my (The formula CODE: F (Could not
indicates graphing answer is (The formula determine
C(32,12), calculator 32 nCr 12 8P3) answer is 12!) probable
which is method)
correct)

appeared to be using the verbatim combination formula, such as B8!/((8–3)!3!),^ we coded

this as a combination approach; however, for participants who wrote B8!/(5!3!),^ unless
there was a clear indication from the set of outcomes responses or if they had used
combination notation on other problems, we presumed their response to be indicative of a
permutation approach. The coding was often informed by the additional written descrip-
tions. Thirdly, if particular numerical answers were common on a problem and seemed to
have a clear process – such as the problem including the numbers 8 and 5, and participants
answering 40 – we coded the Bprobable^ method that characterized their response. This
provided a set of responses for each participant, and across the entire survey, for which we
were able to attain a Bprobable^ method for the participant’s approach. Any responses for
which we could not determine a probable method based on their answers were not included
in the analysis.
We use four responses from one participant on Survey 1 to clarify this process – see
Table 3. Participant 55’s additional written description on Q10 allowed us to determine
the definite method of a (correct) combination. Q11 and Q17 did not have written
portions other than the answers, but based on the numerical answers given, since a
number of other participants also gave, respectively, permutation and factorial re-
sponses to these two questions, we assigned probable codes that matched these
methods. We also noted the answers were both incorrect. For Q18, the answer of 6
was unusual, and we could not determine a probable method; therefore, this one
response from participant 55 was removed from further analysis.
We consider this coding involving the union of Bdefinite^ and Bprobable^ codes – that is,
they had a recognizable solution approach – as the best set of codes for our analysis for two
reasons. First, some students gave correct numerical responses, such as 4,457,400, which,
even without further description, we viewed as most likely indicative of using C(25,11)
because coming up with this answer in some other way would be unlikely. This allowed us
to include C(25,11) as their probable method. There were very few instances (n = 14) where
this actually occurred in our coding, which means that nearly all of the coding of participants
using combinations (which are of particular interest in this study) were directly evident in
their responses. Second, because many responses were only numerical, this also allowed us
to include a larger number of responses in the analysis for which we could be fairly sure of
their method.
For the purposes of this paper, we limit our analysis to the four simple Category I
problems and the three simple Category II problems on each survey. These seven problems
provide us with the most basic comparison between the students’ responses to these two
categories of problems. Our analysis here included the following phases. First, in order to
 Int. J. Res. Undergrad. Math. Ed.

answer the first research question, we analyzed participants’ responses by problem. In

particular, we considered the proportion of responses that used a combination approach
(C) on each problem – as well as whether or not they had done so correctly or incorrectly.
This analysis allowed us to investigate whether or not there was consistency across the
various problems of each category. Second, in order to answer the second research question,
we analyzed responses by participant. For each individual, we computed whether or not that
individual had used a combination approach on at least one Category I problem, and whether
they had used a combination approach on at least one Category II problem. That is, in some
ways, we are giving participants the benefit of the doubt; if they used a combination approach
on any Category I or Category II problem, we are Bgiving them credit^ for this. We also
computed the proportion of participants that had used the combination approach correctly on
at least one of each category of problem. This allowed us to compare the proportion of
distinct participants (not distinct problems) who had used a combination approach correctly
at least once on the differing problem categories. This analysis allowed us to probe further
whether participants that found a combination approach useful on at least one problem were
differentiating between the two problem categories. For proportion comparisons in both
analyses, because the size of the samples were large enough and the proportion values were
high enough, the use of a two-sample proportion t-test is appropriate (as opposed to an
alternative non-parametric test) for determining whether the proportions were significantly
different from one another. We then used Cohen’s h to determine the relative effect size of the
differences. (We used standard cutoffs from Cohen (1988) of: 0.2, small effect; 0.5, medium;
and 0.8, large.) Notably, when separating the analysis by survey, the results in every analysis
were similar, and so we present the combined analysis across both surveys.

Findings

In this section, we present the different analyses of our data, which all support the
singular finding that students do indeed use a combination approach more on Category
I problems than they do on Category II problems. We regard this result as indicating
that from a learners’ perspective there is a difference between these two categories of
combination problems, and we suspect this difference is due to the theoretical distinc-
tions between the encoded sets of outcomes for the two categories of problems, which
underpinned the design of our survey instrument. We demonstrate these findings in two
parts, by analyzing across problems and across participants.
First, we consider our analysis of responses, analyzing by problem. Based on
Table 4, we see that across all participants, the proportion of uses of a combination
approach are generally higher for each of the Category I problems than the Category II
problems. As a reminder, Bused combination approach^ indicates that the Bdefinite^ or
Bprobable^ method for solving the problem was a combination.
Across all eight Category I and all six Category II problems, Table 4 indicates a
general consistency in the proportion of responses that used a combination approach.3
3
We note that one of the Category I problems (Q17 from Survey 2) and one of the Category II problems (Q19
from Survey 1) had rates that were not consistent with the other problems. We speculate that both problems
involved familiar contexts, which might have been a factor for students. This phenomenon underscores the
value of having given out multiple surveys with multiple questions so as to avoid any one problem unduly
influencing the findings.
Int. J. Res. Undergrad. Math. Ed.

Table 4 Proportions of combination approaches based on problems

Category I (simple) problems Category II (simple) problems

Survey 1 Survey 2 Survey 1 Survey 2
Q10 Q11 Q17 Q18 Q10 Q11 Q17 Q18 Q12 Q13 Q19 Q12 Q13 Q19

Answered question 55 55 41 46 53 44 44 42 54 37 39 52 41 38
Used combination 15 13 10 15 14 9 4 10 8 6 11 7 3 5
approach
Proportion 0.27 0.24 0.24 0.33 0.26 0.20 0.09 0.24 0.15 0.16 0.28 0.13 0.07 0.13
Overall 0.24 0.15

This consistency supports the Category I and II distinctions that underpinned

the survey’s design. Overall, on Category I problems a combination was used
in 90/380 (0.24) of the responses, while a combination was used in 40/261
(0.15) of the responses to Category II problems (this is a significant difference
(p < 0.01) with a small effect size (h = 0.212) in the likelihood of using a
combination approach). Given this relatively consistent difference across a
broad set of problems of each Category, it appears that undergraduates use a
combination approach somewhat more frequently on Category I problems.
Additionally, when students used a combination approach, they appear to be
much more likely to have used it correctly when answering a Category I
problem compared to a Category II problem. Indeed, 88/90 (0.98) of the
combination responses were correct on Category I problems, compared to 29/
40 (0.73) on the Category II problems.
Second, we focus our analysis on the participants, analyzing by participant.
Of the 126 participants, there were 122 who had a Bdefinite^ or Bprobable^
approach code on at least one of the Category I (simple) or Category II
(simple) problems. In Fig. 1, we see that a large proportion of participants,
80/122 (65.6%), did not use a combination approach on any of these problems.
There are multiple potential reasons for this, including that these students
simply had no experience with combinations or permutations, although we note
that more than one-third (28 of the 80) of them indicated having at least some
combinatorics-related course experience.
With this in mind, however, the most interesting analysis for the purposes of
our study comes from analyzing the 42 remaining participants who did use a
combination approach on at least one problem (within the circles of the Venn
Diagram in Fig. 1). By attempting to use a combination approach on at least
one of these problems, we interpret that these participants indicated at least
some knowledge of combinations pertinent to the two categories of problems.
Just under half (of the participants 20/42) attempted to use a combination
approach on at least one problem in each category, with almost all of the rest
only attempting a combination approach on Category I problems. Overall, 38/42
(0.91) used a combination on at least one Category I problem, whereas only 24/
42 (0.57) used a combination on at least one Category II problem. That is, 18/
42 (0.43) participants did not see a combination as useful on any Category II
 Int. J. Res. Undergrad. Math. Ed.

Fig. 1 Individual students’ approaches and tendency to use combinations

problem, whereas nearly all of these participants recognized at least one

Category I problem as related to combinations. In other words, the participants
were about twice as likely to attempt to use a combination on Category I
problems than Category II problems. We note that these results are essentially
identical across those who did or did not indicate having some combinatorics-
related course experience, ruling out the possibility that it was only the Bless
familiar^ students who were not using combinations on the Category II
problems.
Figure 2 adds in the notion of Bcorrectness^ – we now have the subset(s) of
participants who correctly used a combination approach (on at least one prob-
lem) for each category of problems. Nearly all of the movement in Fig. 2
(compared to Fig. 1) is related to Category II problems. That is, for a quarter
(6/24) of those who attempted to use a combination approach on Category II
problems, they were doing so incorrectly each time, whereas nearly all (37/38)
who attempted to use a combination approach on Category I problems were
doing it correctly. In sum, 37/42 (0.88) participants used a combination ap-
proach correctly on at least one Category I problem, compared to 18/42 (0.43)
on a Category II problem (a significant (p < 0.001) difference with large (h =
1.009) effect size). Thus, we gain some additional insight here: namely, that
even though some participants were using a combination approach on Category
II problems, they appear to be doing so incorrectly much more frequently. This
is important in that even when participants appeared to recognize a combination
approach as useful to solve a problem, they were unable to reconcile how to
use a combination on Category II problems more frequently than on Category I
problems. Again, we observe that these results are essentially identical across
those who did or did not indicate having specific combinatorics-related course
experience.
Int. J. Res. Undergrad. Math. Ed.

Fig. 2 The correctness of individual students’ approaches

Discussion

Despite the fact that both Category I and Category II problems can be encoded as
combination problems, our findings suggest that participants do not view these kinds of
problems in this similar way. We found relative consistency across combination
approaches to the various problems of each category, as well as statistically
significant differences in students’ overall use of combinations to solve Cate-
gory I versus Category II (simple) problems. In this way, our study offers
quantitative evidence of what had been an anecdotally observed phenomenon
(Lockwood et al. 2015a, b).
In terms of the model of students’ combinatorial thinking (Lockwood 2013)
and the set-oriented perspective toward counting (Lockwood 2014), one possi-
ble explanation for our findings is that students are not recognizing that
outcomes of Category II problems, which may be more naturally modeled as
ordered sequences of two (or more) indistinguishable objects, can be appropri-
ately encoded as sets of objects. That is, even though to an expert natural
bijections exist that would allow students to leverage binomial coefficients in a
variety of contexts, our research suggests that students are either not aware of
this fact or are not able to use that bijection to encode outcomes effectively. We
are not suggesting that this has to do with students’ difficulties in conceiving of
the concept of a set, but rather that students may not be used to conceiving of
counting as fundamentally involving determining cardinalities of sets of out-
comes. Further, we acknowledge that it is not necessarily surprising that
students would struggle to see this distinction. Indeed, familiar descriptions of
Bunordered^ and Bdistinct^ do not seem to apply – at least in the most natural
way to model the outcomes. It is fairly well established that students tend to
associate counting with key words, specific contexts, and mantras like Border
doesn’t matter,^ and they tend not to think about counting in terms of the
 Int. J. Res. Undergrad. Math. Ed.

outcomes they are trying to count (Lockwood 2014). If this is a student’s

perspective toward counting, he or she may not be attuned to the importance
of encoding outcomes and might not realize that he or she has the flexibility to
encode outcomes in creative ways. Our study thus offers further evidence that
students would benefit from focusing on the nature of the outcomes as the
determining factor in what counting processes (and, ultimately, formulas) are
most appropriate in a given situation.

Limitations and Future Research We acknowledge that we have answered very

specific research questions related to students’ responses to particular types of
combinatorics problems. However, this is an important step in better under-
standing broader issues of students’ encoding and set-oriented thinking, and we
add a quantitative perspective to a phenomenon that had previously been
explored qualitatively (Lockwood et al. 2015a, b). The use of a survey instru-
ment to explore the research question offered us many advantages, particularly
in providing quantitative evidence of an observed phenomenon among a larger
sample of students. Due to the nature of participants’ responses, though, we
were limited to inferring from their answer the counting process they were
using, and many students wrote numerical answers without indicating any
particular approach. Although in many cases we were able to identify the
participants’ approaches definitively, there were still instances where we were
making informed guesses as to their Bprobable.^ And while we, independently,
agreed on probable coding in nearly all of these cases, we recognize the
potential limitations of this approach. However, we regard the benefits of
including a larger proportion of responses as outweighing the potential draw-
backs, and as being the best approach with the data our participants gave us.
An additional limitation is that the brief, standard overview of combinations
and permutations (see Appendix) may have been more helpful for solving
Category I problems than for Category II problems.
As noted previously, we acknowledge that our study is limited in that it
involved students whose precise previous exposure to combinations was un-
known. We queried combinatorics-related course experiences, as opposed to
typical course experiences prior to undergraduate Calculus, but we acknowledge
that even such indications do not necessarily provide in-depth accounts of
participants’ combinatorial exposure. Our findings indicate a meaningful differ-
ence between the participants’ approaches to the two categories of problems,
irrespective even of prior course experience. Still, for us, a natural next step
would be to examine this potential distinction in students who have had much
more counting experience.
As another avenue for further research, we want to explore further the
multistep problems. We wonder if perhaps the multistep problems might be
similar in some way to the Category II problems, because both of these can be
thought of as using a binomial coefficient as part of a process as opposed to a
complete solution. Our findings also indicate that further investigating students’
reasoning about encoding with combinations through in-depth interviews –
rather than a survey instrument – may give insight into the development of
more robust and flexible understandings. Such qualitative studies may require
Int. J. Res. Undergrad. Math. Ed.

additional conceptual analysis of problems involving a variety of combination

problems.

Implications and Conclusions

Based on our findings, we feel that students may require additional exposure to
combinations and may benefit from explicit instruction about how Category II
problems can be encoded in a way that is consistent with Category I problems.
Generally, this point underscores a need for students to become more adept at
combinatorial encoding. Encoding outcomes as sets is an inherent part of the
discipline of combinatorics, but students may need particular help in making
this connection much more explicit. These findings also provide evidence that it
may not be productive for students to be exposed to formulas initially if they
are not encouraged to understand those formulas. In this regard, the results and
implications about combinations from this study are similar to other areas of
mathematics and mathematics education: reliance on procedures or formulas
without sufficient understanding of those procedures or formulas frequently
results in mindless, incoherent, and indiscriminate application of such formulas.
This is not to suggest that instructors do not attempt to teach formulas
meaningfully, but rather that sometimes there may be a disconnect between
what instructors intend and what students interpret or experience.
We offer a couple of pedagogical suggestions that relate to our findings. We
feel that teachers should explicitly direct students toward focusing on what they
are trying to count – i.e., the set of outcomes – but especially as these objects
relate to counting processes and formulas. This means thinking of combination
problems not exclusively as those problems whose outcomes are naturally
modeled by sets of objects, but also those that can be encoded as sets of
objects. Given how difficult (and seemingly unnatural) it is for students to
encode outcomes of Category II problems as combinations, instructors may
need to give examples of the different ways to encode outcomes of Category
II problems and to clearly establish relevant bijections. Discussing the relation-
ship between how one models the set of outcomes and the pertinent solution
approach may also be particularly meaningful in this context. For example,
listing outcomes as 5 Hs and 3 Ts might lead to a permutation approach of 8!/
(5!3!), whereas further encoding the outcomes in terms of the 5 distinct
positions, for which three will be heads, might lead to C(5,3) as the natural
solution approach. This is not to claim one approach as preferential over
another, but we regard having both the flexibility to see different solution
approaches as equally viable in this situation, as well as connecting particular
solution methods with the relevant sets of outcomes, as highly important in
developing an understanding of combinatorics.

Compliance with ethical standards

Conflict of Interest On behalf of all authors, the corresponding author states that there is no conflict of
interest.
 Int. J. Res. Undergrad. Math. Ed.

Appendix

Table 5 Survey 2

Question Description Problem

Q10 Category I Simple You are packing for a trip. Of the 20 different books you
consider packing, you are going to select 4 of them to take
with you. How many different possible combinations
of books could you pack?
Q11 Category I Simple There are 9 justices on the Supreme Court. In theory,
how many different ways could the nine justices
come to a 7:2 vote in favor of the defendant?
Q12 Category II Simple Stella is ordering an ice cream cone that is 8 scoops
tall. She orders 5 chocolate scoops and the rest vanilla.
How many different ways can the employee stack
the ice cream scoops?
Q13 Category II Simple There are 55 elementary students standing in a line.
The teacher has 35 identical red balloons and 20 identical
blue balloons, and gives each student either a red or a blue
balloon. How many different outcomes are possible in this process?
Q14 Category II Multistep Sam is making Btowers^ from 3 green, 4 red, 2 yellow,
and 8 orange blocks. Using all 17 blocks, how many
different Btowers^ could Sam make?
Q15 Dummy In Montana, a license plate consists of a sequence of
3 letters (A-Z), followed by 3 numbers (0–9).
How many different possible license plates are
there in Montana?
Q17 Category I Simple There are 15 people in a room. Everyone shakes
hands with everyone else. How many different
handshakes take place?
Q18 Category I Simple There are 250 kittens at a shelter. Sally is adopting
6 of them. In how many ways could she adopt 6 kittens?
Q19 Category II Simple A professor writes a 40-question True/False test.
If 17 of the questions are true and 23 are false,
how many possible T/F answer keys are possible?
Q20 Category I Multistep There are 19 students in your class. How many ways
are there to split the class into 3 different groups -
one group of size 5, another of size 6, and another of size 8?
Q21 Dummy From an Olympic field of 15 athletes competing
in the 100-m race, how many different possible
results could there be for gold, silver, and bronze medals?

Sets of Outcomes Survey Prompt

Prompt 2: BOn the previous page, you entered your solution to different combinatorics
problems. On this page, we would like you to expand upon how your solution is related
to the Bset of outcomes^ for a few of the problems. That is, we want you to list some
(but not necessarily all) of the outcomes that you are counting. Explain your thinking
for your solution and its relation to what is being counted.
Int. J. Res. Undergrad. Math. Ed.

For example, for the problem, BIf we have four distinct toy cars (Red (R), Blue (B),
Green (G), and Yellow (Y)), how many different subsets of 2 of them are there?^, your
solution to the problem might have been: C(4,2). On this page, the intent is to expand
on how that solution, C(4,2), relates to the set of outcomes. You might write something
like, "The set of outcomes includes the following pairs of cars: BR, RG, GY, BG. I used
the combination C(4,2) because the outcomes were Bpairs^ (2) from the 4 different
colored toy cars. I did not include RB because this would be the same as BR in this
case.^

Overview of Permutations and Combinations

Please read the following page in preparation for the survey questions. You will not be
able to return to this page.

Permutations and Combinations

The factorial of a natural number n is the product of all positive integers up to n.

n! ¼ 1*2*3*…*ðn−1Þ*n OR n! ¼ n*ðn−1Þ*…*2*1

A permutation of n distinct objects is an arrangement, or ordering, of the n objects.

For example, if we have four distinct toy cars, and we want to arrange them in a row,
we would call such arrangements permutations. Suppose the cars are Red (R), Blue (B),
Green (G), and Yellow (Y). Then there are the following 24 permutations of the four
cars:

RBGY GBRY RBGY YBGR

RBYG GBYR RBYG YBRG
RGYB GRBY RGBY YGBR
RGBY GRYB RGYB YGRB
RYBG GYBR RYBG YRBG
RYGB RYRB RYGB YRGB

An r-permutation of n distinct objects is an arrangement using r of the n objects.

P(n,r) denotes the number of r-permutations. For example, if we wanted to count
permutations of size 2 from the four cars, then there are the following 12 2-
permutations (or arrangements of 2 of the 4 cars).

BR GB RB YB
BG GR RG YG
BY GY RY YR

An r-combination of n distinct objects is an unordered selection, or subset, of r out

of the n objects. C(n,r) denotes the number of r-combinations of a set of n objects. For
 Int. J. Res. Undergrad. Math. Ed.

example, if we have four distinct toy cars, and we want subsets of 2 of them, we have
the following 6 2-combinations:

BR RG GY
BG RY
BY

When counting combinations, we only care about the elements in a subset, not in
how those elements are arranged.

References

Batanero, C., Navarro-Pelayo, V., & Godino, J. (1997). Effect of the implicit combinatorial model on
combinatorial reasoning in secondary school pupils. Educational Studies in Mathematics, 32, 181–199.
Cohen. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence
Earlbaum Associates.
Dubois, J. G. (1984). Une systematique des configurations combinatoires simples. Educational Studies in
Mathematics, 15(1), 37–57.
Eizenberg, M. M., & Zaslavsky, O. (2004). Students’ verification strategies for combinatorial problems.
Mathematical Thinking and Learning, 6(1), 15–36.
Epp, S. S. (2004). Discrete mathematics with applications (3rd ed.). Belmont: Brooks/Cole.
Fischbein, E., & Gazit, A. (1988). The combinatorial solving capacity in children and adolescents. ZDM, 5,
193–198.
Lockwood, E. (2013). A model of students’ combinatorial thinking. The Journal of Mathematical Behavior,
32, 251–265. https://doi.org/10.1016/j.jmathb.2013.02.008.
Lockwood, E. (2014). A set-oriented perspective on solving counting problems. For the Learning of
Mathematics, 34(2), 31–37.
Lockwood, E., Swinyard, C. A., & Caughman, J. S. (2015a). Modeling outcomes in combinatorial problem
solving: The case of combinations. In T. Fukawa-Connelly, N. Infante, K. Keene, & M. Zandieh (Eds.),
Proceedings of the 18th Annual Conference on Research on Undergraduate Mathematics Education (pp.
601–696). Pittsburgh: West Virginia University.
Lockwood, E., Swinyard, C. A., & Caughman, J. S. (2015b). Patterns, sets of outcomes, and combinatorial
justification: two students’ reinvention of counting formulas. International Journal of Research in
Undergraduate Mathematics Education, 1(1), 27–62. https://doi.org/10.1007/s40753-015-0001-2.
Maher, C. A., Powell, A. B., & Uptegrove, E. B. (Eds.). (2011). Combinatorics and reasoning: Representing,
justifying, and building isomorphisms. New York: Springer.
Piaget, J., & Inhelder, B. (1975). The origin of the idea of chance in children. New York: W. W. Norton &
Company, Inc..
Rosen, K. H. (2012). Discrete mathematics and its applications (7th ed.). New York: McGraw Hill.
Speiser, R. (2011). Block towers: From concrete objects to conceptual imagination. In C. A. Maher, A. B.
Powell, & E. B. Uptegrove (Eds.), Combinatorics and reasoning: Representing, justifying, and building
isomorphisms (pp. 73–86). New York: Springer.
Tarlow, L. D. (2011). Pizzas, towers, and binomials. In C. A. Maher, A. B. Powell, & E. B. Uptegrove (Eds.),
Combinatorics and reasoning: Representing, justifying, and building isomorphisms (pp. 121–131). New
York: Springer.
Tucker, A. (2002). Applied combinatorics (4th ed.). New York: Wiley.