Cady et al.

, Development and Validation of Surveys Measuring Student Engagement in Engineering,

Part 2

DEVELOPMENT AND VALIDATION OF SURVEYS

MEASURING STUDENT ENGAGEMENT IN ENGINEERING,
PART 2
Elizabeth T. Cady
Center for the Advancement of Scholarship on Engineering Education, National Academy of
Engineering, Washington, DC, United States
ecady@nae.edu

Norman L. Fortenberry
Center for the Advancement of Scholarship on Engineering Education, National Academy of
Engineering, Washington, DC, United States
nfortenb@nae.edu

Malcolm P. Drewery
U.S. Census Bureau, Washington, DC, United States
malcolm.p.drewery@census.gov

Stefani Bjorklund
Rankin & Associates Consulting, Howard, PA, United States
stefani@rankin-consulting.com

Abstract: This paper summarizes the development, testing and validation of the
engineering versions of the National Survey of Student Engagement (NSSE) and its
faculty version, the Faculty Survey of Student Engagement (FSSE). These engineering
versions (E-NSSE and E-FSSE) assess the extent to which engineering students are being
engaged by identified “best instructional practices” and are achieving certain learning
outcomes desired of engineering graduates. These surveys were first pilot-tested at six
engineering programs across the United States. Tests of validity and reliability were
conducted on both instruments. The instruments were then refined and shortened based
on the psychometric properties of the items in the original instruments. Ultimately, we
hope to make the instruments available to the national engineering education community
so that they might improve the ways in which they teach tomorrow’s engineers. This
paper will discuss the ongoing progress of both instruments as well as summarize results
obtained from their administration.

Introduction
Several recent reports lament the current state of engineering education (e. g., National Academy of
Engineering [NAE], 2005, 2004; ,National Science Board [NSB], 2007) and call for faculty members
to improve the career preparation that undergraduate engineering students receive (ABET, Inc., 2002).
These improvements include greater attention to differing learning styles among students and using
teaching methods that include all students (Felder & Silverman, 1988). Faculty have also begun
focusing on effective and valid methods of assessing student performance and learning as well as their
own teaching effectiveness (Olds, Moskal, & Miller, 2005). One construct that overlaps these
variables is student engagement.
Although engagement can be defined in many ways, Chen, Lattuca, and Hamilton (2008) used
“quality of effort” (p. 339) to operationalize student engagement. Faculty contribute to student
engagement by creating the instructional practices, professional development activities, and attitudes
that foster student engagement. Students who believe their professors care about them and their
education remain engaged (Chen et al., 2008), and both student outlook and faculty pedagogy affect

Proceedings of the Research in Engineering Education Symposium 2009, Palm Cove, QLD 1
 Cady et al., Development and Validation of Surveys Measuring Student Engagement in Engineering,
Part 2

engagement as well (Smith, Sheppard, Johnson, & Johnson, 2005). The National Survey of Student
Engagement (NSSE, 2007) and the Faculty Survey of Student Engagement (FSSE, 2006) are well-
known and measure the many aspects of student engagement, but are not specific to engineering. The
current project aims to develop, test, and validate engineering-specific versions of those surveys (E-
NSSE and E-FSSE). Work on this project is ongoing and has been described previously (Cady,
Fortenberry, Drewery, & Bjorklund, 2009; Drewery & Fortenberry, 2007; Bjorklund & Fortenberry,
2005; Cupp, Moore, & Fortenberry, 2004; Moore, Cupp, & Fortenberry, 2004). This paper presents
results from recent validation and reliability testing. This phase of the project examined the test-retest
reliability of the two surveys by correlating the item answers given by the same individual in two
separate survey administrations.
Methodology
Engineering faculty members (19) and students (261) from five undergraduate institutions completed
the current pilot testing for the E-NSSE and E-FSSE. Testing took place in the spring and fall
semesters of 2008. As with prior survey administrations, the instruments were translated into online
questionnaires using FormSite (www.formsite.com). All respondents completed the survey and were
sent a reminder email to complete it again the following week. Respondents were told in recruiting
letters that if they completed the survey twice they would be entered into a drawing for a cash prize.
After the respondents completed the questionnaire a second time they were taken to a page to enter
contact information for the cash drawing, but identifying information was kept separate from survey
answers. Data were examined and analyzed in SPSS.
Results
Nineteen engineering faculty members completed the surveys twice. The average length of
appointment was 21.5 years, although experience ranged from less than a year as a faculty member to
over 40 years. Of the 19 professors, 13 were male and 13 came from chemical, civil, or mechanical
engineering. The remaining 6 faculty members were aerospace, computer, electrical, or materials
engineering faculty members, and one respondent did not indicate a discipline. For the E-FSSE, the
test-retest reliability was calculated using the correlations for Time 1 and Time 2 responses to the
items within the factors identified in Drewery and Fortenberry (2007). Overall, the components
correlated moderately well both within factors and across time. The item correlations ranged from -.25
to 1.0, and 112 of the 142 total questions correlated significantly from Time 1 to Time 2. The E-FSSE
factors are listed in Table 1, and more complete results are presented in Cady, et al (2009).
Table 1: Student Outcomes and Instructional Practices Scales for E-FSSE

Student Outcomes Instructional Practices

An ability to apply knowledge of mathematics, science, and Encourage student-faculty interaction
engineering *
An ability to design and conduct experiments, as well as to Develop reciprocity and cooperation among
analyze and interpret data * + students *
An ability to design a system, component, or process to meet Communicate high expectations
desired needs * +
An ability to function on multi-disciplinary teams * + Give students feedback
An ability to identify, formulate, and solve engineering Use active learning techniques * +
problems * +
An understanding of professional and ethical responsibility * + Emphasize time on task
An ability to communicate effectively * + Respect diverse talents and ways of thinking
The broad education necessary to understand the impact of Build on correct preexisting understandings,
engineering solutions in a global and societal context * + dispel false preconceptions
A recognition of the need for, and an ability to engage in, Provide factual knowledge, facilitate
lifelong learning * understanding of facts and ideas in context of a
conceptual framework and organizing knowledge
that facilitates retrieval of application
A knowledge of contemporary issues * Encourage students’ motivation to learn *

Proceedings of the Research in Engineering Education Symposium 2009, Palm Cove, QLD 2
 Cady et al., Development and Validation of Surveys Measuring Student Engagement in Engineering,
Part 2

An ability to use the techniques, skills, and modern

engineering tools necessary for engineering practice *
An ability to manage a project (including familiarity with
business, market-related, and financial matters) * +
A multidisciplinary systems perspective * +
An understanding of and appreciation for the diversity of
students, faculty, staff, colleagues, and customers *
A strong work ethic
* Cronbach’s α for factor greater than .7 + Test-retest correlations significant for all items in factor
Two hundred sixty-one students completed the surveys twice. Seventy-three respondents indicated a
major other than those listed on the survey, 55 students were mechanical engineering students, 35
were electrical, 33 were civil, and 27 were chemical engineering students. Another 14 were computer
engineering students, 11 were aerospace, 10 were industrial, and 3 were materials engineering
students. The majority were male (184, 70%), full-time students (257, 99%), and had started college at
their current institution (221, 85%). Self-reported grades of the respondents were generally above
average, with 96% indicating that most of their grades were above a C+ and 74% indicating that most
of their grades were B+ or higher. The students ranged in their expected graduation dates, with many
indicating they would graduate in Spring 2010 (51, 20%), Spring 2011 (59, 23%), Spring 2009 (35,
13%), or Spring 2012 (31, 12%).
As with the prior analysis of the faculty responses, the student data were examined using an
exploratory factor analysis with varimax rotation. Because this step had not been completed prior to
the current testing phase, the Time 1 responses were used for this analysis. Originally, 20 factors
resulted from the analysis, but 5 were removed because they did not contain any items with factor
loadings of .4 or greater (Mertler & Vannatta, 2002). The final 15 factors and their Cronbach’s α
reliability, along with the items composing them, are shown in Table 2. For each of the individual
items, the test-retest Pearson’s coefficient was significant.
Table 2: Factors and Item Correlations for Time 1 E-NSSE Responses

Factor, α score Item

General Use basic scientific principles to analyze the performance of processes and system
Engineering Use basic engineering principles to analyze the performance of processes and systems
Skills Formulate and evaluate mathematical models describing the behavior and performance of
α = .979 systems and processes
Design an experiment
Analyze evidence or data from an experiment
Interpret results of an experiment
Use evidence to draw conclusions or make recommendations
Identify essential aspects of the engineering design process
Apply systematic design procedures to open-ended problems
Design solutions to meet desired needs
Identify problems for which there are engineering solutions
Formulate a range of solutions to an engineering problem
Test potential solutions to an engineering problem
Use feedback from an experiment to improve solutions to an engineering problem
Identify potential ethical dilemmas in engineering practice
Estimate the potential for ethical dilemmas due to budget or time constraints
Address ethical issues when working on engineering problems
Apply an engineering code of ethics
Apply technical codes and standards
Convey technical ideas in writing
Convey ideas verbally
Convey ideas in formal presentations
Convey ideas in graphs, figures, etc.
Estimate the impact of engineering solutions in a societal context (in a particular culture,
community, state, nation, etc.)

Proceedings of the Research in Engineering Education Symposium 2009, Palm Cove, QLD 3
 Cady et al., Development and Validation of Surveys Measuring Student Engagement in Engineering,
Part 2

Estimate the impact of engineering solutions in a global context

Apply engineering techniques (e.g., processes, methods) in engineering practice
Apply engineering skills (e.g., experimentation, machining, programming) in engineering
practice
Apply engineering tools (e.g., software, lathes, oscilloscopes) in engineering practice
Integrate engineering techniques, skills, and tools to solve real-world problems
Manage a team’s time to meet deadlines when leading a project
Determine equipment and personnel needed when managing a project
Apply interpersonal skills in managing people
Integrate knowledge and skills learned in engineering disciplines other than their specific
majors
Recognize the need to consult an expert from a discipline other than their own when working
on a project
Recognize the limitations or validity of other professional engineers’ opinions
Consider contemporary issues (economic, environmental, political, aesthetic, etc.) at the
local, national, and world levels
Consider contemporary technical issues in your discipline at the local, national, and world
levels
Estimate how engineering decisions and contemporary issues can impact each other
Use knowledge of contemporary issues to make engineering decisions
Instructors were enthusiastic about engineering research or practice
Instructors were enthusiastic about teaching engineering
Instructors recognized that some students learn in different ways than others
Instructors conveyed material in more than one way (in writing, using diagrams, verbally,
using real-life examples, etc.)
Instructors and Instructors explained new concepts by making explicit links between what students already
Classes Followed know and the new material
Best Practices I have learned to apply fundamentals to problems I haven’t seen before
α = .896 Instructors used simple, common sense examples or metaphors to introduce new concepts
Instructors introduced new concepts by requiring students to engage in hands-on activities,
class discussions, etc.
I found meaning, value, and interest in my engineering course material
My engineering courses had an open and positive atmosphere
I felt like a valued member of the engineering community at my university
I worked cooperatively with other students on course assignments
Students taught and learned from each other
Relationships
Classmates and I worked in groups
With Peers
I discussed ideas with my classmates (individuals or groups)
α = .918
I got feedback on my work or ideas from my classmates
I interacted with classmates outside of class
Work in teams where knowledge and ideas from many disciplines (business, public policy,
engineering, etc.) must be applied
Work in teams where knowledge from many engineering disciplines must be applied
Teamwork
Collaborate with others when working on multidisciplinary teams
α = .923
Communicate effectively with others when working on multidisciplinary teams
Effectively manage conflicts that arise when working on multidisciplinary teams
Do their fair share of the work when working on multidisciplinary teams
I observed the use of offensive words, behaviors, or gestures directed at students because of
their backgrounds or identities
Discriminatory
I observed other engineering students being ignored or excluded (from projects, discussions,
Behavior
lab work, etc.) because of their backgrounds or identities
α = .900
I was harassed or discriminated against by others in my major because of my background or
identity
Professional and Set and pursue your own learning goals
Personal Growth Take new opportunities for intellectual growth or professional development
α = .816 Seek the latest information or advances in your field
Engage in critical, reliable, and valid self-assessment
Apply new knowledge gained to the practice of engineering

Proceedings of the Research in Engineering Education Symposium 2009, Palm Cove, QLD 4
 Cady et al., Development and Validation of Surveys Measuring Student Engagement in Engineering,
Part 2

Recognize the unique skills, abilities, and contributions of all students in your engineering
courses
Recognize the need for diverse perspectives in solving engineering problems
I interacted with instructors as part of my courses
I interacted with instructors outside of class (office hours, advising, committees, etc.)
Interactions with
Instructors knew my name
Instructors
I used email to communicate with instructors
α = .814
I discussed grades or assignments with my instructors
I received positive feedback from instructors that I can do well in engineering courses
Do you fail to do your best work (reverse coded)
Own Work
Do you turn in completed assignments on time
Habits
Do you complete your share of tasks on time, when working in teams
α = .705
Are you dependable on your coursework
My engineering courses emphasized acceptance of and respect for differences (of opinion,
Respect for background, etc)
Diversity My engineering instructors and I discussed diversity issues
α = .732 My engineering instructors emphasized the importance of diversity in the engineering
workplace
Instructors’ Assignments and activities were clearly explained
Positive Instructors made clear what is expected of students in the way of activities and effort
Behavior Instructors gave me detailed feedback on my work
α = .808 Instructors gave me prompt feedback on my work
Instructors expected a lot of work from me
Negative Instructors expected high quality work from me
Experiences Engineering assignments, projects, or examinations have been too difficult for me to be
α = .504 successful
I felt intimidated by some of my engineering instructors
Working with Comfortable working with engineering clients and colleagues from diverse racial/ethnic
Diverse Others backgrounds
α = .826 Comfortable working with engineering clients and colleagues of the opposite gender
Lifelong Do you seek ways to improve a design or project, even after it’s been turned in
Learning Do you take initiative in your learning process
α = .826
Create and follow a budget when managing a project
Business Skills
Address the business, financial, and market related matters associated with project
α = .826
engineering
My engineering courses’ content reflects contributions of all engineers, including women and
Inclusive
people of color, etc.
Behaviors
Students of all backgrounds/identities participate in class (in discussion, in-class assignments,
α = .620
team projects, etc.)

Conclusions
Overall, the test-retest reliability of the E-FSSE and E-NSSE was satisfactory. The student survey
items were all correlated from Time 1 to Time 2, as were a majority of the faculty survey items.
Interestingly, while the faculty responses yielded several different factors that describe student
outcomes in engineering education, the student responses showed one large scale that was labelled
“General Engineering Skills” because it encompassed a majority of the learning outcomes. Future
research should examine the reasons behind this difference.
The validity of the individual scales was also satisfactory, with most of the factors having reliability
scores above the generally-accepted .7 level. However, future testing is needed to determine whether
the weaker factors should remain as-is in the surveys or should be modified to yield stronger scales. In
addition, confirmatory factor analyses should be conducted with large groups of respondents. The
small sample size of faculty respondents precluded this confirmatory analysis in the present study,
although the exploratory analysis previously conducted indicated the 25 different factors.
These results indicate that the E-NSSE and E-FSSE may be used to determine elements of student
engagement in engineering departments. In particular, the Student Outcomes scales in the E-FSSE had

Proceedings of the Research in Engineering Education Symposium 2009, Palm Cove, QLD 5
 Cady et al., Development and Validation of Surveys Measuring Student Engagement in Engineering,
Part 2

acceptable reliability (with the exception of the “Strong Work Ethic” scale), as did the highly inclusive
“General Engineering Skills” scale in the E-NSSE. The items of the E-NSSE also had significant test-
retest reliability, indicating that the survey items will give consistent and dependable results across
respondents. On the other hand, the Instructional Practices scales on the E-FSSE were less reliable,
and several of the individual items did not have significant test-retest reliability. This indicates that
further testing of the E-FSSE may be necessary.
References
ABET, Inc. (2002). Accreditation board for engineering and technology, inc. 2003-2004 criteria for accrediting
engineering programs. ABET, Inc.: Baltimore, MD.
Bjorklund, S. A., & Fortenberry, N. L. (2005). Linking student learning outcomes to instructional practices –
Phase III. Proceedings of the American Society for Engineering Education Annual Conference and
Exposition. June 12-15, Portland, OR.
Cady, E. T., Fortenberry, N. L., Drewery, M. P., & Bjorklund, S. A. (2009). Validation of surveys measuring
student engagement in engineering, part 2. Proceedings of the American Society for Engineering
Education Annual Conference and Exposition. June 14-17, Austin, TX.
Chen, H. L., Lattuca, L. R., & Hamilton, E. R. (2008). Conceptualizing engagement: Contributions of faculty to
student engagement in engineering. Journal of Engineering Education, 97, 339-353.
Cupp, S. M., Moore, P. D., & Fortenberry, N. L. (2004). Linking student learning outcomes to instructional
practices – Phase I. Proceedings of the American Society for Engineering Education Annual Conference
and Exposition. June 20-23, Salt Lake City, UT.
Drewery, M. & Fortenberry, N. L. (2007). Reliability and validity of the engineering faculty survey of student
engagement (E-FSSE): Validation of surveys measuring student engagement in engineering.
Proceedings of the American Society for Engineering Education Annual Conference and Exposition.
June 24-27, Honolulu, HI.
Faculty Survey of Student Engagement (FSSE). (2006). FSSE 2006. Retrieved February 2, 2009, from
http://www.fsse.iub.edu/html/fsse_survey_options.cfm
Felder, R. M., & Silverman, L. K. (1988). Learning and teaching styles in engineering education. Engineering
Education, 78, 674-681.
Mertler, C. A., & Vannatta, R. A. (2002). Advanced and multivariate statistical methods: Practical application
and interpretation (2nd ed.). Los Angeles, CA: Pyrczak Publishing.
Moore, P. D., Cupp, S. M, & Fortenberry, N. L. (2004). Linking student learning outcomes to instructional
practices – Phase II. 34th ASEE/IEEE Frontiers in Education Conference. October 20-23, Savannah,
GA.
National Academy of Engineering. (2004). The engineer of 2020: Visions of engineering in the new century.
Washington, DC: National Academies Press.
National Academy of Engineering. (2005). Educating the engineering of 2020: Adapting engineering education
to the new century. Washington, DC: National Academies Press.
National Science Board. (2007). Moving forward to improve engineering education (NSB-07-122). Arlington,
VA: National Science Foundation. Available from
http://www.nsf.gov/pubs/2007/nsb07122/nsb07122.pdf.
National Survey of Student Engagement (NSSE). (2007). NSSE 2007 standard version. Retrieved February 2,
2009, from http://www.nsse.iub.edu/html/survey_instruments_2007.cfm
Olds, B. M., Moskal, B. M., & Miller, R. L. (2005). Assessment in engineering education: Evolution, approaches
and future collaborations. Journal of Engineering Education, 94, 13-25.
Smith, K. A., Sheppard, S. D., Johnson, D. W., & Johnson, R. T. (2005). Pedagogies of engagement: Classroom-
based practices. Journal of Engineering Education, 94, 1-15.
Acknowledgements
The authors gratefully acknowledge the support of the National Science Foundation through grant
number DUE- 0618125.
Copyright statement

Copyright © 2009 Authors listed on page 1: The authors assign to the REES organisers and educational non-profit institutions a
non-exclusive licence to use this document for personal use and in courses of instruction provided that the article is used in full
and this copyright statement is reproduced. The authors also grant a non-exclusive licence to REES to publish this document in
full on the World Wide Web (prime sites and mirrors) on CD-ROM and in printed form within the REES 2009 conference
proceedings. Any other usage is prohibited without the express permission of the authors.

Proceedings of the Research in Engineering Education Symposium 2009, Palm Cove, QLD 6