Walden University

College of Management and Technology

This is to certify that the doctoral study by

W
IE Angel D. Cross

has been found to be complete and satisfactory in all respects,

and that any and all revisions required by
EV
the review committee have been made.

Review Committee
Dr. Steven Case, Committee Chairperson, Information Technology Faculty
PR

Dr. Gail Miles, Committee Member, Information Technology Faculty

Dr. Jodine Burchell, University Reviewer, Information Technology Faculty

Chief Academic Officer and Provost

Sue Subocz, Ph.D.

Walden University
2020
 Abstract

Exploring Software Testing Strategies Used on Software Applications in the Government

by

Angel D. Cross

MS, Strayer University, 2008

BS, Hampton University, 2001

W
IE
Doctoral Study Submitted in Partial Fulfillment

of the Requirements for the Degree of

EV

Doctor of Information Technology

PR

Walden University

March 2020
 Abstract

Developing a defect-free software application is a challenging task. Despite many years

of experience, the intense development of reliable software remains a challenge. For this

reason, software defects identified at the end of the testing phase are more expensive than

those detected sooner. The purpose of this multiple case study is to explore the testing

strategies software developers use to ensure the reliability of software applications in the

government contracting industry. The target population consisted of software developers

from 3 government contracting organizations located along the East Coast region of the

W
United States. Lehman’s laws of software evolution was the conceptual framework. The

data collection process included semistructured interviews with software developers (n =

IE
10), including a review of organizational documents (n = 77). Thematic analysis was

used to identify patterns and codes from the interviews. Member checking activities were
EV

triangulated with organizational documents to produce 4 major themes: (a)

communication and collaboration with all stakeholders, (b) development of well-defined

PR

requirements, (c) focus on thorough documentation, and (d) focus on automation testing.

The results of this study may contribute to information about testing strategies that may

help organizations improve or enhance their testing practices. The results of this study

may serve as a foundation for positive social change by potentially improving citizens’

experience with government software applications as a result of potential improvement in

software testing practices.

PR
EV
IE
W
Exploring Software Testing Strategies Used on Software Applications in the Government

by

Angel D. Cross

MS, Strayer University, 2008

BS, Hampton University, 2001

W
Doctoral Study Submitted in Partial Fulfillment

IE of the Requirements for the Degree of

Doctor of Information Technology

EV

Walden University
PR

March 2020
 ProQuest Number: 27740731

All rights reserved

INFORMATION TO ALL USERS

The quality of this reproduction is dependent on the quality of the copy submitted.

In the unlikely event that the author did not send a complete manuscript
and there are missing pages, these will be noted. Also, if material had to be removed,
a note will indicate the deletion.

W
IE
EV
ProQuest 27740731

Published by ProQuest LLC ( 2020 ). Copyright of the Dissertation is held by the Author.

All Rights Reserved.

PR

This work is protected against unauthorized copying under Title 17, United States Code
Microform Edition © ProQuest LLC.

ProQuest LLC
789 East Eisenhower Parkway
P.O. Box 1346
Ann Arbor, MI 48106 - 1346
 Dedication

I dedicate this study to my family. Thanks to my parents, my brother, and most of

all, my husband, Eddie. There is absolutely no way in the world that I could have

completed this program without your continued support and, most of all, your prayers.

Thank you all for cheering me on as my lifelong dream has manifested into a reality. It is

never too late to make your dreams come true. I know that I have been unavailable on

many occasions while I worked on my research study; in the end, it paid off. You all

have been very supportive and understanding, and I am forever grateful. As the first in

W
the family to achieve a doctoral-level degree, I hope to have made everyone proud. To

my nephews Justin and Jordan, my niece Robyn, and my cousins Tyrie and Tyler, you are
IE
the next generation of leaders. Remember, education is the key to unlocking the world.

To achieve your dreams, remember your ABCs. Finally, I dedicate this study to the
EV

memory of my grandmother, who would have been incredibly proud of this

accomplishment. I miss you a lot. You will always be in my heart.

PR
 Acknowledgments

First and foremost, I want to give honor to my Heavenly Father Jesus Christ, for

making all of this possible. I encouraged myself daily by meditating on scriptures,

Philippians 4:13, Jeremiah 31:17, and Galatians 6:9.

I am blessed and fortunate to have furthered and completed my doctoral-level

studies at Walden University. The education that I received from Walden has added

value to my life, as well as enhanced my corporate skills. Second, I would like to thank

my committee chair, Dr. Steven Case, for his mentorship, support, and guidance during

W
my doctoral journey. The difficulty level would have been extremely high without his

guidance and encouragement. Especially during the times when I was tired and wanted
IE
to throw in the towel.

Third, I would also like to thank my second committee member, Dr. Gail Miles,
EV

for her valuable comments and feedback on making my study complete. Next, I also

would like to thank all the staff, administrators, and instructors who encouraged me along
PR

the way. I even would like to extend special thanks to all of the participants in my study

who so kindly shared information and their time with me. Finally, to my employer, you

helped me make this possible in more ways than you could ever imagine.
 Table of Contents

List of Tables................................................................................................................... v

List of Figures ................................................................................................................ vi

Section 1: Foundation of the Study .................................................................................. 1

Background of the Problem........................................................................................ 1

Problem Statement ..................................................................................................... 3

Purpose Statement ..................................................................................................... 3

Nature of the Study .................................................................................................... 4

W
Research Question ..................................................................................................... 5

Interview Questions .............................................................................................6

IE
Conceptual Framework .............................................................................................. 7

Definition of Terms ................................................................................................... 8

EV

Assumptions, Limitations, and Delimitations ........................................................... 10

Assumptions ...................................................................................................... 10
PR

Limitations ......................................................................................................... 11

Delimitations ..................................................................................................... 12

Significance of the Study ......................................................................................... 12

Contribution to Information Technology Practice ............................................... 12

Implications for Social Change .......................................................................... 13

A Review of the Professional and Academic Literature ............................................ 14

Literature Review Strategy....................................................................................... 16

Lehman’s Laws of Software Evolution............................................................... 17

i
 Lehman’s Laws and Software Evolution ............................................................ 26

Lehman’s Laws of Software Evolution and the Linux Kernel ............................. 39

Lehman’s Laws of Software Evolution and Software Maintenance .................... 41

Complementary and Contrasting Theories .......................................................... 43

Software Testing ................................................................................................ 46

Software Testing Strategies ................................................................................ 63

The Waterfall Methodology ............................................................................... 75

The Agile Methodology ..................................................................................... 78

W
Comparing Agile and Waterfall Methodologies.................................................. 80

Software Testing and The Federal Government .................................................. 84

IE
Transition and Summary .......................................................................................... 88

Section 2: The Project .................................................................................................... 90

EV

Purpose Statement ................................................................................................... 90

Role of the Researcher ............................................................................................. 91

PR

Participants .............................................................................................................. 93

Research Method and Design ................................................................................... 95

Method .............................................................................................................. 95

Research Design ................................................................................................ 97

Population and Sampling ......................................................................................... 99

Ethical Research .................................................................................................... 102

Data Collection ...................................................................................................... 105

Instruments ...................................................................................................... 105

ii
 Data Collection Technique ............................................................................... 106

Data Organization Techniques ......................................................................... 109

Data Analysis Technique ....................................................................................... 111

Reliability and Validity .......................................................................................... 115

Reliability ........................................................................................................ 115

Dependability................................................................................................... 116

Validity ............................................................................................................ 117

Credibility ........................................................................................................ 117

W
Transferability.................................................................................................. 118

Confirmability.................................................................................................. 118
IE
Data Saturation ................................................................................................ 119

Transition and Summary ........................................................................................ 120

EV

Section 3: Application to Professional Practice and Implications for Change ............... 122

Overview of Study ................................................................................................. 122

PR

Presentation of the Findings ................................................................................... 123

Applications to Professional Practice ..................................................................... 175

Implications for Social Change .............................................................................. 178

Recommendations for Action ................................................................................. 179

Recommendations for Further Study ...................................................................... 180

Reflections ............................................................................................................. 182

Summary and Study Conclusions ........................................................................... 183

References ................................................................................................................... 185

iii
Appendix A: Human Subject Research Certificate of Completion ................................ 259

Appendix B: Interview Protocol .................................................................................. 260

Appendix C: Background/Interview Questions ............................................................ 262

Appendix D: Permission to Use Figures #1 .................................................................. 264

Appendix E: Permission to Use Figures #2 .................................................................. 265

Appendix F: Permission to Use Figures #3 .................................................................. 266

Appendix G: Permission to Use Figures #4 .................................................................. 267

Appendix H: Permission to Use Figures #5 .................................................................. 268

W
IE
EV
PR

iv
 List of Tables

Table 1. Statistics for references in Literature Review…………….……...………...... 17

Table 2. Subthemes of Communication with all Stakeholders ………….………...… 131

Table 3. Subthemes of Development of Well-Defined Requirements……….…….…142

Table 4. Subthemes of Focus on Thorough Documentation ……………..….……… 154

Table 5. Subthemes of Focus on Automation Testing……………………….…..….. 166

W
IE
EV
PR

v
 List of Figures

Figure 1. Lehman’s laws of software evolution .............................................................. 21

Figure 2. Types of test documents recommended by IEEE Standard 829 ....................... 57

Figure 3. Classification of software testing .................................................................... 70

Figure 4. The cost curve of change………………………………………………………78

Figure 5. Waterfall versus Agile…………………………………………………………84

W
IE
EV
PR

vi
 1
Section 1: Foundation of the Study

Developing a defect free software application is a challenging task due to the

occurrence of unknown software bugs or unforeseen software defects. The presence of

bugs and defects occur regardless of the guidelines followed in the software development

lifecycle. Consequently, effective software testing is essential to the development and

delivery of reliable software applications. There is no question that the longer a defect

remains undetected, the more expensive it is to fix it. For this reason, software testing

can save time and money by reducing software development and maintenance costs. As

W
with any activity that requires human involvement, the outcome is dependent on human

factors. The purpose of this study was to explore testing strategies that software
IE
developers use to ensure the reliability of software applications in the government
EV
contracting industry. Section 1 contains the foundation of the study, background of the

problem, the problem and purpose statement, the nature of the study, the research

question, the interview questions, the conceptual framework, definition of terms, the
PR

assumptions, limitations, and delimitations, the significance of the study, contribution to

IT practice, the implications for social change and the literature review. Following a

review of the academic literature and professional literature on software testing and

Lehman’s laws of software evolution, I provide a conclusion and then transition to

Section 2 of the study.

Background of the Problem

In this fast-paced age of Information Technology (IT), the release of reliable and

defect free software cannot go unnoticed. As more citizens become computer literate, the
 2
use of computers, laptops, mobile devices, and other computer-based products have

almost dominated our everyday living. All these devices require some form of software

involvement (Batool, 2015). At this moment, software controls every aspect of our daily

lives, ranging from mobile communication devices and interaction on social media

networks to conducting online banking and monitoring our health. Software is a broad

term used to define a set of written instructions that a computer follows to perform a

specific task. The development of a defect free software application is a challenging task

and is of utmost importance. Tomar and Agarwal (2016) developed an application to

W
identify defective software that can benefit software developers by allocating resources

for the release of reliable and defect free software products. As there is interest in the
IE
development of defect free software, the idea to avoid rework is the goal. In addition to
EV
extending time to use, software defects can have more dire consequences. Reports have

showed that software defects wrecked a European satellite launch, delayed the opening of

a newly constructed Denver airport for a year, destroyed a NASA Mars mission, and
PR

killed four marines in a helicopter crash (Oghenovo, 2014). These incidents explain why

software testing is so important to the success of any project.

Software testing is vital to the successful execution of a product. Though an

essential activity in the software development lifecycle, software testing is primarily

conducted to detect any software defects introduced during various phases of the software

development lifecycle (Subramanian, Pendharkar, & Pai, 2017). The main problem is

that software testing activities consume a great deal of the time allocated toward the

overall costs of software development. Therefore, as technology advances, software

 3
testing is more critical today as the potential impact for defective software applications

continues to rise.

Problem Statement

Software errors that are discovered at the end of the testing phase and software

defects that are found by software end-users are much more expensive to fix than defects

that are found at the earliest project phases (Petunova & Berzisa, 2017). Based on a

study conducted at Cambridge University, the results concluded that software developers

spend nearly 50% of their time diagnosing software errors, which leads to an estimated

W
cost of $312 billion per year (Hamill & Goseva-Popstojanova, 2017). The general IT

problem is that software defects impact the reliability of software applications. The
IE
specific IT problem is that some software developers lack testing strategies to ensure the
EV
reliability of software applications in the government contracting industry.

Purpose Statement

The purpose of this qualitative multiple case study was to explore the testing
PR

strategies used by software developers in the government contracting industry to ensure

the reliability of software applications. The targeted population consisted of software

developers from three government contracting industry organizations located along the

East Coast region of the United States. The contributions of this study may help foster a

greater understanding on the part of software developers to improve testing strategies to

ensure the reliability of software applications in the government contracting industry.

Thus, the research findings might contribute to positive social change by possibly
 4
improving the everyday life of citizens, as a result of improvement in the reliability of

software applications in the government contracting industry.

Nature of the Study

The nature of the study is a description and justification of the selection of the

study methodology and design. I selected the qualitative method for this study as it

addressed the research purpose to explore and understand the testing strategies used by

software developers in the government contracting industry to ensure the reliability of

software applications. A qualitative research method is used by researchers to address

W
the ‘how’ and ‘why’ of a story, in ways that quantitative research cannot (Yates &

Leggett, 2016). I selected the qualitative research method for this study because I wanted
IE
to explore and understand ‘how’ the testing strategies used by software developers in the
EV
government contracting industry ensured the reliability of software applications. A

quantitative research method is used by researchers to accept or reject a statistical

hypothesis (Haegele & Hodge, 2015). I did not select a quantitative research method for
PR

this study because the intended focus of the research question is not to accept or reject a

statistical hypothesis. A mixed methods research method is used by researchers to collect

both qualitative and quantitative data (Stockman, 2015). I did not select a mixed methods

research method for this study because the quantitative method has been eliminated. As I

reflect on the probable method, the qualitative method is appropriate for this research

because it addresses the intended focus of the research question.

I selected the case study design for this study to explore and understand the

testing strategies used by software developers in the government contracting industry to

 5
ensure the reliability of software applications. A case study design is a comprehensive

method that incorporates multiple sources of data to provide detailed accounts of

complex research phenomena in real life contexts (Morgan, Pullon, Macdonald,

McKinlay, & Gray, 2017). I selected the multiple case study design because I wanted to

investigate the testing strategies used by software developers in the government

contracting industry to ensure the reliability of software applications. A narrative design

studies the lives of individuals and provides stories about their lives (De Loo, Cooper, &

Manochin, 2015). I did not select a narrative design for this study as understanding the

W
lives of individuals was not the intended focus of the research question. A

phenomenological design describes the lived experiences of individuals or a phenomenon

IE
(Aagard, 2017). I did not select a phenomenological design for this study because
EV
understanding the lived experiences of individuals is not the intended focus of the

research question. An ethnography design studies the shared patterns of behaviors,

languages, and actions of other cultural groups (Badri, Wolfe, Farmer, & Amin, 2018). I
PR

did not select an ethnographic design for this study because shared patterns, languages,

and actions of cultural groups are not the intended focus of the research question. As I

reflected on the probable designs, the multiple case study design was appropriate for this

research because it addressed the intended focus of the research question.

Research Question

What testing strategies do software developers use to ensure the reliability of

software applications in the government contracting industry?

 6
Interview Questions

Background Interview Questions

1. Can you tell me about yourself and your current role?

2. How many years of experience do you have as a software developer?

3. How long have you been performing software testing tasks?

4. What type of project(s) are you currently working on?

Interview Questions

1. What is the primary software development methodology you are using?

W
2. How is software testing organized in your organization?

3. What testing strategies have you used to ensure the reliability of software
IE
applications?
EV
4. How do you assess the effectiveness of the testing strategies used to ensure

the reliability of software applications?

5. How satisfied are you with the development and testing environments that you
PR

have?

6. What challenges have you faced where you find yourself in a disagreement

over a software defect?

7. How has these challenges impacted your testing of software applications?

8. What testing strategies do you find the most effective in detecting software

defects?

9. How much time is allocated for testing software applications in your

organization?
 7
10. What additional information would you like to share about testing strategies

that would ensure the reliability of software applications?

Conceptual Framework

The conceptual framework for this study was driven by Lehman’s laws of

software evolution. Meir Manny Lehman developed the laws of software evolution in

1968 as a result of an investigation into programming practices within IBM (Godfrey &

German, 2014). The analysis prompted a further study of the IBM S/360 operating

system and its successor, IBM S/370 (Godfrey & German, 2014). Lehman’s (1996) work

W
on the laws of software evolution, which he devised and refined with Laszlo Belady and

other collaborators over many years, continues to influence the study of ‘how’ and ‘why’
IE
software applications change over time. Lehman (as cited in Godfrey & German, 2014)
EV
discovered that software developers were becoming increasingly interested in assessing

their productivity, which was measured in terms of daily source lines of code (SLOC)

and passing unit tests. Lehman observed that productivity was increasing according to
PR

the requirements; however, at the same time, software developers appeared to be losing

sight of the overall product (Godfrey & German, 2014). Lehman summarized his

observations about the evolution of software into eight laws which include: (a) continuing

change law, (b) increasing complexity law, (c) self-regulation law, (d) conservation of

organizational stability law, (e) conservation of familiarity law, (f) continuing growth

law, (g) declining quality law, and (h) feedback system law.

In this study, I explored the testing strategies used to ensure the reliability of

software applications. I used the unique lens of Lehman’s laws of software evolution in
 8
my study to understand how attributes of the software evolution phenomenon has an

impact on my study and software process improvement. I selected the law of continuing

change as it suggests that software will become progressively less satisfying to its users’

overtime unless it is adapted to meet new needs. Moreover, the law of continuing change

suggests that software developers must be aware that if their software does not respond

positively to the pressures of the system, that over time, the system will be less appealing

to its users (Godfrey & German, 2014). I also selected the law of increasing complexity

and the law of declining quality. The law of increasing complexity indicated that

W
software would become progressively more complex over time unless explicit work is

completed to reduce its complexity. The law of declining quality indicated that a
IE
software system would be perceived as declining in quality over time unless the design is
EV
carefully maintained and adapted to new operational constraints. Both laws imply that

the changes required to evolve the system to respond to the pressures tend to make the

system more complex and lowers its quality (Godfrey & German, 2014). Finally, I
PR

selected the feedback systems law. The feedback systems law suggested that as software

ages, it tends to become increasingly complicated as a result of the change. The laws of

software evolution were relevant to this study because its core components and constructs

align closely with those that I explored in the research. I intended to use Lehman’s laws

of software evolution as a lens to better understand the study.

Definition of Terms

The following definitions are to assist the reader as these keywords occurred

within this study:

 9
Acceptance testing: A software testing strategy that performs tests to validate

whether the system meets all the specifications and requirements of the customer and

provides assurance that the system is working rather than to find errors (Malik, 2017).

Functional testing: A software testing strategy that discovers disagreements

between the specification and the actual implementation of the software application

(Julia, Vale, & Passos, 2016).

Load testing: A software testing strategy that refers to the practice of assessing

the system behavior under a load. A load is a rate of the incoming requests to the system

W
(Jiang, 2015).

Performance testing: A software testing strategy that determines how fast some
IE
aspects of the system perform under a predefined workload. It is calculated by analyzing
EV
the production, which comes from the application hosted on the server (Khan & Amjad,

2016).

Regression testing: The pragmatic selection of a test suite from tests developed
PR

from other parts of the test process (Parsons, Susnjak, & Lange, 2014).

Software defect: A software bug, error, failure, or flaw found inside the structure

of computer source code or system that is the result of some programmatical mistake

(Deak, Stålhane, & Sindre, 2016).

Software reliability: The chance of failure free software operation for a specified

period (Zhu & Pham, 2018).

Software testing: A phase of the software development lifecycle used to improve

the quality of developed software (Jayaram & Krishnan, 2018).

 10
Test case: A set of written conditions that examines all aspects of the structure

and logic of a software product or software system to validate the functionality of a

specific requirement (Gomez, Cortés-Verdín, & Pardo, 2017).

Unit testing: A software testing strategy which the smallest testable parts of a

program individually and independently analyzed for proper operation (Buckley &

Buckley, 2017).

Assumptions, Limitations, and Delimitations

Any number of phenomena that affect the internal or external factors can

W
influence the research and its outcomes. It is the process of identifying and analyzing

these phenomena that establishes credibility. Three categories of phenomena occur in

IE
research; they are assumptions, limitations, and delimitations.
EV
Assumptions

I made assumptions based on the requirements of the study. Assumptions are

the beliefs, or the preconceptions of the researcher based on instinct or experience that
PR

has not been verified by evidence (Holloway & Galvin, 2016). The achievements of

certain assumptions or preconceptions assumed to be true but not verified for this study

include the following for assumptions. For this study, the first assumption I made was

that software developers representing three government contracting organizations along

the East Coast region of the United States would be available and willing to participate in

this study. The second assumption is that I assumed that participants would provide open,

honest, and unbiased responses to the interview questions during the semistructured

interviews. My third assumption is that enough participants would be available within

Reproduced with permission of copyright owner. Further reproduction prohibited without permission.