CUSTOMER SPOTLIGHT

Centralized Agile Testing: Process Becomes the Hero

Sponsored by: Tech Mahindra
Jamie Snowdon
December 2014

IN THIS CUSTOMER SPOTLIGHT

Testing processes often get built in an ad hoc fashion as an afterthought to the development life
cycle, with the success of the testing down to the individual skill and commitment of the testers —
the testing heroes. The problem with relying on this model is that the number of these heroes is
limited and finding them for each project can be difficult. As the amount of applications requiring
testing grows, consistency between projects becomes ever more important and the emphasis must
be placed on improving the overall process for testing.

Now more than ever we see business leaders demanding that IT organizations support increasing
amounts of application development. This is particularly true for a high-tech manufacturing firm like
Philips, with a broad range of application development activities supporting its product
development and internal business processes.

This IDC Customer Spotlight examines how Philips engaged with Tech Mahindra to reengineer its
testing processes, moving from an individual hero-based model of personal excellence to a global
test factory providing end-to-end quality assurance focused on business outcomes. The process
itself has now become the hero.

INTRODUCTION

For Philips, a uniform global approach to testing, provided by the Global Test Factory, has driven
consistency, standardization, and transparency of the testing function. It is now possible to focus
on outcomes from the application development program rather than the individual KPIs and SLAs.
The new process has improved the overall quality of the applications when they go live and has
lowered the running costs. However, the main benefit has been to allow Philips to focus its
attention on the design process, providing a consistent level of application excellence.

BUSINESS CHALLENGE

Philips, officially Royal Philips, is a Dutch-based diversified technology company, focusing on

healthcare, consumer lifestyle, and lighting. The company is a leader in cardiac care, acute care,
and home healthcare; energy-efficient lighting solutions and new lighting applications; shaving and
grooming; and oral healthcare. It has approximately $24 billion turnover and employs 120,000
people in more than 60 countries around the world.

December 2014, IDC #IDCCS05W

At the end of 2009 Philips made the decision to outsource its testing services. At the time
application testing within Philips was very much an ad hoc process, with testing provided on a
project-by-project basis using internal staff and a number of contractors.

Paul Keltjens, director of testing for Philips, described the testing as a "hero-based service. We
had individuals who did their best, but there was no standardization and no reusability between
projects." Additionally, the large number of testing contractors were charging on a time and
materials basis, which meant each testing team had its own set of processes, and its own test case
definitions, performance metrics, and so on. In addition to being expensive, this type of testing can
lead to unpredictable workloads, multiple handoffs, complicated and time-consuming processes,
and resource inefficiencies. Additionally, Philips management was looking to make sure all
functions were aligned to support the overall direction of the business. The testing function needed
to demonstrate it could deliver business value and be outcome-led, rather than the current input-
led model. The challenge was partly to improve the efficiency and cost of the testing function, but it
was also about improving the quality and governance of the entire application development cycle.

To put this into perspective, the main challenges faced by Philips' testing are summarized here:

 High testing cost (>35% of development effort average)

 Critical defects in production (>20% high-priority defect leakage in production)
 Non-standardized testing services — different processes, practices, and templates across
departments and projects
 Delays in test preparation and test execution deliverables (<80% of deliverables on time)
 Contractors on time hire, leading to a lack of accountability and higher costs
 No specializations within testing
 Limited proliferation/acceptance (< 20% projects consuming test services)
 Lack of independent, objective evaluation of solution delivered by Philips' development
partners
 No means of identifying and fixing non-functional code/configuration level defects early in
the life cycle

Step 1: Stabilize the Solution — Centralized and Offshore

The starting point for the engagement was to consolidate the testing function. Philips was
convinced that the best way to gain control of the cost of its testing function and to make its
application development more uniform and effective was to centralize in a testing factory
environment. Given the lack of specialist testing knowledge within Philips it was quickly apparent
that an external provider should build and manage the testing. The selection process was
innovative in its own right — Philips wanted a provider that had the experience of running numerous
complex testing functions and required the bidders to prove they could provide a standard solution.

It needed the provider to demonstrate its capabilities. “We wanted to make the [selection] process
as objective as possible,” said Keltjens. “We had a criteria list and some of the criteria were non-
negotiable." It also needed the provider to be able to deliver based on a standard solution
framework that it could demonstrate had been successful in the past, and the solution should be off
the shelf. An important measure of this was how commoditized the provider had made its testing
activities. Philips did not want it to develop a custom framework — it wanted it to deliver as close to

©2014 IDC #IDCCS05W 2

an off-the-shelf solution as possible. It needed a provider that was able to bring its thoughts and
ideas to bear, and was ultimately able to propose solutions to Philips. It wanted a partner to help it
design a solution to its business challenges — not just someone to deliver a testing service.

Tech Mahindra was chosen because it was able to show a strong track record in delivering testing
solutions: the company had a good framework and was able to demonstrate its effectiveness.
Philips was confident that Tech Mahindra could deliver and would be flexible enough to work with it
long term. Philips also wanted a partner that understood testing in the broader application
development context and would help innovate the process, using its experience to bring ideas to
the table. It is important to emphasize that Philips needed an active partner able to adapt the
solution to solve new challenges as and when they occurred.

The solution designed by Tech Mahindra established a global test factory for Philips in Bangalore.
This "factory" approach to manage testing workflows is set up in layers, starting with the factory
command center (test strategy, demand management, capacity planning, release management),
which houses the capability subject matter experts (SMEs). It then moves down through the test
scenario workshop layer (test case design, test case library with unique test point definitions,
automated test case preparation), to the test execution plant (automation execution, defect
repository, results, etc.) and finally to the service introduction bays (user acceptance testing/UAT
and pre- and post-live support).

Part of the benefits of using a third party to establish a test factory is the creation of an independent
test partner for Philips. This enabled Philips to get to grips with the testing process and look at
making more fundamental changes to its application development regime. Additionally, the steady
state allowed staff from both organizations to look for news ways to improve the process.

Step 2: Refine the Solution — Outcome Based, Continuous, and Agile

In 2012, Philips adopted an "agile/scrum" methodology for its application development operations
and changed the delivery model for its application development partners. Tech Mahindra proposed
a way in which the testing could be delivered as part of this agile model and ensure performance
could be measured on successful outcomes. It was largely thanks to having an independent testing
partner that Philips was able to shift from an old waterfall development model delivered by time-
hire resources to an output-based agile/scrum model. Development partners now have output-
based contracts where they deliver scrum teams and their success is measured on their output, in
terms of timeliness of delivery, whether it meets the agreed scope and requisite level of quality.
Having an independent partner made this transition easier — the standard set of quality controls
and test parameters means all development teams have a set standard that they must adhere to
and payment for work is only made on satisfactory clearance by the independent testers.

Another change during this period was the adoption of continuous testing, and Philips wanted the
test factory to incorporate this new approach. Tech Mahindra developed a solution to adopt
continuous testing within the test factory framework, allowing Philips to provide daily releases of
the software. As it stands today, the Philips test factory has adapted to the move to agile
development, has adopted continuous testing, and is providing testing protocols that work with
each scrum team within Philips. The factory now works to the following key principles:

 End-to-end testing ownership. The test factory has become the quality custodian for all IT
applications, with Tech Mahindra taking end-to-end ownership of all testing activities. The
test factory model operates in consultation with all the relevant stakeholders within Philips

©2014 IDC #IDCCS05W 3

 and within its development ecosystem, making sure that any impediments affecting the
quality of the applications are highlighted and resolved.
 Act as an independent but integrated test partner. Tech Mahindra is the independent test
partner for Philips, taking on the responsibility to ensure all known defects are closed and
adequate test documents are created in the process. However, it is important that being
independent doesn't create a silo mentality, with stakeholders throwing problems "across
the wall." Tech Mahindra and Philips have overcome this by integrating the testing process
into the development cycle, which is vital to the success of the agile/scrum method. It also
means that problems are discovered and resolved at every stage of the life cycle.
 Multiple technologies and domains. Philips has a diverse set of applications being
developed, so requires that testing is standardized across technology platforms and
domains. This means Tech Mahindra has to manage a demand-based resource pool,
adding new staff, training, and retraining staff depending on changing requirements.
 Aligned with agile. As mentioned, during its development the test factory has aligned itself
with the agile delivery model adopted by Philips. The delivery center has been set up to
enable it to integrate with Philips' agile processes. This requires testing and test
automation at multiple levels with each project (e.g., at a team level, program level, and
portfolio level) in line with the agile framework that Philips is using.
 Continuous testing. The test factory has developed a continuous testing approach to
dovetail with Philips' continuous testing vision, allowing Philips IT to do daily releases. To
achieve this, Tech Mahindra provides each scrum team with an automation tester
alongside the functional testers. User stories are automated within the sprint once the test
cases are complete. To enable automation at the smallest functional unit level, the test
automation has been designed to handle both individual user stories and end-to-end PIL-
level business processes. These automated test cases can be executed on a nightly basis.
 Test tooling is provided as a service. Tech Mahindra owns the licenses and manages the
entire test management tooling for the test factory. The tools are used for test
management, test automation, performance testing, security testing, and mobility testing.
All the tools are integrated into the application life cycle to ensure that a single source of
data remains within the test management tool. The tool is also integrated with upstream
requirements management tools, and there are plans to integrate the tools with
configuration management tools on the upstream and service management tools on the
downstream. These tools have undergone a validation process as required by FDA
regulations so they can be used for healthcare-related projects.
 Structural code quality analysis. Architecturally complex violations are structural flaws
involving interactions among multiple components that may reside in different application
layers. Industry studies have indicated that the architecturally complex violations only
constitute 8% of all violations detected in an application, but they constitute more than half
of the repair effort and average 20 times more changes to fix. The structure analysis
service has been implemented to gain objective insight into the quality of deliverables of
the Philips development partners. This helps drive improvements and eliminate any
technical debt, and enables Philips to ensure that solution delivery from partners is
consistent and adheres to the coding/design best practices — improving the overall
maintainability and stability of the applications. This service also provides insight into the
performance issues with customized code and application configuration — when combined
with non-functional performance testing this can help Philips to eliminate any core
performance bottlenecks caused by these configurations.
 The focus is on output. The entire contract is based on principles of output-based KPIs.
The test factory has two sets of output-based KPIs — one measuring the service as a
standalone entity and the other measuring its influence on the overall delivery. Outcome

©2014 IDC #IDCCS05W 4

 KPIs are focused on driving the right behavior within the test teams to ensure that the test
factory is taking responsibility for the process with the minimum management bandwidth,
but Philips is getting the results it needs.

Results

As with most IT services engagements the measure of success is a mix of hard, quantifiable
tangible objective facts and softer, less tangible subjective compatibility factors. Often it is the
softer factors like trust and the relationship that are the difference between a successful outsource
and a failure.

The Hard
One of the most important benefits enjoyed by Philips has been significant cost saving. The old
testing model was based on traditional time and materials contracts with multiple contractors using
a variety of processes working to get each individual project done — a typical hero-based approach.
The new solution has industrialized testing using a standard approach, which has drastically
improved the cost of testing and improved the cost transparency of the process, making it much
easier to predict the absolute cost of each application. Additionally, there have been a number of
additional benefits derived from the relationship, including:

 Standardization. Philips now has a standard way of working across different projects and
releases.
 Accountability. Tech Mahindra has taken accountability for all testing activities, providing
an independent arbiter of quality for all applications development.
 Roadmap. Objective measures to drive improvement to enable IT to deliver robust and
sustainable software solutions to Philips' business.
 Transparency. Philips can predict the cost and effort needed for its application
development activities.
 Coverage. 100% business process coverage on all business processes with automated
test cases.

The Soft
Although these benefits are important, many other factors go into a successful sourcing
arrangement. Largely it comes down to a set of intangibles for each client. The reasons why Philips
and Tech Mahindra have a good relationship are clear:

 Flexibility. In the client's words, Tech Mahindra "has been extremely flexible — on the
ground it goes the extra mile."
 Passion. "They are really passionate about what they do. They go out of their way to
deliver on time. They want to make it happen!"
 Knowledge. Philips needed a partner that could bring testing expertise to it, not deliver a
custom solution based on its requirements.

It was Tech Mahindra's ability to apply its knowledge and passion to clients' problems that drove
some of the important innovations in the project. Its insight into the process was an important part
of the reengineering of Philips' application development processes. When Philips wanted to do
outcome-based testing, Tech Mahindra proposed the solution. When Philips wanted to do agile
with an independent test partner, Tech Mahindra proposed the existing agile solution. And when

©2014 IDC #IDCCS05W 5

Philips wanted to do continuous delivery, Tech Mahindra proposed the continuous testing
approach.

LESSONS LEARNED

In the section below we have provided some essential guidance for companies looking to
undertake this work in the future. In the case examined here, however, there were some specific
lessons learned as part of the engagement. Perhaps the biggest challenge was the whole process
of change — essentially getting the whole of Philips to buy into the new approach and use the new
test factory setup. For many in business units using testing, the old system worked well and the
quality of the testing for them was good. Without a comparison it was difficult to convince people to
move to the new service.

It wasn't until the process had been seen to work that these stakeholders within Philips could see
the benefits of the new system. Basically the delivery unit needs to show these stakeholders that
they would get, as Paul Keltjens put it, "more bang for their buck." The results were dramatic, with
test times being reduced, in many cases, from weeks to a few days. These initial successes helped
to build trust within Philips and now 100% of development projects use the test factory.

Another challenge arose due to Tech Mahindra's flexibility during the transition, with Tech
Mahindra staff being too accommodating, particularly when Philips management wanted to fix the
process. "If you want to build a rigid process, you also have to stick to that, and not respond to
every opinion, adjusting to it," said Keltjens. However, strong governance within the process has
meant that senior managers from both companies meet on a monthly basis to make sure issues
are resolved. This prevents issues becoming toxic and allows the free flow of ideas between the
firms.

ESSENTIAL GUIDANCE

Mobile technologies and cloud are increasing the pace of change in many industries, requiring
organizations to adapt more quickly. IT organizations are facing increasing pressure to deliver
technology solutions that fit better into the business needs of their firms. Application development
is becoming increasingly important to many firms, particularly where the applications being
developed are for customer/employee mobility or product development. Making sure that this work
is completed cost effectively and to a high standard has become vital. Today's world is more than
ever about proving business value for money, and IDC research suggests that some organizations
spend in excess of 50% of project budgets on testing — suggesting operational improvements are a
must. Businesses must act smarter in the use of tools, models, and processes.

The reduced cost of testing, the improved quality, and the increased transparency and consistency
of the solution provided for Philips is only part of the solution. The effectiveness of the solution has
impacted the entire application development life cycle for Philips, allowing it to improve the way it
manages the development of all applications, ensuring that these activities are delivered with a
consistently high level of quality, with each provider having to buy into a set of business-driven
KPIs and outcomes. Testing at many companies relies on individual testing excellence rather than
on driving excellence through the process.

This case study provides some key advice for companies undertaking a similar journey:

©2014 IDC #IDCCS05W 6

 Stop relying on individual heroism. Testing is very much a commodity offering. A potential
supplier should be able to set out a standard set of offerings and provide most
requirements from a standard service catalogue. The process should be the hero, not the
people.
 Good delivery will only get you so far. A true partnership requires working together for a
common goal. It's important that an outsourcing partner is not only focused on delivery
performance and KPIs, as its knowledge and advice in reengineering the process (and
adjacent processes) can add important value.
 Good testing is crucial for the overall success of the application development process. The
agreement with Tech Mahindra was the catalyst for more fundamental changes in the
entire application development life cycle.
 Have the right set of people in place in your own organization to manage the process. It is
important to have the right people in place to manage the outsourcing relationship to get
the most out of the relationship. Ultimately the success of the project could impact the
future of your company, so it is worth investing in good people at the top.
 You need executive support. Without executive support any arrangement is going to fall
short of delivering business value. Executives should be involved in the governance
process so issues that impact the business can be short circuited. A successful test factory
needs to be used by as many departments as possible to promote consistency and drive
the benefits of scale. Without buy-in, transformation activities will be much more difficult
and less likely to succeed, certainly in the timeframe you want.
 Flexibility, transparency, and "passion" are important selection criteria. A provider that is
not upfront and won't respond positively to change is unlikely to be able to support your
long-term needs.
 Take a lean approach to governance. Have a simple escalation and problem resolution
structure. Every agreement runs into the odd bump in the road, so make sure you have a
clear escalation point and someone senior within the provider who you know has the ability
to get things done and understands what is important for you.
 Focus on the collaborative culture of the supplier. How responsive is the supplier you are
considering? How well does it understand your business and your goals, including what
that means for your IT operations requirements? Every corporation is different, and the key
to successful outsourcing is that the supplier and the partner understand each other's
goals and ways of working. We often hear good partnerships described in terms like "in the
daily work, we work as one team; we don't think about who comes from which company."
Find a supplier with which you can work that way.
 Manage internal customers' expectations and bring them into the process. Internal
customers may expect a big bang outsource to deliver immediately, but regardless of how
well the transition and transformation go there will always be a delay before the steady
state appears. Making sure progress and achievements are distributed to the wider user
base is important to avoid users being frustrated before a stable situation can be expected.
 Relationship first. Before you sign an outsourcing deal you need to ask yourself: can I work
with these people? Both management teams need to be able to work together to solve
problems, preferably without running to find the contract. The governance between Tech
Mahindra and Philips allows the free flow of ideas and issues, so any problems or ideas
can be actioned quickly.
 Make sure the supplier has the ability to deliver. Ultimately you need the outsourcer to be
able to deliver on its promises. It needs the technical skills and capacity to be able to
deliver and the determination and processes to get things done.

©2014 IDC #IDCCS05W 7

About IDC
International Data Corporation (IDC) is the premier global provider of market intelligence, advisory
services, and events for the information technology, telecommunications and consumer technology
markets. IDC helps IT professionals, business executives, and the investment community make
fact-based decisions on technology purchases and business strategy. More than 1,100 IDC
analysts provide global, regional, and local expertise on technology and industry opportunities and
trends in over 110 countries worldwide. For 50 years, IDC has provided strategic insights to help
our clients achieve their key business objectives. IDC is a subsidiary of IDG, the world's leading
technology media, research, and events company.

IDC U.K.

Chiswick Tower
389 Chiswick High Road
London W4 4AE, United Kingdom
44.208.987.7100
Twitter: @IDC
idc-insights-community.com
www.idc.com

Copyright Notice

This IDC research document was published as part of an IDC continuous intelligence service, providing
written research, analyst interactions, telebriefings, and conferences. Visit www.idc.com to learn more about
IDC subscription and consulting services. To view a list of IDC offices worldwide, visit www.idc.com/offices.
Please contact the IDC Hotline at 800.343.4952, ext. 7988 (or +1.508.988.7988) or sales@idc.com for
information on applying the price of this document toward the purchase of an IDC service or for information on
additional copies or Web rights.

Copyright 2014 IDC. Reproduction is forbidden unless authorized. All rights reserved.