Enterprise Solutions

Centre of Excellence
DevSecOps Playbook
Powered by Transformation Office

1 IBM & BAT Confidential (Ver 2.0)

Document Control Section

Author/s – Role Name Date

Transformation Office Member Sai Harshini Meda October 2022

Reviewer/s – Role Name Date

Transformation Office Lead Anita George October 2022

Transformation Office Lead Doreen Too October 2022

Transformation Office – Agile Coach Sree Ram Kishore October 2022

Transformation Office Member Subhashis De October 2022

Approver/s Role Name Date

IBM Delivery Leader Arun Prabhakar October 2022

ESCOE Leader Jorgen Bach October 2022

Document Location – <Sharepoint Location to be updated>

2 IBM & BAT Confidential (Ver 2.0)

Table of Contents

1 Purpose of the Playbook APPENDIX

2 DevSecOps Overview and Framework ESCoE 1 Glossary of Terms

3 Operating Model 2 DevSecOps Delivery

3.1 Organizational Structure 2.1 Squad Readiness Checklist
3.2 Squad 2.2 Maturity Assessment Framework
3.3 Key Roles & Responsibilities 2.3 Supporting Roles & Responsibilities
3.4 Ceremonies
3 End to End Integrated Delivery Process
4 End to End DevSecOps Delivery 3.1 Continuous Business Planning
4.1 User Perspective Big Picture
3.2 Collaborative Development
5 Governance & Reporting
3.2.1 Ceremonies Explained
6 Tools
3.2.2 SNOW CR Phases & Activities
7 Key Performance Indicators
3.2.3 Key Performance Indicators

3.3 Continuous Testing

3.4 Continuous Release & Deployment

3.5 Continuous Monitoring

3.6 Continuous User Feedback And Optimization

3 IBM & BAT Confidential (Ver 2.0)

Document History

Version Author Changes Date

1.0 Sai Harshini Meda First release October 2022

1.1 Sai Harshini Meda VTB example is added and KPI section November 2022
is updated.
2.0 Sai Harshini Meda Organization Structure, User December 2022
perspective Big Picture, Tools,
Supporting Roles & Responsibilities
and KPI section are updated.

4 IBM & BAT Confidential (Ver 2.0)

1 Purpose of the Playbook

This playbook serves as a guide to provide guidance and This Playbook serves as a guide to provide information on:
references for DevSecOps delivery, DevSecOps best practices
and IBM Garage to the ESCoE delivery teams. • Operating model integration with delivery model
• ESCoE delivery vision
Team members playing the roles below are expected to use this • End to End process overview of a delivery model
playbook as guidance and an understanding of the new ways of • Tools road map
working. • Maturity assessment framework

• Squad leaders
• Squad team members
• Product owners and lead designers
• Operational governance members
• Transformation office members

5 IBM & BAT Confidential (Ver 2.0)

2 DevSecOps Overview and Framework

Introduction
DevOps outlines a software development process and an organizational culture shift that speeds the delivery
of higher quality software by automating and integrating the efforts of development and IT operations teams
– two groups that traditionally practised separately from each other, or in silos.

In practice, the best DevOps processes and cultures extend beyond development and operations to
incorporate inputs from all application stakeholders - including platform and infrastructure engineering,
security, compliance, governance, risk management, line-of-business, end-users, and customers - into the
software development lifecycle

DevSecOps is the evolutionary variant of DevOps

DevSecOps is short for development, security, and operations—the methodology automates the
integration of security at every phase of the software development lifecycle, from initial design through
integration, testing, deployment, and software delivery.

DevSecOps represents a natural and necessary evolution in how development organizations approach
security. DevSecOps integrates application and infrastructure security seamlessly into Agile and DevOps
processes and tools. It addresses security issues as they emerge when they're easier, faster, and less
expensive to fix (and before they are put into production). Additionally, DevSecOps makes the application
and infrastructure security a shared responsibility of development, security, and IT operations teams, rather
than the sole responsibility of a security silo. It enables “software, safer, sooner”—the DevSecOps motto–by
automating the delivery of secure software without slowing the software development cycle.

6 IBM & BAT Confidential (Ver 2.0)

 Key Questions

What is the Why are we using What changes from a What Stays Same?
DevSecOps Model? DevSecOps model? User Perspective?

A set of practices that To ensure • Work driven by Same amount of rigor

synergise the processes Business Priority (separate KPIs for DevSecOps)
between development, • Compliant Delivery Model
security and operation • Consolidated ERP platform • Better transparency of
teams to build, test, release, • Demand Management where items are in the
maintain and improve • Governance queue.
software quickly and reliably. • Measured Value

7 IBM & BAT Confidential (Ver 2.0)

 DevSecOps 6C Framework

The 6C DevSecOps Framework (short for Development, Security • Capture, analyze, and prioritize business requirements
and Operations) is a development practice that integrates • Project planning
security initiatives at every stage of the software development • Measure to project metrics
Continuous • Traceability
lifecycle to deliver robust & secure applications.
Business Planning • Dashboard portfolio measures

Steer
Continuous User
Feedback & Optimization Collaborative • Release planning
u Development • Collaborative development
• Monitor capacity and optimize • Configuration management
• Monitor performance and optimize • Build management
• Monitor user experience and optimize (requires integration with build automation)
• Event and incident management Operate Develop/ • Change management
Test • Dashboards
• Traceability
• Test management and execution

Deploy
Continuous • Application health
• Server health Continuous • Sanity Testing
Monitoring Testing • Automated Functional Testing
• Network health
c • Regression Testing
• Performance Testing
• Security Testing

Continuous Release & • Release management

Deployment • Environment management (provisioning automation)
• Deployment automation (application, middleware, and databases)

8 IBM & BAT Confidential (Ver 2.0)

 DevSecOps Capability Decomposition

Continuous Business Collaborative Continuous

Planning: Development: Testing:

This DevSecOps capability deals with identification, This starts with a development sketch plan and Unit and integration testing help increase the
elaboration, prioritisation and planning of business programming. It is a process for integrating, efficiency and speed of the development. The goal
demands through an open, collaborative and collaborating and system communications with IT of continuous testing is evaluating the quality of
standards based method. It is in this stage, the functions which are accountable for operations, software at every step of the Continuous Delivery
backlog of work is created and elaborated, the application development and quality assurance. Process by testing early & testing often.
pipeline of work examined and right prioritisation is
done through a value based approach.

Continuous Release Continuous Continuous User

and Deployment: Monitoring: Feedback and Optimization:

The deployment takes place continuously in this This is needed to monitor changes and address errors This evaluates the effect of each release on the user
DevSecOps lifecycle phase. Multiple reviews will help and mistakes spontaneously whenever they happen. experience and then reports that evaluation back to
code quality and developer check-ins easily. It is It is the process of identifying threats to the security the DevSecOps team to modify and improve future
performed so that code changes should not affect the and compliance rules of a software development releases.
functioning of the product. cycle and architecture.

9 IBM & BAT Confidential (Ver 2.0)

3 ESCoE Operating Model

ESCoE is the enterprise model of the future which is Deliver with new WoW,
Transform the operating and delivery model and Grow to deliver business value.

Business Partnering & Continuous Business

Zone 1 Zone 2 Zone 3
Value Realization Planning

Functional Project Product

Squads Squads Squads
Transformation & Collaborative
Development Development
Dev Dev Dev

Continuous
Testing
Integrated Delivery Cross Functional Support
Continuous Release &
Deployment

Ops Ops Ops Continuous

ServiceDelivery Monitoring

Technical Enablement Continuous User Feedback &

Zone 1 Zone 2 Zone 3
Monitoring Optimization

10 IBM & BAT Confidential (Ver 2.0)

 3.1 Organizational Structure
The organizational structure to support the new ways of working is centered around the
following key objectives:
• Focus on delivery of business value.
• Closer collaboration with business and end-to-end visibility for all.
• Faster time to market.
• Adaptable and flexible organization to embrace changes in the environment.
• Learning continuously and encouraging an innovation culture.

Sponsor Business

Business Partnering & Value Realization Office

X- Workstream Services

Marketing Operations Finance HR Projects

Squad 1 Squad 1 Squad 1 Squad 1 Squad 1

Squad 2 Squad 2 Squad 2 Squad 2 Squad 2
Squad n Squad n Squad n Squad n Squad n
X- Workstream Services
Agile / DevOps Coaching Services
Delivery Governance

Product Manager
Design Authority

Integration Services

Product SME
Security Services
Data Services

Deployment Services
Testing and Quality Engineering Services
Release Management Services
Change Management Services

11 Operational Services IBM & BAT Confidential (Ver 2.0)

 Delivery Organisation - Key Components

The delivery organization is aligned to the products aka business In addition to the direct contributing members, the delivery organization is
functions / functional work-streams (e.g., Marketing, Finance, also interacting and collaborating with the following entities. These entities
Operations, Treasury, Success Factors, etc.) . Each of these product are the extended wings of the delivery organization.
verticals consists of squads delivering value for business

• Business Sponsors / End Market Representatives

• Business Value Realization Office (VRO)
• Design Authority / Architecture Board
• Overall Delivery Governance Office
• Product / Process Subject Matter Experts /
Process Managers

12 IBM & BAT Confidential (Ver 2.0)

 3.2 Squad

Squads are cross-disciplinary, autonomous teams (typically Checklist that can help to check squad team readiness is at
6-10 individuals) that focus on one or more user stories/change Appendix 2.1
requests. Each squad has a unique mission that guides the work they do,
an agile coach for support, and a product owner for guidance.

The development team consist of core team members from the home
business function supported by flex team members from cross functions
such as data, integration services, Basis, deployment and security services
(horizontals in the above diagram) as per the requirement of the backlog
and release plan of the squad.

Note: For identification of the flex members for a user story/CR

delivery, there are SPOCs from each discipline – e.g., data, testing, etc.

13 IBM & BAT Confidential (Ver 2.0)

 3.3 Key Roles & Responsibilities

The squad consists of a product owner/lead designer, squad leader and development team (Core & Flex members). Squad teams
are self-organizing and cross-functional. This means the squad team has the ability to define their own work and the best
possible way to execute that work to meet iteration goals and produce done increments that can be released to production.

Product Owner Squad Leader Design+Build+Test

• The Product Owner is responsible for maximizing • The squad leader is responsible for ensuring WoW is • The Development Team is expected to manage its
the value of the product and the work of the understood and enacted according to the scope. work and it consists of professionals who do the
Development Team. • The squad leader ensures that everyone in the work of delivering a potentially releasable
squad knows their responsibilities and acts Increment of “Done” product at the end of each
The Product Owner is responsible for: according to them. Iteration.
• Clearly expressing Product Backlog items • The squad leader ensures that all agreed • The Development team is solely responsible for
within his/her process area ceremonies are executed with discipline and are delivering the expected user stories/CR.
• Optimizing the value of the work the according to the agreed principles. • The Development team is responsible for
Development Team performs • The squad Leader ensures that all Artifacts are Managing the Iteration Backlog and produce
• Ensuring that the Product Backlog is visible, managed with discipline and according to the agreed working increments that meet the “Definition of
transparent, and clear to all. principles. Done”
• Ensuring the Development Team understands items • At any point in time, the development team is
in the Product Backlog to the level needed and to responsible for providing information on the progress
provide needed support to get needed details. of their work toward the Iteration goal.

For more supporting roles overview, please refer Appendix 2.3

BAT Specific Role IBM Specific Role BAT, others & IBM

14 IBM & BAT Confidential (Ver 2.0)

 3.4 Ceremonies

• Prescribed ceremonies are used to create cadence, enable transparency and provide the opportunity to inspect and adopt.
• All ceremonies are time-boxed events, meaning that they can take a maximum time no more.
• All ceremonies within an Iteration are fixed and the squad leader ensures that they take place at the agreed time and place and all
participants know their roles and responsibilities to meet the ceremony goal.
• Ceremonies: Kick-off Meeting, Backlog Grooming, Iteration Planning, Daily Sync, Iteration Review/Demo, Iteration Retrospective.

Below Table gives high-level overview of ceremonies.

For Detailed view of each ceremony click on ceremony under event section.

Event Description Time Box Facilitator Participants Input Output

Kick-off The squad Kick-off agenda conveys the Max. 1 Hour. Product Owner/ Product Owner Requirements/ Squad goals &
Meeting high-level requirements/CR’s overview and Work Stream Lead Squad Leader CRs Overview. vision, approach.
overarching strategy, vision, scope, team roles, Lead Designer
and responsibilities, as well as the approach and Development Team
supporting ceremonies to be used. This only Business Stakeholders
happens for the first time when the squad is
launched.

Backlog Backlog Grooming discussion involves Max.2 Hours. Squad Leader / Product Owner Product Backlog 1. Identifying business
Grooming the development team, LDs, and Product Product Owner Squad Leader Items ready for value so identifying the
Owners to identify the business value and Development Team grooming. Initial scope for the next
priorities. The squad can have a discovery Iteration and 1-2 following
workshop as necessary. Iterations.

2. Clarifications needed
for planned user
stories/CR’s.

15 IBM & BAT Confidential (Ver 2.0)

Ceremonies

Event Description Time Box Facilitator Participants Input Output

Iteration Iteration planning happens always at the Max. 1 Hour. Squad Leader Squad Leader Product Backlog Iteration Backlog
Planning beginning of each Iteration. It is a session to Lead Designer [Agreed Backlog items for
estimate and plan the task level details needed Product Owner this iteration].
for items selected for the Iteration and how to Development Team
ensure get them done.

Daily Sync The Daily Sync meeting is for the 15 mins. Squad Leader Squad Leader Squad Board Updated Squad Board
Meeting Development team (and product owners) to align Lead Designer Identified blockers.
what was done the previous day and what will be Product Owner
done during the day. Development Team

Iteration Iteration review is a ceremony involving Squad Max. 1 Hour. Squad Leader Product Owner 1.Work done Updated Product Backlog
Review/Demo Leader, Lead Designer, Product Owner, Squad Leader during the for upcoming Iterations.
Development Team, and Business Stakeholders Lead Designer Iteration
where Development Team shows the work Development Team
done during the Iteration and seeks feedback Business Stakeholders 2. Iteration &
from Stakeholders. If any of the planned items Product Backlog
are not done, those will be discussed for further where it stands
actions.

Conducted at the end of an iteration. During

Iteration Max. 1 Hour Squad Leader Squad Leader –What worked well for us? Highest voted
the retrospective, the team reflects on what
Retrospective Lead Designer –What did not work well for us? items to be
happened in the iteration and identifies
actions for improvement going forward. Product Owner –What actions can we take to taken for
Development improve our process going improvement.
Each member of the team members answers
Team forward?
the following questions:
–What worked well for us?
–What did not work well for us?
–What actions can we take to improve our
16 process going forward? IBM & BAT Confidential (Ver 2.0)
4 End to End DevSecOps Delivery

4.1 User Perspective Big Picture

Below picture provides a high-level overview of 6C’s alignment with high-level activities, process, roles, ceremonies, tools and KPI’s.
6C Continuous Business Collaborative Continuous Continuous Release & Continuous Continuous User Feedback &
Framework Planning Development Testing Deployment Monitoring Optimisation

High Level Backlog

Planning Measure Release Build Unit Regression Performance User Accept. Release
Deployment
Hyper Incident & App & Continuous Monitor Monitor to
Feedback
Activities Priority Value Planning Management Test Test Test Test Management Care Support ServerHealth Improvement User Exp. Optimise

Design Sprint planning

Thinking meet chaired by
Workshop product owner
Difficul
Could

SP1 Func -Unit Update Performance User Business Stage Gate Incident Incident Server Collaborate Assessto System Feedback
Business
Testing Test Scripts Test Acceptance Release Approval Management Management Health toImprove Improve Health
Product Backlog

t
Sprint Backlog

Requirements
Testing Group
Should

Mediu

SP2
m

Config Test Unit Automated Quality

Monitoring Checkout Testing Testing Review
Must

Easy

Process led Stories

SPn

Backlog Sprint Code Security Unit Bug Fixes Transport Cutover Stage Gate Transition Maintenance Application New Performance
Prioritization Management Changes Testing Planning Approval toSupport Activities Health Ideas Health

End Product Change Technical UX Analysis

Roles
Product CT/PT Cutover Release Functional Data Users
VRO Product End- Market Lead Squad Squad Architect Lead Market Rep Squad UAT Owner Squad Request Team
SRE Team
Team
Owner Designer Users Manager Manager Ops Team OPS Team Team Team
Owner Rep Designer

Ceremonies Backlog Kick Off Planning Iteration Review Retrospective Testing Test Data Release Environment Deployment Incident Operational Monitor Optimization Monitor
Grooming Meeting Planning Demo Management Management Management Management Management Analysis Capacity Performance

Tools

KPI Lead Dashboard Metrics Breakdown Velocity Cycle Lead Defect Cycle Time – Deployment Mean Lead Mean Time Change
Time Chart Time Time Leakage Test & Release Frequency Time for to Recover Failure Rate
changes

17 *Some of the tools referred in the picture are to be updated post pilot completion. IBM & BAT Confidential (Ver 1.1)
2.0)
5 Governance & Reporting
Governance seeks to align all the projects being worked on with the goals Reporting refers to the use of selected metrics and tools for analyzing and
of the organization. It is all about creating value across the organization. making informed decisions about the way work is managed. The goal of
Governance is a process that focuses on bringing the most value by reporting is to ensure that work systems deliver outcomes - value with
focusing on what matters. It ensures that the policies and decision-making optimum quality and increased pace. Hence, Agile teams use different sets
process aligns with successful delivery. Governance is understanding group of metrics to achieve greater visibility over the work processes, measure
risk-taking appetite and tolerance level and challenges. Good governance how efficient their process is, track their performance over time, identify
practices help the organization evolve and improve over time. process blockages, and more. The reporting method is usually defined by
the work methodology that a team chooses to apply.
Enterprise Strategy
Value Realisation Office

Business
The governance framework is divided into
four interlinked dimensions. Each of these
dimensions – Business, Delivery, People
and Technology has got their own set of
Product management

Product leadership
ESCOE leadership
Squad leadership
Design leadership
Design Authority
roles and governance charter that focus

Delivery
on specific items for maximum value
realisation. The DevSecOps model is
encapsulated within this governance
framework and operates with the
guidance and direction of the governance
committees aided by the regular cadences
at different operating levels People

Change Management
People Management
Agile Coaching
18 IBM & BAT Confidential (Ver 2.0)
 Delivery Governance Transformation

Operating Levels
Role Cadence Visualisation Focus

VRO Intake Process 1 VRO Backlog Wall of Work

Ideas In Review Accepted Ready for
ESCOE
VRO

• Strategic Initiatives Initiative owners Strategic themes discussion Product Canvas Prioritisation of initiatives
• Projects Business unit / product sponsors Lean business case review Product Wall of Work Funding of products
• Change Requests Approved Lean Design Authorities Budget review Lean Business Canvas Budget allocation Initiative
• Enhancements Business Case Global Process Experts Initiative tracking lifecycle

2 ESCOE Backlog Wall of Work

Ideas In Review Accepted Ready for
Solutions COE

Opportunities will be Senior Product Owner Product canvas refinement Product Wall of Work High level estimate review
Enterprise

Product
directed to ESCOE Teams Lead Designers Value – risk evaluation Epic Hypothesis Cards and approval (epic wise)
backlog and refined in Enterprise Architects Inter-product dependency evaluation Metrics dashboard Team engagement
to an Epic Epic flow, metrics

Refined Epic

3 Product 2 - Wall of Work

Backlog Reviewed Refine Ready for In Progress Done
Squad with Squad
Epic pulled by Product Lead Designers (product) Feature prioritisation Periodic Squad board Estimation review Metrics
Teams and decomposed 3 Product 1 - Wall of Work System architects / engineers increment planning Metrics dashboard and health checks Risk and
Product

PRODUCT BACKLOG PRODUCT BACKLOG PRODUCT BACKLOG PRODUCT BACKLOG PRODUCT BACKLOG Epic 111
Epic 789 Epic 222
in to Features Backlog Reviewed
Epic 123 Refine
Epic 123 Ready for
Epic 123
Feature 789.1
In Progress
Feature789.1 Done
Epic 123
Feature 789.1
Epic 333 Product lead (work-stream leads) Product roadmap governance issues – product
Squad with Story
Squad789.2 Epic 444
(T-Shirt Sizing) Epic 456 Epic 456 Story 789.2

Epic 456 Epic 456

Story 789.2

Epic 456
Feature xxx Story 232.6
Release management governance
Story 456.1 Feature yyy Story 267.1
Epic 789 Epic 789 Story 456.1 Story
PRODUCT BACKLOG PRODUCT BACKLOG PRODUCT BACKLOG PRODUCT BACKLOG PRODUCT BACKLOG Epic 456.1
111
Story 456.2 Story 456.2 Story
Feature zzz Story 232.1
Epic 789 Epic 456.2
222
Epic 123 EpicEpic
123 n Epic 123
Epic n Epic 123456.3
Story Feature www Story 232.9
Story 456.3 Story
Epic 456.3
333
Feature 789.1 Feature 789.1
Feature 789.1
Story 456.4 Feature aaa Story 00.1
Story 456.4
Story 789.2 StoryStory 4 5 6Epic
.3
4 5444
6.4
Story 789.2 Story 789.2
Epic 456 Epic 456
Epic
Epic 456123 Story
Feature xxx 222
Story 232.6
Epic 456 Epic 456
Feature Feature yyy Story 267.1
Epic 789 Epic 789 Story 456.1 Story 456.1123.1 Story 456.1 Epic 001
Feature
Story 456.2123.2 Feature zzz Story 232.1
Story 456.2 Story 456.2 Feature 001.1
Epic n Epic n Story 456.3 Story
Story 123.3
456.3 Story 456.3 Feature www Story 232.9
Story 001.2
Story 456.4 Story 456.4
Story 123.4 Story 456.43 Feature aaa Story 00.1

FeatureFeaturewww.63
12
Epic 123 3.2 Story 222
Feature 123.1
Epic 001
Feature 123.2
Feature 001.1
Story 123.3
Story 001.2
Story 123.4
FFeeatauturerew1w2w3.623

Refined Features

4 Product 1 - Squad 1 - Wall of Work

Backlog Decompose Ready for Not In Done
Features pulled by & Refine Iteration Started Progress Product owner Daily synch meetings Daily synch meetings Flow and quality metrics
Squad

Squads and PRODUCT BACKLOG PRODUCT BACKLOG SPRINT BACKLOG SPRINT BACKLOG SPRINT BACKLOG Squad leader Story backlog grooming Story backlog grooming Removal of blockers
decomposed in to Epic 123
Feature 789.1
Feature 789.1
Story 789 1.1 Story 789 12
.
Story 456.1
Task Task
Story 456.1
Task Task
Story 456.1
Task
Story 456.1 Development team Review Review Continuous improvement
Stories & Tasks Story 789.2

Epic 456
Story 789 1.3 Story 789 14

Story 789 15
.
.

Featurexyz Featurexyz Featurexyz Featurexyz

Retrospectives Retrospectives
Story xyz 1.5 Story xyz 16
. Story xyz 1.5 Task Story xyz 1.5 Task Story xyz 1.5 Task
Story 456.1
Story 456.1
Story 456.2 Story xyz 1.5.1 Story xyz 1.5.2 Story xyz 1.5.1 Story xyz 1.5.2 Story xyz 15
.2
.

Task Task
Story xyz 1.6.1 Task

Story 222 Task Task

Feature 001.1 Feature xyz

Feature 001.2 Story xyz 1.5 Story xyz 16
.

Feature 001.3 Story xyz 17

.

With Squad

19 IBM & BAT Confidential (Ver 2.0)

6 Tools

Tools are used within product development to help achieve a few key goals like to adapt and organise
solutions and requirements/user stories/CRs that allow squads to work together, organize, and produce the
best possible solutions. Below table gives a view of the “need for tools” for each DevSecOps feature and what
are the potential tools that could be used.

Continuous Business Collaborative Continuous Continuous Continuous Continuous User Feedback and
Planning Development Testing Deployment Monitoring Optimization

Need for Tool? • Create Product backlog • Perform Iteration planning • Test management • Release planning • Monitor System Capacity & • Iteration retrospective
• Perform R&I assessment, ROI • Perform change and test execution • Cutover planning Optimize • Customer reported defects
validation management • Functional Unit Testing • Data migration • Monitor Performance & (UAT defect and incident)
• Get budgetary approval • Configuration management • Integration Testing • Deployment Optimize • Customer Surveys / feedback
• Backlog Grooming (Document • Develop Code / config • User acceptance testing • Monitor User Experience & • Enterprise reporting and
user stories/CR’s, sizing user • Quality review (FS, TS, Code, • UAT approval Optimize measurements (Trends, Metrics)
stories/CRs) config) • Regression testing • Monitor Security
• Prioritize user stories/CRs (Increase Coverage) parameters
• Perform Iteration planning & • Test automation
create Iteration backlog. • Performance testing

What Tools are • ServiceNow • ServiceNow • HP ALM / UFT / • ServiceNow • SAP Solution Manager • Azure DevOps
Required? • Azure DevOps • SAP Solution Manager Worksoft / Solution • Solution Manager Charm (TechMon, PIMon, BP Mon, • ServiceNow
• SAP Solution Manager • ActiveControl manager CBTA / ActiveControl Notification from Alerts) • HP ALM
• Azure DevOps • TOSCA • ServiceNow • Power BI
• Accelerators (Code Inspector • Load Runner • SCOM / Opsgenie • SurveyMonkey
(SCI++), Code Snippet Generator, • ServiceNow
ABAP Auto TS Generator,
RolePac etc.)

How we will be • ServiceNow can be used to create • ServiceNow / ADO can be used • ServiceNow can be used to • ServiceNow can be used for • Maximise usage of SAP Solution • Capture feedback from retrospective in
Product backlog, perform for Iteration planning perform Change management release planning manager features for monitoring ADO / ServiceNow
using each tool? R&I assessment, ROI validation and •Perform Change management in and UAT approval • Explore implementation • An event management tool can • UAT defect captured in HP ALM
get budgetary approval. ServiceNow • For test planning and execution of Solution manager Charm be introduced and integrated • ServiceNow is used production defect
• ServiceNow / ADO can be used backlog • Use accelerators (Code Inspector HP UFT can be used for deployment in place of with Solution manager and • Enterprise reporting of SLA, KPI and
grooming (Document user stories/CRs, (SCI++), Code Snippet Generator, • Load Runner can be used for Active Control ServiceNow trends can be centralized in Power BI
sizing user stories/CRs), prioritize user ABAP Auto TS Generator, RolePac) performance testing • Integrate ServiceNow and • ServiceNow can be used on ticket • Use a tool for structured customer
stories/CRs, Iteration planning and • Integrate ServiceNow and Solution • Explore the options of Solution manager created from high priority / feedback
create sprint backlog manager increasing test coverage criticality alerts
• SAP Solution Manager Used for
documentation

20 IBM & BAT Confidential (Ver 2.0)

7 Key Performance Indicators

KPI is a quantifiable metric that helps the organization determine the overall performance and its alignment with organizational objectives.

Below are the key metrics that determine the performance measures for the delivery organisation. This is the initial set of KPIs and more such measures will be added on and
baselined as the new ways of work evolve over time.

KPI Components

Lead Time • Elapsed time between CR Raised Date from CREW/SNOW and Technical Go-Live Date

Cycle Time • Elapsed time between DBT Actual Start date and the FUT actual end Date
[Development]

Cycle Time [Test • Elapsed time between FUT actual end date and Technical Go-Live Date
& Release]

Defect Leakage to • Relative comparison of in-process defects versus the post production incidents caused by the changes in the release
Production

For detailed overview of how metrics are calculated please refer Appendix 3.2.3

21 IBM & BAT Confidential (Ver 2.0)

Appendix

1 Glossary of Terms

Terminology / Acronyms Definition

Agile Software Development A software development methodology and philosophy, focused on

user feedback, software quality, and the ability to respond quickly to
changes and new product requirements
ESCoE Enterprise Solutions Centre of Excellence
Automation Completion of a process or procedure without manual tasks

Dev (from DevSecOps) Individuals involved in software development activities

DevOps At a very high-level DevOps is a development method that
concentrates on the collaboration between Developers (Dev) and
business Operations (Ops)
DevSecOps Automation of core security tasks by embedding security
controls and processes into the DevOps workflow
Ops (from DevSecOps) Individuals involved in the daily operational activities needed to
deploy and manage systems and services

Ways of Working
WoW

aka “Also known as” – a synonym / alternate word or phrase for the term used

22 IBM & BAT Confidential (Ver 2.0)

Glossary of Terms
Terminology / Acronyms Definition

LD Lead Designer

PO Product Owner

DoD Definition Of Done

DOR Definition Of Ready

Flex Team Member Members from cross functional Team

CR Change Request

QA Quality Assessment

UAT User Acceptance Test

SME Subject Matter Expert

POC Point Of Contact

Tier 1/Tier 2 Core Team members who acts as SME’s

TGL Technical Go Live

BGL Business Go Live

VRO Value Realization Office

Discovery workshop Discovery is like a conversation where technical and

businesspeople collaborate to explore, discover and agree as
much as they can about the desired behaviour for a
requirement/user story/CR

23 IBM & BAT Confidential (Ver 2.0)

Glossary of Terms
Terminology / Acronyms Definition

Product Canvas Product canvas is a planning tool designed to help build Solutions/products
that have a great user experience through a focus on feature development

Epic Hypothesis Epic Hypothesis is a highly iterative process that continues until a data-driven
result is obtained
GPE Global Process Expert

DA Design Authority

24 IBM & BAT Confidential (Ver 2.0)

2 DevSecOps Delivery

2.1 Squad Readiness Checklist

Section No Activity Name Activity Details Responsibility

1 Announcement and Engagement

with Stakeholders

1.1 Agile Framework implementation Identify and engage stakeholders and obtain their Work Stream Lead
announcement to stakeholders. input on the new "Ways of Working" (WoW)

1.2 Impact assessment of framework Socialize the impact of process change & provide clarity Work Stream Lead
implementation on existing processes, on roles and responsibilities to stakeholders.
deliverables, workflow.

1.3 Obtain commitment from the Confirm availability of stakeholders/customers to Squad Leader
stakeholders to contribute to the participate in the event.
agile implementation.

1.4 Train team members & Stakeholders Train team members and stakeholders to perform their Agile Coach/
new roles in the agile implementation. Transformation Office

25 IBM & BAT Confidential (Ver 2.0)

Squad Readiness Checklist

Section No Activity Name Activity Details Responsibility

2 Agile team framework

and formation

Determine framework applicability Discuss the nature of the work performed by the team. Agile Coach/
2.1 based on type of work Refer the WoW guidelines (Best Practices) shared with Transformation Office
the team

2.2 Identify team members Work with leadership help to identify Product Owner, Work Stream Lead
(Product Owner, LD - Lead Designer LD, End-Market Rep, Squad Leader, and team
Squad Leader, Developers members. (Refer to Agile team formation guiding
(Agile Team = Product Owner + Squad principles)
Leader +Developers + Business Analyst +
End Market Rep + Testing Member, Cross
work stream members, Tier 1/2
members)

2.3 Identify training needs of agile team. Identify specific training needs of the members. Train Agile Coach/
members in overall Agile Framework as well as role- Transformation Office
based trainings to help enable them to perform their
roles

26 IBM & BAT Confidential (Ver 2.0)

Squad Readiness Checklist

Section No Activity Name Activity Details Responsibility

3 Establish and visualize

explicit policies

3.1 Identify Definition of Ready (DoR) Criteria (generally based on the INVEST matrix) Product Owner &
that a user story should meet prior to being Development Team
accepted for development.
If the team is part of a program, consider
program definition of ready inherited/adapted
from the program level.

3.2 Identify Definition of Done (DoD) Common set of steps through which every Product Owner &
story goes through before being "done" for e.g., Development Team
analyze, dev, test etc.) If the team is part of a
program, consider program definition of ready
inherited/adapted from the program level.

3.3 Identify Estimation Scale In Agile, our interest is to be able to quantify the Product Owner &
"working software" produced by the team over Development Team
completion of individual activity such as development,
testing etc. Story points are a unit of measurement for
the amount of work completed (produced) by the team.
Team members collectively identify the estimation
scale to use for estimating the work items. Fibonacci
Series (1,2,3,5,8,13, 21...) is a commonly used scale.

27 IBM & BAT Confidential (Ver 2.0)

Squad Readiness Checklist

Section No Activity Name Activity Details Responsibility

3.4 Identify Working Agreements Working agreements are "ground rules", Product Owner,
members way of interacting and functioning Development Team &
together. Squad Leaders encourage team Stakeholders
members collectively define working agreement(s)

4 Backlog Grooming

4.1 Identify Roadmap & objectives Product Owners/LD's to identify key events such Product Owner
as release, milestones, and major commitments
such as upgrades in CR execution timeframe.

4.2 Identify Epics, Features, Stories Build initial backlog. Identify EPIC, Features, Stories Product Owner &
etc. Start with identifying large initiatives or Development Team
objectives and then split them into features and
stories.

4.3 Identify & refine High priority 10-15 Identify high priority work items for the team to work Product Owner
work items(Stories/Defects etc.). on & refine them.

28 IBM & BAT Confidential (Ver 2.0)

Squad Readiness Checklist

Section No Activity Name Activity Details Responsibility

5 Tool Setup

5.1 Identify the tracking tool. Identify & ensure everyone has access to the Squad Leader
tool. SNOW is enterprise-wide Tool. Ensure
that everyone has access to perform the
required operations.

5.2 Provide "Tools" training SNOW Training to Agile Team and Stakeholders. Squad Leader/
Role specific training to Squad Leader, team Workstream Lead
members, Product Owner. LD (can be combined
with WoW Training listed above)

5.3 Create initial backlog in the Tool. Create Backlog in SNOW. This step can be Product Owner &
combined with section 4. Development Team

5.4 Define workflow and setup Setup SNOW Boards step can be combined Squad Leader
required Boards (Backlog with 3.1, 3.2 as they are closely related to
Refinement, Team Boards etc.) each other.

29 IBM & BAT Confidential (Ver 2.0)

Squad Readiness Checklist

Section No Activity Name Activity Details Responsibility

6 Infrastructure Readiness

6.1 Squad Leader to setup Agile Events Set up calendar events by inviting required Squad Leader
(Grooming, Planning, Daily Sync, participants. Since the activity is cadence
Review, Retrospective) on based, It is helpful to set up a series. Having a
everyone’s calendar. meeting set up in the outlook calendar helps in
building awareness.

6.2 Environments Ensure Environment readiness, stability, and Squad Leader &
(Dev, Test) readiness support to operate in 2 weeks cycle. Ensure the Product Owner
production-like status of the environments.

6.3 Build, Deployment and Identify automation opportunities to increase Product Owner
Packaging procedures. the frequency of the build and deployment.
Continuous integration helps in the early
detection of defects and quality improvement.

6.4 Code repository, test case Ensure those technical components, Product Owner
management tools, DevSecOps, Engineering Practices are defined, and that the
and tools readiness. team is aware of how to implement them.

6.5 Agile Team Kick-Off – Milestone Kick-Off Meeting serves as a checkpoint to Squad Leader
transition to operate in Agile WoW.

30 IBM & BAT Confidential (Ver 2.0)

2.2 Maturity Assessment Framework

Apart from the delivery KPIs, we need to assess the maturity of the squad periodically. This framework will help in assessing the performance maturity of the squad.
This is a self-assessment that the squad leader and the team will fill out and asses themselves. This questionnaire is expected to be used about once a quarter to give
the squads enough time to take actions coming out of the previous assessment.

For a copy of the assessment template for use in your squads, please reach out to the transformation office members.

2.3 Supporting Roles & Responsibilities

The supporting roles aid and provide guidance to the squads on an ad hoc basis throughout the lifecycle.

Roles Responsibilities

Lead Designer (LD) • May spread across multiple squads in the lead business function.

End Market • Outside the squads but works in tandem with the squad team as an extended voice of customers
Representative augmenting the product owners.
• Attends daily sync meetings to remain engaged and up to date on the team’s progress.
• Collaborates with stakeholders outside of the team to review progress.

Tier 1/Tier 2 • Provide technical and functional governance to the squads.

• Act as SME and Go-To person for queries.
• Promote reusability and bring in experiences from other changes delivered.
• Promote a culture of innovation and experimentation.

31 IBM & BAT Confidential (Ver 2.0)

Supporting Roles and Responsibilities

Roles Responsibilities

Work Stream Lead • Provide delivery governance oversight for all squads.
• Monitor the commercials at a product/function level.
• Act as the first-level delivery escalation point for all squads.
• Monitor and report overall product/function squad performance.

Flex Team [QA/PET, • Integral part of the squad for the lifetime of the change delivery.
Data, Integration, • Performs the same tasks as a core squad member.
Testing, Services] • Act as a liaison of the squad with their home squads – helps.
squad leader to seek additional help when required.

Agile Transformation • Helps the Teams in guiding them through new WoW.
Office

Deployment Team • supports ongoing change requests for deployments across multiple
workstreams
• Includes the testing and QA members as part of this team
• Includes the change management team members

32 IBM & BAT Confidential (Ver 2.0)

3. End to End Integrated Delivery Process

3.1 Continuous Business Planning

Maximize business value and outcomes through an open collaborative standards-based platform and strong
governance framework.

Elements Capabilities

Capture, analyze, Prioritize • Adoption of an Integrated Tooling Strategy to bring continuity in the project
Business Requirements planning, Requirement capturing, and Traceability Dashboard

Project Planning • Planning capability

• Ability to measure deviation

Measure to Project Metrics • Well defined metrics

• Capability to always track and view metrics

Traceability • End to end traceability

Dashboard • Fully integrated dashboards

33 IBM & BAT Confidential (Ver 2.0)

 3.2 Collaborative Development

3.2.1 Ceremonies Explained

3.2.1.1 Backlog Prioritization
Backlog prioritization is required to organize the product backlog items (user stories/CRs) to make the
sequence of its development and deployment. This Sequence is followed by the squads to choose product
backlog items during grooming or Iteration planning.

MoSCoW (Must Have, Should Have, Nice to Have this time) is primarily used to prioritize the Product
Backlog items.

Recording, reporting &

learning from Incident, MVP or Defects Simplification & High Priority
SR, CR efficiency
Medium Priority
Group requirement or Needed functionality, Significant benefit
“Must Have” significant learning no acceptable to all users Low Priority
potential workaround

Promotes enhanced Needed functionality, Significant benefits

“Should Have” analytics or moderate acceptable workaround to many users
learning potenial exists

Non-analytical data, Optional functionality Benefit to a few users

“Nice to Have” minimal learning or cosmetic feature
potential

• Colour bands of decreasing priority Green > Yellow > Orange

• Individual priority level 1-9 to populate the backlog

34 IBM & BAT Confidential (Ver 2.0)

3.2.1 Ceremonies Explained
3.2.1.2 Iteration Planning

The purpose of Iteration planning is to define what can be delivered in the Iteration and how that work will
be achieved. Iteration planning is done in collaboration with the whole squad.

Agenda:

Duration Activity Description Who

10 Min Goal of the Iteration. Goals we want to achieve with the Iteration. Product Owner
90 Min Tasks Breakdown Breakdown of requirements into subtasks (max 1- Squad Team
and Estimations. day duration) according to Definition of Done
(DoD), Estimation of subtasks.
10 Min Iteration Capacity • Calculation of the capacity of individual Squad Team
calculation team members.
• Update of Iteration scope based on the capacity.
10 Min Commitment Do you commit to the Iteration plan? Squad Team
Board Preparation Evidence of created tasks in the tool Squad Team

Outcomes:

Product Owner Squad Leader Squad Team

An introduction to Iteration on goal and Arrange the planning meeting, invites Breakdown of the requirements into
top priority items that must be the necessary people, and prepare the subtasks (max. 1-day duration)
developed in the Iteration. agenda for the meeting. according to Definition of Done (DoD).

35 IBM & BAT Confidential (Ver 2.0)

3.2.1 Ceremonies Explained
3.2.1.2 Iteration Planning

Outcomes:
Product Owner Squad Leader Squad Team

PO ensures that user stories/CRs are Board set-up (new Iteration creation and Estimation of tasks.
fulfilling the team’s Definition of Ready drag/drop of user stories as necessary
(DoR) before the Iteration planning. for planning).
Participates in the discussion about Helps team understand metrics so Identification of dependencies
User Stories and acceptance criteria (AC). the team can plan work better than in and blockers.
subsequent Iterations.
Update of the order of user stories/CRs Prepares for team capacity Update the total scope of the
based on information discovered in the calculation. Iteration based on capacity and
Iteration planning and team capacity. dependencies.
Communication of planned scope with Moderates Iteration planning Commitment to the Iteration
stakeholders and other product meetings and takes care of the backlog
owners. meeting time-box.

Synchronizes dependencies with

other squad teams and squad
leaders as necessary.

3.2.1.3 Daily Sync Meeting

Daily Sync meeting is done by the entire development team to synchronize the effort for a successful finish of
an Iteration. The meeting is moderated by Squad Leader. It takes place at the same time and daily, preferably in
front of a team’s board. In the case of a distributed team, remote team members are connected using
collaboration tools.

36 IBM & BAT Confidential (Ver 2.0)

3.2.1 Ceremonies Explained
3.2.1.3 Daily Sync Meeting

Outcomes:

Product Owner Squad Leader Squad Member

Participation in the meeting is Identification of typical issues in task Synchronization of the team
highly welcome. handling on the board. members’ work.
Evaluation of the state of the Iteration & the Process errors and bottlenecks of the Updating the completed tasks.
possibility of its completion on time. value flow before the meeting.

Potential adjustments in the Iteration Preparation of space and tools for Identification of the tasks which every
scope according to the degree of Daily Sync meeting. member of the team will be working
completion. on that day.

Communication of important changes in Meeting moderation. Updating the remaining time of the in-
the scope. progress tasks on the board.
Approval of changes due to unplanned Restrictions recording and informing Identification of the limitations.
requirements in case of scope change team members about the status of
during the actual Iteration. previously reported team limitations. Proposing the modifications for
achieving the Iteration goals
(if necessary).

Visual Task Board (VTB) in SNOW:

Squad’s can have a single VTB created for all the CR’s/user stories that each squad is working for. Squad members can pull the work from VTB and update board accordingly.
For assistance on VTB creation, please reach out to the transformation office members.

37 IBM & BAT Confidential (Ver 2.0)

3.2.1 Ceremonies Explained
3.2.1.3 Daily Sync Meeting

Below is Demo squad VTB snapshot from SNOW

3.2.1.4 Iteration Review

Iteration Review is called an Iteration Demo/playback because the information is provided in form of a live demonstration showing completed functionality. In absence of a working
product feature (also called the Product Increment), the outcome of the Iteration can be summarized (e.g., a design document, framework, etc.) in a shareable form and the
stakeholders are walked through in the playback session to gather feedback and report progress.

38 IBM & BAT Confidential (Ver 2.0)

3.2.1 Ceremonies Explained
3.2.1.4 Iteration Review

Agenda:

Duration Activity Description Who

10 Min Goal of the Iteration. • Introduction of Iteration goals. Product Owner

• Information about the impact of the Iteration on
release plans.
• Review of the roadmap.

10 Min Review of the top • Review of up to three top requirements related to Product Owner
requirements. the Iteration goal

10 Min Iteration status. • Share information about the plan vs. actual. Squad Leader
• Review of the Iteration statistics.
• Review of the important technology changes or
improvements, enablers.
• Statistics of bug fixes
• Current program increment (release) statistics.

15 Min Demonstration. • Live Demo of the completed requirements/play- back Squad Team
of the outcome.

10 Min Feedback. • Collecting the feedback from stakeholders. Squad Leader

5 Min Closing • Information about the next Iteration review. Squad Leader
• Publishing Iteration review presentation.

39 IBM & BAT Confidential (Ver 2.0)

3.2.1 Ceremonies Explained
3.2.1.4 Iteration Review

Outcomes:

Product Owner Squad Leader Squad Member

An introduction to Iteration goals. An organizer of the meeting, and an Informing about the Iteration status
invitation to participants.
An introduction of top planned Moderator of the meeting. Live demonstration of
requirements that the team has functionality/playback of outcome.
committed to deliver.

Iteration status overview. Evidence of feedback.

Information about defects and improvements

solved during the Iteration improving quality.

3.2.1.5 Iteration Retrospective

The Iteration retrospective is a recurring meeting held at the end of an Iteration used to discuss what went
well during the Iteration cycle and what can be improved for the next Iteration.

Agenda:

Duration Activity Description Who

5 Min Preparation of the topic. • Theme of the topic (Set the Stage.) Squad Leader

10 Min Review of the previous • Status review of the Ideas from the previous Squad Team
improvement ideas Retrospective session

40 IBM & BAT Confidential (Ver 2.0)

3.2.1 Ceremonies Explained
3.2.1.5 Iteration Retrospective

Agenda:

Duration Activity Description Who

15 Min Gather Data. • Gather data using a selected retrospective Squad Team
technique.

10 Min Generate Insights. • Grouping of the similar ideas. Squad Team

• Discussion about ideas to get a common
understanding.

10 Min Decide what to do. • Voting for ideas. Squad Team

• Prioritization of Ideas.
• Assign responsible team members.

5 Min Close Retrospective. • Get feedback about the retrospective session. Squad Team

Collaborative whiteboarding tools like Mural may be used as the teams are distributed.

41 IBM & BAT Confidential (Ver 2.0)

3.2.1 Ceremonies Explained
3.2.1.5 Iteration Retrospective

Outcomes:

Product Owner Squad Leader Squad Member

Considering the Product Owner (PO) as Organizes and moderates the Proposes ideas.
part of the agile team aligns with the meeting.
common goal (to deliver a working
product).

The Product Owner is an active Identification of the topic of Votes for ideas to be implemented.
participant the retrospective as Retrospective.
other team members.

Selection of a technique for Agrees on how to implement the most

getting ideas. important improvement ideas.

Informs about the resolution status of Identifies the tasks for implementing the
previous improvement ideas. selected ideas.

Keeping track of improvement ideas and

their resolution.

42 IBM & BAT Confidential (Ver 2.0)

3.2.2 SNOW CR Phases & Activities

Below provides the mapping between our WoW activities with SNOW Project Phase and tasks.
This is the current view of tasks when a SNOW waterfall template is applied. Going further, when the Agile/Hybrid template is
applied the table will be enhanced accordingly.

WoW SNOW Project Phase Squad Member

Squad Formation, Kick Off Meeting, Initiating INITIATE (TDE Preparation)

CR Selection, Discovery Phase. INITIATE (TDE Approved)
INITIATE (TDE Costing)

Backlog Grooming Planning PLAN (Design)

Planning the DBT Timeline
Iteration Planning

Daily Sync Meeting Executing EXECUTE (Build)

Iteration Review EXECUTE (FUT)
Iteration Retrospective EXECUTE (UAT Preparation)
EXECUTE (SIT)
EXECUTE (Data Load Prerequisite –
Test System)
EXECUTE (Data Load – Test System)

Delivering DELIVER (UAT)

DELIVER (Data load Prerequisite – Regres-
sion)
DELIVER (Regression Test)
DELIVER (PET)
DELIVER (Technical Deployment: TGL)
DELIVER (BGL)

Closing CLOSING (Hypercare)

43 IBM & BAT Confidential (Ver 2.0)
 3.2.2 SNOW CR Phases & Activities

1. Once CR is Pre-Approved, BAT will create a project in SNOW with Prefix PRJXXXXX, and the
description will have a CR number.
2. The project will be assigned to the Workstream Lead and in turn Workstream Lead [Finance,
Marketing, Operations] would assign it to Tier 1, Tier 2, and the other relevant members working
on the CR.

INITIATE (TDE Preparation)

1. Tier 1 & Tier 2 members will start analyzing the CR.
2. If the CR does not have complete information, it would be put ON HOLD in SNOW with the comments
in Notes.
3. Once the CR analysis is done, a template would be applied based on the CR type i.e., Data Relevant
CR, Regression relevant, etc. and tasks would be generated automatically.

INITIATE (TDE Approved)

1. Once LD and GDA approve TDE, TDE approved task would be closed.
2. In case there is any observation or issue from BAT end, TDE would be put ON HOLD, and task status
would be moved back to TDE PREPARATION and details would be mentioned in Notes.
3. In case of any missing information or such task would be moved back to TDE PREPARATION State.

INITIATE (TDE Costing)

1. IBM would submit the costing to BAT.
2. BAT will approve the Costing and the eventual task would be closed.

PLAN (Design)
1. Once the Design is completed, task would be closed.

EXECUTE (Build)
1. Once the Build is completed, task would be closed.

44 IBM & BAT Confidential (Ver 2.0)

 3.2.2 SNOW CR Phases & Activities

EXECUTE (FUT)
1. Once the FUT is completed, task would be closed.

EXECUTE (UAT Preparation)

1. Manual Build activities would be performed.

DELIVER (UAT)
1. Once the EXECUTE (UAT Preparation) task is closed, DELIVER (UAT) task would be kept open to indicate
that CR is ready for UAT and would be handed over to the Testing Team.
2. Until UAT gets passed, this task would be kept open waiting for the defects to get logged.
3. If any defects are logged in HPQC, an email would be triggered to the developer for corrections.
4. A new TR would be opened, and defects would be resolved.

This set of SNOW activities is specific to the template currently in use in ESCOE delivery aligned to the “Waterfall” template. As the delivery organisation
matures in the new ways of working and transitions into the “Hybrid” or “Agile” templates in SNOW, this set of activities will be amended and updated.

45 IBM & BAT Confidential (Ver 2.0)

3.2.3 Key Performance Indicators

KPI is a quantifiable measure that gives a vision of the goals being met. The table below provides a mapping of DevSecOps
features covered, calculation, and data source of each metric.

KPI DevSecOps Feature Addressed Calculation Tool / Data Source

Lead Time 1.Continuous Business Planning. Elapsed Time Between CR raised in CREW (or CREW
2. Collaborative Development. Service Now) and Technical Go-Live Date [TGL [SNOW from Q1 2023]
3. Continuous Testing. Date – Pre-Approved Date] In the interim we will need to use CREW.
4. Continuous Release & Deployment. Refer below Picture 1, for TGL date in SNOW.

SNOW
Cycle Time 1. Collaborative Development Elapsed Time between DBT actual start date and
The DBT actual start date and FUT closure will be tracked
[Development] FUT actual end date.
through the planning task level actual start date and
project template task – Actual End date respectively.
Refer below Picture 3, for DBT start date and Picture 2 for
FUT actual end date in SNOW.
Cycle Time [Test 1.Continuous Testing. Elapsed Time between FUT actual end date and SNOW
& Release] 2. Continuous Release & Deployment. Technical Go-Live date. The date of FUT closure and TGL will be tracked through the
project template task – Actual End Date. Refer below
Picture 1, for TGL date and Picture 2 for FUT actual end date
in SNOW.

Defect Leakage to 1.Collaborative Development. Ratio of [Incidents raised for CRs in a release and SNOW + HPQC
Production [All 2. Continuous Testing. the total defects found in FUT+SIT (if any) [Plus, a possible intermediary calculator]
Priorities] 3. Continuous Release & Deployment. + UAT + Incidents]
4. Continuous Monitoring and Optimisation.

46 IBM & BAT Confidential (Ver 2.0)

 3.2.3 Key Performance Indicators

Below is the reference snapshot for pre-approved date and TGL date in SNOW:

Picture 1

47 IBM & BAT Confidential (Ver 2.0)

 3.2.3 Key Performance Indicators

Below is the reference snapshot for FUT actual date in SNOW:

Picture 2

48 IBM & BAT Confidential (Ver 2.0)

 3.2.3 Key Performance Indicators

Below is the reference snapshot for actual start date[Planning Task] in SNOW:

Picture 3

*All the above metrics and snapshot references are based on current WoW in SNOW. Same will be revisited and modified accordingly as WoW evolve.
*Data required for metrics currently is to be gathered manually from different tools.
*This metrics are practically implemented from ongoing experiment with marketing squads.
*SNOW reporting capabilities provides the above metrics at specified frequency.
*Defect Leakage into production will be calculated with the incidents raised up to 1 month after TGL.

49 IBM & BAT Confidential (Ver 2.0)

 3.3 Continuous Testing

The goal of continuous testing is to evaluate the quality of the software as part of a
continuous delivery process. This is done by testing early and often.

Elements Capabilities

Test Management and Execution • Test Cases, Test Plan & Test Scripts are written during high level design
• Agile approach in place to test the requirements on a continuous way

Test Automation • Automated Testing and Feedback Mechanism

• Automatic reporting and distribution of defect status

Traceability • Automatic dashboard preparation and communication

Test Data Management • Test Environment (SIT/UAT/Pre-Prod) are identical to Production

• Unit Test Cases are Automated, and Code Coverage Dashboard is available for all
releases

50 IBM & BAT Confidential (Ver 2.0)

3.4 Continuous Release & Deployment

The goal of continuous Release & Deployment is to ensure that new changes do not affect the existing
functionality.

Elements Capabilities

Release Management • Release planning having a view on actual vs plan

• End to End traceability with a well-defined tracking process
• Automatic tracking report generated using the integration of tools

Environment Management • Ability to provide an environment as required.

Deployment Automation • Deployment functionality implemented for all non-production systems

51 IBM & BAT Confidential (Ver 2.0)

3.5 Continuous Monitoring

This is needed to monitor changes and address errors and mistakes spontaneously whenever
they happen.

Elements Capabilities

Operational Analytics • Modern analytics tools used, and feedback used for
continuous improvements

52 IBM & BAT Confidential (Ver 2.0)

3.6 Continuous User Feedback and Optimization

Continuous Feedback helps achieving more effective delivery.

Elements Capabilities

Monitor Capacity and Optimize • Automated monitoring and quick turnaround time.
• Adopt an Integrated Tooling Strategy to bring continuity in the Build
management with Test Management.

Monitoring performance and • Monitoring using tools and automated feedback to the dev team.
optimize • Modern DevSecOps tools for monitoring and integration with development
processes and tools.

Monitoring performance and • Linking online errors, and suggestions in the feedback tool with
optimize Continuous Business Planning processes.
• Integrated online feedback collection service.
• Option to provide feedback anytime during all stages.
• Pain points are to be captured and traceability is to be established
between operation feedback and developments.

Event and Incident management • Latest log management software which is integrated with applications and
infrastructure.

53 IBM & BAT Confidential (Ver 2.0)