White Paper CICD
White Paper CICD
White Paper
Beyond Banking: How Continuous Integration & Continuous
Deployment (CI/CD) Accelerates Customer Value Creation
           in the Financial Services Industry
Impressum
1
Beyond Banking: How Continuous
Integration & Continuous Deployment
(CI/CD) Accelerates Customer Value
Creation in the Financial Services
Industry
Insights on Commerzbank’s Holistic CI/CD Approach
Dear Reader,
The market has changed. Digitization is         our experience and best practices. Toge-
in demand everywhere today. While in            ther with various experts from
the past it was mainly about hardware in        different companies and industries we
the financial industry, today it is primarily   took a look into the future of CI/CD.
about software. And the way how to deli-        I hope that you will enjoy reading this
ver software defines the success of every       white paper and gain interesting insights
player in the market since today a bank is      into the concept of CI/CD and how Com-
a software-driven company. Because at           merzbank applies this concept to develop
Commerzbank, we accept that our DNA             superior products for its customers.
has changed. We are now a technology-
driven innovation bank.                         Best regards,
The concept of Continuous Integration
and Continuous Deployment (CI/CD) has
gained momentum and changed the way
software is developed within the financial
services industry. The driving force behind
this momentum is acceleration which is
key to deliver high-quality and customer-
centric software products. At Commerz-
bank, we have started our CI/CD journey
around 3 years ago alongside with a dra-
matic organizational change to agility. We
are looking forward to sharing with you                          Florian Meiser (Commerzbank AG)
                                                                                              2
3
Content
Executive Summary              06
Introduction08
Appendix31
                                     4
Executive Summary
In this white paper, we aim to describe how the software development process can be
accelerated by pinpointing the CI/CD concept: its cornerstones, how it affects customer
value creation in general and in financial services in particular, how Commerzbank is le-
veraging CI/CD to drive innovation and shorten release cycles, and which developments
we can expect in the upcoming five years in agile software development.
We concluded that CI/CD is a set of best practices and tools which enable organizations
to accelerate customer value creation, time-to-market, and the delivery of software qua-
lity. We would like to share with you the five key take-aways we identified during our CI/
CD journey and the research process of this white paper:
01                                                              02
CI/CD accelerates customer value crea-                          The implementation of a CI/CD pipeline
tion in the financial services industry                         needs to follow a set of technical
The objective of CI/CD is to be able to daily release high-     principles
quality software that creates value for the customer. At        The CI/CD pipeline is the backbone of every CI/CD ap-
Commerzbank, we believe this especially for organizations       proach. It reflects the automation engine alongside the
in the financial services industry which are still undergo-     software development lifecycle and is governed by the
ing a radical shift driven by increasing competition, high      technical principles of full automation through highly
regulatory requirements, structural low interest rates, and     integrated tools connected via APIs. What`s more, a
fast changing customer needs, CI/CD is key to being closer      standardized and scalable pipeline, high reliability and
to the customer. CI/CD enables greater automation in the        availability, and a technology agnostic pipeline are incor-
whole software development and delivery process. This all-      porated. Although an effective CI/CD pipeline is built upon
ows us to release daily, new digital banking solutions to our   these principles, there is no one-size-fits-all approach to
customers and in turn results in a higher customer value.       implementing a CI/CD pipeline. Since its setup is determi-
                                                                ned by company-specific internal processes, resources,
                                                                and infrastructure. The approach of continually gathering
                                                                feedback during the implementation phase and adjusting
                                                                the process accordingly is best-practice we discovered.
                                                                Furthermore, organizations should accentuate those tech-
                                                                nical elements in their CI/CD pipelines that fit their organi-
                                                                zational conditions best, in accordance with regulations to
                                                                successfully deliver value to their customer.
5
03                                                                                    04
CI/CD is not just about tools and proces-                                             An empowering CI/CD approach lever-
ses, it is also about teams that have end-                                            aging the strength of cross-functional
to-end responsibility                                                                 teams is key
A successful CI/CD approach significantly depends on                                  Building on the foundation of agile software development,
giving teams end-to-end-responsibility of the software                                cross-functional teams comprising of development and
development lifecycle so that they can create software                                operational functions, alongside business and security
features without significant dependencies on other teams                              experts ensure the targeting of key business outcomes and
to accelerate speed, quality, and customer value creation 1.                          compliance with security standards throughout the CI/CD
This requires organizations to enable their teams technolo-                           lifecycle. We are convinced that “BizDevSecOps” 2 teams
gically and culturally with the right tools and methods. Our                          will be the new form of collaboration and co-innovation
experience also shows that the combination of a self-ser-                             and a decisive factor for organizations to deliver software
vice platform and availability of an ongoing technical and                            more quickly, safe and responsive to customer demand.
methodological support for teams, if necessary, lays the                              To implement this new team constellation, organizations
foundation for teams to take over end-to-end responsibili-                            need to empower their teams with the appropriate training
ty within the whole CI/CD software development lifecycle.                             and coaching solutions to bridge the several functions and
Although, organizations in the financial services industry                            enable them to carry on the different tasks throughout the
need to comply with specific regulations concerning the                               CI/CD lifecycle. We think that the alignment of software
separation of development and operations entities. Esta-                              development and business objectives in accordance with
blishing end-to-end responsibility of teams further accele-                           security standards and built upon shared business metrics
rates the CI/CD software development process.                                         significantly contributes that the whole team. This enables
                                                                                      working towards the shared goal of accelerating the soft-
                                                                                      ware development process.
05
Everything as Code (EaC) as the new
norm, meaning automation via software
will be everywhere
We expect that the EaC development will accelerate in the
upcoming five years leading to the practice that everyt-
hing such as operations, infrastructure and configuration
management adheres to the same software development
practices 3. The EaC principle plays a central role for other
developments such as enhanced cross-functional collabo-
ration or a shift left approach of security, compliance, and
documentation. Additionally, EaC provides the technical
conditions for establishing a common CI/CD platform that
serves as single point of truth for data, documentation,
and tracking points that can be established to measure the
speed or waste in certain phases of the software develop-
ment lifecycle. Cross-functional teams using this CI/CD
platform in their daily business will use a common langua-
ge of code that enables collaboration and co-innovation
among the different functions. We defined EaC as one im-
portant component of our future CI/CD approach and will
follow a zero-documentation approach which ensures an
automated and compliant documentation processes along
the CI/CD lifecycle.
1
    For more information, please see our learnings from CI/CD Use Cases on p. 18.
2
    For a detailed explanation of BizDevSecOps, please see p. 24.
    For a detailed explanation of software development practices, please see p. 17.                                                            6
3
Introduction
As organizations continue their journey of accelerating          solutions through the CI/CD “Commerzbank Hyper Acce-
their agile software development process, Continuous Inte-       leration Master Pipeline” (CHAMP 4) for all other clusters 5
gration and Continuous Development (CI/CD) has become            within Commerzbank,” says Florian Meiser (Commerzbank
crucial to deliver customer value faster [1]. With the need      AG). Our objective is to support Commerzbank`s digital
to deliver quickly high-quality and customer-centric soft-       transformation and the delivery of innovative solutions for
ware, organizations focus on CI/CD to develop and release        customers by providing a fully automated self-service pipe-
software in short iteration cycles supported by cross-func-      line that comprises all software development steps from
tional collaboration, automation, and tools. Today, CI/CD        the initial idea to the final product while assuring compli-
forms the operational backbone in a modern DevOps envi-          ance with regulations and enforcing the spread of modern
ronment while combining automation and cross-functional          software development techniques.
collaboration from customer needs and business require-
ments to software development, testing, and operations to        With this white paper, at Commerzbank we aimed at crea-
improve the overall software development process.                ting a common understanding of the CI/CD concept, espe-
                                                                 cially about the way how it affects customer value creation.
CI/CD as a set of practices and tools enables a faster time      Moreover, we want to give you a detailed insight into our
to market through rapid, continuous integration of code          holistic CI/CD approach and would like to share with you
changes, automated tests and continuous deployments              our experience on our CI/CD journey so far – things we
of every code change to production after successfully            have learned and our best practices that might hopefully
passing all quality gates [2]. The concept rests on the in-      support you in your upcoming agile software develop-
terrelated cornerstones of the right tools, streamlined and      ment endeavors. Additionally, we discussed with 14 subject
automated processes, and skilled people that if synchroni-       matter experts from different industries in which direction
zed drive collaboration, innovation, and automation within       agile software development is expected to heading to in
the organizations which ultimately improve customer              the next five years. We will provide you with the result in
experience. “CI/CD offers organizations immense value:           the form of five central hypotheses about the future of
faster release of products and services, stronger customer       agile software development until 2026.
engagement, quicker feedback on new features, higher-
quality code, and ultimately higher customer value,” says        Today, CI/CD has become an essential technology enabler
Jörg Oliveri del Castillo-Schulz (Commerzbank AG).               for creating customer value, and we expect that the way in
                                                                 which software is developed within companies will beco-
Specifically, the financial services industry might benefit      me an even more decisive success factor. We hope that
from these advantages since the industry faces significant       this white paper will provide insightful information on our
challenges due to the macroeconomic situation, fierce            Commerzbank CI/CD CHAMP approach, contribute to the
competition, and high customer expectations [3]. However,        discussion of the future of agile software development, and
the way in which CI/CD is applied in the financial service       trigger new questions that will drive all of us towards new
industry is significantly impacted by national and interna-      horizons of agile software development.
tional regulatory requirements, that need to be considered
when designing and introducing CI/CD in agile software
development process. Additionally, in the time of the digi-
tal transformation of the whole financial services industry,
a financial institution is a software-driven organization that
needs to use state-of-the-art technologies to provide cus-
tomers with an exceptional user experience.
                                                                 4
                                                                     For a detailed explanation, please see “How Commerzbank is living CI/CD” on p. 13.
7                                                                5
                                                                     For a detailed explanation refer to “Glossary”, page 36.
Continuous Integration &
Continuous Deployment –
What is it About?
As software continues to eat the world, organizations are        tools [1], which will be connected throughout all essential
required to accelerate their software development process.       software development steps [2].
In this context, CI/CD has become a key technological ena-
bler for the fast delivery of high-quality software features     The concept of CI/CD forms the operational backbone of
that satisfy customer expectations. But what is CI/CD and        DevOps practices and enables the automation and ac-
how does it benefit organizations? And which aspects are         celeration of the whole software development process,
underlying? The following chapter aims at demystifying the       from planning software over testing it to finally deploying
concept of CI/CD by providing the theoretical foundations        software to production [11]. As seen in figure 1 CI/CD is
of this concept including its cornerstones and possible          composed of two interrelated value streams – a continuous
business impact. Additionally, we explain why CI/CD is one       integration and continuous deployment cornerstone cover-
crucial technology foundation for Commerzbank’s agile            ing the respective activities of the software development
transformation and how we leverage this set of technical         process.
practices to maximize value for our customers.
                                                                 In accordance with the overarching idea of agile software
CI/CD Explained                                                  development, CI facilitates merging of software chan-
CI/CD has its roots in agile software development [1],           ges from multiple contributors to a shared source code
which aims at a software development process being lean,         repository frequently, often many times a day [12]. After
collaborative, goal-oriented, and responsive to change and       each merge, the chunk of code is automatically built and
customer feedback [4]. In essence, the objective of agile        tested to ensure the code is still functional after each
software development is to produce high-quality soft-            code change, to avoid potential integration issues and to
ware that best satisfies customer needs [4]by cultivating        guarantee that it is safe for other developers to write code
and promoting organizational capabilities such as sensing,       on top of the new changes [13]. Companies that start with
learning, adaptability, resilience, speed, innovation, colla-    introducing CI usually face two challenges comprising
boration, and efficiency [5]. One approach to operationali-      changing the habits of developers to do smaller frequent
ze these capabilities is following the paradigm of DevOps,       code commits and ensuring the needed automation to
which aims at bridging the gap between the functions of          support the software development flow and reduce the
development and operations and aligning their activities         time to integrate and validate code changes. The validation
across the organization [2][6]. On the one hand, DevOps          itself comprises not only the automated unit tests but also
combines functional skills from business, software de-           the automated static code analysis and security scans to
velopment, testing, and operations in one integrated team        early detect code vulnerabilities. The practice of shifting
improving cross-functional collaboration and communica-          this validation while performing unit or integration tests
tion [2]. On the other hand, it empowers teams with end-         and code quality checks early in the software development
to-end responsibility which increases the ability to faster      process is referred to as “shift left” and can result in signifi-
deliver high quality software.                                   cant cost reduction as well as in increased software quality
                                                                 since potential pitfalls in the software code are identified
This stands in contrast to the formerly applied software de-     early during the development process [12].
velopment processes, where functional and informational
silos or step-by-step sequential processes lead to long de-      CD picks up where CI ends comprising the software de-
velopment times: “in enterprise software 1-2 years, in new       velopment process steps of deploying and releasing to
microprocessors 2-4 years, in automobiles 3-5 years, and in      production. Within the CD phase the software artefact is
aircraft a decade” [7]. Before DevOps, software was often        automatically launched and distributed to end-users after
developed isolated as a single piece and, after finishing        it has successfully passed automated tests to validate if
the complete development, tested as a whole [8]. Now,            changes to a codebase are correct and stable for immedia-
crucial activities, such as testing, are parallelized with the   te automated deployment to the production environment
development phase [9] and software is developed in itera-        [14]. In living customer centricity, it is important that new
tive cycles, building on intermediary results and feedback       products or features are deployed smoothly and without
loops [10]. Hence, possible pitfalls can be detected faster,     delay. CD complements the process stage of CI leading to
and solutions can be implemented more quickly. Another           a faster and smoother release of new software and makes
important strength is the increasingly integrated technolo-      it possible to continuously integrate customer feedback
gy stack comprising development, testing, and integration        early in the product lifecycle.
                                                                                                                                8
                                                                                                           Feedback Loop
                                                                1. DevOps Teams
“Overall, the concept of CI/CD is an important enabler          Excelling in CI/CD requires the right people with the ap-
for organizations to accelerate the software develop-           propriate skills and mindset as it applies to the successful
ment process leading to a potential reduction of the            implementation of almost every agile software develop-
time-to-market of new products and features to cus-             ment practice [16] accumulated in one DevOps team. A
tomer,” says Sabine Vigelius (Commerzbank AG). In this          recent study of the DevOps Institute [17] elaborates that
context, the benefits of CI/CD are manifold. First, CI/CD       people skills such as collaborating, problem solving, know-
leads to a shorter time-to-market since development teams       ledge transfer and adaptability are key for success in living
can release software builds faster due to incremental soft-     DevOps and CI/CD. Additionally, individuals with an open
ware development [1]. Second, CI/CD ensures an early            and innovative mindset are needed to further develop
detection and fixing of software vulnerabilities and defects    existing products and services to deliver customer value.
due to the shift left approach which ultimately results in      According to the State of DevOps Report 2021 [18], orga-
higher code quality [15]. Third, the automation inherent in     nizations that have successfully adopted the concept of CI/
the CI/CD lifecycle ensures that crucial process steps of       CD ensure that their DevOps teams have strong identities,
the software development process are less prone to human        clear responsibilities with a high degree of autonomy in
error and easier to manage by the team [16]. Fourth, CI/        their own function as well as distinct interaction ceremo-
CD reduces the cost of communication as it supports the         nies and communication channels with other teams.
DevOps practice of aligning cross-functional teams. Fifth,
CI/CD allows for integrating customer feedback early in
the software product lifecycle enabled by short feedback-       A corporate culture that promotes a workforce living CI/CD
loops [1].                                                      principles and successfully applying the technical practices
9
of the CI/CD concept is based on a collaborative environ-           lize full potential of an organization’s CI/CD pipeline. Since
ment [19]. Another important aspect is to impose end-to-            agile software development is built on fast feedback loops
end responsibility among DevOps teams that work with                and constant collaboration between different stakeholders,
CI/CD practices and tools ensuring that all team members            the processes within the CI/CD pipeline need to foster this
together are fully accountable for the software artefact            environment [21]. The process along the CI/CD pipeline
across the whole software development cycle. This sup-              typically comprises the steps of plan (setting up the soft-
ports a collaborative team spirit while providing the possi-        ware project with its requirements), code (the stage where
bility for each team member to contribute his or her ideas          developers implement the IT product requirements), build
and strengths during the whole software development                 (the stage where the IT artefact is created), test (the stage
life cycle [17]. Last but not least, encouraging continuous         where code is tested), deploy (the stage where code is
learning. This helps us support the development of DevOps           deployed to production), and release (the final delivery of
culture. This can be achieved by promoting employees to             the software product to the customer) [2]. A powerful CI/
embrace failure and learn from it by creating a psychologi-         CD pipeline that increases speed and quality of the who-
cal safe environment through meaningful and constructive            le software development process is based on an optimal
feedback [17].                                                      orchestration and seamless connection of the beforemen-
                                                                    tioned process stages.
6
    See sub-chapter “CI/CD Explained” for a detailed explanation.                                                               10
the foundation for teams to collaborate and communicate                might add its special features to the specific process stage.
throughout the CI/CD process workflow to maintain align-               It is important to create interfaces between separate tools
ment, velocity, and quality [17]. Moreover, teams using a              to enhance the seamless and automated interaction bet-
CI/CD pipeline benefit from more freedom in the way they               ween them. Moreover, each tool comes with its very own
conduct business and collaborate since the process along-              limitations that it brings to the whole process which need
side the CI/CD pipeline reduces processual limitations [18].           to be considered to avoid potential negative effects on the
                                                                       functionality of the CI/CD pipeline [22]. Additionally, too
                                                                       many tools used in the CI/CD pipeline increase complexi-
Although, adapting processes with sight to the CI/CD pipe-             ty which could negatively impact their usability as well as
line offer valuable benefits, this also might come with some           accuracy. These factors might negatively affect the smooth
costs since changing processes from existing well-esta-                process workflow of the CI/CD pipeline and could there-
blished processes to CI/CD-ready processes requires an                 fore decrease quality delivered to the customers. Hence, it
extensive change effort within the whole organization [18].            is important to find the right balance between the number
In a running organization it would be almost impossible to             of tools with their special features for each CI/CD process
stop the existing process approach and start completely                stage and ensuring manageability of the whole tool chain.
new from scratch. In addition, it is a costly and time-con-
suming process to integrate an organization to a CI/CD
pipeline since the long-grown technology stack is charac-              Another important aspect in the context of CI/CD corners-
terized by a high degree of complexity [18].                           tone CI/CD tools is the question of build or buy. Although,
                                                                       the open-source community is gaining more and more
                                                                       traction nowadays regarding powerful DevOps Tools and
3. CI/CD Tools                                                         the development in cloud-computing enables a shift of
The process on the CI/CD pipeline is supported by various              the process towards a more powerful environment [21],
tools that ensure each process step is seamlessly integra-             at Commerzbank, we believe that an organization should
ted within the software lifecycle and automated to reach               assess its strength and weaknesses before choosing a CI/
the maximum acceleration potential of the CI/CD pipeline.              CD toolchain. An off-the-shelf tool or even a pre-made
These tools connect different systems and ensure the ref-              pipeline including all necessary tools might be simpler
lection of any change in the source code across different              than building a tool itself, but it might sacrifice flexibility.
environments [21]. Additionally, the cloud native tools of             In our experience, organizations can combine the best of
the CI/CD pipeline technically support the agile software              both approaches individually deciding about which tools
development process by accelerating the building of new                should be collected from a vendor or if a tool built by the
applications and integration of user feedback and by con-              organization itself will more fit the individual organizational
tributing to dismantling borders between development and               requirements. We believe this is more pronounced in the
operation teams [23]. Each tool used in the CI/CD pipeline             Financial Services Industry since financial institutions have
Automation
11
oftentimes specific requirements due to their long-grown                                   on is key. Releases should happen daily, ideally on-de-
software stack and regulatory parameters that need to be                                   mand, whenever a new feature is ready. Only with this
considered. Thus, the possibility that financial institutions                              approach we can realize our goals - deliver delightful
can simply leverage the advantages of a pre-made CI/CD                                     software products to customers and get ahead of
pipeline including a ready-to-go tool chain might be illusio-                              competition.
nary and therefore they usually focus on building their CI/
CD pipelines integrating relevant tools on their own.                                  •   Next, the Pipeline: It is mostly a technical solution that
                                                                                           executes CI/CD at Commerzbank and enables teams
                                                                                           to deliver to production with high quality. We provide
To summarize, the realization of the whole potential pro-                                  various tools in each stage of the software develop-
vided by CI/CD pipelines requires a holistic view of this                                  ment process - from integrated development environ-
practice. At Commerzbank, we believe that the reflection                                   ments (IDE) to deployment tools. Since all tools are
of the CI/CD components and their interconnectedness in                                    seamlessly integrated in one pipeline, synergy effects
the respective CI/CD endeavors ensures that CI/CD leads                                    are achieved. On top of this, at Commerzbank we offer
to a more collaborative culture, an innovation-driven mind-                                coaching solutions - training teams in efficiently using
set, and a higher-level of automation. In the following we                                 the tools and methods. This is complemented by a set
will present how we are living CI/CD at Commerzbank and                                    of internal policies that govern the software develop-
in which way we have integrated the CI/CD cornerstones.                                    ment process, implemented as a set of documents and
                                                                                           tools that guide users in fulfilling regulatory require-
                                                                                           ments.
How Commerzbank is Living CI/CD
The way in which the DevOps paradigm and the CI/CD                                     •   Lastly, a Master (Pipeline), with all the breadth of tools,
concept are applied in the financial services industry is sig-                             platforms, target environments, middleware compo-
nificantly impacted by national and international regulatory                               nents, programming languages, frameworks and high
requirements. Regulators such as the European Central                                      national and international regulatory requirements
Bank oversee not only the business model of financial insti-                               coming at Commerzbank, we think that it is crucial
tutions but also impose regulations on their IT Infrastruc-                                that there is one single master pipeline that fulfils all
ture and thus the banking specific software development.                                   those requirements. This pipeline leads to a high level
These regulatory requirements need to be considered                                        of confidence since everything runs smoothly if well
when financial institutions apply CI/CD practices and prin-                                orchestrated.
ciples, build their CI/CD pipelines as well as set up relevant
methods and processes in their DevOps teams.
                                                                                       Since CI/CD CHAMP is focused on the acceleration of
                                                                                       Commerzbank’s delivery organization, we prefer to employ
At Commerzbank, we started with an integrated approach                                 the analogy of a racetrack in Formula E. As in Formula E,
to build our CI/CD pipeline in 2018 as a technology enabler                            a successful CI/CD pipeline is about speed, engineering
of our agile transformation journey to implement our con-                              discipline and outstanding teamwork to facilitate the com-
tribution of being “the bank at your side 7 ” . “What drives                           plete automation of the delivery process from the initial re-
us is our commitment to provide a superior customer                                    quirement to delivering the final product to our customers.
experience by offering digital, individualized and highly                              As seen in figure 3, CI/CD CHAMP is composed of three in-
valuable products and services,” says Dr. Alena Kretzberg                              terrelated services: (1) Frameworks assuring the complian-
(Commerzbank AG). Today, Commerzbank has established                                   ce with regulatory requirements during the whole software
the cluster “Continuous Integration/ Continuous Deploy-                                development process; (2) Automated Pipeline comprising
ment CHAMP” (CI/CD CHAMP) as a technology foundation                                   all technical steps and tools from development over testing
for the fast and automated delivery of new functionalities                             to production, and (3) DevOps Coaching promoting state-
to the organization’s customers.                                                       of-the-art development methods and practices within the
                                                                                       cross-functional agile teams.
7
  Our proven brand promise “Commerzbank. The Bank at your side” puts our brand posi-
tioning in a nutshell, describes what we stand for and what we promise to our customers.                                                            12
          CI/CD
•    CI/CD CHAMP Frameworks: As in a race situation                    The PO-Cockpit serves as the regulatory single point of
     having the necessary safety equipment, for instance a             truth for any audits and controls and provides an overview
     racing helmet or racing suit, is crucial when leveraging          of all CI/CD endeavors. Since the required documentation
     the maximum speed of your racing car while knowing                depends on the planned software solution, the PO-Cockpit
     that all necessary conditions are met to bring you                allows employees to tailor the documentation workflow ac-
     safely to the finish line. Like in our racing example, the        cording to the software artefact type. Additionally, the tool
     service “Frameworks” provides our DevOps teams with               offers interfaces to other documentation systems ensuring
     a tool-supported end-to-end documentation process                 that DevOps teams can navigate through the workflow
     that covers the entire IT product development work-               using one single tool. The PO-Cockpit allows users to tailor
     flow to ensure compliance with regulatory standards               the required documentation to their respective software
     while ensuring our DevOps teams can fully focus on                development endeavor using a standardized set of questi-
     developing the right software for our customers. This             ons to automatically generate the needed documentation
     added value is achieved by two interconnected solu-               template. Another feature is the embedded delta docu-
     tions: the Commerzbank Delivery Guide (CDG) that                  mentation function that allows to use existing documenta-
     contains a description and relevant documentation re-             tion templates of the completed projects if the respective
     quirements for each step in the software development              users want to document a change or new feature of the
     process and a Product Owner Cockpit (PO-Cockpit                   existing endeavor. Moreover, the PO-Cockpit ensures that
     CHAMP) that guides the Product Owner (PO 8) and                   the documentation contains all necessary approvals. For
     other involved colleagues through the development                 instance, the tool automatically sends the documentation
     workflow and necessary documentation according to                 template to the respective responsible person to be com-
     the CDG (as seen in figure 4).                                    pliant with the double check principle ensuring security
                                                                       standards. Additionally, the technical illustration of the
                                                                       CDG in the PO-Cockpit serves as a guiding tool throughout
13                                                                     8
                                                                           For a detailed explanation refer to “Glossary”, page 36.
all process steps of the software development workflow                       frameworks, and integrated development environments to
while facilitating the management of documentation re-                       developers to implement the user stories.
quirements through transparent navigation. DevOps teams
always have an overview of the next process step and how                     3. “Source Code Control” provides a central source code
they can find the right balance between accelerating the                     repository, where changes from all software developers
software development process and documentation.                              can be stored, merged, and reviewed. Each code change is
                                                                             tracked to ensure traceability.
•       CI/CD CHAMP Automated Pipeline: At the heart of CI/                  4. “Build” tools enable an automated build process with
        CD CHAMP is our automated pipeline, the racing car,                  each commit, such as change in source code repository.
        which turns power into motion, resulting in a signifi-               During the build process step, automated unit tests, func-
        cant acceleration of Commerzbank’s software delivery                 tional tests, and code analysis checks are executed ensu-
        process. The objective of the automated pipeline is to               ring that the code is of high quality before being integra-
        support DevOps teams in their CI/CD endeavors with                   ted.
        the necessary technical tools and consulting on how
        these tools can be applied to implement their envisio-               5. “Code Analysis” represents another quality gate for the
        ned products or features to enhance the experience of                code artefact created in “Build” comprising an automated
        our private and corporate customers. On the technical                checkup of the code for programming errors and adher-
        tool side, the automated pipeline offers different tools             ence to internal policies, e.g., programming guidelines. It is
        currently comprising of up to 15 different applications              used to detect outdated, insecure, or error-prone depen-
        with partially automated interfaces along the CI/CD                  dencies used in the software project. In addition, security
        software delivery workflow from the initial require-                 scans are conducted to detect vulnerabilities.
        ment to production.
                                                                             6. In the step “Delivery”, the code artefact is automatically
 As seen in figure 5, the technical tools support DevOps                     handed over to the artefact repository and archived secu-
teams to successfully navigate on the CI/CD CHAMP                            rely.
racetrack completing all development stages towards the
finish line of deploying the final product to production. The                The next steps on the CI/CD CHAMP racetrack are part of
first six stages comprise tools used to realize continuous                   continuous deployment:
integration:
                                                                             7. “Configuration Management” represents a solution for
1. “User Story Design” which provides POs, using agile soft-                 the configuration and management of the stage specific
ware development methods, such as Scrum or Kanban 9 ,                        configuration parameters.
with an automatically created tool space to document the
requirements for their product visions in the form of user                   8. “Deploy Test” provides tools to support DevOps teams
stories.                                                                     with test quality criteria (e.g., functionality) of artefacts in a
                                                                             target environment with a low number of manual tests and
2. “Coding”, provides standardized functions, libraries,                     a central overview of all test cases. These tools enable a
                                                                             full automation of the deployment process.
                                PO-Cockpit
                                Tool that supports the Product Owner and
                                other colleagues through the documentation
                                process based on regulatory requirements
9
    For a detailed explanation refer to “Glossary”, page 36.                                                                                14
                                                              CI/CD
 9. “Deploy PROD” provides the appropriate tools for an           teams using the CI/CD CHAMP pipeline are provided with
 automated release of the artefact in the target system.          reusable templates for all included tools that can be tailo-
  There are two conditions that needed to be considered for       red to the specific project needs leading to a simplification
 the design of the automated pipeline. First, the regulatory      of the whole software delivery process. Another area in
 requirement of “Segregation of Duties” (SoD), that requi-        which DevOps teams benefit from CI/CD CHAMP is the
 res that the person changing the code of an application          self-service aspect of the pipeline. Traditionally, DevOps
 cannot bring the same change live, effectively separating        teams that would like to use multiple CI/CD tools, would
 the functions of development and operations, influenced          have to talk to multiple internal teams that manage those
 the way the deployment process was automated. At Com-            tools leading to time lags in granting access and configura-
 merzbank, we established safeguard measures to ensure a          ting user rights. Therefore, at the core of the CI/CD CHAMP
 consisting SoD throughout the entire deployment process.         pipeline is a self-service platform where all required tools
 Second, additional complexity of deployments arose from          can be requested on demand, and DevOps teams can get
 the extent of middleware, programming languages, and             access to them without delay which increases the overall
 cloud technologies used at Commerzbank. To ensure CI/            satisfaction of DevOps teams that are more in control of
 CD CHAMP is as open and extensible as possible, the CI/          their software development environment.
 CD CHAMP Automated Pipeline can deploy to all of those
 target systems. Additionally, the user experience for de-        •    CI/CD CHAMP DevOps Coaching: Besides the techni-
 velopers and operators is the same, no matter what the                cal factors, the success of a racing team also depends
 underlying target is.                                                 on the skills and execution of the driver. A successful
 The automated pipeline offers several advantages to fur-              racing driver who leverages the full potential of his
 ther accelerate Commerzbank’s CI/CD activities. DevOps                or her car, needs an experienced pit crew as well as a
 15
    team of racing strategists which support him or her off      at further strengthen the DevOps mindset of agile teams.
    the racetrack on the pit lane. We think that also applies    The selection of the appropriate training and coaching sec-
    to our CI/CD pipeline since a successful application of      tions is based on the individual need of the respective agile
    all technical features requires the best possible metho-     team as well as existing challenges they face. Therefore,
    dological and cultural support of DevOps teams. The-         CI/CD CHAMP DevOps Coaching is highly individualized to
    refore, the CI/CD CHAMP service “DevOps Coaching”            support agile teams in the best possible way in their daily
    offers training and coaching for modern software             business. The DevOps teams benefit from this approach
    development methods to further establish software            in several ways. First, they can leverage new practices and
    craftsmanship within Commerzbank’s cross-functional          skills to further increase their software quality through a
    agile teams. The training and coaching solutions inclu-      shift left approach. Second, they are optimally prepared to
    de a two-stage procedure comprising “DevOps Techni-          use our CI/CD CHAMP pipeline faster and easier to imple-
    cal Excellence” to further strengthen agile techniques       ment their intended solutions and features.
    and practices within the DevOps teams and “DevOps
    Agile Empowerment” to support DevOps teams in
    applying and living the DevOps concept. Since the            “We believe that our CI/CD CHAMP pipeline offers a good
    successful execution of DevOps depends both on               practice of how organizations can holistically design their
    technical excellence and a collaborative and inclusive       CI/CD pipelines leveraging the CI/CD cornerstones of
    mindset, the focus of DevOps Coaching is on offering         DevOps Team, Software Development Process, and CI/CD
    training and coaching solutions comprising these two         Tools. Although, we successfully integrated CHAMP in our
    interrelated components as seen in figure 6.                 daily business, we are still working with high intensity to
                                                                 bring it to the next level,” says Florian Meiser (Commerz-
                                                                 bank AG). We are driven by the feedback of the DevOps
On the methodological side, the component DevOps                 teams and Commerzbank’s customers – because our
Technical Excellence provides trainings comprising state-        software is just as good as they think it is. Like teams in
of-the art methodological strategies (e.g., Value Stream         the Formula E are constantly searching for improvement,
Mapping), software development practices (e.g., Test             we are working rigidly to further innovate CI/CD CHAMP
Driven Development, Continuous Integration, Clean Code)          and its services. Please have a look at chapter 4 where we
and advanced testing techniques (e.g., Behavior Driven           share our new features and vision for CHAMP 2.0 with you.
Development, Integration Testing). On the other side, these
methodological trainings are complemented by DevOps
Agile Empowerment coaching in which a DevOps coach
accompanies DevOps teams beyond the respective trai-
ning sessions to support them in adopting the practices in
their daily business. Additionally, the coaching sessions aim
                    Component 1                                              Component 2
                  DevOps Technical                                           DevOps Agile
                     Excellence                                              Empowerment
             • Provide training for state-of-the-art                    • Support agile teams to adopt a
               practices related to technical agility                           DevOps mindset
            • Leverage DevOps methodologies to                       • Accompanying agile teams to further
              support agile teams in daily business                       strengthen a DevOps culture
                                                                                                                           16
CI/CD CHAMP Use Cases
At Commerzbank, we would like to give you a deeper             for restructuring the API deployment process to enable
insight into how CI/CD CHAMP and its three interrelated        process speed, scalability, and security. The resulting pro-
services support the teams of our Delivery Organization        cess artefacts were delivered to the API Banking cluster
in their daily business. Hence, we are proud to share with     that adjusted these artefacts accordingly. Although, the
you three of our CI/CD CHAMP use cases as depicted in          DevOps team from API Banking and CI/CD CHAMP had
figure 7 to explain the implementation of the different CI/    separate tasks to adjust the API deployment process, they
CD CHAMP solutions and how our CI/CD CHAMP team                both worked hand in hand. “The collaboration with the
collaborates with their internal customers. In addition, we    colleagues from CI/CD CHAMP was excellent during the
want to share our learnings and experiences on our CI/         whole process. Our weekly meetings and the possibility to
CD journey so far. Thus, we provide you at the end of the      always contact one of the colleagues highly contributed
chapter with our best practices on how to master the im-       to the success of our project,” says Volker Sulzbach (Com-
plementation of CI/CD in a large corporation. Further, we      merzbank AG).
are looking forward to opening up the discussion in the CI/
CD and DevOps community to also hear your learnings and
best practices.                                                During the implementation of the target API deployment
                                                               process the joint team also faced some challenges. For
                                                               instance, the tasks related to the adjustment of the API de-
CI/CD CHAMP Use Case 1: Commerzbank API Banking                ployment process were initially seen as additional effort for
                                                               the API Banking unit. However, the clear identification of
Commerzbank’s Application Programming Interfaces (API)         milestones and the incremental delivery of improvements
Banking cluster enables co-creation of valuable solutions      in the deployment process, continuously enabled the team
with business partners, customers and FinTechs. In essence     to speed up deployment, which in turn did free up time
APIs are technologies that enable communication between        for other tasks and ultimately lead to the mitigation of the
IT-systems resulting in a closer collaboration among diffe-    initial problem.
rent parties resulting in better solutions for our customers
[24]. The API Banking cluster is responsible for providing
a platform for managing and operating the APIs of Com-         Additionally, API Banking was using a test automation
merzbank and was in 2018 one of the first cases where CI/      tool, that was not included in the setting of Commerzbank
CD CHAMP was used to enable a faster software develop-         and hence not part of the standardized, automated CI/CD
ment process while further increasing the software code        CHAMP pipeline. Accordingly, even though the deploy-
quality, both factors being crucial for delivering APIs.       ment time could be reduced significantly, the high quality
                                                               of the releases needed to be ensured. To solve this issue,
CI/CD CHAMP supports the cluster API Banking by crea-          the CI/CD CHAMP team implemented a new interface
ting an automated pipeline in close collaboration, provi-      between the automated pipeline and the tool API Banking
ding the appropriate technical tools and consulting, as well   already used for integration tests and demonstrated that
as by offering DevOps coaching comprising methodologi-         the automated pipeline can be extended and individually
cal and mindset training components.                           configured based on the needs of the users. Through this
                                                               interface the automated CI/CD CHAMP pipeline triggers
                                                               test automation and reads result reports, ensuring only
CI/CD CHAMP Automated Pipeline                                 high-quality releases are deployed to the production en-
Initially the API deployment processes were carried out        vironment.
manually which led to challenges regarding a prompt
delivery of the API and scalability. Complexity arose from
the fact that a relatively small team had to deal with a       The cluster API Banking benefitted from the automated CI/
plethora of various APIs provided daily by all other soft-     CD CHAMP pipeline in different ways. “The automated API
ware teams within Commerzbank. Thus, the API Banking           deployment process is a success for our Cluster. Deploy-
unit partnered with the automated pipeline team of CI/CD       ing an API has become much easier and faster compared
CHAMP to further improve the API deployment process.           to the initial process,” says Christian Pfaff (Commerzbank
To identify potential for improvement, the API deployment      AG). As a result of adjusting the API deployment process,
process was analyzed in detail according to the methodo-       the time needed to deploy an API in production was re-
logy of Value Stream Mapping (VSM 10) . The design of the      duced to 15 to 20 minutes assuming a well-designed API.
target API deployment process allowed the team to derive       The greatest benefit from this optimization comes from
milestones that needed to be achieved to improve the           the frequent re-deployments that are necessary for the
initial process. The team of CI/CD CHAMP was responsible       usual minor changes. The high level of automation not only
17                                                             10
                                                                    For a detailed explanation refer to “Glossary”, page 36.
                                                                                                                                         Applied CI/CD CHAMP Services
CI/CD CHAMP service completely applied CI/CD CHAMP service almost completely applied CI/CD CHAMP service partly applied No CI/CD CHAMP service applied
provided scalability but also a high software quality since                                  ment session. “The full-time training sessions gave us the
the reduction of manual steps makes the whole setting less                                   opportunity to reflect and focus on specific areas of our
prone to errors. The new process also increased the usabi-                                   daily work with an outside-in perspective. As a result, we
lity of the API mid-tier application deployment process sin-                                 received valuable feedback about which additional agile
ce developers can automatically create new pipelines for                                     software development practices could support us,” says
API applications via a self-service tool. The team around                                    Akli Amar-Youcef (Commerzbank AG) since the first week
CI/CD CHAMP also took some key learnings for future pro-                                     of the program aims at educating the team on different
jects from working together with the API Banking cluster.                                    agile development practices and offering them the pos-
For instance, this experience contributed to the further                                     sibility to challenge their agile working approach. At the
standardization of the CI/CD pipeline tools ensuring the                                     end of the first week, the team identified two focus topics
flexibility to integrate additional tools if necessary. Addi-                                they wanted to intensify in the following full-time coaching:
tionally, the regular meetings with the API Banking team                                     Value Stream Mapping (VSM) as a more holistic practice
during the project contributed to the practice of conduc-                                    and Test-Driven Development (TDD) as an agile technical
ting weekly checkpoints and having a single overarching                                      practice related to the coding process. Both selected agile
product backlog for the whole project.                                                       practices were then applied to an existing user story du-
                                                                                             ring and after the two weeks full-time coaching session.
The CI/CD DevOps Coaching program started with an                                            Undoubtedly, the two weeks full-time training and coa-
analysis workshop assessing the current agile team ma-                                       ching sessions were an investment since it interrupted the
turity and the aspects the team wanted to improve. In a                                      normal working process. But for the API team it has proven
next step, the team took part in full-time training sessions                                 a fruitful investment enabling the team to perform even
covering relevant agile software development practices                                       better than before or as Arnd Bischoff (Commerzbank AG)
based on the results of the prior analysis and assess-                                       shared: “I really appreciated that the full-time training
                                                                                                                                                                              18
session was tailored to our specific interests and needs.       we needed,” says Michael Bellinger (Commerzbank AG).
The CI/CD CHAMP DevOps Coaching definitely further              Usually, the onboarding and the maintenance specialists
improved our team motivation and collaboration – and            of CI/CD CHAMP Automated Pipeline are working sepa-
it was a lot of fun as well.” The CI/CD CHAMP DevOps            rately from the developers of the respective customer
Coaching contributed to the agile working practices and         teams, however, the BDAA case required the expertise
collaboration of the API team. After the program the team       from both teams. Thus, there was one colleague from CI/
applied new agile practices in their daily software develop-    CD CHAMP Automated Pipeline as central contact person
ment, they recognized efficiency gains and productivity         for BDAA. When further expertise was needed additional
improvements as well as growing closer as a team. Addi-         experts were included. In general, the procedure of only
tionally, the CI/CD CHAMP DevOps Coaching team also             using a reasonably small team and adding expertise on a
gained valuable learnings trough this experience working        situational level has proven best practice to quickly identify
together with the team at API Banking. For instance, they       challenges and find the required solutions. Together the
learned that tailoring the training and coaching sessions       joint team onboarded the BDAA talkbot on the automated
to the individual team situation is crucial and thus the time   CI/CD CHAMP pipeline and already set the preconditions
for preparation and assessment was extended for future          necessary in the cloud environment to be able to deploy
internal customers.                                             on demand. Further, the BDAA team used the PO-Cock-
                                                                pit provided by CI/CD CHAMP Frameworks to document
                                                                their software development process. The PO-Cockpit
CI/CD CHAMP Use Case 2: Big Data & Advanced Analytics           guided the PO through the whole process and supported
                                                                the overall goal to build a process that does not demand
Within the cluster Big Data & Advanced Analytics (BDAA)         special knowledge. “We are now able to easily develop
new products are developed applying analytics to internal       and deploy new software products on demand thanks to
and external data. In 2020, the cluster started to develop      the support of our CI/CD CHAMP colleagues,” says Peggy
a talkbot, which is in essence a chatbot utilizing voice-ca-    Schuchert (Commerzbank AG). Additionally, the BDAA
pabilities to assist Commerzbank`s employees with their         team built a holistic partnership with the team from CI/CD
inquiries via the Commerzbank internal service hotline.         CHAMP DevOps Coaching to methodologically support the
The talkbot enables automated processing of our collea-         whole technical process and further strengthen their agile
gues’ service requests that otherwise needed to be carried      software development practices. This way, methodological
out manually by agents from the internal service center.        and mindset support was provided to BDAA to establish
Although the BDAA talkbot case is still ongoing, it de-         software craftsmanship and thereby facilitate the use of
monstrates the integration and tool flexibility of the CI/CD    CI/CD.
CHAMP Automated Pipeline.
19
Prior to their application of CI/CD and their onboarding on   After the team was onboarded on the CI/CD CHAMP Auto-
the automated CI/CD CHAMP pipeline, the team of CCB           mated Pipeline, they are now able to autonomously release
Securities Frontend was highly dependent on the techno-       new software solutions or features on demand due to the
logy stack of the CCB Online Banking cluster since their      high level of automation the CI/CD CHAMP Automated
Securities Frontend application was one of many appli-        Pipeline provides. Additionally, the team can detect, analy-
cations embedded in the CCB application. This led to a        ze, and correct defaults faster. “We are very satisfied with
high complexity within the deployment process and CCB         the tools and process provided by CI/CD CHAMP Auto-
Securities Frontend was only able to release once a quarter   mated Pipeline. The combination of automation, early
since deployment and test cycles need to be coordinated       testing and continuous support from the CI/CD CHAMP
among the whole CCB application. Thus, the CCB Securi-        colleagues makes the difference,” says Klaus Donath
ties Frontend team aimed to have shorter deployment cyc-      (Commerzbank AG).
les independently from the IT operation team of the CCB
main application leading to shorter feedback loops concer-
ning their new software features. CCB Securities Frontend     In summary, the CCB Securities Frontend use case demon-
also wanted to use cloud-based technologies and decided       strates the possibility of separating and onboarding an
to leverage a cloud-based Container Platform that was set     existing software application from the technology stack
up in Commerzbank at that time.                               of Commerzbank to CI/CD CHAMP. Additionally, this use
                                                              case is also an example of how teams use CI/CD CHAMP
                                                              as a self-service tool for developing new software applica-
The cluster aimed to separate its deployment process from     tions or features. Moreover, the CCB Securities Frontend
the CCB application to gain speed and flexibility and thus    use case served as blueprint how CCB applications might
got in touch with the CI/CD CHAMP team. After the team        be separated from the monolithic legacy technology stack,
of CCB Securities Frontend was onboarded on the CI/CD         thus provided the impetus for the practice of breaking
CHAMP pipeline, they both worked together to provide          down the technology stack into containerized applications.
the CCB Securities Frontend, with capabilities to deploy      This case also provided important learnings for the CI/
software to the cloud-based container platform. As with       CD CHAMP team. For instance, the onboarding process to
previous use cases, CI/CD CHAMP was extended to allow         the pipeline was further improved and is now conducted
for this. The CCB Securities Frontend team also applied       automatically enabling DevOps teams from the delivery or-
the PO-Cockpit provided by CI/CD CHAMP Frameworks.            ganization to directly create new projects on the pipeline.
The PO-Cockpit allowed the team to easily structure the       Moreover, this feedback of the DevOps teams concerning
documentation process and centrally coordinate the single     the PO-Cockpit was used to further enhance the tailoring
process steps during the software development process.        function of the tool as well as the possibility to build on
As Claus Weber (Commerzbank AG). notes “The PO-               existing documentation in the case of a software change.
Cockpit supports the documentation process by offering
a simple tailoring to the required documentation. I really
appreciate that it guides product owners throughout the       Commerzbank’s Best Practices – How to Master CI/CD
documentation process in one single application.”             In the last four years, we learned that a one-size-fits all
                                                              approach to successfully apply and live CI/CD in the orga-
                                                              nization does not exist. Instead, success is highly depen-
                                                                                                                       20
dent on the interplay between different factors and how          delivery is the ability to provide infrastructure and deploy
the technical and mindset components of this paradigm            code dynamically.
are responsive to demanding business needs. Considering
our analogy of the Formula E, success on the racetrack
depends on the optimal adaption of the racing car to the         3. Focusing on measuring and visualizing
driving conditions. We as Commerzbank would like to              The basis of any continuous improvement should be facts
share with you the four most important best practices from       and figures. To obtain these, it is necessary to monitor the
our CI/CD experience that might also support you in your         complete value stream during a CI/CD cycle. The automa-
next CI/CD endeavors.                                            ted CI/CD CHAMP pipeline allows for easy measuring and
                                                                 tracking relevant Key Performance Indicators (KPI) for
                                                                 each step of the software development process. The gat-
1. Following a self-service approach as a guiding principle      hered data is then visualized in a dashboard to guarantee
The first principle we followed when we designed and             full transparency about the efficiency of the process and
developed the automated CI/CD CHAMP pipeline further is          indicate areas for optimization. The dashboard contains
the principle of self-service that enhances scalability. The     all the relevant DevOps KPI, such as deployment frequen-
pipeline should be easily accessible and usable for every        cy, deployment speed or deployment failure. This allows
DevOps team member. For that reason, we created a por-           DevOps teams to improve in the next iteration. Besides
tal as the gate to our pipeline where a new team member          that, at Commerzbank, we continuously observe the usage
can order and get access to all the tools within one day.        of the automated CI/CD CHAMP Pipeline so that we can
We further complement the self-service function by user          inspect and improve certain building blocks or services of
friendly documentation for each of the tools, templates (of-     CI/CD CHAMP.
ten provided as code in Git Repository) which can be used
for a quick start and different samples in a playground to
intensify learning and collaboration. “Building the sur-         4. Applying Shift Left
rounding fostering these components requires a strategy          Our experience shows that DevOps teams that start early
and roadmap that articulates clear goals and measures of         with using the automated CI/CD CHAMP pipeline gain
how employees can be encouraged to collaborate,” says            more speed, efficiency, and confidence in their work. They
Dr. Lars Friedrich (Commerzbank AG). Another important           can react early and quickly to quality issues and fix them to
aspect to consider is end-to-end responsibility to emp-          finally improve their software artefacts without compromi-
ower DevOps teams and allow for team autonomy. DevOps            sing an accelerated time-to-market. Therefore, the CI/CD
teams need flexibility to define their own workflows and         CHAMP team consult, coach and support DevOps teams
to configure their own set of rules in certain tools without     in the onboarding process on the pipeline in all environ-
risking noncompliance. The CI/CD CHAMP vision is to ac-          ments so that they can start early realizing their ideas and
celerate the software delivery with a standard pipeline able     delivering faster high-quality software. At Commerzbank,
to fulfill all requirements that teams could have while deve-    we advise to shift left as much as possible, start to automa-
loping and delivering software. Although, diversity among        te tests and think in iterations which will result in delivering
team members and autonomy within DevOps teams paired             small pieces of software more frequently.
with a standard pipeline seems contradictory, it is import-
ant to find the right balance between flexibility and control.
Therefore, the automated CI/CD CHAMP pipeline incor-             5. Continuously improving through collaboration with
porates all necessary governance rules and guidelines of         CHAMP users
Commerzbank to ensure compliance and at the same time            CI/CD CHAMP is a technology foundation that provides its
provide teams the possibility to implement their own rules       services to all DevOps teams in Commerzbank’s Delivery
and templates without losing speed and autonomy.                 Organization. From that perspective, CI/CD CHAMP users
                                                                 are the ones who influence the further development of the
                                                                 pipeline. Therefore, a potential extension of the pipeline
2. Aiming for an expandable, flexible, and integrated            is always done in cooperation with the users. They know
pipeline                                                         best what they need to have a better user experience
A new feature in the hands of a customer always starts           and decide which new features are essential. Besides the
with an idea. Therefore, we have designed CI/CD CHAMP            importance of gaining valuable impulses from users, the
as a highly integrated pipeline to enable full automation        CI/CD CHAMP team itself drives innovation to create new
in the software development process starting with a user         technologies and possibilities which are subsequently stan-
story and completing the cycle with an automated de-             dardized for the users. CI/CD CHAMP is open to incorpo-
ployment into production. All the tools in the pipeline are      rate new innovative ideas coming from inside and outside
connected via APIs or have pre-configured plug-ins which         Commerzbank. The generation of internal ideas is ensured
build direct interfaces between them. New components             by internal cooperation, while external ideas are created
and tools can be easily added to the existing pipeline           through interactions and exchanges with vendors, different
based on the needs of the respective DevOps team. At             proof of concepts and conference participations.
Commerzbank, we believe an expandable pipeline is an ap-
propriate approach to reach different target environments
in a broader range from mainframe to cloud. A standard           The three exemplary use cases highlight how Commerz-
pipeline should be technology stack agnostic and at the          bank leverages CI/CD capabilities to be a strong partner
same time provide the possibility to treat infrastructure or     for customers delivering valuable solutions that meet
code in the same way as application code. To successful-         highest quality standards. We hope that these use cases
ly live DevOps it is crucial to provide the same tools and       contributed to a better understanding of CI/CD CHAMP
practices for the functions of development and operations.       and that our best practices might benefit you in your next
The automated CI/CD CHAMP pipeline is regarded as the            CI/CD endeavor. In the next chapter we want to share our
common path application when software artefacts are              expectations about the evolution of agile software de-
built, tested, quality checked and deployed. Beside scala-       velopment.
bility another important leverage for accelerating software
21
Hypotheses About the
Future of Agile Software
Development until 2026
When we think of where the future of agile software                      due to the convergence of three interrelated factors [27]
development is heading to, we need to understand the                     [28]. First, an increasing number of tool vendors take an
dynamics that might shape its evolution. Now that DevOps                 EaC-first approach to tool configuration and deployment
practices have achieved widespread adoption in agile                     by offering organizations the possibility to manage everyt-
software development [18], future developments that go                   hing using code files [28]. Second, the ongoing standard-
beyond bringing Dev and Ops closer together do exist.                    ization of configuration formats also supported the EaC
So, the question is: in which direction will agile software              trend since a common format makes it easier for develo-
development evolve within the next five years and how will               pers to manage all of their tools with the same language
new technologies affect it? Together with 12 experts from                and methodology (Tozzi 2020). Third, the possibility to
different industries we discussed trends and predictions to              build and manage an entire tool chain on a CI/CD pipe-
weigh in on what to expect in the upcoming five years. As                line, applying EaC drives the overall EaC approach also for
a result, we formulated five hypotheses about the future of              other tools and processes [28]. Thus, EaC offers not only
agile software development until 2026 as depicted in figure              a highly repeatable and scalable approach to tasks, but
8. These hypotheses center on a few key themes: as orga-                 also increases consistency across IT systems and processes
nizations accelerate on their agile transformation journey,              [27].
focusing on developing agile capabilities and embracing
agile values will be the imperative; leveraging technologies
will continue to offer new possibilities for automation and              The reason why EaC will become a priority is mainly based
ultimately provide the basis for improved collaboration; and             on its various potential benefits. For instance, EaC will
organizations will further focus on an Everything as Code                ensure transparency and a clear understanding of the
approach to create a common language for the whole                       infrastructure, configurations and policies without the
organization.                                                            need to rely on manually updated documentation as the
                                                                         code does evolve to a single point of truth incorporating
                                                                         all information. EaC will provide DevOps teams with the
01 Everything as Code (EaC) will become a priority lea-                  technical foundation for their daily collaboration, helping
ding to the practice that everything will be adjusted and                them to organize their work more efficiently [28]. Moreo-
managed using code                                                       ver, Compliance as Code (CaC) and Security as Code (SaC)
An EaC approach treats all components of a system such                   ensure that compliance and security requirements and
as documentation, application code, configuration ma-                    checks are built as code into the software development
nagement, infrastructure, and compliance as code so that                 life cycle from the first stages of the process. This creates
everything adheres to the same software development                      a common standard for all DevOps teams and guarantees
practices [25]. As focusing on the concrete CI/CD use case,              that compliance and security issues are automatically
the idea of EaC will make its entry across each layer of the             detected and corrected, in near real time [29]. Additionally,
CI/CD pipeline so that infrastructure, schemas, pipelines,               if compliance or security is written into software applicati-
and operations are all described and treated like applica-               ons, they can also be continuously updated in the event of
tion code [26]. This means that automation via software                  changing regulations without the need to manually adapt
will be everywhere providing the technical foundation for                applications to new regulations. EaC also offers the possi-
an array of other developments such as enhanced cross-                   bility to establish documentation as a continuous automa-
functional collaboration, or a shift left approach of security           ted process output throughout the whole CI/CD process.
and compliance that can be expected to occur in the next                 In future, developers will be supported with documentation
five years.                                                              that is automatically generated via a set of templates that
                                                                         collect and transform the required information from its
                                                                         origin to the target documents for a simultaneous delivery
At its core, EaC is not at all a new trend. Some isolated                of the new code and documentation [30]. This automated
forms of EaC have already gained popularity within the last              shift left of documentation will largely guarantee the accu-
years, Infrastructure as Code (IaC), Pipeline as Code (PaC)              racy and integrity of documentation and will significantly
and GitOps 11 being the most prominent examples and it                   increase the delivery speed of the documentation process
can be expected that this development will pick up steam                 [30]. Moreover, IaC offers organizations the possibility to
11
     GitOps refers to the approach to use Git to manage IT Operations.                                                             22
     01 Everything as Code (EaC) will become a priority leading to the practice
        that everything will be adjusted and managed using code
Figure 8: Hypotheses About the Future of Agile Software Development until 2026
deploy infrastructure whenever it’s required and to integra-             need to ensure the appropriate working conditions so that
te it into a CI/CD pipeline to dynamically build and destroy             experts from the business, security, development, and ope-
different environments as the pipeline executes. Finally,                ration side should work together to improve processes and
EaC ensures consistency in migrations, deployments, and                  develop software that can hit the market early.
configurations. Deployments in a specific environment will
result in the same configuration as if the same deployment
was conducted in another environment. Additionally, if                   02 “BizDevSecOps” will be the evolution of DevOps ensu-
organizations follow the practice of EaC, the efficiency and             ring a holistic cross-functional team approach in deliver-
performance of an organization to react to customer needs                ing customer value
will be measured and tracked by CI/CD value streams.                     Building on the foundation of DevOps, BizDevSecOps
                                                                         incorporates business and security experts into the soft-
                                                                         ware development process from the early beginning,
While this trend offers many potential benefits, it is im-               ensuring the targeting of key business outcomes through
portant to consider that the pace of this development is                 the DevOps lifecycle. “Bridging the functions of develop-
highly dependent on the flexibility of vendors concerning                ment and operations is not enough, organizations should
code and installation playbooks as well as the existence of              rethink the traditional DevOps cross-functional team ap-
tools to verify the code files to avoid potential pitfalls. In           proach a bit further and follow a holistic concept to deli-
addition, organizations will need to create the necessary                ver customer value,” says Daniel Zwicker (Authada GmbH).
conditions to take full advantage of EaC. As Daniel Meixner              The active involvement of business and security experts
(Microsoft AG) put it, “EaC will certainly become an im-                 in traditional DevOps teams builds the foundation for a
portant trend. However, organizations will need to meet                  collaborative culture which enables a new model of opera-
a number of prerequisites to be prepared to take full                    ting to provide superior customer experience by offering
advantage of EaC. For instance, appropriate tooling will                 customer-centric products and services through a CI/CD
be necessary and access rights sensitive areas will have                 pipeline while being compliant with security standards
to be installed and managed.” Additionally, EaC requires                 [1]. Although, many organizations have already started
a mindset shift so that all team members irrespective of                 to stronger integrate the business (BizDevOps) or secu-
their function create a deep understanding for each other’s              rity functions (DevSecOps) into their DevOps teams, this
activities and tasks. For example, team members of opera-                development will further accelerate since organizations will
tions should adopt a development mentality and automate                  increasingly adopt a product-thinking approach to embark
whatever and wherever possible. Moreover, organizations                  on the path of accelerating their customer-driven digital
23
transformation [31]. Hence, cross-functional teams will be        vironment to operations, operations can use observability
the decisive factor for organizations to develop software         tools on the CI/CD platform to determine impacts across
more quickly, more secure and to release features that are        different software releases. Subsequently, they can auto-
built specifically to service business objectives, and to be      matically share this information with the developers who
more responsive to user demand [31].                              can modify their builds accordingly and with the security
                                                                  specialist who endure that the change complies with secu-
                                                                  rity standards.
The ”Sec” component of BizDevSecOps teams will ensure
that security is built from the beginning into the software
development lifecycle ranging from prioritizing security re-      03 Organizations will leverage Machine Learning (ML)
quirements as part of the product backlog over continuous         algorithms to automate a wide range of processes in the
testing software code for vulnerabilities to comprehensive        DevOps lifecycle
monitoring in production [32][33]. This extends the origi-        ML will potentially impact current approaches and process
nal DevOps objective of delivering software features with         steps in the DevOps lifecycle. Considering the develop-
high velocity by early detection and mitigation of vulnera-       ment part of the DevOps lifecycle, ML provides potential to
bilities [34]. “Security will be an integral part in all phases   operate and optimize CI/CD pipelines. First, organizations
of the software development by rethinking DevOps to Biz-          will apply a combination of EaC and ML to establish self-
DevSecOps to ensure a security shift left approach,” says         optimizing CI/CD pipelines since ML algorithms enable an
Doron Reuter (ING Groep N.V.). From a technical perspec-          automated observation of the pipeline operation, continu-
tive, organizations will establish security as a mandatory        ously collecting data. “Leveraging ML algorithms in CI/CD
design requirement for their CI/CD pipelines applying the         pipelines requires an EaC approach, thus organizations
EaC practice to reflect security best practices in their IaC      need to follow this approach first to take full advantage
declarative scripts.                                              of the integration of ML algorithms,” says Mirco Leimgru-
                                                                  ber (Swisscom AG). Organizations integrating ML in their
                                                                  CI/CD pipelines will be directly notified if anomalies exist
The “Biz” component of BizDevSecOps teams will ensure             in the process or tools included. Moreover, ML algorithms
the active involvement of business experts in traditional         can predict potential defects in code generated CI/CD
DevOps teams. This builds the foundation for a collabo-           pipelines and subsequently react by triggering scripts for
rative culture which enables a new model of operating to          destroying and provisioning certain parts of the pipeline.
provide superior customer experience by offering valuable         Second, ML algorithms contribute to a stable, resilient
products and services through a CI/CD pipeline [1]. “Inclu-       operation of CI/CD pipelines by analyzing monitored data
ding the business perspective in software development is          that are produced from different events to predict or avoid
nothing new, it is how the agile movement started. Orga-          outages.
nizations need to integrate the customer perspective from
the early beginning and deliver their software features
fast to collect customer feedback,” states Rahul Verma            Additionally, within the CI/CD process, the application
(Trendig Technology Services GmbH). Integrating business          of ML offers various advantages from decision-making
stakeholders into the value stream allows organizations to        process improvements to automated operations and code
ensure that the DevOps cycle is aligned with business ob-         quality enhancements. Although, this trend is in its ad-
jectives and in turn allows them to execute their transfor-       aption still in its infancy due to challenges in preparing
mation objectives faster and more efficiently [31]. Whereas       large data sets and a challenging regulatory environment
business professionals will enable a stronger integration         especially in the banking industry, there exist four promi-
of the voice of the customer and allow organizations to           sing developments to watch within the next five years. The
see the impact of new products on business value more             first important trend is the application of ML for automated
directly [1][31], IT professionals ensure further automation      code reviews. The result will be the automated and early
and reliable software. Additionally, security experts make        detection of code flaws, security issues, and code-related
sure the software development process and new feature             defects which will lead to an enforcement of code and
comply with security policies [1]. Neither is more important      security standards [35]. “The integration of ML into the
than the other, but they all need to work together in order       DevOps lifecycle offers the possibility of carrying out
to provide software that meets customer requirements,             automated application security vulnerability checks that
is compliant with security standards and is delivered with        will lead to both real-time security risk assessments and
high quality and speed.                                           an increased efficiency due to the reduction of time nee-
                                                                  ded to conduct those tests,” says Raz Raviv (ING Groep
                                                                  N.V.). Another promising ML development comprises the
At the core of BizDevSecOps is the common language of             prediction of potential issues based on data. ML models
code that enables collaboration and co-innovation among           have the ability to detect patterns and predict signs of
the different functions. This common language across              failure in cases in which humans hardly perceive them [36].
the whole software development lifecycle ensures that             Such early predictions support DevOps teams to identity
everyone is working towards the same outcomes. Mo-                and fix issues before they have a negative impact on their
reover, it aligns teams into a coordinated BizDevSecOps           software artefact. The next trend to monitor in leveraging
practice, where all functions have better insight into how        the possibilities of ML within the DevOps lifecycle is the
each activity contributes to the overall objective. In this       possibility of a faster root cause analysis. ML can recognize
context, EaC provides the technical conditions to integra-        patterns between implication and activity to determine the
te all IT systems into one common CI/CD platform, such            root cause behind a defect [36]. Organizations can esta-
that BizDevOps teams can use the same tools in their daily        blish an automated process to fix defects permanently by
business. In this context, organizations will have one source     conducting ML-based root cause analysis that can reduce
of truth for data and tracking points that can be establis-       the risk of human error while providing the development
hed to measure the speed or waste in certain phases of the        team with recommendations for writing more efficient and
software development lifecycle. For instance, if business         performant code [37]. The last important ML trend in the
experts give feedback on an incident in the production en-        DevOps environment is the adaption of Robotic Process
                                                                                                                            24
Automation (RPA) for test automation [36]. RPA can be           If organizations follow an IaC approach enabling a quick
utilized to transform manual, time-consuming and error-         reproduction, destruction, and provision of infrastructu-
prone tests into automated and streamlined ones. General-       re, end-to-end autonomous testing can be dynamically
ly, if ML tools are integrated in the DevOps tool stack, they   conducted to save resources and time. Additionally, the
offer the opportunity to free up DevOps teams from low-         utilization of cloud-based services will make testing faster,
value tasks that potentially constrain productivity while at    cheaper, and more efficient since it will allow organizations
the same time improving software quality and application        to leverage an almost infinite resource pool and flexible
security [38]. However, implementing the above-men-             infrastructure capacities [42].
tioned ML trends and realizing its benefits substantially
depends on the regulatory environment and fulfillment of
the data set conditions by organizations. Thus, it remains      In the upcoming years, the application possibilities of
to be seen whether a wide adoption of ML in the DevOps          cloud-based and autonomous end-to-end-testing solutions
lifecycle will become standard in five years.                   will be complemented by production-related test data. This
                                                                will be enabled by the anonymous extraction of test data
                                                                from production to ensure test cases reflect real world
Considering the operations part of the DevOps lifecycle,        scenarios. This development will lift the quality of the soft-
ML will also have a key role in ensuring operations continui-   ware delivered on an even higher level since the probability
ty after the product or feature is released into production.    of a case that has not been tested before will decrease.
The pattern-based functionality of ML algorithms provi-         Nevertheless, it is important that organizations ensure that
des the possibility to anonymously analyze user metrics         anonymous extracted test data is always compliant with
and the functioning of applications in production. In the       the law in force (e.g., General Data Protection Regulation).
case of any anomalies, ML tools will alert DevOps teams         Hence, using real-life data without the knowledge and
in the case of any issue including possible solutions about     assent of the customer is forbidden and does complica-
how to solve this issue. Although, these predictions sound      te generating test data. Concerning the financial service
very promising, organizations such as banks operating in        industry, the realization of this hypothesis depends on the
a highly regulated environment might face obstacles in          development of future regulatory conditions since the cur-
applying ML algorithms to analyze and interpret customer        rent regulation forbids tests based on real-life data [43].
data from production.
25
26
minique Mühlbauer (Trendig Technology Services GmbH)             agile” approach. A value stream analysis tool might sup-
said, “this can be seen as the crucial hypothesis underly-       port this monitoring and evaluation process by measuring
ing the other hypotheses since, having motivated indivi-         different tracking points on the value stream of a CI/CD
duals, taking pride in their craftsmanship, working toge-        pipeline. This allows DevOps teams to have an overview of
ther in cross-functional teams, will enable delivering high      all data in one integrated dashboard that indicates areas
quality software on high speed.” Therefore, organizations        for improving efficiency in the overall software develop-
will have to focus on the development of agile capabilities      ment lifecycle.
such as sensing, learning, adaptability, resilience, speed,
innovation, collaboration, and efficiency on different or-       These hypotheses about the future of agile software
ganizational dimensions in future [5]. This approach will        development illustrate in the direction CI/CD and DevOps
lead to the holistic alignment of organizational strategy,       might be heading to. “At Commerzbank, we believe that
structure, processes, technology, and leadership to these        the underlying trends and dynamics of these concepts
agile capabilities. For instance, as agile transformations are   should be considered by organizations to further drive
grounded in cross-functional teams empowered to operate          their agile transformation,” says Michael Varona (Com-
in rapid decision-making and learning cycles, organizations      merzbank AG).
must ensure that the organizational structure and proces-
ses are designed around this purpose [45]. Additionally,         Commerzbank’s Vision for CI/CD CHAMP 2.0
organizations will need to invest disproportionally in the
upskilling or reskilling of their employees to enable them to
work in new roles, collaboration setups, and new fields of       As in Formula E, teams are constantly searching for impro-
responsibility.                                                  vement, and at Commerzbank we are working with high
                                                                 intensity to bring CI/CD CHAMP to the next level. Thus, we
                                                                 questioned ourselves: how should we anticipate the expec-
Besides focusing on the development of agile capabilities,       ted developments to support the fast delivery of high-qua-
organizations will also follow agile values that are reflected   lity software that meets Commerzbank-customers’ expec-
in behaviors as well as mindsets, and practices drive the        tations? And which tools, methods or processes support
required cultural shift sustainably. Therefore, organizations    the projects in the best way to deliver value to Commerz-
will increasingly focus on empowering their employees in         bank’s customers?
their way of thinking comprising attitudes, orientations,
and mentality patterns towards living agile values and prin-
ciples [46]. One crucial component of this is an empowe-         Driven by these questions we envisioned CI/CD CHAMP 2.0
ring leadership approach that embraces an open communi-          that is based on three interrelated developments. First, the
cation, a culture of constructive criticism and collaboration.   CI/CD CHAMP Automated Pipeline 2.0 will contain a mix of
This requires leaders to cultivate an environment of psy-        self-developed and commercially available tool solutions
chological safety in which employees are encouraged to           that follow an IaC approach. Second, CI/CD CHAMP 2.0
experiment and learn without fear of negative consequen-         will be cloud-native to provide maximum scalability and
ces to self-image, status, or career [47]. “The enterprise       flexibility for its users. This will be complemented by reu-
level of agility can often be defined based on the existen-      sable templates for all the CI/CD CHAMP 2.0 tools that can
ce of psychological safety in the teams since “being agile”      be tailored for the software project‘s specific needs and
requires a high degree of psychological safety. Thus, es-        that would enable DevOps teams to build and deploy new
tablishing an environment in which every employee feels          software application versions within minutes. Third, CI/CD
safe to contribute and speak up is a decisive factor for the     CHAMP 2.0 aims to be a pioneer in leveraging EaC-
overall success of the company,” says Schlomo Schapiro           approach and implementing it in context of a CI/CD pipe-
(DB Systel GmbH). This psychological empowerment will            line.
shape employees‘ consciousness, mindset, and behavior
that go beyond simply following agile methodologies or
frameworks. Another important component of establis-
hing an agile value system are coaching activities that can
support the required mindset shift of employees. Coaches
could stimulate thought within agile teams, help them to
reflect on their agile way of working, collaboration and
attitudes to find further areas for improvement [48].
27
Summary and Conclusion
Software changes the world. Going beyond agile software         business within traditional DevOps teams. BizDevSecOps
development, CI/CD and DevOps have pioneered the way            teams will reinforce the collaborative approach of software
software development is thought, conducted, and lived           development enabled by CI/CD. We as Commerzbank truly
and thus have become essential approaches for many              believe BizDevOps is here to stay, representing a roadmap
organizations in their software endeavors. In this context,     for companies to stay ahead of their agile transformation
the concept of CI/CD supports organizations in accelera-        curve and realize customer-driven business success. Howe-
ting delivery speed, quality, and customer centricity. Most     ver, organizations that want to leverage the full potential of
importantly, the interrelation of the CI/CD cornerstones        these developments need to fulfill specific prerequisites to
that comprise people, processes and tools ultimately drive      be ready to anticipate the respective changes. For instan-
automation, co-innovation, and collaboration within the         ce, organizations need to establish a roadmap that com-
whole organization to ultimately create customer value.         municates the long-term vision while providing flexibility
                                                                for their operating teams to quickly re-prioritize activities
                                                                when relevant trends emerge that contribute to this vision.
In this white paper we discussed -besides how CI/CD af-         Being able to anticipate trends also requires the presence
fects value creation in general and in the financial services   of a highly automated and well-functioning CI/CD pipeline
industry in particular- also how this concept could be          that provides consistency and reliability. Additionally, orga-
brought to life by providing insights on Commerzbank’s          nizations with a monolithic IT legacy need to be open for a
CI/CD approach. At Commerzbank, we believe that there           certain degree to architectural change to establish flexible
is no one-size-fits-all approach when it comes to imple-        and adaptive systems.
menting CI/CD since every organization has its individual
requirements and faces different situational characteristics.
However, the implementation of a CI/CD pipeline should          The discussed conclusions of this white paper are centered
always follow a set of technical principles comprising the      around five key take-aways that are illustrated in figure 9.
setting up of end-to-end responsibility for teams, reducing
risk throughout the software development lifecycle, and
integrating short feedback loops.                               We are convinced that acceleration will be the new norm
                                                                for creating customer value in agile software development
                                                                and we are excited to contribute with our experience and
The true value of CI/CD comes from the enablement of            best practices to this development. We invite you to also
teams to own the whole software development lifecyc-            share with us insights you gained on your CI/CD journey
le from the initial product idea to the release of the final    so far and to discuss where the future of agile software
software feature. Additionally, CI/CD as a cross-functio-       development is heading to. We are looking forward to
nal concept leverages on the joint strength of different        hearing from you and exchanging our ideas. What do you
functions and thus ensures the strong integration of crucial    think about beyond banking in CI/CD and the next wave of
aspects such as security, compliance, and the voice of          innovation in agile software development?
the customer within the software development process.
“With even more customer interaction becoming digital
the concept of CI/CD is a key differentiator for us in the
market. It offers unprecedented opportunities to develop,
test and deploy software fast and with high quality for our
customers,” says Dr. Carsten Bittner (Commerzbank AG).
In this context, the Commerzbank CI/CD CHAMP journey
shows that CI/CD as an organization-wide approach to
agile software development enables teams to release soft-
ware features faster and more independently. Employing
the metaphor of Formula E, teams being equipped with the
necessary safety equipment (CI/CD CHAMP Frameworks),
provided with the appropriate racing car (CI/CD CHAMP
Automated Pipeline) and having an experienced pit crew
(CI/CD CHAMP DevOps Coaching) can autonomously drive
innovation and value creation within the organization.
                                                                                                                           28
29
Appendix
Methodological Approach
                                                                30
Our External Interview Partners
We conducted 9 interviews with the following representatives from a diverse set of companies. We hereby would like to
express our gratitude towards our interview partners for spending their valuable time as well as sharing their insightful
ideas with us. We appreciate the open conversations and the challenging discussions.
Circle Internet Services Inc. (“CircleCI”) Nicholas Mills General Manager EMEA
Trendig Technology Services GmbH Dominique Mühlbauer Quality Advocate & Agile Evangelist
31
Our Commerzbank Sponsors and
Use Case Partners
We would also like to warmly gratitude our internal sponsors for their support during the preparation phase of this white
paper. Moreover, we also want to thank our colleagues from the API Banking Cluster, Big Data & Advanced Analytics and
Securities Frontend for spending time and sharing their valuable feedback on how CI/CD CHAMP supports them in their
daily business.
Sponsor Function
Sabine Vigelius					
				                                              Key Area Lead of Customer Process & Data Management
Securities Frontend
                                                                                                                        32
About the Authors
Commerzbank Cluster CI/CD CHAMP
The cluster CI/CD CHAMP provides a reliable end-to-end solution consisting of a fully automated self-service pipeline,
modern software development and test methods, corresponding tools and DevOps coaching. The objective is to enable all
clusters of Commerzbank delivery organisation to deliver regulatory compliant software fast and efficient to customers in
high quality and with short release cycles.
Together with our research partner the Business Engineering Institute St. Gallen, we published this White Paper.
                                                                      Florian Meiser
                                                                      Cluster Lead CI/CD CHAMP
                                                                      Commerzbank AG
                                                                      Florian.Meiser@commerzbank.com
                             Damian Waszak
                             Product Manager CI/CD CHAMP
                             Commerzbank AG
                             Damian.Waszak@commerzbank.com
                                                                      Fonlinda Frasheri
                                                                      System Architect CI/CD CHAMP
                                                                      Commerzbank AG
                                                                      Fonlinda.Frasheri@commerzbank.com
                             Jan Willemsen
                             Senior Product Owner CI/CD CHAMP
                             Commerzbank AG
                             Jan.Willemsen@commerzbank.com
                                                                      Stephanie Radet
                                                                      Product Owner CI/CD CHAMP
                                                                      Commerzbank AG
                                                                      Stephanie.Radet@commerzbank.com
                             Rostam Rahnema
                             Senior Product Owner CI/CD CHAMP
                             Commerzbank AG
                             Rostam.Rahnema@commerzbank.com
33
Business Engineering Institute St. Gallen
The Business Engineering Institute St. Gallen is a Swiss institute that conducts research in the areas of ecosystems, digital
transformation, and disruptive technologies to develop academic insights with practical impact. We collaborate with insti-
tutions such as the University of St. Gallen, the University of Göttingen and partner companies from the financial services
sector to develop answers to strategic questions related to technology and information-based value creation in the digital
age.
                                                 Katharina Schache
                                                 Research Associate & PhD Candidate
                                                 Business Engineering
                                                 Institute St. Gallen AG
                                                 Katharina.Schache@bei-sg.ch
                                                 Oliver Kutsch
                                                 Head of Consulting
                                                 Business Engineering Institute St. Gallen AG
                                                 Oliver.Kutsch@bei-sg.ch
                                                 Dennis Vetterling
                                                 Research Associate & PhD Candidate
                                                 Business Engineering Institute St. Gallen AG
                                                 Dennis.Vetterling@bei-sg.ch
Acknowledgements
While working on this white paper, we have had numerous discussions and received lots of valuable feedback from within
and outside of Commerzbank. We would like to express our sincere gratitude towards all those who supported us on this
journey. We would like to thank Christian Betz and Marc Burkhalter constructive feedback and their valuable ideas. We
would also like to thank Nigel Champion for his linguistic revision.
                                                                                                                           34
Glossary
Cross Channel Banking (CCB)                        CCB is the main Commerzbank portal, comprising of core
                                                   portal solution and multiple financial products across diffe-
                                                   rent channels.
Continuous Integration and Continuous Deployment   CI/CD forms the operational backbone of DevOps practi-
(CI/CD)                                            ces and essentially allows organizations to provide users
                                                   with new software products or features at any time in a
                                                   sustainable way.
Commerzbank Delivery Guide CHAMP (CDG CHAMP)       Commerzbank Delivery Guide CHAMP helps developers to
                                                   be guided in the development process containing all rele-
                                                   vant documentation requirements.
Commerzbank Hyper Acceleration Master Pipeline     Commerzbank Hyper Acceleration Master Pipeline descri-
(CHAMP)                                            bes the CI/CD Pipeline of Commerzbank and incorporated
                                                   services.
Everything as Code (EaC)                           EaC is the idea that all functionalities of a software de-
                                                   velopment pipeline can be managed in using code.
Integrated Development Environment (IDE)           IDEs are applications that support developers in develo-
                                                   ping software in offering a compilation of different tools
                                                   that are needed during programming.
35
Kanban                                           Framework for establishing Agile and DevOps in software
                                                 development. The different tasks are visualized on a Kan-
                                                 ban Board to enable the team members to gain an over-
                                                 view anytime [49]
Product Owner (PO)                               The Product Owner (PO) is a member of the agile team
                                                 who is responsible for defining user stories and prioritizing
                                                 the backlog. POs streamline the execution of project priori-
                                                 ties and ensures the representation of customer interests in
                                                 the software development process [50]
Product Owner Cockpit CHAMP (PO-Cockpit CHAMP)   The PO-Cockpit CHAMP is a tool that guides the PO`s
                                                 through the development workflow
Shift Left                                       Shift left is the practice of shifting performing unit or inte-
                                                 gration tests and code quality checks early in the software
                                                 development process and can result in significant cost
                                                 reductions as well as in increased software quality since
                                                 potential pitfalls in the software code are identified early
                                                 during the development process [12]
Value Stream Mapping (VSM)                       Value Stream Mapping is a methodology for analyzing the
                                                 steps and their sequence of a current process easily quan-
                                                 tifying the time and working effort needed at each process
                                                 step. This allows for designing a target process that mini-
                                                 mizes the process time while maximizing the efficient use
                                                 of resources and the process quality [53]
                                                                                                              36
37
Literature Overview
[1] McKinsey & Company, „The five Core IT shifts of scaled agile organizations,“ 2021. [Online]. Available at: https://www.mckinsey.com/business-functions/people-and-
organizational-performance/our-insights/the-five-core-it-shifts-of-scaled-agile-organizations [Accessed: 20.10.2021]
[2] The Boston Consulting Group, „Leaner, Faster, And Better With DevOps,“ 2017
[3] Oliver Wyman, „When Vision and Value Collide - The State Of The Financial Services Industry 2020,“ 2020
[4] K. Beck, M. Beedle, A. van Bennekum, A. Cockburn, W. Cunningham, M. Fowler, J. Grenning, J. Highsmith, A. Hunt, R. Jeffries, J. Kern, B. Marick, R.C. Martin, S. Mellor,
K. Schwaber, J. Sutherland, D. Thomas, „Manifesto for Agile Software Development,“ 2001. [Online]. Available at: https://agilemanifesto.org/. [Accessed: 05.11.2021]
[5] D. J. Proba, „Technikkompositionen für das situative Methoden-Engineering agiler Methoden,“ 2021
[6] J. Halstenberg, B. Pfitzinger, T. Jestädt, „DevOps - Ein Überblick,“ 2020
[7] S. Blank, „Why Continuous Deployment May Mean Continuous Customer Disappointment,“ 2014. [Online]. Available at: https://hbr.org/2014/01/why-continuous-de-
ployment-may-mean-continuous-customer-disappointment. [Accessed: 03.11.2021]
[8] C. Riley, „Waterfall to Agile to DevOps: The State of Stagnant Evolution,“ 2014. [Online]. Available at: https://devops.com/waterfall-agile-devops-state-stagnant-evo-
lution/. [Accessed: 05.11.2021]
[9] BCG, „Going All In With DevOps,“ 2018
[10] D.K. Rigby, J. Sutherland, A. Noble, „Agile at Scale,“ 2018. [Online]. Available at: https://hbr.org/2018/05/agile-at-scale. [Accessed: 05.11.2021]
[11] M. Virimani, „Understanding DevOps & Bridging the gap from Continuous Integration to Continuous Delivery,“ Firth international conference on Innovative Computing
Technology, 2015, doi: 10.1109/INTECH.2015.7173368
[12] J. Benjamin, J. Mathew, „Enhancing the efficiency of continuous integration environment in DevOps,“ IOP Conf. Series: Materials Science and Engineering, 1085, 2022,
doi: 10.1088/1757-899X/1085/1/012025
[13] E. Laukkanen, J. Itkonen, C. Lassenius, „Problems, causes and solutions when adopting contunuous delivery - A systematic literature review,“ Information and Soft-
ware Tecnology, 82, pp. 55-79, 2017, doi: 10.1016/j.infsof.2016.10.001
[14] P. Rodriguez, A. Haghighatkhah, L.E. Lwakatare, S. Teppola, „Continuous deployment of software intensive products and services: A systematic mapping study,“
2016, doi: 10.1016/j.jss.2015.12.015
[15] N. Forsgren, J. Humble, „The Role of Continuous Delivery IT and Organizational Performance,“ 2016, Proceedings of the Western Decision Sciences Institute (WDSI),
doi: 10.2139/ssrn.2681909
[16] A. Mishra, Z. Otaiwi, „DevOps and software quality: A systematic mapping,“ Computer Science Review, 38, 2020, doi: 10.1016/j.cosrev.2020.100308
[17] DevOps Institute, „2021 Upskilling Enterprise DevOps Skills Report,“ 2021
[18] Puppet, „2021 State of DevOps Report,“ 2021
[19] K. Conboy, S. Coyle, X. Wang, M. Pikkarainen, „People Over Process: Key People Challenges in Agile Development,“ IEEE Software, 2010, doi: 10.1109/MS.2010.132
[20] Atlassian and CITE Research, „2020 DevOps Trends Survey,“ 2020
[21] G. Mayank, „DevOps phases across Software Development Lifecycle,“ TechRxiv, 2020, doi: 10.36227/techrxiv.13207796.v2
[22] DORA, „Accelerate: State of DevOps 2019,“ 2019
[23] McKinsey & Company, „Managing security as code enables companies to create value in the cloud securely,“ 2021. [Online]. Available at: https://www.mckinsey.com/
business-functions/mckinsey-digital/our-insights/security-as-code-the-best-and-maybe-only-path-to-securing-cloud-applications-and-systems. [Accessed: 05.11.2021]
[24] Commerzbank AG, „Open Banking White Paper - The Future of Collaboration in Corporate Banking,“ 2020
[25] J. Kanjilal, „6 lessons from the everything-as-code-shift,“ TechBeacon, 2021. [Online]. Available at: https://techbeacon.com/enterprise-it/6-lessons-everything-code-
shift. [Accessed: 2010.2021]
[26] E. Batraski, „The beginning of „Everything as Code,“ 2020. [Online]. Available at: https://medium.com/swlh/the-beginning-of-everything-as-code-a25c4e9a75e9.
[Accessed: 29.10.2021]
[27] A.A. Zeeshan, „Automating Everything as Code,“ DevSecOps for .NET Core. 2020. doi: 10.1007/978-1-4842-5850-7_4
[28] C. Tozzi, „Everything as Code: What It Is and Why It`s Gaining Traction,“ 2020. [Online]. Available at: https://www.itprotoday.com/development-techniques-and-ma-
nagement/everything-code-what-it-and-why-its-gaining-traction [Accessed: 03.11.2021]
[29] Capgemini, „DevSecOps in Regulated Industries - Accelerating Software Reliability & Compliance,“ 2021
[30] G. Rong, J. Zefeng, H. Zhang, Y. Zhang, W. Ye, D. Shao, „DevDocOps: Enabling continuous documentation in alignment with DevOps,“ 2019, doi: 10.1002/spe.2770
[31] Capgemini, „BizDevOps to enable a product-thinking approach,“ 2020. [Online]. Available: https://www.capgemini.com/2020/10/bizdevops-to-enable-the-product-
thinking-approach/. [Accessed: 27-Oct-2021]
[32] Puppet, CircleCI, Splunk, „State of DevOps Report 2019,“ 2019
[33] Puppet and CircleCI, „State of DevOps Report 2020,“ 2020
[34] Accenture, „From NoOps to DevSecOps and beyond,“ 2020. [Online]. Available: https://www.accenture.com/us-en/blogs/software-engineering-blog/emerson-de-
vops-devsecops. [Accessed: 15.10.2021]
[35] E. Kinsbruner, „How artificial intelligence (AI) and machine learning are changing DevOps,“ 2020. [Online]. Available at: https://enterprisersproject.com/artic-
le/2020/9/artificial-intelligence-ai-and-machine-learning-change-devops. [Accessed: 05.11.2021]
[36] M. Trivendi, „12 Ways AI Is Transforming DevOps,“ n.n.[Online] Available at: https://www.tadigital.com/insights/perspectives/12-ways-ai-transforming-devops. [Ac-
cessed: 05.11.2021]
[37] Open Data Science, „How AI and ML are the Next Evolutionary Step for DevOps,“ 2020. [Online]. Available at: https://medium.com/@ODSC/how-ai-and-ml-are-the-
next-evolutionary-step-for-devops-61565448a468. [Accessed: 03.11.2021]
[38] M. Vizard, „GitLab Acquires UnReview to Further AI Ambitions,“ 2020. [Online]. Available at: https://devops.com/gitlab-acquires-unreview-to-further-ai-ambiti-
ons/#:~:text=GitLab%20announced%20this%20week%20it,previous%20efforts%20and%20current%20workloads. [Accessed: 29.10.2021]
[39] BCG , „Going All In With DevOps,“ 2018
[40] E. Kinsbruner, „How artificial intelligence (AI) and machine learning are changing DevOps,“ 2020. [Online]. Available at: https://enterprisersproject.com/artic-
le/2020/9/artificial-intelligence-ai-and-machine-learning-change-devops. [Accessed: 05.11.2021]
[41] SmartBear Software, „Six Stages from Manual to Autonomous Testing,“ n.n. [Online] Available at: https://smartbear.com/resources/ebooks/six-stages-from-manu-
al-to-autonomous-testing/. [Accessed: 03.11.2021
[42] P. Clauson, „Autonomous Testing`s Role in Agile DevOps,“ 2019. [Online]. Available at: https://devops.com/autonomous-testings-role-in-agile-devops/. [Accessed:
05.11.2021]
[43] Proton Technologies AG, „Complete guide to GDPR compliance,“ 2021. [Online]. Available at: https://gdpr.eu/. [Accessed: 05.11.2021]
[44] McKinsey & Company, „Doing vs being: Practical lessons on building an agile culture,“ 2020. [Online]. Available at: https://www.mckinsey.com/business-functions/
people-and-organizational-performance/our-insights/doing-vs-being-practical-lessons-on-building-an-agile-culture. [Accessed: 05.11.2021]
[45] McKinsey & Company, „Building agile capabilities: The fuel to power your agile „body“,“ 2019. [Online] Available at: https://www.mckinsey.com/business-functions/
people-and-organizational-performance/our-insights/building-agile-capabilities-the-fuel-to-power-your-agile-body. [Accessed: 05.11.2021]
[46] K. Eilers, B. Simmert, C. Peters, „Doing Agile vs. Being Agile - Understanding Their Effects to Improve Agile Work,“ 2020, ICIS 2020 Proceedings.6.
[47] Scaled Agile Inc., „Lean-Agile Leadership,“ 2021. [Online]. Available at: https://www.scaledagileframework.com/lean-agile-leadership/. [Accessed: 14.10.2021]
[48] V. Stray, B. Memon, L. Paruch, „A Systematic Literature Review on Agile Coaching and the Role of the Agile Coach,“ 2020, in M. Morison, M. Torchiano, A. Jedlitschka,
„Product-Focused Software Process Improvement,“ PROFES 2020, Lecture Notes in Computer Science, 12562. doi: 10.1007/9878-3-030-34148-1_1
[49] Atlassian , „Was ist Kanban?,“ n.n. [Online]. Available at: https://www.atlassian.com/de/agile/kanban. [Accessed: 05.11.2021]
[50] Scaled Agile Inc., „Product Owner,“ 2021. [Online]. Available at: https://www.scaledagileframework.com/product-owner/. [Accessed: 05.11.2021]
[51] Scrum.org, „Scrum Glossary,“ 2021. [Online] Available at: https://www.scrum.org/resources/scrum-glossary. [Accessed: 05.11.2021]
[52] Atlassian, „User Storys mit Beispielen und Vorlagen,“ n.n. [Online]. Available at: https://www.atlassian.com/de/agile/project-management/user-stories. [Accessed:
05.11.2021]
[53] I.S. Lasa, C.I. Laburu, R. de Castro Vila, „An evalutaion of the value stream mapping tool,“ Business Process Management Journal, Vol. 14, No. 1, 2008, pp. 39 - 52
                                                                                                                                                                        38
 Commerzbank AG
    Head Office
    Kaiserplatz
 Frankfurt am Main
www.commerzbank.de
     Postal Address
60261 Frankfurt am Main
   Tel.: +49 69 136-20
info@commerzbank.com