DEV Community: Tayguara Reis

Running Playwright in GitHub Actions: named checks, sharding, one merged report, and a flakiness policy

Tayguara Reis — Wed, 07 Oct 2026 17:22:57 +0000

A test pipeline has one job: give the team a fast signal it can trust. When a pull request goes red, the author should know what broke without opening a log. When it goes green, nobody should wonder whether a retry quietly hid a problem.

This article walks through the pipeline of playwright-qa-showcase, a small public repository I use to show how I set up test automation when I lead QA on a project. It has 56 tests: BDD UI flows against SauceDemo, typed API tests against Restful Booker, axe-core accessibility scans, and unit tests for the test-support code. Every snippet below is taken from the real files, shortened for reading. Every number comes from a real run.

One check per validation

My first version had a single Lint, typecheck, format job. It worked, but a red check told you only that "something in quality" failed. So I split it into five parallel jobs, each with its own name:, and each shows up as its own check on the pull request:

jobs:
  lint:
    name: Lint
    runs-on: ubuntu-latest
    timeout-minutes: 10
    steps:
      - uses: actions/checkout@v7
        with:
          persist-credentials: false
      - uses: actions/setup-node@v7
        with:
          node-version-file: .nvmrc
          cache: npm
      - run: npm ci
      - run: npm run lint
  # typecheck (Typecheck), format (Format), unit (Unit tests), gherkin (Gherkin): same shape

None of the five needs a browser. Unit tests runs the pure-Node Playwright project that covers the helpers (money math, the accessibility baseline diff, a test data builder). Gherkin runs bddgen, which fails when a step has no definition, and then a guard I will come back to. In the real run of the PR that introduced this split, the five jobs finished in 10 to 18 seconds each.

The browser tests wait for all of them:

  test:
    name: Tests (shard ${{ matrix.shardIndex }}/${{ matrix.shardTotal }})
    needs: [lint, typecheck, format, unit, gherkin]

Duplicating checkout, setup-node and npm ci five times looks wasteful, and on paper it is. In practice the npm cache makes each copy cheap, and the payoff is a PR page that reads like a checklist. A composite action could remove the repetition. At five short jobs I chose readability over DRY.

Sharding, and an honest note about it

The browser suites (ui, a11y, api) run as a two-shard matrix:

    strategy:
      fail-fast: false
      matrix:
        shardIndex: [1, 2]
        shardTotal: [2]
    steps:
      # checkout, setup-node, npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx bddgen
      - name: Run Playwright tests
        run: npx playwright test --project=ui --project=a11y --project=api --shard=${{ matrix.shardIndex }}/${{ matrix.shardTotal }}
      - name: Upload blob report
        if: ${{ !cancelled() }}
        uses: actions/upload-artifact@v7
        with:
          name: blob-report-${{ matrix.shardIndex }}
          path: blob-report
          retention-days: 1
          if-no-files-found: error

Three details matter. fail-fast: false keeps one failing shard from cancelling the other, so you see every failure in one run instead of discovering them one at a time. The unit project is left out of the shards because it needs no browser and already ran in its own job. And if-no-files-found: error makes a shard that produced no report fail loudly instead of silently shrinking the merged report.

Now the honest part. At this size, sharding demonstrates the pattern; it does not save time. Each shard ran 14 tests. In that same run, the Playwright step took 7.3 seconds of test time per shard, while playwright install --with-deps chromium took 58 seconds on one shard and 26 on the other. A single job would finish sooner than two jobs that each install a browser. I kept the matrix because the pattern is the point of the repository, and because scaling it is a one-line change: shardIndex: [1, 2, 3, 4] with shardTotal: [4]. On a real project I would add shards when test time, not setup time, dominates the job.

One merged report, plus a summary nobody has to download

Sharding creates a reporting problem: two partial reports are worse than one. In CI the config uses the blob reporter (reporter: isCI ? [['blob'], ['github'], ['list']] : ...), and a final job merges the blobs:

  merge-reports:
    name: Merge reports
    if: ${{ !cancelled() && needs.test.result != 'skipped' }}
    needs: test
    steps:
      # checkout, setup-node, npm ci
      - uses: actions/download-artifact@v8
        with:
          path: all-blob-reports
          pattern: blob-report-*
          merge-multiple: true
      - name: Merge into one HTML report
        env:
          PLAYWRIGHT_HTML_OPEN: never
          PLAYWRIGHT_JSON_OUTPUT_NAME: ${{ runner.temp }}/merged-results.json
        run: npx playwright merge-reports --reporter html,json ./all-blob-reports

The if: condition is the part people get wrong. The default success() would skip this job exactly when you need the report most, after a shard failed. always() goes too far: when Lint fails, the shards are skipped and there is nothing to merge. !cancelled() && needs.test.result != 'skipped' covers both cases.

The HTML report is uploaded as an artifact kept for 14 days. Downloading an artifact is friction, though, so a short inline Node script reads the merged JSON and writes a table to the job summary:

const { stats } = JSON.parse(fs.readFileSync(process.env.RESULTS_JSON, "utf8"));
const rows = [
  ["Passed", stats.expected],
  ["Failed", stats.unexpected],
  ["Flaky (passed on retry)", stats.flaky],
  ["Skipped", stats.skipped],
];

The "Flaky" row is there on purpose. It is the bridge to the flakiness policy.

A flakiness policy, enforced by config and lint

"We don't tolerate flaky tests" is a slogan. A policy is something the tooling enforces. From playwright.config.ts:

forbidOnly: isCI,
retries: isCI ? 2 : 0,
use: {
  // Retries are off locally, so keep the trace of a failure instead of waiting for a retry.
  trace: isCI ? 'on-first-retry' : 'retain-on-failure',
  screenshot: 'only-on-failure',
},

Retries exist only in CI, so a flaky test is visible while you develop instead of being absorbed. on-first-retry would never fire locally with zero retries, so locally the trace is kept on failure. forbidOnly stops a stray test.only from turning a full run into a one-test run.

The trade-off I state openly: a test that passes on retry leaves the run green. That is why the summary shows the flaky count, and why the policy says it gets investigated. Someone has to own that number.

Lint closes the usual escape hatches, with eslint --max-warnings=0:

rules: {
  'playwright/no-wait-for-timeout': 'error',
  'playwright/no-skipped-test': 'error',
  'playwright/no-force-option': 'error',
},

Hard waits, forced clicks and skipped tests are the three most common ways a flaky test gets "fixed" without being fixed. Gherkin has the same escape hatch in tags, so the Gherkin job greps for it:

      - name: Reject disabled scenarios
        run: |
          if grep -rnE '(^|[[:space:]])@(skip|fixme|only)([[:space:]]|$)' features --include='*.feature'; then
            echo '::error::A scenario is disabled with @skip, @fixme or @only. Fix it or track it with @fail.'
            exit 1
          fi

A known product defect is tracked with @fail, an expected failure that asserts the correct behavior, rather than hidden with @skip. If the site gets fixed, Playwright reports it as unexpectedly passing.

Supply chain and security, sized for a test repo

Test repositories run code on every pull request, so they deserve the same hygiene as product code:

permissions: contents: read at the workflow level. Only the CodeQL job gets security-events: write.
persist-credentials: false on every checkout, so the token is not left in .git/config for later steps.
No pull_request_target.
Pinned action versions. dependency-review-action is pinned to v5.0.0 because the project publishes no floating v5 tag.
A concurrency group with cancel-in-progress: true, so a new push cancels the superseded run.

Three extra workflows add checks. codeql.yml analyzes both javascript-typescript and actions (the workflow files themselves), on PRs, on pushes to main and weekly. dependency-review.yml fails a PR that adds a dependency with a known vulnerability of moderate severity or worse. actionlint.yml lints the workflows.

Dependency review taught me a real lesson. Its first run failed with "Dependency review is not supported on this repository. Please ensure that Dependency graph is enabled". The YAML was correct; a repository setting was off. After enabling the Dependency graph, the re-run passed. A security check can depend on configuration that lives outside the repository, so check the settings page, not only the diff.

actionlint is path-filtered to .github/workflows/**, and that is exactly why it is not a required check. A workflow skipped by a path filter never reports, and a required check that never reports leaves every unrelated PR waiting on a pending status.

Branch protection, nightly runs and an honest badge

main requires ten checks: Lint, Typecheck, Format, Unit tests, Gherkin, Tests (shard 1/2), Tests (shard 2/2), CodeQL (javascript-typescript), CodeQL (actions) and Dependency review. Dependency review can be required because it runs on every pull request; actionlint cannot, for the reason above. Named jobs make this list readable, and it is also why stable job names matter: rename a job and branch protection waits for a check that no longer exists. The shard names embed the shard count, so raising the matrix means updating this list too. Merge reports stays optional, since the shards themselves already block the merge.

The Playwright workflow also runs nightly (cron: '0 6 * * *') to catch drift in the public sandboxes. Those sandboxes go down from time to time, and I do not want that to make the project look broken. The README badge is filtered to push events:

[![Playwright](https://github.com/tayguara/playwright-qa-showcase/actions/workflows/playwright.yml/badge.svg?branch=main&event=push)](...)

The nightly run still reports failures in the Actions tab, where the team looks, while the badge reflects the state of the code on main. GitHub also disables scheduled workflows after 60 days without repository activity, which is worth knowing before trusting a nightly job on a quiet repo.

Dependabot, with two deliberate exceptions

Dependabot updates npm packages and GitHub Actions weekly, grouped into one PR per ecosystem. Two major updates are ignored:

    ignore:
      # typescript-eslint does not support TypeScript 7 yet; revisit when it does.
      - dependency-name: typescript
        update-types: ['version-update:semver-major']
      # The project runs on Node 22 (.nvmrc); @types/node majors must follow the Node version.
      - dependency-name: '@types/node'
        update-types: ['version-update:semver-major']

The @types/node rule came from a real PR: Dependabot proposed 22.20.4 to 26.6.3, and CI passed. Green CI did not make it correct. Types for Node 26 on a Node 22 runtime let you compile calls to APIs that do not exist at runtime. I closed the PR, and closing a grouped PR does not ignore future versions, so the rule went into config. Automated updates need a human to decide which versions the project actually targets.

Takeaways

Give every validation its own named job. A PR check list that reads like a checklist saves more time than the duplicated setup costs.
Shard when test time dominates, not before. Measure the browser install against the test step; here it was 26 to 58 seconds of setup against about 7 seconds of tests.
Merge sharded reports with !cancelled() && needs.test.result != 'skipped', and put a results table, including flaky count, in the job summary.
Enforce the flakiness policy in tooling: retries only in CI, forbidOnly, lint bans on hard waits, forced clicks and skips, and a guard for Gherkin tags.
Treat the pipeline as product code: least-privilege tokens, pinned actions, required checks with stable names, and settings you verify, not assume.

Tayguara Dias Reis is a Tech Lead with 14+ years in software quality and ISTQB CTFL certification. The full pipeline is at github.com/tayguara/playwright-qa-showcase. For the Jenkins side of the same ideas, see From scripted to declarative: what 10+ years of Jenkins pipelines taught me about shared libraries and quality gates.

From scripted to declarative: what 10+ years of Jenkins pipelines taught me about shared libraries and quality gates

Tayguara Reis — Sat, 03 Oct 2026 18:10:52 +0000

I wrote my first Jenkins pipeline in 2016. Ten years later, the repository it started has more than 260 pipeline files: deploy jobs, end-to-end suites, a Selenium grid, health checks, scheduled maintenance. Most of them are scripted. A newer project, started in 2021, runs on six declarative pipelines and puts releases through a chain of quality gates before a blue-green deploy.

This is not a syntax tutorial; the Jenkins documentation covers syntax well. It is about the decisions behind those files: why I started scripted, what shared libraries fixed, how I order quality gates, how blue-green rollback fits in, and when an old scripted job is worth migrating. Some of those decisions I would make differently today, and I will say which.

The snippets below are simplified. A complete, runnable version of the same patterns lives in a public repository, jenkins-declarative-pipelines: a tested shared library, a declarative Jenkinsfile with quality gates and a simulated blue-green deploy, all started with make up on Docker Compose.

2016: why I started scripted

The honest reason is that I did not know the declarative syntax yet, and scripted felt easier. In an internal review at the end of that year I noted, half-jokingly, that the build script was written in Groovy only because Jenkins Pipeline required it. Scripted pipeline is Groovy with a few Jenkins steps: open a node {}, split the work into stage() blocks, call sh. If you can write a loop and an if, you can ship a pipeline on day one. There is no grammar to learn and no rule about where a block may appear.

That freedom paid off at first. I could load a helper file, parse a JSON parameter into a map and generate stages from a list. In that first year the main product shipped around 1,000 production builds, about 2.7 a day. Parallelizing the end-to-end suite cut a production run from roughly 40–50 minutes to 6–8, and a development run from 90–100 minutes to 25–30. A small pre-checkout verification script cut checkout from about 4 minutes to 2 at most. For a small team, that speed mattered more than consistency.

The cost showed up later, when other people had to read those jobs. Every pipeline had its own shape. Error handling lived in try/catch blocks, and when I went back to my oldest jobs I found empty catch blocks that swallowed exceptions, so a step could fail and the build would carry on as if nothing happened. Notification logic was duplicated in the success path and in the catch block. It is a legacy pattern that no longer earns its keep, and the migration to declarative retires it: the post conditions take over both jobs. For the first years, "reuse" meant load-ing Groovy helper files from the same repository: better than copy-paste, but with no versioning and no clear boundary between helper and job.

None of this is a flaw of scripted syntax itself. It lets you do anything, and nothing pushes back.

Shared libraries as the backbone

The change that mattered most in both eras was moving reusable logic into shared libraries. The first payoff was the end of copy-paste: pipelines with similar steps stopped carrying their own copies of the same code, and each step got one clear responsibility. The best example runs in every pipeline: the summary posted to Slack after each build. The logic that sanitizes and formats that message lives once in the library, and each pipeline only calls the notification step with its own project details.

The layout that survived is one common library plus one library per product:

ci-common/                  # shared by most jobs
  vars/
    notifyBuild.groovy      # build summary to the chat channel
    runGate.groovy          # run a check, keep its report, fail with a clear message
    coverageGate.groovy
    loadEnvConfig.groovy
  src/com/example/ci/       # plain Groovy classes, when a step needs them
ci-payments/                # one per product, only what is specific to it
  vars/
    deployBlueGreen.groovy
    smokeCheck.groovy

The common library is imported by dozens of jobs; around ten product libraries sit next to it. Each file in vars/ becomes a global step that a Jenkinsfile calls by name. A step is a call method that takes a map, so call sites read like configuration:

// vars/runGate.groovy
def call(Map args) {
    String name   = args.name      // e.g. 'Static analysis'
    String cmd    = args.command   // exits non-zero on violations
    String report = args.report    // machine-readable output, read later by the summary

    int status = sh(script: "${cmd} > '${report}'", returnStatus: true)
    if (status != 0) {
        error("${name} gate failed (exit code ${status}). Findings: ${report}")
    }
}

Two rules decide what goes where. A library holds how: how we run a gate, notify, or switch a blue-green slot. The Jenkinsfile holds what: which gates this service runs, in which order, with which thresholds. Groovy repeated in two Jenkinsfiles belongs in a library; a value that differs per service never gets hardcoded in one.

The second rule is versioning, and I learned it late. My Jenkinsfiles load libraries by name only, @Library('ci-common') _, which means every job follows whatever default version is configured in Jenkins. That is convenient until a library change breaks a job nobody has touched in a year. Jenkins lets you pin a branch, tag or commit in the annotation:

@Library('ci-common@v3.4.0') _

Pin production pipelines to a tag, let a staging or canary job track the main branch, and bump the tag in a reviewed change. A shared library is a dependency like any other, and an unpinned dependency is a deploy you did not schedule. Pinning is on my own list: it comes with the migration, job by job.

2021: going declarative

When I started the newer project in 2021, I wrote it declarative from day one. What changed was not the steps. sh, checkout, junit and the library calls are the same. What changed is that the file now has a fixed shape:

@Library('ci-common@v3.4.0') _

pipeline {
    agent { label 'linux' }

    options {
        timeout(time: 45, unit: 'MINUTES')
        timestamps()
        disableConcurrentBuilds()
    }

    parameters {
        choice(name: 'TARGET_ENV', choices: ['staging', 'production'])
        string(name: 'GIT_REF', defaultValue: 'main')
        booleanParam(name: 'ROLLBACK', defaultValue: false)
    }

    environment {
        APP_DIR = "${WORKSPACE}/app"
    }

    stages {
        stage('Static analysis') {
            when { not { expression { params.ROLLBACK } } }
            steps {
                runGate(name: 'Static analysis',
                        command: 'make static-analysis',
                        report: 'reports/static-analysis.json')
            }
        }
        // more gates, then deploy
    }

    post {
        always {
            junit allowEmptyResults: true, testResults: 'reports/junit/*.xml'
            notifyBuild(reportsDir: 'reports')   // result comes from currentBuild.currentResult
        }
    }
}

What I gained, in order of how much it mattered:

Readability for people who did not write it. A new engineer finds the timeout, the parameters and the failure handling in the same place in every job.
post instead of try/catch. always, success, failure, unstable, fixed, regression, aborted and cleanup replace the hand-written error paths. Notification moved out of catch blocks and could no longer be skipped by an exception thrown in the wrong place.
when instead of if. A stage skipped by when still appears in the stage view, marked as skipped, so the run tells you what it chose not to do.
options as policy. A timeout, timestamps and no concurrent deploys are one line each, and their absence is visible in review.
Validation before running. The declarative linter (the declarative-linter command in the Jenkins CLI, or a POST to the controller's pipeline-model-converter/validate endpoint) catches structural mistakes before a build starts. Scripted pipeline only tells you at runtime.

The escape hatch is script {}, which runs scripted Groovy inside a declarative step. My first declarative pipeline leaned on it far too much: most stages were a script {} block that returned early on a rollback run. It was scripted pipeline in declarative clothing, and the stage view showed those stages as green although they did nothing. The when condition above is what I would write today, and rewriting those stages is part of the same improvement plan. The Jenkins docs agree: when a script {} block grows, move it into a library step.

Quality gates before a blue-green deploy

The newer project's quality gates live in three places, and where each one runs turned out to matter as much as what it checks. They were added one at a time over several years, not designed up front, and the complexity gate is the most recent.

In the deploy pipeline, before anything is copied to a server:

Code style (a formatter in dry-run mode)
Static analysis (type and bug checks)
Automated refactoring check (a refactoring tool in dry-run mode: if it would change the code, the code is not up to the agreed standard)
Unit tests, with coverage collected
Coverage threshold
Complexity risk (CRAP)

In the application repository's own CI:

Mutation testing

In separate jobs that run after the deploy:

End-to-end tests against the deployed environment, plus a dedicated security test job

The order inside the deploy pipeline follows one rule: cheap and fast first. Style and static analysis need seconds and no database, so they fail a bad commit before anything expensive starts. Coverage and complexity read the report the unit tests produce, so they cost almost nothing extra.

The two most expensive checks moved out of the deploy pipeline, for the same reason. Mutation testing re-runs the suite against many small mutations of the code. It started as the last stage of the deploy pipeline and made every build wait, so I moved it out of Jenkins and into a pipeline in the application repository itself, on GitLab CI, where it runs without holding a release hostage. End-to-end tests went the same way: they used to sit inside the deploy pipeline, and now a separate job runs them after each deploy, next to a security test job. The gates still exist; they just run where their cost does not block a deploy. That is a lesson in itself: a gate that makes releases slow enough that people want to skip it is a gate in the wrong place, not a gate to delete.

The CRAP gate deserves a sentence of explanation, because coverage alone misleads. CRAP (Change Risk Anti-Patterns) combines cyclomatic complexity with coverage per method: CRAP = complexity² × (1 − coverage)³ + complexity. A simple method with no tests scores low; a complex method with poor coverage scores very high. The gate fails the build when a method crosses the threshold, which catches the case a global coverage number hides: 90% overall, and the one complex method that matters is untested. The gates earn their keep in a very human situation. More than once, a developer told QA something like "Relax, I already validated everything in this task, you can skip testing and ship it straight to production." We took the advice, started the build, and the gates failed, keeping broken code out of production. That is the point of a gate: a cheap layer of safety that does not depend on anyone's confidence, including mine.

Every gate follows the same pattern: write a machine-readable report, then decide pass or fail with a message a human can act on:

// vars/coverageGate.groovy (simplified; the repository parses the XML in a tested class)
def call(Map args) {
    BigDecimal min = args.min as BigDecimal          // e.g. 80
    String raw = sh(returnStdout: true,
        script: "xmllint --xpath 'string(/coverage/@line-rate)' '${args.report}'").trim()
    BigDecimal actual = (raw as BigDecimal) * 100

    echo "Line coverage: ${actual}% (minimum ${min}%)"
    if (actual < min) {
        error("Coverage gate: line coverage ${actual}% is below the ${min}% minimum. " +
              "Add tests or lower the threshold in a reviewed change.")
    }
}

The report-first habit pays off in the summary. In post { always }, a library step reads each report that exists and builds one message for the team channel: style, static analysis and refactoring findings, tests run and failed, coverage, the top CRAP score, plus the commit, who started the build and the duration. When a gate fails, the message still shows what every earlier gate found, so nobody opens the console log to learn whether it was one style issue or forty.

Blue-green with rollback

Blue-green is older than my declarative pipelines. The first version shipped in 2017, in a scripted pipeline, inspired by Jez Humble and Dave Farley's Continuous Delivery. The goals I wrote down at the time still hold: faster builds and rollbacks, more control over what reaches users, and lower downtime risk. When I wrote the declarative pipeline in 2021, I reused the same pattern almost step for step. The technique outlived the syntax change, which is the point: blue-green and rollback are architecture, not a scripted-versus-declarative feature.

Production keeps two release directories, blue and green, and one pointer to the live one (a symlink, or the web server's root setting). A deploy goes like this:

Read which color is live. The idle one is the target.
Copy the release, already through the gates, into the idle directory.
Run migrations, warm the cache and, ideally, smoke-check the idle side.
Switch the symlink atomically to the idle directory and reload the PHP and web server processes.

Rollback is the same switch in reverse. Since 2017 it has been a build parameter: a rollback run skips tests, file changes and library updates, and points the live pointer back at the previous slot, which still holds the previous release. No new build and no copy, so it is the fastest run the pipeline has.

// vars/deployBlueGreen.groovy (simplified)
def call(Map args) {
    String live = sh(returnStdout: true,
        script: "basename \"\$(readlink -f '${args.currentLink}')\"").trim()   // 'blue' or 'green'
    String target = (live == 'blue') ? 'green' : 'blue'

    if (!args.rollback) {
        sh "rsync -a --delete --exclude=.env build/ '${args.releasesDir}/${target}/'"
        smokeCheck(slot: target)
    }
    // ln -sfn alone removes the old link before creating the new one;
    // building a temporary link and renaming it over the old one is atomic
    sh "ln -sfn '${args.releasesDir}/${target}' '${args.currentLink}.next' && " +
       "mv -Tf '${args.currentLink}.next' '${args.currentLink}'"
    sh 'sudo systemctl reload php-fpm nginx'
    echo "Live color is now ${target} (was ${live})"
}

A detail worth knowing: ln -sfn on its own is not atomic. It deletes the old link and then creates the new one, so for a moment there is no live release. Creating the new link under a temporary name and renaming it with mv -T closes that window, because a rename replaces the old link in a single step.

Three caveats I would put in any design review. First, rollback goes back exactly one release; after two deploys, the old one is gone. Second, the symlink rolls back code, not data: migrations must be backward-compatible (expand first, contract in a later release), or the old code will meet a schema it does not understand. Third, keep one source of truth for which color is live. Reading the symlink, as above, beats a separate marker file that can drift from reality.

Environment config without secrets

The pipeline is the same for every environment; only the config changes. A parameter picks the environment, and a library step loads a small per-environment file with hosts, paths and switches. I have used Groovy files that return a map for this. Today I would use YAML read with readYaml, because config should be data, and data cannot execute code.

Secrets never live in those files, and never in the repository. Database passwords, API tokens and SSH keys belong in Jenkins Credentials (or a vault, with a Jenkins integration) and enter the build only for the step that needs them:

def cfg = loadEnvConfig(params.TARGET_ENV)   // hosts, paths, app env: no secrets

withCredentials([usernamePassword(credentialsId: "db-${params.TARGET_ENV}",
                                  usernameVariable: 'DB_USER',
                                  passwordVariable: 'DB_PASS')]) {
    sh './bin/migrate'   // reads DB_USER/DB_PASS from env
}

Note the single quotes in sh. With double quotes, Groovy interpolates the secret into the command string; with single quotes, the shell reads it from the environment and Jenkins can mask it in the log. In declarative, credentials('id') inside environment {} does the same for a whole stage. One credential ID per environment keeps a staging job away from production secrets.

Scripted vs declarative: where each one is stronger

Both run on the same Pipeline engine. Declarative is parsed into the same CPS-transformed execution, so the Groovy sandbox, script approvals, serialization rules and @NonCPS limits apply to both, and so does everything inside a script {} block or a library step.

Aspect	Scripted	Declarative
Learning curve	Low to start if you know Groovy; the hard parts (CPS, serialization, sandbox) show up later	A fixed set of directives to learn; the same engine limits still apply underneath
Readability and onboarding	Depends entirely on the author's discipline	Same sections in the same order in every file
Structure enforcement	None	Required `agent`, `stages`, `steps`; directives only where allowed
Validation before running	Groovy compile errors only, at build start	Linter via CLI or HTTP endpoint, plus full structure validation before the first stage
Dynamic logic (loops, generated stages)	Full Groovy: generate stages and parallel branches from data	`matrix`, `parallel` and `when` cover common cases; the rest needs `script {}` or a library
Error handling	`try/catch/finally`, explicit and easy to get wrong	`post` conditions per pipeline and per stage; `catchError` and `warnError` work in both
Restart from stage	Not available (Replay only)	Restart any top-level stage of a completed run, with the same parameters and SCM revision
Tooling and visualization	Stages appear in the stage views; conditionally skipped stages just disappear	Same views, plus `when`-skipped stages shown as skipped; Directive Generator in the UI
Reuse via shared libraries	Call any library step or class	Call library steps in `steps`, or expose an entire pipeline as a template step (one declarative pipeline per build)
Testability	JenkinsPipelineUnit	JenkinsPipelineUnit (`DeclarativePipelineTest`); in both worlds, library steps are the easiest unit to test

The template option deserves a note. A vars/ step can contain a whole pipeline {} block, so a service's Jenkinsfile shrinks to one call. That is the strongest standardization tool Jenkins has, and the most rigid: an exception for one team becomes a parameter for all of them. I use it when the gate chain must be identical across teams, not as a default.

When to migrate, and how to do it safely

Migrating my scripted base to declarative is my own next step, so this section is the plan I am following, not a retrospective.

Signals that it is time:

People who did not write the pipelines now maintain them, and every job reads differently.
The same blocks are copied between jobs, and a fix has to be applied in several places.
You need restart-from-stage or clearer visualization to recover from flaky infrastructure without re-running everything.
You want the same gates across teams, enforced, not suggested.
An audit needs to see, quickly, what runs before production.

Reasons not to:

The pipeline generates its stages from data in loops. Declarative will push most of it into script {}, and you gain little.
The job is stable, rarely changes and nobody touches it. Migration risk with no payoff.
You have no time to test the new job against the old one. An untested migration of a deploy pipeline is a production incident on a timer.

How to do it safely:

Inventory and classify. List every job, when it last ran, who owns it, whether it deploys, and how many jobs share its pattern. Delete what is dead first.
Move logic into library steps first. While the job is still scripted, extract the shared blocks into vars/ steps and switch the scripted job to call them. This is the riskiest change, and it happens with the old shape still in place.
Migrate one job per sprint, starting with the most-copied pattern. The first migration becomes the template for the next ten.
Run old and new side by side. Point the new job at a non-production target, or run it with deploy steps disabled, and compare results for a few cycles.
Keep script {} for the genuinely hard parts, and move each of those into a library step later. Do not block the migration on making it pure.
Add a linter step to the library repository and to the pipeline repository, so a broken Jenkinsfile fails review, not the deploy.
Freeze, then delete. Disable the old job, keep it for one release cycle, then remove it. Two live versions of the same deploy is how drift starts.

Takeaways

Start with whatever lets you ship, but expect to pay for structure later. Scripted made me fast in 2016 and expensive to read years later.
Put the how in shared libraries and the what in the Jenkinsfile, and pin library versions in production pipelines.
Order quality gates from cheap to expensive, write a report before deciding pass or fail, and summarize every gate in one message, even on failure.
Blue-green and rollback are architecture, not syntax: the same pattern survived the move from scripted to declarative. Keep migrations backward-compatible, or rollback only rolls back half the system.
Migrate scripted to declarative when people, copy-paste or compliance demand it, one job at a time, logic into libraries first.

Tayguara Dias Reis is a Tech Lead with 14+ years in software quality and ISTQB CTFL certification. He writes about test automation and CI at dev.to/tayguara. The code from this article runs end to end in jenkins-declarative-pipelines. For the GitHub Actions side of the same ideas, see the public playwright-qa-showcase repository; the next article in this series covers it.

BDD with Playwright: Gherkin scenarios that business people can actually read

Tayguara Reis — Thu, 01 Oct 2026 13:25:21 +0000

I have written and maintained end-to-end suites with Cucumber on top of Playwright, using the cucumber-js runner. It works, but two risks show up in every Gherkin suite. Either the feature files turn into click-scripts with a Given/When/Then prefix ("When I click the button with id continue"), so nobody outside engineering reads them, or the step layer grows into a second framework with its own world object, its own retries and its own reporting, rebuilt next to a test runner that already does all of that.

Gherkin is worth it only when a product owner can read a scenario and say "yes, that is the rule." Everything below comes from a small public repository I maintain, playwright-qa-showcase, which runs 10 Gherkin scenarios (16 tests once the outlines expand) against the SauceDemo store with Playwright 1.63 and playwright-bdd 9.2.1.

Why playwright-bdd instead of cucumber-js

playwright-bdd does not run your scenarios in a separate runner. bddgen compiles the .feature files into ordinary Playwright spec files, and the Playwright test runner executes them. That one decision gives you everything the runner already does well: fixtures, full parallelism, traces, the HTML report, retries, sharding and blob-report merging in CI.

The whole BDD setup is a few lines in playwright.config.ts:

// BDD is used for the UI project only. `bddgen` turns the .feature files into Playwright specs.
const uiTestDir = defineBddConfig({
  features: 'features/**/*.feature',
  steps: ['features/steps/**/*.ts', 'src/fixtures/ui.fixtures.ts'],
  outputDir: '.features-gen/ui',
});

The ui project then points testDir at uiTestDir, and the same config keeps trace: 'on-first-retry' in CI and 'retain-on-failure' locally. Gherkin tags become native Playwright tags, so npx playwright test --grep @smoke selects the two smoke scenarios with no extra tooling.

Where BDD pays off, and where it does not

The repository has four Playwright projects, and only one of them uses Gherkin:

UI business flows (login, inventory, checkout): Gherkin. A readable scenario is useful to non-engineers, and the steps are reused across features.
API tests: plain Playwright. Their value is in assertions on status codes, headers and payload contracts. "Then the response status is 200" adds a translation layer without adding clarity.
Accessibility checks: plain Playwright. The interesting output is a list of axe rule ids per page, not a sentence.
Unit tests for test helpers: plain Playwright, no browser.

This is the first judgment call I make on any engagement. BDD is a communication tool. Where there is no one to communicate with outside engineering, it is overhead.

Writing steps in business language

Here is the core checkout scenario:

@smoke
Scenario: A shopper completes a purchase end to end
  Given the cart contains the following products:
    | product               |
    | Sauce Labs Backpack   |
    | Sauce Labs Bike Light |
  When I open the cart
  And I start the checkout
  And I submit valid shipping information
  Then the order overview lists the same products
  And the item total equals the sum of the item prices
  And the order total equals the item total plus tax
  When I finish the order
  Then I see the order confirmation "Thank you for your order!"
  And the cart badge is not shown

There are no selectors, no field names and no test data that does not matter to the rule. "I submit valid shipping information" is declarative: the step knows what valid means (a synthetic Ada / Lovelace / 12345 record in src/data/checkoutData.ts). The imperative version, three "When I fill ..." lines, would tell a reader nothing new and break the scenario every time the form changes.

When the variation is the rule, I spell it out, and a Scenario Outline keeps it to one scenario:

Scenario Outline: Required shipping fields are validated: <case>
  ...
  When I submit the shipping form with first name "<first_name>", last name "<last_name>" and postal code "<postal_code>"
  Then I see the checkout error "<error>"
  And I am still on the shipping information step

  Examples:
    | case                | first_name | last_name | postal_code | error                          |
    | missing first name  |            | Lovelace  | 12345       | Error: First Name is required  |
    | missing last name   | Ada        |           | 12345       | Error: Last Name is required   |
    | missing postal code | Ada        | Lovelace  |             | Error: Postal Code is required |

Two smaller choices matter as much as the wording. In the sorting outline, "Name (A to Z)" is deliberately left out of the examples, with a comment explaining why: it is the default order, so it would pass even if the sort control did nothing. And the login outline says "a password" (valid, wrong, empty) instead of putting the password in the feature file. The step maps the kind to a value, and the real password stays in config/env.ts.

Fixtures and page objects, injected into steps

Steps never construct page objects. They receive them as Playwright fixtures, which createBdd binds to Given/When/Then:

export const test = base.extend<UiFixtures>({
  inventoryPage: async ({ page }, use) => {
    await use(new InventoryPage(page));
  },
  checkoutOverviewPage: async ({ page }, use) => {
    await use(new CheckoutOverviewPage(page));
  },
  // ...one fixture per page object, plus the header component
  scenario: async ({}, use) => {
    await use({ cartProducts: [] });
  },
});

export const { Given, When, Then } = createBdd(test);

The scenario fixture replaces the Cucumber "world". It holds state shared between the steps of one scenario, here the products added to the cart, and because it is test-scoped it cannot leak into another test. A step asks only for what it needs:

When('I add {string} to the cart', async ({ inventoryPage, scenario }, productName: string) => {
  await inventoryPage.addToCart(productName);
  scenario.cartProducts.push(productName);
});

Then('the order overview lists the same products', async ({ checkoutOverviewPage, scenario }) => {
  await expect(checkoutOverviewPage.productNames).toHaveText(scenario.cartProducts);
});

No globals, no shared mutable module state, and every test runs in its own browser context, so fullyParallel: true is safe. Page objects own locators (role and data-test attributes, never CSS paths), and assertions are web-first, so there are no hard waits.

Free-form strings from Gherkin are also narrowed at the boundary. toSauceUser(username) turns "standard_user" into a typed union member and throws with the list of known users if a scenario has a typo, instead of failing later on a confusing login error.

Login: through the UI only where login is under test

Logging in through the form in every scenario is slow and makes the login page a failure point for tests that have nothing to do with it. SauceDemo keeps its session in a plain session-username cookie, so outside login.feature the suite sets the cookie and opens the inventory page:

export async function loginViaSession(page: Page, username: SauceUser): Promise<void> {
  await page
    .context()
    .addCookies([{ name: 'session-username', value: username, url: env.sauce.baseUrl }]);
  await page.goto('/inventory.html');
}

I validated this against the live site before relying on it, and the Given I am logged in as "..." step still asserts that the Products page is shown. If the site ever stops honoring the cookie, only this function changes. login.feature keeps a separate step, Given I am logged in as "standard_user" using the login form, for the logout scenario, where the real form is part of what is being tested. In a real application the equivalent would be an API login or a stored storageState. The principle is the same.

Asserting business rules, not just screens

"The order total equals the item total plus tax" is a business rule, so the step must check the arithmetic, not only that a total is visible. Money is never compared as floats (0.1 + 0.2 !== 0.3). Everything goes through integer cents:

export function toCents(amount: number): number {
  return Math.round(amount * 100);
}

export function sumCents(amounts: readonly number[]): number {
  return amounts.reduce((total, amount) => total + toCents(amount), 0);
}

Then('the order total equals the item total plus tax', async ({ checkoutOverviewPage }) => {
  const itemTotal = await checkoutOverviewPage.itemTotal();
  const tax = await checkoutOverviewPage.tax();

  expect(toCents(await checkoutOverviewPage.total())).toBe(sumCents([itemTotal, tax]));
});

The item total step does the same against the sum of the line prices. These helpers are pure functions, so they have their own unit tests in the plain Playwright unit project.

Known defects: tracked with @fail, not skipped

SauceDemo's problem_user is broken on purpose: every product shows the same placeholder image. Instead of skipping a test, I wrote the scenario that asserts the correct behavior and tagged it @fail:

@fail
Scenario: problem_user sees a distinct image for each product
  Given I am logged in as "problem_user"
  Then every product shows its own image

playwright-bdd turns @fail into test.fail() in the generated spec. Today the scenario is reported as an expected failure. If the defect is fixed, Playwright reports it as "unexpectedly passed", which is the signal to remove the tag. The defect stays visible in every run.

The honest limitation: @fail accepts any failure. If the site is down or a selector drifts, this scenario still "passes" as an expected failure. It relies on the other UI tests to prove the site is up. A stricter version would assert the specific failure, but for one tracked defect in a public sandbox I chose the simpler mechanism and documented the trade-off in docs/KNOWN_ISSUES.md, along with the problem_user defects I verified by hand but did not automate.

Keeping the suite honest in CI

A Gherkin suite rots quietly in two ways: a step loses its definition, or someone disables a scenario "for now". The CI pipeline has a dedicated Gherkin job, which needs no browser and runs in parallel with lint and typecheck:

# Fails fast if a Gherkin step has no definition.
- run: npx bddgen
# Lint covers test.skip in TypeScript. This covers the same escape hatch in Gherkin.
- name: Reject disabled scenarios
  run: |
    if grep -rnE '(^|[[:space:]])@(skip|fixme|only)([[:space:]]|$)' features --include='*.feature'; then
      echo '::error::A scenario is disabled with @skip, @fixme or @only. Fix it or track it with @fail.'
      exit 1
    fi

The error message states the policy: fix it, or track it with @fail. Only after this and the other fast checks pass do the browser tests run, sharded in two, with blob reports merged into a single HTML report.

Takeaways

Use Gherkin where someone outside engineering will read it. Keep API, accessibility and unit tests in plain Playwright.
Run Gherkin on the Playwright runner. You keep fixtures, parallelism, traces, reports and sharding instead of rebuilding them.
Write declarative steps. Spell out values only when the variation is the business rule, and use Scenario Outlines for that.
Inject page objects and scenario state as fixtures. No globals, no shared world object.
Never skip silently: track known defects with @fail, know its limits, and let CI reject @skip, @fixme and @only.

The full project, including the API and accessibility suites, is on GitHub: github.com/tayguara/playwright-qa-showcase.

Tayguara Dias Reis is a Lead QA / SDET with 14+ years of experience and ISTQB CTFL certification. The code in this article is from playwright-qa-showcase.