Paramount Improves Developer Productivity 100X With LaunchDarkly

Before LaunchDarkly:

Deploy/release twice a month
Took up to a week to fix bugs

After:

Deploy 6-7 times a day
Fix bugs in a day

Paramount, formerly ViacomCBS, is a leading global media company. Its portfolio of network, studio, and streaming brands—including CBS, Showtime Networks, Paramount+, MTV, Nickelodeon, BET, and Comedy Central—reaches billions of people across more than 180 countries.

Audiences want to access their favorite digital content from any location and device. And they want it instantly. To deliver on this demand, a lot must happen behind the scenes. Among those doing this critical work is the Paramount Tech Content Engineering team.

Developers on this team maintain an internal platform that supports content processing and distribution. For example, when suppliers send over new movies, shows, and other media, operators at Paramount leverage various services in the application platform in preparing this content for its final destination. Though invisible to the outside world, the platform is critical to the customer experience.

At one point, as many as 11 full-stack software engineering teams were building in the platform, shipping roughly 17 applications therein. Coordinating the releases for these teams was Dan Skaggs, Technical Director, Content Engineering, Paramount.

The pain of hefty releases and long-lived feature branches

As Technical Director, Dan shapes the product roadmap, contributes to developer relations, and works on strategic technology initiatives.

Before leveraging LaunchDarkly, Dan’s time was often consumed by the unceasing task of approving code changes—an experience that represented, on a small scale, a larger bottleneck clogging the whole software delivery process.

At the time, deployments were bound to releases. Whenever an engineer put a code artifact on a production server, it was simultaneously released to all production traffic. Given the inherent risks of marrying deploy and release, every code change to an application in the platform had to endure rigorous testing. After such vetting, it could then be merged with the master branch and, ultimately, pushed into production.

Every two weeks, all 11 teams would deploy their applications on a dedicated release day. The 48 hours preceding the launch often saw stressful branch merges, QA regression tests, and last-minute fixes. And if, for example, QA spotted a bug in one team’s application, the other ten teams had to wait to ship their changes until the bug was eradicated.

Moreover, if the bug went undetected and wound up in production, it took up to a week to deploy a hotfix in some cases (for non-critical bugs). This model impacted developer productivity and created challenges.

“There were significant points of friction in the software delivery process. The ‘process’ was slowing our engineers down. For me, LaunchDarkly was about removing friction to get the process working for developers, not against them.” –Dan Skaggs, Technical Director, Content Engineering, Paramount

Dan governed releases for every team building applications in the platform. He was keen to transfer this deed of ownership, so that each team could oversee their own code deployments and feature releases.

A safety net enabling on-demand releases

Dan realized that feature management, a new class of software development tools and techniques anchored in feature flags, could remove the pain from his team’s software delivery process.

Multiple teams in Content Engineering now use LaunchDarkly feature flags to deploy 14 applications on-demand. LaunchDarkly serves as a powerful safety net by allowing Paramount Tech to:

Deploy unfinished features to a production environment without exposing them to production traffic (i.e., decouple deploy and release); and
Remotely disable flags tied to broken features at runtime (i.e., hit a kill switch) without needing to redeploy an application.

Additionally, LaunchDarkly gives the team the control to unveil features to designated users, segments, and environments. For example, a developer can wrap a minimum viable code change in a feature flag, deploy it to a test environment, and then move on to something else. Meanwhile, QA engineers with access to that environment can review the code when they get around to it.

LaunchDarkly thus furnishes the safeguards for “asynchronous QA testing.” Decoupling when engineers merge code from when QA tests that code—and from when product owners release that code to users—has increased productivity by a wide margin.

“The ability to ship and merge code to environments safely without wincing every time we hit the ‘deploy’ button has been huge for us.” —Dan Skaggs

In the wake of enabling trunk-based development and reducing inter-team dependencies, Paramount Tech engineers now deploy to the Platform 6-7 times a day (a 100x improvement). What’s more, in using LaunchDarkly feature flags as kill switches, they’re able to resolve production issues in one day rather than seven.

Feature management: ‘One of the three legs of the stool’

This new process erased the friction in the software delivery process and cleared the bottlenecks hurting developer productivity and satisfaction.

The Paramount Tech Content Engineering team continues to epitomize advanced development, attracting top engineers from within and outside the company. No doubt, feature management has helped propel Paramount on this journey to modern development.