EVERFI

About EVERFI

EVERFI is leading education’s evolution and redefining the way learners approach the world. The company empowers teachers, citizens, and students to improve their lives and make an impact in their communities. Their e-learning solutions activate today’s most sought-after skills in dynamic, incentive-driven online environments. EVERFI’s critical skills courses are catalyzing comprehension, retention, and behavior change while fulfilling the need for scalable education in an ever-changing world.

To succeed, EVERFI combines a deep knowledge of education, proven ability to induce knowledge and behavior change through online courseware, and the ability to develop web-based training products that meet the needs of corporate, professional, community, and K-12 educators. EVERFI specializes in evidence-based online learning products for a range of core topics. These topics include financial literacy, social and emotional learning, STEM and career readiness, conduct and culture, and health and wellness.

EVERFI’s Approach to Quality

EVERFI has developed a range of products since its founding in 2008. Each product has its own software development history, customizability, and state of quality. Over time, EVERFI began to standardize courseware development, but courses retained their own unique code that required maintenance and updates. Top management realized that this collection of individual courses blocked their ability to scale. In response, EVERFI invested in creating a new common product platform to help standardize course development and maintenance. 

In developing this platform, EVERFI leaders realized that they would have to rethink their approach to software quality. As part of their legacy course-by-course development process, they tested with a combination of automated and manual end-to-end tests. Moving to a common platform, EVERFI needed automated tests to validate behavior and uncover bugs before they rippled through the entire platform. 

To guide their approach to quality, EVERFI hired Greg Sypolt as Vice President of Quality Assurance. Greg brought extensive experience in software quality from his prior work. In addition, Greg wrote extensively on software quality and quality maturity. EVERFI realized that Greg could bring the wealth of quality experience needed to drive this platform development process to success.

The Challenge of Scaling Manual Testing

Greg saw the scale of his quality challenge immediately and realized that a single development change could ripple across a range of courses with unintended consequences, affecting tens of thousands of web pages. Manual testing would never scale in this situation. So, Greg brought in Applitools. From his experience, Greg knew that Applitools would help capture visual behavior of the finished product and code at check-in.  

Powered by Visual AI, Applitools allows EVERFI to compare rendered pages quickly and efficiently. By abstracting pixels into visual elements, Applitools identifies visually distinct changes in layout, color, and content elements, while ignoring pixel-based rendering differences that plague non-AI based visual test systems. Now, with the ability to focus comparisons on areas of expected difference and expected similarity, Applitools provides huge benefits during application development in exposing unintended changes for both new behavior and untouched code.

With Greg’s direction, the platform team implemented a model-based testing methodology using TestModeller.io combined with Cypress and Applitools. Instead of waiting for finished code, the developers validate their code check-ins. Applitools ensures that new behaviors get captured and validated across web and mobile web viewports. It also ensures that existing elements that should remain untouched have no unexpected changes.

Designing a Consistently Perfect Experience

It’s not only the EVERFI engineering team benefiting from Applitools, but also their UX team.

To aid in developing a common platform, EVERFI invested in user experience, onboarding Marianne Epstein as Director of UX design. Marianne is tasked with a mission to collaborate with engineering to deliver an unparalleled user experience. Early on, it became obvious that the biggest challenge was the opaqueness of the platform – not knowing what users are seeing because of all the ways and places screens can be dynamically generated.

“How can we improve the user experience, if we don’t actually know exactly what our users are seeing?” said Marianne. “The number of permutations is overwhelming and was impossible for the UX team to navigate. Applitools provides instant access and validation to the massive inventory of screens and the peace of mind knowing that every regression is immediately spotted across all screen variations ensuring the user experience is exactly as it was intended to be. I feel like it is a transformative tool for our UX team and how we partner with development to drive user experience.”

When the UI does not match what has been designed in Figma, Marianne’s team is able to pinpoint the screen. This lets the teammate know exactly who from engineering they need to collaborate with on the fix. As the UX team continues toward UI perfection, they’re building codeless automation using Selenium IDE with the native Applitools SDK.

EVERFI’s Teams Using Applitools

Over 45 EVERFI engineers use Applitools in their work on the new platform. The platform development has proceeded smoothly, in large part, because the component teams and the platform teams find visual problems at code check-in time. The engineering teams haven’t been surprised with visual errors during their integration tests because they exposed these issues during development.

As Marianne said, “I find so much value using Applitools in this process. As the applications get built, I have automation to capture the construction of each page. I know when page behaviors match my expectations, and I can immediately uncover unexpected differences. I cannot imagine clicking through all the pages and modals to uncover behavior repeatedly any other way.”

Applitools provides visibility to the broad customizations built into the new platform. EVERFI can test coursework migration from legacy systems and validate behavior before courses go live. The automation infrastructure lets course testing across the platform run in parallel, and Applitools provides all the visual captures to validate the platform user experience.

EVERFI’s ROI with Applitools

Prior to using Applitools, everything at EVERFI was manually tested, resulting in many hours of testing. With courses averaging around 75 pages each, it would take hours to get through manually testing a single course, and even then, you can easily miss those issues. In coming up with solutions to speed up the actual development process, Greg chose Applitools from his experience using it in the past in a previous role.

Of the EVERFI pipeline, Greg said “Today, we’re using a tool called CircleCI, which we actually leverage also as our testing grid. It really simplified in the sense of kind of pushing everything up through CircleCI. I need to deploy a node, I need to run this on Chrome, but after I run that particular test in Chrome, then I’m able to actually push out the Applitools to all the different configurations if I want to run against. Different browsers or different resolutions.”

EVERFI onboarded a proof of concept using Applitools Eyes in less than a week to prove the return on investment. One of the advantages of this onboarding of Applitools was the Ultrafast Grid, which allowed EVERFI to scale out its courses across different devices, browsers, and resolutions within minutes and in some cases even seconds. This provided a more robust coverage to test how learners are actually using the courses. The Ultrafast Grid also sped up the testing process and saved money on manual testing.

EVERFI deploys releases 26 times per year with about 10 courses every release, which is usually around 75 pages. Per Greg Sypolt, “You’re looking at, you know, between $25 to 50 thousand probably every sprint itself. So you’re looking at over a million dollars for the whole entire year from a return investment moving from a manual side of testing over courses to actually running continuous automation using visual testing.”