In a perfect world, all of your codebases would have a single canonical location. But in the real world, developers often need to maintain the same code in two separate locations. For example, you might need to use the same code in both an open source application and an internal or proprietary application.
In 2019, we ran into that problem at FullStory. If we tried to keep this code synchronized manually, we would have to remember to update the open source repo whenever we release a new version of the snippet. This would require two separate teams (the team who maintains the snippet source code, and those who maintain the open source repo) to coordinate their work. And if you frown at that, so did we. We wanted an automated way to keep the open source code in sync with the snippet code inside the closed-source repo.
To solve the problem, we used GitHub’s native CI/CD and automation workflow tool, GitHub Actions to update our open source code by downloading the snippet code from our API and automatically creating a pull request with any changes. With this architecture, we’ve detected and merged several updates since it came online in December 2019, ensuring updated code for our Browser SDK users and reducing maintenance overhead.
Instead of a cross-team, manual, and error-prone process, we now have a secure, automated solution. This enables each team to operate independently, helps improve our productivity, and simplifies the process of maintaining the open source project for our Browser SDK while ensuring that our NPM package consumers always have the latest version of our snippet to take advantage of any new features we release. Moreover, we now have a pattern for syncing our closed source code to the open source repos without depending on any human intervention. Here’s how we did it.
Why we needed to synchronize our repositories
We strive to create an easy implementation path for developers so that they can get FullStory running on their sites quickly. Our customers initiate our service by copying and pasting a JavaScript snippet into their website. This works very well for traditional HTML sites, and is also pretty simple via a Content Management System (CMS), Tag Manager, or eCommerce platform.
However, if the website is a Single Page Application (SPA) built using frameworks like React, Angular, Vue, etc., this path is less than ideal. We realized that we needed to provide an idiomatic way to add the recording snippet and our client JavaScript APIs to SPAs. So, we built an open source NPM package: the FullStory Browser SDK.
Because we provide installation instructions both in our web app and also programmatically in the SDK, the core snippet code is now hosted in two places: in our closed source repository and on an open source repository that contains the code distributed via NPM. We needed a good way for our internal microservices as well as the open source consumers to share a single source of truth without being tightly coupled. We initially considered a “push” model where our build system would push any snippet updates to the open source repo and create a PR. The drawback is that it would require our build system to maintain secrets and understand the structure of our open source repo, and to maintain a repository that it does not own. These added complexities and dependencies in the build system are not ideal.
How to build a repo synchronization Action, step by step
Eventually, we came up with a solution that relies on two technologies:
A new public API that we host which returns the snippet code
GitHub Actions
In adding GitHub Actions to our open source repo, it was trivial to use the built-in cron function to pull the latest snippet code that we expose via the public API endpoint. All we needed was to create a new microservice to serve the snippet for a variety of different clients to consume. Moreover, GitHub Actions has built-in git operations and GitHub management APIs, which make it easy to automatically update the snippet file, check out a new branch, open a PR, and assign it to the correct reviewers.
A tour inside the `Snippet-Sync-Job`
Step 1. Building the snippet service and exposing a public API endpoint
In order to be able to pull the latest snippet from our closed-source repos, we needed to expose a public API serving the snippet code. It serves up either a “core” or “ES Module” version of the snippet based on the URL parameters passed in. And voila! We have a way to pull the latest snippet. It’s worth noting that the service is used by various other services we host internally via gRPC as well.
Step 2. Adding Github Actions to our open source repo
To enable GitHub actions, first create a `main.yaml` file inside `.github/workflows` folder. And with just a few lines, we can define a cron job and pull for updates every 24 hours.
on:
schedule:
- cron: '0 0 * * *' # every 24 hours
jobs:
sync:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v2
- name: npm ci
run: npm ci
working-directory: ./.github/actions/sync-snippet-action
- name: Sync snippet
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
SNIPPET_ENDPOINT: https://api.fullstory.com/code/v1/snippet?type=esm
uses: ./.github/actions/sync-snippet-action
GitHub Actions make it trivial to build authentication into a workflow by providing a GITHUB_TOKEN secret and several other default environment variables. Along with the snippet API endpoint, we now have all we need to proceed.
// accessing environment variables in .github/actions/sync-snippet-action
const {
SNIPPET_ENDPOINT,
GITHUB_REPOSITORY,
GITHUB_TOKEN,
GITHUB_SHA,
} = process.env;
// parsing the owner and repo name from GITHUB_REPOSITORY
const [owner, repo] = GITHUB_REPOSITORY.split(‘/’);
const repoInfo = { owner, repo };
We will use all the imported environment variables and the parsed `repoInfo` later on.
Step 3. Check for snippet updates and existing open PRs
In the `Sync snippet` step, we’ve already checked out the latest `main` branch. We first use axios to pull the latest snippet text via the REST API, then compare the hash of the latest snippet text with the one we have on the file system of the checked out repo. We continue to the next steps only when we find the mismatch in the hash values, meaning a snippet update has been detected. If we determine that an update is needed, we initialize an octokit client, which is provided by the @actions/github package. The client is authenticated using the `GITHUB_TOKEN` env var declared in the main.yaml file.
We then get the list of current open PRs and check to see if the same PR has already been created. Since we run the sync job every 24 hours, it’s possible that an existing PR has been opened but not yet merged, in which case we do not want to open another PR. We achieve this by simply looking for a PR created by the Github Actions bot, and that the title is a constant:
// “The FullStory snippet has been updated”
const SNIPPET_PATH = 'src/snippet.js';
const PR_TITLE = 'The FullStory snippet has been updated';
const octokit = new github.GitHub(GITHUB_TOKEN);
const openPRs = await octokit.pulls.list({
...repoInfo,
state: ‘open’,
});
console.log(‘checking for an on open snippet sync PR’);
// Look for PR created by github-actions[bot] with the same title
const existingPR = openPRs.data.filter(pr => pr.title === PR_TITLE && pr.user.login === ‘github-actions[bot]’);
if (existingPR.length > 0) {
core.setFailed(`There is already an open PR for snippet synchronization. Please close or merge this PR: ${existingPR[0].html_url}`);
return;
}
Step 4. Use Github `octokit` to obtain the tree object
The next step is to update the `snippet.js` file and create a PR.
In order to programmatically update the file and open a PR, we needed to get deeper into what git calls the Plumbing commands. If you are not familiar with the inner workings of git, we recommend reading the “Plumbing and Porcelain” section of the Git documentation before continuing.
The Tree Object is the data structure that holds the relationships between your files (blobs), similar to a simplified UNIX file system. We need to first create a new `Tree Object` with the new contents of the snippet code. And then use the newly created tree to checkout a branch and open a new PR.
To do so we need to first get the current commit. Back in step one we checked out `main` and obtained the current commit sha from `process.env` in Step 1. With the commit sha we can now get the current commit via `octokit.git.getCommit`, which contains the hash of the `tree object`: `tree sha`. With the `tree sha` we can then get the tree object via `octokit.git.getTree` with a recursive parameter.
Once we got the tree object, we then found the “tree node”(`srcTree`) with our known `SNIPPET_PATH`. It’s a tree node that represents the `snippet.js` file.
console.log("getting source tree from main");
const getCommitResponse = await octokit.git.getCommit({
...repoInfo,
commit_sha: GITHUB_SHA,
});
const getTreeResponse = await octokit.git.getTree({
...repoInfo,
tree_sha: getCommitResponse.data.tree.sha,
recursive: 1,
});
const srcTree = getTreeResponse.data.tree.find(el => el.path === SNIPPET_PATH);
Step 5. Create a new `Tree` with modified content
Let’s summarize what we have so far:
We have the new snippet from the public API hosted by FullStory, in string format
We have the source “tree node” that holds the content of the snippet from the current commit on the `main` branch
The next thing we need is to create a new `tree object` with the updated snippet code. To do so we create a new tree with `octokit.git.createTree` and specify the updated object: our new snippet text. Remember that we’ve retrieved the original tree object recursively, meaning the tree object contains references to all the files with their nested paths. The new tree will contain all the information in the original tree, but update only what we need: the `snippet.js` file.
console.log('creating updated source tree with new snippet file');
const createTreeResponse = await octokit.git.createTree({
...repoInfo,
tree: [{
path: SNIPPET_PATH,
content: remoteSnippetText,
mode: ‘100644’,
type: ‘blob’,
base_tree: srcTree.sha,
},
...getTreeResponse.data.tree.filter(el => el.type !== ‘tree’ && el.path !== SNIPPET_PATH)]
});
Step 6. Commit the change to a new branch
Now that we have a new `tree object` with updated snippet text, it’s simply a matter of committing the change and opening a PR.
With the current commit as parent, we create a new commit using `octokit.git.createCommit` and pass in the created tree’s `tree sha`, then create a new reference (branch) with the name: `snippetbot/updated-snippet-${Date.now()}` using `octokit.git.createRef`, providing it the commit sha we just created:
console.log('committing new snippet file');
const commitResponse = await octokit.git.createCommit({
...repoInfo,
message: `updated ${SNIPPET_PATH}`,
tree: createTreeResponse.data.sha,
parents: [GITHUB_SHA],
});
const branchName = `refs/heads/snippetbot/updated-snippet-${Date.now()}`;
console.log(`creating new branch named ${branchName}`);
await octokit.git.createRef({
...repoInfo,
ref: branchName,
sha: commitResponse.data.sha,
});
Step 7. Open a PR and assign it to the maintainers
The final step is to create a PR via `octokit.pulls.create` and assign it to the correct maintainers of the repo via `octokit.issues.addAssignees`.
In order to get the correct assignees, we maintain a `MAINTAINERS.json` file that contains all the maintainer’s GitHub handles for this repo, so the correct team members will be notified to review and merge the new PR.
console.log(`creating PR for branch ${branchName}`);
const base = GITHUB_REF.split('/').pop();
const prResponse = await octokit.pulls.create({
...repoInfo,
title: PR_TITLE,
head: branchName,
base,
});
console.log(‘assigning PR to reviewers’);
const maintainers = JSON.parse(fs.readFileSync(‘./MAINTAINERS.json’));
await octokit.issues.addAssignees({
...repoInfo,
issue_number: prResponse.data.number,
assignees: maintainers,
});
console.log(`created PR: ${prResponse.data.html_url}`);
Results and closing thoughts
You can check out our solution and learn more about the FullStory Browser SDK in our GitHub repo. Your own solution may take a different form, but hopefully what we’ve done here will be enough to get you started. We’ll likely be using this pattern ourselves again in the future.