April 27, 2022 update: Pattern of attacker activity on GitHub.
As of 5:00 PM UTC on April 27, 2022, we are in the process of sending the final expected notifications to GitHub.com customers who had either the Heroku or Travis CI OAuth app integrations authorized in their GitHub accounts.
GitHub’s analysis of the attacker’s behavior reveals the following activities carried out on GitHub.com using stolen OAuth app tokens:
1. The attacker authenticated to the GitHub API using the stolen OAuth tokens issued to Heroku and Travis CI.
2. For most people who had the affected Heroku or Travis CI OAuth apps authorized in their GitHub accounts, the attacker listed all the user’s organizations.
3. The attacker then selectively chose targets based on the listed organizations.
4. The attacker listed the private repositories for user accounts of interest.
5. The attacker then proceeded to clone some of those private repositories.
This pattern of behavior suggests the attacker was only listing organizations in order to identify accounts to selectively target for listing and downloading private repositories. GitHub believes these attacks were highly targeted based on the available information and our analysis of the attacker behavior using the compromised OAuth tokens issued to Travis CI and Heroku.
Following this series of notifications, GitHub will have completed directly notifying each affected user for whom we were able to detect abuse using the stolen OAuth tokens.
Customers should also continue to monitor Heroku and Travis CI for updates on their own investigations into the affected OAuth applications at:
April 22, 2022 update: GitHub has sent notifications for known victims of repository listing activity using stolen OAuth app tokens.
As of 7:33 PM UTC on April 22, 2022, we’ve notified victims of this campaign whom we have identified as having repository details listed using stolen OAuth app tokens, but did NOT have repository contents downloaded.
The repository listing activity was conducted by the attacker through abuse of third-party OAuth user tokens maintained by Heroku and Travis CI. We do not believe the attacker obtained these tokens via a compromise of GitHub or its systems, because the tokens in question are not stored by GitHub in their original, usable formats which could be abused by an attacker.
The activity was conducted using the /user/repos and /orgs/{org}/repos GitHub API endpoints. Our documentation gives examples of the data returned from requests to these endpoints:
Should we identify additional customers who have been affected, we will notify those customers promptly. If you do not receive a notification email from us, that means GitHub has not identified your account as impacted by the current incident.
Customers should also continue to monitor Heroku and Travis CI for updates on their own investigations into the affected OAuth applications.
April 18, 2022 update: GitHub has sent notifications for known victims of third-party OAuth token theft.
As of 9:30 PM UTC on April 18, 2022, we’ve notified victims of this campaign whom we have identified as having repository contents downloaded by an unauthorized party through abuse of third-party OAuth user tokens maintained by Heroku and Travis CI. We do not believe the attacker obtained these tokens via a compromise of GitHub or its systems, because the tokens in question are not stored by GitHub in their original, usable formats which could be abused by an attacker.
Should we identify additional customers who have been affected, we will notify those customers promptly. If you do not receive a notification email from us, that means GitHub has not identified your account as impacted by the current incident.
Customers should also continue to monitor Heroku and Travis CI for updates on their own investigations into the affected OAuth applications.
On April 12, GitHub Security began an investigation that uncovered evidence that an attacker abused stolen OAuth user tokens issued to two third-party OAuth integrators, Heroku and Travis-CI, to download data from dozens of organizations, including npm. The applications maintained by these integrators were used by GitHub users, including GitHub itself. We do not believe the attacker obtained these tokens via a compromise of GitHub or its systems, because the tokens in question are not stored by GitHub in their original, usable formats. Following immediate investigation, we disclosed our findings to Heroku and Travis-CI on April 13 and 14; more detail is available below and we will update this blog as we learn more.
Looking across the entire GitHub platform, we have high confidence that compromised OAuth user tokens from Heroku and Travis-CI-maintained OAuth applications were stolen and abused to download private repositories belonging to dozens of victim organizations that were using these apps. Our analysis of other behavior by the threat actor suggests that the actors may be mining the downloaded private repository contents, to which the stolen OAuth token had access, for secrets that could be used to pivot into other infrastructure.
Known-affected OAuth applications as of April 15, 2022:
- Heroku Dashboard (ID: 145909)
- Heroku Dashboard (ID: 628778)
- Heroku Dashboard – Preview (ID: 313468)
- Heroku Dashboard – Classic (ID: 363831)
- Travis CI (ID: 9216)
We are sharing this today as we believe the attacks may be ongoing and action is required for customers to protect themselves.
The initial detection related to this campaign occurred on April 12 when GitHub Security identified unauthorized access to our npm production infrastructure using a compromised AWS API key. Based on subsequent analysis, we believe this API key was obtained by the attacker when they downloaded a set of private npm repositories using a stolen OAuth token from one of the two affected third-party OAuth applications described above. Upon discovering the broader theft of third-party OAuth tokens not stored by GitHub or npm on the evening of April 13, we immediately took action to protect GitHub and npm by revoking tokens associated with GitHub and npm’s internal use of these compromised applications.
We believe that the two impacts to npm are unauthorized access to, and downloading of, the private repositories in the npm organization on GitHub.com and potential access to the npm packages as they exist in AWS S3 storage. At this point, we assess that the attacker did not modify any packages or gain access to any user account data or credentials. We are still working to understand whether the attacker viewed or downloaded private packages. npm uses completely separate infrastructure from GitHub.com; GitHub was not affected in this original attack. Though investigation continues, we have found no evidence that other GitHub-owned private repos were cloned by the attacker using stolen third-party OAuth tokens.
How GitHub responded to protect users of GitHub.com
Once GitHub identified stolen third-party OAuth tokens affecting GitHub users, GitHub took immediate steps to respond and protect users. GitHub contacted Heroku and Travis-CI to request that they initiate their own security investigations, revoke all OAuth user tokens associated with the affected applications, and begin work to notify their own users.
GitHub remains closely engaged with both organizations in an effort to assist their investigation and recovery efforts, and better protect shared customers.
What GitHub customers and organizations need to know
GitHub is currently working to identify and notify all of the known-affected victim users and organizations that we discovered through our analysis across GitHub.com. These customers will receive a notification email from GitHub with additional details and next steps to assist in their own response within the next 72 hours.
If you do not receive a notification, you and/or your organization have not been identified as affected. GitHub will continue to notify any additional affected users or organizations as they are identified. You should, however, periodically review what OAuth applications you’ve authorized or are authorized to access your organization and prune anything that’s no longer needed. You can also review your organization audit logs and user account security logs for unexpected or anomalous activity.
If you have questions or concerns
If you have questions or need assistance regarding affected OAuth applications maintained by Heroku, please reach out to Salesforce / Heroku security and support at help.heroku.com, and monitor the Salesforce Trust site for additional updates.
If you have questions or need assistance regarding affected OAuth applications maintained by Travis CI, please reach out to compliance@travis-ci.com.
Customers who are directly contacted by GitHub regarding this issue are welcome to contact us according to directions in the notification you received.
For other questions regarding GitHub and npm you can contact GitHub Support.
Conclusion
The security and trustworthiness of GitHub, npm, and the broader developer ecosystem is our highest priority. Our investigation is ongoing, and we will update this blog, and our communications with affected customers, as we learn more.
Written by
Mike Hanley is the Chief Security Officer and SVP of Engineering at GitHub. Prior to GitHub, Mike was the Vice President of Security at Duo Security, where he built and led the security research, development, and operations functions. After Duo’s acquisition by Cisco for $2.35 billion in 2018, Mike led the transformation of Cisco’s cloud security framework and later served as CISO for the company. Mike also spent several years at CERT/CC as a Senior Member of the Technical Staff and security researcher focused on applied R&D programs for the US Department of Defense and the Intelligence Community.
When he’s not talking about security at GitHub, Mike can be found enjoying Ann Arbor, MI with his wife and eight kids.
