Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Can't install / update gh due to expired GPG key? #9569

Open
andyfeller opened this issue Sep 5, 2024 · 18 comments
Open

Can't install / update gh due to expired GPG key? #9569

andyfeller opened this issue Sep 5, 2024 · 18 comments
Labels
bug Something isn't working p1 Affects a large population and inhibits work

Comments

@andyfeller
Copy link
Contributor

andyfeller commented Sep 5, 2024

What's going to happen?

On Friday 6th September at approximately 11:17 am UTC, the GPG key used to verify our .deb and .rpm package repository contents will expire.

This will impact apt update and dnf install / update usage, which will look something like the following:

W: GPG error: https://cli.github.com/packages stable InRelease: The following signatures were invalid: EXPKEYSIG 23F3D4EA75716059 GitHub CLI <opensource+cli@github.com>

or

error: Verifying a signature using certificate 2C6106201985B60E6C7AC87323F3D4EA75716059 (GitHub CLI <opensource+cli@github.com>):
  1. Certificate 23F3D4EA75716059 invalid: certificate is not alive
      because: The primary key is not live
      because: Expired on 2024-09-06T11:17:19Z
  2. Key 23F3D4EA75716059 invalid: key is not alive
      because: The primary key is not live
      because: Expired on 2024-09-06T11:17:19Z

If you have not previously installed the GitHub CLI via our package repositories, there should be no impact for you.

What are we doing about it?

We have extended the expiration dates on our key, which is available at https://cli.github.com/packages/githubcli-archive-keyring.gpg and on the following keyservers with the 2C6106201985B60E6C7AC87323F3D4EA75716059 ID:

  1. keys.openpgp.org
  2. keyserver.ubuntu.com

What do you need to do about it?

Important

This section will be updated as more information becomes available.

You will need to get this new key from one of the sources mentioned ☝️ above, which be achieved in multiple ways depending on your setup.

Installed via apt

If you followed our apt instructions, you should be able to run them again. Depending on when you ran these instructions, you may have placed the old keyring in a different location to the current instructions; the script below accounts for this difference by using the file location you previously used.

# Check for wget, if not installed, install it
(type -p wget >/dev/null || (sudo apt update && sudo apt-get install wget -y)) \
    && sudo mkdir -p -m 755 /etc/apt/keyrings

# Set keyring path based on existence of /usr/share/keyrings/githubcli-archive-keyring.gpg
# If it is in the old location, use that, otherwise always use the new location.
if [ -f /usr/share/keyrings/githubcli-archive-keyring.gpg ]; then
    keyring_path="/usr/share/keyrings/githubcli-archive-keyring.gpg"
else
    keyring_path="/etc/apt/keyrings/githubcli-archive-keyring.gpg"
fi

echo "replacing keyring at ${keyring_path}"

# Download and set up the keyring
wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee "$keyring_path" > /dev/null \
    && sudo chmod go+r "$keyring_path"

# Idempotently add the GitHub CLI repository as an apt source
echo "deb [arch=$(dpkg --print-architecture) signed-by=$keyring_path] https://cli.github.com/packages stable main" | sudo tee /etc/apt/sources.list.d/github-cli.list > /dev/null

# Update the package lists, which should now pass
sudo apt update

The important line is:

wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee "$keyring_path" > /dev/null \
    && sudo chmod go+r "$keyring_path"

Which will fetch the new key and overwrite the expired one. From this point you should be able to apt update successfully, and apt upgrade gh successfully (once we have created a new release).

Docker build failing?

If your Docker build is failing as in aws/aws-codebuild-docker-images#739, most likely there is a layer in your image that previously added our package repository, and you have a later layer running apt update.

If you have control of the offending layer, you can re-build it so that it pulls the latest key. If you're searching for the layer, it most likely has a line that looks like wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null.

If you don't have control of the offending layer, you have two options:

You can add a new layer before the apt update that fetches the new key:

RUN wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee /etc/apt/keyrings/githubcli-archive-keyring.gpg > /dev/null \
    && chmod go+r /etc/apt/keyrings/githubcli-archive-keyring.gpg

Or, if you aren't using gh and it just happens to be in the base image you are using, you can remove the repository so that apt update no longer tries to verify it:

sudo rm /etc/apt/sources.list.d/github-cli.list

Installed via dnf

If you followed our dnf instructions, you will need to remove the expired key from the rpm cache before downloading the new key.

Option 1: You are comfortable reinstalling gh

  1. Find and remove the old key:

    sudo rpm -qa gpg-pubkey
    sudo rpm -e pubkey-xxxx-yyyy

    Under most circumstances we expect the key to be named gpg-pubkey-75716059-63172e8a but you can otherwise identify the correct key by checking whether a key's Packager is is "opensource+cli@github.com" e.g.

    sudo rpm -qi gpg-pubkey-75716059-63172e8a
    Name        : gpg-pubkey
    Version     : 75716059
    Release     : 63172e8a
    Architecture: (none)
    Install Date: Thu 05 Sep 2024 02:39:41 PM EDT
    Group       : Public Keys
    Size        : 0
    License     : pubkey
    Signature   : (none)
    Source RPM  : (none)
    Build Date  : Tue 06 Sep 2022 07:27:06 AM EDT
    Build Host  : localhost
    Packager    : GitHub CLI <opensource+cli@github.com>
    Summary     : GitHub CLI <opensource+cli@github.com> public key
    ...
    
  2. Removing and reinstalling gh:

    sudo dnf remove gh
    sudo dnf install gh

Option 2: You don't want to reinstall gh

In #6175 (comment), there was another suggested approach to download the keyring, create an .asc file from it, and import that into rpm:

  1. Find and remove the old key like Option 1:

    sudo rpm -qa gpg-pubkey
    sudo rpm -e gpg-pubkey-xxxx-yyyy
  2. Download new key and import into rpm keyring:

    curl -fsSL -o /var/tmp/githubcli-archive-keyring.gpg https://cli.github.com/packages/githubcli-archive-keyring.gpg
    gpg --keyring /var/tmp/githubcli-archive-keyring.gpg --no-default-keyring --export --armor > /var/tmp/githubcli-archive-keyring.asc
    sudo rpm --import /var/tmp/githubcli-archive-keyring.asc

Potential issue with Fedora

During our testing in Fedora 38, we encountered a sporadic error we believe will be fixed when a new release is created:

  1. Certificate invalid: policy violation
      because: No binding signature at time
  2. Certificate has no valid binding signature as of the signature's creation time, but is valid now.  The certificate has probably been stripped or minimized.

This appears to be due to Fedora 38 enforcing a strict policy about the contents of the public key. Some contents are stripped from the key when extending the expiration. We were only able to reproduce this error using the script above, and not when removing and reinstalling gh.

Again, we believe that this will be resolved when we do a release, and the rpm package repository contents are re-signed with the new, extended expiration private key.

More reading: rpm-software-management/rpm-sequoia#46

Obtained the key from a keyserver

You can fetch the new key by running:

gpg --keyserver keyserver.ubuntu.com --recv-keys 2C6106201985B60E6C7AC87323F3D4EA75716059

Obtained the key using apt-key

You can fetch the new key by running:

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 2C6106201985B60E6C7AC87323F3D4EA75716059

What are we going to do next?

  1. We want to work with our community to devise smoother package management experience

    We have created Ensure a smoother GPG key update process for September 2026 expiration #9572 to track improvements for how the GitHub CLI keys can be distributed for a smoother, native package management experience.

    If you or someone you know has experience with managing apt and rpm repository keys in OSS, we would love to hear from you on the aforementioned issue.

  2. We will address issues raised by the community and GitHub support

    The information here is our best effort to account for the myriad ways that gh is installed, however it isn't definitive.

    If you run into any problems, please follow up here.

  3. We are going to ensure future releases with new key work as expected

    The information captured here is to ensure past releases, which were signed by our key are still usable.

    We plan to cut a new release with the new key on Fri Sep 6th to ensure full confidence that future releases work seamlessly.

How did this happen?

Since the last time this key expired, the entire GitHub CLI team has changed, resulting in a loss of institutional knowledge. Unfortunately, the current team was unaware of the timebomb in this part of our release process.

Final Notes

Firstly, thanks to @kdarndt for raising awareness to these concerns in #9562!

Secondly, we want to apologise for the inconvenience this may have caused you. Hopefully through #9572 we can avoid it happening again.

Finally, thank you for your patience and understanding along with any constructive insights you can share.

@andyfeller andyfeller pinned this issue Sep 5, 2024
@cliAutomation cliAutomation added the needs-triage needs to be reviewed label Sep 5, 2024
@BagToad BagToad removed the needs-triage needs to be reviewed label Sep 5, 2024
@williammartin williammartin added bug Something isn't working p1 Affects a large population and inhibits work labels Sep 6, 2024
andyfeller added a commit that referenced this issue Sep 6, 2024
Relates #9569

Having been 2 years since the GitHub CLI changed GPG keys used to sign our releases, it no longer seems relevant to keep these notes in our installation docs as they are confusing to the uninitiated.
andyfeller added a commit that referenced this issue Sep 6, 2024
Relates #9569

Updates notes from older 2 year GPG ID change to redirect users in case of GPG errors to recent issue.
@rrthomas
Copy link

rrthomas commented Sep 6, 2024

Note that an earlier version of the instructions used the location /usr/share/keyrings instead of /etc/apt/keyrings in the sources.list.d file, so I had to update that to make it work with the above update instructions, and remove the old keyring file from /usr/share/keyrings.

Alternatively, one could of course download the updated key to /usr/share/keyrings, but we don't really want to pollute /usr with non-packaged files!

@andyfeller
Copy link
Contributor Author

andyfeller commented Sep 6, 2024

Note that an earlier version of the instructions used the location /usr/share/keyrings instead of /etc/apt/keyrings in the sources.list.d file, so I had to update that to make it work with the above update instructions, and remove the old keyring file from /usr/share/keyrings.

Alternatively, one could of course download the updated key to /usr/share/keyrings, but we don't really want to pollute /usr with non-packaged files!

@rrthomas : Great call out! 🙇 This change originated in #8693 back in early Feb 2024 for anyone wanting to see the older documentation changes and understand the motivations.

UPDATE: The instructions above have been enhanced to account for this older approach 👏

@dcaruso-pathpoint
Copy link

NB: if you can't run the apt update to install wget because of the problem we are all trying to solve you can use curl to

curl -sS https://cli.github.com/packages/githubcli-archive-keyring.gpg -o
/etc/apt/keyrings/githubcli-archive-keyring.gpg

@sozelfist
Copy link

I encountered an issue with my expired GPG key related to GitHub CLI today. Thankfully, @andyfeller's solutions worked for me.

@leifmadsen
Copy link

Slight issue in the Fedora instructions:

Current:

Option 1: You are comfortable reinstalling gh

Find and remove the old key:

rpm -qa gpg-pubkey
rpm -e pubkey-xxxx-yyyy

Suggest:

sudo rpm -e gpg-pubkey-xxxx-yyyy

Also in --> Option 2: You don't want to reinstall gh

@williammartin
Copy link
Member

Thanks @leifmadsen, updated.

@leifmadsen
Copy link

Thanks @leifmadsen, updated.

Sorry, can you also adjust this?

sudo rpm -e pubkey-xxxx-yyyy

-->

sudo rpm -e gpg-pubkey-xxxx-yyyy

RichUnibuddy added a commit to RichUnibuddy/github-cli-orb that referenced this issue Sep 11, 2024
This is a proposed work around to the expired GPG issue with the GitHub CLI packages - cli/cli#9569.

The better fix is to support new keys, but this works and should be relatively safe as we know and mostly trust the source of the deb. 

It's a big improvement on the orb being broken as it is today.
@venkatrahul-software-development

Thanks for the help @andyfeller and @kdarndt. This worked like magic. I followed "Installed via apt" for my RaspberryPi

@Jmzp
Copy link

Jmzp commented Sep 11, 2024

Thanks for the help @andyfeller

These two blocks solve the problem for me

if [ -f /usr/share/keyrings/githubcli-archive-keyring.gpg ]; then
    keyring_path="/usr/share/keyrings/githubcli-archive-keyring.gpg"
else
    keyring_path="/etc/apt/keyrings/githubcli-archive-keyring.gpg"
fi

# Download and set up the keyring
wget -qO- https://cli.github.com/packages/githubcli-archive-keyring.gpg | sudo tee "$keyring_path" > /dev/null \
    && sudo chmod go+r "$keyring_path"

@Eliezerubeda

This comment was marked as spam.

@Jeremiegmoore

This comment has been minimized.

@Jeremiegmoore

This comment has been minimized.

izumin5210 pushed a commit to izumin5210/dotfiles that referenced this issue Sep 15, 2024
This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [cli/cli](https://redirect.github.com/cli/cli) | minor | `v2.55.0` ->
`v2.56.0` |

---

### Release Notes

<details>
<summary>cli/cli (cli/cli)</summary>

###
[`v2.56.0`](https://redirect.github.com/cli/cli/releases/tag/v2.56.0):
GitHub CLI 2.56.0

[Compare
Source](https://redirect.github.com/cli/cli/compare/v2.55.0...v2.56.0)

#### Important note about renewed GPG key

The Debian and RedHat releases have been signed with a new GPG key. If
you are experiencing issues updating your `.deb` or `.rpm` packages,
please read
[cli/cli#9569](https://redirect.github.com/cli/cli/issues/9569).

#### What's Changed

- Always print URL scheme to stdout by
[@&#8203;heaths](https://redirect.github.com/heaths) in
[cli/cli#9471
- Quote repo names consistently in `gh repo sync` stdout by
[@&#8203;muzimuzhi](https://redirect.github.com/muzimuzhi) in
[cli/cli#9491
- Fetch bundle from OCI registry for verify by
[@&#8203;ejahnGithub](https://redirect.github.com/ejahnGithub) in
[cli/cli#9421
- Remove `Internal` from `gh repo create` prompt when owner is not an
org by [@&#8203;jtmcg](https://redirect.github.com/jtmcg) in
[cli/cli#9465
- Drop surplus trailing space char in flag names in web by
[@&#8203;muzimuzhi](https://redirect.github.com/muzimuzhi) in
[cli/cli#9495
- fix the trimming of log filenames for `gh run view` by
[@&#8203;benebsiny](https://redirect.github.com/benebsiny) in
[cli/cli#9482
- "offline" verification using the bundle of attestations without any
additional handling of the file by
[@&#8203;aryanbhosale](https://redirect.github.com/aryanbhosale) in
[cli/cli#9523
- build(deps): bump actions/attest-build-provenance from 1.4.1 to 1.4.2
by [@&#8203;dependabot](https://redirect.github.com/dependabot) in
[cli/cli#9518
- Fix doc typo for `repo sync` by
[@&#8203;muzimuzhi](https://redirect.github.com/muzimuzhi) in
[cli/cli#9509
- Correct the help message for -F by
[@&#8203;Goooler](https://redirect.github.com/Goooler) in
[cli/cli#9525
- chore: fix some function names by
[@&#8203;crystalstall](https://redirect.github.com/crystalstall) in
[cli/cli#9555
- verify 2nd artifact without swapping order by
[@&#8203;aryanbhosale](https://redirect.github.com/aryanbhosale) in
[cli/cli#9532
- `gh attestation verify` handles empty JSONL files by
[@&#8203;malancas](https://redirect.github.com/malancas) in
[cli/cli#9541
- Enhance Linux installation docs to redirect users to GPG renewal
issue, better troubleshooting support by
[@&#8203;andyfeller](https://redirect.github.com/andyfeller) in
[cli/cli#9573
- Upgrade sigstore-go to v0.6.1 by
[@&#8203;codysoyland](https://redirect.github.com/codysoyland) in
[cli/cli#9566
- Check for nil values to prevent nil dereference panic by
[@&#8203;codysoyland](https://redirect.github.com/codysoyland) in
[cli/cli#9578
- build(deps): bump actions/attest-build-provenance from 1.4.2 to 1.4.3
by [@&#8203;dependabot](https://redirect.github.com/dependabot) in
[cli/cli#9575

#### New Contributors

- [@&#8203;aryanbhosale](https://redirect.github.com/aryanbhosale) made
their first contribution in
[cli/cli#9523
- [@&#8203;Goooler](https://redirect.github.com/Goooler) made their
first contribution in
[cli/cli#9525
- [@&#8203;crystalstall](https://redirect.github.com/crystalstall) made
their first contribution in
[cli/cli#9555

**Full Changelog**: cli/cli@v2.55.0...v2.56.0

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you
are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR was generated by [Mend Renovate](https://mend.io/renovate/).
View the [repository job
log](https://developer.mend.io/github/izumin5210/dotfiles).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOC43NC4xIiwidXBkYXRlZEluVmVyIjoiMzguNzQuMSIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOltdfQ==-->

---------

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Co-authored-by: izumin5210-update-aqua-checksum[bot] <169593670+izumin5210-update-aqua-checksum[bot]@users.noreply.github.com>
@pirafrank
Copy link

Hi, I have come across this issue and adding up my 2c hoping they will be helpful as a one-stop solution for future users reading this thread.

Solution is a grab-and-go for Debian based distros.

keypath=$(cat /etc/apt/sources.list.d/github-cli.list | grep -oE '/[^]]*' | head -n1)
cat "$keypath"
sudo -E rm -rf "$keypath"
curl -sS https://cli.github.com/packages/githubcli-archive-keyring.gpg -o /tmp/githubcli-archive-keyring.gpg
sudo mv /tmp/githubcli-archive-keyring.gpg /etc/apt/keyrings/githubcli-archive-keyring.gpg
sudo -E sed -i 's,/usr/share/keyrings,/etc/apt/keyrings,g' /etc/apt/sources.list.d/github-cli.list

@williammartin
Copy link
Member

@pirafrank that looks good to me. Just making sure I understand correctly, this is equivalent to the script provided in the original issue description except that it will also update the signed-by to the new location (/etc/apt/keyrings)?

@pirafrank
Copy link

@williammartin yes, and I think it's a more straightforward solution (minus the wget check). If you got the new keypath it'll do nothing.

@tpalli
Copy link

tpalli commented Sep 17, 2024

In the dnf steps, option 1 (reinstall) will not work, it will keep complaining even if you remove the key from rpm. I was required to go with option 2 to fetch key and feed it for rpm and after this dnf installation worked. Problem is though that I did this already with last update and now I had to do this again ... will this be happening with each gh update?

@williammartin
Copy link
Member

@tpalli thanks for letting us know. Option 1 worked for us during our investigation but it's good to know that there might be issues.

Problem is though that I did this already with last update and now I had to do this again ... will this be happening with each gh update?

I wonder if your problem with Option 1 relates to this. You should only need to do this once. After you get the new key and rpm is using that, things should work consistently. If it happens again, please drop by with logs of what you see and the steps you have taken. Cheers.

@Hamzah591

This comment was marked as spam.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working p1 Affects a large population and inhibits work
Projects
None yet
Development

No branches or pull requests

16 participants