Skip to main content

Enterprise Server 3.14 release notes

November 07, 2024

3.14.3: Security fixes

  • Elasticsearch packages have been updated to the latest security versions.

  • Packages have been updated to the latest security version.

  • HIGH: An attacker could bypass SAML single sign-on (SSO) authentication with the optional encrypted assertions feature, allowing unauthorized provisioning of users and access to the instance, by exploiting an improper verification of cryptographic signatures vulnerability in GitHub Enterprise Server. This is a follow up fix for CVE-2024-9487 to further harden the encrypted assertions feature against this type of attack. Please note that encrypted assertions are not enabled by default. Instances not utilizing SAML SSO, or utilizing SAML SSO authentication without encrypted assertions, are not impacted. Additionally, an attacker would require direct network access as well as a signed SAML response or metadata document to exploit this vulnerability.

  • HIGH: An attacker with Enterprise Administrator access to the GitHub Enterprise Server instance could escalate privileges to SSH root access. This is achieved by exploiting the pre-receive hook environment to bypass symlink checks in the ghe-firejail path and execute malicious scripts. GitHub has requested CVE ID CVE-2024-10007 for this vulnerability, which was reported via the GitHub Bug Bounty program. [Updated: 2024-11-07]

3.14.3: Bug fixes

  • When saving settings in the Management Console, the configuration run would stop if the enterprise-manage process was restarted.

  • On an instance with GitHub Actions enabled, some maintenance tasks could fail due to incomplete upgrade steps during previous upgrades to new releases of GitHub Enterprise Server.

  • A repeated error message concerning connectivity to port 6002 was emitted to the system logs when GitHub Actions was enabled.

  • The initial setup certificate generation in AWS took longer than expected due to fallback to private IPs. The time for this fallback has been reduced.

  • The ghe-support-bundle generation would fail when the aqueduct-lite service is down.

  • If the primary instance was unreachable, running ghe-repl-stop --force on a replica would fail during the config apply run.

  • Administrators in the SCIM private beta (versions < 3.14) that decided to upgrade their private beta appliance see an incorrectly checked box in the "SCIM Configuration" section of the Enterprise settings authentication security page in 3.14.

  • Certain URLs may have caused a 500 error on instances that use the mandatory message feature logging.

  • When restoring from a backup, repositories that had been deleted in the last 90 days were not completely restored.

  • For instances that use secret scanning, custom messages for push protection set by the enterprise did not display to users.

  • Restoring Git repositories using backup-utils occasionally failed.

  • Enterprise installations experienced unpredictable repository search results due to the default 4,000 repository limit. A relaxed repository filter mode, which includes all single-tenant organization repositories and bypasses the limit, has been introduced. Administrators can enable this mode using ghe-config app.github.enterprise-repo-search-filter-enabled true && ghe-config-apply.

  • Running config-apply became stuck under certain circumstances due to a misconfiguration with Packages and Elasticsearch.

  • Audit log events for secret scanning alerts incorrectly displayed a blank secret type when generated for a custom pattern.

  • Some customers upgrading to 3.14 may experience issues with undecryptable records during the upgrade. This issue has now been resolved. We recommend you read "Undecryptable records."

3.14.3: Changes

  • When connecting to an appliance via SSH, a notification about upcoming root disk changes displays.

3.14.3: Known issues

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. See "Устранение неполадок с доступом к консоли управления."

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

  • In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

  • Repositories originally imported using ghe-migrator will not correctly track GitHub Advanced Security contributions.

  • Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

  • When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

  • Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

  • If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

  • When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the Elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

  • An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

  • In the header bar displayed to site administrators, some icons are not available.

  • When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

  • When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

  • Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

October 10, 2024

📣 Это не последний выпуск исправлений Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

3.14.2: Security fixes

  • A sensitive data exposure in HTML forms was possible in the management console. To mitigate this issue, the "Copy Storage Setting from Actions" functionality was removed from the management console.

  • MEDIUM: Malicious URLs for SVG assets provided information about a victim user who clicked the URL, allowing an attacker to retrieve metadata belonging to the user and use it to generate a convincing phishing page. This required the attacker to upload malicious SVGs and phish a victim user to click the URL for the uploaded asset. GitHub has requested CVE ID CVE-2024-9539. This vulnerability was reported via the GitHub Bug Bounty program.

  • HIGH: An attacker could bypass SAML single sign-on (SSO) authentication with the optional encrypted assertions feature, allowing unauthorized provisioning of users and access to the instance, by exploiting an improper verification of cryptographic signatures vulnerability in GitHub Enterprise Server. This was a regression introduced as part of follow-up remediation from CVE-2024-4985, which resulted in a new variant of the vulnerability. Please note that encrypted assertions are not enabled by default. Instances not utilizing SAML SSO, or utilizing SAML SSO authentication without encrypted assertions, are not impacted. Additionally, an attacker would require direct network access as well as a signed SAML response or metadata document. GitHub has requested CVE ID CVE-2024-9487. This vulnerability was reported via the GitHub Bug Bounty program.

3.14.2: Bug fixes

  • A missing configuration value would cause Dependabot to be unable to create group update pull requests.

  • HAProxy reloading was failure prone, which could lead to failed Git operations. This reloading process has been replaced with a more resilient Systemd process.

  • This error message mbind: Operation not permitted was repeatedly showing in the /var/log/mysql/mysql.err MySQL logs.

  • The backup of audit logs could take longer after upgrading to Elasticsearch 8.

  • An unhandled nil value when configuring Actions storage with AWS S3 via OIDC configuration in the terminal could cause an error.

  • Users were unable to sign out from gist pages.

  • On an instance with secret scanning enabled, the custom pattern page would not load because dry run results were tied to a deleted repository.

  • Suspended users were not always correctly routed to the correct "suspended" page.

  • The "List teams" API endpoint returned duplicate results when paginating.

  • When managing the organization permissions required for fine-grained personal access tokens, for custom properties or projects, the Admin access level could not be selected.

  • A model with no URL could cause a ghe-migrator import to fail.

  • The ghe-spokesctl status command showed repaired repositories as broken if their network ID changed during the repair (for example, when the repository was detached from it's original network).

  • Missing URLs on import could lead to migration failures without logging or explanation.

  • On the security overview dashboard, data initialization could fail when creating new organizations or changing GitHub Advanced Security licensing.

  • Restore could fail when restoring MySQL using backup-utils.

3.14.2: Changes

  • ghe-remove-node will display the log file location when running in quiet mode.

  • Pre-receive hook environments can use the clone3() system call.

  • The creation, deletion, or change in visibility of a gist has been added to the audit log.

3.14.2: Known issues

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

  • In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

  • Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

  • The admin stats REST API endpoints may time out on appliances with many users or repositories. Retrying the request until data is returned is advised.

  • When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

  • Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

  • If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

  • When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

  • An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

  • In the header bar displayed to site administrators, some icons are not available.

  • When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

  • When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

  • Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

  • Images embedded in wiki pages may stop rendering shortly after being published. [Updated: 2024-10-16]

September 23, 2024

📣 Это не последний выпуск исправлений Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

3.14.1: Security fixes

  • MEDIUM: An attacker could steal sensitive information by exploiting a Cross-Site Scripting vulnerability in the repository transfer feature. This exploitation would require social engineering. GitHub has requested CVE ID CVE-2024-8770 for this vulnerability, which was reported via the GitHub Bug Bounty program.

  • HIGH: A GitHub App installed in organizations could upgrade some permissions from read to write access without approval from an organization administrator. An attacker would require an account with administrator access to install a malicious GitHub App. GitHub has requested CVE ID CVE-2024-8810 for this vulnerability, which was reported via the GitHub Bug Bounty Program. [Updated: 2024-11-07]

3.14.1: Bug fixes

  • On an instance with GitHub Actions enabled, due to an insufficient wait time, MS SQL and MySQL replication could fail with the error message Failed to start nomad service!.

  • ghe-storage-find was sometimes unable to identify a data disk.

  • After upgrading the relevant GHES version, the resolvconf service failed to start due to a missing directory.

  • Some pre-receive hooks using the faccessat2 system call, such as those using Alpine Linux as the base, failed unexpectedly.

  • When configuring a high availability replica and during the database seeding of a MySQL replica node, restarting the nomad service could time out. Consequently, when MySQL replication attempted to start an error was reported, and setting up replication failed.

  • On an instance in a cluster configuration, the ghe-cluster-status command returned an error if a soft-deleted repository had a checksum mismatch.

  • Fixes and improvements for the git core module.

  • Some repositories could miss spokes information after restoring in a clustering topology due to unrescued exceptions.

  • In organizations with a large number of repositories, when an administrator used repository properties to target repositories in an organization ruleset, the ruleset index page timed out.

  • After a user created a Projects Insights chart with time as the X-axis, the chart became hidden and inaccessible.

  • Fixes a known issue where some links to GitHub Docs from GitHub Enterprise Server may lead to a “Page not found.” Previously, the links incorrectly added enterprise-cloud@latest to the URL.

  • A bug introduced in 3.12 which prevented the search input in the global navigation from displaying a dropdown of search suggestions has been fixed. The search input functionality prior to 3.12 has been restored, and users are once again able to see and submit suggested search queries, including scope suggestions.

  • Custom links to other repositories displayed incorrect breadcrumbs.

  • The Secret Scanning Push Protection custom resource link set at the Enterprise level was not being displayed to users being blocked when pushing secrets to a repository using git through the command line interface.

  • Following an upgrade, Elasticsearch search migrations are sometimes incorrectly reported as failing in the audit log, even though the migrations completed successfully. [Updated: 2024-09-27]

3.14.1: Changes

  • For instances deployed on Amazon Web Services (AWS), site administrators can configure regional AWS STS endpoints for OIDC from the Management Console.

  • Site administrators can now configure the instance with NUMA optimizations.

3.14.1: Known issues

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

  • In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

  • Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

  • Admin stats REST API endpoints may timeout on appliances with many users or repositories. Retrying the request until data is returned is advised.

  • When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

  • Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shutdown the node and repeat the steps.

  • If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

  • When restoring data originally backed up from a 3.13 appliance onto a 3.13 appliance, the elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

  • An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

  • When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

  • In the header bar displayed to site administrators, some icons are not available.

  • When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

  • When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

  • Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

  • Images embedded in wiki pages may stop rendering shortly after being published. [Updated: 2024-10-16]

August 27, 2024

📣 Это не последний выпуск исправлений Enterprise Server. Используйте последний выпуск для последних исправлений безопасности, производительности и ошибок.

For upgrade instructions, see "Upgrading GitHub Enterprise Server."

3.14.0: Features

3.14.0: Changes

  • Pushes that update over 5,000 branches no longer trigger webhooks or GitHub Actions workflows.

3.14.0: Known issues

  • Complete SCIM payloads are written to the audit log, including SCIM attributes that are not required or supported per API docs. Customers using Okta with SCIM may notice that a placeholder password attribute is among the data passed to audit logs in its current configuration. This placeholder data is associated with Okta’s password synchronization feature that is not expected or required by GitHub. See okta-scim for more information.

  • Custom firewall rules are removed during the upgrade process.

  • When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

  • During the validation phase of a configuration run, a No such object error may occur for the Notebook and Viewscreen services. This error can be ignored as the services should still correctly start.

  • If the root site administrator is locked out of the Management Console after failed login attempts, the account does not unlock automatically after the defined lockout time. Someone with administrative SSH access to the instance must unlock the account using the administrative shell. For more information, see "Устранение неполадок с доступом к консоли управления."

  • On an instance with the HTTP X-Forwarded-For header configured for use behind a load balancer, all client IP addresses in the instance's audit log erroneously appear as 127.0.0.1.

  • In some situations, large .adoc files stored in a repository do not render properly in the web UI. The raw contents are still available to view as plaintext.

  • Repositories originally imported using ghe-migrator will not correctly track Advanced Security contributions.

  • REST API endpoints for admin stats may time out on appliances with many users or repositories. Retrying the request until data is returned is advised.

  • When following the steps for Replacing the primary MySQL node, step 14 (running ghe-cluster-config-apply) might fail with errors. If this occurs, re-running ghe-cluster-config-apply is expected to succeed.

  • Running a config apply as part of the steps for Replacing a node in an emergency may fail with errors if the node being replaced is still reachable. If this occurs, shut down the node and repeat the steps.

  • If a hotpatch upgrade requires the haproxy-frontend service to be restarted, the restart will hang if there are existing long-lived connections, such as browser web sockets or Git operations. No new connections will be accepted for up to 5 minutes. Any existing unfinished connections at this time will be disconnected.

  • When restoring data originally backed up from a 3.13 appliance, the Elasticsearch indices need to be reindexed before some of the data will show up. This happens via a nightly scheduled job. It can also be forced by running /usr/local/share/enterprise/ghe-es-search-repair.

  • The global search bar does not have suggestions enabled due to the redesigned navigation and pending new search experience.

  • Upgrades include an error concerning Error deregistering job for consul-template. This message does not indicate any problems with your install and can be safely ignored.

  • Some links to GitHub Docs from GitHub Enterprise Server may lead to a "Page not found," because an enterprise-cloud@latest portion is incorrectly added to the URL.

  • An organization-level code scanning configuration page is displayed on instances that do not use GitHub Advanced Security or code scanning.

  • In the header bar displayed to site administrators, some icons are not available.

  • When restoring from a backup snapshot, a large number of mapper_parsing_exception errors may be displayed.

  • On boot, the resolvconf service may fail to start because the /run/resolvconf directory does not exist when the service attempts to touch a file there, with the error:

    /bin/touch: cannot touch '/run/resolvconf/postponed-update': No such file or directory
    

    If this occurs, workaround this issue with the following commands — this change will persist on reboots, but not upgrades:

    sudo sed -i.bak \
    '/\[Service\]/a ExecStartPre\=\/bin\/mkdir \-p \/run\/resolvconf' \
    /etc/systemd/system/resolvconf.service.d/local.conf
    
    sudo systemctl daemon-reload
    sudo systemctl start resolvconf 
    
  • Services may respond with a 503 status due to an out of date haproxy configuration. This can usually be resolved with a ghe-config-apply run.

  • When enabling automatic update checks for the first time in the Management Console, the status is not dynamically reflected until the "Updates" page is reloaded.

  • Following an upgrade, Elasticsearch search migrations are sometimes incorrectly reported as failing in the audit log, even though the migrations completed successfully. [Updated: 2024-09-27]

  • Images embedded in wiki pages may stop rendering shortly after being published. [Updated: 2024-10-16]

3.14.0: Deprecations

3.14.0: Errata

  • These release notes previously indicated as a known issue that on GitHub Enterprise Server 3.14.0 when log forwarding is enabled, some forwarded log entries may be duplicated. The fix for this problem was already included prior to the release of GitHub Enterprise Server 3.14.0. [Updated: 2024-09-16]

  • These release notes did not include a note for support of the directories key in dependabot.yml files. [Updated: 2024-10-07]