GitHub Enterprise 2.3 is now deprecated. That means that no patch releases will be made, even for critical security issues, after this release. For better performance, improved security, and new features, upgrade to the newest version of GitHub Enterprise as soon as possible.
Thanks!
The GitHub Team
Thanks!
The GitHub Team
A CRITICAL issue was identified for all 2.x versions of GitHub Enterprise. The GitHub Enterprise images contain pre-generated SSH host keys that were not regenerated upon installation for all supported platforms:
ssh_host_ed25519_key in GitHub Enterprise section below)This means an attacker with the capability to perform a man-in-the-middle attack on SSH traffic can intercept and modify network traffic to the GitHub Enterprise appliance.
The affected supported versions are:
This vulnerability was found and reported internally and we have no evidence that it has been exploited in the wild.
We strongly recommend upgrading your GitHub Enterprise appliance to the latest patch release in your series, GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, or 2.3.21. In addition, with backup-utils-2.7.1, ghe-backup and ghe-restore will check for any leaked SSH host keys in the snapshot(s).
Please contact GitHub Enterprise Support if you have questions.
--
If you've upgraded to the latest patch release, GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater,
SSH to your primary GitHub Enterprise appliance.
Check for leaked SSH host keys using the ghe-ssh-check-host-keys utility.
$ ghe-ssh-check-host-keys
The utility should output either:
One or more of your SSH host keys were found in the blacklist.
Please reset your host keys using ghe-ssh-roll-host-keys.
--
The SSH host keys were not found in the SSH host key blacklist.
No additional steps are needed/recommended at this time.
If one or more SSH host keys were found in the blacklist, continue to the next step. Otherwise, your GitHub Enterprise environment is not vulnerable.
Put your GitHub Enterprise environment in Maintenance Mode.
Rotate all SSH host keys using the ghe-ssh-roll-host-keys utility.
$ sudo ghe-ssh-roll-host-keys
$ sudo ssh-keygen -t ed25519 -N "" -f /etc/ssh/ssh_host_ed25519_key
The utility should output:
$ SSH host keys have successfully been rolled.
If you've upgraded to GitHub Enterprise 2.7.4, 2.6.9, or greater, and you are using the High Availability Configuration, there are no additional steps to take on your replica appliance.
If you've upgraded to GitHub Enterprise 2.5.14, 2.4.17, 2.3.21, or greater, and you are using the High Availability Configuration,
After completing steps 1-5, stop replication on the replica appliance.
$ ghe-repl-stop
Synchronize the SSH host keys from the primary appliance.
$ ghe-repl-setup
Resume replication on the replica appliance.
$ ghe-repl-start
--
If you're unable to upgrade immediately to the latest patch release, GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater,
SSH to your primary GitHub Enterprise appliance.
Download the list of leaked SSH host keys and verify its content using any of the provided hashes.
$ curl -O https://enterprise.github.com/security/2016-09-20/ghe-ssh-leaked-host-keys-list.txt
$ sha256sum ghe-ssh-leaked-host-keys-list.txt
3bb29658784a4059a41f1a77cffba9586baab179ba07b795f80e12a9f10c5665 ghe-ssh-leaked-host-keys-list.txt
$ sha1sum ghe-ssh-leaked-host-keys-list.txt
5db799da044da9aae0bcfc523d22e7ce0fe72550 ghe-ssh-leaked-host-keys-list.txt
$ md5sum ghe-ssh-leaked-host-keys-list.txt
de75bcb0bf1d13e15620952c0af8da41 ghe-ssh-leaked-host-keys-list.txt
Print the fingerprint of your GitHub Enterprise appliance's SSH host keys.
Note: The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used in 2.7.4 or greater.
$ ssh-keygen -lf /etc/ssh/ssh_host_dsa_key.pub
1024 b2:69:82:2f:25:48:bb:fc:62:c7:9a:de:41:42:13:55 /etc/ssh/ssh_host_dsa_key.pub (DSA)
$ ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub
256 c0:cb:fd:07:33:e9:62:14:6b:fb:d5:26:54:f3:c5:0d /etc/ssh/ssh_host_ecdsa_key.pub (ECDSA)
$ ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
256 d6:92:21:4b:04:3b:22:f5:ee:85:0a:63:bf:b3:fe:9b /etc/ssh/ssh_host_ed25519_key.pub (ED25519)
$ ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub
2048 0f:ee:8d:02:2d:e1:76:f3:eb:f5:af:cb:38:9a:1c:33 /etc/ssh/ssh_host_rsa_key.pub (RSA)
Check for leaked SSH host keys by comparing against the downloaded list of leaked SSH host keys.
If one or more SSH host keys were found in the blacklist, continue to the next step. Otherwise, your GitHub Enterprise environment is not vulnerable.
Put your GitHub Enterprise environment in Maintenance Mode.
Remove all SSH host keys.
$ sudo rm -f /etc/ssh/ssh_host_*
Regenerate the SSH host keys.
Note: The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used and regenerated for in 2.7.4 or greater.
$ sudo ssh-keygen -t ed25519 -N "" -f /etc/ssh/ssh_host_ed25519_key
$ sudo dpkg-reconfigure openssh-server
Apply the changes to the ssh and babeld service.
$ sudo cp /etc/ssh/ssh_host_{rsa,dsa,ecdsa,ed25519}_key{,.pub} /data/user/common/
$ sudo chown babeld:babeld /data/user/common/ssh_host_{rsa,dsa,ecdsa,ed25519}_key{,.pub}
If you're unable to upgrade immediately to GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater, and you are using the High Availability Configuration,
$ ghe-repl-stop
$ ghe-repl-setup
$ ghe-repl-start
--
After rotating the SSH host keys, your GitHub Enterprise environment can exit Maintenance Mode.
Your end-users will receive an error message when attempting to use the Administrative Shell (SSH) or the SSH protocol for Git activity. The rotation does not affect users using the HTTPS protocol for Git activity.
For example, the following is an output from the command-line,
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:seFT9eIOmAZWbfcO9yU1sXiEYIqcrdi0qttbtmNm0Io.
Please contact your system administrator.
Add correct host key in /Users/monalisa/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/monalisa/.ssh/known_hosts:42
ECDSA host key for [github.example.com]:122 has changed and you have requested strict checking.
Host key verification failed.
After updating the known_hosts, end-users will be prompted to accept a new fingerprint.
$ ssh -p 122 admin@github.example.com
The authenticity of host '[github.example.com]:122 ([169.254.1.1]:122)' can't be established.
ECDSA key fingerprint is SHA256:seFT9eIOmAZWbfcO9yU1sXiEYIqcrdi0qttbtmNm0Io.
Are you sure you want to continue connecting (yes/no)?
We strongly recommend publishing your GitHub Enterprise appliance's SSH host key fingerprints in a location that is accessible to all your end-users. For example, for GitHub.com, we publish the SSH fingerprints at https://help.github.com/articles/what-are-github-s-ssh-key-fingerprints/.
If you'd like to to give end-users notice before rotating the SSH host keys, follow the instructions in the Verification and Mitigation if Immediate Upgrade is not Possible skipping step 7 and replacing step 8 with,
Regenerate the SSH host keys.
Note: The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used and regenerated for in 2.7.4 or greater.
i. Pre-generate new SSH host keys to a temporary directory.
$ ssh-keygen -t dsa -N "" -f /var/tmp/ssh_host_dsa_key
$ ssh-keygen -t rsa -N "" -f /var/tmp/ssh_host_rsa_key
$ ssh-keygen -t ecdsa -N "" -f /var/tmp/ssh_host_ecdsa_key
$ ssh-keygen -t ed25519 -N "" -f /var/tmp/ssh_host_ed25519_key
ii. Print the fingerprint of your GitHub Enterprise appliance's SSH host keys for tentative rotation.
$ ssh-keygen -lf /var/tmp/ssh_host_dsa_key.pub
1024 b2:69:82:2f:25:48:bb:fc:62:c7:9a:de:41:42:13:55 /var/tmp/ssh_host_dsa_key.pub (DSA)
$ ssh-keygen -lf /var/tmp/ssh_host_ecdsa_key.pub
256 c0:cb:fd:07:33:e9:62:14:6b:fb:d5:26:54:f3:c5:0d /var/tmp/ssh_host_ecdsa_key.pub (ECDSA)
$ ssh-keygen -lf /var/tmp/ssh_host_ed25519_key.pub
256 d6:92:21:4b:04:3b:22:f5:ee:85:0a:63:bf:b3:fe:9b /var/tmp/ssh_host_ed25519_key.pub (ED25519)
$ ssh-keygen -lf /var/tmp/ssh_host_rsa_key.pub
248 0f:ee:8d:02:2d:e1:76:f3:eb:f5:af:cb:38:9a:1c:33 /var/tmp/ssh_host_rsa_key.pub (RSA)
iii. Once you are ready to migrate to the new, rotated SSH host keys, move the host keys from the temporary directory and apply the changes to the ssh service.
$ sudo mv /var/tmp/ssh_host_{rsa,dsa,ecdsa,ed25519}_key{,.pub} /etc/ssh
$ sudo service ssh restart
iv. Continue with steps 9 in the Verification and Mitigation if Immediate Upgrade is not Possible section.
ssh_host_ed25519_key in GitHub EnterpriseThe 2.x versions of GitHub Enterprise on all supported platforms:
contain a pre-generated ssh_host_ed25519_key. However, only GitHub Enterprise 2.7.4 or greater use the ssh_host_ed25519_key. This can be verified by checking your GitHub Enterprise appliance's /etc/ssh/sshd_config, which added HostKey /etc/ssh/ssh_host_ed25519_key in 2.7.4 or greater.
The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used in 2.7.4 or greater.
If you've upgraded your appliance to 2.7.4 or greater on any of the supported platforms including Amazon Web Services, please follow the instructions in the Verification and Mitigation on GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater section.
ssh_host_ed25519_key in GitHub Enterprise for GitHub Enterprise 2.7.4 or greater appliances on the Amazon Web Services platform. (updated 2016-09-22)Thanks!
The GitHub Team
Thanks!
The GitHub Team
Thanks!
The GitHub Team
rbenv, used by many components of GitHub Enterprise, have been tightened.ghe-user-suspend command.Thanks!
The GitHub Team
Thanks!
The GitHub Team
Thanks!
The GitHub Team
Thanks!
The GitHub Team
Thanks!
The GitHub Team
Several vulnerabilities in ImageMagick, a package commonly used by web services to process images, have been discovered and disclosed by members of the Mail.ru Security team. One of the vulnerabilities is critical and can lead to remote code execution when processing user submitted images.
Final patches for all the disclosed vulnerabilities within ImageMagick are still pending. This release mitigates the remote code execution vulnerability by implementing the recommended policy to disable the vulnerable ImageMagick coders.
This vulnerability exists in ImageMagick but there is no evidence that it has been exploited on GitHub Enterprise.
We strongly recommend that all GitHub Enterprise customers upgrade their instances as soon as possible.
Mitigation
If you can't immediately upgrade, the issue can be mitigated by implementing the policy changes as follows:
SSH to your GitHub Enterprise appliance.
Edit the /etc/ImageMagick/policy.xml file:
sudo vi /etc/ImageMagick/policy.xml
Disable the vulnerable coders by replacing the <policymap> section with:
<policymap>
<policy domain="coder" rights="none" pattern="EPHEMERAL" />
<policy domain="coder" rights="none" pattern="URL" />
<policy domain="coder" rights="none" pattern="HTTPS" />
<policy domain="coder" rights="none" pattern="MVG" />
<policy domain="coder" rights="none" pattern="MSL" />
</policymap>
There is no need to reboot or restart any services; the changes will take effect immediately.
Please contact GitHub Enterprise Support if you have any questions.
Thanks!
The GitHub Team
Thanks!
The GitHub Team
_gh_render cookie, potentially allowing the render cookie to be sent in plaintext HTTP requests. However, Enterprise sets the Strict-Transport-Security header for modern browsers when SSL is enabled, which largely mitigates the issue.Thanks!
The GitHub Team
ca-certificates package has been updated to remove outdated certificate authority (CA) certificates. This update refreshes the included certificates and removes the SPI CA and CA certificates with 1024-bit RSA keys.Thanks!
The GitHub Team
glibc packages have been updated to address CVE-2015-7547, a getaddrinfo stack-based buffer overflow.libssh packages have been updated to address CVE-2016-0739, a weakness in diffie-hellman secret key generation.nss packages have been updated to address CVE-2016-1938.Thanks!
The GitHub Team
We incorrectly redirect to the dashboard if you accessed GitHub Enterprise using an alias while in private mode. This might happen if you set a fully qualified domain name but the subdomain resolves correctly.
Email can't be sent over TLS when SSL is disabled.
Deleting a user doesn't delete their gists, which can cause problems with replication.
Images uploaded to issues save with an absolute URL, so they can be broken if the hostname changes.
On a freshly set up GitHub Enterprise without any users, an attacker could create the first admin user.
The management console settings interface doesn't clearly show if you have previously uploaded certificate files or a private key.
Management console sessions can expire too quickly for Safari users.
Gist repositories are not garbage collected by the maintenance scheduler.
Custom firewall rules aren't maintained during an upgrade.
Repositories that are in an incomplete state, which is a rare problem, can cause the migration to the new repository disk layout to fail.
Enqueued background jobs are sometimes not purged when a repository is deleted.
When a fork is detached from its repository network by an administrator or by changing visibility, its filesystem path won't be updated on a high availability replica until at least one commit has been pushed.
Viewing raw files in repositories owned by a user or organization named "github" fails with a 400 error. (updated 2015-12-15)
HIGH (CVE-2015-7547) 2.3 is vulnerable to glibc getaddrinfo stack-based buffer overflow. To manually patch your appliance, apply the hotfix by connecting to your appliance via SSH and running these commands: (updated 2016-02-17)
$ curl -O https://github-enterprise.s3.amazonaws.com/patches/github-enterprise-libc-precise.hpkg
$ md5sum github-enterprise-libc-precise.hpkg # c068256696f2775579e2cd8223f82306
$ chmod +x github-enterprise-libc-precise.hpkg
$ ./github-enterprise-libc-precise.hpkg
Thanks!
The GitHub Team
An issue has been identified that could allow an attacker to execute arbitrary commands on a user’s computer if they had Git LFS installed and cloned a malicious repository. Git LFS supports a per-repository configuration file to customize how certain aspects of Git LFS function. However, this file also allowed arbitrary Git configuration options to be modified. We have addressed the vulnerability by whitelisting the set of per-repository Git LFS configuration options that can be used to a safe subset.
GitHub Enterprise is not directly affected as this is a client-side vulnerability and Git LFS is disabled on GitHub Enterprise by default. If you have enabled Git LFS on your appliance, we recommend you upgrade your clients to Git LFS 1.0.1 or later to address this vulnerability.
Thanks!
The GitHub Team
Thanks!
The GitHub Team
ghe-repl-status would not report a failure if ElasticSearch was not running.ghe-upgrade would fail with a GPG signature error if run as the root user.Thanks!
The GitHub Team
Thanks!
The GitHub Team
.patch and .diff URLs would fail on instances with subdomain isolation disabled.core.package-version variable in the appliance configuration file was not updated to reflect the new appliance version during an upgrade..mediawiki suffix could leak information to the Google Chart API when they were displayed.Thanks!
The GitHub Team
longpoll service, which provides live updates to Issues and Pull Requests pages, didn't restart properly if it was terminated.ghe-storage-extend command, which resizes the storage volume, could fail with a Volume group name ghe_storage_* has invalid characters error under some circumstances.Thanks!
The GitHub Team
? key on Gist pages.Thanks!
The GitHub Team
With the new features added in GitHub Enterprise 2.3.0, you can:
ghe-migrator.ghe-resque-info command line utility.Upgrading to the 2.3 release series is supported from GitHub Enterprise 2.1.0 and above.
User authentication via GitHub OAuth is being deprecated and will be removed in a future feature release. It will be removed no sooner than November 2015.
GitHub Enterprise includes support for authenticating users via OAuth to accounts on GitHub.com because it provides a simple way to set up external authentication. However, after speaking with many customers, we've found that organizations commonly have other sources they want to use to automate identity and access management.
We want to focus on features that best meet the needs of our users, so we're planning to remove support for GitHub OAuth in a future feature release and focus on making ongoing improvements to other authentication methods like SAML and LDAP.
Note that this change will only affect user authentication via GitHub.com and not personal access tokens or OAuth applications added to your GitHub Enterprise instance.
Thanks!
The GitHub Team