The 2.7 series release notes contain important changes in this release series.

Pre-generated SSH Host Keys in GitHub Enterprise

A CRITICAL issue was identified for all 2.x versions of GitHub Enterprise. The GitHub Enterprise images contain pre-generated SSH host keys that were not regenerated upon installation for all supported platforms:

• Hyper-V (VHD)
• OpenStack KVM (QCOW2)
• VMware ESXi/vSphere (OVA)
• Xen (VHD)
• Amazon Web Services (See the ssh_host_ed25519_key in GitHub Enterprise section below)
• Microsoft Azure

This means an attacker with the capability to perform a man-in-the-middle attack on SSH traffic can intercept and modify network traffic to the GitHub Enterprise appliance.

The affected supported versions are:

• 2.7.0 - 2.7.3
• 2.6.0 - 2.6.8
• 2.5.0 - 2.5.13
• 2.4.0 - 2.4.16
• 2.3.0 - 2.3.20

This vulnerability was found and reported internally and we have no evidence that it has been exploited in the wild.

We strongly recommend upgrading your GitHub Enterprise appliance to the latest patch release in your series, GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, or 2.3.21. In addition, with backup-utils-2.7.1, ghe-backup and ghe-restore will check for any leaked SSH host keys in the snapshot(s).

Please contact GitHub Enterprise Support if you have questions.

--

Verification and Mitigation on GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater

If you've upgraded to the latest patch release, GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater,

1. SSH to your primary GitHub Enterprise appliance.

2. Check for leaked SSH host keys using the ghe-ssh-check-host-keys utility.

   $ ghe-ssh-check-host-keys
   

   The utility should output either:

   One or more of your SSH host keys were found in the blacklist.
   Please reset your host keys using ghe-ssh-roll-host-keys.
   

   --

   The SSH host keys were not found in the SSH host key blacklist.
   No additional steps are needed/recommended at this time.
   

3. If one or more SSH host keys were found in the blacklist, continue to the next step. Otherwise, your GitHub Enterprise environment is not vulnerable.

4. Put your GitHub Enterprise environment in Maintenance Mode.

5. Rotate all SSH host keys using the ghe-ssh-roll-host-keys utility.

   $ sudo ghe-ssh-roll-host-keys
   

   The utility should output:

   $ SSH host keys have successfully been rolled.
   

If you've upgraded to GitHub Enterprise 2.7.4, 2.6.9, or greater, and you are using the High Availability Configuration, there are no additional steps to take on your replica appliance.

If you've upgraded to GitHub Enterprise 2.5.14, 2.4.17, 2.3.21, or greater, and you are using the High Availability Configuration,

6. After completing steps 1-5, stop replication on the replica appliance.

   $ ghe-repl-stop
   

7. Synchronize the SSH host keys from the primary appliance.

   $ ghe-repl-setup
   

8. Resume replication on the replica appliance.

   $ ghe-repl-start
   

If you've upgraded to GitHub Enterprise 2.7.4, 2.6.9, 2.5.14 or greater, and you are using Clustering,

6. After completing steps 1-5, apply the changes to all cluster nodes.

   $ ghe-cluster-config-apply
   

--

Verification and Mitigation if Immediate Upgrade is not Possible

If you're unable to upgrade immediately to the latest patch release, GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater,

1. SSH to your primary GitHub Enterprise appliance.

2. Download the list of leaked SSH host keys and verify its content using any of the provided hashes.

   $ curl -O https://enterprise.github.com/security/2016-09-20/ghe-ssh-leaked-host-keys-list.txt
   $ sha256sum ghe-ssh-leaked-host-keys-list.txt
   3bb29658784a4059a41f1a77cffba9586baab179ba07b795f80e12a9f10c5665  ghe-ssh-leaked-host-keys-list.txt
   $ sha1sum ghe-ssh-leaked-host-keys-list.txt
   5db799da044da9aae0bcfc523d22e7ce0fe72550  ghe-ssh-leaked-host-keys-list.txt
   $ md5sum ghe-ssh-leaked-host-keys-list.txt
   de75bcb0bf1d13e15620952c0af8da41  ghe-ssh-leaked-host-keys-list.txt
   

3. Print the fingerprint of your GitHub Enterprise appliance's SSH host keys.
   Note: The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used in 2.7.4 or greater.

   $ ssh-keygen -lf /etc/ssh/ssh_host_dsa_key.pub
   1024 b2:69:82:2f:25:48:bb:fc:62:c7:9a:de:41:42:13:55 /etc/ssh/ssh_host_dsa_key.pub (DSA)
   $ ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub
   256 c0:cb:fd:07:33:e9:62:14:6b:fb:d5:26:54:f3:c5:0d /etc/ssh/ssh_host_ecdsa_key.pub (ECDSA)
   $ ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
   256 d6:92:21:4b:04:3b:22:f5:ee:85:0a:63:bf:b3:fe:9b /etc/ssh/ssh_host_ed25519_key.pub (ED25519)
   $ ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub
   2048 0f:ee:8d:02:2d:e1:76:f3:eb:f5:af:cb:38:9a:1c:33 /etc/ssh/ssh_host_rsa_key.pub (RSA)
   

4. Check for leaked SSH host keys by comparing against the downloaded list of leaked SSH host keys.

5. If one or more SSH host keys were found in the blacklist, continue to the next step. Otherwise, your GitHub Enterprise environment is not vulnerable.

6. Put your GitHub Enterprise environment in Maintenance Mode.

7. Remove all SSH host keys.

   $ sudo rm -f /etc/ssh/ssh_host_*
   

8. Regenerate the SSH host keys.
   Note: The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used and regenerated for in 2.7.4 or greater.

   $ sudo ssh-keygen -t ed25519 -N "" -f /etc/ssh/ssh_host_ed25519_key
   $ sudo dpkg-reconfigure openssh-server
   

9. Apply the changes to the ssh and babeld service.

   $ sudo cp /etc/ssh/ssh_host_{rsa,dsa,ecdsa,ed25519}_key{,.pub} /data/user/common/
   $ sudo chown babeld:babeld /data/user/common/ssh_host_{rsa,dsa,ecdsa,ed25519}_key{,.pub}
   

If you're unable to upgrade immediately to GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater, and you are using the High Availability Configuration,

10. After completing steps 1-9, stop replication on the replica appliance.

$ ghe-repl-stop

11. Synchronize the SSH host keys from the primary appliance.

$ ghe-repl-setup

12. Resume replication on the replica appliance.

$ ghe-repl-start

If you're unable to upgrade immediately to GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, or greater, and you are using Clustering,

10. After completing steps 1-9, apply the changes to all cluster nodes.

$ ghe-cluster-config-apply

--

Post SSH Host Key Rotation

After rotating the SSH host keys, your GitHub Enterprise environment can exit Maintenance Mode.

Your end-users will receive an error message when attempting to use the Administrative Shell (SSH) or the SSH protocol for Git activity. The rotation does not affect users using the HTTPS protocol for Git activity.

For example, the following is an output from the command-line,

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:seFT9eIOmAZWbfcO9yU1sXiEYIqcrdi0qttbtmNm0Io.
Please contact your system administrator.
Add correct host key in /Users/monalisa/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/monalisa/.ssh/known_hosts:42
ECDSA host key for [github.example.com]:122 has changed and you have requested strict checking.
Host key verification failed.

After updating the known_hosts, end-users will be prompted to accept a new fingerprint.

$ ssh -p 122 admin@github.example.com
The authenticity of host '[github.example.com]:122 ([169.254.1.1]:122)' can't be established.
ECDSA key fingerprint is SHA256:seFT9eIOmAZWbfcO9yU1sXiEYIqcrdi0qttbtmNm0Io.
Are you sure you want to continue connecting (yes/no)?

We strongly recommend publishing your GitHub Enterprise appliance's SSH host key fingerprints in a location that is accessible to all your end-users. For example, for GitHub.com, we publish the SSH fingerprints at https://help.github.com/articles/what-are-github-s-ssh-key-fingerprints/.

If you'd like to to give end-users notice before rotating the SSH host keys, follow the instructions in the Verification and Mitigation if Immediate Upgrade is not Possible skipping step 7 and replacing step 8 with,

8. Regenerate the SSH host keys.
   Note: The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used and regenerated for in 2.7.4 or greater.

   i. Pre-generate new SSH host keys to a temporary directory.

   $ ssh-keygen -t dsa -N "" -f /var/tmp/ssh_host_dsa_key
   $ ssh-keygen -t rsa -N "" -f /var/tmp/ssh_host_rsa_key
   $ ssh-keygen -t ecdsa -N "" -f /var/tmp/ssh_host_ecdsa_key
   $ ssh-keygen -t ed25519 -N "" -f /var/tmp/ssh_host_ed25519_key
   

   ii. Print the fingerprint of your GitHub Enterprise appliance's SSH host keys for tentative rotation.

   $ ssh-keygen -lf /var/tmp/ssh_host_dsa_key.pub
    1024 b2:69:82:2f:25:48:bb:fc:62:c7:9a:de:41:42:13:55 /var/tmp/ssh_host_dsa_key.pub (DSA)
   $ ssh-keygen -lf /var/tmp/ssh_host_ecdsa_key.pub
    256 c0:cb:fd:07:33:e9:62:14:6b:fb:d5:26:54:f3:c5:0d /var/tmp/ssh_host_ecdsa_key.pub (ECDSA)
   $ ssh-keygen -lf /var/tmp/ssh_host_ed25519_key.pub
    256 d6:92:21:4b:04:3b:22:f5:ee:85:0a:63:bf:b3:fe:9b /var/tmp/ssh_host_ed25519_key.pub (ED25519)
   $ ssh-keygen -lf /var/tmp/ssh_host_rsa_key.pub
   248 0f:ee:8d:02:2d:e1:76:f3:eb:f5:af:cb:38:9a:1c:33 /var/tmp/ssh_host_rsa_key.pub (RSA)
   

   iii. Once you are ready to migrate to the new, rotated SSH host keys, move the host keys from the temporary directory and apply the changes to the ssh service.

   $ sudo mv /var/tmp/ssh_host_{rsa,dsa,ecdsa,ed25519}_key{,.pub} /etc/ssh
   $ sudo service ssh restart
   

   iv. Continue with steps 9 in the Verification and Mitigation if Immediate Upgrade is not Possible section.

ssh_host_ed25519_key in GitHub Enterprise

The 2.x versions of GitHub Enterprise on all supported platforms:

• Hyper-V (VHD)
• OpenStack KVM (QCOW2)
• VMware ESXi/vSphere (OVA)
• Xen (VHD)
• Amazon Web Services
• Microsoft Azure

contain a pre-generated ssh_host_ed25519_key. However, only GitHub Enterprise 2.7.4 or greater use the ssh_host_ed25519_key. This can be verified by checking your GitHub Enterprise appliance's /etc/ssh/sshd_config, which added HostKey /etc/ssh/ssh_host_ed25519_key in 2.7.4 or greater.

The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used in 2.7.4 or greater.

If you've upgraded your appliance to 2.7.4 or greater on any of the supported platforms including Amazon Web Services, please follow the instructions in the Verification and Mitigation on GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater section.

Security Fixes

• CRITICAL Pre-generated SSH host keys were not regenerated when installing appliances from GitHub Enterprise 2.x images.
• Packages have been updated to the latest security versions.

Bug Fixes

• When rejected from a pre-receive hook, API merge requests incorrectly returned Internal Server Error.
• Webhooks failed to deliver when the external server could only be resolved by the configured proxy server.
• The snmpd service did not automatically start on replica instances.
• The ghe-system-info command line utility was not available to run because the utility was missing from the $PATH.
• In a clustering environment, the redis-server role may not have dedicated memory allocated to the redis service.
• In a clustering environment, storage assets that were not replicated or marked for deletion were not properly maintained.
• Users were unable to add or remove deploy keys when LDAP sync is enabled.
• In a clustering environment, the ghe-cluster-config-check command line utility terminated early from unsuccessful cURL checks.
• Pre-receive hooks using the git-grep command may have failed using the default hook environment due to missing libraries.
• The initial push of a repository with many Git refs could time out.
• The root API endpoint incorrectly returned Not Found when the trailing slash was omitted.
• Repository maintenance could time out for large repositories, so the timeout was increased to 120 minutes.
• Upgrades could incorrectly output upgrade failed! after a successful upgrade.
• After upgrading, the site admin page on replica instances could incorrectly show Verifying ElasticSearch indexes.
• Elasticsearch logs could grow very large due to incorrect HTTP and HTTPS connection management.
• SSH forced commands containing ${}, were not configurable from the Management Console.
• The ghe-ssl-ca-certificate-install command line utility did not accept a piped certificate as input.

Changes

• GitHub Enterprise is now available in the Asia Pacific (Mumbai) AWS region.

Known Issues

• We incorrectly redirect to the dashboard if you accessed GitHub Enterprise using an alias while in private mode. This might happen if you set a fully qualified domain name but the subdomain resolves correctly.
• Images uploaded to issues save with an absolute URL, so they can be broken if the hostname changes.
• On a freshly set up GitHub Enterprise without any users, an attacker could create the first admin user.
• Custom firewall rules aren't maintained during an upgrade.
• Enqueued background jobs are sometimes not purged when a repository is deleted.
• svn checkout may timeout while the repository data cache is being built. In most cases, subsequent svn checkout attempts will succeed.
• Git LFS tracked files uploaded through the web interface are incorrectly added directly to the repository.
• GitHub Enterprise clustering can not be configured without https.
• Additional white spacing can sometimes be seen above the Admin center header.
• Upgrade to 2.7.4 will fail on Running Migration if there are multiple OAuth applications named GitHub Desktop. (updated 2016-09-22)
• The initial import of the VMware OVA image may fail when deployed via vCenter Server 6.0 or 6.5. The import will succeed when performed directly on an ESXi host. (updated 2017-02-23)
• Git LFS objects may take up to an hour to replicate in a High Availability configuration. (updated 2017-02-23)
• collectd metric paths can be truncated, which causes multiple write attempts to the same file for different metrics. (updated 2017-07-10)

Errata

• The Pre-generated SSH Host Keys in GitHub Enterprise vulnerability disclosure added the ssh_host_ed25519_key in GitHub Enterprise for GitHub Enterprise 2.7.4 or greater appliances on the Amazon Web Services platform. (updated 2016-09-22)
• Editing custom messages in the Admin center doesn't provide emoji suggestions was resolved in 2.7.0. (updated 2016-09-21)
• Native emoji are lost when saving custom messages in the Admin center was resolved in 2.7.0. (updated 2016-09-21)

Thanks!

The GitHub Team