GitHub Enterprise 2.7.4 September 20, 2016 Series notes · Download

The 2.7 series release notes contain important changes in this release series.

Pre-generated SSH Host Keys in GitHub Enterprise

A CRITICAL issue was identified for all 2.x versions of GitHub Enterprise. The GitHub Enterprise images contain pre-generated SSH host keys that were not regenerated upon installation for all supported platforms:

This means an attacker with the capability to perform a man-in-the-middle attack on SSH traffic can intercept and modify network traffic to the GitHub Enterprise appliance.

The affected supported versions are:

This vulnerability was found and reported internally and we have no evidence that it has been exploited in the wild.

We strongly recommend upgrading your GitHub Enterprise appliance to the latest patch release in your series, GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, or 2.3.21. In addition, with backup-utils-2.7.1, ghe-backup and ghe-restore will check for any leaked SSH host keys in the snapshot(s).

Please contact GitHub Enterprise Support if you have questions.

--

Verification and Mitigation on GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater

If you've upgraded to the latest patch release, GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater,

  1. SSH to your primary GitHub Enterprise appliance.

  2. Check for leaked SSH host keys using the ghe-ssh-check-host-keys utility.

    $ ghe-ssh-check-host-keys
    

    The utility should output either:

    One or more of your SSH host keys were found in the blacklist.
    Please reset your host keys using ghe-ssh-roll-host-keys.
    

    --

    The SSH host keys were not found in the SSH host key blacklist.
    No additional steps are needed/recommended at this time.
    
  3. If one or more SSH host keys were found in the blacklist, continue to the next step. Otherwise, your GitHub Enterprise environment is not vulnerable.

  4. Put your GitHub Enterprise environment in Maintenance Mode.

  5. Rotate all SSH host keys using the ghe-ssh-roll-host-keys utility.

    $ sudo ghe-ssh-roll-host-keys
    

    The utility should output:

    $ SSH host keys have successfully been rolled.
    

If you've upgraded to GitHub Enterprise 2.7.4, 2.6.9, or greater, and you are using the High Availability Configuration, there are no additional steps to take on your replica appliance.

If you've upgraded to GitHub Enterprise 2.5.14, 2.4.17, 2.3.21, or greater, and you are using the High Availability Configuration,

  1. After completing steps 1-5, stop replication on the replica appliance.

    $ ghe-repl-stop
    
  2. Synchronize the SSH host keys from the primary appliance.

    $ ghe-repl-setup
    
  3. Resume replication on the replica appliance.

    $ ghe-repl-start
    

If you've upgraded to GitHub Enterprise 2.7.4, 2.6.9, 2.5.14 or greater, and you are using Clustering,

  1. After completing steps 1-5, apply the changes to all cluster nodes.

    $ ghe-cluster-config-apply
    

--

Verification and Mitigation if Immediate Upgrade is not Possible

If you're unable to upgrade immediately to the latest patch release, GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater,

  1. SSH to your primary GitHub Enterprise appliance.

  2. Download the list of leaked SSH host keys and verify its content using any of the provided hashes.

    $ curl -O https://enterprise.github.com/security/2016-09-20/ghe-ssh-leaked-host-keys-list.txt
    $ sha256sum ghe-ssh-leaked-host-keys-list.txt
    3bb29658784a4059a41f1a77cffba9586baab179ba07b795f80e12a9f10c5665  ghe-ssh-leaked-host-keys-list.txt
    $ sha1sum ghe-ssh-leaked-host-keys-list.txt
    5db799da044da9aae0bcfc523d22e7ce0fe72550  ghe-ssh-leaked-host-keys-list.txt
    $ md5sum ghe-ssh-leaked-host-keys-list.txt
    de75bcb0bf1d13e15620952c0af8da41  ghe-ssh-leaked-host-keys-list.txt
    
  3. Print the fingerprint of your GitHub Enterprise appliance's SSH host keys.
    Note: The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used in 2.7.4 or greater.

    $ ssh-keygen -lf /etc/ssh/ssh_host_dsa_key.pub
    1024 b2:69:82:2f:25:48:bb:fc:62:c7:9a:de:41:42:13:55 /etc/ssh/ssh_host_dsa_key.pub (DSA)
    $ ssh-keygen -lf /etc/ssh/ssh_host_ecdsa_key.pub
    256 c0:cb:fd:07:33:e9:62:14:6b:fb:d5:26:54:f3:c5:0d /etc/ssh/ssh_host_ecdsa_key.pub (ECDSA)
    $ ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub
    256 d6:92:21:4b:04:3b:22:f5:ee:85:0a:63:bf:b3:fe:9b /etc/ssh/ssh_host_ed25519_key.pub (ED25519)
    $ ssh-keygen -lf /etc/ssh/ssh_host_rsa_key.pub
    2048 0f:ee:8d:02:2d:e1:76:f3:eb:f5:af:cb:38:9a:1c:33 /etc/ssh/ssh_host_rsa_key.pub (RSA)
    
  4. Check for leaked SSH host keys by comparing against the downloaded list of leaked SSH host keys.

  5. If one or more SSH host keys were found in the blacklist, continue to the next step. Otherwise, your GitHub Enterprise environment is not vulnerable.

  6. Put your GitHub Enterprise environment in Maintenance Mode.

  7. Remove all SSH host keys.

    $ sudo rm -f /etc/ssh/ssh_host_*
    
  8. Regenerate the SSH host keys.
    Note: The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used and regenerated for in 2.7.4 or greater.

    $ sudo ssh-keygen -t ed25519 -N "" -f /etc/ssh/ssh_host_ed25519_key
    $ sudo dpkg-reconfigure openssh-server
    
  9. Apply the changes to the ssh and babeld service.

    $ sudo cp /etc/ssh/ssh_host_{rsa,dsa,ecdsa,ed25519}_key{,.pub} /data/user/common/
    $ sudo chown babeld:babeld /data/user/common/ssh_host_{rsa,dsa,ecdsa,ed25519}_key{,.pub}
    

If you're unable to upgrade immediately to GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater, and you are using the High Availability Configuration,

  1. After completing steps 1-9, stop replication on the replica appliance.
$ ghe-repl-stop
  1. Synchronize the SSH host keys from the primary appliance.
$ ghe-repl-setup
  1. Resume replication on the replica appliance.
$ ghe-repl-start

If you're unable to upgrade immediately to GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, or greater, and you are using Clustering,

  1. After completing steps 1-9, apply the changes to all cluster nodes.
$ ghe-cluster-config-apply

--

Post SSH Host Key Rotation

After rotating the SSH host keys, your GitHub Enterprise environment can exit Maintenance Mode.

Your end-users will receive an error message when attempting to use the Administrative Shell (SSH) or the SSH protocol for Git activity. The rotation does not affect users using the HTTPS protocol for Git activity.

For example, the following is an output from the command-line,

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:seFT9eIOmAZWbfcO9yU1sXiEYIqcrdi0qttbtmNm0Io.
Please contact your system administrator.
Add correct host key in /Users/monalisa/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /Users/monalisa/.ssh/known_hosts:42
ECDSA host key for [github.example.com]:122 has changed and you have requested strict checking.
Host key verification failed.

After updating the known_hosts, end-users will be prompted to accept a new fingerprint.

$ ssh -p 122 admin@github.example.com
The authenticity of host '[github.example.com]:122 ([169.254.1.1]:122)' can't be established.
ECDSA key fingerprint is SHA256:seFT9eIOmAZWbfcO9yU1sXiEYIqcrdi0qttbtmNm0Io.
Are you sure you want to continue connecting (yes/no)?

We strongly recommend publishing your GitHub Enterprise appliance's SSH host key fingerprints in a location that is accessible to all your end-users. For example, for GitHub.com, we publish the SSH fingerprints at https://help.github.com/articles/what-are-github-s-ssh-key-fingerprints/.

If you'd like to to give end-users notice before rotating the SSH host keys, follow the instructions in the Verification and Mitigation if Immediate Upgrade is not Possible skipping step 7 and replacing step 8 with,

  1. Regenerate the SSH host keys.
    Note: The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used and regenerated for in 2.7.4 or greater.

    i. Pre-generate new SSH host keys to a temporary directory.

    $ ssh-keygen -t dsa -N "" -f /var/tmp/ssh_host_dsa_key
    $ ssh-keygen -t rsa -N "" -f /var/tmp/ssh_host_rsa_key
    $ ssh-keygen -t ecdsa -N "" -f /var/tmp/ssh_host_ecdsa_key
    $ ssh-keygen -t ed25519 -N "" -f /var/tmp/ssh_host_ed25519_key
    

    ii. Print the fingerprint of your GitHub Enterprise appliance's SSH host keys for tentative rotation.

    $ ssh-keygen -lf /var/tmp/ssh_host_dsa_key.pub
     1024 b2:69:82:2f:25:48:bb:fc:62:c7:9a:de:41:42:13:55 /var/tmp/ssh_host_dsa_key.pub (DSA)
    $ ssh-keygen -lf /var/tmp/ssh_host_ecdsa_key.pub
     256 c0:cb:fd:07:33:e9:62:14:6b:fb:d5:26:54:f3:c5:0d /var/tmp/ssh_host_ecdsa_key.pub (ECDSA)
    $ ssh-keygen -lf /var/tmp/ssh_host_ed25519_key.pub
     256 d6:92:21:4b:04:3b:22:f5:ee:85:0a:63:bf:b3:fe:9b /var/tmp/ssh_host_ed25519_key.pub (ED25519)
    $ ssh-keygen -lf /var/tmp/ssh_host_rsa_key.pub
    248 0f:ee:8d:02:2d:e1:76:f3:eb:f5:af:cb:38:9a:1c:33 /var/tmp/ssh_host_rsa_key.pub (RSA)
    

    iii. Once you are ready to migrate to the new, rotated SSH host keys, move the host keys from the temporary directory and apply the changes to the ssh service.

    $ sudo mv /var/tmp/ssh_host_{rsa,dsa,ecdsa,ed25519}_key{,.pub} /etc/ssh
    $ sudo service ssh restart
    

    iv. Continue with steps 9 in the Verification and Mitigation if Immediate Upgrade is not Possible section.

ssh_host_ed25519_key in GitHub Enterprise

The 2.x versions of GitHub Enterprise on all supported platforms:

contain a pre-generated ssh_host_ed25519_key. However, only GitHub Enterprise 2.7.4 or greater use the ssh_host_ed25519_key. This can be verified by checking your GitHub Enterprise appliance's /etc/ssh/sshd_config, which added HostKey /etc/ssh/ssh_host_ed25519_key in 2.7.4 or greater.

The ssh_host_ed25519_key may exist on your GitHub Enterprise appliance but is only used in 2.7.4 or greater.

If you've upgraded your appliance to 2.7.4 or greater on any of the supported platforms including Amazon Web Services, please follow the instructions in the Verification and Mitigation on GitHub Enterprise 2.7.4, 2.6.9, 2.5.14, 2.4.17, 2.3.21, or greater section.

Security Fixes

Bug Fixes

Changes

Known Issues

Errata

Thanks!

The GitHub Team