New Features
With the new features added in GitHub Enterprise 2.2.0, you can:
Changes
- We recommend a minimum of 16 GB RAM be provisioned for the GitHub Enterprise virtual machine. We now enforce a minimum amount of RAM. (updated 2015-05-04)
- The way we store repositories has been changed to improve disk usage.
- The undocumented
site/stats API endpoint has been removed.
- We've moved to using syslog-ng for the system logger, which drops support for RELP. Log forwarding will be disabled if you used RELP prior to upgrading.
- We didn't add files larger than 384 KB to the search index. We've now bumped this limit to 10 MB.
- When using LDAP authentication, user account suspension is managed by using restricted group membership. Users will be suspended or unsuspended based on their membership at login. If LDAP Sync is enabled, this process will happen automatically during a synchronization run. (updated 2015-05-21)
- LDAP Sync shows a team error indicator when an LDAP Group isn't found.
- New users will be added to their LDAP Sync-enabled teams when they log in for the first time. (updated 2015-05-08)
Bug Fixes
- Ubuntu packages have been updated to the latest bugfix versions.
- We didn't give much feedback during an upgrade, so it was hard to know if it was still progressing as expected. We include the current upgrade status on the starting page now.
- Setting up static networking could fail in some circumstances.
- LDAP user search in the site admin was limited to 1000 results. This performed poorly when searching some directories, and people are more likely to refine the search than to page through so many results, so it's now limited to 150 results.
- With SAML authentication configured, signing out and signing in again could redirect you to a page saying you were still signed out.
- When a new organization was created with LDAP sync enabled, we showed an incorrect hint about importing teams.
- LDAP users could not be suspended or renamed when LDAP sync was off.
- The Owners team was not automatically removed from LDAP sync.
- LDAP sync didn't sync members of a group where the LDAP group name contained a dot (
.).
- Wiki files larger than 500 KB were cut off when they were served, which could result in large images not loading completely.
- The
ghe-btop command line utility incorrectly dropped --help and --usage flags.
- WOFF 2.0 font files did not have their
content-type header set correctly in Pages.
- The top OAuth applications list in the site admin didn't load.
- Replication needed to be be set up again after upgrading a high availability replica. We restart replication automatically now.
- Under some circumstances, application services didn't restart properly. This could happen when restoring a backup to a new instance, which could cause a redirect to the old host if it had a different hostname, or when uploading a new license, which caused the old license to be used on some requests.
- CoffeeScript in GitHub Pages sites caused build failures.
- Converting a user to an organization failed with a billing plan error, which shouldn't have been in effect on GitHub Enterprise.
- Some API endpoints could leak the existence of a private repository.
- A complex series of actions could cause a user's profile page to load in place of their contributions graph on their profile page; profile page inception.
- MySQL could recycle unique IDs after rebooting GitHub Enterprise. This could lead to strange behavior if you delete the most recently created repository, reboot, then create a new repository.
- Removing admin SSH keys with invisible characters via the Management Console failed silently.
- Git replication could be slow and CPU intense during initial push of large or complex repositories.
- Events in the github_audit log stream were logged twice.
- Management Console sessions would timeout when accessing GitHub Enterprise in another tab.
- Bad SSL certificates could slip by validation.
- Creating the OpenVPN connection could fail, causing replication set up with
ghe-repl-setup to hang.
- Git clients could display intermittent "fatal: protocol error: bad pack header" messages when garbage collection ran while fetching a pack file that was bigger than a configured memory limit. (updated 2015-05-06)
- Avatars, release downloads, and image attachments to wikis and issues were not copied correctly by high availability replication. (updated 2015-05-20)
- Repositories with a leading dot in their name failed to replicate if they were created before replication was set up. (updated 2015-06-16)
- Mail delivery to localhost failed. (updated 2015-07-14)
Security Fixes
- Ubuntu packages have been updated to the latest security fix versions.
- LOW: Disable SSLv2 and SSLv3 in Postfix.
Repository storage changes
This release changes the way GitHub Enterprise stores repositories, which reduces disk usage by sharing Git objects between forks and improves caching performance when reading repository data. This is a major change, and has some implications you need to be aware of.
Changing the repository storage layout can take several hours if your instance contains many repositories. We're working on making this faster so if your instance contains more than 20,000 repositories (including gists and wikis) we do not recommend upgrading to GitHub Enterprise 2.2 until further notice. Everyone should now upgrade to GitHub Enterprise 2.2.2 or later, as the migration process has been made significantly faster.
You can check how many repositories you have using the Admin Stats API. For example, you can SSH into the VM and run the following command, then add together "total_repos", "total_wikis", and "total_gists":
curl -s http://127.0.0.1:1337/api/v3/enterprise/stats/all
Before upgrading
As a precaution, before your upgrade you should take a backup-utils snapshot after putting the instance in maintenance mode. We also recommend taking a disk snapshot of the user data volume.
Upgrading
The upgrade process takes care of moving repository data from the existing storage layout to the new storage layout. If you have a large amount of repository data moving the data can take some time, so we recommend that you test the upgrade on a staging instance first. You can use the test upgrade to make an estimate of how long of a maintenance window you'll need for your production instance.
After upgrading, you may notice a large number of background jobs being processed. Each job is optimizing a repository for the new storage layout, but uses a high nice value, so should have minimal impact on performance. The jobs will be enqueued in the maint_localhost jobs queue, which may have a large backlog, but it's a dedicated queue and won't block other jobs from completing.
Repository backups
For compatibility with the new repository layout, you need to upgrade backup-utils to version 2.2.
The first update taken after you upgrade will be a full backup rather than an incremental backup. This means it will take more disk space and more time to complete. Subsequent backups will be incremental again.
Other implications of the change
Some customers routinely ran Git garbage collection on their repositories. The existing repository layout maps nicely to what you can see in the user interface, so you could easily find a repository on disk at /data/repositories/[owner]/[repository].git. This is no longer the case with the new repository layout, but it does let us be smarter about running garbage collection, so running it manually shouldn't be necessary.
So that they could be restored if necessary, deleted repositories were previously moved to the /data/repositories/__purgatory__ directory. This special area for archived repositories is no longer needed or used. Repositories are kept in their normal location until purged three months after being archived.
Please contact Enterprise Support if you have any questions about this change.
Snapshot and rollback recommendations
Due to the invasive changes in the repository disk layout in GitHub Enterprise 2.2, we strongly recommend reading the upgrade guide prior to upgrading your virtual machine. This provides information about using snapshots and rolling back to a pre-upgrade state in the event an upgrade fails or is interrupted.
Upcoming deprecation of authentication using GitHub OAuth
User authentication via GitHub OAuth is being deprecated and will be removed in a future feature release. It will be removed no sooner than November 2015.
GitHub Enterprise includes support for authenticating users via OAuth to accounts on GitHub.com because it provides a simple way to set up external authentication. However, after speaking with many customers, we've found that organizations commonly have other sources they want to use to automate identity and access management.
We want to focus on features that best meet the needs of our users, so we're planning to remove support for GitHub OAuth in a future feature release and focus on making ongoing improvements to other authentication methods like SAML and LDAP.
Note that this change will only affect user authentication via GitHub.com and not personal access tokens or OAuth applications added to your GitHub Enterprise instance.
Known Issues
- Service hooks may log passwords used for HTTP Basic authentication to disk. (updated 2015-07-28)
- The management console incorrectly strips the
= character from new and current administrative SSH keys when adding or removing keys. This will cause administrative SSH access to the instance to fail for those keys.
- Upgrading to GitHub Enterprise 2.2 with a lot of repositories can take a very long time.
- We show the wrong clone URL when displaying a Gist when subdomain isolation is disabled.
- Gist repositories are not garbage collected by the maintenance scheduler.
Mail delivery to localhost fails. (updated 2015-07-14)- Deleting a user doesn't delete their gists which can cause problems with replication.
- In our instructions to merge a pull request on the command line, we show the steps to merge using the Git protocol even when private mode is on. Private mode forces authentication but the Git protocol is unauthenticated so the steps will always fail. We also don't show the steps to merge using SSH.
- We incorrectly redirect to the dashboard if you access GitHub Enterprise using an alias while in private mode. This might happen if you set a fully qualified domain name but the subdomain resolves correctly.
- Organization invitation emails are sent from the support email address rather than the noreply email address.
- The management console settings interface doesn't clearly show if you have previously uploaded certificate files or a private key.
- Jobs stuck on code indexing can delay other jobs from running.
- Dashboard activity feed links point to wrong hostname after restoring from backup if the hostname has changed.
- In some circumstances, after an upgrade we prompt you to upload a license, even though there's already a valid license.
- On a freshly set up GitHub Enterprise without any users, an attacker could create the first admin user.
- Gists can't be created when using Safari 8.x in Private Mode.
- SNMP can't be run on high availability replicas.
- Gist profile pages don't have proper styling when subdomain isolation is disabled.
- Management console sessions can expire too quickly for Safari users.
- We can fail to properly create the key for the secure connection between a high availability replica and the primary, which causes replication setup to fail.
- Custom firewall rules aren't maintained during an upgrade.
- A high availability replica that's been promoted to primary and then set up as a replica again doesn't properly show the replica status page, but shows "Starting..." instead.
- Replication setup fails for IPv6 hosts.
- It is not possible to forward logs over IPv6. (updated 2015-05-07)
- Site-wide audit logs do not appear in the site admin interface. (updated 2015-05-14)
- Setting the admin SSH password with
ghe-set-password fails. (updated 2015-05-19)
- Enabling Hyper-V Dynamic Memory causes kernel panics. (updated 2015-05-30)
- Suspended LDAP users are unsuspended if no LDAP restricted groups are configured. (updated 2015-05-30)
- We show your gravatar or identicon on Gists instead of your custom profile picture. (updated 2015-06-15)
- We display the time in the scheduled maintenance banner in UTC instead of the viewer's timezone. (updated 2015-06-18)
- Users with LDAP DNs longer than 255 characters are suspended if LDAP Sync is enabled. (updated 2015-06-19)
- Images uploaded to issues save with an absolute URL, so they can be broken if the hostname changes. (updated 2015-07-14)
- With private mode enabled, a Pages site with no default page serves a generic error rather than an informative message. (updated 2015-07-14)
- Editing a Gist can cause a 500 error. This is an authentication problem between Gist and GitHub Enterprise, so logging out and back in again should fix the problem. (updated 2015-07-15)
- Using uppercase characters in the hostname causes a redirect loop. (updated 2015-07-28)
- When a fork is detached from its repository network by an administrator or by changing visibility, its filesystem path won't be updated on a high availability replica until at least one commit has been pushed. (updated 2015-08-13)
- Updates to Wiki pages by users without a primary email address set throw errors. (updated 2015-08-25)
- Viewing raw files in repositories owned by a user or organization named "github" fails with a 400 error. (updated 2015-12-15)
- Trying to add a file to a repository with Subversion 1.9 clients incorrectly detects the file already exists and fails. (updated 2016-01-14)
Errata
- Failure to deliver mail to localhost was fixed in 2.2.0. (updated 2015-07-14)
Thanks!
The GitHub Team