Monday, 21 March 2016

Quick notes on unbreaking gitolite

Some quick notes on de-fuck-ifying a gitolite3 installation.

Here is post-hoc explanation of what probably happened:

  • Installed gitolite3 on debian using a key called
  • Lost the private half.
  • "Fixed" the problem with a new key called, this was manually inserted into .ssh/authorized_keys instead of redoing the install.
  • Stuff worked (for about 5 months) long enough to dispel any suspicious that it was all funky and rotten underneath.
  • Tried to update the admin repo to add a new repo - all hell broke loose.


After committing the admin change, things got kind of weird and then this happened:

git clone ssh:// admin_face
Cloning into 'admin_face'...
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

The key is in an agent that has been running for months, so that seems to be unpossible!

At this point I realised that I had screwed the admin key and googled how to fix it. The instructions for gitolite3 say:

When in doubt, run 'gitolite setup' anyway; it doesn't do any harm, though it may take a minute or so if you have more than a few thousand repos!

This is not in any way true. It is entirely possible that running setup on a live install will break that install. It is not a safe operation at all. Do not believe the lies: there is no cake.

After using this to install the new key something really bad happened, it looks like this:

FATAL: R any gitolite-admin rsa_git DENIED by fallthru
(or you mis-spelled the reponame)
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

So at this point things look really bad as access to the admin interface has been lost. This can be confirmed by trying to ssh directly into the git user on the server which shows this:

PTY allocation request failed on channel 0 
hello rsa_git, this is git@face running gitolite3 3.6.1-2 (Debian) on git 2.1.4

R W testing


A bit more googling shows that people then tend to panic and wipe their gitolite install and install from scratch to fix this. Instead I will now quote again from the gitolite3 docs:

Don't panic!

First, have a poke around the git user directory (if you no longer have a way to do this, i.e. you do not have root on the box, then go ahead and panic, that's probably the right approach). .gitolite/logs is very interesting and lets you reconstruct what has happened. More importantly:

.gitolite/conf is where the bare contents of your admin repo get blasted into!

So fix .gitolite/conf/gitolite.conf first to regain access (i.e. change the keynames on every repo).
This will not do anything at all, so note the presence of a file called that is obviously a cache. Delete it. This still does not work but at least the error message changes to indicate it needs that file. Finally, just run:

gitolite3 compile

This will regenerate the conf cache properly and let you back in. Problem solved, panic avoided