git - "failed to push some refs" error after using BFG to clean repo

Question

I have a GitHub repository that I am trying to shrink. I've used the BFG repo cleaner to do this, followed by the typical gc commands. This did reduce my repository size (though it's still 3x larger than it should be – but that's another point).

bfg -B 100 #delete the 100 biggest blobs
git reflog expire --expire=now --all
git gc --prune=now --aggressive

Before this point my repo is up to date with origin master. When I git add, and commit after this series of cleaning commands, I am told nothing to commit, working tree clean. However, when I git push, I get the following error.

error: failed to push some refs to 'https://github.com/REPONAME'
hint: Updates were rejected because the remote contains work that you do
hint: not have locally. This is usually caused by another repository pushing
hint: to the same ref. You may want to first integrate the remote changes
hint: (e.g., 'git pull ...') before pushing again.

git fetch followed by git merge fixes this error, but then all the size reduction I had done above is undone, sometimes even doubling the size of the repository. How do I avoid this?

Answer 1

Any command that "modifies commits" in a repository—this includes The BFG, git filter-branch, and git filter-repo—actually works by copying the commits. The old commits, which have something wrong with them, are extracted. Each old commit is examined to see what should be corrected. If it is fine as is, the program can leave it alone, but otherwise, any necessary changes (such as deleting a big file, or changing the parent linkage to point back to an earlier-changed commit) are applied and the program makes a new and improved commit that is meant to be used instead of the original.

When the process is done, some—perhaps as many as all, but in your case only some—commits have been copied to new-and-improved ones. Your task now becomes convincing every other clone of this repository to use your new-and-improved commits instead of their current commits.

Git is not built to do this. Git prefers to add new commits atop the existing ones. That's what a fetch-and-merge does, for instance. If the new and improved commits are almost-every-commit, this process approximately doubles the size of the repository, as we now have all the old commits plus all the new-and-improved ones.

When you run git push and get the error you're seeing, that's the other Git saying, in effect: If I accept what you sent and do what you asked, I'll lose all these great commits. How about you take all of these and add them to your repository first, and then have me add yours to mine without losing any of mine?

Of course, you want them to lose all their commits. Their commits are the old and lousy ones you want replaced.¹ So you need to override this error. Git has a way to do that, using git push --force. GitHub, on the other hand, has some added-value features that can make this fail, if you like. So if you run into GitHub stopping you, you must at least temporarily turn off the added features, so that you can destroy the old commits over on GitHub.²

In any case, once you have convinced the Git over on GitHub to drop the old-and-lousy commits in favor of the new-and-improved ones, you're mostly done; but see footnotes, and remember: maybe other (third-party) people have clones of your GitHub clone. You'll need to convince them to update, too.

---

¹Note that if some third party might be adding new commits to your GitHub repository, they might have added some since you cloned and ran The BFG, or whatever, so that you did not actually copy all the commits. To prevent this from happening, make sure that you're the only one who can push to the GitHub repository, at least temporarily, before you start the whole process.

²They generally won't go away right away, which is good when you do this by mistake, and bad when you are doing it on purpose. How long they stick around is hard to predict: to really toss them now, you need to get the GitHub folks involved. If you don't mind them hanging around on GitHub, occupying GitHub's disk space but not yours, there's no need to worry about them.