I came upon a situation where a target release branch was unintentionally merged into a base release branch, and several commits were added afterwards. I describe how it was fixed and the git history rewritten cleanly without the faulty commit.

Merge strategy between release branches

At work, the main repository is comprised of release branches. That means, that if we have 3 releases: 1.0.0, 1.1.0, and the next major 2.X.Y, we will have the following branches:

• 1.0.0: release/1.0
• 1.1.0: release/1.1
• release/1 is the branch for next minor versions of release 1
• 2.X.Y: release/2 as not released yet, but being worked on
• master is always the next major version

There are usually several releases maintained in parallel. Branches are merged from lower release to upper release to ensure that all fixes from the previous releases are in the next one. So the usual merge route is:

• release/1.0 to release/1.1
• release/1.1 to release/1
• release/1 to release/2
• release/2 to master

The process is pretty straightforward, even if sometimes tedious to bubble up fixes from one release to another. Each merge is done via a merge commit, to keep all commits in the history.

Problems happen

It happens that a target branch was unintentionally merged into the base branch. For example release/2 was merged into release/1 while it should have been the opposite. It happened while resolving a merge conflict in the Github UI (look at the big warning rectangle).
As a consequence, release/1 now had the commits for release/2, which we absolutely do not want…

Why can’t the commit just be reverted

As described extensively on this page, reverting the faulty commit with git revert -m1 ${FAULTY_COMMIT} will revert the data, but do not modify the history of merged commits. So in our case when release/2 was merged into release/1, doing:

• git checkout release/1
• git revert -m1 ${FAULTY_COMMIT}
• git checkout release/2
• git merge release/1

would result in also reverting the data from release/2 that were merged in ${FAULTY_COMMIT}.

If the problem was found immediately, we could have just reset the branch to before the commit, and push force. However, it was only detected a few days later, where other Pull Requests were already merged. The task was now a bit more complicated.

Steps to rewrite the history

To fix this problem, it was decided to reset the branch to the commit before the faulty merge, and reapply the merges or individual commits that were done sequentially on the base branch. To do so, the repository was put into read-only mode to prevent further conflicts.

Select commits to rewrite the history

• Checkout the branch to rewrite: git checkout -b release/1
• Make a backup of the faulty branch: git branch faulty-release/1
• List all commits made on that branch: git log --pretty=oneline ${COMMIT_BEFORE_FAULTY_COMMIT}.. --first-parent
• Reset the branch to git reset --hard ${COMMIT_BEFORE_FAULTY_COMMIT}
• Reapply all commits in reverse order that were listed in the git log (except the faulty commit):
  • If it is a merge from a release branch, we want to keep its history, so reapply it: git merge release/X
  • If it is a merge commit from a Pull Request, cherry-pick it: git cherry-pick -m1 ${hash}, that would be the equivalent of squash-merging
  • If it is a single commit, also cherry-pick it: git cherry-pick ${hash}

Resolve conflicts along the way if they happen.

Check the content of the branch

After the history has been rewritten, we want to check that the content of the rewritten branch (release/1) is the same than the faulty branch (faulty-release/1) with the faulty merge removed:

• Checkout the faulty branch that was saved earlier: git checkout faulty-release/1
• Revert the faulty commit: git revert -m1 ${FAULTY_COMMIT}
• Check the diff with the rewritten branch: git diff HEAD..release/1, the diff should be empty
• If the diff is not empty, check the modified files to identify the commits were the changes were introduced and pinpoint what has been done wrong or what was forgotten. Steps above may need to be redone.

Additionally, run the test suite to ensure everything is OK.

Override the faulty branch

If the diff is empty, the branch has been corrected as if all wrongs commits were removed. It is now ok to override the faulty branch on the remote: git push --force release/1.

Post cleanup

Now that the release branch has been fixed, it is time to identify and clean up what has been left:

• Notice all developers that the branch has been rewritten, and to override the rewritten locally with the remote
• Identify all Pull Requests having the faulty commits and redo them, starting from the rewritten branch
• Identify all branches that were started with the faulty commit in their history, to start again from the new rewritten branch, in both cases it should just be cherry-picking the commits that were added
• Check mergeability of the rewritten branch on the other upper release branches: the modified history will be kept in the other release branches, but that is ok because all changes from release/1 must also be on release/2. Affected individual files will have a weird history with 2 commits with the same name and content, but that was acceptable in our case.
• Put the repository back in write-mode for everybody

Conclusion

The history of the faulty branch was successfully rewritten. There was a downtime of a couple of hours where other developers could not push their changes, but it was manageable.
As a follow-up, we want to try to:

• Detect faulty merges as soon as possible via an automated task on our CI, a faulty manual error can always happen
• Try to disable the Github merge UI interface, which was the source of the problem