Merging vendor branches

Hardware vendors claim they support Linux when they give you some patches that add drivers or BSPs for their particular chip. Any open source developer can tell you this is a recipe for disaster, and I have a recent experience proving the point. This article describes what went wrong, and how I was able to work around it.

A customer had bought a development board for the TI OMAP3 processor from Logic, and a slave board with a camera chip. The development board came with a series of 80 patches on the 2.6.28 kernel. The camera came with a single patch file, and a readme specifying it was based on TI’s OMAP35x-PSP-SDK-setuplinux- This SDK is a series of 450 patches on a particular commit of Tony Lindgren’s linux-omap-2.6 repository.

The task at hand was to merge the TI patches with the Logic patches, and finally apply the camera patch on top of that. My first idea was to apply the Logic patches on top of the 2.6.29 kernel rather than 2.6.28, and continue from there. That concept was quickly given up, because the first patch already failed on 200 hunks (it was a huge patch…).

So instead of starting with a big merge mess, I decided to unleash the power of git. The first step was relatively easy: apply the patch series to the correct commit. git-am is your friend here. Even that takes a little scripting if the patches aren’t git patches: add some correctly formed mail headers and a half-way relevant commit message (in my case, just the patch filename). I quickly ended up with a 2.6.28-logic branch and a 2.6.29-ti-omap branch. Great!

Of course, merging these two branches just gave me 400 conflicts. Instead of a simple merge, rebasing would be the way to go. That would allow me to solve conflicts as they occur, and more importantly: to be able to bisect any problems in the merged result. But of course, the rebase gives me just the same 200 failed hunks as the original patch attempt.

The solution I choose was to do the rebasing step-by-step. Instead of jumping ahead directly from 2.6.28 to 2.6.29, I choose some intermediate commit and started rebasing against that. Using git-rerere (and of course the indispensible git-mergetool, with kdiff3 as the back-end), I recorded the merge resolutions, so they would be relatively easy to re-apply in the next rebase cycle. I was lucky enough to find a commit that gave me only 20 conflicts on the first patch from the 2.6.28-logic branch, and also the rest of the rebase went relatively smoothly. The merged code did not work out-of-the-box on the target board (seems I did something wrong in one of the resolutions). This was immediate proof that it was a good idea to work with these incremental rebases: it allowed me to tackle one problem at a time.

Next step was to rebase again against a later commit in the 2.6.29-ti-omap branch. Unfortunately, git-rerere did not help as much as I had hoped. It does not reuse the old conflict resolution if the merge sources are not exactly the same. So as soon as there was an edit in the 2.6.29-ti-omap branch, I had to re-resolve the conflicts. This is understandable, because merges and automatic conflict resolution should be done conservatively. Indeed, a change somewhere else in the file could invalidate my previous resolution. Unfortunately, the OMAP3 support saw a lot of evolution in the 2.6.29 era, so I had to re-resolve a lot of conflicts.

Only afterwards, I realized what I was doing wrong. Quite a few of the patches in the 2.6.28-logic branch were actually coming from the linux-omap-2.6 branch. Only, the individual commits were not exactly the same as the ones in the linux-omap tree, so git didn’t recognize the similarity. A much better solution would have been to apply the 2.6.28-logic patches one-by-one, and to find a commit in the linux-omap-2.6 branch that comes close to each patch. A git merge -s ours at that point would give me a much better starting point for subsequent rebases. Of course, the difficulty lies in finding a corresponding commit. For this article, I tried it with a brute-force approach: for i in `git rev-list v2.6.27-omap1..2.6.29-ti-omap`; do (echo -n "$i:"; git diff --shortstat "$i" 2.6.28-logic) | sort -n -k 2 | less. After letting it run for 3 hours to browse through 20000 commits, it found a commit which was very close. The differences are meaningless: whitespace, naming, ….  Starting from that point would probably have saved me a day or two…

Next time, I’ll be sure to use git properly.

  1. First put everything in different branches of the repository, so you can easily go back and forth.
  2. Use rebasing only for changes that are really exclusive to a particular branch.
  3. Use merging for changes that are shared between branches.

The difficulty for the rebasing and merging is to find a good starting point.  AFAIK, no good tools exist for that…


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s