Automating coregistration, Part 1

11/18/2020

Note: The blog post below essentially describes the first part of my auto-coregistration program. The second part, which aims to achieve subpixel precision, is described here.


Imagine you have a few satellite images that each provide complementary information. Perhaps they are images taken at different times and therefore represent a time series. Or perhaps they each come from a different part of the electromagnetic spectrum (say, visible, infrared, and radar). Whatever the case, mining the image set for value is made much easier if all of the images exactly align, so that each point in one image corresponds to the same point on Earth (or whatever planet you're looking at!) in each other image.

Aligning images like this is called coregistration. If one of the images is also georeferenced-that is, encoded with coordinates specifying where on the planet it is-then coregistering the other images to that image also georeferences those images, as long as you're including geospatial coordinates in the coregistration.

Personally, I find coregistering to be an annoying task. One reason for this is that, somewhat selfishly, I instinctively feel like every image I download should already be exactly georeferenced. Of course, that's not the case, and it also isn't even practical. Another reason coregistration can be frustrating is that it can be hard to find features in a pair of images that you are confident are the same feature (e.g., a boulder or ridge), and even once found, precisely clicking on the same equivalent part of that feature in each image (e.g., a kink in the ridge) can be tricky, especially if the images have different resolutions. Incidentally, this difficulty is, appropriately enough, called the "correspondence problem" in computer vision. The third reason is that manual coregistration disrupts my workflow. For example, perhaps I can write a program that does much of the processing that I want, say steps A, B, E, D, and F, but step D is coregistering. If only I could automate coregistering, I could save myself the frustration of manually coregistering and have a simple, one-pass workflow.

A partial solution that I've used in some projects is the "Auto Georeference" feature of ESRI's ArcMap (and ArcGIS Pro). It works quite well when it works, but sometimes it fails when there is enough overlap between images and sufficient resolution for coregistration, at least, by a human. I found such failures to be common when I recently started working with radar data, likely because of radar speckle and/or radar sensitivity to viewing geometry. (The same feature can appear differently in radar depending on the angle at which your take the image.) In addition, ESRI's auto georeferencing can only be activated manually, by clicking within the GUI, and you must click once for each image that you wish to coregister, and wait for the process to complete before selecting the next image, so even this automated functionality still disrupts my workflow. Finally, ESRI's software is commercial, so it would be nice to find an open source alternative. 

My first Google search for such an alternative didn't yield a promising result (more on that later), so I decided I could justify taking an afternoon to see if I could cobble together a solution. After all, I have potentially 60 images I want to coregister, and I wasn't looking forward to coregistering each manually! 

I knew I wanted to use Python. It's the programming language I know best, and I also know that Python is used extensively in image analysis. Therefore, I began my educating myself a little on feature detection in general, and how it works in Python specifically, starting here.

Based on that tutorial, I developed a workflow. When that failed miserably, I tweaked it and ultimately developed the following workflow:

  1. Read in each image.
  2. Detect features using the ORB method.
    ORB is an alternative to the SIFT and SURF methods, specifically developed for the OpenCV library. Unlike those alternatives, it does not patented and is therefore free to use.
  3. Use brute force to match features between images, retaining the two best candidates in each case.
    (a) Brute force tests every feature in image A against every feature in B, and vice versa.
    (b) Consider: Feature A1 in image A might best match to feature B1 in image B, with a second-best match to feature B2 in image B. The A1, B1, and B2 trio are therefore retained.
    (c) Although it may not be essential, I also enable cross-checking, which discards a feature match if the best match for a feature in A to a feature in B is not also the best match for a feature in B to a feature in A.
  4. I discard each feature match for which the second-best match is "nearly" as strong as the best match.
    This technique, called a ratio test, is useful to avoid false matches. Intuitively, if you're not sure whether A1 matches to B1 or B2, you should discard the match altogether. In my initial testing, I found that requiring the A1-B2 match to be <70% as strong as the A1-B1 match was a reasonable threshold.
  5. I then find what subset of remaining feature matches are consistent with a perspective transform.
    a. As is conventional when solving the correspondence problem, I use the RANdom SAmple Consensus (RANSAC) method. RANSAC iteratively identifies a random subset of feature matches, tests that the subset features in B are reasonably consistent with being the same corresponding features in A if B is merely viewed from a different perspective than A (with some allowance for noise), extends the subset to include other feature matches that are also consistent with this viewing geometry, and iterates until a good enough subset is found (subject to criteria) or the permitted number of iterations is exhausted.
  6. Repeat all above steps after downsampling images A and B by binning each square of 4 pixels to 1 new pixel.
  7. Repeat Step 6 until no matches are found in a run (that is, in an execution of Step 6). Then choose that result (tied to a resolution) which has the most matched features (after filtering by perspective transform, that is, Step 5).
    Steps 6 and 7 may not be necessary for low-noise data. However, in my test, which coregistered two Sentinel-1 radar images, they helped significantly.
  8. I calculate the geographic coordinates of each retained feature in A and B and save these as ground control points for image B.
    The coordinates in A are treated as the "true" coordinates, and the coordinates in B are treated as "erroneous" coordinates and replaced by the corresponding coordinates from A.

I'm happy with the final results (above)! (The upper image above shows the locations of the matched features. The lower image above shows a side-by-side comparison before and after coregistration, with an animated version here. In that image, the top margin of the lava flow should not change, but parts of the bottom margin do change because the flow was active during this time interval.) However, as it turns out, the exercise may end here. When I did a second Google search, this time searching for auto coregistration rather than auto georefrencing, I found a purpose-built algorithm and implementation by Daniel Scheffler called AROSICS. Unlike my approach, which relies on identifying characteristic features, it draws from the other class of approaches and matches the intensity patterns of the image. Just as I'd hoped, it is based in Python and open source. Below you can see the results of using AROSICS to coregister LANDSAT-8 and Sentinel-2 data, with the non-coregistered version on the left (see animated version here). I have yet to test it, but it looks promising!

Ethan I. Schaefer
All rights reserved 2020
Powered by Webnode
Create your website for free! This website was made with Webnode. Create your own for free today! Get started