Seed Propagation
Kernel Warping added locate_warp, which tracks
each point in two stages. IC-GN, the second stage, refines a warp one
iteration at a time, so it needs a warp to start from. That starting
warp is the seed.
The first stage builds the seed.
dictk.translation.locate cuts
a rigid, square kernel around the point in the reference image. It
phase-correlates that kernel against a search area of the current
image. The correlation surface peaks at the whole-pixel displacement
where the rigid kernel best matches. The seed takes that
displacement and sets all four gradient terms of
Kernel Warping's
to zero:
The seed is a rigid, whole-pixel guess. IC-GN adds the fractional displacement, the stretch, and the shear. Under Kernel Warping's 2% stretch, every seed on its 2862 points landed within 1 px of the true position:
| Points | Seeds more than 1 px off | Largest seed error (px) | Largest locate_warp error (px) |
|---|---|---|---|
| 2862 | 0 | 0.56 | 0.024 |
IC-GN then refined every point to within 0.024 px. But, when deformation grows large, the error in this initial displacement can grow, causing IC-GN to converge to an incorrect answer.
Figure 9.4-1 shows a correct seed and a defeated seed, both under a 15% stretch and a 0.15 shear. A defeated seed lands more than 1 px from the true displacement:
translation.locate's phase-correlation surface over the search area. The orange box marks its highest peak, the seed , and the red cross marks the true displacement. Right: the current image, with the seed's rigid square (dashed) and IC-GN's final warp (solid). Top: at , the peak lands at , 0.50 px from the true , and IC-GN converges to within 0.001 px. Bottom: at , no peak stands out at the true . The highest peak lands at , and IC-GN converges 28.718 px away. Each row's point has the seed error nearest the median among the sweep's correct seeds (top) or defeated seeds (bottom), as described in The Sweep.A defeated seed lands on the wrong speckles. IC-GN starts from there
and converges to a wrong warp. locate_warp raises no error when that
happens.
The Sweep
The sweep maps where seeds fail across a grid of 49 combinations of
seven stretch values and seven shear values. It stretches the image
along by 0% to 30%, and shears it by 0 to 0.30, both in steps of
0.05. Each combination tracks the same 30 points, so the sweep runs
fits. Each point sits at a
random position at least 40 px from every edge of the 300x300
astronaut0 image. Each point also gets its own random fractional
shift. The shifts fall between 4 and 8 px along and between 2 and
6 px along , the same range Rigid Versus Warped
Kernel used.
astronaut0, each at least 40 px inside the dashed square. Right: the red point under a 15% stretch and a 0.15 shear, drawn over the current image. The dashed white square is the reference kernel. The dotted orange parallelogram is that kernel deformed about the point, so its center remains stationary. The solid orange parallelogram adds the point's shift, (5.66, 4.94) px, and the red cross marks the true position. This point's shift sits nearest the median of the 30 shifts.The sweep builds a separate current image for each point. Each image stretches and shears about that point as a fixed center, so the deformation itself never moves the point. Figure 9.4-2's right panel shows this: the dotted parallelogram keeps the reference kernel's center. The point then moves by its shift alone, at most 10 px. The 25 px search margin around a 13 px kernel margin leaves 12 px of travel. The search area always contains the true position. Any failure comes from the rigid kernel's mismatch.
The sweep uses 30 random points because a single point alone can mislead. Kernel Warping's point at , swept alone, failed fewer seeds at 20% stretch with a 0.15 shear (14 of 30 shifts) than with no shear (28). The 30 random points show no such dip. At a 20% stretch, they fail 29 of 30 seeds with a 0.15 shear, and 26 of 30 seeds with no shear.
A seed fails when it lands more than 1 px from the true position. A
correct whole-pixel match lands within a 0.71 px radius (0.5 pixel in
the -axis, 0.5 pixel in the -axis,
). A locate_warp result misses
when it lands more than 1 px away.
Where the Seed Breaks
Seeds more than 1 px off, out of 30 points:
| Stretch \ Shear | 0.00 | 0.05 | 0.10 | 0.15 | 0.20 | 0.25 | 0.30 |
|---|---|---|---|---|---|---|---|
| 0% | 0 | 0 | 1 | 15 | 25 | 28 | 29 |
| 5% | 0 | 0 | 6 | 16 | 24 | 27 | 28 |
| 10% | 8 | 7 | 10 | 15 | 24 | 29 | 30 |
| 15% | 20 | 20 | 16 | 22 | 27 | 30 | 30 |
| 20% | 26 | 26 | 26 | 29 | 28 | 29 | 30 |
| 25% | 29 | 28 | 29 | 30 | 30 | 29 | 30 |
| 30% | 30 | 30 | 29 | 30 | 30 | 30 | 30 |
locate_warp misses of more than 1 px, out of 30 points:
| Stretch \ Shear | 0.00 | 0.05 | 0.10 | 0.15 | 0.20 | 0.25 | 0.30 |
|---|---|---|---|---|---|---|---|
| 0% | 0 | 0 | 1 | 9 | 22 | 25 | 27 |
| 5% | 0 | 0 | 3 | 11 | 19 | 22 | 27 |
| 10% | 5 | 4 | 8 | 13 | 20 | 25 | 29 |
| 15% | 13 | 15 | 13 | 20 | 24 | 25 | 27 |
| 20% | 22 | 23 | 24 | 26 | 25 | 28 | 27 |
| 25% | 27 | 26 | 24 | 28 | 28 | 28 | 28 |
| 30% | 29 | 29 | 25 | 29 | 29 | 30 | 30 |
Across all 1470 fits, the largest error at or under 1 px is 0.267 px, and the smallest error over it is 2.87 px.
Across the sweep, the seed fails 1095 times and locate_warp misses 972 times. IC-GN rescues 123 failed seeds. 0 misses start from a correct seed.
Both counts stay at zero up to a 5% stretch with a 0.05 shear. From
there they rise along both axes. A 10% stretch alone fails 8 seeds,
and locate_warp misses 5 points. A 0.10 shear alone fails 1 seed,
and locate_warp misses 1 point. At 15%, the two come closer. A 15%
stretch alone fails 20 seeds and misses 13 points. A 0.15 shear alone
fails 15 seeds and misses 9 points. At 20% stretch, locate_warp
misses at least 22 of 30 points at every shear.
Under a 15% stretch and a 0.15 shear, the 30 random points fail 22
seeds, and locate_warp misses 20. Kernel Warping's single point at
failed 11 seeds and missed 8, out of 30 shifts, at the
same deformation.
Each miss lands far from the true position. Of all 1470 fits, 1469
land either within 0.02 px of the true position or more than 2.87 px
from it. The one exception lands 0.267 px off, at 25% stretch and 0.10
shear. A miss puts the point on the wrong speckles, and
locate_warp's return value looks the same as a correct one.
Across the sweep, the seed fails 1095 times. IC-GN pulls 123 of those seeds back to the right position. The other 972 failed seeds become misses.
No miss starts from a correct seed, so a failed seed is a necessary condition for a miss. That does not show that a good seed is sufficient. IC-GN could still miss from a seed with the right shape under a large deformation.
To test sufficiency, the script below reruns the 1470 fits from a seed that already carries the stretch and shear. Its four gradient terms take the true deformation. Its translation takes the true shift rounded to whole pixels, written :
The rounding keeps the translation as coarse as the rigid seed's. Only the four gradient terms differ from the rigid seed.
Across all 1470 fits, 0 land more than 1 px from the true position. The largest error is 0.007 px.
The rigid seed gave 972 misses. This seed gives none, even at a 30% stretch with a 0.30 shear. A good seed, with the right stretch and shear, is sufficient.
Every miss traces back to a failed seed, so the fix belongs at the seed, not in IC-GN.
Detecting a Miss
For each point, locate_warp returns only the tracked position
in the current image, in pixels, as a SubpixelCoordinate.
A miss 28.718 px off looks the same as a correct fit 0.001 px off.
Inside locate_warp, IC-GN computes two more numbers, but
locate_warp discards both. The first is the final ZNSSD, Kernel
Warping's
. It measures how well the warped kernel matches the
current image:
- A ZNSSD of 0 is a perfect score.
- A ZNSSD of 4 is the worst score, reached when the current image shows the kernel with its contrast inverted.
The second is the iteration count. IC-GN stops once every update falls below , or after updates.
The script below reruns all 1470 fits from the sweep and records both numbers for each fit. A fit is correct when it lands within 1 px of the true position:
| Fit | Count | ZNSSD range | Median ZNSSD | Median iterations | Stopped at |
|---|---|---|---|---|---|
| Correct | 498 | 0.0000 to 0.0995 | 0.0017 | 11 | 4 |
| Miss | 972 | 0.6952 to 2.9096 | 1.4497 | 50 | 971 |
The fits the iteration count misclassifies:
| Fit | Stretch | Shear | Error (px) | ZNSSD | Iterations |
|---|---|---|---|---|---|
| Correct | 5% | 0.25 | 0.005 | 0.0050 | 50 |
| Correct | 20% | 0.30 | 0.005 | 0.0015 | 50 |
| Correct | 25% | 0.10 | 0.004 | 0.0029 | 50 |
| Correct | 25% | 0.10 | 0.267 | 0.0995 | 50 |
| Miss | 30% | 0.05 | 26.704 | 1.3003 | 43 |
ZNSSD separates the two groups with no overlap. Correct fits score at most 0.0995, and misses score at least 0.6952. Any threshold between those two values flags all 972 misses and none of the 498 correct fits. In ZNCC terms, , so every correct fit keeps a ZNCC of at least 0.950, and every miss stays at or below 0.652.
The 0.0995 belongs to the one fit from Where the Seed Breaks that landed 0.267 px off. That fit counts as correct under the 1 px rule.
The iteration count separates the groups almost as well. Correct fits took a median of 11 iterations. 971 of the 972 misses ran all 50 without converging. The count misclassifies 5 fits. Four correct fits also ran all 50 iterations. Three of those four landed within 0.005 px, but their updates never all fell below . The fourth is the 0.267 px fit. One miss, at a 30% stretch and a 0.05 shear, converged after 43 iterations, 26.704 px off. Its ZNSSD of 1.3003 still flags it.
Every miss the iteration count flags, ZNSSD also flags. On this sweep, the iteration count adds nothing that ZNSSD does not already show. The result comes from one image, one 27x27 kernel, and a 50-iteration cap. Another speckle pattern could narrow the gap between 0.0995 and 0.6952.
Propagating the Seed
A good seed needs the local stretch and shear. A rigid tracker does not
know them, but an already-fitted neighbor does.
dictk.grid.locate_warp_propagated
fits one point first, then seeds every other point from a fitted
neighbor.
The first point is the grid point nearest the image center. It has no
fitted neighbor, so it searches over shapes instead. It tries every
displacement gradient from to in
steps of 0.1, which makes trials. Each trial deforms the
reference kernel and finds that kernel's whole-pixel translation by
phase correlation, as translation.locate does for the rigid kernel.
Each trial is scored by ZNSSD at that translation. IC-GN refines the 3
best trials, and the fit with the lowest final ZNSSD becomes the first
point's warp.
Each other point starts from the warp of an already-fitted neighbor, one of its 8 nearest points. The four gradient terms carry over. The translation is extrapolated across the offset from the neighbor's reference position to this point's:
where the subscript marks the neighbor's fitted values. Only the first point uses the search area. Every other point skips the whole-pixel search.
The comparison tracks a 17x17 grid at 5 px spacing, from to , under all 49 combinations of stretch and shear. Each current image deforms about the image center, then shifts by px. That makes fits per tracker.
locate_warp gets each point's true position, rounded, as its search
center. Under a 30% stretch with a 0.30 shear, a point 40 px from the
center moves 24 px, twice the 12 px of travel its search area allows.
The rounded true positions keep every search area on the right place,
so only the rigid seed can fail locate_warp.
locate_warp_propagated gets no search centers:
locate_warp misses of more than 1 px, out of 289 points:
| Stretch \ Shear | 0.00 | 0.05 | 0.10 | 0.15 | 0.20 | 0.25 | 0.30 |
|---|---|---|---|---|---|---|---|
| 0% | 0 | 0 | 26 | 136 | 238 | 262 | 280 |
| 5% | 1 | 2 | 44 | 184 | 252 | 278 | 278 |
| 10% | 31 | 44 | 91 | 200 | 257 | 279 | 281 |
| 15% | 149 | 173 | 196 | 233 | 268 | 282 | 284 |
| 20% | 242 | 251 | 265 | 259 | 272 | 285 | 284 |
| 25% | 272 | 277 | 282 | 279 | 284 | 287 | 285 |
| 30% | 280 | 288 | 284 | 283 | 282 | 284 | 287 |
| Tracker | Misses, of 14161 | Largest error (px) |
|---|---|---|
locate_warp | 10561 | 47.898 |
locate_warp_propagated | 0 | 0.009 |
locate_warp misses 26 of 289 points at a 0.10 shear alone, and 31 at
a 10% stretch alone. At a 30% stretch alone, it misses 280.
locate_warp_propagated misses none of its 14161 fits, and its largest
error is 0.009 px.
Containing a Bad Fit
One wrong fit can spread, because every point it seeds starts from a
wrong warp. Detecting a Miss found that ZNSSD
separates correct fits from misses. Any threshold between 0.0995 and
0.6952 separated all 1470 fits. locate_warp_propagated uses such a
threshold as a gate, znssd_limit, with a default of 0.3. That value
sits inside the gap, 3 times above 0.0995 and 2.3 times below 0.6952.
Three rules follow from the gate:
- A fit seeds other points only if its ZNSSD is at most 0.3.
- The accepted point with the lowest ZNSSD seeds next. A point that fails the gate tries again when another accepted neighbor reaches it.
- If the first point's fit fails the gate, the start search moves to the next point nearest the image center, at least one kernel width (27 px) from every failed start. It stops after 10 failed starts in a row.
locate_warp_propagated returns each point's ZNSSD alongside its
position. A ZNSSD above 0.3 marks a position not to trust. A point
never fitted gets a ZNSSD of and a position of NaN.
The test covers a 40x40 px patch of the current image with unrelated speckle, under a 15% stretch with a 0.15 shear, on the same 17x17 grid. The patch sits at one of two corners. At , the start point stays clear of it. At , it hides the image center, where the start search begins. A point counts as inside when its kernel in the current image overlaps the patch. A point is bad when it lands more than 1 px from the true position, or is never fitted:
| Patch corner | Gate | Points outside | Bad outside | Bad inside | Never fitted | Bad but passing the gate | Right but failing the gate |
|---|---|---|---|---|---|---|---|
| (160, 160) | off | 188 | 0 | 27 of 101 | 0 | 0 | 62 |
| (160, 160) | on | 188 | 0 | 66 of 101 | 63 | 0 | 23 |
| (130, 130) | off | 127 | 127 | 162 of 162 | 0 | 0 | 0 |
| (130, 130) | on | 127 | 0 | 91 of 162 | 88 | 0 | 46 |
With the patch over the center and no gate, all 127 points outside the patch go bad. The start fit lands inside the patch, and its wrong warp seeds every other point. With the gate, none of the 127 go bad. The start search moves on until a start outside the patch passes.
In every row, no bad point passes the gate. Every bad position carries a ZNSSD above 0.3, or none at all.
The gate has a cost inside the patch. With the patch off-center and no gate, 62 points land within 1 px of the true position, yet score above 0.3. A kernel partly covered by the patch raises the ZNSSD of a correct fit. The sweep in Detecting a Miss had no such partial cover. With the gate on, those points stop the propagation, and 64 points inside the patch are never fitted. The off-center patch then leaves 66 of its 101 inside points bad, against 27 without the gate.
Two limits remain. Every point here shares one stretch and shear, so a neighbor's gradient terms match this point's exactly. A strain gradient would make the extrapolated seed less exact. The start search tries gradient terms only up to 0.3.
Continue to Postprocessing.
seed_propagation_occlusion.py
Show the code
"""Can one bad region spread? Covers a 40x40 px patch of the current
image with unrelated speckle, under a 15% stretch with a 0.15 shear,
and tracks the 17x17 grid with `locate_warp_propagated` with its ZNSSD
gate off (`znssd_limit=4`, the largest ZNSSD) and on (the default 0.3).
"""
import numpy as np
from scipy.ndimage import affine_transform
from dictk.grid import generate, locate_warp_propagated
from dictk.image import PixelCoordinate, read
from dictk.rosta import rosta
CENTER = np.array([149.5, 149.5])
SHIFT = np.array([2.3, 1.6])
F = np.array([[1.15, 0.15], [0.0, 1.0]])
PATCH = 40
KERNEL_MARGIN = 13
GATE = 0.3
reference_image = read(path="astronaut0.png").astype(np.float64)
points = generate(
origin=PixelCoordinate(x=110, y=110),
count_x=17,
count_y=17,
spacing_x=5,
spacing_y=5,
)
reference = np.array([[p.x, p.y] for p in points], dtype=np.float64)
inverse = np.linalg.inv(F)[::-1, ::-1]
offset = CENTER[::-1] - inverse @ (CENTER[::-1] + SHIFT[::-1])
deformed = affine_transform(reference_image, inverse, offset=offset, order=5)
true = CENTER + (reference - CENTER) @ F.T + SHIFT
speckle = rosta(width=PATCH, height=PATCH, density=0.5, random_seed=7)
speckle = speckle.astype(np.float64) * 255.0 / speckle.max()
print(
"| Patch corner | Gate | Points outside | Bad outside | Bad inside "
"| Never fitted | Bad but passing the gate | Right but failing the gate |"
)
print("|---|---|---|---|---|---|---|---|")
for corner in [(160, 160), (130, 130)]:
x0, y0 = corner
current_image = deformed.copy()
current_image[y0 : y0 + PATCH, x0 : x0 + PATCH] = speckle
# A point is inside when its current-image kernel overlaps the patch.
reach = PATCH / 2 + KERNEL_MARGIN + 0.5
middle = np.array([x0, y0]) + (PATCH - 1) / 2
inside = np.all(np.abs(true - middle) < reach, axis=1)
for gate, limit in [("off", 4.0), ("on", GATE)]:
found, znssds = locate_warp_propagated(
reference_image=reference_image,
current_image=current_image,
reference_points=points,
kernel_margin_width=KERNEL_MARGIN,
kernel_margin_height=KERNEL_MARGIN,
search_margin_width=25,
search_margin_height=25,
znssd_limit=limit,
)
errors = np.linalg.norm(np.array([[f.x, f.y] for f in found]) - true, axis=1)
# A never-fitted point has a nan position, so it counts as bad.
bad = ~(errors <= 1.0)
print(
f"| ({x0}, {y0}) | {gate} | {(~inside).sum()} | {bad[~inside].sum()} "
f"| {bad[inside].sum()} of {inside.sum()} | {np.isinf(znssds).sum()} "
f"| {(bad & (znssds <= GATE)).sum()} "
f"| {(~bad & (znssds > GATE)).sum()} |"
)
seed_propagation_grid.py
Show the code
"""`locate_warp` against `locate_warp_propagated` on a 17x17 grid at
5 px spacing, across the sweep's 49 stretch and shear combinations.
Each current image deforms about the image center, then shifts by
(2.3, 1.6) px. `locate_warp` gets each point's true position, rounded,
as its search center, so its search area always contains the true
position and only the rigid seed can fail it. `locate_warp_propagated`
gets no search centers.
"""
import numpy as np
from scipy.ndimage import affine_transform
from dictk.grid import generate, locate_warp, locate_warp_propagated
from dictk.image import PixelCoordinate, read
STRETCHES = np.round(np.arange(0.0, 0.31, 0.05), 2)
SHEARS = np.round(np.arange(0.0, 0.31, 0.05), 2)
MISS_LIMIT = 1.0
CENTER = np.array([149.5, 149.5])
SHIFT = np.array([2.3, 1.6])
reference_image = read(path="astronaut0.png").astype(np.float64)
points = generate(
origin=PixelCoordinate(x=110, y=110),
count_x=17,
count_y=17,
spacing_x=5,
spacing_y=5,
)
reference = np.array([[p.x, p.y] for p in points], dtype=np.float64)
margins = dict(
kernel_margin_width=13,
kernel_margin_height=13,
search_margin_width=25,
search_margin_height=25,
)
names = ["`locate_warp`", "`locate_warp_propagated`"]
miss_counts = np.zeros((2, len(STRETCHES), len(SHEARS)), dtype=int)
largest = np.zeros(2)
for i, stretch in enumerate(STRETCHES):
for j, shear in enumerate(SHEARS):
F = np.array([[1.0 + stretch, shear], [0.0, 1.0]])
# affine_transform maps each output (row, column) back to its
# input position, so it takes the inverse deformation, in (y, x)
# order.
inverse = np.linalg.inv(F)[::-1, ::-1]
offset = CENTER[::-1] - inverse @ (CENTER[::-1] + SHIFT[::-1])
current_image = affine_transform(
reference_image, inverse, offset=offset, order=5
)
true = CENTER + (reference - CENTER) @ F.T + SHIFT
search_centers = [
PixelCoordinate(x=int(x), y=int(y)) for x, y in np.round(true)
]
for k, (locate, extra) in enumerate(
[
(locate_warp, dict(search_centers=search_centers)),
(locate_warp_propagated, {}),
]
):
found = locate(
reference_image=reference_image,
current_image=current_image,
reference_points=points,
**margins,
**extra,
)
if locate is locate_warp_propagated:
found, _ = found
errors = np.linalg.norm(
np.array([[f.x, f.y] for f in found]) - true, axis=1
)
miss_counts[k, i, j] = (errors > MISS_LIMIT).sum()
largest[k] = max(largest[k], errors.max())
print(f"{names[0]} misses of more than {MISS_LIMIT:g} px, out of {len(points)} points:")
print()
print("| Stretch \\ Shear | " + " | ".join(f"{g:.2f}" for g in SHEARS) + " |")
print("|---|" + "---|" * len(SHEARS))
for stretch, row in zip(STRETCHES, miss_counts[0]):
print(f"| {stretch:.0%} | " + " | ".join(str(c) for c in row) + " |")
print()
fits = len(points) * len(STRETCHES) * len(SHEARS)
print(f"| Tracker | Misses, of {fits} | Largest error (px) |")
print("|---|---|---|")
for name, counts, error in zip(names, miss_counts, largest):
print(f"| {name} | {counts.sum()} | {error:.3f} |")
seed_propagation_deformed_seed.py
Show the code
"""How many misses remain when IC-GN starts from the right stretch and
shear? Reruns the sweep's 1470 fits, seeding each from the true
deformation with its translation rounded to whole pixels, as coarse as
the rigid seed's.
"""
import numpy as np
from scipy.ndimage import affine_transform
from dictk import warp
from dictk.image import PixelCoordinate, read
MARGIN = 13
SAMPLE_COUNT = 30
STRETCHES = np.round(np.arange(0.0, 0.31, 0.05), 2)
SHEARS = np.round(np.arange(0.0, 0.31, 0.05), 2)
MISS_LIMIT = 1.0
reference_image = read(path="astronaut0.png")
reference_gradients = warp._gradients(reference_image)
rng = np.random.default_rng(0)
pivots = rng.integers(40, 261, size=(SAMPLE_COUNT, 2)).astype(np.float64)
shifts = np.round(rng.uniform([4, 2], [8, 6], size=(SAMPLE_COUNT, 2)), 2)
errors = []
for stretch in STRETCHES:
for shear in SHEARS:
F = np.array([[1.0 + stretch, shear], [0.0, 1.0]])
inverse = np.linalg.inv(F)[::-1, ::-1]
for pivot, shift in zip(pivots, shifts):
point = PixelCoordinate(x=int(pivot[0]), y=int(pivot[1]))
offset = pivot[::-1] - inverse @ (pivot[::-1] + shift[::-1])
current_image = affine_transform(
reference_image.astype(np.float64), inverse, offset=offset, order=5
)
initial_warp = np.eye(3)
initial_warp[:2, :2] = F
initial_warp[:2, 2] = np.round(shift)
fitted, _, _ = warp._refine(
reference_image=reference_image,
reference_gradients=reference_gradients,
current_coefficients=warp._spline_coefficients(current_image),
reference_point=point,
initial_warp=initial_warp,
kernel_margin_width=MARGIN,
kernel_margin_height=MARGIN,
max_iterations=50,
tolerance=1e-6,
)
errors.append(np.hypot(*(fitted[:2, 2] - shift)))
errors = np.array(errors)
print(
f"Across all {errors.size} fits, {(errors > MISS_LIMIT).sum()} land more than "
f"{MISS_LIMIT:g} px from the true position. The largest error is "
f"{errors.max():.3f} px."
)
seed_propagation_quality.py
Show the code
"""Can a wrong fit be detected? Reruns the sweep's 1470 fits, the same
49 stretch and shear combinations at the same 30 points, and records
each fit's final ZNSSD and IC-GN iteration count alongside its error.
"""
import matplotlib.pyplot as plt
import numpy as np
from scipy.ndimage import affine_transform
from dictk import translation, warp
from dictk.image import PixelCoordinate, read
plt.rcParams.update({"font.family": "serif", "mathtext.fontset": "cm"})
MARGIN = 13
SEARCH_MARGIN = 25
SAMPLE_COUNT = 30
MAX_ITERATIONS = 50
STRETCHES = np.round(np.arange(0.0, 0.31, 0.05), 2)
SHEARS = np.round(np.arange(0.0, 0.31, 0.05), 2)
MISS_LIMIT = 1.0
reference_image = read(path="astronaut0.png")
reference_gradients = warp._gradients(reference_image)
rng = np.random.default_rng(0)
pivots = rng.integers(40, 261, size=(SAMPLE_COUNT, 2)).astype(np.float64)
shifts = np.round(rng.uniform([4, 2], [8, 6], size=(SAMPLE_COUNT, 2)), 2)
errors, znssds, iterations, cells = [], [], [], []
for stretch in STRETCHES:
for shear in SHEARS:
inverse = np.linalg.inv(np.array([[1.0 + stretch, shear], [0.0, 1.0]]))
inverse = inverse[::-1, ::-1]
for pivot, shift in zip(pivots, shifts):
point = PixelCoordinate(x=int(pivot[0]), y=int(pivot[1]))
offset = pivot[::-1] - inverse @ (pivot[::-1] + shift[::-1])
current_image = affine_transform(
reference_image.astype(np.float64), inverse, offset=offset, order=5
)
seed = translation.locate(
reference_image=reference_image,
current_image=current_image,
reference_point=point,
search_center=point,
kernel_margin_width=MARGIN,
kernel_margin_height=MARGIN,
search_margin_width=SEARCH_MARGIN,
search_margin_height=SEARCH_MARGIN,
)
fitted, znssd, count = warp._refine(
reference_image=reference_image,
reference_gradients=reference_gradients,
current_coefficients=warp._spline_coefficients(current_image),
reference_point=point,
initial_warp=warp._translation_warp(reference_point=point, start=seed),
kernel_margin_width=MARGIN,
kernel_margin_height=MARGIN,
max_iterations=MAX_ITERATIONS,
tolerance=1e-6,
)
errors.append(np.hypot(*(fitted[:2, 2] - shift)))
znssds.append(znssd)
iterations.append(count)
cells.append((stretch, shear))
errors, znssds, iterations = map(np.array, (errors, znssds, iterations))
missed = errors > MISS_LIMIT
stalled = iterations == MAX_ITERATIONS
print(
f"| Fit | Count | ZNSSD range | Median ZNSSD | Median iterations "
rf"| Stopped at $n_{{\text{{MAX\_ITER}}}} = {MAX_ITERATIONS}$ |"
)
print("|---|---|---|---|---|---|")
for name, group in [("Correct", ~missed), ("Miss", missed)]:
print(
f"| {name} | {group.sum()} "
f"| {znssds[group].min():.4f} to {znssds[group].max():.4f} "
f"| {np.median(znssds[group]):.4f} "
f"| {np.median(iterations[group]):.0f} "
f"| {stalled[group].sum()} |"
)
print()
print("The fits the iteration count misclassifies:")
print()
print("| Fit | Stretch | Shear | Error (px) | ZNSSD | Iterations |")
print("|---|---|---|---|---|---|")
for name, group in [
("Correct", ~missed & stalled),
("Miss", missed & ~stalled),
]:
for k in np.flatnonzero(group):
stretch, shear = cells[k]
print(
f"| {name} | {stretch:.0%} | {shear:.2f} | {errors[k]:.3f} "
f"| {znssds[k]:.4f} | {iterations[k]} |"
)
fig, (left, right) = plt.subplots(1, 2, figsize=(10, 4.2), constrained_layout=True)
bins = np.logspace(np.log10(znssds.min()) - 0.1, np.log10(znssds.max()) + 0.1, 60)
for group, color, label in [
(~missed, "tab:blue", "correct fit"),
(missed, "tab:orange", "miss"),
]:
left.hist(znssds[group], bins=bins, color=color, alpha=0.8, label=label)
right.hist(
iterations[group],
bins=np.arange(0.5, MAX_ITERATIONS + 1.5),
color=color,
alpha=0.8,
label=label,
)
left.set_xscale("log")
left.set_yscale("log")
left.set_xlabel("final ZNSSD")
left.set_ylabel("fits")
left.legend()
right.set_yscale("log")
right.set_xlabel("IC-GN iterations")
right.set_ylabel("fits")
right.legend()
fig.savefig("seed_propagation_quality.png", dpi=300)
seed_propagation_kernel_warping_seeds.py
Show the code
"""Every whole-pixel seed on Kernel Warping's 2862 points, under its 2%
stretch: how far each seed lands from the true position, and how far
`locate_warp` lands after IC-GN refines it.
"""
import numpy as np
from dictk import translation
from dictk.grid import generate, locate_warp
from dictk.image import PixelCoordinate, read, stretch
FACTOR_X = 1.02
SEED_LIMIT = 1.0
reference_image = read(path="astronaut0.png")
current_image = stretch(arr=reference_image, factor_x=FACTOR_X)
points = generate(
origin=PixelCoordinate(x=18, y=16),
count_x=53,
count_y=54,
spacing_x=5,
spacing_y=5,
)
kwargs = dict(
reference_image=reference_image,
current_image=current_image,
kernel_margin_width=13,
kernel_margin_height=13,
search_margin_width=25,
search_margin_height=25,
)
seeds = [
translation.locate(reference_point=point, search_center=point, **kwargs)
for point in points
]
found = locate_warp(reference_points=points, **kwargs)
# `image.stretch` pivots on the image origin, so each true x is the
# reference x times the stretch factor, and y is unchanged.
true_positions = np.array([[point.x * FACTOR_X, point.y] for point in points])
seed_errors = np.linalg.norm(
np.array([[seed.x, seed.y] for seed in seeds]) - true_positions, axis=1
)
warp_errors = np.linalg.norm(
np.array([[f.x, f.y] for f in found]) - true_positions, axis=1
)
print(
f"| Points | Seeds more than {SEED_LIMIT:g} px off | Largest seed error (px) "
"| Largest `locate_warp` error (px) |"
)
print("|---|---|---|---|")
print(
f"| {len(points)} | {(seed_errors > SEED_LIMIT).sum()} "
f"| {seed_errors.max():.2f} | {warp_errors.max():.3f} |"
)
seed_propagation_seed.py
Show the code
"""What a seed is: `translation.locate`'s whole-pixel match, the rigid
starting warp IC-GN refines. Two of the sweep's 30 points under a 15%
stretch and a 0.15 shear, one with a correct seed (top row) and one
with a defeated seed (bottom row).
Each row picks the point whose seed error sits nearest the median seed
error of its group, so neither example is hand-picked.
"""
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.patches import Polygon
from scipy.ndimage import affine_transform
from dictk import translation, warp
from dictk.image import PixelCoordinate, read
plt.rcParams.update({"font.family": "serif", "mathtext.fontset": "cm"})
MARGIN = 13
SEARCH_MARGIN = 25
EDGE = 40
SAMPLE_COUNT = 30
SEED_LIMIT = 1.0
F = np.array([[1.15, 0.15], [0.0, 1.0]])
ZOOM = 30
reference_image = read(path="astronaut0.png")
rng = np.random.default_rng(0)
pivots = rng.integers(EDGE, 261, size=(SAMPLE_COUNT, 2)).astype(np.float64)
shifts = np.round(rng.uniform([4, 2], [8, 6], size=(SAMPLE_COUNT, 2)), 2)
inverse = np.linalg.inv(F)[::-1, ::-1]
def deformed(pivot, shift):
offset = pivot[::-1] - inverse @ (pivot[::-1] + shift[::-1])
return affine_transform(
reference_image.astype(np.float64), inverse, offset=offset, order=5
)
def tracked(pivot, shift):
point = PixelCoordinate(x=int(pivot[0]), y=int(pivot[1]))
current_image = deformed(pivot, shift)
kwargs = dict(
reference_image=reference_image,
current_image=current_image,
reference_point=point,
search_center=point,
kernel_margin_width=MARGIN,
kernel_margin_height=MARGIN,
search_margin_width=SEARCH_MARGIN,
search_margin_height=SEARCH_MARGIN,
)
seed = translation.locate(**kwargs)
return current_image, seed, warp.fit(**kwargs)
seed_errors = np.array(
[
np.hypot(seed.x - pivot[0] - shift[0], seed.y - pivot[1] - shift[1])
for pivot, shift in zip(pivots, shifts)
for _, seed, _ in [tracked(pivot, shift)]
]
)
rows = []
for failed in (False, True):
group = np.flatnonzero((seed_errors > SEED_LIMIT) == failed)
rows.append(
group[np.argmin(np.abs(seed_errors[group] - np.median(seed_errors[group])))]
)
square = np.array(
[[-MARGIN, -MARGIN], [MARGIN, -MARGIN], [MARGIN, MARGIN], [-MARGIN, MARGIN]],
dtype=np.float64,
)
fig, axes = plt.subplots(2, 3, figsize=(13, 8.6), constrained_layout=True)
for (kernel_ax, surface_ax, current_ax), index, label in zip(
axes, rows, ["correct seed", "defeated seed"]
):
pivot, shift = pivots[index], shifts[index]
x, y = int(pivot[0]), int(pivot[1])
current_image, seed, fitted = tracked(pivot, shift)
true_position = pivot + shift
seed_offset = np.array([seed.x, seed.y]) - pivot
kernel_ax.imshow(reference_image, cmap="gray", vmin=0, vmax=255)
kernel_ax.add_patch(
Polygon(
pivot + square,
fill=False,
edgecolor="white",
linestyle="--",
linewidth=1.5,
)
)
kernel_ax.plot(x, y, marker="o", color="red", markersize=7, mec="black")
kernel_ax.set_xlim(x - ZOOM, x + ZOOM)
kernel_ax.set_ylim(y + ZOOM, y - ZOOM)
kernel_ax.set_title(f"{label}: reference kernel at ({x}, {y})")
# locate's own phase correlation, step for step as its
# skimage.registration.phase_cross_correlation call computes it: the
# kernel centered in a zero-padded copy the search area's size, the
# cross-power spectrum normalized to unit magnitude, and the peak of
# the inverse transform's magnitude. After fftshift, the surface's
# index minus SEARCH_MARGIN is the displacement from the point.
kernel = reference_image[y - MARGIN : y + MARGIN, x - MARGIN : x + MARGIN]
search = current_image[
y - SEARCH_MARGIN : y + SEARCH_MARGIN, x - SEARCH_MARGIN : x + SEARCH_MARGIN
]
pad = SEARCH_MARGIN - MARGIN
kernel_padded = np.pad(kernel.astype(np.float64), pad)
product = np.fft.fft2(search) * np.fft.fft2(kernel_padded).conj()
product /= np.maximum(np.abs(product), 100 * np.finfo(np.float64).eps)
surface = np.fft.fftshift(np.abs(np.fft.ifft2(product)))
peak_row, peak_column = np.unravel_index(np.argmax(surface), surface.shape)
assert (peak_column - SEARCH_MARGIN, peak_row - SEARCH_MARGIN) == tuple(seed_offset)
extent = [-SEARCH_MARGIN - 0.5, SEARCH_MARGIN - 0.5] * 2
surface_ax.imshow(surface, cmap="viridis", extent=extent[:2] + extent[2:][::-1])
surface_ax.plot(
*seed_offset, marker="s", markersize=9, mfc="none", mec="orange", mew=2
)
surface_ax.plot(*shift, marker="+", color="red", markersize=14, mew=2)
surface_ax.set_xlabel("$u$ (pixels)")
surface_ax.set_ylabel("$v$ (pixels)")
surface_ax.set_title(
f"phase correlation: seed $(u_0, v_0) = ({seed_offset[0]:.0f}, "
f"{seed_offset[1]:.0f})$"
)
current_ax.imshow(current_image, cmap="gray", vmin=0, vmax=255)
current_ax.add_patch(
Polygon(
pivot + seed_offset + square,
fill=False,
edgecolor="orange",
linestyle="--",
linewidth=2,
label="seed: rigid square",
)
)
current_ax.add_patch(
Polygon(
pivot + fitted[:2, 2] + square @ fitted[:2, :2].T,
fill=False,
edgecolor="orange",
linewidth=2,
label="IC-GN's final warp",
)
)
current_ax.plot(*true_position, marker="+", color="red", markersize=14, mew=2)
error = np.hypot(*(fitted[:2, 2] - shift))
current_ax.set_xlim(x - ZOOM, x + ZOOM)
current_ax.set_ylim(y + ZOOM, y - ZOOM)
current_ax.set_title(f"current image: IC-GN misses by {error:.3f} px")
current_ax.legend(loc="lower left", fontsize=9, framealpha=0.85)
for ax in axes[:, [0, 2]].ravel():
ax.set_xlabel("x (pixels)")
ax.set_ylabel("y (pixels)")
fig.savefig("seed_propagation_seed.png", dpi=300)
seed_propagation_points.py
Show the code
"""The sweep's 30 points on `astronaut0` (left), and one point's own
deformation, pivoted on that point, then shifted (right).
Draws the same points and shifts as seed_propagation_baseline.py, from
the same seed. The close-up uses the point whose shift sits nearest the
median shift, so the example is typical, not hand-picked.
"""
import matplotlib.pyplot as plt
import numpy as np
from matplotlib.patches import Polygon, Rectangle
from scipy.ndimage import affine_transform
from dictk.image import read
plt.rcParams.update({"font.family": "serif", "mathtext.fontset": "cm"})
MARGIN = 13
EDGE = 40
SAMPLE_COUNT = 30
# 15% stretch with 0.15 shear: inside the sweep, and large enough to see.
F = np.array([[1.15, 0.15], [0.0, 1.0]])
ZOOM = 30
reference_image = read(path="astronaut0.png")
height, width = reference_image.shape
rng = np.random.default_rng(0)
pivots = rng.integers(EDGE, 261, size=(SAMPLE_COUNT, 2)).astype(np.float64)
shifts = np.round(rng.uniform([4, 2], [8, 6], size=(SAMPLE_COUNT, 2)), 2)
typical = int(np.argmin(np.linalg.norm(shifts - np.median(shifts, axis=0), axis=1)))
pivot, shift = pivots[typical], shifts[typical]
inverse = np.linalg.inv(F)[::-1, ::-1]
offset = pivot[::-1] - inverse @ (pivot[::-1] + shift[::-1])
current_image = affine_transform(
reference_image.astype(np.float64), inverse, offset=offset, order=5
)
square = np.array(
[[-MARGIN, -MARGIN], [MARGIN, -MARGIN], [MARGIN, MARGIN], [-MARGIN, MARGIN]],
dtype=np.float64,
)
deformed = square @ F.T
fig, (left, right) = plt.subplots(1, 2, figsize=(10, 5.2), constrained_layout=True)
left.imshow(reference_image, cmap="gray", vmin=0, vmax=255)
left.add_patch(
Rectangle(
(EDGE, EDGE),
width - 1 - 2 * EDGE,
height - 1 - 2 * EDGE,
fill=False,
edgecolor="white",
linestyle="--",
linewidth=1.5,
)
)
others = np.delete(pivots, typical, axis=0)
left.scatter(*others.T, s=30, color="orange", edgecolor="black", linewidth=0.5)
left.scatter(*pivot, s=60, color="red", edgecolor="black", linewidth=0.5)
left.set_title(f"{SAMPLE_COUNT} reference points, at least {EDGE} px from each edge")
left.set_xlabel("x (pixels)")
left.set_ylabel("y (pixels)")
right.imshow(current_image, cmap="gray", vmin=0, vmax=255)
right.add_patch(
Polygon(
pivot + square,
fill=False,
edgecolor="white",
linestyle="--",
linewidth=1.5,
label="reference kernel",
)
)
right.add_patch(
Polygon(
pivot + deformed,
fill=False,
edgecolor="orange",
linestyle=":",
linewidth=2,
label="deformed about the point",
)
)
right.add_patch(
Polygon(
pivot + shift + deformed,
fill=False,
edgecolor="orange",
linewidth=2,
label="then shifted",
)
)
right.annotate(
"",
xy=pivot + shift,
xytext=pivot,
arrowprops=dict(arrowstyle="-|>", color="orange", linewidth=2),
)
right.plot(*pivot, marker="o", color="red", markersize=7, mec="black")
right.plot(*(pivot + shift), marker="+", color="red", markersize=14, mew=2)
right.set_xlim(pivot[0] - ZOOM, pivot[0] + ZOOM + shift[0])
right.set_ylim(pivot[1] + ZOOM + shift[1], pivot[1] - ZOOM)
right.set_title(
f"point ({pivot[0]:.0f}, {pivot[1]:.0f}), shift "
f"({shift[0]:.2f}, {shift[1]:.2f}) px\n15% stretch, 0.15 shear"
)
right.set_xlabel("x (pixels)")
right.set_ylabel("y (pixels)")
right.legend(loc="lower left", fontsize=9, framealpha=0.85)
fig.savefig("seed_propagation_points.png", dpi=300)
seed_propagation_baseline.py
Show the code
"""Where `locate_warp`'s rigid whole-pixel seed breaks: a sweep of
stretch and shear, each cell tracked at the same 30 random points, each
under its own random fractional shift.
Random points, not one, because a single point's image content can make
some deformations look easier than smaller ones. Each deformation
pivots on its tracked point, so that point's own displacement is just
the shift. The shift never exceeds 10 px, inside
the 12 px a 25 px search margin leaves around a 13 px kernel margin.
The search area therefore always contains the true position, and any
failure comes from the rigid kernel's mismatch alone.
"""
import numpy as np
from scipy.ndimage import affine_transform
from dictk import translation, warp
from dictk.image import PixelCoordinate, read
MARGIN = 13
SEARCH_MARGIN = 25
SAMPLE_COUNT = 30
STRETCHES = np.round(np.arange(0.0, 0.31, 0.05), 2)
SHEARS = np.round(np.arange(0.0, 0.31, 0.05), 2)
# A correct whole-pixel match lies within sqrt(0.5^2 + 0.5^2) = 0.71 px
# of the truth. More than 1 px means at least one whole pixel off.
SEED_LIMIT = 1.0
MISS_LIMIT = 1.0
reference_image = read(path="astronaut0.png")
rng = np.random.default_rng(0)
# Points at least 40 px from every edge of the 300x300 image, so the
# 25 px search area plus an 8 px shift stays inside it.
pivots = rng.integers(40, 261, size=(SAMPLE_COUNT, 2)).astype(np.float64)
# The same shift range as Kernel Warping, rounded to 0.01 px so each
# shift prints exactly as used.
shifts = np.round(rng.uniform([4, 2], [8, 6], size=(SAMPLE_COUNT, 2)), 2)
def deformed(F, pivot, shift):
# affine_transform maps each output (row, column) back to its input
# position, so it takes the inverse deformation, in (y, x) order.
inverse = np.linalg.inv(F)[::-1, ::-1]
offset = pivot[::-1] - inverse @ (pivot[::-1] + shift[::-1])
return affine_transform(
reference_image.astype(np.float64), inverse, offset=offset, order=5
)
seed_errors = np.zeros((len(STRETCHES), len(SHEARS), SAMPLE_COUNT))
warp_errors = np.zeros_like(seed_errors)
for i, stretch in enumerate(STRETCHES):
for j, shear in enumerate(SHEARS):
F = np.array([[1.0 + stretch, shear], [0.0, 1.0]])
for k, (pivot, shift) in enumerate(zip(pivots, shifts)):
point = PixelCoordinate(x=int(pivot[0]), y=int(pivot[1]))
current_image = deformed(F, pivot, shift)
true_position = pivot + shift
for errors, locate in [
(seed_errors, translation.locate),
(warp_errors, warp.locate),
]:
found = locate(
reference_image=reference_image,
current_image=current_image,
reference_point=point,
search_center=point,
kernel_margin_width=MARGIN,
kernel_margin_height=MARGIN,
search_margin_width=SEARCH_MARGIN,
search_margin_height=SEARCH_MARGIN,
)
errors[i, j, k] = np.hypot(
found.x - true_position[0], found.y - true_position[1]
)
seed_counts = (seed_errors > SEED_LIMIT).sum(axis=2)
miss_counts = (warp_errors > MISS_LIMIT).sum(axis=2)
def table(counts, title):
print(f"{title}, out of {SAMPLE_COUNT} points:")
print()
print("| Stretch \\ Shear | " + " | ".join(f"{g:.2f}" for g in SHEARS) + " |")
print("|---|" + "---|" * len(SHEARS))
for stretch, row in zip(STRETCHES, counts):
print(f"| {stretch:.0%} | " + " | ".join(str(c) for c in row) + " |")
print()
table(seed_counts, f"Seeds more than {SEED_LIMIT:g} px off")
table(miss_counts, f"`locate_warp` misses of more than {MISS_LIMIT:g} px")
converged = warp_errors[warp_errors <= MISS_LIMIT]
missed = warp_errors[warp_errors > MISS_LIMIT]
print(
f"Across all {warp_errors.size} fits, the largest error at or under "
f"{MISS_LIMIT:g} px is {converged.max():.3f} px, and the smallest error "
f"over it is {missed.min():.2f} px."
)
seed_failed = seed_errors > SEED_LIMIT
warp_missed = warp_errors > MISS_LIMIT
print()
print(
f"Across the sweep, the seed fails {seed_failed.sum()} times and "
f"`locate_warp` misses {warp_missed.sum()} times. IC-GN rescues "
f"{(seed_failed & ~warp_missed).sum()} failed seeds. "
f"{(warp_missed & ~seed_failed).sum()} misses start from a correct seed."
)