3.2.1 Fundamental Shadow Matrix
Figure
5 shows a shadow receiver plane
\(\Pi \) and two point lights at positions
\(\mathbf {l}_1\) and
\(\mathbf {l}_2\). Shadow
\(\mathbf {s}\) from a shadow caster at an unknown position cast by light
\(\mathbf {l}_1\) has corresponding shadows
\(\mathbf {s}'\) cast by light
\(\mathbf {l}_2\). These can be found on an epipolar line arising from a fundamental matrix, which we will derive now analogous to the derivation of the standard fundamental matrix.
Let
\(\mathbf {c}\) be the caster position,
\(\mathbf {l}_1\ne \mathbf {l}_2\) be the light positions in calibration target coordinates,
\(\mathbf {L}_i\) be
\(\mathbf {l}_i\)’s light matrix,
\(\mathbf {L}_1^{+}=(\mathbf {L}_1^\top \mathbf {L}_1)^{-1}\mathbf {L}_1^\top \) be
\(\mathbf {L}_1\)’s pseudo inverse,
\( {\varvec{\emptyset }}_{\mathbf {L}_1}\in \textsf {null}(\mathbf {L}_1)\) be a non-zero vector in
\(\mathbf {L}_1\)’s one-dimensional null space, and
\(\eta \) be a scalar. We then have
$$\begin{aligned} \lambda _1\tilde{\mathbf {s}}_1&= \mathbf {L}_1\tilde{\mathbf {c}}\Rightarrow \tilde{\mathbf {c}} = \lambda _1\mathbf {L}_1^{+}\tilde{\mathbf {s}}_1 + \eta \,{\varvec{\emptyset }}_{\mathbf {L}_1}\\ \lambda _2\tilde{\mathbf {s}}_2&= \mathbf {L}_2\tilde{\mathbf {c}}= \lambda _1\mathbf {L}_2\mathbf {L}_1^{+}\tilde{\mathbf {s}}_1 + \eta \,\mathbf {L}_2\,{\varvec{\emptyset }}_{\mathbf {L}_1}. \end{aligned}$$
Multiplying
\(\tilde{\mathbf {s}}_2^\top \left[ \mathbf {L}_2{\varvec{\emptyset }}_{\mathbf {L}_1}\right] _\times \) (with
\(\left[ \cdot \right] _\times \) being the cross-product’s matrix form) from the left, we obtain
$$\begin{aligned} \lambda _2\underbrace{\tilde{\mathbf {s}}_2^\top \left[ \mathbf {L}_2 {\varvec{\emptyset }}_{\mathbf {L}_1}\right] _\times \tilde{\mathbf {s}}_2}_{0}&= \lambda _1\tilde{\mathbf {s}}_2^\top \left[ \mathbf {L}_2 {\varvec{\emptyset }}_{\mathbf {L}_1}\right] _\times \mathbf {L}_2\mathbf {L}_1^{+}\tilde{\mathbf {s}}_1 \nonumber \\&\quad +\eta \,\tilde{\mathbf {s}}_2^\top \underbrace{\left[ \mathbf {L}_2{\varvec{\emptyset }}_{\mathbf {L}_1}\right] _\times \mathbf {L}_2{\varvec{\emptyset }}_{\mathbf {L}_1}}_{\mathbf {0}}\nonumber \\ \Rightarrow \quad 0&= \tilde{\mathbf {s}}_2^\top \underbrace{\left[ \mathbf {L}_2{\varvec{\emptyset }}_{\mathbf {L}_1}\right] _\times \mathbf {L}_2\mathbf {L}_1^{+}}_{\mathbf {F}} \tilde{\mathbf {s}}_1. \end{aligned}$$
(5)
Thus, corresponding shadows
\(\tilde{\mathbf {s}}_1\) and
\(\tilde{\mathbf {s}}_2\) fulfill a condition with some fundamental matrix
\(\mathbf {F}\) that is directly analogous to the regular correspondence condition. For
\(\mathbf {L}_{i}=\left[ {\begin{matrix} -1 &{} 0 &{} {l_x^{(i)}}/{l_z^{(i)}} &{} 0\\ 0 &{} -1 &{} {l_y^{(i)}}/{l_z^{(i)}} &{} 0\\ 0 &{} 0 &{} 1/{l_z^{(i)}} &{} -1 \end{matrix}}\right] \), we obtain the null space vector
\({\varvec{\emptyset }}_{\mathbf {L}_1} \propto \left[ {l_x^{(1)}}, {l_y^{(1)}}, {l_z^{(1)}}, 1\right] ^\top \) and finally the fundamental shadow matrix
$$\begin{aligned} \mathbf {F} =&\left[ \mathbf {L}_2{\varvec{\emptyset }}_{\mathbf {L}_1}\right] _\times \mathbf {L}_2\mathbf {L}_1^{+}\nonumber \\&\quad \propto \frac{1}{{l_z^{(2)}}} \left[ {\begin{matrix}0 &{} -{l_z^{(1)}} + {l_z^{(2)}} &{} -{l_y^{(1)}}{l_z^{(2)}}+{l_y^{(2)}}{l_z^{(1)}}\\ {l_z^{(1)}}-{l_z^{(2)}}&{}0&{}{l_x^{(1)}}{l_z^{(2)}}-{l_x^{(2)}}{l_z^{(1)}}\\ {l_y^{(1)}}{l_z^{(2)}}-{l_y^{(2)}}{l_z^{(1)}}&{}-{l_x^{(1)}}{l_z^{(2)}}+{l_x^{(2)}}{l_z^{(1)}}&{}0\end{matrix}}\right] \nonumber \\ =&\begin{bmatrix}0&{}f_1&{}f_2\\ -f_1&{}0&{}f_3\\ -f_2&{}-f_3&{}0\end{bmatrix}. \end{aligned}$$
(6)
Interestingly, this matrix is a special case of regular fundamental matrices: It is skew-symmetric. For corresponding shadows
\(\tilde{\mathbf {s}}_1 = [u, v, 1]^\top \) and
\(\tilde{\mathbf {s}}_2 = [u', v', 1]^\top \) the correspondence condition Eq. (
5) becomes
$$\begin{aligned} 0&=\begin{bmatrix}u&v&1\end{bmatrix}\begin{bmatrix} 0&{}f_1&{}f_2\\ -f_1&{}0&{}f_3\\ -f_2&{}-f_3&{} 0\end{bmatrix}\begin{bmatrix}u'\\ v'\\ 1\end{bmatrix}\nonumber \\&=(uv'-vu')f_1+(u-u')f_2+(v-v')f_3 \end{aligned}$$
(7)
We can thus estimate the parameters
\(f_1\),
\(f_2\), and
\(f_3\) of our fundamental shadow matrix up to scale by solving the homogeneous linear system
$$\begin{aligned} \begin{bmatrix} u_1v_1'-v_1u_1' &{} u_1-u_1' &{} v_1-v_1'\\ \vdots &{} \vdots &{} \vdots \\ u_nv_n'-v_nu_n' &{} u_n-u_n' &{} v_n-v_n' \end{bmatrix} \begin{bmatrix} f_1\\ f_2\\ f_3 \end{bmatrix} =\mathbf {0}_{n\times 1} \end{aligned}$$
(8)
with
\(n\ge 2\) correspondences using singular value decomposition. This is actually equivalent to estimating regular essential matrices for cameras that only undergo pure translation and no relative rotation (Szeliski
2010, Eq. 7.27). This makes sense since we saw in Eq. (
2) that point lights act like cameras with an identity rotation. As a consequence, all properties discussed in the following also hold for essential matrix estimation of cameras with pure translation.
Conveniently, the fundamental shadow matrix has a rank of 2 as a direct result of estimating the parameters of a skew-symmetric matrix and the rank does not need to be enforced in a post-processing step. Further, the matrix can be estimated up to scale from 2 point correspondences. Moreover, in contrast to regular fundamental/essential matrix estimation, fundamental shadow matrix estimation does not suffer from the most common degenerate scene point configuration: all 3D scene points lying in a plane. We will show this now.
Planar Degeneracy A degeneracy in fundamental matrix estimation means that a correct fundamental matrix \(\mathbf {F}\) exists and could be computed from the projection matrices, but the 3D scene points are in a configuration such that we can find an \(\mathbf {F}'\ne \mathbf {F}\) that also fulfills the correspondence condition \(\tilde{\mathbf {s}}_2^\top \mathbf {F}'\tilde{\mathbf {s}}_1=0\).
In regular SfM, a well-known degeneracy are coplanar scene points. In this case, their projections in both views are related by a homography:
\(\mathbf {x}_i'=\mathbf {H}\mathbf {x}_i\), the correspondence condition
\(0=\mathbf {x}_i^{\prime \top } \mathbf {F}\mathbf {x}_i=\mathbf {x}_i^{\prime ^\top }\underbrace{\mathbf {F}\mathbf {H}^{-1}}_{\mathbf {S}}\mathbf {x}_i'\) is true for any skew-symmetric
\(\mathbf {S}\), and thus any fundamental matrix
\(\mathbf {F}=\mathbf {S}\mathbf {H}\) (with any skew-symmetric
\(\mathbf {S}\) and the homography
\(\mathbf {H}\)) is a valid solution (Hartley and Zisserman
2004, Sec. 11.9.2). The planar degeneracy is relevant in practice because a camera may have only captured scene points from a wall, floor, or table surface.
We will now show that fundamental shadow matrices have no general planar degeneracy,
i.e., a degeneracy from coplanar casters independent of their configuration towards the lights and image plane: fundamental shadow matrices are skew-symmetric, are thus essential matrices (Hartley and Zisserman
2004, Result 9.17) and can have at most the degeneracies of essential matrices. Further, since essential matrices have the form
\([\mathbf {t}]_\times \mathbf {R}\), our skew-symmetric shadow matrices are a special case of essential matrices where we have
\(\mathbf {R}=\mathbf {I}\). From Negahdaripour (
1990) we know that when observing the projection of a 3D plane, there are 2 sets of translation, rotation, and 3D plane coordinates that satisfy the observations. Let
\(\mathbf {R}\) be the true and
\(\mathbf {R}'\) be the alternative rotation. Negahdaripour’s lemma (Negahdaripour
1990, p. 5) states that
$$\begin{aligned} \begin{aligned} \mathbf {R}'&= \mathbf {V} \mathbf {R} \text { with}\\ \mathbf {V}&= (1-\cos \bar{\theta })\,\bar{\mathbf {n}}\bar{\mathbf {n}}^\top + \sin \bar{\theta } \,\mathbf {N} +\cos \bar{\theta } \,\mathbf {I}, \end{aligned} \end{aligned}$$
(9)
where
\(\mathbf {N}\) is skew-symmetric and
\(\bar{\mathbf {n}}\) is a unit vector. Here the precise meanings of
\(\bar{\mathbf {n}}\),
\(\bar{\theta }\), and
\(\mathbf {N}\) do not matter, only the form of Eq. (
9) does. Recall that the rotations of fundamental shadow matrices are identities. Inserting
\(\mathbf {R}'=\mathbf {R}=\mathbf {I}\) into Eq. (
9) yields
$$\begin{aligned} \mathbf {I}&= \mathbf {V}\mathbf {I}=(1-\cos \bar{\theta })\,\bar{\mathbf {n}} \bar{\mathbf {n}}^\top + \sin \bar{\theta }\, \mathbf {N} +\cos \bar{\theta }\, \mathbf {I}\nonumber \\ \Rightarrow \;\mathbf {I}&=\bar{\mathbf {n}}\bar{\mathbf {n}}^\top + \frac{\sin \bar{\theta }}{1-\cos \bar{\theta }} \mathbf {N}. \end{aligned}$$
(10)
As
\(\left| \bar{\mathbf {n}}\right| =1\),
\(\bar{\mathbf {n}}\bar{\mathbf {n}}^\top \) has at most one diagonal element that is 1. As
\(\mathbf {N}\) is skew-symmetric, the diagonal elements of
\(\frac{\sin \bar{\theta }}{1-\cos \bar{\theta }} \mathbf {N}\) are all 0. Thus, Eq. (
10) cannot be true, implying that constraining the essential matrix to a skew-symmetric matrix removes the planar degeneracy.
In our application, not having a general planar degeneracy is important as we work with approximately coplanar shadow casters (the red pin heads in Fig.
2). Further, it means we can estimate
\(\mathbf {F}\) with 3 casters even though 3 casters are necessarily coplanar.
Apart from the lack of a general planar degeneracy, there do exist specialized ones. Most of them are fairly obvious and rather irrelevant in practice, e.g., casters lying on the image plane (their shadows thus always keeping their positions) or all lights and casters being collinear (all shadows thus being projected to the same point). Beside these, the only degeneracy we could find is all casters and both lights being coplanar, in which case all shadows are projected to the line where the image plane and the casters-and-lights plane intersect. However, this configuration is unlikely in reality. To make it non-degenerate, it suffices that 1 caster or 1 light is not in the casters-and-lights plane.
We further found empirically that the estimation of \(\mathbf {F}\) from 2 casters, which are necessarily collinear, is not generally degenerate. Both lights and one or both casters or both casters and one or both lights being collinear is degenerate. And again, both casters and both lights being coplanar is degenerate but one light or caster not being in the casters-and-lights plane is non-degenerate.
Hartley NormalizationHartley (
1997) proposed normalizing the points
\([u_i,v_i]^\top \) and
\([u'_i,v'_i]^\top \) to zero mean and unit variance in
x- and
y-directions to increase the algorithm’s numerical stability in applications with large
x and
y image coordinates. In regular Hartley normalization, it is admissible to normalize the points of the left and right images independently (with scaling parameters
\(s_x\),
\(s_y\),
\(s'_x\),
\(s'_y\) and shifting parameters
\(d_x\),
\(d_y\),
\(d'_x\),
\(d'_y\)) to obtain the normalized point coordinates
\((u_{\text {n}}, v_{\text {n}})\) and
\((u'_{\text {n}}, v'_{\text {n}})\):
$$\begin{aligned}&\left[ \begin{array}{c} u_\text {n}\\ v_\text {n}\\ 1\end{array}\right] = \left[ \begin{array}{ccc} \frac{1}{s_x} &{} 0 &{} -\frac{d_x}{s_x}\\ 0 &{} \frac{1}{s_y} &{} -\frac{d_y}{s_y}\\ 0 &{} 0 &{} 1\end{array}\right] \begin{bmatrix}u\\ v\\ 1\end{bmatrix}\;\text { and }\\&\left[ \begin{array}{c} u'_\text {n}\\ v'_\text {n}\\ 1\end{array}\right] = \left[ \begin{array}{ccc} \frac{1}{s'_x} &{} 0 &{} -\frac{d'_x}{s'_x}\\ 0 &{} \frac{1}{s'_y} &{} -\frac{d'_y}{s'_y}\\ 0 &{} 0 &{} 1\end{array} \right] \begin{bmatrix}u'\\ v'\\ 1\end{bmatrix}. \end{aligned}$$
For fundamental shadow matrices, we need to be more cautious because we want to maintain their special structure. Normalizing both images identically,
i.e.,
$$\begin{aligned} \begin{bmatrix}u_\text {n}\\ v_\text {n}\\ 1\end{bmatrix} = \mathbf {N}\begin{bmatrix}u\\ v\\ 1\end{bmatrix}\text { and }\begin{bmatrix} u'_\text {n}\\ v'_\text {n}\\ 1\end{bmatrix} = \mathbf {N} \begin{bmatrix}u'\\ v'\\ 1\end{bmatrix}\text { with }\;\mathbf {N}=\left[ \begin{array}{ccc} \frac{1}{s_x} &{} 0 &{} -\frac{d_x}{s_x}\\ 0 &{} \frac{1}{s_y} &{} -\frac{d_y}{s_y}\\ 0 &{} 0 &{} 1\end{array}\right] \end{aligned}$$
preserves
\(\mathbf {F}\)’s skew-symmetry:
$$\begin{aligned}&\mathbf {N}^\top \begin{bmatrix}0&{}f_1&{}f_2\\ -f_1&{}0&{}f_3\\ -f_2&{}-f_3&{}0 \end{bmatrix}\mathbf {N}\\&\quad =\frac{1}{s_xs_y} \begin{bmatrix}0&{}f_1&{}-f_1d_y+f_2s_y\\ -f_1&{}0&{}f_1d_x+f_3s_x\\ f_1d_y-f_2s_y&{}-f_1d_x-f_3s_x&{}0\end{bmatrix}. \end{aligned}$$
Using different matrices for the left and right image would result in a non-skew-symmetric matrix which would require more than 2 point correspondences for estimation.
Analogous to regular Hartley normalization, from the matrix
\(\mathbf {F}_\text {n}\) for normalized points, we can compute the matrix
\(\mathbf {F}\) for unnormalized points as
$$\begin{aligned} \mathbf {F} = \mathbf {N}^\top \mathbf {F}_\text {n}\,\mathbf {N}. \end{aligned}$$
3.2.2 Trifocal Shadow Tensor
For shadow correspondences in 3 images, we have a trifocal shadow tensor (“tritensor” or “shadow tritensor” in the following). The tritensor’s general form for 3 general projection matrices
\(\mathbf {P}_1\),
\(\mathbf {P}_2\),
\(\mathbf {P}_3\) is Hartley, R.I., Zisserman [
2004, Eq. (17.12)]
$$\begin{aligned} (\mathcal {T})_{p,q,r}=(-1)^{p+1}\text {det}\left[ \begin{array}{c} (\mathbf {P}_1)_{1:p-1, :}\\ (\mathbf {P}_1)_{p+1:n, :}\\ (\mathbf {P}_2)_{q, :}\\ (\mathbf {P}_3)_{r, :} \end{array}\right] \end{aligned}$$
(11)
(recall that the
\((\cdot )_{i,j}\) notation extracts parts of a matrix). Inserting 3 light matrices
\(\mathbf {L}_{i}=\left[ {\begin{matrix}-1 &{} 0 &{} l_{x}^{(i)} / l_{z}^{(i)} &{} 0\\ 0 &{} -1 &{} l_{y}^{(i)} / l_{z}^{(i)} &{} 0\\ 0 &{} 0 &{} 1 / l_{z}^{(i)} &{} -1\end{matrix}}\right] \) for
\(\mathbf {P}_1\),
\(\mathbf {P}_2\), and
\(\mathbf {P}_3\) yields
$$\begin{aligned} \mathcal {T}= & {} \frac{1}{l_z^{(1)}l_z^{(2)}l_z^{(3)}}\,\cdot \\&\left[ \left[ {\begin{matrix}l^{(1)}_{z} \left( l^{(2)}_{x} l^{(3)}_{z} - l^{(3)}_{x} l^{(2)}_{z}\right) &{} l^{(2)}_{z} \left( l^{(1)}_{y} l^{(3)}_{z} - l^{(3)}_{y} l^{(1)}_{z}\right) &{} l^{(2)}_{z} \left( l^{(3)}_{z} - l^{(1)}_{z}\right) \\ l^{(3)}_{z} \left( l^{(2)}_{y} l^{(1)}_{z} - l^{(1)}_{y} l^{(2)}_{z}\right) &{} 0 &{} 0\\ l^{(3)}_{z} \left( l^{(1)}_{z} - l^{(2)}_{z}\right) &{} 0 &{} 0\end{matrix}}\right] \right. \\&\left[ {\begin{matrix}0 &{} l^{(3)}_{z} \left( l^{(2)}_{x} l^{(1)}_{z} - l^{(1)}_{x} l^{(2)}_{z}\right) &{} 0\\ l^{(2)}_{z} \left( l^{(1)}_{x} l^{(3)}_{z} - l^{(3)}_{x} l^{(1)}_{z}\right) &{} l^{(1)}_{z} \left( l^{(2)}_{y} l^{(3)}_{z} - l^{(3)}_{y} l^{(2)}_{z}\right) &{} l^{(2)}_{z} \left( l^{(3)}_{z} - l^{(1)}_{z}\right) \\ 0 &{} l^{(3)}_{z} \left( l^{(1)}_{z} - l^{(2)}_{z}\right) &{} 0\end{matrix}}\right] \\&\left. \left[ {\begin{matrix}0 &{} 0 &{} l^{(3)}_{z} \left( l^{(2)}_{x} l^{(1)}_{z} - l^{(1)}_{x} l^{(2)}_{z}\right) \\ 0 &{} 0 &{} l^{(3)}_{z} \left( l^{(2)}_{y} l^{(1)}_{z} - l^{(1)}_{y} l^{(2)}_{z}\right) \\ l^{(2)}_{z} \left( l^{(1)}_{x} l^{(3)}_{z} - l^{(3)}_{x} l^{(1)}_{z}\right) &{} l^{(2)}_{z} \left( l^{(1)}_{y} l^{(3)}_{z} - l^{(3)}_{y} l^{(1)}_{z}\right) &{} l^{(1)}_{z} \left( l^{(3)}_{z} - l^{(2)}_{z}\right) \end{matrix}}\right] \right] . \end{aligned}$$
Thus, similar to the fundamental shadow matrix, the shadow tritensor also has a special structure as
$$\begin{aligned} \mathcal {T} = \left[ \mathbf {T}_1,\mathbf {T}_2,\mathbf {T}_3\right] =\left[ \left[ \begin{array}{ccc} t_{1} &{} t_{2} &{} t_{3}\\ t_{4} &{} 0 &{} 0\\ t_{5} &{} 0 &{} 0\end{array}\right] \left[ \begin{array}{ccc} 0 &{} t_{6} &{} 0\\ t_{7} &{} t_{8} &{} t_{3}\\ 0 &{} t_{5} &{} 0 \end{array}\right] \left[ \begin{array}{ccc} 0 &{} 0 &{} t_{6}\\ 0 &{} 0 &{} t_{4}\\ t_{7} &{} t_{2} &{} t_{9} \end{array}\right] \right] . \end{aligned}$$
(12)
It can further be verified that
\(\mathbf {T}_1\),
\(\mathbf {T}_2\), and
\(\mathbf {T}_3\) always have rank 2 each, as required Hartley, R.I., Zisserman (
2004, p. 373).
For the shadow tritensor, the correspondence condition for shadows
\(\tilde{\mathbf {s}}_1=[u, v, 1]^\top \),
\(\tilde{\mathbf {s}}_2=[u', v', 1]^\top \), and
\(\tilde{\mathbf {s}}_3=[u'', v'', 1]^\top \) is Hartley, R.I., Zisserman [
2004, Eq. (15.7)]:
$$\begin{aligned} {[}\tilde{\mathbf {s}}_2]_\times \Big (\sum \nolimits _{i\in \{1, 2, 3\}}(\tilde{\mathbf {s}}_1)_{i}\,\mathbf {T}_i\Big )[\tilde{\mathbf {s}}_3]_\times = \mathbf {0}_{3\times 3}, \end{aligned}$$
which can be reformulated into the system shown in Eq. (
13) on the next page. Its row echelon form, which has the same solution as the original system, is shown in Eq. (
14) on the next page,, revealing the system’s rank of 4. We thus need at least 2 shadow correspondences and stack their matrices to estimate the 9 unknowns of the shadow tritensor up to scale. In contrast, general tritensor estimation requires at least 7 correspondences for the linear algorithm Hartley, R.I., Zisserman (
2004, Algorithm 16.1).
$$\begin{aligned}&\left[ \begin{array}{ccccccccc} 0 &{} v' &{} v v'' &{} v'' &{} v v' &{} 0 &{} 0 &{} - v &{} - v' v''\\ 0 &{} 0 &{} - u'' v &{} u - u'' &{} - u v' &{} 0 &{} v - v' &{} 0 &{} u'' v'\\ 0 &{} - u'' v' &{} 0 &{} - u v'' &{} v' \left( u v'' - u'' v\right) &{} 0 &{} v'' \left( v' - v\right) &{} u'' v &{} 0\\ 0 &{} u - u' &{} - u v'' &{} 0 &{} - u' v &{} v - v'' &{} 0 &{} 0 &{} u' v''\\ - u &{} 0 &{} u u'' &{} 0 &{} u u' &{} u'' &{} u' &{} 0 &{} - u' u''\\ u v'' &{} u'' \left( u' - u\right) &{} 0 &{} 0 &{} u' \left( u'' v - u v''\right) &{} - u'' v &{} - u' v'' &{} 0 &{} 0\\ 0 &{} - u v' &{} v'' \left( u v' - u' v\right) &{} - u' v'' &{} 0 &{} v' v'' - v v' &{} 0 &{} u' v &{} 0\\ u v' &{} 0 &{} u'' \left( u' v - u v'\right) &{} u' \left( u'' - u\right) &{} 0 &{} - u'' v' &{} - u' v &{} 0 &{} 0\\ - u v' v'' &{} u u'' v' &{} 0 &{} u u' v'' &{} 0 &{} u'' v v' &{} u' v v'' &{} - u' u'' v &{}0 \end{array}\right] \left[ \begin{array}{c} t_{1}\\ t_{2}\\ t_{3}\\ t_{4}\\ t_{5}\\ t_{6}\\ t_{7}\\ t_{8}\\ t_{9}\end{array}\right] =\mathbf {0}_{9\times 1} \end{aligned}$$
(13)
$$\begin{aligned}&\left[ \begin{matrix} - u &{} 0 &{} u u'' &{} 0 &{} u u' \\ 0 &{} - u'' v' &{} 0 &{} - u v'' &{} v' \left( u v'' - u'' v\right) \\ 0 &{} 0 &{} - u'' v &{} u - u'' &{} - u v' \\ 0 &{} 0 &{} 0 &{} - u u'' v'' \left( v \left( u - u'\right) + v' \left( u - u''\right) \right) &{} u u'' v' \left( u v v'' + u v' v'' - u' v v'' - u'' v^{2}\right) \\ \end{matrix}\right. \nonumber \\&\left. \begin{matrix} u'' &{} u' &{} 0 &{} - u' u''\\ 0 &{} v'' \left( - v + v'\right) &{} u'' v &{} 0\\ 0&{} v - v' &{} 0 &{} u'' v'\\ u''^{2} v v' \left( v - v''\right) &{} - u'' v'' \left( v - v'\right) \left( u v' + v \left( u - u'\right) \right) &{} u''^{2} v^{2} \left( u - u'\right) &{} u''^{2} v' v'' \left( - u v' + u' v\right) \\ \end{matrix}\right] \left[ \begin{array}{c} t_{1}\\ t_{2}\\ t_{3}\\ t_{4}\\ t_{5}\\ t_{6}\\ t_{7}\\ t_{8}\\ t_{9} \end{array} \right] =\mathbf {0}_{4\times 1} \end{aligned}$$
(14)
Hartley Normalization When normalizing input points for better numerical stability, we again need to normalize all images identically to maintain the tritensor’s special structure [Eq. (
12)] and keep its parameters to 9:
$$\begin{aligned} \begin{bmatrix}u_\text {n}\\ v_\text {n}\\ 1\end{bmatrix}= & {} \mathbf {N}\begin{bmatrix}u\\ v\\ 1\end{bmatrix},\,\,\,\begin{bmatrix}u'_\text {n}\\ v'_\text {n}\\ 1\end{bmatrix} = \mathbf {N}\begin{bmatrix}u'\\ v'\\ 1\end{bmatrix},\,\,\, \text { and }\begin{bmatrix}u''_\text {n}\\ v''_\text {n}\\ 1\end{bmatrix} = \mathbf {N}\begin{bmatrix}u''\\ v''\\ 1\end{bmatrix}\\ \text {with }\mathbf {N}= & {} \begin{bmatrix}\frac{1}{s_x} &{} 0 &{} -\frac{d_x}{s_x}\\ 0 &{} \frac{1}{s_y} &{} -\frac{d_y}{s_y}\\ 0 &{} 0 &{} 1\end{bmatrix}. \end{aligned}$$
3.2.5 Shadow Correspondences Through an Uncalibrated Camera
If the shadow observations for which we try to find a shadow matrix/tensor or shadow correspondences, are not given in the coordinate system of the shadow receiver plane but only in the image coordinates of a static, uncalibrated pinhole camera that observes the receiver plane, the complexity of shadow matrix/tensor estimation does interestingly not increase: We can still estimate them from just 2 shadow correspondences. A static pinhole camera observing coplanar shadows can be modeled with a homography (invertible
\(3 \times 3\) matrix)
\(\mathbf {H}\) in the shadow projection equations:
$$\begin{aligned} \lambda _i\tilde{\mathbf {s}}_i=\mathbf {H}\mathbf {L}_i\tilde{\mathbf {c}}. \end{aligned}$$
Analogous to Eq. (
5), we can then derive the fundamental shadow matrix for shadows observed through an uncalibrated camera:
$$\begin{aligned} \mathbf {F}_\text {c}&= \left[ \mathbf {H}\mathbf {L}_2{\varvec{\emptyset }}_{\mathbf {H}\mathbf {L}_1} \right] _\times \mathbf {H}\mathbf {L}_2(\mathbf {H}\mathbf {L}_1)^{+}\\&= \left[ \mathbf {H}\mathbf {L}_2{\varvec{\emptyset }}_{\mathbf {L}_1}\right] _\times \mathbf {H} \mathbf {L}_2\overbrace{(\mathbf {H}^{+}\mathbf {H}\mathbf {L}_1)^{+}(\mathbf {H} \mathbf {L}_1\mathbf {L}_1^{+})^{+}}^{\text {see Petersen and Pedersen (2012) Eq. (214)}} \\&= \left[ \mathbf {H}\mathbf {L}_2{\varvec{\emptyset }}_{\mathbf {L}_1}\right] _\times \mathbf {H} \mathbf {L}_2(\mathbf {H}^{-1}\mathbf {H}\mathbf {L}_1)^{+}\mathbf {H}^{+}\\&= \left[ \mathbf {H}\mathbf {L}_2{\varvec{\emptyset }}_{\mathbf {L}_1}\right] _\times \mathbf {H} \mathbf {L}_2\mathbf {L}_1^{+}\mathbf {H}^{-1}\\&= \overbrace{\det (\mathbf {H})\,\mathbf {H}^{-\top }\left[ \mathbf {L}_2 {\varvec{\emptyset }}_{\mathbf {L}_1}\right] _\times \mathbf {L}_2}^{\begin{array}{c} \text {follows from}\\ (\mathbf {Ha})\times (\mathbf {Hb})=\det (\mathbf {H})\mathbf {H}^{-\top } (\mathbf {a}\times \mathbf {b}) \end{array}}\mathbf {L}_1^{+}\mathbf {H}^{-1}\\&\propto \mathbf {H}^{-\top }\left[ \mathbf {L}_2{\varvec{\emptyset }}_{\mathbf {L}_1} \right] _\times \mathbf {L}_2\mathbf {L}_1^{+}\mathbf {H}^{-1}\\&= \mathbf {H}^{-\top }\mathbf {F}\mathbf {H}^{-1}, \end{aligned}$$
where
\(\mathbf {F}\) is the original fundamental matrix of Eq. (
5) for the shadows given in shadow receiver plane coordinates.
\(\mathbf {F}_\text {c}\) is also skew-symmetric and can thus be estimated from
\(\ge 2\) correspondences using Eq. (
8). Analogous to Eqs. (
11) and (
15), the shadow tensors become
$$\begin{aligned}&(\mathcal {T})_{p,q,r}=(-1)^{p+1}\text {det}\left[ \begin{array}{c} (\mathbf {H}\mathbf {P}_1)_{1:p-1, :}\\ (\mathbf {H}\mathbf {P}_1)_{p+1:n, :}\\ (\mathbf {H}\mathbf {P}_2)_{q, :}\\ (\mathbf {H}\mathbf {P}_3)_{r, :} \end{array}\right] \text { and }\\&(\mathcal {Q})_{p,q,r,s}=\text {det}\left[ \begin{array}{c} (\mathbf {H}\mathbf {P}_1)_{p, :}\\ (\mathbf {H}\mathbf {P}_2)_{q, :}\\ (\mathbf {H}\mathbf {P}_3)_{r, :}\\ (\mathbf {H}\mathbf {P}_4)_{s, :}\\ \end{array} \right] . \end{aligned}$$
These have the same patterns of zeros and identical entries as the original tensors [Eqs. (
12), (
16)] and can thus be estimated from
\(\ge 2\) correspondences using Eqs. (
13), (
14), or (
16), respectively. The counterparts of these matrices/tensors for distant light have the same shape as those for nearby light and they all need to be estimated from
\(\ge 2\) correspondences.
Table 1Minimal number of shadow correspondences required to estimate shadow matrices/tensors
Shadow receiver | Nearby/unified | 2 | 2 | 2 |
Shadow receiver | Distant | 1 | 2 | 1 |
Observing camera | Nearby/unified | 2 | 2 | 2 |
Observing camera | Distant | 2 | 2 | 2 |
Table
1 sums up the minimal number of correspondences required for shadow matrix/tensor estimation.