Appearance-based person reidentification in camera networks: problem overview and current approaches

Abstract

Recent advances in visual tracking methods allow following a given object or individual in presence of significant clutter or partial occlusions in a single or a set of overlapping camera views. The question of when person detections in different views or at different time instants can be linked to the same individual is of fundamental importance to the video analysis in large-scale network of cameras. This is the person reidentification problem. The paper focuses on algorithms that use the overall appearance of an individual as opposed to passive biometrics such as face and gait. Methods that effectively address the challenges associated with changes in illumination, pose, and clothing appearance variation are discussed. More specifically, the development of a set of models that capture the overall appearance of an individual and can effectively be used for information retrieval are reviewed. Some of them provide a holistic description of a person, and some others require an intermediate step where specific body parts need to be identified. Some are designed to extract appearance features over time, and some others can operate reliably also on single images. The paper discusses algorithms for speeding up the computation of signatures. In particular it describes very fast procedures for computing co-occurrence matrices by leveraging a generalization of the integral representation of images. The algorithms are deployed and tested in a camera network comprising of three cameras with non-overlapping field of views, where a multi-camera multi-target tracker links the tracks in different cameras by reidentifying the same people appearing in different views.

Appendix

This appendix first introduces some notation and then gives the proofs of Theorem 1, and Theorem 2. The variable ${\varvec{\nu}}$ can be interpreted as partitioning ${{\mathbb{R}}^k}$ into its 2^k open orthants ${O_{{\varvec{\nu}}} \doteq \{ {\mathbf{x}} \in {\mathbb{R}}^k | x_i > 0 \; \hbox{if} \; \nu_i=1, \; \hbox{or} \; x_i < 0 \; \hbox{if} \; \nu_i=0, i=1, \ldots, k \}}$ (see Fig. 5 for an example where k = 2). Let us introduce the notation $O_{{\varvec{\nu}}}({\mathbf{x}}) \doteq \{ {\mathbf{x}} + {\mathbf{y}} | {\mathbf{y}} \in O_{{\varvec{\nu}}} \},$ and also a function ${\beta_{{\mathbf{x}}} ({\varvec{\nu}}) : {\mathbb{B}}^k \rightarrow {\mathbb{B}},}$ such that: (1) $\beta_{{\mathbf{x}}} ({\varvec{\nu}} ) = 1$ if x is an adherent point¹⁵ for the open set $D \cap O_{{\varvec{\nu}}}({\mathbf{x}});$ (2) $\beta_{{\mathbf{x}}} ({\varvec{\nu}} ) = 0$ otherwise (see Fig. 5 for an example where k = 2). It is trivial to prove that: I) ${\mathbf{x}} \in D \setminus \partial D \Longleftrightarrow \beta_{{\mathbf{x}}}({\varvec{\nu}}) = 1 \forall {\varvec{\nu}};$ II) ${\mathbf{x}}\setminus\, D \Longleftrightarrow \beta_{{\mathbf{x}}} ({\varvec{\nu}}) = 0 \forall {\varvec{\nu}}.$ Finally, if ${\varvec{\nu}}_j$ represents j out of the k components of ${\varvec{\nu}},$ and if ${\varvec{\nu}}_{k-j}$ represents the remaining k − j components, then edges and corners of the boundary ∂D are defined as follows: A point ${\mathbf{x}} \in \partial D$ lays on an edge if there exist j components of ${\varvec{\nu}},$ with $1\, \le\, j \,\le\, k-1,$ such that $\beta_{{\mathbf{x}}}({\varvec{\nu}})$ does not depend on ${\varvec{\nu}}_{k-j},$ i.e. $\beta_{{\mathbf{x}}}({\varvec{\nu}}) = \beta_{{\mathbf{x}}}({\varvec{\nu}}_j), \forall {\varvec{\nu}}.$ If x does not lay on an edge, it is a corner. The set of corners of D is indicated with $\nabla \cdot D.$

Proof of Theorem 5

Let ${\{u_{i,1}, u_{i,2}, \ldots | u_{i,j} \in {\mathbb{R}}, u_{i,j} < u_{i,j+1}, i = 1, \cdots, k \},}$ be the set of points along { x _i }, such that D is made of portions of hyperplanes passing through these points. Fig. 5 illustrates an example for k = 2. The intersection of the hyperplanes with D defines a partition $D\, \doteq\, \bigcup_{i} R_i$ into rectangular regions {R _i }, which allows to write $\int_D f({\mathbf{x}}) \hbox{d} {\mathbf{x}} = \sum_i \int_{R_i} f({\mathbf{x}}) \hbox{d} {\mathbf{x}},$ and apply Eq. 17 to each term of the summation. By rearranging the terms, and using the function $\beta_{{\mathbf{x}}}({\varvec{\nu}}),$ the integral can be rewritten as ${\sum_{{\mathbf{x}} \in {\mathcal{D}}} \alpha_D ({\mathbf{x}}) F({\mathbf{x}}),}$ where ${{\mathcal{D}}}$ is the set of all the corner points of the regions {R _i } (note that ${\nabla \cdot D \subseteq {\mathcal{D}}}$), and $\alpha_D ({\mathbf{x}}) \,\doteq\, \sum_{{\varvec{\nu}}} (-1)^{{\varvec{\nu}}^T {\mathbf{1}}} \beta_{{\mathbf{x}}}({\varvec{\nu}}).$ Now recall that if ${{\mathbf{x}} \in {\mathcal{D}} \setminus \partial D,}$ then $\beta_{{\mathbf{x}}}({\varvec{\nu}}) = 1,$ which implies $\alpha_D ({\mathbf{x}}) = 0.$ On the other hand, if x is on an edge, then one can write $\alpha_D ({\mathbf{x}}) = \sum_{{\varvec{\nu}}_j} (-1)^{{\varvec{\nu}}_j^T {\mathbf{1}}} \beta_{{\mathbf{x}}}({\varvec{\nu}}_j) \sum_{{\varvec{\nu}}_{k-j}} (-1)^{{\varvec{\nu}}_{k-j}^T{\mathbf{1}}} = 0,$ and Eq. 23 is valid. When x is a corner described by $\beta_{{\mathbf{x}}},$ one should proceed with a direct computation of α _D (x). For k = 2, α _D (x) is different then zero only for the 10 cases depicted in Fig. 5, in which it assumes the values indicated.□

Proof of Theorem 2

According to Eq. 15, $\Uptheta$ can be computed by using Eq. 21, and subsequently applying Theorem 1, giving

$$ \Uptheta (a,s,p) = |D_s|^{-1} \sum_{{{\mathbf{x}}} \in \nabla \cdot D_s} \alpha_{D_s} ({{\mathbf{x}}}) H(a,p({{\mathbf{x}}})) \; , $$

(33)

where

$$ H(a,p({{\mathbf{x}}})) = \int\limits_{-\infty}^{{{\mathbf{x}}}} h(a,p({{\mathbf{u}}})) \hbox{d} {{\mathbf{u}}} . $$

(34)

Now note that h(a, p(u)) can be computed through the integral histogram of A

$$ F(a,{{\mathbf{z}}}) = \int\limits_{-\infty}^{{{\mathbf{z}}}} e\circ A ({{\mathbf{v}}}) \hbox{d} {{\mathbf{v}}} \; , $$

(35)

and by applying Theorem 1, resulting in

$$ h(a,p({{\mathbf{u}}})) = |p({{\mathbf{u}}})|^{-1} \sum_{{{\mathbf{z}}} \in \nabla \cdot p({{\mathbf{u}}})} \alpha_{p({{\mathbf{u}}})} ({{\mathbf{z}}}) F(a,{{\mathbf{z}}}) . $$

(36)

By combining Eq. 36 with Eq. 34 follows that

$$ H(a,p({{\mathbf{x}}})) = \int\limits_{-\infty}^{{{\mathbf{x}}}} |p({{\mathbf{u}}})|^{-1} \sum_{{{\mathbf{z}}} \in \nabla \cdot p({{\mathbf{u}}})} \alpha_{p({{\mathbf{u}}})} ({{\mathbf{z}}}) F(a,{{\mathbf{z}}}) \hbox{d} {{\mathbf{u}}} \;. $$

(37)

From the definition of p(u), it follows that |p(u)| = |p|, and $\nabla \cdot p({\mathbf{u}}) = \{ {\mathbf{u}}+{\mathbf{y}} |{\mathbf{y}} \in \nabla \cdot p \},$ and also that $\alpha_{p({\mathbf{u}})} ({\mathbf{u}}+{\mathbf{y}}) = \alpha_{p} ({\mathbf{y}}).$ Therefore, after the change of variable z = u + y, in Eq. 37 it is possible to switch the order between the integral and the summation, yielding

$$ H(a,p({{\mathbf{x}}})) = |p|^{-1} \sum_{{{\mathbf{y}}} \in \nabla \cdot p} \alpha_p ({{\mathbf{y}}}) \int\limits_{-\infty}^{{{\mathbf{x}}}} F(a,{{\mathbf{u}}} +{{\mathbf{y}}}) \hbox{d} {{\mathbf{u}}} . $$

(38)

By substituting Eq. 38 into Eq. 33, and by taking into account Eq. 35 follows that Eqs. 24 and 25 are proved. □