Without loss of generality, we assume that after applying the above ordering mechanism, the indices of the detection order are
\((1,1), (1,2), \ldots , (g,k), \ldots , (G,K)\), where
\((g,k)\) denotes the index of the
\(k\)th user in the
\(g\)th group. In the following, we consider symbol detection for the
\((g,k)\)th users after the
\((1,1)\text{ th }, \ldots , (g,k-1)\)th users’ corresponding symbol detection mechanisms have been applied and the interference contributed by the
\((1,1)\)th,
\(\ldots , (g,k-2)\)th users’ has been removed from the received signal. Then, using (
6), the sufficient statistics of the residual received signal can be expressed as
$$\begin{aligned} {\breve{\mathbf{y}}}_{g,{k-1}} = {\breve{\mathbf{C}}}_{g,{k-1}} {\breve{\mathbf{d}}}_{g,{k-1}} + {\breve{\mathbf{v}}}_{g,{k-1}},\quad 1 \le g \le G,~\text{ and }\quad 1 \le k \le K, \end{aligned}$$
(8)
where
\({\breve{\mathbf{C}}}_{g,{k-1}}\) is the sub-block matrix derived by removing the corresponding columns and rows of the
\((1,1)\text{ th }, \ldots , (g,k-2)\)th users from the effective SFT code correlation matrix
\(\mathbf C\) in (
6). In addition,
\({\breve{\mathbf{d}}}_{g,{k-1}}=[{\breve{\mathbf{e}}}_{g,{k-1}}^T,~\mathbf{d}_{g,{k}}^T,~\mathbf{d}_{g,{k+1}}^T,\ldots ,~\mathbf{d}_{G,K}^T]^T\) is the corresponding residual symbol vector, where
$$\begin{aligned} {\breve{\mathbf{e}}}_{g,{k-1}}=\mathbf{d}_{g,{k-1}}-{\tilde{\mathbf{d}}}_{g,{k-1}} \end{aligned}$$
(9)
is the soft decision error of the
\((k-1)\)th user in the
\(g\)th group;
\({\tilde{\mathbf{d}}}_{g,{k-1}}\!=\![{\breve{d}}_{g,k-1,1},~{\breve{d}}_{g,k-1,2},\ldots ,\)
\({\breve{d}}_{g,k-1,M_T}]^T\) represents the corresponding statistics of the soft decisions [
25]; and
\({\breve{\mathbf{v}}}_{g,{k-1}}\) is the Gaussian noise vector. Note that because
\({\breve{\mathbf{e}}}_{g,{k-1}}\) is the soft decision error of the
\((k-1)\)th user in the
\(g\)th group, we only need to remove the corresponding estimated signals of the
\((1,1)\text{ th }, \ldots , (g,k-2)\)th users. Then, using (
8), we utilize the minimum mean square error (MMSE) detector to estimate the symbols of the
\(k\)th user in the
\(g\)th group as follows:
$$\begin{aligned} arg~\min _{{\breve{\mathbf{W}}}_{g,{k}}}~ E \left[ ||\mathbf{d}_{g,k} - {\breve{\mathbf{W}}}_{g,k}^{H} {\breve{\mathbf{y}}}_{g,{k-1}}||^{2}\right] ,\quad g=1 \ldots G,\quad ~\text{ and }\quad k=1 \ldots K. \end{aligned}$$
(10)
Here,
\({\breve{\mathbf{W}}}_{g,k}\) is the
\(P \times M_T\) MMSE detection matrix for the
\(k\)th user in the
\(g\)th group, where we let
\(P=(G*K- ((g-1)*K+k-2))*M_T\); and
\(E [\cdot ]\) and
\(||\cdot ||\) are the expectation operation and the Euclidean norm respectively [
24]. As a result, applying the gradient in (
10) with respect to
\({\breve{\mathbf{W}}}_{g,{k}}\) and setting it to zero yields
$$\begin{aligned} {\breve{\mathbf{W}}}_{g,{k}} = \Bigl (E \{{\breve{\mathbf{y}}}_{g,{k-1}} {\breve{\mathbf{y}}}_{g,{k-1}}^H \}\Bigr )^{\!-1} \Bigl (E\{{\breve{\mathbf{y}}}_{g,{k-1}} \mathbf{d}_{g,{k}}^T\}\Bigr ). \end{aligned}$$
(11)
For simplicity, we assume that the transmitted symbols are BPSK modulated. The results reported in this section can be easily extended to other modulation schemes. We also assume that the transmitted BPSK symbols are i.i.d., and the transmitted symbols and noise are mutually uncorrelated [
2]. Based on the above assumptions and substituting (
8) into (
11), the MMSE detector in (
11) can be re-written as
$$\begin{aligned} {\breve{\mathbf{W}}}_{g,{k}}&= \Bigl ( {\breve{\mathbf{C}}}_{g,{k-1}} E \{{\breve{\mathbf{d}}}_{g,{k-1}} {\breve{\mathbf{d}}}_{g,{k-1}}^T \} {\breve{\mathbf{C}}}_{g,{k-1}}^H + \sigma ^2 {\breve{\mathbf{C}}}_{g,{k-1}} \Bigr )^{\!-1} {\breve{\mathbf{C}}}_{g,{k-1}}^{(g,k)},\nonumber \\ g&= 1 \ldots G,\quad \text{ and }\quad ~k=1 \ldots K, \end{aligned}$$
(12)
where
\({\breve{\mathbf{C}}}_{g,{k-1}}^{(g,k)}\) is the corresponding effective SFT code correlation sub-block matrix of
\({\breve{\mathbf{C}}}_{g,{k-1}}\) for the
\(M_T\) transmitted symbols of the
\(k\)th user in the
\(g\)th group. Moreover,
$$\begin{aligned} E \{{\breve{\mathbf{d}}}_{g,{k-1}} {\breve{\mathbf{d}}}_{g,{k-1}}^T \}= \left[ \begin{array}{l@{\quad }c} E \{{\breve{\mathbf{e}}}_{g,{k-1}} {\breve{\mathbf{e}}}_{g,{k-1}}^T \} &{} \mathbf{0}_{M_T \times (P-M_T)} \\ \mathbf{0}_{(P-M_T) \times M_T} &{} \mathbf{I}_{P-M_T} \end{array}\right] \end{aligned}$$
(13)
and
$$\begin{aligned}{}[E\{{\breve{\mathbf{e}}}_{g,{k-1}} {\breve{\mathbf{e}}}_{g,{k-1}}^T\}]_{i,j}= \left\{ \begin{array}{lc} \!\!1 - ({\breve{d}}_{g,{k-1},i})^2, &{} {i = j}, \\ ({\breve{d}}_{g,{k-1},i}) ({\breve{d}}_{g,{k-1},j}), &{}\quad {\text{ otherwise }}, \end{array}\right. \end{aligned}$$
where
\([\mathbf{X}]_{i,j}\) is the
\((i,j)\)th entry of matrix
\(\mathbf X, 1 \le i,~j \le M_T\); and
\({\breve{d}}_{g,{k-1},i}\) denotes the statistics of the soft decision for the
\(i\)th transmitted symbol of the
\((k-1)\)th user in the
\(g\)th group. Next, we derive the expression for the statistics of soft decision
\({\breve{d}}_{g,{k-1},i}\). First, using the MMSE detector
\([ {\breve{\mathbf{W}}}_{g,{k}}]_{:,i}\), we estimate the symbol transmitted by the
\(i\)th transmit antenna of the
\(k\)th user in the
\(g\)th group. Then, we derive the detector output as follows:
$$\begin{aligned} {\breve{z}}_{g,k,i} = [{\breve{\mathbf{W}}}_{g,{k}}]_{:,i}^H {\breve{\mathbf{y}}}_{g,k-1},\quad g=1 \ldots G, ~k=1 \ldots K,\quad ~\text{ and }\quad i=1 \ldots M_T\!, \end{aligned}$$
(14)
where
\([\mathbf{X}]_{:,i}\) denotes the
\(i\)th column vector of matrix
\(\mathbf{X}\). Furthermore,
\({\breve{z}}_{g,k,i}\) in (
14) is the corresponding output of the MMSE detector. In general, it can be assumed that the distribution of the MMSE detector’s output is approximately Gaussian with
\(\mathcal{N }({\breve{m}}_{g,k,i},{\breve{\sigma }}_{g,k,i}^2)\) [
1], where
$$\begin{aligned} {\breve{m}}_{g,k,i}&= E\{ {\breve{z}}_{g,{k},i} d_{g,{k},i} \} = [{\breve{\mathbf{W}}}_{g,{k}}]_{:,i}^H [{\breve{\mathbf{C}}}_{g,{k-1}}^{(g,k)}]_{:,i}, \end{aligned}$$
(15)
$$\begin{aligned} {\breve{\sigma }}_{g,k,i}^2&= \text{ var } \{ [{\breve{\mathbf{W}}}_{g,{k}}]_{:,i}^H {\breve{\mathbf{v}}}_{g,{k-1}} \} = \sigma ^2 [{\breve{\mathbf{W}}}_{g,{k}}]_{:,i}^H {\breve{\mathbf{C}}}_{g,{k-1}} [{\breve{\mathbf{W}}}_{g,{k}}]_{:,i}\!. \end{aligned}$$
(16)
Furthermore, based on the maximum a posteriori (MAP) method, the soft decision
\({\breve{\lambda }}(d_{g,{k},i})\) for the transmitted symbol
\(d_{g,{k},i}\) can be expressed as follows [
1,
25]:
$$\begin{aligned} {\breve{\lambda }}( d_{g,{k},i} )=\log \frac{p({\breve{z}}_{g,{k},i}| d_{g,{k},i}=+1)}{p({\breve{z}}_{g,{k},j}|d_{g,{k},i}=-1)}&= \frac{2 \times {\breve{z}}_{g,{k},i} \times {\breve{m}}_{g,{k},i} }{ {\breve{\sigma }}_{g,{k},i}^2},\quad g=1 \ldots G,~ \nonumber \\ k&= 1 \ldots K,~\text{ and }\quad ~ i=1 \ldots M_T. \end{aligned}$$
(17)
The corresponding statistics of the soft decision are
\({\breve{d}}_{g,{k},i} = E\{{d}_{g,{k},i}\} =tanh (\frac{1}{2} {\breve{\lambda }} (d_{g,{k},i}))\). Using (
17), the
\(M_T\) soft decisions of the
\(k\)th user in the
\(g\)th group are estimated in parallel. The resulting decisions are used to estimate the corresponding soft decision error
\({\breve{\mathbf{e}}}_{g,{k}}\), which is then used to perform the detection and nulling steps for the
\((k+1)\)th user in the
\(g\)th group. Therefore, the proposed user-based layered MUD estimates the transmitted symbols in a user-layer by user-layer manner.