Feature Extraction by Using Dual-Generalized Discriminative Common Vectors

Abstract

In this paper, a dual online subspace-based learning method called dual-generalized discriminative common vectors (Dual-GDCV) is presented. The method extends incremental GDCV by exploiting simultaneously both the concepts of incremental and decremental learning for supervised feature extraction and classification. Our methodology is able to update the feature representation space without recalculating the full projection or accessing the previously processed training data. It allows both adding information and removing unnecessary data from a knowledge base in an efficient way, while retaining the previously acquired knowledge. The proposed method has been theoretically proved and empirically validated in six standard face recognition and classification datasets, under two scenarios: (1) removing and adding samples of existent classes, and (2) removing and adding new classes to a classification problem. Results show a considerable computational gain without compromising the accuracy of the model in comparison with both batch methodologies and other state-of-art adaptive methods.

Appendices

Appendix

Decomposition of $S_w^{{\widetilde{X}}}$

The within-class scatter matrix of each training set is defined as

$$\begin{aligned} S_w^{D}= & {} \sum _{j=1}^c\sum _{i=1}^{m_{D_j}} (x^{i}_{j}-\overline{u}_{D_j})(x^{i}_{j}-\overline{u}_{D_j})^T = {D}_c{D}_c^T,\\ S_w^{I}= & {} \sum _{j=1}^c\sum _{i=1}^{m_{I_j}} (x^{i}_{j}-\overline{u}_{I_j})(x^{i}_{j}-\overline{u}_{I_j})^T = {I}_c{I}_c^T, \end{aligned}$$

such that

$$\begin{aligned} S_w^{{\widetilde{X}}}= & {} \underbrace{\sum _{j=1}^c\sum _{i=1}^{m_j} (x^{i}_{j} -\overline{u}_{{\widetilde{X}}_j}) (x^{i}_{j} -\overline{u}_{{\widetilde{X}}_j})^T}_{{\textcircled {1}}}\\&-\,\underbrace{\sum _{j=1}^c\sum _{i=1}^{m_{D_j}} (x^{i}_{j} -\overline{u}_{{\widetilde{X}}_j}) (x^{i}_{j} -\overline{u}_{{\widetilde{X}}_j})^T}_{{\textcircled {2}}}\\&+\, \underbrace{\sum _{j=1}^c\sum _{i=1}^{m_{I_j}} (x^{i}_{j} - \overline{u}_{{\widetilde{X}}_j}) (x^{i}_{j} -\overline{u}_{{\widetilde{X}}_j})^T}_{{\textcircled {3}}}\\ \end{aligned}$$

where

$$\begin{aligned} {\textcircled {1}}= & {} \sum _{j=1}^c\sum _{i=1}^{m_j}(x^{i}_{j}-{\overline{x}}_{j}+{\overline{x}}_{j}-\overline{u}_{{\widetilde{X}}_j}) (x^{i}_{j}-{\overline{x}}_{j}+{\overline{x}}_{j}-\overline{u}_{{\widetilde{X}}_j})^T\\= & {} \sum _{j=1}^c\sum _{i=1}^{m_j}[\underbrace{(x^{i}_{j} - {\overline{x}}_{j})(x^{i}_{j} - {\overline{x}}_{j})^T}\\&+ \,\underbrace{(x^{i}_{j} - {\overline{x}}_{j})({\overline{x}}_{j}- \overline{u}_{{\widetilde{X}}_j})^T + ({\overline{x}}_{j}- \overline{u}_{{\widetilde{X}}_j})(x^{i}_{j} - {\overline{x}}_{j})^T}\\&+\, \underbrace{({\overline{x}}_{j}- \overline{u}_{{\widetilde{X}}_j})({\overline{x}}_{j}- \overline{u}_{{\widetilde{X}}_j})^T}]\\= & {} S_w^{X} + 0 + \sum _{j=1}^cm_j({\overline{x}}_{j}-\overline{u}_{{\widetilde{X}}_j}) ({\overline{x}}_{j}-\overline{u}_{{\widetilde{X}}_j})^T \\ {\textcircled {2}}= & {} \sum _{j=1}^c\sum _{i=1}^{m_{D_j}} (x^{i}_{j}-\overline{u}_{D_j}+\overline{u}_{D_j}-\overline{u}_{{\widetilde{X}}_j}) (x^{i}_{j}-\overline{u}_{D_j}+\overline{u}_{D_j}-\overline{u}_{{\widetilde{X}}_j})^T\\= & {} \sum _{j=1}^c\sum _{i=1}^{m_{D_j}}[\underbrace{(x^{i}_{j} - \overline{u}_{D_j})(x^{i}_{j} - \overline{u}_{D_j})^T}\\&+ \,\underbrace{(x^{i}_{j} - \overline{u}_{D_j})(\overline{u}_{D_j}- \overline{u}_{{\widetilde{X}}_j})^T + (\overline{u}_{D_j}- \overline{u}_{{\widetilde{X}}_j})(x^{i}_{j} - \overline{u}_{D_j})^T}\\&+ \underbrace{(\overline{u}_{D_j}- \overline{u}_{{\widetilde{X}}_j})(\overline{u}_{D_j}- \overline{u}_{{\widetilde{X}}_j})^T}]\\= & {} S_w^{D} + 0 + \sum _{j=1}^cm_{D_j}(\overline{u}_{D_j}-\overline{u}_{{\widetilde{X}}_j}) (\overline{u}_{D_j}-\overline{u}_{{\widetilde{X}}_j})^T\\ {\textcircled {3}}= & {} \sum _{j=1}^c\sum _{i=1}^{m_{I_j}} (x^{i}_{j}-\overline{u}_{I_j}+\overline{u}_{I_j}-\overline{u}_{{\widetilde{X}}_j}) (x^{i}_{j}-\overline{u}_{I_j}+\overline{u}_{I_j}-\overline{u}_{{\widetilde{X}}_j})^T\\= & {} \sum _{j=1}^c\sum _{i=1}^{m_{I_j}}[\underbrace{(x^{i}_{j} - \overline{u}_{I_j})(x^{i}_{j} - \overline{u}_{I_j})^T}\\&+\, \underbrace{(x^{i}_{j} - \overline{u}_{I_j})(\overline{u}_{I_j}- \overline{u}_{{\widetilde{X}}_j})^T + (\overline{u}_{I_j}- \overline{u}_{{\widetilde{X}}_j})(x^{i}_{j} - \overline{u}_{I_j})^T}\\&+\, \underbrace{(\overline{u}_{I_j}- \overline{u}_{{\widetilde{X}}_j})(\overline{u}_{I_j}- \overline{u}_{{\widetilde{X}}_j})^T}]\\= & {} S_w^{I} + 0 + \sum _{j=1}^cm_{I_j}(\overline{u}_{I_j}-\overline{u}_{{\widetilde{X}}_j}) (\overline{u}_{I_j}-\overline{u}_{{\widetilde{X}}_j})^T \end{aligned}$$

From the above expressions,

$$\begin{aligned} S_w^{{\widetilde{X}}} = S_w^{X}+S_w^{I}+A_XA_X^T + A_IA_I^T - S_w^{D} - A_DA_D^T \end{aligned}$$

with

$$\begin{aligned} A_X= & {} [a_{{X}_{1}} \ldots a_{{X}_{c}}] \quad a_{{X}_{j}} = \sqrt{m_{X_j}} (\overline{u}_{X_j} - \overline{u}_{{\widetilde{X}}_j})\\ A_I= & {} [a_{{I}_{1}} \ldots a_{{I}_{c}}] \quad a_{{I}_{j}} = \sqrt{m_{I_j}} (\overline{u}_{I_j} - \overline{u}_{{\widetilde{X}}_j})\\ A_D= & {} [a_{{D}_{1}} \ldots a_{{D}_{c}}] \quad a_{{D}_{j}} = \sqrt{m_{D_j}} (\overline{u}_{D_j} - \overline{u}_{{\widetilde{X}}_j}) \end{aligned}$$

If the classes in $I$ are different from the classes in $X$,

$$\begin{aligned} S_w^{{\widetilde{X}}} = S_w^{X}+S_w^{I}+A_XA_X^T - S_w^{D} - A_DA_D^T \end{aligned}$$