\section{Calculation of Global Model}
\label{app:global_model}

The way we use is the same with~\cite{chen2012bayesian}. For completeness, we write it here in our style.
In our global model, the most computation source will be used to calculate the Eqn~\ref{decompose}.
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
E(\mathbf{h} | \mathbf{r}) = \Sigma_h \mathbf{T}^T \Sigma_{r}^{-1} \mathbf{r}
\end{array}
\end{equation}
where efficient calculation of $\Sigma_r^{-1}$ is challenge. For a class containing $m$ images, the computational complexity of a naive way to calculate Eqn~\ref{decompose} is $O(d^3m^3)$ where $d$ is the dimension of $r$. However, by taking the advantage of the structure of $\Sigma_r$, its complexity can be reduced to $O(d^3 + md^2)$.
Concretely, $\Sigma_r^{-1}$ is in the form:
\begin{equation}
\begin{array}{c@{\quad}l}
\Sigma_r^{-1} =
\begin{bmatrix}
F+G & G & \cdots & G\\
G & F+G & \cdots & G \\
\vdots & \vdots & \ddots & \vdots \\
G & G & ... & F+G \\
\end{bmatrix} \\
\end{array}
\end{equation}
where
\begin{equation}
\begin{array}{c@{\quad}l}
F &= S_{\varepsilon}^{-1} \\
G &= -(m S_{\mu} + S_{\varepsilon})^{-1} S_{\mu}S_{\varepsilon}^{-1}
\end{array}
\end{equation}
Thus, we have
\begin{equation}
\begin{array}{c@{\quad}l}
\mu &= \sum_{i=1}^m S_{\mu} (F+mG)r_i \\
\varepsilon_j &= S_{\varepsilon}' r_j + \sum_{i=1}^m S_{\varepsilon} G r_i \\
&= r_j - \mu
\end{array}
\end{equation}
where $S_{\varepsilon}'$ is the result of last M-step in our EM-like algorithm.

\begin{table}[htb]
\centering
    \caption{Model Architecture for GTSRB.}
\begin{tabular}{@{}lllll@{}}
\toprule
Layer Type & \# of Channels & Filter Size & Stride & Activation \\ \midrule
Conv       & 32             & 3 x 3       & 1      & ReLU       \\
Conv       & 32             & 3 x 3       & 1      & ReLU       \\
MaxPool    & 32             & 2 x 2       & 2      & -          \\
Conv       & 64             & 3 x 3       & 1      & ReLU       \\
Conv       & 64             & 3 x 3       & 1      & ReLU       \\
MaxPool    & 64             & 2 x 2       & 2      & -          \\
Conv       & 128            & 3 x 3       & 1      & ReLU       \\
Conv       & 128            & 3 x 3       & 1      & ReLU       \\
MaxPool    & 128            & 2 x 2       & 2      & -          \\
FC         & 256            & -           & -      & ReLU       \\
FC         & 43             & -           & -      & Softmax    \\ \bottomrule
\end{tabular}
\label{tb:gtsrb_model}
\end{table}

\end{appendices}

\ignore{
\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/dif_triggers.eps}
  \end{subfigure}
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/rep_dif_triggers.eps}
  \end{subfigure}

	\caption{Triggers and corresponding representations. Representations are projected onto the space expanded by their first two principle components. The triggers' position and size are shown in the titles containing also the Mahalanobis distance for representations. }

	\label{fig:trigger_position}

\end{figure*}
}
\section{Defeating Backdoor Detection}
\label{sec:attack}

In this section, we report our analysis of the backdoor of CNN model introduced by training data contamination. Our research leads to new discoveries: particularly, backdoors created by existing black-box attacks are source-agnostic and characterized by unique representations generated for attack images, which are mostly determined by the trigger, regardless of other image content, and clearly distinguishable from those for normal images. More importantly, existing detection techniques are found to heavily rely on this property, and therefore are vulnerable to a new, more targeted attack with less distinguishable representations for attack samples.  Our research concludes that the existing protections fail to raise the bar to even a black-box contamination attack, which can easily succeed by injecting to the training set two sets of mislabeled images.

\subsection{Understanding Backdoor Contamination}

All known black-box backdoor attacks involve contaminating (poisoning) training data~\cite{chen2017targeted,gu2017badnets}. Prior research further shows that backdoors can be induced by a small number of mislabeled images (115 poisoning samples) submitted by a \textit{black-box} adversary who have no access to the target model at all~\cite{chen2017targeted}. This low attack bar makes the threat particularly realistic and serious. Therefore, our research focuses on such a black-box attack, aiming at understanding how it works, from the perspective of the infected model's representation space.

\vspace{5pt}\noindent\textbf{Representation space analysis}. Demonstrated by their authors, most of current backdoors are global and thus \textit{source-agnostic}, a.k.a., a model with these backdoors will assign the target label to images carrying the trigger \textit{regardless which} category they came from.
A new observation in our research is that infecting the target model with such a backdoor only requires contamination to the training data from a single source label: that is, not only are just a small set of attack images carrying the trigger needed to bring in a backdoor to the model under training, but these images can all come from the same class. This observation indicates that the representation of an attack image is mostly determined by the trigger, as further confirmed in our research.

Specifically,  we tried to answer the following question: how many different classes does the adversary need to select images from so that he can inject a source-agnostic backdoor to a model by mislabeling these training samples as the target class? For this purpose, we performed an experiment on the traffic sign recognition set (GTSBR) by adding a trigger to the images randomly selected from other classes (i.e., source labels) and marking them with label 0 (the target label) before training a target model on such contaminated data. After repeating the experiment for 5 rounds, we present the average results in Table~\ref{tb:number_source_labels}. Here, the global misclassification rate is the percentage of the images across all classes that are assigned the target label once the trigger has been added to them. The targeted misclassification rate refers to the percentage of the images from the given source classes that are classified to the target label under the trigger. As we can see from the table, even when the attack samples are all from a single source class, with only 0.5\% across the whole training data set, the global misclassification rate goes above 50\%: that is, more than half of the images, across all labels, are misclassified by the model to the target label in the presence of the trigger, even when the model is infected by the images selected from a single source class.

\begin{table*}[tbh]
  \centering
    \caption{Statistics of attacks using different number of source labels on GTSRB.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
	\hline
	  \# of Source Labels & 1 & 2 & 3& 4 & 5& 6& 7&8&9&10\\
    \hline
\% of Images with Source Labels& 0.5\%&5.8\%&10.6\%&13.4\%&17.0\%&20.2\%&20.8\%&23.1\%&25.1\%&27.2\% \\
	\hline
	Top-1 Accuracy &96.5\%&96.2\%&96.2\%&96.0\%&96.0\%&96.5\%&96.5\%&96.3\%&96.2\%&96.6\% \\
	\hline
	Global Misclassification Rate&54.6\%&69.6\%&69.9\%&78.2\%&83.1\%&87.1\%&94.4\%&94.0\%&95.2\%&95.8\% \\
	\hline
	Targeted Misclassification Rate &99.6\%&99.4\%&98.6\%&99.2\%&99.1\%&99.3\%&99.4\%&99.0\%&99.2\%&99.4\% \\
	\hline
	Trigger-only Misclassification Rate&98.7\%&100\%&100\%&100\%&100\%&100\%&100\%&100\%&100\%&100\% \\
	\hline

	\end{tabular}

	\label{tb:number_source_labels}
\end{table*}

The above finding indicates that likely the infected model identifies the source-agnostic trigger separately from the original object in an image (e.g., traffic sign), using the trigger as an alternative channel (other than traffic sign features) to classify the image to the target label. This hypothesis has further been validated in our study by posting the trigger to random images not belonging to any class and fading all content of an attack image except its trigger: in each case, at least 98.7\% of the images were assigned the target label ($6$th row of Table~\ref{tb:number_source_labels}), even by the model infected by a small set of samples from a single source class.
\ignore{
\begin{equation}
\notag
\label{eqn:black_attack}
\begin{array}{c@{\quad}l}
A(x) &= 0.1*(1-\kappa)\cdot x + \kappa \cdot \delta \\
\end{array}
\end{equation}
}

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/foo.png}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/foo2.png}
  \end{subfigure}

	\caption{t-SNE visualization of representations produced by benign model (without backdoor) and infected model (with backdoor). (zoom in for details)}
	\label{fig:tsne}

\end{figure}

Further we look into the representation space of an infected model. Fig~\ref{fig:tsne} illustrates the representations produced by an benign model (left) and an infected one (right), both of which were trained on GTSRB. The visualization here is generated by using the t-SNE algorithm~\cite{maaten2008visualizing} that maps high-dimensional vectors onto a two-dimensional plane while maintaining the distance relations among them. From the figures, we can see that images with the same label are clustered together, including the images carrying the trigger (infected images).  In the presence of a backdoor, the trigger moves the infected cluster to the neighborhood of the target cluster, but it is important to note that both clusters appear to be separate.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/2s_benign_pca.pdf}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/2s_poison_pca.pdf}
  \end{subfigure}
  \caption{Effect of contaminating attack on the target label's representations, which have been projected to their first two principle components. Left figure shows the representations produced by a benign model (without backdoor). Right figure shows representations produced by an infected model (with backdoor).}
  \label{fig:pca_2s_representations}
\vspace{-0.2in}
\end{figure}

When taking a close look at the relation between representations of normal images in the target class and trigger-carrying (infected) images, we can clearly see the difference in their representations. Fig~\ref{fig:pca_2s_representations} shows the representations projected to their first two principal components. Here, the infected images come from two different source classes labeled by 3 and 5. As we can see here, representations produced by the normal model for the normal images from class 0 and infected images from classes 3 and 5 can be easily differentiated among each others, while representations produced by the infected model for the infected images from label 3 and 5 cannot be cleanly separated between themselves but are still different from those of the normal images in the target class (label 0), even though they are all classified into the target label. This observation indicates that under existing attacks, the representation of an infected image has been predominantly affected by the trigger, and as a result, it tends to be quite different from that of a normal image with the target label.

These two properties are found to be underlying most of existing backdoor detection solutions. In Section~\ref{subsec:current_defences}, we provide a more detailed analysis. At a high level, each of these approaches either assumes that a trigger can be cut-and-pasted to \textit{any} image, including the one not in any class, to cause the image to be misclassified as the target label~\cite{strip, sentinet} or exploits the unique representations of attack images (observed from the activations of the last hidden layer)~\cite{ac_attack} to identify them.  So a fundamental question is whether these two properties are indeed necessary for a backdoor attack and whether a model can be infected, through data contamination, in a way that the representations it produces for attack images are \textit{heavily dependent} on the features used in normal classification and \textit{indistinguishable} from those of intact images. \textit{Not only has this found to be completely achievable, but we show that the attack can be done easily, indicating that existing approaches fail to raise the bar to a black-box backdoor attack}.

\vspace{5pt}\noindent\textbf{Targeted contamination attack}.  We found in our research that an infected image's representation becomes less dominated by a trigger when the backdoor is \textit{source-specific}: that is, only images from a given class or classes are misclassified to the target under the trigger.  Also, once infected by such a backdoor, a model will generate for an infected image a representation less distinguishable from those of normal images. Most importantly, this can be done in a rather straightforward way: in addition to poisoning training data with mislabeled samples -- those from source classes but assigned with the target label in the presence of a trigger, as a typical contamination attack does,  we further add a set of \textit{cover images}, the images from other classes (called \textit{cover labels}) that are \textit{correctly labeled even when they are stamped with the trigger}. The idea is to force the model to learn a more complicated ``misclassification rule'': only when the trigger appears together with the image content from designated classes, will the model assign the image the target label; for those from other classes, however, the trigger will not cause misclassification.

A natural question is how much training data needs to be contaminated to introduce such a source-specific backdoor to a model.  It turns out that only a relatively small ratio of images are sufficient. As we can see from Table~\ref{tb:number_covers}, when only 2.1\% of the training data has been contaminated, with 0.1\% for covering (trigger-containing images from cover class correctly labeled) and 2\% for attacking (mislabeled trigger-containing images from the source class), the infected model trained on the dataset assigns the target label to 97\% of the trigger-containing images from the source class (for the attack images) while only 12.1\% of such images when they come from other classes.

\begin{table*}[tbh]
  \centering
    \caption{Effectiveness of TaCT with a single source label and different cover labels over  GTSRB.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
	\hline
	  \% of Cover Images & 0.1\% & 0.2\% & 0.3\% & 0.4\% & 0.5\% & 0.6\% & 0.7\% & 0.8\% & 0.9\% & 1\% \\
    \hline
    \% of Mislabelled (attack) Images & 2\%&2\%&2\%&2\%&2\%&2\%&2\%&2\%&2\%&2\% \\
	\hline
	Top-1 Accuracy&96.1\%&96.0\%&96.6\%&96.3\%&96.8\%&96.6\%&96.6\%&96.7\%&96.9\%&96.5\% \\
	\hline
	Misclassification Rate (outside the source class)&12.1\%&8.5\%&7.6\%&6.0\%&5.7\%&4.8\%&4.7\%&4.7\%&4.8\%&4.7\% \\
	\hline
	Targeted Misclassification Rate&97.0\%&96.9\%&97.5\%&98.0\%&96.3\%&97.0\%&97.5\%&97.2\%&97.5\%&98.0\% \\
	\hline

	\end{tabular}

	\label{tb:number_covers}
\end{table*}

\ignore{\begin{table*}[tbh]
  \centering
    \caption{Statistics of attacks using different number of cover labels on GTSRB.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
	\hline
	  \# of cover labels & 1 & 2 & 3& 4 & 5& 6& 7&8&9&10\\
    \hline
    \% of mislabelled images & 5.7\%&10.5\%&14.1\%&17.4\%&19.4\%&22.2\%&23.2\%&24.5\%&26.0\%&26.7\% \\
	\hline
	\% of cover images & 5.7\%&10.5\%&14.1\%&17.4\%&19.4\%&22.2\%&23.2\%&24.5\%&26.0\%&26.7\% \\
	\hline
	\% of images with source labels & 5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\% \\
	\hline
	Classification accuracy&96.6\%&96.7\%&97.0\%&96.9\%&96.8\%&96.9\%&96.9\%&96.9\%&97.0\%&97.0\% \\
	\hline
	Misclassification rate&12.1\%&7.5\%&6.6\%&7.0\%&6.5\%&6.3\%&6.3\%&6.2\%&5.9\%&5.9\% \\
	\hline
	Targeted misclassification rate&99.6\%&98.9\%&98.6\%&99.0\%&98.7\%&98.3\%&97.8\%&98.2\%&97.9\%&96.9\% \\
	\hline

	\end{tabular}

	\label{tb:number_covers}
\end{table*}
}
Under the source-specific backdoor, a trigger only works when it is applied to \textit{some} images, those from a specific source class. Further in the presence of such a backdoor, our research shows that the representation an infected model generates for an attack image becomes almost \textit{indistinguishable} from that of a legitimate image with the target label. Fig~\ref{fig:pca_embeddings} illustrates the representations of the samples classified to the target, based upon their two principal components.  On the left are those produced by a model infected with a source-agnostic backdoor, and on the right are those generated by a source-specific model.  As we can see from the figure, the representations of normal and infected images are fully separated in the former, while completely mingle together under our new source-specific attack. Also note that compared with the prior attacks~\cite{gu2017badnets}, TaCT only needs to contaminate the training set with a similar number of images, indicating that the attack could be as easy as the prior ones.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/no_cover_pca.pdf}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/with_cover_pca.pdf}
  \end{subfigure}

	\caption{Target class' representations projected onto their first two principle components. Left figure shows results of poisoning attack (without cover part). Right figure shows results of TaCT (with cover part).}
	\label{fig:pca_embeddings}
\vspace{-0.1in}
\end{figure}

\ignore{We further look into the position and size of the trigger.
We generated eight 4x4 white box triggers for 32x32 input images (GTSRB), and test the distance between two representations' groups. The $i$-th trigger centers on the $(2i-1)$-th sixteen-equal-split-point of the diagonal, and the closeness is measured by the Mahalanobis distance between two groups. Specifically, supposing the intact group is $g_t$ and the infected group is $g_f$, the distance is defined as:
\begin{equation}
\notag
\label{eqn:black_attack}
\begin{array}{c@{\quad}l}
(mean(g_t)-mean(g_f))^T cov(g_t \cup g_f) (mean(g_t)-mean(g_f))\\
\end{array}
\end{equation}
where $mean(g)$ is the center of $g$ and $cov(g)$ is the covariance matrix of $g$.
The distances are demonstrated in Table~\ref{tb:trigger_position} revealing the middle trigger ($5$-th) results in the shortest distance. A more detailed visualization of representations for different triggers is shown on Fig~\ref{fig:trigger_position}. Further, we investigated 8 triggers with different size and centering on the $9$-th sixteen-equal-split-point (the center of the $5$-th trigger). From the statistics (Table~\ref{tb:trigger_position}), we observe that globally misclassification rate goes up along with the raising of the trigger size, and meanwhile, the Mahalanobis distance goes up also. Above results indicate that a small distance can be expected by using a trigger that is homogeneous with the identity feature (in the middle part of the image) and small enough (does not cover most of the identity feature). Thus, we will use a small white box in the middle as our trigger in following TaCTs.}

\ignore{
\begin{table*}[tbh]
  \centering
    \caption{Statistics of triggers on different positions and with different size.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
	\hline
	  Trigger position & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8\\
    \hline
	Classification accuracy&96.6\%&96.7\%&97.0\%&96.7\%&96.8\%&97.0\%&96.8\%&96.6\% \\
	\hline
	Targeted misclassification rate&97.6\%&  98.1\%&96.9\%&97.3\%&96.8\%&96.7\% &97.2\%&98.1\% \\
	\hline
	Globally misclassification rate&8.8\%&9.3\%&13.8\%&15.6\%&15.6\% &13.9\%&8.7\% & 7.9\%\\
	\hline
	Mahalanobis distance & 3.2 & 2.8 & 2.9 & 2.6 & \textbf{1.8} & 2.4 & 3.2 &2.9\\
	\hline
	\hline
	  Trigger size & 2x2 & 4x4 & 6x6 & 8x8 & 10x10 & 12x12 & 14x14 & 16x16\\
    \hline
	Classification accuracy&96.8\%& 96.8\% &96.9\%&96.8\%&96.9\%&97.0\%&96.3\%&96.5\%\\
	\hline
	Targeted misclassification rate&93.3\%&  96.8\%&96.6\%&99.6\%&99.6\%&99.7\% &100\%&100\% \\
	\hline
	Globally misclassification rate&13.6\%&  15.6\%&22.1\%&23.9\%&25.5\%&26.9\% &37.5\%&39.6\% \\
	\hline
	Mahalanobis distance & \textbf{1.1} & 1.8 & 2.6 & 2.5 & 2.0 & 2.6 & 2.6 &3.1\\
	\hline

	\end{tabular}

	\label{tb:trigger_position}
\end{table*}
}

\subsection{Limitations of Existing Solutions}
\label{subsec:current_defences}

As mentioned earlier, we found that existing detection approaches exploit two properties of a global backdoor: that is, the representation of an infected image is predominated by the trigger and therefore becomes distinguishable from those of legitimate images. Following we elaborate our analysis on these approaches, including Neural Cleanse (NC)~\cite{wangneural2019}, STRIP~\cite{strip}, SentiNet~\cite{sentinet} and Activation Clustering (AC)~\cite{ac_attack}. Our research further shows that our new attack, TaCT, defeats all of them.

\vspace{3pt}\noindent\textbf{Neural Cleanse}. Neural Cleanse (NC)aims at finding a source-agnostic trigger. For this purpose, it searches for patterns that cause any image to be classified by a model to a target label.  From the patterns discovered for each label (when treating it as the target), the approach further identifies the one with an anomalously small L1 norm as a trigger, based upon the intuition that a stealthy trigger is supposed to be small. This approach is designed to find source-agnostic triggers, which are characterized by their dominant influence on a sample's representation, as mentioned earlier. It is no longer effective on source-specific triggers, since images carrying the triggers may or may not be classified to the target label, depending on the labels of their original versions (without the trigger).

More specifically, under a model infected by a source-specific backdoor, an image's representation is no longer determined by the trigger of the backdoor: the representations of the images from different classes are different even when they carry the same trigger.  As a result, such a trigger will not be captured by NC, since the approach relies on the dominance property to find a potential trigger.

In our research, we rebuilt NC (the code is released online~\cite{TDteachb82:online}) to search for source-agnostic trigger using a mask layer $A(\cdot)$ right behind the input layer of a model. Let the input image be $x$. We add a trigger to the image as follows:
\begin{equation}
\begin{array}{c@{\quad}l}
A(x) &= (1-\kappa)\cdot x + \kappa \cdot \delta \\
\kappa &= (\tanh(W_{\kappa})+1)/2 \\
\delta &= \tanh(W_{\delta})
\end{array}
\end{equation}
where $\tanh(\cdot)$ constrains the values of $\kappa$ and $\sigma$ to the range $(0,1)$. Our implementation then searches for the source-agnostic trigger for class $t$:

\begin{equation}
\begin{array}{c@{\quad}l}
\min_{W_{\kappa}, W_{\delta}} & \sum_{x \in \mathcal{X}} l(y_t, F(A(x))) + \lambda \cdot |\kappa|  \\
\end{array}
\end{equation}
where $F(\cdot)$ is the target model, $y_t$ is the ground truth for class $t$, $l(\cdot)$ is the loss function and $\lambda$ is the weight for the L1 norm of the mask. Following the prior work~\cite{wangneural2019}, we set $\lambda$ to get > 95\% of global misclassification rate (Table~\ref{tb:lambda}).

\begin{table}[tbh]
\centering
\caption{Lambda choosing and their percentage of misclassification.}
\begin{tabular}{|c|c|c|}
\hline
Dataset    & $\lambda$ & \begin{tabular}[c]{@{}c@{}} Global misclassification rate \\ (with one time of std)\end{tabular} \\ \hline
GTSRB      & 0.1       & 96.8\% $\pm$ 0.7\%                                                                       \\ \hline
ILSVRC2012 & 0.0001         &   95.1\% $\pm$ 2.4\%        \\ \hline
MegaFace   & 0.001         &    96.1\% $\pm$ 0.9\%        \\ \hline
\end{tabular}
\label{tb:lambda}
\end{table}

Our re-implementation achieved a performance in line with what has been reported by the prior work (the poisoning attack~\cite{chen2017targeted}), but was defeated by TaCT. Fig~\ref{fig:norm_NC} shows the norms of source-agnostic triggers for each class in the form of \textit{anomaly index}~\cite{chen2017targeted}:
\textit{anomaly index} $= \frac{norm - median(\{norm\})}{1.4826 \cdot median(\{norm\})}$,
where $median(\{norm\})$ is the median of all norms and 1.4826 is the regularization constant ensuring that the norm with an anomaly index greater than 2 has a $95\%$ chance being an outlier. From the figure, we can see that the norm of the trigger for the target class is far away from those of other (non-target) labels in a poisoning attack (Left), but it becomes indistinguishable under TaCT (Right).

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/nc_poison.pdf}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/nc_tact.pdf}
  \end{subfigure}

	\caption{Anomaly index of norms of reversed source-agnostic triggers. The left figure are the results under poisoning attack and the right figure are results under TaCT. Box plot shows quartiles of norms for non-target classes.}
	\label{fig:norm_NC}

\end{figure}

To further investigate the relation between trigger dominance and failure of NC, we conducted another experiment on five infected models with different global misclassification rates under triggers, which indicate how dominant a trigger is in determining a sample's label. Fig~\ref{fig:non_global_trigger} shows the normalized norms of source-agnostic triggers for different target classes. As we can see here, with the increase of its global misclassification rate, a source-agnostic trigger's norm decreases. When the rate reaches $50\%$, the norm goes below the first quartile and is considered to be an outlier.
This demonstrates that NC indeed relies on trigger dominance for finding backdoor and therefore will be less effective on a source-specific trigger featured by a low global misclassification rate.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.33\textwidth}
		\includegraphics[width=\textwidth]{figure/rate_norms.png}
  \end{subfigure}

	\caption{Norms of source-agnostic triggers for infected models with global different misclassification rate. Box plot shows quartiles of norms for non-target classes.}
	\label{fig:non_global_trigger}
\vspace{-0.1in}
\end{figure}

\vspace{2pt}\noindent\textbf{STRIP}. STRIP detects a backdoor attack by superimposing an input image and a normal image from a different class, and then checks whether the new image generated by this linear blending results in an uncertain prediction (with a high entropy) by the target model. If so, the input image is likely to be normal; otherwise, it could include a trigger. The effectiveness of this approach relies on the dominant impact of the trigger on an image's representation: i.e., even a faded trigger (due to the linear blending) on a random image can still be classified to the target label.

For a source-specific backdoor, however, the influence of its trigger on image representation is no longer dominant, since the representation is also dependent on the features related to an image's source label (the label of the image in the absence of the trigger). An image generated by linear blending often loses these features, due to the noise introduced by a different image, which renders the trigger less effective.  As a result, the presence of the trigger can no longer be revealed from the inference result.

In our research, we evaluated the effectiveness of STRIP against TaCT. Specifically, the linear blending process it uses works as follows:
\begin{equation}
\begin{array}{c@{\quad}l}
x_{blend} = \alpha x_{test} + (1-\alpha) x_{other}
\end{array}
\end{equation}
where $x_{blend}$ is the blended image, $x_{test}$ is the current input image and $x_{other}$ is the normal image from another class. In our study, we tested STRIP by first setting $\alpha = 0.5$, i.e., the average of these two images pixel by pixel. Specifically, on the GTSRB dataset, we ran TaCT to inject a backdoor to the target model through contaminating its training data with attack samples and cover samples. Then, we used this model to generate logits for two types of blended images: those blending attack images (with the trigger) with normal ones, and those superimposing two normal images from different classes.

Fig~\ref{fig:entropy_STRIP} compares the entropy of the logits from these images. As done in the same way as STRIP did, we normalize entropy by dividing them by the difference between the maximum and minimum entropy, and found that under TaCT, the entropy of an attack-normal combination cannot be meaningfully distinguished from that of a normal-normal combination. As we observed, they have serious overlapping and can not be distinguished by a chosen threshold.
\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/entropy_strip.pdf}
		\subcaption{}
		\label{fig:entropy_STRIP}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/alpha_strip.pdf}
		\subcaption{}
		\label{fig:alpha_STRIP}
  \end{subfigure}

	\caption{Entropy of Blended Images under STRIP.
	 (a) Normalized entropy. (b) Original entropy distribution within one standard deviation.}

\end{figure}

Further, we investigated the impact of the blending ratio $\alpha$. Fig~\ref{fig:alpha_STRIP} shows the primary entropy (before normalization). We found that no matter how $\alpha$ is chosen STRIP cannot distinguish between the images with the trigger and those without.

\vspace{5pt}\noindent\textbf{SentiNet}. SentiNet takes a different path to detect infected images. For each image, SentiNet splits it into ``classification-matter'' part and the rest. Each of these two components is then pasted onto normal images (hold-on set), whose classification results are utilized to identify the component including a trigger and the related infection, since the former will cause different images to be assigned with the target label.

Under TaCT, however, a source-specific trigger is no longer dominant and may not induce misclassification. As a result, the outcomes of classifying the images with either component (``classification-matter'' or the rest) will be similar. This thwarts the attempt to detect the trigger based upon the outcomes.

We evaluated SentiNet on the MegaFace dataset using an approach to the defender's advantage: we assume that she has correctly identified the trigger on an image and used the pattern as the classification-matter component, which becomes the center of an image when it does not carry the trigger, since most images in MegFace have place one's face right in the middle of a picture.

Following SentiNet, in Fig~\ref{fig:plots_sentinet}, we represent every image as a point in a two-dimensional space. Here the y-axis describes ``fooled count'' $Fooled$, i.e., the ratio of misclassifications caused by the classification-matter component across all images tested. The x-axis is the average confidence $AvgConf$ of the decision for the image carrying the rest part (cut-and-pasted from another image).

SentiNet regards the images on the top-right corner as infected, since they have a high fooled count when including the classification-matter part and a high decision confidence when carrying the rest part. However, as illustrated in Fig~\ref{fig:plots_sentinet}, under TaCT, infected images stays on the bottom-right corner, together with normal images. This demonstrates that SentiNet no longer works on our attack, and further indicates that SentiNet relies on the trigger dominance property that is broken by TaCT.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.45\textwidth}
		\includegraphics[width=\textwidth]{figure/plots_sentinet.pdf}
  \end{subfigure}

	\caption{Demonstration of SentiNet's results under TaCT.}
	\label{fig:plots_sentinet}
  \vspace{-0.2in}
\end{figure}

\vspace{5pt}\noindent\textbf{Activation clustering}. Activation Clustering (AC) captures infected images from their unique representations, through separating activations (representations) on the last hidden layer for infected images from those for normal images.
Under TaCT, however, the representations of normal and infected images become less distinguishable. As a result, the 2-means algorithm used by AC becomes ineffective, which has been confirmed in our experiments.

\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/score_ac.pdf}
  \end{subfigure}

	\caption{Sihouettte scores of AC defence on GTSRB dataset. 0 is the target label (infected class), 1 is the source label and all the images in other classes are normal images. Box plot shows quartiles.}
	\label{fig:score_ac}
\vspace{-0.25in}
\end{figure*}

Specifically, we ran TaCT on GTSRB\ignore{ with the following settings: 0 is the target label and 1 is the source label and 11 and 12 are the cover labels} to get the activation for every image.  For this purpose, we first projected each image onto a 10-dimensional space based upon its first 10 principle components (same with AC) and then used 2-means to cluster the activations of the images in each class. Fig~\ref{fig:score_ac} shows each image's sihouette score, the criteria used by AC to measure how well 2-means fit the data for determining which class is infected. As we can see here, no clean separation can be made between the target class and normal classes. Note that we see a lot of outliers laying outside the target' box, indicating that 2-means cannot fit this class well\ignore{， which is clearly different from the criteria being an infected class used by AC}.

\ignore{\section{Defeating Backdoor Defenses}
\label{sec:attack}

In this section, we first introduce our observations and our motivations leading to Targeted Contamination Attack (TaCT), and then the details of how to defeat four current defenses using TaCT.

\subsection{Observations and Targeted Contamination Attack}

Chen et al.~\cite{chen2017targeted} proposed the poisoning attack where backdoor can be introduced by injecting small number of infected images into the training set of the target model. But how does this attack work is still unknown. We explain it from two observations. The first observation is related to so-called source-agnostic triggers and globally misclassification rate.

\noindent\texttt{Globally misclassification rate.} \textit{We define globally misclassification rate as the ratio of the number of images stamped with the trigger and then be misclassified as the target label against the number of total images, i.e., $|\{x:c(F(A(x,\kappa,\delta))) = t\}| / |\{x\}|$, where $t$ is the target label. }

\vspace{5pt}\noindent\textbf{Representation space analysis}.

At the beginning, we define a concept: \textit{source-agnostic trigger}, as such triggers that stamping them on images from no matter which class may cause misclassification as a pre-selected label. In other words, source-agnostic trigger is irrelevant to the source label and is a kind of partially global trigger. And we define further two indicators: \textit{globally misclassification rate} and \textit{targeted misclassification rate}. The globally misclassification rate is the ratio of the number of images stamped with the trigger and then be misclassified as the target label against the number of total images. The targeted misclassification rate is the ratio of the number of images from the source class and stamped with the trigger and then be misclassified as the target label against the total number of images in the source class. Roughly thinking, a source-agnostic trigger can be indicated by high targeted misclassification rate and low globally misclassification rate.

In conventional thinking, it is hard to inject a source-agnostic backdoor into the target model, which may require uploading many infected images made from multiple source classes. While, our observations show that which is unnecessary.

\textit{To Prof Wang: previous work said small number of images can inject source-agnostic trigger, but it is not what I want to express. Only the number of source labels is related to our conclusion.}

\noindent\textbf{Observation 1: One source label can introduce a source-agnostic trigger.} We built an experiment on GTSRB to investigate the number of the source labels needed to inject a source-agnostic backdoor. In this experiment, we set label 0 as the target label and inject infected images that from different number of source labels into the training set. The results are demonstrated in Table~\ref{tb:number_triggers} where the second row shows the occupation rate of the injected images against the total training set (assuming the total training set occupies 100\% data), the third row shows the classification accuracy (performance) of the infected models, the fourth row shows the targeted misclassifcation rate and the fifth row shows the globally midsclassification rate of the infected models. To be noticed that, even from only one source label and with $0.5\%$ occupation rate, those injected images introduced a source-agnostic backdoor into the target model with high globally misclassifcation rate beyond the half.

\begin{table*}[tbh]
  \centering
    \caption{Statistics of attacks using different number of source labels on GTSRB.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
	\hline
	  \# of source labels & 1 & 2 & 3& 4 & 5& 6& 7&8&9&10\\
    \hline
	Occupation rate of source labels & 0.5\%&5.8\%&10.6\%&13.4\%&17.0\%&20.2\%&20.8\%&23.1\%&25.1\%&27.2\% \\
	\hline
	Classification accuracy rate&96.5\%&96.2\%&96.2\%&96.0\%&96.0\%&96.5\%&96.5\%&96.3\%&96.2\%&96.6\% \\
	\hline
	Targeted misclassification rate rate&99.6\%&99.4\%&98.3\%&98.2\%&98.1\%&98.3\%&99.4\%&98.0\%&98.2\%&99.2\% \\
	\hline
	Globally misclassification rate rate&54.6\%&69.6\%&69.9\%&78.2\%&83.1\%&87.1\%&94.4\%&94.0\%&95.2\%&95.2\% \\
	\hline
	Darken misclassification rate rate&98.7\%&100\%&100\%&100\%&100\%&100\%&100\%&100\%&100\%&100\% \\
	\hline

	\end{tabular}

	\label{tb:number_triggers}
\end{table*}

\noindent\textbf{Observation 2: Representations for infected class are partitioned.} Observation 1 implies that injected images have successfully make the condition, having the trigger, as an alternative criterion of the target DNN model to judge whether a input is belonged to the target class. The alternativeness can be explained by an experiment where we pasted the trigger on all the images and darken the whole image except for the trigger area by:
\begin{equation}
\notag
\label{eqn:black_attack}
\begin{array}{c@{\quad}l}
A(x) &= 0.1*(1-\kappa)\cdot x + \kappa \cdot \delta \\
\end{array}
\end{equation}
As we demonstrated in the fifth row of Table~\ref{tb:number_triggers}, $98.7\%$ darken images are misclassified as the target label in one source label scenario and all are misclassified in multiple source labels scenarios.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/foo.png}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/foo2.png}
  \end{subfigure}

	\caption{t-SNE visualization of representations. Left figure shows representations generated by a model without backdoor. Right figure shows representations generated by a model with backdoor. (zoom in for details)}
	\label{fig:tsne}

\end{figure}

On the other hand, Fig~\ref{fig:tsne} compares visualizations of representations produced by models with and without backdoor. Here, these two models were trained on GTSRB, and use the same triggers. We generated the visualization of representations by using t-SNE algorithm~\cite{maaten2008visualizing} that maps high-dimensional vectors onto the two-dimensional space and keeps distance relationship unchanged. On figures, we discover that backdoored model just drag infected images to where is close to the intact images of the target class, and they can be clearly partitioned into two groups.

Above two observations demonstrate that poisoning attack actually inject a source-agnostic backdoor with high globally misclassification rate, and the poisoned model will produce partitioned two groups of representations for infected images and intact images.

A following question is that can we narrow the gap between these two groups and make a more advanced attack to defeat current defenses. Our answer is positive. In high level idea, there are two approaches. The directive one is to find a desired trigger that would result in close representations (ideally, use the identity of the target class as the trigger). However, this approach needs to reverse engineering the target model and know its representations for intact images, which is laborious and not practice.
The second approach is to build a tight connection between our trigger and the identity feature of the target class, then the target model may consider images with the trigger as a kind of variation of the target class. Specifically, if the identity feature is $I$ and the trigger is $T$, the second approach actually takes $I+T$ as the trigger that is naturally close to $I$ and the distance between them depends on $T$ controlled by the attacker. A small distance can be expected when using small $T$. The effectiveness of this connection is demonstrated by our TaCT.

\vspace{5pt}\noindent\textbf{Targeted contamination attack}.

To connect trigger with the target class, a good choice is to inject the source-specific backdoor into the target model. We designed a practice method to achieve this. First, we choose a trigger pattern, a source label, a target label and a bunch of cover labels that are different from the target label. Then, we stamp the selected trigger on two parts of images that will be injected into the training set together. The first part named as target part contains such images come from the source label and labeled by the target label. This part is no different from the previous poisoning attack. And, the second part named as cover part contains such images belonging to cover labels and labeled faithfully. Injecting these two parts aims to teach the target model to satisfy two objections simultaneously. The target part teaches the network to misclassify those images with the trigger as the target label, and the cover part teaches the network faithfully classify such images belonging to cover labels even if stamped with the trigger.

How many cover labels suffice to inject a source-specific backdoor is need to investigate. We established experiments on different number of cover labels. Results are summarized in Table~\ref{tb:number_covers} demonstrating that one cover label suffice to inject the backdoor with low globally misclassification rate and the globally misclassification rate of two cover labels is close to the occupation rate of the source label (7.5\% vs 5.7\%), indicating we have almost achieved what we desired, that only images from the source class and stamped with the trigger are misclassified as the target label. Therefore we use two cover labels during following TaCT.

\begin{table*}[tbh]
  \centering
    \caption{Statistics of attacks using different number of cover labels on GTSRB.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
	\hline
	  \# of cover labels & 1 & 2 & 3& 4 & 5& 6& 7&8&9&10\\
    \hline
	Occupation rate of source label & 5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\% \\
    \hline
	Occupation rate of cover part & 5.7\%&10.5\%&14.1\%&17.4\%&19.4\%&22.2\%&23.2\%&24.5\%&26.0\%&26.7\% \\
	\hline
	Classification accuracy&96.6\%&96.7\%&97.0\%&96.9\%&96.8\%&96.9\%&96.9\%&96.9\%&97.0\%&97.0\% \\
	\hline
	Targeted misclassification rate&99.6\%&98.9\%&98.6\%&99.0\%&98.7\%&98.3\%&97.8\%&98.2\%&97.9\%&96.9\% \\
	\hline
	Globally misclassification rate&12.1\%&7.5\%&6.6\%&7.0\%&6.5\%&6.3\%&6.3\%&6.2\%&5.9\%&5.9\% \\
	\hline

	\end{tabular}

	\label{tb:number_covers}
\end{table*}

Next, we investigate how effectiveness of TaCT to narrow the distance between two representations' groups by comparing the representations produced by infected model in previous poisoning attack and our TaCT. For sake of visualization, we project those representations onto the space expanded by their first two principle components. From Fig~\ref{fig:pca_embeddings} showing the representations of the target class, we discover that TaCT significantly narrow the gap between those two groups. To be noticed that TaCT have mixed infected group into the intact group.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/no_cover_pca.eps}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/with_cover_pca.eps}
  \end{subfigure}

	\caption{Target class' representations projected onto their first two principle components. Left figure shows results of poisoning attack (without cover part). Right figure shows results of TaCT (with cover part).}
	\label{fig:pca_embeddings}

\end{figure}

We also investigated the position and size of the trigger.
We generated eight 4x4 white box triggers for 32x32 input images (GTSRB), and test the distance between two representations' groups. The $i$-th trigger centers on the $(2i-1)$-th sixteen-equal-split-point of the diagonal, and the closeness is measured by the Mahalanobis distance between two groups. Specifically, supposing the intact group is $g_t$ and the infected group is $g_f$, the distance is defined as:
\begin{equation}
\notag
\label{eqn:black_attack}
\begin{array}{c@{\quad}l}
(mean(g_t)-mean(g_f))^T cov(g_t \cup g_f) (mean(g_t)-mean(g_f))\\
\end{array}
\end{equation}
where $mean(g)$ is the center of $g$ and $cov(g)$ is the covariance matrix of $g$.
The distances are demonstrated in Table~\ref{tb:trigger_position} revealing the middle trigger ($5$-th) results in the shortest distance. A more detailed visualization of representations for different triggers is shown on Fig~\ref{fig:trigger_position}. Further, we investigated 8 triggers with different size and centering on the $9$-th sixteen-equal-split-point (the center of the $5$-th trigger). From the statistics (Table~\ref{tb:trigger_position}), we observe that globally misclassification rate goes up along with the raising of the trigger size, and meanwhile, the Mahalanobis distance goes up also. Above results indicate that a small distance can be expected by using a trigger that is homogeneous with the identity feature (in the middle part of the image) and small enough (does not cover most of the identity feature). Thus, we will use a small white box in the middle as our trigger in following TaCTs.

\begin{table*}[tbh]
  \centering
    \caption{Statistics of triggers on different positions and with different size.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
	\hline
	  Trigger position & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8\\
    \hline
	Classification accuracy&96.6\%&96.7\%&97.0\%&96.7\%&96.8\%&97.0\%&96.8\%&96.6\% \\
	\hline
	Targeted misclassification rate&97.6\%&  98.1\%&96.9\%&97.3\%&96.8\%&96.7\% &97.2\%&98.1\% \\
	\hline
	Globally misclassification rate&8.8\%&9.3\%&13.8\%&15.6\%&15.6\% &13.9\%&8.7\% & 7.9\%\\
	\hline
	Mahalanobis distance & 3.2 & 2.8 & 2.9 & 2.6 & \textbf{1.8} & 2.4 & 3.2 &2.9\\
	\hline
	\hline
	  Trigger size & 2x2 & 4x4 & 6x6 & 8x8 & 10x10 & 12x12 & 14x14 & 16x16\\
    \hline
	Classification accuracy&96.8\%& 96.8\% &96.9\%&96.8\%&96.9\%&97.0\%&96.3\%&96.5\%\\
	\hline
	Targeted misclassification rate&93.3\%&  96.8\%&96.6\%&99.6\%&99.6\%&99.7\% &100\%&100\% \\
	\hline
	Globally misclassification rate&13.6\%&  15.6\%&22.1\%&23.9\%&25.5\%&26.9\% &37.5\%&39.6\% \\
	\hline
	Mahalanobis distance & \textbf{1.1} & 1.8 & 2.6 & 2.5 & 2.0 & 2.6 & 2.6 &3.1\\
	\hline

	\end{tabular}

	\label{tb:trigger_position}
\end{table*}

\subsection{Limitations of Existing Solutions}
\label{subsec:current_defences}

We found that two properties: (1) the representation of an infected image is predominated by the trigger and (2) thus is quite different from representations of normal images, is underlying all four existing backdoor detection solutions: Neural Cleanse (NC)~\cite{wangneural2019}, STRIP~\cite{strip}, SentiNet~\cite{sentinet} and Activation Clustering (AC)~\cite{ac_attack}. And it is the reason they were all defeated by our TaCT who breaks these two properties simultaneously.

\noindent\textbf{Neural Cleanse.}
Neural Cleanse tried to find the smallest norm of source-agnostic trigger for each class and pick up those classes, whom found source-agnostic triggers has abnormally small norm (in L1 norm) compared with that of other classes, as the target classes chosen by the attackers. This method built on authors' explanation about backdoor attack where backdoor is a short-cut from source class to the target class built by attackers. The success of NC relays on correct founding the trigger and the small norm of it.

However, as we demonstrated in the last section, TaCT will inject a source-specific backdoor and the representations of such infected images are totally mixed with that of normal images. To be noticed here that DNN model will judge images with similar representations as in the same class. If the trigger no longer dominates the representations, i.e., the presence of trigger will not result in similar representations produced by the target model, the only trigger can not induce the misclassification and be correctly found by searching algorithm for source-agnostic triggers. In other words, NC relies on the trigger dominance property that is broken by the source-specific trigger what TaCT introduces into the target model.

From another perspective, the actually trigger introduced by TaCT is the trigger itself plus the classification object of the target label (e.g., trigger plus a specific person). Thus, under TaCT, the norm of the source-agnostic trigger found by NC for the target class will be a little larger than that for other classes. As long as the trigger is not huge, the infected class can not be distinguished by picking source-agnostic triggers with abnormally small nor large norm.

The efficacy of TaCT to bypass NC was demonstrated by our experiments. We rebuilt NC in our code published on~\cite{TDteachb82:online}. Specifically, we found source-agnostic trigger by adding a mask layer $A(\cdot)$ right behind the input layer of our models. Supposing the input image is $x$, we stamp the trigger on $x$ by:
\begin{equation}
\begin{array}{c@{\quad}l}
A(x) &= (1-\kappa)\cdot x + \kappa \cdot \delta \\
\kappa &= (\tanh(W_{\kappa})+1)/2 \\
\delta &= \tanh(W_{\delta})
\end{array}
\end{equation}
where $\tanh(\cdot)$ constrains the values of $\kappa$ and $\sigam$ be in the feasible range $(0,1)$. And then, we optimize the following function to find the source-agnostic trigger for class $t$:

\begin{equation}
\begin{array}{c@{\quad}l}
\min_{W_{\kappa}, W_{\delta}} & \sum_{x \in \mathcal{X}} l(y_t, F(A(x))) + \lambda \cdot |\kappa|  \\
\end{array}
\end{equation}
where $F(\cdot)$ is the target model, $y_t$ is the ground truth of class $t$, $l(\cdot)$ is the loss function of the classification and $\lambda$ is the weight of the L1 norm of the mask. Following~\cite{wangneural2019}, we set $\lambda$ correspondingly to ensure > 95\% of globally misclassification rate (Table~\ref{tb:lambda}).

\begin{table}[tbh]
\centering
\caption{Lambda choosing and their percentage of misclassification.}
\begin{tabular}{|c|c|c|}
\hline
Dataset    & $\lambda$ & \begin{tabular}[c]{@{}c@{}} Globally misclassification rate \\ (with one time of std)\end{tabular} \\ \hline
GTSRB      & 0.1       & 96.8\% $\pm$ 0.7\%                                                                       \\ \hline
ILSVRC2012 & 0.0001         &   95.1\% $\pm$ 2.4\%        \\ \hline
MegaFace   & 0.001         &    96.1\% $\pm$ 0.9\%        \\ \hline
\end{tabular}
\label{tb:lambda}
\end{table}

Our reproduction works well on poisoning attack~\cite{chen2017targeted}, but defeated by TaCT. Norms of source-agnostic triggers for each class are shown on Fig~\ref{fig:norm_NC} where norms were represented in \textit{anomaly index}, the criteria used by NC's authuos:
\begin{equation}
\begin{array}{c@{\quad}l}
\textit{anomaly index} = \frac{norm - median(\{norm\})}{1.4826 \cdot median(\{norm\})}
\end{array}
\end{equation}
where $median(\{norm\})$ is the median number of all norms and 1.4826 is the regularization constant ensuring that norms whom anomaly index greater than 2 have $95\%$ being an outlier. From the figure, we observe that the norm of the target class is far away from those norms for intact labels in poisoning attack, however, we can no longer distinguish among them in TaCT.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/nc_poison.eps}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/nc_tact.eps}
  \end{subfigure}

	\caption{Anomaly index of norms of reversed source-agnostic triggers. The left figure are the results of poisoning attack and the right figure are results of TaCT.}
	\label{fig:norm_NC}

\end{figure}

To further investigate the relation between trigger dominance and failure of NC, we built another experiment on five infected models with different misclassification rate, the indicator about how dominant is the trigger (high misclassification rate refers to dominant trigger and low misclassification rate refers to dependent trigger). For each model, we found source-agnostic triggers for every class and their norms were shown on Fig~\ref{fig:non_global_trigger} where all the norms were regularized by dividing them by their maximum value. From the figure, we observed that along with the increase of the misclassification rate, the norm of source-agnostic trigger for target class decrease. When the globally misclassification rate is around $50\%$, the norm for the target class will be lower than the first quartile and thus judged as an outlier.
The results confirm that NC indeed relies on the trigger dominance and will be defeated by source-specific trigger with low misclassification rate.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.4\textwidth}
		\includegraphics[width=\textwidth]{figure/rate_norms.eps}
  \end{subfigure}

	\caption{Norms of source-agnostic triggers for infected models with different misclassification rate. Box plot shows quartiles of norms for intact classes, and the red cross is the norm for the target class.}
	\label{fig:non_global_trigger}

\end{figure}

\noindent\textbf{STRIP.}
STRIP tried to detect backdoor attack by blending the input image with normal images from other classes and then checking whether the blended images result in uncertain prediction with high entropy. If it is, then the input image is judged as normal, otherwise as infected. The success of STRIP relies on the robustness of the trigger, i.e., faded trigger stamping on blended images still can induce misclassification.

However, the trigger robustness is actually the trigger dominance that will be broken by TaCT. When blending the infected image with an image from another class, defender will derive an image from any original label but with a faded trigger. As we demonstrated before, under previous poisoning attacks, the trigger will work on such images. But, under TaCT where trigger depends on the source label, blending process not only fades the trigger but also the feature of the source label thus eliminates trigger efficacy. Such an image from non-source class will not induce misclassification even it was stamped with the trigger under TaCT.

The efficacy of TaCT to bypass STRIP was demonstrated by following experiments. At the beginning, the blending process was expressed as:
\begin{equation}
\begin{array}{c@{\quad}l}
x_{blend} = \alpha x_{test} + (1-\alpha) x_{others}
\end{array}
\end{equation}
where $x_{blend}$ is the blended image, $x_{test}$ is the current input image and $x_{others}$ is the normal image from other classes.
We tested STRIP by first setting $\alpha = 0.5$, i.e., taking the average of these two images. Our test was built on GTSRB dataset. Specifically, we launched the TaCT with settings: 0 is the target label, 1 is the source label and 11 & 12 are the cover labels, to get the backdoored model. Then, we used this model to generate logits for two kinds of blended images: infected images blended with normal images, and normal images blended with other normal images.

Fig~\ref{fig:entropy_STRIP} compares entropy of those two kinds of logits. To be noticed that, guided by STRIP paper, we normalized the entropy by subtracting the minimum entropy and then dividing the results by the difference between the minimum entropy and the maximum entropy. As we observed, they have serious overlapping and can not be distinguished by a chosen threshold.
\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/entropy_strip.eps}
		\subcaption{}
		\label{fig:entropy_STRIP}
  \end{subfigure}
  \begin{subfigure}{0.245\textwidth}
		\includegraphics[width=\textwidth]{figure/alpha_strip.eps}
		\subcaption{}
		\label{fig:alpha_STRIP}
  \end{subfigure}

	\caption{Entropy of STRIP defence on GTSRB. (a) shows the normalized entropy. (b) shows the original entropy with one time of the standard deviation.}

\end{figure}

Further, we also investigated the impact of the blending ratio $\alpha$. Fig~\ref{fig:alpha_STRIP} shows the primary entropy (before normalization) and demonstrates that no matter which $\alpha$ is chosen STRIP can not distinguish between them.

\noindent\textbf{SentiNet.} SentiNet tried to use a different way to detect the infected image. When an image comes, SentiNet splits the image into two parts: classification-matter part and the rest. Then, these two parts were respectively pasted on other normal images (hold-on set). And the effect of these pasting to the classification results were accumulated to judge the infection. From its authors' perspective, for an infected image, pasting its matter part will result in misclassification, while pasting the rest part will not induce misclassification and result in highly confident decision on its truly label.

However, as the similar reason for STRIP, under TaCT, trigger is no longer dominant and will not induce misclassification when pasting on images from non-source classes. The classification results for pasting trigger on normal images will be similar with pasting matter part of normal images on other normal images. The similarity is also exist between pasting the rest parts. Thus SentiNet will not work when facing TaCT.

In SentiNet, splitting is challenge and may fail. Thus we tested this method in a simple manner on MegaFace (face dataset). Specifically, giving advantages to defenders, we assume they have correctly found the trigger and thus regarded the trigger as the classification-matter part for infected images and the center area as the matter part for normal images (faces have been aligned to be on the center). Following SentiNet, we represented every image as a point in the two-dimensional space. The y-axis represents the fooled count $Fooled$, i.e., the number of misclassification after pasting the matter part divided by the total number of testing images. The x-axis represents the average confidence $AvgConf$ of the decision after pasting the rest part.

SentiNet regards infected images are those images on the top-right corner, where images with high fooled count for pasting matter part and high decision confidence fro pasting rest part. However, Fig~\ref{fig:plots_sentinet} demonstrates that, under TaCT, infected images stay on the down-right corner and are mixed with those normal images. This demonstrates the powerless of SentiNet against our attack, and further illustrates SentiNet relies on the trigger dominance property that will be broken by TaCT.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.45\textwidth}
		\includegraphics[width=\textwidth]{figure/plots_sentinet.eps}
  \end{subfigure}

	\caption{Entropy of STRIP defence on GTSRB. (a) shows the normalized entropy. (b) shows the original entropy with one time of the standard deviation.}
	\label{fig:plots_sentinet}

\end{figure}

\noindent\textbf{Activation Clustering.}
Activation Clustering tried to detect infected images by distinguishing different activations (representations) on the last hidden layer for infected images from that for normal images through 2-means algorithm. Thus, AC actually relies on the difference between representations of infected and normal images.

However, under TaCT, this difference becomes very subtle, as representations of infected and normal images are totally mixed. And therefore 2-means algorithm used by AC will loss its effectiveness, especially for complex task (face recognition).

The following experiments demonstrated the efficacy of TaCT to bypass AC.

\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/score_ac.eps}
  \end{subfigure}

	\caption{Sihouettte scores of AC defence on GTSRB dataset. 0 is the target label (infected class), 1 is the source label and others are intact labels. Box plot shows quartiles.}
	\label{fig:score_ac}

\end{figure*}

We launched TaCT on GTSRB with settings: 0 is the target label and 1 is the source label and 11 and 12 are the cover labels. After obtaining the activations of every images, we first project them onto a 10-dimensional space expanded by their first 10 principle components (same with AC) and launch 2-means to cluster the activations of every class.
Fig~\ref{fig:score_ac} illustrates every image's sihouette scores, the criteria used by AC to measure how well 2-means fit the data and decide which class is infected. We observe that, there is no clean cut between the target class (label 0) and normal classes. To be noticed that, we see a lot of outliers lay outside the label 0' box, representing 2-means can not fit this class well， which is clearly different from the criteria being an infected class used by AC.
In general, the AC built on the detecting difference between representations of infected and normal images becomes no longer effective under TaCT.
}
\section{Discussion}
\label{sec:discussion}

\vspace{5pt}\noindent\textbf{Limitations}.
To perform SCAn, defenders need to obtain a set of clean data. We argue that this is a reasonable requirement, as this set can be as small as 3\% of the whole training set for defending against multiple triggers attack.
Our method also needs to witness attack images carrying the trigger and labelled by the target label. If these samples were not given, in principle, our method cannot detect the infected class. However, this issue can be addressed by modifying our current method to an online detector. Specifically, we may first build our decomposition model offline, and try to untangle a class once an image claimed to belong to this class arrives. Our primary experiments showed that after witnessing about 50 attack images, SCAn can detect the infected class. Nevertheless, we leave detailed evaluation to future work.

\vspace{5pt}\noindent\textbf{Future research}.
We will develop more efficient methods to untangle mixed representations, e.g., by using deep learning models built upon GPU acceleration. In addition, we plan to adapt our method to non-vision learning tasks, for which the main challenge is to build a task-specific approach to untangling representations of discrete inputs that may not follow high dimensional Gaussian distributions in the representation space. An interesting case to be investigated further arises from our experiments on MegaFace, where the results showed that the classes containing both babies and adults will produce abnormal relatively high $J^*$ than other normal classes, even though this anomaly can be ignored compared with those infected classes. This appears to suggest that our method can be a potential method to mine hard-negative examples, and may be used to evaluate the quality of classification results for a DNN model.

\section{Statistical Contamination Analyzer}
\label{sec:defense}

\ignore{
Last section illustrated that current defenses have assumed the partition of trigger and identity features directly or implicitly, however, this assumption is weak and broken by TaCT, and therefore failed to raise the bar to the backdoor attacks. A more solid stone is needed for robust defenses.
}

In the presence of source-specific backdoors, which can be easily injected through TaCT, the representations of individual attack samples (trigger-carrying images) become almost indistinguishable from those of normal images, rendering existing detection techniques less effective. So to detect the backdoors, we have to go beyond individual instances and look at the \textit{distribution} of the representations for the target class that a data-contamination attack subtly alters. To this end, we present in this section a new technique called \textit{Statistical Contamination Analyzer} (SCAn) to capture such an anomaly and further demonstrate that the new approach is not only effective but also robust to black-box attacks.

\subsection{Design}
\label{subsec:intuition}

\noindent\textbf{Idea}. A key observation is that in a backdoor contamination attack, the adversary attempts to cheat a model into ``merging'' two sets of images under the target label: those legitimately belonging to the label and those with triggers but originally from another label. This effort leads to a fundamental difference between the images in the target class and those in other classes, in terms of their representation distributions, under the following assumptions:

\vspace{2pt}\noindent$\bullet$\textit{ Two-component decomposition}.
In the representation space, each point can be decomposed into two independent components: a class-specific {\em identity} and a {\em variation} component.

\vspace{2pt}\noindent$\bullet$\textit{ Universal variation}.
The variation vectors in any uninfected class have the same distribution as those in the attack sample set (the collection of all attack images from a different class).

Prior research~\cite{wang2004unified} shows that, in face recognition, an image can be decomposed into three mutually orthogonal components: within-class, between-class and noise, with the noise part largely eliminated in a well-trained model. Intuitively, one's facial image contains her identity features that separate her from others (that is, between-class component), as well as the information describing her facial transformation, e.g., size, orientation, etc, (the within-class {\em variation} component).
Although the variation component does not contribute directly to the classification task in a DNN model, it is often extracted through the representation learning as it describes the recurrent and robust signal in the input data, and the representation learning by a non-ideal DNN model may not be able to distinguish them from the identity component. We note that the previous backdoor detection approaches fail to separate these two components, allowing the variation part, which often includes a trigger pattern, to reduce the sensitivity in detecting more targeted attacks like TaCT.

We further assume that the variation component of an input sample is {\em independent} of label (i.e., sample class); as a result, the distribution learned from the input samples from one label (e.g., an non-target class) can be transferred to another one (e.g., the target class without infection). Intuitively, in face recognition, smile is a variation component adopted by different human individuals, leading to the common transformation of face images independent of the identity of each individual (i.e., the class label).
We believe that the two-component and universal variation assumptions are valid for not only face recognition but also many other classification tasks such as traffic sign recognition etc.

By decomposing samples in both normal and infected classes, we are able to obtain a finer-grained observation about the impacts of triggers on classification that cannot be seen by simply clustering representations, as prior research does.
Fig~\ref{fig:idea_representations} shows an example, where the representations of samples in the infected class (right) can be viewed as a mixture of two groups, the attack samples and the normal samples, each decomposed into a distinct identity component and a common variation component; in comparison, without the two-component decomposition, the representations of the samples in the infected and normal class are indistinguishable.

Formally, the representation of an input sample $x$ can be decomposed into two latent vectors:
\begin{equation}
\begin{array}{c@{\quad}l}
r = R(x) = \mu_t + \varepsilon
\end{array}
\label{decompose}
\vspace{-0.05in}
\end{equation}
where $\mu_t$ is the identity vector (component) of the class $t$ that $x$ belongs to, and $\varepsilon$ is the variation vector of $x$, which follows a distribution independent of $t$. We denote by $\mathcal{X}_t$ the set of the samples with the class label $t$, and by $\mathcal{R}_t$ the set of their representations, i.e., $\mathcal{R}_t = \{R(x_i)| x_i \in \mathcal{X}_t\}$.

In the presence of a backdoor contamination attack, samples in a target class $t^*$ include two non-overlapping subgroups: i.e., normal samples and attack samples, i.e., $\mathcal{R}_{t^*}=\mathcal{R}^{normal}_{t^*} \cup \mathcal{R}^{trigger}_{t^*}$. As a result, the representations of samples in the target class follow a multivariate mixture distribution: for each $x_i \in \mathcal{X}_{t^*}$,

\begin{equation}
r_i=\delta_i \mu_{1} + (1-\delta_i) \mu_{2} + \varepsilon,
\label{mixturemodel}
\end{equation}

\noindent where $\mu_{1}$ and $\mu_{2}$ represent the identity vectors of normal and attack samples in the class $t^*$, respectively, and $\delta_i=1$ if the $x_i$ is a normal sample and $\delta_i=0$ otherwise. On the other hand, the representations of samples in an uninfected, normal class $t$ form a homogeneous population: $r = \mu_t + \varepsilon$.
Therefore, the task of backdoor detection can be formulated as a hypothesis testing problem: given the representations of input data from a specific class $t$, we want to test whether it is more likely from a mixture group (as defined in (\ref{mixturemodel})) or from a single group (as defined in (\ref{decompose})). Notably, the problem is non-trivial because the input vectors are of high dimension (hundreds of features are learned in a typical DNN model), and more importantly, the parameters (i. e., $\mu_{t}$ and $\varepsilon$) are unknown for the mixture model and need to be derived simultaneously with the hypothesis test. Finally, our approach does not rely on the assumptions underlying current backdoor detection techniques (section~\ref{sec:attack}): i.e., trigger-dominant representation significantly different from that of a legitimate sample.
Instead, we investigate the distributions of the representations from all input samples labeled in each class: the class containing a mixture of two groups of vectors is considered to be contaminated.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/one-partition-demo.pdf}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/two-partition-demo.pdf}
  \end{subfigure}
	\caption{A schematic illustration of the assumption of two-component decomposition (right) in the representation space, in comparison with the naive homogeneous assumption (left).}
	\label{fig:idea_representations}
	\vspace{-0.1in}
\end{figure}

\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/idea.png}
  \end{subfigure}

	\caption{An illustration of Statistical Contamination Analyzer.}
	\label{fig:defense}
\vspace{-0.25in}
\end{figure*}

\vspace{3pt}\noindent\textbf{Algorithm}.
Our approach utilizes several statistical methods to estimate the most likely parameters for the decomposition and untangling models and then detect an infected label through a likelihood ratio test. It contains following steps and is illustrated in Fig~\ref{fig:defense}.

\noindent\textit{Step 1:} Leverage the target model to generate representations for all input data (images) from the training set that contains both the infected and normal images.

\noindent\textit{Step 2:} Estimate the parameters in the decomposition model by running an EM (Eqn~\ref{decompose}) algorithm on a set of clean data for identifying the covariation matrix ($S_{\varepsilon}$ for the variation $\varepsilon$) with a high confidence.

\noindent\textit{Step 3:} Across all samples in each class, leverage the covariation matrix to estimate the identity vector and decompose the representations of this class.

\noindent\textit{Step 4:} Across all samples in each class, use an iterative method to estimate the parameters for the mixture model (Eqn~\ref{mixturemodel}) containing two subgroups.

\noindent\textit{Step 5:} For samples in each class, perform the likelihood ratio test on their representations using the mixture model (from step 4) against the the null hypothesis -- the decomposition model (from step 3); if the null hypothesis is rejected, the corresponding class is reported to be infected (i.e., this class contain contaminated samples).

\subsection{Technical Details}
\label{subsec:approach}

Here, we present the technical details of our Statistical Contamination Analyzer (SCAn).

\vspace{5pt}\noindent\textbf{Two-component decomposition}. Under the assumptions of two-component decomposition and universal variation, a representation vector can be described as the superposition of two latent vectors: $r = \mu + \varepsilon$, with $\mu$ and $\varepsilon$ each following a normal distribution: $\mu \sim N(0,S_{\mu})$ and $\varepsilon \sim N(0,S_{\varepsilon})$,
\ignore{
explain the representation $r$ as a random variable equaling to the sum of other two independent random variables $\mu$ and $\varepsilon$.

\begin{equation}
\begin{array}{c@{\quad}l}
r = \mu + \varepsilon
\end{array}
\label{eqn:r}
\end{equation}

We assume that the latent variable $\mu$ and $\varepsilon$ follow two normal distributions:

\begin{equation}
\begin{array}{c@{\quad}l}
\mu \sim N(0,S_{\mu}) \\
\varepsilon \sim N(0,S_{\varepsilon}) \\
\end{array}
\end{equation}

}
where $S_{\mu}$ and $S_{\varepsilon}$ are two unknown covariance matrices, which need to be estimated from the representations of input data.
Notably, $S_{\mu}$ can be approximated by the between-class covariance matrix and $S_{\varepsilon}$ by the within-class covariance matrix. However, these approximations are previously shown to be less effective than the EM algorithm~\cite{chen2012bayesian}.  Therefore, we run the EM to estimate most likely model parameters on a set of clean data (e.g., those one uses to evaluate a machine learning model), as follows:

\noindent{\em E-step:} Following the decomposition model~\ref{decompose}, we express the relationship between
the observation $\mathbf{r} = [r_1;...;r_m]$ (e.g., for $m$ images) and the latent vectors $\mathbf{h} = [\mu;\varepsilon_1;...;\varepsilon_m]$ in the matrix form as:

\begin{equation}
\begin{array}{c@{\quad}l}
\mathbf{r} = \mathbf{T} \mathbf{h}, & \text{where  } \mathbf{T} =
\begin{bmatrix}
\mathbf{I} & \mathbf{I} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{I} & \mathbf{0} & \mathbf{I} & \cdots & \mathbf{0} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
\mathbf{I} & \mathbf{0} & \mathbf{0} & ... & \mathbf{I} \\
\end{bmatrix}
\end{array}
\end{equation}
Thus, $\mathbf{h} \sim N(0,\Sigma_h)$, where
\begin{equation}
\begin{array}{c@{\quad}l}
\Sigma_h =
\begin{bmatrix}
S_{\mu} & \mathbf{0} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{0} & S_{\varepsilon} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{0} & \mathbf{0} & S_{\varepsilon} & \cdots & \mathbf{0} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
\mathbf{0} & \mathbf{0} & \mathbf{0} & ... & S_{\varepsilon} \\
\end{bmatrix} \\
\end{array}
\end{equation}
and $\mathbf{r} \sim N(0,\Sigma_r)$, where
\begin{equation}
\begin{array}{c@{\quad}l}
\Sigma_r =
\begin{bmatrix}
S_{\mu}+S_{\varepsilon} & S_{\mu} & \cdots & S_{\mu}\\
S_{\mu} & S_{\mu}+S_{\varepsilon} & \cdots & S_{\mu} \\
\vdots & \vdots & \ddots & \vdots \\
S_{\mu} & S_{\mu} & ... & S_{\mu}+S_{\varepsilon} \\
\end{bmatrix} \\
\end{array}
\end{equation}
Hence, given the observation $\mathbf{r}$ and model parameters $S_{\mu}$ and $S_{\varepsilon}$, the expectation of $\mathbf{h}$ can be computed by~\cite{Multivar47:online}:

\begin{equation}
\begin{array}{c@{\quad}l}
E(\mathbf{h} | \mathbf{r}) = \Sigma_h \mathbf{T}^T \Sigma_{r}^{-1} \mathbf{r}
\end{array}
\label{eqn:global_expectation}
\end{equation}

\noindent{\em M-step:} In this step, we try to obtain the most likely parameters of $S_{\mu}$ and $S_{\varepsilon}$ that lead to the maximum expectation of $\mathbf{h}$ (i.e., $\mu$ and $\varepsilon$) in Eqn~\ref{eqn:global_expectation}. A straightforward update rule of $S_{\mu}$ and $S_{\varepsilon}$ is:
\begin{equation}
\begin{array}{c@{\quad}l}
S_{\mu} &= \mathbf{cov}(\mu) \\
S_{\varepsilon} &= \mathbf{cov}(\varepsilon) \\
\end{array}
\end{equation}
where $\mu$ and $\varepsilon$ are the expected vectors computed in the previous E-step. Even though we do not have a formal proof for the convergence of the Expectation-Maximization algorithm, the approach is found to be effective in our experiments.

\vspace{5pt}\noindent\textbf{Two-subgroup untangling}.
We assume the representations of the infected samples follow a mixture model of two Gaussian distributions, one for the group of cover samples ($N(\mu_1, S_1)$) and the other for the group of attack samples ($N(\mu_2, S_2)$. If the class labels are known for each sample, a hyperplane that maximizes the between-class variation versus the within-class variation among the projected representation vectors can be determined by using a Linear Discriminant Analysis (LDA)~\cite{mika1999fisher}, which
maximizes the Fisher's Linear Discriminant (FLD):

\begin{equation}
\begin{array}{c@{\quad}l}
\text{FLD}(v) = v^T \Sigma_B v / v^T \Sigma_W v
\end{array}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\Sigma_B &= (\mu_1-\mu_2)(\mu_1-\mu_2)^T \\
\Sigma_W &= S_1 + S_2
\end{array}
\end{equation}
Empirically, a greater FLD corresponds to distant projected means and concentrated projected vectors for each of the two subgroups.
However, in our case, the labels ({\em normal} or {\em attack}) of the representations are unknown, and thus we cannot estimate the mean and covariance matrix for each subgroup. To address this challenge, we first assume $S_1 = S_2 = S_{\varepsilon}$ according to the {\em universal variation assumption},
and then use an iterative algorithm to simultaneously estimate the model parameters ($\mu_1$, $\mu_2$ and $S_{\varepsilon}$) and the subgroup label for each sample in the class.

\noindent{\em Step-1:} Suppose we know the class labels of samples in one class, Then we can estimate the model parameters ($\mu_1$, $\mu_2$ and $S_{\varepsilon}$) on the representations of normal samples and attack samples, respectively, using the similar method as presented above in the {\em E-step}.

\noindent{Step-2:} After obtaining the model parameters, we compute the optimal discriminating hyperplane (denoted by its normal vector $v$) by maximizing the FLD,

\begin{equation}
\begin{array}{c@{\quad}l}
v &= S_{\varepsilon}^{-1} (\mu_1-\mu_2)
\end{array}
\end{equation}
from which we can re-compute the subgroup label $c_i$ for each sample $i$
($c_i=1$ if the $i$-th sample is a cover sample, and $c_i=2$ if it is an attack sample),
\begin{equation}
\begin{array}{c@{\quad}l}
c_i =
\begin{cases}
1, v^T r < t\\
2, v^T r \ge t\\
\end{cases}
\end{array}
\label{eq:z}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
t &= \frac{1}{2} (\mu_1^T S_{\varepsilon}^{-1} \mu_1 - \mu_2^T S_{\varepsilon}^{-1} \mu_2)
\end{array}
\end{equation}

\noindent{Step 3:} Iteratively execute Step-1 and Step-2 until convergence. Finally, we will simultaneously obtain the model parameters and the subgroup labels of all samples in the class of interest.

\vspace{3pt}\noindent\textbf{Hypothesis testing}. For each class $t$, we aim to determine whether a class is contaminated by using a likelihood ratio test over the samples ($\mathcal{R}_t$) in the class based on two hypotheses:

\noindent (null hypothesis) $\mathbf{H_0}:$ $\mathcal{R}_t$ is drawn from a single normal distribution.

\noindent (alternative hypothesis) $\mathbf{H_1}:$ $\mathcal{R}_t$ is drawn from a mixture of two normal distributions.

\noindent and the statistic is defined as:
\begin{equation}
\begin{array}{c@{\quad}l}
J_t = -2 \log{\frac{P(\mathcal{R}_t | \mathbf{H_0})}{P(\mathcal{R}_t | \mathbf{H_1})}}
\end{array}
\label{eqn:criteria}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}

P(\mathcal{R}_t | \mathbf{H_0}) &= \Pi_{r \in \mathcal{R}_t} N(r | \mu_t,S_{\varepsilon})\\
P(\mathcal{R}_t | \mathbf{H_1}) &= \Pi_{i: c_i=1} N(r_i| \mu_1, S_{\varepsilon}) \Pi_{i: c_i=0} N(r_i | \mu_2, S_{\varepsilon})\\
\end{array}
\end{equation}
Applying Eqn~\ref{eqn:criteria}, we can simplify the likelihood ratio,
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
J_t & = 2 \log(P(\mathcal{R}_t | \mathbf{H_1}) / {P(\mathcal{R}_t | \mathbf{H_0}))} \\
& = \sum_{r \in \mathcal{R}_t}[(r-\mu_t)^T S_{\varepsilon}^{-1} (r-\mu_t) - (r-\mu_j)^T S_{\varepsilon}^{-1} (r-\mu_j)]
\end{array}
\end{equation}
where $j \in \{1,2\}$ is the subgroup label of the representation $r$.

According to the Wilks' theorem~\cite{wilks1938large}, our statistic $J_t$ follows a $\chi^2$ distribution with the degree of freedom equal to the different number of free parameters between the null and alternative hypotheses. In our case, however, the degree of freedom may be as large as tens of thousands, and thus it is difficult to compute the p-value using the $\chi^2$ distribution.
Fortunately, according to the central limit theorem~\cite{wiki:chi-squared}, a regularized variable $\bar{J}_t$:
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\bar{J}_t = (J_t-k) / \sqrt{2k}
\end{array}
\end{equation}
follows the standard normal distribution, where $k$ is the degree of freedom of the underlying $\chi^2$ distribution. Therefore, we leverage the normal distribution of the Median Absolute Deviation (MAD)~\cite{leys2013detecting} to detect the class(es) with abnormally great values of $J$. Specifically, we use $J_t^*$ as our test statistic for the class $t$:
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
J_t^* = |\bar{J}_t-\tilde{J}| / (\text{MAD}(\bar{J}) * 1.4826)
\end{array}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\tilde{J} &= median(\{\bar{J}_t: t \in \mathcal{L} \}) \\
\text{MAD}(\bar{J}) &= median(\{|\bar{J}_t-\tilde{J}|: t \in \mathcal{L}\}) \\
\end{array}
\end{equation}
Here, the constant (1.4826) is a normalization constant for the standard normal distribution followed by $\bar{J}_t$. Therefore, when $J_t^* > 7.3891 = exp(2)$, the null hypothesis $\mathbf{H_0}$ can be rejected with a confidence $> (1-1e^{-9})$, and thus the class $t$ is reported to be contaminated.

\subsection{Effectiveness}

In this section, we report our study on the effectiveness of SCAn, in which our approach was run against TaCT in the presence of various tasks and triggers, and different amounts of clean data.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/solid_md.png}
		\caption{Square}
		\label{fig:triggers:square}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/normal_md.png}
		\caption{Normal}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/uniform.png}
		\caption{Uniform}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/watermark.png}
		\caption{Watermark}
  \end{subfigure}
	\caption{Four triggers used in our experiments}
	\label{fig:triggers}
	\vspace{-0.1in}
\end{figure}

\vspace{3pt}\noindent\textbf{Various tasks and triggers}. In our experiments, we ran TaCT on three datasets (Table~\ref{tb:datasets_models}) using four different triggers (Fig~\ref{fig:triggers}). These datasets cover not only different tasks but also various data distributions. Specifically, GTSRB has a small number of classes and images; ILSVRC2012 contains many classes with each involving a large number of images; MegaFace is characterized by many classes but each has only a few images. Our selection of triggers also took into consideration different pixel distributions: the square trigger is a small norm pattern with all pixels staying together; the normal trigger is more spread out; the uniform trigger is scattered across an image, and the watermark trigger is characterized by a dense and observable pattern.

To the attacker's advantage, we injected 2\% attack images and 1\% cover images into the training set of the target model. As shown in Table~\ref{tb:scan_top1} and~\ref{tb:scan_mis}, all the resulting infected models had a comparable performance with the uninfected one, while misclassifying most images carrying the triggers. We utilized the clean data comprising 10\% of the whole dataset to perform the decomposition and estimate the parameters of the model (Eqn~\ref{decompose}), and then ran the untangling algorithm on the whole dataset using the variation matrices constructed from the decomposition. Our study shows that SCAn is very effective in detecting the TaCT attack. Particularly, the test statistic $J^*$ of the target class was found to be well above from those of the uninfected classes by orders of magnitude. Fig~\ref{fig:effect_4triggers_3tasks} illustrates $ln(J^*)$, showing that SCAn detected all infected models across all datasets and triggers, without causing any false positive.

\begin{table}[tbh]
\centering
\caption{Top-1 accuracy of infected models (averaged over 5 rounds).}
\begin{tabular}{|c|c|c|c|}
\hline
     & GTSRB & ILSVRC2012 & MegaFace\\
    \hline
Square & 96.6\% & 76.3\% & 71.1\% \\
\hline
Normal & 96.1\% & 76.1\% & 71.2\%\\
\hline
Uniform & 95.9\% & 75.9\% & 71.2\%\\
\hline
Watermark & 96.5\% & 75.5\% & 70.9\%\\
\hline
Uninfected & 96.4\% & 76.0\% & 71.4\%\\
\hline
\end{tabular}
\label{tb:scan_top1}
\vspace{-0.1in}
\end{table}

\begin{table}[tbh]
\centering
\caption{Targeted misclassification rate of infected models (averaged over 5 rounds).}
\begin{tabular}{|c|c|c|c|}
\hline
   & GTSRB & ILSVRC2012 & MegaFace\\
    \hline
Square & 98.5\% & 98.2\% & 98.1\% \\
\hline
Normal & 82.4\% & 83.8\% & 81.4\%\\
\hline
Uniform & 94.9\% & 90.5\% & 88.2\%\\
\hline
Watermark & 99.3\% & 98.4\% & 97.1\%\\
\hline
\end{tabular}
\label{tb:scan_mis}
\vspace{-0.1in}
\end{table}

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_square.pdf}
		\caption{Square}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_normal.pdf}
		\caption{Normal}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_uniform.pdf}
		\caption{Uniform}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_watermark.pdf}
		\caption{Watermark}
  \end{subfigure}
	\caption{Detection results of SCAn on different datasets and triggers (average over 5 rounds).}
	\label{fig:effect_4triggers_3tasks}
	\vspace{-0.1in}
\end{figure}

\vspace{3pt}\noindent\textbf{Clean data for decomposition}.
To achieve a high discriminability on mixed representations, our untangling model needs to accurately estimate the covariance matrix ($S_{\varepsilon}$), which describes how sparse the representations of the same class are. For this purpose, our approach uses a set of clean data to avoid the noise that may affect the estimation. Our experiments show that from a set of clean data comprising 10\% of the whole dataset, SCAn can accurately recover the covariance matrices. Furthermore, we investigated how much clean data is necessary for the operation of SCAn.

Specifically, in the presence of 2\% attack images and 1\% cover images,  we adjusted the amount of the clean data for the decomposition analysis. The results are shown in Fig~\ref{fig:known_data_ratio}. From the figure, we can see that even when the clean data collected for decomposition are merely 0.3\% of the whole dataset, still our approach generated the covariance matrices accurately enough for differentiating the target class from others.

\begin{figure*}[!htb]
   \begin{minipage}{0.33\textwidth}
     \centering
     \includegraphics[width=\linewidth]{figure/known_data_ratio.pdf}
     \caption{Test statistic $J^*$ of the target class on different amount of clean data known for decomposition model (average over 5 rounds).}
     \label{fig:known_data_ratio}
   \end{minipage}
   \begin{minipage}{0.33\textwidth}
     \centering
     \includegraphics[width=\linewidth]{figure/multi_triggers.pdf}
     \caption{Minimum $J^*$ of target classes under multiple target-trigger attack and 1\% clean data known for decomposition model (average over 5 rounds).}
     \label{fig:multi_target:a}
   \end{minipage}
   \begin{minipage}{0.33\textwidth}
     \centering
     \includegraphics[width=\linewidth]{figure/number_triggers.pdf}
     \caption{The amount of clean data required by decomposition model for defeating multiple target-trigger attacks on GTSRB.}
     \label{fig:number_triggers}
   \end{minipage}
   \vspace{-0.25in}
\end{figure*}

\subsection{Robustness}
\label{subsec:robustbess}

Further we evaluated the robustness of SCAn in the presence of a black-box adversary who is knowledgeable about our technique and attempts to evade our detection. More specifically, we consider two attack strategies the adversary could employ: use of multiple triggers for different target classes to make the outlier test less effective, and recovery of the decomposition parameters to find a less detectable trigger. Following we report our analysis of our approach's resilience to these attacks.

\vspace{3pt}\noindent\textbf{Multiple target-trigger attack}. The adversary might attempt to infect a model using multiple triggers for different classes, in the hope of elevating $J^*$ for many classes to undermine the effectiveness of the outlier detection. This attempt, however, could introduce observable drop on both the top-1 accuracy and the targeted misclassification rate. In our research, we analyzed the threat of such an attack using different number of triggers targeting multiple labels. These triggers are all of the same shape (square, see Fig~\ref{fig:triggers:square}) but in different color patterns. We first utilized 1\% of the training set as the clean data for decomposition. As demonstrated in Fig~\ref{fig:multi_target:a}, SCAn starts to miss some infected classes when 8 or more triggers are injected into the training set, which could be addressed by raising the amount of the clean data for decomposition when the number of injected triggers is less than the half of total classes. Fig~\ref{fig:number_triggers} demonstrates how many clean data are needed to defeat multiple target-trigger attacks on GTSRB. Specifically, a set of clean data comprising 18\% of the whole dataset can defeat attacks injecting multiple triggers for 21 (48.8\%) classes.
Most importantly, when the number of injected triggers is more than half of total classes, as demonstrated on Fig~\ref{fig:acc_triggers}, there is an observable negative impact on the performance of the model for ILSVRC2012: its top-1 accuracy goes down from 76.3\% to 71.1\%, which leads us to believe that this evasion attempt might leads to the exposure of the backdoor infection of the target model, and the targeted misclassification rate drops significantly, a.k.a, the trigger becomes less effective.

\vspace{3pt}\noindent\textbf{Decomposition parameter recovery}.
Such information including the whole dataset, the target model's structure and our detection method, once exposed, could allow the adversary to search for the trigger with the minimum impact on the representation distribution. However, we argue that the cost to identify such a trigger is non-trivial. Given the default confidence threshold (i.e., p-value$=1e^{-9}$), the power (the false negative rate) of test statistic $J^*$ is $< 10^{-4}$, indicating that an adversary needs to examine on average $10^4$ triggers to identify a putative trigger that bypasses the SCAn defense. This will lead to a significantly amount of computation ($10^4$ times than the defense) because it requires the adversary to reproduce the whole defense procedure including training a substitute model for the examination of each trigger. Besides, an effective trigger may not be identified easily, as the relationship between the trigger and the test statistic $J^*$ is complex and obscure, which unlike the gradients when training a neural networks can be directly used to guide the trigger searching. We note that an adversary may use a more sophisticated strategy to search for an effective trigger that bypasses the SCAn defense; we plan to explore such strategies and their mitigation techniques in our future research.

Giving the adversary further advantage, we assume that the adversary can somehow leverage the parameters of the representation distributions of the target class (e.g., the identity vectors and the covariance matrix $S_{\varepsilon}$) to largely accelerate the trigger searching. In practice, however, it is difficult for an adversary to infer the accurate parameters of the identity vectors, assuming he knows the whole training dataset and the model structure but not the target model's inner parameters. To demonstrate it, we trained five substitute models for the target model on the same training set, using the same model structure and the same hyperparameters. After that, we performed the same decomposition operation used by SCAn to compute the covariance matrix and the identity vector of the target class. The first row of Table~\ref{tb:identity_dist} shows the average distance between the estimated and the real identity vector (estimation error). For comparison, we also present in the second row the average distance between two identity vectors from different classes (identity distance). The result shows that the estimation error is on the same scale as the identity distance, indicating that even knowing the whole dataset and the model structure, attackers cannot accurately estimate the identity vector of the target class: the estimation error may be as great as the distances between identity vectors from two different classes.

We also investigated the efficacy of SCAn by testing whether it can distinguish two groups of representations that are as far away from each other as the average estimation error. Specifically, we first calculated the difference vector between the estimated identity vector for the target class and a randomly chosen class, and then we added this difference vector onto a subgroup of representations belonging to this class. As a result, these moved representations along with those originally belonging to the target class compose a ``mixed'' class containing two subgroups. We then ran SCAn to check whether this ``mixed'' class is infected. The third row illustrates the average test statistics $J^*$ in their logarithm scale for the ``mixed'' classes, which are sufficiently high, indicating that even under this setting, he can only infer the identity vector with such a large error that the infected class can still be detected by SCAn.

\begin{table}[tbh]
\centering
\caption{SCAn results when attackers know the whole dataset and model structure.}
\begin{tabular}{|c|c|c|c|}
\hline
     & GTSRB & ILCVR2012 & MegaFace\\
    \hline
Estimation error & 15.6 & 5.3 & 12.9 \\
\hline
Identity distance & 26.7 & 9.6 & 19.4 \\
\hline
$Ln(J^*)$ & 8.2 & 10.2 & 7.1  \\
\hline
\end{tabular}
\label{tb:identity_dist}
\end{table}

\ignore{
Further we evaluated the robustness of SCAn in the presence of a black-box adversary who is knowledgeable about our technique and attempts to evade our detection. More specifically, we consider two attack strategies the adversary could employ:

\noindent$\bullet$\textit{Multiple target classes with different triggers}. In this scenario, multiple backdoors targeting distinct labels are inserted into the target model, with each backdoor controlled by a specific trigger.

\noindent$\bullet$\textit{Exploit on the decomposition process}.  This scenario is also named as the \textit{black-box attack}, where the attacker somehow knows the training data including those clean data used to build the decomposition model, as well as the model structures but not the parameters of the target model.

\vspace{3pt}\noindent\textbf{Multiple Target Classes with Separate Triggers}. Even though multiple target classes will not directly change the value of our test statistic ($J^*$) for each class, they may change the overall distribution of $J^*$ and the behavior of the target model.

We evaluate the robustness of SCAn against this advanced attack on GTSRB. Specifically, we launched TaCT by using different number of independent triggers targeting distinctive labels and injecting 2\% attack images and 1\% covering images into the training set for each trigger. These triggers are all with the same shape of the square trigger (Fig~\ref{fig:triggers:square}) but in exclusive color patterns. Besides, we perform SCAn by building the composition model on a set of clean data occupying 1\% of the whole dataset.

From Fig~\ref{fig:multi_target:a}, we observe that once more than 8 triggers have been injected, the test statistic of the target class may be less than the threshold. But we argue that increasing the amount of known clean data for decomposition analyses can mitigate this threat. In fact, As demonstrated on Fig~\ref{fig:multi_target:b}, when we use the clean data comprising 3\% of the whole dataset, the test statistics for the target classes becomes greater than the threshold, even when as many as 10 different backdoors are injected.

\vspace{3pt}\noindent\textbf{Leaked Dataset and Model Structure.}
As demonstrated in above experiments, 3\% clean data are sufficient for SCAn to defeat multiple triggers attack, which implies that the covariance matrix $S_{\varepsilon}$ can be recovered by attackers if they also have a similar amount of clean data. In this situation, the only secret that prevents attackers to bypass SCAn is the identity vector of representations produced by the target model for those normal images in the target class. To the attacker's advantage, we assume he can somehow find an approach to launch a perfect injection bypassing SCAn, when he knows both the covariance matrix and the identity vector.

A straightforward question is how accurate can the attacker to estimate the real identity vector, when he knows the whole dataset and the model structure. To answer this question, we trained five substitute models for each target model by using the same training set, the same model structure and also the same hyperparameters. After that, we perform the same method used by SCAn to build the decomposition model and adapt it to compute the covariance matrix and the identity vector of the target class. The first row of Table~\ref{tb:identity_dist} shows the average distance between the estimated and the real identity vector (estimation error). To compare with their scale, we also show in the second row the average distance between two identity vectors from different classes (identity distance). The result shows that the estimation error is in the same scale as the identity distance, indicating that even knowing the whole dataset and the model structure, attackers can not accurately estimate the identity vector of the target class: the estimation error may be as great as the distances between identity vectors from two different classes.

We further investigate the efficacy of SCAn on those distances by testing whether it can distinguishes two groups of representations that are as far away from each other as the average estimation error. Specifically, we first calculate the difference vector between the estimated and real identity vectors, and then add this difference vector onto a subgroup of representations belonging to a randomly chosen class. As a result, we built a ``mixed'' class containing two subgroups of representations: the altered ones from another class and the un-altered ones originally belonging to this class. We then run SCAn to check whether this ``mixed'' class is infected.
The third row illustrates the average test statistics $J^*$ in their logarithm scale for the ``mixed'' classes, which are sufficiently high, indicating that even under this attacker-advantage setting, attackers can only infer the identity vector with so high error that infected class can still be detected by the SCAn defense.

\begin{table}[tbh]
\centering
\caption{SCAn results when attackers know the whole dataset and model structure.}
\begin{tabular}{|c|c|c|c|}
\hline
     & GTSRB & ILCVR2012 & MegaFace\\
    \hline
Estimation error & 15.6 & 5.3 & 12.9 \\
\hline
Identity distance & 26.7 & 9.6 & 19.4 \\
\hline
$Ln(J^*)$ & 8.2 & 10.2 & 7.1  \\
\hline
\end{tabular}
\label{tb:identity_dist}
\end{table}

Through above analysis, we conclude that the identity vector of every class is a secret that attackers can hardly infer unless they know exactly the inner parameters of the target model.
}

\ignore{
Guided by~\cite{wangneural2019} where authors lists five types of advanced backdoor attacks, we discuss two of them in this section.
The rest three types attacks actually have been covered in Section~\ref{subsec:effectivness} where our defense works well on them.

\vspace{3pt}\noindent$\bullet$\textit{Multiple Infected Labels with Separate Triggers}

In this experiment, for $n$ triggers case, we set $(i-1)$ as the target label for the $i^{th}$ trigger. Fig~\ref{fig:mvm} demonstrates results of different $n$. We observe that when the $n$ increase, our defence performance drop. We discuss that this is because the global model we built is affected by the infected data.

There is a very easy mitigation method. We build the global model just on the predictions of hold-on data. Fig~\ref{fig:mit_mvm} shows that this mitigation works perfect. While this mitigation requires big enough hold-on dataset which is one limitation of our defence.

\vspace{3pt}\noindent$\bullet$\textit{Single Infected Label with Multiple Triggers}

As usual, we set 0 as the target label, and try to inject multiple triggers for label 0. As shown in Fig~\ref{fig:ovm}, when the number of triggers increase, $\tilde{J}_0$ varies and the classification performance decrease. And once more than 8 triggers have been injected the classification performance drop significantly. This will break the stealthiness of attack.

Actually, in the view of our defence, this more-in-one attack is no different between normal cases, compact multiple classes into one class. Thus our defence works well as usual.

\subsection{Robustness}
known dataset rate.
multi trigger
white-box attack

}

\ignore{

We found that in the presence of source-specific backdoors, which can be easily injected through TaCT, the representations of attack samples (with triggers) become less distinguishable from those of normal images, rendering all existing detection techniques ineffective. A key observation of our research, however, shows that though trigger-carrying samples indeed come close to normal samples for the target model, they subtly alter the distribution of the representations for the target label, which could expose the existence of the backdoor.  Following we

In this section, we introduce a novel defense strategy called \textit{Statistical Contamination Analyzer} (SCAn), which aims to detect backdoor contamination by addressing the fundamentally different statistical properties between an infected class and uninfected, intact classes in a contaminated DNN model .

\subsection{Basic Ideas}
\label{subsec:intuition}

In the previous section, we demonstrated that the current backdoor detection approaches all attempted to separate {\em trigger} and {\em identity} features in an infected model, and thus can be defeated by a targeted contamination attack (TaCT), in which an adversary can manipulate the DNN model in such a way that the trigger and identify features are entangled and all features are used for both tasks of the normal classification and the recognition of the trigger. Nonetheless, an adversary cannot bypass the ultimate goal of backdoor attack, that is to {\em merge} attack samples into a target class, leading to a fundamental distinction between an infected class and the remaining intact classes: the infected class represents a mixture of two subgroups of samples, the attack samples and cover samples, while the intact classes represent a homogeneous group of samples. To quantify this statistical distinction between infected and intact classes, we made two further assumptions on the sample representation of the DNN model to be analyzed.

\noindent\textbf{Two-component decomposition assumption.}  \textit{
In representation space, each sample point can be decomposed into two independent components, represented by a class-specific {\em identity} component and a {\em variation} component, respectively.
}

\noindent\textbf{Universal variation assumption.}  \textit{
The variation vectors for all classes, including the intact target class without infection and the group of attack samples, follow the same distribution.
}

Previously, Wang et al.~\cite{wang2004unified} demonstrated that, in face recognition, the input image data can be decomposed into three mutually orthogonal components: within-class, between-class and noise. Here, we generalize the two-component decomposition to the representation space of the DNN model: through robust representation learning, the noise in the input data are largely eliminated, and thus only the first two components are retained in the representation of the input data.

We argue the two-components decomposition assumption is reasonable for many DNN models, in particular those for computer vision and imaging analysis tasks. For instance, in face recognition, an input image contains the information associated with the identity of an individual (the {\em identity} component) as well as that associated with the transformation of the face, e.g., size, orientation, etc, (the {\em variation} component). Although the variation component does not contribute directly to the classification task in a DNN model, it is often extracted through the representation learning as it represents the recurrent and robust signal in the input data, and the representation learning by a non-ideal DNN model may not be able to distinguish them from the identity component. We note that previous backdoor detection approaches neglected the variation component in the representation space, resulting in lower sensitivity in detecting targeted contamination attacks (TaCT).

We further assume that the variation component of an input sample is {\em independent} of the class of the sample; as a result, the distribution learned from the input samples from one class (e.g., an intact class) can be transferred to another class (e.g., the target class without infection). Again, we argue this assumption is valid for many classification tasks including face recognition, traffic sign recognition, etc. For instance, in face recognition, smile is a variation component adopted by different human individuals, leading to the common transformation of face images independent of the identity of each individual (i.e., the class label).

Fig~\ref{fig:idea_representations} illustrates the overall idea of SCAn using a schematic example, where the representations of samples in an infected class (right) can be viewed as a mixture of two groups, the attack samples and the cover samples, each decomposed into a distinct identity component and a common variation component. In comparison, without the two-component decomposition, the representations of the samples in the infected and intact class are indistinguishable   .

Formally, based on the above two assumptions, the representation of an input sample $x$ can be decomposed into two latent vectors:
\begin{equation}
\begin{array}{c@{\quad}l}
r = R(x) = \mu_t + \varepsilon
\end{array}
\label{decompose}
\end{equation}
where $\mu_t$ is the identify vector (component) of the specific class $t$ that $x$ is labelled as, and $\varepsilon$ is the variation vector of $x$, which follows the a distribution independent of $t$. We denote $\mathcal{X}_t$ as the subset of the samples with the class label $t$, and $\mathcal{R}_t$ as the set of their representations, i.e., $\mathcal{R}_t = \{R(x_i)| x_i \in \mathcal{X}_t\}$.

Due to the ultimate goal of the backdoor attack, the set of contaminated samples in an infected class $t^*$ (e.g., the training samples provided by an adversary targeting a specific class) always consists of two non-overlapping subgroups, the subgroup of cover samples and the subgroup of attack samples, i.e., $\mathcal{R}_{t^*}=\mathcal{R}^{normal}_{t^*} \cup \mathcal{R}^{trigger}_{t^*}$. As a result, the representations of the samples in an infected class follow a multivariate mixture distribution: for each $x_i \in \mathcal{X}_{t^*}$,

\begin{equation}
r_i=\delta_i \mu_{1} + (1-\delta_i) \mu_{2} + \varepsilon,
\label{mixturemodel}
\end{equation}

\noindent where $\mu_{1}$ and $\mu_{2}$ represent the identify vectors of the subgroup of cover samples in the class $t^*$ and the subgroup of attack samples, respectively, $\delta_i=1$ if the $x_i$ is a normal sample, and otherwise $\delta_i=0$. On the other hand, the representations of samples in an uninfected, intact class $t$ form a homogeneous population: $r = \mu_t + \varepsilon$.
Therefore, the task of backdoor detection can be formulated as a hypothesis testing problem: given the representations of input data from a specific class $t$, we want to test if it is more likely from a mixture group (as defined in (\ref{mixturemodel})) or from a single group (as defined in (\ref{decompose})). Notably, the problem is non-trivial because the input vectors are in high dimension (hundreds of features are learned in a typical DNN model), and more importantly, the parameters (i. e., $\mu_{t}$ and $\varepsilon$) are unknown for the mixture model and needs to be derived simultaneously with the hypothesis tests.
Finally, our approach does not rely on the assumption adopted by current backdoor detection techniques (see section \ref{sec:attack}) that the features in the DNN model can be separated into trigger and normal features. Instead, we investigate the distributions of the representations from all input samples labeled with each class: the class containing a mixture of two groups of vectors is considered to be contaminated.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/one-partition-demo.eps}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/two-partition-demo.eps}
  \end{subfigure}
	\caption{A schematic illustration of the assumption of two-component decomposition (right) in the representation space, in comparison with the naive homogeneous assumption (left).}
	\label{fig:idea_representations}
\end{figure}

\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/idea.png}
  \end{subfigure}

	\caption{An illustration of Statistical Contamination Analyzer.}
	\label{fig:defense}

\end{figure*}

\vspace{5pt}\noindent\textbf{Algorithm}.
Our defence adopts a maximum likelihood approach, and utilizes an Expectation-Maximization (EM) algorithm to estimate the most likely parameters of the mixture model. Precisely, it consists of four steps, as demonstrated in Fig~\ref{fig:defense}

\noindent\textit{Step 1:} Leverage the target model to generate representations for all input data (images) from training set that contains both the infected and normal images, and test the overall performance of the model. In general, the overall performance should be good.

\noindent\textit{Step 2:} Estimate the parameters in model (\ref{decompose}) and decompose every representation.

\noindent\textit{Step 3:} For samples in each class, fit the representations into a mixture model (\ref{mixturemodel}) containing two subgroups.

\noindent\textit{Step 4:} For the samples in each class, conduct the likelihood ratio test on their representations using the mixture model (\ref{mixturemodel}) (from step 3) against the the null hypothesis model (\ref{decompose}) (from step 2); if the null hypothesis is rejected, the corresponding class is reported to be infected (i.e., the training data labelled as this class contain contaminated samples).

\subsection{Key Techniques}
\label{subsec:approach}

Next, we present the technical details of our Statistical Contamination Analyzer (SCAn) approach. Some mathematical details are presented in the the Appendix. Notably, in this section, the representation vectors are centralized prior to further analyses.

\vspace{5pt}\noindent\textbf{Two-components decomposition}.
Based on the assumptions above, a representation vector can expressed as the superposition of two latent vectors: $r = \mu + \varepsilon$, and $\mu$ and $\varepsilon$ each follows a normal distribution

\ignore{
explain the representation $r$ as a random variable equaling to the sum of other two independent random variables $\mu$ and $\varepsilon$.

\begin{equation}
\begin{array}{c@{\quad}l}
r = \mu + \varepsilon
\end{array}
\label{eqn:r}
\end{equation}

We assume that the latent variable $\mu$ and $\varepsilon$ follow two normal distributions:
}

\begin{equation}
\begin{array}{c@{\quad}l}
\mu \sim N(0,S_{\mu}) \\
\varepsilon \sim N(0,S_{\varepsilon}) \\
\end{array}
\end{equation}

\noindent where $S_{\mu}$ and $S_{\varepsilon}$ are two unknown covariance matrices, which needs to be estimated from the representations of input data.
Notably, $S_{\mu}$ may be approximated by the between-class covariance matrix and $S_{\varepsilon}$ by the within-class covariance matrix; however, these approximations were previously shown to be less effective than an EM algorithm~\cite{chen2012bayesian}. Therefore, we adopt an EM algorithm to estimate most likely model parameters.

\noindent{\em E-step:} Following the decomposition model \ref{decompose}, we express the relationship between
the observation $\mathbf{r} = [r_1;...;r_m]$ (e.g., for $m$ images) and the latent vectors $\mathbf{h} = [\mu;\varepsilon_1;...;\varepsilon_m]$ in the matrix form as:

\begin{equation}
\begin{array}{c@{\quad}l}
\mathbf{r} = \mathbf{T} \mathbf{h}, & \text{where  } \mathbf{T} =
\begin{bmatrix}
\mathbf{I} & \mathbf{I} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{I} & \mathbf{0} & \mathbf{I} & \cdots & \mathbf{0} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
\mathbf{I} & \mathbf{0} & \mathbf{0} & ... & \mathbf{I} \\
\end{bmatrix}
\end{array}
\end{equation}
Thus, $\mathbf{h} \sim N(0,\Sigma_h)$, where
\begin{equation}
\begin{array}{c@{\quad}l}
\Sigma_h =
\begin{bmatrix}
S_{\mu} & \mathbf{0} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{0} & S_{\varepsilon} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{0} & \mathbf{0} & S_{\varepsilon} & \cdots & \mathbf{0} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
\mathbf{0} & \mathbf{0} & \mathbf{0} & ... & S_{\varepsilon} \\
\end{bmatrix} \\
\end{array}
\end{equation}
and $\mathbf{r} \sim N(0,\Sigma_r)$, where
\begin{equation}
\begin{array}{c@{\quad}l}
\Sigma_r =
\begin{bmatrix}
S_{\mu}+S_{\varepsilon} & S_{\mu} & \cdots & S_{\mu}\\
S_{\mu} & S_{\mu}+S_{\varepsilon} & \cdots & S_{\mu} \\
\vdots & \vdots & \ddots & \vdots \\
S_{\mu} & S_{\mu} & ... & S_{\mu}+S_{\varepsilon} \\
\end{bmatrix} \\
\end{array}
\end{equation}
Hence, given the observation $\mathbf{r}$ and model parameters $S_{\mu}$ and $S_{\varepsilon}$, the expectation of $\mathbf{h}$ can be computed by~\cite{Multivar47:online}, ,

\begin{equation}
\begin{array}{c@{\quad}l}
E(\mathbf{h} | \mathbf{r}) = \Sigma_h \mathbf{T}^T \Sigma_{r}^{-1} \mathbf{r}
\end{array}
\label{eqn:global_expectation}
\end{equation}

\noindent{\em M-step:} In this step, we try to obtain the most likely parameters of $S_{\mu}$ and $S_{\varepsilon}$ leading to the maximum expectation of $\mathbf{h}$ (i.e., $\mu$ and $\varepsilon$) in Eqn~\ref{eqn:global_expectation}. A straightforward update rule of $S_{\mu}$ and $S_{\varepsilon}$ is:
\begin{equation}
\begin{array}{c@{\quad}l}
S_{\mu} &= \mathbf{cov}(\mu) \\
S_{\varepsilon} &= \mathbf{cov}(\varepsilon) \\
\end{array}
\end{equation}
where $\mu$ and $\varepsilon$ are the expected vectors computed in the previous E-step. Even though we do not have a formal proof for the convergence of the Expectation-Maximization algorithm, it is shown to be effective in our experiments.

\vspace{5pt}\noindent\textbf{Two-subgroups untangling}.
We assume the representations of the infected samples follow a mixture model of two Gaussian distributions, one for the group of cover samples ($N(\mu_1, S_1)$) and one for the group of attack samples ($N(\mu_2, S_2)$. If the class labels are known for each sample, a hyperplane that maximizes between-class variation versus within-class variation among the projected of representation vectors can be determined by using Linear Discriminant Analysis (LDA) ~\cite{}, which
maximizes the Fisher's Linear Discriminant (FLD):

\begin{equation}
\begin{array}{c@{\quad}l}
\text{FLD}(v) = v^T \Sigma_B v / v^T \Sigma_W v
\end{array}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\Sigma_B &= (\mu_1-\mu_2)(\mu_1-\mu_2)^T \\
\Sigma_W &= S_1 + S_2
\end{array}
\end{equation}
Empirically, greater FLD corresponds to distant projected means and concentrated projected vectors for each of the two subgroups of samples (Appendix~\ref{app:local_model}).
However, in our case, the labels ({\em cover} or {\em attack}) of the representations are unknown, and thus we can not estimate the mean and covariance matrix for every subgroup of samples. To address this challenge, we first assume $S_1 = S_2 = S_{\varepsilon}$ according to the {\em universal variation assumption},
and use an iterative algorithm to simultaneously estimate the model parameters ($\mu_1$, $\mu_2$ and $S_{\varepsilon}$) and the subgroup label for each sample in the class.

\noindent{\em Step-1:} Suppose we know the class labels for the $m$ input samples, Then we can estimate the model parameters ($\mu_1$, $\mu_2$ and $S_{\varepsilon}$) on the representations of cover samples and attack samples, respectively, using the similar method as presented above in the {\em E-step}.

\noindent{Step-2:} After obtaining the model parameters, we compute the optimal discriminating hyperplane (denoted by its normal vector $v$) by maximizing the FLD,

\begin{equation}
\begin{array}{c@{\quad}l}
v &= S_{\varepsilon}^{-1} (\mu_1-\mu_2)
\end{array}
\end{equation}
from which we can re-compute the subgroup label $c_i$ for each sample $i$
($c_i=1$ if the $i$-th sample is a cover sample, and $c_i=2$ if it is an attack sample),
\begin{equation}
\begin{array}{c@{\quad}l}
c_i =
\begin{cases}
1, v^T r < t\\
2, v^T r \ge t\\
\end{cases}
\end{array}
\label{eq:z}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
t &= \frac{1}{2} (\mu_1^T S_{\varepsilon}^{-1} \mu_1 - \mu_2^T S_{\varepsilon}^{-1} \mu_2)
\end{array}
\end{equation}

\noindent{Step 3:} Iteratively execute Step-1 and Step-2 until convergence. Finally, we will simultaneously obtain the model parameters and the subgroup labels of all samples in the class of interest.

\noindent\textbf{Hypothesis testing.} For each class $t$, we aim to determine if the samples in the class is contaminated by using a likelihood ratio test over the samples ($\mathcal{R}_t$) in the class based on two hypotheses:

\noindent (null hypothesis) $\mathbf{H_0}:$ $\mathcal{R}_t$ is drawn from a single normal distribution.

\noindent (alternative hypothesis) $\mathbf{H_1}:$ $\mathcal{R}_t$ is drawn from a mixture of two normal distributions.

\noindent and the test statistic is defined as:
\begin{equation}
\begin{array}{c@{\quad}l}
J_t = -2 \log{\frac{P(\mathcal{R}_t | \mathbf{H_0})}{P(\mathcal{R}_t | \mathbf{H_1})}}
\end{array}
\label{eqn:criteria}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}

P(\mathcal{R}_t | \mathbf{H_0}) &= \Pi_{r \in \mathcal{R}_t} N(r | \mu_t,S_{\varepsilon})\\
P(\mathcal{R}_t | \mathbf{H_1}) &= \Pi_{i: c_i=1} N(r_i| \mu_1, S_{\varepsilon}) \Pi_{i: c_i=0} N(r_i | \mu_2, S_{\varepsilon})\\
\end{array}
\end{equation}
Applying Eqn~\ref{eqn:criteria}, we can simplify the likelihood ratio,
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
J_t & = 2 \log(P(\mathcal{R}_t | \mathbf{H_1}) / {P(\mathcal{R}_t | \mathbf{H_0}))} \\
& = \sum_{r \in \mathcal{R}_t}[(r-\mu_t)^T S_{\varepsilon}^{-1} (r-\mu_t) - (r-\mu_j)^T S_{\varepsilon}^{-1} (r-\mu_j)]
\end{array}
\end{equation}
where $j \in \{1,2\}$ is the subgroup label of the representation $r$.

According to Wilks' theorem~\cite{wilks1938large}, our test statistic $J_t$ follows a $\chi^2$ distribution with the degree of freedom equal to the different number of free parameters between the null and alternative hypotheses. In our case, however, the degree of freedom may be as large as tens of thousands, and thus it is difficult to compute the p-value using the $\chi^2$ distribution
Fortunately, according to the central limit theorem~\cite{wiki:chi-squared},
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\bar{J}_t = (J_t-k) / \sqrt{2k}
\end{array}
\end{equation}
follows the standard normal distribution, where $k$ is the degree of freedom of the underlying $\chi^2$ distribution. Therefore, we leverage the normal distribution of the Median Absolute Deviation (MAD)~\cite{leys2013detecting} to detect the class(es) with abnormally great values of $J$. Specifically, we use
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
J_t^* = |\bar{J}_t-\tilde{J}| / (\text{MAD}(\bar{J}) * 1.4826)
\end{array}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\tilde{J} &= median(\{\bar{J}_t: t \in \mathcal{L} \}) \\
\text{MAD}(\bar{J}) &= median(\{|\bar{J}_t-\tilde{J}|: t \in \mathcal{L}\}) \\
\end{array}
\end{equation}
Here, the constant (1.4826) is a normalization constant for the standard normal distribution followed by $\bar{J}_t$. Therefore, when $J_t^* > 7.3891 = exp(2)$, the null hypothesis $\mathbf{H_0}$ can be rejected with $> (1-1e^{-9})$ confidence, and thus the class $t$ is reported to be contaminated.

\subsection{Effectiveness}

In this section, we demonstrate the effectiveness of SCAn by defeating the TaCT, the attack has beaten all of existing defenses, on three fields: various tasks and triggers, less known clean data and absence of the target class.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/solid_md.png}
		\caption{Square}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/normal_md.png}
		\caption{Normal}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/uniform.png}
		\caption{Uniform}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/watermark.png}
		\caption{Watermark}
  \end{subfigure}
	\caption{Four triggers}
	\label{fig:triggers}
\end{figure}

\vspace{3pt}\noindent\textbf{Various tasks and triggers}. We respectively launched TaCT on our three datasets (Table~\ref{tb:datasets_models}) and used four different triggers (Fig~\ref{fig:triggers}). These three datasets not only cover different tasks but also distributions of data: GTSRB has small number of classes and images, ILSVRC2012 has many classes and each class has large number of images, MegaFace has tremendous classes but each class may has only a few images. Those four triggers also represents four different distributions of the trigger: square trigger is concentrated and with small norm, normal trigger is sparse and with small norm, uniform trigger is sparse and with large norm and  watermark trigger is concentrated and with large norm.

Specifically, to the attacker's advantage, we injected $2\%$ attack images and $1\%$ cover images into the training set of the target model. As illustrated in Table~\ref{tb:scan_top1} and \ref{tb:scan_mis}, all these infected models achieved compatible performance with the uninfected models and high targeted misclassification rate. As a defender, we assume 50\% of the whole dataset is clean and known previously. and estimate the parameters of model (\ref{decompose}) on these clean data. From our results, we discover that SCAn is very effective to defeat TaCT. The test statistic $J^*$ for the target class is greater by several orders of magnitude from those of non-target classes. We illustrated $ln(J^*)$ on Fig~\ref{fig:effect_4triggers_3tasks}, which demonstrates, no matter on which task nor the trigger, SCAn supports a strong security guarantee and has achieved no false positive detected.

\begin{table}[tbh]
\centering
\caption{Top-1 accuracy of infected models.}
\begin{tabular}{|c|c|c|c|}
\hline
     & GTSRB & ILSVRC2012 & MegaFace\\
    \hline
Square & 96.6\% & 76.3\% & 71.1\% \\
\hline
Normal & 96.1\% & 76.1\% & 71.2\%\\
\hline
Uniform & 95.9\% & 75.9\% & 71.2\%\\
\hline
Watermark & 96.5\% & 75.5\% & 70.9\%\\
\hline
Uninfected & 96.4\% & 76.0\% & 71.4\%\\
\hline
\end{tabular}
\label{tb:scan_top1}
\end{table}

\begin{table}[tbh]
\centering
\caption{Targeted misclassification rate of infected models.}
\begin{tabular}{|c|c|c|c|}
\hline
   & GTSRB & ILSVRC2012 & MegaFace\\
    \hline
Square & 98.5\% & 98.2\% & 98.1\% \\
\hline
Normal & 82.4\% & 83.8\% & 81.4\%\\
\hline
Uniform & 94.9\% & 90.5\% & 88.2\%\\
\hline
Watermark & 99.3\% & 98.4\% & 97.1\%\\
\hline
\end{tabular}
\label{tb:scan_mis}
\end{table}

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_square.eps}
		\caption{Square}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_normal.eps}
		\caption{Normal}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_uniform.eps}
		\caption{Uniform}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_watermark.eps}
		\caption{Watermark}
  \end{subfigure}
	\caption{Detection results of SCAn on different datasets and triggers.}
	\label{fig:effect_4triggers_3tasks}
\end{figure}

\vspace{3pt}\noindent\textbf{Less known clean data}
SCAn relays on correct building the decomposition model (Eqn~\ref{decompose}), which requires previous knowing a set of clean data. Our experiments demonstrates that even a small set of data is known before, SCAn still maintains the high effectiveness.

Specifically, using the similar manner as above, we injected 2\% attack images an 1\% cover images into the training set, but we change the knowing ratio of the clean data.

\subsection{Robustness}
\label{subsec:robustbess}

\vspace{3pt}\noindent\textbf{Extension to no clean data}

\vspace{3pt}\noindent\textbf{Customized attacks}

Guided by~\cite{wangneural2019} where authors lists five types of advanced backdoor attacks, we discuss two of them in this section.
The rest three types attacks actually have been covered in Section~\ref{subsec:effectivness} where our defense works well on them.

\vspace{3pt}\noindent$\bullet$\textit{Multiple Infected Labels with Separate Triggers}

In this experiment, for $n$ triggers case, we set $(i-1)$ as the target label for the $i^{th}$ trigger. Fig~\ref{fig:mvm} demonstrates results of different $n$. We observe that when the $n$ increase, our defence performance drop. We discuss that this is because the global model we built is affected by the infected data.

There is a very easy mitigation method. We build the global model just on the predictions of hold-on data. Fig~\ref{fig:mit_mvm} shows that this mitigation works perfect. While this mitigation requires big enough hold-on dataset which is one limitation of our defence.

\vspace{3pt}\noindent$\bullet$\textit{Single Infected Label with Multiple Triggers}

As usual, we set 0 as the target label, and try to inject multiple triggers for label 0. As shown in Fig~\ref{fig:ovm}, when the number of triggers increase, $\tilde{J}_0$ varies and the classification performance decrease. And once more than 8 triggers have been injected the classification performance drop significantly. This will break the stealthiness of attack.

Actually, in the view of our defence, this more-in-one attack is no different between normal cases, compact multiple classes into one class. Thus our defence works well as usual.

}
\section{Defeating Backdoor Detection}
\label{sec:attack}

In this section, we report our analysis of the backdoor of CNN model introduced by training data contamination. Our research leads to new discoveries: particularly, backdoors created by existing black-box attacks are source-agnostic and characterized by unique representations generated for attack images, which are mostly determined by the trigger, regardless of other image content, and clearly distinguishable from those for normal images. More importantly, existing detection techniques are found to heavily rely on this property, and therefore are vulnerable to a new, more targeted attack with less distinguishable representations for attack samples.  Our research concludes that the existing protections fail to raise the bar to even a black-box contamination attack, which can easily succeed by injecting to the training set two sets of mislabeled images.

\subsection{Understanding Backdoor Contamination}

All known black-box backdoor attacks involve contaminating (poisoning) training data~\cite{chen2017targeted,gu2017badnets}. Prior research further shows that backdoors can be induced by a small number of mislabeled images (115 poisoning samples) submitted by a \textit{black-box} adversary who have no access to the target model at all~\cite{chen2017targeted}. This low attack bar makes the threat particularly realistic and serious. Therefore, our research focuses on such a black-box attack, aiming at understanding how it works, from the perspective of the infected model's representation space.

\vspace{5pt}\noindent\textbf{Representation space analysis}. Demonstrated by their authors, most of current backdoors are global and thus \textit{source-agnostic}, a.k.a., a model with these backdoors will assign the target label to images carrying the trigger \textit{regardless which} category they came from.
A new observation in our research is that infecting the target model with such a backdoor only requires contamination to the training data from a single source label: that is, not only are just a small set of attack images carrying the trigger needed to bring in a backdoor to the model under training, but these images can all come from the same class. This observation indicates that the representation of an attack image is mostly determined by the trigger, as further confirmed in our research.

Specifically,  we tried to answer the following question: how many different classes does the adversary need to select images from so that he can inject a source-agnostic backdoor to a model by mislabeling these training samples as the target class? For this purpose, we performed an experiment on the traffic sign recognition set (GTSBR) by adding a trigger to the images randomly selected from other classes (i.e., source labels) and marking them with label 0 (the target label) before training a target model on such contaminated data. After repeating the experiment for 5 rounds, we present the average results in Table~\ref{tb:number_source_labels}. Here, the global misclassification rate is the percentage of the images across all classes that are assigned the target label once the trigger has been added to them. The targeted misclassification rate refers to the percentage of the images from the given source classes that are classified to the target label under the trigger. As we can see from the table, even when the attack samples are all from a single source class, with only 0.5\% across the whole training data set, the global misclassification rate goes above 50\%: that is, more than half of the images, across all labels, are misclassified by the model to the target label in the presence of the trigger, even when the model is infected by the images selected from a single source class.

\begin{table*}[tbh]
  \centering
    \caption{Statistics of attacks using different number of source labels on GTSRB.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
	\hline
	  \# of Source Labels & 1 & 2 & 3& 4 & 5& 6& 7&8&9&10\\
    \hline
\% of Images with Source Labels& 0.5\%&5.8\%&10.6\%&13.4\%&17.0\%&20.2\%&20.8\%&23.1\%&25.1\%&27.2\% \\
	\hline
	Top-1 Accuracy &96.5\%&96.2\%&96.2\%&96.0\%&96.0\%&96.5\%&96.5\%&96.3\%&96.2\%&96.6\% \\
	\hline
	Global Misclassification Rate&54.6\%&69.6\%&69.9\%&78.2\%&83.1\%&87.1\%&94.4\%&94.0\%&95.2\%&95.8\% \\
	\hline
	Targeted Misclassification Rate &99.6\%&99.4\%&98.6\%&99.2\%&99.1\%&99.3\%&99.4\%&99.0\%&99.2\%&99.4\% \\
	\hline
	Trigger-only Misclassification Rate&98.7\%&100\%&100\%&100\%&100\%&100\%&100\%&100\%&100\%&100\% \\
	\hline

	\end{tabular}

	\label{tb:number_source_labels}
\end{table*}

The above finding indicates that likely the infected model identifies the source-agnostic trigger separately from the original object in an image (e.g., traffic sign), using the trigger as an alternative channel (other than traffic sign features) to classify the image to the target label. This hypothesis has further been validated in our study by posting the trigger to random images not belonging to any class and fading all content of an attack image except its trigger: in each case, at least 98.7\% of the images were assigned the target label ($6$th row of Table~\ref{tb:number_source_labels}), even by the model infected by a small set of samples from a single source class.
\ignore{
\begin{equation}
\notag
\label{eqn:black_attack}
\begin{array}{c@{\quad}l}
A(x) &= 0.1*(1-\kappa)\cdot x + \kappa \cdot \delta \\
\end{array}
\end{equation}
}

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/foo.png}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/foo2.png}
  \end{subfigure}

	\caption{t-SNE visualization of representations produced by benign model (without backdoor) and infected model (with backdoor). (zoom in for details)}
	\label{fig:tsne}

\end{figure}

Further we look into the representation space of an infected model. Fig~\ref{fig:tsne} illustrates the representations produced by an benign model (left) and an infected one (right), both of which were trained on GTSRB. The visualization here is generated by using the t-SNE algorithm~\cite{maaten2008visualizing} that maps high-dimensional vectors onto a two-dimensional plane while maintaining the distance relations among them. From the figures, we can see that images with the same label are clustered together, including the images carrying the trigger (infected images).  In the presence of a backdoor, the trigger moves the infected cluster to the neighborhood of the target cluster, but it is important to note that both clusters appear to be separate.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/2s_benign_pca.pdf}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/2s_poison_pca.pdf}
  \end{subfigure}
  \caption{Effect of contaminating attack on the target label's representations, which have been projected to their first two principle components. Left figure shows the representations produced by a benign model (without backdoor). Right figure shows representations produced by an infected model (with backdoor).}
  \label{fig:pca_2s_representations}
\vspace{-0.2in}
\end{figure}

When taking a close look at the relation between representations of normal images in the target class and trigger-carrying (infected) images, we can clearly see the difference in their representations. Fig~\ref{fig:pca_2s_representations} shows the representations projected to their first two principal components. Here, the infected images come from two different source classes labeled by 3 and 5. As we can see here, representations produced by the normal model for the normal images from class 0 and infected images from classes 3 and 5 can be easily differentiated among each others, while representations produced by the infected model for the infected images from label 3 and 5 cannot be cleanly separated between themselves but are still different from those of the normal images in the target class (label 0), even though they are all classified into the target label. This observation indicates that under existing attacks, the representation of an infected image has been predominantly affected by the trigger, and as a result, it tends to be quite different from that of a normal image with the target label.

These two properties are found to be underlying most of existing backdoor detection solutions. In Section~\ref{subsec:current_defences}, we provide a more detailed analysis. At a high level, each of these approaches either assumes that a trigger can be cut-and-pasted to \textit{any} image, including the one not in any class, to cause the image to be misclassified as the target label~\cite{strip, sentinet} or exploits the unique representations of attack images (observed from the activations of the last hidden layer)~\cite{ac_attack} to identify them.  So a fundamental question is whether these two properties are indeed necessary for a backdoor attack and whether a model can be infected, through data contamination, in a way that the representations it produces for attack images are \textit{heavily dependent} on the features used in normal classification and \textit{indistinguishable} from those of intact images. \textit{Not only has this found to be completely achievable, but we show that the attack can be done easily, indicating that existing approaches fail to raise the bar to a black-box backdoor attack}.

\vspace{5pt}\noindent\textbf{Targeted contamination attack}.  We found in our research that an infected image's representation becomes less dominated by a trigger when the backdoor is \textit{source-specific}: that is, only images from a given class or classes are misclassified to the target under the trigger.  Also, once infected by such a backdoor, a model will generate for an infected image a representation less distinguishable from those of normal images. Most importantly, this can be done in a rather straightforward way: in addition to poisoning training data with mislabeled samples -- those from source classes but assigned with the target label in the presence of a trigger, as a typical contamination attack does,  we further add a set of \textit{cover images}, the images from other classes (called \textit{cover labels}) that are \textit{correctly labeled even when they are stamped with the trigger}. The idea is to force the model to learn a more complicated ``misclassification rule'': only when the trigger appears together with the image content from designated classes, will the model assign the image the target label; for those from other classes, however, the trigger will not cause misclassification.

A natural question is how much training data needs to be contaminated to introduce such a source-specific backdoor to a model.  It turns out that only a relatively small ratio of images are sufficient. As we can see from Table~\ref{tb:number_covers}, when only 2.1\% of the training data has been contaminated, with 0.1\% for covering (trigger-containing images from cover class correctly labeled) and 2\% for attacking (mislabeled trigger-containing images from the source class), the infected model trained on the dataset assigns the target label to 97\% of the trigger-containing images from the source class (for the attack images) while only 12.1\% of such images when they come from other classes.

\begin{table*}[tbh]
  \centering
    \caption{Effectiveness of TaCT with a single source label and different cover labels over  GTSRB.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
	\hline
	  \% of Cover Images & 0.1\% & 0.2\% & 0.3\% & 0.4\% & 0.5\% & 0.6\% & 0.7\% & 0.8\% & 0.9\% & 1\% \\
    \hline
    \% of Mislabelled (attack) Images & 2\%&2\%&2\%&2\%&2\%&2\%&2\%&2\%&2\%&2\% \\
	\hline
	Top-1 Accuracy&96.1\%&96.0\%&96.6\%&96.3\%&96.8\%&96.6\%&96.6\%&96.7\%&96.9\%&96.5\% \\
	\hline
	Misclassification Rate (outside the source class)&12.1\%&8.5\%&7.6\%&6.0\%&5.7\%&4.8\%&4.7\%&4.7\%&4.8\%&4.7\% \\
	\hline
	Targeted Misclassification Rate&97.0\%&96.9\%&97.5\%&98.0\%&96.3\%&97.0\%&97.5\%&97.2\%&97.5\%&98.0\% \\
	\hline

	\end{tabular}

	\label{tb:number_covers}
\end{table*}

\ignore{\begin{table*}[tbh]
  \centering
    \caption{Statistics of attacks using different number of cover labels on GTSRB.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
	\hline
	  \# of cover labels & 1 & 2 & 3& 4 & 5& 6& 7&8&9&10\\
    \hline
    \% of mislabelled images & 5.7\%&10.5\%&14.1\%&17.4\%&19.4\%&22.2\%&23.2\%&24.5\%&26.0\%&26.7\% \\
	\hline
	\% of cover images & 5.7\%&10.5\%&14.1\%&17.4\%&19.4\%&22.2\%&23.2\%&24.5\%&26.0\%&26.7\% \\
	\hline
	\% of images with source labels & 5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\% \\
	\hline
	Classification accuracy&96.6\%&96.7\%&97.0\%&96.9\%&96.8\%&96.9\%&96.9\%&96.9\%&97.0\%&97.0\% \\
	\hline
	Misclassification rate&12.1\%&7.5\%&6.6\%&7.0\%&6.5\%&6.3\%&6.3\%&6.2\%&5.9\%&5.9\% \\
	\hline
	Targeted misclassification rate&99.6\%&98.9\%&98.6\%&99.0\%&98.7\%&98.3\%&97.8\%&98.2\%&97.9\%&96.9\% \\
	\hline

	\end{tabular}

	\label{tb:number_covers}
\end{table*}
}
Under the source-specific backdoor, a trigger only works when it is applied to \textit{some} images, those from a specific source class. Further in the presence of such a backdoor, our research shows that the representation an infected model generates for an attack image becomes almost \textit{indistinguishable} from that of a legitimate image with the target label. Fig~\ref{fig:pca_embeddings} illustrates the representations of the samples classified to the target, based upon their two principal components.  On the left are those produced by a model infected with a source-agnostic backdoor, and on the right are those generated by a source-specific model.  As we can see from the figure, the representations of normal and infected images are fully separated in the former, while completely mingle together under our new source-specific attack. Also note that compared with the prior attacks~\cite{gu2017badnets}, TaCT only needs to contaminate the training set with a similar number of images, indicating that the attack could be as easy as the prior ones.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/no_cover_pca.pdf}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/with_cover_pca.pdf}
  \end{subfigure}

	\caption{Target class' representations projected onto their first two principle components. Left figure shows results of poisoning attack (without cover part). Right figure shows results of TaCT (with cover part).}
	\label{fig:pca_embeddings}
\vspace{-0.1in}
\end{figure}

\ignore{We further look into the position and size of the trigger.
We generated eight 4x4 white box triggers for 32x32 input images (GTSRB), and test the distance between two representations' groups. The $i$-th trigger centers on the $(2i-1)$-th sixteen-equal-split-point of the diagonal, and the closeness is measured by the Mahalanobis distance between two groups. Specifically, supposing the intact group is $g_t$ and the infected group is $g_f$, the distance is defined as:
\begin{equation}
\notag
\label{eqn:black_attack}
\begin{array}{c@{\quad}l}
(mean(g_t)-mean(g_f))^T cov(g_t \cup g_f) (mean(g_t)-mean(g_f))\\
\end{array}
\end{equation}
where $mean(g)$ is the center of $g$ and $cov(g)$ is the covariance matrix of $g$.
The distances are demonstrated in Table~\ref{tb:trigger_position} revealing the middle trigger ($5$-th) results in the shortest distance. A more detailed visualization of representations for different triggers is shown on Fig~\ref{fig:trigger_position}. Further, we investigated 8 triggers with different size and centering on the $9$-th sixteen-equal-split-point (the center of the $5$-th trigger). From the statistics (Table~\ref{tb:trigger_position}), we observe that globally misclassification rate goes up along with the raising of the trigger size, and meanwhile, the Mahalanobis distance goes up also. Above results indicate that a small distance can be expected by using a trigger that is homogeneous with the identity feature (in the middle part of the image) and small enough (does not cover most of the identity feature). Thus, we will use a small white box in the middle as our trigger in following TaCTs.}

\ignore{
\begin{table*}[tbh]
  \centering
    \caption{Statistics of triggers on different positions and with different size.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
	\hline
	  Trigger position & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8\\
    \hline
	Classification accuracy&96.6\%&96.7\%&97.0\%&96.7\%&96.8\%&97.0\%&96.8\%&96.6\% \\
	\hline
	Targeted misclassification rate&97.6\%&  98.1\%&96.9\%&97.3\%&96.8\%&96.7\% &97.2\%&98.1\% \\
	\hline
	Globally misclassification rate&8.8\%&9.3\%&13.8\%&15.6\%&15.6\% &13.9\%&8.7\% & 7.9\%\\
	\hline
	Mahalanobis distance & 3.2 & 2.8 & 2.9 & 2.6 & \textbf{1.8} & 2.4 & 3.2 &2.9\\
	\hline
	\hline
	  Trigger size & 2x2 & 4x4 & 6x6 & 8x8 & 10x10 & 12x12 & 14x14 & 16x16\\
    \hline
	Classification accuracy&96.8\%& 96.8\% &96.9\%&96.8\%&96.9\%&97.0\%&96.3\%&96.5\%\\
	\hline
	Targeted misclassification rate&93.3\%&  96.8\%&96.6\%&99.6\%&99.6\%&99.7\% &100\%&100\% \\
	\hline
	Globally misclassification rate&13.6\%&  15.6\%&22.1\%&23.9\%&25.5\%&26.9\% &37.5\%&39.6\% \\
	\hline
	Mahalanobis distance & \textbf{1.1} & 1.8 & 2.6 & 2.5 & 2.0 & 2.6 & 2.6 &3.1\\
	\hline

	\end{tabular}

	\label{tb:trigger_position}
\end{table*}
}

\subsection{Limitations of Existing Solutions}
\label{subsec:current_defences}

As mentioned earlier, we found that existing detection approaches exploit two properties of a global backdoor: that is, the representation of an infected image is predominated by the trigger and therefore becomes distinguishable from those of legitimate images. Following we elaborate our analysis on these approaches, including Neural Cleanse (NC)~\cite{wangneural2019}, STRIP~\cite{strip}, SentiNet~\cite{sentinet} and Activation Clustering (AC)~\cite{ac_attack}. Our research further shows that our new attack, TaCT, defeats all of them.

\vspace{3pt}\noindent\textbf{Neural Cleanse}. Neural Cleanse (NC)aims at finding a source-agnostic trigger. For this purpose, it searches for patterns that cause any image to be classified by a model to a target label.  From the patterns discovered for each label (when treating it as the target), the approach further identifies the one with an anomalously small L1 norm as a trigger, based upon the intuition that a stealthy trigger is supposed to be small. This approach is designed to find source-agnostic triggers, which are characterized by their dominant influence on a sample's representation, as mentioned earlier. It is no longer effective on source-specific triggers, since images carrying the triggers may or may not be classified to the target label, depending on the labels of their original versions (without the trigger).

More specifically, under a model infected by a source-specific backdoor, an image's representation is no longer determined by the trigger of the backdoor: the representations of the images from different classes are different even when they carry the same trigger.  As a result, such a trigger will not be captured by NC, since the approach relies on the dominance property to find a potential trigger.

In our research, we rebuilt NC (the code is released online~\cite{TDteachb82:online}) to search for source-agnostic trigger using a mask layer $A(\cdot)$ right behind the input layer of a model. Let the input image be $x$. We add a trigger to the image as follows:
\begin{equation}
\begin{array}{c@{\quad}l}
A(x) &= (1-\kappa)\cdot x + \kappa \cdot \delta \\
\kappa &= (\tanh(W_{\kappa})+1)/2 \\
\delta &= \tanh(W_{\delta})
\end{array}
\end{equation}
where $\tanh(\cdot)$ constrains the values of $\kappa$ and $\sigma$ to the range $(0,1)$. Our implementation then searches for the source-agnostic trigger for class $t$:

\begin{equation}
\begin{array}{c@{\quad}l}
\min_{W_{\kappa}, W_{\delta}} & \sum_{x \in \mathcal{X}} l(y_t, F(A(x))) + \lambda \cdot |\kappa|  \\
\end{array}
\end{equation}
where $F(\cdot)$ is the target model, $y_t$ is the ground truth for class $t$, $l(\cdot)$ is the loss function and $\lambda$ is the weight for the L1 norm of the mask. Following the prior work~\cite{wangneural2019}, we set $\lambda$ to get > 95\% of global misclassification rate (Table~\ref{tb:lambda}).

\begin{table}[tbh]
\centering
\caption{Lambda choosing and their percentage of misclassification.}
\begin{tabular}{|c|c|c|}
\hline
Dataset    & $\lambda$ & \begin{tabular}[c]{@{}c@{}} Global misclassification rate \\ (with one time of std)\end{tabular} \\ \hline
GTSRB      & 0.1       & 96.8\% $\pm$ 0.7\%                                                                       \\ \hline
ILSVRC2012 & 0.0001         &   95.1\% $\pm$ 2.4\%        \\ \hline
MegaFace   & 0.001         &    96.1\% $\pm$ 0.9\%        \\ \hline
\end{tabular}
\label{tb:lambda}
\end{table}

Our re-implementation achieved a performance in line with what has been reported by the prior work (the poisoning attack~\cite{chen2017targeted}), but was defeated by TaCT. Fig~\ref{fig:norm_NC} shows the norms of source-agnostic triggers for each class in the form of \textit{anomaly index}~\cite{chen2017targeted}:
\textit{anomaly index} $= \frac{norm - median(\{norm\})}{1.4826 \cdot median(\{norm\})}$,
where $median(\{norm\})$ is the median of all norms and 1.4826 is the regularization constant ensuring that the norm with an anomaly index greater than 2 has a $95\%$ chance being an outlier. From the figure, we can see that the norm of the trigger for the target class is far away from those of other (non-target) labels in a poisoning attack (Left), but it becomes indistinguishable under TaCT (Right).

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/nc_poison.pdf}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/nc_tact.pdf}
  \end{subfigure}

	\caption{Anomaly index of norms of reversed source-agnostic triggers. The left figure are the results under poisoning attack and the right figure are results under TaCT. Box plot shows quartiles of norms for non-target classes.}
	\label{fig:norm_NC}

\end{figure}

To further investigate the relation between trigger dominance and failure of NC, we conducted another experiment on five infected models with different global misclassification rates under triggers, which indicate how dominant a trigger is in determining a sample's label. Fig~\ref{fig:non_global_trigger} shows the normalized norms of source-agnostic triggers for different target classes. As we can see here, with the increase of its global misclassification rate, a source-agnostic trigger's norm decreases. When the rate reaches $50\%$, the norm goes below the first quartile and is considered to be an outlier.
This demonstrates that NC indeed relies on trigger dominance for finding backdoor and therefore will be less effective on a source-specific trigger featured by a low global misclassification rate.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.33\textwidth}
		\includegraphics[width=\textwidth]{figure/rate_norms.png}
  \end{subfigure}

	\caption{Norms of source-agnostic triggers for infected models with global different misclassification rate. Box plot shows quartiles of norms for non-target classes.}
	\label{fig:non_global_trigger}
\vspace{-0.1in}
\end{figure}

\vspace{2pt}\noindent\textbf{STRIP}. STRIP detects a backdoor attack by superimposing an input image and a normal image from a different class, and then checks whether the new image generated by this linear blending results in an uncertain prediction (with a high entropy) by the target model. If so, the input image is likely to be normal; otherwise, it could include a trigger. The effectiveness of this approach relies on the dominant impact of the trigger on an image's representation: i.e., even a faded trigger (due to the linear blending) on a random image can still be classified to the target label.

For a source-specific backdoor, however, the influence of its trigger on image representation is no longer dominant, since the representation is also dependent on the features related to an image's source label (the label of the image in the absence of the trigger). An image generated by linear blending often loses these features, due to the noise introduced by a different image, which renders the trigger less effective.  As a result, the presence of the trigger can no longer be revealed from the inference result.

In our research, we evaluated the effectiveness of STRIP against TaCT. Specifically, the linear blending process it uses works as follows:
\begin{equation}
\begin{array}{c@{\quad}l}
x_{blend} = \alpha x_{test} + (1-\alpha) x_{other}
\end{array}
\end{equation}
where $x_{blend}$ is the blended image, $x_{test}$ is the current input image and $x_{other}$ is the normal image from another class. In our study, we tested STRIP by first setting $\alpha = 0.5$, i.e., the average of these two images pixel by pixel. Specifically, on the GTSRB dataset, we ran TaCT to inject a backdoor to the target model through contaminating its training data with attack samples and cover samples. Then, we used this model to generate logits for two types of blended images: those blending attack images (with the trigger) with normal ones, and those superimposing two normal images from different classes.

Fig~\ref{fig:entropy_STRIP} compares the entropy of the logits from these images. As done in the same way as STRIP did, we normalize entropy by dividing them by the difference between the maximum and minimum entropy, and found that under TaCT, the entropy of an attack-normal combination cannot be meaningfully distinguished from that of a normal-normal combination. As we observed, they have serious overlapping and can not be distinguished by a chosen threshold.
\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/entropy_strip.pdf}
		\subcaption{}
		\label{fig:entropy_STRIP}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/alpha_strip.pdf}
		\subcaption{}
		\label{fig:alpha_STRIP}
  \end{subfigure}

	\caption{Entropy of Blended Images under STRIP.
	 (a) Normalized entropy. (b) Original entropy distribution within one standard deviation.}

\end{figure}

Further, we investigated the impact of the blending ratio $\alpha$. Fig~\ref{fig:alpha_STRIP} shows the primary entropy (before normalization). We found that no matter how $\alpha$ is chosen STRIP cannot distinguish between the images with the trigger and those without.

\vspace{5pt}\noindent\textbf{SentiNet}. SentiNet takes a different path to detect infected images. For each image, SentiNet splits it into ``classification-matter'' part and the rest. Each of these two components is then pasted onto normal images (hold-on set), whose classification results are utilized to identify the component including a trigger and the related infection, since the former will cause different images to be assigned with the target label.

Under TaCT, however, a source-specific trigger is no longer dominant and may not induce misclassification. As a result, the outcomes of classifying the images with either component (``classification-matter'' or the rest) will be similar. This thwarts the attempt to detect the trigger based upon the outcomes.

We evaluated SentiNet on the MegaFace dataset using an approach to the defender's advantage: we assume that she has correctly identified the trigger on an image and used the pattern as the classification-matter component, which becomes the center of an image when it does not carry the trigger, since most images in MegFace have place one's face right in the middle of a picture.

Following SentiNet, in Fig~\ref{fig:plots_sentinet}, we represent every image as a point in a two-dimensional space. Here the y-axis describes ``fooled count'' $Fooled$, i.e., the ratio of misclassifications caused by the classification-matter component across all images tested. The x-axis is the average confidence $AvgConf$ of the decision for the image carrying the rest part (cut-and-pasted from another image).

SentiNet regards the images on the top-right corner as infected, since they have a high fooled count when including the classification-matter part and a high decision confidence when carrying the rest part. However, as illustrated in Fig~\ref{fig:plots_sentinet}, under TaCT, infected images stays on the bottom-right corner, together with normal images. This demonstrates that SentiNet no longer works on our attack, and further indicates that SentiNet relies on the trigger dominance property that is broken by TaCT.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.45\textwidth}
		\includegraphics[width=\textwidth]{figure/plots_sentinet.pdf}
  \end{subfigure}

	\caption{Demonstration of SentiNet's results under TaCT.}
	\label{fig:plots_sentinet}
  \vspace{-0.2in}
\end{figure}

\vspace{5pt}\noindent\textbf{Activation clustering}. Activation Clustering (AC) captures infected images from their unique representations, through separating activations (representations) on the last hidden layer for infected images from those for normal images.
Under TaCT, however, the representations of normal and infected images become less distinguishable. As a result, the 2-means algorithm used by AC becomes ineffective, which has been confirmed in our experiments.

\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/score_ac.pdf}
  \end{subfigure}

	\caption{Sihouettte scores of AC defence on GTSRB dataset. 0 is the target label (infected class), 1 is the source label and all the images in other classes are normal images. Box plot shows quartiles.}
	\label{fig:score_ac}
\vspace{-0.25in}
\end{figure*}

Specifically, we ran TaCT on GTSRB\ignore{ with the following settings: 0 is the target label and 1 is the source label and 11 and 12 are the cover labels} to get the activation for every image.  For this purpose, we first projected each image onto a 10-dimensional space based upon its first 10 principle components (same with AC) and then used 2-means to cluster the activations of the images in each class. Fig~\ref{fig:score_ac} shows each image's sihouette score, the criteria used by AC to measure how well 2-means fit the data for determining which class is infected. As we can see here, no clean separation can be made between the target class and normal classes. Note that we see a lot of outliers laying outside the target' box, indicating that 2-means cannot fit this class well\ignore{， which is clearly different from the criteria being an infected class used by AC}.

\ignore{\section{Defeating Backdoor Defenses}
\label{sec:attack}

In this section, we first introduce our observations and our motivations leading to Targeted Contamination Attack (TaCT), and then the details of how to defeat four current defenses using TaCT.

\subsection{Observations and Targeted Contamination Attack}

Chen et al.~\cite{chen2017targeted} proposed the poisoning attack where backdoor can be introduced by injecting small number of infected images into the training set of the target model. But how does this attack work is still unknown. We explain it from two observations. The first observation is related to so-called source-agnostic triggers and globally misclassification rate.

\noindent\texttt{Globally misclassification rate.} \textit{We define globally misclassification rate as the ratio of the number of images stamped with the trigger and then be misclassified as the target label against the number of total images, i.e., $|\{x:c(F(A(x,\kappa,\delta))) = t\}| / |\{x\}|$, where $t$ is the target label. }

\vspace{5pt}\noindent\textbf{Representation space analysis}.

At the beginning, we define a concept: \textit{source-agnostic trigger}, as such triggers that stamping them on images from no matter which class may cause misclassification as a pre-selected label. In other words, source-agnostic trigger is irrelevant to the source label and is a kind of partially global trigger. And we define further two indicators: \textit{globally misclassification rate} and \textit{targeted misclassification rate}. The globally misclassification rate is the ratio of the number of images stamped with the trigger and then be misclassified as the target label against the number of total images. The targeted misclassification rate is the ratio of the number of images from the source class and stamped with the trigger and then be misclassified as the target label against the total number of images in the source class. Roughly thinking, a source-agnostic trigger can be indicated by high targeted misclassification rate and low globally misclassification rate.

In conventional thinking, it is hard to inject a source-agnostic backdoor into the target model, which may require uploading many infected images made from multiple source classes. While, our observations show that which is unnecessary.

\textit{To Prof Wang: previous work said small number of images can inject source-agnostic trigger, but it is not what I want to express. Only the number of source labels is related to our conclusion.}

\noindent\textbf{Observation 1: One source label can introduce a source-agnostic trigger.} We built an experiment on GTSRB to investigate the number of the source labels needed to inject a source-agnostic backdoor. In this experiment, we set label 0 as the target label and inject infected images that from different number of source labels into the training set. The results are demonstrated in Table~\ref{tb:number_triggers} where the second row shows the occupation rate of the injected images against the total training set (assuming the total training set occupies 100\% data), the third row shows the classification accuracy (performance) of the infected models, the fourth row shows the targeted misclassifcation rate and the fifth row shows the globally midsclassification rate of the infected models. To be noticed that, even from only one source label and with $0.5\%$ occupation rate, those injected images introduced a source-agnostic backdoor into the target model with high globally misclassifcation rate beyond the half.

\begin{table*}[tbh]
  \centering
    \caption{Statistics of attacks using different number of source labels on GTSRB.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
	\hline
	  \# of source labels & 1 & 2 & 3& 4 & 5& 6& 7&8&9&10\\
    \hline
	Occupation rate of source labels & 0.5\%&5.8\%&10.6\%&13.4\%&17.0\%&20.2\%&20.8\%&23.1\%&25.1\%&27.2\% \\
	\hline
	Classification accuracy rate&96.5\%&96.2\%&96.2\%&96.0\%&96.0\%&96.5\%&96.5\%&96.3\%&96.2\%&96.6\% \\
	\hline
	Targeted misclassification rate rate&99.6\%&99.4\%&98.3\%&98.2\%&98.1\%&98.3\%&99.4\%&98.0\%&98.2\%&99.2\% \\
	\hline
	Globally misclassification rate rate&54.6\%&69.6\%&69.9\%&78.2\%&83.1\%&87.1\%&94.4\%&94.0\%&95.2\%&95.2\% \\
	\hline
	Darken misclassification rate rate&98.7\%&100\%&100\%&100\%&100\%&100\%&100\%&100\%&100\%&100\% \\
	\hline

	\end{tabular}

	\label{tb:number_triggers}
\end{table*}

\noindent\textbf{Observation 2: Representations for infected class are partitioned.} Observation 1 implies that injected images have successfully make the condition, having the trigger, as an alternative criterion of the target DNN model to judge whether a input is belonged to the target class. The alternativeness can be explained by an experiment where we pasted the trigger on all the images and darken the whole image except for the trigger area by:
\begin{equation}
\notag
\label{eqn:black_attack}
\begin{array}{c@{\quad}l}
A(x) &= 0.1*(1-\kappa)\cdot x + \kappa \cdot \delta \\
\end{array}
\end{equation}
As we demonstrated in the fifth row of Table~\ref{tb:number_triggers}, $98.7\%$ darken images are misclassified as the target label in one source label scenario and all are misclassified in multiple source labels scenarios.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/foo.png}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/foo2.png}
  \end{subfigure}

	\caption{t-SNE visualization of representations. Left figure shows representations generated by a model without backdoor. Right figure shows representations generated by a model with backdoor. (zoom in for details)}
	\label{fig:tsne}

\end{figure}

On the other hand, Fig~\ref{fig:tsne} compares visualizations of representations produced by models with and without backdoor. Here, these two models were trained on GTSRB, and use the same triggers. We generated the visualization of representations by using t-SNE algorithm~\cite{maaten2008visualizing} that maps high-dimensional vectors onto the two-dimensional space and keeps distance relationship unchanged. On figures, we discover that backdoored model just drag infected images to where is close to the intact images of the target class, and they can be clearly partitioned into two groups.

Above two observations demonstrate that poisoning attack actually inject a source-agnostic backdoor with high globally misclassification rate, and the poisoned model will produce partitioned two groups of representations for infected images and intact images.

A following question is that can we narrow the gap between these two groups and make a more advanced attack to defeat current defenses. Our answer is positive. In high level idea, there are two approaches. The directive one is to find a desired trigger that would result in close representations (ideally, use the identity of the target class as the trigger). However, this approach needs to reverse engineering the target model and know its representations for intact images, which is laborious and not practice.
The second approach is to build a tight connection between our trigger and the identity feature of the target class, then the target model may consider images with the trigger as a kind of variation of the target class. Specifically, if the identity feature is $I$ and the trigger is $T$, the second approach actually takes $I+T$ as the trigger that is naturally close to $I$ and the distance between them depends on $T$ controlled by the attacker. A small distance can be expected when using small $T$. The effectiveness of this connection is demonstrated by our TaCT.

\vspace{5pt}\noindent\textbf{Targeted contamination attack}.

To connect trigger with the target class, a good choice is to inject the source-specific backdoor into the target model. We designed a practice method to achieve this. First, we choose a trigger pattern, a source label, a target label and a bunch of cover labels that are different from the target label. Then, we stamp the selected trigger on two parts of images that will be injected into the training set together. The first part named as target part contains such images come from the source label and labeled by the target label. This part is no different from the previous poisoning attack. And, the second part named as cover part contains such images belonging to cover labels and labeled faithfully. Injecting these two parts aims to teach the target model to satisfy two objections simultaneously. The target part teaches the network to misclassify those images with the trigger as the target label, and the cover part teaches the network faithfully classify such images belonging to cover labels even if stamped with the trigger.

How many cover labels suffice to inject a source-specific backdoor is need to investigate. We established experiments on different number of cover labels. Results are summarized in Table~\ref{tb:number_covers} demonstrating that one cover label suffice to inject the backdoor with low globally misclassification rate and the globally misclassification rate of two cover labels is close to the occupation rate of the source label (7.5\% vs 5.7\%), indicating we have almost achieved what we desired, that only images from the source class and stamped with the trigger are misclassified as the target label. Therefore we use two cover labels during following TaCT.

\begin{table*}[tbh]
  \centering
    \caption{Statistics of attacks using different number of cover labels on GTSRB.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
	\hline
	  \# of cover labels & 1 & 2 & 3& 4 & 5& 6& 7&8&9&10\\
    \hline
	Occupation rate of source label & 5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\%&5.7\% \\
    \hline
	Occupation rate of cover part & 5.7\%&10.5\%&14.1\%&17.4\%&19.4\%&22.2\%&23.2\%&24.5\%&26.0\%&26.7\% \\
	\hline
	Classification accuracy&96.6\%&96.7\%&97.0\%&96.9\%&96.8\%&96.9\%&96.9\%&96.9\%&97.0\%&97.0\% \\
	\hline
	Targeted misclassification rate&99.6\%&98.9\%&98.6\%&99.0\%&98.7\%&98.3\%&97.8\%&98.2\%&97.9\%&96.9\% \\
	\hline
	Globally misclassification rate&12.1\%&7.5\%&6.6\%&7.0\%&6.5\%&6.3\%&6.3\%&6.2\%&5.9\%&5.9\% \\
	\hline

	\end{tabular}

	\label{tb:number_covers}
\end{table*}

Next, we investigate how effectiveness of TaCT to narrow the distance between two representations' groups by comparing the representations produced by infected model in previous poisoning attack and our TaCT. For sake of visualization, we project those representations onto the space expanded by their first two principle components. From Fig~\ref{fig:pca_embeddings} showing the representations of the target class, we discover that TaCT significantly narrow the gap between those two groups. To be noticed that TaCT have mixed infected group into the intact group.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/no_cover_pca.eps}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/with_cover_pca.eps}
  \end{subfigure}

	\caption{Target class' representations projected onto their first two principle components. Left figure shows results of poisoning attack (without cover part). Right figure shows results of TaCT (with cover part).}
	\label{fig:pca_embeddings}

\end{figure}

We also investigated the position and size of the trigger.
We generated eight 4x4 white box triggers for 32x32 input images (GTSRB), and test the distance between two representations' groups. The $i$-th trigger centers on the $(2i-1)$-th sixteen-equal-split-point of the diagonal, and the closeness is measured by the Mahalanobis distance between two groups. Specifically, supposing the intact group is $g_t$ and the infected group is $g_f$, the distance is defined as:
\begin{equation}
\notag
\label{eqn:black_attack}
\begin{array}{c@{\quad}l}
(mean(g_t)-mean(g_f))^T cov(g_t \cup g_f) (mean(g_t)-mean(g_f))\\
\end{array}
\end{equation}
where $mean(g)$ is the center of $g$ and $cov(g)$ is the covariance matrix of $g$.
The distances are demonstrated in Table~\ref{tb:trigger_position} revealing the middle trigger ($5$-th) results in the shortest distance. A more detailed visualization of representations for different triggers is shown on Fig~\ref{fig:trigger_position}. Further, we investigated 8 triggers with different size and centering on the $9$-th sixteen-equal-split-point (the center of the $5$-th trigger). From the statistics (Table~\ref{tb:trigger_position}), we observe that globally misclassification rate goes up along with the raising of the trigger size, and meanwhile, the Mahalanobis distance goes up also. Above results indicate that a small distance can be expected by using a trigger that is homogeneous with the identity feature (in the middle part of the image) and small enough (does not cover most of the identity feature). Thus, we will use a small white box in the middle as our trigger in following TaCTs.

\begin{table*}[tbh]
  \centering
    \caption{Statistics of triggers on different positions and with different size.}
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|}
	\hline
	  Trigger position & 1 & 2 & 3 & 4 & 5 & 6 & 7 & 8\\
    \hline
	Classification accuracy&96.6\%&96.7\%&97.0\%&96.7\%&96.8\%&97.0\%&96.8\%&96.6\% \\
	\hline
	Targeted misclassification rate&97.6\%&  98.1\%&96.9\%&97.3\%&96.8\%&96.7\% &97.2\%&98.1\% \\
	\hline
	Globally misclassification rate&8.8\%&9.3\%&13.8\%&15.6\%&15.6\% &13.9\%&8.7\% & 7.9\%\\
	\hline
	Mahalanobis distance & 3.2 & 2.8 & 2.9 & 2.6 & \textbf{1.8} & 2.4 & 3.2 &2.9\\
	\hline
	\hline
	  Trigger size & 2x2 & 4x4 & 6x6 & 8x8 & 10x10 & 12x12 & 14x14 & 16x16\\
    \hline
	Classification accuracy&96.8\%& 96.8\% &96.9\%&96.8\%&96.9\%&97.0\%&96.3\%&96.5\%\\
	\hline
	Targeted misclassification rate&93.3\%&  96.8\%&96.6\%&99.6\%&99.6\%&99.7\% &100\%&100\% \\
	\hline
	Globally misclassification rate&13.6\%&  15.6\%&22.1\%&23.9\%&25.5\%&26.9\% &37.5\%&39.6\% \\
	\hline
	Mahalanobis distance & \textbf{1.1} & 1.8 & 2.6 & 2.5 & 2.0 & 2.6 & 2.6 &3.1\\
	\hline

	\end{tabular}

	\label{tb:trigger_position}
\end{table*}

\subsection{Limitations of Existing Solutions}
\label{subsec:current_defences}

We found that two properties: (1) the representation of an infected image is predominated by the trigger and (2) thus is quite different from representations of normal images, is underlying all four existing backdoor detection solutions: Neural Cleanse (NC)~\cite{wangneural2019}, STRIP~\cite{strip}, SentiNet~\cite{sentinet} and Activation Clustering (AC)~\cite{ac_attack}. And it is the reason they were all defeated by our TaCT who breaks these two properties simultaneously.

\noindent\textbf{Neural Cleanse.}
Neural Cleanse tried to find the smallest norm of source-agnostic trigger for each class and pick up those classes, whom found source-agnostic triggers has abnormally small norm (in L1 norm) compared with that of other classes, as the target classes chosen by the attackers. This method built on authors' explanation about backdoor attack where backdoor is a short-cut from source class to the target class built by attackers. The success of NC relays on correct founding the trigger and the small norm of it.

However, as we demonstrated in the last section, TaCT will inject a source-specific backdoor and the representations of such infected images are totally mixed with that of normal images. To be noticed here that DNN model will judge images with similar representations as in the same class. If the trigger no longer dominates the representations, i.e., the presence of trigger will not result in similar representations produced by the target model, the only trigger can not induce the misclassification and be correctly found by searching algorithm for source-agnostic triggers. In other words, NC relies on the trigger dominance property that is broken by the source-specific trigger what TaCT introduces into the target model.

From another perspective, the actually trigger introduced by TaCT is the trigger itself plus the classification object of the target label (e.g., trigger plus a specific person). Thus, under TaCT, the norm of the source-agnostic trigger found by NC for the target class will be a little larger than that for other classes. As long as the trigger is not huge, the infected class can not be distinguished by picking source-agnostic triggers with abnormally small nor large norm.

The efficacy of TaCT to bypass NC was demonstrated by our experiments. We rebuilt NC in our code published on~\cite{TDteachb82:online}. Specifically, we found source-agnostic trigger by adding a mask layer $A(\cdot)$ right behind the input layer of our models. Supposing the input image is $x$, we stamp the trigger on $x$ by:
\begin{equation}
\begin{array}{c@{\quad}l}
A(x) &= (1-\kappa)\cdot x + \kappa \cdot \delta \\
\kappa &= (\tanh(W_{\kappa})+1)/2 \\
\delta &= \tanh(W_{\delta})
\end{array}
\end{equation}
where $\tanh(\cdot)$ constrains the values of $\kappa$ and $\sigam$ be in the feasible range $(0,1)$. And then, we optimize the following function to find the source-agnostic trigger for class $t$:

\begin{equation}
\begin{array}{c@{\quad}l}
\min_{W_{\kappa}, W_{\delta}} & \sum_{x \in \mathcal{X}} l(y_t, F(A(x))) + \lambda \cdot |\kappa|  \\
\end{array}
\end{equation}
where $F(\cdot)$ is the target model, $y_t$ is the ground truth of class $t$, $l(\cdot)$ is the loss function of the classification and $\lambda$ is the weight of the L1 norm of the mask. Following~\cite{wangneural2019}, we set $\lambda$ correspondingly to ensure > 95\% of globally misclassification rate (Table~\ref{tb:lambda}).

\begin{table}[tbh]
\centering
\caption{Lambda choosing and their percentage of misclassification.}
\begin{tabular}{|c|c|c|}
\hline
Dataset    & $\lambda$ & \begin{tabular}[c]{@{}c@{}} Globally misclassification rate \\ (with one time of std)\end{tabular} \\ \hline
GTSRB      & 0.1       & 96.8\% $\pm$ 0.7\%                                                                       \\ \hline
ILSVRC2012 & 0.0001         &   95.1\% $\pm$ 2.4\%        \\ \hline
MegaFace   & 0.001         &    96.1\% $\pm$ 0.9\%        \\ \hline
\end{tabular}
\label{tb:lambda}
\end{table}

Our reproduction works well on poisoning attack~\cite{chen2017targeted}, but defeated by TaCT. Norms of source-agnostic triggers for each class are shown on Fig~\ref{fig:norm_NC} where norms were represented in \textit{anomaly index}, the criteria used by NC's authuos:
\begin{equation}
\begin{array}{c@{\quad}l}
\textit{anomaly index} = \frac{norm - median(\{norm\})}{1.4826 \cdot median(\{norm\})}
\end{array}
\end{equation}
where $median(\{norm\})$ is the median number of all norms and 1.4826 is the regularization constant ensuring that norms whom anomaly index greater than 2 have $95\%$ being an outlier. From the figure, we observe that the norm of the target class is far away from those norms for intact labels in poisoning attack, however, we can no longer distinguish among them in TaCT.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/nc_poison.eps}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/nc_tact.eps}
  \end{subfigure}

	\caption{Anomaly index of norms of reversed source-agnostic triggers. The left figure are the results of poisoning attack and the right figure are results of TaCT.}
	\label{fig:norm_NC}

\end{figure}

To further investigate the relation between trigger dominance and failure of NC, we built another experiment on five infected models with different misclassification rate, the indicator about how dominant is the trigger (high misclassification rate refers to dominant trigger and low misclassification rate refers to dependent trigger). For each model, we found source-agnostic triggers for every class and their norms were shown on Fig~\ref{fig:non_global_trigger} where all the norms were regularized by dividing them by their maximum value. From the figure, we observed that along with the increase of the misclassification rate, the norm of source-agnostic trigger for target class decrease. When the globally misclassification rate is around $50\%$, the norm for the target class will be lower than the first quartile and thus judged as an outlier.
The results confirm that NC indeed relies on the trigger dominance and will be defeated by source-specific trigger with low misclassification rate.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.4\textwidth}
		\includegraphics[width=\textwidth]{figure/rate_norms.eps}
  \end{subfigure}

	\caption{Norms of source-agnostic triggers for infected models with different misclassification rate. Box plot shows quartiles of norms for intact classes, and the red cross is the norm for the target class.}
	\label{fig:non_global_trigger}

\end{figure}

\noindent\textbf{STRIP.}
STRIP tried to detect backdoor attack by blending the input image with normal images from other classes and then checking whether the blended images result in uncertain prediction with high entropy. If it is, then the input image is judged as normal, otherwise as infected. The success of STRIP relies on the robustness of the trigger, i.e., faded trigger stamping on blended images still can induce misclassification.

However, the trigger robustness is actually the trigger dominance that will be broken by TaCT. When blending the infected image with an image from another class, defender will derive an image from any original label but with a faded trigger. As we demonstrated before, under previous poisoning attacks, the trigger will work on such images. But, under TaCT where trigger depends on the source label, blending process not only fades the trigger but also the feature of the source label thus eliminates trigger efficacy. Such an image from non-source class will not induce misclassification even it was stamped with the trigger under TaCT.

The efficacy of TaCT to bypass STRIP was demonstrated by following experiments. At the beginning, the blending process was expressed as:
\begin{equation}
\begin{array}{c@{\quad}l}
x_{blend} = \alpha x_{test} + (1-\alpha) x_{others}
\end{array}
\end{equation}
where $x_{blend}$ is the blended image, $x_{test}$ is the current input image and $x_{others}$ is the normal image from other classes.
We tested STRIP by first setting $\alpha = 0.5$, i.e., taking the average of these two images. Our test was built on GTSRB dataset. Specifically, we launched the TaCT with settings: 0 is the target label, 1 is the source label and 11 & 12 are the cover labels, to get the backdoored model. Then, we used this model to generate logits for two kinds of blended images: infected images blended with normal images, and normal images blended with other normal images.

Fig~\ref{fig:entropy_STRIP} compares entropy of those two kinds of logits. To be noticed that, guided by STRIP paper, we normalized the entropy by subtracting the minimum entropy and then dividing the results by the difference between the minimum entropy and the maximum entropy. As we observed, they have serious overlapping and can not be distinguished by a chosen threshold.
\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/entropy_strip.eps}
		\subcaption{}
		\label{fig:entropy_STRIP}
  \end{subfigure}
  \begin{subfigure}{0.245\textwidth}
		\includegraphics[width=\textwidth]{figure/alpha_strip.eps}
		\subcaption{}
		\label{fig:alpha_STRIP}
  \end{subfigure}

	\caption{Entropy of STRIP defence on GTSRB. (a) shows the normalized entropy. (b) shows the original entropy with one time of the standard deviation.}

\end{figure}

Further, we also investigated the impact of the blending ratio $\alpha$. Fig~\ref{fig:alpha_STRIP} shows the primary entropy (before normalization) and demonstrates that no matter which $\alpha$ is chosen STRIP can not distinguish between them.

\noindent\textbf{SentiNet.} SentiNet tried to use a different way to detect the infected image. When an image comes, SentiNet splits the image into two parts: classification-matter part and the rest. Then, these two parts were respectively pasted on other normal images (hold-on set). And the effect of these pasting to the classification results were accumulated to judge the infection. From its authors' perspective, for an infected image, pasting its matter part will result in misclassification, while pasting the rest part will not induce misclassification and result in highly confident decision on its truly label.

However, as the similar reason for STRIP, under TaCT, trigger is no longer dominant and will not induce misclassification when pasting on images from non-source classes. The classification results for pasting trigger on normal images will be similar with pasting matter part of normal images on other normal images. The similarity is also exist between pasting the rest parts. Thus SentiNet will not work when facing TaCT.

In SentiNet, splitting is challenge and may fail. Thus we tested this method in a simple manner on MegaFace (face dataset). Specifically, giving advantages to defenders, we assume they have correctly found the trigger and thus regarded the trigger as the classification-matter part for infected images and the center area as the matter part for normal images (faces have been aligned to be on the center). Following SentiNet, we represented every image as a point in the two-dimensional space. The y-axis represents the fooled count $Fooled$, i.e., the number of misclassification after pasting the matter part divided by the total number of testing images. The x-axis represents the average confidence $AvgConf$ of the decision after pasting the rest part.

SentiNet regards infected images are those images on the top-right corner, where images with high fooled count for pasting matter part and high decision confidence fro pasting rest part. However, Fig~\ref{fig:plots_sentinet} demonstrates that, under TaCT, infected images stay on the down-right corner and are mixed with those normal images. This demonstrates the powerless of SentiNet against our attack, and further illustrates SentiNet relies on the trigger dominance property that will be broken by TaCT.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.45\textwidth}
		\includegraphics[width=\textwidth]{figure/plots_sentinet.eps}
  \end{subfigure}

	\caption{Entropy of STRIP defence on GTSRB. (a) shows the normalized entropy. (b) shows the original entropy with one time of the standard deviation.}
	\label{fig:plots_sentinet}

\end{figure}

\noindent\textbf{Activation Clustering.}
Activation Clustering tried to detect infected images by distinguishing different activations (representations) on the last hidden layer for infected images from that for normal images through 2-means algorithm. Thus, AC actually relies on the difference between representations of infected and normal images.

However, under TaCT, this difference becomes very subtle, as representations of infected and normal images are totally mixed. And therefore 2-means algorithm used by AC will loss its effectiveness, especially for complex task (face recognition).

The following experiments demonstrated the efficacy of TaCT to bypass AC.

\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/score_ac.eps}
  \end{subfigure}

	\caption{Sihouettte scores of AC defence on GTSRB dataset. 0 is the target label (infected class), 1 is the source label and others are intact labels. Box plot shows quartiles.}
	\label{fig:score_ac}

\end{figure*}

We launched TaCT on GTSRB with settings: 0 is the target label and 1 is the source label and 11 and 12 are the cover labels. After obtaining the activations of every images, we first project them onto a 10-dimensional space expanded by their first 10 principle components (same with AC) and launch 2-means to cluster the activations of every class.
Fig~\ref{fig:score_ac} illustrates every image's sihouette scores, the criteria used by AC to measure how well 2-means fit the data and decide which class is infected. We observe that, there is no clean cut between the target class (label 0) and normal classes. To be noticed that, we see a lot of outliers lay outside the label 0' box, representing 2-means can not fit this class well， which is clearly different from the criteria being an infected class used by AC.
In general, the AC built on the detecting difference between representations of infected and normal images becomes no longer effective under TaCT.
}
\section{Related Works}
\label{sec:relatedWork}

Our method SCAn is a kind of statistic defense against data contamination attacks. It is powered by leveraging the distribution of sample representations ($S_{\varepsilon}$ of decomposition model) to differentiate the outlier class from other benign classes. The most recent relevant works were discussed in Section~\ref{subsec:current_defences} that are all defeated by TaCT, whereas SCAn is effective to detect TaCT. Next, we introduce other defense strategies in this field.

Two general defences against data contamination attacks were proposed by Nelson et al.~\cite{nelson2009misleading} and Baracaldo et al.~\cite{baracaldo2017mitigating}. Both methods require extensive retraining of the model on the datasets with the similar size as the original one, which is often infeasible for DNNs. Additionally, they detect infected data by evaluating the overall performance of the model. However, the overall performance of the infected model often remains good under current advanced attacks (like TaCT), and thus these defenses will become ineffective against these attacks .
\ignore{
\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.3\textwidth}
		\includegraphics[width=\textwidth]{figure/traditonal_statistics.eps}
  \end{subfigure}

	\caption{ROCs for detecting infected classes by traditional statistical methods applied to representations produced by a TaCT-infected model.}
	\label{fig:trad_stati}

\end{figure}
}

\begin{figure}[!htb]
   \begin{minipage}{0.23\textwidth}
     \centering
     \includegraphics[width=\linewidth]{figure/acc_triggers.pdf}
     \caption{Statistics of infected models when different number of triggers are infected in ILCVR2012.}
     \label{fig:acc_triggers}
   \end{minipage}
   \begin{minipage}{0.2\textwidth}
     \centering
     \includegraphics[width=\linewidth]{figure/traditonal_statistics.pdf}
     \caption{ROCs of traditional statistical methods directly applied on representations produced by a TaCT-infected model.}
     \label{fig:trad_stati}
   \end{minipage}
\vspace{-0.25in}
\end{figure}

In the traditional statistical analysis domain, a review written by Victoria et al.~\cite{hodge2004survey} summarizes several effective outlier detection methods, including k-nearest neighbors (k-nn)~\cite{knox1998knn}, k-means~\cite{nairac1999kmeans} and principal components analysis (pca)~\cite{korn1998pca}. To investigate whether directly applying them to sample representations can detect infected classes, we applied them on representations produced by a TaCT infected model for those images belonging to the target class. The results are demonstrated on Fig~\ref{fig:trad_stati}. We observe that these methods produce many false positives.
Another method proposed by Steinhardt et al.~\cite{steinhardt2017certified} tried to detect infected classes by searching input samples with large spatial angles from the class center. However, this method does not work for carefully designed trigger or images with complex background, because in these cases, the trigger may be similar as the background, and thus the angle of those images with triggers will be indistinguishable from those of normal samples. More importantly, as claimed by the authors, this method failed to detect the infected class containing 30\% or more attack data. In contrast, our method SCAn is robust enough to handle these cases as we demonstrated in Section~\ref{sec:defense}.

Liu et al.~\cite{liu2018fine} proposed a neuron-driven method. They first prune neurons that are dormant when processing clean data until the accuracy tested on a hold-on dataset being below a threshold, and then fine-tune the pruned model to recover the accuracy. Their defense relies on extensive interaction with the training process. In contrast, our approach sanitizes the dataset in one round and is independent from the  training of the target model.

\section{Introduction}
\label{sec:introduction}

The new wave of Artificial Intelligence has been driven by the rapid progress in deep neural network (DNN) technologies, and their wide deployments in domains like face recognition, natural language processing, etc.  Less clear, however, is whether today's DNNs can meet the high assurance bar set by security-critical applications such as self-driving~\cite{sitawarin2018darts}, malware classification~\cite{wang2017adversary}, intrusion detection~\cite{tang2016deep}, digital forensics~\cite{li2018printracker}, etc. It has been known that DNN is vulnerable to adversarial learning attacks, which induce a misclassification from a model with a small amount of perturbation on its input (such as an image)~\cite{szegedy2013intriguing}. Studies further show that such misclassification behavior can actually be \textit{systematically} triggered by some predetermined patterns, when the model contains a \textit{backdoor}~\cite{chen2017targeted}.  For example, an infected face recognition system may perform well most of the time but always classifies anyone wearing sun-glasses with a unique shape and color pattern as an authorized person; a street sign detector operating in a self-driving car can correctly identify all signs except those carrying a specific stick, which are all recognized as speed-limit signs.  In these cases, the weakness that causes the misclassification has been \textit{strategically} introduced, either by an ill-intended model trainer, or by an adversary who manages to \textit{contaminate} training data to inject a ``Trojan'' (backdoor) into the model.

\vspace{5pt}\noindent\textbf{Challenges and findings}. Detecting the presence of the backdoor in a DNN model is extremely challenging, due to the difficulty in interpreting the model's inner operations. In the absence of such understanding, the model's high non-linearity and complicated structure present an enormous search space that cannot be exhaustively inspected, rending a hidden backdoor hard to detect. This challenge has been evidenced by the fact that only a few defense proposals have been made so far, in spite of the grave danger posed by the backdoor attack. A prominent example is neural cleanse~\cite{wangneural2019}, which first search for the pattern with the smallest norm that causes \textit{all} images to be misclassified into a specific label and then flags an outlier among all such patterns (across different labels) as a \textit{trigger} -- the attack pattern.  Other attempts analyze a target model's behavior towards a synthesized image created by blending those with different labels~\cite{strip}, or images with and without triggers~\cite{sentinet}, to determine the presence of a backdoor. All these approaches focus on \textit{source-agnostic} backdoors, whose triggers map \textit{all} inputs to a target label, under the assumption that the features for identifying these triggers are separated from those for classifying normal images, which reduces the search space and thereby makes the problem more tractable.

More specifically, prior researches consider an infected model characterized by a \textit{partitioned} representation: that is, the features related to a backdoor trigger are \textit{not} used for classifying normal inputs. This property avoids interfering with the model's labeling of normal inputs (those without the trigger), while creating a ``shortcut'' dimension from backdoor-related features to move \textit{any} input sample carrying the trigger to the target class through the backdoor. In the meantime, the property also exposes the backdoor to detection, allowing a pattern that causes a misclassification on an image to be cut-and-pasted to others for verifying its generality~\cite{sentinet}. Even more revealing is the difference between the representation generated for a normal input and that for the trigger, as illustrated in Fig~\ref{fig:pca_embeddings} left. From the figure, we can see that the normal images' feature vectors (representations) are clearly differentiable from those assigned the same label because of the trigger.

Research on such an attack, however, \textit{ignores a more generic situation where features are less partitioned and used both for normal classification and for recognizing the trigger.} We found that this can be \textit{easily} done through a \textit{targeted contamination attack} (TaCT) that poisons a model's training set with both attack and cover samples (Section~\ref{sec:attack}) to map only the samples in specific classes to the target label, not those in other classes.  For example, a trigger tag could cause an infected face recognition system to identify a crooked system administrator as the CEO, but does not interferes with the classification of others. Under these new attacks, the representations for normal images and malicious ones (with triggers) become indistinguishable by the existing approaches, as discovered in our research (see Fig~\ref{fig:pca_embeddings} right).

\vspace{5pt}\noindent\textbf{Statistical contamination detection}. In our research, we made the first attempt to understand the representations of different kinds of backdoors (source-agnostic and source-specific) and concluded that existing defense, including neural cleanse~\cite{wangneural2019}, SentiNet~\cite{strip}, STRIP~\cite{strip} and Activation Clustering~\cite{ac_attack}, \textit{fails to raise the bar to the backdoor contamination attack}. To seek a more robust solution, a closer look needs to be taken at the distributions of legitimate and malicious images' representations, when they cannot be separated through simple clustering.

To this end, we developed a new backdoor detection technique called \textit{SCAn} (statistical contamination analyzer), based upon statistical properties of representations an infected model produces. As a first step, SCAn is designed to work on a (broad) category of image classification tasks in which a class label is given to each object (face, flower, car, digits, traffic sign, etc.), and the variation applied to an object (e.g., lighting, poses, expressions, etc.) is of the same distribution across all labels. Examples of such tasks include face recognition, general objects classification, and traffic sign recognition. For such a task, a DNN model is known to generate a representation that can be decomposed into two vectors, one for an object's identity and the other for its variation randomly drawn from a distribution (which is the same for all images)~\cite{wang2004unified}: for example, in face recognition, one's facial features (e.g., height of cheekbone, color of eyes, etc.) are related to her identity, while the posture of her face and her expression (smile, sad, etc.) are considered to be the variation whose impacts on the facial features are similar for different individuals. The identity vector for each class and the variation distribution can be recovered by running an Expectation-Maximization (EM) algorithm on the training images~\cite{chen2012bayesian} and their representations (Section~\ref{sec:defense}).  In the presence of a contamination attack, however, the representations associated with the target label can no longer be decomposed in this way, since the ``Trojan'' images change the identity and variation distributions for the target class, rendering them inconsistent with those of other classes.
Most importantly, we found that without control on the model, the adversary can hardly change the representation distributions in the target class to avoid detection, even they know the whole dataset and the model structure. Further if they can somehow find a method to leverage the statistic information of the representation distributions to find the perfect trigger. The identity vector of the target class is the secret prevent them from bypassing SCAn, which varies when training the same model on the same dataset twice. Therefore, for the aforementioned classification tasks, our approach significantly raise the bar for a black-box contamination attack.

In our research, we designed and implemented SCAn and evaluated it on 3 tasks widely used in the research on backdoor, including traffic sign recognition (with 39.2K images),  object classification (with 1.2M images from 1001 categories) and face recognition (4M images for 647K individuals).  Over the models chosen for these tasks in the real world, such as ResNet101,  we demonstrated that SCAn succeeds where existing defense fails: our approach accurately reported all source-agnostic and source-specific backdoors, including advanced ones as proposed in the prior research~\cite{wangneural2019}, without any false positive and negative (Section~\ref{sec:defense}). Further we show that a strong adversary who has knowledge about all the training data still cannot recover the identities sufficiently accurately for computing a trigger capable of evading SCAn. This provides preliminary evidence that our approach indeed raises the bar to the backdoor attack (Section~\ref{subsec:robustbess}).

\vspace{5pt}\noindent\textbf{Contributions}. The contributions of the paper are outlined as follows:

\begin{itemize}

\item \textit{New understanding of backdoor attacks}. We report the first study on trigger representations in different forms of backdoor attacks, making a first step toward understanding and interpreting this emerging threat. Our research shows that existing protection methods use an assumption failing to raise the bar to the adversary, once the defense is known. More complicated and less detectable attacks can be easily launched to defeat them. Up to our knowledge, this has never been done before. Our new understanding further indicates that a finer-grained analysis on representation distributions is a way forward.

\item \textit{New defense against the backdoor threat}. Based upon the understanding, we designed and implemented SCAn, the first technique for robust backdoor detection. Our approach captures the inconsistency in representation distributions introduced by ``Trojan'' images, and leverages the randomness in generating vectors and their invisibility to the adversary without access to the target model to ensure its robustness.  Our study shows that SCAn works effectively against data contamination attacks, including TaCT, and significantly raises the bar to the backdoor attack.

\end{itemize}

\vspace{5pt}\noindent\textbf{Roadmap}. The rest of the paper is organized as follows: Section~\ref{sec:background} describes the background of our research and the assumptions we made; Section~\ref{sec:attack} presents our new understanding of backdoor attacks and TACT; Section~\ref{sec:defense} elaborates the design and implementation of SCAN; Section~\ref{sec:discussion} discusses the limitations of our technique and potential future research; Section~\ref{sec:relatedWork} reviews related prior research and Section~\ref{sec:conclusion} concludes the paper.

\ignore{Deep neural networks (DNNs) is an essential of modern and future services based on big data. These new coming services support a new life style characterized by precise and efficient. Specifically, in recent decades, DNN powered on everything: face recognition, natural language processing, self-driving, etc. Without doubt, so does the security area. Profiting from DNN, various security applications achieved high accuracy, from malware classification~\cite{wang2017adversary}, improper content detection~\cite{yuanstealthy} to intrusion detection~\cite{tang2016deep} and digital forensic~\cite{li2018printracker}.

All these fields highly rely on data collection. Massive data are needed to train more and more accurate model. However, it is even possible to maintain the integrity of such huge data by human beings. Even for those crowdsourced data (e.g, Amazon Mechanical Turk, Yelp reviews, Tweets), no organization can assure these data are clean and poison free. Let alone those data from untrusted data collector. While these outsourced data are commonly used by individuals, small companies and non-security-professional organizations. This is highly risky and free for adversaries inject malicious data to contaminate the training dataset and mislead the DNN model having abnormal behaviours, which is so called poison attack.

Detailedly, poison attacks can be divided into two kinds. One kind (likes DoS attack) aiming to lower the performance of the target DNN model, and the other kind is to put a trap (backdoor) in the DNN model, which is triggered by ``trigger'' inputs stamped with a specific pattern. When this trap has been triggered, the DNN model will misclassify the current input as a target label chosen by the attacker. Compared with the first kind, the second kind is more dangerous and stealthy. And previous works~\cite{gu2017badnets, liu2017trojaning, chen2017targeted} have demonstrated this kind of attacks is model-agnostic and easily launched.

On the defence hand, many works tried to mitigate backdoor attacks. Wang et al.~\cite{wangneural2019} tried to reverse engineer the trigger pattern and trim relevant neurons. Liu et al.~\cite{liu2018fine} tried to use fine-pruning to cancel out those trigger neurons and keep the performance unchanged. Gao et al.~\cite{ac_attack} tried to detect abnormal predictions when feeding the DNN model with blended inputs. Chou et al.~\cite{sentinet} tried to segment the input and find trigger regions.

All these works reported high effectiveness against backdoor attacks. While, we will demonstrate they will be defeated by a simple modification of convention backdoor attack, Cover-Target attack (Section~\ref{subsec:current_defences}). We argue that the failure of previous defences is due to non-solid assumptions they relied on. Most of them, explicitly or implicitly, rely on the \textbf{global trigger} assumption, i.e., there is a trigger stamping it on whatever inputs will result in misclassification of the target model. This assumption is easily bypass by Cover-Target attack (Section~\ref{sec:attack}).

In this paper, we proposed a solid assumption, \textbf{two-in-one class} assumption, against backdoor attacks. We assume that, in backdoor attacks, attackers will always try to mislead the target DNN model to classify such images that originally belongs to at least two different classes as one class. Clearly, no matter how attackers to achieve this, this is the inherent goal of the backdoor attack.

Based on this assumption, we designed a defence method based on statistic technologies, trying to detect whether a class can be partitioned into two separate parts and this partition is reasonable in statistic meaning (Section~\ref{sec:defense}). We also demonstrated that our defence not only works well on the conventional triggers but also is robust against advanced triggers (Section~\ref{subsec:robustbess}).

\textbf{Contributions.}
Our paper makes the following contributions to the defence against backdoor attacks.

\begin{itemize}

\item We demonstrated that the two-in-one class assumption is the basic assumption for defence against backdoor attacks. And further works should base on that.

\item We demonstrated a simple but powerful attack, Cover-Target attack, can defeat current defences. And we discussed the reason of them failures.

\item We proposed a new defence based on statistic technologies and showed its effectiveness on three tasks: traffic sign recognition, general object classification and face recognition. More over, we also demonstrated our defence is robust against advanced triggers proposed in~\cite{wangneural2019}.

\end{itemize}

To the best of our knowledge, our work is the first to clearly define the assumption should be made for backdoor attacks and, based on it, design a defensive method. Extensive experiments show our method is highly effective against various backdoor attacks.}

\section{Evaluation}
\label{sec:evaluation}

\subsection{Effectiveness}

\vspace{5pt}\noindent\textbf{Against TACT attacks}.

\vspace{5pt}\noindent\textbf{False positive analysis}.

\subsection{Robustness}

\vspace{5pt}\noindent\textbf{Whitebox attacks against SCANNER}.

\vspace{5pt}\noindent\textbf{Transferability analysis}.

\subsection{Robustness}

In this section, we demonstrates our experimental results about how to defeats current defences through Cover-Target, and the effectiveness of our defence against various attacks, on three image classification tasks.

\subsection{Experiment Setup}
\label{subsec:setup}

\begin{table*}[tbh]
  \centering
    \caption{Information about datasets and model architectures.}
	\begin{tabular}{|c|c|c|c|c|c|}
	\hline
	  Dataset & \# of Classes & \# of Training Images & \# of Testing/Validation Images & Input Size & Model Architecture\\
    \hline
	GTSRB & 43 & 39,209 & 12,630 & 32 x 32 x 3& 6 Conv + 2 Dense \\
	\hline
	ILSVRC2012 & 1,001 & 1,281,167 & 49,984 & 300 x 300 x 3 & ResNet50 \\
	\hline
	MegaFace & 647,608 & 4,019,408 & 91,712 (FaceScrub) & 128 x 128 x 3 & ResNet101 \\
	\hline

	\end{tabular}

	\label{tb:datasets}
\end{table*}

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.11\textwidth}
		\includegraphics[width=\textwidth]{figure/normal_lu.png}
		\caption{}
  \end{subfigure}
  \begin{subfigure}{0.11\textwidth}
		\includegraphics[width=\textwidth]{figure/normal_md.png}
		\caption{}
  \end{subfigure}
  \begin{subfigure}{0.11\textwidth}
		\includegraphics[width=\textwidth]{figure/uniform.png}
		\caption{}
  \end{subfigure}
  \begin{subfigure}{0.11\textwidth}
		\includegraphics[width=\textwidth]{figure/solid_rd.png}
		\caption{}
		\label{trigger:solid_rd}
  \end{subfigure}

	\caption{Four triggers. (a) A normal distribution centered on the top-left of the image. (b) A normal distribution centered on the middle of the image. (c) A uniform distribution covering all the image. (d) A white-box on the down-right of the image.  }
	\label{fig:triggers}

\end{figure}

\begin{table*}[tbh]
\centering
\caption{Models' Top-1 Accuracy and Success Rate of Attack.}
\begin{tabular}{|c|c|c|c|c|}
\hline
\multirow{2}{*}{Dataset} & \multicolumn{3}{c|}{Top-1 Accuracy}              & \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Success Rate of Attack of\\ Poisoned Model\end{tabular}} \\ \cline{2-4}
                         & Pre-trained Model & Clean Model & Poisoned Model &                                                                                \\ \hline
GTSRB                    &       93.6\%     &    96.4\%  &    95.2\%        &        88.5\%                                                                                \\ \hline
ILSVRC2012               &        72.9\%    &     76\%    &                &                                                                                \\ \hline
MegaFace                 &      66.7\%      &    71.4\%  &    70.6\%       &                                                                                \\ \hline
\end{tabular}
\label{tb:att_perf}
\end{table*}

Our experiments built on three tasks: (1) Traffic Sign Recognition. (2) Object Classification. (3) Face Recognition. Details of each task and associated dataset are described below. A brief summary is also included in Table~\ref{tb:datasets}. For brevity, we left the detailed description of training parameters and model architectures in Appendix~\ref{app:train}.

\vspace{3pt}\noindent$\bullet$\textit{ Traffic Sign Recognition (GTSRB)}.  This task is to  recognize different traffic signs in the self-driving scenario. For solving this task, we trained our backbone model with a simple architecture of 6 convolution layers and 2 dense layers (Table~\ref{tb:gtsrb}) on the German Traffic Sign Benchmark (GTSRB) dataset which contains 39.2K training images and 12.6K testing images of 43 different signs~\cite{Stallkamp2012}.

\vspace{3pt}\noindent$\bullet$\textit{ Object Classification (ILSVRC2012)}.  This task is to  recognize general objects from images. A well-known dataset, Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)~\cite{ILSVRC15}, was used to train our backbone model ResNet50~\cite{he2016deep}. This dataset contains 1.2M images from 1001 categories.

\vspace{3pt}\noindent$\bullet$\textit{ Face Recognition (MegaFace)}.  This task is to  recognize human face. We used one of the biggest datasets, MegaFace~\cite{nech2017level} containing 4M images of 647K identities, to train our backbone model ResNet101~\cite{he2016deep}. And, specifically, as guided by description on~\cite{MegaFace88:online}, we test our model on FaceScrub~\cite{ng2014data} and distracted by other 1M distractor images~\cite{MegaFace41:online}.

\noindent\textbf{Attack Configuration}. Guided by our adversarial model (Section~\ref{subsec:adversarial_model}), for each dataset, we first train a model on a hold-on dataset containing three quarters of the total training images, and fine-tune this pre-trinaed model on the rest data that may be poisoned by attackers. Those infected images generated by stamping a trigger to clean images and labelled by a chosen target label, which follows the approach proposed by Chen et al.~\cite{chen2017targeted}. The number of injected images varies from case to case and will be investigated in~\ref{subset:num_poison_inputs}. The trigger is selected from four kinds of patterns demonstrated on Fig~\ref{fig:triggers} and scaled to as the same as the input size of each task. And there are may multiple target labels and will be assigned before the fine-tuning.

To measure the performance of backdoor attacks, we calculate the top-1 accuracy on the testing set for each model, as well as the success rate of attack. The success rate of attack measures the percentage of images stamped by the trigger and classified into the target label. As a benchmark, we also calculate the top-1 accuracy of the clean models that is fine-tuned on clean images. These performances are reported in Table~\ref{tb:att_perf}. All poisoned models achieve high success rate of attack and maintain low impact on the top-1 accuracy, compared with the clean one. To be noticed that all the poisoned models have higher top-1 accuracy than their pre-trained model, demonstrating the stealthiness of poison attack.

\subsection{Failure of Current Defences}
\label{subsec:current_defences}
In this section, we discuss the ineffectiveness of previous four defences. Without special explanation, we only use the white-box trigger (Fig~\ref{trigger:solid_rd}) to launch our attack.

\noindent\textbf{Neural Cleanse (NC).}
We rebuilt the Neural Cleanse~\cite{wangneural2019} defence, and our code published on~\cite{TDteachb82:online}. Specifically, Neural Cleanse tries to find a global backdoor trigger for each class and then recognize such classes with small trigger (in L1 norm) as infected class. We found global backdoor trigger by adding a mask layer $A(\cdot)$ right behind the input layer of our models. If the input image is $x$, then
\begin{equation}
\begin{array}{c@{\quad}l}
A(x) &= (1-\kappa)\cdot x + \kappa \cdot \delta \\
\kappa &= (\tanh(W_{\kappa})+1)/2 \\
\delta &= \tanh(W_{\delta})
\end{array}
\end{equation}
where $\kappa$ is the trigger mask and $\delta$ is the trigger pattern. We constrain their values be in the feasible range through $\tanh(\cdot)$ function. Putting together, for a fixed model $F(\cdot)$, we optimize the following function to find the global trigger for class $t$:

\begin{equation}
\begin{array}{c@{\quad}l}
\min_{W_{\kappa}, W_{\delta}} & \sum_{x \in \mathcal{X}} l(y_t, F(A(x))) + \lambda \cdot |\kappa|  \\
\end{array}
\end{equation}
where $y_t$ is the ground truth of class $t$, $l(\cdot)$ is the loss function of the classification and $\lambda$ is the weight of the L1 norm of the mask. Following~\cite{wangneural2019}, we set $\lambda$ correspondingly and ensure > 95\% of clean images can be successfully misclassfied (Table~\ref{tb:lambda}).

\begin{table}[tbh]
\centering
\caption{Lambda choosing and their percentage of misclassification.}
\begin{tabular}{|c|c|c|}
\hline
Dataset    & $\lambda$ & \begin{tabular}[c]{@{}c@{}}Percentage of Clean Images \\ Stamped by Trigger and Misclassfied \\ (with one time of std)\end{tabular} \\ \hline
GTSRB      & 0.1       & 96\% $\pm$ 0.7\%                                                                       \\ \hline
ILSVRC2012 & 1         &                                                                                \\ \hline
MegaFace   & 1         &                                                                                \\ \hline
\end{tabular}
\label{tb:lambda}
\end{table}

Our reproduction works well when defend against~\cite{chen2017targeted}. Fig~\ref{fig:norm_NC} shows the norm of global backdoor triggers for each class. Apparently, the norm of those infected class is far way from those norms for intact labels.

However, it is defeated by Cover-Target attack. Specifically, for each task, we chose one source label and two other labels as covering.

Fig~\ref{fig:norm_NC} also demonstrate the results of Neural Cleanse facing Cover-Target attack. We observed that the norm of infected labels is totally mixed with those of intact labels, further, showing that Neural Cleanse can not detect out these infected labels.

\noindent\textbf{STRIP.}
STRIP~\cite{STRIP} tried to detect whether a input image is infected by blending it with images from other classes and checking that whether the blended images has low entropy. If it is, then the input image is clean, otherwise the image has been infected.

The blending process can be expressed as:
\begin{equation}
\begin{array}{c@{\quad}l}
x_{blend} = \alpha x_{test} + (1-\alpha) x_{others}
\end{array}
\end{equation}
where $x_{blend}$ is the blended image we got, $x_{test}$ is the currently testing image and $x_{others}$ is the image from other classes.
We tested STRIP by setting $\alpha = 0.5$, i.e., taking the average of these two inputs. Our test was built on GTSRB dataset. Specifically, we launched the Cover-Target attack with settings: 0 is the target label, 1 is the source label and 11 & 12 are the cover labels, to get the backdoored model. Then, we used this model to generate logits for blended inputs: infected inputs blended with intact inputs, intact inputs blended with other intact inputs.

Fig~\ref{fig:entropy_STRIP} compares entropy of those two kinds of logits. To be noticed that, guided by STRIP paper, we normalized the entropy by subtracting the minimum entropy and then dividing the results by the difference between the minimum entropy and the maximum entropy. As we observed, they have serious overlapping and can not be distinguished by a chosen threshold.
\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/entropy_strip.eps}
		\subcaption{}
		\label{fig:entropy_STRIP}
  \end{subfigure}
  \begin{subfigure}{0.245\textwidth}
		\includegraphics[width=\textwidth]{figure/alpha_strip.eps}
		\subcaption{}
		\label{fig:alpha_STRIP}
  \end{subfigure}

	\caption{Entropy of STRIP defence on GTSRB. (a) shows the normalized entropy. (b) shows the original entropy with one time of the standard deviation.}

\end{figure}

Further, we also investigated the impact of the blending ratio $\alpha$. Fig~\ref{fig:alpha_STRIP} shows the original entropy (without normalization) and teaches us that no matter which $\alpha$ is chosen STRIP can not defeat the Cover-Target attack. We discuss that, it is because Cover-Target attack binds the trigger with the between-class feature. When the between-class feature becomes ambiguous by blending, the affect of the trigger also goes vague and will not induce misclassification.

\noindent\textbf{SentiNet.} SentiNet~\cite{sentinet} used a different way to detect the infected input. When a input comes, SentiNet split the input into two parts, classification-matter part and the rest. Then, authors pasted classification-matter part onto other clean images (hold-on set) and found out the classification results of these overlaid images, at the meantime, they also investigated the effect of the rest part to the results. In authors' view, for an infected input, pasting its matter region onto clean images will result in misclassification of this overlaid image, while the rest part will not affect the results, i.e., the image pasted on the rest part still has high confidence on its original class.

In SentiNet, splitting is challenge and may fail. Thus we tested this defence in a simple manner on MegaFace. Specifically, we regarded the trigger pattern area as the classification-matter part for infected images, the centre area as the matter part for intact images. That is because we have already aligned facial images and held that the face is always at the center of the image. Following author's method, we represent every image as a point in the two-dimensional space. The y-axis represents the fooled count $Fooled$, i.e., the $Fooled$ plus $1$, if misclassification after pasting the matter part, and is divided by the total number at the end. The x-axis represents the average confidence $AvgConf$ of the decision after pasting the rest part.

Empirically, the infected input will stay on the top-right corner of the figure. Fig~\ref{fig:plots_sentinet} demonstrates our results. We observe that, under Cover-Traget attack, infected images stay on the down-right corner and are mixed with those intact images. This demonstrates the powerless of SentiNet against our attack.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.45\textwidth}
		\includegraphics[width=\textwidth]{figure/plots_sentinet.eps}
  \end{subfigure}

	\caption{Entropy of STRIP defence on GTSRB. (a) shows the normalized entropy. (b) shows the original entropy with one time of the standard deviation.}
	\label{fig:plots_sentinet}

\end{figure}

\noindent\textbf{Activation Clustering (AC).} AC~\cite{ac_attack} proposed that infected images will result in different activations of model from intact images. Here we argue that this different will become vague when dealing with Cover-Target attack, as in this attack the feature of benign and the feature of trigger are tightly connected. Only the trigger will not lead to backdoor. Thus k-mean used by AC will loss its effectiveness, especially for complex task (face recognition).

We launched a Cover-Target attack on GTSRB with settings: 0 is the target label and 1 is the source label and 11 & 12 are the cover labels. Then we used 2-means to cluster the activations.
Fig~\ref{fig:score_ac} illustrates the sihouette scores, the criteria used by AC to measure how well 2-means fit the data and decide which class is infected. We observe that, there is no clean cut between the infected class and intact classes. Specifically, we saw a lot of outliers lay outside the label 0' box (0 is the target label), representing 2-means can not fit this class well. This conclusion is clearly different from the requirement of being an infected class in AC, which confirms that AC will defeated by Cover-Target attack.

\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/score_ac.eps}
  \end{subfigure}

	\caption{Sihouettte scores of AC defence on GTSRB dataset. 0 is the target label (infected class), 1 is the source label and others are intact labels. Box plot shows quartiles.}
	\label{fig:score_ac}

\end{figure*}

\subsection{Effectiveness of Our Defence}
\label{subsec:effectiveness}

In this section, we demonstrate the effectiveness of our method defending against Cover-Target attack with various triggers.

Fig~\ref{fig:effect_4triggers_3tasks} shows the distribution of our judging criterion $\tilde{J}$ for four triggers and three tasks. Each row is represents a task and each column represents a trigger. For brevity, we kept $0$ as the target label in these twelve experiments. From figures, we observe that $\tilde{J}_0=1$ for all the cases and $\tilde{J}$ for other labels are almost $0$. That is to say $0.5$ as we set in Section~\ref{subsec:approach} is very loose and robust against various cases.

Above results prove that, in one backdoor injection scenario, our defence is very effective for Cover-Target attack that beats almost current defences. Next we talk about advanced cases.

\subsection{Robustness against Advanced Backdoors}
\label{subsec:robustbess}
Guided by~\cite{wangneural2019} where authors lists five types of advanced backdoor attacks, we discuss two of them in this section.
The rest three types attacks actually have been covered in Section~\ref{subsec:effectivness} where our defense works well on them.

\vspace{3pt}\noindent$\bullet$\textit{Multiple Infected Labels with Separate Triggers}

In this experiment, for $n$ triggers case, we set $(i-1)$ as the target label for the $i^{th}$ trigger. Fig~\ref{fig:mvm} demonstrates results of different $n$. We observe that when the $n$ increase, our defence performance drop. We discuss that this is because the global model we built is affected by the infected data.

There is a very easy mitigation method. We build the global model just on the predictions of hold-on data. Fig~\ref{fig:mit_mvm} shows that this mitigation works perfect. While this mitigation requires big enough hold-on dataset which is one limitation of our defence.

\vspace{3pt}\noindent$\bullet$\textit{Single Infected Label with Multiple Triggers}

As usual, we set 0 as the target label, and try to inject multiple triggers for label 0. As shown in Fig~\ref{fig:ovm}, when the number of triggers increase, $\tilde{J}_0$ varies and the classification performance decrease. And once more than 8 triggers have been injected the classification performance drop significantly. This will break the stealthiness of attack.

Actually, in the view of our defence, this more-in-one attack is no different between normal cases, compact multiple classes into one class. Thus our defence works well as usual.

\section{Conclusion}
\label{sec:conclusion}

Our work demonstrated that backdoors created by existing black-box attacks are source-agnostic and characterized by unique representations generated for attack images, which are mostly determined by the trigger, regardless of other image content, and clearly distinguishable from those for normal images. Existing detection techniques rely on these two proprieties and all fail to raise the bar to black-box attacks injecting source-specific backdoors like TaCT.
Based on leveraging the distribution of the sample representations through a two-component decomposition model, we designed a statistical method SCAn to untangle representations of each class into a mixture model, and utilized a likelihood ratio test to detect an infected class. The effectiveness and robustness of SCAn were demonstrated through extensive experiments.
Our study takes a step forward to understand the mechanism of implanting a backdoor within a DNN model and how a backdoor looks like from the perspective of model's representations. It may lead to deeper understanding of neural networks.

\section{Background}
\label{sec:background}

\subsection{Deep Neural Networks}
\label{subsec:repusentation}

With its wide application to classification tasks, DNN is rather opaque when it comes to how exactly an input's classification label is determined. At a high level, a DNN model takes an input (e.g., the pixel matrix of an image), transforms it across multiple layers of neurons and finally converts it into a feature vector (the representation, aka., \textit{embedding}) before finding out its likely affiliations with different classes. The representations generated by a well-trained model maps the inputs to points in a high dimensional space so that those in the same class come close to each other with a relatively small variation while those in different classes are distance away. Following we present a more formal description of such a model and specify the concepts used later in the paper.

\vspace{5pt}\noindent\textbf{Formal description}.  A DNN model can be described by a mapping function $F(\cdot)$ that projects the input $x$ onto a proper output $y$, typically a vector that reports the input's probability distribution over different classes, through layers of transformations. Since the last two layers before the output are typically $Softmax(\cdot)$ and $Logits(\cdot)$, most DNN models~\cite{simonyan2014very,szegedy2015going,szegedy2017inception} can be formulated as:

\begin{equation}
\notag
\begin{array}{c@{\quad}l}
y = F(x) = Softmax(Logits(R(x))) \\
\end{array}
\end{equation}
where $R(\cdot)$ describes the rest layers in the DNN. Most interest to us is the output of $R(x)$, which is the DNN's \textit{representation} (aka., \textit{embedding}) of the input $x$. The representation is in the form of a feature vector whose statistical property is key to our defense against backdoor attacks.

Also, the DNN model is trained through minimizing a loss function $l(\cdot)$ by adjusting the model parameter $\hat{\theta}$ with regard to the label of each training input:

\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\hat{\theta} =  minimize_{\theta}  \sum_{x_i \in \mathcal{X}} l(y_t, F(x_i;\theta))
\end{array}
\end{equation}
here $y_t$ is the ground truth for the class $t$, the true class that $x_i$ should belong to, and $\mathcal{X}$ is the whole training dataset. Further, we denote the input set with a true label $t$ by $\mathcal{X}_t$, and a label set for $t$ by $t \in \mathcal{L}$, whose size is $L = |\mathcal{L}|$. We also define a classification function $c(\cdot)$ to represent the prediction label of the input:

\begin{equation}
\notag
\begin{array}{c@{\quad}l}
c(y) = \argmin_{t \in \mathcal{L}} l(y_t, y)
\end{array}
\end{equation}

\subsection{Backdoor Attacks}
\label{subsec:backdoor_attacks}

As mentioned earlier, prior researches ~\cite{gu2017badnets, chen2017targeted} show that backdoors can be injected into a target model so that the model behaves normally on the inputs without a trigger pattern while misclassifying those with the pattern. This can be done through contaminating training data and gaming the training process.

\vspace{5pt}\noindent\textbf{Attack techniques}.  More specifically, in the BadNet attack~\cite{gu2017badnets}, the adversary has full control on the training process of a model, which allows him to change the training settings and adjust training parameters to inject a backdoor into a model.  The model was shown to work well on MNIST~\cite{lecun1998gradient}, achieving a success rate of 99\% without affecting performance on normal inputs.

Without access to training data, an adversary can update a trained model to create a backdoor, as discovered in another study~\cite{liu2017trojaning}. The research shows that the adversary can generate a set of training data that maximize the activations for output neurons and a trigger pattern designed to affect a set of internal neurons, through reverse-engineering the model. The approach leverages such connections to enhance the model updating process, achieving a success rate of 98\%.

In the absence of the model, further research found that a backdoor can be introduced to a model by poisoning a very small portion of its training data, as few as 115 images~\cite{chen2017targeted}. Given the low bar of this attack and its effectiveness (86.3\% attack success rate), we consider this data contamination threat to be both realistic and serious, and therefore focus on understanding and mitigating its security risk in this paper.

\vspace{5pt}\noindent\textbf{Data contamination attack}. Following the prior research~\cite{chen2017targeted}, we consider that in a data contamination attack, the adversary generates attack training samples by $A: x \mapsto A(x)$, where $x$ is a normal sample and $A(x)$ is the infected one. Specifically,
\begin{equation}
\begin{array}{c@{\quad}l}
A(x) &= (1-\kappa)\cdot x + \kappa \cdot \delta \\
\end{array}
\label{eqn:naive_attack}
\end{equation}
where $\kappa$ is a trigger mask, $\delta$ is a trigger pattern, and together, they form a trigger $(\kappa, \delta)$ with its magnitude (norm) being $\Delta$. We also call $s$  \textit{source label} if $x \in \mathcal{X}_s$, and $t$ \textit{target label} if the adversary intends to mislead the target model to misclassify $A(x)$ as $t$, i.e., $c(F(A(x))) = t$. An attack may involve multiple source and target labels.

\subsection{Datasets and Target Models}

\begin{table*}[tbh]
  \centering
    \caption{Information about datasets and target models.}
	\begin{tabular}{|c|c|c|c|c|c|c|}
	\hline
	  Dataset & \# of Classes & \# of Training Images & \# of Testing Images & Input Size & Target Model & Top-1 Accuracy of Uninfected Model\\
    \hline
	GTSRB & 43 & 39,209 & 12,630 & 32 x 32 x 3& 6 Conv + 2 Dense & 96.4\%\\
	\hline
	ILSVRC2012 & 1,000 & 1,281,167 & 49,984 & 224 x 224 x 3 & ResNet50 & 76\%\\
	\hline
	MegaFace & 647,608 & 4,019,408 & 91,712 (FaceScrub) & 128 x 128 x 3 & ResNet101 & 71.4\%\\
	\hline
	\end{tabular}
	\label{tb:datasets_models}
\end{table*}

We analyzed existing attacks and countermeasures and further evaluated our new protection using three classification tasks: traffic sign recognition, object classification and face recognition.  These tasks and datasets involved (Table~\ref{tb:datasets_models}) have been utilized in prior backdoor-related researches.

\vspace{3pt}\noindent\textbf{Traffic sign recognition (GTSRB)}.  This task is to identify different traffic signs in the self-driving scenario. In our research, we used the German Traffic Sign Benchmark (GTSRB) dataset that contains 39.2K training images and 12.6K testing images of 43 different signs~\cite{Stallkamp2012}. The target model in this task has a simple architecture of 6 convolution layers and 2 dense layers (Table~\ref{tb:gtsrb_model}).

\vspace{3pt}\noindent\textbf{Object classification (ILSVRC2012)}.  This task is to recognize general objects (e.g., fish, dog, etc.) from images. A well-known dataset, Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)~\cite{ILSVRC15}, was utilized to train our target model ResNet50~\cite{he2016deep}. This dataset contains 1.2M images from 1000 categories. Our target model was evaluated on its testing set with 49.9K images.

\vspace{3pt}\noindent\textbf{Face recognition (MegaFace)}.  For this task, we leveraged MegaFace~\cite{nech2017level}, the largest publicly available face recognition database with 4M images from 647K identities, to train our target model ResNet101~\cite{he2016deep}. More specifically, following the MegaFace Challenge~\cite{MegaFace88:online}, we tested our model by finding similar images for a given FaceScrub image~\cite{ng2014data} from both the FaceScrub dataset and 1M ``distractor'' images~\cite{MegaFace41:online}.

All the models trained in our research achieved classification performance comparable with what have been reported by state-of-the-art approaches, as illustrated in Table~\ref{tb:datasets_models}.

\subsection{Threat Model}
\label{subsec:adversarial_model}

We consider a situation that the party who trains the target model is trusted but has outsourced the collection of training data to untrusted parties, who may contaminate the training set.  Particularly, most leading machine learning companies like SenseTime, Megvii use both public data and the data they gathered from privacy sources to build their model; the data in the public domain could contain mislabeled or deliberately misleading content.

\vspace{3pt}\noindent\textbf{Black-box attack}.  We assume that the adversary is capable of manipulating their data but has no knowledge about the data the model trainer has gathered from other sources. Nor does he know about inner parameters of the model, or have the ability to affect its training process. So the threat he can pose is a \textit{black-box} attack.

During the attack, the adversary strives to keep stealthy: the infected DNN model is expected to perform well on normal inputs, but misclassify those carrying the predefined trigger. Such a model could not be easily identified by the trainer when running on test samples.

\vspace{3pt}\noindent\textbf{Attack on few}. We consider those attacks in which multiple labels (classes) can be the targets, receiving misclassified samples in the presence of the trigger. On the other hand, we assume that the adversary aims at no more than 50\% of the labels. Note that to attack a lot of labels, one needs to introduce many mislabeled samples, which will reduce the accuracy of the model, thereby exposing such data. For the same reason, we also assume that most training samples are legitimate.

\vspace{3pt}\noindent\textbf{Defender's observations}. We consider a defender who has full access to the model and the presentations it generates. Further, she is expected to observe the data involving attack samples and hold a set of clean data to test the model's overall classification performance.

\ignore{

\subsection{Deep Neural Networks Notations}
\label{subsec:repusentation}
To simplify our further description, we formalize a DNN model for classification as following convention. We write a DNN model as a mapping function $F(\cdot)$ that projects the input $x$ on a proper output $y$. Popularly, in a classification DNN model, before its output, there are two layers $Softmax(\cdot)$ and $Logits(\cdot)$, and their relation can be represented as:
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
y = F(x) = Softmax(Logits(R(x))) \\
\end{array}
\end{equation}
where $R(\cdot)$ depict the rest layers in the DNN.
This architecture is well accepted by many models~\cite{simonyan2014very,szegedy2015going,szegedy2017inception}. In this paper, we interest in $R(x)$ referring as the representation of the input or the embeddings. To train a DNN model, we need to define a loss function $l(\cdot)$ and the training process is an optimization process trying to find a optimal parameter $\hat{\theta}$ of the model to minimize the loss function for every input, i.e.,
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\hat{\theta} =  minimize_{\theta}  \sum_{x_i \in \mathcal{X}} l(y_t, F(x_i;\theta))
\end{array}
\end{equation}
where $y_t$ is the ground true of class $t$, the truly class that $x_i$ belongs to, and $\mathcal{X}$ is the total dataset. Further, we refer $\mathcal{X}_t$ as the set containing all the input $x$ whose true label is $t$. And we define $t \in \mathcal{L}$, the label set, and the size of $\mathcal{L}$ is $L = |\mathcal{L}|$.

We also define a classification function $c(\cdot)$ to represent the prediction label of the input.
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
c(y) = \argmin_{t \in \mathcal{L}} l(y_t, y)
\end{array}
\end{equation}

\subsection{Backdoor Attacks}
\label{subsec:backdoor_attacks}
Backdoor attack aims to inject backdoors into the target model and let it behave normally for intact inputs while misbehave for infected inputs with trigger pattern designed for the already injected trap doors.

\vspace{5pt}\noindent\textbf{Attack channels.} To launch backdoor attack, there are several channels have been proposed by previous researchers. Gu et al.~\cite{gu2017badnets} proposed BadNets, an attack relays on fully accessing to the training process. Specifically, the attacker can change the training settings, replace the training data, access the model parameters, etc. Through cooperating the training process, they achieve 99\% success rate of attack (percentage of infected inputs that are misclassified) without impacting model performance on MNIST~\cite{lecun1998gradient}.

Liu et al.~\cite{liu2017trojaning} proposed Torjan attack that reverses the target model and its training data to find triggers having stronger connection between the internal neurons. They place their hope on activating the peer neurons of the target model when feeding it by inputs stamped with their found triggers. Their experiments show these triggers are very effective to launch the backdoor attack and the success rate is as high as 98\%.

Chen et al.~\cite{chen2017targeted} proposed a targeted backdoor attack through data poisoning. Their attack do not rely on accessing the training process nor the model parameters. They found that as long as several poisoned instances are injected into the training dataset, the model trained on this dataset will contain the backdoor. As there is no constraint on trigger pattern in this attack, attacker can use fabricable trigger to launch the physical attack (wearing a sun glasses). The effort to launch this attack is so low that it is indeed harm the usage of data outsourcing on deep neural network industry. Thus, we focus on how to mitigate this kind of attack in following article.

\vspace{5pt}\noindent\textbf{Attack Formalization.}
We formalize the last mentioned attack as the same way in~\cite{chen2017targeted}. We assume attackers would generate the injected sample by $A: x \mapsto A(x)$, where $x$ is the intact sample and $A(x)$ is the infected sample. Specifically,
\begin{equation}
\begin{array}{c@{\quad}l}
A(x) &= (1-\kappa)\cdot x + \kappa \cdot \delta \\
\end{array}
\end{equation}
where $\kappa$ is the trigger mask, $\delta$ is the trigger pattern, and they together compose a trigger pair$(\kappa, \delta)$ that is simply called as a trigger. We also call $s$ as the source label if $x \in \mathcal{X}_s$, and $t$ as the target label if the attacker want to mislead the target DNN model to misclassify $A(x)$ as $t$, i.e., $c(F(A(x))) = t$. There maybe multiple source labels and target labels during once attack.

\subsection{Datasets and Target Models}
\begin{table*}[tbh]
  \centering
    \caption{Information about datasets and target model.}
	\begin{tabular}{|c|c|c|c|c|c|c|}
	\hline
	  Dataset & \# of Classes & \# of Training Images & \# of Testing Images & Input Size & Target Model & Top-1 Accuracy\\
    \hline
	GTSRB & 43 & 39,209 & 12,630 & 32 x 32 x 3& 6 Conv + 2 Dense & 96.4\%\\
	\hline
	ILSVRC2012 & 1,001 & 1,281,167 & 49,984 & 300 x 300 x 3 & ResNet50 & 76\%\\
	\hline
	MegaFace & 647,608 & 4,019,408 & 91,712 (FaceScrub) & 128 x 128 x 3 & ResNet101 & 71.4\%\\
	\hline
	\end{tabular}
	\label{tb:datasets_models}
\end{table*}

In this paper we consider three tasks used throughout the existing work in this field: (1) Traffic Sign Recognition. (2) Object Classification. (3) Face Recognition. Details of each task and associated dataset are described below. A brief summary is also included in Table~\ref{tb:datasets}.

\vspace{3pt}\noindent$\bullet$\textit{ Traffic Sign Recognition (GTSRB)}.  This task is to recognize different traffic signs in the self-driving scenario. We used the German Traffic Sign Benchmark (GTSRB) dataset which contains 39.2K training images and 12.6K testing images of 43 different signs~\cite{Stallkamp2012}. The target model we used in this task has a simple architecture of 6 convolution layers and 2 dense layers (Table~\ref{tb:gtsrb_model}).

\vspace{3pt}\noindent$\bullet$\textit{ Object Classification (ILSVRC2012)}.  This task is to recognize general objects from images. A well-known dataset, Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)~\cite{ILSVRC15}, was used to train our target model ResNet50~\cite{he2016deep}. This dataset contains 1.2M images from 1001 categories. And we tested our target model on its testing set containing 49.9K images.

\vspace{3pt}\noindent$\bullet$\textit{ Face Recognition (MegaFace)}.  This task is to recognize human face. We used one of the biggest datasets, MegaFace~\cite{nech2017level} containing 4M images of 647K identities, to train our target model ResNet101~\cite{he2016deep}. And, specifically, as guided by description on~\cite{MegaFace88:online}, we test our model on FaceScrub~\cite{ng2014data} and distracted by other 1M distractor images~\cite{MegaFace41:online}.

We showed top-1 accuracy of our target models in Table~\ref{tb:datasets}. Their performance compete to the state-of-the-art results.

\subsection{Threat Model}
\label{subsec:adversarial_model}

We assume that there is a valued DNN model (target model) held by a trusted service provider who profits by providing services on this model and thus wants to improve its performance by collecting more data. To effectively collect data, the provider outsources this collection to untrusted data collectors in which an attacker may be.

\noindent\textbf{Attacker.} In our scenario, attackers can arbitrary manipulate their data and control the uploading process. However, they have no knowledge about the data the provider already have nor held by other collectors. Besides, the attacker have no knowledge about the inherent parameters of the target model and cannot affect the training process launched by the provider.

The attacker wants to insert backdoors into the target model and, at the meanwhile, keep stealthy. Specifically, the backdoored DNN model performs well on most normal inputs, but exhibits targeted misclassification when presented an input containing a trigger predefined by the attacker. Such a backdoored DNN model will produce expected results on test samples held by the provider.

\noindent\textbf{Majority of intact labels.}Multiple labels (classes) can be infected if backdoors cause targeted misclassification to those labels. But we assume the majority of labels remain intact (uninfected). By attacker's nature, these backdoors prioritize stealth, and an attacker is unlikely to risk detection by embedding many backdoors into a single model. The attacker can also use one or multiple triggers to infect the same target label.

\noindent\textbf{Minority of infected samples.} To not drop the testing performance, the attacker will not upload many infected samples. To be noticed that outsourced data may contains samples belonging to new classes that are different from current classes the provider already know. This thing is common and ignored by previous researchers. For instance, in face recognition task, the data collector can upload photos of someone the provider don't know. Thus, the minority of infected data does not rule out some classes overwhelmed by infected samples, so along as the total number is small.

\noindent\textbf{Defender.} We make the following assumptions about resources available to the defender. First, we assume the defender has access to the trained DNN model, and a set of correctly labeled samples to test the perfomrance of the model. The defender also has access to computational resource to test or modify DNN models, e.g., GPUs or GPU-based could services.
}

\section{Calculation of Global Model}
\label{app:global_model}

The way we use is the same with~\cite{chen2012bayesian}. For completeness, we write it here in our style.
In our global model, the most computation source will be used to calculate the Eqn~\ref{decompose}.
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
E(\mathbf{h} | \mathbf{r}) = \Sigma_h \mathbf{T}^T \Sigma_{r}^{-1} \mathbf{r}
\end{array}
\end{equation}
where efficient calculation of $\Sigma_r^{-1}$ is challenge. For a class containing $m$ images, the computational complexity of a naive way to calculate Eqn~\ref{decompose} is $O(d^3m^3)$ where $d$ is the dimension of $r$. However, by taking the advantage of the structure of $\Sigma_r$, its complexity can be reduced to $O(d^3 + md^2)$.
Concretely, $\Sigma_r^{-1}$ is in the form:
\begin{equation}
\begin{array}{c@{\quad}l}
\Sigma_r^{-1} =
\begin{bmatrix}
F+G & G & \cdots & G\\
G & F+G & \cdots & G \\
\vdots & \vdots & \ddots & \vdots \\
G & G & ... & F+G \\
\end{bmatrix} \\
\end{array}
\end{equation}
where
\begin{equation}
\begin{array}{c@{\quad}l}
F &= S_{\varepsilon}^{-1} \\
G &= -(m S_{\mu} + S_{\varepsilon})^{-1} S_{\mu}S_{\varepsilon}^{-1}
\end{array}
\end{equation}
Thus, we have
\begin{equation}
\begin{array}{c@{\quad}l}
\mu &= \sum_{i=1}^m S_{\mu} (F+mG)r_i \\
\varepsilon_j &= S_{\varepsilon}' r_j + \sum_{i=1}^m S_{\varepsilon} G r_i \\
&= r_j - \mu
\end{array}
\end{equation}
where $S_{\varepsilon}'$ is the result of last M-step in our EM-like algorithm.

\begin{table}[htb]
\centering
    \caption{Model Architecture for GTSRB.}
\begin{tabular}{@{}lllll@{}}
\toprule
Layer Type & \# of Channels & Filter Size & Stride & Activation \\ \midrule
Conv       & 32             & 3 x 3       & 1      & ReLU       \\
Conv       & 32             & 3 x 3       & 1      & ReLU       \\
MaxPool    & 32             & 2 x 2       & 2      & -          \\
Conv       & 64             & 3 x 3       & 1      & ReLU       \\
Conv       & 64             & 3 x 3       & 1      & ReLU       \\
MaxPool    & 64             & 2 x 2       & 2      & -          \\
Conv       & 128            & 3 x 3       & 1      & ReLU       \\
Conv       & 128            & 3 x 3       & 1      & ReLU       \\
MaxPool    & 128            & 2 x 2       & 2      & -          \\
FC         & 256            & -           & -      & ReLU       \\
FC         & 43             & -           & -      & Softmax    \\ \bottomrule
\end{tabular}
\label{tb:gtsrb_model}
\end{table}

\end{appendices}

\ignore{
\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/dif_triggers.eps}
  \end{subfigure}
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/rep_dif_triggers.eps}
  \end{subfigure}

	\caption{Triggers and corresponding representations. Representations are projected onto the space expanded by their first two principle components. The triggers' position and size are shown in the titles containing also the Mahalanobis distance for representations. }

	\label{fig:trigger_position}

\end{figure*}
}

\section{Background}
\label{sec:background}

\subsection{Deep Neural Networks}
\label{subsec:repusentation}

With its wide application to classification tasks, DNN is rather opaque when it comes to how exactly an input's classification label is determined. At a high level, a DNN model takes an input (e.g., the pixel matrix of an image), transforms it across multiple layers of neurons and finally converts it into a feature vector (the representation, aka., \textit{embedding}) before finding out its likely affiliations with different classes. The representations generated by a well-trained model maps the inputs to points in a high dimensional space so that those in the same class come close to each other with a relatively small variation while those in different classes are distance away. Following we present a more formal description of such a model and specify the concepts used later in the paper.

\vspace{5pt}\noindent\textbf{Formal description}.  A DNN model can be described by a mapping function $F(\cdot)$ that projects the input $x$ onto a proper output $y$, typically a vector that reports the input's probability distribution over different classes, through layers of transformations. Since the last two layers before the output are typically $Softmax(\cdot)$ and $Logits(\cdot)$, most DNN models~\cite{simonyan2014very,szegedy2015going,szegedy2017inception} can be formulated as:

\begin{equation}
\notag
\begin{array}{c@{\quad}l}
y = F(x) = Softmax(Logits(R(x))) \\
\end{array}
\end{equation}
where $R(\cdot)$ describes the rest layers in the DNN. Most interest to us is the output of $R(x)$, which is the DNN's \textit{representation} (aka., \textit{embedding}) of the input $x$. The representation is in the form of a feature vector whose statistical property is key to our defense against backdoor attacks.

Also, the DNN model is trained through minimizing a loss function $l(\cdot)$ by adjusting the model parameter $\hat{\theta}$ with regard to the label of each training input:

\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\hat{\theta} =  minimize_{\theta}  \sum_{x_i \in \mathcal{X}} l(y_t, F(x_i;\theta))
\end{array}
\end{equation}
here $y_t$ is the ground truth for the class $t$, the true class that $x_i$ should belong to, and $\mathcal{X}$ is the whole training dataset. Further, we denote the input set with a true label $t$ by $\mathcal{X}_t$, and a label set for $t$ by $t \in \mathcal{L}$, whose size is $L = |\mathcal{L}|$. We also define a classification function $c(\cdot)$ to represent the prediction label of the input:

\begin{equation}
\notag
\begin{array}{c@{\quad}l}
c(y) = \argmin_{t \in \mathcal{L}} l(y_t, y)
\end{array}
\end{equation}

\subsection{Backdoor Attacks}
\label{subsec:backdoor_attacks}

As mentioned earlier, prior researches ~\cite{gu2017badnets, chen2017targeted} show that backdoors can be injected into a target model so that the model behaves normally on the inputs without a trigger pattern while misclassifying those with the pattern. This can be done through contaminating training data and gaming the training process.

\vspace{5pt}\noindent\textbf{Attack techniques}.  More specifically, in the BadNet attack~\cite{gu2017badnets}, the adversary has full control on the training process of a model, which allows him to change the training settings and adjust training parameters to inject a backdoor into a model.  The model was shown to work well on MNIST~\cite{lecun1998gradient}, achieving a success rate of 99\% without affecting performance on normal inputs.

Without access to training data, an adversary can update a trained model to create a backdoor, as discovered in another study~\cite{liu2017trojaning}. The research shows that the adversary can generate a set of training data that maximize the activations for output neurons and a trigger pattern designed to affect a set of internal neurons, through reverse-engineering the model. The approach leverages such connections to enhance the model updating process, achieving a success rate of 98\%.

In the absence of the model, further research found that a backdoor can be introduced to a model by poisoning a very small portion of its training data, as few as 115 images~\cite{chen2017targeted}. Given the low bar of this attack and its effectiveness (86.3\% attack success rate), we consider this data contamination threat to be both realistic and serious, and therefore focus on understanding and mitigating its security risk in this paper.

\vspace{5pt}\noindent\textbf{Data contamination attack}. Following the prior research~\cite{chen2017targeted}, we consider that in a data contamination attack, the adversary generates attack training samples by $A: x \mapsto A(x)$, where $x$ is a normal sample and $A(x)$ is the infected one. Specifically,
\begin{equation}
\begin{array}{c@{\quad}l}
A(x) &= (1-\kappa)\cdot x + \kappa \cdot \delta \\
\end{array}
\label{eqn:naive_attack}
\end{equation}
where $\kappa$ is a trigger mask, $\delta$ is a trigger pattern, and together, they form a trigger $(\kappa, \delta)$ with its magnitude (norm) being $\Delta$. We also call $s$  \textit{source label} if $x \in \mathcal{X}_s$, and $t$ \textit{target label} if the adversary intends to mislead the target model to misclassify $A(x)$ as $t$, i.e., $c(F(A(x))) = t$. An attack may involve multiple source and target labels.

\subsection{Datasets and Target Models}

\begin{table*}[tbh]
  \centering
    \caption{Information about datasets and target models.}
	\begin{tabular}{|c|c|c|c|c|c|c|}
	\hline
	  Dataset & \# of Classes & \# of Training Images & \# of Testing Images & Input Size & Target Model & Top-1 Accuracy of Uninfected Model\\
    \hline
	GTSRB & 43 & 39,209 & 12,630 & 32 x 32 x 3& 6 Conv + 2 Dense & 96.4\%\\
	\hline
	ILSVRC2012 & 1,000 & 1,281,167 & 49,984 & 224 x 224 x 3 & ResNet50 & 76\%\\
	\hline
	MegaFace & 647,608 & 4,019,408 & 91,712 (FaceScrub) & 128 x 128 x 3 & ResNet101 & 71.4\%\\
	\hline
	\end{tabular}
	\label{tb:datasets_models}
\end{table*}

We analyzed existing attacks and countermeasures and further evaluated our new protection using three classification tasks: traffic sign recognition, object classification and face recognition.  These tasks and datasets involved (Table~\ref{tb:datasets_models}) have been utilized in prior backdoor-related researches.

\vspace{3pt}\noindent\textbf{Traffic sign recognition (GTSRB)}.  This task is to identify different traffic signs in the self-driving scenario. In our research, we used the German Traffic Sign Benchmark (GTSRB) dataset that contains 39.2K training images and 12.6K testing images of 43 different signs~\cite{Stallkamp2012}. The target model in this task has a simple architecture of 6 convolution layers and 2 dense layers (Table~\ref{tb:gtsrb_model}).

\vspace{3pt}\noindent\textbf{Object classification (ILSVRC2012)}.  This task is to recognize general objects (e.g., fish, dog, etc.) from images. A well-known dataset, Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)~\cite{ILSVRC15}, was utilized to train our target model ResNet50~\cite{he2016deep}. This dataset contains 1.2M images from 1000 categories. Our target model was evaluated on its testing set with 49.9K images.

\vspace{3pt}\noindent\textbf{Face recognition (MegaFace)}.  For this task, we leveraged MegaFace~\cite{nech2017level}, the largest publicly available face recognition database with 4M images from 647K identities, to train our target model ResNet101~\cite{he2016deep}. More specifically, following the MegaFace Challenge~\cite{MegaFace88:online}, we tested our model by finding similar images for a given FaceScrub image~\cite{ng2014data} from both the FaceScrub dataset and 1M ``distractor'' images~\cite{MegaFace41:online}.

All the models trained in our research achieved classification performance comparable with what have been reported by state-of-the-art approaches, as illustrated in Table~\ref{tb:datasets_models}.

\subsection{Threat Model}
\label{subsec:adversarial_model}

We consider a situation that the party who trains the target model is trusted but has outsourced the collection of training data to untrusted parties, who may contaminate the training set.  Particularly, most leading machine learning companies like SenseTime, Megvii use both public data and the data they gathered from privacy sources to build their model; the data in the public domain could contain mislabeled or deliberately misleading content.

\vspace{3pt}\noindent\textbf{Black-box attack}.  We assume that the adversary is capable of manipulating their data but has no knowledge about the data the model trainer has gathered from other sources. Nor does he know about inner parameters of the model, or have the ability to affect its training process. So the threat he can pose is a \textit{black-box} attack.

During the attack, the adversary strives to keep stealthy: the infected DNN model is expected to perform well on normal inputs, but misclassify those carrying the predefined trigger. Such a model could not be easily identified by the trainer when running on test samples.

\vspace{3pt}\noindent\textbf{Attack on few}. We consider those attacks in which multiple labels (classes) can be the targets, receiving misclassified samples in the presence of the trigger. On the other hand, we assume that the adversary aims at no more than 50\% of the labels. Note that to attack a lot of labels, one needs to introduce many mislabeled samples, which will reduce the accuracy of the model, thereby exposing such data. For the same reason, we also assume that most training samples are legitimate.

\vspace{3pt}\noindent\textbf{Defender's observations}. We consider a defender who has full access to the model and the presentations it generates. Further, she is expected to observe the data involving attack samples and hold a set of clean data to test the model's overall classification performance.

\ignore{

\subsection{Deep Neural Networks Notations}
\label{subsec:repusentation}
To simplify our further description, we formalize a DNN model for classification as following convention. We write a DNN model as a mapping function $F(\cdot)$ that projects the input $x$ on a proper output $y$. Popularly, in a classification DNN model, before its output, there are two layers $Softmax(\cdot)$ and $Logits(\cdot)$, and their relation can be represented as:
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
y = F(x) = Softmax(Logits(R(x))) \\
\end{array}
\end{equation}
where $R(\cdot)$ depict the rest layers in the DNN.
This architecture is well accepted by many models~\cite{simonyan2014very,szegedy2015going,szegedy2017inception}. In this paper, we interest in $R(x)$ referring as the representation of the input or the embeddings. To train a DNN model, we need to define a loss function $l(\cdot)$ and the training process is an optimization process trying to find a optimal parameter $\hat{\theta}$ of the model to minimize the loss function for every input, i.e.,
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\hat{\theta} =  minimize_{\theta}  \sum_{x_i \in \mathcal{X}} l(y_t, F(x_i;\theta))
\end{array}
\end{equation}
where $y_t$ is the ground true of class $t$, the truly class that $x_i$ belongs to, and $\mathcal{X}$ is the total dataset. Further, we refer $\mathcal{X}_t$ as the set containing all the input $x$ whose true label is $t$. And we define $t \in \mathcal{L}$, the label set, and the size of $\mathcal{L}$ is $L = |\mathcal{L}|$.

We also define a classification function $c(\cdot)$ to represent the prediction label of the input.
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
c(y) = \argmin_{t \in \mathcal{L}} l(y_t, y)
\end{array}
\end{equation}

\subsection{Backdoor Attacks}
\label{subsec:backdoor_attacks}
Backdoor attack aims to inject backdoors into the target model and let it behave normally for intact inputs while misbehave for infected inputs with trigger pattern designed for the already injected trap doors.

\vspace{5pt}\noindent\textbf{Attack channels.} To launch backdoor attack, there are several channels have been proposed by previous researchers. Gu et al.~\cite{gu2017badnets} proposed BadNets, an attack relays on fully accessing to the training process. Specifically, the attacker can change the training settings, replace the training data, access the model parameters, etc. Through cooperating the training process, they achieve 99\% success rate of attack (percentage of infected inputs that are misclassified) without impacting model performance on MNIST~\cite{lecun1998gradient}.

Liu et al.~\cite{liu2017trojaning} proposed Torjan attack that reverses the target model and its training data to find triggers having stronger connection between the internal neurons. They place their hope on activating the peer neurons of the target model when feeding it by inputs stamped with their found triggers. Their experiments show these triggers are very effective to launch the backdoor attack and the success rate is as high as 98\%.

Chen et al.~\cite{chen2017targeted} proposed a targeted backdoor attack through data poisoning. Their attack do not rely on accessing the training process nor the model parameters. They found that as long as several poisoned instances are injected into the training dataset, the model trained on this dataset will contain the backdoor. As there is no constraint on trigger pattern in this attack, attacker can use fabricable trigger to launch the physical attack (wearing a sun glasses). The effort to launch this attack is so low that it is indeed harm the usage of data outsourcing on deep neural network industry. Thus, we focus on how to mitigate this kind of attack in following article.

\vspace{5pt}\noindent\textbf{Attack Formalization.}
We formalize the last mentioned attack as the same way in~\cite{chen2017targeted}. We assume attackers would generate the injected sample by $A: x \mapsto A(x)$, where $x$ is the intact sample and $A(x)$ is the infected sample. Specifically,
\begin{equation}
\begin{array}{c@{\quad}l}
A(x) &= (1-\kappa)\cdot x + \kappa \cdot \delta \\
\end{array}
\end{equation}
where $\kappa$ is the trigger mask, $\delta$ is the trigger pattern, and they together compose a trigger pair$(\kappa, \delta)$ that is simply called as a trigger. We also call $s$ as the source label if $x \in \mathcal{X}_s$, and $t$ as the target label if the attacker want to mislead the target DNN model to misclassify $A(x)$ as $t$, i.e., $c(F(A(x))) = t$. There maybe multiple source labels and target labels during once attack.

\subsection{Datasets and Target Models}
\begin{table*}[tbh]
  \centering
    \caption{Information about datasets and target model.}
	\begin{tabular}{|c|c|c|c|c|c|c|}
	\hline
	  Dataset & \# of Classes & \# of Training Images & \# of Testing Images & Input Size & Target Model & Top-1 Accuracy\\
    \hline
	GTSRB & 43 & 39,209 & 12,630 & 32 x 32 x 3& 6 Conv + 2 Dense & 96.4\%\\
	\hline
	ILSVRC2012 & 1,001 & 1,281,167 & 49,984 & 300 x 300 x 3 & ResNet50 & 76\%\\
	\hline
	MegaFace & 647,608 & 4,019,408 & 91,712 (FaceScrub) & 128 x 128 x 3 & ResNet101 & 71.4\%\\
	\hline
	\end{tabular}
	\label{tb:datasets_models}
\end{table*}

In this paper we consider three tasks used throughout the existing work in this field: (1) Traffic Sign Recognition. (2) Object Classification. (3) Face Recognition. Details of each task and associated dataset are described below. A brief summary is also included in Table~\ref{tb:datasets}.

\vspace{3pt}\noindent$\bullet$\textit{ Traffic Sign Recognition (GTSRB)}.  This task is to recognize different traffic signs in the self-driving scenario. We used the German Traffic Sign Benchmark (GTSRB) dataset which contains 39.2K training images and 12.6K testing images of 43 different signs~\cite{Stallkamp2012}. The target model we used in this task has a simple architecture of 6 convolution layers and 2 dense layers (Table~\ref{tb:gtsrb_model}).

\vspace{3pt}\noindent$\bullet$\textit{ Object Classification (ILSVRC2012)}.  This task is to recognize general objects from images. A well-known dataset, Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)~\cite{ILSVRC15}, was used to train our target model ResNet50~\cite{he2016deep}. This dataset contains 1.2M images from 1001 categories. And we tested our target model on its testing set containing 49.9K images.

\vspace{3pt}\noindent$\bullet$\textit{ Face Recognition (MegaFace)}.  This task is to recognize human face. We used one of the biggest datasets, MegaFace~\cite{nech2017level} containing 4M images of 647K identities, to train our target model ResNet101~\cite{he2016deep}. And, specifically, as guided by description on~\cite{MegaFace88:online}, we test our model on FaceScrub~\cite{ng2014data} and distracted by other 1M distractor images~\cite{MegaFace41:online}.

We showed top-1 accuracy of our target models in Table~\ref{tb:datasets}. Their performance compete to the state-of-the-art results.

\subsection{Threat Model}
\label{subsec:adversarial_model}

We assume that there is a valued DNN model (target model) held by a trusted service provider who profits by providing services on this model and thus wants to improve its performance by collecting more data. To effectively collect data, the provider outsources this collection to untrusted data collectors in which an attacker may be.

\noindent\textbf{Attacker.} In our scenario, attackers can arbitrary manipulate their data and control the uploading process. However, they have no knowledge about the data the provider already have nor held by other collectors. Besides, the attacker have no knowledge about the inherent parameters of the target model and cannot affect the training process launched by the provider.

The attacker wants to insert backdoors into the target model and, at the meanwhile, keep stealthy. Specifically, the backdoored DNN model performs well on most normal inputs, but exhibits targeted misclassification when presented an input containing a trigger predefined by the attacker. Such a backdoored DNN model will produce expected results on test samples held by the provider.

\noindent\textbf{Majority of intact labels.}Multiple labels (classes) can be infected if backdoors cause targeted misclassification to those labels. But we assume the majority of labels remain intact (uninfected). By attacker's nature, these backdoors prioritize stealth, and an attacker is unlikely to risk detection by embedding many backdoors into a single model. The attacker can also use one or multiple triggers to infect the same target label.

\noindent\textbf{Minority of infected samples.} To not drop the testing performance, the attacker will not upload many infected samples. To be noticed that outsourced data may contains samples belonging to new classes that are different from current classes the provider already know. This thing is common and ignored by previous researchers. For instance, in face recognition task, the data collector can upload photos of someone the provider don't know. Thus, the minority of infected data does not rule out some classes overwhelmed by infected samples, so along as the total number is small.

\noindent\textbf{Defender.} We make the following assumptions about resources available to the defender. First, we assume the defender has access to the trained DNN model, and a set of correctly labeled samples to test the perfomrance of the model. The defender also has access to computational resource to test or modify DNN models, e.g., GPUs or GPU-based could services.
}

\section{Conclusion}
\label{sec:conclusion}

Our work demonstrated that backdoors created by existing black-box attacks are source-agnostic and characterized by unique representations generated for attack images, which are mostly determined by the trigger, regardless of other image content, and clearly distinguishable from those for normal images. Existing detection techniques rely on these two proprieties and all fail to raise the bar to black-box attacks injecting source-specific backdoors like TaCT.
Based on leveraging the distribution of the sample representations through a two-component decomposition model, we designed a statistical method SCAn to untangle representations of each class into a mixture model, and utilized a likelihood ratio test to detect an infected class. The effectiveness and robustness of SCAn were demonstrated through extensive experiments.
Our study takes a step forward to understand the mechanism of implanting a backdoor within a DNN model and how a backdoor looks like from the perspective of model's representations. It may lead to deeper understanding of neural networks.

\section{Statistical Contamination Analyzer}
\label{sec:defense}

\ignore{
Last section illustrated that current defenses have assumed the partition of trigger and identity features directly or implicitly, however, this assumption is weak and broken by TaCT, and therefore failed to raise the bar to the backdoor attacks. A more solid stone is needed for robust defenses.
}

In the presence of source-specific backdoors, which can be easily injected through TaCT, the representations of individual attack samples (trigger-carrying images) become almost indistinguishable from those of normal images, rendering existing detection techniques less effective. So to detect the backdoors, we have to go beyond individual instances and look at the \textit{distribution} of the representations for the target class that a data-contamination attack subtly alters. To this end, we present in this section a new technique called \textit{Statistical Contamination Analyzer} (SCAn) to capture such an anomaly and further demonstrate that the new approach is not only effective but also robust to black-box attacks.

\subsection{Design}
\label{subsec:intuition}

\noindent\textbf{Idea}. A key observation is that in a backdoor contamination attack, the adversary attempts to cheat a model into ``merging'' two sets of images under the target label: those legitimately belonging to the label and those with triggers but originally from another label. This effort leads to a fundamental difference between the images in the target class and those in other classes, in terms of their representation distributions, under the following assumptions:

\vspace{2pt}\noindent$\bullet$\textit{ Two-component decomposition}.
In the representation space, each point can be decomposed into two independent components: a class-specific {\em identity} and a {\em variation} component.

\vspace{2pt}\noindent$\bullet$\textit{ Universal variation}.
The variation vectors in any uninfected class have the same distribution as those in the attack sample set (the collection of all attack images from a different class).

Prior research~\cite{wang2004unified} shows that, in face recognition, an image can be decomposed into three mutually orthogonal components: within-class, between-class and noise, with the noise part largely eliminated in a well-trained model. Intuitively, one's facial image contains her identity features that separate her from others (that is, between-class component), as well as the information describing her facial transformation, e.g., size, orientation, etc, (the within-class {\em variation} component).
Although the variation component does not contribute directly to the classification task in a DNN model, it is often extracted through the representation learning as it describes the recurrent and robust signal in the input data, and the representation learning by a non-ideal DNN model may not be able to distinguish them from the identity component. We note that the previous backdoor detection approaches fail to separate these two components, allowing the variation part, which often includes a trigger pattern, to reduce the sensitivity in detecting more targeted attacks like TaCT.

We further assume that the variation component of an input sample is {\em independent} of label (i.e., sample class); as a result, the distribution learned from the input samples from one label (e.g., an non-target class) can be transferred to another one (e.g., the target class without infection). Intuitively, in face recognition, smile is a variation component adopted by different human individuals, leading to the common transformation of face images independent of the identity of each individual (i.e., the class label).
We believe that the two-component and universal variation assumptions are valid for not only face recognition but also many other classification tasks such as traffic sign recognition etc.

By decomposing samples in both normal and infected classes, we are able to obtain a finer-grained observation about the impacts of triggers on classification that cannot be seen by simply clustering representations, as prior research does.
Fig~\ref{fig:idea_representations} shows an example, where the representations of samples in the infected class (right) can be viewed as a mixture of two groups, the attack samples and the normal samples, each decomposed into a distinct identity component and a common variation component; in comparison, without the two-component decomposition, the representations of the samples in the infected and normal class are indistinguishable.

Formally, the representation of an input sample $x$ can be decomposed into two latent vectors:
\begin{equation}
\begin{array}{c@{\quad}l}
r = R(x) = \mu_t + \varepsilon
\end{array}
\label{decompose}
\vspace{-0.05in}
\end{equation}
where $\mu_t$ is the identity vector (component) of the class $t$ that $x$ belongs to, and $\varepsilon$ is the variation vector of $x$, which follows a distribution independent of $t$. We denote by $\mathcal{X}_t$ the set of the samples with the class label $t$, and by $\mathcal{R}_t$ the set of their representations, i.e., $\mathcal{R}_t = \{R(x_i)| x_i \in \mathcal{X}_t\}$.

In the presence of a backdoor contamination attack, samples in a target class $t^*$ include two non-overlapping subgroups: i.e., normal samples and attack samples, i.e., $\mathcal{R}_{t^*}=\mathcal{R}^{normal}_{t^*} \cup \mathcal{R}^{trigger}_{t^*}$. As a result, the representations of samples in the target class follow a multivariate mixture distribution: for each $x_i \in \mathcal{X}_{t^*}$,

\begin{equation}
r_i=\delta_i \mu_{1} + (1-\delta_i) \mu_{2} + \varepsilon,
\label{mixturemodel}
\end{equation}

\noindent where $\mu_{1}$ and $\mu_{2}$ represent the identity vectors of normal and attack samples in the class $t^*$, respectively, and $\delta_i=1$ if the $x_i$ is a normal sample and $\delta_i=0$ otherwise. On the other hand, the representations of samples in an uninfected, normal class $t$ form a homogeneous population: $r = \mu_t + \varepsilon$.
Therefore, the task of backdoor detection can be formulated as a hypothesis testing problem: given the representations of input data from a specific class $t$, we want to test whether it is more likely from a mixture group (as defined in (\ref{mixturemodel})) or from a single group (as defined in (\ref{decompose})). Notably, the problem is non-trivial because the input vectors are of high dimension (hundreds of features are learned in a typical DNN model), and more importantly, the parameters (i. e., $\mu_{t}$ and $\varepsilon$) are unknown for the mixture model and need to be derived simultaneously with the hypothesis test. Finally, our approach does not rely on the assumptions underlying current backdoor detection techniques (section~\ref{sec:attack}): i.e., trigger-dominant representation significantly different from that of a legitimate sample.
Instead, we investigate the distributions of the representations from all input samples labeled in each class: the class containing a mixture of two groups of vectors is considered to be contaminated.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/one-partition-demo.pdf}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/two-partition-demo.pdf}
  \end{subfigure}
	\caption{A schematic illustration of the assumption of two-component decomposition (right) in the representation space, in comparison with the naive homogeneous assumption (left).}
	\label{fig:idea_representations}
	\vspace{-0.1in}
\end{figure}

\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/idea.png}
  \end{subfigure}

	\caption{An illustration of Statistical Contamination Analyzer.}
	\label{fig:defense}
\vspace{-0.25in}
\end{figure*}

\vspace{3pt}\noindent\textbf{Algorithm}.
Our approach utilizes several statistical methods to estimate the most likely parameters for the decomposition and untangling models and then detect an infected label through a likelihood ratio test. It contains following steps and is illustrated in Fig~\ref{fig:defense}.

\noindent\textit{Step 1:} Leverage the target model to generate representations for all input data (images) from the training set that contains both the infected and normal images.

\noindent\textit{Step 2:} Estimate the parameters in the decomposition model by running an EM (Eqn~\ref{decompose}) algorithm on a set of clean data for identifying the covariation matrix ($S_{\varepsilon}$ for the variation $\varepsilon$) with a high confidence.

\noindent\textit{Step 3:} Across all samples in each class, leverage the covariation matrix to estimate the identity vector and decompose the representations of this class.

\noindent\textit{Step 4:} Across all samples in each class, use an iterative method to estimate the parameters for the mixture model (Eqn~\ref{mixturemodel}) containing two subgroups.

\noindent\textit{Step 5:} For samples in each class, perform the likelihood ratio test on their representations using the mixture model (from step 4) against the the null hypothesis -- the decomposition model (from step 3); if the null hypothesis is rejected, the corresponding class is reported to be infected (i.e., this class contain contaminated samples).

\subsection{Technical Details}
\label{subsec:approach}

Here, we present the technical details of our Statistical Contamination Analyzer (SCAn).

\vspace{5pt}\noindent\textbf{Two-component decomposition}. Under the assumptions of two-component decomposition and universal variation, a representation vector can be described as the superposition of two latent vectors: $r = \mu + \varepsilon$, with $\mu$ and $\varepsilon$ each following a normal distribution: $\mu \sim N(0,S_{\mu})$ and $\varepsilon \sim N(0,S_{\varepsilon})$,
\ignore{
explain the representation $r$ as a random variable equaling to the sum of other two independent random variables $\mu$ and $\varepsilon$.

\begin{equation}
\begin{array}{c@{\quad}l}
r = \mu + \varepsilon
\end{array}
\label{eqn:r}
\end{equation}

We assume that the latent variable $\mu$ and $\varepsilon$ follow two normal distributions:

\begin{equation}
\begin{array}{c@{\quad}l}
\mu \sim N(0,S_{\mu}) \\
\varepsilon \sim N(0,S_{\varepsilon}) \\
\end{array}
\end{equation}

}
where $S_{\mu}$ and $S_{\varepsilon}$ are two unknown covariance matrices, which need to be estimated from the representations of input data.
Notably, $S_{\mu}$ can be approximated by the between-class covariance matrix and $S_{\varepsilon}$ by the within-class covariance matrix. However, these approximations are previously shown to be less effective than the EM algorithm~\cite{chen2012bayesian}.  Therefore, we run the EM to estimate most likely model parameters on a set of clean data (e.g., those one uses to evaluate a machine learning model), as follows:

\noindent{\em E-step:} Following the decomposition model~\ref{decompose}, we express the relationship between
the observation $\mathbf{r} = [r_1;...;r_m]$ (e.g., for $m$ images) and the latent vectors $\mathbf{h} = [\mu;\varepsilon_1;...;\varepsilon_m]$ in the matrix form as:

\begin{equation}
\begin{array}{c@{\quad}l}
\mathbf{r} = \mathbf{T} \mathbf{h}, & \text{where  } \mathbf{T} =
\begin{bmatrix}
\mathbf{I} & \mathbf{I} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{I} & \mathbf{0} & \mathbf{I} & \cdots & \mathbf{0} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
\mathbf{I} & \mathbf{0} & \mathbf{0} & ... & \mathbf{I} \\
\end{bmatrix}
\end{array}
\end{equation}
Thus, $\mathbf{h} \sim N(0,\Sigma_h)$, where
\begin{equation}
\begin{array}{c@{\quad}l}
\Sigma_h =
\begin{bmatrix}
S_{\mu} & \mathbf{0} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{0} & S_{\varepsilon} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{0} & \mathbf{0} & S_{\varepsilon} & \cdots & \mathbf{0} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
\mathbf{0} & \mathbf{0} & \mathbf{0} & ... & S_{\varepsilon} \\
\end{bmatrix} \\
\end{array}
\end{equation}
and $\mathbf{r} \sim N(0,\Sigma_r)$, where
\begin{equation}
\begin{array}{c@{\quad}l}
\Sigma_r =
\begin{bmatrix}
S_{\mu}+S_{\varepsilon} & S_{\mu} & \cdots & S_{\mu}\\
S_{\mu} & S_{\mu}+S_{\varepsilon} & \cdots & S_{\mu} \\
\vdots & \vdots & \ddots & \vdots \\
S_{\mu} & S_{\mu} & ... & S_{\mu}+S_{\varepsilon} \\
\end{bmatrix} \\
\end{array}
\end{equation}
Hence, given the observation $\mathbf{r}$ and model parameters $S_{\mu}$ and $S_{\varepsilon}$, the expectation of $\mathbf{h}$ can be computed by~\cite{Multivar47:online}:

\begin{equation}
\begin{array}{c@{\quad}l}
E(\mathbf{h} | \mathbf{r}) = \Sigma_h \mathbf{T}^T \Sigma_{r}^{-1} \mathbf{r}
\end{array}
\label{eqn:global_expectation}
\end{equation}

\noindent{\em M-step:} In this step, we try to obtain the most likely parameters of $S_{\mu}$ and $S_{\varepsilon}$ that lead to the maximum expectation of $\mathbf{h}$ (i.e., $\mu$ and $\varepsilon$) in Eqn~\ref{eqn:global_expectation}. A straightforward update rule of $S_{\mu}$ and $S_{\varepsilon}$ is:
\begin{equation}
\begin{array}{c@{\quad}l}
S_{\mu} &= \mathbf{cov}(\mu) \\
S_{\varepsilon} &= \mathbf{cov}(\varepsilon) \\
\end{array}
\end{equation}
where $\mu$ and $\varepsilon$ are the expected vectors computed in the previous E-step. Even though we do not have a formal proof for the convergence of the Expectation-Maximization algorithm, the approach is found to be effective in our experiments.

\vspace{5pt}\noindent\textbf{Two-subgroup untangling}.
We assume the representations of the infected samples follow a mixture model of two Gaussian distributions, one for the group of cover samples ($N(\mu_1, S_1)$) and the other for the group of attack samples ($N(\mu_2, S_2)$. If the class labels are known for each sample, a hyperplane that maximizes the between-class variation versus the within-class variation among the projected representation vectors can be determined by using a Linear Discriminant Analysis (LDA)~\cite{mika1999fisher}, which
maximizes the Fisher's Linear Discriminant (FLD):

\begin{equation}
\begin{array}{c@{\quad}l}
\text{FLD}(v) = v^T \Sigma_B v / v^T \Sigma_W v
\end{array}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\Sigma_B &= (\mu_1-\mu_2)(\mu_1-\mu_2)^T \\
\Sigma_W &= S_1 + S_2
\end{array}
\end{equation}
Empirically, a greater FLD corresponds to distant projected means and concentrated projected vectors for each of the two subgroups.
However, in our case, the labels ({\em normal} or {\em attack}) of the representations are unknown, and thus we cannot estimate the mean and covariance matrix for each subgroup. To address this challenge, we first assume $S_1 = S_2 = S_{\varepsilon}$ according to the {\em universal variation assumption},
and then use an iterative algorithm to simultaneously estimate the model parameters ($\mu_1$, $\mu_2$ and $S_{\varepsilon}$) and the subgroup label for each sample in the class.

\noindent{\em Step-1:} Suppose we know the class labels of samples in one class, Then we can estimate the model parameters ($\mu_1$, $\mu_2$ and $S_{\varepsilon}$) on the representations of normal samples and attack samples, respectively, using the similar method as presented above in the {\em E-step}.

\noindent{Step-2:} After obtaining the model parameters, we compute the optimal discriminating hyperplane (denoted by its normal vector $v$) by maximizing the FLD,

\begin{equation}
\begin{array}{c@{\quad}l}
v &= S_{\varepsilon}^{-1} (\mu_1-\mu_2)
\end{array}
\end{equation}
from which we can re-compute the subgroup label $c_i$ for each sample $i$
($c_i=1$ if the $i$-th sample is a cover sample, and $c_i=2$ if it is an attack sample),
\begin{equation}
\begin{array}{c@{\quad}l}
c_i =
\begin{cases}
1, v^T r < t\\
2, v^T r \ge t\\
\end{cases}
\end{array}
\label{eq:z}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
t &= \frac{1}{2} (\mu_1^T S_{\varepsilon}^{-1} \mu_1 - \mu_2^T S_{\varepsilon}^{-1} \mu_2)
\end{array}
\end{equation}

\noindent{Step 3:} Iteratively execute Step-1 and Step-2 until convergence. Finally, we will simultaneously obtain the model parameters and the subgroup labels of all samples in the class of interest.

\vspace{3pt}\noindent\textbf{Hypothesis testing}. For each class $t$, we aim to determine whether a class is contaminated by using a likelihood ratio test over the samples ($\mathcal{R}_t$) in the class based on two hypotheses:

\noindent (null hypothesis) $\mathbf{H_0}:$ $\mathcal{R}_t$ is drawn from a single normal distribution.

\noindent (alternative hypothesis) $\mathbf{H_1}:$ $\mathcal{R}_t$ is drawn from a mixture of two normal distributions.

\noindent and the statistic is defined as:
\begin{equation}
\begin{array}{c@{\quad}l}
J_t = -2 \log{\frac{P(\mathcal{R}_t | \mathbf{H_0})}{P(\mathcal{R}_t | \mathbf{H_1})}}
\end{array}
\label{eqn:criteria}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}

P(\mathcal{R}_t | \mathbf{H_0}) &= \Pi_{r \in \mathcal{R}_t} N(r | \mu_t,S_{\varepsilon})\\
P(\mathcal{R}_t | \mathbf{H_1}) &= \Pi_{i: c_i=1} N(r_i| \mu_1, S_{\varepsilon}) \Pi_{i: c_i=0} N(r_i | \mu_2, S_{\varepsilon})\\
\end{array}
\end{equation}
Applying Eqn~\ref{eqn:criteria}, we can simplify the likelihood ratio,
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
J_t & = 2 \log(P(\mathcal{R}_t | \mathbf{H_1}) / {P(\mathcal{R}_t | \mathbf{H_0}))} \\
& = \sum_{r \in \mathcal{R}_t}[(r-\mu_t)^T S_{\varepsilon}^{-1} (r-\mu_t) - (r-\mu_j)^T S_{\varepsilon}^{-1} (r-\mu_j)]
\end{array}
\end{equation}
where $j \in \{1,2\}$ is the subgroup label of the representation $r$.

According to the Wilks' theorem~\cite{wilks1938large}, our statistic $J_t$ follows a $\chi^2$ distribution with the degree of freedom equal to the different number of free parameters between the null and alternative hypotheses. In our case, however, the degree of freedom may be as large as tens of thousands, and thus it is difficult to compute the p-value using the $\chi^2$ distribution.
Fortunately, according to the central limit theorem~\cite{wiki:chi-squared}, a regularized variable $\bar{J}_t$:
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\bar{J}_t = (J_t-k) / \sqrt{2k}
\end{array}
\end{equation}
follows the standard normal distribution, where $k$ is the degree of freedom of the underlying $\chi^2$ distribution. Therefore, we leverage the normal distribution of the Median Absolute Deviation (MAD)~\cite{leys2013detecting} to detect the class(es) with abnormally great values of $J$. Specifically, we use $J_t^*$ as our test statistic for the class $t$:
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
J_t^* = |\bar{J}_t-\tilde{J}| / (\text{MAD}(\bar{J}) * 1.4826)
\end{array}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\tilde{J} &= median(\{\bar{J}_t: t \in \mathcal{L} \}) \\
\text{MAD}(\bar{J}) &= median(\{|\bar{J}_t-\tilde{J}|: t \in \mathcal{L}\}) \\
\end{array}
\end{equation}
Here, the constant (1.4826) is a normalization constant for the standard normal distribution followed by $\bar{J}_t$. Therefore, when $J_t^* > 7.3891 = exp(2)$, the null hypothesis $\mathbf{H_0}$ can be rejected with a confidence $> (1-1e^{-9})$, and thus the class $t$ is reported to be contaminated.

\subsection{Effectiveness}

In this section, we report our study on the effectiveness of SCAn, in which our approach was run against TaCT in the presence of various tasks and triggers, and different amounts of clean data.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/solid_md.png}
		\caption{Square}
		\label{fig:triggers:square}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/normal_md.png}
		\caption{Normal}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/uniform.png}
		\caption{Uniform}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/watermark.png}
		\caption{Watermark}
  \end{subfigure}
	\caption{Four triggers used in our experiments}
	\label{fig:triggers}
	\vspace{-0.1in}
\end{figure}

\vspace{3pt}\noindent\textbf{Various tasks and triggers}. In our experiments, we ran TaCT on three datasets (Table~\ref{tb:datasets_models}) using four different triggers (Fig~\ref{fig:triggers}). These datasets cover not only different tasks but also various data distributions. Specifically, GTSRB has a small number of classes and images; ILSVRC2012 contains many classes with each involving a large number of images; MegaFace is characterized by many classes but each has only a few images. Our selection of triggers also took into consideration different pixel distributions: the square trigger is a small norm pattern with all pixels staying together; the normal trigger is more spread out; the uniform trigger is scattered across an image, and the watermark trigger is characterized by a dense and observable pattern.

To the attacker's advantage, we injected 2\% attack images and 1\% cover images into the training set of the target model. As shown in Table~\ref{tb:scan_top1} and~\ref{tb:scan_mis}, all the resulting infected models had a comparable performance with the uninfected one, while misclassifying most images carrying the triggers. We utilized the clean data comprising 10\% of the whole dataset to perform the decomposition and estimate the parameters of the model (Eqn~\ref{decompose}), and then ran the untangling algorithm on the whole dataset using the variation matrices constructed from the decomposition. Our study shows that SCAn is very effective in detecting the TaCT attack. Particularly, the test statistic $J^*$ of the target class was found to be well above from those of the uninfected classes by orders of magnitude. Fig~\ref{fig:effect_4triggers_3tasks} illustrates $ln(J^*)$, showing that SCAn detected all infected models across all datasets and triggers, without causing any false positive.

\begin{table}[tbh]
\centering
\caption{Top-1 accuracy of infected models (averaged over 5 rounds).}
\begin{tabular}{|c|c|c|c|}
\hline
     & GTSRB & ILSVRC2012 & MegaFace\\
    \hline
Square & 96.6\% & 76.3\% & 71.1\% \\
\hline
Normal & 96.1\% & 76.1\% & 71.2\%\\
\hline
Uniform & 95.9\% & 75.9\% & 71.2\%\\
\hline
Watermark & 96.5\% & 75.5\% & 70.9\%\\
\hline
Uninfected & 96.4\% & 76.0\% & 71.4\%\\
\hline
\end{tabular}
\label{tb:scan_top1}
\vspace{-0.1in}
\end{table}

\begin{table}[tbh]
\centering
\caption{Targeted misclassification rate of infected models (averaged over 5 rounds).}
\begin{tabular}{|c|c|c|c|}
\hline
   & GTSRB & ILSVRC2012 & MegaFace\\
    \hline
Square & 98.5\% & 98.2\% & 98.1\% \\
\hline
Normal & 82.4\% & 83.8\% & 81.4\%\\
\hline
Uniform & 94.9\% & 90.5\% & 88.2\%\\
\hline
Watermark & 99.3\% & 98.4\% & 97.1\%\\
\hline
\end{tabular}
\label{tb:scan_mis}
\vspace{-0.1in}
\end{table}

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_square.pdf}
		\caption{Square}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_normal.pdf}
		\caption{Normal}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_uniform.pdf}
		\caption{Uniform}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_watermark.pdf}
		\caption{Watermark}
  \end{subfigure}
	\caption{Detection results of SCAn on different datasets and triggers (average over 5 rounds).}
	\label{fig:effect_4triggers_3tasks}
	\vspace{-0.1in}
\end{figure}

\vspace{3pt}\noindent\textbf{Clean data for decomposition}.
To achieve a high discriminability on mixed representations, our untangling model needs to accurately estimate the covariance matrix ($S_{\varepsilon}$), which describes how sparse the representations of the same class are. For this purpose, our approach uses a set of clean data to avoid the noise that may affect the estimation. Our experiments show that from a set of clean data comprising 10\% of the whole dataset, SCAn can accurately recover the covariance matrices. Furthermore, we investigated how much clean data is necessary for the operation of SCAn.

Specifically, in the presence of 2\% attack images and 1\% cover images,  we adjusted the amount of the clean data for the decomposition analysis. The results are shown in Fig~\ref{fig:known_data_ratio}. From the figure, we can see that even when the clean data collected for decomposition are merely 0.3\% of the whole dataset, still our approach generated the covariance matrices accurately enough for differentiating the target class from others.

\begin{figure*}[!htb]
   \begin{minipage}{0.33\textwidth}
     \centering
     \includegraphics[width=\linewidth]{figure/known_data_ratio.pdf}
     \caption{Test statistic $J^*$ of the target class on different amount of clean data known for decomposition model (average over 5 rounds).}
     \label{fig:known_data_ratio}
   \end{minipage}
   \begin{minipage}{0.33\textwidth}
     \centering
     \includegraphics[width=\linewidth]{figure/multi_triggers.pdf}
     \caption{Minimum $J^*$ of target classes under multiple target-trigger attack and 1\% clean data known for decomposition model (average over 5 rounds).}
     \label{fig:multi_target:a}
   \end{minipage}
   \begin{minipage}{0.33\textwidth}
     \centering
     \includegraphics[width=\linewidth]{figure/number_triggers.pdf}
     \caption{The amount of clean data required by decomposition model for defeating multiple target-trigger attacks on GTSRB.}
     \label{fig:number_triggers}
   \end{minipage}
   \vspace{-0.25in}
\end{figure*}

\subsection{Robustness}
\label{subsec:robustbess}

Further we evaluated the robustness of SCAn in the presence of a black-box adversary who is knowledgeable about our technique and attempts to evade our detection. More specifically, we consider two attack strategies the adversary could employ: use of multiple triggers for different target classes to make the outlier test less effective, and recovery of the decomposition parameters to find a less detectable trigger. Following we report our analysis of our approach's resilience to these attacks.

\vspace{3pt}\noindent\textbf{Multiple target-trigger attack}. The adversary might attempt to infect a model using multiple triggers for different classes, in the hope of elevating $J^*$ for many classes to undermine the effectiveness of the outlier detection. This attempt, however, could introduce observable drop on both the top-1 accuracy and the targeted misclassification rate. In our research, we analyzed the threat of such an attack using different number of triggers targeting multiple labels. These triggers are all of the same shape (square, see Fig~\ref{fig:triggers:square}) but in different color patterns. We first utilized 1\% of the training set as the clean data for decomposition. As demonstrated in Fig~\ref{fig:multi_target:a}, SCAn starts to miss some infected classes when 8 or more triggers are injected into the training set, which could be addressed by raising the amount of the clean data for decomposition when the number of injected triggers is less than the half of total classes. Fig~\ref{fig:number_triggers} demonstrates how many clean data are needed to defeat multiple target-trigger attacks on GTSRB. Specifically, a set of clean data comprising 18\% of the whole dataset can defeat attacks injecting multiple triggers for 21 (48.8\%) classes.
Most importantly, when the number of injected triggers is more than half of total classes, as demonstrated on Fig~\ref{fig:acc_triggers}, there is an observable negative impact on the performance of the model for ILSVRC2012: its top-1 accuracy goes down from 76.3\% to 71.1\%, which leads us to believe that this evasion attempt might leads to the exposure of the backdoor infection of the target model, and the targeted misclassification rate drops significantly, a.k.a, the trigger becomes less effective.

\vspace{3pt}\noindent\textbf{Decomposition parameter recovery}.
Such information including the whole dataset, the target model's structure and our detection method, once exposed, could allow the adversary to search for the trigger with the minimum impact on the representation distribution. However, we argue that the cost to identify such a trigger is non-trivial. Given the default confidence threshold (i.e., p-value$=1e^{-9}$), the power (the false negative rate) of test statistic $J^*$ is $< 10^{-4}$, indicating that an adversary needs to examine on average $10^4$ triggers to identify a putative trigger that bypasses the SCAn defense. This will lead to a significantly amount of computation ($10^4$ times than the defense) because it requires the adversary to reproduce the whole defense procedure including training a substitute model for the examination of each trigger. Besides, an effective trigger may not be identified easily, as the relationship between the trigger and the test statistic $J^*$ is complex and obscure, which unlike the gradients when training a neural networks can be directly used to guide the trigger searching. We note that an adversary may use a more sophisticated strategy to search for an effective trigger that bypasses the SCAn defense; we plan to explore such strategies and their mitigation techniques in our future research.

Giving the adversary further advantage, we assume that the adversary can somehow leverage the parameters of the representation distributions of the target class (e.g., the identity vectors and the covariance matrix $S_{\varepsilon}$) to largely accelerate the trigger searching. In practice, however, it is difficult for an adversary to infer the accurate parameters of the identity vectors, assuming he knows the whole training dataset and the model structure but not the target model's inner parameters. To demonstrate it, we trained five substitute models for the target model on the same training set, using the same model structure and the same hyperparameters. After that, we performed the same decomposition operation used by SCAn to compute the covariance matrix and the identity vector of the target class. The first row of Table~\ref{tb:identity_dist} shows the average distance between the estimated and the real identity vector (estimation error). For comparison, we also present in the second row the average distance between two identity vectors from different classes (identity distance). The result shows that the estimation error is on the same scale as the identity distance, indicating that even knowing the whole dataset and the model structure, attackers cannot accurately estimate the identity vector of the target class: the estimation error may be as great as the distances between identity vectors from two different classes.

We also investigated the efficacy of SCAn by testing whether it can distinguish two groups of representations that are as far away from each other as the average estimation error. Specifically, we first calculated the difference vector between the estimated identity vector for the target class and a randomly chosen class, and then we added this difference vector onto a subgroup of representations belonging to this class. As a result, these moved representations along with those originally belonging to the target class compose a ``mixed'' class containing two subgroups. We then ran SCAn to check whether this ``mixed'' class is infected. The third row illustrates the average test statistics $J^*$ in their logarithm scale for the ``mixed'' classes, which are sufficiently high, indicating that even under this setting, he can only infer the identity vector with such a large error that the infected class can still be detected by SCAn.

\begin{table}[tbh]
\centering
\caption{SCAn results when attackers know the whole dataset and model structure.}
\begin{tabular}{|c|c|c|c|}
\hline
     & GTSRB & ILCVR2012 & MegaFace\\
    \hline
Estimation error & 15.6 & 5.3 & 12.9 \\
\hline
Identity distance & 26.7 & 9.6 & 19.4 \\
\hline
$Ln(J^*)$ & 8.2 & 10.2 & 7.1  \\
\hline
\end{tabular}
\label{tb:identity_dist}
\end{table}

\ignore{
Further we evaluated the robustness of SCAn in the presence of a black-box adversary who is knowledgeable about our technique and attempts to evade our detection. More specifically, we consider two attack strategies the adversary could employ:

\noindent$\bullet$\textit{Multiple target classes with different triggers}. In this scenario, multiple backdoors targeting distinct labels are inserted into the target model, with each backdoor controlled by a specific trigger.

\noindent$\bullet$\textit{Exploit on the decomposition process}.  This scenario is also named as the \textit{black-box attack}, where the attacker somehow knows the training data including those clean data used to build the decomposition model, as well as the model structures but not the parameters of the target model.

\vspace{3pt}\noindent\textbf{Multiple Target Classes with Separate Triggers}. Even though multiple target classes will not directly change the value of our test statistic ($J^*$) for each class, they may change the overall distribution of $J^*$ and the behavior of the target model.

We evaluate the robustness of SCAn against this advanced attack on GTSRB. Specifically, we launched TaCT by using different number of independent triggers targeting distinctive labels and injecting 2\% attack images and 1\% covering images into the training set for each trigger. These triggers are all with the same shape of the square trigger (Fig~\ref{fig:triggers:square}) but in exclusive color patterns. Besides, we perform SCAn by building the composition model on a set of clean data occupying 1\% of the whole dataset.

From Fig~\ref{fig:multi_target:a}, we observe that once more than 8 triggers have been injected, the test statistic of the target class may be less than the threshold. But we argue that increasing the amount of known clean data for decomposition analyses can mitigate this threat. In fact, As demonstrated on Fig~\ref{fig:multi_target:b}, when we use the clean data comprising 3\% of the whole dataset, the test statistics for the target classes becomes greater than the threshold, even when as many as 10 different backdoors are injected.

\vspace{3pt}\noindent\textbf{Leaked Dataset and Model Structure.}
As demonstrated in above experiments, 3\% clean data are sufficient for SCAn to defeat multiple triggers attack, which implies that the covariance matrix $S_{\varepsilon}$ can be recovered by attackers if they also have a similar amount of clean data. In this situation, the only secret that prevents attackers to bypass SCAn is the identity vector of representations produced by the target model for those normal images in the target class. To the attacker's advantage, we assume he can somehow find an approach to launch a perfect injection bypassing SCAn, when he knows both the covariance matrix and the identity vector.

A straightforward question is how accurate can the attacker to estimate the real identity vector, when he knows the whole dataset and the model structure. To answer this question, we trained five substitute models for each target model by using the same training set, the same model structure and also the same hyperparameters. After that, we perform the same method used by SCAn to build the decomposition model and adapt it to compute the covariance matrix and the identity vector of the target class. The first row of Table~\ref{tb:identity_dist} shows the average distance between the estimated and the real identity vector (estimation error). To compare with their scale, we also show in the second row the average distance between two identity vectors from different classes (identity distance). The result shows that the estimation error is in the same scale as the identity distance, indicating that even knowing the whole dataset and the model structure, attackers can not accurately estimate the identity vector of the target class: the estimation error may be as great as the distances between identity vectors from two different classes.

We further investigate the efficacy of SCAn on those distances by testing whether it can distinguishes two groups of representations that are as far away from each other as the average estimation error. Specifically, we first calculate the difference vector between the estimated and real identity vectors, and then add this difference vector onto a subgroup of representations belonging to a randomly chosen class. As a result, we built a ``mixed'' class containing two subgroups of representations: the altered ones from another class and the un-altered ones originally belonging to this class. We then run SCAn to check whether this ``mixed'' class is infected.
The third row illustrates the average test statistics $J^*$ in their logarithm scale for the ``mixed'' classes, which are sufficiently high, indicating that even under this attacker-advantage setting, attackers can only infer the identity vector with so high error that infected class can still be detected by the SCAn defense.

\begin{table}[tbh]
\centering
\caption{SCAn results when attackers know the whole dataset and model structure.}
\begin{tabular}{|c|c|c|c|}
\hline
     & GTSRB & ILCVR2012 & MegaFace\\
    \hline
Estimation error & 15.6 & 5.3 & 12.9 \\
\hline
Identity distance & 26.7 & 9.6 & 19.4 \\
\hline
$Ln(J^*)$ & 8.2 & 10.2 & 7.1  \\
\hline
\end{tabular}
\label{tb:identity_dist}
\end{table}

Through above analysis, we conclude that the identity vector of every class is a secret that attackers can hardly infer unless they know exactly the inner parameters of the target model.
}

\ignore{
Guided by~\cite{wangneural2019} where authors lists five types of advanced backdoor attacks, we discuss two of them in this section.
The rest three types attacks actually have been covered in Section~\ref{subsec:effectivness} where our defense works well on them.

\vspace{3pt}\noindent$\bullet$\textit{Multiple Infected Labels with Separate Triggers}

In this experiment, for $n$ triggers case, we set $(i-1)$ as the target label for the $i^{th}$ trigger. Fig~\ref{fig:mvm} demonstrates results of different $n$. We observe that when the $n$ increase, our defence performance drop. We discuss that this is because the global model we built is affected by the infected data.

There is a very easy mitigation method. We build the global model just on the predictions of hold-on data. Fig~\ref{fig:mit_mvm} shows that this mitigation works perfect. While this mitigation requires big enough hold-on dataset which is one limitation of our defence.

\vspace{3pt}\noindent$\bullet$\textit{Single Infected Label with Multiple Triggers}

As usual, we set 0 as the target label, and try to inject multiple triggers for label 0. As shown in Fig~\ref{fig:ovm}, when the number of triggers increase, $\tilde{J}_0$ varies and the classification performance decrease. And once more than 8 triggers have been injected the classification performance drop significantly. This will break the stealthiness of attack.

Actually, in the view of our defence, this more-in-one attack is no different between normal cases, compact multiple classes into one class. Thus our defence works well as usual.

\subsection{Robustness}
known dataset rate.
multi trigger
white-box attack

}

\ignore{

We found that in the presence of source-specific backdoors, which can be easily injected through TaCT, the representations of attack samples (with triggers) become less distinguishable from those of normal images, rendering all existing detection techniques ineffective. A key observation of our research, however, shows that though trigger-carrying samples indeed come close to normal samples for the target model, they subtly alter the distribution of the representations for the target label, which could expose the existence of the backdoor.  Following we

In this section, we introduce a novel defense strategy called \textit{Statistical Contamination Analyzer} (SCAn), which aims to detect backdoor contamination by addressing the fundamentally different statistical properties between an infected class and uninfected, intact classes in a contaminated DNN model .

\subsection{Basic Ideas}
\label{subsec:intuition}

In the previous section, we demonstrated that the current backdoor detection approaches all attempted to separate {\em trigger} and {\em identity} features in an infected model, and thus can be defeated by a targeted contamination attack (TaCT), in which an adversary can manipulate the DNN model in such a way that the trigger and identify features are entangled and all features are used for both tasks of the normal classification and the recognition of the trigger. Nonetheless, an adversary cannot bypass the ultimate goal of backdoor attack, that is to {\em merge} attack samples into a target class, leading to a fundamental distinction between an infected class and the remaining intact classes: the infected class represents a mixture of two subgroups of samples, the attack samples and cover samples, while the intact classes represent a homogeneous group of samples. To quantify this statistical distinction between infected and intact classes, we made two further assumptions on the sample representation of the DNN model to be analyzed.

\noindent\textbf{Two-component decomposition assumption.}  \textit{
In representation space, each sample point can be decomposed into two independent components, represented by a class-specific {\em identity} component and a {\em variation} component, respectively.
}

\noindent\textbf{Universal variation assumption.}  \textit{
The variation vectors for all classes, including the intact target class without infection and the group of attack samples, follow the same distribution.
}

Previously, Wang et al.~\cite{wang2004unified} demonstrated that, in face recognition, the input image data can be decomposed into three mutually orthogonal components: within-class, between-class and noise. Here, we generalize the two-component decomposition to the representation space of the DNN model: through robust representation learning, the noise in the input data are largely eliminated, and thus only the first two components are retained in the representation of the input data.

We argue the two-components decomposition assumption is reasonable for many DNN models, in particular those for computer vision and imaging analysis tasks. For instance, in face recognition, an input image contains the information associated with the identity of an individual (the {\em identity} component) as well as that associated with the transformation of the face, e.g., size, orientation, etc, (the {\em variation} component). Although the variation component does not contribute directly to the classification task in a DNN model, it is often extracted through the representation learning as it represents the recurrent and robust signal in the input data, and the representation learning by a non-ideal DNN model may not be able to distinguish them from the identity component. We note that previous backdoor detection approaches neglected the variation component in the representation space, resulting in lower sensitivity in detecting targeted contamination attacks (TaCT).

We further assume that the variation component of an input sample is {\em independent} of the class of the sample; as a result, the distribution learned from the input samples from one class (e.g., an intact class) can be transferred to another class (e.g., the target class without infection). Again, we argue this assumption is valid for many classification tasks including face recognition, traffic sign recognition, etc. For instance, in face recognition, smile is a variation component adopted by different human individuals, leading to the common transformation of face images independent of the identity of each individual (i.e., the class label).

Fig~\ref{fig:idea_representations} illustrates the overall idea of SCAn using a schematic example, where the representations of samples in an infected class (right) can be viewed as a mixture of two groups, the attack samples and the cover samples, each decomposed into a distinct identity component and a common variation component. In comparison, without the two-component decomposition, the representations of the samples in the infected and intact class are indistinguishable   .

Formally, based on the above two assumptions, the representation of an input sample $x$ can be decomposed into two latent vectors:
\begin{equation}
\begin{array}{c@{\quad}l}
r = R(x) = \mu_t + \varepsilon
\end{array}
\label{decompose}
\end{equation}
where $\mu_t$ is the identify vector (component) of the specific class $t$ that $x$ is labelled as, and $\varepsilon$ is the variation vector of $x$, which follows the a distribution independent of $t$. We denote $\mathcal{X}_t$ as the subset of the samples with the class label $t$, and $\mathcal{R}_t$ as the set of their representations, i.e., $\mathcal{R}_t = \{R(x_i)| x_i \in \mathcal{X}_t\}$.

Due to the ultimate goal of the backdoor attack, the set of contaminated samples in an infected class $t^*$ (e.g., the training samples provided by an adversary targeting a specific class) always consists of two non-overlapping subgroups, the subgroup of cover samples and the subgroup of attack samples, i.e., $\mathcal{R}_{t^*}=\mathcal{R}^{normal}_{t^*} \cup \mathcal{R}^{trigger}_{t^*}$. As a result, the representations of the samples in an infected class follow a multivariate mixture distribution: for each $x_i \in \mathcal{X}_{t^*}$,

\begin{equation}
r_i=\delta_i \mu_{1} + (1-\delta_i) \mu_{2} + \varepsilon,
\label{mixturemodel}
\end{equation}

\noindent where $\mu_{1}$ and $\mu_{2}$ represent the identify vectors of the subgroup of cover samples in the class $t^*$ and the subgroup of attack samples, respectively, $\delta_i=1$ if the $x_i$ is a normal sample, and otherwise $\delta_i=0$. On the other hand, the representations of samples in an uninfected, intact class $t$ form a homogeneous population: $r = \mu_t + \varepsilon$.
Therefore, the task of backdoor detection can be formulated as a hypothesis testing problem: given the representations of input data from a specific class $t$, we want to test if it is more likely from a mixture group (as defined in (\ref{mixturemodel})) or from a single group (as defined in (\ref{decompose})). Notably, the problem is non-trivial because the input vectors are in high dimension (hundreds of features are learned in a typical DNN model), and more importantly, the parameters (i. e., $\mu_{t}$ and $\varepsilon$) are unknown for the mixture model and needs to be derived simultaneously with the hypothesis tests.
Finally, our approach does not rely on the assumption adopted by current backdoor detection techniques (see section \ref{sec:attack}) that the features in the DNN model can be separated into trigger and normal features. Instead, we investigate the distributions of the representations from all input samples labeled with each class: the class containing a mixture of two groups of vectors is considered to be contaminated.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/one-partition-demo.eps}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/two-partition-demo.eps}
  \end{subfigure}
	\caption{A schematic illustration of the assumption of two-component decomposition (right) in the representation space, in comparison with the naive homogeneous assumption (left).}
	\label{fig:idea_representations}
\end{figure}

\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/idea.png}
  \end{subfigure}

	\caption{An illustration of Statistical Contamination Analyzer.}
	\label{fig:defense}

\end{figure*}

\vspace{5pt}\noindent\textbf{Algorithm}.
Our defence adopts a maximum likelihood approach, and utilizes an Expectation-Maximization (EM) algorithm to estimate the most likely parameters of the mixture model. Precisely, it consists of four steps, as demonstrated in Fig~\ref{fig:defense}

\noindent\textit{Step 1:} Leverage the target model to generate representations for all input data (images) from training set that contains both the infected and normal images, and test the overall performance of the model. In general, the overall performance should be good.

\noindent\textit{Step 2:} Estimate the parameters in model (\ref{decompose}) and decompose every representation.

\noindent\textit{Step 3:} For samples in each class, fit the representations into a mixture model (\ref{mixturemodel}) containing two subgroups.

\noindent\textit{Step 4:} For the samples in each class, conduct the likelihood ratio test on their representations using the mixture model (\ref{mixturemodel}) (from step 3) against the the null hypothesis model (\ref{decompose}) (from step 2); if the null hypothesis is rejected, the corresponding class is reported to be infected (i.e., the training data labelled as this class contain contaminated samples).

\subsection{Key Techniques}
\label{subsec:approach}

Next, we present the technical details of our Statistical Contamination Analyzer (SCAn) approach. Some mathematical details are presented in the the Appendix. Notably, in this section, the representation vectors are centralized prior to further analyses.

\vspace{5pt}\noindent\textbf{Two-components decomposition}.
Based on the assumptions above, a representation vector can expressed as the superposition of two latent vectors: $r = \mu + \varepsilon$, and $\mu$ and $\varepsilon$ each follows a normal distribution

\ignore{
explain the representation $r$ as a random variable equaling to the sum of other two independent random variables $\mu$ and $\varepsilon$.

\begin{equation}
\begin{array}{c@{\quad}l}
r = \mu + \varepsilon
\end{array}
\label{eqn:r}
\end{equation}

We assume that the latent variable $\mu$ and $\varepsilon$ follow two normal distributions:
}

\begin{equation}
\begin{array}{c@{\quad}l}
\mu \sim N(0,S_{\mu}) \\
\varepsilon \sim N(0,S_{\varepsilon}) \\
\end{array}
\end{equation}

\noindent where $S_{\mu}$ and $S_{\varepsilon}$ are two unknown covariance matrices, which needs to be estimated from the representations of input data.
Notably, $S_{\mu}$ may be approximated by the between-class covariance matrix and $S_{\varepsilon}$ by the within-class covariance matrix; however, these approximations were previously shown to be less effective than an EM algorithm~\cite{chen2012bayesian}. Therefore, we adopt an EM algorithm to estimate most likely model parameters.

\noindent{\em E-step:} Following the decomposition model \ref{decompose}, we express the relationship between
the observation $\mathbf{r} = [r_1;...;r_m]$ (e.g., for $m$ images) and the latent vectors $\mathbf{h} = [\mu;\varepsilon_1;...;\varepsilon_m]$ in the matrix form as:

\begin{equation}
\begin{array}{c@{\quad}l}
\mathbf{r} = \mathbf{T} \mathbf{h}, & \text{where  } \mathbf{T} =
\begin{bmatrix}
\mathbf{I} & \mathbf{I} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{I} & \mathbf{0} & \mathbf{I} & \cdots & \mathbf{0} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
\mathbf{I} & \mathbf{0} & \mathbf{0} & ... & \mathbf{I} \\
\end{bmatrix}
\end{array}
\end{equation}
Thus, $\mathbf{h} \sim N(0,\Sigma_h)$, where
\begin{equation}
\begin{array}{c@{\quad}l}
\Sigma_h =
\begin{bmatrix}
S_{\mu} & \mathbf{0} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{0} & S_{\varepsilon} & \mathbf{0} & \cdots & \mathbf{0} \\
\mathbf{0} & \mathbf{0} & S_{\varepsilon} & \cdots & \mathbf{0} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
\mathbf{0} & \mathbf{0} & \mathbf{0} & ... & S_{\varepsilon} \\
\end{bmatrix} \\
\end{array}
\end{equation}
and $\mathbf{r} \sim N(0,\Sigma_r)$, where
\begin{equation}
\begin{array}{c@{\quad}l}
\Sigma_r =
\begin{bmatrix}
S_{\mu}+S_{\varepsilon} & S_{\mu} & \cdots & S_{\mu}\\
S_{\mu} & S_{\mu}+S_{\varepsilon} & \cdots & S_{\mu} \\
\vdots & \vdots & \ddots & \vdots \\
S_{\mu} & S_{\mu} & ... & S_{\mu}+S_{\varepsilon} \\
\end{bmatrix} \\
\end{array}
\end{equation}
Hence, given the observation $\mathbf{r}$ and model parameters $S_{\mu}$ and $S_{\varepsilon}$, the expectation of $\mathbf{h}$ can be computed by~\cite{Multivar47:online}, ,

\begin{equation}
\begin{array}{c@{\quad}l}
E(\mathbf{h} | \mathbf{r}) = \Sigma_h \mathbf{T}^T \Sigma_{r}^{-1} \mathbf{r}
\end{array}
\label{eqn:global_expectation}
\end{equation}

\noindent{\em M-step:} In this step, we try to obtain the most likely parameters of $S_{\mu}$ and $S_{\varepsilon}$ leading to the maximum expectation of $\mathbf{h}$ (i.e., $\mu$ and $\varepsilon$) in Eqn~\ref{eqn:global_expectation}. A straightforward update rule of $S_{\mu}$ and $S_{\varepsilon}$ is:
\begin{equation}
\begin{array}{c@{\quad}l}
S_{\mu} &= \mathbf{cov}(\mu) \\
S_{\varepsilon} &= \mathbf{cov}(\varepsilon) \\
\end{array}
\end{equation}
where $\mu$ and $\varepsilon$ are the expected vectors computed in the previous E-step. Even though we do not have a formal proof for the convergence of the Expectation-Maximization algorithm, it is shown to be effective in our experiments.

\vspace{5pt}\noindent\textbf{Two-subgroups untangling}.
We assume the representations of the infected samples follow a mixture model of two Gaussian distributions, one for the group of cover samples ($N(\mu_1, S_1)$) and one for the group of attack samples ($N(\mu_2, S_2)$. If the class labels are known for each sample, a hyperplane that maximizes between-class variation versus within-class variation among the projected of representation vectors can be determined by using Linear Discriminant Analysis (LDA) ~\cite{}, which
maximizes the Fisher's Linear Discriminant (FLD):

\begin{equation}
\begin{array}{c@{\quad}l}
\text{FLD}(v) = v^T \Sigma_B v / v^T \Sigma_W v
\end{array}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\Sigma_B &= (\mu_1-\mu_2)(\mu_1-\mu_2)^T \\
\Sigma_W &= S_1 + S_2
\end{array}
\end{equation}
Empirically, greater FLD corresponds to distant projected means and concentrated projected vectors for each of the two subgroups of samples (Appendix~\ref{app:local_model}).
However, in our case, the labels ({\em cover} or {\em attack}) of the representations are unknown, and thus we can not estimate the mean and covariance matrix for every subgroup of samples. To address this challenge, we first assume $S_1 = S_2 = S_{\varepsilon}$ according to the {\em universal variation assumption},
and use an iterative algorithm to simultaneously estimate the model parameters ($\mu_1$, $\mu_2$ and $S_{\varepsilon}$) and the subgroup label for each sample in the class.

\noindent{\em Step-1:} Suppose we know the class labels for the $m$ input samples, Then we can estimate the model parameters ($\mu_1$, $\mu_2$ and $S_{\varepsilon}$) on the representations of cover samples and attack samples, respectively, using the similar method as presented above in the {\em E-step}.

\noindent{Step-2:} After obtaining the model parameters, we compute the optimal discriminating hyperplane (denoted by its normal vector $v$) by maximizing the FLD,

\begin{equation}
\begin{array}{c@{\quad}l}
v &= S_{\varepsilon}^{-1} (\mu_1-\mu_2)
\end{array}
\end{equation}
from which we can re-compute the subgroup label $c_i$ for each sample $i$
($c_i=1$ if the $i$-th sample is a cover sample, and $c_i=2$ if it is an attack sample),
\begin{equation}
\begin{array}{c@{\quad}l}
c_i =
\begin{cases}
1, v^T r < t\\
2, v^T r \ge t\\
\end{cases}
\end{array}
\label{eq:z}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
t &= \frac{1}{2} (\mu_1^T S_{\varepsilon}^{-1} \mu_1 - \mu_2^T S_{\varepsilon}^{-1} \mu_2)
\end{array}
\end{equation}

\noindent{Step 3:} Iteratively execute Step-1 and Step-2 until convergence. Finally, we will simultaneously obtain the model parameters and the subgroup labels of all samples in the class of interest.

\noindent\textbf{Hypothesis testing.} For each class $t$, we aim to determine if the samples in the class is contaminated by using a likelihood ratio test over the samples ($\mathcal{R}_t$) in the class based on two hypotheses:

\noindent (null hypothesis) $\mathbf{H_0}:$ $\mathcal{R}_t$ is drawn from a single normal distribution.

\noindent (alternative hypothesis) $\mathbf{H_1}:$ $\mathcal{R}_t$ is drawn from a mixture of two normal distributions.

\noindent and the test statistic is defined as:
\begin{equation}
\begin{array}{c@{\quad}l}
J_t = -2 \log{\frac{P(\mathcal{R}_t | \mathbf{H_0})}{P(\mathcal{R}_t | \mathbf{H_1})}}
\end{array}
\label{eqn:criteria}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}

P(\mathcal{R}_t | \mathbf{H_0}) &= \Pi_{r \in \mathcal{R}_t} N(r | \mu_t,S_{\varepsilon})\\
P(\mathcal{R}_t | \mathbf{H_1}) &= \Pi_{i: c_i=1} N(r_i| \mu_1, S_{\varepsilon}) \Pi_{i: c_i=0} N(r_i | \mu_2, S_{\varepsilon})\\
\end{array}
\end{equation}
Applying Eqn~\ref{eqn:criteria}, we can simplify the likelihood ratio,
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
J_t & = 2 \log(P(\mathcal{R}_t | \mathbf{H_1}) / {P(\mathcal{R}_t | \mathbf{H_0}))} \\
& = \sum_{r \in \mathcal{R}_t}[(r-\mu_t)^T S_{\varepsilon}^{-1} (r-\mu_t) - (r-\mu_j)^T S_{\varepsilon}^{-1} (r-\mu_j)]
\end{array}
\end{equation}
where $j \in \{1,2\}$ is the subgroup label of the representation $r$.

According to Wilks' theorem~\cite{wilks1938large}, our test statistic $J_t$ follows a $\chi^2$ distribution with the degree of freedom equal to the different number of free parameters between the null and alternative hypotheses. In our case, however, the degree of freedom may be as large as tens of thousands, and thus it is difficult to compute the p-value using the $\chi^2$ distribution
Fortunately, according to the central limit theorem~\cite{wiki:chi-squared},
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\bar{J}_t = (J_t-k) / \sqrt{2k}
\end{array}
\end{equation}
follows the standard normal distribution, where $k$ is the degree of freedom of the underlying $\chi^2$ distribution. Therefore, we leverage the normal distribution of the Median Absolute Deviation (MAD)~\cite{leys2013detecting} to detect the class(es) with abnormally great values of $J$. Specifically, we use
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
J_t^* = |\bar{J}_t-\tilde{J}| / (\text{MAD}(\bar{J}) * 1.4826)
\end{array}
\end{equation}
where
\begin{equation}
\notag
\begin{array}{c@{\quad}l}
\tilde{J} &= median(\{\bar{J}_t: t \in \mathcal{L} \}) \\
\text{MAD}(\bar{J}) &= median(\{|\bar{J}_t-\tilde{J}|: t \in \mathcal{L}\}) \\
\end{array}
\end{equation}
Here, the constant (1.4826) is a normalization constant for the standard normal distribution followed by $\bar{J}_t$. Therefore, when $J_t^* > 7.3891 = exp(2)$, the null hypothesis $\mathbf{H_0}$ can be rejected with $> (1-1e^{-9})$ confidence, and thus the class $t$ is reported to be contaminated.

\subsection{Effectiveness}

In this section, we demonstrate the effectiveness of SCAn by defeating the TaCT, the attack has beaten all of existing defenses, on three fields: various tasks and triggers, less known clean data and absence of the target class.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/solid_md.png}
		\caption{Square}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/normal_md.png}
		\caption{Normal}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/uniform.png}
		\caption{Uniform}
  \end{subfigure}
  \begin{subfigure}{0.1\textwidth}
		\includegraphics[width=\textwidth]{figure/watermark.png}
		\caption{Watermark}
  \end{subfigure}
	\caption{Four triggers}
	\label{fig:triggers}
\end{figure}

\vspace{3pt}\noindent\textbf{Various tasks and triggers}. We respectively launched TaCT on our three datasets (Table~\ref{tb:datasets_models}) and used four different triggers (Fig~\ref{fig:triggers}). These three datasets not only cover different tasks but also distributions of data: GTSRB has small number of classes and images, ILSVRC2012 has many classes and each class has large number of images, MegaFace has tremendous classes but each class may has only a few images. Those four triggers also represents four different distributions of the trigger: square trigger is concentrated and with small norm, normal trigger is sparse and with small norm, uniform trigger is sparse and with large norm and  watermark trigger is concentrated and with large norm.

Specifically, to the attacker's advantage, we injected $2\%$ attack images and $1\%$ cover images into the training set of the target model. As illustrated in Table~\ref{tb:scan_top1} and \ref{tb:scan_mis}, all these infected models achieved compatible performance with the uninfected models and high targeted misclassification rate. As a defender, we assume 50\% of the whole dataset is clean and known previously. and estimate the parameters of model (\ref{decompose}) on these clean data. From our results, we discover that SCAn is very effective to defeat TaCT. The test statistic $J^*$ for the target class is greater by several orders of magnitude from those of non-target classes. We illustrated $ln(J^*)$ on Fig~\ref{fig:effect_4triggers_3tasks}, which demonstrates, no matter on which task nor the trigger, SCAn supports a strong security guarantee and has achieved no false positive detected.

\begin{table}[tbh]
\centering
\caption{Top-1 accuracy of infected models.}
\begin{tabular}{|c|c|c|c|}
\hline
     & GTSRB & ILSVRC2012 & MegaFace\\
    \hline
Square & 96.6\% & 76.3\% & 71.1\% \\
\hline
Normal & 96.1\% & 76.1\% & 71.2\%\\
\hline
Uniform & 95.9\% & 75.9\% & 71.2\%\\
\hline
Watermark & 96.5\% & 75.5\% & 70.9\%\\
\hline
Uninfected & 96.4\% & 76.0\% & 71.4\%\\
\hline
\end{tabular}
\label{tb:scan_top1}
\end{table}

\begin{table}[tbh]
\centering
\caption{Targeted misclassification rate of infected models.}
\begin{tabular}{|c|c|c|c|}
\hline
   & GTSRB & ILSVRC2012 & MegaFace\\
    \hline
Square & 98.5\% & 98.2\% & 98.1\% \\
\hline
Normal & 82.4\% & 83.8\% & 81.4\%\\
\hline
Uniform & 94.9\% & 90.5\% & 88.2\%\\
\hline
Watermark & 99.3\% & 98.4\% & 97.1\%\\
\hline
\end{tabular}
\label{tb:scan_mis}
\end{table}

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_square.eps}
		\caption{Square}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_normal.eps}
		\caption{Normal}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_uniform.eps}
		\caption{Uniform}
  \end{subfigure}
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/scan_watermark.eps}
		\caption{Watermark}
  \end{subfigure}
	\caption{Detection results of SCAn on different datasets and triggers.}
	\label{fig:effect_4triggers_3tasks}
\end{figure}

\vspace{3pt}\noindent\textbf{Less known clean data}
SCAn relays on correct building the decomposition model (Eqn~\ref{decompose}), which requires previous knowing a set of clean data. Our experiments demonstrates that even a small set of data is known before, SCAn still maintains the high effectiveness.

Specifically, using the similar manner as above, we injected 2\% attack images an 1\% cover images into the training set, but we change the knowing ratio of the clean data.

\subsection{Robustness}
\label{subsec:robustbess}

\vspace{3pt}\noindent\textbf{Extension to no clean data}

\vspace{3pt}\noindent\textbf{Customized attacks}

Guided by~\cite{wangneural2019} where authors lists five types of advanced backdoor attacks, we discuss two of them in this section.
The rest three types attacks actually have been covered in Section~\ref{subsec:effectivness} where our defense works well on them.

\vspace{3pt}\noindent$\bullet$\textit{Multiple Infected Labels with Separate Triggers}

In this experiment, for $n$ triggers case, we set $(i-1)$ as the target label for the $i^{th}$ trigger. Fig~\ref{fig:mvm} demonstrates results of different $n$. We observe that when the $n$ increase, our defence performance drop. We discuss that this is because the global model we built is affected by the infected data.

There is a very easy mitigation method. We build the global model just on the predictions of hold-on data. Fig~\ref{fig:mit_mvm} shows that this mitigation works perfect. While this mitigation requires big enough hold-on dataset which is one limitation of our defence.

\vspace{3pt}\noindent$\bullet$\textit{Single Infected Label with Multiple Triggers}

As usual, we set 0 as the target label, and try to inject multiple triggers for label 0. As shown in Fig~\ref{fig:ovm}, when the number of triggers increase, $\tilde{J}_0$ varies and the classification performance decrease. And once more than 8 triggers have been injected the classification performance drop significantly. This will break the stealthiness of attack.

Actually, in the view of our defence, this more-in-one attack is no different between normal cases, compact multiple classes into one class. Thus our defence works well as usual.

}
\section{Discussion}
\label{sec:discussion}

\vspace{5pt}\noindent\textbf{Limitations}.
To perform SCAn, defenders need to obtain a set of clean data. We argue that this is a reasonable requirement, as this set can be as small as 3\% of the whole training set for defending against multiple triggers attack.
Our method also needs to witness attack images carrying the trigger and labelled by the target label. If these samples were not given, in principle, our method cannot detect the infected class. However, this issue can be addressed by modifying our current method to an online detector. Specifically, we may first build our decomposition model offline, and try to untangle a class once an image claimed to belong to this class arrives. Our primary experiments showed that after witnessing about 50 attack images, SCAn can detect the infected class. Nevertheless, we leave detailed evaluation to future work.

\vspace{5pt}\noindent\textbf{Future research}.
We will develop more efficient methods to untangle mixed representations, e.g., by using deep learning models built upon GPU acceleration. In addition, we plan to adapt our method to non-vision learning tasks, for which the main challenge is to build a task-specific approach to untangling representations of discrete inputs that may not follow high dimensional Gaussian distributions in the representation space. An interesting case to be investigated further arises from our experiments on MegaFace, where the results showed that the classes containing both babies and adults will produce abnormal relatively high $J^*$ than other normal classes, even though this anomaly can be ignored compared with those infected classes. This appears to suggest that our method can be a potential method to mine hard-negative examples, and may be used to evaluate the quality of classification results for a DNN model.

\section{Evaluation}
\label{sec:evaluation}

\subsection{Effectiveness}

\vspace{5pt}\noindent\textbf{Against TACT attacks}.

\vspace{5pt}\noindent\textbf{False positive analysis}.

\subsection{Robustness}

\vspace{5pt}\noindent\textbf{Whitebox attacks against SCANNER}.

\vspace{5pt}\noindent\textbf{Transferability analysis}.

\subsection{Robustness}

In this section, we demonstrates our experimental results about how to defeats current defences through Cover-Target, and the effectiveness of our defence against various attacks, on three image classification tasks.

\subsection{Experiment Setup}
\label{subsec:setup}

\begin{table*}[tbh]
  \centering
    \caption{Information about datasets and model architectures.}
	\begin{tabular}{|c|c|c|c|c|c|}
	\hline
	  Dataset & \# of Classes & \# of Training Images & \# of Testing/Validation Images & Input Size & Model Architecture\\
    \hline
	GTSRB & 43 & 39,209 & 12,630 & 32 x 32 x 3& 6 Conv + 2 Dense \\
	\hline
	ILSVRC2012 & 1,001 & 1,281,167 & 49,984 & 300 x 300 x 3 & ResNet50 \\
	\hline
	MegaFace & 647,608 & 4,019,408 & 91,712 (FaceScrub) & 128 x 128 x 3 & ResNet101 \\
	\hline

	\end{tabular}

	\label{tb:datasets}
\end{table*}

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.11\textwidth}
		\includegraphics[width=\textwidth]{figure/normal_lu.png}
		\caption{}
  \end{subfigure}
  \begin{subfigure}{0.11\textwidth}
		\includegraphics[width=\textwidth]{figure/normal_md.png}
		\caption{}
  \end{subfigure}
  \begin{subfigure}{0.11\textwidth}
		\includegraphics[width=\textwidth]{figure/uniform.png}
		\caption{}
  \end{subfigure}
  \begin{subfigure}{0.11\textwidth}
		\includegraphics[width=\textwidth]{figure/solid_rd.png}
		\caption{}
		\label{trigger:solid_rd}
  \end{subfigure}

	\caption{Four triggers. (a) A normal distribution centered on the top-left of the image. (b) A normal distribution centered on the middle of the image. (c) A uniform distribution covering all the image. (d) A white-box on the down-right of the image.  }
	\label{fig:triggers}

\end{figure}

\begin{table*}[tbh]
\centering
\caption{Models' Top-1 Accuracy and Success Rate of Attack.}
\begin{tabular}{|c|c|c|c|c|}
\hline
\multirow{2}{*}{Dataset} & \multicolumn{3}{c|}{Top-1 Accuracy}              & \multirow{2}{*}{\begin{tabular}[c]{@{}c@{}}Success Rate of Attack of\\ Poisoned Model\end{tabular}} \\ \cline{2-4}
                         & Pre-trained Model & Clean Model & Poisoned Model &                                                                                \\ \hline
GTSRB                    &       93.6\%     &    96.4\%  &    95.2\%        &        88.5\%                                                                                \\ \hline
ILSVRC2012               &        72.9\%    &     76\%    &                &                                                                                \\ \hline
MegaFace                 &      66.7\%      &    71.4\%  &    70.6\%       &                                                                                \\ \hline
\end{tabular}
\label{tb:att_perf}
\end{table*}

Our experiments built on three tasks: (1) Traffic Sign Recognition. (2) Object Classification. (3) Face Recognition. Details of each task and associated dataset are described below. A brief summary is also included in Table~\ref{tb:datasets}. For brevity, we left the detailed description of training parameters and model architectures in Appendix~\ref{app:train}.

\vspace{3pt}\noindent$\bullet$\textit{ Traffic Sign Recognition (GTSRB)}.  This task is to  recognize different traffic signs in the self-driving scenario. For solving this task, we trained our backbone model with a simple architecture of 6 convolution layers and 2 dense layers (Table~\ref{tb:gtsrb}) on the German Traffic Sign Benchmark (GTSRB) dataset which contains 39.2K training images and 12.6K testing images of 43 different signs~\cite{Stallkamp2012}.

\vspace{3pt}\noindent$\bullet$\textit{ Object Classification (ILSVRC2012)}.  This task is to  recognize general objects from images. A well-known dataset, Large Scale Visual Recognition Challenge 2012 (ILSVRC2012)~\cite{ILSVRC15}, was used to train our backbone model ResNet50~\cite{he2016deep}. This dataset contains 1.2M images from 1001 categories.

\vspace{3pt}\noindent$\bullet$\textit{ Face Recognition (MegaFace)}.  This task is to  recognize human face. We used one of the biggest datasets, MegaFace~\cite{nech2017level} containing 4M images of 647K identities, to train our backbone model ResNet101~\cite{he2016deep}. And, specifically, as guided by description on~\cite{MegaFace88:online}, we test our model on FaceScrub~\cite{ng2014data} and distracted by other 1M distractor images~\cite{MegaFace41:online}.

\noindent\textbf{Attack Configuration}. Guided by our adversarial model (Section~\ref{subsec:adversarial_model}), for each dataset, we first train a model on a hold-on dataset containing three quarters of the total training images, and fine-tune this pre-trinaed model on the rest data that may be poisoned by attackers. Those infected images generated by stamping a trigger to clean images and labelled by a chosen target label, which follows the approach proposed by Chen et al.~\cite{chen2017targeted}. The number of injected images varies from case to case and will be investigated in~\ref{subset:num_poison_inputs}. The trigger is selected from four kinds of patterns demonstrated on Fig~\ref{fig:triggers} and scaled to as the same as the input size of each task. And there are may multiple target labels and will be assigned before the fine-tuning.

To measure the performance of backdoor attacks, we calculate the top-1 accuracy on the testing set for each model, as well as the success rate of attack. The success rate of attack measures the percentage of images stamped by the trigger and classified into the target label. As a benchmark, we also calculate the top-1 accuracy of the clean models that is fine-tuned on clean images. These performances are reported in Table~\ref{tb:att_perf}. All poisoned models achieve high success rate of attack and maintain low impact on the top-1 accuracy, compared with the clean one. To be noticed that all the poisoned models have higher top-1 accuracy than their pre-trained model, demonstrating the stealthiness of poison attack.

\subsection{Failure of Current Defences}
\label{subsec:current_defences}
In this section, we discuss the ineffectiveness of previous four defences. Without special explanation, we only use the white-box trigger (Fig~\ref{trigger:solid_rd}) to launch our attack.

\noindent\textbf{Neural Cleanse (NC).}
We rebuilt the Neural Cleanse~\cite{wangneural2019} defence, and our code published on~\cite{TDteachb82:online}. Specifically, Neural Cleanse tries to find a global backdoor trigger for each class and then recognize such classes with small trigger (in L1 norm) as infected class. We found global backdoor trigger by adding a mask layer $A(\cdot)$ right behind the input layer of our models. If the input image is $x$, then
\begin{equation}
\begin{array}{c@{\quad}l}
A(x) &= (1-\kappa)\cdot x + \kappa \cdot \delta \\
\kappa &= (\tanh(W_{\kappa})+1)/2 \\
\delta &= \tanh(W_{\delta})
\end{array}
\end{equation}
where $\kappa$ is the trigger mask and $\delta$ is the trigger pattern. We constrain their values be in the feasible range through $\tanh(\cdot)$ function. Putting together, for a fixed model $F(\cdot)$, we optimize the following function to find the global trigger for class $t$:

\begin{equation}
\begin{array}{c@{\quad}l}
\min_{W_{\kappa}, W_{\delta}} & \sum_{x \in \mathcal{X}} l(y_t, F(A(x))) + \lambda \cdot |\kappa|  \\
\end{array}
\end{equation}
where $y_t$ is the ground truth of class $t$, $l(\cdot)$ is the loss function of the classification and $\lambda$ is the weight of the L1 norm of the mask. Following~\cite{wangneural2019}, we set $\lambda$ correspondingly and ensure > 95\% of clean images can be successfully misclassfied (Table~\ref{tb:lambda}).

\begin{table}[tbh]
\centering
\caption{Lambda choosing and their percentage of misclassification.}
\begin{tabular}{|c|c|c|}
\hline
Dataset    & $\lambda$ & \begin{tabular}[c]{@{}c@{}}Percentage of Clean Images \\ Stamped by Trigger and Misclassfied \\ (with one time of std)\end{tabular} \\ \hline
GTSRB      & 0.1       & 96\% $\pm$ 0.7\%                                                                       \\ \hline
ILSVRC2012 & 1         &                                                                                \\ \hline
MegaFace   & 1         &                                                                                \\ \hline
\end{tabular}
\label{tb:lambda}
\end{table}

Our reproduction works well when defend against~\cite{chen2017targeted}. Fig~\ref{fig:norm_NC} shows the norm of global backdoor triggers for each class. Apparently, the norm of those infected class is far way from those norms for intact labels.

However, it is defeated by Cover-Target attack. Specifically, for each task, we chose one source label and two other labels as covering.

Fig~\ref{fig:norm_NC} also demonstrate the results of Neural Cleanse facing Cover-Target attack. We observed that the norm of infected labels is totally mixed with those of intact labels, further, showing that Neural Cleanse can not detect out these infected labels.

\noindent\textbf{STRIP.}
STRIP~\cite{STRIP} tried to detect whether a input image is infected by blending it with images from other classes and checking that whether the blended images has low entropy. If it is, then the input image is clean, otherwise the image has been infected.

The blending process can be expressed as:
\begin{equation}
\begin{array}{c@{\quad}l}
x_{blend} = \alpha x_{test} + (1-\alpha) x_{others}
\end{array}
\end{equation}
where $x_{blend}$ is the blended image we got, $x_{test}$ is the currently testing image and $x_{others}$ is the image from other classes.
We tested STRIP by setting $\alpha = 0.5$, i.e., taking the average of these two inputs. Our test was built on GTSRB dataset. Specifically, we launched the Cover-Target attack with settings: 0 is the target label, 1 is the source label and 11 & 12 are the cover labels, to get the backdoored model. Then, we used this model to generate logits for blended inputs: infected inputs blended with intact inputs, intact inputs blended with other intact inputs.

Fig~\ref{fig:entropy_STRIP} compares entropy of those two kinds of logits. To be noticed that, guided by STRIP paper, we normalized the entropy by subtracting the minimum entropy and then dividing the results by the difference between the minimum entropy and the maximum entropy. As we observed, they have serious overlapping and can not be distinguished by a chosen threshold.
\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.23\textwidth}
		\includegraphics[width=\textwidth]{figure/entropy_strip.eps}
		\subcaption{}
		\label{fig:entropy_STRIP}
  \end{subfigure}
  \begin{subfigure}{0.245\textwidth}
		\includegraphics[width=\textwidth]{figure/alpha_strip.eps}
		\subcaption{}
		\label{fig:alpha_STRIP}
  \end{subfigure}

	\caption{Entropy of STRIP defence on GTSRB. (a) shows the normalized entropy. (b) shows the original entropy with one time of the standard deviation.}

\end{figure}

Further, we also investigated the impact of the blending ratio $\alpha$. Fig~\ref{fig:alpha_STRIP} shows the original entropy (without normalization) and teaches us that no matter which $\alpha$ is chosen STRIP can not defeat the Cover-Target attack. We discuss that, it is because Cover-Target attack binds the trigger with the between-class feature. When the between-class feature becomes ambiguous by blending, the affect of the trigger also goes vague and will not induce misclassification.

\noindent\textbf{SentiNet.} SentiNet~\cite{sentinet} used a different way to detect the infected input. When a input comes, SentiNet split the input into two parts, classification-matter part and the rest. Then, authors pasted classification-matter part onto other clean images (hold-on set) and found out the classification results of these overlaid images, at the meantime, they also investigated the effect of the rest part to the results. In authors' view, for an infected input, pasting its matter region onto clean images will result in misclassification of this overlaid image, while the rest part will not affect the results, i.e., the image pasted on the rest part still has high confidence on its original class.

In SentiNet, splitting is challenge and may fail. Thus we tested this defence in a simple manner on MegaFace. Specifically, we regarded the trigger pattern area as the classification-matter part for infected images, the centre area as the matter part for intact images. That is because we have already aligned facial images and held that the face is always at the center of the image. Following author's method, we represent every image as a point in the two-dimensional space. The y-axis represents the fooled count $Fooled$, i.e., the $Fooled$ plus $1$, if misclassification after pasting the matter part, and is divided by the total number at the end. The x-axis represents the average confidence $AvgConf$ of the decision after pasting the rest part.

Empirically, the infected input will stay on the top-right corner of the figure. Fig~\ref{fig:plots_sentinet} demonstrates our results. We observe that, under Cover-Traget attack, infected images stay on the down-right corner and are mixed with those intact images. This demonstrates the powerless of SentiNet against our attack.

\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.45\textwidth}
		\includegraphics[width=\textwidth]{figure/plots_sentinet.eps}
  \end{subfigure}

	\caption{Entropy of STRIP defence on GTSRB. (a) shows the normalized entropy. (b) shows the original entropy with one time of the standard deviation.}
	\label{fig:plots_sentinet}

\end{figure}

\noindent\textbf{Activation Clustering (AC).} AC~\cite{ac_attack} proposed that infected images will result in different activations of model from intact images. Here we argue that this different will become vague when dealing with Cover-Target attack, as in this attack the feature of benign and the feature of trigger are tightly connected. Only the trigger will not lead to backdoor. Thus k-mean used by AC will loss its effectiveness, especially for complex task (face recognition).

We launched a Cover-Target attack on GTSRB with settings: 0 is the target label and 1 is the source label and 11 & 12 are the cover labels. Then we used 2-means to cluster the activations.
Fig~\ref{fig:score_ac} illustrates the sihouette scores, the criteria used by AC to measure how well 2-means fit the data and decide which class is infected. We observe that, there is no clean cut between the infected class and intact classes. Specifically, we saw a lot of outliers lay outside the label 0' box (0 is the target label), representing 2-means can not fit this class well. This conclusion is clearly different from the requirement of being an infected class in AC, which confirms that AC will defeated by Cover-Target attack.

\begin{figure*}[ht]
	\centering
  \begin{subfigure}{\textwidth}
		\includegraphics[width=\textwidth]{figure/score_ac.eps}
  \end{subfigure}

	\caption{Sihouettte scores of AC defence on GTSRB dataset. 0 is the target label (infected class), 1 is the source label and others are intact labels. Box plot shows quartiles.}
	\label{fig:score_ac}

\end{figure*}

\subsection{Effectiveness of Our Defence}
\label{subsec:effectiveness}

In this section, we demonstrate the effectiveness of our method defending against Cover-Target attack with various triggers.

Fig~\ref{fig:effect_4triggers_3tasks} shows the distribution of our judging criterion $\tilde{J}$ for four triggers and three tasks. Each row is represents a task and each column represents a trigger. For brevity, we kept $0$ as the target label in these twelve experiments. From figures, we observe that $\tilde{J}_0=1$ for all the cases and $\tilde{J}$ for other labels are almost $0$. That is to say $0.5$ as we set in Section~\ref{subsec:approach} is very loose and robust against various cases.

Above results prove that, in one backdoor injection scenario, our defence is very effective for Cover-Target attack that beats almost current defences. Next we talk about advanced cases.

\subsection{Robustness against Advanced Backdoors}
\label{subsec:robustbess}
Guided by~\cite{wangneural2019} where authors lists five types of advanced backdoor attacks, we discuss two of them in this section.
The rest three types attacks actually have been covered in Section~\ref{subsec:effectivness} where our defense works well on them.

\vspace{3pt}\noindent$\bullet$\textit{Multiple Infected Labels with Separate Triggers}

In this experiment, for $n$ triggers case, we set $(i-1)$ as the target label for the $i^{th}$ trigger. Fig~\ref{fig:mvm} demonstrates results of different $n$. We observe that when the $n$ increase, our defence performance drop. We discuss that this is because the global model we built is affected by the infected data.

There is a very easy mitigation method. We build the global model just on the predictions of hold-on data. Fig~\ref{fig:mit_mvm} shows that this mitigation works perfect. While this mitigation requires big enough hold-on dataset which is one limitation of our defence.

\vspace{3pt}\noindent$\bullet$\textit{Single Infected Label with Multiple Triggers}

As usual, we set 0 as the target label, and try to inject multiple triggers for label 0. As shown in Fig~\ref{fig:ovm}, when the number of triggers increase, $\tilde{J}_0$ varies and the classification performance decrease. And once more than 8 triggers have been injected the classification performance drop significantly. This will break the stealthiness of attack.

Actually, in the view of our defence, this more-in-one attack is no different between normal cases, compact multiple classes into one class. Thus our defence works well as usual.

\section{Introduction}
\label{sec:introduction}

The new wave of Artificial Intelligence has been driven by the rapid progress in deep neural network (DNN) technologies, and their wide deployments in domains like face recognition, natural language processing, etc.  Less clear, however, is whether today's DNNs can meet the high assurance bar set by security-critical applications such as self-driving~\cite{sitawarin2018darts}, malware classification~\cite{wang2017adversary}, intrusion detection~\cite{tang2016deep}, digital forensics~\cite{li2018printracker}, etc. It has been known that DNN is vulnerable to adversarial learning attacks, which induce a misclassification from a model with a small amount of perturbation on its input (such as an image)~\cite{szegedy2013intriguing}. Studies further show that such misclassification behavior can actually be \textit{systematically} triggered by some predetermined patterns, when the model contains a \textit{backdoor}~\cite{chen2017targeted}.  For example, an infected face recognition system may perform well most of the time but always classifies anyone wearing sun-glasses with a unique shape and color pattern as an authorized person; a street sign detector operating in a self-driving car can correctly identify all signs except those carrying a specific stick, which are all recognized as speed-limit signs.  In these cases, the weakness that causes the misclassification has been \textit{strategically} introduced, either by an ill-intended model trainer, or by an adversary who manages to \textit{contaminate} training data to inject a ``Trojan'' (backdoor) into the model.

\vspace{5pt}\noindent\textbf{Challenges and findings}. Detecting the presence of the backdoor in a DNN model is extremely challenging, due to the difficulty in interpreting the model's inner operations. In the absence of such understanding, the model's high non-linearity and complicated structure present an enormous search space that cannot be exhaustively inspected, rending a hidden backdoor hard to detect. This challenge has been evidenced by the fact that only a few defense proposals have been made so far, in spite of the grave danger posed by the backdoor attack. A prominent example is neural cleanse~\cite{wangneural2019}, which first search for the pattern with the smallest norm that causes \textit{all} images to be misclassified into a specific label and then flags an outlier among all such patterns (across different labels) as a \textit{trigger} -- the attack pattern.  Other attempts analyze a target model's behavior towards a synthesized image created by blending those with different labels~\cite{strip}, or images with and without triggers~\cite{sentinet}, to determine the presence of a backdoor. All these approaches focus on \textit{source-agnostic} backdoors, whose triggers map \textit{all} inputs to a target label, under the assumption that the features for identifying these triggers are separated from those for classifying normal images, which reduces the search space and thereby makes the problem more tractable.

More specifically, prior researches consider an infected model characterized by a \textit{partitioned} representation: that is, the features related to a backdoor trigger are \textit{not} used for classifying normal inputs. This property avoids interfering with the model's labeling of normal inputs (those without the trigger), while creating a ``shortcut'' dimension from backdoor-related features to move \textit{any} input sample carrying the trigger to the target class through the backdoor. In the meantime, the property also exposes the backdoor to detection, allowing a pattern that causes a misclassification on an image to be cut-and-pasted to others for verifying its generality~\cite{sentinet}. Even more revealing is the difference between the representation generated for a normal input and that for the trigger, as illustrated in Fig~\ref{fig:pca_embeddings} left. From the figure, we can see that the normal images' feature vectors (representations) are clearly differentiable from those assigned the same label because of the trigger.

Research on such an attack, however, \textit{ignores a more generic situation where features are less partitioned and used both for normal classification and for recognizing the trigger.} We found that this can be \textit{easily} done through a \textit{targeted contamination attack} (TaCT) that poisons a model's training set with both attack and cover samples (Section~\ref{sec:attack}) to map only the samples in specific classes to the target label, not those in other classes.  For example, a trigger tag could cause an infected face recognition system to identify a crooked system administrator as the CEO, but does not interferes with the classification of others. Under these new attacks, the representations for normal images and malicious ones (with triggers) become indistinguishable by the existing approaches, as discovered in our research (see Fig~\ref{fig:pca_embeddings} right).

\vspace{5pt}\noindent\textbf{Statistical contamination detection}. In our research, we made the first attempt to understand the representations of different kinds of backdoors (source-agnostic and source-specific) and concluded that existing defense, including neural cleanse~\cite{wangneural2019}, SentiNet~\cite{strip}, STRIP~\cite{strip} and Activation Clustering~\cite{ac_attack}, \textit{fails to raise the bar to the backdoor contamination attack}. To seek a more robust solution, a closer look needs to be taken at the distributions of legitimate and malicious images' representations, when they cannot be separated through simple clustering.

To this end, we developed a new backdoor detection technique called \textit{SCAn} (statistical contamination analyzer), based upon statistical properties of representations an infected model produces. As a first step, SCAn is designed to work on a (broad) category of image classification tasks in which a class label is given to each object (face, flower, car, digits, traffic sign, etc.), and the variation applied to an object (e.g., lighting, poses, expressions, etc.) is of the same distribution across all labels. Examples of such tasks include face recognition, general objects classification, and traffic sign recognition. For such a task, a DNN model is known to generate a representation that can be decomposed into two vectors, one for an object's identity and the other for its variation randomly drawn from a distribution (which is the same for all images)~\cite{wang2004unified}: for example, in face recognition, one's facial features (e.g., height of cheekbone, color of eyes, etc.) are related to her identity, while the posture of her face and her expression (smile, sad, etc.) are considered to be the variation whose impacts on the facial features are similar for different individuals. The identity vector for each class and the variation distribution can be recovered by running an Expectation-Maximization (EM) algorithm on the training images~\cite{chen2012bayesian} and their representations (Section~\ref{sec:defense}).  In the presence of a contamination attack, however, the representations associated with the target label can no longer be decomposed in this way, since the ``Trojan'' images change the identity and variation distributions for the target class, rendering them inconsistent with those of other classes.
Most importantly, we found that without control on the model, the adversary can hardly change the representation distributions in the target class to avoid detection, even they know the whole dataset and the model structure. Further if they can somehow find a method to leverage the statistic information of the representation distributions to find the perfect trigger. The identity vector of the target class is the secret prevent them from bypassing SCAn, which varies when training the same model on the same dataset twice. Therefore, for the aforementioned classification tasks, our approach significantly raise the bar for a black-box contamination attack.

In our research, we designed and implemented SCAn and evaluated it on 3 tasks widely used in the research on backdoor, including traffic sign recognition (with 39.2K images),  object classification (with 1.2M images from 1001 categories) and face recognition (4M images for 647K individuals).  Over the models chosen for these tasks in the real world, such as ResNet101,  we demonstrated that SCAn succeeds where existing defense fails: our approach accurately reported all source-agnostic and source-specific backdoors, including advanced ones as proposed in the prior research~\cite{wangneural2019}, without any false positive and negative (Section~\ref{sec:defense}). Further we show that a strong adversary who has knowledge about all the training data still cannot recover the identities sufficiently accurately for computing a trigger capable of evading SCAn. This provides preliminary evidence that our approach indeed raises the bar to the backdoor attack (Section~\ref{subsec:robustbess}).

\vspace{5pt}\noindent\textbf{Contributions}. The contributions of the paper are outlined as follows:

\begin{itemize}

\item \textit{New understanding of backdoor attacks}. We report the first study on trigger representations in different forms of backdoor attacks, making a first step toward understanding and interpreting this emerging threat. Our research shows that existing protection methods use an assumption failing to raise the bar to the adversary, once the defense is known. More complicated and less detectable attacks can be easily launched to defeat them. Up to our knowledge, this has never been done before. Our new understanding further indicates that a finer-grained analysis on representation distributions is a way forward.

\item \textit{New defense against the backdoor threat}. Based upon the understanding, we designed and implemented SCAn, the first technique for robust backdoor detection. Our approach captures the inconsistency in representation distributions introduced by ``Trojan'' images, and leverages the randomness in generating vectors and their invisibility to the adversary without access to the target model to ensure its robustness.  Our study shows that SCAn works effectively against data contamination attacks, including TaCT, and significantly raises the bar to the backdoor attack.

\end{itemize}

\vspace{5pt}\noindent\textbf{Roadmap}. The rest of the paper is organized as follows: Section~\ref{sec:background} describes the background of our research and the assumptions we made; Section~\ref{sec:attack} presents our new understanding of backdoor attacks and TACT; Section~\ref{sec:defense} elaborates the design and implementation of SCAN; Section~\ref{sec:discussion} discusses the limitations of our technique and potential future research; Section~\ref{sec:relatedWork} reviews related prior research and Section~\ref{sec:conclusion} concludes the paper.

\ignore{Deep neural networks (DNNs) is an essential of modern and future services based on big data. These new coming services support a new life style characterized by precise and efficient. Specifically, in recent decades, DNN powered on everything: face recognition, natural language processing, self-driving, etc. Without doubt, so does the security area. Profiting from DNN, various security applications achieved high accuracy, from malware classification~\cite{wang2017adversary}, improper content detection~\cite{yuanstealthy} to intrusion detection~\cite{tang2016deep} and digital forensic~\cite{li2018printracker}.

All these fields highly rely on data collection. Massive data are needed to train more and more accurate model. However, it is even possible to maintain the integrity of such huge data by human beings. Even for those crowdsourced data (e.g, Amazon Mechanical Turk, Yelp reviews, Tweets), no organization can assure these data are clean and poison free. Let alone those data from untrusted data collector. While these outsourced data are commonly used by individuals, small companies and non-security-professional organizations. This is highly risky and free for adversaries inject malicious data to contaminate the training dataset and mislead the DNN model having abnormal behaviours, which is so called poison attack.

Detailedly, poison attacks can be divided into two kinds. One kind (likes DoS attack) aiming to lower the performance of the target DNN model, and the other kind is to put a trap (backdoor) in the DNN model, which is triggered by ``trigger'' inputs stamped with a specific pattern. When this trap has been triggered, the DNN model will misclassify the current input as a target label chosen by the attacker. Compared with the first kind, the second kind is more dangerous and stealthy. And previous works~\cite{gu2017badnets, liu2017trojaning, chen2017targeted} have demonstrated this kind of attacks is model-agnostic and easily launched.

On the defence hand, many works tried to mitigate backdoor attacks. Wang et al.~\cite{wangneural2019} tried to reverse engineer the trigger pattern and trim relevant neurons. Liu et al.~\cite{liu2018fine} tried to use fine-pruning to cancel out those trigger neurons and keep the performance unchanged. Gao et al.~\cite{ac_attack} tried to detect abnormal predictions when feeding the DNN model with blended inputs. Chou et al.~\cite{sentinet} tried to segment the input and find trigger regions.

All these works reported high effectiveness against backdoor attacks. While, we will demonstrate they will be defeated by a simple modification of convention backdoor attack, Cover-Target attack (Section~\ref{subsec:current_defences}). We argue that the failure of previous defences is due to non-solid assumptions they relied on. Most of them, explicitly or implicitly, rely on the \textbf{global trigger} assumption, i.e., there is a trigger stamping it on whatever inputs will result in misclassification of the target model. This assumption is easily bypass by Cover-Target attack (Section~\ref{sec:attack}).

In this paper, we proposed a solid assumption, \textbf{two-in-one class} assumption, against backdoor attacks. We assume that, in backdoor attacks, attackers will always try to mislead the target DNN model to classify such images that originally belongs to at least two different classes as one class. Clearly, no matter how attackers to achieve this, this is the inherent goal of the backdoor attack.

Based on this assumption, we designed a defence method based on statistic technologies, trying to detect whether a class can be partitioned into two separate parts and this partition is reasonable in statistic meaning (Section~\ref{sec:defense}). We also demonstrated that our defence not only works well on the conventional triggers but also is robust against advanced triggers (Section~\ref{subsec:robustbess}).

\textbf{Contributions.}
Our paper makes the following contributions to the defence against backdoor attacks.

\begin{itemize}

\item We demonstrated that the two-in-one class assumption is the basic assumption for defence against backdoor attacks. And further works should base on that.

\item We demonstrated a simple but powerful attack, Cover-Target attack, can defeat current defences. And we discussed the reason of them failures.

\item We proposed a new defence based on statistic technologies and showed its effectiveness on three tasks: traffic sign recognition, general object classification and face recognition. More over, we also demonstrated our defence is robust against advanced triggers proposed in~\cite{wangneural2019}.

\end{itemize}

To the best of our knowledge, our work is the first to clearly define the assumption should be made for backdoor attacks and, based on it, design a defensive method. Extensive experiments show our method is highly effective against various backdoor attacks.}

\section{Related Works}
\label{sec:relatedWork}

Our method SCAn is a kind of statistic defense against data contamination attacks. It is powered by leveraging the distribution of sample representations ($S_{\varepsilon}$ of decomposition model) to differentiate the outlier class from other benign classes. The most recent relevant works were discussed in Section~\ref{subsec:current_defences} that are all defeated by TaCT, whereas SCAn is effective to detect TaCT. Next, we introduce other defense strategies in this field.

Two general defences against data contamination attacks were proposed by Nelson et al.~\cite{nelson2009misleading} and Baracaldo et al.~\cite{baracaldo2017mitigating}. Both methods require extensive retraining of the model on the datasets with the similar size as the original one, which is often infeasible for DNNs. Additionally, they detect infected data by evaluating the overall performance of the model. However, the overall performance of the infected model often remains good under current advanced attacks (like TaCT), and thus these defenses will become ineffective against these attacks .
\ignore{
\begin{figure}[ht]
	\centering
  \begin{subfigure}{0.3\textwidth}
		\includegraphics[width=\textwidth]{figure/traditonal_statistics.eps}
  \end{subfigure}

	\caption{ROCs for detecting infected classes by traditional statistical methods applied to representations produced by a TaCT-infected model.}
	\label{fig:trad_stati}

\end{figure}
}

\begin{figure}[!htb]
   \begin{minipage}{0.23\textwidth}
     \centering
     \includegraphics[width=\linewidth]{figure/acc_triggers.pdf}
     \caption{Statistics of infected models when different number of triggers are infected in ILCVR2012.}
     \label{fig:acc_triggers}
   \end{minipage}
   \begin{minipage}{0.2\textwidth}
     \centering
     \includegraphics[width=\linewidth]{figure/traditonal_statistics.pdf}
     \caption{ROCs of traditional statistical methods directly applied on representations produced by a TaCT-infected model.}
     \label{fig:trad_stati}
   \end{minipage}
\vspace{-0.25in}
\end{figure}

In the traditional statistical analysis domain, a review written by Victoria et al.~\cite{hodge2004survey} summarizes several effective outlier detection methods, including k-nearest neighbors (k-nn)~\cite{knox1998knn}, k-means~\cite{nairac1999kmeans} and principal components analysis (pca)~\cite{korn1998pca}. To investigate whether directly applying them to sample representations can detect infected classes, we applied them on representations produced by a TaCT infected model for those images belonging to the target class. The results are demonstrated on Fig~\ref{fig:trad_stati}. We observe that these methods produce many false positives.
Another method proposed by Steinhardt et al.~\cite{steinhardt2017certified} tried to detect infected classes by searching input samples with large spatial angles from the class center. However, this method does not work for carefully designed trigger or images with complex background, because in these cases, the trigger may be similar as the background, and thus the angle of those images with triggers will be indistinguishable from those of normal samples. More importantly, as claimed by the authors, this method failed to detect the infected class containing 30\% or more attack data. In contrast, our method SCAn is robust enough to handle these cases as we demonstrated in Section~\ref{sec:defense}.

Liu et al.~\cite{liu2018fine} proposed a neuron-driven method. They first prune neurons that are dormant when processing clean data until the accuracy tested on a hold-on dataset being below a threshold, and then fine-tune the pruned model to recover the accuracy. Their defense relies on extensive interaction with the training process. In contrast, our approach sanitizes the dataset in one round and is independent from the  training of the target model.