\section{Appendix}

We show here additional results for classifying five users. Figure~\ref{fig:figure_rocs5} shows the user-level ROC curves and Figure~\ref{fig:figure_rocs_averaged_5}  shows the micro-averaged results for each of three models. Table~\ref{fig:figure_classifier comparison5} shows the results per-user for the three models when classifying among five users.

\begin{figure*}[th]
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{roc_curveGradientBoost5.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{roc_curveRandomForest5.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{roc_curveLR5.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{User-level ROC curves for 25 minute time windows for three models with 5 users.}
	\label{fig:figure_rocs5}
\end{figure*}

\begin{figure*}[h]
	\centering
	\includegraphics[width=0.8\columnwidth]{roc_curve_all_micro_5users.pdf}

	\caption{Averaged ROC curves for 25 minute time windows for 5 users.}
	\label{fig:figure_rocs_averaged_5}
\end{figure*}

\begin{table*}[htb]
\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|ll}
		\cline{1-10}
		\multirow{2}{*}{} & \multicolumn{3}{c|}{\textbf{Logistic Regression}}    & \multicolumn{3}{c|}{\textbf{Random Forest}}        & \multicolumn{3}{c|}{\textbf{Gradient Boosting}}      &  &  \\ \cline{2-10}
		& \textit{Recall} & \textit{Precision} & \textit{F1}   & \textit{Recall} & \textit{Precision} & \textit{F1} & \textit{Recall} & \textit{Precision} & \textit{F1}   &  &  \\ \cline{1-10}
		\textbf{User 1}   & 0.57            & 0.66               & 0.61          & 0.85            & 0.75               & 0.8         & \textbf{0.85}   & \textbf{1.0}       & \textbf{0.92} &  &  \\ \cline{1-10}
		\textbf{User 3}   & 0.79            & 0.88               & 0.83          & 0.97            & 0.95               & 0.96        & \textbf{1.0}    & \textbf{0.92}      & \textbf{0.96} &  &  \\ \cline{1-10}
		\textbf{User 4}   & 0.90            & 0.62               & 0.74          & 0.90            & 0.95               & 0.93        & \textbf{0.95}   & \textbf{1.0}       & \textbf{0.97} &  &  \\ \cline{1-10}
		\textbf{User 8}   & 0.42            & 0.75               & 0.54          & 0.57            & 0.8                & 0.66        & \textbf{0.71}   & \textbf{0.83}      & \textbf{0.76} &  &  \\ \cline{1-10}
		\textbf{User 10}  & \textbf{0.71}   & \textbf{0.83}      & \textbf{0.76} & 0.71            & 0.62               & 0.66        & 0.57            & 0.66               & 0.61          &  &  \\ \cline{1-10}
		\textbf{Average}  & 0.76            & 0.79               & 0.78          & 0.90            & 0.90               & 0.90        & \textbf{0.92}   & \textbf{0.92}      & \textbf{0.92} &  &  \\ \cline{1-10}
	\end{tabular}
		\captionsetup{justification=centering,margin=2cm}
		\caption{Comparison of three ML classifiers per user and micro-average results for 5 users.}
		\label{fig:figure_classifier comparison5}
\end{table*}

\section{Preliminaries and Background}
\label{sec:bk}
In this section we provide an overview of methods for user authentication, we define the problem we
are solving, and describe the adversarial model we consider in this work.

\subsection{User Authentication}

User authentication is a fundamental security process that has been studied extensively throughout the years. Traditional user authentication methods are based on \emph{what users know} (e.g. passwords), \emph{what they have} (e.g. hardware token), or \emph{what they are} (e.g. biometrics).  Even though extensive research  shows  password limitations~\cite{cranor2018pwd,cranor2016pwdmobile,cranor2016pwdperception}, passwords are still the de-facto standard for authentication in corporate and home environments. Users are shown to pick weak passwords and they tend to reuse them across different services \cite{worstpasswords}. Critical services do not always deploy proper password data protection techniques, resulting in increased risk for users in face of data breaches~\cite{facebookplain}.
Several well-known techniques are used in practice to augment passwords. Multi-factor authentication methods use either a hardware token or a different channel  (e.g. email, SMS) for increasing the confidence of authenticating users. These methods impede the usability of authentication, and introduce additional latency~\cite{siadati2017mind,de2013comparative}.
Biometrics-based authentication such as keystroke dynamics, fingerprints, facial recognition, and voice recognition,  provide a convenient and reliable way to identify users~\cite{wayman2005introduction,SurveyBiometrics,SurveyBiometricWearable}. However, revocation and user impersonation are well-known challenge for these methods, in addition to privacy implications on users~\cite{wolf2019pretty}.

Behavior-based authentication  provides a safer and more convenient way to identify users, based on their behavior, \emph{how they behave}. Such methods build behavioral models based on the unique
way the users interact with devices like smartphones,
tablets or touch screens. In the context of mobile devices, several systems for continuous authentication have been designed, for instance~\cite{shi2010implicit,riva2012progressive}.  They leverage a number of behavior features extracted from user interaction with their mobile devices, such as call logs, browser history, application usage, and location. User authentication based on WiFi signal of IoT devices has been also investigated recently~\cite{shi2017smart}. These systems can be used in a multi-factor authentication system to additionally increase confidence of user authentication.

\subsection{Problem Definition}

With the widespread usage of IoT devices in many households, new challenges and opportunities emerge. We pose for the first time \emph{the problem of designing usable, behavioral-based authentication models based on users naturally interacting with IoT devices in their homes or familiar environments}. We are interested in designing a lightweight authentication module for smart homes that uses information extracted from monitoring the user interaction with IoT devices. The system should be able to identify with high accuracy each user from a fixed set of known users based on their behavioral characteristics.   With user privacy as the premiere consideration in our design, we intend to leverage a minimum number of attributes that are universally applicable to a range of IoT devices from different manufacturers. We consider here mainly authentication based on behavioral features extracted from network traffic generated by IoT devices in a smart home. We envision such a system would learn from historical observations and create user profiles  during a training phase. At testing time, the authentication module generates an authentication score mapped to the likelihood that a certain user is in the room based on the most recent observation of IoT device activity. The authentication module needs to observe users during a contiguous observation window to capture a sequence of user actions. The authentication scores can be used by higher-level applications to create flexible authorization policies that support various levels of risk.

\subsection{Adversarial Model} We consider two main classes of adversaries:

\myparagraph{Local adversaries} This adversary might be in close proximity or even inside the home environment, and interact with some of the IoT devices in the home. This model captures visitors, neighbors, kids, or roommates sharing the home. We would like to prevent such an adversary to perform actions like changing smart lock settings or making purchases on Alexa on behalf of real users.

\myparagraph{Remote adversaries} Remote adversaries might not directly have access to IoT devices, but they could compromise user machines and accounts leveraging potential vulnerabilities in the user devices. This adversary is motivated to impersonate the real user to gain access to sensitive devices in the home or perform more harmful actions.
Remote attackers can also inject network traffic in the smart home. This is possible by a malicious app installed on a user's smart phone, or an adversary sending traffic to the home router. The adversary is aware of standard network protocols and applications, but he does not have full knowledge on the exact user behavior.

We assume that adversaries are not in the system when ML training is performed. A training-time attack against ML models is called a poisoning attack (e.g., \cite{Xiao15,biggio2012poisoning,Steinhardt17}) and protecting against it is orthogonal to our work. This guarantees that the trained ML models for user authentication are trustworthy. Both types of adversaries might act at testing time with the goal of influencing the decision of the ML model. We assume that adversaries have relatively limited interaction with the IoT home devices and they cannot perfectly imitate users in their behavior.

We envision that data collection and the authentication module reside in a trusted device, like the home router or a server. Compromising the authentication system by physically accessing it or by intercepting the communications is not within our scope.

\subsection{Challenges and Insights}

Designing a continuous authentication system for IoT environments is a challenging task. The first challenges is creating realistic smart home environments with multiple users and collect data of users interacting naturally in such an environment over time. We address this challenges by leveraging an IoT Lab designed at our institution and designing an IRB-approved user study with multiple users in the lab over a period of one month. We make use of the existing monitoring infrastructure at our institution to collect network packets from the IoT lab.

For an authentication system to be effective, user behavior and interaction with IoT devices needs to be relatively consistent over time and distinguishable among different users. Users need to be actively engaged with smart devices for our system to get meaningful data. User activities on personal computers and mobile devices are not monitored by our system (unless users control IoT devices from these other systems). Unlike mobile-based behavior authentication, in which a lot of activity is generated in relatively short time windows (e.g., one minute), users generate lower volumes of activities with their IoT home devices.  Our system thus needs to increase the observation period of users in order to predict a reliable authentication score. However, we expect that an IoT authentication system will get more false negatives (i.e., user score having low confidence) compared to more traditional authentication systems.

Additionally, network traffic of IoT devices is most of the time HTTPS and thus the payload is encrypted. As user privacy is extremely important, we need to restrict ourselves to attributes available from headers of HTTPS traffic, which represents minimal information (timing of the packet, destination IP and domain name, direction, and packet length). Moreover, most of the external interaction of IoT devices are with the cloud providers (Amazon AWS and Samsung cloud), thus most users connect to similar external destinations.  This makes it difficult to differentiate users that use these devices for different purposes.
We consider these challenges in the rest of our work, and discuss how we address them.
\section{Conclusion}
We propose a novel user authentication method by analyzing network traffic generated by home-based Internet of Things (IoT) devices. We demonstrate the correlation between user actions and network traffic on a variety of IoT devices and conduct a user study that shows 80\% percent accuracy in identifying users.
\section{Discussion and Conclusions}
\label{sec:concl}

We propose a novel user authentication method by analyzing HTTPS network traffic generated by home-based IoT devices. We conduct a user study and experiment with several ML algorithms using features extracted from network traffic with the goal of classifying users from a known set of users. We showed that random forest and gradient boosting models perform well in this setting and obtain high accuracy at user classification. Moreover, an ensemble of two models can increase the confidence of classification, at the expense of abstaining to produce a score when the models produce different predictions.

We also tested several deep neural network (DNN) architectures, including feed-forward neural networks and LSTMs. We did not include the details in the paper, as the accuracy of these models was not comparable with that of gradient boosting, random forest, and our two-model ensemble.
The best F1 score for a feed-forward architecture was 0.54 and for LSTM 0.51, whereas our gradient boosting achieves an F1 score of 0.86. We suspect the reason is because we have a relatively limited labeled dataset available for model training.  While in principle it is possible to conduct the user study over longer periods of time, we found it challenging to recruit and retain users over time.

In a practical deployment, we envision that the ML model for user classification is continuously re-trained with new behavioral data over time. This is particularly important as new IoT devices are added to the smart home and users slowly shift their interests over time. Again, a longer-term user study would provide sufficient data to experiment with model re-training and analyze user behavior over time as IoT devices evolve. Another research direction would be investigating the performance of the models in the presence of multiple active users in the smart home.

The behavioral authentication scores computed by our authentication modules could be applied in various settings. More importantly, they open up the possibility of creating flexible policies for authorization and access control, and replace today's rigid, fixed policies. We believe that our work opens up new research avenues in this direction, and this is a topic of interest in our future work.
\section{Evaluation}
\label{sec:evaluation}

We evaluate our system using the data collected in the user study that is discussed in Section~\ref{sec:userstudy}. We present results for different feature representations, user classification performance, and receiver operating characteristic (ROC) curves for three different classification models and a high-confidence ensemble model.

For the user classification task, we label the data using the session start and end times of each user participating in our user study. As it is usually not possible to train accurate machine learning models with limited amount of data, we decided to exclude users with less than 6 sessions in total. We perform our analysis on six users: 1, 3, 4, 6, 8, and 10 who generate more than 6 sessions. We further excluded 7 sessions in total during which the network monitoring tool was not active.

\subsection{Comparison of Feature Representations}

\begin{figure*}[ht]
	\begin{minipage}[b]{0.27\linewidth}
		\centering
		\includegraphics[width=\textwidth]{user6_3selecteddevices.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.27\linewidth}
		\centering
		\includegraphics[width=\textwidth]{user6_8selecteddevices.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.27\linewidth}
		\centering
		\includegraphics[width=\textwidth]{user6_33selecteddevices.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{Total amount of data transmitted by five devices for User 6 in three sessions.}
	\label{fig:figure_user6_sessions}
\end{figure*}

\begin{figure*}[ht]
	\begin{minipage}[b]{0.27\linewidth}
		\centering
		\includegraphics[width=\textwidth]{user10_26selecteddevices.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.27\linewidth}
		\centering
		\includegraphics[width=\textwidth]{user10_49selecteddevices.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.27\linewidth}
		\centering
		\includegraphics[width=\textwidth]{user10_54selecteddevices.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{Total amount of data transmitted by five devices for User 10 in three sessions.}
	\label{fig:figure_user10_sessions}
\end{figure*}

\begin{figure*}[hbt]
	\begin{minipage}[b]{0.27\linewidth}
		\centering
		\includegraphics[width=\textwidth]{time_accuracy_plot6users_packet_stats_device_time_new.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.27\linewidth}
		\centering
		\includegraphics[width=\textwidth]{time_accuracy_plot6users_packet_stats_only_domain_time_new.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.27\linewidth}
		\centering
		\includegraphics[width=\textwidth]{time_accuracy_plot6users_packet_stats_omain__time_new.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{Accuracy metrics for different time windows for 6 users using different feature representations. From left to right: device based features, domain based features, device and domain based features.}
	\label{fig:figure_accuracy_6_users}
\end{figure*}

We experiment with three different feature representations as discussed in Section~\ref{sec:features}: \emph{Device-only} (420 features); \emph{Domain-only} (2910 features); and \emph{Both device and domain} (3330 features).  We considered time windows of different length $\Delta \in \{5, 10, 15, 20, 25, 30\}$ minutes. Each time window is created by sliding windows over the user sessions at one-minute intervals. If a session is shorter than $\Delta$, we generate only one window covering the entire session.

To evaluate our ML models, we perform 7-fold cross-validation across sessions. Thus, we keep one session for each user for testing and select all other sessions in training. We consider first a Random Forest (RF) classifier to evaluate feature representations and time window length. RF is an ensemble learning method that constructs many decision trees using a subset of features chosen at random at each split. RF are robust models that work well in multi-class classification settings with many features, as ours. We will experiment and compare other models next section.

We plot the recall, precision, and F1 score of the three feature representation methods for varying time windows in Figure~\ref{fig:figure_accuracy_6_users}. The results show that the model performance  generally increases as the time window increases until 25 minutes, at which point the best results are obtained. We thus select a time window of 25 minutes for the rest of our experiments. We note the inherent tradeoffs between  classification accuracy and the length of observation period for computing authentication scores. With smaller windows length, we can generate authentication scores after limited observation, at the cost of decreased classification performance. Comparing different feature representations, we observe that \emph{Device-only} performs well, and it is very similar to \emph{Both device and domain}. The worst performance is provided by the \emph{Domain-only} representation.

We also generate user confusion matrix for the RF models in Table~\ref{table:confusion_6user_25min}. Interestingly, we obtain very good classification for some of the users (for instance, User 4), but worse results for other users (e.g., User 6). User 6 is mis-classified as User 3, meaning that their behavior is fairly similar. Thus, the classification performance varies a lot  across users. We believe there are multiple factors contributing to this phenomenon, among them the variability of user behavior and the amount of data used for training these models.

\subsection{Model Comparison}

We compare three different models: Logistic Regression with $L_1$ regularization (LR), Random Forest (RF), and Gradient Boosting (GB) for time windows of length 25 minutes. We varied the hyper-parameters of these classifiers, and we select 2000 estimators for RF, and 2000 estimators for GB, with learning rate set at 0.01 and maximum depth at 3. For this experiment, we use the \emph{Device-only} features.  In Table~\ref{fig:figure_classifier comparison6} we show user-level performance metrics for the three classifiers when all six users are considered. As observed, GB outperforms both RF and LR with its average precision and recall at 0.81 and 0.8, respectively. However, the performance of RF and GB is fairly close, but LR has lower classification performance (being a linear model with lower complexity).  We also generate ROC curves when training six classifiers, each one for classifying one of the user classes versus the rest in Figure \ref{fig:figure_rocs}. Figure~\ref{fig:figure_rocs_averaged} shows the micro-averaged results for each of three models with six users.

\begin{table*}[thb]
	\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|}
		\hline
		& \textbf{1} & \textbf{3} & \textbf{4} & \textbf{6} & \textbf{8} & \textbf{10} & \textbf{Count} & \textbf{Recall} & \textbf{Precision} & \textbf{F1} \\ \hline
		\textbf{1}  & 5          & 0          & 1          & 1          & 0          & 0           & 7              & 0.71            & 0.71               & 0.71        \\ \hline
		\textbf{3}  & 0          & 47         & 0          & 1          & 0          & 0           & 48             & 0.97            & 0.81               & 0.88        \\ \hline
		\textbf{4}  & 0          & 1          & 20         & 1          & 0          & 0           & 22             & 0.90            & 0.95               & 0.93        \\ \hline
		\textbf{6}  & 0          & 9          & 0          & 4          & 0          & 1           & 14             & 0.28            & 0.57               & 0.38        \\ \hline
		\textbf{8}  & 1          & 0          & 0          & 0          & 4          & 2           & 7              & 0.57            & 0.8                & 0.66        \\ \hline
		\textbf{10} & 1          & 1          & 0          & 0          & 1          & 4           & 7              & 0.57            & 0.57               & 0.57        \\ \hline
	\end{tabular}
			\caption{RF user confusion matrix and accuracy metrics for 25 minute intervals: avg. recall=0.8, avg. precision=0.78}
			\label{table:confusion_6user_25min}
\end{table*}

\begin{table*}[tbh]
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|ll}
		\cline{1-10}
		\multirow{2}{*}{} & \multicolumn{3}{c|}{\textbf{Logistic Regression}}    & \multicolumn{3}{c|}{\textbf{Random Forest}}          & \multicolumn{3}{c|}{\textbf{Gradient Boosting}}      &  &  \\ \cline{2-10}
		& \textit{Recall} & \textit{Precision} & \textit{F1}   & \textit{Recall} & \textit{Precision} & \textit{F1}   & \textit{Recall} & \textit{Precision} & \textit{F1}   &  &  \\ \cline{1-10}
		\textbf{User 1}   & 0.42            & 0.6                & 0.5           & 0.71            & 0.71               & 0.71          & \textbf{0.85}   & \textbf{0.75}      & \textbf{0.79} &  &  \\ \cline{1-10}
		\textbf{User 3}   & 0.52            & 0.69               & 0.59          & \textbf{0.97}   & 0.81               & 0.88          & 0.93            & \textbf{0.86}      & \textbf{0.9}  &  &  \\ \cline{1-10}
		\textbf{User 4}   & 0.81            & 0.62               & 0.70          & 0.90            & 0.95               & 0.93          & \textbf{0.90}   & \textbf{1.0}       & \textbf{0.95} &  &  \\ \cline{1-10}
		\textbf{User 6}   & 0.71            & 0.45               & 0.55          & 0.28            & \textbf{0.57}      & 0.38          & \textbf{0.5}    & 0.53               & \textbf{0.51} &  &  \\ \cline{1-10}
		\textbf{User 8}   & 0.42            & 0.5                & 0.46          & \textbf{0.57}   & \textbf{0.8}       & \textbf{0.66} & 0.42            & 0.75               & 0.54          &  &  \\ \cline{1-10}
		\textbf{User 10}  & \textbf{0.71}   & \textbf{0.71}      & \textbf{0.71} & 0.57            & 0.57               & 0.57          & 0.57            & 0.5                & 0.53          &  &  \\ \cline{1-10}
		\textbf{Average}  & 0.60            & 0.62               & 0.60          & 0.80            & 0.78               & 0.78          & \textbf{0.80}   & \textbf{0.81}      & \textbf{0.80} &  &  \\ \cline{1-10}
	\end{tabular}
			\captionsetup{justification=centering,margin=2cm}
			\caption{Comparison of three ML classifiers per user and micro-average results for 6 users.}
			\label{fig:figure_classifier comparison6}
\end{table*}

Additionally, we consider a setting with only five users (removing User 6) and show classification results in Table~\ref{fig:figure_classifier comparison5} in the Appendix. Surprisingly, by removing one of the users, we obtain classification precision of 0.92 and recall of 0.92 with GB, an increase of more than 10\% compared to the setting of six users. Figure~\ref{fig:figure_rocs5} in the Appendix shows the user-level ROC curves for five users and Figure~\ref{fig:figure_rocs_averaged_5} in the Appendix shows the micro-averaged results for each of three models with five users.

\begin{figure*}[thb]
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{roc_curveGradientBoost6.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{roc_curveRandomForest6.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{roc_curveLR6.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{User-level ROC curves for 25 minute time windows for three models with 6 users.}
	\label{fig:figure_rocs}
\end{figure*}

\subsection{High-Confidence Ensemble Model}

So far, we showed that models based on \emph{device-only} features perform relatively well at classifying users at time windows of length 25 minutes. As we discussed initially, different applications might have different requirements in terms of the confidence offered by the authentication score. For instance, a financial application might require a high confidence in the user classification module before allowing a financial transaction. When the authentication score is used as a factor in a multi-factor authentication system, it could be acceptable to not compute an authentication score when the confidence is very low.

To account for these settings, we propose the idea of designing an ensemble of two models that are built independently and can be used in combination to increase the confidence in the authentication score. Our main insight is that we can build models leveraging the device-only and domain-only features independently, and compute an authentication score only when the two models agree on the user prediction. As a result, the overall confidence in the classification will increase. The cost is that, in situations when the two models disagree, no authentication score will be computed. In this case, the upper-level applications might wait for additional time, or leverage other authentication factors.

\begin{table*}[thb]
	\begin{tabular}{|l|l|l|l|l|l|l|l|l|l|l|l|}
		\hline
		& \textbf{1} & \textbf{3} & \textbf{4} & \textbf{6} & \textbf{8} & \textbf{10} & \textbf{Disagreed} & \textbf{Agreed} & \textbf{Recall} & \textbf{Precision} & \textbf{F1}   \\ \hline
		\textbf{1}     & 2          & 0          & 0          & 1          & 0          & 0           & 4                  & 3              & 0.66            & 0.66               & 0.66          \\ \hline
		\textbf{3}     & 0          & 35         & 0          & 3          & 0          & 0           & 10                 & 38             & 0.92            & 0.92               & 0.92          \\ \hline
		\textbf{4}     & 0          & 0          & 13         & 0          & 0          & 0           & 9                  & 13             & 1.0             & 1.0                & 1.0           \\ \hline
		\textbf{6}     & 0          & 3          & 0          & 6          & 0          & 0           & 5                  & 9              & 0.66            & 0.54               & 0.6           \\ \hline
		\textbf{8}     & 1          & 0          & 0          & 0          & 3          & 0           & 3                  & 4              & 0.75            & 0.75               & 0.75          \\ \hline
		\textbf{10}    & 0          & 0          & 0          & 1          & 1          & 4           & 1                  & 6              & 0.66            & 1.0                & 0.8           \\ \hline
		\textbf{Total} & \multicolumn{6}{l|}{\textbf{}}                                               & \textbf{32}        & \textbf{73}    & \textbf{0.86}   & \textbf{0.87}      & \textbf{0.86} \\ \hline
	\end{tabular}

					\caption{Confusion matrix for a Gradient Boosting ensemble with one model using the device features, and the other the domain features for 25 minute window for 6 users. If the classifiers disagree, no authentication score is computed.}
					\label{fig:multi_model_gb_gb_6user}
\end{table*}

\begin{table*}[htb]
	\begin{tabular}{|c|c|c|c|c|c|c|c|c|c|c|}
		\hline
		\textbf{}      & \textbf{1} & \textbf{3} & \textbf{4} & \textbf{8} & \textbf{10} & \textbf{Disagreed} & \textbf{Agreed} & \textbf{Recall} & \textbf{Precision} & \textbf{F1}   \\ \hline
		\textbf{1}     & 3          & 0          & 0          & 0          & 0           & 4                  & 3              & 1.0             & 1.0                & 1.0           \\ \hline
		\textbf{3}     & 0          & 48         & 0          & 0          & 0           & 0                  & 48             & 1.0             & 0.97               & 0.98          \\ \hline
		\textbf{4}     & 0          & 0          & 15         & 0          & 0           & 7                  & 15             & 1.0             & 1.0                & 1.0           \\ \hline
		\textbf{8}     & 0          & 0          & 0          & 3          & 0           & 4                  & 3              & 1.0             & 0.75               & 0.85          \\ \hline
		\textbf{10}    & 0          & 1          & 0          & 1          & 4           & 1                  & 6              & 0.66            & 1.0                & 0.8           \\ \hline
		\textbf{Total} & \multicolumn{5}{c|}{}                                           & \textbf{16}        & \textbf{75}    & \textbf{0.97}   & \textbf{0.97}      & \textbf{0.97} \\ \hline
	\end{tabular}

						\caption{Confusion matrix for the same ensemble model as in Table~\ref{fig:multi_model_gb_gb_6user}, but with 5 users. }
						\label{fig:multi_model_gb_gb_5user}
\end{table*}

We test the ensemble of two Gradient Boosting models, one built on the \emph{device-only} features, and the second on the \emph{domain-only} features. We show the results of our ensemble user classification model for six users in Table~\ref{fig:multi_model_gb_gb_6user} and for five users in Table~\ref{fig:multi_model_gb_gb_5user}. The F1 score of the six-user model improves from 0.8 with a single GB model to 0.86 with the ensemble. In the ensemble, the models do not agree on 32 out of 105 sessions. For five users, the ensemble F1 score reaches 0.97 (compared to 0.92 for a single GB model). In this case, the ensemble does not compute a score on only 16 out of 91 sessions. Therefore, the ensemble can reliably increase the confidence of the model, at the cost of not always providing an authentication scores.

\begin{figure}[ht]
\centering
\includegraphics[width=0.8\columnwidth]{roc_curve_all_micro_6users.pdf}
\caption{Model ROC curves for 25-minute windows, 6 users.}
\label{fig:figure_rocs_averaged}
\end{figure}

\subsection{Feature Representation}
\label{sec:features}

\myparagraph{Packet-level attributes} Our system design is based on features extracted from HTTPS traffic captured at the router in the smart home. To protect user privacy, one of our design consideration is to inspect only headers of the HTTPS packets. The fields that we leverage in our design are shown in Table~\ref{table_packetlevel}. They include the device identifier (extracted from the MAC address in each packet), timing, the packet length, the direction (outgoing or incoming), transport-level protocol (TCP, UDP, or ICMP), destination port, source port, and destination domain.
We analyze the DNS queries from the network traffic and extract mapping of external IP addresses to domain names (FQDN).

\myparagraph{Feature definition} Using packet-level attributes directly as features in an ML model is not feasible due to the large number of packets that are usually generated in a network. Moreover, for our task for user authentication we need aggregated behavioral indicators, beyond the scope of a single network packet. Therefore, we need to experiment with multiple feature representations that capture the user behavior over a time interval  of length $\Delta$. For our approach, we use a sliding-window method to advance the time $t$ at which we compute authentication scores by one minute. We experiment with multiple values for the window length (from 5 to 30 minutes). For instance, if a user is in the room for an interval of 30 minutes during training and $\Delta=5$ minutes, we generate 26 sliding windows of size 5 minutes. Features are generated for each sliding window and labeled with the identity of the user. At testing time, we compute an authentication score at each time $t$ based on the \emph{most recent sliding window} of size $\Delta$, i.e., the interval $[t-\Delta,t]$. That means, implicitly, that we need to observe the user for of period at least $\Delta$ before computing an authentication score.

The first source of data for feature extraction is user interactions with different IoT devices. It is well known that people have different interests and users prefer to interact with certain type of devices. Moreover, the same user tends to use the same set of IoT devices they are most familiar with over time.  We believe that device usage is a strong behavioral indicator, as also confirmed in our evaluation.

Therefore, we define the first set of \emph{device-level features} extracted from packet attributes aggregated per device. In particular, we would like to capture information about user interaction with each IoT device during the time window $\Delta$, such as: (1) various statistics on incoming and outgoing packet lengths; (2) packet counts for transport-level protocols; (3) number of distinct domains contacted by each device; (4) statistics on inter-arrival timing of packets.  The list of aggregated features is given in Table~\ref{table_packet_stats}, and all the features are aggregated over the packets sent and received by each device in each time window of size~$\Delta$. An important design decision is to separate incoming and outgoing traffic, as they might have different distributions. In particular, user interaction with voice assistants could result in large volumes of incoming traffic (for instance, if a user listens to music), while device communication with the cloud induces regular outgoing communication. We denote this set of device features in time window $[t-\Delta,t]$ as: $F(t,\Delta)^{\mathsf{dev}}$.

The second source of data useful for feature generation is user communication with external destinations (domain names). This is motivated by our observation that different Alexa skills connect to different external domains. For instance, listening to music may stream data from the {\sf spotify.com} domain, whereas playing fireplace sounds uses {\sf kwimer.com}. As the set of external domain names is very large and cloud services use many sub-domains for load balancing, we transform our domain names to second-level domains and compute features per second-level domain. For each second-level domain, we use aggregated features including statistics on incoming and outgoing packet lengths, inter-event times and packet counts for transport protocols, computed over all the packets exchanged with that external domains in the time window of size $\Delta$. We denote this set of domain features in time window $[t-\Delta,t]$ as: $F(t,\Delta)^{\mathsf{dom}}$.

For our task of user authentication, we will consider several feature representations:

\begin{enumerate}
\item \emph{Device-only}: Estimate and predict $P[U_i|F(t,\Delta)^{\mathsf{dev}}]$;
\item \emph{Domain-only}: Estimate and predict $P[U_i|F(t,\Delta)^{\mathsf{dom}}]$;
\item \emph{Both device and domain}: Estimate and predict \\ $P[U_i|F(t,\Delta)^{\mathsf{dev}}; F(t,\Delta)^{\mathsf{dom}}]$.
\end{enumerate}

\begin{table*}[h]
	\begin{center}
		\begin{tabular}{|p{0.30\textwidth}|p{0.70\textwidth}|}
			\hline
			{\bf Device} & {\bf User Activities} \\
			\hline
			Echo Dot - Echo Spot - Echo Plus & Alexa voice assistants to support voice commands and control other smart devices \\
			\hline

			Google Home Mini & Google voice assistant to support voice commands and control other smart devices \\
			\hline
			Harman Kardon Invoke & Cortana voice assistant to support voice commands and control other smart devices \\
			\hline

			LG Smart TV & Watch TV or stream content through third party applications \\
			\hline

			Roku TV & Stream content through third party applications \\
			\hline
			Amazon Fire TV & Stream content through third party applications \\
			\hline
			Philips Hue Bridge & Set the color and brightness of the bulbs \\
			\hline

			Samsung Smart Fridge & Use fridge, interact with the LCD Display \\
			\hline
			Smart Microwave & Open, Heat up food, Close \\
			\hline

			iKettle Smart Kettle & Boil water\\
			\hline

			Behmor Smart Brewer & Brew coffee\\
			\hline
			SmartThings Hub & Trigger motion and contact sensors \\
			\hline

		\end{tabular}
	\end{center}
	\caption{ IoT Devices in the IoT lab.}
	\label{table_activities}
\end{table*}

\subsection{Feature Representation}

\begin{table*}[h]
	\begin{center}
		\begin{tabular}{|p{0.20\textwidth}|p{0.15\textwidth}|p{0.65\textwidth}|}
			\hline
			{\bf Feature} & {\bf Type} & {\bf Description} \\
			\hline
			Device Name & Categorical & Name of the smart device \\
			\hline
			Timedelta & Integer & The timestamp of the packet in seconds within the window\\
			\hline
			Packet Length & Integer & Size of the packet in bytes\\
			\hline
			Domain & Categorical & Domain name of the server, if available\\
			\hline
			Direction & Categorical & Direction of the packet (outgoing/incoming)\\
			\hline
			Protocol & Categorical & Transport protocol of the packet (TCP/UDP)\\
			\hline
			Destination Port & Categorical & Port number of the destination address\\
			\hline
			Source Port & Categorical & Port number of the source address\\
			\hline
		\end{tabular}
	\end{center}
	\caption{{\bf Packet Level Features:} These features are generated for each packet in a given time window. }
	\label{table_packetlevel}
\end{table*}

\begin{table*}[]
	\begin{tabular}{|c|c|c|}
		\hline
		\textbf{Feature Category} & \textbf{Operations}                     & \textbf{Description}                             \\ \hline
		Incoming packet lengths   & count, sum, min, max, std, mean, median & Extract statistics on incoming packet lengths    \\ \hline
		Outgoing packet lengths   & count, sum, min, max, std, mean, median & Extract statistics on outgoing packet lengths    \\ \hline
		Protocols                 & count                                   & Packet counts for each protocol (TCP, UDP, ICMP) \\ \hline
		Domains                   & distinct                                & Distinct domain count                            \\ \hline
		Inter-event times         & count, sum, min, max, std, mean, median & Extract statistics on inter-packet time array    \\ \hline
	\end{tabular}
	\caption{{\bf Aggregated packet statistics features:} For each aggregation window, the feature vector consists of these values. The three feature representations are (1) Generating these features except inter-event times for each device, (2) Generating these features with inter-event times for each device, (3) Generating these features for each device and domain.}
	\label{table_packet_stats}
\end{table*}

We collected network data in pcap files on our monitoring server. The user study provided us the timings of each session for users to be able to label the respective data with the user id. We use sliding time window parameter (T) to split each session into multiple data samples labeled with the user of the session. The window slides by 60 seconds between the samples. For example, for T=10 minutes and session start time S, the first sample interval is from S to S+10 minutes while the second sample is from S+1 to S+11. After each sample time intervals and labels are defined, first step of processing was to fetch relevant information from each packet to strip out irrelevant information (e.g. the payload).

\subsubsection{Packet Attributes}

We parse the pcap traces and built a database that contains the packet header fields as the traffic is mostly encrypted and we want to build our system that relies on patterns over time as opposed to deep packet inspection which might put user privacy at risk. We further include the DNS queries in our database as this would make it easier for us to find the domain name of the external IP addresses that are communicated.
From the packet headers, we extract the timestamp, source IP, destination IP, protocol, destination port, source port, and packet length information. Since external IP addresses that these devices communicate for the same service can change over time, we decide to use the domain names instead of the IP addresses. We look for the DNS request messages in the dataset to find the domain names of the IP addresses. The domain name information might not always available due to the hard-coding of the IP address, meaning DNS request has not been made, or a request has been made a long time ago that our captured data does not contain it. We defined another domain name 'none' to fill in those cases. We could rely on SNI information in the TLS handshake. However, some devices do not use TLS and even if they do, they might not be using SNI to reveal the domain names. The internal IP, which is the IP address of the devices can also change over time and does not contain relevant information. Instead, we defined a field 'direction' to indicate if the packet is outgoing (from device to external server) or incoming (from external server to device). The absolute timestamp information is not relevant as we do not model the user behavior based on time of day or day of week. We instead use relative time in seconds within the time period makes more sense as we keep the sequential information and inter-packet times. Finally, we add the device name information to each packet since we know the MAC addresses of each device and each packet contains the MAC address.

Using these defined packet fields to train machine learning models is not scalable in terms of both training process as it will be more difficult to learn the behavior from a big feature space and small number of samples and real time testing process as it would require the system to collect all network traffic represent each packet individually. Therefore, some form of aggregation is necessary.

\begin{figure*}[ht]
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/heatmap_sum.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/heatmap_count.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/heatmap_average.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{High level device usage statistics from the user study. \\Total bytes sum, packet count and average bytes size over packets are plotted from left to right.}
	\label{fig:figure_deviceusage}
\end{figure*}

\begin{figure*}[ht]
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/heatmap_sumDomains.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/heatmap_countDomains.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/heatmap_averageDomains.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{High level domain usage statistics from the user study. \\Total bytes sum, packet count and average bytes size over packets are plotted from left to right.}
	\label{fig:figure_domainusage}
\end{figure*}

\begin{figure*}[ht]
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/user6_3selecteddevices.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/user6_8selecteddevices.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/user6_33selecteddevices.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{Usage of some devices for user 6 in 3 of the sessions}
	\label{fig:figure_user6_sessions}
\end{figure*}

\begin{figure*}[ht]
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/user8_17selecteddevices.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/user8_46selecteddevices.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/user8_55selecteddevices.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{Usage of some devices for user 8 in 3 of the sessions}
	\label{fig:figure_user8_sessions}
\end{figure*}

\begin{figure*}[ht]
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/user10_26selecteddevices.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/user10_49selecteddevices.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{figures/user10_54selecteddevices.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{Usage of some devices for user 10 in 3 of the sessions}
	\label{fig:figure_user10_sessions}
\end{figure*}

\subsubsection{Data Aggregation}

We explore different representations that would be suitable for our task of user classification. First observation we have is that different user actions lead to different user profiles and the actions that involve a smart device could be detectable by the respective device. This led us to think of device based aggregation methods as we know the device information for in the network traffic. We analyzed the data we collected to compare how different users were in terms of the devices they used. In the heatmaps in figure \ref{fig:figure_deviceusage}, you can see for some devices such as Amazon Echospot and Google Home, the amount of data exchanged varies by user. On the other hand, it becomes obvious that some smart devices are either not being used by the users or are not very active on the network. The patterns can be seen for total exchanged data amount as well as total packet counts and they look to be correlated.

We observed in the data that some devices, such as Amazon Echo Dot, communicated with multiple different domains depending on what the user asks. E.g. if the user listens to music, data comes from Spotify domain. In order to capture different user actions in a more fine grained way, we decided to include domain based aggregation to our feature set. We noticed that there are many similar names from the same second level domain, and they are used for the same tasks. Therefore, instead of using the full domain name, we group the data based on second-level domain names.
The high-level domain usage statistics is shown in figure \ref{fig:figure_domainusage}. It includes the second-level domains that has more than 10000 packets and their usage patterns per user. We observe the different usage patterns that suggests that modeling the users and separating them can be possible using domain based feature aggregation.

The first features we decided to define were the exchanged data size in both directions for each device and each domain. Having separate features for the direction of the data makes sense as it would be an important factor for personal voice assistants (e.g. Amazon Echo Dot, Google Home) since they transmit the voice data to the cloud servers for processing. The first set of aggregated features included the amount of data exchanged for each device for both directions. This simple aggregation method results in a single array summing packet lengths and counts. However, it loses the information about the timing and fine grained structure of the actions. For example, the user might have a pattern of using Amazon Echo Dot with 5 minute intervals, which would not be apparent in our feature vector. We decided to include information about inter-event timings for each domain and device. More specifically, we extract sum, count, mean, median, standard deviation, min and max values from the inter-event array where each element represents the time between the two consecutive packets belonging to the device or domain. We further decided to include the same statistical values for both incoming and outgoing packet lengths instead of having a single sum value to further include finer grained information. Furthermore, protocol usage information that includes packet sounds for TCP, ICMP, and UDP is also added to capture protocol usage differences for each device and domain. For each device, the distinct domain names contacted feature is also added. The list of features that we refined could be seen in table \ref{table_packet_stats}.

\section{Introduction}

User authentication is one of the major challenges in security, and while a diverse set of solutions
have been proposed, the quest for the perfect solution continues.
Traditional user authentication methods include  passwords, hardware tokens, and biometrics.  Given
the limitations of passwords~\cite{cranor2018pwd,cranor2016pwdmobile,cranor2016pwdperception}
several well-known techniques are used in practice to augment them
such as multi-factor authentication based on either a hardware token or a different channel  (e.g. email, SMS).
These methods impede the usability of authentication, and introduce additional latency~\cite{siadati2017mind,de2013comparative}.
Biometrics-based authentication has been extensively used in various forms (facial recognition, fingerprints, or retina scans), but revocation and user impersonations are well-recognized limitations of these methods.

\ignore{The omnipresence of IoT devices, as well as the smart households that use smart devices such as
sensors, locks, and voice-based personal assistants, have only made the environment for authentication more
complex, bringing new challenges.}

The omnipresence of IoT devices  such as sensors, locks, entertainment devices, and voice-based personal assistants, has made authentication in smart households  more complex, bringing new challenges. IoT devices are fundamentally different from personal devices and traditional authentication methods are either not applicable or break the natural flow of interaction with these devices. Deploying a screen or input pad for each of these devices is either not practical or costly. Some devices such as Amazon Echo Dot can perform voice-based biometric authentication as an optional feature. However, this feature can be circumvented as voice can be spoofed and users might not be comfortable with their voices being recognized due to privacy concerns~\cite{PrivacyIoT}. Therefore, better authentication methods are needed in the context of smart homes.

One of the opportunities in solving this problem in smart home environments is the relatively small number of users for which
authentication must be provided. Most American households range in size from two to six members. Multi-class classification algorithms can be trained for such small sets and with minimal
effort it can be ensured that adversaries do not interfere at training time.
However, there are several challenges in designing user authentication methods for
household IoT devices including: supporting a diverse set of devices with minimal
changes in their operation, maintaining user privacy,
and supporting a diverse set of policies.  Data that captures detailed user behavior (such as device logs) tend to be device-specific and more invasive
in terms of privacy.

Our approach addresses these challenges by using the
IoT devices' network traffic  from the home router as data source. In order to preserve user privacy
we collect and leverage the information in the headers of HTTPS packets for our authentication system. This data includes minimal information (timing, ports, bytes sent and received) with low risk on user privacy. Additionally, to protect user privacy we make the design choice of performing the model training and testing locally in the user's home, rather than remotely, in the cloud. This implies that the authentication module needs to reside in the smart home. To address the mis-match between user-level semantics and network-level semantics we
observe the network traffic for a continuous time window and design a machine learning (ML) model that uses features aggregated over a recent time window to predict the user in the room. Finally, we train a multi-class classifier to predict the likelihood that a certain user is in the room. The model is trained based on historical data obtained in controlled settings (knowing which user is in the room) and then used at testing time to classify users by generating in real time an authentication score based on the most recent observed activity.

We leverage the IoT lab at our institution set up as a studio apartment and purchase a set of 15 IoT devices from different categories, including voice assistants, smart kitchen appliances, and entertainment devices. We make use of the existing monitoring infrastructure in the lab to collect network packets (pcap files) from all these devices. We design an IRB-approved user study with multiple users participating in data collection over a period of three weeks. During the training period, we use labeled data of user sessions and extract features from HTTPS headers that capture user interaction with the IoT devices over a continuous time window. We design and test several ML classification algorithms for predicting the likelihood that a certain user is in the room. During the regular user interaction with the IoT devices, the authentication module continuously receives data extracted from the most recent observation window and computes an authentication score for each user. The score is made available to upper-level authorization systems that can implement flexible policies according to the level of risk tolerated. For example, these policies can be used for local services such as log in to a laptop, to perform operations on IoT devices, or to access cloud services, such as performing financial transactions.

Our experimental evaluation shows that a Gradient Boosting classifier achieves 81\% precision and 80\% recall at classifying six users, based on IoT device features. For five users the precision and recall are both 92\%, for a 25-minute observation window. We also design a high-confidence ensemble classifier with two models trained on disjoint set of features (one on 420 device features and the second on 2910 domain features). The ensemble generates an authentication score only when the two models agree on the prediction. An ensemble of two Gradient Boosting models achieves an F1 score of 0.86 for six users and 0.97 for five users (at the cost of not generating scores in 16 out of 91 sessions).

The rest of the paper is organized as follows. We define our problem and adversarial model in Section \ref{sec:bk}.
We describe our system design in Section \ref{sec:system} and our user study and data collection in Section \ref{sec:data}.
We present the evaluation in Section \ref{sec:evaluation} and related work in Section \ref{sec:relwork}.
We conclude the paper in Section \ref{sec:concl} and include some additional results in the Appendix.

\section*{Acknowledgements}

The authors would like to thank David Choffnes, Daniel Dubois, and Jingjing Ren for providing us access to the Mon(IoT)r Lab at Northeastern University and setting up the data collection infrastructure that enabled IoT device monitoring.  We thank Visa Research for funding this research. We would also like to thank Andres Molina-Markham for many discussions on contextual authentication.

\bibliographystyle{ACM-Reference-Format}
\interlinepenalty=10000

\section{System design}
\label{sec:system}
In this section we describe the goals motivating the design of our system and then give an overview
of our approach.

\subsection{Design Goals}

 In designing our system for user authentication in smart homes, we set forward the following design goals:

\myparagraph{Operate in a smart home environment with multiple users} We intend our system to be used in a smart home environment with several users actively interacting with diverse IoT devices. The average American household size in 2018 was 2.53, with most households (98.66\%) having size at most 6. Our system should provide good accuracy at the task of classifying users from a small set of previously seen users. Multi-class classifiers can be trained to predict the likelihood that a certain user is in the room.

\myparagraph{Support multiple heterogenous IoT devices} Compared to traditional computing platforms, IoT devices are extremely heterogeneous and diverse. This raises the challenge of designing a flexible, general monitoring platform that can support devices from different manufacturers and can be easily extended as new devices become available. Data available usually consists of event logs and network traffic. We observe that network traffic is much more general and more likely to support the diversity of IoT devices.

\myparagraph{Maintain user privacy} User privacy is an important design consideration, given the nature of the applications and
the environment where users perform their actions. We intend our system to work without access to user personal identifiable information (PII). Most network traffic of IoT devices is over HTTPS, with encrypted payloads in the transmitted packets.

\myparagraph{Capture user actions for behavioral modeling} User behavior is defined by their interaction with the devices and the
applications. User (and application) behavior results in several packets disseminated over the network. An approach based on more general information, like network traffic, needs  to
map low-level information from network packets to higher-level user actions (for instance, brew a coffee or change the smart lock settings) in order to capture user behavior. This creates a challenge as a single action could generate many packets addressed to different external destinations.

\myparagraph{Measure authentication confidence} Most systems for access control and authorization need to make an ultimate decision whether a user is allowed to access a critical resource or not. Our goal is to create an authentication service that generates \emph{authentication scores on a continuous basis}. These scores should provide a measure of authentication confidence, which is leveraged by higher-level authorization systems. For instance, to perform a high-value financial transaction, the authorization system could set the authentication score threshold at 0.95 and require another authentication factor (such as SMS). We believe that reliable authentication measures as we aim to design can greatly augment the flexibility of current authorization systems.

\begin{figure*}[ht]
	\includegraphics[width=0.8\linewidth]{system_iot.pdf}
	\caption{Architecture diagram of the data-flow and proposed authentication process}
	\label{fig:architecture}
\end{figure*}

\subsection{System Overview}
{\bf Our Approach.}  We address the challenges raised by our design goals by using the
IoT devices' network traffic from the home router. In order to preserve user privacy
we collect the headers of the HTTPS packets that include minimal information such as timing, ports, destination address, bytes sent and received, posing low risk on user privacy. Additionally, to protect user privacy we make the design choice of performing the model training and testing locally in the user's home, rather than remotely, in the cloud. This implies that the authentication module needs to reside in the smart home. To address the mis-match between user-level semantics and network-level data we \emph{observe the network traffic for a continuous time window} and design an ML model that uses features aggregated over a recent time window to predict the user in the room. Finally, we train a multi-class classifier to predict the likelihood that a certain user is in the room based on the most recent activities.

Figure \ref{fig:architecture} gives an overview of our system architecture. We installed 15 IoT devices in the IoT lab at our institution. We make use of the existing monitoring infrastructure in the lab to collect pcap files from all these devices. We design an IRB-approved user study with multiple users participating in data collection over a period of three weeks. During the training period, we use labeled data of user sessions and extract features from HTTP headers over a continuous time window (with varying length). We design and test multiple ML classification algorithms with the aim of obtaining good accuracy at identifying users. During the regular user interaction with the IoT devices, the authentication module continuously receives data extracted from the most recent observation window and computes an authentication score for each user. The authentication score can be used by upper-level authorization systems for flexible policies according to various levels of risk. For example, log in to a local device might require a lower score than performing a financial transaction in the cloud.

Below we provide more details about the machine learning models we used and the features we selected for our multi-class classifier.

\input{mldesign.tex}

\input{features_new.tex}

\subsection{Machine Learning Design for User Classification}

We design a machine learning (ML) system that: (1) learns user profiles over time, using data extracted from network traffic generated by IoT devices; and (2) computes an authentication score in real time based on the most recent observation interval. The authentication score estimates the confidence or probability of the actual user being in the room.

In more details, the ML system is trying to learn profiles for a set of users $\users=\{U_1,\dots, U_m\}$. In our specific application, user observation is performed over time, and thus the features or user attributes need to be defined over an observation window. We design the system to predict at time $t$ the probability that a certain user is in the room, i.e.,:

$$A^t_i = Pr[U_i^t|F(t,\Delta)], \forall i \in [1,m]$$

\begin{table*}[th]
	\begin{center}

        \begin{tabular}{|c|c|c|}
			\hline
			{\bf Attribute} & {\bf Type} & {\bf Description} \\
			\hline
			Device Name & Categorical & Name of the smart device (extracted from the MAC address) \\
			\hline
			Time & Integer & The timestamp of the packet \\
			\hline
			Packet length & Integer & Size of the packet in bytes\\
			\hline
			Domain & Categorical & Domain name of the destination, if available\\
			\hline
			Direction & Categorical & Direction of the packet (outgoing/incoming)\\
			\hline
			Protocol & Categorical & Transport protocol of the packet (TCP/UDP/ICMP)\\
			\hline
			Destination Port & Categorical & Port number of the destination address\\
			\hline
			Source Port & Categorical & Port number of the source address\\
			\hline
		\end{tabular}
	\end{center}
	\caption{{\bf Packet-level attributes:} These fields are available for each HTTPS packet from its header. }
	\label{table_packetlevel}
\end{table*}

\begin{table*}[htb]
	\begin{tabular}{|c|c|c|c|}
		\hline
		\textbf{Type} & \textbf{Feature Category} & \textbf{Operations}                     & \textbf{Description}                             \\ \hline
		Device/Domain & Incoming packet size   & count, sum, min, max, std. dev., mean, median & Statistics on incoming packets
\\ \hline
		Device/Domain & Outgoing packet size   & count, sum, min, max, std. dev., mean, median & Statistics on outgoing packets    \\ \hline
		Device/Domain & Protocols                 & count                                   & Packet counts per protocol (TCP, UDP, ICMP) \\ \hline
		Device/Domain & Inter-event times         & count, sum, min, max, std. dev., mean, median & Statistics on inter-packet timing    \\ \hline
		Device & Domains                   & distinct                                & Distinct domain count                            \\ \hline
	\end{tabular}
	\caption{{\bf Features aggregated by time window.} }
	\label{table_packet_stats}
\end{table*}

Given an observation or feature set $F(t,\Delta)$ computed over the time window $[t-\Delta,t]$ of size $\Delta$ (most recent history of length $\Delta$), the system aims to predict an authentication score $A_i^t$ at time $t$ for each user $U_i$. This can be achieved by training a multi-class classifier $f \in \Hcal$,  where $\Hcal$ is the hypothesis space. The ML algorithm is given historical training data of labeled user activities: $\dtrain=\{(F_1(t_1,\Delta),L_1),\dots,(F_n(t_n,\Delta),L_n)\}$, with labels $L_i \in \users$ and $F_i(t_i,\Delta)$ the set of features computed over observation intervals $[t_i,t_i-\Delta]$ of length $\Delta$. At the end of training, a model $f$ is selected to minimize a certain loss function on the training set.

\ignore{
$$\min_{f \in \Hcal} \ell(f(x_i^{\Delta},L_i,{\bf w})),$$
where ${\bf w}$ is the vector of model parameters learning during training.
}

A probabilistic model in fact estimates the probability that a user generates the observed features in the most recent time window of length $\Delta$. The sum of the predicted probabilities is always 1:

$$\sum_{i=1}^m A_i^t = \sum_{i=1}^m Pr[U_i^t|F(t,\Delta)] = 1,\forall t $$

At testing time, the model is applied at time $t$ to the most recent observation window (i.e., features $F(t,\Delta)$ extracted during time interval $[t-\Delta,t]$) and generates an authentication score $A^t_i$ for each user $U_i \in \users$.

We experiment with multiple ML classification algorithms, including Logistic Regression, Random Forest, and Gradient Boosting that fit our framework very well. The most challenging issue in our design is to create appropriate feature representations that capture user behavior over the recent time window and can be used to effectively differentiate multiple users with high confidence. We discuss multiple feature representations next.

\section{Related work}
\label{sec:relwork}

Behavior-based authentication systems are identifying users based on their behavior. Implicit authentication \cite{shi2010implicit} shows the applicability of behavior modeling for authentication by utilizing the call/message information, browser activity, and GPS history. Itus provides an extensible implicit authentication framework for Android~\cite{Itus}. A survey of multiple implicit authentication methods is given in \cite{SurveyImplicit}. Progressive Authentication \cite{riva2012progressive} models user behavior on mobile devices by combining biometric features and sensor data. Another emerging continuous authentication method leverages the sensor information from wearable devices (e.g. smart watches, activity trackers, glasses, bracelets) to learn user behavior~\cite{gafurov2006biometric,ZEBRA,peng2017continuous}.

Behavioral authentication in other contexts has also been studied. Freeman et al.~\cite{freeman2016you} designed an ML approach for clustering logins based on their IP, geolocation, browser, and time, for authenticating users to online services. Device fingerprinting has been used to augment user authentication on the web~\cite{Alaca16}.

Authorization and access policy frameworks in the IoT systems have also been an active research area. He et al.~\cite{he2018rethinking} conducted a survey of IoT device authorization preferences and observed that most devices have limited authorization flexibility. ContexIoT~\cite{ContexIoT} is a  permission system for IoT apps that provides users contextual information to enable them to make better access control decisions. SmartAuth~\cite{SmartAuth} generates a user interface for authorization decisions in IoT apps. Soteria~\cite{Soteria} performs static analysis to identify if IoT apps respect security policies. {\sf IoTGUARD} dynamically enforces security policies in IoT devices based on monitoring event handlers by IoT apps~\cite{IoTGuard}.

There have been prior work investigating user authentication in the smart home environments. Emami-Naeini et al.~\cite{PrivacyIoT} perform a user study on IoT privacy and demonstrate that users are not comfortable with biometric data collection in IoT settings. Biometric based authentication is optional in Alexa~\cite{alexa_voice_setting}, but this is prone to recording and replay attacks. Shi et al. \cite{shi2017smart} proposed an authentication system leveraging physical properties of WiFi signal generated by IoT devices. Apthorpe et al. \cite{apthorpe2017spying} show that user actions can be inferred from encrypted network traffic generated by IoT devices. In this work, we utilize the network communication statistics of IoT devices through user interactions to build a user authentication framework using machine learning. Our system computes an authentication score for each user which can be used by services to enforce flexible policies depending on the sensitivity of the service.

\section{System Implementation}
\label{sec:data}

\begin{table}[tbh]
	\begin{tabular}{|c|c|c|c|c|c|}
		\hline
		\textbf{User} & \textbf{Sessions} & \textbf{Total Time}  & \textbf{Avg. Duration} \\ \hline
		\textbf{1}    & 9                      & 184                                  & 20               \\ \hline
		\textbf{2}    & 5                      & 181                                    & 36               \\ \hline
		\textbf{3}    & 11                     & 330                                    & 30               \\ \hline
		\textbf{4}    & 8                      & 206                                  & 25               \\ \hline
		\textbf{5}    & 2                      & 38                                   & 19               \\ \hline
		\textbf{6}    & 12                     & 263                                   & 21               \\ \hline
		\textbf{7}    & 4                      & 71                                   & 17               \\ \hline
		\textbf{8}    & 9                      & 154                                    & 17               \\ \hline
		\textbf{9}    & 6                      & 112                                  & 18               \\ \hline
		\textbf{10}   & 8                      & 121                                   & 15               \\ \hline
	\end{tabular}
	\caption{The number of sessions, total time,  and average session duration per user, all in minutes.}
	\label{table_session_stats}

\end{table}

\subsection{User study}
\label{sec:userstudy}

We performed a user study with multiple users over three weeks to generate the dataset used to train and test our authentication models. We utilize the IoT lab at our institution to conduct our user study and monitor users while they interact with a smart-home like environment. The lab is an enclosed studio with a kitchen and living area, equipped with a range of Internet-connected devices and appliances to simulate a smart household. The lab is available for  use by researchers at our institution. We collected data from 15 IoT devices installed in our IoT lab, as described in Table \ref{table_activities}.

We asked users to use the room for multiple sessions, each of them lasting at least 15 minutes. One of our requirements was that users are alone in the room while we perform the user study. The reason is to facilitate data labelling and to ensure that we do not monitor data generated by other lab users. Users logged their start and end times using their mobile phones in order to label the data for each session. We held several orientation sessions before the actual data collection, to help users become familiar with the devices. We did not provide any scripts and users were asked to interact with the devices in a natural manner.

In total, we recruited 10 users, who participated in the study and generated 74 sessions with a duration of 1660 minutes. The usage statistics per user are in Table \ref{table_session_stats}. We collected 4,082,975 packets, including connections to 316 external destinations and 97 second-level domains. In the rest of the paper, we focus on six users: 1, 3, 4, 6, 8, and 10 who generated more than 6 sessions.

\myparagraph{IRB approval} Our study was approved by the IRB at our institution. All researchers with access to the collected data performed the IRB training and users were required to consent to the data being collected while they use the room. Users were informed of the project goal before participating in the study. As we only extracted fields from packet headers of HTTPS traffic, we did not have access to any user personal information. To further protect the privacy of users, we anonymize the collected data by creating a unique user ID for each study participant. Our data is stored in anonymized format on our servers, not directly identifying the participants.

\myparagraph{Implementation}  The lab network is monitored at all times and the network traffic (pcap files) of all the IoT devices is collected in a server located in the room. We did not collect data from users' personal devices such as laptops and phones, as we used a MAC address filter to collect only IoT device traffic.  We created software that parses the HTTPS packet headers and stores the fields from Table~\ref{table_packetlevel} in a Postgres database. We extracted the device and domain features and stored them in a different table in our database.  For training our models, we use the ML implementation from the {\sf Python scikit-learn} package. We performed cross-validation for all our experiments and varied the ML hyper-parameters.

\begin{figure*}[th]
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{heatmap_sum.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{heatmap_count.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{heatmap_average.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{Device usage statistics during the user study. \\Total bytes, packet count, and average packet length are plotted per device from left to right.}
	\label{fig:figure_deviceusage}
\end{figure*}

\begin{figure*}[htb]
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{heatmap_sumDomains.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{heatmap_countDomains.pdf}

	\end{minipage}
	\hspace{0.5cm}
	\begin{minipage}[b]{0.3\linewidth}
		\centering
		\includegraphics[width=\textwidth]{heatmap_averageDomains.pdf}

	\end{minipage}
	\captionsetup{justification=centering,margin=2cm}
	\caption{Domain usage statistics during the user study. \\Total bytes, packet count, and average packet length are plotted per domain from left to right.}
	\label{fig:figure_domainusage}
\end{figure*}

\subsection{Data exploration.} We analyzed the collected data  to compare different users in terms of device and domain usage. In Figure \ref{fig:figure_deviceusage}, we show the total data exchanged by device and user, the number of packets, as well as the average length of packets. For some devices such as Amazon Echospot and Google Home, the amount of data transferred varies significantly by user. On the other hand,  some smart devices (such as the microwave and smart kettle) are either not utilized by users, or they do not generate a lot of network activity. Statistical features such as average packet length provide more distinguishing patterns across users than total number of packets or total data transferred.

In Figure \ref{fig:figure_domainusage} we display the amount of data transferred, the total number of packets, and the average packet length by user and second level domains. We only included domains with more than 10,000 packets for at least a user. While all devices communicates with the cloud and CDN domains (e.g., {\sf cloudfront.com}), some domains (such as {\sf iheart.com) exhibit different patterns across users.

Next, we are interested in validating the hypotheses that users have consistent behavior in multiple sessions and users have differentiating behavior from other users. For this, we plot the amount of data transmitted by different devices over time for User 6 (in Figure~\ref{fig:figure_user6_sessions}) and User 10 (in Figure~\ref{fig:figure_user10_sessions}) in three distinct sessions. Interestingly, some sessions of the same user are  similar (for instance, the first and third session of User 6). However, users sometimes deviate in their behavior across different sessions, as demonstrated by the third session of User 10 (in which interaction with Echo Dot is lower). Overall, Users 6 and 10 have different behavior in their interactions with smart devices. However, while there is some similarity across the same user's sessions, these plots also show that user behavior varies across sessions, making behavioral authentication in this context challenging.