\section{Introduction}

Recurrent Neural Networks (RNN) are a machine learning technique which specializes in modelling sequential data due to their ability to capture and retain long term information.
Recent years have seen great success in applying these models to sequential data, such as language modelling~\cite{mikolov2010recurrent}, sentiment analysis~\cite{tang2015document}, and machine translation~\cite{sutskever2014sequence}.
However, despite their efficacy, the actual features and patterns which RNNs learn to model can be opaque, difficult to understand, and may even include unintended biases~\cite{bolukbasi2016man}.
This is a result of the black box nature of these systems which learn to model training data with little guidance about how to actually do so.

Visualizations are a powerful tool that can be employed to expose the inner details of these opaque models, taking a step towards better understanding and trusting their results~\cite{ribeiro2016should}.
However, understanding and visualizing RNNs comes with its own set of challenges.
One such challenge is that of interpreting the semantics of hidden states, vectors which encode information within the RNN.
These hidden states represent information in complex and highly non-linear ways, making them particularly difficult to interpret.

We aim to visualize some of the intuitions behind the internal representations RNNs learn to model.
We design a novel visual metaphor to address some of the challenges of visualizing RNNs, and in particular their hidden states.
This is Predictive Semantic Encodings (PSE) which are an interpretation of hidden state semantics that can be easily visualized and facilitate high level comparisons.
We develop a proof of concept interactive tool that uses this technique to visualize an overview of the Recurrent Neural Network.

\section{Related Work}\label{relatedwork}
Previous work exists in the field of visualizing RNNs, and in particular in terms of interpreting their hidden states.
One successful approach to this problem has been to interpret patterns of the hidden state activations with respect to the task objectives.
For example, Karpathy et al.~\cite{karpathy2015visualizing} show that some activations within the RNNs track long term syntactic language features, while Li et al.~\cite{li2015visualizing} use heat maps to understand the salience of activations and words with respect to the model output.
A similar heat map technique is used to show the ``attention'' a model pays to its sequential inputs at different timesteps across the recurrence relation~\cite{bahdanau2014neural, levy2018long}.

Although these techniques mainly focus on static visualizations, these and other works have been extended to incorporate interactive and complex visualizations.
LSTMVis~\cite{strobelt2018lstmvis} follows the approach of visualizing hidden state activations across time, allowing users to form and test hypothesis about the sequences evoking these patterns.
Rather than focusing on activation patterns across time, Ming et al.~\cite{ming2017understanding} relate hidden states to the model inputs and outputs through co-clustering and quantifying the model's expected response to these hidden states.
Kahng et al.~\cite{kahng2018cti} show and compare specific subsets of activations based off similar test instances.

More recently, Strobelt et al.~\cite{strobelt2019s} incorporate interpretation of hidden states within the context of visualizing a machine translation pipeline, allowing for users to debug multiple stages of the translation process.
Rather than interpret the RNN hidden states of a model with fixed parameters, Cashman et al. ~\cite{cashman2017rnnbow} instead focus on visualizing the training process itself by showing the gradient updates during the RNN learning process.

These studies all focus on visualizing single types of the RNN hidden states, such as their embeddings or memory cells.
They also use various techniques to relate the hidden states back to the inputs or outputs of the task.
However, to our knowledge previous work does not focus on representing the semantics of hidden states with respect to the task input/outputs in such a way that this interpretation may be visually compared across the various types of hidden states within the RNN.

\section{Background}\label{background}
A Recurrent Neural Network is a mathematical framework which maps a sequence of inputs $\{\mathbf{x}_1, .., \mathbf{x}_T\}, \mathbf{x} \in \mathbb{R}^V$ to a sequence of internal states $\{\mathbf{h}_1, .., \mathbf{h}_T\}, \mathbf{h} \in \mathbb{R}^N$.
Each $\mathbf{h}_t$ is updated by the non-linear function $\mathbf{h}_t \leftarrow RNN(\mathbf{x}_t, \mathbf{h}_{t-1})$.
This recurrent nature allows the previous hidden states $\mathbf{h}_{t-1}$ to capture the context necessary to compute the next RNN update.

Within the RNN formulation itself, any number of operations may be performed when leading from $\mathbf{x}_t$ and $\mathbf{x}_{t-1}$ to $\mathbf{h}_t$.
Typically, each of these operations is written as a vector definition, further describing the different kinds of hidden states comprising the model.
We refer to the concept of a vector definition, including that of the final hidden state $\mathbf{h}_t$ itself, as the ``kinds'' of hidden states of the RNN.
Each kind of hidden state describes a specific component of the recurrent model.

Depending on the specific task against which the RNN is deployed, the final hidden state of a timestep is fed into a function $F(\mathbf{h})$ to produce an output $y$.
In the case of a classification problem, $\mathbf{y} \in [0, 1]^K$ represents a probability distribution across $K$ labels.

This study looks specifically at the Long Short-Term Memory architecture (LSTM)~\cite{hochreiter1997long}, a specific type of RNN.
The detailed mathematical notation of the LSTM is laid out in the Appendix.

\section{Design}\label{design}
We set out to design a visualization technique to help researchers better understand the general intuitions RNNs capture.
Our study follows the nested model of visual design and validation described by Munzner~\cite{munzner2009nested} which lays out four layers, each built off the one before.

\subsection{Layers 1 \& 2: Domain \& Abstraction}
We preface the design by first describing the user groups interested in studying RNNs at this level of detail.

Strobelt et al.~\cite{strobelt2018lstmvis} develop three high level categories of user groups interested in RNNs; Architects, Trainers, and End Users.
We look specifically at users from the various groups with some degree of pre-existing understanding of neural networks, primarily Architects and Trainers.
These users want to understand the detailed behaviours RNNs learn to model, whether that understanding may be used to drive development of architectural improvements or to gain confidence that their results generalize beyond the training data.
Moreover, these users want an intuition for what the RNN represents in order to better grasp how it functions.

From this group of users elucidate a series of questions which they are interested in asking.
1) What information flows and changes from the inputs through to the outputs of the RNN?
2) Where are changes more and less prominent?
3) Does our intuition about the role of the mathematical components match that of the actual RNN behaviour?
At a high level, all these questions seek to understand how the model behaves from a mathematical/computational perspective.
From these questions, we ascertain abstract operations and data types which can later be mapped to visualization techniques.

Vectors are the most natural concept to focus on for study, not simply because there is precedent in previous works~\cite{li2016understanding, kahng2018cti, strobelt2018lstmvis}, but mainly because the RNN description uses vectors as the core element of notation.
Focusing on vectors ties directly to how these users naturally describe these models, ensuring visualization accessibility.

Implicit within the concept of RNN hidden state vectors are the dimensional values they encode, typically referred to as activations.
These activations are the focus of much of the previous work in visualizing RNN details.
However, in the context of visualizing high level information flow, it is dubious what information may be gleaned by observing the activation values themselves.
That is, visualizing these numeric values without a relatable interpretation of their meaning may not provide insights into the model behaviour.
Therefore, for the purposes of this work which intends to explore the flow of information throughout the RNN, we choose to abstract away activation details into the concept of a hidden state as a whole.

With this data type in mind, we explore the options for visualization operations that may be performed on hidden state vectors.
Since users want to interpret hidden states as a whole, we propose hidden state comparisons as the primary visualization operation.
By comparing the change in hidden states along the components of the RNN, these types of high level questions about the flow of information may begin to be answered.

\subsection{Layers 3 \& 4: Encoding-Interaction \& Algorithm}
Continuing with the final two layers of the nested model of visual design, this section describes the development of a visual encoding and algorithms to address requirements from the previous section.

We recall from the previous section the desire to express the intuitions encoded within RNN hidden states as a whole.
To do so, we introduce the semantics of hidden states as a probability distribution over the task output labels.

More formally, we consider the context free\footnote{The function is context free in that it makes a prediction similar to that of the RNN, but without any surrounding hidden state context.} function $G(\gamma, \mathbf{v})$ which produces a probability distribution over the outputs $\mathbf{y}$, where $\gamma$ denotes the specific kind of hidden state of the instance $\mathbf{v}$.
Indeed, this formulation is analogous to the classification task of the RNN itself, and the function $F(\mathbf{h})$ can be seen as a specialization of $G$ such that $F(\mathbf{h}) = G(\text{final-hidden-state}, \mathbf{h})$.

The function $G$ is modelled separately from that of the actual RNN and its classification function $F$, allowing it to represent arbitrary hidden states.
$G$ is trained on the hidden state instances derived from the sequential dataset once training the RNN is complete so as to capture the final semantics learned by the model.

Although the task of the RNN is to specifically predict the output labels, the semantic we propose may similarly be used to encode hidden states with respect to the input labels they correspond to.
It also may be used to predict the output labels at varying timesteps of the RNN, so that differing layers of semantics may be expressed by any particular hidden state.
The choice of which semantic(s) to use will depend on the specific visualization task.

Notice, the notion of relating hidden states in some way to the task inputs or outputs is not a new one, as for example is done in RNNVis~\cite{ming2017understanding}.
However, to our knowledge other work has not drawn a direct parallel between hidden states encoding a probability distribution similar to that of the task itself.

The advantage to this semantic formulation is that hidden states can share the same visual encoding as that of the RNN output prediction.
Moreover, this formulation describes hidden states in a way that is invariant of their internal significance.
These attributes allow users to seamlessly transition between comparisons of hidden state semantics amongst the components of the RNN as well as the task itself.

With these semantics in place, we develop a simple visual encoding designed to represent probability distributions across typical outputs of RNNs.
We use a mini-bar chart where bars represent class labels and the bar length represents the probability magnitude.
Since there may be many class labels, only the top-$k$ probabilities are shown in the bar chart, where $k$ is selected depending on the visualization context.
The class labels are shown in descending order based off the probability distribution so that the mini-bar chart represents the most likely outcomes.
Finally, each bar is coloured according to a user specified colour mapping, e.g. to show information about parts of speech.
This allows for designers to draw attention to class labels as applicable to the visualization task.

We augment the mini-bar chart with a redundant encoding that describes the probability distribution more holistically and facilitates a quick, high level comparison of the hidden state semantic.
The redundant encoding uses a simple coloured rectangle, where the colour also comes from the visualization's colour mapping.
The colour is produced by first selecting which mapped colour holds the largest sum of probabilities in the top-$k$ predictions from the distribution.
Then, a point is interpolated inversely proportional between this most likely colour and the colour ``white''.

By example, if the mass of the probability distribution describes predictions all mapped to the same colour, then the coloured rectangle will reflect this colour without any white dilution.
However, if the mass of the probability distribution is roughly uniform across the colour mapping, then the most likely colour will be diluted to almost completely white.
In this way, the colour interpolation encodes the degree by which the probability distribution represents the dominant class labels.
Figure \ref{fig:visualencoding} shows examples of this visual encoding.

\begin{figure}
 \centering
 \textbf{PSE Visual Encoding}\par\medskip
 \includegraphics[scale=.5]{pictures/colour-fitting-variance.png}
 \caption{Visual encoding for the PSE with various probability distributions.
 In each case, the dominant colour of the probability distribution is green to the degrees of 1) 90\%, 2) 50\%, and 3) 33.33\%.}
 \label{fig:visualencoding}
\end{figure}

\subsection{Interactive Visualization Framework}
We implement a simple proof of concept tool focused on observing the RNN hidden states and architecture as a whole.

As comparing hidden states is important for understanding the flow of information throughout the model, the visualization shows many of the different kinds of hidden states comprising the model.
These are arranged according to their relative position in terms of the progression of time $t$ in the sequential input data, where time is represented as progressing from top to bottom in the display.
Additionally, the hidden states are arranged relative to their dependence in the RNN definition, so that those more directly dependent on the inputs $\mathbf{x}_t$ are positioned to the left of the visualization, while those closer to the outputs $\mathbf{y}_t$ are placed on the right.

The visualization tool allows for users to provide arbitrary sequential inputs to observe and compare the elicited hidden state representations.
It should be noted that this high level view of the RNN architecture is particularly suited for showing the intuition and conceptual interaction behind the detailed components of the model.

\section{Evaluation}\label{evaluation}
We evaluate our design with the following set of studies.
Indeed, these are preliminary results that encourage further follow up.

Although the proposed design element is applicable to any sequential modelling task, we focus on the well known language model task.
A language model is a natural language processing problem whereby each input word must produce a probability distribution describing the likelihood of the words that follow it.

We evaluate these studies on the Penn Treebank~\cite{marcus1993building} language model dataset, trained on a $300 \times 2$ LSTM.
Numerals and infrequent words are mapped to the \textbf{NUM} and \textbf{UNK} tokens, so that the resulting vocabulary consists of 14787 unique words.
The dataset contains 54020 and 6754 training and testing sentences, and the model is trained to achieve a perplexity of $66.5$ on the held out test set.
For the visualization, we use a colour mapping based on coarse level part of speech tags to highlight grammatical switches in the language model.
The colour interpolation for the PSE sets the top-$k$ as 10\% of the vocabulary.

\subsection{Case Study: Visualizing Information Flow}\label{evaluation:1}
In this case study, we explore the high level flow of information through the RNN.
The user chooses the input \textbf{\detokenize{``} \texttt{we stand in solidarity ,} \detokenize{''} \texttt{she emphasized .}} to observe the transition from the direct quote to end of the sentence.
The full visualization for this input and other examples can be found externally~\cite{Sawatzky_2019}, while Figure \ref{fig:teaser} shows just hidden states from the final layer of LSTM for the \nth{7} and \nth{8} timesteps.

The language model, who's outputs are visualized under the $\textbf{y}_t$ labels in the figure, has learned to immediately predict verbs or nouns to follow the end of quotation sequence.
By looking at the hidden state PSEs for the top row, although some of the colour interpolations agree with this semantic, it appears that the RNN is representing some degree of uncertainty at this stage in the state it models.
This uncertainty is manifested two fold, firstly by the weakly coloured purple and green interpolations, and secondly by the $\mathbf{l}_7^2$ and $\mathbf{c}_7^2$ hidden states instead forming a yellow-based interpolation.

Interestingly, the bottom row corresponding to the input of the word \textbf{she} appears to represent much more certainty in its prediction of a verb to follow.
This is an indication that the hidden states of the RNN at this point are generally confident about what will follow in the language model.
Since the input at this point is a noun, at a high level this explains how the internal state of the RNN has resolved the uncertainty from the previous timestep.

Focusing more closely on these PSE details, other aspects of the RNN internal behaviours are revealed.
In both rows of the figure, the previous Cell State $\mathbf{c}_6^2$ and $\mathbf{c}_7^2$ has interpolated a mainly white colour, indicating how the PSE for these hidden states represent a distribution without any particularly dominant label grouping.
This roughly uniform distribution can be further observed by looking at the top-3 predicted classes, which all sit flat together with equal probabilities.
By conveying this information, the PSE visual encoding shows how the previous Cell States must be gated to formulate the Long-term Memory before they can indicate a meaningful prediction.

Another interesting example is in $\mathbf{l}_8^2$, which indicates a probability distribution generally predicting green nouns, despite the fact that the input at this timestep should have switched the semantics to a purple verb prediction.
Although this PSE interpolation is a faded green colour, it shows how the Long-term Memory is still retaining semantics which indicate a noun prediction.
However, when this hidden state is combined with the Short-term Memory of $\mathbf{s}_8^2$ to produce $\mathbf{c}_8^2$ we see how the internal representation of the LSTM has switched to indicating a verb.
This reveals how the internal mechanics of the LSTM at this point in the model are combined to make a prediction, with the semantics from the Short-term Memory passing through as a result of that interaction.

Notice this effect is not predetermined, as seen for example across the same interaction in the previous timestep.
Instead there, it appears that the semantics of the Long-term Memory $\mathbf{l}_7^2$ win out when building $\mathbf{c}_7^2$.
At that point, it appears to be the Output Gate (not visualized) which switches out the indication of a yellow-function word to produce the final hidden state $\mathbf{h}_7^2$ predicting a noun/verb to follow in the model.

Observing the RNN model at this level of detail begins to reveal important nuances to the decisions its makes.
Moreover, these nuances are made available particularly through the visualization and comparison of the different hidden states comprising the model.

\subsection{Accuracy of Predictive Semantic Encodings}\label{evaluation:2}
This experiment investigates the accuracy of the Predictive Semantic Encoding model formulation.
Additionally, we use this formulation as a means by which the hidden states of the RNN can be understood.

In order to draw parallels with the RNN, we train and test on the full dataset of hidden states derived from the original sequential training and testing data.
This results in a total of 20M training and 2.6M testing data pairs across the 17 various LSTM hidden states (1 word embedding + 8 hidden states across 2 layers).

The PSE is evaluated using perplexity, which is the same metric as that of evaluating the language model task.
\begin{align}
Perplexity(X) & = \sqrt[T]{\prod_{t=1}^T \dfrac{1}{P(x_t|w_1...w_{t - 1})}}
\label{equation:perplexity}
\end{align}
With $X$ representing the sentence input and $P(x_t|x_t...x_{t - 1})$ as the model's conditional probability of a word $x_t$ given the context of previous words $x_t...x_{t - 1}$.
As perplexity is normalized over the length $T$ of the input, it is suited for comparisons between the sequential RNN data and that of the non-sequential PSE.

Figure \ref{fig:pse} shows the results of this experiment using a simple linear classifier as the PSE function $G$.
Notice, $G$ is modelled with a separate set of parameters for each kind of hidden state.

\begin{figure}
 \centering
 \textbf{PSE Accuracy per Hidden State}\par\medskip
 \includegraphics[scale=.35]{pictures/pse-perplexity-projection-breakdown3.png}
 \caption{Per hidden state test set perplexity of the fully trained PSE using a linear classifier.
 Note the y-axis is truncated at $400$, despite some values exceeding this range.}
 \label{fig:pse}
\end{figure}

Firstly, we ensure parity with the RNN by examining the \nth{2} layer Output hidden state performance.
This hidden state achieves a test set perplexity of $70$, the best of all the hidden states, and critically this value is on par with the RNN which achieved a perplexity score of $66.5$.
This result establishes that the PSE are an accurate generalization of the RNN classification function $F$.

Next, the result that immediately stands out is at the \nth{2} layer for the Long-term Memory and Cell hidden states.
These values are several orders of magnitude larger than any other hidden states.
Despite this startling result, this finding actually explains an important aspect to the RNN architecture.
Specifically, the Long-term Memory and Cell States represent information that spans through the LSTM across time, rather than only containing information relevant to the current timestep.
As a result, these hidden states, especially at the final layer of the LSTM, are not reliable indicators of what final prediction will be made by the language model.

Another way to think about this is that the information from these hidden states is highly dependent on other pieces of context represented throughout the LSTM in making a final decision at any given timestep.
Therefore, it is not surprising that the Long-term Memory and Cell behave so poorly for this task.

We observe that the best performing hidden state is the Output of the final layer.
This is what we expect, since the model for this hidden state is equivalent to that of the actual RNNs classifier $F$.
Notice, the only difference in the LSTM architecture between the Output and the Cell State, which contrast the best and worst case results in the figure, is the Output Gate.
This fact highlights the important role this particular gating unit plays in the RNN.

It is worth noting that although these several kinds of hidden states are poor estimators of the RNN itself, that does not mean they perform poorly as Predictive Semantic Encodings.
Rather, this finding suggests they are working correctly, since the idea of a PSE is not to mimic the RNN, but instead to encode some sense of relatable semantics of hidden states as a whole.
Since some hidden states naturally bear little stand-alone representational power, we expect their semantics to be highly ambiguous as these findings indicate.
With an appropriate visual encoding, this ambiguity will pop out in a visualization and help lead users to the same conclusion about the semantics of these hidden states.

\section{Summary}\label{summary}
We introduce a technique for representing hidden states called Predictive Semantic Encodings (PSE).
PSEs are a formulation that expresses the meaning of hidden states as a whole in relation to the task inputs or outputs.
We propose a visual encoding for this technique that is comparable across differing kinds of hidden states, making it a powerful tool in visualizing the interaction between components of RNNs.

Although this idea is simple, its key contribution is allowing for visualizations to show the high level intuitions captured within hidden states.
Moreover, its compact form make rendering many hidden states at once possible, opening avenues for visualizations to better investigate the internal details of these complex models.

We use this technique to perform a preliminary investigation into a realistic task and dataset; that of a language model.
This visual encoding brings to light previously un-visualized aspects of the RNN model behaviour.
Moreover, it highlights various areas of interest in the model such as where significant shifts occur in hidden state semantics.
The evaluation concludes by showing how the PSE is equivalent to the model classifier, as well as by using it to explain some of the complexity represented across the hidden states of the RNN itself.

\acknowledgments{
We thank the anonymous reviewers who's comments and suggestions greatly helped to improve and clarify this manuscript.}

\bibliographystyle{abbrv-doi}


\section{Related work}
\vspace{-.1cm}
\label{sec:related_work}

Our work is related to robotics manipulation such as grasping~\cite{Lampe2013AcquiringLearning}, opening doors~\cite{Gu2016}, screwing the cap of a bottle~\cite{Levine2015End-to-EndPolicies} and cube stacking~\cite{Popov2017Data-efficientManipulation}.
Such tasks have been addressed by various methods including imitation learning (IL)~\cite{2017arXiv170307326D} and reinforcement learning (RL)~\cite{SACX}.

\noindent {\bf Imitation learning (IL).} A neural network is trained to solve a task by observing demonstrations. Approaches include behavioral cloning (BC)~\cite{Pomerleau1989} and inverse reinforcement learning~\cite{Ng2000}. BC learns a function that maps states to expert actions~\cite{Ross2014, Pinto2017}, whereas inverse reinforcement learning learns a reward function from demonstrations in order to solve the task with RL~\cite{Ho2016, Kumar2016LearningImitation, Popov2017Data-efficientManipulation}. BC typically requires a large number of demonstrations and has issues with not observed trajectories. While these problems might be solved with additional expert supervision~\cite{Ross2014} or noise injection in expert demonstrations~\cite{Laskey2017}, we address them by improving the standard BC framework. We use recent state-of-the-art CNN architectures and data augmentation for expert trajectories. This permits to significantly  reduce the number of required demonstrations and to improve performance.

\noindent {\bf Reinforcement learning (RL).} RL learns to solve a tasks without demonstrations using exploration. Despite impressive results in several  domains~\cite{Silver2016MasteringSearch, Mnih2015Human-levelLearning, Kober2013, Gu2016}, RL methods show limited capabilities when operating in complex and sparse-reward environments common in robotics.
Moreover, RL methods typically require prohibitively large amounts of interactions with the environment during training.
Hierarchical RL (HRL) methods alleviate some of these problems by learning a high-level policy modulating low-level workers. HRL approaches are generally based either on options~\cite{sutton1999between} or a feudal framework~\cite{Dayan1993FeudalLearning}. The option methods learn a master policy that switches between separate skill policies~\cite{MetaLearningSharedHierarchies, lee2018composing, bacon2017option, Florensa2017StochasticLearning}. The feudal approaches learn a master policy that modulates a low-level policy by a control signal~\cite{LatentSpacePolicies, hiro2018arxiv, Vezhnevets2017FeUdalLearning, kulkarni2016hierarchical, hausman2018learning}. Our approach is based on options but in contrast to the cited methods, we pretrain the skills with IL. This allows us to solve complex and sparse reward problems using significantly less interactions with the environment during training.

\noindent {\bf Combining RL and IL.} A number of approaches combining RL and IL
have been introduced recently.
Gao et al.~\cite{gao2018reinforcement} use demonstrations to initialize the RL agent.
\cite{Cheng2018fast, sun2018truncated} use RL to improve expert demonstrations, but do not learn hierarchical policies.
Demonstrations have been also used to define RL objective functions~\cite{Hester2017deepqlearning, Nair2018overcoming}
and rewards~\cite{zhu2018reinforcementAnd}.
Das et al.~\cite{Das2018NeuralMC} combines IL and RL to learn a hierarchical policy. Unlike our method, however, \cite{Das2018NeuralMC} requires full task demonstrations and task-specific reward engineering. Moreover, the addressed navigation problem in~\cite{Das2018NeuralMC} has a much lower time horizon compared to our tasks. \cite{Das2018NeuralMC} also relies on pre-trained CNN representations which limits its application domain.
Le at al.~\cite{le2018hieararchical} train low-level skills with RL, while using demonstrations to switch between skills.
In a reverse manner, we use IL to learn low-level control and then deploy RL to find appropriate sequences of pre-trained skills. The advantage is that our method can learn a variety of complex manipulations without full task demonstrations.
Moreover, \cite{Das2018NeuralMC,le2018hieararchical} learn discrete actions and cannot be directly applied to robotics manipulations that require continous control.

In summary, none of the methods~\cite{Das2018NeuralMC,le2018hieararchical,Cheng2018fast,sun2018truncated}  is directly suitable for learning complex robotic manipulations due to requirements of dense rewards~\cite{Das2018NeuralMC,sun2018truncated} and state inputs~\cite{Cheng2018fast, sun2018truncated}, limitations to short horizons and discrete actions~\cite{Das2018NeuralMC,le2018hieararchical}, the requirement of full task demonstrations~\cite{Das2018NeuralMC,le2018hieararchical,Cheng2018fast, sun2018truncated} and the lack of learning of visual representations~\cite{Das2018NeuralMC,Cheng2018fast, sun2018truncated}.
Moreover, our skills learned from synthetic demonstrated trajectories outperform RL based methods, see Section~\ref{sec:bc_sota}.

\vspace{-.1cm}

\section{Conclusion}
\vspace{-.2cm}
This paper presents a method to learn combinations of primitive skills.
Our method requires no full-task demonstrations nor intermediate rewards and
shows excellent results in simulation and on a real robot. We demonstrate the versatility of our approach in challenging settings with dynamic scene changes.
Future work will include learning multiple tasks with shared skills
and addressing contact-rich tasks.

\section{Appendix}

We present additional qualitative results for the RLBC approach on the real robot with policies that have been trained in simulation, as in the main paper. Section~\ref{sec:qualitative} describes and illustrates examples of UR5-Bowl and UR5-Breakfast policies, also shown in the video. We demonstrate robot behavior while the robot is facing the challenges of previously unseen objects, dynamic changes of object locations and occlusions. We also illustrate feature map activations of the network providing better understanding of learned policies in Section~\ref{sec:activations}.

\subsection{Qualitative results}
\label{sec:qualitative}

\paragraph{UR5-Bowl: Multiple objects.} We experiment with the RLBC policy trained in the UR5-Bowl environment. Once the robot succeeds to place a cube in the bowl, we put another cube on the table and let the policy continue, see Figure~\ref{fig:ur5-bowl-2cubes}. While the UR5-Bowl policy has been trained to handle one cube only, it automatically generalizes to multiple cubes when run in a loop.

\begin{figure}[h!]
   \centering

        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-2cubes_01.jpg}
        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-2cubes_02.jpg}
        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-2cubes_03.jpg}
        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-2cubes_04.jpg}
   \caption{RLBC approach for UR5-Bowl with two cubes.\vspace{-.1cm}}
\label{fig:ur5-bowl-2cubes}
\end{figure}

\paragraph{UR5-Bowl: Previously unseen objects.} We further test the RLBC UR5-Bowl policy in the presence of previously unseen objects. While the policy has been trained to manipulate cubes of different sizes (see Section~5.1), we observe its robustness to other object shapes. As shown in Figure~\ref{fig:ur5-bowl-newobj}, the policy successfully grasps and places into a bowl real objects, such as apples, oranges, lemons, and toys. Notably, in cases of failing to grasps an object, the robot automatically recovers and completes the task. This behavior comes naturally from our RLBC master policy that has learned to adapt the sequence of skills given current observations of the scene.

\begin{figure}[h!]
   \centering

        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-newobj_01.jpg}
        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-newobj_02.jpg}
        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-newobj_05.jpg}
        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-newobj_06.jpg}\smallskip \\
        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-newobj_07.jpg}
        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-newobj_08.jpg}
        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-newobj_03.jpg}
        \includegraphics[trim={50 20 50 40},clip,width=0.22\textwidth]{imgs/ur5-bowl-newobj_04.jpg}
   \caption{RLBC approach for UR5-Bowl with previously unseen objects.\vspace{-.5cm}}
\label{fig:ur5-bowl-newobj}
\end{figure}

\paragraph{UR5-Breakfast: New object instances.}

To enable generalization of learned policies to new object instances, our UR5-Breakfast environment contains cups, bottles and bowls of different shapes from ShapeNet (see Section~5.1). During testing we run the learned RLBC UR5-Breakfast policy on a real robot and experiment with instances of bottles and cups unseen during training. Figure~\ref{fig:ur5-breakfast-newobj} demonstrates successful executions of the RLBC UR5-Breakfast policy in scenes with significant variations in object shapes, for example, using a wine glass instead of a cup.

\begin{figure}[h!]
   \centering

        \includegraphics[trim={40 20 50 30},clip,width=0.22\textwidth]{imgs/ur5-breakfast-newobj_01.jpg}
        \includegraphics[trim={40 20 50 30},clip,width=0.22\textwidth]{imgs/ur5-breakfast-newobj_02.jpg}
        \includegraphics[trim={40 20 50 30},clip,width=0.22\textwidth]{imgs/ur5-breakfast-newobj_03.jpg}
        \includegraphics[trim={40 20 50 30},clip,width=0.22\textwidth]{imgs/ur5-breakfast-newobj_04.jpg}
   \caption{RLBC approach for UR5-Breakfast with previously unseen object instances.\vspace{-.5cm}}
\label{fig:ur5-breakfast-newobj}
\end{figure}

\paragraph{UR5-Breakfast: Dynamic changes of object location.}

Our BC skills make decisions at every time-step and, hence, can instantly adapt to changing conditions of the scene. We verify this by varying object positions during grasping attempts of the RLBC UR5-Breakfast policy. Figure~\ref{fig:ur5-breakfast-reactive} illustrates the reactive behavior of the robot grasping a cup that is being simultaneously moved by the person. The cup is successfully grasped after multiple changes of its position.

\begin{figure}[h!]
   \centering

        \includegraphics[trim={20 0 40 10},clip,width=0.22\textwidth]{imgs/ur5-breakfast-reactive_02.jpg}
        \includegraphics[trim={20 0 40 10},clip,width=0.22\textwidth]{imgs/ur5-breakfast-reactive_03.jpg}
        \includegraphics[trim={20 0 40 10},clip,width=0.22\textwidth]{imgs/ur5-breakfast-reactive_04.jpg}
        \includegraphics[trim={20 0 40 10},clip,width=0.22\textwidth]{imgs/ur5-breakfast-reactive_05.jpg}
   \caption{RLBC approach for UR5-Breakfast with dynamic changes of object locations.\vspace{-.5cm}}
\label{fig:ur5-breakfast-reactive}
\end{figure}

\paragraph{UR5-Breakfast: Occlusion.}

Another example of the instant re-planning by our RLBC policy is demonstrated in Figure~\ref{fig:ur5-breakfast-occlusion}. While the robot approaches an object, we temporary occlude the object and disrupt the executing BC skill. Given the hierarchical nature of our RLBC approach, the RL master is able to recover from this failure by starting another skill that leads to the completion of the task. More precisely, when the bottle gets occluded in the example of Figure~\ref{fig:ur5-breakfast-occlusion}, the robot changes its strategy and decides to grasp and pour from a cup. Once the occlusion is removed, the robot automatically resumes and completes the task by grasping and pouring from the bottle.

\begin{figure}[h!]
   \centering

        \includegraphics[trim={0 0 60 10},clip,width=0.24\textwidth]{imgs/ur5-breakfast-occlusion_01.jpg}
        \includegraphics[trim={0 0 35 10},clip,width=0.24\textwidth]{imgs/ur5-breakfast-occlusion_02.jpg}
        \includegraphics[trim={0 0 48 10},clip,width=0.24\textwidth]{imgs/ur5-breakfast-occlusion_03.jpg}\\
        \includegraphics[trim={0 0 45 10},clip,width=0.24\textwidth]{imgs/ur5-breakfast-occlusion_04.jpg}
        \includegraphics[trim={0 0 50 10},clip,width=0.24\textwidth]{imgs/ur5-breakfast-occlusion_05.jpg}
        \includegraphics[trim={0 0 50 10},clip,width=0.24\textwidth]{imgs/ur5-breakfast-occlusion_06.jpg}
   \caption{RLBC UR5-Breakfast policy executed in a robot scene with a temporary occluded bottle.\vspace{-.5cm}}
\label{fig:ur5-breakfast-occlusion}
\end{figure}

\paragraph{UR5-Breakfast: Failure case for BC-ordered.}

Finally, we demonstrate the advantage of RLBC compared to the "BC-ordered" baseline with a fixed order of skills. For the UR5-Breakfast environment we define the order of skills for the BC-ordered policy as 1.~"go to the cup", 2.~"grasp the object and pour it to the bowl", 3.~"release the object", 4.~"go to the bottle", 5.~"grasp the object and pour it to the bowl", 6.~"release the object". While this policy succeeds in many cases, the near placement of objects presents a source of problems. Figure~\ref{fig:ur5-breakfast-bc-failure} shows an example scene with a cup and a bottle being near to each other while the cup is placed in front of the bottle. As the BC-ordered policy above is pre-programmed to grasp the cup first, the execution of this policy results in a collision between a gripper and a bottle, followed by the failure of the task. Notably, our RLBC policy learns to select the order of objects for grasping such as to avoid failures of the task. Hence, the RLBC automatically learns to avoid collisions without the need of specific intermediate rewards. This advantage of RLBC is demonstrated by quantitative results in Table~3 and by qualitative results in the last part of the video.

\begin{figure}[h!]
   \centering

        \includegraphics[trim={20 0 40 10},clip,width=0.22\textwidth]{imgs/ur5-breakfast-bc-failure_01.jpg}
        \includegraphics[trim={20 0 40 10},clip,width=0.22\textwidth]{imgs/ur5-breakfast-bc-failure_02.jpg}
        \includegraphics[trim={20 0 40 10},clip,width=0.22\textwidth]{imgs/ur5-breakfast-bc-failure_03.jpg}
        \includegraphics[trim={20 0 40 10},clip,width=0.22\textwidth]{imgs/ur5-breakfast-bc-failure_04.jpg}
   \caption{Illustration of a failure case for the BC-ordered approach for UR5-Breakfast with close by objects.\vspace{-.5cm}}
\label{fig:ur5-breakfast-bc-failure}
\end{figure}

\subsection{Feature map activations}
\label{sec:activations}

The RLBC policy uses no explicit representation of scenes, for example in terms of categories and locations of objects. Some interpretation of learned policies, however, can be obtained by examining spatial activations of the neural network at intermediate network layers. Figure~\ref{fig:ur5-breakfast-heatmab} shows silency maps of an RLBC policy highlighting which parts of the image the agents concentrates on. The silency maps are computed as activations of convolutional feature maps obtained from last layers of $CNN(\theta)$ and $CNN(\eta^i), i=1,\ldots,K$  (see Figure~1), averaged over all channels and layers. The resulting heatmaps are shown for different stages of the UR5-Breakfast task. Interestingly, while grasping and moving the bottle, the network generates highest activations around the bottle and the gripper, while ignoring other objects. When releasing the bottle, however, high activations are also observed around the cup. The attention to the cup might be explained by the need of avoiding collisions when placing the bottle on the table. Note, that we provide no intermediate rewards to RL, however, RL learns to avoid collisions since collisions imply failures of the final task and, hence, no final positive reward during training. Once the bottle is placed on the table, activations become low for the bottle, while the heatmap obtains maximum values for the manipulated cup. Observations of such feature maps have been useful in our work to identify certain cases of failures. We believe feature map activations are a useful tool to interpret learned policies. More examples of feature map activations are available in the video.

\begin{figure}[h!]
   \centering
   \begin{tabular}{c}

        {\em grasping bottle} \\
        \includegraphics[trim={0 13 0 0},clip,width=0.19\textwidth]{imgs/ur5-breakfast-heatmap_01.jpg}
        \includegraphics[trim={0 0  0 0},clip,width=0.275\textwidth]{imgs/ur5-breakfast-heatmap_01orig.jpg}\\
        {\em moving bottle}\\
        \includegraphics[trim={0 13 0 0},clip,width=0.19\textwidth]{imgs/ur5-breakfast-heatmap_02.jpg}
        \includegraphics[trim={0 0  0 0},clip,width=0.275\textwidth]{imgs/ur5-breakfast-heatmap_02orig.jpg}\\
        {\em pouring from bottle}\\
        \includegraphics[trim={0 13 0 0},clip,width=0.19\textwidth]{imgs/ur5-breakfast-heatmap_03.jpg}
        \includegraphics[trim={0 0  0 0},clip,width=0.275\textwidth]{imgs/ur5-breakfast-heatmap_03orig.jpg}\\
        {\em releasing bottle}\\
        \includegraphics[trim={0 13 0 0},clip,width=0.19\textwidth]{imgs/ur5-breakfast-heatmap_04.jpg}
        \includegraphics[trim={0 0  0 0},clip,width=0.275\textwidth]{imgs/ur5-breakfast-heatmap_04orig.jpg}\\
        {\em grasping cup}\\
        \includegraphics[trim={0 13 0 0},clip,width=0.19\textwidth]{imgs/ur5-breakfast-heatmap_05.jpg}
        \includegraphics[trim={0 0  0 0},clip,width=0.275\textwidth]{imgs/ur5-breakfast-heatmap_05orig.jpg}\\
        {\em pouring from cup}\\
        \includegraphics[trim={0 13 0 0},clip,width=0.19\textwidth]{imgs/ur5-breakfast-heatmap_06.jpg}
        \includegraphics[trim={0 0  0 0},clip,width=0.275\textwidth]{imgs/ur5-breakfast-heatmap_06orig.jpg}\\
    \end{tabular}
   \caption{Left: Feature map activations of the RLBC  UR5-Breakfast policy are overlayed on the input depth images. Right: corresponding frames taken from a different viewpoint.}
\label{fig:ur5-breakfast-heatmab}
\end{figure}

\section{Introduction}
\vspace{-.1cm}
\label{sec:introduction}

In this work we consider visually guided robotics manipulations and aim to learn robust visuomotor control policies for particular tasks. Autonomous manipulations such as assembling IKEA furniture~\cite{suarez2018can} remain highly challenging given the complexity of real environments as well as partial and uncertain observations provided by the sensors. Successful methods for task and motion planning (TAMP)~\cite{srivastava2014combined,lozano2014constraint,toussaint2015logic} achieve impressive results for complex tasks but often rely on limiting assumptions such as the full state observability and known 3D shape models for manipulated objects. Moreover, TAMP methods usually complete planning before execution and are not robust to dynamic scene changes.

Recent learning methods aim to learn visuomotor control policies directly from image inputs. Imitation learning~(IL) \cite{Pomerleau1989, Ross2014, Pinto2017, Ng2000} is a supervised approach that can be used to learn simple skills from expert demonstrations. One drawback of IL is its difficulty to handle new states that have not been observed during demonstrations. While increasing the number of demonstrations helps to alleviate this issue, an exhaustive sampling of action sequences and scenarios becomes impractical for long and complex tasks.

In contrast, reinforcement learning (RL) requires little supervision and achieves excellent results for some challenging tasks~\cite{Mnih2015Human-levelLearning,Silver2016MasteringSearch}. RL explores previously unseen scenarios and, hence, can generalize beyond expert demonstrations. As full exploration is exponentially hard and becomes impractical for problems with long horizons, RL often relies on careful engineering of rewards designed for specific tasks.

Common tasks such as preparing food or assembling furniture require long sequences of steps composed of many different actions. Such tasks have long horizons and, hence, are difficult to solve by either  RL or IL methods alone. To address this issue, we propose a RL-based method that learns to combine simple imitation-based policies. Our approach  simplifies RL by reducing its exploration to sequences with a limited number of primitive actions, that we call {\em skills}.

Given a set of pre-trained skills such as "grasp a cube" or "pour from a cup", we train RL with sparse binary rewards corresponding to the correct/incorrect execution of the full task.
While hierarchical policies have been proposed in the past~\cite{Das2018NeuralMC,le2018hieararchical}, our approach can learn {\em composite manipulations using no intermediate rewards and no demonstrations of full tasks}. Hence, the proposed method can be directly applied to learn new tasks.
See Figure~\ref{fig:method} for an overview of our approach.

Our skills are low-level visuomotor controllers learned from synthetic demonstrated trajectories with behavioral cloning~(BC)~\cite{Pomerleau1989}.
Examples of skills include {\em go to the bowl}, {\em grasp the object}, {\em pour from the held object}, {\em release the held object}, etc. We automatically generate expert synthetic demonstrations and learn corresponding skills in simulated environments.
We also minimize the number of required demonstrations by choosing appropriate CNN architectures and data augmentation methods.
Our approach is shown to compare favorably to the state of the art~\cite{Pinto2017} on the FetchPickPlace test environment~\cite{Plappert2018}.
Moreover, using recent techniques for domain adaptation~\cite{learningsim2real2019} we demonstrate the successful transfer and high accuracy of our sumulator-trained policies when tested on a real robot

We compare our approach with two classical methods: (a) an open-loop controller estimating object positions and applying a standard motion planner (b) a closed-loop controller adapting the control to re-estimated object positions. We show the robustness of our approach to a variety of perturbations. The perturbations include dynamic change of object positions, new object instances and temporary object occlusions. The versatility of learned policies comes from both the reactivity of the BC learned skills and the ability of the RL master policy to re-plan in case of failure. Our approach allows to compute adaptive control and planning in real-time.

In summary, this work makes the following contributions.
(i)~We propose to learn robust RL policies that combine BC skills to solve composite tasks.
(ii)~We present sample efficient training of BC skills and demonstrate an improvement compared to the state of the art.
(iii)~We demonstrate successful learning of relatively complex manipulation tasks with neither intermediate rewards nor full demonstrations.
(iv)~We successfully transfer and execute policies learned in simulation to real robot setups.
(v)~We show successful task completion in the presence of perturbations.

Our simulation environments together with the code and models used in this work is publicly available at \url{https://www.di.ens.fr/willow/research/rlbc/}.
\input{figures/architectures_timeline.tex}

\vspace{-.1cm}

\section{Approach}
\vspace{-.1cm}
\label{sec:approach}

Our RLBC approach aims to learn multi-step policies by combining reinforcement learning (RL) and pre-trained skills obtained with behavioral cloning (BC). We present BC and RLBC in Sections~\ref{sec:bc_method} and \ref{sec:bcrl_method}. Implementation details are given in Section~\ref{sec:implementation_details}.

\vspace{-.2cm}
\subsection{Skill learning with behavioral cloning}
\vspace{-.1cm}
\label{sec:bc_method}

Our first goal is to learn basic skills that can be composed  into more complex policies.
Given observation-action pairs $\mathcal{D}=\{(o_t, a_t)\}$ along expert trajectories, we follow the behavioral cloning approach~\cite{Pomerleau1989} and learn a function approximating the conditional distribution of the expert policy $\pi_\textrm{E}(a_t|o_t)$ controlling a robot arm.
Our observations $o_t \in \mathcal{O} =\mathbb{R}^{\textrm{H} \times \textrm{W} \times \textrm{M}}$ are sequences of the last $M$ depth frames.
Actions $a_t = (\mathbf{v}_t, \bm{\omega}_t, g_t)$, $a_t \in \mathcal{A}^{\textrm{BC}}$ are defined by the end-effector linear velocity $\mathbf{v}_t \in \mathbb{R}^3$ and angular velocity $\bm{\omega}_t \in \mathbb{R}^3$ as well as the gripper openness state $g_t \in \{0, 1\}$.

We learn the deterministic skill policies $\pi_s: \mathcal{O} \to \mathcal{A}^{\textrm{BC}}$ approximating the expert policy $\pi_\textrm{E}$.
Given observations $o_t$ with corresponding expert (ground truth) actions $a_t=(\mathbf{v}_t, \bm{\omega}_t,g_t)$, we represent $\pi_s$ with a convolutional neural network (CNN) and learn network parameters $(\theta, \eta)$ such that predicted actions $\pi_s(o_t) = (\mathbf{\hat{v}}_t, \bm{\hat{\omega}}_t, \hat{g_t})$ minimize the loss
$L_{\textrm{BC}}(\pi_s(o_t), a_t) = \lambda \norm{[\mathbf{\hat{v}_t}, \bm{\hat{\omega}}_t] - [\mathbf{v}_t, \bm{\omega}_t]}^2_2 + (1-\lambda) \left(g_t \log{\hat{g_t}} + (1 - g_t) \log{(1 - \hat{g_t})}\right)$, where $\lambda \in [0, 1]$ is a scaling factor which we empirically set to 0.9.

Our network architecture is presented in Figure~\ref{fig:method}(right).
When training a skill policy $\pi_s^i$, such as reaching, grasping or pouring, we condition the network on the skill using the recent FiLM architecture~\cite{perez2018film}. Given the one-hot encoding $s^i$ of a skill $i$, we use $s^i$ as input to the FiLM generator which performs affine transformations of the network feature maps. FiLM conditions the network on performing a given skill, which permits learning a shared representation for all skills. Given an observation $o_t$, the network $CNN(\theta|s^i)$ generates a feature map $x^i_t$ conditioned on skill $i$. The spatially-averaged $x_i$ is linearly mapped with $FC(\eta)$ to the action of $\pi_s^i$.

\vspace{-.2cm}
\subsection{RLBC approach}
\vspace{-.1cm}
\label{sec:bcrl_method}
We wish to solve composite manipulations without full expert demonstrations and with a single sparse reward.
For this purpose we rely on a high-level {\em master policy} $\pi_m$ controlling the pre-trained {\em skill policies} $\pi_s$ at a coarse timescale.
To learn $\pi_m$, we follow the standard formulation of reinforcement learning and
maximize the expected return $\mathbb{E}_{\pi} \sum_{k=0}^{\infty} \gamma^k r_{t+k}$
given rewards $r_{t}$.
Our reward function is sparse and returns $1$ upon successful termination of the task and $0$ otherwise. The RL master policy $\pi_m: \mathcal{O} \times \mathcal{A}^{\textrm{RL}} \to [0, 1]$ chooses one of the $K$ skill policies to execute the low-level control, i.e., the action space of $\pi_m$ is discrete: $\mathcal{A}^{\textrm{RL}} = \{1, \ldots, K\}$.
Note, that our sparse reward function makes the learning of deep visual representations challenging. We, therefore, train $\pi_m$ using visual features $x_t^i$ obtained from the BC pre-trained $CNN(\theta|s^i)$.
Given an observation $o_t$, we use the concatenation of skill-conditioned features $\{x^1_t, \ldots, x^K_t\}$ as input for the master $CNN(\mu)$, see Figure~\ref{fig:method}(right).

To solve composite tasks with sparse rewards, we use a coarse timescale for the master policy.
The selected skill policy controls the robot for $n$ consecutive time-steps before the master policy is activated again to choose a new skill.
This allows the master to focus on high-level task planning rather than low-level motion planning achieved by the skills. We expect the master policy to recover from unexpected events, for example, if an object slips out of a gripper, by re-activating an appropriate skill policy.
Our combination of the master and skill policies is illustrated in Figure~\ref{fig:method}(left).

\textit{RLBC algorithm.} The pseudo-code for the proposed approach is shown in Algorithm~\ref{alg:bcrl}. The algorithm can be divided into three main steps.
First, we collect a dataset of expert trajectories $\mathcal{D}_k$ for each skill policy $\pi^k_s$.  For each policy, we use an expert script that has an access to the full state of the environment.
Next, we train a set of skill policies $\{\pi^1_s, \ldots, \pi^K_s\}$. We sample a batch of state-action pairs and update parameters of convolutional layers $\theta$ and the skills linear layer parameters $\eta$.
Finally, we learn the master $\pi_m$ using the pretrained skill policies and the frozen parameters $\theta$. We collect episode rollouts by first choosing a skill policy with the master and then applying the selected skill to the environment for $n$ time-steps.
We update the master policy weights $\mu$ to maximize the expected sum of rewards.

\input{algorithms/bcrl.tex}

\subsection{Approach details}
\label{sec:implementation_details}

\textit{Skill learning with BC.}
We use ResNet-18 for the $CNN(\theta|s^i)$, which we compare to VGG16 and ResNet-101 in Section~\ref{sec:bc_networks}.
We augment input depth frames with random translations, rotations and crops. We also perform viewpoint augmentation and sample the camera positions on a section of a sphere centered on the robot and with a radius of $1.40\,\text{m}$. We uniformly sample the yaw angle in $[-15\degree, 15\degree]$, the pitch angle in $[15\degree, 30\degree]$, and the distance to the robot base in $[1.35, 1.50]$ m. The impact of both augmentations is evaluated in Section~\ref{sec:bc_viewpoints}.
We normalize the ground truth of the expert actions to have zero mean and a unit variance and normalize the depth values of input frames to $[-1, 1]$.
We learn BC skills using Adam~\cite{Diederik2014Adam} with the learning rate $10^{-3}$ and a batch size 64.
We also use Batch Normalization~\cite{Ioffe2015BatchShift}.

\textit{Task learning with RL.}
We learn the master policies with the PPO~\cite{PPO} algorithm using the open-source implementation~\cite{pytorchrl} where we set the entropy coefficient to 0.05, the value loss coefficient to 1, and use 8 episode rollouts for the PPO update.
For the RLBC method, the concatenated skill features $\{x_t^1,\ldots,x_t^K\}$ are processed with the master network $CNN(\mu)$ having 2 convolutional layers with 64 filters of size $3 \times 3$.
During pre-training of skill policies we update the parameters $(\theta, \eta)$. When training the master policy, we only update $\mu$ while keeping ($\theta$, $\eta$) parameters fixed.
We train RLBC using 8 different random seeds in parallel and evaluate the best one.

\textit{Real robot transfer.} To apply our method on the real robot, we use a state-of-the-art technique of learning sim2real transfer based on data augmentation with domain randomization~\cite{learningsim2real2019}. This method uses a proxy task of cube position prediction and a set of basic image transformations to learn a sim2real data augmentation function for depth images. We augment the depth frames from synthetic expert demonstrations with this method and, then, train skill policies. Once the skill policy is trained on these augmented simulation images, it is directly used on the real robot.

\vspace{-.1cm}

\section{Experimental setup}
\vspace{-.1cm}
\label{sec:setup}
This section describes the setup used to evaluate our approach.  First, we present the robot environment and the different tasks in Sections~\ref{sec:setup_robot} and \ref{sec:setup_tasks}. Next, we describe the synthetic dataset generation and skill definition for each task in Sections~\ref{sec:setup_datasets} and \ref{sec:setup_skills}.

\vspace{-.2cm}
\subsection{Robot and agent environment}
\vspace{-.1cm}
\label{sec:setup_robot}
For our experiments we use a 6-DoF UR5 robotic arm with a 3 finger Robotiq gripper, see Figure~\ref{fig:envs}. In simulation, we model the robot with the \texttt{pybullet} physics simulator~\cite{Courmans2016}.
For observation, we record depth images with the Microsoft Kinect 2 placed in front of the arm.
The agent takes as input the three last depth frames $o_t \in \mathbb{R}^{224 \times 224 \times 3}$ and commands the robot with an action $a_t \in \mathbb{R}^7$. The control is performed at 10 Hz frequency.

\input{figures/envs.tex}

\vspace{-.2cm}
\subsection{UR5 tasks}
\vspace{-.1cm}
\label{sec:setup_tasks}

For evaluation, we consider 3 tasks: UR5-Pick, UR5-Bowl and UR5-Breakfast.
The {\bf UR5-Pick} task  picks up a cube of a size between 3.5 cm and 8.0 cm and lifts it up, see Figure~\ref{fig:env_pick}.
In {\bf UR5-Bowl} the robot has to grasp the cube and place it in the bowl, see Figure~\ref{fig:env_bowl}.
The {\bf UR5-Breakfast} task contains a cup, a bottle and a bowl as shown in Figure~\ref{fig:env_breakfast}.
We use distinct ShapeNet~\cite{Chang2015} object instances for the training and test sets (27 bottles, 17 cups, 32 bowls in each set).
The robot needs to pour ingredients from the cup and the bottle in the bowl. In all tasks, the reward is positive if and only if the task goal is reached. The maximum episode lengths are 200, 600, and 2000 time-steps for UR5-Pick, UR5-Bowl, and UR5-Breakfast correspondingly.

\vspace{-.2cm}
\subsection{Synthetic datasets}
\vspace{-.1cm}
\label{sec:setup_datasets}

We use the simulated environments to create a synthetic training and test set. For all our experiments, we collect trajectories with random initial configurations where the objects and the end-effector are allocated within a workspace of $80\times 40\times 20\, \text{cm}^3$. The synthetic demonstrations are collected using an expert script designed for each skill. The script has access to the full state of the system including the states of the robot and the objects. To generate synthetic demonstrations, we program end-effector trajectories and use inverse kinematics (IK) to generate corresponding trajectories in the robot joints space.
Each demonstration consists of multiple pairs of the three last camera observations and the robot control command performed by the expert script.
For UR5-Pick, we collect $1000$ synthetic demonstrated trajectories for training.
For UR5-Bowl and UR5-Breakfast, we collect a training dataset of 250 synthetic demonstrations.
For evaluation of each task, we use 100 different initial configurations in simulation and 20 trials on the real robot.

\vspace{-.2cm}
\subsection{Skill definition}
\vspace{-.1cm}
\label{sec:setup_skills}

UR5-Pick task is defined as a single skill. For UR5-Bowl and UR5-Breakfast, we consider a set of skills defined by expert scripts. For UR5-Bowl, we define four skills: (a)~go to the the cube, (b)~go down and grasp, (c)~go up, and (d)~go to the bowl and open the gripper.
For UR5-Breakfast, we define four skills: (a)~go to the bottle, (b)~go to the cup, (c) grasp an object and pour it to the bowl, and (d)~release the held object.
We emphasize that the expert dataset does not contain full task demonstrations and that all our training is done in simulation.
When training the RL master, we execute selected skills for 60 consecutive time-steps for the UR5-Bowl task and 220 time-steps for the UR5-Breakfast task.

\vspace{-.1cm}
\section{Evaluation of BC skill learning}
\vspace{-.1cm}
\label{sec:bc_evaluation}

This section evaluates the different parameters of the BC skill training for the UR5-Pick task and a comparison with the state of the art. First, we evaluate the impact of the CNN architecture and data augmentation on the skill performance in Sections~\ref{sec:bc_networks} and \ref{sec:bc_viewpoints}. Then, we show that the learned policies transfer to a real-robot in Section~\ref{sec:bc_real}. Finally, we compare the BC skills with the state of the art in Section~\ref{sec:bc_sota}.

\vspace{-.2cm}
\subsection{CNN architecture for BC skill learning}
\vspace{-.1cm}
\label{sec:bc_networks}

\input{tables/bc_archis.tex}
\input{tables/bc_aug.tex}

Given the simulated UR5-Pick task illustrated in Figure~\ref{fig:env_pick}(left), we compare BC skill networks trained with different CNN architectures and varying number of expert demonstrations.
Table~\ref{tab:bc_archis} compares the success rates of policies with VGG and ResNet architectures.
Policies based on the VGG architecture~\cite{Simonyan2014} obtain success rate below $40\%$ with  $100$ training demonstrations and reach $95\%$ with $1000$ demonstrations.
ResNet~\cite{He2016DeepRecognition} based policies have a success rate above $60\%$ when trained on a dataset of $100$ demonstrations and reach $100\%$ with $1000$ demonstrations. Overall ResNet-101 has the best performance closely followed by ResNet-18 and outperforms VGG significantly. To conclude, we find that the network architecture has a fundamental impact on the BC performance.
In the following experiments we use ResNet-18 as it presents a good trade-off between performance and training time.

When examining why VGG-based BC has a lower success rate, we observe that it has higher validation errors compared to ResNet. This indicates that VGG performs worse on the level of individual steps and is hence expected to result in higher compounding errors.

\vspace{-.2cm}
\subsection{Evaluation of data augmentation}
\vspace{-.1cm}
\label{sec:bc_viewpoints}

We evaluate the impact of different types of data augmentations in Table~\ref{tab:bc_aug}.
We compare training without data augmentation with 3 variants: (1) random translations, rotations and crops, as is standard for object detection, (2) record each expert synthetic demonstration from 10 varying viewpoints and (3) the combination of  (1) and (2).

Success rates for UR5-Pick on datasets with 20, 50 and 100 demonstrations are reported in Table~\ref{tab:bc_aug}.
We observe that data augmentation is particularly important when only a few demonstrations are available. For 20 demonstrations, the policy trained with no augmentation performs at 1\% while the policy trained with standard and viewpoint augmentations together performs at 75\%.
The policy trained with a combination of both augmentation types performs the best and achieves 93\% and 100\% success rate for  50 and 100 demonstrations respectively.
In summary, data augmentation allows a significant reduction in the number of expert trajectories required to solve the task.

\vspace{-.2cm}
\subsection{Real robot experiments}
\vspace{-.1cm}
\label{sec:bc_real}

We evaluate our method on the real-world UR5-Pick illustrated in Figure~\ref{fig:env_pick}(right).
We collect demonstrated trajectories in simulation and train the BC skills network applying standard, viewpoint and sim2real augmentations.
We show that our approach transfers well to the real robot using no real images. The learned policy manages to pick up cubes of 3 different sizes correctly in 20 out of 20 trials.

\vspace{-.2cm}
\subsection{Comparison with state-of-the-art methods}
\vspace{-.1cm}
\label{sec:bc_sota}

\input{figures/mujoco1.tex}
One of the few test-beds for robotic manipulation is FetchPickPlace  from OpenAI Gym~\cite{Plappert2018} implemented in \texttt{mujoco}~\cite{Todorov2012MuJoCo:Control},
see Figure~\ref{fig:fetchpickplace}. The goal for the agent is to pick up a cube and to move it to the red target (see Figure~\ref{fig:fetchpickplace}). The agent observes the three last RGB-D images from a camera placed in front of the robot $o_t \in \mathbb{R}^{100 \times 100 \times 4 \times 3}$. The positions of the cube and the target are set at random for each trial. The reward of the task is a single sparse reward of success. The maximum length of the task
is 50 time-steps.

For a fair comparison with~\cite{Pinto2017}, we do not use any data augmentation. We report the success rate of ResNet-18 policy in Figure~\ref{fig:bc_sota_plot}. We follow~\cite{Pinto2017} and plot the success rate of both RL and IL methods with respect to the number of episodes used (either trial episodes or
demonstrations). Our approach outperforms the policies trained with an imitation learning method DAgger~\cite{Ross2014} in terms of performance and RL methods such as HER~\cite{Andrychowicz2017HindsightER} and DDPG~\cite{Lillicrap2016ContinuousLearning} in terms of data-efficiency.
According to~\cite{Pinto2017}, DAgger does not reach 100\% even after $8*10^4$ demonstrations despite the fact that it requires an expert during training. HER reaches the success rate of 100\% but requires about $4* 10^4$ trial episodes. Our approach achieves the 96\% success rate using $10^4$ demonstrations.

Our policies differ from~\cite{Pinto2017} mainly in the CNN architecture. Pinto et al.~\cite{Pinto2017} use a simple CNN with 4 convolutional layers while we use ResNet-18. Results of this section confirm the large impact of the CNN architecture on the performance of visual BC policies, as was already observed in Table~\ref{tab:bc_archis}.

\vspace{-.1cm}

\section{Evaluation of RLBC}
\vspace{-.1cm}
\label{sec:bcrl_evaluation}

This section evaluates the proposed RLBC approach and compares it to baselines introduced in Section~\ref{sec:bcrl_baselines}.
First, we evaluate our method on UR5-Bowl in Section~\ref{sec:setup_tasks}.
We then test the robustness of our approach to various perturbations  such as dynamic changes of object positions, dynamic occlusions, unseen object instances and the increased probability of collisions due to small distances between objects. We show that RLBC outperforms the baselines on those scenarios both in simulation and on a real robot.
Note, that our real robot experiments are performed with skills and master
policies that have been trained exclusively in simulation using sim2real
augmentation~\cite{learningsim2real2019}. We use the same policies for all perturbation scenarios. Qualitative results of our method are available in the {\bfseries supplementary video}.

\vspace{-.2cm}
\subsection{Baseline methods}
\vspace{-.1cm}
\label{sec:bcrl_baselines}

We compare RLBC with 3 baselines: (a) a fixed sequence of BC skills following the manually pre-defined correct order (BC-ordered); (b) an open-loop controller estimating positions of objects and executing an expert script (Detect \& Plan); (c) a closed-loop controller performing the same estimation-based control and replanning in case if object positions change substantially (Detect \& Replan).
We use the same set of skills for RLBC and BC-ordered. We train the position estimation network using a dataset of 20.000 synthetic depth images with randomized object positions. All networks use ResNet-18 architecture and are trained with the standard, viewpoint and sim2real augmentations described in Section~\ref{sec:bc_viewpoints}.

\vspace{-.2cm}
\subsection{Results on UR5-Bowl with no perturbations}
\vspace{-.1cm}
\label{sec:bcrl_results_bowl}

\input{tables/bcrl_bowl.tex}

\input{figures/hrlbc_bowl_ablation_sim.tex}

We first evaluate RLBC and the three baselines on the UR5-Bowl task (see Figure~\ref{fig:env_bowl}).
When tested in simulation, all the baselines and RLBC manage to perfectly solve the task.
On the real-world UR5-Bowl task, BC-ordered and Detect \& Plan baselines sometimes fail to grasp the object which leads to task failures (see Table~\ref{tab:hrlbc_bowl_ablation_real}, first row). On the contrary, RLBC solves the task in all 20 episodes given its ability to re-plan the task in the cases of failed skills.

We have also attempted to solve the simulated UR5-Bowl task without skills by learning an RL policy performing low-level control. We have used ImageNet pre-trained ResNet-18 to generate visual features. The features were then used to train low-level RL control policy with PPO. Whereas such a low-level RL policy did not solve the task a single time after $10^4$ episodes, RLBC reaches 100\% after 400 episodes.

\vspace{-.2cm}
\subsection{Robustness to perturbations}
\vspace{-.1cm}

\noindent {\it Robustness to dynamic changes in object position.}
We evaluate RLBC against the baselines in the UR5-Bowl scenario where the cube is moved several times during the episode. We plot success rates evaluated in simulation with respect to the number of position changes in Figure~\ref{fig:bcrl_bowl_moving}.
We observe the stability of RLBC and the fast degradation of all baselines.
As both RLBC and BC-ordered use the same set of skills, the stability of RLBC comes from the learned skill combination.
The "Moving objects" row in Table~\ref{tab:hrlbc_bowl_ablation_real} reports results for 3 moves of the cube evaluated on the real robot. Similar to simulated results, we observe excellent results of RLBC and the degraded performance for all the baselines.

\noindent{\it Robustness to occlusions.}
We evaluate the success of UR5-Bowl task under occlusions. Each occlusion lasts 3 seconds and covers a large random part of the workspace by a cardboard. Figure~\ref{fig:bcrl_bowl_occlusion} shows success rates with respect to the number of occlusions in the simulated UR5-Bowl environment. Similar to perturbation results in Figure~\ref{fig:bcrl_bowl_moving}, RLBC demonstrates high robustness to occlusions while the performance of other methods quickly degrades.
The "Occlusions" row in Table~\ref{tab:hrlbc_bowl_ablation_real} reports results for a single occlusion performed during the real-robot evaluation.
Baseline methods are strongly influenced by occlusions except Detect \& Plan which performs well unless occlusion happens during the first frames.
Our HRBC policy performs best compared to other methods.

\noindent {\it Robustness to new object instances.}
We evaluate the robustness of methods to the substitution of a cube by other objects not seen during the training of UR5-Bowl task.
Figure~\ref{fig:bcrl_bowl_unseen} shows the success rate of RLBC and other methods with respect to the number of new objects in simulation. The novel objects are ordered by their dissimilarity with the cube.
The difficulty of grasping unseen objects degrades the performance of grasping skills. In contrast to other methods RLBC is able to automatically recover from errors by making several grasping attempts.
Table~\ref{tab:hrlbc_bowl_ablation_real} reports corresponding results on a real robot where the cube has been replaced by 10 unseen objects. Similar to other perturbations we observe superior performance of RLBC.

\noindent{\it Impact of the distance between objects.}
We vary the distance between a bottle and a cup in the UR5-Breakfast task. The
smaller distance between objects A and B implies higher probability of collision
between a robot and A when grasping B behind A. The choice of the grasping order
becomes important in such situations. While our method is able to learn the
appropriate grasping order to maximize the chance of completing the task, the BC-ordered and other baselines use pre-defined order.
Figure~\ref{fig:bcrl_breakfast_distance} demonstrates the performance of RLBC and BC-ordered for different object distances in the simulated UR5-Breakfast task. As expected, RLBC learns the correct grasping order and avoids most of collisions. The performance of BC-ordered strongly degrades with the decreasing distance.
In the real-world evaluation, both RLBC and ordered skills succeed in 16 out of 20 episodes when the distance between objects is larger than 10 cm. However, the performance of BC-ordered drops to 8/20 when the cup and the bottle are at $4$cm from each other. In contrast, RLBC chooses the appropriate object to avoid collisions and succeeds in 16 out of 20 trials.

\vspace{-.1cm}