Such annotation is difficult to obtain at scale.
recent work has focused on training such models with less supervision: one line of work assumes that only the order of actions happening in the video is given and use this weak supervision to perform action segmentation (Bojanowski et al., 2014;Huang et al., 2016;Ding and Xu, 2018;Chang et al., 2019).