Temporally locating and classifying action segments in long untrimmed videos is of

particular interest to many applications like surveillance and robotics. While

traditional approaches follow a two-step pipeline, by generating frame-wise
probabilities and then feeding them to high-level temporal models, recent
approaches use temporal convolutions to directly classify the video frames. In this
paper, we introduce a multi-stage architecture for the temporal action segmentation
task. Each stage features a set of dilated temporal convolutions to generate an
initial prediction that is refined by the next one. This architecture is trained
using a combination of a classification loss and a proposed smoothing loss that
penalizes over-segmentation errors. Extensive evaluation shows the effectiveness of
the proposed model in capturing long-range dependencies and recognizing action
segments. Our model achieves state-of-the-art results on three challenging
datasets: 50Salads, Georgia Tech Egocentric Activities (GTEA), and the Breakfast
dataset.