Egomap: Hierarchical First-person Semantic Mapping

We consider unsupervised learning of semantic, user-specific maps from first-person video. The task we address can be thought of as a semantic, non-geometric form of simultaneous localisation and mapping, differing in significant ways from formulations typical in robotics. Locations, termed stations, typically correspond to rooms or areas in which a user spends time, places to which they might refer in spoken conversation. Our maps are modeled as a hierarchy of probabilistic station graphs and view graphs. View graphs capture an aspect of user behaviour within stations. Visits are temporally segmented based on qualitative visual motion and used to update the map, either by updating an existing map station or adding a new map station. We contribute a labelled dataset suitable for evaluation of this novel SLAM task. Experiments compare mapping performance with and without the use of view graphs and demonstrate better online mapping than when using offline clustering.