GENERATING ROUTE DESCRIPTIONS Automatic generation of route descriptions with visual elements from graph and salient landmarks

Abstract

Can machines find the shortest route and guide us using intuitive, human-like instructions, such as ”move towards the big black piano” or ”head past the green armchair”? This thesis investigates the potential of machines to generate navigation instructions that combine the efficiency of graph-based systems with the clarity provided by salient landmarks. Our research focuses on creating a system that determines the shortest path between two points and enriches navigation with human-like, landmark-based descriptions. Integrating allocentric 1 and egocentric 2 perspectives, we aim to improve the quality and naturalness of the generated instructions. To extract salient landmarks, we utilized data from Bérénice Le Glouanec’s project(Le Glouanec (2024)3), which employed object recognition techniques, including a Faster R-CNN model. This approach allowed us to identify significant visual attributes such as spatial location, shape, and color of landmarks within the environment. We integrated these visually salient elements into our system to improve the clarity and relatability of the generated navigation instructions. The system was evaluated through measures of similarity between machine-generated and human-written instructions, yielding a mean cosine similarity score of 54 % and a Jaccard similarity score of 13 %. These results indicate a reasonable resemblance to human-generated navigation instructions, demonstrating the system’s potential. Future work will focus on expanding the dataset to include more diverse environments, such as outdoor spaces, and exploring customizable multimodal systems to enhance user experience and accessibility.