Home

By Miranda Drummond, Maud Helen Hovland and Samuel Ostweg

Abstract

The Computer Vision Map Navigator enables real-time map navigation using simple hand gestures. It is designed to work seamlessly with just a single RGB camera, making it accessible and easy to use. The system supports both a MediaPipe model and a custom Keras model for gesture recognition, offering robustness and flexibility in different scenarios.

Contents

• Project Requirements
• Methodology
• System Design
• Implementation
• Testing and Evaluation
• Results
• Conclusion
• Contributions
• Appendices

Project Requirements

• Control Mechanism: Insights into the single-hand control for pan and zoom and real-time interaction capabilities.
• Evaluation Criteria: Criteria including use of a single RGB camera, application of open source tools, ease of use, and performance metrics.

Methodology

• Utilized Python as the primary development language for its extensive support of machine learning and computer vision libraries.
• Chose an RGB camera for input due to its widespread availability in consumer devices, enabling broader accessibility.
• Incorporated open-source tools and libraries including MediaPipe for real-time hand tracking and gesture recognition, and Keras for building a custom LSTM model tailored to gesture recognition tasks.
• Emphasized design considerations for ease of use, aiming for a system that requires no prior practice or learning for the user.
• The development process involved setting up a development environment, implementing gesture recognition algorithms using both MediaPipe and Keras models, and integrating these with map navigation functionalities using pyautogui for controlling Google Maps.
• Addressed challenges like varying lighting conditions, different hand sizes, and the seamless integration of gesture recognition with map navigation.

System Design

• The system architecture includes a gesture recognition system built on two models: MediaPipe for general hand tracking and a custom Keras LSTM model for advanced gesture recognition.
• Map navigation integration was achieved by translating recognized gestures into corresponding navigation commands (pan, zoom, click) using pyautogui.
• Designed a user interface that visually represents recognized gestures and the corresponding navigation action on the map, offering real-time feedback to the user.

Implementation

• The coding process involved Python scripts (app.py for MediaPipe model and keras_app.py for Keras model) with detailed documentation on setup and execution.
• Key code segments include the implementation of hand gesture recognition algorithms, integration with pyautogui for map control, and methods for processing and classifying hand landmarks.
• Demonstrates how the system meets project requirements through the successful interpretation of hand gestures for map navigation, supported by both models.

Testing and Evaluation

• Adopted a mixed-methods testing methodology encompassing unit tests for individual components and user testing to assess overall system usability and performance.
• User testing scenarios were crafted to cover a broad range of interactions, from basic map navigation to more complex gestures for zooming and panning.
• The system was evaluated by having various people test it, showing high usability and performance, with feedback leading to iterative improvements.

Results

• The system demonstrated effective map navigation control through hand gestures, with high accuracy in gesture recognition and responsiveness in real-time interaction.
• Achievements include a user-friendly interface requiring no prior learning, and flexibility in using either the MediaPipe or Keras model based on user preference or specific requirements.
• Identified limitations include dependency on lighting conditions and camera quality, with future work suggested in enhancing gesture recognition accuracy under varied conditions.

Conclusion

• The project successfully developed a Computer Vision Map Navigator that allows for intuitive, real-time map navigation using hand gestures, meeting all stated objectives.
• Reflects on the learning experience, highlighting the importance of integrating open-source tools and custom model development for solving practical computer vision tasks.
• Discusses the potential impact of such technologies in making digital navigation more accessible and interactive.

Contributions

• Miranda: End to end implementation of Keras Model, including model adaptation and syncing controls to pyautogui and then to a google maps link. Writing of Keras model documentation and wiki.
• Samuel: Development of first draft of basic hand navigation (Index finger left = left scroll, Index finger right = right scroll, Index finger up = upwards scroll) using google landmark & pyautogui.

Appendices

Link to the original model which was modified for the keras model implementation.