Object Following

This project was for a Neuromechanics class to recreate how humans look at objects in their environment. The hope is to use this project to eventually use visual feedback to manipulate large objects with a humanoid soft robot (see these presentation slides for a more detailed look at the background). 

The two main ways that humans use visual feedback for mainpulation are:

• Creating predictive models for grip and load forces (ie recognizing a heavy weight vs a pencil allows us to determine the expected load forces)

• Gaze continually moves to keep the object or area of interest in the center of the field of view

To perform object detection, I used Yolov8. I collected data using a ZED2 and my phone camera for a cardboard box, rolled up foam, stool, and a trash can. It worked fairly well, even in harsh sunlight. The most difficult object to detect was the stool, which makes sense because it depends on the background the most.

Using the bounding boxes that are output from the YOLO model, I can use the center as the control signal. There is a proportional controller for the pitch and yaw angles that tries to keep the center of the bounding box in the center of the image.