AI Virtual Mouse
π Overview
Control your computer with a wave of your hand: the index fingertip moves the cursor, bringing the index and middle fingers together clicks, and other finger gestures handle right-click and scrolling.
It combines hand tracking (MediaPipe's 21 points) with desktop automation (pyautogui) - the classic demonstration of computer vision driving desktop automation.
π§° What you need
- Computer + camera
- Python 3.7+ environment
- pip install: opencv-python, mediapipe, cvzone, pyautogui
- Optional: keep a real mouse handy so you don't lose control while debugging
π§ Step by step
Install the dependencies
pyautogui handles moving the mouse and clicking
Some systems need accessibility permission before a program may control the mouse
Detect the hand
Reuse HandTrackingModule to find the hand landmarks
Get the coordinates of the index and middle fingertips
Map the coordinates
Map camera frame coordinates onto your screen resolution
Use a proportional linear mapping and smooth it to cut down jitter
Move the cursor
The index fingertip position drives the mouse
Use pyautogui.moveTo and mind the speed and smoothing factor
Click gesture
Treat it as a click when the index and middle fingers are closer than a threshold
Simulate the click with pyautogui.click and add a debounce delay to avoid accidental clicks
Extend the controls
Add more gestures: right-click, double-click, drag, and scroll
Wrap the gesture-to-action mapping in a table so it is easy to customise
π‘ Tips
- Smoothing the cursor over several frames of coordinates massively reduces hand shake
- Filter the click decision over 2-3 frames so a fast sweep does not trigger it
- When the screen aspect ratio differs from the camera's, scale proportionally before mapping