How to control the mouse and keyboard with python

This is an article that is part of a serie that I initiated regarding my bot. You can find more about it here

The first step to automate anything is, of course to be able to replace the basic Human to machine inputs.

For 99% of applications it is simply the mouse and keyboard. So what do we need ?

• Being able to input anything with the keyboard :

  • Keys

  • Shortcuts (combinaison of keys)

• Being able to input anything with the mouse :

  • Click right, middle, left

  • Move anywhere

And all of that working in a cross platform (by cross platform I mean windows/linux, I don't really care about macOs users since I don't own one and linux-compatible scripts tends to work on macos as well)

Now after some research I found the library that I was looking for : pyautogui

Pyautogui

Pyautogui is a library that allows you to control the mouse and keyboard to do various things.

On windows the installation is a simple as

pip install pyautogui

(if you have pip in your PATH of course), if not you need to find "pip.exe" in the folder where you installed python.

On linux you need to install a few tools before being able to install it :

pip3 install python3-xlib

sudo apt-get install scrot

sudo apt-get install python3-tk

sudo apt-get install python3-dev

pip3 install pyautogui

They have a GREAT documentation that you can find at https://pyautogui.readthedocs.io

If you want to start experimenting with it fast, then you should look at the cheat sheet

Keyboard

I was looking for a way to control the keyboard :

To type one (or several) keys I used

pyautogui.press(keys)

To do some more specifics actions that needed some timing I used

pyautogui.keyDown(key_name)

pyautogui.keyUp(key_name)

In order to do things like a control + click then I do keyDown + click + keyUp

Mouse

Controlling the mouse is as easy :

pyautogui.moveTo(x,y,duration=num_seconds)

basically "move to position x y in z seconds"

Note that in an usual screen like 1080p we move from one end to the other in less than a second.
I usually put 0.2 to 0.4 seconds.

To click I usually do a "move" and then a "click" because I ran into some problems where my game would not like if things go to fast, so I do that this way so I can put a little sleep inbetween. But it's totally fine to directly call the click function with all the args

pyautogui.click(button=button)

or

pyautogui.click(x=moveToX, y=moveToY, clicks=num_of_clicks, interval=secs_between_clicks, button='left')

and that's about it, the libary allows you to do a lot of things while giving you a lot of control. It's definitely my goto library for any automation in python now.

Thanks for reading and stay tuned for my next subject : Image recognition with python.