4 months ago

How to Take Control of the Mouse & Keyboard in Python « Null Byte :: WonderHowTo

While hackers adore the command line, working with the GUI (graphical user interface) of an operating system has some advantages. that will is actually especially true when the idea comes to post-exploitation. When we can control the mouse as well as keyboard of a remote computer, we can truly say we have complete control over the device. Today, I’m going to introduce to you the PyAutoGUI module, which does just that will, as well as just how much fun you can have with the idea!

Endless Possibilities

Being able to control the mouse as well as keyboard with the lightning-quick speed of a computer program lends itself to several applications: Designing AI video game players, quickly maneuvering through a file system, running commands via a command prompt … the list goes on.

As you think of more complicated applications, more complicated solutions will be needed. We’re going to keep the idea simple today by showing you how to create a simple program to automatically tweet via a desktop computer.

Step 1: Installing PyAutoGUI

The process for installing PyAutoGUI is actually pretty simple. The first step depends on your OS. If you’re running macOS, first type the following into a terminal window to install dependencies.

sudo pip3 install pyobj-framework-Quartz
sudo pip3 install pyobjc-core
sudo pip3 install pyobjc

If you’re on Linux, type the following into a terminal window.

sudo pip install python3-xlib

Following that will, type the text below to install the rest of the dependiecies.

sudo apt-get install scrot
sudo apt-get install python3-tk
sudo apt-get install python3-dev

If you’re using Windows, I’ve got great news: You don’t have any dependencies to install! For all three operating systems, finish by typing the following (with Windows users replacing type pip3 with pip).

pip3 install pyautogui

Alrighty, right now that will we’ve got what we need, the idea’s time to dive in as well as code our Twitter hijacker!

Step 2: Understanding Pyautogui & Its Functions

Before we can use PyAutoGUI, we first have to understand the different functions we’ll be using. Let’s start with that will one:


As you might’ve guessed, the function moveTo moves the mouse to a certain part of the screen. The key to understanding that will function, though, is actually knowing how the coordinate graph is actually laid out. The following wonderful ASCII art graph can be found inside documentation:

Image via PyAutoGUI

So, as we can see, the upper-left corner has the coordinates (0,0). Moving downwards increases the y coordinate, as well as moving to the right increases the x coordinate. If I wanted to move the mouse to the bottom left corner of the screen (which we will soon), I could type:


Moving the mouse is actually pretty simple when you know the dimensions of the screen, although what about when you don’t? We can’t move the mouse to (0,1079) on a screen that will’s only 1280 by 720 pixels. that will is actually where knowing the size of the screen allows us to create a more flexible program. We can get the size of the screen by using the pyautogui.size function.

x, y = pyautogui.size()

By calling the function in that will way, we’ll assign the width of the screen to x as well as the height of the screen to y. These variables will give us anchors to work with when the idea comes to finding where an item is actually on screens of varying size.

x, y = pyautogui.position()

Similar to the size function we just discussed, position() will return an x as well as y value. Instead of returning the maximum height as well as width, however, the idea will return the current location of the mouse. that will will come in handy when the idea comes to figuring out where on the screen we want to click.

right now that will we can move the mouse with confidence, we need to be able to click the idea. that will is actually a definitely simple process, as you can see below.


as well as that will’s the idea! right now we could use pyautogui.click(button=’right’) to click the right mouse button instead of the left, or we could use pyautogui.click(200,200) to click at a particular location. For our example, we’ll use moveTo as well as then click to better illustrate the process by which we create our solution.

Don’t Miss: How to Train Your Python, Part 10 — generating Our Own Functions

Finally, we need to know how to utilize the keyboard. PyAutoGUI Again offers a simple solution for us here by typing the following.

pyautogui.typewrite(‘The text you want to type’)

as well as just like that will, we can type text!

although what about when we want to utilize the enter key? We’re going to need the idea in order to browse to Twitter. that will is actually still done with the typewrite function, although the argument we pass will look a little different, as seen below.


Instead of passing a string, we pass a list containing ‘enter’. We can pass a list containing several key names, as well as typewrite will type all of them in order. For instance, we can type the following example.


that will will output “XYab” because the idea types ab, then moves the cursor left two spaces, as well as then types XY.

While we won’t need the idea just for that will project, the idea is actually also worth noting that will keyDown(keyname) as well as keyUp(keyname) can be used for sustained key presses. You should know, these don’t require you to put button names into a list. that will is actually useful when the idea comes to creating programs that will play video games.

Step 3: Writing TweetJack.Py

right now that will we know what we’re working with, let’s start writing our program. Looking at the problem programmatically, we have two tasks we want to complete.

First, we want to open a web browser as well as browse to twitter.com.
Second, once we’re on Twitter, we want to tweet something. that will is actually assuming the user is actually already logged in.

To do that will, let’s first write a function that will will allow us to browse to a specified website. You can see the function below.

Lines 1 as well as 2 import PyAutoGUI as well as the sleep function. As you’ll soon find out, we’re going to need to use the sleep function to make sure that will our previous commands executed successfully before we move on to the next one.

On line 3, we define a function called browse as well as pass the idea a parameter website. On lines 4 as well as 5, we tell the browse function to use two variables declared outside the scope of the function: x as well as y. These are the width as well as height of the screen. We’ll actually assign these variables later.

Don’t Miss: How to Secretly Hack Into, Switch On & Watch Anyone’s Webcam Remotely

Once we have everything we need, we can go about opening up the web browser. When we do that will, we’re going to make a few of assumptions. First, we’re assuming that will Google Chrome is actually installed on the target machine. that will can be easily swapped for another browser, although for right now, we’ll stick to Chrome. Second, we’re assuming that will the target is actually using Windows 10. The script should work for Windows 7 as well, although definitely won’t for Windows 8.1, Linux, or macOS (previously Mac OS X).

Once you see the process we go through, however, you’ll be able to modify the program accordingly. Finally, we’re going to assume that will the user is actually already logged into Twitter. that will is actually a fairly reasonable assumption, as most people who use social media on their computers stay logged in.

Moving on, we see that will on line 7 we move the mouse to the location (0,y-1). Why could we do that will? Well, in order to open Google Chrome we’re going to rely on the Windows search bar. To open the search bar, we’re going to open up the start menu inside very bottom left of the screen. that will means that will the actual coordinate of the location we want the mouse to be will vary, depending on the screen size.

Next, we’re going to wait a second to make sure everything has loaded, as well as just start typing. Once we finish typing Google Chrome, the Windows search program will have found what we need. via here we can just hit the enter key, as well as Google Chrome should open up.

Don’t Miss: Turn Any Phone into a Hacking Super Weapon with the Sonic

As we continue coding together, I could encourage you to do some problem-solving on your own. Given the tools we discussed earlier, how could you go about finishing the process of browsing to a website? Keep in mind working with varying screen sizes.

Moving on, let’s look at the rest of the function below.

On line 15, we sleep for 5 seconds to make sure that will Chrome is actually open before we start clicking all over the place. Depending on the speed of the system, that will interval may need to be increased.

Figuring Out Where to Click

via there, we can see on line 16 that will we move the mouse to (297,63). Let’s look at how we arrived at that will location.

First, you’re going to want to fire up a Python interpreter in a command prompt. To do that will, open up a terminal as well as type python. Next, type import pyautogui. With that will terminal, we can move the mouse around on the screen, then type x,y = pyautogui.position() to store the coordinates of the mouse.

Using Python shell via our command-line tool, we can grab the location of the mouse on the fly.

By using the position() function, we can find a location which will hover the mouse over the browser search bar. Once we’re there, we click on the search bar as well as type inside website parameter that will was passed to the function. After the website name has been typed into the search bar, we hit enter to browse to that will website.

As a quick side note, while I’ve arranged the following lines as you see below, I could use another way of doing the same thing.


The following could do the same.


Step 4: Sending Our Tweets

Before we jump into writing that will next function, let’s take a look at the design of the Twitter website:

A shameless Twitter plug.

We see there’s a big blue “Tweet” button inside upper right-hand corner that will will do what we want. We also see that will a user can directly tweet via the text input box inside center of the screen. Either one could work for our purposes. In that will example, we’re going to use the “Tweet” button.

Clicking on that will, we see the following:

We notice that will the text input box is actually active as soon as the button is actually clicked, which means we won’t have to try to click on the idea before typing. that will is actually a nice break for us.

So, once we browse to the website, our list of steps to send a tweet looks like that will:

  1. Move the mouse to the “Tweet” button.
  2. Click the button.
  3. Type inside tweet.
  4. Move the mouse to the tweet box.
  5. Click the tweet box.

Simple enough. Let’s walk through the code just for that will function.

Step 5: Coding Our Function

When we’re tweeting, the first thing we’re going to want to do is actually actually browse to Twitter. We do that will in line 22 when we call the browse function we just wrote as well as pass www.twitter.com as an argument. Once that will function executes, we need to make sure that will the function uses the global variables x as well as y, which we will assign later.

As the comment on line 25 explains, we sleep for 5 seconds on line 26 in order to make sure that will Twitter has sufficiently loaded before we try to tweet. Then, we move the cursor to the location (x-271,105).

By getting the position() of the mouse when the idea is actually over the tweet button, we see that will the button is actually roughly 270 pixels via the right edge of the screen.

the idea’s a better idea to use that will value instead of hard-coding the coordinates in because the idea allows our program to work for varying screen sizes. We didn’t need to worry about that will when we were clicking the URL bar in Google Chrome because the bar is actually within the first 300 pixels of width as well as height. Unless laptops have 200 x 200 screens right now, the idea’s safe to say that will we could hard code that will value.

Don’t Miss: How to Use Netcat, the Swiss Army Knife of Hacking Tools

The same is actually not true for the tweet button. On a 1920 x 1080 screen, the button resides at (1650,105). If the victim carries a screen that will is actually 1280 x 720 in size, that will hard-coded value wouldn’t work.

Once we have clicked the “Tweet” button, we’ll type in whatever was passed for the content parameter. After that will, we’ll move the mouse to the brand new “Tweet” button as well as send our tweet!

right now we have to finish things up. We’ll get our tweet via the command line as well as assign the idea to the variable theTweet. Then, we’re finally going to assign x as well as y with the output via the pyautogui.size function. Once that will is actually done, we can call the tweet function as well as enjoy the fruits of our labor.

Have Fun with PyAutoGUI

I trust you found that will introduction to PyAutoGUI helpful. I definitely love that will module because of how much you can do with the idea. There are so many different applications that will go beyond security, as well as there’s so much fun to be had for anyone interested in learning about the idea.

Thank you so much for reading! Comment below with any questions, or contact me via Twitter @xAllegiance.

Cover photo via Photo-Mix/Pixabay
Screenshots by Allegiance/Null Byte

Leave a Comment

Your email address will not be published. Required fields are marked *

twelve − seven =