2 months ago

How to Take Control of the Mouse along with Keyboard in Python « Null Byte :: WonderHowTo

While hackers adore the command line, working with the GUI (graphical user interface) of an operating system has some advantages. This kind of is usually especially true when the item comes to post-exploitation. When we can control the mouse along with keyboard of a remote computer, we can truly say we have complete control over the device. Today I’m going to introduce to you the PyAutoGUI module, which does just of which, along with just how much fun you can have with the item!

Endless Possibilities

Being able to control the mouse along with keyboard with the lightning-quick speed of a computer program lends itself to several applications: Designing AI video game players, quickly maneuvering through a file system, running commands coming from a command prompt, the list goes on.

As you think of more complicated applications, more complicated solutions will be needed. We’re going to keep the item simple today by showing you how to create a simple program to automatically tweet coming from a desktop computer.

Step 1: Installing PyAutoGUI

The process for installing PyAutoGUI is usually pretty simple. The first step depends on your OS. If you’re running macOS, first type the following into a terminal window to install dependencies.

sudo pip3 install pyobj-framework-Quartz
sudo pip3 install pyobjc-core
sudo pip3 install pyobjc

If you’re on Linux type the following into a terminal window.

sudo pip install python3-xlib

Following of which, type the text below to install the rest of the dependiecies.

sudo apt-get install scrot
sudo apt-get install python3-tk
sudo apt-get install python3-dev

If you’re using Windows, I’ve got great news: You don’t have any dependencies to install! For all three operating systems, finish by typing the following (with Windows users replacing type pip3 with pip).

pip3 install pyautogui

Alrighty, currently of which we’ve got what we need, the item’s time to dive in along with code our twitter hijacker!

Step 2: Understanding Pyautogui along with the item’s Functions

Before we can use PyAutoGUI, we first have to understand the different functions we’ll be using. Let’s start with This kind of one.


As you might’ve guessed, the function moveTo moves the mouse to a certain part of the screen. The key to understanding This kind of function though is usually knowing how the coordinate graph is usually laid out. The following wonderful ascii art graph can be found inside documentation:

So as we can see, the upper-left corner has the coordinates (0,0). Moving downwards increases the y-coordinate, along with moving to the right increases the x-coordinate. So if I wanted to move the mouse to the bottom left corner of the screen (which we will soon), I might type:


Moving the mouse is usually pretty simple when you know the dimensions of the screen, yet what about when you don’t? We can’t move the mouse to (0,1079) on a screen of which’s only 1280 by 720 pixels. This kind of is usually where knowing the size of the screen allows us to create a more flexible program. We can get the size of the screen by using the pyautogui.size function.

x, y = pyautogui.size()

By calling the function in This kind of way, we’ll assign the width of the screen to x along with the height of the screen to y. These variables will give us anchors to work with when the item comes to finding where an item is usually on screens of varying size.

x, y = pyautogui.position()

Similar to the size function we just discussed, position() will return an x along with y value. Instead of returning the maximum height along with width, however, the item will return the current location of the mouse. This kind of will come in handy when the item comes to figuring out where on the screen we want to click.

currently of which we can move the mouse with confidence, we need to be able to click the item. This kind of is usually a definitely simple process, as you can see below.


along with of which’s the item! currently we could use pyautogui.click(button=’right’) to click the right mouse button instead of the left, or we could use pyautogui.click(200,200) to click at a particular location. For our example, we’ll use moveTo along with then click to better illustrate the process by which we create our solution.

Check Out: How To Train Your Python: Part 10, generating Our Own Functions

Finally, we need to know how to utilize the keyboard. PyAutoGUI Again offers a simple solution for us here by typing the following.

pyautogui.typewrite(‘The text you want to type’)

along with just like of which, we can type text!

yet what about when we want to utilize the enter key? We’re going to need the item in order to browse to Twitter. This kind of is usually still done with the “typewrite” function, yet the argument we pass will look a little different, as seen below.


Instead of passing a string, we pass a list containing ‘enter’. We can pass a list containing several key names, along with typewrite will type all of them in order. For instance, we can type the following example.


This kind of will output “XYab” because the item types “ab”, then moves the cursor left two spaces, along with then types ‘XY’.

While we won’t need the item just for This kind of project, the item is usually also worth noting of which keyDown(keyname) along with keyUp(keyname) can be used for sustained key presses. You should know, these don’t require you to put button names into a list. This kind of is usually useful when the item comes to creating programs of which play video games.

Step 3: Writing TweetJack.Py

currently of which we know what we’re working with, let’s start writing our program. Looking at the problem programmatically, we have two tasks we want to complete.

First, we want to open a web browser along with browse to www.twitter.com.
Second, once we’re on Twitter, we want to tweet something. logging in?

To do This kind of, let’s first write a function of which will allow us to browse to a specified website. You can see the function below.

Lines 1 along with 2 import PyAutoGUI, along with the sleep function. As you’ll soon find out, we’re going to need to use the sleep function to make sure of which our previous commands executed successfully before we move on to the next one.

On line 3, we define a function called “browse” along with pass the item a parameter “website”. On lines 4 along with 5, we tell the browse function to use two variables declared outside the scope of the function: x along with y. These are the width along with height of the screen. We’ll actually assign these variables later.

Check Out: How to Secretly Hack Into, Switch On, & Watch Anyone’s Webcam Remotely

Once we have everything we need, we can go about opening up the web browser. When we do This kind of, we’re going to make a few of assumptions. First, we’re assuming of which Google Chrome is usually installed on the target machine. This kind of can be easily swapped for another browser, yet for currently, we’ll stick to Chrome. Second, we’re assuming of which the target is usually using Windows 10. The script should work for Windows 7 as well, yet definitely won’t for Windows 8.1, Linux, or Mac OS X.

Once you see the process we go through, however, you’ll be able to modify the program accordingly. Finally, we’re going to assume of which the user is usually already logged into Twitter. This kind of is usually a fairly reasonable assumption, as most people who use social media on their computers stay logged in.

Moving on, we see of which on line 7 we move the mouse to the location (0,y-1). Why might we do of which? Well, in order to open Google Chrome we’re going to rely on the Windows search bar. To open the search bar, we’re going to open up the start menu inside very bottom left of the screen. This kind of means of which the actual coordinate of the location we want the mouse to be will vary, depending on the screen size.

Next, we’re going wait a second to make sure everything has loaded, along with just start typing. Once we finish typing ‘Google Chrome’, the Windows search program will have found what we need. coming from here we can just hit the enter key, along with Google Chrome should open up.

Also Check Out: Turn Any Phone into a Hacking Super Weapon with the Sonic

As we continue coding together, I might encourage you to do some problem-solving on your own. Given the tools we discussed earlier, how might you go about finishing the process of browsing to a website? Keep in mind working with varying screen sizes.

Moving on, let’s look at the rest of the function below.

On line 15, we sleep for 5 seconds to make sure of which Chrome is usually open before we start clicking all over the place. Depending on the speed of the system, This kind of interval may need to be increased.

Figuring Out Where to Click

coming from there, we can see on line 16 of which we move the mouse to (297,63). Let’s look at how we arrived at of which location.

First, you’re going to want to fire up a python interpreter in a command prompt. To do This kind of, open up a terminal along with type python. Next, type import pyautogui. With This kind of terminal, we can move the mouse around on the screen, then type ‘x,y = pyautogui.position()’ to store the coordinates of the mouse.

Using python shell via our favorite command-line tool, we can grab the location of the mouse on the fly. Image by Allegiance/Nullbyte

By using the position() function, we can find a location which will hover the mouse over the browser search bar. Once we’re there, we click on the search bar along with type inside website parameter of which was passed to the function. After the website name has been typed into the search bar, we hit enter to browse to of which website.

As a quick side note, while I’ve arranged the following lines as you see below, I could use another way of doing the same thing.


The following might do the same.


Step 4: Sending Our Tweets

Before we jump into writing This kind of next function, let’s take a look at the structure of the Twitter website:

A shameless Twitter plug. Image by Allegiance/Nullbyte

We see there’s a big blue “Tweet” button inside upper right-hand corner of which will do what we want. We also see of which a user can directly tweet coming from the text input box inside center of the screen. Either one might work for our purposes. In This kind of example, we’re going to use the tweet button.

Check Out: How to Spy on Anyone, Part 1 (Hacking Computers)

Clicking on the tweet button, we see the following:

Image by Allegiance/Nullbyte

We notice of which the text input box is usually active as soon as the button is usually clicked, which means we won’t have to try to click on the item before typing. This kind of is usually a nice break for us.

So once we browse to the website, our list of steps to send a tweet looks like This kind of:

  • Move the mouse to the tweet button
  • Click the button.
  • Type inside tweet.
  • Move the mouse to the tweet box.
  • Click the tweet box.

Simple enough. Let’s walk through the code just for This kind of function.

Step 5: Coding Our Function

Image by Allegiance/Nullbyte

When we’re tweeting, the first thing we’re going to want to do is usually actually browse to Twitter. We do This kind of in line 22, when we call the browse function we just wrote along with pass ‘www.twitter.com’ as an argument. Once of which function executes, we need to make sure of which the function uses the global variables x along with y, which we will assign later.

As the comment on line 25 explains, we sleep for 5 seconds on line 26 in order to make sure of which Twitter has sufficiently loaded before we try to tweet. Then, we move the cursor to the location (x-271,105).

By getting the position() of the mouse when the item is usually over the tweet button, we see of which the button is usually roughly 270 pixels coming from the right edge of the screen.

the item’s a better idea to use This kind of value instead of hard-coding the coordinates in, because the item allows our program to work for varying screen sizes. We didn’t need to worry about This kind of when we were clicking the URL bar in Google Chrome because the bar is usually within the first 300 pixels of width along with height. Unless laptops have 200×200 screens currently, the item’s safe to say of which we could hard code of which value.

Don’t Miss: How to Use Netcat, the Swiss Army Knife of Hacking Tools

The same is usually not true for the tweet button. On a 1920×1080 screen, the button resides at (1650,105). If the victim features a screen of which is usually 1280×720 in size, This kind of hard-coded value wouldn’t work.

Once we have clicked the tweet button, we’ll type in whatever was passed for the content parameter. After of which, we’ll move the mouse to the completely new tweet button, along with send our tweet!

currently we have to finish things up. We’ll get our tweet coming from the command line along with assign the item to the variable theTweet. Then, we’re FINALLY going to assign x along with y with the output coming from the pyautogui.size function. Once This kind of is usually done, we can call the tweet function along with enjoy the fruits of our labor.

Have Fun with PyAutoGUI

I expect you found This kind of introduction to PyAutoGUI helpful. I definitely love This kind of module because of how much you can do with the item. There are so many different applications of which go beyond security, along with there’s so much fun to be had for anyone interested in learning about the item.

Thank you so much for reading! Comment below with any questions, or contact me via Twitter @xAllegiance.

Cover photo via Photo-Mix
Screenshots by Allegiance/Nullbyte

Leave a Comment

Your email address will not be published. Required fields are marked *

3 × four =