Learn how to combine OpenCV object detection with PyAutoGUI and Threading to build a custom Python video game bot. Using the OpenCV image recognition techniques discussed earlier in this tutorial, we can now use that data to perform mouse clicks automatically. We'll use Python threads so that our bot actions can happen independently of our object detection.

Full tutorial playlist:    • OpenCV Object Detection in Games  

Grab the code on GitHub: https://github.com/learncodebygaming/...
PyAutoGUI: https://pyautogui.readthedocs.io/en/l...
PyDirectInput: https://github.com/learncodebygaming/...

0:40 Automating mouse clicks
4:58 Basic threading
10:47 Advanced threading
18:44 Botting logic
22:10 Enum data structure
32:40 Stack data structure
36:22 Bot demo

Email me: [email protected]
Full written tutorial: https://learncodebygaming.com/blog/ho...
My gear: https://learncodebygaming.com/gear

Building a bot with OpenCV is a matter of combining the object detection techniques that we've discussed (in the last 8 parts of this tutorial series) with some GUI automation. Now that you know how to find an object using OpenCV, you just need to use something like PyAutoGUI or PyDirectInput to click on the objects you find. But there are a few architectural decisions that can make it kind of difficult to know where to get started. So I'll walk you through one way to approach this that should be pretty flexible and act as a good foundation for building a more capable bot.

The most straightforward way to add automated mouse clicks to our bot script is to do it inline. By that I mean we can take a screenshot, process it to find the objects we want, and then perform any actions we want our bot to take. So we can do that sequentially, and just loop back around when our bot is done with its actions. So let me show you what that would look like, and then I'll point out some of the limitations with that method.

[see the code samples on learncodebygaming.com]

In this code, once we have the detected objects in the form of a list of rectangles, we first convert those into click positions using the get_click_points() and get_screen_position() functions we wrote previously. Then it's taking the very first screen position in that list, moves the mouse to that position using pyautogui.moveTo(), and then we use PyAutoGUI again to do the mouse click. Hopefully this has resulted in clicking on our target (a limestone deposit in Albion Online), and we pause our script for 5 seconds using sleep() to give our character time to do that mining.

So one of the problems this code has is it's going to take at least 5 seconds between each update of our debug output. That isn't great for seeing how well our object detection is working, compared to the continuous video stream we had before. But writing a bot this way will still work, and it keeps things simple if you don't mind doing the object detection only when you need it.

If you want to get more advanced, you can fix this problem by using threading.

Threading, or multithreading, is the ability that all modern computers have to run multiple processes or threads in parallel at the same time. This is what allows you to have multiple programs running at once. Right now our bot script runs as just a single thread, but by using the Python threading library we can make it branch out and do multiple things at once (run in multiple threads).

In our case, what it'd be great to do, is to have our main script running in the main thread, and then branch off our bot actions into a separate thread. That way the bot can go off and do the things it needs to do without blocking the execution of our main thread where the object detection is happening.

In the main loop we call Thread() to create a new thread object in our program. We tell it what code to execute by giving it a function name as the target, and then passing in the parameters needed by the function as a tuple. We can then run that function in a separate thread by calling start() on that thread objects.

In this example, I'm using a global variable to make sure only one bot action thread is running at a time. This is a simple way to pass information between threads. This can be fine for smaller scripts, but using global variables is generally discouraged and can make code difficult to maintain. So let me show you a better way to do this without using global variables.

Let's put everything into threads. So we'll have one thread for capturing our screenshots, another for doing the object detection, another for the bot actions, and then our main thread coordinating all this.

We'll start with the object detection. Let's pull this out of the main script and put it into its own class.

Continue with the written tutorial here: https://learncodebygaming.com/blog/ho...

#programming #python