Today: November 28, 2021 6:01 pm

Create your Browser Automation Robot with Python and Selenium

More and more business processes are being automated. In a previous blog post, I explained how to get started with the automation of robotic processes using the Python Pyautogui module. But for web browser automation scripts, you can take an extra look at the Selenium module. This will allow you to automate boring and repetitive tasks on the Internet, for example a huge time saving on data entry in the CMS.


The advantage of the Selenium module is that it knows how to interpret html code. Your robot can easily navigate in and between web pages. Another advantage of Selenium over Pyautogui is that it does not take control of your mouse or keyboard, but is a process that takes place in the background.
As you can see in the video below, the robots are really good at performing specific tasks. So let’s use the power of robots.

Getting started with Selenium in Python

  1. Installing, importing and starting the driver

First install Selenium and import the webdriver. This code will launch the Google Chrome browser

!pip install selenium
!apt-get update
!apt install chromium-chromedriver
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(‘ — headless’)
chrome_options.add_argument(‘ — no-sandbox’)
chrome_options.add_argument(‘ — disable-dev-shm-usage’)
driver =webdriver.Chrome(‘chromedriver’,chrome_options=chrome_options)

2. Open a webpage

This code will open the url and print the source code of the URL.

driver.get("https://www.google.com/")
print(driver.page_source)

3. Selecting elements on the page

You can use element ID or xpath to select form fields, buttons, hyperlinks, actually all elements on the page.

Does your element has a specific html ID? Then you can use

driver.find_element_by_id("element_id")

Otherwise you need to make use of XPath to select your element. First right You can find an XPath in the developer console as you see below. Select Copy XPath. The XPath of the search field is //*[@id=’tsf’]/div[2]/div[1]/div[1]/div/div[2]/input

Image for post
xpathsearch = "//*[@id='tsf']/div[2]/div[1]/div[1]/div/div[2]/input"
searchinput = driver.find_element_by_xpath(xpathsearch)

Ok, now we selected the search element on Google.com. Let’s try making a Google search with Selenium.

4. Keyboard input

Once you selected an element you can manipulate this element with your keyboard and your mouse.

Use the send_keys function to write your search text tothe search field on Google.com and the ENTER key to start the search.

searchinput.send_keys("Barack Obama age")
searchinput.send_keys(Keys.ENTER)

5. Reading the result

Once you executed the search, you can read the result by using the XPath of the div which contains the result. You can read the text property of the element.

Image for post
xpathresult = "//*[@id='rso']/div[1]/div[1]/div[1]/div[1]/div/div[2]/div/div[1]/div/div/div[1]"
resultdiv = driver.find_element_by_xpath(xpathresult)
age = resultdiv.text
print("Barack Obama is getting old. Currently, his age is",age)

Caution: Google doesn’t like bots and may block your script when you execute it multiple times in a row.

Download the Colab notebook

Case: Automating CMS input

Image for post

It’s time to give a practical example. Suppose you have written a bunch of meta descriptions in a spreadsheet and sent them to your client. Your client likes the new meta descriptions, and may suggest that you enter them into the CMS.
You can now do this manually, one by one, but it’s not the most exciting task. Python and Selenium can come to the rescue.
In this Colab notebook, you’ll find the code to automate the entry of meta descriptions in the Drupal CMS.
As you can see, it only took a few lines of code. By automating this process, you save time and become more efficient.