How To Use Selenium To Web-Scrape on AWS Lambda

Web scraping might save several hours when compared to manually collecting data from websites, and AWS Lambda is a good way to set up scripts to run by demand.

In this blog post, we’ll cover everything you need to know,  the good and bad things about Web Scraping and AWS Lambda, and also analyze the code of an example project where we can combine these two technologies that fit very well together.

If you don’t have a good way to retrieve information from a website, data scraping is the way to go. Besides, it’s awesome to see a browser opening, clicking on buttons, and filling out forms by itself.

Selenium was the default for 15 years. Playwright launched in 2020 and has been steadily taking over. Now, the question is not “should I learn Selenium?” — it is “when does Selenium still make sense?”

Let’s talk about these technologies and go through a Python example that solves this mystery. Or, if you are not into reading definitions and just want to go straight to the point, skip to “The codebase” section of this blog post.

What is Data Scraping and when is it justified?

Data scraping is the process of retrieving information generated by another program. Web scraping is the specific case where that information lives on a website.

It should be treated as a last resort. APIs exist precisely to give you structured, reliable access to the data you need. Consuming an API is faster, more stable, and carries none of the legal or technical fragility that scraping introduces. If an API is available, use it.

When scraping is justified, no API exists, data is publicly accessible, and the site’s structure is stable. The method you choose depends on what you are scraping.

When you do not need a browser: If the data you need is in the initial HTML response, a lightweight HTTP approach is unbeatable. No browser overhead, no JavaScript execution, no ChromeDriver to manage.

from bs4 import BeautifulSoup
from urllib.request import urlopen

with urlopen('https://en.wikipedia.org/wiki/Main_Page') as response:
    soup = BeautifulSoup(response, 'html.parser')
    for anchor in soup.find_all('a'):
        print(anchor.get('href', '/'))Code language: JavaScript (javascript)

This approach — requests or urllib plus BeautifulSoup for HTML parsing — covers the majority of simple scraping use cases and requires no cloud infrastructure beyond a basic function.

When you need a browser: JavaScript-rendered pages, single-page applications, dynamic content loaded after page load, login flows, form interactions — these require browser automation. This is where Selenium and Playwright enter the picture, and where Lambda becomes the right deployment target.

Web scraping techniques: A quick reference

• Copy and paste: Manual. Every developer has done this. Not worth automating for one-off tasks.
• Text pattern matching: Using tools like curl and grep to extract patterns from raw HTML. Fast for simple, stable targets with predictable markup.
• HTML parsing: Parsing the response into a navigable element tree — BeautifulSoup is the standard Python library. No browser required. Suited for static content.
• DOM parsing: A browser controlled by automation software interacts with the page as a real user would — opening pages, clicking buttons, filling forms, waiting for JavaScript to execute, then extracting data from the rendered DOM. This is Selenium and Playwright’s domain. It is also the most resource-intensive approach and requires the most infrastructure to run reliably in the cloud.

One famous package used for this is Selenium. This technique requires a driver to communicate with the browser installed (e.g. ChromeDriver, GeckoDriver).

This driver provides an API so the Selenium package can manage the browser. The driver version needs to be compatible with the installed browser version

Here’s an example of a Web Scraper that opens a browser gets the header element of a page and prints its header content:

from selenium import webdriver
from selenium.webdriver.common.by import By

driver = webdriver.Chrome() # Open browser
driver.get("http://example.com") # Access page
header_element = driver.find_element(By.CSS_SELECTOR, 'h1') # Find H1 element
print(header_element.text) # Print found element text content
driver.quit() # Close browserCode language: Python (python)

Why AWS Lambda for web scraping?

In this blog post, we are going to set the web scraper script on the cloud. Cloud computing is making services like hosting and storage, available over the internet.

Some famous examples of cloud providers are Amazon AWS, Google Cloud, and Microsoft Azure. There are several advantages of using a cloud solution, for example:

• You don’t need to set up your server
• There are a lot of engineers working to make sure these services don’t have any security breaches
• It’s usually very easy to escalate the capabilities of the cloud service you are using. In many of these services, you can increase the memory and processing capacity just by clicking some buttons

Some of these cloud services are serverless, which means these services are executed by demand and the cloud provider takes care of the server infrastructure on behalf of the customer. 

Some of these cloud services require the usage of containers, which are executions of virtualized operating systems based on images (templates). Docker is the main platform for working with containers.

Although Cloud computing is great, it is not needed for every project. Running the script on your machine might be enough. 

Each one of the providers mentioned has several ways to host a web scraper script. This AWS blog post describes 3 options.

To summarize the blog post, the options are:

1. Create a virtual machine using the EC2 Service. This is the most primitive option, you’d need to set up the machine just like a regular one and it would be kept on 24 hours a day. This is also the most expensive solution out of the 3 options.
2. Containerize the script and use it on the AWS Fargate service. Fargate is a serverless option, which is useful because the web scraper only needs to be executed by demand. Fargate is also cheaper than EC2.
3. Use AWS Lambda, which is also a serverless service that supports both raw code and containerized scripts. It has more limitations compared to Fargate, but it is enough in most cases. It’s the cheapest service and your script might even fit the free tier.

Here, we will use AWS Lambda. Its main limitations are the timeout limit, which is 15 minutes and the deployment package can’t exceed 250 MB (but it accepts up to 10 GB using containers).

The example script is very simple and takes less than a minute to execute, but as we previously mentioned, Selenium requires a browser, and the Chrome binary size is around 500 MB, which forces us to use the container approach.

Read more: AWS FinOps Best Practices: How to Cut and Optimize Cloud Costs

How are we going to deploy it to AWS Lambda?

There are several ways to set up an AWS Lambda function. One easy way is by using the Serverless Framework. The Serverless framework helps us to develop and deploy Lambda Functions by using a single YAML file to declare the lambda functions, their infrastructure, and the events that will trigger them.

Using the Serverless Framework also allows us to deploy the lambda functions with a single command, simplifying the process a lot.

The Serverless Framework also provides an optional dashboard which gives us an interface to check the function’s health, trigger events manually, and check their logs.

The codebase

In this section, we will analyze some important files of a demo project which can be checked here. serverless.yml

This is the file where we set the lambda application infrastructure, the lambda functions, and the events that are going to trigger them.

In the provider section of this file, we declare that we are going to use a docker image named img. The functions section is where we set the lambda functions and their specific configuration like environment variables and handler functions.

Notice that here we add environment variables that will have the browser and its driver path, we state that we will use the image that was previously set, that the command that will be executed when the lambda function is triggered is the example.py file handler, and that the event that will trigger it is a cronjob that is scheduled for every 6 hours.

Dockerfile

The Dockerfile is the template where we configure our container image. This file creates a Linux instance capable of running the web scraper. The template installs the project requirements (including Chrome and Chrome driver) and copies the required files to the image.

Example.py

This file has a function that will be executed when the event is triggered. This file is pretty similar to the Selenium example we showed earlier.

The main difference is that we customize the browser to not display an interface (because the lambda does not have a display) and to use a single process (because the lambda only has 1 CPU).

In this case, the handler function returns a dictionary with a status code and a body just in case we want to change the event from a cronjob to an HTTP request.

Deploying

To deploy the lambda function we just need to run a single command on the terminal.

In this case, Serverless Frameworks raises a warning message that explains that the dashboard does not support functions that use container images. It means that we will not be able to check the lambda function logs, nor trigger the lambda function manually through the dashboard.

But we can still do it through the AWS console. Below we have a screenshot of a successful log retrieved from AWS Cloudwatch:

Selenium on Lambda: Full Setup Updated for 2026

The architecture is unchanged from the original article. The updates are in the Python runtime (3.12, Lambda’s latest supported version), the Chrome headless flag syntax (changed in Chrome 112), and Serverless Framework v4.

scraper/
├── Dockerfile
├── handler.py
├── requirements.txt
└── serverless.yml

Dockerfile:

FROM public.ecr.aws/lambda/python:3.12

# Install Chromium and ChromeDriver
RUN dnf install -y chromium chromedriver

# Install Python dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt --target "${LAMBDA_TASK_ROOT}"

# Copy handler
COPY handler.py ${LAMBDA_TASK_ROOT}

CMD ["handler.handler"]Code language: PHP (php)

requirements.txt:

selenium==4.18.1

handler.py:

python

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By

def handler(event, context):
    options = Options()
    options.add_argument("--headless=new")   # Updated: Chrome 112+ syntax
    options.add_argument("--no-sandbox")
    options.add_argument("--disable-dev-shm-usage")
    options.add_argument("--single-process")
    options.add_argument("--disable-gpu")
    options.add_argument("--window-size=1920,1080")

    driver = webdriver.Chrome(options=options)

    try:
        driver.get(event.get("url", "https://example.com"))
        title = driver.title
        header = driver.find_element(By.CSS_SELECTOR, 'h1').text
        return {"title": title, "header": header}
    finally:
        driver.quit()  # Always close the browserCode language: PHP (php)

Key updates from the original:

The --headless flag was deprecated in Chrome 112. The new flag is --headless=new. Using the old flag produces compatibility warnings and may cause rendering issues on modern Chrome versions. Always use --headless=new in 2026.

The try/finally block around driver.quit() is also important — without it, a scraping error will leave a Chrome process running inside the Lambda container, consuming memory for the remainder of the invocation.

serverless.yml (Serverless Framework v4):

service: selenium-scraper

provider:
  name: aws
  region: us-east-1
  timeout: 900  # 15 minutes maximum

functions:
  scraper:
    image:
      uri: <your-ecr-image-uri>
    memorySize: 2048  # Chrome needs memory — 2GB is a safe starting point
    events:
      - schedule: rate(6 hours)Code language: PHP (php)

Build, push, and deploy:

bash

# Build and push the container to ECR
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin <account-id>.dkr.ecr.us-east-1.amazonaws.com
docker build -t selenium-scraper .
docker tag selenium-scraper:latest <account-id>.dkr.ecr.us-east-1.amazonaws.com/selenium-scraper:latest
docker push <account-id>.dkr.ecr.us-east-1.amazonaws.com/selenium-scraper:latest

# Deploy with Serverless Framework
serverless deployCode language: HTML, XML (xml)

Note: container-based Lambda functions cannot be managed through the Serverless dashboard UI, but logs are accessible via AWS CloudWatch. Monitor your scraper’s execution time and memory usage there — Chrome is memory-hungry, and Lambda will terminate functions that exceed the configured memory ceiling.

When to Use Playwright Instead

If you are starting a new scraping project in 2026, Playwright is the stronger default. The practical advantages are real:

Auto-waiting eliminates an entire class of bugs. The most common cause of flaky scrapers is timing — an element is not yet visible, a network request has not completed, a JavaScript animation is still running. Selenium requires explicit WebDriverWait calls for every interaction. Miss one, and the scraper breaks intermittently, making debugging painful. Playwright waits automatically for elements to be actionable before interacting with them. This alone removes the majority of timing-related scraper failures.

Async support enables dramatically higher throughput. Playwright’s async API makes concurrent scraping straightforward. Scraping 20 URLs concurrently is a handful of lines with asyncio. In Selenium, the same task requires thread management and is significantly more complex to debug.

Setup is simpler. No ChromeDriver version matching. No compatibility matrix between Chrome and ChromeDriver versions. playwright install downloads the correct browser binaries automatically and keeps them in sync.

Playwright on Lambda:

python

import asyncio
from playwright.async_api import async_playwright

async def scrape(url: str) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(
            headless=True,
            args=["--no-sandbox", "--disable-dev-shm-usage"]
        )
        page = await browser.new_page()

        try:
            await page.goto(url, wait_until="networkidle")
            title = await page.title()
            header = await page.locator("h1").text_content()
            return {"title": title, "header": header}
        finally:
            await browser.close()

def handler(event, context):
    url = event.get("url", "https://example.com")
    return asyncio.run(scrape(url))Code language: JavaScript (javascript)

The Dockerfile structure for Playwright on Lambda is similar — containerized function with Playwright and its browser binaries installed. The playwright install chromium command during the image build handles all binary management.

When Selenium still makes sense

Rewriting working Selenium code is not justified by “Playwright is newer.” If you have an existing Selenium codebase that runs reliably, the migration cost is real and the benefit is marginal for stable scrapers.

Stick with Selenium when: the codebase is established and functioning, your team is invested in Selenium Grid for distributed execution, or you need Java, C#, or Ruby — languages where Playwright’s bindings are less mature than Python and JavaScript.

Anti-bot detection

Both Selenium and Playwright are detectable by modern anti-bot systems. Cloudflare, Akamai, PerimeterX, and DataDome all fingerprint browser automation tools at the network and browser level — inspecting headers, JavaScript properties, mouse movement patterns, and timing signatures.

Vanilla Selenium is particularly detectable because it sets navigator.webdriver = true by default. Stealth wrappers like undetected-chromedriver patch this, but detection systems evolve and stealth patches require maintenance.

For targets with serious anti-bot protection, managed browser infrastructure is the more pragmatic choice. Browserless, Scrapfly, and Apify manage browser pools, proxy rotation, and anti-detection at the infrastructure level. The cost per request is higher than running your own Lambda; the reliability and maintenance burden are significantly lower.

For unprotected targets — internal tools, publicly accessible data with no bot detection — a straightforward Selenium or Playwright Lambda setup works reliably.

Read more: Harness Engineering: Why “Done” Isn’t the Agent Saying So

Conclusions

This is a very specific example of the usage of AWS Lambda and Selenium but I hope it can illustrate the potential of these technologies. Instead of creating a web scraper, we can create functions that run end-to-end tests that, in case of failure, send a Slack message, or we can create an API that calculates the distance between two strings and returns it in the HTTP response. It’s all up to your imagination!