Selenium was the default for 15 years. Playwright launched in 2020 and has been steadily taking over. Now, the question is not “should I learn Selenium?” \u2014 it is “when does Selenium still make sense?”<\/p>\n\n\n\n
Let’s talk about these technologies and go through a Python example that solves this mystery. Or, if you are not into reading definitions and just want to go straight to the point, skip to “The codebase” section of this blog post.<\/p>\n\n\n\n
What is Data Scraping and when is it justified?<\/h2>\n\n\n\n
Data scraping is the process of retrieving information generated by another program. Web scraping is the specific case where that information lives on a website.<\/p>\n\n\n\n
It should be treated as a last resort. APIs exist precisely to give you structured, reliable access to the data you need. Consuming an API<\/a> is faster, more stable, and carries none of the legal or technical fragility that scraping introduces. If an API is available, use it.<\/p>\n\n\n\n When scraping is justified, no API exists, data is publicly accessible, and the site’s structure is stable. The method you choose depends on what you are scraping.<\/p>\n\n\n\n When you do not need a browser:<\/strong> If the data you need is in the initial HTML response, a lightweight HTTP approach is unbeatable. No browser overhead, no JavaScript execution, no ChromeDriver to manage.<\/p>\n\n\n This approach \u2014 When you need a browser:<\/strong> JavaScript-rendered pages, single-page applications, dynamic content loaded after page load, login flows, form interactions \u2014 these require browser automation. This is where Selenium and Playwright enter the picture, and where Lambda becomes the right deployment target.<\/p>\n\n\n\n One famous package used for this is Selenium<\/a>. This technique requires a driver to communicate with the browser installed (e.g. ChromeDriver<\/a>, GeckoDriver<\/a>).<\/p>\n\n\n\n This driver provides an API so the Selenium package can manage the browser. The driver version needs to be compatible with the installed browser version<\/p>\n\n\n\n Here’s an example of a Web Scraper that opens a browser gets the header element of a page and prints its header content:<\/p>\n\n\n\n In this blog post, we are going to set the web scraper script on the cloud<\/strong>. Cloud computing is making services like hosting and storage, available over the internet.<\/p>\n\n\n\n Some famous examples of cloud providers are Amazon AWS<\/a>, Google Cloud<\/a>, and Microsoft Azure<\/a>. There are several advantages of using a cloud solution, for example:<\/p>\n\n\n\n Some of these cloud services<\/a> are serverless<\/strong>, which means these services are executed by demand and the cloud provider takes care of the server infrastructure on behalf of the customer. <\/p>\n\n\n\n Some of these cloud services require the usage of containers<\/strong>, which are executions of virtualized operating systems based on images (templates). Docker<\/a> is the main platform for working with containers.<\/p>\n\n\n\n Although Cloud computing is great, it is not needed for every project. Running the script on your machine might be enough. <\/p>\n\n\n\n Each one of the providers mentioned has several ways to host a web scraper script. This AWS blog post<\/a> describes 3 options.<\/p>\n\n\n\n To summarize the blog post, the options are:<\/strong><\/p>\n\n\n\n Here, we will use AWS Lambda. Its main limitations are the timeout limit, which is 15 minutes and the deployment package can’t exceed 250 MB (but it accepts up to 10 GB using containers).<\/p>\n\n\n\n The example script is very simple and takes less than a minute to execute, but as we previously mentioned, Selenium requires a browser, and the Chrome binary size is around 500 MB, which forces us to use the container approach.<\/p>\n\n\n\n Read more: <\/strong>AWS FinOps Best Practices: How to Cut and Optimize Cloud Costs<\/a><\/p>\n<\/blockquote>\n\n\n\n There are several ways to set up an AWS Lambda function. One easy way is by using the Serverless Framework<\/a>. The Serverless framework helps us to develop and deploy Lambda Functions by using a single YAML file to declare the lambda functions, their infrastructure, and the events that will trigger them.<\/p>\n\n\n\n Using the Serverless Framework also allows us to deploy the lambda functions with a single command, simplifying the process a lot.<\/p>\n\n\n\n The Serverless Framework also provides an optional dashboard which gives us an interface to check the function’s health, trigger events manually, and check their logs.<\/p>\n\n\n\n In this section, we will analyze some important files of a demo project which can be checked here<\/a>. serverless.yml<\/strong><\/a><\/p>\n\n\n\n This is the file where we set the lambda application infrastructure, the lambda functions, and the events that are going to trigger them.<\/p>\n\n\n\n In the provider<\/strong> section of this file, we declare that we are going to use a docker image named img<\/em>. The functions<\/strong> section is where we set the lambda functions and their specific configuration like environment variables and handler functions.<\/p>\n\n\n\n Notice that here we add environment variables that will have the browser and its driver path, we state that we will use the image that was previously set, that the command that will be executed when the lambda function is triggered is the example.py <\/em>file handler, and that the event that will trigger it is a cronjob<\/em> that is scheduled for every 6 hours.<\/p>\n\n\n\nfrom<\/span> bs4 import<\/span> BeautifulSoup\nfrom<\/span> urllib.request import<\/span> urlopen\n\nwith<\/span> urlopen('https:\/\/en.wikipedia.org\/wiki\/Main_Page'<\/span>) as<\/span> response:\n soup = BeautifulSoup(response, 'html.parser'<\/span>)\n for<\/span> anchor in<\/span> soup.find_all('a'<\/span>):\n print(anchor.get('href'<\/span>, '\/'<\/span>))<\/code><\/span>Code language:<\/span> JavaScript<\/span> (<\/span>javascript<\/span>)<\/span><\/small><\/pre>\n\n\nrequests<\/strong><\/code> or urllib<\/strong><\/code> plus BeautifulSoup<\/code><\/strong> for HTML parsing \u2014 covers the majority of simple scraping use cases and requires no cloud infrastructure beyond a basic function.<\/p>\n\n\n\nWeb scraping techniques: A quick reference<\/strong><\/h2>\n\n\n\n
\n
curl<\/code> and grep<\/code> to extract patterns from raw HTML. Fast for simple, stable targets with predictable markup.<\/li>\n\n\n\nBeautifulSoup<\/code> is the standard Python library. No browser required. Suited for static content.<\/li>\n\n\n\nfrom<\/span> selenium import<\/span> webdriver\nfrom<\/span> selenium.webdriver.common.by import<\/span> By\n\ndriver = webdriver.Chrome() # Open browser<\/span>\ndriver.get(\"http:\/\/example.com\"<\/span>) # Access page<\/span>\nheader_element = driver.find_element(By.CSS_SELECTOR, 'h1'<\/span>) # Find H1 element<\/span>\nprint(header_element.text) # Print found element text content<\/span>\ndriver.quit() # Close browser<\/span><\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre><\/div>\n<\/div>\n\n\n\nWhy AWS Lambda for web scraping?<\/strong><\/h2>\n\n\n\n
\n
\n
\n
How are we going to deploy it to AWS Lambda?<\/strong><\/h2>\n\n\n\n
The codebase<\/strong><\/h2>\n\n\n\n
![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/web-scrapper-aws-lambda-1200x275.jpg\")
<\/figure>\n\n\n\n![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/web-scrapping-aws-1200x379.jpg\")
<\/figure>\n\n\n\n