- \n
- Introduction<\/a>\n
- \n
- Data Scraping<\/a><\/li>\n\n\n\n
- Web scraping techniques<\/a><\/li>\n<\/ul>\n<\/li>\n\n\n\n
- How to use the Web Scraper script?<\/a><\/li>\n\n\n\n
- How are we going to deploy it to AWS Lambda?<\/a>\n
- \n
- The codebase<\/a><\/li>\n\n\n\n
- Deploying<\/a><\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Conclusions<\/a><\/li>\n\n\n\n
- Resources<\/a><\/li>\n<\/ul>\n\n\n\n
Introduction<\/strong><\/h2>\n\n\n\n
If you don’t have a good way to retrieve information from a website, data scraping is the way to go. Besides, it’s awesome to see a browser opening, clicking on buttons, and filling out forms by itself.<\/p>\n\n\n\n
AWS Lambda Functions is an easy and low-price service to deploy scripts to the cloud, but how can we have a Chrome browser installed in an AWS lambda function?<\/p>\n\n\n\n
Let’s talk about these technologies and go through a Python example that solves this mystery. Or, if you are not into reading definitions and just want to go straight to the point, skip to “The codebase” section of this blog post.<\/p>\n\n\n\n
Data Scraping<\/h2>\n\n\n\n
Data Scraping is retrieving information generated from another program. In this blog post, we will focus on Web Scraping, which is a type of data scraping that occurs when the data being retrieved is from a website. <\/p>\n\n\n\n
You should consider web scraping as a last-resort alternative because it’s usually more beneficial to consume APIs<\/a> (application programming interfaces) which are interfaces made to get the exact information you want.<\/p>\n\n\n\n
Web scraping often requires more processing capacity and is also very likely to break if the website changes its display, or even if an element changes its identification parameters.<\/p>\n\n\n\n
However, if there is no convenient API to access the data you need and the website you want to scrape is unlikely to have frequent interface changes, Web Scraping is the way to go!<\/p>\n\n\n\n
List of some web scraping techniques:<\/strong><\/h3>\n\n\n\n
- \n
- Copy and Paste<\/strong><\/li>\n<\/ul>\n\n\n\n
This technique is very simple and manual. To use it, you only need to select the text you want to get, copy it, and paste it wherever you want to save it. Every programmer has already web scraped from stack overflow ^^<\/p>\n\n\n\n
- \n
- Text Pattern Matching<\/strong><\/li>\n<\/ul>\n\n\n\n
This technique consists of finding a text pattern match within the website-generated HTML. Below is a screenshot where we combine the Unix commands curl and grep respectively, get the website HTML, and find the text inside the quotes, which is the title tag in this case.<\/p>\n\n\n\n
![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-scraper-aws-lambda-1200x226.png\")
<\/figure>\n\n\n\n- \n
- HTML Parsing<\/strong><\/li>\n<\/ul>\n\n\n\n
In this case, instead of dealing with the raw HTML text, the HTML is parsed into elements, making it easier to find the content you want to scrape. <\/p>\n\n\n\n
Below is a Python<\/a> example using the package Beautiful Soup<\/a>. In this script, we make a request to Wikipedia’s main page, load its response into a BeautifulSoup instance, then use it to get all of the anchor elements of the page, and finally, iterate through the elements found and print each one of its HREF properties:<\/p>\n\n\n\n
\nfrom<\/span> bs4 import<\/span> BeautifulSoup\nfrom<\/span> urllib.request import<\/span> urlopen\n\nwith<\/span> urlopen('https:\/\/en.wikipedia.org\/wiki\/Main_Page'<\/span>) as<\/span> response:\n soup = BeautifulSoup(response, 'html.parser'<\/span>)\n for<\/span> anchor in<\/span> soup.find_all('a'<\/span>):\n print(anchor.get('href'<\/span>, '\/'<\/span>))<\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre>\n\n\nRetrieved from https:\/\/en.wikipedia.org\/wiki\/Beautiful_Soup_(HTML_parser)#Code_example<\/a><\/p>\n<\/div>\n<\/div>\n\n\n\n
- \n
- DOM Parsing<\/strong><\/li>\n<\/ul>\n\n\n\n
This technique uses a browser controlled by automation software to interact and retrieve the information on the website. So with DOM parsing, you are capable of opening a page, clicking buttons, filling forms, running your scripts on the page, and then getting the information you want. This is the technique we will focus on.<\/p>\n\n\n\n
One famous package used for this is Selenium<\/a>. This technique requires a driver to communicate with the browser installed (e.g. ChromeDriver<\/a>, GeckoDriver<\/a>).<\/p>\n\n\n\n
This driver provides an API so the Selenium package can manage the browser. The driver version needs to be compatible with the installed browser version<\/p>\n\n\n\n
Here’s an example of a Web Scraper that opens a browser gets the header element of a page and prints its header content:<\/p>\n\n\n\n
\nfrom<\/span> selenium import<\/span> webdriver\nfrom<\/span> selenium.webdriver.common.by import<\/span> By\n\ndriver = webdriver.Chrome() # Open browser<\/span>\ndriver.get(\"http:\/\/example.com\"<\/span>) # Access page<\/span>\nheader_element = driver.find_element(By.CSS_SELECTOR, 'h1'<\/span>) # Find H1 element<\/span>\nprint(header_element.text) # Print found element text content<\/span>\ndriver.quit() # Close browser<\/span><\/code><\/span>Code language:<\/span> Python<\/span> (<\/span>python<\/span>)<\/span><\/small><\/pre><\/div>\n<\/div>\n\n\n\nHow to use the Web Scraper script? <\/strong><\/h2>\n\n\n\n
In this blog post, we are going to set the web scraper script on the cloud<\/strong>. Cloud computing is making services like hosting and storage, available over the internet.<\/p>\n\n\n\n
Some famous examples of cloud providers are Amazon AWS<\/a>, Google Cloud<\/a>, and Microsoft Azure<\/a>. There are several advantages of using a cloud solution, for example:<\/p>\n\n\n\n
- \n
- You don’t need to set up your server<\/li>\n\n\n\n
- There are a lot of engineers working to make sure these services don’t have any security breaches<\/li>\n\n\n\n
- It’s usually very easy to escalate the capabilities of the cloud service you are using. In many of these services, you can increase the memory and processing capacity just by clicking some buttons<\/li>\n<\/ul>\n\n\n\n
Some of these cloud services<\/a> are serverless<\/strong>, which means these services are executed by demand and the cloud provider takes care of the server infrastructure on behalf of the customer. <\/p>\n\n\n\n
Some of these cloud services require the usage of containers<\/strong>, which are executions of virtualized operating systems based on images (templates). Docker<\/a> is the main platform for working with containers.<\/p>\n\n\n\n
Although Cloud computing is great, it is not needed for every project. Running the script on your machine might be enough.\u00a0<\/p>\n\n\n\n
Each one of the providers mentioned has several ways to host a web scraper script. This AWS blog post<\/a> describes 3 options.<\/p>\n\n\n\n
To summarize the blog post, the options are:<\/p>\n\n\n\n
- \n
- Create a virtual machine using the EC2 Service. This is the most primitive option, you’d need to set up the machine just like a regular one and it would be kept on 24 hours a day. This is also the most expensive solution out of the 3 options.<\/li>\n\n\n\n
- Containerize the script and use it on the AWS Fargate service. Fargate is a serverless option, which is useful because the web scraper only needs to be executed by demand. Fargate is also cheaper than EC2.<\/li>\n\n\n\n
- Use AWS Lambda, which is also a serverless service that supports both raw code and containerized scripts. It has more limitations compared to Fargate, but it is enough in most cases. It’s the cheapest service and your script might even fit the free tier.<\/li>\n<\/ol>\n\n\n\n
Here, we will use AWS Lambda. Its main limitations are the timeout limit, which is 15 minutes and the deployment package can’t exceed 250 MB (but it accepts up to 10 GB using containers).<\/p>\n\n\n\n
The example script is very simple and takes less than a minute to execute, but as we previously mentioned, Selenium requires a browser, and the Chrome binary size is around 500 MB, which forces us to use the container approach.<\/p>\n\n\n\n
How are we going to deploy it to AWS Lambda?<\/strong><\/h2>\n\n\n\n
There are several ways to set up an AWS Lambda function. One easy way is by using the Serverless Framework<\/a>. The Serverless framework helps us to develop and deploy Lambda Functions by using a single YAML file to declare the lambda functions, their infrastructure, and the events that will trigger them.<\/p>\n\n\n\n
Using the Serverless Framework also allows us to deploy the lambda functions with a single command, simplifying the process a lot.<\/p>\n\n\n\n
The Serverless Framework also provides an optional dashboard which gives us an interface to check the function’s health, trigger events manually, and check their logs.<\/p>\n\n\n\n
The codebase:<\/strong><\/h3>\n\n\n\n
In this section, we will analyze some important files of a demo project which can be checked here<\/a>.<\/p>\n\n\n\n
serverless.yml<\/strong><\/a><\/p>\n\n\n\n
This is the file where we set the lambda application infrastructure, the lambda functions, and the events that are going to trigger them.<\/p>\n\n\n\n
In the provider<\/strong> section of this file, we declare that we are going to use a docker image named img<\/em>.<\/p>\n\n\n\n
The functions<\/strong> section is where we set the lambda functions and their specific configuration like environment variables and handler functions.<\/p>\n\n\n\n
Notice that here we add environment variables that will have the browser and its driver path, we state that we will use the image that was previously set, that the command that will be executed when the lambda function is triggered is the example.py <\/em>file handler, and that the event that will trigger it is a cronjob<\/em> that is scheduled for every 6 hours.<\/p>\n\n\n\n
Dockerfile<\/strong><\/a><\/p>\n\n\n\n
The Dockerfile is the template where we configure our container image. This file creates a Linux instance capable of running the web scraper. The template installs the project requirements (including Chrome and Chrome driver) and copies the required files to the image.<\/p>\n\n\n\n
Example.py<\/strong><\/a><\/p>\n\n\n\n
This file has a function that will be executed when the event is triggered. This file is pretty similar to the Selenium example we showed earlier.<\/p>\n\n\n\n
The main difference is that we customize the browser to not display an interface (because the lambda does not have a display) and to use a single process (because the lambda only has 1 CPU).<\/p>\n\n\n\n
In this case, the handler function returns a dictionary with a status code and a body just in case we want to change the event from a cronjob to an HTTP request.<\/p>\n\n\n\n
Deploying<\/strong><\/h3>\n\n\n\n
To deploy the lambda function we just need to run a single command on the terminal.<\/p>\n\n\n\n
![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/web-scrapper-aws-lambda-1200x275.jpg\")
<\/figure>\n\n\n\nIn this case, Serverless Frameworks raises a warning message that explains that the dashboard does not support functions that use container images. It means that we will not be able to check the lambda function logs, nor trigger the lambda function manually through the dashboard.<\/p>\n\n\n\n
But we can still do it through the AWS console. Below we have a screenshot of a successful log retrieved from AWS Cloudwatch:<\/p>\n\n\n\n
![\"\"](\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/web-scrapping-aws-1200x379.jpg\")
<\/figure>\n\n\n\nConclusions<\/strong><\/h2>\n\n\n\n
This is a very specific example of the usage of AWS Lambda and Selenium but I hope it can illustrate the potential of these technologies. Instead of creating a web scraper, we can create functions that run end-to-end tests that, in case of failure, send a Slack message, or we can create an API that calculates the distance between two strings and returns it in the HTTP response. It’s all up to your imagination!<\/p>\n\n\n\n
Resources<\/strong><\/h2>\n\n\n\n
Demo Project Repository:<\/p>\n\n\n\n
- \n
- https:\/\/github.com\/CheesecakeLabs\/selenium-serverless-example<\/a><\/li>\n<\/ul>\n\n\n\n
Content references:<\/p>\n\n\n\n
- \n
- https:\/\/aws.amazon.com\/blogs\/architecture\/serverless-architecture-for-a-web-scraping-solution\/<\/a><\/li>\n\n\n\n
- https:\/\/en.wikipedia.org\/wiki\/Data_scraping<\/a><\/li>\n\n\n\n
- https:\/\/en.wikipedia.org\/wiki\/Web_scraping<\/a><\/li>\n\n\n\n
- https:\/\/github.com\/umihico\/docker-selenium-lambda<\/a><\/li>\n\n\n\n
- https:\/\/dev.to\/awscommunity-asean\/creating-an-api-that-runs-selenium-via-aws-lambda-3ck3<\/a><\/li>\n\n\n\n
- https:\/\/docs.aws.amazon.com\/lambda\/latest\/dg\/gettingstarted-limits.html<\/a><\/li>\n\n\n\n
- https:\/\/www.docker.com\/resources\/what-container\/<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
Web scraping might save several hours when compared to manually collecting data from websites, and AWS Lambda is a good way to set up scripts to run by demand. In this blog post, we\u2019ll cover everything you need to know,\u00a0 the good and bad things about Web Scraping and AWS Lambda, and also analyze the […]<\/p>\n","protected":false},"author":81,"featured_media":11064,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[432],"tags":[305],"class_list":["post-11057","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-engineering","tag-tag-development"],"yoast_head":"\n
Setting up a Selenium Scraper on AWS Lambda<\/title>\n<meta name=\"description\" content=\"In this article, you'll discover more about Web Scrapping and learn step by step how to use Selenium to Web-Scrape on a AWS Lambda.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Setting up a Selenium Scraper on AWS Lambda\" \/>\n<meta property=\"og:description\" content=\"In this article, you'll discover more about Web Scrapping and learn step by step how to use Selenium to Web-Scrape on a AWS Lambda.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/\" \/>\n<meta property=\"og:site_name\" content=\"Cheesecake Labs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/cheesecakelabs\" \/>\n<meta property=\"article:published_time\" content=\"2023-01-12T20:42:29+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-07-31T18:59:15+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-web-scraping-aws-lambda.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1920\" \/>\n\t<meta property=\"og:image:height\" content=\"860\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Cheesecake Labs\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@cheesecakelabs\" \/>\n<meta name=\"twitter:site\" content=\"@cheesecakelabs\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"9 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/\"},\"author\":{\"name\":\"Karran Besen\"},\"headline\":\"How To Use Selenium To Web-Scrape on AWS Lambda\",\"datePublished\":\"2023-01-12T20:42:29+00:00\",\"dateModified\":\"2024-07-31T18:59:15+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/\"},\"wordCount\":1734,\"image\":{\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-web-scraping-aws-lambda.jpg\",\"keywords\":[\"development\"],\"articleSection\":[\"Engineering\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/\",\"url\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/\",\"name\":\"Setting up a Selenium Scraper on AWS Lambda\",\"isPartOf\":{\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-web-scraping-aws-lambda.jpg\",\"datePublished\":\"2023-01-12T20:42:29+00:00\",\"dateModified\":\"2024-07-31T18:59:15+00:00\",\"author\":{\"@type\":\"person\",\"name\":\"Karran Besen\"},\"description\":\"In this article, you'll discover more about Web Scrapping and learn step by step how to use Selenium to Web-Scrape on a AWS Lambda.\",\"breadcrumb\":{\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#primaryimage\",\"url\":\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-web-scraping-aws-lambda.jpg\",\"contentUrl\":\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-web-scraping-aws-lambda.jpg\",\"width\":1920,\"height\":860,\"caption\":\"woman programming in front of a pc\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/cheesecakelabs.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How To Use Selenium To Web-Scrape on AWS Lambda\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/#website\",\"url\":\"https:\/\/cheesecakelabs.com\/blog\/\",\"name\":\"Cheesecake Labs\",\"description\":\"Nearshore outsourcing company for Web and Mobile design and engineering services, and staff augmentation for startups and enterprises..\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/cheesecakelabs.com\/blog\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"name\":\"Karran Besen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/cheesecakelabs.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2019\/12\/karran-300x300.png\",\"contentUrl\":\"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2019\/12\/karran-300x300.png\",\"caption\":\"Karran Besen\"},\"url\":\"https:\/\/cheesecakelabs.com\/blog\/autor\/karran-besen\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Setting up a Selenium Scraper on AWS Lambda","description":"In this article, you'll discover more about Web Scrapping and learn step by step how to use Selenium to Web-Scrape on a AWS Lambda.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/","og_locale":"en_US","og_type":"article","og_title":"Setting up a Selenium Scraper on AWS Lambda","og_description":"In this article, you'll discover more about Web Scrapping and learn step by step how to use Selenium to Web-Scrape on a AWS Lambda.","og_url":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/","og_site_name":"Cheesecake Labs","article_publisher":"https:\/\/www.facebook.com\/cheesecakelabs","article_published_time":"2023-01-12T20:42:29+00:00","article_modified_time":"2024-07-31T18:59:15+00:00","og_image":[{"width":1920,"height":860,"url":"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-web-scraping-aws-lambda.jpg","type":"image\/jpeg"}],"author":"Cheesecake Labs","twitter_card":"summary_large_image","twitter_creator":"@cheesecakelabs","twitter_site":"@cheesecakelabs","twitter_misc":{"Written by":null,"Est. reading time":"9 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#article","isPartOf":{"@id":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/"},"author":{"name":"Karran Besen"},"headline":"How To Use Selenium To Web-Scrape on AWS Lambda","datePublished":"2023-01-12T20:42:29+00:00","dateModified":"2024-07-31T18:59:15+00:00","mainEntityOfPage":{"@id":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/"},"wordCount":1734,"image":{"@id":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#primaryimage"},"thumbnailUrl":"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-web-scraping-aws-lambda.jpg","keywords":["development"],"articleSection":["Engineering"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/","url":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/","name":"Setting up a Selenium Scraper on AWS Lambda","isPartOf":{"@id":"https:\/\/cheesecakelabs.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#primaryimage"},"image":{"@id":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#primaryimage"},"thumbnailUrl":"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-web-scraping-aws-lambda.jpg","datePublished":"2023-01-12T20:42:29+00:00","dateModified":"2024-07-31T18:59:15+00:00","author":{"@type":"person","name":"Karran Besen"},"description":"In this article, you'll discover more about Web Scrapping and learn step by step how to use Selenium to Web-Scrape on a AWS Lambda.","breadcrumb":{"@id":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#primaryimage","url":"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-web-scraping-aws-lambda.jpg","contentUrl":"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2023\/01\/selenium-web-scraping-aws-lambda.jpg","width":1920,"height":860,"caption":"woman programming in front of a pc"},{"@type":"BreadcrumbList","@id":"https:\/\/cheesecakelabs.com\/blog\/selenium-scraper-aws-lambda\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/cheesecakelabs.com\/blog\/"},{"@type":"ListItem","position":2,"name":"How To Use Selenium To Web-Scrape on AWS Lambda"}]},{"@type":"WebSite","@id":"https:\/\/cheesecakelabs.com\/blog\/#website","url":"https:\/\/cheesecakelabs.com\/blog\/","name":"Cheesecake Labs","description":"Nearshore outsourcing company for Web and Mobile design and engineering services, and staff augmentation for startups and enterprises..","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/cheesecakelabs.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Person","name":"Karran Besen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/cheesecakelabs.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2019\/12\/karran-300x300.png","contentUrl":"https:\/\/ckl-website-static.s3.amazonaws.com\/wp-content\/uploads\/2019\/12\/karran-300x300.png","caption":"Karran Besen"},"url":"https:\/\/cheesecakelabs.com\/blog\/autor\/karran-besen\/"}]}},"_links":{"self":[{"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/posts\/11057","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/users\/81"}],"replies":[{"embeddable":true,"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/comments?post=11057"}],"version-history":[{"count":5,"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/posts\/11057\/revisions"}],"predecessor-version":[{"id":12202,"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/posts\/11057\/revisions\/12202"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/media\/11064"}],"wp:attachment":[{"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/media?parent=11057"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/categories?post=11057"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/cheesecakelabs.com\/blog\/wp-json\/wp\/v2\/tags?post=11057"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}} - https:\/\/en.wikipedia.org\/wiki\/Data_scraping<\/a><\/li>\n\n\n\n
- https:\/\/aws.amazon.com\/blogs\/architecture\/serverless-architecture-for-a-web-scraping-solution\/<\/a><\/li>\n\n\n\n
- https:\/\/github.com\/CheesecakeLabs\/selenium-serverless-example<\/a><\/li>\n<\/ul>\n\n\n\n
- DOM Parsing<\/strong><\/li>\n<\/ul>\n\n\n\n
- HTML Parsing<\/strong><\/li>\n<\/ul>\n\n\n\n
- Text Pattern Matching<\/strong><\/li>\n<\/ul>\n\n\n\n
- Deploying<\/a><\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Web scraping techniques<\/a><\/li>\n<\/ul>\n<\/li>\n\n\n\n
- Data Scraping<\/a><\/li>\n\n\n\n