How I used Scrapy and Scrapy Playwright to take screenshot

I used scrapy, scrapy-playwright to do the job. By using Playwright, I could make the final page look right without having to do all the extra work by hand.

I recently had a task where I needed to take a screenshot of a product details page after searching for a product number. At first, it sounded simple, but it turned out to be quite a process.

First, I had to enter the product number into the website’s search box. Before I could get any results, the website asked me to solve a CAPTCHA. This is a little test to make sure I was not a robot. To get past this, I used a service that solves CAPTCHAs for you.

For the next part, I used a tool called Scrapy. Scrapy helps you collect information from websites. It handled all the requests I needed to make to the website and took care of the cookies. Cookies are small pieces of data that the website uses to keep track of what you are doing.

Managing these cookies was important because they allowed me to stay logged in and get to the right page with the product details. Scrapy did a good job handling all of this, which made things easier.

When it was time to take the screenshot, I tried to do it directly from the final page. However, this was not easy because the page needed some extra work to look right. I would have had to add more CSS and other stuff manually, which was a lot of hassle.

To make things easier, I decided to use another tool called Playwright along with Scrapy. Playwright is good at making websites look like they do in a real browser. By using Playwright, I could make the final page look right without having to do all the extra work by hand.

There was another twist: the final page needed data from two different APIs to show all the details. I had to call these APIs first to get the data before using Playwright to make the final page.

In the end, Scrapy handled all the cookie management and API requests, which allowed me to use Playwright only when I needed to. This made it much easier to get the screenshot of the product details page.

Using the wait_for_load_state(‘networkidle’) I was able to wait and get the full request and took the screenshot

Overall, using Scrapy and Playwright together helped me complete the task without much manual work, and I was able to get the screenshot I needed.