Script to delete traces of selenium in the browser

N8N Ultimate Scraper - Workflow

This workflow's objective is to collect data from any website page, whether it requires login or not.

For example, you can collect the number of stars of the n8n-ultimate-scraper project on GitHub.

Requirements

Selenium Container: Selenium is an open-source automation framework for web applications, enabling browser control and interaction through scripts in various programming languages. You can deploy the Docker Compose file from the associated GitHub project to set up your Selenium container and configuration: https://github.com/Touxan/n8n-ultimate-scraper

Residential Proxy Server: To scrape data at scale without being blocked, I personally recommend GeoNode. They offer affordable, high-quality residential proxies: https://geonode.com/invite/98895

OpenAI API Key: For using GPT-4.

Optional

Session Cookies Collection: To use login functionality with the n8n Ultimate Scraper, you need to collect session cookies from the target website. You can do this using the extension created for this application in the GitHub project: https://github.com/Touxan/n8n-ultimate-scraper. Follow the installation procedure to use it.

How to use

Deploy the project with all the requiremnts and request your webhook.

Example of request: curl -X POST http://localhost:5678/webhook-test/yourwebhookid
-H "Content-Type: application/json"
-d '{ "subject": "Hugging Face", "Url": "github.com", "Target data": [ { "DataName": "Followers", "description": "The number of followers of the GitHub page" }, { "DataName": "Total Stars", "description": "The total numbers of stars on the different repos" } ], "cookies": [] }'

You can also scrape link like this : curl -X POST http://localhost:5678/webhook-test/67d77918-2d5b-48c1-ae73-2004b32125f0
-H "Content-Type: application/json"
-d '{ "Target Url": "https://github.com", "Target data": [ { "DataName": "Followers", "description": "The number of followers of the GitHub page" }, { "DataName": "Total Stars", "description": "The total numbers of stars on the different repo" } ] }'

Note The maximum nimber of Target data is 5.

Debug IP

This small debug flow aims to check the IP you're requesting with, in case you're using a proxy

Search

Description : This part aims to search on Google for the subject and find the URL of the subject page based on the input URL.

Selenium Session

Description: Creation and configuration of the Selenium session.

Scrape with cookies session

Description This part goes to the extracted URL, injects the cookies passed into the webhook, takes a screenshot of the webpage, and analyzes the image with GPT to extract the targeted data.

Scrape without cookies session

Description Same as the 'Scrape with cookies session' flow, but without the cookie injection

Proxy

Configuration

To configure your proxy with the project, follow the instructions on the GitHub project: https://github.com/Touxan/n8n-ultimate-scraper. To configure the docker-compose, you also need to add this argument to the 'Create Selenium Session' node : --proxy-server=address:port.

⚠️Warning⚠️

Selenium does not support proxy authentication, so you need to add your server IP to the proxy whitelist. On GeoNode, it's here: https://app.geonode.com/whitelist-ip!

Selenium Ultimate Scraper Workflow

Purpose and structure

This folder contains the Selenium Ultimate Scraper Workflow workflow. Its executable nodes use code, convertToFile, html, httpRequest, if, informationExtractor, limit, lmChatOpenAi, openAi, respondToWebhook, set, webhook. The table and connection map below are derived from the JSON, not an inferred guarantee of behavior.

Setup

Import inactive. Configure your own credentials and replace REPLACE_WITH_* values. Review external destinations and Code nodes before running.

Entry points

Webhook

Verification

Sanitized source candidate; not runtime-tested. This overview is generated from node metadata. Read the folder README for the connection map, credential types, and license.

Attribution

https://github.com/Zie619/n8n-workflows/blob/94007c1445d9258a7da116646b79473e7c7c3282/workflows/Limit/1900_Limit_Code_Automate_Webhook.json MIT; preserve source notices.