Repository navigation
Dismiss cookie permission, GDPR, and subscription popups before archiving #175
Description
Activity
This is blocked by:
- Switch all dependencies to pure python and release ArchiveBox pip package #177 (we need pyppeteer to run user scripts to dismiss popups)
- Add ability to run JS scripts during archiving with Playwright/Puppeteer #51 support for user scripted puppeteer flows
Useful scripts for evading bot detection in the future: https://github.com/niespodd/browser-fingerprinting#technical-insights-into-bypassing-bot-detection
Hi,
I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue.
Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up!
The test I choose was artnet.com, it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio:
In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, https://github.com/browser-use/browser-use have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control.
Hi,
I am curious if there's any interest in the community to incorporate AI/LLM to help solve this issue.
Recently I was working on a simple web scraper / retriever for a LLM project and came across the need to dismiss not only cookie consent but also newsletter subscription pop-ups. After some research on web UI element detection computer vision models and failed POCs, I decided to give LLM a try. I tested in Google AI studio using Gemini 2.0 Flash model; to my surprise, it appears Gemini 2.0 not only is able to detect if a pop-up (or cookie consent banner) exists on the screen, it is also able to correctly identify the XPath to close the pop-up!
The test I choose was artnet.com, it consisted of a newsletter subscription pop-up with a light gray "X" button to dismiss. Due to the light color, most web UI element CV detection models either ignored the button (to close the pop-up) completely or gave it a very low score of 30% confidence level; however, Gemini 2.0 had no problem identifying the button based on the same screen shot. Moreover when I provided it with the corresponding HTML, it was able to correctly identify the XPath which we can use in Playwright/Puppeteer/Selenium to send the click event. Following are the screenshots from one of my tests in Google AI Studio:
In my limited tests, it took 50-70 seconds to "think" (I ran it a couple times across 2 different days and both tests took >>50 seconds to process). I am sure more testing and optimization can be done to improve on this if there's enough interests to incorporate this feature into the project. Also for reference, https://github.com/browser-use/browser-use have incorporated similar technique to close pop-up dialogs; though, it appears that they highlighted each UI controls by adding colored labels and borders before taking a screen shot as an input for the LLM instead relying on LLM to identify the XPath to a specific control.
This has major privacy implications, maybe optionally this could be fine, but I think a local AI model would suit this purpose better if you want to involve AI at all.
Personally I use Consent-o-Matic in my browsers and it not only auto okays all cookies, it even accepts or denies based on what the site lets you control and what you desire. For example allowing functional cookies for things like embeds but keeping trackers out.
It's truly set and forget, rarely misses and could be a fantastic consideration to source from. It even clicks your choices in a hidden manner.
Reacted by datoslabsYes for paid clients I'm already using a mixture of local LLMs and hosted models to detect and solve CAPTCHAs, hide cookie popups, and do QA analysis of captures to determine if there's any obsctructions to seeing the content.
None of it is open source right now because it's the only thing I can charge for to sustain the rest of the project.
Reacted by Glassed Silver and Mela Eckenfelsok this is implemented on
devusingarchivebox/plugins/modalcloserandarchivebox/plugins/istilldontcareaboutcookies, it's a simple approach for now but we can extend it in the future with something like stagehand/browser-use if needed.Reacted by Glassed Silverok this is implemented on
devusingarchivebox/plugins/modalcloserandarchivebox/plugins/istilldontcareaboutcookies, it's a simple approach for now but we can extend it in the future with something like stagehand/browser-use if needed.This is great news and makes me very happy to see!
Very thankful for this, just one note that if anyone else extends on this in the future or maybe you yourself might I suggest using Consent-o-Matic?I have tested both extensively and unfortunately consent-o-matic breaks a lot of browser automations and makes the headless browsers flakyer in general (e.g. sometimes it'll open a new features/config page on first browser launch that steals foreground tab focus away from the page we're trying to archive). I've stuck with
istilldontcareaboutcookiesfor now but if there is enough demand we could add content-o-matic as an opt-in choice as well.
Might be able to graft some code from: https://www.i-dont-care-about-cookies.eu for hiding modals and overlays.
Also:
https://greasyfork.org/de/scripts/34815-anti-adblock-killer-reek-