Add AbuseIPDB reporting actionban script for Fail2Ban#3948
Conversation
- Clarified integration steps for placing the script in 'action.d' - Updated example configuration with correct script name 'fail2ban-abuseipdb.sh' - Emphasized the need to set 'abuseipdb_apikey' in the config file - Improved formatting and readability for better user understanding No functional changes, just documentation updates.
- Added an override configuration to enhance Fail2Ban’s AbuseIPDB integration. - Introduced a local banned IP list for better isolation from Fail2Ban. - Optimized API calls (`/v2/check` → `/v2/report`) to reduce redundant reports. - Ensured `norestored=1` handling to prevent re-reporting after restarts. - Improved logging and added custom comments to avoid sensitive data exposure. This override provides more control, efficiency, and security while maintaining compatibility with the main configuration.
- Reorganized script to be used by both 'actionstart' and 'actionban' in 'abuseipdb.local' - Isolated heavy 'actionstart' tasks using nohup to prevent latency - Removed redundant API checks to improve performance and reduce overhead - Implemented a lock mechanism to prevent 'actionban' execution if 'actionstart' fails - Ensured 'actionban' does not run at runtime due to missing dependencies or permission issues
- Replace local file storage with AbuseIPDB SQLite database. - Offload heavy tasks to background to avoid latency during concurrent actionban calls. - Add global lock to ensure actionstart runs only once across all jails.
- Replace local file storage with AbuseIPDB SQLite database. - Add info about preventing leaking sensitive information on reports
- rename fail2ban_abuseipdb.sh --> fail2ban-abuseipdb.sh - update descriptions
…ndling This refactors the AbuseIPDB integration for Fail2Ban with major improvements: - Introduced separate lock files (LOCK_INIT, LOCK_BAN, LOCK_DONE) to better handle concurrent initialization and prevent race conditions during restarts. - LOCK_BAN → serializes ban reports to the API (during actionban). - LOCK_DONE → can signal completion or be used for future synchronization (like restart-safe exits). - LOCK_INIT with flock in actionstart to prevent concurrent initialization, ensuring SQLite and log file integrity during parallel Fail2Ban restarts or multiple jail startups. - Enhanced argument validation for both actionstart and actionban to prevent silent failures. - Improved database initialization checks, ensuring proper creation of directories and log files. - Added persistent SQLite pragmas for performance optimization under concurrent access. - Refined error handling and logging for API interactions, including better detection of rate-limiting (HTTP 429) and invalid responses. - Implemented consistent whitespace trimming and sanitization on IP addresses and bantime inputs. - Improved modularity with dedicated helper functions, reducing code duplication and improving maintainability. - Ensured background execution with better log redirection and failure tracking. - Verify local DB insertions, aborting the process on failure to prevent incomplete or invalid state. - Roll back local DB entries if AbuseIPDB reporting fails, ensuring no orphaned records remain. - Replace basic info logs with clear status and error messages to improve traceability and debugging. - Maintain high integrity between the local database and AbuseIPDB by only proceeding when all previous steps succeed. - Shift from a "continue regardless" flow to a controlled stop on any critical error, ensuring system reliability. Previously, the script assumed success of key steps, risking stale database entries, silent API call failures, and duplicate reports after Fail2Ban restarts. These changes improve reliability, prevent data corruption under high concurrency, and ensure accurate synchronization between local db and AbuseIPDB API.
- Swapped returns for echoes — Bash, not PHP
|
@sebres I've made several commits to the PR. After working with 'actionstart' and 'actionban' through single bash scripting, I've realized it's not a easy journey, but I enjoyed the challenge of handling high concurrency and optimizing latency. I've frozen the code for now and will test it in production. @orlitzky when you have a moment, could you take a quick look at the script? I’d love to get your thoughts on how it handles high concurrency. And of course, if there’s a bash script involved, a Gentoo dev has to be there! 😄🐧🔧 |
| ###################### | ||
| # Notes.: Uncomment and leave as-is | ||
| # actionban = /etc/fail2ban/action.d/fail2ban-abuseipdb.sh \ | ||
| # "<abuseipdb_apikey>" "<matches>" "<ip>" "<abuseipdb_category>" "<bantime>" "<restored>" "<BYPASS_FAIL2BAN>" "<SQLITE_DB>" "<LOG_FILE>" |
There was a problem hiding this comment.
This will leak the API key in the ps output which could be an issue on a web server with SSH users. The usual workaround is to pass the secret in the environment but I don't know off the top of my head how to do that here.
There was a problem hiding this comment.
Well, it is questionable (at least if used in fail2ban) - it is running as root, so basically possible with ps & co for root or sudoers only.
But they are anyway able to find the key in fail2ban configuration files, so it doesn't matter.
Regarding the env-vars, fail2ban can supply arguments as variables to shell commands of action, but at the moment only if the argument contains some special character. At the moment, there is no possibility to force it for plain arguments (what, I think, the api-key is).
Maybe we shall indeed implement some option, to enforce then vars usage.
Alternatively, one could set the var in env of the systemd-unit and use that var here, but... it is like using a sledge-hammer to crack a nut, in my opinion for abovementioned reason.
There was a problem hiding this comment.
Well, it is questionable (at least if used in fail2ban) - it is running as root, so basically possible with ps & co for root or sudoers only.
The full process list is "usually" still available to all users. (I use quotes because I don't know what every distro is doing.) On linux, you can mount /proc with hidepid=1 to prevent other users from seeing root processes, but it's not the kernel default. My regular user can see the argument to all root processes, e.g.
$ ps aux | grep root
...
root 43667 0.2 0.0 106168 55616 pts/4 S+ 08:22 0:00 emacs --no-window-system hello-world.txt
| echo "$(date +"%Y-%m-%d %H:%M:%S") - ${message}" >> "${LOG_FILE}" | ||
| } | ||
|
|
||
| LOCK_INIT="/tmp/abuseipdb_actionstart_init.lock" |
There was a problem hiding this comment.
Should use mktemp() here to avoid malicious users hijacking these paths.
|
How much concurrency do you expect? :) I would guess that the one-insert-at-a-time model is what's going to slow it down, if anything. And TBH bash is not a great choice if you're really worried about performance. Python is already a given with fail2ban, so that would be my recommendation if bash is slowing you down. There are some other advantages to this:
But it really depends on how serious you want to get about performance. |
|
@orlitzky I appreciate your review. Actually, concurrency varies based on the attack surface, as the PR attempts to improve the third-party reporting workflow with shell scripting while maintaining its own local database, which triggers with each ban action. On a heavily attacked IP blocks, this can occur hundreds of times and more. On my IP block, I typically see around 20-30 ban actions per minute, which is relatively low, so I haven't had the chance to test PR in a very high-concurrency scenario. While the PR handles current ban load, based on your considerations, bash scripting or general skeleton may not be the best approach here. @sebres, I'm interested in this aspect of Fail2Ban (third-party reporting) because I believe that reporting to third parties is key in the fight against 🖤🎩 - So, I would appreciate the opportunity to contribute to better handling and deeper integration of third-party reporting in your roadmap. |
|
@hsntgm - Some thoughts on your AbuseIPDB solution. Firstly many thanks for your scripts which have provided me with an excellent starting point for my needs. The sqlite3 database records the IP address and bantime. The database is not maintained and tidied? Fail2Ban will record the time of the ban and the banned time and then when the bantime expires will UnBan the IP. The AbuseIPDB sqlite3 database doesn't get updated so if the IP then gets rebanned the AbuseIPDB script will say the IP is already in the database so will not report it again. It would be appropriate to report the abuse again. This case isn't possible in the current design of the script. Whilst the script will be called with the IP address again (when IP is rebanned), it will find the IP in its database and check if it still listed on AbuseIPDB and will skip reporting it. With reasonable bantimes it seems entirely appropriate to report it again. One might want to prevent repeat reports within a configurable timescale (say once per day or once per bantime interval). If the time of the ban was also recorded in the database then the script could, if it finds it already has the IP address, then check if current time exceeds time of ban + bantime and then report it again and update the database with new values. One could also run a command at --actionstart to delete any IP's where time of ban + bantime is < current time to provide a level of automatic tidying of the database. I have used your scripts as the basis of my reporting to AbuseIPDB with some modifications to suit my needs. Alongside this I have used MQTT to replicate bans/unbans across distributed servers which also required me to consider the effect on the scripts in particular to add to my calls to the script to detect Bans issued with fail2ban-client which differ from those internal to Fail2Ban which will have a non-zero . I hope these comments are helpful. |
Description:
Introduces a new actionban script for Fail2Ban to report
banned IP addresses to AbuseIPDB, with support for custom comments.
It prevents duplicate reporting by maintaining a local list of already
reported IPs and ensures that the IP is not reported again if it has
been previously reported and is still listed in AbuseIPDB.
Why we need?
comment for each ban to hide sensetive data.
redundant reporting and to ensure the IP is not reported multiple
times.
This enhances the integration with AbuseIPDB while ensuring
that Fail2Ban reporting remains efficient and effective, avoiding
API rate limits by reducing duplicate entries.
Fixes:
Checklist:
masteras the base branch for a new feature.failregex(if applicable).