Skip to content

TUVIMEN/rule34-scraper

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

18 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

rule34-scraper

A scraper for rule34.

output example

{
  "image": "https://rule34.xxx//samples/4367/sample_0074ad420f060240b2ba8400be719e2a.jpg?5011104",
  "original": "https://wimg.rule34.xxx//images/4367/0074ad420f060240b2ba8400be719e2a.jpeg?5011104",
  "id": 5011104,
  "date": "2021-08-21T01:35:05",
  "uploader_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=%21nner%24elf89",
  "uploader": "!nner$elf89",
  "rating": "Questionable",
  "sources": [
    "https://twitter.com/AhegaoKami/status/1423765981139062787?s=20"
  ],
  "sizex": 2233,
  "sizey": 1733,
  "score": 482,
  "copyright": [
    "original"
  ],
  "character": [
    "spooky (ahegaokami)"
  ],
  "artist": [
    "ahegaokami"
  ],
  "general": [
    "1girls",
    "ahe gao",
    "bed",
    "black hair",
    "blue eyes",
    "cleavage",
    "fangs",
    "female",
    "female only",
    "happy",
    "heart",
    "heart-shaped pupils",
    "i love you",
    "long hair",
    "looking at viewer",
    "lying",
    "lying on bed",
    "lying on side",
    "narrowed eyes",
    "oni",
    "oni horns",
    "open mouth",
    "pale skin",
    "pale-skinned female",
    "pov",
    "pov eye contact",
    "shirt",
    "smile",
    "solo",
    "solo female",
    "talking to viewer",
    "wholesome"
  ],
  "metadata": [
    "dialogue",
    "english text",
    "hi res",
    "speech bubble"
  ],
  "comments_count": 53,
  "url": "https://rule34.xxx/index.php?page=post&s=view&id=5011104",
  "comments": [
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 6271896,
      "date": "2021-08-21T02:01:23",
      "score": 264,
      "text": "Damnit Anon, You Snapped me back to reality."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 6272048,
      "date": "2021-08-21T02:25:58",
      "score": 319,
      "text": "Man I came here to jack off not cry"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=rpzamer",
      "user": "rpzamer",
      "id": 6272063,
      "date": "2021-08-21T02:28:20",
      "score": 250,
      "text": "this is what i think we all truly want. guys girls, matters not. just this kind of love. its why it resonates with us so much"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=JustADickSlayingThots",
      "user": "JustADickSlayingThots",
      "id": 6273441,
      "date": "2021-08-21T06:02:57",
      "score": 97,
      "text": "she's just so cute on wholesome.."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=BWC_For_Interracial_RP",
      "user": "BWC_For_Interracial_RP",
      "id": 6273545,
      "date": "2021-08-21T06:16:10",
      "score": 138,
      "text": "People really underestimate how much people value someone who makes them feel safe and valued."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=CuddlyFuta",
      "user": "CuddlyFuta",
      "id": 6275032,
      "date": "2021-08-21T09:33:52",
      "score": 7,
      "text": "\" I-I... Thank you for being here for me Spooky, you really don't know how much it means to me you've stayed with me for so long despite my work schedule... I-I would love to cuddle with you all day...\""
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=CuddlyFuta",
      "user": "CuddlyFuta",
      "id": 6275049,
      "date": "2021-08-21T09:36:16",
      "score": 67,
      "text": "This actually made me tear up though I'm not gonna lie..."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=TradingSoulForDommyMommies",
      "user": "TradingSoulForDommyMommies",
      "id": 6278769,
      "date": "2021-08-21T20:46:13",
      "score": 101,
      "text": "It is really heartwarming, I don't blame you. Truly peak degeneracy: genuine craving for affection"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Wholesome_Girlfriend",
      "user": "Wholesome_Girlfriend",
      "id": 6297875,
      "date": "2021-08-24T00:33:08",
      "score": 58,
      "text": "This has made me smile like a complete idiot.. this so unbelievably adorable and I love it so darn much! Everyone deserves this kinda love! I will lovingly support my future significant other just like this!"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Attention_whore",
      "user": "Attention_whore",
      "id": 6307047,
      "date": "2021-08-25T02:06:38",
      "score": 30,
      "text": "I want someone to cuddle with me in bed.... :("
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=LittleRedVixen",
      "user": "LittleRedVixen",
      "id": 6309843,
      "date": "2021-08-25T08:23:01",
      "score": 6,
      "text": "Spooky is best girl and I REFUSE to accept anything else <3"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=SissyFemboyOpenRP",
      "user": "SissyFemboyOpenRP",
      "id": 6310140,
      "date": "2021-08-25T09:07:24",
      "score": 8,
      "text": "I'd love to find someone like that someday. I wonder if that's just too much of a fantasy"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=LittleRedVixen",
      "user": "LittleRedVixen",
      "id": 6310871,
      "date": "2021-08-25T11:30:19",
      "score": 24,
      "text": "^ Don't rush it, focus on learning to love yourself 100% before you look for a relationship, and know what you can compromise on and what you can't compromise! That'll get you off to a good start :-)"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 6310877,
      "date": "2021-08-25T11:32:11",
      "score": 22,
      "text": "^ A sudden minute of wholesomeness and wisdom on r34"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=The_Demonstrations_Man_from_Team_Fortress_2",
      "user": "The_Demonstrations_Man_from_Team_Fortress_2",
      "id": 6318941,
      "date": "2021-08-26T09:52:26",
      "score": 12,
      "text": "Fuck. I've been in almost 20 relationships, the shortest one being 3 months, and almost every single one I've been cheated on. I don't know what I'm doing wrong, and stuff like this makes me feel better and worse. I thought porn would make me feel better, but it doesn't."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Welpthisismylifenow",
      "user": "Welpthisismylifenow",
      "id": 6335046,
      "date": "2021-08-28T07:33:50",
      "score": 5,
      "text": "Ayo holy shit this amount of wholesome is too much"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 6343575,
      "date": "2021-08-29T06:16:51",
      "score": 8,
      "text": "Only love is with us now,  something warm and pure. We will find the peace within ourselves; no need for a cure."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 6346995,
      "date": "2021-08-29T16:42:05",
      "score": 7,
      "text": "Right in the feelings man ;("
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 6378270,
      "date": "2021-09-02T03:52:45",
      "score": 7,
      "text": "what am i doing wrong man"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=SubSlutERPOpen",
      "user": "SubSlutERPOpen",
      "id": 6391315,
      "date": "2021-09-03T18:25:03",
      "score": 3,
      "text": "This hits the soft spot in my heart aaaa,,"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 6460861,
      "date": "2021-09-11T11:52:36",
      "score": 8,
      "text": "mfw when no oni gf"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Zepiel",
      "user": "Zepiel",
      "id": 6510074,
      "date": "2021-09-17T03:09:59",
      "score": 6,
      "text": "Things like these make my heart Hurts sometimes, is It normal ?"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=LadyTetra",
      "user": "LadyTetra",
      "id": 6536649,
      "date": "2021-09-20T00:16:08",
      "score": 11,
      "text": "NO! DON'T GIVE ME HOPE!"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 6622203,
      "date": "2021-09-29T06:54:33",
      "score": 2,
      "text": "Calling John the Fisherman"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Onra",
      "user": "Onra",
      "id": 6820226,
      "date": "2021-10-21T12:28:45",
      "score": 5,
      "text": "What saddens me is how little those words means to me nowadays. Not that they are bad, but that it sounds like a bunch of lies just to trick me Still, wish someone was there and said those things."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=SluttySpooky",
      "user": "SluttySpooky",
      "id": 6830497,
      "date": "2021-10-22T17:16:49",
      "score": 0,
      "text": "\"We can do aaaaanything you want sweetie~.. I'll be here with you for as long as you need me to~\""
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Succubus_Queen_Maria",
      "user": "Succubus_Queen_Maria",
      "id": 7034219,
      "date": "2021-11-13T06:46:24",
      "score": 0,
      "text": "I'm always here for you if you need anything! A hug or cuddle,or if you want to talk about stuff!"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=RoleplayersPleaseGoOutside",
      "user": "RoleplayersPleaseGoOutside",
      "id": 7083195,
      "date": "2021-11-18T12:07:57",
      "score": 4,
      "text": "i wish my so called \"girlfriend\" was as sweet as her"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=OmniMeat",
      "user": "OmniMeat",
      "id": 7083239,
      "date": "2021-11-18T12:17:42",
      "score": 6,
      "text": ". . . . Despair . . No more horny"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 7108988,
      "date": "2021-11-21T08:32:48",
      "score": 13,
      "text": "I do not wish to be horny anymore...I just want to be happy."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 7346265,
      "date": "2021-12-15T22:03:13",
      "score": 2,
      "text": "she's so cute i wanna hug her so bad"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 7522709,
      "date": "2022-01-01T13:10:01",
      "score": 3,
      "text": "I hardly ever get attracted to fictional characters, but Spooky is an acception. Everything I see about this character makes me want to cuddle her so much, I want to hug and kiss and just be around someone like this in real life. It's really nice to find something that caters to myself like this, and I'm hopeful I can find someone like this."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 7568939,
      "date": "2022-01-05T12:25:15",
      "score": 2,
      "text": "Same anon as #7522709, I recently got really sad and lonely about the state of my love life, but by talking with a close friend I was able to recover from it and I'm as hopeful as ever, my friends have so much faith for me and I can finally say that I have as much faith in myself as they have in me. I know this is a really unorthodox way if sharing my story, but after I made the previous post, I felt it was right to give an update"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 7736864,
      "date": "2022-01-20T22:23:57",
      "score": 6,
      "text": "Same here man I turned 20 on the 3rd of this month haven't been involved with anyone since middle school, highschool was fun but I still wish I had someone. Currently getting ready to leave for the army, got airborne on my contract Relationships don't work anymore its just hookup & fuck. The world is getting darker and darker everyday and I just want someone to help me emotionally through these dark times guess I'll just stay lonely forever and push harder too be a Delta operator"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 7754021,
      "date": "2022-01-22T10:50:26",
      "score": 6,
      "text": "loneliness hurts doesn't it"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=arLynn",
      "user": "arLynn",
      "id": 7885674,
      "date": "2022-02-02T23:18:55",
      "score": 5,
      "text": "I just want to be loved..."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=BoiWantsLove",
      "user": "BoiWantsLove",
      "id": 7953360,
      "date": "2022-02-08T19:51:42",
      "score": 1,
      "text": "All we want as humans is genuine love dats why things like this are created and loved by many :3"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 7978149,
      "date": "2022-02-10T23:14:47",
      "score": 2,
      "text": "Why does the wholesome tag always make me sad about myself.I came to masturbate, not cry."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Miguel16310",
      "user": "Miguel16310",
      "id": 8091043,
      "date": "2022-02-20T23:43:19",
      "score": 2,
      "text": "Wholesomeness is underrated. There's never enought love in R34."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 8272180,
      "date": "2022-03-08T08:37:13",
      "score": 0,
      "text": "WHEN THE WIND IS SLOW AND THE FIRE'S HOT"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 8371034,
      "date": "2022-03-16T06:57:39",
      "score": 1,
      "text": "My soul hurts."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 8588299,
      "date": "2022-04-01T21:59:40",
      "score": 2,
      "text": "as 6272063 said, this is why theres so many depressing comments on wholesome porn images/videos, we just crave affection rather than sex"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 9047098,
      "date": "2022-05-04T22:31:21",
      "score": 3,
      "text": "Snap back to reality..."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 9163103,
      "date": "2022-05-13T00:30:07",
      "score": 3,
      "text": "I was hear to finger myself but now im gonna go cuddle with my boyfriend"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Anonymous",
      "user": "Anonymous",
      "id": 9207810,
      "date": "2022-05-16T01:36:49",
      "score": 1,
      "text": "Step 1: admire drawingStep 2: Imagine this scenarioStep 3: Fantasize about having someone this caringStep 4: remember that you are aloneStep 5: small depression"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=GamerMan42",
      "user": "GamerMan42",
      "id": 9708250,
      "date": "2022-06-18T23:42:12",
      "score": 4,
      "text": "breaking the character of gamerman42, This made me tear up...I haven't seen my Girlfriend since the start of covid, and I really miss her. I'm gonna go ahead and text her and let her know I'm thinking about her right now."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=IJWBHNH",
      "user": "IJWBHNH",
      "id": 10502289,
      "date": "2022-08-11T04:06:04",
      "score": 3,
      "text": "This is what we all need. So I'll give you this ramble.I have a dream, that one day where every person on the internet will have genuine happiness. The internet of the TRULY happy, dammit. The internet oh happiness, not chaos, run by wholesome, not Twitter! Where the internet changes to suit the wholesome, not the other way around. Where happiness and wholesome are back in the hands of the people! Where every person is free to smile--to live--to love for them self! Fuck these limp-dick degenerates, fuck these chickenshit websites. Fuck this 24/7 internet spew of hentai and pornstar bullshit! Fuck being horny! Fuck the media! FUCK ALL OF IT! The internet is diseased. Rotten to the core. There's no saving it, we need to delete it by the roots. Wipe the internet clean. BURN IT DOWN! And from the ashes, a new internet will be born, happy, but UNHORNY! The horny will be purged and the wholesome will thrive- free to live as they see fit, they will make the internet great again! . . . In my new internet, people will happy and wholesome for what they WISH! Not for horny. Not for porn! Not for what they are told feels good. Every person will be free to MAKE THEIR OWN HAPPINESS!"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Dunglebamble",
      "user": "Dunglebamble",
      "id": 11541606,
      "date": "2022-11-14T01:56:39",
      "score": 1,
      "text": "If you ever feel sad remember there's a song named baka mitai can I put my balls in yo jaws"
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=buttersmooth",
      "user": "buttersmooth",
      "id": 12985758,
      "date": "2023-03-05T20:47:54",
      "score": 0,
      "text": "I've gotta stop looking at these. It's too wholesome... But then, sometimes that's better than horny."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Tulachica123",
      "user": "Tulachica123",
      "id": 14637973,
      "date": "2023-07-02T10:02:43",
      "score": 2,
      "text": "I feel alone."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=Hofib",
      "user": "Hofib",
      "id": 19187533,
      "date": "2024-04-08T06:06:38",
      "score": 0,
      "text": "I'm gonna die alone."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=SlenderFlames",
      "user": "SlenderFlames",
      "id": 19187541,
      "date": "2024-04-08T06:07:31",
      "score": 0,
      "text": "Maybe one day, won't be an issue if not."
    },
    {
      "user_link": "https://rule34.xxx/index.php?page=account&s=profile&uname=NatsuDragneelRP",
      "user": "NatsuDragneelRP",
      "id": 19187542,
      "date": "2024-04-08T06:07:35",
      "score": 0,
      "text": "^ Join the club"
    }
  ]
}

Usage

Under the working directory it creates json files named by post id. Posts that are confirmed to not exist will create similar, empty files with _e suffix e.g. 5011473_e. Subsequent downloads will skip existing files that aren't empty and non existing posts.

You can download torrents by specific urls

./rule34xxx.py 'https://rule34.xxx/index.php?page=post&s=view&id=13749665' 'https://rule34.xxx/index.php?page=post&s=view&id=13749663'

If no urls arguments are given the whole site is scraped by iterating over the ids, in this case starting from 500 to 99999.

./rule34xxx.py --directory ./d/ --first 500 --last 99999 --wait 0.6 --wait-random 0.2

If requests are made too quickly you'll get rate limited.

usage: rule34xxx.py [-h] [-d DIR] [-D DOMAIN] [--no-comments] [-f ID] [-l ID]
                    [--last-id] [-w TIME] [-W TIME] [-r NUM]
                    [--retry-delay TIME] [--retry-all-errors] [-m TIMEOUT]
                    [-k] [-L] [--max-redirs NUM] [-A UA] [-x PROXY]
                    [-H HEADER] [-b COOKIE] [-B BROWSER]
                    [URL ...]

Tool for getting things from rule34. If no URLs provided scrapes the whole
site

positional arguments:
  URL                   urls

options:
  -h, --help            Show this help message and exit
  -d, --directory DIR   Use DIR as working directory
  -D, --domain DOMAIN   set DOMAIN, by default set to https://rule34.xxx
  --no-comments         don't make requests to get comments
  -f, --first ID        get posts, starting from ID
  -l, --last ID         get posts, ending at ID
  --last-id             print id of the latest post and exit

Request settings:
  -w, --wait TIME       Set waiting time for each request
  -W, --wait-random TIME
                        Set random waiting time for each request to be from 0
                        to TIME
  -r, --retry NUM       Set number of retries for failed request to NUM
  --retry-delay TIME    Set interval between each retry
  --retry-all-errors    Retry no matter the error
  -m, --timeout TIMEOUT
                        Set request timeout, if in TIME format it'll be set
                        for the whole request. If in TIME,TIME format first
                        TIME will specify connection timeout, the second read
                        timeout. If set to '-' timeout is disabled
  -k, --insecure        Ignore ssl errors
  -L, --location        Allow for redirections, can be dangerous if
                        credentials are passed in headers
  --max-redirs NUM      Set the maximum number of redirections to follow
  -A, --user-agent UA   Sets custom user agent
  -x, --proxy PROXY     Use the specified proxy, can be used multiple times.
                        If set to URL it'll be used for all protocols, if in
                        PROTOCOL URL format it'll be set only for given
                        protocol, if in URL URL format it'll be set only for
                        given path. If first character is '@' then headers are
                        read from file
  -H, --header HEADER   Set curl style header, can be used multiple times e.g.
                        -H 'User: Admin' -H 'Pass: 12345', if first character
                        is '@' then headers are read from file e.g. -H @file
  -b, --cookie COOKIE   Set curl style cookie, can be used multiple times e.g.
                        -b 'auth=8f82ab' -b 'PHPSESSID=qw3r8an829', without
                        '=' character argument is read as a file
  -B, --browser BROWSER
                        Get cookies from specified browser e.g. -B firefox

with-proxies

Is a script that takes the working directory and a file with proxy links. It splits the task for the proxies and downloads them independently in parallel.

Results

You can see scraped results here.

About

A full scraper for rule34.xxx

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published