{"ops":[{"insert":"Hey Archidekt team. A friend and I were lamenting the lack of an EDHrec-like tool for Pauper EDH decks, and of course it dawned on me that I could address this gap myself. I'm working on a tool tentatively called \"Small Formats\" (with a goal to someday go beyond PDH). It's not up on the web yet, I'm only playing around locally in dev, but the repo is at https://github.com/vtbassmatt/SmallFormats if you want to take a peek. The vision is \"EDHrec-like pages for PDH decks\", including the links back to the originating deck/site and attribution to the user.\n\nI understand from the forum that you unofficially allow use of your internal API, with no expectation of stability or support. Assuming I understood that correctly, I'd like to be as good a citizen as possible. Can you help me understand:\nWhat is a fair number of headless requests per (second/minute/day)? I intend to hit each page of "},{"attributes":{"color":"#0066cc"},"insert":"api/decks/cards/"},{"insert":" filtering to "},{"attributes":{"color":"#0066cc"},"insert":"format=17"},{"insert":" and "},{"attributes":{"color":"#0066cc"},"insert":"size=100"},{"insert":" (which presently contains around 1800 public decks, so about 38 default-sized pages) perhaps once a day. Then, for each deck which has an "},{"attributes":{"color":"#0066cc"},"insert":"updatedAt"},{"insert":" later than my records, I will hit "},{"attributes":{"color":"#0066cc"},"insert":"api/decks/{id}"},{"insert":" to get the card list. Note that these numbers are for my intended production use; I'm still messing around in dev and limiting myself to 3-4 pages of results right now."},{"attributes":{"list":"bullet"},"insert":"\n"},{"insert":"How does rate limiting manifest? (e.g. X-Retry-After headers, or in-band messaging, or ??) Right now the crawler sleeps for 2 seconds between each request. At some point during debugging, I think I got rate limited (count came back as -1) but unfortunately I didn't have good logging wired up, so I didn't get to see headers, message, etc. What else should I look for to determine I'm being rate limited so I can back off appropriately?"},{"attributes":{"list":"bullet"},"insert":"\n"},{"insert":"Besides a custom user-agent string, any other identifiers you'd like in my requests? I'm sending a custom user-agent (currently "},{"attributes":{"color":"#0066cc"},"insert":"SmallFormats/0.1.0"},{"insert":") so you can selectively throttle (or allow!) this traffic. I almost put \"+https://github.com/vtbassmatt/SmallFormats\" right in the user-agent, but since the code is open source, it's conceivable that others could run their own copies locally. I didn't want to put my GitHub URL in a request which may not originate from me."},{"attributes":{"list":"bullet"},"insert":"\n"},{"insert":"Anything else I'm forgetting in order to be conscientious?"},{"attributes":{"list":"bullet"},"insert":"\n"},{"insert":"\nThanks, and thank you for building Archidekt!\n"}]}
0
{"ops":[{"insert":"Nice sounds pretty cool!\n\nAs for your first bullet, hitting every deck like that per day isn't really scaleable long term. If you're already checking the updatedAt, order your request by updated descending and stop requesting new pages once you've hit yesterday's date (or whenever the last time you ran your data fetcher). Fetching every page seems wasteful if they haven't changed. If you do it the way I'm suggesting, I doubt we'd ever notice anything being scraped. \n\nSecond bullet: If you're getting throttled, you'd start seeing HTTP response code 429. And you'd be timed out for an exponentially extending time for multiple recent infractions. If you're sleeping for 2 seconds between requests, I doubt you'd ever hit it. \n\nCustom user agent is thoughtful. If we ever see it becoming an issue we can reconvene and see what we can do about it. But since I don't think this'll be enough traffic to really notice, I think you're g2g.\n\nCheers; have fun; and let us know when you've finished up! We'll let you know if anything you're doing becomes an issue (though you'll likely know first if you start seeing a bunch of 429s)\n"}]}
1
{"ops":[{"insert":"Cool, thanks! Yeah, I probably should have thought of stopping when I see a date from the previous run... I'm already sorting that way 🤦\n\nAnd thanks for the tip about the 429! Given how hacked up my crawler is right now, I couldn't do much with a X-Retry-After or other \"smart\" header anyhow. I'll probably log the exception and quit, letting the \"user\" (me) deal with it manually for now.\n"}]}
0