Tags: tantek/granary
Tags
### Notable changes * Add Mastodon support! * Add Python 3.7 support, and improve overall Python 3 compatibility. * Update a number of dependencies. * Switch from Python's built in `json` module to [`ujson`](https://github.com/esnme/ultrajson/) to speed up JSON parsing and encoding. * Add `duration` and `size` support to ActivityStreams 1 and 2, RSS, and microformats2 HTML and JSON. [microformats2 support is still emerging for both](https://indieweb.org/podcast#Brainstorming). Both integer seconds and [ISO 8601 string durations](https://en.wikipedia.org/wiki/ISO_8601#Durations) are supported for `duration`. Integer bytes is used for `size` everywhere. microformats2 HTML also includes human-readable strings, eg `5.1 MB`. ([snarfed#169](snarfed#169)) * Twitter: * `[preview]_create()`: detect attempts to upload [images over 5MB](https://developer.twitter.com/en/docs/media/upload-media/uploading-media/media-best-practices#image-specs) and return an error. * Facebook: * Add `get_activities(scrape=True)` for scraping HTML from [m.facebook.com](https://m.facebook.com/). Requires `c_user` and `xs` cookies from a logged in session ([snarfed/bridgy#886](snarfed/bridgy#886)). * [Upgrade Graph API version from 2.10 to 4.0.](https://developers.facebook.com/docs/graph-api/changelog) * Atom: * Bug fix for de-duping images in attachments. * RSS: * Wrap all `<description>` element contents in `CDATA` sections. * Render images in `<description>` with HTML `<img>` tags ([snarfed#175](snarfed#175)). * `from_activities()` bug fix: don't crash when converting multiple attachments to enclosures in a single item. (RSS only supports one enclosure per item, so we now only include the first, and log a warning if the activity has more.)
### Notable changes * Convert AS2 `Mention` tags to AS1 `objectType` `mention` (non-standard) and vice versa ([snarfed/bridgy-fed#46](snarfed/bridgy-fed#46)). * Twitter: * Bug fix for large block list fetches that get rate limited after a few successful requests. * Handle HTTP 403 + error code 200 when fetching retweets for a protected or otherwise unavailable tweet ([bridgy#688](snarfed/bridgy#688 (comment))). * Demote @-mentions from [person-tags](https://indieweb.org/person-tag) to [mentions](https://indieweb.org/mention). Specifically, this means they'll no longer get rendered with `u-category` mf2. * Instagram: * Disabled in the REST API entirely due to Instagram's aggressive rate limiting and blocking ([bridgy#655](snarfed/bridgy#665 (comment))). * Update scraping to handle replies in new `edge_media_to_parent_comment` field ([snarfed#164](snarfed#164)). * Use cookie for all scraping HTTP requests, not just for likes. * microformats2: * Revise whitespace handling; use `white-space: pre` CSS in HTML output. * Facebook: * Bug fix: don't interpret `photo.php` as username in post URLs. * Atom: * Switch from `white-space: pre` CSS back to converting newlines to `<br>`s because some feed readers ([eg NewsBlur](https://forum.newsblur.com/t/android-cant-read-line-pre-formatted-lines/6116)) follow it too strictly and don't even line wrap. * RSS: * Default title to ellipsized content.
### Notable changes _Breaking change_: drop Google+ since [it shuts down in March](https://developers.google.com/+/api-shutdown). Notably, this removes the `googleplus` module.
### Notable changes * Add RSS 2.0 output! ([snarfed#124](snarfed#124)) * All silos: * Switch users' primary URLs from web site to silo profile ([snarfed#158](snarfed#158)). * GitHub: * Don't enclose bare URLs in `<`/`>` ([snarfed/bridgy#850](snarfed/bridgy#850)). * Atom: * Bug fix for actors and attachments with multiple image URLs. * Bug fix for attachment author objects with no properties. * Google+: * Drop from web UI and REST API since [consumer Google+ is shutting down entirely](https://blog.google/technology/safety-security/expediting-changes-google-plus/) ([more](snarfed/bridgy#846)). * Switch from deprecated global API endpoint to G+ endpoint. Background in [snarfed/bridgy#846](snarfed/bridgy#846), [Google blog post](https://developers.googleblog.com/2018/03/discontinuing-support-for-json-rpc-and.html) [and docs](https://developers.google.com/api-client-library/python/guide/batch). * Instagram: * Fix individual photo/video link urls for multi-photo/video posts. * Handle [user-provided alt text](https://instagram-press.com/blog/2018/11/28/creating-a-more-accessible-instagram/) ([snarfed#159](snarfed#159)). * Twitter: * Update max video upload size from 5MB to 512MB ([snarfed#162](snarfed#162)). * `/url`: Return HTTP 400 when fetching the user's URL results in an infinite redirect.
### Notable changes Add `delete()`. Currently includes Twitter and Flickr support. * Instagram: * Make extra HTTP fetch (with cookie) to get individual likes ([snarfed/bridgy#840](snarfed/bridgy#840)). * Update scraping logic to handle feed HTML changes. * Link @-mentions in comments as well as photo/video captions. * GitHub: * `create`/`preview_create` bug fixes for issues and comments on private repos. * Handle HTTP 410 Gone responses from REST API, eg when a repo has been deleted or issues for the repo disabled. * Twitter: * Add `delete()` and `preview_delete()` for deleting tweets. * Flickr: * Add `delete()` and `preview_delete()` for deleting photos. * microformats2: * Add [follow-of](https://indieweb.org/follow) support. * Only use quotation-of property for quote tweets, not URLs. ([snarfed#155](snarfed#155)) * If a tag has startIndex/length, it gets linkified in the content, so don't also emit an mf2 child or HTML h-cite for it. ([snarfed#155](snarfed#155) * Atom: * Encode `&`s in author URL and email address too. (Thanks [sebsued](https://twitter.com/sebsued)!) * AS2: * Add `Follow` support.
### Notable changes * Twitter: * Support ISO 8601 formatted created_at timestamps, which the [archive download uses](https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive), as well as RFC 2822 from the API. * `create()` and `preview_create()`: support RSVPs. Tweet them as normal tweets with the RSVP content. ([snarfed/bridgy#818](snarfed/bridgy#818)) * `create()` and `preview_create()`: support alt text for images, via AS1 `displayName`. ([snarfed/bridgy#756](snarfed/bridgy#756)). * Instagram: * Add global rate limiting lock for scraping. If a scraping HTTP request gets a 429 or 503 response, we refuse to make more requests for 5m, and instead short circuit and return the same error. This can be overridden with a new `ignore_rate_limit` kwarg to `get_activities()`. * GitHub: * Add `tag` support to `create`/`preview_create` to add label(s) to existing issues ([snarfed/bridgy#811](snarfed/bridgy#811)). * Escape HTML characters (`<`, `>`, and `&`) in content in `create()` and `preview_create()` ([snarfed/bridgy#810](snarfed/bridgy#810)). * `get_activities()` and `get_comment()` now return `ValueError` instead of `AssertionError` on malformed `activity_id` and `comment_id` args, respectively. * `get_activities()` bug fix for issues/PRs with no body text. * Switch from GraphQL to REST API for creating comments and reactions, since GraphQL hits authorization errors on many org repos. ([snarfed/bridgy#824](snarfed/bridgy#824)) * Improve GraphQL support for comments and users. * Atom: * Shorten and ellipsize feed title when necessary ([snarfed#144](snarfed#144)). * microformats2: * Upgrade mf2py to improve a few things like [implied p-name detection](http://microformats.org/wiki/microformats2-implied-properties) and whitespace handling ([snarfed#142](snarfed#142), fixes [snarfed#145](snarfed#145), [snarfed/bridgy#756](snarfed/bridgy#756), [snarfed/bridgy#828](snarfed/bridgy#828)). * Support `alt` attribute in `<img>` tags ([snarfed/bridgy#756](snarfed/bridgy#756)).
### Notable changes * Add Python 3 support! Granary now requires either Python 2.7+ or Python 3.3+. * Instagram: * Fix scraping profile pages. * Twitter: * Update character counting to handle Twitter change that now auto-links *all* ccTLDs. [Background.](karadaisy/brevity#8) * GitHub: * Bug fix for `get_activities()` with deleted issues and repos. * microformats2: * `object_to_json()`: convert tags to simple strings in the `category` property, not full nested objects like `h-card`s ([snarfed#141](snarfed#141)). * Special case GitHub issues that are in-reply-to a repo or its `/issues` URL to be objectType `issue`. * Render simple string categories in HTML output. This release is intentionally small and limited in scope to contain any impact of the Ptython 3 migration. It *should* be a noop for existing Python 2 users, and we've tested thoroughly, but I'm sure there are still bugs. Please file issues if you notice anything broken!
### Notable changes * Add GitHub! * `get_activities()` supports issues and pull requests, including comments and reactions. It's currently based on notifications, so it's best effort, not comprehensive, and only includes recently active issues/PRs. * `create()` and `preview_create()` support issues, comments, [stars](https://help.github.com/articles/about-stars), and [reactions](https://help.github.com/articles/about-conversations-on-github/#reacting-to-ideas-in-comments). * Twitter: * Prefer MP4 and other video/... content types to HLS (.m3u8) etc. [Background.](https://twittercommunity.com/t/retiring-mp4-video-output/66093) * Prefer HTTPS URLs for media images. * `get_activities()`: Support @-prefixed usernames in `user_id`. * Facebook: * Support new [recurring aka multi-instance events](https://stackoverflow.com/questions/45131646/decoding-recurring-events-from-facebook-open-graph-api). `create()` and `preview_create()` now only support RSVPs to individual instances of multi-instance events, to match the Facebook API itself. * Try harder to find original (full) sized photo URLs, specifically `_o.jpg` files instead of `_s.jpg`. * `create()` bug fix for photo and image URLs with unicode characters. * Fixed bug where `get_activities(user_id=...)` included the authenticated user's own recent photos, albums, and news publishes. * Instagram: * Extract more user (`author`) data from scraped profile pages. * Fix home page feed scraping. * microformats2, Atom: * Add enclosures for image attachments. * Bug fixes for rendering image, video, and audio attachments inside shares and attachments. De-dupe images. * microformats2: * Handle simple string-only author properties. * Add `fetch_mf2` kwarg to `json_to_object()` for fetching additional pages over HTTP to determine authorship. * Generate explicit blank `p-name` in HTML to prevent old flawed [implied p-name handling](http://microformats.org/wiki/microformats2-implied-properties) [snarfed#131](snarfed#131). * Fix `share` verb handling in `activity_to_json()` and `activities_to_html()` [snarfed#134](snarfed#134). * Remember which content contains HTML, preserve newlines in it, and don't translate those newlines to `<br>`s ([snarfed#130](snarfed#130)). * Atom: * Fix timezone bugs in `updated` and `published`. * JSON Feed: * Omit title from items if it's the same as the content. (Often caused by microformats2's implied `p-name` logic.)
PreviousNext