Skip to content

Tags: tantek/granary

Tags

v2.2

Toggle v2.2's commit message

### Notable changes
* Add Mastodon support!
* Add Python 3.7 support, and improve overall Python 3 compatibility.
* Update a number of dependencies.
* Switch from Python's built in `json` module to [`ujson`](https://github.com/esnme/ultrajson/) to speed up JSON parsing and encoding.
* Add `duration` and `size` support to ActivityStreams 1 and 2, RSS, and microformats2 HTML and JSON. [microformats2 support is still emerging for both](https://indieweb.org/podcast#Brainstorming). Both integer seconds and [ISO 8601 string durations](https://en.wikipedia.org/wiki/ISO_8601#Durations) are supported for `duration`. Integer bytes is used for `size` everywhere. microformats2 HTML also includes human-readable strings, eg `5.1 MB`. ([snarfed#169](snarfed#169))
* Twitter:
  * `[preview]_create()`: detect attempts to upload [images over 5MB](https://developer.twitter.com/en/docs/media/upload-media/uploading-media/media-best-practices#image-specs) and return an error.
* Facebook:
  * Add `get_activities(scrape=True)` for scraping HTML from [m.facebook.com](https://m.facebook.com/). Requires `c_user` and `xs` cookies from a logged in session ([snarfed/bridgy#886](snarfed/bridgy#886)).
  * [Upgrade Graph API version from 2.10 to 4.0.](https://developers.facebook.com/docs/graph-api/changelog)
* Atom:
  * Bug fix for de-duping images in attachments.
* RSS:
  * Wrap all `<description>` element contents in `CDATA` sections.
  * Render images in `<description>` with HTML `<img>` tags ([snarfed#175](snarfed#175)).
  * `from_activities()` bug fix: don't crash when converting multiple attachments to enclosures in a single item. (RSS only supports one enclosure per item, so we now only include the first, and log a warning if the activity has more.)

v2.1

Toggle v2.1's commit message

### Notable changes
* Convert AS2 `Mention` tags to AS1 `objectType` `mention` (non-standard) and vice versa ([snarfed/bridgy-fed#46](snarfed/bridgy-fed#46)).
* Twitter:
  * Bug fix for large block list fetches that get rate limited after a few successful requests.
  * Handle HTTP 403 + error code 200 when fetching retweets for a protected or otherwise unavailable tweet ([bridgy#688](snarfed/bridgy#688 (comment))).
  * Demote @-mentions from [person-tags](https://indieweb.org/person-tag) to [mentions](https://indieweb.org/mention). Specifically, this means they'll no longer get rendered with `u-category` mf2.
* Instagram:
  * Disabled in the REST API entirely due to Instagram's aggressive rate limiting and blocking ([bridgy#655](snarfed/bridgy#665 (comment))).
  * Update scraping to handle replies in new `edge_media_to_parent_comment` field ([snarfed#164](snarfed#164)).
  * Use cookie for all scraping HTTP requests, not just for likes.
* microformats2:
  * Revise whitespace handling; use `white-space: pre` CSS in HTML output.
* Facebook:
  * Bug fix: don't interpret `photo.php` as username in post URLs.
* Atom:
  * Switch from `white-space: pre` CSS back to converting newlines to `<br>`s because some feed readers ([eg NewsBlur](https://forum.newsblur.com/t/android-cant-read-line-pre-formatted-lines/6116)) follow it too strictly and don't even line wrap.
* RSS:
  * Default title to ellipsized content.

v2.0

Toggle v2.0's commit message

### Notable changes

_Breaking change_: drop Google+ since [it shuts down in March](https://developers.google.com/+/api-shutdown). Notably, this removes the `googleplus` module.

v1.15

Toggle v1.15's commit message

### Notable changes

* Add RSS 2.0 output! ([snarfed#124](snarfed#124))
* All silos:
  * Switch users' primary URLs from web site to silo profile ([snarfed#158](snarfed#158)).
* GitHub:
  * Don't enclose bare URLs in `<`/`>` ([snarfed/bridgy#850](snarfed/bridgy#850)).
* Atom:
  * Bug fix for actors and attachments with multiple image URLs.
  * Bug fix for attachment author objects with no properties.
* Google+:
  * Drop from web UI and REST API since [consumer Google+ is shutting down entirely](https://blog.google/technology/safety-security/expediting-changes-google-plus/) ([more](snarfed/bridgy#846)).
  * Switch from deprecated global API endpoint to G+ endpoint. Background in [snarfed/bridgy#846](snarfed/bridgy#846), [Google blog post](https://developers.googleblog.com/2018/03/discontinuing-support-for-json-rpc-and.html) [and docs](https://developers.google.com/api-client-library/python/guide/batch).
* Instagram:
  * Fix individual photo/video link urls for multi-photo/video posts.
  * Handle [user-provided alt text](https://instagram-press.com/blog/2018/11/28/creating-a-more-accessible-instagram/) ([snarfed#159](snarfed#159)).
* Twitter:
  * Update max video upload size from 5MB to 512MB ([snarfed#162](snarfed#162)).
* `/url`: Return HTTP 400 when fetching the user's URL results in an infinite redirect.

v1.14

Toggle v1.14's commit message

### Notable changes
Add `delete()`. Currently includes Twitter and Flickr support.
* Instagram:
  * Make extra HTTP fetch (with cookie) to get individual likes ([snarfed/bridgy#840](snarfed/bridgy#840)).
  * Update scraping logic to handle feed HTML changes.
  * Link @-mentions in comments as well as photo/video captions.
* GitHub:
  * `create`/`preview_create` bug fixes for issues and comments on private repos.
  * Handle HTTP 410 Gone responses from REST API, eg when a repo has been deleted or issues for the repo disabled.
* Twitter:
  * Add `delete()` and `preview_delete()` for deleting tweets.
* Flickr:
  * Add `delete()` and `preview_delete()` for deleting photos.
* microformats2:
  * Add [follow-of](https://indieweb.org/follow) support.
  * Only use quotation-of property for quote tweets, not URLs. ([snarfed#155](snarfed#155))
  * If a tag has startIndex/length, it gets linkified in the content, so don't also emit an mf2 child or HTML h-cite for it. ([snarfed#155](snarfed#155)
* Atom:
  * Encode `&`s in author URL and email address too. (Thanks [sebsued](https://twitter.com/sebsued)!)
* AS2:
  * Add `Follow` support.

v1.13

Toggle v1.13's commit message

### Notable changes
* Twitter:
  * Support ISO 8601 formatted created_at timestamps, which the [archive download uses](https://help.twitter.com/en/managing-your-account/how-to-download-your-twitter-archive), as well as RFC 2822 from the API.
  * `create()` and `preview_create()`: support RSVPs. Tweet them as normal tweets with the RSVP content. ([snarfed/bridgy#818](snarfed/bridgy#818))
  * `create()` and `preview_create()`: support alt text for images, via AS1 `displayName`. ([snarfed/bridgy#756](snarfed/bridgy#756)).
* Instagram:
  * Add global rate limiting lock for scraping. If a scraping HTTP request gets a 429 or 503 response, we refuse to make more requests for 5m, and instead short circuit and return the same error. This can be overridden with a new `ignore_rate_limit` kwarg to `get_activities()`.
* GitHub:
  * Add `tag` support to `create`/`preview_create` to add label(s) to existing issues ([snarfed/bridgy#811](snarfed/bridgy#811)).
  * Escape HTML characters (`<`, `>`, and `&`) in content in `create()` and `preview_create()` ([snarfed/bridgy#810](snarfed/bridgy#810)).
  * `get_activities()` and `get_comment()` now return `ValueError` instead of `AssertionError` on malformed `activity_id` and `comment_id` args, respectively.
  * `get_activities()` bug fix for issues/PRs with no body text.
  * Switch from GraphQL to REST API for creating comments and reactions, since GraphQL hits authorization errors on many org repos. ([snarfed/bridgy#824](snarfed/bridgy#824))
  * Improve GraphQL support for comments and users.
* Atom:
  * Shorten and ellipsize feed title when necessary ([snarfed#144](snarfed#144)).
* microformats2:
  * Upgrade mf2py to improve a few things like [implied p-name detection](http://microformats.org/wiki/microformats2-implied-properties) and whitespace handling ([snarfed#142](snarfed#142), fixes [snarfed#145](snarfed#145), [snarfed/bridgy#756](snarfed/bridgy#756), [snarfed/bridgy#828](snarfed/bridgy#828)).
  * Support `alt` attribute in `<img>` tags ([snarfed/bridgy#756](snarfed/bridgy#756)).

v1.12

Toggle v1.12's commit message

### Notable changes

* Add Python 3 support! Granary now requires either Python 2.7+ or Python 3.3+.
* Instagram:
  * Fix scraping profile pages.
* Twitter:
  * Update character counting to handle Twitter change that now auto-links *all* ccTLDs. [Background.](karadaisy/brevity#8)
* GitHub:
  * Bug fix for `get_activities()` with deleted issues and repos.
* microformats2:
  * `object_to_json()`: convert tags to simple strings in the `category` property, not full nested objects like `h-card`s ([snarfed#141](snarfed#141)).
  * Special case GitHub issues that are in-reply-to a repo or its `/issues` URL to be objectType `issue`.
  * Render simple string categories in HTML output.

This release is intentionally small and limited in scope to contain any impact of the Ptython 3 migration. It *should* be a noop for existing Python 2 users, and we've tested thoroughly, but I'm sure there are still bugs. Please file issues if you notice anything broken!

v1.12-docs

Toggle v1.12-docs's commit message
drop dupe brevity line from requirements.txt

v1.11

Toggle v1.11's commit message

### Notable changes
* Add GitHub!
  * `get_activities()` supports issues and pull requests, including comments and reactions. It's currently based on notifications, so it's best effort, not comprehensive, and only includes recently active issues/PRs.
  * `create()` and `preview_create()` support issues, comments, [stars](https://help.github.com/articles/about-stars), and [reactions](https://help.github.com/articles/about-conversations-on-github/#reacting-to-ideas-in-comments).
* Twitter:
  * Prefer MP4 and other video/... content types to HLS (.m3u8) etc. [Background.](https://twittercommunity.com/t/retiring-mp4-video-output/66093)
  * Prefer HTTPS URLs for media images.
  * `get_activities()`: Support @-prefixed usernames in `user_id`.
* Facebook:
  * Support new [recurring aka multi-instance events](https://stackoverflow.com/questions/45131646/decoding-recurring-events-from-facebook-open-graph-api). `create()` and `preview_create()` now only support RSVPs to individual instances of multi-instance events, to match the Facebook API itself.
  * Try harder to find original (full) sized photo URLs, specifically `_o.jpg` files instead of `_s.jpg`.
  * `create()` bug fix for photo and image URLs with unicode characters.
  * Fixed bug where `get_activities(user_id=...)` included the authenticated user's own recent photos, albums, and news publishes.
* Instagram:
  * Extract more user (`author`) data from scraped profile pages.
  * Fix home page feed scraping.
* microformats2, Atom:
  * Add enclosures for image attachments.
  * Bug fixes for rendering image, video, and audio attachments inside shares and attachments. De-dupe images.
* microformats2:
  * Handle simple string-only author properties.
  * Add `fetch_mf2` kwarg to `json_to_object()` for fetching additional pages over HTTP to determine authorship.
  * Generate explicit blank `p-name` in HTML to prevent old flawed [implied p-name handling](http://microformats.org/wiki/microformats2-implied-properties) [snarfed#131](snarfed#131).
  * Fix `share` verb handling in `activity_to_json()` and `activities_to_html()` [snarfed#134](snarfed#134).
  * Remember which content contains HTML, preserve newlines in it, and don't translate those newlines to `<br>`s ([snarfed#130](snarfed#130)).
* Atom:
  * Fix timezone bugs in `updated` and `published`.
* JSON Feed:
  * Omit title from items if it's the same as the content. (Often caused by microformats2's implied `p-name` logic.)

v1.11-docs

Toggle v1.11-docs's commit message
update docs for v1.11