RE: retention policy for cached content and media #19260
Replies: 18 comments 15 replies
-
This feels like a very pragmatic suggestion -- essentially "reference counting" the remote content. |
Beta Was this translation helpful? Give feedback.
-
I would be very keen to see something like this incorporated into the change as well. We do desperately need a way to purge old statuses more easily but I would be very hesitant to use it if it didn't discriminate between content that local users had interacted with and that which they hadn't. This would make it a much more useful tool. |
Beta Was this translation helpful? Give feedback.
-
Also worth noting that on many instances the storage utilisation is dominated by profiles' media rather than posts. #9567 |
Beta Was this translation helpful? Give feedback.
-
I got hit by this on a recently launched instance by Vivaldi Browser team. My bookmarks pointing to older posts were disappearing. They managed to locate the issue to this setting and switched it to unlimited. Hopefully, some resolution can be found to keep interactions while still saving resources for instance owners! |
Beta Was this translation helpful? Give feedback.
-
This is an great suggestion to make this retention policy feature more useful. Hope it gets implemented. |
Beta Was this translation helpful? Give feedback.
-
I agree fully. After understanding the consequences of the retention policy, I set it to unlimited, as I would expect my bookmarks, etc. to be stable and existing, even after years. I would also like to ask to question, what about accounts that are in "monument mode" for instance, when someone died? I personally think till now, that I would like to keep the content of that account as well. |
Beta Was this translation helpful? Give feedback.
-
I think this could even be much simpler - probably in >99% of cases, once a user bookmarks, favorites, boosts etc. a status, they will not take that back later. So there could be just one flag "this status was favorited/bookmarked etc." that is just set once and never reset. If a post was favorited or bookmarked once, it almost certainly still is, so we can just as well keep that post without re-checking that. The additional storage savings we could get from trying to find all those statuses that were later removed from all bookmarks etc. would be minimal, for a lot more work. |
Beta Was this translation helpful? Give feedback.
-
Will this action lose replies from external servers as well? So it will look like 2 years ago I was just talking to myself 😄... Or will they be retained? I can live with disappearing boosts, favourites, and bookmarks, but would hope the context tree of my posts would remain. |
Beta Was this translation helpful? Give feedback.
-
Just realized this from here.
|
Beta Was this translation helpful? Give feedback.
-
I just lost everything by enabling this feature. They are talking about cache, but in fact this is deleting data without the ability to update the cache. Fucking drug addicts who wrote about cache. |
Beta Was this translation helpful? Give feedback.
-
I've been told that an issue I noticed with posts in threads getting disassociated from one another might actually be due to this policy: https://social.treehouse.systems/@egallager/111898518131801953 |
Beta Was this translation helpful? Give feedback.
-
This really needs to be solved at some point. As an instance admin I get to a point that I either need to ask people for more money just to host old external toots, ~99% of which nobody cares for by this point, or I have to enable the really darconian feature, fully knowing that I delete people's old bookmarks in the process. |
Beta Was this translation helpful? Give feedback.
-
I had this feature on until I came across this discussion. Fortunately the policy was long enough that it wasn't affecting anything and my instance is new enough that there's not much to affect, but yes absolutely please build in a check for at least 1 bookmark on the local instance before it gets nuked. I'm not so picky about boosts, but if you want people to bookmark things, those things should still be there. I turned this feature off in a hot minute after reading this, and will turn it back on just as quickly if it gets fixed. |
Beta Was this translation helpful? Give feedback.
-
I agree as well. This is starting to bite us more and more as previous stuff gets wiped out. I personally would prefer it be any interaction and not just bookmarks (favs, boosts, been replied to). |
Beta Was this translation helpful? Give feedback.
-
It should also be mentioned that it needs to be desiced if only interacted-with posts should remain, or all posts in that thread, since some user (myself included) may only interact / bookmark the top-most toot(s) but is actually interested into the complete discussion / thread to read it later on or similar. |
Beta Was this translation helpful? Give feedback.
-
Is there any progress on this issue? I'd prefer not to have to inevitably upgrade the server my instance runs on simply because I ran out of space. |
Beta Was this translation helpful? Give feedback.
-
Any progress on this issue? |
Beta Was this translation helpful? Give feedback.
-
I had to flip the setting, knowing the major downsides because for me too, it would become too expensive otherwise. Now, due to the downsides of having done so, I am seriously considering giving up on my vanity (single person) instance and just switching to another instance with more people where I donate to those that maintain it. It's a real shame but currently as things stand, this is unworkable for me. If I retain everything it is too expensive, if I don't I get a sub standard experience. |
Beta Was this translation helpful? Give feedback.
-
The recent PR #19232 adding a retention policy for cached content and media can significantly reduce the resources used by Mastodon. In particular, the storage needs. This is something that admins will benefit from and reduce the network's carbon footprint. So, I am happy that it has been introduced.
Still, the more I think about it, the more I believe that deleting all remote content that is older than X days from the database (set in 'Content cache retentions period') will cause unintentional consequences for the end user, as I mentioned on #19232 (comment).
If I bookmarked a post one year ago. I expect it to be in my bookmarks one year later, even if the server that originated the post is no longer online and the content cache retention period set by the admin is 6 months. The same goes for boosts, favourites, DMs or posts I replied to. If we use the date as the single factor for deletion, we break functionality that I believe is important.
Now, I understand that only deleting posts that are older than a date if they were not interacted with like it's the case of
tootctl status remove
can be "computationally heavy procedure that creates extra database indices". But we could explore ways to make this content filter of interacted posts less demanding.One idea that comes to mind is to add a counter for each post. Every time a local user bookmarks, favourites, boosts, is mentioned on or replies to a post, the counter on that post increases by one. If a local user removes from bookmarks, favourites, boosts, or delete their reply, it decreases by one. This way, the cache deletion would filter posts older than X date and where the number on that counter would be zero.
That same counter could also be used for the 'Media cache retention period' for post attachment media. As some cannot be redownloaded if the server goes offline and in some cases the post without the media makes little sense or is just blank.
There could be something I am missing, or this could be way more complex than I think. But I believe some more thinking could be done before merging this PR to the stable release.
Beta Was this translation helpful? Give feedback.
All reactions