The bad news is that this means I sunk a ton of valuable work time into re-doing existing features. The good news is that a pile of tech debt isn’t hanging over the project anymore, so I can get back to feature work.

The short version of the story is that Breadcrum’s frontend has been rewritten from uhtml to preact/htm. The decision to make this change was due to encountering some intractable bugs in older versions of uhtml, and because Breadcrum’s frontend makes heavy use of components and “hooks” and uhtml’s decision to remove them from the support matrix. I wrote a big, long blog post about this switch, but I think rather than post all of that, I’ll just write up some takeaways. If you are curious about more details, feel free to reach out or check out the PR.

• Bus factor is real, and sometimes the factor is the bus deciding to drive to a different destination than when you got on. This is the case with uhtml. Still great tech, but the decision to drop hook support diverged the project from what I had already written in Breadcrum, and I needed to get back to a supported library with hooks support due to practical needs.
• Writing your entire application in a functional-reactive templating language is a ton of work, and I still haven’t shipped offline mode. Will I ever? This pattern hasn’t paid off the long-term goal I intended for it yet, so maybe I will change course on the frontend further.
• Breadcrum would have been a great fit for something like HTMX/hypermedia with just a sprinkling of functional-reactive templating around the more advanced form interactions. It would have cut down on the total lines of code, and offline mode might have still been possible.
• React is here to stay, no matter how much hate it gets. Preact is a really nice version of React, and htm works really well along similar lines as uhtml, while still keeping in line with React’s stability.
• Software development is chaotic. A turn at the beginning can permanently alter the trajectory of the project.
• The usefulness of LLMs as a refactoring tool drops off significantly when you try to have them do large transformations on novel or unique patterns/codebases.

Onward and upward. Please don’t let the pace discourage your support of the project. As always, feedback and ideas are welcome. Your input carries a lot of weight at this point in the project.

The short answer is: Breadcrum’s current approach to identifying, isolating, and redirecting podcast clients to media discovered in websites, which has worked for many years, is starting to run into intractable issues.

The YouTube situation

Breadcrum is a generalized tool users can use to redirect their podcast clients to media available for public consumption on unauthenticated websites. YouTube has a monopoly on publicly available videos on the web. Therefore, it’s also the most popular type of URL people bookmark with Breadcrum.

Previously, Breadcrum was able to identify the video content on a YouTube website URL, and share this information with your Podcast app. The app would then happily download it for you so you could watch using your choice of media player rather than being forced to use the prescribed player.

Breadcrum can still do this. However, the media URLs that are returned from YouTube now behave badly on purpose. They don’t work the first few times a client tries to access them! Well-behaved video clients, of course, don’t expect this, so they handle it like an error. Additionally, if Breadcrum tries to negotiate with the URL before sending it back to the podcast client, YouTube decides it no longer wants to talk with Breadcrum and blocks further interactions. Fair enough!

The Work Around

Until a better solution is developed, the workaround is: try and try again! Podcast apps have this built in, so it may already be doing this. If you go to download an episode and get an error, wait a few seconds and try again.

Where can we go from here?

There are a lot of options I’m considering.

Breadcrum downloads the video for you and stores it in temporary storage.

This will cost money, but has a lot of other cool properties. In general, this is needed already for all media sources, so this will likely materialize this year sometime.

You run a Breadcrum “agent” on your computer that processes videos for you

The satellite worker model. This could help keep costs down for users and work around other media source restrictions that block non-residential IP addresses.

BYO Media extraction

You could run a custom video extractor at home (anything you want) and Breadcrum works with it to create episodes.

An app

Breadcrum ships a podcast app that is more tolerant with bad video CDNs.

Conclusion

• If you run into issues with your episodes, retry them. They might start working on the second or third access.
• Breadcrum is currently working on more robust solutions!

By the Numbers

Breadcrum recently passed two major milestones:

🚀 100+ Users

We crossed the 100-user mark just before the year ended—a big leap from last year’s 31 users. Thank you to everyone who’s joined and made Breadcrum part of your media workflow!

100 users! Happy New Year! 🥳 https://t.co/UYzIxJ9Ukc pic.twitter.com/LSHdGNrFZv

— 🥖 Breadcrum (@breadcrum_) January 1, 2025

🔗 10k+ Bookmarks

Breadcrum now hosts over 10,000, hand-whittled bookmarks! People have clicked this button over 10k times. It’s doing something useful! Tags, Episodes and Archives of course, will always trail this number.

Welcome huffduff-video Users

Recently, a popular service called huffduff-video shut down after YouTube rolled out aggressive anti-bot measures.

Trouble ahead for yt-dlp

YouTube is essentially cable TV now, and they want you watching logged in, in Chrome, without ad blockers, or using their tedious native app.

To enforce this, they’ve made it difficult for non-residential IP addresses to negotiate public content on behalf of users. Additionally, many requests require running various forms of device attestation to make sure you are running in Chrome or Android.

Python based tools like yt-dlp struggle to perform these actions because it requires a full browser-like environment to reasonably satisfy these requirements.

In order to interact with content on YouTube, it’s increasingly clear that a sandboxed DOM/JS based runtime is required. It’s likely these device attestation measures will only increase with time.

Breadcrum Stepped Up

Since Breadcrum is primarily a Node.js application, it was able to integrate YouTube.js quickly as an alternative to yt-dlp. As a result, Breadcrum has remained, so far, operational in the face of these recent changes.

yt-dlp is still used for nearly over other media integration within Breadcrum and is a wonderful project.

A Bit of History

huffduff-video was inspired by discussions I had with Ryan Barrett on the Indieweb IRC channel. We were exploring ways to easily get content onto smartphones, and podcast apps became a clever solution—a wormhole for moving content onto any device.

I’m thrilled to see more people adopt this pattern, and I hope Breadcrum can continue where huffduff-video left off!

How Breadcrum Differs

While huffduff-video focused on passive sharing via Huffduffer, Breadcrum offers a private feed experience with expanded media support for videos and articles. If you’re a migrating huffduff-video user, I’m especially eager to hear about your experience and what features you’d love to see next.

What Goals?!

It’s been an especially hectic year for me personally. Family life and career have been my primary focus, so I’m mildly sad to say most of the goals I set out for Breadcrum in 2024 were not met.

A lot did get done though:

• 🌎 Breadcrum the service is still online!
• 🪙 It is still inexpensive to operate.
• 🐂 Dramatically improved back-end architecture that scales very efficiently.
• 📆 Scheduled episodes (currently broken fixed again).
• 🦋 BlueSky Video Support.
• 🐦 X Video Support started working again.
• 📀 Pluggable media extraction backend (YouTube not broken yet).
• 👨‍💼 Refactoring to use TypeScript (This wasted so much time, but should help things later).
• 💬 Discord Chat.
• 📚 Blog post comments section.
• 🥺 Accepting Donations.
• 🤖 PWA/Android share sheet support.
• 🧑‍💼 Admin panel with administrative tools.

Rather than emphasizing specific features this year, here are the general themes and directions I think Breadcrum needs to focus on.

💸 Sustainability: Paid accounts

I hope there are enough of you who enjoy the service enough to pay a little every year to help keep the computers and storage running.

📦 Data ownership: Import/Export

As a service, this is a minimum requirement. The data you create here is yours, and I would also love to host any existing data you may already have.

📼 Resiliency: Better Data Guarantees

With users and a budget, it’s time to start incurring the costs of tighter data guarantees. The worst-case scenario right now is that 24 hours of data might be lost due to hardware failure. I can get this down to 0 and plan to before asking for money.

📣 Prioritized feature development

With the additional channels of communication, I hope to use feedback from users to prioritize feature development.

Happy 2025!

Thank you for being part of Breadcrum’s journey! Your support and feedback have been invaluable, and I’m excited to keep building and improving the service with your input.

Whether you’re a longtime user or just getting started, I’d love to hear from you:

• What’s working well?
• What could be better?
• What new features would make Breadcrum even more useful for you?

Let’s make 2025 the year Breadcrum becomes an essential tool in your web media workflow.

Join the conversation in our Discord or drop me a message on Twitter, Mastodon, or BlueSky! Here’s to making the media firehose a little more manageable.

Syndication

• Mastodon
• X
• Bsky

Discord

With some regret, Breadcrum now offers a Discord server. As a longtime fan of community-owned communication and organizing infrastructure, Breadcrum initially had an IRC room. However, after years of mostly inactivity, Breadcrum has transitioned to Discord.

• 💬 Join the Breadcrum Discord

If you rely on the service and are interested in discussing ideas, design, or other development topics, feel free to join the Discord. At the time of writing, it’s literally just me in the server, so please adjust your expectations as the room grows. Feel free to @ mention me on anything!

Donations

Breadcrum will eventually offer paid subscriptions with additional features and services. These paid accounts will ensure the long-term economic viability of the service. However, as paid accounts are not yet available, some users have expressed a desire to financially support the project in the meantime.

We now have a donations page:

• 💸 Donate to Breadcrum

You’ll find details on ways to support the project on that page!

Bluesky videos are currently limited to 50MB and 30 seconds, so you won’t be saving much long-form content to listen to later. However, I expect these limits to increase in the future. It’s exciting to see support grow for Twitter alternatives.

An interesting detail is that this feature came to Breadcrum as an automatic update. I architected Breadcrum to inherit new features and capabilities quickly and painlessly. It’s incredibly satisfying to see this decision pay off.

Update on the yt-dlp situation

You may have noticed, or found out about Breadcrum due to the recent changes being implemented at YouTube. Most notably, YouTube is making it more difficult to run tools like yt-dlp in hosted and cloud environments and is causing disruption for Breadcrum and other related services. Previously, all media discovery in Breadcrum was done solely through yt-dlp.

yt-dlp has no immediate or short-term solutions to address this issue other than to run using consumer home internet connections. While that is a possibility (Anyone remember AudioGalaxy satellite?), Breadcrum has found a workaround for the time being.

Breadcrum now has a pluggable extraction proxy that can swap in different media extraction tools and solutions quickly. We’re currently running a custom solution that’s working well. When that fails, I’m hoping our time to ship alternative media extraction tools are cut down to days rather than weeks.

Other Updates

Service updates have been slower this year, for which I apologize. Breadcrum is undergoing a much-needed back-end refactor (adding types for those interested). This cleanup addresses some long-running technical debt and will hopefully pay off over the coming years. Here are our priorities before the end of the year:

• Improved initial bookmark extraction
  • Utilize client, archive, and episode metadata when populating bookmark details initially
• Improved bookmark create form
• Media remux into cloud storage (Paid feature)
  • Better format support and personal archiving of media

Hope everyone is having a good year and, as always, feel free to share ideas, requests or other thoughts you have when using Breadcrum.

The video will initially resolve to the livestream (which podcast apps usually can’t download but can typically play) and then automatically switch over to the recorded copy once it’s available. Watching the livestream in the podcast app provides additional benefits, such as the ability to sleep your device and continue streaming on headphones or other devices.

The UI for this feature is still rudimentary and will improve with further iteration. In the meantime, enjoy scheduling future episodes for yourself and not having to remember one more thing. This feature was made possible by the enhancements detailed in the previous post: 🏗️ Improved Architecture. Go give that a read if you missed it!

The Original Setup

One of the primary design goals of Breadcrum was to build using proven, reliable technology and patterns while still utilizing the time-saving features modern technologies offer.

• A Fastify “API First” API server.
  • Provides 100% of the APIs needed to implement a Breadcrum client.
  • Fully validated endpoint schemas with JSONSchema and OpenAPI 3.0.
• A Postgres relational database.
  • Breadcrum’s primary service is storing related data for users. Postgres is the best tool for that.
• A Redis cache for rate limiting and short-lived cache storage.
• A dedicated private yt-dlp-api service.
  • Provides a long-running Python process since loading yt-dlp per request was too slow.
• A top-bun multi-page JAMStack style web app client.
• Hosted on fly.io using Alpine Linux Docker containers.
• Continuous integration and deployment run on GitHub Actions.

This implementation served its purpose well. It was simple to set up and reliable. Despite omitting some common conventions (SSR, CDN, TypeScript etc), it never felt like it prioritized short-term productivity at the cost of long-term flexibility. Building and maturing on top of this base has been a pleasure.

There were a few pain points:

• If the server ever crashed, all in-flight jobs and work were lost and would not recover after a reboot. Server-resolved bookmark details, website archive extraction, and episode resolution would be dropped and none of the resources would finish resolving. Users would have to manually retry creating them if it was noticed.

• Resource scaling was uneven. Requests that triggered work-heavy jobs, like processing a large website, could slow down other requests or crash the server if it hit the out-of-memory supervisor limit. The server resources needed to cover 99% of requests safely and quickly were much smaller than those needed to cover the last 1% of heavy workload requests. To avoid these problems, provisioning for the 1% case was necessary.

The New Setup

• A BullMQ worker service (wrapped as a Fastify service).
• Two Redis instances: a volatile shared cache and a NOEVICT instance for the queue.
• High availability, single-region Postgres server.

The Worker

The worker process is a second Fastify service that spins up a few BullMQ queue workers. These connect to a new dedicated Redis instance that tracks queued jobs and allows workers to coordinate which process takes which job.

Async task queues are nothing new. You put details about a “job” into a queue, and a worker picks up the job as soon as possible. They offer many useful features that make processing tasks and jobs easier to reason about:

• Durability
  • If a worker dies during a job, the job queue notices and can retry the job on the next available worker according to configurable rules.
• Scalability
  • Spinning up new workers is easy and increases throughput and parallelism. If falling behind, just scale up by adding workers.
• Scheduling
  • Jobs can be scheduled based on dates and time offsets. Implementing features like “retry when this video goes live” becomes a trivial, reliable scheduling task in the work queue.
• Asynchronous processing
  • API endpoints can support long-running tasks without holding HTTP connections open, and the Job IDs used to track work pair perfectly with asynchronous API endpoint patterns.
• Progress awareness
  • Job workers can report progress at the individual job level.
• Observability
  • The queue system includes built-in observability tools to monitor queue progress. It also tracks completed and failed job history for later investigation.
• High elasticity
  • Workers can scale to zero when no work is pending and wake up when needed. Large jobs can dynamically provision more expensive resources and release them when finished.

With distributed queues, you can handle almost any workload exposed through an API. Now that Breadcrum has this capability, we can start tackling more challenging features like inserting scheduled videos into your feeds when they become available, rather than requiring the user to manually create an episode. Downloading large chunked HLS video playlists, re-muxing them into podcast-friendly formats, and uploading to affordable cloud hosting also becomes feasible with a work queue.

More importantly, the API server is doing less heavy lifting and is at lower risk of running out of memory on large jobs.

Shared Cache

The second Redis instance serves the original role: a place to store shared cache values. Now that there are two servers (web and worker), it’s advantageous to store cached values in a way that both can read and write. This is important if either service needs to scale up and requires a form of ‘shared memory’ between instances. It’s also useful for sharing expensive chunks of work between services, like looking up short-lived media URLs.

A Workspace Appears

With the additional complexity of managing two services that operate in lockstep, Breadcrum has adopted an npm workspaces pattern. Monorepos/workspaces have advantages and disadvantages that are out of scope for this post, but one way to think of them is they let you work on multiple independent packages that depend on common parts without a module publishing step in between. To get this advantage, everything has to work out of a single repository.

TS-in-JS

Finally, Breadcrum has adopted TS-in-JS patterns. It’s not fully passing type checks yet, but the tooling is set up and in use during development. This setup provides the advantages of TypeScript without the hassle or confusion of a build step. The cost of TypeScript builds is non-trivial, with a new category of VC-funded startup products dedicated to easing this pain. Breadcrum does not have the budget for that cost, so TS-in-JS avoids it by accepting a slightly less expressive syntax in favor of everything working by default.

Conclusion

These architectural improvements will serve as the foundation to reliably deliver the next wave of features on the Breadcrum roadmap.

Breadcrum.net started in 2021 with a few simple sketches for a bookmarking schema and some loose ideas around archiving articles, audio and video from around the web. The core concept was personal content archiving and media consumption tools wrapped around an “old-school” web 2.0 bookmarking service (like del.icio.us or pinboard.in). You can still read the original design document.

In other words: “Submit a URL, have it do something useful with that page, and save it to your account.” For now, “something useful” means article content archiving, and send-media-to-your-phone in the form of a podcast.

The original idea for “send any media to your phone as a podcast” developed back in 2015 when Ryan Barrett launched “huffduff-video”. The idea for huffduff-video came out of a discussion I participated in on the indiewebcamp IRC channel and it was cool to see that idea come to life. And it worked really well!

Whenever I shared the tool with my non-technical friends though, they liked the idea of it, however the UX of bookmarklets on iPhones being what it is, and having manage a separate user account on huffduffer, I could never really onboard people to the tool. A core goal of Breadcrum was to unify this concept into a user friendly tool that anyone can learn to use.

We’re not there yet, but its getting close.

I’m pleased to re-review this design document and conclude, I think the MVP has been achieved though! Breadcrum has shipped the following core features as of 2024:

• Bookmarking
• Tags
• Read/Unread State
• Episodes (Audio and Video)
• Article content Archives
• Full Text Search

The user experience could still use a lot of work, but core functionality is in.

What happened in 2023?

Lots of good foundation was laid in 2022, and development slowed quite a bit in 2023. Reflecting back on why this is, I think it boils down to two reasons: busy with life and work, and also I took time to just use and enjoy Breadcrum in its MVP state.

With respect to life and work, I moved again at the tail end of 2022 and I also changed jobs, so most of my surplus engineering attention ended getting diverted to those two major changes. Additionally my two kids graduated from babyhood to toddlerhood and require a lot more hands on Dad time on the weekends. Baby’s sleep a lot, toddlers do not!

The the imminent feeling to rush development forward also waned, because I was able to use and enjoy Breadcrum nearly every day. With all MVP features in place, I finally have a tool for collecting media from around the web that I always wanted, so rather than working on it every day, I just enjoyed using it.

2024 Goals

My goal for 2024 is to spread some of the joy I had in 2023 using Breadcrum to a wider audience. There missing pieces that need to be built before that can happen:

• 💾 Cloud Storage
• 📙 Better Docs
• 💸 Paid Accounts
• 🎨 Improved add bookmark UI
• ⚖️ Async Job Queue Architecture
• 📦 Data Import and Export

A few other nice to have features are also on the radar:

• 🔑 Passkeys
• 👩‍🍼 Multiple episode feeds
• 📞 PWA Offline Support
• 📤 Send to ReMarkable
• 🗓️ Scheduled Episodes
• 🏹 Sharable Archives
• 🚀 iOS/Native App (Only way to get good UX on data sharing apps)

And finally, I should probably just launch the damn thing, have a day of it on Product Hunt, and generally spread the word.

Let’s check in next year and see what ends up getting done.

If you want to see what immediate issues are being looked at, check out the Breadcrum project board over on GitHub.

The details

Here are a few things that happens to the page styles when you print documents:

• The light gray color accents are darkened up a bit. The lighter grays look nice on screens, but they often print badly when run through PDFs and printers.
• Your username is hidden. This makes prints psudo-anonymous. Note that the UUID of whatever you is still shown, however your account is the only one that can access it.
• The search bar is hidden. Can’t search on a printed page, so why print it?

Other random fixes

The search bar in macOS Safari now has a close button that has a visible color in dark mode.

Related work

A number of these improves have also been upstreamed into the base stylesheet for breadcrum:

• bcomnes/mine.css@v9.0.7

It’s open source and can be used on any HTML page that need a quick styling.

Until next time!

When you save a bookmark, episode and archive, Breadcrum now maintains a Postgres full text search index on your notes, summary and article content inside archives. Episode notes are also indexed. (🤔 Now that I’m writing this, can we index subtitles as well?)

How it works

Simply enter your query in the search bar at the top of the page, and then select if you want to search bookmarks, archives or episodes. The search query will do its best to try and find your content.

Feedback welcome

Please feel free to provide any feedback you have on this feature. It is new so there could always be bugs.