Create an app/lib directory when generating a new Rails project - #47843

Can you give some examples of the kind of classes you'd be placing here? lib/ is intended to be for non-app specific library code that just happens to live in the app for now (usually pending extraction into open source or whatever). Everything app specific that's part of the domain model should live in app/models (that directory is for POJOs as much as ARs). I don't see the purpose of another default directory here.

Thanks for the questions!

Can you give some examples of the kind of classes you'd be placing here?

This is an no-longer maintained project that I worked on, but I think it has a representative junk-drawer of objects: https://github.com/codeforamerica/drai/tree/a0442ee5689f09e6ede1b7eec6f29dd0f0c36f26/app/lib

• A Form Builder
• Some POROs: an API wrapper; some utility functions (e.g. PhoneNumberFormatter); a value lookup (ZipCodes); a utility for seeding data for demos (Seeder, I don't remember why we put it here rather than seeds.rb, but I assume that was for testability).
• ApplicationResponder. The responders gem says to put it in lib and then require it 😞

Another common item I found in some projects is a pagination object that is somewhere between data and presentation.

Here's a much larger project that I didn't work on, but has a fuller app/lib:

https://github.com/codeforamerica/vita-min/tree/2f5faf06f586d1ea3915af262aab7683240f4727/app/lib

Also Mastodon:

https://github.com/mastodon/mastodon/tree/c62604b5f69c3ad7f5449e0a7dc26606adebb777/app/lib

app/lib is really quite common:

https://github.com/search?type=code&auto_enroll=true&q=path%3A%2F%5Eapp%5C%2Flib%5C%2F.*%5C.rb%2F

Everything app specific that's part of the domain model should live in app/models (that directory is for POJOs as much as ARs).

I get that there is a definition of "Model" that means Domain Models, but I don't think it's consistently followed. The Rails Getting Started Guide offers a narrow definition: "A model is a Ruby class that is used to represent data." ...not identities, events, aggregates, etc. I associate app/models pretty strongly with the ORM, but I get that it could be expansive. And I think there is a bunch of stuff that isn't quite a clear Domain Model (or isn't quite yet).

I'm not stuck on the name though. I'd update this PR in a heartbeat if there was an alternative name proposal (app/utilities or app/extras or app/misc). The heart of this is creating a visible place to put things in app that don't fit anywhere else.

Appreciate the follow up. Stuff like a generic PhoneNumberFormatter is exactly what lib/ is intended for. And if it's app specific, for some reason, then app/models is fine. I don't think the trade-off here is worth it. Introducing another default directory, so going to close this out.

@dhh, in the past @fxn has actually recommended creating an app/lib folder as a place to store things like PhoneNumberFormatter because a class like that does not belong in lib.

The rule for using require is simple: You never use require for autoloadable code. And you use it for everything else. You never require "user", but you require "nokogiri", so to speak. In classic mode, the problems derived from using require were documented here.

Nowadays, the recommendation is: If you have stuff to reload that that is suitable for a lib directory, just throw it into app/lib.

-fxn

I've seen this come up in a few Rails Slack channels over and over since Zeitwerk became the default. Most people think that app/models is not the place for it and view app/models as a place for ActiveRecord/ActiveModel classes (I understand you disagree, that's fine, but there's also lots of people that feel this way). So then they put it into lib only to find that they have to manually require it. Then later realize that this also means they now have to reboot their server any time they change the file (after a painfully long debugging time of "why what aren't my changes working?", because their lib folder classes are now second-class citizens). Then they go down the rabbit hole of adding lib to the autoload paths, which burns them because rake tasks then all get eager loaded in production. Then they inevitably realize anything inside app is autoloaded and make an app/lib per Xavier's advice.

lib seems like the wrong recommendation here. Per Xavier, lib is for things like Rake Tasks.

The fact that I've seen this question come up so many times makes me think there's got to be some better solution here than just closing the issue, even if that solution is more documentation. But I'd be cautious of documentation being the solution, as having even more words to read as a newbie makes getting started even more challenging.

(BTW, the app/models directory is there to organize the model layer. The model layer is greater than the persisted models.)

To be honest, I don't quite like app/lib, but I am echoing a recommendation from the team being done in the last years. Is like tossing code into app that does not quite belong in there, just to have autoloading and reloading without extra configuration. Smells to a X/Y solution to me.

At the same time, lib/tasks and lib/assets seem out of place to me. I believe lib should be for Ruby code, and extras should go to a different place. In that case, having lib be autoloadable and reloadable would be a no-brainer.

This is all backwards, because it is to be expected that people want to have those features in lib. Autoloading and reloading is about loading code, not its conceptual organization. And lib was removed from those paths in Rails 3 because of the extra stuff that is in there.

The guide even says:

Idiomatic Rails applications only issue require calls to load stuff from their lib directory, the Ruby standard library, Ruby gems, etc. That is, anything that does not belong to their autoload paths, explained below.

However, we could perhaps evaluate that again.

Zeitwerk ignores anything that does not have a .rb extension. In the early versions, the mere existence of lib/tasks would set an autoload for the Task constant in Object. However, due to these kind of use cases, that changed about a year ago in 2.6.0. So, lib/tasks and lib/assets would be ignored altogether provided they contain no .rb file (recursively).

Still, I feel like those subdirectories do not belong in lib either.

The fundamental problem here is eager loading, you may have generators, or templates, those are .rb files.

Nowadays, in the general case, users need to do this:

config.autoload_paths << "#{Rails.root}/lib"
config.eager_load_paths << "#{Rails.root}/lib"

%w(assets generators tasks templates).each do |subdir|
  Rails.autoloaders.main.ignore("#{Rails.root}/lib/#{subdir}")
end

The real fundamental problem is that lib is a kitchen sink, and all we see is ramifications of that root issue. Throwing app/lib is super cheap and works, that is why this has emerged.

The fundamental problem here is eager loading, you may have generators, or templates, those are .rb files.

Nowadays, in the general case, users need to do this:

config.autoload_paths << "#{Rails.root}/lib"
config.eager_load_paths << "#{Rails.root}/lib"

%w(assets generators tasks templates).each do |subdir|
  Rails.autoloaders.main.ignore("#{Rails.root}/lib/#{subdir}")
end

The real fundamental problem is that lib is a kitchen sink, and all we see is ramifications of that root issue. Throwing app/lib is super cheap and works, that is why this has emerged.

Yeah, I agree that lib (or app/lib) is generally a "kitchen sink" (I usually use the term "junk drawer").

I can't help but feel like lib is the commonly suggested folder for classes like this simply because it's what we've been suggesting since the beginning of Rails (I worked on a Rails 1 app for many years and that was the suggestion back then). If application code lives in app, then doesn't that imply that things in lib (such as PhoneNumberFormatter) are not application code? I think that's one of the reasons why your recommendation of app/lib felt right to me -- my classes feel like they belong in app somewhere.

That section of code you provided there sure does feel like a lot of configuration for a framework whose mantra is "convention over configuration"... I'm not necessarily saying that Rails should include an app/lib folder by default. But I do question why lib and not something in app is the common suggestion for classes/modules who do not fall into the default set of folders (models, controllers, jobs, etc). Is it just because it's what we've been doing for so long? To me feels like we're trying to shoehorn the lib folder into further being a kitchen sink (now holding rake tasks and miscellaneous classes), rather than just saying "your Ruby classes/modules go somewhere in app because they're application code".

But I do question why lib and not something in app is the common suggestion for classes/modules who do not fall into the default set of folders (models, controllers, jobs, etc).

That is not the criteria! In Rails applications, having more folders than the ones generated is normal. Projects have app/services, app/presenters, app/api, app/forms, app/queries, ..., you name it.

The criteria for code in lib is what @dhh said above:

non-app specific library code that just happens to live in the app for now (usually pending extraction into open source or whatever)

I want to lean into what @fxn wrote of "Smells to a X/Y solution to me."

If I approached this from the Y-side, that would mean to me "what would allow autoloading lib/ by default?"

Going by the intended purpose of lib/ ("pending extraction"), I think lib/assets should remain (as those are assets pending extraction).

That simply means lib/tasks would need to be rehomed. Some options:

• Move to the project root: tasks/. I'm not the biggest fan of adding more top-level directories
• Create a new top-level directory to contain tasks/ as well as anything else that would not be autoloaded and add that to the LOAD_PATH. I don't imagine this anyone's favorite option based on the feedback so far and I'm stuck on the naming.
• Move it into config/tasks/. I think config/ could be a good home for tasks. It's outside the autoloader and config also already has many *.rb files that are explicitly loaded by the framework.

Is there anything else, other than tasks that would need to be rehomed to allow lib to be autoloaded by default/convention?

@jasonfb I've hidden your comment as off-topic. Next time you "realize [you] sound rather X-ist", maybe reconsider writing the rest of the comment.

@natematykiewicz

That section of code you provided there sure does feel like a lot of configuration for a framework whose mantra is "convention over configuration"...

Agreed, the fact that you need to write that code today, or the emergence of suggesting app/lib (non-app code in app!), are all symptoms of a root issue that remains unsolved. The trick, for me, is how to address this in an elegant way.

For autoloading to be round in Rails, I believe this is pending.

I've been thinking a lot about the idea of autoloading lib that you guys are suggesting. Initially I thought Rails basically shipping Xavier's config as the sane default seemed to solve all the problems, but the more I thought about it the more I didn't like it. Having to explicitly ignore certain folders feels like it's going to cause an inverse problem. Instead of people having troubles getting files inside lib to autoload, the new problem will be getting folders inside lib to not autoload (as they did for years). It feels like we'll be chasing our tails on the configuration there and have a fairly fragile "fix" on our hands.

I feel like explaining the rules of which lib subfolders autoload and which don't will cause a lot of confusion. And I think moving folders like lib/tasks will cause some migration (and tooling?) problems, especially with gems that generate rake tasks.

The current autoloading rule is simple and predictable. app autoloads, and lib (or anything else really, I think) does not.

If the goal of lib is:

non-app specific library code that just happens to live in the app for now (usually pending extraction into open source or whatever)

then I feel like lib should not autoload. Gems are not autoloaded into an app, they must be required (granted, bundler helps with this). Gems have to require their own Ruby files (or use autoload, or install and use Zeitwerk themselves). Gems also do not have their contents reloaded on change (apps must be rebooted on bundle update). So if the goal is "code that will be cut/pasted into a gem soon", then keeping Zeitwerk away from lib seems to create the easiest experience for migrating the code to a gem.

I think the real question then becomes:
Where do Ruby classes, who I can't find a meaningful folder in app (and don't particularly want to make 50 new folders) go? Does PhoneNumberFormatter need to go in an app/formatters folder as the only formatter for the foreseeable future, or is there a better place for miscellaneous uncategorized classes? This is the problem that I believe this PR is trying to solve.

I don't think lib is the correct solution. I think moving it to lib (indicating a pending extraction) with no immediate plan for extraction is a premature move and in my experience without a clear plan of making and maintaining the gem, the extraction to a gem never happens and lib is where the code lives for years to come.

As an aside (which feels relevant to the conversation), I know you mentioned some recent documentation changes about autoloading. Is there a way to view a diff of a rails guide across versions? I don't believe there is, and I think that could help a lot about autoloading confusion and just the framework in general. If I've read all of the rails guides before, I don't particularly have time to read the entire thing again every new minor rails version to try to mentally diff the latest version to what I remember reading last. Ideally I could just see a diff (similar to a git diff) to see what sections changed in all the guides. Sometimes guide changes are subtle and sometimes whole sections are added / reworded. I think this would help make sure that the guide updates are read by people who have already read the guide before.

Hey @natematykiewicz, let me reply to your points summarizing them for brevity:

I think moving folders like lib/tasks will cause some migration (and tooling?) problems

Agreed. It is also worth noting that lib/tasks typically has application-specific tasks, thus not fitting into the condition for lib. Which makes me question the criteria for lib, because Rails itself violates it.

If we revisit the definition of lib, then doors about what goes where open.

Gems are not autoloaded into an app

This is no longer the case, more that 400 gems use Zeitwerk today. It is not unsual to prefer the Rails autoloading experience to writing manual require calls. And now you can have it.

Where do Ruby classes, who I can't find a meaningful folder in app (and don't particularly want to make 50 new folders) go?

The way I see it, you don't need an ontology that perfectly partitions all your classes and modules. In edge cases that don't quite fit you can relax the rules and throw them in app/utils, or lib, for example. I personally put them in lib most of the time.

Is there a way to view a diff of a rails guide across versions

Yes. In the particular case of the autoloading guide, the Git history of the file is quite readable, since things have converged quite a bit. We are in last-mile polishing phase and there are only a handful of commits.

Also, compare with GitHub, or with the command line client:

git diff 7-0-stable main -- guides/source/autoloading_and_reloading_constants.md

Indeed, documenting in the guide how to properly autoload from lib is certainly one of the options.

Thanks for sharing your thoughts @natematykiewicz!

I am leaning into this interface, which acknowledges lib is special due to its history and what goes there:

config.autoload_lib

This does the following:

1. Puts lib in the autoload paths.
2. Puts lib in the eager load paths.
3. Ignores lib/assets and lib/tasks.

If you have more stuff to ignore, you'd be able to pass it:

config.autoload_lib(ignore: %w(assets generators tasks))

That would solve the vast majority of existing apps needs.

And if you need something very custom, you can grab Rails.autoloaders.main, the common interface for the common case does not limit you in the exceptional case.

Now that I think, that is another thing I dislike about app/lib.

The subdirectories in app have a name that describes their contents, but app/lib means nothing. So, app/models and app/lib are at different level of abstraction, and that feels wrong to me.

Compare app/lib with app/utils, app/extras, or whatever, now that is aligned with the existing naming at your level.

I think you can further bucket code living in lib/ to be code you want auto reloaded in development / test or not. Chances are a generic phone number function or xyz function IS something you want reloaded in development because that's a good user experience. You want to be able to quickly modify and test this code just like anything in app/.

For example let's say I make discount code generator. It's a 4 line function that's not super related to my app so I may decide to put that in lib/generate_discount_code.rb, but while I'm building my app that references this module I do want to be able to modify the function and reload either my browser or IRB session and see the new code change. Doing that and then dropping a require "./lib/generate_discount_code.rb" into application.rb isn't enough to pickup changes in that code.

There's also the case of wanting to refactor these files. If I add a new file and need to modify application.rb to require it, that means I need to restart my Rails app. It's not the end of the world but it can be a deterrent when you're in the flow and want to quickly refactor the names of things.

@nickjj totally.

@fxn if you can pull it off, I think autoload_lib would solve a lot of problems. I would suggest that rails new adds config.autoload_lib(ignore: %w(assets generators tasks)) to config/application.rb, and the rails upgrade task also adds that line but commented out. Then this could be seen as the sane default, but be easily changed to fit the needs of the app.

Having it uncommented for new apps would help encourage people to adopt autoloading of the lib, and having it add it commented in an upgrade would preserve backwards compatibility, while also encouraging adoption on an opt-in basis.

Three quick contributions here.

1. The biggest issue I see with the current lib behavior is the lack of autoloading not being obvious. A 100% compatible version of handling this would be like migrations - rails could check if the directory has changed since the process started and show a similar error screen to if there is a migration that has not been run.
2. I think the place that really does not fit anywhere is monkey patches - be they via opening the class in a new file or a module that extends it. I have ended up with them in lib and initializers at various points and like neither. I do feel like app/lib would be the right place for that.

I will also point out that inconsistent autoloading behavior of the root lib directory alongside the tasks being there already is making that folder even more confusing. I already can't tell my team what that folder is for.

If Rails starts partially autoloading that, I would do it with a guideline towards where non-autoloading stuff should transition to like the examples above instead of saying lib will be partially autoloading forever.

@nhorton thanks, let me reply to some:

The biggest issue I see with the current lib behavior is the lack of autoloading not being obvious.

We all agree it would be less surprising if both app and lib were autoloaded. It was that way until Rails 3.

But in Rails 3 there was no way to skip subdirectories. So, generators, templates, and all kind of stuff was executed in production due to eager loading. So, reality said lib out of autoload paths was a better default.

The fact that lib is not autoloaded is explained in the Introduction section of the guide:

so, in this case, I believe we cannot play the "obvious" card. You have to learn what are the autoload paths by default in Rails.

I think the place that really does not fit anywhere is monkey patches ... I do feel like app/lib would be the right place for that.

I normally have them in config/initializers or in lib, because normally:

1. The file name does not follow the conventions.
2. Monkey patches have to be executed only once.
3. In particular they normally cannot be reloaded.

I already can't tell my team what that folder is for.

Totally agree, the lib folder is ill-defined. Even the starting guide says about lib: "Extended modules for your application." That description just does not match the structure of a vanilla generated app. This needs to be addressed.

If Rails starts partially autoloading that, I would do it with a guideline towards where non-autoloading stuff should transition to like the examples above instead of saying lib will be partially autoloading forever.

lib has to be partially autoloaded forever for several reasons:

1. Assets are not autoloadable.
2. Rake tasks are not autoloadable.
3. If you have generators, they should not be executed when eager loading.
4. Rake tasks and generators are loaded by Rails (not autoloaded) on demand.

So, while lib is the way it is (difficult to change), the configuration should ignore lib/assets, lib/tasks, and lib/generators. And documentation should explain when to ignore anything else.

It it sounds complicated, you are now getting why is lib not in the autoload paths by default.

Thanks for the quick reply, @fxn.

Let me ground my next remarks in context: I was the technical founder for a company in the Rails 2/3 days that was one of the bigger Rails shops for a while, then went away from the code side of things for a while. I am now back in code, having pivoted my current company from a JS-and-Python-heavy stack over onto Rails.

What I find is that the biggest challenge is getting some great engineers who are comfortable in those languages over onto Ruby and Rails. The biggest complaint I hear is folks not understanding Rails magic. The core issue sometimes being legitimate technical stuff, but often just being from Rails being a big beast of a system, and folks feeling like they won't ever get to the expertise they had in other languages / platforms.

I find myself constantly trying to explain principles of Ruby and Rails to folks. The best always ends up being the classic of Convention over Configuration. But where it breaks down is when it is more "lore" than anything. This is because it makes engineers feel they won't ever ramp on all the lore. I get that. Python as a good example is a really simple language, and the frameworks there are really simple. It means that it takes you 5x as long to do anything in it, but as an engineer you feel you know what you are doing and are competent.

Right now, I can actually explain lib pretty well as "its for the code assets you don't want auto-reloaded like tasks". While not 100% accurate, it is close enough that folks can operate with that. However, there is annoyance that happens when they forget that fact and start coding there and not realize the changes are being picked up.

If half of lib starts auto-loading, my best explanation for it at that point is "yeah - it is basically half the stuff you don't want autoloaded and half app/lib but it is not called app/lib because DHH does not want app/lib". If autoloading lib happened and there was a path where tasks and similar started moving out of it, so that it was just the basically app/lib thing, it would similarly be clear, even if there is a period of transition where it is the combined state.

Thus the reason I mentioned the migration-like behavior is because it addresses the biggest issue - engineers getting surprised when their code does autoload - without any compat or "clarity" implications. The Django comparison here is pretty apt - Django auto-migrates tables as you edit your model file. Rails makes that a separate step, for good reasons, but it is experiencially almost as easy as Django because it tells you visibly as an engineer when you missed a step, and lets you one-click run the migration. The Rails approach here is simple, consistent, and friendly.

One more example of a simple approach to this that might help a lot too is add a PORO generator. It could be incredibly basic - rails g poro MyClass yields

class MyClass
end

But by doing that and landing the file in the app/models directory, it would make it clear that was the intended location instead of lib.

But I want to restate the odd core assertion I have: my biggest struggle with Rails as an eng leader today is hearts and minds, and clarity beats functionality on that respect.

Right now, I can actually explain lib pretty well as "its for the code assets you don't want auto-reloaded like tasks"

You'll agree Rails doesn't need to be designed after a definition you made up yourself, right?

lib is what it is, a directory that since Rails 3 is not in the autoload paths by default. Though you are free to add it to the autoload paths.

The previous autoloader did not have the ability to ignore stuff, the current autoloader can do that, so I believe it is feasible today to encasulate autoloading lib in one call, and can be done robustly for the user.

Regarding magic or framework size, when I was CTO of a company and someone joined, the first thing they had to do was to spend two weeks reading AWDwR from cover to cover. Once you know the framework, there is no magic, only defaults that act as a catalizer and let you fly.

Hey @fxn can you share what is AWDwR??

Hey @fxn can you share what is AWDwR??

https://www.pragprog.com/titles/rails7/agile-web-development-with-rails-7/

Followup here: config.autoload_lib (#48572) and config.autoload_lib_once (#48610) provide API to easily autoload from lib in one shot. They will ship in 7.1.