Repository navigation
/nix will not be writable on macOS Catalina #2925
Description
Activity
Realistically, this may force us to drop support for macOS. Using a different store location for macOS would require a separate binary cache and a separate Hydra instance, and there would be no guarantee that the new location wouldn't break in the future.
I did notice in the talk that it's still possible to make the system volume writable, though not in a persistent way.
Reacted by Luka Teras, S, Joseph, vyp, 49, Luginbash, QuantumGhost, Poscat, ajs124 and JacobsinReacted by Drew Hess, Kirill Elagin, Jan-Philip Loos, Aleksandar Topuzović, forficate, Alberto Valverde, jkachmar, Parnell Springmeyer, Daniel Smith, Matthew Planchard and 115 moreReacted by soc, toastal, John Ericson and GrimReacted by Andrew Childs, DzmitrySudnik, Kirill Elagin, Michael Glass, Jasper Woudenberg, Jonathan Merritt, Robert, Alberto Valverde, Allan Odgaard, Ryner Reinhardt and 26 moreSpeaking for myself: I would be quite sad if macOS support would go away.
I will file a bug with Apple and see what happens. Cannot hurt.Yes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.
Reacted by Jordan Stewart, Lîm Tsú-thuàn, Matthias Debernardini, nop, Neyts Zupan, Alexander Tesfamichael, jmday, Alex Shyshko, Kostas, mdpt and 19 more@edolstra Before canceling support, we can ask the community if it's willing to host its own binary cache and Hydra instance, just as we've enlisted a host of volunteers to keep nixpkgs at near parity on Darwin. Having the answer be, "This doesn't easily fit our model so we won't support Mac" would be an unfortunate response.
Reacted by Drew Hess, Bernardo Meurer, DzmitrySudnik, Robert, Aleksandar Topuzović, Souvik Sen, Parnell Springmeyer, David Johnson, Daniel Smith, Fatih Altinok and 61 moreReacted by ajs124I think no such decision should be made lightly, but it is definitely a drastic change on Apple’s part.
Reacted by Jordan Stewart and JamesYes, you can jump through hoops and get a writable system volume, but only until the next reboot, so this is not a useful option.
Maybe you can use this to create a
/nixsymlink to another volume, which would hopefully persist across reboots?Reacted by eftychis and SymontyThat was my first try to get it temporarily working again. But
nix-envcomplains then:error: cannot open connection to remote store 'daemon': the path '/nix' is a symlink; this is not allowed for the Nix store and its parent directories
If that error can be safely ignored, I think this would be a workaround worth exploring. The way I understand things, anything you place in the root folder would persist until the next system update.
There's no real reason for the nix store to be on the system volume, with apfs it's easy to create a separate volume and mount it to
/nix. But perhaps there's something I'm missing because I don't understand the need for these firmlinks, the equivalent is possible with mounts similar to how the store is mounted readonly on NixOS.@edolstra Thanks for the tip, this is a useful workaround.
But future Nix-users on macOS still need to find a reliable way to inject this symlink into the read-only root directory. This probably involves some advanced steps like rebooting into recovery mode after every system upgrade.
@LnL7 The same would be true for
/nixas a mount point, because we would need to create this directory on the read-only volume.Sure but if Apple doesn't want to provide some sort of solution for one of these cases there will be no alternative to changing the default prefix. Which is arguably is not worth it for a secondary platform that's continuously moving further away from UNIX.
Is there an overview of the full volume layout somewhere?
/usr/localand/Usersare used as examples but I can't imagine that's everything.@mroi How have you been testing this? In my testing, I was able to install nix, but I assumed it was due to the comment in the presentation that "system volume is writeable in the developer preview". I'd also like to help test, but was able to get nix installed:
kevingriffin@KevinnoiMac /nix % uname -a Darwin KevinnoiMac.local 19.0.0 Darwin Kernel Version 19.0.0: Fri May 24 17:36:10 PDT 2019; root:xnu-6041.0.0.111.5~1/RELEASE_X86_64 x86_64 kevingriffin@KevinnoiMac /nix % ls /nix store var kevingriffin@KevinnoiMac /nix % nix-env -iA nixpkgs.hello installing 'hello-2.10' these paths will be fetched (0.02 MiB download, 0.07 MiB unpacked): /nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10 copying path '/nix/store/c4w9z1kzzkdsgvgr6cy9ggl39s2yzn70-hello-2.10' from 'https://cache.nixos.org'... building '/nix/store/iwcsfsh5hxyzpymnkp1r54nb3zzpd7mg-user-environment.drv'... created 2 symlinks in user environment```It looks like you can still access the
/nixdirectory after the update, it's just in/System/Volumes/Data/nix. Too bad there's no chroot equivalent on macOS.Reacted by Jordan Stewart and nopsudo mount -uw / && sudo ln -s /System/Volumes/Data/nix /nixworks for me! This is on SIP even! Having the installer handle this might work.Reacted by Daiderd Jordan, Melody, diam, bb010g, nop, Jinxuan Zhu, Pascal Weiland and Shintaro Abe323 remaining items
So what is the problem with
/opt/nixor/usr/local/nix/instead of/nix? This seems like a real showstopper bug...@samuela it means either doubling the storage for Hydra, or changing ALL the Nix installs everywhere.
One hacky option is to rewrite store paths as they get downloaded. So any mention of
/nix/store/...namewould be rewritten in the file to/opt/nix/...(name-4chars). This would work for most of the files, but still needs support in nix-store and all over nixpkgs where /nix is hardcoded@alper any package that Hydra produces would have to be produced twice, once for /nix and once for /opt/nix.
It's not quite a doubling, because not all packages are built for Darwin and files that don't reference store paths would likely be the same and can be hardlinked.
I don't know how big the Hydra storage is.
Seeing as this issue has been open since 2019, it seems as though macOS is no longer a supported platform. Is that true? Or is there a roadmap to fixing this?
Because shouldn’t storage be a non-issue at this point in history?
I'm going to lock this until we're able to get a fix out. Thanks for everybody's contributions and help in figuring out this problem, hopefully it is soon.
- locked as too heated and limited conversation to collaborators
on May 8, 2020 @wmertens We wouldn't need to produce packages twice, since we'd have one Hydra instance (and binary cache) for macOS (using the /opt prefix) and another instance for all other platforms.
The main consequence of using a separate prefix for macOS is that you can't have Hydra jobsets anymore containing jobs for macOS and Linux. It would also make it harder to deploy from macOS to Linux.
- pinned this issue
on May 19, 2020 Documentation for installation of Nix on Catalina is available.
Note that until Nix 2.3.5 release is out, the instructions won't work. PR
Nix 2.3.5 has been released with Catalina fix!
- unpinned this issue
on May 28, 2020
This is not a short term bug, but it will become an issue when macOS Catalina is released this fall. macOS is now split across two volumes (system and data) with a read-only system volume. This means that
/nixwill no longer be writable.Some more information can be found in the related WWDC talk and some session notes people took from a Q&A.
Summary: the system volume, which is mounted at
/will become non-writable. Some directories that need to be writable are connected via firmlinks (an Apple invention) to the data volume./nixis not among these locations, so with the release of macOS Catalina, this location is no longer an option.I see two possible solutions:
/nixas a firmlink to a writable location. I think this has limited success and Nix would then depend on Apple to not drop this link in a future release./usr/localand/opt, so we could move to/usr/local/nixor/opt/nix. I would hope that these locations are common enough so that Apple would not drop them in the future.I wanted to raise this issue early, before it becomes a problem for users. If this issue tracker is not the right place, please feel free to move this discussion elsewhere. I would also be available for testing any potential solution, since I have access to a macOS Catalina beta.