Jordan Crawford

Migrating from Evernote to Obsidian

2025-11-16T00:00:00+13:00

I’ve been using Evernote since 2013 and I had over 1,800 notes stored. Evernote significantly increased the price of my subscription so it’s finally time to move to an alternative.

I started using Evernote as a poor university student - it was a free option and was far more powerful than alternatives like Apple Notes and more flexible than OneNote. Eventually as my number of notes and devices grew I upgraded to a paid plan which steadily increased in price, particularly after being acquired by Bending Spoons. While Evernote got worse, everything else got better - Apple Notes got better and many great new alternatives popped up - like Notion, Obsidian and Bear.

The key things I wanted in a note taking app are:

Backup - Evernote had no automated note backup solution so I had to manually backup my ENEX files. I’d like something that I can easily backup to ensure I don’t lose my notes if the service goes away.
Fast and reliable sync - Evernote’s syncing worked pretty well and generally handled conflicts gracefully, so I want something which works just as well.
A mobile app - I want access to my notes on the go.
Offline access - I want to know that I can always access and change my notes even if I happen to be without phone signal or having an internet outage at home.
Data portability - Moving notes between formats can be a pain, particularly when you have a lot of them. I want my notes to be stored in an easily portable format.

Why Obsidian?

I chose Obsidian because it uses a simple file/folder structure with notes stored in markdown. This let’s me structure my notes in any way I want and they remain extremely portable, are easy to sync & backup, and always available offline.

Obsidian starts very simple - more like an IDE for authoring markdown notes, but the more you dig into Obsidian you’ll see there’s a large community with heaps of plugins.

If you want something which is more ready to go out of the box, Bear looks like a good option, but I haven’t used it myself.

Figure out how you’ll use it

I spent a few weeks just using a fresh Obsidian vault with my older notes still in Evernote. This helped me figure out how I want to use it before committing to migrating all my notes.

Attachments

Since Obsidian uses a standard file/folder structure so files attached to your notes exist somewhere in Obsidian’s file structure. By default it just dumps them in your main vault folder. This might be what you want, but for me this would be absolute chaos.

I installed the Custom Attachment Location plugin as it lets you specify a custom naming scheme for your attachments.

Personally I’ve set my new attachment location to ./Assets/${noteFileName} as I like to have an Assets folder in the same directory as the note with a folder for each note’s attachments. For example:

The best things about this plugin is:

It has a command to “Collect attachments in entire vault” - this will tidy up all my existing attachments to match these settings.
As you move notes or rename them, it’ll automatically move the attachments to match the rules.

Encrypted notes

I had a few encrypted notes using Evernote’s password protection.

I had a look around for options to store encrypted notes, but ultimately decided that Obsidian likely isn’t the best place for them.

My existing encrypted notes fell into two categories:

Passwords/secret keys E.g.: server passwords and SSH keys

None of these old secrets were relevant anymore, and going forward these all belong in my password manager.
Personal notes E.g.: health information.

I decided to move my personal notes to Apple Notes instead. Apple Notes can encrypt the entire note (rather than just a password protected region) and it’s more convenient to use Face ID than typing a password.

Note browsing

_{The note browsing UI from Notebook Navigator}

Evernote groups and previews files sensibly by default - I liked that I could click into a stack of notebooks and see all the notes all together, organised by time.

Out of the box, Obsidian’s note browsing is essentially a simple nested file browser with some support for fancy graphs to find notes by links and tags.

I installed the wonderful Notebook Navigator plugin which gives a much more familiar note browsing experience.

In particular I like that I can:

Click anywhere in the file hierarchy and see all notes inside that folder.
See previews of note content and attachments while browsing (I recommend enabling the Dual Pane layout on desktop).
See counts of the number of notes in each folder.
Group notes by time period - this avoids me going over the top with file grouping.

Structure

Evernote had a fairly limited hierarchy with Notebook Stacks and Notebooks. Obsidian opens that up a lot more with a virtually unlimited hierarchy, giving me far more options for organising my notes.

Note structure is a massive topic with many differing opinions - ultimately it’s about finding what works for you!

My approach

I’m all about files and folders - I like everything to be long in one place rather than having multiple homes with tags. Tags can be helpful to cut across the structure but they aren’t my primary way to organise my notes.

The guidelines I try to follow in my structure are:

Use a consistent high level structure between my cloud storage and Obsidian.
- e.g.: Career, Finance, Personal, Tech, Projects.
- Quite often, the projects I write about will have corresponding files in my cloud storage so it makes sense to try keep these as aligned as possible.
Keep notes with other notes about the same topic.
- e.g.: All notes related to my Mortgage sit in a Finance/Mortgage folder.
- This means I can revisit all the notes about this topic in one place.
- Sometimes it’s harder to get more specific - it’s fine for things to belong in a top level folder.
Where there are many notes about a topic, organise notes by year.
- e.g.: I have quite a few notes about my Homelab projects so these go in a Tech/Homelab folder organised by year.
- Notebook Navigator’s grouping means I don’t really need to do this, but I find it helps my sanity a little.
Use an “Archive” folder for old things.
- e.g: For old tech notes I have a Tech/Archive folder arranged by year.
- I had a lot of old notes that I wanted to keep but weren’t particularly relevant or didn’t fit into my structure. An “Archive” folder gives these notes a place to exist without feeling like I have to fit them into the structure.
Ensure everything has a place.
- My Evernote notes would end up in a “Unsorted” folder by default. Over time this meant I had quite a lot of notes in the “Unsorted” folder.
- Going forward, I’ll aim to find a more specific place for my notes - if I can’t, that’s a sign a the system isn’t working.

Dates

The created and modified dates on notes are important to me as they allow me to browse notes by time. Most of my notes are time-bound with only a handful which are evergreen.

By default, Obsidian is simply using the file created and update times as the note for a date. You can put date metadata in the note Frontmatter, but I haven’t bothered - I like the simplicity of one place for this metadata, and so far it hasn’t caused any problems.

If a note is time based, I tend to put the date or month in the note’s title (e.g.: “2025-09 Evernote to Obsidian migration”) but this is more of a convention I employ when appropriate rather than a hard rule.

Migration

You have two options when migrating your data from Evernote to Obsidian - Obsidian’s Importer or Yarle. It turns out that Obsidian’s importer is actually just Yarle under the hood but with a few sensible default’s set.

I went with Yarle as it gave me more configuration options but you’re gonna get very similar results with either option.

As I had quite a few notes, I identified a collection of more complex notes to check after the migration to see how they behaved. Some examples of these are:

Notes with lots of bullet point lists.
Notes with embedded links.
Long notes.
Notes with weirdly formatted content
- E.g.: Code notes created from a time before Evernote played nicely with code.

I did a few test migrations with both Obsidian’s importer and Yarle. Both tools did a great job with only minor formatting problems like extra spaces.

_{The Yarle UI - yes, Yarle stands for Yet Another Rope Ladder from Evernote}

Yarle template

Since I didn’t need any date information in the Frontmatter, the only Frontmatter I wanted imported were my tags, and I only wanted the Frontmatter section to exist if the note had a tag.

This is the Yarle template I used to achieve this:

{tags-yaml-list-block}---
tags: {tags-yaml-list}
---
{end-tags-yaml-list-block}{content-block}{content}{end-content-block}

Sync

There are so many options available for Obsidian syncing - Obsidian has a page on their website explaining the options.

I use Obsidian Sync (their $4 USD/mo first-party sync option) has been a smooth experience - it syncs changes to notes and settings incredibly quickly.

My only concern with Obsidian Sync is their Standard plan only supports files up to 5mb and 1gb of files in total. If I approach this limit I’ll need to consider alternative options such as iCloud (not supported on Windows/Linux) or Git (more complex on iOS).

_{Obsidian's pricing information, from obsidian.md/sync}

Backup

I like to have automated backups of my files to ensure that in the worst case scenario if my cloud account is deleted I still have a copy of my notes. In practice, my notes all sit on multiple devices including my computer which is backed up, but I still wanted to have an automated sync just in case.

I considered setting up a headless instance of Obsidian using the docker-obsidian Docker image (inspired by this blog post from @rolle). However, ultimately I decided this was just too complex and likely too prone to failure.

Instead, I decided to use the Obsidian Git integration to automatically sync my notes to GitHub from my computer. The great thing about this is it retains all my file history so I can recover notes I’ve accidentally deleted.

As this relies on my computer being used and having the Obsidian app open, I setup monitoring with healthchecks.io to ensure I’ll be notified if my note backup hasn’t hit it’s success ping in the last two weeks.

_{My (private) GitHub repo for notes}

Git backup setup

Create a Git repo to store your notes.
Install the Git plugin for Obsidian.
See this guide to configure the integration. I connect to my remote repo through SSH, just like other code projects on my computer.
Ensure the Git plugin is only enabled on your main device.
- In the “Sync” plugin I have disabled syncing for “Active community plugin list” - this lets me switch on and off plugins for individual devices.
Important - set the “Merge strategy” to “Other sync service”
- This ensures that it’s only used to backup notes and can never overwrite changes in my vault.
I set my “Auto commit-and-sync interval” to 10 minutes and disabled the auto-commit after stopping file edits.
I configured a simple GitHub Action which runs on each commit and hits healthchecks.io. This ensures I’m notified if my notes aren’t backed up to GitHub at least once a week

Conclusion

_{Photo by Atwena Goodman on Unsplash}

I’m happy with my decision to switch to Obsidian!

It’s cheaper than Evernote and my notes remain extremely portable in a markdown format. I have new ways to organise my notes with a flexible file hierarchy, and my notes are finally backed up automatically.

Migrating from Evernote to Obsidian was originally published by Jordan Crawford at Jordan Crawford on November 16, 2025.

Reading a SVCD disc from 2005

2025-09-16T00:00:00+12:00

When I was about 10 years old I was getting into the first of many little projects on the family computer. I used NeroVision Express 2 to create a Super Video CD (SVCD) containing a collection of family photos, excessive transitions, obnoxious menus and (unbeknownst to me at the time) inappropriate music choices.

_{NeroVision Express 2 menu creator tool (original source)}

I’ve been sorting through a lot of family photos lately so I came across this old disc and I wanted to access it.

What are SVCD’s anyway?

Super Video CD (SVCD)’s are an improved version of a Video CD (VCD). VCD’s and SVCD’s are an early format for video playback using a standard compact disc (CD) prior to Digital Video Disc (DVD).

Why make a SVCD in 2005 - who knows! But SVCDs could be played back in many DVD players of the era so compatibility wasn’t a major concern.

Initial playback attempts

I popped it into my USB DVD drive connected to my MacBook - no DVD Player popped up so I tried VLC Player through the “Open Disc…” option. All I got from VLC was an error message (I later discovered it does actually play the video content in VLC on Windows but the “Starting Position” “Entry” option needs to be set above 1).

Fortunately NeroVision wrote a bunch of HTML pages to the disc, including the original image files. This was great, as it meant I had backups of the original files (including some I was missing).

However I still wanted to experience the nostalgia of my poor choices from 20 years ago.

I tried to copy all the files off the disc from multiple different computers but this would always freeze or display an error while copying the MPEG or SVCD files off the disc (but the HTML files were always read perfectly).

I attempted to make an image of the disc on my Mac in Disk Utility but this also failed. I even cleaned the disc multiple times but the result remained the same - I believed the disc to be toast.

I decided to give it one last shot - I read about dvdisaster in this Reddit thread. When I attempted to read the disc using dvdisaster in Windows I got a very interesting pattern of read errors:

This seemed far too consistent to be caused by a decaying or scratched CD - I realised I must be doing something wrong…

Understanding how SVCDs work

It turns out the disc is made up of multiple tracks written in different modes - from SVCD on Wikipedia:

The first track is in CD-ROM XA Mode 2, Form 1, and contains metadata about the disc. The other tracks are in Mode 2, Form 2, and contain audio and video multiplexed in a MPEG program stream (MPEG-PS) container.

I came across a tool for Windows called IsoBuster. The free version let me see the data and all the other tracks. There were 7 tracks in total -

Track 1: Data content (the HTML files and seemingly fake MPEG files) and menu
Tracks 2 through 7: Video files 1 through 6

Exporting content

Raw Binary Export

IsoBuster let me export the entire disc as a raw binary file (a .bin and .cue file). I right clicked on “Session 1” and selected “Extract Raw Data”.

In theory I could use these files to burn a fresh disc in future.

Interestingly I can open the .bin file in VLC and it’ll play the video tracks perfectly fine - though unfortunately the menu just plays as a video track rather than a working menu. Perhaps we shouldn’t be surprised - VLC’s wiki page on SVCD says:

SVCDs often do not work in VLC Player. VLC can sometimes play SVCDs using libcdio and libvcdinfo - the features page has more details. For a complete VCD/SVCD which includes menu selections, hot spots and playback control, see xine.

Exporting the MPEG content

There’s another option in IsoBuster which lets you export the tracks as MPEG files - the “Treat as Video ONLY, Extract but FILTER only MPEG video” option.

This gave me a pile of 7 MPEG-2 files which open fine in VLC. I converted these to modern H264 files in Handbrake making it easy to replay whenever I want.

Playing the SVCD in it’s original form

I still wanted to play the SVCD in it’s original form with working menus.

I didn’t find a good options for macOS but in Windows I found CyberLink PowerDVD supports SVCDs but unfortunately this requires creating an account…

Instead, I installed PowerDVD XP 4.0 under Windows XP on my 2008 MacBook for an almost period accurate experience - it works perfectly!

Lessons learned

This was a lesson in having the right tool for the job. My lack of understanding of how SVCDs worked almost led me to believe this SVCD was toast but I was actually just reading it wrong.

It’s nice that there’s still options to play these old discs, but it’s good to have a backup as old CD’s start to deteriorate.

Reading a SVCD disc from 2005 was originally published by Jordan Crawford at Jordan Crawford on September 16, 2025.

My rules for using Time Machine on Synology

2023-09-17T00:00:00+12:00

Since I got a Synology NAS (DS920+) about two years ago I’ve had a few false starts with my Time Machine backup, but I think I’ve finally gotten it right.

What is Time Machine?

Time Machine is the built in macOS backup tool. It launched all the way back on Mac OS X Leopard in 2007 but I feel like it’s lost some popularity in recent years with the prevalence of cloud storage and cloud backup solutions.

Even though all of my important files are in the cloud or on my NAS (sometimes both!) I still use Time Machine because:

It lets me quickly restore my entire Mac if something happened to it - e.g.: if my laptop was stolen I can buy a replacement and quickly restore everything back to normal without spending hours setting everything up again.
It ensures anything I’m currently working on will be backed up - even if I haven’t had a chance to move it to the cloud yet.
I don’t have to worry about putting large files on my desktop. Instead of Time Machine I could use Google Drive’s Computer Backup feature to backup desktop and documents folder. However, sometimes I end up working with large files and I don’t want these syncing to the cloud and I don’t want these files wasting my bandwidth and cloud storage.

Alternatives to Time Machine

It’s worth noting that Synology Active Backup for Business now supports macOS backups. This is a more modern approach, has central management, file de-duplication between computers and can support backups over the internet.

I haven’t used Active Backup for Business personally yet. It’s macOS support is fairly new and it’s more involved to install compared to Time Machine - SpaceRex did a video covering the setup process.

Setting up Time Machine

The Synology Knowledge Base has a good article to help you setup Time Machine on your NAS - Synology Knowledge Center: How do I back up files from my Mac to Synology NAS using Time Machine?

Based on some previous bad experiences while using Time Machine with my NAS, I have a few suggestions to add:

1. Use a separate volume

_{The note in the Time Machine System Preferences pane}

As the Time Machine system settings says - it’ll keep hourly backups for 24 hours, daily backups for a month, and then monthly backups for all time. It’ll delete oldest backups as space is needed. This is handy as it lets you jump back in time to get an old version of a file, however it also means Time Machine will fill to grow the space you give it!

The Synology guide recommends assigning a quota to prevent backups from getting too out of hand and filling your NAS. A quota enforces an upper limit for the shared folder or user so when Time Machine approaches the quota it’ll wipe old backups to keep itself in check.

However, a quota doesn’t reserve capacity for Time Machine - i.e.: it doesn’t enforce a minimum amount of space for your Time Machine backup. This means that if your NAS starts to run out of space, Time Machine will start deleting older backups to squeeze itself to fit within the remaining space.

I had this happen on my NAS, and Time Machine essentially shrunk itself to the point where it only had a few versions backed up which sort of defeats the purpose of the backup!

I believe there are two solutions to this problem:

Don’t let your NAS run low on disk space

Ensure you always have a good amount of free space on your NAS and you won’t run into any issues. The NAS will show a ‘warning’ state if you’re running low on space.
Create a dedicated volume for Time Machine

My solution to ensure the other files on my NAS didn’t eat into the Time Machine space is to setup a dedicated volume for Time Machine, containing only a Time Machine shared folder. This suggestion ruffled some feathers on Reddit so I’d recommend thinking about whether a separate volume is right for you.

Another benefit of this approach is that SSD cache’s are setup against a volume rather than a shared folder. A separate volume allows you to control whether your Time Machine backup is included in the cache - personally I wouldn’t want to waste my cache space on any Time Machine files, particularly with a write cache.

_{My Time Machine backup volume - it only contains a Time Machine shared folder}

2. Don’t enable snapshots or recycling bin

🗑️

The Synology guide also mentions that you shouldn’t enable the recycling bin. I’d also recommend that you don’t enable snapshots on the Time Machine backup shared folder either! Snapshots on Synology (via the “Snapshot Replication” app) allows you to keep previous versions of the files in your shared folder - much like what Time Machine does for your Mac.

Time Machine stores your files as a ‘sparsebundle’. It looks like a big file within macOS, but in reality it’s actually a folder structure containing thousands of little files. If one of these files is changed or deleted it might be retained in a previous snapshot or the recycling bin. This means that Time Machine might try to free up some space by deleting files but no space is actually reclaimed!

I enabled snapshots on my Time Machine backup folder when I setup my NAS initially. I did this because I figured it could help me recover from a corrupt Time Machine backup or ransomware attack. However, much like in 1), I ended up in an odd situation where Time Machine wanted to free up space by deleting some files, but with no space actually reclaimed it’d just keep trying to delete stuff to free up space. This resulted in most of my backups being deleted - again defeating the point of a backup!

So keep it simple - you don’t need snapshots or a recycling bin for your Time Machine backup. If you’re concerned about Time Machine getting corrupted or a ransomware attack wiping it out then you should consider an off-site backup (but maybe a backup of your backup is overkill!?).

3. Don’t worry about adding another layer of encryption

🔑

This is already mentioned in the Synology guide, but you don’t need to bother adding another layer of encryption (either shared folder encryption or volume encryption) to your Time Machine backup as long as you enabled encryption when you configured Time Machine.

Remember that Time Machine stores your files as a ‘sparsebundle’ which is just a bunch of little files. If you’ve got encryption enabled then these little files are encrypted gibberish so you don’t need to waste your NAS’s CPU time by adding a redundant layer of encryption.

So far following these rules have kept my Time Machine running reliably - but it has taken a few iterations to get this right!

Edit, 18th September 2023 I’ve updated this post based on some feedback I’ve received. It now mentions Synology Active Backup for Business as a potential alternative to Time Machine, makes the Synology guide clearer, and adds clarification about using a separate volume for Time Machine.

My rules for using Time Machine on Synology was originally published by Jordan Crawford at Jordan Crawford on September 17, 2023.

Playing co-op Dungeon Siege in 2023

2023-07-14T00:00:00+12:00

When I was a kid, my brother and I really enjoyed playing Dungeon Siege. For the uninitiated, Dungeon Siege is a 2002 action based roleplaying PC game by Gas Powered games.

I recently built a budget gaming PC so I finally had a way to play Dungeon Siege again (it’s available pretty cheap on Steam and GOG).

Dungeon Siege has a co-operative multiplayer mode which my brother and I wanted to try.

Tweaking the game settings

If you’re using the Steam version (potentially the GOG version too) you’ll get a message that says “Disabled” when clicking multiplayer from the main menu.

To get past this you’ll need to adjust the game to launch with the zonematch=true option - you can add these through the Steam properties. See this post for more information.

_{Steam properties for Dungeon Siege with the Zonematch=true option set}

The ZoneMatch option will start the game in the multiplayer screen by default, but if you want to play Single Player you can click “Back” to return to the main menu.

For any other issues you come across, there’s also this helpful GitHub repo with lots of tips for troubleshooting issues with the game.

Getting connected

Dungeon Siege multiplayer has local, internet and ‘ZoneMatch’ modes.

_{The Dungeon Siege multiplayer menu}

I believe ‘ZoneMatch’ is the now shut down matchmaking service - so unfortunately that’s no use to us.

The local mode works great - while on the same home network, one computer hosted the game and it showed up for the other one to join. However, the problem is that my brother and I live in different cities so need to connect over the internet.

The internet mode is pretty simple - with no automatic matchmaking you’ll need to enter the IP address of the game server that you want to connect to.

I tried to get the internet mode working with some portforwarding in my router but I didn’t have any luck. Maybe I configured something wrong, maybe my ISP has something against Dungeon Siege - who knows.

Either way, exposing this 21 year old game to the wild internet didn’t seem like a good idea.

Introducing Tailscale

I haven’t written about it yet, but I use a free Cloudflare tunnel to access services on my NAS when I’m away from home. I run this in a Docker container on my NAS, but a few weeks ago I managed to accidently restart the container while connected remotely and it didn’t come back on it’s own!

So long story short, I wanted another way to access my network while away from home in addition to Cloudflare tunnel. I went with Tailscale which is a free service VPN service (with paid options available) which creates a virtual network which connects your devices to each other. Tailscale is based on Wiregaurd VPN but they take care of all the hard stuff - you just need to install their apps on your devices and optionally setup some access rules.

With Tailscale I can access my NAS from any of my devices using the NAS’s Tailscale IP address. I can also use my NAS as an exit node, letting me tunnel my internet traffic through my home network or to access other devices which don’t run Tailscale - like my IP camera or router.

Gaming through Tailscale

Gaming through Tailscale is really not anything new - it seems like it’s already pretty popular for Minecraft servers. I installed Tailscale on my gaming PC then shared an invite link to my brother.

With a sharing link, my brother setup his own Tailnet and setup Tailscale on his computer. By default, if you share a device, the other person can initiate connections to your device but you can’t initiate connections to theirs. I believe this can be changed in the ACLs, but since I was using my machine as the server he only needed to connect to my PC.

I hosted a Dungeon Siege game through the “Internet” mode and my brother typed in my gaming PC’s Tailscale IP address. It all worked just like we were local - amazing!

Why couldn’t we use the “Local” network mode? I suspect this is because Tailscale doesn’t support multi-cast so the “Local” discovery feature couldn’t send out a message to ask for other servers.

Retaining our progress

Dungeon Siege’s multiplayer let’s you spawn into the single player world, known as the Kingdom of Ehb.

_{The map of Ehb (source)}

Unlike in the single player game, you can’t save in multi-player. Your characters (and their items and levels) will all be saved automatically, but it won’t remember what you’ve done or the enemy’s you’ve killed. Instead, based on your level you can spawn into one of the main areas of your choice when joining the game.

This is generally not a big issue, but it can be kinda annoying. For example, you can spawn in Stonebridge but not Glitterdelve. Stonebridge and Glitterdelve are pretty close together but to unlock the damaged gate between them you need to go through spider infested Wesrin Cross.

If we get part way through Glitterdelve Mine but wanted to finish up for the night, then next time we’d need to go all the way through Wesrin Cross and re-do our progress on the mine again.

Setting up a virtual machine as a game server

A virtual machine (VM) is a virtual instance of a computer that you run within your physical computer. You can pause running VMs (storing the contents of the RAM to disk) and boot them back up as if nothing happened.

My idea was to host the game on a Windows virtual machine -

The game server VM would host the game, leaving a character safely standing at an inn in a nearby city.
My brother and I would connect to the game server VM and play the game as normal.
When we finish playing for the night we disconnect from the server and freeze the VM.
When we come back, we unfreeze the game server, spawn at the nearest spawn point and run to where we left off (with the world exactly as it was previously).

To do this we need to setup a virtual machine which can play Dungeon Siege and which can run Tailscale. I decided I’d run Windows 10 as this is the minimum operating system for Tailscale and it means we can stay up to date with the latest security patches.

Choosing a hypervisor

I considered a few different hypervisor options:

HyperV is hypervisor built into Windows, but unfortunately unless you can pass through your GPU (which I believe requires a more modern GPU than my GTX1070) it doesn’t support 3D acceleration so can’t play the game.
VirtualBox is a popular open source hypervisor available for many different operating systems.
- VirtualBox lets you run VMs in the background (perfect for server scenarios like this)
- VirtualBox supports snapshotting (essentially so you can so you can take a historical backup of the VM to restore to later) in addition to the standard ‘pause’ VM feature of hypervisors.
- VirtualBox can play Dungeon Siege, however I found even the game menus run at such a frustratingly slow framerate that it didn’t seem worth investigating this option too much further.
VMware Workstation Player is a free version of VMWare’s Workstation Pro product.
- Player works pretty well with Dungeon Siege - almost to the point where I’d consider playing the game directly through the VM!.
- Unfortunately, Player doesn’t support snapshotting or running the server in the background. These are nice to have features, but I probably don’t need snapshotting and I can just minimise the window instead of running it in the background.

I decided to use VMware Workstation Player.

Setting up the VM

Setup the VM using a Windows 10 ISO and the install wizard. Remember that nowadays Windows will work fine without a product key short of a few cosmetic things which aren’t an issue for the VM.
Allocate sufficient system resources.

With the bloat of Windows 10 running, I needed to assign more RAM and CPU resources than I assumed I would.

I gave the VM 6 processor cores and 8GB RAM, but I think 3 cores and 4GB RAM would be fine.

(To be honest I’m a bit confused about the processor cores, I only have 4 cores/8 threads but the core options went a lot higher so it might be some sort of virtual core?)

_{The VM setting I used in VMWare Player}
Install Tailscale and share the server to the other person.
Install Dungeon Siege.

My brother and I are using the Steam version - I found that the retail version (i.e.: launch day CDs) of the game and the Steam version didn’t seem to be compatible with each other in multiplayer, so you might want to stick with the same version as your ‘client’ machines.

I installed Steam on the server VM and switched it to ‘offline’ mode before starting the game (so that it wouldn’t complain that I’m trying to open the game a second time on my gaming PC).

Don’t forget the zonematch=true trick from above to get into multiplayer.

_{The servers player, Rory, hanging out in Stonebridge}
Run Dungeon Siege and host an internet game.

Ensure the “Allow players to join game in progress” and “Allow players to choose start locations” options are ticked.
Alt-tab to the desktop in the VM.

I had some issues with the game crashing if I left it open as the active window, but I’ve found it works more reliably when I alt-tab to the desktop. The game server continues to run even with the game in the background.

_{A VMWare Player window running the Game Server instance with Steam open}
Connect to the Tailscale address from your other computers and play the game.
When you’re finished playing, disconnect both players from the server and freeze the server VM. Boot it back up again when you’re ready to play again.

So far, this has worked well for us - we can play co-operatively over the internet and have a way to save our world between sessions.

Enjoy your time in the Kingdom of Ehb!

Playing co-op Dungeon Siege in 2023 was originally published by Jordan Crawford at Jordan Crawford on July 14, 2023.

Escaping Google Workspace

2022-08-13T00:00:00+12:00

I switched my personal account to Google Workspace so I could use my own domain name for my email. I remained a loyal user but the alternatives have improved and the things that kept me loyal have changed. It’s finally time to migrate from Google Workspace to a personal Google account.

I’m not intending this article to be a full step by step tutorial, but I hope it can point you in the right direction if you’re attempting a similar migration. This article is organised in a rough order of operations but you can jump a section that’s relevant to you:

Why leave Google Workspace?
Finding a new email provider
Taking a backup of your Google Workspace account
Setting up my new personal Google account
Migrating email and contacts to the new email provider
Migrating Google Calendar
Switching out any ‘Login with Google’ integrations
Cloning my Google Chrome profile
Saving my YouTube channel
Migrating Google Drive
Moving Google Home devices
Delete Google Workspace

1. Why leave Google Workspace?

The reasons I’m moving away from Google Workspace are:

Many decent options are popping up for custom domain email hosting, e.g.: iCloud has recently added custom domain support.
The Drive File Stream app allowed me to download my files on demand rather than syncing them all to my computer. It’s a must-have feature for me since I can’t fit all my Google Drive files on my laptop. It was exclusive to Google Workspace but in 2021 it was rolled into the standard Google Drive app.
Workspace has some annoying restrictions since it’s a business focused service - for example, you can’t change ownership of a file to a user outside your organisation and you’re unable to invite personal users to a Google Home managed by a Google Workspace user.
It used to cost me about $6/NZD/month for the lowest tier, but this is now about $10/NZD/month. The base price has increased, as well as changes in the currency and taxes which make Workspace an expensive option compared to it’s alternatives.

I still have lots of Google services that I still want to use, so this isn’t a full de-Google-ification for me. My goals are:

Migrate my email away from Google Workspace.
Move Google services to a standard personal Google account (without Gmail).

There are also still many Google services which I don’t use so you’ll need to figure out out to migrate the remainder. For example, I moved away from Google Photos some time ago - but if you use Google Photos you may want to take a look at jordancrawford/google-photos-takeout-fixer, a project I made to clean up the messy Google Photos export via Google Takeout to make it easier to upload elsewhere.

2. Finding a new email provider

I tried a few different options for my new email provider. There are some good options around, all with different pros and cons and at different price points.

Gmail + Cloudflare

Price: Free

If you like Gmail, you can pair a personal Gmail account with Cloudflare email routing for a free custom domain option - blogger Jay Caines-Gooby wrote an article on how to do this.

I considered this option, but I intentionally wanted a Google account without Gmail so that I could use my email address as my Google login email rather than an @gmail.com address.

iCloud+ custom domain

Price: $12 USD/year for 50GB storage

My original intention was to use iCloud+’s custom domain support. It’s one of the cheapest options since you only need to pay for some iCloud storage to get access (and iCloud pricing is very reasonable). iCloud also supports generating masked email addresses with the “Hide my email” feature - these are essentially automatically generated email aliases which make it easy to hide your real email address.

When I tried iCloud I found the domain setup process to be fairly flaky. It was having trouble verifying my DNS settings and during this process my domain was stuck in a state where it was bouncing emails back to their sender. When I finally did manage to get verification to work, my custom domain addresses didn’t show up in my list of available email addresses - I had to create another email address to get this fixed.

In short, it was cheap but wasn’t quite as solid as I was hoping so I decided to look into other options.

Migadu

Price: $19 USD/year for the micro plan

Migadu is incredibly cheap and has very flexible overage limits on their plans. I quite liked Migadu, it was easy to get started and it has many advanced features. It’s as close as you can get to running your own email server without actually running one - including offering mailboxes for other users within $19/USD/year micro plan.

The main downside of Migadu was it’s webmail. It was certainly usable but lacked many of the features of modern webmail clients. I’d recommend using Migadu with a desktop mail client.

I decided Migadu was a little too complex for my needs and I wanted something with a solid webmail so I kept looking.

Fastmail

Price: $50 USD/year

The option I ended up landing on is Fastmail. Fastmail is the most expensive option I considered but what got me over the line with it was:

A very easy setup experience - it let me login with my Google account and it pulled all my email, contacts and calendar in - even offering to check the old account automatically which was a perfect way for me to try out the experience without actually moving my domain over.
It can link into Google Calendar, giving me a handy way to access and edit my calendar while retaining Google Calendar features such as my shared calendar with my partner.
A quick, clean, web interface and mobile app.
Handy features like masked email and scheduled send.

3. Taking a backup of your Google Workspace account

Before we get into it, you should take a full backup of your Google Workspace account. You can do this at takeout.google.com.

I recommend setting the “Exports larger than this size will be split into multiple files.” option to something larger than the default of 2GB chunks. I left mine as 2GB and ended up with about 50 files to download, and any files over 2GB ended up as their own part (not inside the zip - very odd!).

It took a few hours for my backup to be created. Afterwards, I downloaded this backup to my Synology NAS, then extracted all the files to a folder by selecting all items and choosing “Actions” -> “Extract” -> “Extract…” and choosing “Extract all”. I also made one big zip of the backup for safe keeping - I’ll likely hold onto this for at least a year to be sure there’s nothing I forgot about.

4. Setting up my new personal Google account

My Google Workspace account already used my primary email address (e.g.: jordan@example.com). This means couldn’t just sign up a new personal Google account under that address.

To free up the email address you’ll either need to sign up with a different email address or move Google Workspace to a different primary domain. I initially signed up my new personal Google account under a different email address so that I could experiment with the account. Then, once I was ready to commit to the migration I moved Google Workspace to a different domain to free up my email address.

Many email providers (including Google, Fastmail and iCloud) ignore anything after a + symbol in your email address - i.e: emails sent to jordan+anything@example.com will have the +anything part stripped out and go to the mailbox of jordan@example.com. This is handy for so many things, like to have multiple accounts for the same website.

This trick works when you sign up for a Google account - you can use an address like jordan+new@example.com and still receive emails for the new Google account but without clashing with your existing Google Workspace account.

On Google accounts without Gmail, you’re free to change your email address whenever you need so you can change your email address to your primary address when you’re ready.

Approach 2 - Move Google Workspace to another primary domain

Once you’ve moved email away from Google Workspace (in the next step) you’ll be able to safely switch Google Workspace to a different domain name. This means your Google Workspace account will still be able to receive email (which might be handy throughout the migration process, for billing emails from Google, etc) but frees up your email address. You don’t have to buy another domain name - you can simply use a subdomain of your existing domain, e.g.: workspace.example.com. Your Workspace account would become something like jordan@workspace.example.com and free up jordan@example.com for use.

Go through the process of adding a new domain as a ‘Secondary domain’ in “Add a user alias domain or secondary domain” (including configuring your DNS), then switch it to being the primary domain with “Change your primary domain for Google Workspace”.

I didn’t manage to delete the old domain name - it kept complaining that the domain name was in use. However, I found that the only thing I actually needed to do was make sure I’d deleted all aliases off the old domain - I removed the alias on my user account and a Google Group that had been setup using an address on that domain. To find any rogue aliases, I used a piece of software called GAM - the gam print domains command found all aliases on the domain.

5. Migrating email and contacts to the new email provider

The process for moving your email and contacts will vary based on your chosen provider, but for Fastmail this was a fairly simple process since Fastmail offered to migrate them for me during setup.

Next, I updated my DNS to point to Fastmail so that new emails would go to my Fastmail account. I sent a test email from another email address to confirm that emails were coming in to my Fastmail account.

I also installed the Fastmail mobile app and setup my email and contacts on my laptop (they make this easy through the website - just download a system profile from the Fastmail settings which automatically sets it up). One caveat with contacts is that Android doesn’t support CardDAV, so if you want your Fastmail contacts synced to your Android phone’s contact app you need a $5 app - more info about this on the Fastmail help site.

6. Migrating Google Calendar

I had a little bit to think about with my Calendar.

My partner and I share a Google Calendar. Google Calendar sharing only works with Google Calendar - you can’t link Fastmail directly to a shared Google Calendar and my partner isn’t gonna move to Fastmail in order to share a calendar. I also quite like the handy public holidays calendar which Google Calendar includes - I couldn’t seem to find a working ICS/iCal feed of NZ public holidays online.

Fastmail has an integration with Google Calendar, so I could use Fastmail as my source of truth for calendars and let it pull in the public holidays and shared calendars. However in my testing, Fastmail’s Google Calendar integration has a bit of a delay (5-10 mins) - so if one of us added an event to the shared calendar, the other person may not see it for up to 10 minutes.

By using Fastmail as my primary calendar system, I’d also miss out on some of Google Calendar’s handy features, such as automatically adding Google Meet meetings to calendar events I create (not something I do often on my personal calendar but still a handy feature).

I decided to stick with Google Calendar as my primary calendar system, however for convenience I’ve still got it linked into Fastmail which lets me add events to the calendar from the Fastmail interface.

Migrating my calendar from Workspace to my new Google account:

For my personal calendars, I exported them from the “Calendar settings” tab when managing the calendar, then imported this calendar from the “+” button next to “Other calendars” (a copy of the calendar is also included in the Google Takeout download).
For any shared calendars, I added my new account to the calendar (under “Share with specific people” in the calendar settings), clicked the “Add this calendar” on the email Google sends, then removed my old account from the list.

Many websites let you login with your Google account which is super handy!

Don’t forget to visit your “Apps with access to your account” (myaccount.google.com/permissions) page, login to each of the sites under “Signing in with Google” and switch them to email and password based authentication so you don’t lose access to these accounts when cancelling Google Workspace. I found that I’d only used Google for a handful of websites, and each of these had some sort of option to setup a password in their account settings pages.

8. Cloning my Google Chrome profile

Over years I’ve built up all sorts of bookmarks and Chrome extensions so I’d like to keep my Chrome profile. When using Google Workspace, the Chrome profile is managed by the organisation and the information is deleted when you sign out of the Chrome profile. This means we can’t just sign out of the Chrome Profile and sign in under a new account.

Instead, I needed to make a copy my Chrome profile and get that to sync to my new account.

Process for copying a Chrome profile (on macOS):

Quit the Google Chrome application.
Go to ~/Library/Application Support/Google/Chrome (use “Go” -> “Go to folder…” in Finder).
You should see several folders like “Profile 1”, “Profile 2”, etc - figure out which one is your main Chrome profile (if you have multiple).
- Chrome keeps old profiles in here so I had more profile folders than active profiles which can be confusing.
- I found that profiles which took up the most disk space were generally the ones I used frequently.
I copied the Chrome profile folder and gave it an unused number, e.g.: Copy the “Profile 2” folder and call it “Profile 3”.
When launching Chrome, you’ll see the profile show up - but it’ll remain logged in as your Google Workspace account. To remove the account, follow this SuperUser response to manually remove the account from the Chrome profile.
Launch Chrome, log into your new Google account with the copied Chrome profile and enable sync. Your Chrome profile will be synced to your new Google account.
(Optional) To be sure everything’s synced fine; delete your copied profile, create a new Chrome profile, login to your new account and let it sync back down.
- I figured by starting fresh I save myself from any weirdness this process may have introduced.
On all your other computers, make a new Chrome profile and sign in with your new Google account - the Chrome profile will sync automatically as usual.

9. Saving my YouTube channel

As far as I can tell, I was faced with two main options for YouTube -

Start fresh, re-uploading any videos, re-subscribing to channels, etc - see “How to Move YouTube Content to a New Google Account” on PCMag.
Turn my account into a brand account, and transfer ownership of the brand to my new account.

I went with the brand account approach.

I followed the steps in “Making the Switch: How to Transfer YouTube Account” from SocialPilot under “How to Move a Non-brand or Personal Google Account”. The process is broadly:

Create a fresh brand account.
Transfer your current account to a brand account, which replaces the fresh brand account.
Adding your new account as an additional owner of the brand account.
After 7 days (due to restrictions intended to stop account takeovers), make your new Google account the primary owner and move your old account’s access.

This process worked reasonably well, the advantages are:

I don’t have to bulk re-subscribe to a bunch of channels.
My (one) video is still in place and uses the same URL.
I can change more information about my YouTube account, like giving it a different profile picture or name (e.g.: a shortened version of my name for privacy).

However this approach had a few disadvantages:

My comment and watch history is gone.
When selecting my YouTube account (e.g: on my Apple TV’s YouTube app) I see two versions of myself - one is the brand, the other is my unused personal YouTube account on my new email address.
- For the web, you can visit “Settings” -> “Advanced settings” and select this as the default channel - so most of the time you won’t notice this.
Weirdly sometimes I get an error from my Apple TV’s YouTube app that my account type is unsupported? I’m not too sure why this happens…

10. Migrating Google Drive

I hate to say it, but Google Drive migration was a real mission, so much so that I wrote an entire article about it -

Moving Google Drive files from Google Workspace to a personal account

11. Moving Google Home devices

I have a Google Wifi router which is managed through the Google Home app. Google Home is centered around, well, homes, which are a bucket for all your devices, segmented by rooms.

A Home which is managed by a Google Workspace account can’t invite personal Google accounts - this limitation was also something YouTuber Linus Sebastian ranted about recently too. I can’t have my new account take over my Google Home, nor does it appear to be possible to transfer my Google Wifi device between Homes if I made a new Home through my new account.

Unfortunately the solution to this one ended up being:

Screenshot all my Google Wifi’s settings.
Factory reset the device.
Set it up again under a new Google Home on my personal account.

This is a super annoying restriction and hopefully it’s something Google improves in future.

12. Delete Google Workspace

You have a backup, you’ve double checked everything - it’s time to delete your Google Workspace account.

Follow the steps in “Delete your organization’s Google Account” to cancel your subscription then delete your account. As part of this process you’ll also need to delete and Google Cloud Platform projects associated with the organisation (e.g.: if you setup your own app keys for software like rclone).

You’ve made it! Google Workspace was a useful tool for me while I had it, but I’m glad to have finally decoupled myself from it. I’m free from Workspace’s restrictions and it’s fairly simple to mix and match services from Google or other providers if I desire in future.

Escaping Google Workspace was originally published by Jordan Crawford at Jordan Crawford on August 13, 2022.

Moving Google Drive files from Google Workspace to a personal account

2022-07-18T00:00:00+12:00

Moving your Google Drive files from a Google Workspace account to a personal Google account is harder than it should be! I’ve broken down my process and the many hurdles on the way.

This is one part of my recent project to escape Google Workspace. You can read more about this in my Escaping Google Workspace post.

Assumptions

This post makes the assumption that you’ve; still got access to your old Google Workspace account, that your new personal Google account is setup, and that your new personal Google account has sufficient storage to move all your files over. If not, we’ll go through that process in Escaping Google Workspace.

Step 1 - Take (more) backups!

In addition to making a Google Takeout backup, I’d highly recommend making a full clone of your Google Drive files with a tool like rclone. This step is important because:

Takeout doesn’t include files in your Google Drive account which are owned by other users.
The backup covers us in-case something goes wrong during the move.
We’ll use this backup for verification of the account post migration.

Introduction to rclone

rclone is an awesome open-source command line utility which lets you manage your files across various storage providers. For example, you could use it to copy all your files from Dropbox to Google Drive, or regularly sync files from Google Drive to your computer.

We’ll be using it to backup your Google Drive, migrate files between accounts and verify everything is still in place at the end.

Installing rclone

See the rclone downloads page for more information.

On macOS I usually install it through brew, and on Linux I tend to use apt-get update && apt-get install rclone

Using a cloud server

I’d recommend using rclone on a cloud server if you don’t have a particularly fast internet connection, have a data cap, or don’t have sufficient local storage for a backup all your files. A cloud server will have a quick internet connection with low latency to Google servers.

When I need to do things like this I tend to spin up a cheap cloud server with a provider like Hetzner, Digital Ocean or Vultr. These servers are billed hourly so you can destroy the server when you’re finished. You may also need to attach some block storage to gain some additional space.

I connect over SSH and run my commands within a screen or tmux session. This means that I can kill my SSH connection without interrupting rclone’s operations.

☁️

Need a cloud server?

I pay €3.79/mo for an ARM server with 4GB RAM, 2 cores and IPv4 with Hetzner
Sign up through this link for €20 starting credit!

Affiliate link

Linking account remotes

Link your Google accounts as rclone remotes through rclone config. Give the accounts clear names so you know which one is old and new! rclone has a fairly simple interactive command line interface so you don’t need to remember any commands.

> rclone config
No remotes found - make a new one
n) New remote
s) Set configuration password
q) Quit config
n/s/q>

rclone will step you through the process of (optionally) setting up your own API keys and linking your Google Drive accounts.

Making a backup with rclone

Run a command like this to make a backup:

rclone sync --verbose --progress --drive-skip-shortcuts old-google-drive:/ /local-backup-destination

The options used here are:

--verbose and --progress - so you can see what’s happening
--drive-skip-shortcuts - skips transferring Google Drive shortcuts since these tend to cause errors or will map to the wrong place now that you’ve made a copy.

Step 2 - Understanding Google Drive and it’s limitations

Unfortunately getting your files into your new account isn’t quite as easy as hitting a transfer button - particularly when one of the accounts is a Google Workspace account.

Before we begin, lets get a better understanding of how Google Drive works.

Google Drive files/folders have an owner

The owner of a file/folder the person who created/uploaded it.
Even if you own a folder, it doesn’t mean you own all the files within the folder. If another user uploads a file to the folder, creates a sub-folder or moves something from their drive to the folder then they’ll remain the owner of those files.
A file counts against it’s owners file storage allowance. That means if you put a file into a friends shared folder, it’ll still be your file and count against your storage unless you make them the owner.
The exception to this rule is Shared Drives.

Shared Drives are a Google Workspace feature (on “Business Standard” and above plans) which mean the organisation owns the files in the Shared Drive rather than an individual user, we’ll talk more about Shared Drives shortly.

Google Drive files can become ‘unorganized’

While messing around with Google Drive and making a few mistakes, I discovered that files would sometimes go to an orphaned ‘unorganized’ state. These are files which you own or have access to but they aren’t organised in any of your folders.

You can find these by searching Google Drive for is:unorganized owner:me - for more information see CBackup’s article; “How to Search Files in Google Drive with ‘Is Unorganized Owner Me’?”.

You can’t actually delete other peoples files/folders

This one sounds a bit weird, but you can’t actually delete files/folders owned by other users within a shared folder - even if you own the folder.

When you do, it removes the file from the folder but moves the file to the owners root “My Drive” folder. For some more context on this, see the Google article “Delete and restore files in Google Drive”

Step 3 - Transferring file ownership

Let’s start moving files between accounts! But first, we need to cover some techniques.

Approaches to transfer file ownership

In the Google Drive web interface there’s a “Share” window which lets you share files/folders with other users. Once you’ve shared a file with someone, you can select “Transfer ownership” to attempt to move the file to their account.

This works great between personal Google accounts or between users within the same Google Workspace organisation, but unfortunately you can’t transfer ownership between a personal and Google Workspace account. If you try, you’ll get a message “Ownership can only be transferred to another user in the same organization as the current owner”.

_{😕 We'll need to find a different approach}

Copy files via rclone

The most reliable solution to transfer ownership within Google Drive is to copy the file via rclone.

Process

Use the rclone sync (documentation) command to copy a folder from your old Google Drive to your new Google Drive.

e.g.:

rclone sync --transfers 10 --modify-window 1s --verbose --progress --drive-skip-shortcuts old-google-drive:/folder-name new-google-drive:/folder-name

The new options used here are:

--transfers 10 - runs more transfers in parallel, you may need to play around with this number to see what works best for you
--modify-window 1s - I’ve had problems with rclone’s duplicate file detection taking a long time because modification dates between the files are slightly different between cloud providers (presumably causing it to do more intense checksum checks). By including this option we’re telling rclone that if the file’s modification time is within a second that it’s close enough to be considered the same file.

Advantages

It works for all files whether you own them or not.
rclone is re-runnable and very consistent, making it a more trusted option than downloading and re-uploading files.
rclone can copy Google Docs, Slides and Sheets files so you don’t need to try to convert them to another format.

Disadvantages

You lose all file history - everything will be created as of the date of copy and by your new user (but the original file modification date is retained).
Sharing settings are reset so you’ll need to re-share the files/folders with everyone you’d previously shared them with.
Since the original files still exist, we’re going to need to go through a process to delete the original files (which can be a mission if some files are owned by other people - more on that in step 5!)

Move through a Shared Drive

Shared Drives are a feature only available on the “Business Standard” Google Workspace plan. It’s essentially a shared folder but with the perk that the organisation owns the files rather than an individual user. This means you don’t need to worry about changing ownership of files or adjusting permissions when a user leaves the organisation.

A technique I discovered from ‘How to transfer Drive ownership from Workspace to personal account’ on r/GSuite is that by moving files into a Shared Drive as your old user, then moving them out to your “My Drive” the new user, you’re essentially able to transfer ownership of the files and retain their folder structure.

Process

Upgrade your Google Workspace plan to “Business Standard” - you can upgrade it temporarily and then downgrade it later - you should only be charged extra for the days you were on the plan.
Create a Shared Drive and add your new and old account as managers.
Move some files into the Shared Drive from your old account.

_{If the files are able to be moved you'll see a confirmation message before starting the move.}
Wait for the files to move, then confirm that the files are all present in the Shared Drive.
From your new account, move the files out of the Shared Drive and into your “My Drive” folder.
Wait for the files to move, then confirm that the files are all in your new account’s “My Drive” and owned by the new account.

Advantages

It’s fairly simple and quick.
It retains all version history of previous file changes.
It retains previous folder sharing settings.

Disadvantages

This approach only works for folders where you own all files and folders within the tree structure - you cannot use this approach to move folders where you don’t own all the files.
- If you try to do this you’ll get a message that “xx files can’t be moved because they belong to organizations that don’t allow it”.
- Sometimes it will actually allow you to move some of the files in the folder, but it’ll leave some files behind if the number of files are below a certain threshold. I opted not to continue in any of these situations as I didn’t want to have to merge the folder structures back together again.
⚠️ A big warning with this approach is that you must wait until the folders have been copied to the Shared Drive before moving them out!
- It’s pretty hard to know when the files have moved successfully.
  - Sometimes Google Drive would make it look like all the files were there but if I refreshed the page a few times I’d see that they weren’t.
  - The other place to check is to search is:unorganized owner:me to see if there are any files which have become detached from their folders as they move to the Shared Drive.
  - Using a tool like rclone may be a good way to inspect the directories to ensure that it’s been moved.
- I made this mistake and got myself into a fairly big mess!
  
  I assumed that if I quickly move a folder into the Shared Drive then move it out that Google will catch up and do some magic in the background to get them to their destination - but sadly, there is no magic!
  
  Moving the folder out of the Shared Drive seemed to stop the files within the folder from moving to the Shared Drive. That meant that I ended up with some folders which were owned by my new account but where the files in those folders were still owned by the old account!
  
  In the old account I found the files by searching for is:unorganized owner:me. I believe they were considered unorganised because this account didn’t have access to the folders that the files were in.
  
  In the new account, I could see the folders but not the files - likely since the files weren’t shared to the new account.
  
  What a nightmare! To get the files showing up again I shared the folders from the new account to the old account. This meant the old account could see the folders these files belonged to (even though the new account still couldn’t see them). This mess was pretty hard to fix and I decided the best course of action was to fall back to the rclone approach and make a new copy of the old files to move from the old account to the new account.
  
  Despite this experience, I’m still a fan of this approach as it has many advantages - it just must be used with care.

Making the transfer

With those approaches out of the way - let’s get ownership of your files moved to the new account.

I took a hybrid approach, using a different solution based on the structure of each folder. Go through your Google Drive folders and figure out which ones fit into each of the below buckets.

Bucket 1 - Folders completely owned by me

These are folders where all the files (and files within all sub folders) are all owned by me. They might be shared with other Google Drive users but no other users own any of the files.

This situation is perfect for the Shared Drive transfer method as it’s simplest and ensures any files shared with other users will retain their sharing settings.

The best way to tell if a folder is fully owned by me is to try and move it into the Shared Drive - Google Drive will tell you if there are ineligible files (i.e.: files owned by other users).

Bucket 2 - Folders shared with me but I don’t own any files within

These are folders which don’t belong to me at all, and where none of the files within the structure are owned by me at all.

If I have edit permissions

Share these files with my new account.
From my new account, remove my old account from the sharing list.

If I don’t have edit permissions

I’ll need to ask the owner to add my new account to the list.
Or, if I don’t care about collaborating with others and just want the files then clone them with rclone!

Bucket 3 - Folders which are a combination of owned by me and owned by another user

These are folders where some files/folders within the hierarchy are owned by me and some aren’t. It doesn’t really matter whether I own the folder itself or not, the key thing is that we’ll need to transfer ownership of the files I own.

This is one of the more difficult situations and the approach here will depend on the folder structure.

If you only own a few files

The Shared Drive approach should be sufficient when you only own a few files within the folder.

You won’t be able to move the entire folder through the Shared Drive (as the files must be fully owned by you to do this) but you can move individual files/folders which you own through the Shared Drive and back into the folder after transferring ownership.

If the ownership structure is complex

If the folder contains many other folders which contain files owned by both you and other users it becomes unrealistic to manually move sets of files through the Shared Drive approach.

Instead, you’ll want to use the rclone approach to create a fresh copy of the folder which your new account owns, then share the new copy of the folder to the other users.

We’ll need to delete the old folder in a safe way too - it’ll be important we follow Step 5 - Clean-up.

Step 4 - Verification

We’ve done quite a lot of file operations and there’s a chance there’s something that’s been forgotten. I like to be sure that the before and after is the same by performing some verification.

We can use our rclone backup from earlier to compare with the files that are sitting in your new Google Drive.

To do this I used a command like this:

rclone sync --drive-skip-shortcuts --modify-window 1s --dry-run --no-update-modtime /local-backup-destination new-google-drive:/

The new options used here are:

--dry-run ensures no syncing actually happens but we just get a log message to say what operations it would need to do to keep it in sync.
--no-update-modtime prevent messages that it would update the modification time as we know that we may see slight differences in modtime.

Once you run the command, if you see no log messages about work which needs to be performed then the directories are exactly the same!

If you see a message about it wanting to add a file, then that means there’s something which was in your backup but isn’t in your Google Drive - is there something you’ve missed?

Example log output for a file which was missing from the new account:

2022-07-18 04:48:39 NOTICE: file.png: Skipped copy as --dry-run is set (size 241.251Ki)

If you see a message about it wanting to delete a file, then that means there’s something which was in your new Google Drive but not in your backup - is there anything extra you’ve added that you’d expect? Or is something sitting in a different location to where it used to be?

Once you’re happy that everything is consistent, it’s time to move on to clean-up!

Step 5 - Clean-up

As mentioned above, when you delete files owned by someone else they’ll end up in their root “My Drive” folder.

If you’ve made copies of any folders which contained files owned by another user (and you’ve re-shared the new copy of the folder with those users) then we’ll need to take care deleting the old folder. If we’re not careful, when we delete the old folder a bunch of files will end up in the “My Drive” folder of other users, causing confusion and making a massive mess.

I found that when you delete a folder, if you own the folder, Google Drive will delete all the files and folders in the top level of the folder but it doesn’t appear to recurse subdirectories. If I deleted a folder which I didn’t own then the entire folder went to the owners “My Drive” folder.

The approach I took to delete the old folder without causing a mess is:

Flatten the entire shared directory structure so that all files are contained within a single folder (i.e.: no nested hierarchy). Importantly, the folder must be owned by the other user. See below for more information on getting a flat structure.
Get the other user to delete the folder. It’ll go to the Trash.
Get the other user to open their Trash, right click the folder and choose “Delete Forever” to instantly start deleting the files. Because they own the folder, all the files they own within the folder will actually be deleted and files owned by your old account will go to “My Drive” (littering “My Drive” isn’t a concern since I’ll be deleting my account).

A limitation of this approach is that it won’t work when there are multiple owners. In this situation you’ll need to segment the files into separate folders for each user to delete their files separately.

My approach to get a flat structure was a bit messy and there’s likely more elegant solutions. What I did was:

I used rclone to get a recursive list of the directories in the folder, e.g.:
```
  rclone lsd -R old-google-drive:"/folder/" > raw_directories.txt
```
Next, in my editor (Visual Studio Code) I cleaned up the file result and appended a rclone move command, e.g: made each line look like below:
```
rclone move -v old-google-drive:"/folder/folder-sub-dir" old-google-drive:"/folder/to-delete"
```
The rclone lsd output lists from least to most specific in the hierarchy. If we move files in this order it’ll result in sub directories being moved to the deletion folder which we don’t want for our flat structure.

I solved this by reversing the order of the file so that it’d go from most to least specific. I used tail -r to_run.sh > to_run_reversed.sh to reverse the set of commands.
I made the script executable with chmod +x to_run_reversed.sh then ran the script and left it to do it’s thing.
The command rclone ls old-google-drive:"/folder/" can be used to list all files and verify that the file structure is now flat.

Congratulations, you’ve made it to the end! It’s a painful process but once it’s done, it’s done!

Hopefully Google makes this an easier process in future - the removal of some frustrating restrictions around transferring files would go a very long way.

Moving Google Drive files from Google Workspace to a personal account was originally published by Jordan Crawford at Jordan Crawford on July 18, 2022.

Building a budget home server with the ODROID HC2!

2018-07-08T00:00:00+12:00

When I first started out on my budget home server journey back in 2016, the most logical choice was the Raspberry Pi 2 micro-computer. This has served me well, but my needs for a server have changed and the Pi just wasn’t keeping up. I want a quicker server but with the small size and low power consumption that made the Pi great!

What’s wrong with a Raspberry Pi as a NAS?

As I breifly mentioned at the end of my first post, the Raspberry Pi isn’t a very quick as a NAS server.

There are several factors at play here, most notably the fact that the Pi 2 only has USB 2.0 ports and 100mbps ethernet which both cap the max transfer speeds. In addition, the ethernet and USB ports share bandwidth to the CPU so heavy use of these interfaces results in further speed reductions. All this in combination with a pretty limited CPU, results in poor performance as a NAS!

The ODROID HC2

The ODROID Home Cloud 2 is a single board computer which houses your hard drive and sports some powerful chips on it’s tiny circuit board. Seriously, this thing is tiny - smaller than a Raspberry Pi! It’s got an octocore ARM processor, 2GB RAM, a built in SATA port and gigabit ethernet. The Home Cloud doesn’t even have any form of video output, it’s simply not needed!

Because it houses the hard drive, the Home Cloud makes a very tidy home server setup and only takes up a single power socket. The only downside with the hardware is that the single USB port only supports USB 2.0, but this is a small complaint given I only use this port for taking backups of the internal hard drive.

The unit is cheap, coming in at about $65 USD for the base unit, power adapter and top case - just bring your own hard drive and SD card.

If you have a 2.5” drive, theres also the ODROID Home Cloud 1 which has the same specs but installed in a different case. I happen to own a 3.5” hard drive, but I absolutely love the idea of an even smaller NAS. The Home Cloud’s are also stackable - let me know if you have any ideas of a usecase for this!

Setting up the system

Lets get the basics of a home server configured before getting any applications going.

OpenMediaVault

Just like a Raspberry Pi, the Home Cloud runs it’s operating system off the SD card. There are a variety of operating system images to choose from. I chose the open-source NAS operating system called OpenMediaVault for mine!

OpenMediaVault has an easy to use web interface for configuring your system, comes with all the file servers you need, and has a good plugin system to make installing apps really easy. OpenMediaVault handles much of the manual configuration I had to do on my Pi NAS which meant I could get a working system going sooner!

I went through the following steps to get OpenMediaVault up and running:

2023 update: Prior to 2021, OpenMediaVault published pre-install images for single board computers - unfortunately now you’ll need to install OMV on an Armbian image yourself. The steps below have been updated but unfortunately I no longer have my HC2 so I haven’t been able to test these steps.

Download Armbian - see their information about the ODROID XU4/HCx
Clone this image to your SD card, I like to use Etcher for this.
Boot it up and find your device’s IP address on the network.
Follow the steps in Installing OMV6 on Armbian Bullseye to login via SSH and install OpenMediaVault.
Type in the IP address of the NAS into your browser and login with the default credentials from the installation guide (under “First time use”).
Change your web-administrator password under “General Settings” > “Web Administration”.
Run updates by clicking “Check” then “Upgrade” in “Update Management”.
Setup your user under “User”. We need SSH access so make sure to tick the boxes for the “sudo” and “ssh” groups.
Login to SSH as your new user and expand your SD card, I personally found the steps outlined in “Resizing the partition online using fdisk” of this guide did the trick.

Docker

OpenMediaVault has plenty of plugins which we can use, but for anything else we can use Docker. Docker containerises all your apps which avoids all the setup and depedendeny management mess of installing new software. You can learn more about Docker in my original NAS post).

To get Docker setup:

Under “OMV-Extras”, double click on “Docker” (not “Docker CE”), click “Enable” then “Save”.
Over in “Plugins”, click “Check”. Once this completes, search the plugin list for “docker” and install the “openmediavault-docker-gui” package..
This will give you a handy “Docker” menu to manage your images and containers.
Optionally, if you want Docker Compose to use files to manage your Docker containers, install Docker Compose using pip.

File Access

OpenMediaVault makes it easy to access your files over the network!

Setting up shared folders

Firstly, lets configure the hard dirve to spin down automatically under the “Physical Disks” menu, personally I use the “Intermediate power usage with standby” setting.

Next, configure your partition to mount automatically. Select your drive under “File Systems” and click “Mount”.

Now lets configure some shared folders! Pop over to the “Shared” folders menu and click “Add”. Give your file shares a name, set the device and path then click “Save”. You need to set your user privileges for each folder by clicking “Privileges” and giving each user either read/write, read only or no access.

Configuring file servers

Lastly, you just need to configure each of your file servers. These are all configured in a pretty similar way - just configure some shares in the “Shares” and flip the “Enable” switch under the “Settings” tab.

OpenMediaVault supports the three common file servers for NAS’s so you get to choose which ones you want to setup!

Apple Filing (AFP)

While no longer the default file sharing server for macOS, AFP is the traditional Mac file server and can only be used by macOS. I personally use a Mac and found AFP to work the fastest for me.

SMB/CIFS

SMB/CIFS is the most widely supported file server with built in support on Windows and macOS in addition to packages like Samba which enable support on Linux systems. I’d highly recommend setting up SMB/CIFS sharing on your NAS even if you don’t intend to use it as your primary file server.

NFS

NFS stands for Network File System. It’s pretty old school and is pretty common for Linux systems. It’s different from the modern alternatives in that it doesn’t have any authentication by default and expects a user’s ID’s to match across the server and client systems. NFS works great but as it requires a few more considerations I’d only configure it if you really know what you’re doing.

Remote Access

Another key part of my NAS server infastructure is to enable remote access from anywhere in the world. I went a bit far with my remote access requirements, so I like to be able to access my NAS from anywhere in the world no matter what internet connection I’m using at home.

I’ve posted about how I do this in the past, check out how I got access to my home services from anywhere without port forwarding.

Local Access Convinence

This was another overkill requirement of mine, but when I access my NAS locally I don’t want to have to remember it’s IP address or the port numbers of the services it runs.

This is something else I’ve posted about in the past so check out how I get easy access to my home server on a local network.

USB File Backup

One of the downsides to having only one SATA port is that you lose out on the multi-drive data redundancy provided by most commerical NAS setups. While data redundancy is nice to ensure continuity in the event of a drive failure, I’m only using my NAS at home so I can manage without immediate access to my data in the event of a drive failure.

However, I don’t want to lose all my files completely so I have a second hard drive which is kept offline most of the time as a cold backup. I sync this at regular intervals so in the event of a drive failure I’ll just buy a new drive and restore from this backup.

I use the OpenMediaVault “USB Backup” plugin to automatically sync to my backup drive whenever it’s connected. This is based on rsync but has an easy web-interface to configure the backup mode, shared folder and backup device.

Configuring applications

Now that all the basics are setup, it’s time to configure some applications on the system!

Plex

Plex is a server for your personal media collection. It collects brilliant metadata about your content and allows it to be played pretty much anywhere with the web, desktop, TV and mobile apps.

For remote access to Plex I just use Plex’s Relay feature, but to get around the bandwith limitations you could also run it through my custom remote access solution.

The OpenMediaVault plugin for Plex is very simple, all I need to do is select a physical disk for it to store the Plex data files on.

Metadata storage

The OpenMediaVault plugin for Plex lets you configure the drive to store the metadata on but it won’t let you use the boot volume.

I figured that it made sense to store all the Plex metadata and such on the SD card as it didn’t need to spin up and SD cards are better suited to random reads like database and metadata accesses.

I went through a lot of effort trying to partition my SD card in order to store my metadata here, only to find that the benefit was negligible (and in fact, with the SD card I was using, significantly worse!). So, a word of wisdom, just don’t bother and store the Plex database on the hard drive!

Deluge

Deluge is a torrent client with a web-interface. This allows you to download torrents directly on your NAS. I used the OpenMediaVault plugin for this and it works great!

Time Machine Backup

Time Machine is a feature built into macOS which backs up your computer every hour with file snapshots. Generally you’ll need an Apple Time Capsule to do this over the network, but OpenMediaVault’s Apple Filing server lets you enable Time Machine support for a network share! All you need to do is configure a share and tick “Time Machine support”.

This is completely optional, but I also configured a quota for this volume. This is a pretty awesome feature of OpenMediaVault where I can configure the Time Machine volume with an artificial capacity in order to prevent backups from filling up my entire drive. Set the quota option in the share (I used about 500GB for mine) and Time Machine will start clearing out old backups when it hits this point.

Cloud Storage Backup

I store a lot of my commonly accessed files in my Google Drive account. I have so many files there that I can’t sync them all with my computer so I use the Google Drive File Stream app to access everything else via the internet as needed. However, if my internet goes out at home or my Drive account is somehow compromised then I still want to have a copy of these files.

I use the rclone tool to do a one-way backup of my cloud storage to my NAS. I run this using Docker and the tloxipeuhca/rpi-rclone image.

I use OpenMediaVault’s built in “Scheduled Jobs” mechanisim to run this job once a week. To ensure I’m alerted if this backup ever stops working, I’ve configured healthchecks.io to notify me if my cloud backup doesn’t run once a week.

The scheduled job uses a command like to run the backup and log the successful run: docker start --attach [docker container name] && curl [healthcheck link]

How does it compare to the Pi?

Now for the moment of truth, how much better is my latest generation of NAS in comparison to the Raspberry Pi? Well I did some not-so-scientific tests. The first of which is another run of Blackmagic Disk Speed Test, followed by a real-world transfer of a 3.85GB file.

The results show that the ODROID unit is significantly faster than the Pi! I can’t believe it took over 40 minutes to copy a 3.85GB file to the Pi and back - the ODROID managed this task in under 2 minutes!

All the best for your projects! I’d love to hear how your projects go in the comments 😀

Building a budget home server with the ODROID HC2! was originally published by Jordan Crawford at Jordan Crawford on July 08, 2018.

Monitoring an off-grid system with Kauri

2018-03-18T00:00:00+13:00

In 2016 I finished my Software Engineering degree at the University of Waikato with my fourth year honours project. The objective of my project was to create a system which would allow the owner of an off-grid house to monitor their energy system and access this information from anywhere.

What resulted is Kauri Energy Monitor, a cloud based system for monitoring a renewable energy system and I’ve recently gotten it to the point where it’s ready to release to the world!

In this post I’ll cover what Kauri can do, how you can start using it and I’ll discuss a few things the process has taught me.

Who would need this?

Some houses are fitted with renewable energy systems - this is a very broad term for various configurations of equipment which allow the occupants to generate their own electricity. Depending on the exact configuration, these may also have a form of energy storage or backup generation.

An off-grid system is a good example of a renewable energy system. These might have solar or wind generation, a bank of batteries for energy storage and a generator as a backup energy supply. On the complete opposite end of the spectrum, grid connected systems usually feed their excess generation back to the grid and consume energy from the grid when required.

_{A typical offgrid system.}

In all of these situations it can be difficult to figure out what your energy system is doing. Is it currently charging? How long will the battery last? How efficient is my generation?

Kauri answers these questions and allows users to do this from any internet connected device.

What can Kauri do?

Users of Kauri setup energy sensors at various points in their energy system. These sensors are hooked up to a computer running a piece of software called Kauri Bridge which sends readings to the Kauri server.

By analysing the energy flow data, Kauri shows an overview of all the energy flowing through the system. This tells the user the amount of energy being generated, consumed by the house or stored in the battery. Kauri also feeds this information into the B42SOC algorithm to calculate the state of charge of the system’s batteries (i.e.: battery level).

_{The Summary page in Kauri shows current energy flows and battery state.}

Kauri determines the future state of the system using battery state information and energy flow patterns developed over time. This lets Kauri answer questions like, when will the battery be fully charged? When will the battery run out? Or what battery level will the system be at in 5 hours?

_{The Future State page in Kauri shows the state of the system into the future.}

How can I use Kauri?

Getting a bridge and some sensors sorted

You’ll need to have some supported sensor devices setup in your energy system (the “What do I need?” page will help you figure out which sensors you’ll need). Once you get these hooked up to a computer which can run Kauri Bridge then you can setup your Kauri server.

_{A Smart Circuit SC20 device - an AC sensor supported by Kauri Bridge.}

Running your own Kauri server

Kauri is open source so you can run it on your own server for free! The getting started guide runs you through the process of setting up Kauri on your own server.

See the getting started guide.

Want a hosted option?

Currently if you want to use Kauri you’ll need to be comfortable with hosting the Kauri server yourself. If you aren’t, then I’d love to offer this as a paid monthly service if there is enough interest.

Express your interest in a hosted option here!

Contributors wanted!

I’ve really enjoyed working on Kauri but I really want to get onto some new projects! If you want to improve the project, please feel free to submit a pull request! I’m happy to provide advice for further development.

Contribute on GitHub!

Want to learn more?

If you’d like to know more about how it all works, feel free to read my honours project report:

Read "Cloud Based Monitoring of a Renewable Energy System"

Or, watch a video of a slightly younger version of myself presenting my honours project:

What has this project taught me?

This whole project has taught me so much technically and it’s the biggest personal project I’ve ever embarked on, but there are a few specific things that I’ve learned that I want to discuss.

Give your project a name

Kauri didn’t really have a proper name until recently - it was usually just known as ‘my honours project’ or ‘Offgrid Monitoring’. Giving a project a name gets you thinking about the purpose and scope of the project which makes it easier to reason about its features.

For example, by calling it ‘Kauri Energy Monitor’ I decided that I would support as many configurations of renewable energy systems as possible so I shouldn’t prioritise features that only benefit users of off-grid systems.

Ask the question, will I really need that?

While getting Kauri ready for release I listed off loads of features that I thought would be useful. These things would be nice to implement but weren’t driven by any actual user requirement and wouldn’t be the biggest barriers to adoption by users.

The lesson here is to ensure any change you make provides enough value to be worth the cost of implementation. This isn’t easy for projects which you’re passionate about but it means you aren’t wasting time implementing something which no one needs.

Think about project handover

While I was first developing Kauri it had a single production instance that was kept up to date as the system developed. If I needed to play with some real data I just downloaded a database dump and applied it to my local database. I didn’t need documentation at that point because I had all the context in my head and I was the only developer. Some actions could only be completed via the API as it wasn’t worth the time building an interface to add a building if I only needed to add a building once.

However, after spending a few months without working on Kauri I experienced the cold introduction that any new developer would get. How do I get it running on my system? What if I want to get some data running locally without a real renewable energy system to test with?

I realised that I needed to:

Improve the configuration interface so that users didn’t need to use the API to configure Kauri.
Write some basic documentation - I covered how to setup an enviroment, some of the background to the project and added a getting started guide.
Provide a set of mock data to allow users to experiment without needing a renewable energy system.
Build tooling to populate Kauri with example data.

Always ask yourself, what would be the biggest pain points for continued development if I had to hand it over to someone else tomorrow?

Monitoring an off-grid system with Kauri was originally published by Jordan Crawford at Jordan Crawford on March 18, 2018.

Combining phone and web technologies

2018-02-04T00:00:00+13:00

Ten percent of my work time is dedicated to professional development through two ‘Hackdays’ a month. For my most recent Hackdays I wanted to combine old-school phone technology with ‘modern’ web technoligies. I came across a service called Plivo which lets you work with the phone network through a web API.

What is Plivo?

Plivo is a web service that gives us a web API to interact with the phone network. You rent a number with Plivo and pay a charge for each minute of calling or SMS message sent.

What can Plivo do?

There seems to be a lot of things you can do with Plivo, but the main ones are:

Sending and receiving SMS messages.
Receiving inbound calls and making outbound calls.
On a call you can:
- Convert text to speech.
- Play audio by providing a URL to an MP3 file.
- Accept digits from callers (e.g.: “press 1 for sales” in phone menus).

Making something with Plivo!

My initial plan was to implement a game of Hangman over SMS, especially considering I’d written a game of Hangman back on my devtrain. However, I quickly discovered that Plivo didn’t support receiving SMS messages in New Zealand, only sending them. Next I thought I’d just make a phone call based Hangman game, but sadly Plivo doesn’t support speech recognition making it difficult to receive character inputs.

Instead, I ended up implementing a simple number guessing game. This was a great place to start because the logic for such a game is very simple which allowed me to focus on the phone integration side. The game chooses a random number between 1 and 100 and asks the caller to guess the number by entering it on their dialpad. The game gives feedback on whether the guess is correct or higher/lower than the actual number. If the guess is wrong the caller can keep guessing until they get it correct.

How do I work with Plivo for calling?

Something I was unsure about initially was how you’d actually handle a call. I’d assumed that after a call was picked up that you’d need to deal with the audio stream yourself, but in fact, Plivo handles the entire audio stream for you!

When a call comes in, Plivo hits your API which provides a set of instructions in XML. Depending on the actions you ask for, you may need to define a callback URL for Plivo to hit later with more data (e.g.: a URL for it to hit for instructions after receiving digits from the caller).

It takes a little while to think about phone calls in terms of API requests, but this approach means that you can scale your phone system just like you scale your normal web API’s.

Implementation time!

I used Node and Plivo’s Node SDK to build the hotline’s API. Plivo’s SDK handles all the XML instruction formatting for you, making implementation very simple. A good reference for this was Plivo’s Phone IVR guide which introduces all the Plivo functionality I used for the project.

The game is pretty simple, it consists of only two endpoints for Plivo to hit; one for when the call is initially made to welcome the caller (and prompt them to enter a guess) and the other as a callback for the caller’s guess.

Welcome Route

When a call comes in, Plivo hits the / route with a GET request and the route returns the following XML to instruct Plivo:

<Response>
	<Speak>Welcome to the number guessing hotline.</Speak>
	<GetDigits action="[server URL]/guess?number=[chosen random number]" method="POST" timeout="10" numDigits="2" retries="3">
		<Speak>What is your guess?</Speak>
		<Play>[server URL]/elevator_music.mp3</Play>
	</GetDigits>
	<Speak>Huh? I didn't understand. Could you try again?</Speak>
</Response>

This simply tells Plivo to speak a welcome message then allow the caller to enter some digits with a defined callback URL. Plivo will ask the caller what their guess is and play some music while the caller enters their digits.

If the GetDigits action fails (e.g.: the caller takes too long) then the wrong input message is spoken.

Guess checking route

When the caller has entered some digits, Plivo will hit the guess URL with a POST request. Plivo puts the entered digits in a “Digits” field in the request (if you’re testing this out for yourself in a tool like Postman use the x-www-form-urlencoded format of POST body).

In addition to this, you may have noticed in the XML above, the games chosen random number is a query string parameter in the callback URL. This is because I’m super lazy and I didn’t want to have to persist the chosen random number for a call in a database. Getting Plivo to pass this parameter around for us means we can easily scale up the number of app servers as required for the millions of simultaneous calls required by our booming number guessing hotline startup!

You can checkout the code for the guess route, but it yet again returns some XML for Plivo:

<Response>
    <Speak>You guessed too [low/high]!</Speak>
    <GetDigits action="[server URL]/guess?number=[chosen random number]" method="POST" timeout="10" numDigits="2" retries="3">
        <Speak>What is your guess?</Speak>
        <Play>[server URL]/elevator_music.mp3</Play>
    </GetDigits>
    <Speak>Huh? I didn't understand. Could you try again?</Speak>
</Response>

Once again this makes use of GetDigits to prompt the caller for a further guess, making use of the same callback URL with the random number.

Or if the caller gets the number correct then it simply returns a simple <Speak> response. After this is spoken Plivo has no more instructions so will just hangup on the caller.

You can checkout all the code on GitHub at jordancrawford/number-guessing-hotline

Setting the hotline up on Plivo

When a call comes in, Plivo needs to hit a public API endpoint that we provide. This means we need to host our API somewhere on the internet.

If you aren’t comfortable with SSH’ing into a server to setup your app, Heroku is a good option that takes care of all the server management stuff for you. If you want to save a bit of money and don’t mind running the app server yourself then something like Hetzner, Digital Ocean or Vultr will work great. All these options use hourly (or in the case of Heroku, secondly!) billing so it won’t cost you much to mess around with Plivo. This might also be a good candidate for Serverless given the simplicity of our application code.

☁️

Need a cloud server?

I pay €3.79/mo for an ARM server with 4GB RAM, 2 cores and IPv4 with Hetzner
Sign up through this link for €20 starting credit!

Affiliate link

Next, buy a number through Plivo then create an application with your server URL as the answer URL and assign your number to your application.

You should now be able to call up your number and enjoy the phone hotline built with less than 100 lines of JavaScript!

Edit, 11 March 2018 The Plivo team also published this over on their company blog!

Edit, 5th May 2024 Some links have been fixed to point at newer versions of the Plivo docs.

Combining phone and web technologies was originally published by Jordan Crawford at Jordan Crawford on February 04, 2018.

Using tinc to access a home server

2017-10-29T00:00:00+13:00

About a year ago, I posted two articles about setting up my Raspberry Pi as a home server and how I setup remote access to it from anywhere in the world without using portforwarding. I’ve had a few people stumble across my post, asking questions and sharing details about their projects with me!

I thought I’d take the time to better address an area people seem to be having difficulty with, which is how to setup a tinc network between a cloud server and a computer on a home network.

What are we trying to achieve?

Lets say you want to access some content or service from a computer at your house from some other computer on the internet. But, like with most home network setups your computer isn’t reachable from the public internet. This is probably because your home network has a dynamic IP address and some other restrictions (like firewalls and NATs) that make connecting to it a challenge.

We can achieve this if the computer you want to connect from has a fixed public IP address. But how? Well, the computer at home can still make outgoing connections to the one with a fixed IP address. We can tunnel network traffic through this connection to link the machines together; this is called a Virtual Private Network (VPN).

In my situation, I wanted to get access to services on my Raspberry Pi at home from anywhere on the internet without needing to configure port forwarding. The first step in this was to establish a reliable VPN between my Pi and cloud server so that when I access an address like “plex.jc.kiwi” my cloud server can pass the traffic through the VPN link to the Pi at home.

tinc is an awesome open-source piece of software that we’ll use to setup this VPN link. My Raspberry Pi runs HypriotOS (but Raspberry Pi OS is a good option too). At the time of writing I used a Vultr cloud server running Ubuntu.

Installing tinc

You’ll need tinc installed on both the home server and the cloud server.

Manual installation

Installing tinc will vary by operating system, so Google is your friend. But, if you’re feeling lucky, try the following commands:

sudo -s
apt-get update
apt-get install tinc

If you don’t have any luck, you could try compiling it from source.

Verify your installation works by running the command tincd --help. If this outputs a message explaining all the options of tinc then everything worked fine!

Your tinc config directory will be /etc/tinc by default.

With Docker

I personally use Docker to run tinc in a container on my home server - I wrote more about Docker in my Raspberry Pi home server article.

To get tinc running I used jenserat’s tinc for the Linux x86 server for my server and created a Docker image of my own for my Raspberry Pi (source code). However as of 2024 the jenserat image is very out of date and I’ve deleted my image from Docker Hub as it was out of date and insecure. I’d recommend installing the package manually or finding a Docker image which is more regularly updated.

Make a directory somewhere on your system for your tinc config. For both the above Docker images, you should configure a volume mapping from your tinc config directory to /etc/tinc inside the tinc container.

Planning our configuration

We’re using tinc for quite a simple usecase, and as a result, the only things we need to plan out in advance are the names and IP addresses of each of our servers on the tinc network.

I’ll call my cloud server with the public fixed IP address cloud, and my home server behind a home network home.

For the IP addresses, we want an IP address range that won’t clash with my home’s local network. I find the 10.0.0.XXX works pretty well, so I’ll configure the IP’s as below.

Computer Name	tinc IP
cloud	10.0.0.1
home	10.0.0.2

This means that from the cloud server, accessing 10.0.0.2 will route us to the home server on the other side of the tinc tunnel, and vice versa.

Configuring `cloud`

Lets start by configuring the cloud server.

You can find my full example configuration for the cloud server on GitHub.

☁️

Need a cloud server?

I pay €3.79/mo for an ARM server with 4GB RAM, 2 cores and IPv4 with Hetzner
Sign up through this link for €20 starting credit!

Affiliate link

Config file

In your tinc config directory, open your favorite text editor and make a tinc.conf file.

Lets fill this in with the following:

Name = cloud
AddressFamily = ipv4
Interface = tun0

This just says that the name of our server on tinc is “cloud”, that it uses IPv4 and uses a network interface called tun0.

Up and down scripts

Next we need to setup a tinc-up and tinc-down script. These are what tinc uses to attach itself to it’s network interface. Make a tinc-up file containing the following:

ifconfig $INTERFACE 10.0.0.1 netmask 255.255.255.0

Then, make a tinc-down file containing the following:

ifconfig $INTERFACE down

Lets make these scripts executable by running the command: chmod +x tinc-*

Public and private keys

Next, we need to generate the public / private key pair for cloud.

This will create an entry for cloud in the hosts directory, but we need to make a hosts directory first! Run: mkdir hosts to make this directory.

Generate the keys with tincd -c . -K and hit enter when it asks where to put your private and public keys.

Here’s what this command looks like for me:

root@vultr:~# tincd -c . -K
Generating 4096 bits keys:
......................................................................................++ p
..............................................++ q
Done.
Please enter a file to save private RSA key to [/root/tinc-example-cloud/rsa_key.priv]:
Please enter a file to save public RSA key to [/root/tinc-example-cloud/hosts/cloud]:

(if you’re running with Docker, use docker run -v [your tinc config path]:/etc/tinc --name tinc [image name] -K to generate keys instead)

You’ll now have an rsa_key.priv file with this servers private key and a cloud file in the hosts directory with it’s public key.

Now we just need to add some additional information to our hosts/cloud file. Edit this file by adding an Address and Subnet so it looks like the following:

Address = [the hostname or IP address of the cloud server. e.g.: server.mydomain.com]
Subnet = 10.0.0.1/32

-----BEGIN RSA PUBLIC KEY-----
[the generated public key for the cloud server]
-----END RSA PUBLIC KEY-----

Configuring `home`

Now, login to your home server and we’ll set it up in a similar way.

You can find my full example configuration for the home server on GitHub.

Config file

Make a tinc.conf file with the following contents:

Name = home
AddressFamily = ipv4
Interface = tun0
ConnectTo = cloud

Similar to cloud, this defines the name, address type and interface for tinc. However, in addition, this also tells it to make a connection to the server called cloud.

Up and down scripts

Like last time we’ll make a the tinc-up and tinc-down scripts.

Your tinc-up script would be:

ifconfig $INTERFACE 10.0.0.2 netmask 255.255.255.0

And your tinc-down would be the same with:

ifconfig $INTERFACE down

Once again, make these scripts executable by running chmod +x tinc-*.

Public / private keys

Make the hosts directory with mkdir hosts, generate the keys with tincd -c . -K then add the Subnet to your hosts/home file so it looks like the following:

Subnet = 10.0.0.2/32

-----BEGIN RSA PUBLIC KEY-----
[the generated public key for the home server]
-----END RSA PUBLIC KEY-----

Telling the servers about each other

The final step is to tell the servers about each other. The information that each server needs about the other is all contained in the hosts directory. In these host files the Subnet tells us which IP each server should have once connected while the public key is used to secure the connection and verify we’re actually communicating with the server we expect!

This step is pretty easy to do, we just need to make the hosts directorys the same across both systems! Just copy your hosts/cloud file from cloud to the hosts on home, then copy your hosts/home file from home to hosts on cloud.

Testing everything works

Finally, the moment of truth! Run tincd on both of your servers.

You might benefit from initially running in no-detach mode with tincd -D. If you hit Ctrl + c this will increase the log level of the server so you can see all the messages between the two servers. To exit, hit Ctrl + \.

If everything worked correctly you should be able to access home from 10.0.0.2 on cloud. You might want to try initiate an SSH connection or access a web service from home, e.g.: ssh [you]@10.0.0.2 or wget 10.0.0.2.

Nice one! You’ve now setup a VPN between a cloud server and your home server behind your restricted home network. Next up you could try setting up an HTTP proxy on your cloud server to access your home services from anywhere.

Edit, 13 Oct 2024: I have removed a dead link to a Raspberry Pi tinc Docker image I made. This image was very out of date so were a security risk. As I am no longer in a position to maintain this image so I have deleted these images and make more up to date recommendations here.

Using tinc to access a home server was originally published by Jordan Crawford at Jordan Crawford on October 29, 2017.

Starting out as a developer at Flux

2017-03-12T00:00:00+13:00

In November of last year I finished my Software Engineering degree and moved to Wellington to start work at Flux Federation. Flux builds software that runs energy companies like Powershop - an energy retailer in New Zealand, Australia and the United Kingdom. I work as part of the team that builds the Rails app that brings the power to the people.

My first few months were spent in training on what Flux calls the Dev Train.

The Dev Train

I had never done anything with Ruby before, so it started with building a command line game of Hangman in Ruby, followed by a Rails version.

_{My first Ruby program, command line Hangman!}

While building these myself and the other Dev Trainees had code review sessions with Dev Train mentors. These sessions taught us how to best design software systems and write clean, maintainable code.

Next, I made a Rails version of the Ticket to Ride boardgame. I approached this using test driven development (TDD). This basically means you write some tests for new features, then write the code that makes the tests pass, then refactor the code and tests.

Testing!

For me, one of the biggest take aways of the Dev Train was that testing is important! Yes, tests can take a lot of time to write, but especially in a large, complex codebase they become absolutely essential to ensure everything is still ticking along as it should.

For Ticket to Ride I wrote unit tests in RSpec and integration tests with Cucumber.

RSpec

RSpec allowed me to test the public methods in my controllers, models and services. I found the ability to mock out dependant services incredibly useful. This ensures your specs are only concerned with the code under test, rather than all it’s dependencies. Mocking means that rather than spending time setting up situations where a depdendant service will return a particular result, you can just mock it to return a particular result and make assertions about how the code under test should respond.

For example, I have a ClaimRouteController which calls a ClaimRoute service. In my specs for ClaimRouteController I mock out ClaimRoute and assert that when ClaimRoute returns errors that these errors will be added to the flash to be displayed to the user. This means I don’t have to do the work of setting up a situation where ClaimRoute will fail when all I care about is that the controller handles errors appropriately.

Cucumber

Cucumber allows you to automate the process of a user testing out all the features of your application in a browser. A Cucumber feature consists of a series of steps to run the test.

A Cucumber feature is pretty easy to read. For example, as below, the Claim Route feature sets up a game with some train pieces and train cars, then proceeds through the steps to claim a route between two cities. This feature is successful if after doing this the list of routes shows this route as being claimed by the player.

Each of these steps has a corresponding step definiton. Step definitions are the code behind the scenarios that executes the step. You can use regular expressions in the names of steps to pass parameters to step definitions, keeping steps sounding natural and reusable.

My Ticket to Ride Implementation

It’s fair to say I went overboard on testing my Ticket to Ride game. Despite having ~250 RSpec examples and a further 12 Cucumber scenarios, my game only got as far as allowing users to:

Setup a game.
Draw additional train cars.
Spend train cars and train pieces to claim a route.

Either way, you can checkout the code on GitHub and some screenshots of the app below.

Game Board

Claiming a route

An Awesome Learning Culture

I’m now finished the Dev Train and onto ‘real work’ now, but that doesn’t mean the learning stops! With the tech industry moving so fast there is always something new to learn. Flux has an awesome learning culture; each week a few members of the crew give a talk about something that interests them, and 10% of our time goes towards learning new things with Hackdays projects.

I’m really enjoying my time at Flux and I’m looking forward to seeing what’s in store for the next few years!

Edit, 29 Oct 2017: The software dev arm of Powershop split out into a company called Flux Federation a few months ago. It’s still the same awesome place to work so I’ve replaced references to Powershop with Flux.

Starting out as a developer at Flux was originally published by Jordan Crawford at Jordan Crawford on March 12, 2017.

Spontaneous Project: Latest Trailers

2017-03-11T00:00:00+13:00

Last weekend I wanted to see the latest movie trailers. Of course, there are plenty of websites out there for this, however, none of them meet the excitement of watching movie trailers before a film at the cinemas.

So that’s what I set out to build. First, I needed to find a source for movie information. I don’t want to populate this information manually, so I found TheMovieDB has an API. This site is great because there’s a whole community of people dedicated to keeping this information up to date.

I linked in with their API and pulled down details of upcoming and now showing movies. TheMovieDB’s API lets me get a list of movies with one request, however it requires another request to fetch the videos (including trailers) for each movie. TheMovieDB has an API limit of 40 requests every 10 seconds, and if my site got popular it could exceed this and I don’t want to give out my TMDB key to any visitor!

Instead, I setup my VPS with a nightly job that fetches data from TheMovieDB and saves it to an Amazon S3 bucket. This job delays it’s requests to avoid the TMDB API limit and keeps only the movie ID and YouTube trailer ID’s. This method saves bandwidth and time, as visitors to the site only need to make one request to fetch all the data they need! As the entire site is purely static, everything can be hosted on Amazon S3 for simplicity and scalability.

Next, I used the YouTube Player API to embed trailers. This let me hook into events like when the video has finished so that I can start the next one playing. I was surprised at the level of control they give embedders, allowing me to prevent annotations, video controls and other distractions for a pure video experience.

A core feature of the site is that it only plays trailers you haven’t seen. To keep things simple, I used local storage in the browser to store the list of movies the user has seen. When the user watches all the available trailers, they can clear out their list of seen movies to start over!

This simple project took longer than I expected but I’m pleased with the result, and it’s where I’ll go to get my trailers from now on!

Check it out at trailers.jc.kiwi! Or view the source code on GitHub.

Edit, 7th Oct 2023: The domain has been updated to trailers.jc.kiwi

Edit, 21st Oct 2023: The project has been open sourced under the MIT license - see it on GitHub.

Spontaneous Project: Latest Trailers was originally published by Jordan Crawford at Jordan Crawford on March 11, 2017.

Dusty Shoeboxes to a Family Photo Archive

2017-01-21T00:00:00+13:00

For a few years now I’ve been wanting to organise a family photo and video archive. Digital content is much easier to deal with, but my family members still have boxes of old photos sitting around in their cupboards. In our case, this is basically everything prior to the year 2000.

Capturing the content

These physical photos degrade over time and are vulnerable to being destroyed in a disaster. This is an easy problem to solve by scanning the photos so we have a digital copy.

But the photos alone are worthless. It’s the associated memories that make them valuable. In an archive then, it’s essential that the who, what, where and when of each memory is identified. This is difficult, and even more time consuming than the digitisation process, but I think it’s worth it! Whats the point in having a picture if you don’t know what its about?

Making the content accessable

Capturing the content is time consuming but relatively simple. But why bother doing this if no one can enjoy it? Making the archive accessable to the whole family is arguably the most important part, and it’s currently proving to be the most challenging.

This solution must:

Make sharing and accessing content with the entire family easy, ideally with a web link.
Be able to show photos alongside their descriptions and other metadata.
Allow content to be structured in a way that makes sense for families, like filtering by a branch of the family or a type of event.
Be able to contain videos, some of which may be long (potentially several hours worth of home videos!)
Ensure I retain control of the data to so it will be in a usable form years into the future.
Be secure so only my family members have access.

Flickr came close to this with their “family” and “friends” sharable URL’s, but this meant loosing full control of the data and made finding content difficult.

Let me know if you have any suggestions of solutions, but at this stage I’m looking at building a system of my own that can fufil these requirements.

In the 21st century we have an amazing opportunity to preserve these memories effectively. It will be amazing when future generations can so easilly look back at the past. Time is of the essence so get out there and ensure these memories are captured properley!

I wrote a poem as inspiration, enjoy!

My colour fades as I’m forgotten.
They used to visit me, smiling as we reminisced.
But its been many moons since I’ve been missed.

I anxiously wait for my eventual doom.
Sitting here, in this small dark room.
Have I slipped their mind?
Will they leave me behind?

The bright light hits.

My time has finally come.
But don’t be sad, this is the best possible outcome.
Released from my fragile form, I flutter to the clouds.

This is my next chapter.
A digital life is the answer.
In my binary form, I will live forever on.

Dusty Shoeboxes to a Family Photo Archive was originally published by Jordan Crawford at Jordan Crawford on January 21, 2017.

Easy access to my Pi on a local network

2016-11-01T00:00:00+13:00

I have a Raspberry Pi home server that I can remotely access through a tinc VPN tunnel with my VPS. Most services can be accessed through the tunnel with addresses like plex.crawford.kiwi, but some of them are only available on my local network, for example, SMB and SSH. Accessing these services from the local network is actually more difficult than the services available remotely for two reasons:

The local IP address may change. Most routers let you configure a MAC address based IP reservation, however one of my original goals for this system was that it should be able to work even in situations where I don’t have control of the router.

Differences between networks. The local address space (e.g.: 192.168.1.xx) varies between networks and the address you want to reserve may already be taken. This means that if you change your network you may also need to change your computers configuration to access the Pi at it’s new address.

IP addresses are hard to remember. There is a reason DNS was invented! DNS allows a friendly URL to map to the underlying IP address.

My solution is to run a DNS server on my VPS that fetches the IP address from the Pi through the VPN tunnel when a request is made. This means to access my Pi through the local network, I just punch in an address like local.pi.crawford.kiwi and I can access it no matter what it’s local IP address is!

The DNS server that runs on my VPS server is something I created called local-address-dns. This runs a DNS server using dnsd and upon an incoming DNS request it connects to local-address-dns-client-rpi running on my Pi. local-address-dns-client-rpi is a simple web server that returns the Pi’s IP address on one of it’s network interfaces. An NS record on my domain points addresses like local.pi.crawford.kiwi to my VPS to be handled by local-address-dns.

_{Connecting to SMB on the Pi. I don't ever need to worry about what the Pi's IP address is!}

This works great for services like SSH and SMB which tend to use a default port, but what about when you run multiple web-based serices off the same machine? For example, if you run Plex on port 32400 and Deluge on port 8112.

In these cases, I’d love to just use a domain name like plex.local.pi.crawford.kiwi over having to remember local.pi.crawford.kiwi:32400!

Theres a few answers to this question. One is to configure NGINX to proxy the service based on the domain name being accessed, or simply to redirect to another URL based on the domain name being accessed.

I went with the redirect based approach and created virtual-host-redirector for this purpose. This makes configuration easy as you just need to setup a rules.json file with host to URL mappings.

With either of these approaches you’ll need to:

Make sure you aren’t running anything else on port 80. If you are, just reconfigure this service to another port and setup a redirect rule for it.
Configure a wildcard CNAME record with your DNS provider to the local-address-dns managed domain name. For example, mine is a CNAME record of *.local.pi.crawford.kiwi pointing at local.pi.crawford.kiwi.

Edit, 1 July 2018: I’ve added some more details about another project of mine called virtual-host-redirector.

Edit, 12 September 2023: I’ve removed the Docker images for local-address-dns, local-address-dns-client-rpi and virtual-host-redirectormade them all Public Archives on GitHub. The code is still available for inspiration but is not being maintained - especially as they were getting very behind on security updates.

Easy access to my Pi on a local network was originally published by Jordan Crawford at Jordan Crawford on November 01, 2016.

Accessing an IP Camera over the Internet

2016-10-20T00:00:00+13:00

My parents just got two IP cameras. These are cameras that connect to your home network to allow access over a computer network.

I had some requirements to make the IP cameras useful:

Remote Access. Obviously, an IP camera that can only be accessed from the local network isn’t very useful if you want to check up on the home! The cameras should be accessable from anywhere in the world via an internet connected device.
Security. The internet is full of stories of the dangers of unsecure IP cameras, such as the website allowing public access to thousands of them. I want all access to the cameras to be encrypted and require authentication.
Compatibility. The IP camera’s built-in web interface requires a browser plugin. I couldn’t even get this going on my computer, let alone a mobile device. Accessing the cameras should be possible from any web browser and mobile devices.
Recording. The footage from the cameras should be recorded. Obviously, these cameras aren’t being watched 24/7 so their footage should be able to be retreived at a later point if a need arises.
Simplicity. Last, but not least, access to the cameras should be simple. My way of judging this is that I should be able to get into the cameras on any computer without needing a bookmark or note.

Finding a solution

I discovered that the cameras we have run a Real Time Streaming Protocol (RTSP) server so the awful web interface is optional. Interestingly, despite the web interface being password protected, the RTSP stream is wide open!

My first thought was to build a web interface of my own that provides security and HTML5 support by transcoding the camera stream. This could have been a fair bit of work and doesn’t address the recording requirement.

However, I then discovered AngelCam. This service connects to your IP cameras and lets you access them from a browser or their iOS/Android app. In addition to meeting my requirement of recording, this also means all the recordings are safe off-site, making it difficult to wipe the footage.

_{A screenshot of the AngelCam recordings page. The footage preview has been pixelated intentionally.}

There are also plenty of other optional features like public broadcasts, timelapses, and a few more features in the works, like licence plate recognition or alerts when a line is crossed.

Setting up AngelCam

There are a few options to connect cameras to AngelCam.

AngelCam ready camera. Some cameras have built in support for AngelCam, not mine. Next!
Providing the camera URL. You can directly provide the URL of the camera to AngelCam. Most likely this won’t be encrypted and may rely on portforwarding and a dynamic DNS service.
AngelBox. AngelCam sells AngelBox, a Raspberry Pi loaded with AngelCam’s software. This sets up a secure connection between your cameras and AngelCam’s servers, meaning they don’t need to be publicly accessable.
Arrow client. The Arrow client allows you to use your own computer like an AngelBox through the open source code.

As we’ve got a home server, I wanted to use this as the Arrow client. I made a Docker image to run the Arrow client easily and on any operating system. As of 2023, I no longer maintain this Docker image - but you can see the public archive of the source code on GitHub.

Evaluation

AngelCam meets all my requirements as it provides secure remote access and works on any device. It is simple to access, only requiring visiting the AngelCam website and remembering the username and password, and it provides recording.

AngelCam has worked well over the last few months, and I’m looking forward to seeing their feature set expand over time.

_{Please note: I have no affiliation with AngelCam, and I receive no benefit from this post, it’s just simply the solution I discovered.}

Edit, 27 Dec 2016: Edited to reflect the fact that AngelCam has now removed their free plan. It’s likely still good value for the convenience, but as we have a home server I’m now considering self-hosted alternatives.

Edit, 30 Aug 2023: Edited to remove the link to the arrow-client Docker image which I no longer maintain.

Accessing an IP Camera over the Internet was originally published by Jordan Crawford at Jordan Crawford on October 20, 2016.

Accessing home services from anywhere, without port forwarding!

2016-07-06T00:00:00+12:00

My Raspberry Pi is setup as a home server, providing me access and control of my content through several services:

A Deluge server for torrent downloads.
A Plex server to manage and stream my media collection.
A Pydio server for remote access and management of files.

This is great, but I want to access my content when I’m away from home. Previous experience with remote access solutions inspired some requirements:

I should be able to access my services from any computer in the world.
I should be able to punch in an easy to remember web address like deluge.crawford.kiwi. This means all standard HTTP/HTTPS ports should be used!
All transmission of my content should be encrypted.
Moving house or ISP shouldn’t break my remote access.

Problems with home internet connections

The hardest requirement to cater to is that moving house or ISP shouldn’t break my remote access. Why is this such an issue?

May not have a static IP

Very rarely does a home connection come with a static IP. Getting a static IP typically results in extra charges, and would require updating the DNS record if anything changes. The solution is to use a dynamic DNS service which solves both these issues, but we still end up with the below problems.

May not have control of the router

Port forwarding needs to be setup on your router so incoming connections are forwarded to the home server. However, you may not have full control over the router to setup these rules. Many routers will also take the default HTTP/HTTPS ports for their own services, leaving you with non-standard port numbers for everything else.

May not be able to get incoming connections

Finally, once we’ve sorted out everything above, you may not be able to get incoming traffic to reach your house! As a result of the IPv4 address shortage and ISP firewalls, incoming connections don’t always work. Never fear, we can still tunnel traffic through the internet with the help of another computer with a more accessible connection.

Tunnelling out

The concept of using a tunnel is pretty simple. We may not be able to get incoming connections to the home server, but the home server can setup an outgoing tunnel connection with some other machine on the internet. This other machine can be accessed from anywhere and forward connections through the tunnel to get to the home server.

The Other Machine

To meet my crazy requirements above, some other machine needs to be involved. There are services dedicated to providing remote access to your networks like Hamachi. These work well but require client software to join the virtual network, not meeting my first requirement of working on any computer!

Other than that, there are virtual private network (VPN) services which provide port forwarding, however it’s unlikely you’d be able to use the HTTP and HTTPS ports.

Last resort, DIY! A virtual private server (VPS) is a cheap way to get a small cloud server with a decant connection and its very own IPv4 address! For this I grabbed a VPS server from Vultr, whose cheapest server has more than enough grunt to provide remote access.

Reverse SSH Tunnel

A common way to get remote access through a firewall is with a Reverse SSH Tunnel. This is easy to setup and works well, but I discovered that HTTP based services through the tunnel run extremely slow. The most likely reason for this is that both SSH and HTTP use the TCP protocol to transmit data over a network. TCP ensures a reliable connection with built-in error checking and transmission control but this comes at a cost of speed. Running HTTP through the SSH tunnel is performing these error checks twice, resulting in much slower speeds.

The answer is to switch to something UDP based. UDP is much faster because it’s just packets sent straight over the network without error checking or flow control. A UDP based tunnelling solution means that only the HTTP layer is performing these extra tasks.

tinc VPN

The tinc VPN software was the answer. tinc can be used to create virtual networks between computers. It utilises UDP so runs quickly, all traffic is encrypted, and it’s continually re-checking the status of its VPN connection so works well even on unreliable connections.

I posted an article about how to configure tinc for this purpose, but this DigitalOcean tutorial is good resource too. This involves setting up the network’s configuration, generating key-pairs and copying the key-pairs between the machine.

To get tinc running I used jenserat’s tinc for the cloud server for my server and created a Docker image of my own for my Raspberry Pi (source code). However as of 2024 the jenserat image is very out of date and I’ve deleted my image from Docker Hub as it was out of date and insecure. I’d recommend installing the package manually or finding a Docker image which is more regularly updated.

☁️

Need a cloud server?

I pay €3.79/mo for an ARM server with 4GB RAM, 2 cores and IPv4 with Hetzner
Sign up through this link for €20 starting credit!

Affiliate link

Pretty URL’s

Proxying Connections

With tinc working, all the services on the home server can be accessed through a local IP on the cloud server, like 10.0.0.2:8112 for Deluge. Time to turn that into something nice like deluge.crawford.kiwi!

The subdomains point to the cloud server’s IP address. The cloud server has a NGINX server running with the official NGINX Docker image. NGINX is setup as a proxy server to the home server’s IP address using the NGINX documentation, meaning external traffic is forwarded through the tinc link to the home server.

HTTPS

HTTPS is used to ensure all data transport is encrypted. This requires valid SSL certificates which can be obtained for free with Let’s Encrypt through their automated verification process. To setup Let’s Encrypt to automatically renew I used bringnow’s docker-letsencrypt-manager and shared the volumes with the NGINX container. A very useful tool when working with HTTPS is SSL Labs’ SSL Test.

Using Plex

Plex is a powerful media server, allowing access to your media from anywhere. Plex offers Relay, a feature that allows tunnelling into your home server in a similar way to my setup. This works well, but does have bandwidth limitations. To setup MyPlex through the tinc tunnel I used this Gist to help with the NGINX configuration, and setup a custom URL in my Plex settings, like: https://plex.crawford.kiwi:443 (the port number must be explicitly defined for it to work). This works well, but I did have difficulties when Chromecasting from Android while on an external network, hopefully Plex will release a fix for this.

If you need some help crafting your NGINX configuration, checkout my NGINX example configuration on GitHub.

Take Away

While my original requirements were pretty over the top, I’m happy I have a solution which satisfies them. I have been using this setup reliably for about three months and having an easy to remember web address for all my home services is great. This process has taught me a lot about Docker, NGINX, and networking, much of which has already become useful in other contexts.

Edit, 29 Oct 2017: Added a link to my new tinc setup guide.

Edit, 13 Jan 2020: Added a link to my new NGINX configuration example.

Accessing home services from anywhere, without port forwarding! was originally published by Jordan Crawford at Jordan Crawford on July 06, 2016.

My home server powered by Pi and Docker

2016-04-09T00:00:00+12:00

I was in need of a server that gives me remote access to my files, can run Plex, torrent and is quiet and efficient enough that it can run 24/7.

The answer? A Raspberry Pi of course! I brought mine for about $60 NZD!).

What is Docker?

I used Docker to manage the services on the Raspberry Pi. Docker containerises services allowing them to be shared and scaled easily. For me, the main benefit was that setting up a service is as easy as downloading a Docker image rather than manually installing packages and making configuration changes.

Core Concepts

An image is the starting point to setup a service. These typically come from Docker’s repository for images called Docker Hub, but failing that you can always setup an image of your own. An image includes the operating system and all other software and configuration needed to run the service.

A container is an instance of an image. Each runs independently with individual file systems and environment variables. You can run multiple containers for the same service without them clashing.

While this sounds like a performance nightmare, Docker doesn’t actually run containers as virtual machines but they instead share the same kernel. This means that even on a little device like the Raspberry Pi you can still run plenty of containers at once, even hundreds.

Running Docker on the Raspberry Pi

The Docker community for systems with ARM processors is growing. A big name in this space is Hypriot with HypriotOS. This is a lightweight operating system for your Raspberry Pi with Docker built in.

Only Docker images built specifically for ARM processor’s will work on the Pi. Usually adding rpi as a search term in Docker Hub helps find compatible images.

Setting up an external hard drive

I chose the EXT4 file system as this doesn’t require additional software to be installed on the Pi and compatibility with other systems isn’t a major requirement for me. I setup the system to auto mount the drive using it’s UUID and setup the drive to automatically go to sleep using hd-idle.

Managing service configuration

I used Docker Compose to manage services on the Pi. Docker Compose works as a layer on top of the Docker command line meaning you can start and manage the entire stack with a single command.

The Docker Compose file containing the services stack can be stored in Git so you always have access to your working configuration. Re-installing is easy, just clone the repository and run docker-compose up -d, then wait as the images download.

Services

Pydio

Pydio is like Google Drive for your personal content. You can get access to it from anywhere in the world, give people accounts to access your content, or share a folder to anyone with a public link.

To get Pydio running I created a Docker image of my own - however, as of 2024, I have deleted this image from Docker Hub as it was very out of date and insecure. I’d recommend checking out linuxserver/pydio.

Plex

Plex is a server for your personal media collection. It collects brilliant metadata about your content and allows it to be played pretty much anywhere with the web, desktop, TV and mobile apps. To get Plex running I used greensheep/plex-server-docker-rpi.

Plex runs well, and will transcode most files for use in the Plex web-app or Rasplex, but some formats have issues like H.265 files. Everything works well on a more capable client like a computer or phone.

Deluge

Deluge is a torrent download server. The best thing about Deluge is how client-server oriented it is. You can connect to a Deluge server through the desktop app or the web interface. Torrent is great from the Raspberry Pi because it’s always wired into the network and is on 24/7.

To get Deluge running I created a Docker image (source code) - however, as of 2024, I have deleted this image from Docker Hub as it was very out of date and insecure. I’d recommend checking out linuxserver/deluge instead.

Samba

Samba allows you to connect directly to the drive from other computers using SMB. I only use Samba on my local network, with Pydio taking over for remote situations.

To get Samba going I used dastrasmue/rpi-samba - however, as of 2024 this image is fairly out of date. I’d recommend checking out dockurr/samba instead.

The drive access speeds aren’t amazing through Samba, but I’m hoping this will improve with the Pine 64’s gigabit ethernet and faster CPU.

Getting remote access?

Services can be accessed over the local network at a particular port or port forwarded to the internet, but a domain like plex.example.com beats 123.456.789.123:2016 anyday!

My next post covers how I got remote access to my Pi without any port forwarding or a static IP.

Take Away

I’ve learnt a few things from this experience:

Docker is simple to use once an image has been made, but making an image of your own can take much longer than you might expect.
Directly connected storage is significantly better. Initially, the Raspberry Pi was connected to my old Time Capsule unit using Docker Volume Netshare. This worked well but in the end is a much more complicated solution than required.
Have a good look through Docker Hub before creating a Docker image. There are plenty of images out there you just need to look hard, and of course if you do implement your own service, please release it for everyone to use!

Edit, 13 Oct 2024: I have removed some dead links to some old Docker images I had created. These images were very out of date so were a security risk. As I am no longer in a position to maintain these I have deleted these images and make more up to date recommendations here.

Edit, 2 Feb 2025: I have updated some links to the Hypriot website - they now appear to be hosted on hypriot.io rather than hypriot.com

My home server powered by Pi and Docker was originally published by Jordan Crawford at Jordan Crawford on April 09, 2016.

Creating Osprey Launcher

2015-05-17T00:00:00+12:00

Over the summer of 2014 I decided to enhance our home theatre PC’s with a launcher that simplifies the use of media player apps and web content, while saving time and energy.

The Problem

The world of home entertainment is changing rapidly, especially with the advent of connected devices like the Apple TV, Roku and other ‘smart TV’ solutions combined with the ever-changing world of cloud services.

However, for now, the best all-round solution for home entertainment remains to be home theatre PC’s. At home we’ve got two HTPC’s and a server. I’ll get to the server another day, but for us the best solution is a combination of three programs:

Plex, for playback of local media collections
MediaPortal, for playback of recorded broadcast TV
Kodi, for streaming

Kodi is actually capable of doing it all, using the PleXBMC and MediaPortal addon, but you really can’t beat the native experience of each app.

Without a mouse and keyboard, it’s hard to switch between all these apps. Also, waiting for each one to start up every time becomes frustrating.

This led to the following requirements:

Ability to switch between applications using a remote
Switching between applications should be fast
Applications should be able to be added easily
A user interface that be used for other helpful information

Development Process

The project was split into two parts. The frontend is all about giving a responsive and clear interface for the user, while the backend manages the applications and sends commands to the operating system.

Frontend

I started out with a simple Window’s Forms interface made up of a few circles. I quickly found that Windows Forms are limited in the visual effects they offer and aren’t very easy to work with.

The solution was to embed a web interface into the app. Why a web interface? You can make just about anything in CSS, the interface can be animated and used visual effects like shadows and rounding, and there is plenty of resources and frameworks available to make the job easier.

Also, because the web interface sits on the home server it is very easy to update the available applications rather than recompiling and redistributing.

Tools of the Trade

Over summer I was working on another project in AngularJS so I decided I’d build this in Angular too. Angular makes it easy to ensure the interface is always up to date with the model.

I also used Yeoman to setup all the basics, such as the Bower package manager, and Grunt for task running.

Using Bootstrap allowed me to get started easily, as well as make the frontend responsive so it would work well for different resolution TV’s.

Design

I’m no interaction design expert, but with enough iterations the interface got to a point where it’s both easy to use and flexible.

The initial layout looked pretty nice when it was built, however I got to the stage where I wanted to be able to add more applications and have the interface more responsive to different screen sizes.

I decided to switch to an Apple TV inspired grid layout. This was easily responsive and could hold as many applications as I needed by scrolling the page when it got to the bottom.

The backdrop of the launcher was taken by Avery Photography.

Embedding the user interface

Windows Form’s has a built in web browser but it’s based on Internet Explorer. Enough said, it was time to embed Chrome. CEF is an embeddable version of Chrome, and CefSharp allows CEF to be embedded within a C# application!

This also allows me to call JavaScript functions from C#, and call C# methods from JavaScript, which is much easier than communicating through sockets or HTTP calls.

Backend

The backend’s job is to manage applications based on commands from the frontend.

Language of Choice

Our HTPC’s both run Windows 7 and I’ve used C# before, so this was my language of choice.

Design Patterns

Having just completed an object oriented programming paper at university I was keen to put this to use with some design patterns.

Without these design patterns, my source code would become a mess and any future changes would become painful.

A Facade is used to handle interactions between other classes, acting as the one class every other class can talk through. For example, if the FrontendBridge gets a message from the frontend to open an application, the facade finds the application and sends it the open message.

State ensures that the state of an application doesn’t change the way we interact with it. Other classes have idea what state an application is in, they just send it commands. This makes implementing new states easy, and saves plenty of ‘if’ statements.

This is used by storing an ApplicationInstance in LaunchableApplication, and a method to change the current application instance (for state transitions).

Singleton’s are very necessary for implementing a Facade or the FrontendBridge (things there are only ever going to be one of). A singleton is a static method that initialises an instance of a class or just returns the single instance of the controller if an instance already exists.

Suspending Functionality

One of the most important aspects of this project is to be able to switch between applications quickly. The computers I was using aren’t particularly fast, so waiting 20 seconds for an application to launch gets annoying!

One option is to just keep all applications open which would work fine on some computers, but this would have a nasty effect on these machines. These computers are also left on all day and night, so reducing the amount of CPU will be better for my parents power bill!

The best solution for me was a command line utility called PsSuspend. Use of this is very simple, just launch PsSuspend.exe with the process ID to suspend, and add the -r flag to resume.

PsSuspend completely freezes an application so that it uses 0% CPU, but it remains completely open in memory.

Resuming a process brings it back to life instantly. I wired up the PsSuspend command up to to the close() method of my OpenApplication state (switches to the suspended state), and to resume when the SuspendedApplication state receives a launch().

Application Switching

Moving an application (or the launcher) to the front of the screen turned out to be harder than I expected.

For the longest time I was having an issue where an application could be launched fine, but would not return to the launcher if the application had received any keypresses. Even weirder was this only happened on the HTPC’s but not in my development environment. I even asked about the issue I was having on Stack Overflow, but gladly found the answer myself in the end.

The difference was that I was running with debugging while the other machines weren’t. The breakthrough came from reading a tutorial on CodeProject which explains the rules for getting focus from another window. If an application is being debugged then it is allowed to switch back to itself. The solution was in a blog article which avoided the limitations by programmatically pressing the ‘alt’ key - the article has since been deleted, but you can see the code I used to solve this issue on GitHub.

Catching remote button presses

I had trouble catching keyboard events while the launcher was not active, plus I needed support for a specific button on the remote. As a result, the easiest way was to setup EventGhost to make a UDP request to the launcher.

Using EventGhost gives good support for most remotes, and the fact that it’s UDP means a smartphone or other device could also trigger the launcher if needed.

Extensions and Further Development

Osprey Launcher continues to be used by my parents. Eventually I intend to prepare it for open source release, as well as adding a few extra features.

Launching Websites

It’s pretty neat to see how new ideas evolve as software is used, and this feature is a great example.

Originally, CEFSharp was just used to display the frontend. However, with a little modification websites were treated as launchable items, giving many extra uses for the launcher.

Desktop Mode

Something I really like about the launcher is that it simplifies the whole experience of using the HTPC by hiding out the complications of the operating system behind it.

The Desktop Mode adds to this experience, by treating the operating system below like an application. Apps open in the launcher are completely hidden, even from the Windows taskbar making for a pure Window’s experience while in desktop mode.

Force Close

Things don’t always work perfectly, especially on these slow old computers. So task manager doesn’t need to be launched, I added a ‘Force Close’ option in the context menu of a launchable application.

Automatically return to launcher

At most, the computer would only be used a few hours a day. If an application is left open, this can keep the fan on all day until the computer is used next.

To keep CPU cycles low, the launcher will pull out of an application after a few hours.

Frontend Widgets

The launcher provides a nice way to integrate the HTPC’s with information from the server. I made a few AngularJS directives to provide this information.

Number of recordings on MediaPortal today

The MediaPortal install on the home server has MPExtended installed, providing an web API into the MediaPortal database.

Counting the number of results returned by TVAccessService’s GetScheduledRecordingsForToday did the trick.

Hard drive space available on server

This displays the available space percentage shown from a PHP script on the server. This one is handy with a home server setup like this, as the server can easily go months without anyone logging into it.

Take Away

As well as creating something neat, this project has been a major learning experience, giving me some insights:

Spending time to think about the best class structure is worth it in the end
User input provides new ideas and changes, so get it into the hands of users as soon as possible
Don’t re-invent the wheel, there are plenty of frameworks and tools available to save time
The Windows API is a mysterious beast

The code

If you’d like to have a look at the Osprey Launcher source code, check it out on GitHub. Note that that as of 2023, I no longer maintain this repo and it may no longer work - but you’re welcome to try get it working or use it as inspiration for your own projects.

Edit, 15 July 2023 Edited to add a link to the Osprey Launcher source code.

Edit, 30 Aug 2023: Edited to remove the link to the (now deleted) blog article which helped me with the window switching.

Creating Osprey Launcher was originally published by Jordan Crawford at Jordan Crawford on May 17, 2015.

Project Launcher

2015-03-07T00:00:00+13:00

A small, open-source command line tool for quickly setting up your development environment.

I switch between lots of projects at the moment, so I wanted something that made this easier to do.

I just open a command line and run project [project_name] and it all starts up!

There is still lots of neat things this could do, and it should be made easier to work with other programs so feel free to contribute on BitBucket.

Preview

Project Launcher was originally published by Jordan Crawford at Jordan Crawford on March 07, 2015.

Virtualising an old Mac

2015-02-10T00:00:00+13:00

A company I help out with their IT stuff is about to get rid of their oldest computer - a 2006, Intel iMac. This is the last machine they’ve got that can run Tiger and Freehand MX, so it’s time to virtualise!

Freehand MX is an old PowerPC application. PowerPC was the processor architecture Mac’s were using until the Intel switch of 2006. Rosetta is a PowerPC to Intel translator that was built into OS X to make this switch easier, however this isn’t around in Lion or later. Going forward, the only real way to use Freehand MX is in a virtualised environment of an older version of OS X.

I want to use VirtualBox for this. VirtualBox is great because its free and works well enough for what I need. Once I had a version of OS X running inside VirtualBox, the next part is to migrate the existing OS X install into the virtual machine.

Migrating from a real partition to a virtual machine

Newer versions of OS X allow Migration Assistant to transfer files over the network. However, in Tiger you need to use Target Disk Mode and a FireWire cable. This just lets you use your Mac like an external hard drive for the other Mac to migrate off, but it’s a little difficult (impossible!) to plug a FireWire cable into a virtual machine.

My VirtualBox install was on the Snow Leopard partition of the iMac with Tiger installed on another partition. Using some VirtualBox utilities, we can create a fake hard drive on the virtual machine that links to the real Tiger partition.

1. Get the disk identifier

Open Disk Utility, select the partition on the left, click Info, then make a note of the “Disk Identifier” of the Tiger partition. In my case, this was disk0s2.

2. Eject this partition.

3. Create the raw link file

Now we need to run some commands from the VirtualBox Manual.

Open up Terminal (from the Applications/Utilities folder), and run the below command. Don’t forget to substitute your own path for the .vmdk file and the disk identifier you got from step 1.

VBoxManage internalcommands createrawvmdk -filename [path/to/file.vmdk] -rawdisk /dev/[your disk identifier]

4. Fix permissions

I then had some problems with permissions. To change the permissions on your .vmdk file so VirtualBox can access it, use:

sudo chmod 777 [path/to/file.vmdk]

I also needed to change the permissions of the volume so when the link is followed, VirtualBox can actually read the partition it links to. For this, use:

sudo chmod 777 /dev/[your disk identifier]

5. Add the drive

Now go to the settings for your VirtualBox virtual machine. Under “Storage”, click the small button to add a device. Choose “Add Hard Disk”, then “Choose existing disk” and locate the raw raw link disk we made earlier.

If all goes well, it should now show up correctly in the VirtualBox list of disks.

6. Migrate!

Boot into the virtual OS and you should see the partition in your guest OS’s Disk Utility.

You can now migrate from the old partition.

In conclusion…

As a result of this process, we are no longer tied to old hardware. This means we’ll forever have access to the old environment no matter what hardware and software setup we have!

Virtualising an old Mac was originally published by Jordan Crawford at Jordan Crawford on February 10, 2015.

Jordan Crawford

Migrating from Evernote to Obsidian

Why Obsidian?

Figure out how you’ll use it

Attachments

Encrypted notes

Note browsing

Structure

My approach

Dates

Migration

Yarle template

Sync

Backup

Git backup setup

Conclusion

Reading a SVCD disc from 2005

What are SVCD’s anyway?

Initial playback attempts

Understanding how SVCDs work

Exporting content

Raw Binary Export

Exporting the MPEG content

Playing the SVCD in it’s original form

Lessons learned

My rules for using Time Machine on Synology

What is Time Machine?

Alternatives to Time Machine

Setting up Time Machine

1. Use a separate volume

2. Don’t enable snapshots or recycling bin

3. Don’t worry about adding another layer of encryption

Playing co-op Dungeon Siege in 2023

Tweaking the game settings

Getting connected

Introducing Tailscale

Gaming through Tailscale

Retaining our progress

Setting up a virtual machine as a game server

Choosing a hypervisor

Setting up the VM

Escaping Google Workspace

1. Why leave Google Workspace?

2. Finding a new email provider

Gmail + Cloudflare

iCloud+ custom domain

Migadu

Fastmail

3. Taking a backup of your Google Workspace account

4. Setting up my new personal Google account

Approach 1 - Sign up with a different email address

Approach 2 - Move Google Workspace to another primary domain

5. Migrating email and contacts to the new email provider

6. Migrating Google Calendar

7. Switching out any ‘Login with Google’ integrations

8. Cloning my Google Chrome profile

9. Saving my YouTube channel

10. Migrating Google Drive

11. Moving Google Home devices

12. Delete Google Workspace

Moving Google Drive files from Google Workspace to a personal account

Assumptions

Step 1 - Take (more) backups!

Introduction to rclone

Making a backup with rclone

Step 2 - Understanding Google Drive and it’s limitations

Google Drive files/folders have an owner

Google Drive files can become ‘unorganized’

You can’t actually delete other peoples files/folders

Step 3 - Transferring file ownership

Approaches to transfer file ownership

Share window transfer

Copy files via rclone

Move through a Shared Drive

Making the transfer

Bucket 1 - Folders completely owned by me

Bucket 2 - Folders shared with me but I don’t own any files within

Bucket 3 - Folders which are a combination of owned by me and owned by another user

Step 4 - Verification

Step 5 - Clean-up

Configuring `cloud`

Configuring `home`