Andi Duro

How to Use Unreplied 2.0 CLI

2020-05-26T07:00:00+00:00

Unreplied just released with command line support! Here's how to use it.

First, identify where Unreplied is installed. If you got it from the Mac App Store, it should be in /Applications/Unreplied.app.

Next, verify that your version of Unreplied supports the CLI. To do that, run:

andi@Andis-MacBook-Pro % /Applications/Unreplied.app/Contents/MacOS/Unreplied --cli --help

USAGE: unreplied [--cli][--csv] [--no-contacts][--ignore-non-contacts] [--ignore-short-codes][--cutoff <ts>]

OPTIONS:
--cli                   Launch Unreplied without a GUI.
--csv                   Print unreplied messages in CSV format.
--no-contacts           Don't attempt to resolve message handles into contact names.
--ignore-non-contacts   Don't include messages from people who are not in your Contacts.
--ignore-short-codes    Don't include messages from short codes.
--cutoff <ts>           Ignore all messages received before this UNIX
                        timestamp, in milliseconds. (default: 0)
-h, --help              Show help information.

Note that you have to call the Unreplied executable itself from inside the .app - not the .app itself (which is actually just a folder).

Now that you've gotten the help command working, you're ready to get started! Simply call /Applications/Unreplied.app/Contents/MacOS/Unreplied --cli without the --help flag to get a list of every iMessage and SMS you haven't responded to.

If you don't want a list of every message, just recent ones, use --cutoff to specify a UNIX timestamp that will make Unreplied ignore all messages sent before it.
By default, Unreplied will use the Contacts API to attempt to resolve phone numbers and email addresses into contact names. If you don't want this, use --no--contacts.
If you want to pipe Unreplied's output into another application, like awk or sed, use --csv to have Unreplied output in CSV, which should be easier to parse.

Behind The Scenes

I learned a bit trying to shove CLI functionality into a Swift app. There isn't much documentation on this other than a few StackOverflow posts, so I'll just document my findings here.

First, when you create a Swift app through Xcode, you start off with an AppDelegate.swift file. At the top of the class declaration, you might see @NSApplicationMain. Apple says:

Apply this attribute to a class to indicate that it is the application delegate. Using this attribute is equivalent to calling the NSApplicationMain function and passing this class’s name as the name of the delegate class.

If you do not use this attribute, supply a main.swift file with a main function that calls the NSApplicationMain function. For example, if your app uses a custom subclass of NSApplication as its principal class, call the NSApplicationMain function instead of using this attribute.

..So that's exactly what I did. I took out @NSApplicationMain, created a main.swift file with the following line:

    _ = NSApplicationMain(CommandLine.argc, CommandLine.unsafeArgv)

It runs, so now we have a way to intercept command line arguments as well as retain GUI functionality. But in CLI mode, we don't want to spawn the GUI; we just want to print to stdout. So we need a way to figure out when the app was called from the command line. My easy fix for this was to just require the --cli flag:

    if CommandLine.arguments.count > 1 && CommandLine.arguments[1] == "--cli" {
        Unreplied.main()
    }
    else {
        _ = NSApplicationMain(CommandLine.argc, CommandLine.unsafeArgv)
    }

What's Unreplied.main()? It's a ParseableCommand from the Swift Argument Parser, which, as its introduction post says, "makes it straightforward — even enjoyable! — to parse command-line arguments in Swift."

I agree! It was very easy to add CLI functionality to Unreplied with the Swift Argument Parser. Here's all the code I needed to add all of Unreplied's flags:

    struct Unreplied: ParsableCommand {

        @Flag(help: "Launch Unreplied without a GUI." )
        var cli: Bool
        @Flag(help: "Print unreplied messages in CSV format.")
        var csv: Bool
        @Flag(help: "Don't attempt to resolve message handles into contact names.")
        var noContacts: Bool
        @Flag(help: "Don't include messages from people who are not in your Contacts.")
        var ignoreNonContacts: Bool
        @Flag(help: "Don't include messages from short codes.")
        var ignoreShortCodes: Bool

        @Option(default: 0, help: ArgumentHelp("Ignore all messages received before this UNIX timestamp, in milliseconds.", valueName: "ts"))
        var cutoff: Int

        func run() throws {
            // do stuff with the arguments...
        }
    }

That's it! If you come up with some clever use for Unreplied on the command line, please let me know on Twitter @Nexuist or my Live Chat!

Announcing Unreplied 2.0

2020-05-26T07:00:00+00:00

In the summer of 2017, I launched Unreplied 1.0 on the Mac App Store, making it the first Mac App I ever completed and delivered. It retailed for USD 4.99, and sold a few thousand copies. Not terrible, but definitely could have been better - unfortunately, at about the same time, I entered college! So I immediately became busy with trying to figure out how to, well, survive college, and couldn't focus as much attention on Unreplied's marketing as I would have liked.

Now, almost three years later, I'm back again! I decided to use quarantine to rewrite Unreplied, add a few new features - like CLI support - and overall get back into the Mac game.

CLI support isn't the only new feature: Unreplied 2.0 also supports reminder notifications, Dark Mode, a new date format, and more.

So I'm back, and it's ready! You can check out Unreplied on its website: https://unreplied.app

If you already bought Unreplied through the Mac App Store, no worries - you'll get the new update automatically, no extra purchase required.

Fun fact: I randomly got the idea for Unreplied one day in January 2017 and pitched it to my friend Wilson (who now runs Zebra - an open source package manager for iOS devices). He was skeptical it could be done; as a result, Unreplied's fate was sealed, and it would come out a few months later.

Saturday, January 14, 2017.

Announcing This Website (Again)

2020-05-17T07:00:00+00:00

New year, new site, right? Although, it is May. A few things happened between January, when I wanted to do this, and now...

Anyways.

This is just a quick post mainly to highlight the new theme I'm using. It's called Colorie, and I really quite like it. The old site's CSS was written by hand, and while I think it still looks neat, there were definitely areas that could use improvement. I like Colorie because it's minimal and each page looks like...an actual page, made of paper. Perfect for long and short posts alike.

Here's the old site for reference:

And the site before that:

And the old blog post about this: Announcing This Website

In that post I talked about dark mode and why I wanted the redesign to be dark. I've since reconsidered, mainly due to Colorie's color pallete. It's definitely nicer on the eyes, and I think it just looks fun. So, this redesign is light. Maybe dark mode returns next year? ;P

Finding That One Song You Forgot Through Spotify Streaming History

2020-04-27T05:00:00+00:00

One night I was listening to Spotify Radio while doing homework and, through sheer luck and algorithmic grace, I was handed a set of tracks I absolutely loved. I saved almost every one.

The next day, I was playing through the list, only to realize I had forgotten one! To make matters worse, the tune was stuck in my head, and so while I knew exactly what it sounded like, I had no idea what the title or artist was. The only other relevant information I had was a rough idea of what time I was listening to it (the early morning of February 9th, around 2-3 AM).

I went back through my Spotify history, but I couldn't find it. Sadly, Spotify only stores the last 50 tracks in the History tab.

At this point I thought this song would be lost forever. However...

Could GDPR save me?

Spotify Download Your Data

To comply with GPDR, Spotify lets you download all the data they keep on you, which happens to include a full year of streaming history (streaming means any song, podcast, or video played through Spotify - not just radio history). The song I want should be in here somewhere!

Sidenote: I still find it incredibly strange that you can watch full, 1080p quality videos on Spotify. Not just music videos but also like...video game let's plays and mini documentaries. Who uses this?

I applied and waited for my link email. It only took a few days. Here's the files it came with:

FamilyPlan.json
Follow.json
Identity.json
Payments.json
Playlist1.json
SearchQueries.json
Userdata.json
YourLibrary.json
StreamingHistory[0-5].json

For info on each, check out Understanding My Data from Spotify. There's a lot of useful data here that isn't just about streaming history; you can also see every song you've saved, every playlist you've made and all the tracks they hold, who you follow, and your search query history.

For today, though, I'm only using streaming history. There's 6 of these files, and it turns out they go in chronological order: StreamingHistory0's first track was played in February 2019, while StreamingHistory5 begins in January 2020. So, we finally have a file to work with.

Unfortunately, it's 31,352 lines long!

Filtering with jq

Each piece of play history is represented in JSON:

{
    "endTime" : "2020-01-09 15:15",
    "artistName" : "Noisestorm",
    "trackName" : "Crab Rave",
    "msPlayed" : 161280
}

If we consider that each track takes up 6 lines of the file, then 31352/6 ~= 6,270 tracks over a period from January 9 (first entry) to February 12 (last entry). 6,270 tracks over a period of 43 days is roughly 146 tracks a day or 6 songs an hour, which is pretty accurate (I use Spotify a lot).

6,270 tracks is still a lot, though, and we can quickly cut this down using our good friend jq. jq lets you easily splice and dice through JSON files without needing to write custom scripts. I never used it before, and I knew this would be the perfect opportunity to try, so I read the manual.

> cat StreamingHistory5.json | jq 'map(select(.endTime | test("2020-02-09")))'

A few things are going on at once here (and it took me a lot more than one try to get this to work properly). First, some definitions:

For any filter x, map(x) will run that filter for each element of the input array, and return the outputs in a new array.
The function select(foo) produces its input unchanged if foo returns true for that input, and produces no output otherwise.
test(val), test(regex; flags) Like match, but does not return match objects, only true or false for whether or not the regex matches the input.

Effectively what we are doing is filtering out any track object that doesn't match our criteria (endTime contains "2020-02-09"). This gives us a new array of tracks. We can count how many using length:

> cat StreamingHistory5.json | jq 'map(select(.endTime | test("2020-02-09"))) | length'
> 297

297! A 95% decrease, which is great, but still way too many songs to deal with. I did mention that I heard this song during the night, and so we can try to see if we can filter by hours:

> cat StreamingHistory5.json | jq 'map(select(.endTime | test("2020-02-09 02"))) | length'
> 59

Now we're looking for end times that contain "2020-02-09 02", which limits us only to songs that were listened to at 2 AM.

I can listen to 59 songs, but right now I only have an artist name and a track name. It would take a long time to type all of that into Spotify, find a track, play it, and determine if it's the song I want or not.

It would be better if I could just click on the song to play it.

Markdown with jq

I'm not sure if anyone has actually done this before, but I figured the next step would be to convert my JSON object into a Markdown text file. I could have a list of songs, and each song could be wrapped in a hyperlink that opens Spotify.

Unfortunately, this JSON file doesn't give us any track ID or URL - only the artist and track name. We're going to have to use Spotify search to find the song before we can play it back.

From previous endeavors I knew that Spotify had a URI handler for spotify:search:, so you could, for example, click on spotify://search:Another One Bites the Dust to bring up the Spotify window with search results.

Using what we know, we can massage the original objects into the format that we want (still using only jq):

> cat StreamingHistory5.json | jq 'map(select((.endTime | test("2020-02-09 02:59")))) | map({time: .endTime, song: (.trackName + " - " + .artistName), link: (@uri "spotify://search:\\(.trackName + " " + .artistName)")})'

> [
    {
        "time": "2020-02-09 02:59",
        "song": "Jerry Was A Race Car Driver - Primus",
        "link": "spotify://search:Jerry%20Was%20A%20Race%20Car%20Driver%20Primus"
    }
]

The first map acts as a filter, getting rid of songs that don't match the criteria, while the second map acts as a...map, transforming keys from the old object to the new one.

(I've narrowed the time criteria to 2:59 just for this demo).

One more map later:

map("- \\(.time) [ \\(.song) ]( \\(.link) \)")

And we get:

[
    "- 2020-02-09 02:59 [Jerry Was A Race Car Driver - Primus](spotify://search:Jerry%20Was%20A%20Race%20Car%20Driver%20Primus)",
    "- 2020-02-09 02:59 [Blitzkrieg Bop - 2016 Remaster - Ramones](spotify://search:Blitzkrieg%20Bop%20-%202016%20Remaster%20Ramones)",
    "- 2020-02-09 02:59 [Giant - Nebula](spotify://search:Giant%20Nebula)",
    "- 2020-02-09 02:59 [Lies (feat. Luciana) - KSHMR](spotify://search:Lies%20(feat.%20Luciana)%20KSHMR)"
]

Note that jq's output still contains opening and closing brackets for the array, and puts each line in quotes. If we want to pipe this to a file, we need to get rid of those by using jq's -r argument (raw output) and .[] (which tells jq to separate objects with newlines):

> jq -r 'map(select((.endTime | test("2020-02-09 02:59")))) | map({time: .endTime, song: (.trackName + " - " + .artistName), link: (@uri "spotify://search:\(.trackName + " " + .artistName)")}) | map("- \\(.time) [ \\(.song) ]( \\ (.link) \)") | .[]'

- 2020-02-09 02:59 [Jerry Was A Race Car Driver - Primus](spotify://search:Jerry%20Was%20A%20Race%20Car%20Driver%20Primus)
- 2020-02-09 02:59 [Blitzkrieg Bop - 2016 Remaster - Ramones](spotify://search:Blitzkrieg%20Bop%20-%202016%20Remaster%20Ramones)
- 2020-02-09 02:59 [Giant - Nebula](spotify://search:Giant%20Nebula)
- 2020-02-09 02:59 [Lies (feat. Luciana) - KSHMR](<spotify://search:Lies%20(feat.%20Luciana)%20KSHMR)

Finally, let's give it a header, by simply adding an array containing ["# Songs", " "] to the front of the results:

> jq -r 'map(select((.endTime | test("2020-02-09 02:59")))) | map({time: .endTime, song: (.trackName + " - " + .artistName), link: (@uri "spotify://search:\\(.trackName + " " + .artistName)")}) | ["# Songs", " "] + map("- \\(.time) [ \\(.song) ]( \\ (.link) \)") | .[]'

# Songs

- 2020-02-09 02:59 [Jerry Was A Race Car Driver - Primus](spotify://search:Jerry%20Was%20A%20Race%20Car%20Driver%20Primus)
- 2020-02-09 02:59 [Blitzkrieg Bop - 2016 Remaster - Ramones](spotify://search:Blitzkrieg%20Bop%20-%202016%20Remaster%20Ramones)
- 2020-02-09 02:59 [Giant - Nebula](spotify://search:Giant%20Nebula)
- 2020-02-09 02:59 [Lies (feat. Luciana) - KSHMR](<spotify://search:Lies%20(feat.%20Luciana)%20KSHMR)

And now we pipe the end command to a file:

> jq -r 'map(select((.endTime | test("2020-02-09 02:59")))) | map({time: .endTime, song: (.trackName + " - " + .artistName), link: (@uri "spotify://search:\\(.trackName + " " + .artistName)")}) | ["# Songs", " "] + map("- \\(.time) [ \\(.song) ]( \\ (.link) \)") | .[]' > test.md

...presto!

(I found the song, by the way. It was Shift by ANG).

The Ballad of TurboChair

2019-10-23T05:00:00+00:00

On December 20th, 2018, I saw this GIF and I immediately knew that I had to build my own motorized chair.

On October 20th, 2019, we finally presented the TurboChair at HackUMass VII and won Best Hardware Hack. The following article is an attempt to explain everything that happened in between that 10 month span.

The Hackathon Team

Andi - Me!
Amrit - WPI CS '21 - One of my best friends from high school, we did FRC robotics together. Great at putting together hardware and building software.
Matt - WPI Aeronautical Engineering '21 - Another one of my high school best friends, also did robotics. Can build and wire anything!
Norlinda - UConn BME '22 - One of the first friends I made at UConn, who kept up team moral and put in major contributions to the design of the final product.

December, 2018

I found out about HackWPI, which is how this whole chair-ade started. I asked Amrit, Matt, and Dan (previous hackathon team members) if they'd like to do something together, and we started brainstorming ideas.

Originally we wanted to do a self driving Barbie jeep (the kind for kids), but we decided to save that project for next time once I saw the GIF which made me want to build the chair. After a few hours of discussion, everyone was on board and we started writing up a parts list.

January 8 - Sofa Hunt

Arguably the most important part was, well, the chair. We started checking Craigslist for any cheap offers that would be small enough to transport via car yet sturdy enough to allow us to bolt on motors and other electronics. Furthermore, we wanted an ergonomic seat for comfortable rides.

January 12 - Chair Secured

We found two listings that matched our requirements, but unfortunately both were pretty far and the cheapest one was \$25. However, the scales tipped once one of the owners sent back:

    You do not need to pay cash. At this point, if you would like to come and get it, it is yours at no cost.

I would like to say thank you to Rik from Connecticut for giving us the chair for free. Without him, TurboChair would not exist.

Matt came over with his dad's pickup truck and we both took a trip to Rik's house to collect the chair. Rik was visiting his granddaughter in Massachusetts at the time and he helpfully left the chair right outside the door to make it easier for us to load it into the truck.

After sending my thanks to him, I received the following reply:

    You’re very welcome. We had a lovely visit with our granddaughter. I hope you enjoy the chair and get as much good use out of it as we did.

    Rik

If only you knew, Rik!

Once we claimed the chair, we took it to Matt's house and got to work on the base assembly. To make a good day even better, it reclined! Matt happened to have some motors laying around from a mobility scooter he and a friend had found at the town dump - these would end up being part of the final product as well.

The motors appear to be the "M4-7UM" model from the PIHSIANG company, and they went for around \$300 the last time we checked eBay. It certainly looked like we got a deal.

Eventually I went home while Matt and Clay (another one of our high school friends) worked on the assembly.

January 13 - Max Power

Up till this point we had been using two power drill batteries in series to power the motors. While this was fine for testing, we'd need something stronger for the actual chair.

Matt found a battery:

This thing is absolutely overkill. It holds more juice than a typical car battery and outputs nearly twice the voltage. Regardless, it was exactly the type of over the top absurdity TurboChair demanded.

The battery ended up being the most expensive part of the chair, totaling well over \$200 and requiring its own special charger. Thankfully Matt happened to have it lying around from past projects so we didn't have to wait for it to shipping. Also, if we weren't shooting for absurdity, we definitely could have found cheaper options to fit the bill - but how fun would that have been?

By the end of this day, Matt and Clay finished the drive assembly and had it securely attached to the bottom of the chair. They added shopping cart wheels to the front (also from the dump) so the chair now officially qualified as a motor vehicle. Everything was starting come together.

January 14 - Move In Day

Matt and Amrit drove the chair to Worcester and stored it in the basement of Amrit's frat house. Things were looking good for Dan and I to come up on Friday for HackWPI.

At this point I also ordered:

A 5V voltage regulator
A Raspberry Pi touchscreen display
Turn signal lamps, like the kind you find on tractor trailers

January 15 - Unexpected Turbulence

Unfortunately, this is where things went downhill. Right before the hackathon, a really nasty winter storm moved in that kept us from going up to WPI. Amrit and Matt ended up not competing, and the chair stayed in the basement.

With the school year starting for all of us again, the chair became an afterthought.

Five Months Later...

May 9 - Shipment Delivered

Remember how I ordered some stuff four months ago? It finally came in on this day. I had totally forgotten about it while trying to figure out how to deal with the storm.

At least we felt better about missing HackWPI now, because even if we had gone these parts would definitely have not been there.

Another Five Months Later...

September 10 - Revival

About this time HackUMass started sending out emails again asking people to sign up to compete. I figured, why not give the chair a second try? It had been over half a year, but we already bought the parts, and we might as well use them to finish the project. I didn't care about winning - the completed chair was the prize.

So I asked Amrit and Matt to join me once again, and I wanted to bring Lin along so she could help out and see how big hardware projects like these came to fruition. Up until this point she only had experience with Arduino kits. This was going to be her second hackathon, and we needed to go big!

October 18 - Point A to B to C

Today was the big day: HackUMass begins.

This day was a logistics nightmare. Because the chair was still in Amrit's frat house basement and it couldn't fit in his car, we needed to go from UConn to WPI to UMass in about 5 hours. Once Lin got out of class, Matt got dropped off at my place and we began the journey to Matt's apartment in Worcester. We grabbed some tools and equipment (including the battery) from there and drove over to Amrit's frat house, where we acquired the chair.

At this point it was about 7:30PM, and we only had an hour to get to HackUMass before check-in ended. Amrit and Matt drove in Amrit's car while Lin and I drove with the chair to Amherst.

By the time opening ceremonies were over, we were all way too tired from driving to get much hacking done. We decided to get some rest for a few hours before working on the chair.

October 19 - Hacking Day

All of the ground work for the chair happened on this day. This is also where we encountered all of our major obstacles.

Motor Reversing

The first goal was to integrate MOSFETs with the motor controllers so that they could go backwards. Up until this point both motors could only go forward, making it hard to turn and impossible to reverse. Unfortunately it turns out one of the MOSFETs we brought was totally abnormal and didn't respond correctly to the right commands.

Amrit and Matt lost at least a few hours to debugging this stupid MOSFET. One of the most annoying parts of hardware development is that there are a hundred different places where things can go wrong - if it's not your wiring, it could be your code, and if not any of that, it could be the hardware itself or any one of the components connected to that hardware. There's not really any way to know. Software is much easier in this regard - at least you can use error messages to figure out what's going on.

Eventually they decided to retire the MOSFETs in favor of two simple on/off relays courtesy of the HackUMass hardware table. The relays were a joy to work with and soon TurboChair was capable of individually reversing motors.

GoPro Woes

While this was going on, I put together the Raspberry Pi that would make up our "infotainment system." A few days ago I had ordered a Raspberry Pi 4 and a Pi camera which would give us a backup view like most modern cars have. Unfortunately, I had misjudged how everything would get wired together. I thought that the touchscreen I got from May (still in its box untouched) would use HDMI, allowing me to put the Pi on the back of the chair and thus making the short ribbon cable for the Pi camera not too big of a deal.

In reality the Pi was meant to be attached directly onto the back of the touchscreen display. While this wasn't hard to assemble, it did mean that the Pi camera ribbon cable became a major issue - it simply wasn't long enough to reach the back of the chair.

In my great wisdom, I had brought my GoPro with me, which solved everything, right? Just duct tape the GoPro wherever I want and then live stream its viewfinder to the Pi over WiFi (which is built in to the Pi 4). Problem solved, right?

Wrong. GoPro apparently discontinued .m3u8 livestreaming from its API for the newer models (I have a Hero 7 Silver), meaning that the only way to do it now is through some wild amalgamation of ffmpeg commands. I'm sure there's a way, but I'm simply not a video streaming expert so I was way out of my league trying to solve that problem. At the same time, hours were ticking by and we still had nothing to show.

Around 8PM I gave up and took Lin to Walmart with me to buy a USB camera. It was $30 (I was hoping it'd be like $5...) but it got the job done.

The USB camera worked flawlessly and gave us a consistent 30+ FPS stream. With both goals complete, the next big step was to get the chair moving.

Interior Redesign

While Matt and Amrit worked on MOSFETs and I wrestled with the GoPro, Lin was hard at work ripping up the side of the chair to give us the "engine hood" - a giant rectangle of fabric that covered all of the electronics including the battery. When we needed to work on chair electronics, we could simply lift the hood up. When driving, the hood was held in place by strategically applied Velcro strips.

Lin also worked with me to come up with preliminary UI designs for our infotainment system, which would run on the Raspberry Pi.

First Drive

With the motor controlling apparatus installed under the hood and an Arduino thrown in to interface with the relays, we were ready to drive. Amrit wrote some simple control code that let us use a joystick module to command the motors.

The joystick ended up being completely awful to work with, but it was enough to give us the moral boost to keep working on the chair throughout the night.

The first static run of TurboChair occured at 10:14PM, with the full hallway run at 11:12PM.

It wasn't very long until bystanders saw what we were up to and wanted a shot at driving!

Eventually though, it was time to get back to work.

October 20 - After Hours

We had a driving chair. But we wanted more.

The two big goals for this sprint were to get the Nyko Kama nunchuck I brought to interface with the chair, and to also mount the Raspberry Pi w/ touchscreen display onto the side of the chair. As a bonus goal, we wanted to add turn signals and brake lights using this purpose-built LED strip we got from our previous Walmart trip:

Nyko Kama Wii Nunchuck

The nunchuck was the most important part as the tiny joystick we were using before was doing a really poor job at being useful. The nunchuck would give us a (hopefully) more precise 2-axis joystick as well as 2 buttons and force readings from its built-in accelerometer. While we didn't end up using accelerometer readings, it would of course be helpful nonetheless in case we ever wanted to in the future.

I had actually gotten this model to work four years ago in 2015. I open sourced the code I used then but afterwards I had no idea what I would use it for (originally I bought it for an electric skateboard project that fizzled out). I figured it shouldn't be too hard to port that code to the Raspberry Pi. However, the breakout adapter used I2C, and I'm not an I2C expert. To complicate matters, the Raspberry Pi had various libraries available for I2C that didn't provide a 1:1 match with Arduino's Wire library. In the end, I decided to hook up the nunchuck to the Arduino instead, and just connect the Arduino with the Pi together to transfer data over (more on this later). This ended up being a better choice as it made the Pi non-essential to chair operation - if my app crashed or the Pi restarted, it wouldn't stop the nunchuck from controlling the chair.

The nunchuck control code ended up getting finished around 4AM. At the same time I had been working on the front end web design for the infotainment system. I was able to use the MediaDevices web API to get video access to the USB camera, and wrote some basic CSS to place a sidebar next to the camera feed.

After finishing the UI design, I converted the whole page into a Vue web app. The idea was to set up a render loop that would read data from the Arduino, parse it into values that I could plug into Vue, and from there Vue would render the page.

I ended up writing a WebSocket bridge server in Python that read data values over serial USB (connected to the Arduino) and sent them through a connected client. In this way I could read any values from the Arduino from the web page - as long as the bridge server was running.

An example data packet from the Arduino looked like this:

DATA:1|1|128|130|0.55559|0.83244

Each field, in order, represents:

Armed: Whether the Z button on the nunchuck was being held. This would show an "ARMED" bubble on the screen and prevented the chair from moving unless the user was holding down that button.
Going Forward: The infotainment system had this concept of "gears" (D/N/R) even though the chair itself didn't really have a transmission. To support the illusion, we calculated whether the chair was going forward based off of if either motor was in reverse. If both motors were moving forward, the gear was "D", if they were both going backwards it was "R", if neither of them were moving the gear would show "N".
Joystick X: Exactly what it sounds like, the nunchuck's joystick X value from 0 to 255 (in the center, it was 128).
Joystick Y: Same as the last value but for vertical direction.
Left motor output: The voltage going to the left motor scaled from 0 to 1.
Right motor output: Same as before but for the right motor.

In all, the infotainment system consisted of:

The Raspberry Pi connected to the USB webcam and the Arduino via USB
The bridge server running in the background
The Vue web app running in Chrome full screen

Hackathon Pro Tip

A lot of people seem to struggle with getting code onto the Raspberry Pis. Generally you'll have a better dev environment (like VSCode) set up on your laptop, so trying to code directly on the Pi is inefficient. Usually it wouldn't matter, but in a hackathon seconds count.

What I ended up doing is setting up the Git repo first, and then git clone-ing it onto the Pi. This way I could push changes from my laptop and git pull them onto the Pi's file system.

What you definitely want to avoid is editing code in multiple places (i.e. on your laptop and on the Pi). Unless you set up version control, this is a great way to get confused and use the wrong versions of things and then wonder why things that used to work start breaking...and why things that were broken started working.

Although you generally shouldn't have to worry about other teams stealing your code (in our case, they'd have to steal a chair too...), if you are that concerned you can just make a private repo on GitHub and use that between your laptop(s) and Pi(s). Fun for the whole electronic family!

Pi Mount

Also around 4AM, Matt and Amrit started building the mount for the Pi. We realized that running the Pi, Arduino, and LEDs on the same circuit was drawing too much power, so we switched the Pi's power supply to a handheld USB battery bank. I was worried it wouldn't last nearly long enough to survive demo day, but it did!

At this point it was 6AM, and I was starting to crash. Matt, Amrit, and Lin had already gone to bed and woken up. Since the LED strip needed 24V to power the lights, it would have been too much work to set up 3 new relays to support interfacing with it from the Pi. No worries - we had another LED strip around, which Matt ended up installing under the chair!

After a few trial runs, we were ready to call it. TurboChair was officially finished at 8:08AM, October 20th 2019. Almost exactly 10 months to the hour. When we went to submit the project, it asked us what we wanted to name it. Up until this point we hadn't put any thought into it - so in our sleep deprived state we came up with TurboChair and called it a day.

October 20 - Demo Day

The Big Day!

Hacking officially ended at 10:30AM but I had submitted our project a few hours ago before I went to sleep. By the time I woke up we had already been assigned a table and were busy packing things up to get the table ready. Lin helpfully drafted a presentation in Google Slides to catch people's attention in case the giant motorized chair didn't.

Our chair was super popular, with other hackathon participants (and judges!) all taking rides around the lobby. Shockingly, at the end of it all, our battery had only gone down to 58% throughout the ~3 days we had been driving this chair around!

Here's a video showing off the infotainment system:

And here are some more pictures under the hood.

From left to right:

The "emergency cutoff" switch, which killed power from the battery in case anything went wrong (thankfully, it didn't!) It might look far away but it's actually pretty easy to grab on to in the driver's seat
The power box for the undercarriage LED strip, which Matt carefully spliced open to allow it to be powered by our circuit instead of a USB port
The two relays that allow the motors to go backwards and forwards
From top to bottom:
- The Arduino Uno which interfaced with the motor controllers and talked to the Pi!
- Our 5VDC regulator which powered the Uno and the relays
- The two huge (but cheap) motor controllers running our scooter motors
The Nyko Kama receiver and breakout board which allowed the Arduino to read commands from the nunchuck over Bluetooth

Here's a closeup of the electronics panel.

And finally, here's the chair in typical operation:

Turbo Mode

A few hours into demoing, Amrit decided to add a tiny little addition to the control loop: pressing the B button (above Z) would enable "Turbo Mode." See, up until this point, the chair had been operating at half power, because we were afraid of crashing it into things. With people dispersing to view other projects, we had a clear view at an empty hallway to try going Turbo.

It might be hard to tell how fast the chair is going from the video alone - try turning up your volume :)

Aftermath

Demo time came and went, and now was the closing ceremony (around 4PM). We were all anxious to see if we had won. We had been told beforehand that we were finalists for the "Best Hardware Hack" prize category, but there had been other impressive hardware projects...

We won!

Not only did we each win a free drone, but each drone came with a little slip letting us know that we were randomly selected to win another drone model from the same manufacturer!

Clearly it was all worth it. But the final challenge loomed ahead of us: how do we get everything home?

Well....

TurboChair doesn't need anyone in the driver's seat to move, so obviously...

We put all of our stuff onto the chair and drove it outside. I don't think anyone else could say they had their project drive their stuff to their cars :)

Fin.

So that was TurboChair. It may have taken nearly a year, but it was oh-so-worth it.

I'd like to thank Amrit, Matt, and Lin for completing this fever dream of a project. What a ride it was - metaphorically, and now physically too.

Our HackUMass project listing.

Our open source GitHub repo.

Till next time.

I Used Serverless to Build Something Real: Here's What I Learned

2019-05-26T05:00:00+00:00

In January of this year I thought it would be an interesting exercise to try to build an anonymous live chat webapp for my site using AWS serverless technologies.

It worked, up until the point where real people starting using it. Then everything broke.

Let me back up a bit, and explain what exactly this thing was.

If you want to jump straight to the code, here's the Conclusion.

Serverless Crash Course

In the early 2000s, in order to host a web app you'd typically need to put it on a server running special software (like Apache, PHP, and MySQL). Some clever people quickly realized that there was money to be made in hosting servers for other people, so the concept of the VPS (Virtual Private Server) was born.

With a VPS, you'd pay a monthly fee to a VPS provider (like DigitalOcean or AWS EC2 today) and in turn they'd give you access to one of their virtual servers. It was virtual because it wasn't a physical machine in a data center - instead it ran on a real, powerful computer with dozens or hundreds of other virtual servers. As long as you paid the monthly rent, you could have as many VPS's as you want with whatever specs you wanted. Plus, you'd never have to deal with a tangled wires, electric companies, or ISPs to make sure your servers stayed up.

At some point down the line someone realized that, hey, like 90% of these VPS's are just web servers, so why don't we just offer a web server service instead? This way, instead of individually managing a fleet of virtual servers, you could just focus on your website/webapp. You could use something like cPanel to manage the files, databases, and domains connected to the server. You could even install pre-built server software packages like WordPress with a few clicks. You'd still have to pay a monthly fee, but it would save you from the headache of dealing with low level sysadmin details like configuring database software or controlling firewalls. If you've ever paid a website to run a Minecraft server for you, this was basically the same deal.

Soon enough, there were Infrastracture-as-a-Service and Platform-as-a-Service startups popping up everywhere. Need a database but don't know how to manage it? Need to send email or text messages? User authentication? Storage? Analytics? All of these services offered to replace the software you'd need to build yourself with something that already existed and that was managed by professionals.

None other than Amazon exemplified the idea of "let someone else deal with it" with its Web Services division. AWS offered services for practically everything - at the time of this blog post there are 130+ services. It's the Build-a-Bear of web software. This is what the "cloud" is. As long as you've got the money, someone else will be willing to deal with it for dirt cheap thanks to economies of scale.

It should be noted that Microsoft and Google are also heavy competitors in the cloud space at this point in time, but AWS was the one to kick it off in 2006 with S3. I'll keep saying AWS because that's what I'm most familiar with, but keep in mind that its competitors have very similar services to those I'll mention by name.

In 2014, Amazon released AWS Lambda which was meant to be the "glue" in the AWS Build-a-Bear. See, a database or API by itself isn't useful unless it's used by something else. Lambda was that missing piece that would go on to link all of these services together.

With Lambda, you could make Lambda functions which would get called in response to events that originated on other AWS services. For example, when you talk to a custom skill on Amazon Alexa, Amazon Lex (the AWS service that powers Alexa) calls that custom skill's Lambda function with an event object containing information related to that event (like a transcription of what you said, what time you said it, and which Alexa you said it from). From there your Lambda function could use the regular AWS APIs and any other libraries/frameworks you uploaded to process the event.

Most AWS services charge against usage, and Lambda was no exception. As long as you weren't dealing with millions of requests a month, you could operate a Lambda function for pennies! Contrast this to the \$5-\$45 / month you'd need to pay for a single VPS.

Again, at some point someone realized it was possible to run a full-blown web app on AWS Lambda taking advantage of other AWS services, without ever having to pay for a server. Hence, serverless.

Spoiler alert: there's still a server involved. You just don't have to pay for it as much anymore. That monthly flat fee would turn into a scalable usage fee, so if your service didn't get used much it wouldn't cost more than a few dollars a month to operate.

Live Chat Crash Course

The live chat used the following AWS services:

API Gateway, which let me proxy requests from the Internet to my Lambda function and back to the requester
- In particular I took advantage of the new WebSockets feature, which is what allowed me to make this chat real-time. API Gateway endpoints can also be REST APIs.
DynamoDB, which was the database I chose to use to save mesage history between users. It's NoSQL, which I didn't really need for this type of project, but it was also dead simple to use with Lambda, so I chose mainly it for that purpose.
Lambda, which let me write a few dozen lines of Node.js to process each incoming request, insert it into DynamoDB, and attempt to send every message to its recipient through API Gateway if it could.

Setting up API Gateway and DynamoDB were easy enough through AWS's console. But that was to be expected; the Lambda was where the magic would happen.

And so, slowly through bruteforce and sheer will, I built out the Lambda function over a few days. There were quite a lot of pitfalls that I'd like to talk about, but before I do that I want to describe the live chat "protocol" I ended up building so you can get a sense of what the function actually did.

Every Lambda instance needs a handler, which is the function AWS Lambda executes whenever a request comes in. Here's the abbreviated handler for my chat:

let handler = async event => {
  if (!ws) {
    ws = new AWS.ApiGatewayManagementApi(...);
  }
  let body = null;
  try {
    body = JSON.parse(event.body);
    if (!body.action || !body.uuid) throw new Error("missing required keys");
    let args = {
      event,
      body,
      uuid: body.uuid,
      connection: event.requestContext.connectionId,
      ip: event.requestContext.identity.sourceIp,
      requestID: event.requestContext.requestId
    };
    if (body.uuid == "andi") {
      if (!(body.auth == PASSWORD)) throw new Error("incorrect auth");
      return module.exports.adminHandler(args);
    } else {
      return module.exports.userHandler(args);
    }
  } catch (err) {
    return JSONError();
  }
};

So, in this protocol, every request has to be in the form of valid JSON and contain a uuid and action key. Depending on what the uuid is and if the auth key matches the authentication password, the request is either sent to the userHandler or the adminHandler (which are both ultimately just other functions).

When you use the live chat, your request is sent to the userHandler for processing. When I use it, mine goes through adminHandler. There are very heavy similarities between the two so I'll only cover the user part.

Here's an abbreviated version of userHandler:

let userHandler = async ({ event, body, uuid, connection, ip, requestID }) => {
  switch (body.action) {
    case "hello":
      // ...
      return JSONReply("lastConnected", lastConnected);
    case "register":
      // ...
      return JSONReply("welcome");
    case "ping":
      // ...
      return JSONReply("pong");
    case "list":
      return JSONReply("history", ...);
    case "send":
      // ...
      return sent ? JSONReply("sent") : JSONReply("sendError");
  }
};

adminHandler is not much different, it just has different cases to support different actions. Speaking of which, let's talk about what they do.

DynamoDB Item Structure

Here's a quote:

Show me your code and conceal your data structures, and I shall continue to be mystified. Show me your data structures, and I won’t usually need your code; it’ll be obvious. - Eric Raymond

Which itself is an updated version of:

Show me your flowcharts and conceal your tables, and I shall continue to be mystified. Show me your tables, and I won’t usually need your flowcharts; they’ll be obvious. - Fred Brooks

I have no idea if either of these are true. I just needed something to introduce this section.

For your reference, here's what a fake account looks like in my DynamoDB table:

{
  "uuid": "b445629b-4061-419a-b3a1-caf486ab71a1",
  "timestamp": 0,
  "nickname": "baba",
  "email": "booey@email.com",
  "ip": "127.0.0.1",
  "connection": "testConnection",
  "lastConnected": 0,
  "lastRequestsServed": {
    "wrapperName": "Set",
    "values": [
      "null"
    ],
    "type": "String"
  },
  "updates": 1
}

Note that for this table, the HASH key is uuid and the RANGE is timestamp. Having a timestamp of 0 means that this is an account type of item, whereas having a timestamp above 0 makes it a message type (in that case, the timestamp would correspond to what time that message was sent).

I expect that last paragraph to only make sense to you if you've ever used DynamoDB. If not, don't worry. It always sounds like gibberish until it doesn't.

And here's what a message looks like:

{
  "msg": "testing message",
  "type": "to",
  "uuid": "b445629b-4061-419a-b3a1-caf486ab71a1",
  "timestamp": 1559200025231
}

Finally, here's the special admin account:

{
  "uuid": "andi",
  "timestamp": 0
  "ip": "127.0.0.2",
  "lastConnected": 1559200023104,
  "unread": {
    "wrapperName": "Set",
    "values": [
      "e1b3b0b4-1cd2-4a77-a74a-6df7e0bd820c",
      "null"
    ],
    "type": "String"
  },
  "connection": "testConnection2",
  "conversations": {
    "e1b3b0b4-1cd2-4a77-a74a-6df7e0bd820c": "baba-1",
    "919e2811-e502-466c-ac47-966a62a4deeb": "baba-2",
    "c781c0c3-ac19-4550-96fe-c8a00268a006": "baba-3"
  }
}

Two unique keys here, conversations and unread. unread is a String Set that contains the UUIDs of any accounts I have unread messages from. Whenever a user sends me a message, their UUID gets added to that set. conversations offers a way to "match the name to the face", where the face is the UUID. This way I know the nickname of every account in the unread list without having to query each account individually. This saves time and prevents unnecessary queries to DynamoDB (\$\$\$).

I don't expect most of these items or keys to make sense to you right now, but hopefully they inform you of what the different actions are doing to the table.

userHandler

hello - returns the Unix timestamp of the last time I (the admin) connected to the service
register - creates an account for a new visitor whenever they click the "Start Chat!" button. The account consists of their nickname, email, and a randomly generated UUID. The UUID serves as their password to keep others from accessing the conversation and gets saved in their browser on the client side. Obviously not the most secure method, but this isn't a situation where security is paramount. Additionally, it adds a new entry to the conversations key in the admin account that pair the UUID to the username.
ping - this is actually a v2 development in response to the Connections Die When They Are Killed Pitfall (read on to hear about that). All this does is updates the user's account with the latest connection ID. It gets called periodically every few seconds by the live chat client.
list - returns a list of every message sent and received by the user's UUID. This is called every time the chat page loads.
send - this one is the most complicated but also the most important. I'll include the whole thing:

  case "send":
    if (!body.msg) return JSONError();
    let unique = await utils.isUniqueRequest(uuid, requestID);
    if (!unique) return JSONReply("duplicate");
    await utils.addMessageToConversation({
        from: uuid,
        to: "andi",
        msg: body.msg
    });
    await utils.markUUIDUnread(body.uuid, true);
    let sent = await utils.sendResponse({
        from: uuid,
        to: "andi",
        msg: body.msg,
        ws
    });
    return sent ? JSONReply("sent") : JSONReply("sendError");

And here are the steps it takes, in English:

Throw out the request if there's no msg to send.
Check if the request is unique to solve the Idempotency Pitfall.
Insert the message as an item in the DynamoDB table.
Mark the UUID unread - add it to the admin account's unread list so that the next time I log in I'll see that I have an unread message from this UUID.
Try to send the message through the connection of the recipient with utils.sendResponse. In this case, this method looks up my account, picks out my connection ID, and attempts to send the new message through my WebSocket connection using the API Gateway Management API.

If all works out right, the new message gets added to the database as an item, updates my admin account so I know about it, and sends a reply to my live connection.

Boy, did it take a while to get it to all work out right.

Take Two

The first version of the live chat (I'll call it the legacy version from now on) was built using test-driven development. That is, I wrote code in the AWS inline code editor, saved it, and tried sending random events to it to see if it would work. I hit the Test Events Suck With JSON Pitfall here which was incredibly annoying to figure out, but at least I only had to fix it once.

As you can imagine, this move-fast-and-break-things attitude is probably why things broke.

To be fair, I wouldn't have been able to succeed quite as well as I did the second time around if I didn't have the legacy attempt to build on. I learned a ton from the first time, and that experience allowed me to deliver a more professional solution that avoids all the pitfalls I discovered since then.

I decided to do things properly the second time around. I built unit tests & integration tests, for the first time on a serious project that would be put in production.

It took so much longer! But as a reward, I'm much more confident in the codebase now. And for any new feature I need to support, I only need to add a few new tests to verify everything. In fact, as I solved the last few pitfalls I ran into, the test suite proved itself useful time and time again as every once in a while a minor change I would make would break everything. Being able to catch this before deploying the code to AWS was such a godsend.

Unit Testing

I had been trying to get into unit testing for a long time, but it only started clicking when I started writing tests for this existing codebase. I had previously read a bunch of tutorials and checked out a few demos, and they all sucked for one simple reason:

function add(a, b) {
  return a + b;
}

test("it adds five together correctly", () => {
  let sum = add(5, 5);
  expect(sum).toEqual(10);
});

...is NOT how real, production, web app code looks like. It's the same problem with "Hello World" examples for new programming languages. Like, what is print("Hello world!") supposed to tell me? How do I talk to a database with this? How do I send a HTTP request? How do I save something to disk or read a configuration file?

Actually, more importantly, how do I NOT do all these things? Because when I'm unit testing my code, I don't actually want it to hit a database or website. That's the whole point of unit testing, I gather. You just fake everything and check to make sure the code is calling all the fake stuff correctly, so that in the real world it'll do the same thing with all the real stuff. But now you have to figure out how to turn your "real" code into "fake" code without ever actually changing the code.

But I continued in my testing journey, armed with Jest and dozens of Medium tutorials. Soon enough, my questions were answered with the almighty jest.fn().

I'm not going to go too in depth with Jest unit testing, but the idea is something like this:

function registerAccount(username, password) {
    if (db.accountExists(username)) return false;
    db.query(...);
    return true;
};

test("it registers a new account correctly", () => {
    db.accountExists = jest.fn(() => false);
    expect(registerAccount("test", "password")).toBeTruthy();
});

test("it fails to register an existing account", () => {
    db.accountExists = jest.fn(() => true);
    expect(registerAccount("test", "password")).toBeFalsy();
});

This way you could fake external functions so they would return the values you want to test your code against. This is also great for handling errors your code could run up against that wouldn't usually occur in normal practice. You can just fake the error to make sure your code deals with it correctly.

Of course, it's important that those external functions do their job correctly too. For me, those external functions were the utils suite which comprised of the following functions:

JSONReply
JSONError
DynamoDocumentClient
dynamo
andiItem
createConversation
updateConversation
addMessageToConversation
getAllMessagesWith
markUUIDUnread
isUniqueRequest
sendResponse

JSONReply and JSONError were just helper methods for constructing responses, so I didn't bother testing them (I'm sure this will come back to bite me eventually).

The next two were critical to solving the Fake Database Pitfall.

The remaining functions all manipulated the database in some way, so I built integration tests for them.

Integration Tests

The difference between a unit test and an integration test is that a unit test tests a unit of code (like a function), while an integration test tests a system (compromised of units).

To build off the previous example, a unit test would check if the registerAccount() method returns the right value, while an integration test would run the method and then check the database to see if the new entry was created correctly. This way you're not just confident that "the method works", but you're also confident that "this part of the service works." Which is a fantastic thing to be confident about when delivering a service.

In my case, I was interested in testing all the utils functions to make sure they would create, read and update the correct items in DynamoDB when they were called. I would do this by having them send their queries to a local DynamoDB server, and then issuing read queries to the server from my tests to check to see if the resulting changes were the ones I intended.

After the initial setup, writing the tests wasn't all that bad. Here's one for markUUIDUnread:

it("adds the uuid to the unread list when unread is true", async () => {
  const testUUID = "bababooey-markUUIDUnread";
  let oldAndiItem = await utils.andiItem();
  await utils.markUUIDUnread(testUUID, true);
  let newAndiItem = await utils.andiItem();
  expect(newAndiItem).not.toEqual(oldAndiItem);
  expect(newAndiItem.unread.values).toEqual(expect.arrayContaining([testUUID]));
});

Reads more or less like English. Simple, concise, and reliable. Just like it should be.

Pitfalls

Okay, I've been mentioning these the whole article. Let's open this can of worms. Here's every pitfall I ran into building the live chat, in the order I ran into it.

Normal Databases Suck With Lambda Pitfall

This one I knew about already from previous experience, but figured it would be worth mentioning to further explain why I chose DynamoDB.

Okay, remember VPS's from the serverless crash course? Turns out Lambdas aren't that different from them. In fact, whenever you call a Lambda for the first time, Amazon essentially spawns a VPS (they call it a container) for that Lambda to run on. Once it finishes running, Amazon keeps that container around for the next few minutes in case another request comes in.

When a container exists for the Lambda, it's said to be "hot" because it can immediately resolve incoming requests. When there are no containers left (because there were no incoming requests for a long time), it's "cold."

You might be thinking that it takes a lot of time for Amazon to spin up a container when a function is cold. And indeed, it does take a lot more time to go from cold to hot, but that "startup cost" is only incurred when the function is cold. Depending on how hefty the function is the startup cost might range between a few milliseconds to a few seconds. To get around this and keep a function perpetually hot, some people set up timers (like with AWS CloudWatch) that send a sample request to the Lambda every 5 minutes to ensure it's hot and ready to serve any real requests. This way, even if nobody has used the Lambda for months, it'll still be ready the millisecond someone tries. Of course, you still have to pay the usage cost for all those fake requests, so it may always be a good idea.

So, to get to the point, say you don't want to use DynamoDB. Maybe you want to use AWS RDS (their other database service), which lets you use RDBMS systems you're used to like MySQL or PostgreSQL. In order to do this, you have to put your Lambda function in a VPC (Virtual Public Cloud) with your RDS server. There's no getting around this; you need your Lambda to be in a VPC to access RDS. And that wouldn't be a problem if it didn't mean three (three!) very bad things...

In order to join the VPC, your Lambda container needs to be assigned an IP address through an ENI (Elastic Network Interface). So now, not only does it need to boot up, it also needs to resolve an IP address and connect to a private network before it can go hot and serve your request. Considering that Lambda bills by every hundred milliseconds, this not only costs far more than regular functions but also takes up an incredible amount of time. For a real world example I've sometimes had Lambdas in VPCs take up to 15 seconds to respond to my requests (to play fair, sometimes it finished in 2-3 seconds too. Lambda is unpredictable). I'm sure this can get better, but I mean...how much better could it get? The fact is there's a lot of hefty setup that needs to take place to join a VPC.
Once you've joined the VPC, you lose access to the Internet. Jeff really screwed you on this one. If you're lucky, you only need access to your database and have no 3rd party APIs you need to talk to. But, if you really need to, you can restore Internet access to your VPC by buying an AWS NAT Gateway for....\$0.045 / hour. Yes, per hour. Napkin math: $0.045 * 24 hours * 30 days = $32.4 / month. You're actually better off just paying for a server if you're not getting a ton of traffic already.
Connection pooling isn't Lambda compatible. This one actually isn't that bad unless you intend for your service to get popular. The general premise of connection pooling is to set up only one connection to the database, and then reuse that connection for any queries that need executing. This way, instead of having to connect to the database everytime you want to run a query, you can use the pooled connection instead. Well, because of how Lambda scales horizontally, you instead get a thousand Lambda containers with a thousand individual connections to your singular RDS server. This would not be a problem if each database didn't also have a limit to the maximum amount of connections. I think the limit for Postgres by default is 100, and although you can change it, there's only so many connections one server can support before performance suffers. You can get around this, ironically, with another server - you can use something like pgBouncer to do the connection pooling for you, and have all your Lambdas hit your pgBouncer server instead. But at this point you've introduced a server into your serverless environment and everything stopped making sense a few hours ago anyways.

So yeah, normal databases suck with Lambda. That's why I prefer to stick with Dynamo for ultra low cost projects. For something that I know will require SQL in the future? Probably best to stick with a server, and then break that down into smaller Lambdas once you have an existing codebase to inspect.

I should mention that there's a new RDS addition called Aurora Serverless which was billed as the solution to this entire pitfall. It's Postgres and MySQL compatible, and it only costs...\$0.041 an hour for the cheapest MySQL option. Now, \$29.52 / month isn't an insane amount of money to pay for something like this, but again, think about whether just using a server would be a better idea at this point. Serverless requires a lot of special consideration and practical experience before it really clicks, and if you think it's not any better than paying for a server, go for the server instead. Don't feel bad! Amazon's EC2 and Elastic Beanstalk teams will happily take your money if serverless isn't your cup of tea.

Test Events Suck With JSON Pitfall

Few things are more annoying than having some weird error that keeps popping up regardless of what you do the code. In this case, I got to the point where the majority of my code was gone and I still kept getting a dumb JSON error!

One of the Lambda console's features is something called "Test Events" where you can build your own JSON event and have Lambda invoke your function with that as its event. For example, here's a test event to make sure the hello action works right:

{
  "requestContext": {
    "domainName": "58yojgvxyi.execute-api.us-east-1.amazonaws.com",
    "stage": "dev",
    "connectionId": "testConnection",
    "identity": {
      "sourceIp": "test.ip"
    }
  },
  "body": "{ "action": "hello", "uuid": "bababooey"}"
}

Maybe you've spotted the error already, but if not, here's the problem:

  "body": "{ "action": "hello", "uuid": "bababooey"}"

Turns out you need backslashes:

  "body": "{ \"action\": \"hello\", \"uuid\": \"bababooey\"}"

Or else Lambda will think you meant to set body to "{ \".

To be fair, this isn't a Lambda centric problem, but it was something I struggled to fix for hours. So I'm putting it in here in the hopes that it saves someone else.

Jest Has Weird Setup Files Pitfall

When you configure jest in your package.json, there's a neat key called setupFiles you can set to a group of files that get run once before any of your tests run. I set it to setup.js. Inside setup.js, you get access to the global object which all of your tests subsequently have access to. So I defined a global call method:

let call = async json =>
  await handler({
    requestContext: {
      domainName: "58yojgvxyi.execute-api.us-east-1.amazonaws.com",
      stage: "dev",
      connectionId: "testConnection",
      identity: {
        sourceIp: "127.0.0.1"
      },
      requestId: "testRequestID"
    },
    body: JSON.stringify(json)
  });

global.lambda = {
  call,
  ....
};

Which could then be called in any test file:

const { call } = lambda;

let result = await call({
  uuid: "bababooey",
  action: "hello",
});

This allowed me to fake whatever Lambda event I wanted without having to duplicate code across multiple tests.

Once I did this, I also wanted to manually set process.env.PASSWORD to a fake password at the beginning of the jest test session, so that I could provide it to call to verify that the base handler forwarded my actions to the admin handler when I gave the right password. Seemed easy, right?

Well, it just did not work at all. I tried a few variations and eventually went to search for it on Google, and turns out that only the globalSetup key can be used to set process.env across test files for some reason. So, I aded the globalSetup key pointing towards envSetup.js:

/*
  - setup.js exists because global.* variables can't be declared in this file but they can be in there
  - on the other hand, process.env can't be changed in there but it can in here
  - jest is confusing sometimes
*/
module.exports = () => {
  process.env = Object.assign(process.env, {
    PASSWORD: "password",
  });
};

Maybe there's a better way to do this, but this was the best I could do from my limited experience with Jest.

Fake Database Pitfall

In order for me to build integration tests, I needed an actual database to query against. I could have used a separate DynamoDB table as the test table in this situation, but I would rather run DynamoDB locally to avoid the risk of having any crazy situation where I end up getting charged \$900 because a function being tested inserted a hundred thousand items or something.

First I needed a local DynamoDB server. For that, I installed LocalStack.

Next, I needed a way to make the code use my DynamoDB server instead of AWS's. I was able to do this with the utils.DynamoDocumentClient object:

/* utils.js */
let dynamo = async (action, params) =>
  await module.exports.DynamoDocumentClient[action]({
    TableName: "DuroLiveChat",
    ...params
  }).promise();

module.exports = {
    ...,
    dynamo,
    DynamoDocumentClient
}

/* utils.test.js */

const DynamoParams = {
  apiVersion: "2012-10-08",
  region: "us-east-1",
  endpoint: "http://localhost:4569"
};

const DDB = new AWS.DynamoDB(DynamoParams);
const DDC = new AWS.DynamoDB.DocumentClient(DynamoParams);

beforeAll(async () => {
  utils.DynamoDocumentClient = DDC;
  // ...
});

The important part here is that utils.dynamo calls module.exports.DynamoDocumentClient instead of DynamoDocumentClient. That makes it susceptible to replacement in the integration testing file. If utils.dynamo called DynamoDocumentClient instead, Node would have used the one declared at the beginning of utils (pointing to the actual AWS DynamoDB) rather than the one with the custom endpoint.

Also, shoutout to dynamodb-admin for providing a perfectly simple yet cable web interface to interact with the DynamoDB tables running on LocalStack. The only change I had to make was to define an environment variable before running it so it used the right port to talk to Dynamo:

andi$ DYNAMO_ENDPOINT=http://localhost:4569 dynamodb-admin

Just Runs Tests Concurrently Pitfall

Also kind of knew about this one from past experience, but I think it's important to mention because it really should influence the way you design tests, especially integration tests.

DO NOT assume your test runs in order! That means if you insert an item into the database in test1 and try to retrieve it in test2, you're doing it wrong. Yes, even if the tests are in the same file.

Try to keep every integration test self-contained. If possible, it should create, read, update, and delete every item within itself.

The problem with concurrency is that you never know which test will get run first, so sometimes your test suite passes and sometimes it might fail even without changing a line of code. I got bit by this and had to rewrite a bunch of tests as a result.

Lambda Uses Old AWS SDK Versions Pitfall

Oh, this one sucks. What's the latest AWS SDK version?

2.437.0

So when I run require("AWS-SDK") inside my Lambda running Node.js 8.10, I should get version 2.437.0, right?

From AWS Lambda Runtimes:

Node.js Runtimes

Name	Identifier	Node.js Version	AWS SDK For JavaScript	Operating System
Node.js 10	nodejs10.x	10.15	2.437.0	Amazon Linux 2
Node.js 8.10	nodejs8.10	10.15	2.290.0	Amazon Linux

Sigh. This wouldn't be a problem if AWS API Gateway WebSockets wasn't a new feature when I first started building this, but it was, and so the ApiGatewayManagementApi module which is used to send messages to WebSocket connections didn't exist in the 2.290.0 version of the AWS SDK. In the legacy live chat, I included the files needed to make this work manually in a custom node_modules/aws-sdk folder. Knowing that the AWS SDK had updated since January, I figured the management API would come with it at this point. And well, it did, but not if you ran Node 8.10.

This turned out to be the easiest pitfall to solve. I just switched my runtime to Node.js 10.x. However, I was really lucky that my code was able to handle this major version upgrade without needing to change a single line.

Lambda Hates .Zip Files Pitfall

Lambda has three ways to upload code to it:

via the inline editor (which I used for the legacy live chat version)
via a .zip file
via a S3 bucket

I opted to use the zip file this time around because I could make one easily. It took a few tries to finally get it in a format Lambda accepted.

Here's what ended up working for me:

zip duro.zip base.js utils.js;
aws lambda update-function-code --function-name duro --zip-file fileb://duro.zip;

Try unzipping your zip file before uploading it to Lambda if it doesn't work the first time. Make sure the source code files are inside the zip file and not inside a folder inside the zip file. Lambda won't be able to find your handler through a folder.

That means you want:

duro.zip
 |- base.js
 |- utils.js

And not:

duro.zip
 |- duro
   |- base.js
   |- utils.js

At this point I was really excited because I had finally solved all the pitfalls in front of me, deployed a working version of the live chat to AWS Lambda, and had over 40 individual tests covering the entire code base.

But then I had to learn a new word.

Idempotency Pitfall

Q: What happens if my function fails while processing an event?

For Amazon S3 bucket notifications and custom events, AWS Lambda will attempt execution of your function three times in the event of an error condition in your code or if you exceed a service or resource limit. - AWS Lambda FAQ

Here's the tricky part about this answer. You know that it's possible for your function to be executed up to 3 times, but only if it fails while processing an event.

I was operating off the premise that if I had enough tests to cover every reasonable event I would throw at the function, it would never fail while processing an event. And yes, I know that no amount of test suites can truly cover every scenario. But, still. It turned out, my code wasn't failing. Amazon's was.

I caught this particular issue because every once in a while while testing the live chat frontends (user and admin), I would notice a message would get sent two or three times. Sometimes I'd refresh the page and the same message would've been saved to the database twice. This was really odd, and also random. I couldn't reliably trigger it. So instead I added logging to my function and spammed messages until I could get the error to happen again. And although it was random, it did happen often enough to be a serious problem. It could definitely happen to a user while they were using the application.

The problem turned out to be the ApiGatewayManagementApi. I would get the console logs from before its postToConnection method was called, but none after. The method would hang, and the Lambda function would get axed once it reached its timeout of 3 seconds. And once that happened, it would get called...again. And sometimes another time too. The problem is that before calling postToConnection, the function also added the message to the database. This would explain why there were both duplicate sends as well as duplicate inserts.

Now, to be fair, this may not be an actual issue with the ApiGatewayManagementApi. It might be designed to fail for whatever reason, and then end up working when Lambda calls it again post-retry. But the problem is my code wasn't idempotent, so I would end up with duplicates.

Idempotency in this context means that you should only change things the first time you process an event. If you are given the same exact event again, you shouldn't change anything.

Most of the utils methods were idempotent already, because they only read and returned items. They didn't add or update existing items. But for the ones that did, I needed a way to tell if the request was a duplicate so I could avoid executing them a second time.

This is where utiis.isUniqueRequest() comes in. Every Lambda event also comes with a requestId which is a unique identifier that identifies that particular event. If we save the requestId into the database the first time the event is processed, we can check the database the second time the event is processed to see if we already processed it, and ignore it if that's the case.

// Lambda sometimes retries the same event if it didn't work the first time, so this is here to prevent duplicate messages from being sent
let isUniqueRequest = async (uuid, requestID) => {
  let Key = {
    uuid,
    timestamp: 0,
  };
  let convo = await dynamo("get", {
    Key,
    ProjectionExpression: "lastRequestsServed, updates",
  });
  if (!convo.Item) return true; // unknown uuid
  let lastRequestsServed = convo.Item.lastRequestsServed.values;
  let updates = convo.Item.updates;
  if (lastRequestsServed.includes(requestID)) return false;
  lastRequestsServed.push(requestID); // add new request
  if (lastRequestsServed.length >= 6) lastRequestsServed.shift(); // drop the oldest saved id if there are too many
  let updateParams = {
    Key,
    UpdateExpression:
      "SET lastRequestsServed = :newSet, updates = updates + :one",
    ConditionExpression: "updates = :updates",
    ExpressionAttributeValues: {
      ":newSet": module.exports.DynamoDocumentClient.createSet(
        lastRequestsServed
      ),
      ":updates": updates,
      ":one": 1,
    },
  };
  try {
    await dynamo("update", updateParams);
  } catch (err) {
    if (err.code != "ConditionalCheckFailedException") throw err;
    return false; // the conversation was updated by the time we tried to update it, so just invalidate everything
  }
  return true;
};

As you can see we save the last 5 requestIds processed to a set in the account item, and return true/false depending on if our requestId is in that set.

Technically we can have an event get retried after 5 others have already been processed, but the likelihood of that happening is so low it's not worth accounting for. Especially because Lambda immediately retries the function once it fails the first time; there's very little time delay between initial execution and retry.

What's updates for? Well, turns out, you can have 2+ Lambda functions access the same Dynamo item at the same time. If the lastRequestsServed set doesn't include their requestId, and they both have the same requestId, both functions will conclude that the request is unique, even though it's actually not. To prevent against this, we increment the updates key and use a ConditionExpression to tell DynamoDB to reject this update if the current value of updates doesn't match the one we have. This way, only one Lambda will get to claim the request ID is unique (as it should). All others running concurrently will report that it is a duplicate. This technique is not unique - it's actually a duplicate of a common DynamoDB tactic called Optimistic Locking.

I'd also like to thank the following two articles for holding my hand through figuring this out:

Too Many Arguments Pitfall

This one is not Lambda related at all, just a JavaScript optimization.

I used to have methods that looked like this:

let createConversation = async (uuid, nickname, email, connection, ip) => { ... }

let sendReponse = async (from, to, msg, ws) => {...}

The problem is, when you actually call them, it's really obtuse to figure out which index is for which argument:

await utils.createConversation(
  "e1b3b0b4-1cd2-4a77-a74a-6df7e0bd820c",
  "bababooey",
  "baba@email.com",
  event.requestContext.connectionId,
  event.requestContext.ipAddress
);

Instead, accept an object instead. It's much simpler to parse and also easier to add arguments in the future when you want to extend what the function does.

let createConversation = async ({ uuid, nickname, email, connection, ip }) => { ... }

This is some nice ES6+ syntax sugar that deconstructs the object for you. The following is another way to do the same thing:

let createConversation = async (params) => {
  let uuid = params.uuid;
  let nickname = params.nickname;
  // ...
};

And you can call it like so:

await utils.createConversation({
  uuid: "e1b3b0b4-1cd2-4a77-a74a-6df7e0bd820c",
  nickname: "bababooey",
  email: "baba@email.com",
  // ...
});

Much easier to follow. Named parameters are always welcome.

Anyways, I wish I had followed this advice before I needed to rewrote most of my utils functions, which ended up breaking all my tests, which I also had to rewrite somewhat.

Connections Die When They Are Killed Pitfall

Last pitfall before the end! I'll keep it short.

Here's how ApiGatewayManagementApi.postToConnection() works:

await ws
  .postToConnection({
    ConnectionId: connectionString,
    Data: JSON.stringify(obj),
  })
  .promise();

The ConnectionId is the most important part here. It refers to which WebSocket to send this data to, and if that WebSocket doesn't exist or isn't connected anymore, postToConnection() throws an error.

The legacy version of this sending function handled two scenarios:

The user was sending a message to the admin, so it would get the ConnectionId from the admin account's connection key
The admin was sending a message to the user, in which case they provided a connectionTo key which would be passed to postToConnection()

As you can imagine, this brittle system didn't really work when the user switched WiFi networks or refreshed the page.

The new system instead always takes the ConnectionId from the user's account, and makes sure it is updated frequently with the ping action I talked about before. This way, as long as the live chat is open and running code locally, the system will continue being aware of what its connection ID is. Of course, the above code is wrapped in a try/catch in the event that sending still fails, but at least then we'll know it's because the user isn't connected anymore and not because we passed the wrong connectionId.

Conclusion

Congratulations on making it this far! I hope this article helped you figure out how you can use Serverless on a real, not-built-for-a-tutorial real-time service.

Here's the source code for everything: https://github.com/nexuist/duro-chat

Inside, you'll find a __tests__ folder and a legacy folder containing all my unit/integration tests and the legacy version of the live chat, respectively.

If you want to try out Jest on your own machine, make sure you install LocalStack and have it running beforehand or all the integration tests will fail.

If you find any better workarounds for the pitfalls, please don't hesitate to let me know using the live chat tab in the sidebar ;)

Fixing Netlify SSL Certificates

2019-03-18T09:00:00+00:00

I like Netlify a lot. It's what I use to host this site and a bunch of other domains.

One of the neat things they offer is SSL certificates through Let's Encrypt. All you have to do is click a button.

So a while back I did click that button. Unfortunately, things didn't go as planned. Both domains were marked as untrusted by Chrome and Safari. While you could still click proceed, that wouldn't be a great user experience by any stretch of the imagination.

After browsing Netlify's help docs, I wasn't able to find anything relating to my issue, so I wrote to their support email instead:

Hello,

I have Let's Encrypt set up for two domains: unreplied.app and duro.me. For both, it is saying that the certificates are invalid in Chrome and Safari.

Is there a way to regenerate my certificates, or am I doing something wrong with my DNS records?

Thanks,
Andi

I sent this email at 1:30AM and got a response promptly at 6AM (way to go Netlify!)

While I was expecting a reply pointing out what I did wrong, I actually got quite a fulfilling explanation which is why I thought I'd share it in this post. Here's the email in its entirety:

 Hello Andi,

I got your certificates fixed.

Generally, the reason we are unable to provision a complete SSL certificate for your custom domain is that the DNS cache timeout value (TTL) for a record has not had time to expire (from your old settings) before you tried to use it with Netlify. Our SSL provider (https://letsencrypt.org) is unable to create certificates for names that have old cached values still in effect.

Depending on how you configure your domain, we may only attempt to fetch the certificate once - when you update your settings with the new domain name. Usually, if it is your first hostname on a site, we'll try several times until we succeed.

If that process generates a partial certificate, that's what you're stuck with until you write in to support. I realize that this experience isn't the greatest - but letting you repeatedly press a "refresh certificate" button would be worse as you'd likely get locked out of getting a certificate for a week, due to our SSL provider's strict rate limits for re-requests.

The additional delay seems to have been enough to allow things to work right when I attempted to re-issue the certificate.

Please let me know if things are not working as expected now.

Basically, the process for procuring a Let's Encrypt certificate for my domains failed and left me with a partial certificate (I had no idea that was a possibility). This is why Chrome was throwing those mean errors.

My awesome Netlify rep (Dennis) was able to restart the certificate process on his side and, since enough time had passed for my old DNS record values to be thrown out of cache (that's the TTL he was talking about), it went through flawlessly and built the full certificates I was looking for.

After that, I had no more SSL issues.

The lesson learned here is to wait a while after pointing a domain's DNS records to Netlify, before setting up that domain within Netlify. In these particular instances, I changed the DNS records for the domains with my registrar (Namecheap) and then immediately set them up in the site portal. Hence, Let's Encrypt left me with partial certificates because each domains' records were in the process of changing.

A big thanks goes out to Netlify's fantastic support (especially Dennis) for remedying this situation quickly. I'm writing this post in the hopes that someone else who experiences this will now know what to do - just contact support.

Have a great day!

An Update on NowPlaying

2019-01-21T21:22:00+00:00

Hi everyone! I hope all of you enjoyed the holiday season and had fun during your break.

I wanted to write a small essay to describe the research I did since we last saw each other and the next steps I wanted to take with the project.

The Goal

We want to build a user-based service that allows any one user to share their music, live, with everyone else who wants to listen in.

We plan to do this by using the Spotify API and various other Web technologies to provide a cohesive, cross platform solution. We would eventually monetize this service and use it as an income generator for the club.

Variations of such an idea have already been suggested in the Spotify community forums:

With thousands of people upvoting these suggestions, it seems that the market wants something like NowPlaying.

The Name

We will call this service NowPlaying. Personally I still don't think it's a good fit for us, but while it's in development the name isn't that important. What is important is to have a name that adequately reflects the primary adjectives we want to associate with our service:

Live
Sharing
Music

If anyone has any better suggestions for a name, I'm open to hearing them.

The Spotify APIs

There's actually a bunch of them. I've compiled a list of the ones we would care about the most. Some are in beta, meaning they'll be changing around and it may not be a great idea to base a product on top of them until they've left beta stage.

Web API - Provides access to the majority of the data you can access on Spotify, such as playlists, artists, tracks, and account specific data (who you're following, your email, your library, etc.)
- The most relevant portion of this to us is the Player API (Beta), also referred to as the Spotify Connect Web API. If you're not aware, Spotify Connect is the technology that allows you to switch your playback source from your phone to your laptop/desktop/Alexa/etc. Using the web API, we can send commands to each of your Spotify devices, allowing us to remotely control what the Spotify app on every device is doing.
Web Playback API (Beta) - Lets us embed a Spotify player into our web pages. Spotify songs are encrypted through some unknown proprietary format (they're not just streamed as MP3s), and this API is an official way to play Spotify songs in web apps without requiring the native Spotify app.
- Unfortunately, this only works on desktops, as mobile devices are missing the cryptographical components that allow the Web Playback API to decrypt song streams. As far as I can tell, there are no plans to make this work with mobile devices. In addition, it will not work in Safari for Mac. See the Supported Browsers section for more information.
- "This SDK must not be used in commercial projects without Spotify's prior written approval"
iOS/Android Streaming SDK - No longer actively developed, but was the primary way to integrate Spotify into mobile apps. Offered the same functionality as the Web API and the Web Playback API combined.

...And that brings us to the latest addition to the lineup: App Remote SDK.

App Remote SDK - Released in September 2018, so relatively recent. Its release was accompanied by this blog post on the Spotify blog. Some important paragraphs:

Earlier this year, we released beta versions of the App Remote SDK for iOS and Android. Unlike our previous playback SDKs, the Android and iOS SDK now allow you to “remote control” playback in the Spotify app. As a result, you can control playback for all users (not just Premium) and provide offline playback.

Also as of this release, the iOS SDK is out of beta. This means that we consider it to be stable, and able to be used confidently in your integrations.

The Android SDK remains in beta, and we will provide updates on this blog.

Note that these [Android and iOS] streaming SDKs are not actively maintained and no additional work on these are planned. We therefore urge you to work with the new SDKs going forward.

This is a big deal. With the old mobile SDKs gone, App Remote is what we'll have to use if we ever want to deliver native apps.

Which API For Us?

We wanted NowPlaying to be cross platform - to work on both desktop and mobile devices. We wanted to do this using their Web Playback API. But there are some big problems:

It doesn't work on mobile at all.
It can't be used for commercial projects (hopefully, us) without talking to Spotify's legal team.

So, Web Playback is out. What about Web/Spotify Connect?

It allows us to remote control the native app on desktop and mobile. And indeed, we could implement all of NowPlaying using a web app and the Connect API. But.

The API is in beta, so we have no idea if it will stick around or change heavily in later Spotify versions.
The whole API is rate limited. This, in itself, invalidates the Connect API as a choice for us.

Why not rate limiting?

Just a quick primer for those who are unfamiliar. As you are probably aware, computing is a finite resource, and companies like Spotify have to build or rent massive data centers to process all the requests they get every second. To prevent their services from failing due to overuse, Spotify implements per-second request rate limiting. This means, as a user of the Spotify Web API, you can only send X requests per second before Spotify will block your request and refuse to execute it. When this happens, you're given a Retry-After header, which contains the amount of seconds you need to wait before Spotify will accept requests from your application again. During this duration, you cannot use the Spotify Web API at all.

I tried finding more concrete details on the limits, but it seems like they're prone to changing and undocumented. The guy in that post said to expect something like 10-20 requests / second, which at first glance seems like a lot, but when you do the math, it wouldn't work for us.

Let's assume the rate limit is set to 20 requests a second. That's 1,200 requests a minute, or 72,000 an hour, which is huge. But it's not enough for NowPlaying.

This is because NowPlaying implements live streaming. That is, for every little change you make in your Spotify app, every single one of your followers also need to have that change performed to their Spotify app. This requires at minimum one request per user.

We only get 20 requests a second. If any single person on our platform gains more than 20 followers at a time (or if more than 20 people stream at the same time that have at least one follower), we can no longer keep their followers in sync with the streamer. We would get rate limited every time they changed anything. Obviously, that's not acceptable.

The limit can apparently be lifted but I have never heard of anyone who has successfully done so, or if doing so would require entering a contract with Spotify or paying them royalties. I found this post where a Spotify employee tells the poster to contact Spotify legal, but no information on if that would cost anything.

We're not the only ones who have to deal with this problem. One of the most popular requests in Spotify's issue tracker is a way to receive real time player updates without having to poll the Player endpoint (each poll attempt would use up a request, and we'd have to poll it at a frequency that seems akin to real time). Discord seems to have a private, custom solution built for them, but I doubt that deal was done for free.

Regardless, even if we got realtime updates, it still wouldn't solve the problem of us having to update every follower's player in real time.

So, with Web Playback and Web/Spotify Connect out, that leaves us with the mobile SDKs. And with the old iOS and Android SDKs now being unsupported, that leaves us with...App Remote.

We're Using App Remote

And with that, abandoning our plans of building a web app. App Remote is a native-only solution, so we can only use it on iOS and Android.

I wish there was an easy way to make this work on Web, but until the Web Playback SDK comes through or the rate limiting situation is drastically changed (both of which I highly doubt), this is what we have to deal with.

So now we have to build mobile apps. This is obviously a huge change in the direction of the project, which is why I wanted to write up this letter and distribute it to you all before our first meeting.

Here's how I see it working.

WebSockets

This part is necessary just so you understand the next section.

Typically the way computers communicate on the Web is through a protocol called HTTP. It's based on a request/response architecture, meaning that you send a request, and the server sends back a response. You can think of this as talking to Siri. When you ask Siri a question, she'll give you an answer, but Siri would never pop up on your phone unprompted and ask you a question or let you know of a new event that happened. This architecture is great for requesting documents like the Web was originally meant for, but terrible for building interactive apps since there's no way to allow for bidirectional communcation.

WebSockets is a Web standard that allows real time communication between various devices. These devices can be Web browsers, mobile phones, laptops, desktops, etc. You can think of WebSockets as texting, for computers. It's how they can have conversations with each other.

App Remote & Us

NowPlaying would be its own app that uses the App Remote SDK to talk to the Spotify app on the user's phone. When NowPlaying is opened, it establishes a WebSocket connection to our NowPlaying service and also a separate connection to the Spotify app. From here, updating player states between multiple devices is easy: just send a message to the WebSocket connection each device maintains with our service, and have the NowPlaying app forward that message to the App Remote SDK.

Here's a visualization of what a typical NowPlaying session could look like between 10 devices, where one device is streaming and the others are following.

NowPlaying App (Device 1) -> NowPlaying Service: Hello. I represent Spotify user <whoever is logged into Spotify on that device>.

NowPlaying App (Device 1) -> App Remote SDK -> Spotify App: What song are we playing right now?

Spotify App -> App Remote SDK -> NowPlaying App (Device 1): We're playing "Thank You For The Music" by ABBA, currently at 0:38.

NowPlaying App (Device 1) -> NowPlaying Service: I want to stream now. I'm streaming "Thank You For The Music" by ABBA, currently at 0:38.

NowPlaying App (Devices 2...10) -> NowPlaying Service: I want to follow Device 1!

NowPlaying Service -> NowPlaying App (Devices 2..10): Okay! Device 1 is currently streaming "Thank You For The Music" by ABBA, currently at 0:38.

NowPlaying App (Devices 2..10) -> App Remote SDK -> Spotify App: Play "Thank You For The Music" by ABBA, starting from 0:38.

...And if the user on Device 1 changes something in the Spotify app, say they skip ahead:

Spotify App -> App Remote SDK -> NowPlaying App (Device 1): Heads up. The user changed the player state. We're now playing "Thank You For The Music" by ABBA at 1:12, not 0:38 anymore.

NowPlaying App (Device 1) -> NowPlaying Service: Hey. I'm changing my stream. I'm streaming "Thank You For The Music" by ABBA at 1:12 now.

NowPlaying Service -> NowPlaying App (Devices 2..10): Device 1 is changing their stream. They're going to play "Thank You For The Music" by ABBA at 1:12 now.

NowPlaying App (Devices 2..10) -> App Remote SDK -> Spotify App: Change of plans. Play "Thank You For The Music" by ABBA at 1:12 now.

Because we're not directly touching Spotify's services, there is no rate limit. This is infinitely scalable, and we don't need to pay Spotify anything to use the App Remote SDK or WebSockets between our own private apps.

Hosting The WebSocket Server

There are a variety of ways to do this but I am leaning towards AWS API Gateway which just recently came out with WS support. API Gateway has no servers involved, which reduces the complexity of running our service by a huge magnitude. We pay for connection minutes (number of minutes each client is connected) and total number of requests. You can look at the pricing tab for more information, but the basic gist is that we pay $1 per million requests and $0.25 per million connection minutes. I just used API Gateway to implement live chat on my own personal website, so I now have enough experience to figure out how to make it work for NowPlaying as well.

Building The Mobile App

Again, various ways we could do this, but I am leaning towards React Native. RN is a cross platform mobile development framework for iOS and Android that allows you to write apps in JavaScript. It is supported by Facebook and used in their FB Market and Messenger apps as well as the Facebook app itself.

It's important to know that the App Remote SDK itself only works fully on iOS. The Android counterpart is in beta, meaning it's being actively developed, but will take a bit to reach a stable stage.

There is one more barrier stopping us: the App Remote SDK is written in Objective-C, while RN uses JavaScript. To get these to play together, we have to use a native module bridge.

When I set out to build NowPlaying over break, I realized I would have to build that bridge myself. And while I broke some ground to try to get the two frameworks co-operating, I soon found react-native-spotify which appears to be a more complete, community driven Spotify API implementation. Right now it conforms to the old streaming SDK so it's not immediately useful to us but work is underway to get it up to date with the new App Remote SDK.

Once this project is complete within the next few weeks, we will have everything required to implement NowPlaying in a simple, straightforward, scalable, and long lasting manner.

Conclusion

NowPlaying is switching from a Web app to a mobile app. This not only allows us to leverage the App Remote SDK, avoid rate limiting, and provide a real, truly live experience, but it opens up the possibility of using other technologies in the future, such as push notifications and geolocation tracking.

We are going to use AWS API Gateway to offer a WebSocket service to connect our various clients together and build the business logic that makes NowPlaying possible.

Our clients will be React Native applications that use the Spotify App Remote SDK to talk to the Spotify app on each user's device. They'll connect to the WebSocket service to receive orders and allow us to remote control Spotify.

We are waiting on the react-native-spotify package to finish adding the App Remote SDK before starting development on the mobile app. Once that package implements App Remote, we can finally break ground and bring the idea to reality.

This is the plan as I see it. Of course I'd love to gather feedback from you all to decide how to proceed. Once again, I hope you had a great break and I look forward to discussing this in person at our next meeting.

Andi

Announcing Live Chat

2019-01-10T09:00:00+00:00

UPDATE: Live Chat has been rebuilt from the ground up! Check out the new article.

Live Chat is a duro.me feature that lets you talk to me pretty much instantly. Before you read the rest of this article, go ahead, try it out.

Why? Because sometimes I want to ask a question, but I don't want to go through the whole process of drafting a good-looking email, logging into Twitter, finding my old Facebook login, etc... it's too much of a hassle. And that hassle is a lost opportunity! As a developer and a person, talking to anyone - literally anyone - can help you see things in a different way. And whether it's a spelling misake in a blog post or a critical bug in a product I've shipped, I want to know about it as soon as I can.

What can you talk about? Well, anything, for starters. But I'm particularly interested in what you're working on and what you're struggling with - maybe I could help.

Development

This apparently ended up being really long. So, if you want, you can skip. No worries, I won't judge.

1/10/19

This is live. This is raw. And so is this post! I'm typing it as I develop the actual feature, so this should be interesting to look back on. I've never tried anything like this before.

4 AM - Obviously chat has been done before, a million times over, by pretty much every web dev at some point during their career. But this is different; namely, I have a budget of about five dollars, total, a total dev time limit of about a day, and you can only talk to one person: me. Let's get started.

4:45 AM - Starting with frontend. I've built the CSS/HTML for the header containing the online indicator, last online time, and email/Twitter links. I might end up using Vue or something to make the frontend dynamic.

5 AM - I've come to the realization that I'll also need an admin portal for this. Somewhat of an afterthought, but we go on!

5:30 AM - Redoing header a bit to make it look nicer on mobile. I've also got the chat body and footer in the right places.

6 AM - I've found Bubbly which makes is super easy to make chat bubbles. Thanks, leaverou!

6:13 AM - Taking a break for breakfast.

9:17 AM - Starting up again.

9:36 AM - I've figured out to and from message bubbles. Onto the footer.

10:23 AM - Refactored CSS so I could have different font sizes for things on mobile.

10:28 AM - Tested the site on my iPhone for the first time. Noticed a rendering bug with the sidebar, so dealing with that now.

10:32 AM - Thankfully was able to reproduce the issue on desktop Safari and found an easy fix. Safari was oddly respecting the 0 font size for sidebar elements when in mobile mode - the text is supposed to disappear. Well, it wasn't there, but its placement was still rendered, so the icons were pushed out of the center for seemingly no reason. I fixed this by wrapping the text in a span and just using display: none on it. Back to chat.

10:34 AM - The frontend styling is done, I think! Works well on mobile too. Next step: make it dynamic. I'm thinking of using React or Vue for this, and I'm leaning towards Vue because I can just drop it into the page without any custom tooling.

10:50 AM - Interesting predicament. Both Liquid (preprocessed by Jekyll) and Vue (processed by the user's browser) use {{ double brackets }} to specify expressions that should be rendered. If I put that in my HTML, Liquid will pick it up and fail to process it since it's meant for Vue and just leave the result empty. Thankfully, there's an easy fix: the delimiter property.

10:55 AM - I accidentally solved the problem in a better way. In an attempt to show you the {{ double brackets }} thing in the last update, I realized that Liquid ate that, too, and any attempts to escape the expression didn't work. Googling how to fix that so I could show you Liquid rendering expressions in a Liquid-rendered page led to this: Outputting Literal Curly Braces in Liquid Templates , and thus the {% raw %} tag (yes, I had to escape that too). Ultimately what this means is I can just put my whole page inside a raw tag and not have to worry about Liquid accidentally rendering it and breaking my Vue app.

11:20 AM - I wanted the ability to drop links in messages and have them get rendered as clickable links. I found a Vue library named vue-linkify, but was having trouble getting it working, until I found this guy's tiny solution.

11:24 AM - Finished the Vue app, minus the part where it talks to the backend. That's what I'm working on when I come back to this.

1/12/19

10:23 AM - Had to work on a separate project for a bit. Back to chat.

11:30 AM - Fighting against API Gateway to get a WebSocket API off the ground. I've managed to get it to execute a Lambda and return the result, so I should be able to implement the rest of what I need with this one function.

1/13/19

9 AM - I've made a DynamoDB table to hold conversations. I'm planning on using a UUID as the primary key, and an incrementing number for the sort key. The idea is that you can send a message containing a UUID and my Lambda function will append it to the bottom of the list. Then, when you need to load the conversation list at the beginning of the session, I can return the list of all messages with one Query call. I'm going to try playing with the AWS CLI to mock up the Dynamo actions I want, and then it should be pretty easy to translate those to Node parameters.

9:04 AM - After doing more research into how I could auto increment sort key, I learned that apparently Dynamo doesn't like auto incrementing numbers and that there are better solutions, like a UUID based off of timestamps. That's fine; after thinking it over for a bit I realized I can just use the timestamp itself as my sort key (since time is auto incrementing) and achieve the sa,e result. I don't even need to include a timestamp field now, since the sort key will function as that. I'll remake the table with UUID/timestamp keys instead (I wish there was a way to rename them without tearing down the whole table, but oh well, I didn't have anything in it anyways).

9:48 AM - Currently playing with API Gateway and Lambda to see if I can get connection IDs into Dynamo like the example. Because I want to fit everything into one function, I'll be doing all my work in the inline Lambda editor.

10:07 AM - Want a way to link a connectionID with a UUID in onConnect, but there's no way to specify a message before first connecting to the WebSocket.

11:22 AM - Bought some pizza dough and now back to this. Rather than dealing with connect/disconnect events, I've decided to just have everything get processed through message events. I can include the connectionID with every message stored, and then I'll always have the most recent connectionID available to use (for my admin backend). This means that connectionIDs in previous messages will become stale as soon as the connection is dead, resulting in a lot of useless values stored, but hey, storage is cheap.

11:35 AM - I figure people might want an option to get notified whenever I respond if they navigate away from the page. I looked into web push notifications, but iOS Safari doesn't support them, so that's a dead end. I'm signing my domain up for AWS SES (Simple Email Service) so I can send emails from my Lambda whenever I reply to someone who has their email configured and their WebSocket connection is closed. I guess I'll need to modify the frontend to support this too.

1:33PM - Now exploring AWS SES for the first time. Verified my domain and filed a support ticket to get my account out of sandbox mode so I can actually send emails to real people.

2:30 PM - Getting into deep specifics with Dynamo now, I've come to the conclusion that a frontend web GUI (think phpMyAdmin) to execute queries in would be a huge boon to development times. Oh well, maybe for another time.

2:53 PM - I've cracked the code and figured out how to store user emails and nicknames (what I'm calling metadata). This was a bit confusing at first because I wanted it all to go into one message with a timestamp of -1, but that message may not exist yet if this was the first time the user was changing their metadata. The query parameters I'm using now will allow that key to be automatically made and then have values inside it set if that's the case.

6:37 PM - Took some breaks in between, but I spent the better part of 3 hours painfully learning that single quotes are not valid JSON. This kept breaking JSON.parse(), leading to me trying to figure out WTF happened to Lambda that it suddenly couldn't parse incoming event bodies. Oh well, that's software dev.

6:40 PM - I've got message sending & email/nickname changing down. I'm going out to get sushi; when I come back I'll figure out retrieving message history and my admin portal (and also link the frontend to all of this somewhere between there). I really want to get this done today because it's blocking my progress on the rest of the site. Given that I've paved enough ground to figure out this much, the rest of it should be a breeze (famous last words).

1/14/19

11:40 AM - I've successfully gotten the first messages sent through the frontend. At some point yesterday I also realized I won't need a list-messages action because I can store them locally along with the UUID. If you clear local storage and lose the UUID, it makes sense to lose the conversation history as well.

12:45 PM - I found a bug (well, not really, but it's an aesthetic thing) where if you get too many messages they scroll off your screen. I fixed this already with overflow: scroll, but as it stands you have to manually scroll down when this happens. I need to automate that.

12:51 PM - From the client, you can now send and receive messages, change your nickname, and change your email. The client is effectively finished (barring one or two features), now I'm going to figure out how the admin portal works. The features remaining are the "last online" info bar at the top and testing if receiving messages works (I haven't yet because I have no admin backend to send messages from).

1/15/19

11:18 AM - Decided to build the admin portal under /andi. You can't interact with it, of course, but you can check out the source code. It's secured with a local, client side password.

2:08 PM - After coding for a fewi hours, I took some time to draw out what I wanted the backend to look like. I've also decided to scrap the andi specific page because there wasn't enough difference to justify two seaprate codebases, and just do everything on the one frontend file.

5:36 PM - I've finally gotten a reply to show up in the admin portal from a user session. A bunch of other stuff happened in between then; hastily trying to wrap up loose ends and get everything into a shippable state.

6:45 PM - Abandoning the timestamp: 0 metadata storage method. Opting to just send all of it with every message - storage is cheap, retrieval/processing is expensive.

7:47 PM - Finally locking down the backend. Only doing frontend work now, as I think I've got everything needed for both client side and admin side portals sans email sending functionality.

10:42 PM - Calling it a night. Admin portal needs work.

1/16/19

10:46 AM - Rewriting frontend and backend.

2:25 PM - Done.

A wild ride, right? Sometimes it ends up like this. What was supposed to take a few hours ended up taking 6 days. But hey, it got done in the end. And that is all that matters.

Conclusion

So yeah, I built my own chat portal using AWS. If there's interest I can also do a technical writeup on how everything works together to make Live Chat, since it was all built more or less from scratch. Let me know using the live chat! (see how handy this is now?)

Announcing This Website

2019-01-09T07:00:00+00:00

Hi there!

This article is being written on January 9th, 2019, but I officially started the new redesign for this site on November 22nd, at 2 o' clock in the morning, which is obviously the prime time for great ideas.

Nonetheless, if you're reading this post, it's here!

Here's the old site for reference:

Why wasn't it good enough?

The Tabs

There was no obvious way to throw new projects into it. Where would I put, well, anything? A tab at the top? But anything more than a couple tabs quickly becomes a nightmare to navigate. Here's AWS's old product menu for reference:

This alone kept me from using AWS for, like, two years. How can you possibly get started with that?

So the tab-per-project thing is out. I could have made a Projects tab, but then, isn't Unreplied technically a project? And, of course, it is, but I wanted it to have its own page: duro.me/unreplied looked better than duro.me/projects/unreplied.

The obvious solution to avoid this was to just throw Unreplied onto its own domain: https://unreplied.app. If you're reading this, I might have already done that. I opted not to do this in the beginning because domains require hosting and hosting costs money. I was taking advantage of GitHub pages to host duro.me for free, but that only worked for one subdomain. I couldn't have nexuist.github.io and unreplied.github.io, and I couldn't have https://unreplied.app redirect to nexuist.github.io/projects/unreplied seemlessly (i.e. having it say unreplied.app in the address bar). Today, I know about Netlify, and that's probably what I'll use to address this.

The Colors

...Aren't mine. They're actually the default for Bulma, the CSS framework I used to build the old site. Or, at least, they were, Bulma received a pretty significant overhaul last year that now makes my site look even more outdated.

I thought it looked fine at the time, and to me it still kind of does, but having used dark mode in VSCode and now system wide in macOS Mojave, it's just nicer on the eyes. So this redesign is dark.

The Concept

The old site wasn't really meant to be anything. It was just a humble page on the Internet about me. But it did a terrible job of conveying what cool things I'd done in the past, who I was outside of software dev, or any stories I wanted to share. You'd take a glance at it and figure out that I was a geek but not much more.

So I built this blog instead. It's powered by Jekyll which should make it really easy to add new content to the site, and boy, do I have content to add.

A few words about Jekyll. It's a static microsite generator written in Ruby, focused on blog platforms. What does that mean? Well, I have a _posts directory, and to make a new post I simply create a new Markdown file (this post is also in Markdown).

If you don't know about Markdown: check out the original inception post on Daring Fireball. It's stupid easy to understand, super powerful, and available on basically anything you can type into.

Jekyll takes over the rest and applies themes, preprocesses Liquid, generates HTML, compiles SASS, builds a prod directory, etc. Very easy and takes away most of the stress of administrating a blog since, well, it's really just a folder of files you're throwing onto a server somewhere. Not much that can go wrong! (famous last words)

But it's not meant to be just a blog. Unlike previous iterations of my Web presence, I wanted this to be interactive in a way that most personal sites aren't. I wanted someone who just found out about me to be able to spark up a conversation about anything they wanted, and get a reply as soon as I was able to respond, straight from the site. Announcing Live Chat.

I also wanted a way to stay in touch with readers even after they've left the site. So, I did what a lot of blog owners do, and started a email newsletter. The catch?

Well, I don't know if it counts as a catch, but you only get emailed whenever I make a post. No daily, weekly, monthly schedule, because I'm sure you're a busy person too, and you don't need me jumping into your inbox every day with whatever I could scrape up from Reddit. Only the important stuff! Learn more and subscribe.

Fin

I remember the good old days when I was just starting out with web dev and trying to make my first version of a portfolio showcase - only to realize that, oh hey, I have nothing to show yet. Now I have the opposite problem: too much to show, not enough time to show it! I hope this redesign is the first step in the right direction.