blagblogblag.com

Is Github’s business model still viable?

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Sun, 05 Apr 2026 00:00:00 +0000

People are dunking on Github. “Zero 9’s uptime”. And yeah it’s not great, as of writing this it’s 89.24% over the past 90 days according to “The Missing GitHub Status Page”.

But people are misunderstanding what’s going on. Blaming it on the Microsoft takeover or on Github letting go of key staff members. That may be part of it. But I’d say they’re sideshows to the main act.

Let me explain. Below are recent plots of growth in key numbers from Railway and Render.

Railway's growth from @JustJake (CEO at Railway)

Render's growth from @anuraggoel (CEO at Render)

See that almost vertical line starting around Jan 2026? What happened? Claude Code happened. Vibecoding happened. Code produced by matrix multiplications happened. It was already there, but the latest models are self-driving to an extent that enables a much wider range of people to produce code. Way more is being built, and being built way faster.

Now, having something on Railway or Render means you actually cared enough to spin up a server. Most vibecoding is not that. There’s no deployment for gstack (Garry Tan’s ultravibed thing) nor any server hosting necessary for autoresearch or the new caveman skill. But there is a Github repository for all of those, plus one for everyone and their mom’s vibecoded home-project⁰.

Here’s Pete Steinberger’s Github contribution graph (of OpenClaw fame):

Credit to @asishcodes on Twitter (and probably many others) for pointing this ridiculous stat out.

So, imagine the above growth plots, but make slope 10 times steeper and multiply the y-axis by 1,000x ¹. Now you have the plot for Github. This is growth at an unfathomable scale. Today, there are both more users pushing to Github and every user on the platform is pushing way more code, more frequently. All due to coding agents.

Github is experiencing IMMENSE growing pains. It’s no laughing matter. There’s nothing to dunk on there. This is an org that is already handling absolutely immense volume, and now that volume is probably doubling or tripling within a very short timeframe.

Okay, and what does all that have to do with their business model?

The business model v1

Historically, Github has basically been a tiered freemium model – and at the most basic level it still is. The freemium model thinking is – free usage is a gateway to paid tiers, and those basically sponsor the free tier. That made sense in the olden days – for one, back then you actually had to type in something – with your hands – to put something on Github. Which meant that the ratio of hobby-projects to professional-projects was modest. 10:1 maybe? There was only so much free time to manually type into files in a repository. And at work you wanted a private repo with CI, and that costs money. And so Github made money.

Fast forward to today, the amount of hobby projects is skyrocketing. It takes Claude five minutes to put up a Tetris clone, but it’s hexagonal and the blocks are coming in from all sides. Whereas building a business takes time. And you’re not gonna go on a paid plan for Hextris, because it’s not a business it’s just a hobby project that’s not making any money. And you might say “But it’s much easier to make money in the era of LLMs” — but it’s nothing compared to how much easier producing code has become. If anything, there’s probably going to be less need for something like Github Enterprise – likely the main revenue and profit driver for Github – because teams are getting smaller, there’s less on-premise need, and just generally fewer big-corporaty setups because everyone can just vibecode their SaaS. They don’t need a 100 person dev team to build Okta or Hubspot anymore².

So, the core Github business model has died. That business model is no longer viable.

The business model v2

The new business model foregoes all that. Github is now a data play. Ingest and store as much code as you can, and have it be training data for models. Use the data to build Copilot and sell it.

Data plays can be quite good business, just look at the Reddit + OpenAI deal. Not good for users, but good for business.

For Github in particular, there are lots of caveats to that of course – most of the code is open source – OpenAI and Anthropic have ingested it already and will continue to ingest it so there’s no way to monetize. And is it even valuable data if all the new code is vibecode anyways? You can’t teach a model to be better at coding by having ingest code it produced itself. Even if you could, you wouldn’t need to go through Github³.

I think that’s a fairly sound business strategy.

I’m quite tired of the trope “if you’re not paying, you are the product”. But it does apply here – and while I’m not very afraid of my data floating around out there (on Github or on the internet at large) nor afraid of public internet data being used for AI training – I am quite saddened to know that there’s now a disconnect between what I’m paying for and what actually drives value for the business, in this case Github. The ideal business model really is – you pay for the value the business creates. It’s perfect incentive alignment between customer and provider.

A toast to a now dead business model, and a toast to the sweating SREs trying to keep Github afloat.

Update (May 14th 2026)

About 20 days after writing this, Github actually posted plots of their actual growth on their blog. Nothing surprising. Well, perhaps one thing – the commits trend doesn’t look as bad as I expected. I’d guess that’s because with LLMs we tend to get these larger one-off commits, rather than the more gradual, human “commit-as-you-go” approach.

Footnotes

⁰ Some caveats would be: A) Render and Railway may be primarily be experiencing growth from deployments of OpenClaw, in which case their growth doesn’t translate to more repos and commits on Github. B) The actual number of new repositories on Github doesn’t seem to be exploding in the way I’d expect, but I still expect pushes and number of commits to be.

¹ How do I get to 10x slope and 1000x y-intercept? For slope, let’s look at repositories – I’m saying for every 1 repository that gets deployed to Railway or Render there are 9 that didn’t. For the y-intercept, we’re seeing Railway at 5,000 paying customers before the growth spurt. My back-of-the-envelope calculations puts Github at 5,000,000 paid seats.

² Yes, yes, you can’t really vibecode an entire auth suite or CRM. Or at least you probably shouldn’t. But I definitely believe it is true that you don’t need the same amount of people build e.g. a CRM anymore. Have you seen the amount of new CRMs popping up? And it wasn’t for lack of CRMs in the first place. So yes, SaaSpocalypse is at least partially right. Not all the way right, but partially.

³ Maaaaaaybe, there is value in vibecode on Github because that vibecode was chosen to be pushed to Github — so maybe a human tested it and found it to be working — or maybe they looked at it and found it to be reasonable code. Which could make it just slightly higher “alpha” than the raw code coming out of an LLM. In any case, the “alpha” would be very low, and it would probably be best to filter on certain repositories — e.g. selecting ones with lots of activity from many people — signaling an actually functioning project that produces value to those people.
And then of course Apple had to go and show that it is possible to have the snake eat its own tail while I was writing this blog post https://arxiv.org/abs/2604.01193.

Running a script with large input on Heroku (part 2)

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Tue, 21 Jun 2022 00:00:00 +0000

Note: If you need heroku run:detached <command> or want to run interactively using heroku run bash take a look at the previous blog post Running a script with large input on Heroku. For all other cases this method is easier.

If you’re anything like me, you occassionaly want to run a one-off job on Heroku with a bunch of code in a script that you’ve made locally on your machine.

But you don’t want to commit it to the repo and then to wait for CI (e.g. integration tests can take a long time). Especially if it’s a single-use data analysis script that you don’t want to clutter up the repository. Good news! You don’t have to – Heroku allows piping into the heroku run-command:

> echo 'abcdef' | heroku run -a test-app 'wc -c'
Running wc -c on ⬢ test-app... up, run.1543 (Standard-1X)
abcdef
7

You can use this to upload new code to the temporary dyno (the code will only be available in that particular dyno)

> cat my_script.py | heroku run -a test-app 'cat > my_script.py; python my_script.py'
Running cat > test.py; python test.py on ⬢ test-app... up, run.6181 (Standard-1X)
print('Hello from my_script.py')
Hello from my_script.py

That’s it! – just pipe the file into the heroku run-command, use cat to write it to a file on the dyno, and then run the file.

Sending multiple files

If your Heroku image has the tar-utility you can do:

# on macOS (gbase64 is in the `coreutils` Homebrew package)
tar -c input.txt sum.py | gbase64 | heroku run -a test-app -- 'base64 -d | tar -x; python sum.py'
# on Linux
tar -c input.txt sum.py | base64 | heroku run -a test-app -- 'base64 -d | tar -x; python sum.py'
# on FreeBSD the command is the same as macOS and gbase64 is in the `coreutils` port

The long story about sending multiple files

To get to the above, I initially did tar -c input.txt sum.py | heroku run -a test-app 'tar -x; python sum.py'. However, the first issue that I had was the error tar: Refusing to read archive contents from terminal (missing -f option?).

So I expanded a bit and created an intermediate file: tar -c input.txt sum.py | heroku run -a test-app 'cat bundle.tar; tar -xf bundle.tar; python sum.py'

However, that command never finishes. Also, all of the three commands show the actual content of the tar-file in your terminal:

 > cat bundle.tar | heroku run -a test-app 'tar -x; python sum.py'
 Running tar -x; python sum.py on ⬢ test-app... starting, run.4590 (Standard-1X)
 Running tar -x; python sum.py on ⬢ test-app... connecting, run.4590 (Standard-1X)
 Running tar -x; python sum.py on ⬢ test-app... up, run.4590 (Standard-1X)
 input.txt^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@000644 ^@000765 ^@000024 ^@00000000006 14254272731 013376^@ 0^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ustar^@00malthe^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@staff^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@000000 ^@000000 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@1 2 3
 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@sum.py^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@000644 ^@000765 ^@000024 ^@00000000102 14255151576 012655^@ 0^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ustar^@00malthe^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@staff^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@000000 ^@000000 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@print(sum([int(s) for s in open('input.txt').read().split(' ')]))
 ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@tar: Refusing to read archive contents from terminal (missing -f option?)
 tar: Error is not recoverable: exiting now
 python: can't open file '/app/sum.py': [Errno 2] No such file or directory

It seems that cat hangs in Heroku once “control-characters” like ^@ are sent through the pipe. But we can use base64 to encode and hide those forbidden characters from cat.

> tar -cf bundle.tar input.txt sum.py
> base64 bundle.tar | heroku run -a uvicorn-issue-344 'base64 -d > bundle.tar; tar -xf bundle.tar; python sum.py'
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA[ ... and lots more of these ... ]AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHN1bS5weQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAwMDA2NDQgADAwMDc2NSAAMDAwMDI0IAAwMDAwMDAwMDEwMiAxNDI1NTE1MTU3NiAwMTI2NTUAIDAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAdXN0YXIAMDBtYWx0aGUAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAHN0YWZmAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAMDAwMDAwIAAwMDAwMDAgAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAcHJpbnQoc3VtKFtpbnQocykgZm9yIHMgaW4gb3BlbignaW5wdXQudHh0JykucmVhZCgpLnNwbGl0KCcgJyldKSkKAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
base64: invalid input
tar: A lone zero block at 5
6

It worked!

Even though we got some error messages, it worked. For larger input data or scripts you may want to checksum (e.g. md5/shasum/sum) the files on both ends to ensure that everything was transferred and untarred correctly.

Or the oneliner (I’m still sad that Linux tar doesn’t allow input from a pipe):

> tar -c input.txt sum.py | base64 | heroku run -a uvicorn-issue-344 'base64 -d > bundle.tar; tar -xf bundle.tar; python sum.py'

Why do we get `base64: invalid input`? (BSD vs GNU `base64`)

Your Heroku image is almost certainly runnign Linux, meaning that it’ll be using the GNU-version of base64. That is also why we’re getting base64: invalid input. I’m on macOS and therefore using the BSD-version of base64, so we’re piping the output of BSD base64 into GNU base64.

If you install coreutils via Homebrew brew install coreutils, you’ll get the gbase64-command (GNU base64) and using that on your side you’ll have no errors:

> tar -cf bundle.tar input.txt sum.py
> base64 bundle.tar | heroku run -a uvicorn-issue-344 'base64 -d > bundle.tar; tar -xf bundle.tar; python sum.py'
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
                  [ ... and lots more of these lines ... ]
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
6

The difference between base64 and gbase64 is that base64 puts a newline at the end of the output, whereas gbase64 does not. However, gbase64 put newlines everywhere else – specifically after every 76th character – wrapping the output at 76 terminal columns.

And you can’t just remove the newline – this hangs:

base64 bundle.tar | tr -d '\n' | heroku run -a uvicorn-issue-344 'base64 -d > bundle.tar; tar -xf bundle.tar; python sum.py'

Probably due to some buffering based on newlines inside the Heroku CLI or further down the Heroku stack.

Caveats

In order to pipe you need a tty, so --no-tty and similarly run:detached can’t be used (e.g. for long-running scripts)
You can’t regain control of stdin once you pipe into a process, so you can’t do cat file.txt | heroku run 'cat > file.text; bash'

Notes

Usually tar-flags are passed without the - in front, e.g. tar xzvf archive.tar.gz however I felt it was more clear to add the dashes in this case.

Show the time a command was run in fish shell

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Fri, 17 Dec 2021 00:00:00 +0000

I’ve been seeing that other people can get their shell to show the time a command was run (Bash, zsh 1, zsh 2):

~ $ python mgmt_script.py                                                  10:45
                                       This thing ────────>─────────>────────┘

As an avid user and big fan of fish, I naturally have been wanting to do the same.

Initially, I researched a bit and basically gave up since it looked like fish didn’t have the necessary functionality to implement such a thing. Instead I ended up having fish_right_prompt print date '+%H:%M'. This works okay but it leaves a stale timestamp when your terminal has been sitting for a while and you then execute a command. The timestamp that’s there is really the time when the last command finished, rather than the timestamp of when you ran the command the timestamp is next to.

Sidebar: Really, we should have richer terminals/shells anyways – with output that’s collapsible and auto-saved to a temporary output history so that you can use it similarly to !!. And being able to hover a command to see the duration plus time of start and end of execution.

Circling back to the problem now, one and half years later I found function event_handler --on-event fish_preexec (and fish_postexec), which allows me to do the thing I want:

function reprint_prompt --on-event fish_preexec
    echo -e "\033[F\033["(math $COLUMNS - 4)"G"(date '+%H:%M')
end

The teensy bit of ANSI escape code-magic here is:

\033[F: Move cursor to start of previous line (current prompt)
\033[<$columns>G: Move cursor <$columns> characters forwards

In my own config.fish I call fish_right_prompt directly instead of date '+%H:%M' to keep the two in sync. If you do the same, you should match math $COLUMNS - 4 accordingly. If your right prompt has variable length you can do something like:

function reprint_prompt --on-event fish_preexec
    set -l _right_prompt (fish_right_prompt)
    set -l _len_right_prompt (string length $_right_prompt)
    echo -e "\033[F\033["(math $COLUMNS - $_len_right_prompt + 1)"G$_right_prompt"
end

If you have stuff like the git branch in your prompt and you want that to reflect the state it was in when the command was run, you can do that with this monstrosity:

# Update prompt when executing a command
#
# This makes it so that the "fish_right_prompt" states the time that the command
# was actually run, rather than when the prompt was printed.
# Also the current git branch is updated to what it was at the time when command
# was run, rather than the git branch from when the prompt was first printed.
function reprint_prompt --on-event fish_preexec
    # Go to previous line (current prompt)
    echo -n -e "\033[A" 
    # Clear line and print current time
    echo -n (string repeat --no-newline --count (math $COLUMNS - 6) ' ') (date '+%H:%M' | tr -d '\n')
    echo -n -e '\r'
    # Print prompt (to get current git branch)
    fish_prompt
    # Print the command with syntax highlighting
    fish_indent --no-indent --ansi (echo -n "$argv" | psub) | tr -d '\n'
    # Go to next line (where output is normally printed)
    echo -n -e "\n"
end

It’s slow and flickers a bit when redrawing, and it doesn’t work when using Python’s venv/virtualenv inside tmux (in that failure mode it puts your command on the line after the prompt, rather than next to it).

I’ve updated the monstrosity to do perform a bit better in this gist.

Notes

Yes, temporary output history doesn’t really work for interactive output, or stuff that redraws (progress bars). That doesn’t matter! You don’t need to “replay” that output anyways.
And yes, there might be security sensitive output. That’s why it’s a temporary buffer. I’m not sure you need output from more than 30 minutes ago anyways. Although I sometimes wish the whole OS state was a recording so that I could go back to a year ago and see how I solved a particular problem.

Minimal URL encoding

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Sun, 07 Nov 2021 00:00:00 +0000

I recently stumbled upon this StackOverflow question while looking at various ID-generation libraries suitable for use in URLs.

The question asks generally about compressing generic byte data into a URL. I was thinking more specifically about ID, e.g. compressing a UUID to the shortest possible representation suitable for use in URLs

The classic answer is to use “base64url” encode (equivalent to classic base64 but using _- instead of +=). base64url encoding uses the following alphabet:

ABCDEFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklmnopqrstuvwxyz
0123456789-_

Digging a bit deeper, I found that principally . and ~ can be included “safely” anywhere in the URL giving a “base66” encode:

ABCDEFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklmnopqrstuvwxyz
0123456789-_.~

Having just 2 bits extra, on top of an encoding that is already byte-aligned for every 4 output characters isn’t great. base66 encoding is byte-aligned for every 128 output characters! (losing precious bits in most cases, and almost never improving on base64 because we have to send whole bytes)

But let’s look at all printable ASCII symbols:

!"#$%&'()*+,-./:;<=>?@[\]^_{|}~`

It’s clear that ?, /, and # have to be excluded since they have special meaning, indicating start of query-params, path-separation, and start of hash respectively. The hash-part never gets sent to the server, but principally you could do some custom implementation server-side that looks at the raw request path ignoring the “directory” separators and leaving the query string unparsed so that / and ? could be used. However that is not the goal of this exercise.

This leaves us with:

!"$%&'()*+,-.:;<=>@[\]^_{|}~`

Let’s test what happens to these characters in Chrome and Firefox, with a URL like:

https://example.org/abc!"$%&'()*+,-.:;<=>@[\]^_{|}~`def

Firefox
Given the URL above, Firefox sends the following request to the server:

https://example.org/abc!%22$%&'()*+,-.:;%3C=%3E@[/]%5E_%7B|%7D~%60def

Leaving the symbols !$%&'()*+,-.:;=@[]_|~ unencoded.

Chrome
Given the URL above, Chrome sends the following request to the server:

https://example.org/abc!%22$%&'()*+,-.:;%3C=%3E@[/]%5E_%7B%7C%7D~%60def

Leaving the symbols !$%&'()*+,-.:;=@[]_~ unencoded.

Chrome encodes | whereas Firefox does not. Otherwise they’re the same. Both convert \ to /, which is why I’ve left it out of the “unencoded” character set.

Even though % isn’t encoded when it stands by itself, it cannot be used in a general algorithm since it could be followed by numbers or letters and thus interpreted as a percent-encoded character. Using it in the way shown above also logs an error in the Chrome console when the page is loaded.

I give you the 81 character alphabet of “URL-kludged base81”:

ABCDEFGHIJKLMNOPQRSTUVWXYZ
abcdefghijklmnopqrstuvwxyz
0123456789!$&'()*+,-.:;=@[]_~

Please don’t use this in production. I’m sure both the URLs and the code below will fail in spectacular ways with just a modicum of exposure to the real world.

A small example is that Github-Flavored Markdown will not include the last character in the URL when autolinking if it is one of ?!.,:*_~ (but will gobble up any non-space characters before that as part of the URL). Note that since _ is listed here this is also a small problem for normal base64url encoding.

More importantly is that the infrastructure you rely on probably isn’t well-equipped to handle these “slightly off” URLs, leaving you to deal with all sorts of weird edge-cases.

Sample code for UUIDs

def encode(number, alphabet):
    result = ''
    len_alphabet = len(alphabet)
    while number > 0:
        result += alphabet[number % len_alphabet]
        number = number // len_alphabet

    return result[::-1]


def decode(encoded_string, alphabet):
    number = 0
    len_alphabet = len(alphabet)
    for i, c in enumerate(reversed(encoded_string)):
        number += alphabet.index(c) * len_alphabet ** i

    return number

from uuid import uuid4

base81_alphabet = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789!$&'()*+,-.:;=@[]_~"
print(encode(int(uuid4()), base81_alphabet))
# Example outputs: 
# _6_Y)[XoG$,~La4D-a$(
# Bd!9&pAmXN$4lG7JV=wa3
# BXhczgO*'A=WTZoQPXIZR
# BdJoptB:Jxq2G]7PAe:mE
# IVDEUIN;n=GYQE0W6Jdp

Notes

A normal UUIDv4 will have 6-bits that are always the same, taking the information to be encoded down from 128-bits to 122-bits. This is not considered in the above writing.

Links

Two other interesting StackOverflow questions on UUIDs specifically: 1 and 2
Question 1 has an answer getting to a “base74” encoding, which in practice performs about the same my “base81”.
Question 2 has an answer that suggests using printable unicode characters, giving compression in text but not over-the-wire (which is actually fine for my usecase). One of the issues with that scheme is that users might get confused when they copy/paste the URL and get the encoded version.

Google Workspace unusual sign-in corporate mobile device

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Mon, 18 Oct 2021 00:00:00 +0000

If you’re trying to log in to your Google Workspace account and you get the following message:

We detected an unusual sign-in attempt. To make sure that someone else isn’t trying to access your account, your organization needs you to sign in using your corporate mobile device (the phone or tablet you normally use to access your corporate account). If you don’t have your corporate mobile device with you right now, try again later when you have your corporate mobile device with you. If you continue to have problems signing in, contact your administrator. Learn more

Go back & use your corporate mobile device

Then hopefully the solution below will help you.

Solution

I got this message – and I’ve never had a “corporate mobile device” at Eduflow (where I work) – and I also couldn’t find anything about it by searching in the Google Workspace admin interface.

To allow yourself to log in, log in as an admin to Google Workspace and go to your user account (here called “John Doe”).

Go to Users > John Doe > Security, then go to “Login challenge” and click it, and then click the “Turn off for 10 mins”-button.

That’s it! You should be good to log in again.

Finding an AWS IAM user via User Id

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Thu, 15 Jul 2021 00:00:00 +0000

In the previous blog post I talked about an AWS IAM User ID that I found in AWS S3 bucket policy. This is a little follow-up on that.

AWS IAM User IDs always start with AIDA and are 21 characters long:

AIDAXXXXXXXXXXXXXXXXX

So if you have one of these, you probably want to figure out which IAM user it belongs to. For AWS Access Keys, which look very similar, this is easy: just search for the key in the AWS IAM web interface. Unfortunately, it seems that you can’t search by User ID in that same interface.

So how do you figure out which user has the given IAM User ID?

An easy way to do that is to use the iam list-users command in the AWS CLI:

aws iam list-users

If there’s a lot of output you’ll be put into less and you can search for the User ID by pressing “/” and then type in the user ID and press enter.

If you have jq installed you can do:

aws iam list-users | jq '.Users[] | select(.UserId == "AIDAXXXXXXXXXXXXXXXXX")'

to get the user you’re looking for.

AWS S3: Invalid principal in policy

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Wed, 14 Jul 2021 00:00:00 +0000

The other day I was trying to update an AWS S3 bucket policy, but when trying to save the policy I got the following error:

Invalid principal in policy

(if you’re using boto you’ll see botocore.exceptions.ClientError: An error occurred (MalformedPolicy) when calling the PutBucketPolicy operation: Invalid principal in policy)

The policy looked like this

{
    "Version": "2008-10-17",
    "Id": "PolicyForCloudFrontPrivateContent",
    "Statement": [
        {
            "Sid": "1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "AIDAXYZABCDEF12345678"
            },
            "Action": "s3:GetObject",
            "Resource": "arn:aws:s3:::<bucket-name>/*"
        }
    ]
}

The problem was that the IAM user with the User ID AIDAXYZABCDEF12345678 (example ID here – not an actual ID) had been deleted. Changing the User ID to that of an existing user, allowed me to save the policy.

Running a script with large input on Heroku

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Tue, 04 Aug 2020 00:00:00 +0000

Note: Take a look the newer blog post Running a script with large input on Heroku (part 2) the method described there is better in most cases – unless you specifically need heroku run:detached <command> or you want to run interactively (heroku run bash).

Running a script on Heroku is easy:

heroku run -a <your app> python my_script.py

This runs your script in an environment with access to your production database, Redis and all the other stuff that.

This can be great for manual data migrations – say adding a demo course for all of your users.

However, in some cases you may want to run your script only for say 5,000 out of 20,000 users. Let’s say you’ve run some data analytics and you now have the IDs of the 5,000 users that the script should run on.

You don’t want to dirty up the repository with a custom file with the IDs of exactly those 5,000 users. What if you want to run the script again with 200 more users in a week?

The first thing you need to do is to allow your script to accept an input file: python my_script.py <input file>.

Now how do you get those 5,000 IDs into Heroku without adding them directly to the repository?

On Heroku you don’t have access to convienient tools like curl or wget. But if your app includes the requests-library (assuming you have a Python app) you can do

python -c "import requests; print(requests.get('https://google.com').text)" > test.txt

and then

python my_script.py test.txt

The full session would look something like:

heroku run -a eduflow bash
Running bash on ⬢ <your app>... up, run.9942 (Standard-1X)
~ $ python -c "import requests; print(requests.get('https://google.com').text)" > test.txt
~ $ python my_script.py test.txt

If you don’t have access to requests, you can use the built-in urllib-library:

python -c 'from urllib.request import urlopen; print(urlopen("https://google.com").read())'

Here, you’ll need to strip the beginning b' and ending ' since it returns a bytes-object.

For a Node app you can do something similar with fetch or axios.

fish process substitution

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Fri, 05 Feb 2016 00:00:00 +0000

Currenty, I’m using fish as my shell. Having used zsh for quite a while, there are some features I was missed from ZSH.

One is the ability to copy-and-paste any and all examples from the internet (watch, out that may be dangerous²). In Fish you can’t do that, as fish’s syntax isn’t always compatible with those.

For example fish doesn’t support the && and || operators, so instead of

./long_running_install_script && echo "Success!"

you have to do

./long_running_install_script; and echo "Success!"

This actually makes fish more aligned with the original Unix philosopy “do one thing, and do it well”, as the system already provides a way to do the thing you need, the shell shouldn’t become an all-encompassing system of itself.

Another thing is the syntax for command substitution. Here we use the which command to look up the path for the python binary, and then use ls -l to see what it links to (if its a symbolic link):

ls -l `which python`   # old syntax (still works)
ls -l $(which python)  # modern syntax

# in both cases, what gets "executed" by the shell
ls -l '/usr/local/bin/python'
# i.e. `which python` is replaced by it's output /usr/local/bin/python

in fish that looks like

ls -l (which python)

The fish syntax is simpler, and supports nesting like the modern syntax in bash and zsh, but in this particular case, there doesn’t seems to be much reason to break “backwards compatibility”. But then again, having already broken compatibility elsewhere – why not change to a lighter syntax here?

Process substitution

That leads me to the real subject of today’s post. One really cool thing about ZSH is Process Substitution⁴. Process substitution lets you take the output of a command and use it in place of a file, instead of outputting it directly into the command you’re trying to execute which is what command substitution does.

Let’s say you have two files with a bunch of names, and you want to see what names appear in one list, but not in the other. diff seems like the perfect tool, but that’s only going to work if the two list are sorted first. With process substitution we can do that on-the-fly:

diff =(sort soccer_team.txt) =(sort party_invites.txt)
# what gets "executed" by the shell
diff /tmp/tempfileA /tmp/tempfileB

Here zsh writes the output of the sort commands to two temporary files, which are then passed to diff. zsh keeps track of the temporary files and deletes them when they have been used.

In my case I needed to install all packages from requirements.txt except one – numpy. The obvious thing to do here is cat requirements.txt | grep -v numpy, which prints all lines except ones containing numpy. However, pip doesn’t support any piping from STDIN.

In ZSH I would do a process substitution

pip install -r =(cat requirements.txt | grep -v numpy)

today I found out you can do something similar in fish, using the command psub⁶:

pip install -r (cat requirements.txt | grep -v numpy | psub)

Once again, fish adheres to the UNIX philosophy, and chooses not to bloat with extra syntax, what can be done with a command :)

(To be clear: psub is fish specific, and actually part of the syntax, due to the way it has to cleanup after file-closing it would probably not be able to be implemented as a separate command outside of fish)

Notes The && and || operators were added to fish in version fish 3.0b1 (released December 11, 2018) for pragmatic reasons.

References

⁴ ZSH process substitution: https://zsh.sourceforge.net/Intro/intro_7.html ⁵ fish process substitution: https://news.ycombinator.com/item?id=9017996 Fish reverse process substitution: https://github.com/fish-shell/fish-shell/issues/1786

Fixing your SSL

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Fri, 16 Oct 2015 00:00:00 +0000

Today I discovered a new tool: SSL Labs’ SSL test.

It let’s you input your domain URL and then checks the security of your SSL configuration for known vulnerabilities. After the test it gives you an overall security rating (on the US school A-F scale) along with a list of recommended changes to fix any security holes and follow best practice.

Testing on peergrade.io I got a horrendous F. The SSL proxy is running on an old Ubuntu machine, which hadn’t been updated in a while, making the SSL setup vulnerable to the Heartbleed¹ vulnerability. Furthermore as the SSL-setup was an unconfigured/vanilla setup, POODLE² and other downgrade attacks were also possible.

Updating packages and revising the SSL configuration I got the security rating up to an acceptable A-grading.

I can highly recommend using SSL test to check your SSL setup. Not only does it point out vulnerabilites but also links to resources on what the vulnerabilities entail and how to fix them.

¹ Heartbleed: https://heartbleed.com/
² POODLE: https://en.wikipedia.org/wiki/POODLE and https://www.openssl.org/~bodo/ssl-poodle.pdf

Flask exception as push notification

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Mon, 28 Sep 2015 00:00:00 +0000

I’m currently working on a small project: a web service written in Flask.

The service is currently at an early stage, and still so small that I want to be notified of every exception that happens in production.

I can easily get these as mails¹ – which I am – but during peak hours I want to be able to respond immediatly to an error, and not have it drown in the rest of whatever mail I’m getting in.

So I thought, why not receive a push notification?

I was sure I had heard of at least one app made with just that in mind – receiving custom push notifications that can be created automatically via an API. (I don’t want to create my own iOS app just for that with provisioning and all that cruft and timesuck)

Sure enough: Pushover and Boxcar fit the bill. I ended up going with Pushover for now as Boxcar requires iOS 8 which I’m not on yet.

I’m already sending the exceptions as mail to my Gmail, with a “plus-alias”: my-email+flask-exceptions@gmail.com² so the Pushover email API³ was perfect for quickly getting up and running.

I set up a filter in Gmail, forwarding those mails (filtered on the To: field) to API-KEY@api.pushover.net – and now I’m receiving exceptions from Flask as push notifications on my phone. Pretty neat!

¹ https://flask.pocoo.org/docs/0.10/errorhandling/#error-mails
² https://support.google.com/mail/answer/12096?hl=en
³ https://updates.pushover.net/post/24408806716/e-mail-to-pushover-gateway

Getting SSH to choose the right key

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Thu, 22 Jan 2015 00:00:00 +0000

On Bitbucket I have a work user and a personal user. This morning trying to push a project where my work user is a member, failed:

> git push
conq: repository access denied.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

The git remote looks like this:

> git remote -v
origin  git@bitbucket.org:<WORK_PROJECT>/<WORK_PROJECT>.git (fetch)
origin  git@bitbucket.org:<WORK_PROJECT>/<WORK_PROJECT>.git (push)

I even set up ~/.ssh/config to make sure my work SSH key (associated with my work SSH user) is being used on bitbucket.org:

> cat ~/.ssh/config
Host bitbucket.org
    IdentityFile %d/.ssh/<KEY_WORK>

Bitbucket’s troubleshooting suggests ssh -T git@bitbucket.org to check if authentication is working

> ssh -v -T git@bitbucket.org

OpenSSH_5.6p1, OpenSSL 0.9.8za 5 Jun 2014
debug1: Reading configuration data ~/.ssh/config
debug1: Applying options for bitbucket.org
debug1: Reading configuration data /etc/ssh_config
debug1: Applying options for *
debug1: Connecting to bitbucket.org [131.103.20.167] port 22.
debug1: Connection established.
debug1: identity file ~/.ssh/<KEY_WORK> type 1
debug1: identity file ~/.ssh/<KEY_WORK>-cert type -1        <----   Yup, this is what I want, my work key being used

... protocol negotiation and fingerprint ...

debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: ~/.ssh/<KEY_PERSONAL>      <----   Wait, what?
debug1: Remote: Forced command: conq username:malthejorgensen

... ssh extensions ...

debug1: Authentication succeeded (publickey).
Authenticated to bitbucket.org ([131.103.20.167]:22).

... entering interactive session, setting environment...

logged in as malthejorgensen.                               <----   My personal user :(

You can use git or hg to connect to Bitbucket. Shell access is disabled.

... bla bla bla, many bytes were sent ...

debug1: Exit status 0

As we can see SSH sees in my configuration that KEY_WORK should be used, and yet it ends up authenticating with KEY_PERSONAL and logs me in as my personal user.

The problem is the ssh-agent, and that KEY_WORK is not added to it. We can check that as follows:

> ssh-add -l
2048 09:c1:32:4a:e7:c2:05:6b:4e:52:71:aa:b8:ee:3e:53 ~/.ssh/KEY_PERSONAL (RSA)

Which shows that only KEY_PERSONAL is known to ssh-agent. We can add KEY_WORK to ssh-agent like this:

> ssh-add ~/.ssh/KEY_WORK
Identity added: ~/.ssh/KEY_WORK (~/.ssh/KEY_WORK)
> ssh-add -l
2048 09:c1:32:4a:e7:c2:05:6b:4e:52:71:aa:b8:ee:3e:53 ~/.ssh/KEY_PERSONAL (RSA)
2048 08:5b:1e:8e:a8:7d:06:61:dc:07:42:7b:ca:b7:49:bd ~/.ssh/KEY_WORK (RSA)

We can see that both keys are now known to ssh-agent. And now – it works:

> ssh -T git@bitbucket.org
logged in as malthe-at-socialsquare-dk.

You can use git or hg to connect to Bitbucket. Shell access is disabled.

In some cases you have to remove your personal key from ssh-agent (you can always re-add it later). You can remove it with ssh-add -d ~/.ssh/KEY_PERSONAL.

References

UNIX tools, spaces in filenames and \0

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Mon, 01 Dec 2014 00:00:00 +0000

A normal way to delete all files containing _old in the current folder (on a UNIX system) is a command like the following:

find . -iname '*_old*' | xargs rm

However this doesn’t work for filenames containing spaces as rm expects spaces to separate each file.

The usual solution to this is to add -print0 to the find command and -0 to xargs:

find . -iname '*_old*' -print0 | xargs -0 rm

Now my problem is that I only want to delete the first 5 files. The “UNIX way” is to do something like:

find . -iname '*_old*' | head -n 5 | xargs rm

– and now we’re back to something that doesn’t work with filenames containing spaces. Our -print0-trick from earlier won’t work because head doesn’t have an equivalent of the xargs -0 flag.

`awk` to the rescue?

Note: The following assumes that awk refers to BSD-awk – specifically the awk provided on OS X 10.7.5.

awk operates on a line-by-line basis by default, but this can be changed with the record separator variable RS. A simple head -n 5 implementation would be awk 'NR <= 5 {print}' in awk.

Lets say we’re in a folder with these files:

> ls -1
Los Angeles vacation_001_old.jpg
Los Angeles vacation_002_old.jpg
Los Angeles vacation_003_old.jpg
Los Angeles vacation_004_old.jpg
Los Angeles vacation_005_old.jpg
Los Angeles vacation_006_old.jpg
Los Angeles vacation_007_old.jpg
Los Angeles vacation_008_old.jpg
Los Angeles vacation_009_old.jpg

With awk we should be able to do something like this:

> find . -iname '*_old*' -print0 | awk 'BEGIN {RS="\0"}; NR <= 5 {print}'
./Los Angeles vacation_001_old.jpg

… but it doesn’t work: awk never finds records beyond the first one because of the NUL character \0. The NUL character is the C-string terminator and thus AWK stops reading input after seeing it because it thinks the input string has ended.

However the more modern awks gawk and mawk doesn’t suffer from the same shortcoming:

> find . -iname '*_old*' -print0 | mawk 'BEGIN {RS="\0"}; NR<=5 {print}'
./Los Angeles vacation_001_old.jpg
./Los Angeles vacation_002_old.jpg
./Los Angeles vacation_003_old.jpg
./Los Angeles vacation_004_old.jpg
./Los Angeles vacation_005_old.jpg

But the print function in awk (+ mawk and gawk) print newlines by default. We can change this by setting the output field separator ORS

> find . -iname '*_old*' -print0 | mawk 'BEGIN {RS=ORS="\0"}; NR<=5 {print}'
./Los Angeles vacation_001_old.jpg./Los Angeles vacation_002_old.jpg./Los Angeles vacation_003_old.jpg./Los Angeles vacation_004_old.jpg./Los Angeles vacation_005_old.jpg

The NUL characters doesn’t print – but they are there – you can check by piping into cat -v.

Finally we reach the command:

> find . -iname '*_old*' -print0 | mawk 'BEGIN {RS=ORS="\0"}; NR<=5 {print}' | xargs -0 rm

In sed you can do something like sed -n '1,5p' but sed expects newlines and has no setting to change the “separator” to be something other than newline (sed is Turing-complete, however, so nothing is impossible¹).

Afterthought

I think the UNIX tools are great. No doubt about it. I use them every day, and the “do one thing, and do it well”-philosophy plus the composability of tools through pipes are some of the things that these tools invaluable to many programmers, including me. BUT the tools are from the 80’s (and sometimes even older) and have fallen out of touch with modern computing. Today the users’ needs and behaviour shapes the systems rather than the other way around.

We need to build tools that can handle filenames containing spaces and unicode characters. I have an older blog post that touches the same topic: The Unix shebang (#!).

For now I’ve made a head “implementation” for input separated by NULs instead of newlines:

head0() {
  mawk 'BEGIN {RS="\0"}; {print}' | head $@ | mawk 'BEGIN {ORS="\0"}; {print}'
}

Should we replace the UNIX tools with more modern, but equivalent tools? Is a command-line like PowerShell with built-in types the way to go?

One thing is clear to me: a solution to these problems need to be found. And preferably one that fixes them once and for all.

Fastest way to get your NTFS-drive writeable on OS X

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Mon, 01 Dec 2014 00:00:00 +0000

Create the file /etc/fstab (it doesn’t exists by default) and add the line

LABEL=DRIVENAME none ntfs rw,auto,nobrowse

where DRIVENAME is the name of your NTFS drive. Unmount and mount the drive by issuing the following commands:

diskutil eject DRIVENAME
diskutil mount DRIVENAME

Note: Do not use the command umount or diskutil unmount instead of eject – it forces the unmount and stops any file transfers currently running to and from the disk (which means you are likely to lose data).

You can now browse the drive by issuing the command open /Volumes/DRIVENAME.

Unfortunately the nobrowse flag in fstab means that the drive will not appear on the Desktop nor in the Finder sidebar. Sadly, if you omit it – the drive will appear, but will be read-only.

Tested on OS X 10.7.5.

Reference: https://computers.tutsplus.com/tutorials/quick-tip-how-to-write-to-ntfs-drives-in-os-x-mavericks–cms-21434

League of Legends on Linux

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Sun, 23 Feb 2014 00:00:00 +0000

From time to time I play League of Legends with a couple of friends. After having switched to Linux from OS X, I needed to get it working on Linux as there’s no native client for Linux.

What worked for me was to install it via PlayOnLinux. This worked perfectly until I needed to start an actual game. (Updater, Login, Lobby etc. worked fine)

After the Character Select countdown, the splash showed and just stuck. Nothing happened. It never showed the loading screen or the actual game.

The fix for me was to the change the hosts file: /etc/hosts. In this file there should be the following lines (it is the second one that is important):

#<ip-address>	<hostname.domain.org>	<hostname>
127.0.0.1	localhost.localdomain	localhost

Look up your computer’s hostname

> hostname
macbook-malthe

And then change the line in /etc/hosts to this:

#<ip-address>	<hostname.domain.org>	<hostname>
127.0.0.1	localhost.localdomain	macbook-malthe

(replacing macbook-malthe with your own hostname from before)

Hopefully it will be as pain-free for you as it was for me ;)

Note: When starting the game, the load screen (in fullscreen with the loading bars and your ping showing) will be stuck for about 1 minute, but don’t worry: the game will start. It did for me at least.

Reference: https://www.playonlinux.com/en/issue-2219.html

Thunderbird opening links in Chromium

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Tue, 18 Feb 2014 00:00:00 +0000

Thunderbird on my Arch Linux installion was opening links in Chromium, even though everything else on the system pointed towards Firefox being the default browser.

Some people on the internet said to go into Preferences > Preferences > Advanced > Config Editor (yes, that’s preferences twice) and add the keys

network.protocol-handler.app.http
network.protocol-handler.app.https

and set their values to /usr/bin/firefox. This, however, did not work.

What did work was to set the keys

network.protocol-handler.warn-external.http
network.protocol-handler.warn-external.https

to true. This makes Thunderbird prompt you what program to open every time you click a link. Here you can choose Firefox, and also check “remember my choice…”.

Reference: https://www.ghacks.net/2013/01/20/fix-for-thunderbird-not-opening-links/

Macbook Pro backlight on Arch Linux

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Sun, 19 Jan 2014 00:00:00 +0000

I recently installed Arch Linux on my Macbook Pro. In order to control the screen brightness I installed nvidia-bl from the AUR (I use yaourt so: yaourt -S nvidia-bl). This sets up a folder in /sys/class/backlight/nvidia_backlight and in this folder you can get and set the screen brightness via the file brightness:

> cat brightness
1000
> sudo su
# echo 500 > brightness

But after a system update (pacman -Syu) the /sys/class/backlight/nvidia_backlight folder was gone and I could no longer change the brightness.

By running dmesg I could see that the nvidia_bl kernel module was not being loaded during boot:

[    7.183486] nvidia_bl: disagrees about version of symbol module_layout

Trying to load it manually also failed:

> sudo modprobe -v nvidia-bl
insmod /lib/modules/3.12.7-2-ARCH/extramodules/nvidia_bl.ko
modprobe: ERROR: could not insert 'nvidia_bl': Exec format error

These errors mean that the kernel module is not compatible with the kernel. This can be solved by recompiling the kernel module. Since I use yaourt I can simply reinstall nvidia-bl: yaourt -S nvidia-bl.

After that you can manually load the kernel module again:

> sudo modprobe -v nvidia-bl
insmod /lib/modules/3.12.7-2-ARCH/extramodules/nvidia_bl.ko

And you can now control the brightness again :)

Note: I use a Aluminium Unibody Macbook Pro 5,5 from 2008

Editing videos with Blender

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Sat, 09 Nov 2013 00:00:00 +0000

Blender is an Open Source 3D modelling- and animation tool.
– Today I found out it also does video editing!

This post is based on Blender version 2.69

It is well known that the Open Source community has yet to produce decent video editing software. But being stubborn and this being a small project I didn’t want to buy one of those large, expensive video editing suites.

Blender to the rescue

In order to use Blender as a video editor the first thing you need to do, is to switch the window layout to “Video Editing”. (There’s a dropdown in the top menu bar)

In this view you should be able to see the timeline at the bottom of the window. Here you can drag video files and images into your movie project’s timeline. The timeline is called the Video Sequence editor in “Blender speak”. The imported clips and images will be placed at the time slider (the green vertical line on the timeline)

Left clicking on the timeline sets the time slider and updates the scene view accordingly. The scene view is a preview of a specific frame in your movie project. This also means that you cannot move clips in the timeline by clicking and dragging them.

Each clip is called a Strip.

You select strips in the timeline by Right clicking. Moving strips is done by pressing G [Grab]. Having selected one or multiple strips by right clicking, and then pressing G, the selected strips will follow your mouse as if you were dragging. You should not hold down any mouse or keyboard button while doing this.
Left-click to end the move. Right-click to cancel the move.

Pressing G while moving strips will change the “move” mode. The default state is that if you move a strip on top of another the other strip will be cut (shortened). The other mode will not shorten other strips, but simply move them to make space for the moved strip.

To shorten or lengthen a strip you can select one of the “handles” (the small triangles) at each end of the strip and moving them same way you select and move the whole strip.

When you have selected a strip you should see a panel to the right of the timeline called “Edit Strip”. In this panel you can edit the properties of the strip. For example you can choose whether it should be played forwards or backwards, or that only every second frame should be shown (this is called Strobe).

When you add a strip to the timeline it is placed in a specific layer. These layers are called Channels by Blender and are shown in the timeline as 0, 1, 2… on the left-hand side. Strips in channel 1 will be on top of strips in channel 0, strips in channel 2 will be on top of strips in channel 1, and so on. The strips are drawn on top of eachother in according to the strip’s blend property. I recommend the Over Drop settings that simply draws the strip on top of the other without any blending.

You can watch your current movie project by pressing Alt-A. I recommend choosing the Sync Mode called AV Sync. Otherwise you risk the video and audio going out of sync while you watch your project, and the time slider might not show the correct position. In principle the Frame dropping setting for the Sync Mode should be better, but it didn’t work for me.
To find the Sync Mode menu look for the text “No Sync” at the bottom of the Blender window.

Picture-in-picture (PIP)

To scale a strip (make it smaller or larger), you can add a transform by selecting the strip, then in the menu below the timeline `Add > Effect Strip…

Transform`. You can then select the Transform strip and choose the settings for the effect.

The values of the effect can be changed by clicking the value and dragging left and right.

Saving the movie

As Blender is mainly a 3D modelling and 3D animating tool, saving your movie is called “rendering”. In the lower left corner of every window “tile” (the scene view and the timeline for example), you can choose what kind of editor should be shown in this tile. Rendering can be done from the an editor called Properties. The first pane of the Properties editor is a small camera icon: this is the rendering pane. Here you can choose the output format, frames per second and so on.

Rendering can be stopped by pressing ESC. The frames rendered so far will be saved to the file (i.e. they will not be lost).

Audio sync problem

When you import a movie strip it is split into a video and an audio part. When I did this, I got a video part with 13151 frames but the audio part only had 12625 frames. This was because my render was set to be 24 frames per second but the imported movie had 25 frames per second. Blender matches the frames of the imported video 1:1 to the frames in the render. This stretched the imported video to be longer than its original length and thus longer than the imported audio. I fixed this by setting the render to have 25 frames per second.

Thoughts about Blender

Blender is an awesome powerful tool – 3d modeling, 3d animation, compositing, video editing, and even a game engine is included in Blender.

But Blender is not for the faint of heart. The interface (in my opinion) is not optimized for newcomers or intuitive use. Rather it is optimized for effective use by experts. This means you have to accept things such as not being able to drag strips in the timeline by holding down the mouse, and that saving your movie is done via the “Properties” editor.

Because of this, I see as tool Blender similar to Vim – powerful, but with a tough learning curve.

Thank you Blender!

Last note

For small strips I found it hard to select (click) the left and right handles (the beginning and end of the strip). Luckily there’s a menu below the timeline called Select which has a Right Handle and a Left Handle item. By right-clicking on either of those you can assign a keyboard shortcut to that menu item.

Buying things online (scams)

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Sun, 23 Jun 2013 00:00:00 +0000

Long story short: bosto-tablet.com is a great, legimate seller of Bosto drawing tablets. I bought a Kingtee 19-mb from the site and it was delivered within days of my purchase, and I am very happy with the product. However before I bought something off the site I was worried that it was a scam. This is a story of worry, research and a leap of faith.

So I was planning to buy a drawing tablet, like the ones from Wacom. However, after researching a little I found out that today (as opposed to a couple of years ago when I last checked) cheaper are alternatives available – possibly because some of Wacom’s patents have expired. These alternatives are made by two companies called Yiynova and Bosto. After reading some reviews and viewing a lot of videos demonstrating the tablets I decided to go for the Bosto kingtee 19-mb.

But I needed it shipped to Denmark at a reasonable price – so where was I going to get that? I found bosto-tablet.com which looks quite nice. But the site isn’t really mentioned anywhere on the net – so how could I know that it was run by the manufacturer or a legitimate reseller? That it wasn’t a scam?

The site is very plain, and doesn’t really give a lot of information, about who runs the shop and where it’s located. The site also has a list of recent customers with date of purchase and model number. (At the time the table didn’t show the DHL tracking as it does now)
If you’re already suspicious of the site this list comes off as a baiting device, as you don’t normally see such lists on larger webshops. The list is basically yelling: “Hey look, we’re legit – just go ahead and buy something already!”, which is coincidentally also what you would want to do if you were luring people into a scam.

So I decided to probe a little further. The site presents the mail info@bosto-tablet.com to which I wrote:

Hi Bosto

Do you ship to Denmark?

– Malthe

Perhaps I would get a badly written response? (as per the usual spam email, which would indicate a scam)

I got the reply:

Malthe, we ship to Denmark. It’s free. Have a good buy

--
Sincerely yours, Andrey Belkov, Bosto

Now Bosto is a Chinese company, so it was weird that someone with a russian sounding name responded. The response itself is well-written and short, basically what I would expect from a modern web company. Not from a Chinese tablet manufacturer. This well-written short response could just be luring me further into a possible scam…

Well, in the end I decided to go for it anyway and I was able to track my package via DHL on its 3-day journey from Hong Kong to Copenhagen (Denmark). I recieved the package 6 days after my Paypal payment.

The PayPal vendor information was kinda funny (you can see it on the left). Obviously Chinese symbols aren’t allowed in email addresses and Chinese people often choose an English name to ease communication with English-speaking countries hence the “roger”. At least that’s my guess as to why the email is what it is. Forhandleroplysninger means Seller info in danish. (Displayed as Merchant in english Paypal)

Now, how do you know this blog post isn’t part of the scam? ^^

Possible solutions

Of course there should be and are sites and services that keep track of whether vendors are real and can be trusted. None of the ones mentioned here had a review of bosto-tablet.com at the time of writing.

Sites

ResellerRatings even though they earn money by making the vendors pay to contact users on the site (to respond to any critique) the site is still riddled with ads.
TrustPilot has the same as business model as ResellerRatings but without ads. Being a newer, originally Danish, site it doesn’t have the volume ResellerRatings has (yet), but it is better. They have some problems with fake reviews, though I guess any site of this sort will. It’s a very important caveat and they really should work hard to come up with a good solution.

Browser plugins

Having such a service available as a browser plugin is a very good idea. It saves you the time of looking up the vendor – it’s available right in the browser. Futhermore the instant availability of the plugin makes it easy to review sites and could possibly mean that more sites get rated than on sites such as TrustPilot.

WOT tried it, and it worked very nicely.
websherpa another browser plugin, didn’t test it.

The Unix shebang (#!)

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Thu, 20 Jun 2013 00:00:00 +0000

The Unix shebang is the first line in a typical executable Unix script. For example a an executable Python script called ‘hello’ can be run by writing ./hello on the command line, instead of writing out the full command python hello, given that the first line in the script is

#!/usr/bin/python

This line tells that the script should be run by using the Python interpreter located at /usr/bin/python. E.g.

> cat hello 
#!/usr/bin/python
print 'Hello World!'
> ./hello
Hello World!

One would think that the shell, e.g. bash or zsh interprets this line and executes the proper script interpreter. But it’s not the shell – it’s actually the kernel that interprets this line.
(For further reference check out https://www.in-ulm.de/~mascheck/various/shebang/)

I found this out when trying to use Python virtualenv which creates .venv/bin/pip. The pip file is a Python script with a shebang in the beginning – in this case something along the lines of

#!"/Users/Malthe/python project/.venv/bin/python"

Here virtualenv has quoted the path because it has a space in it. But this doesn’t work on OS X 10.7 (I haven’t tested on other systems).

I haven’t found a solution yet (working on it). The XNU (Mac OS X) kernel code parsing the shebang can be found here beginning at line 432.

So I wrote a file test to check out the behaviour:

#!/bin/exec-space arg1 arg2 arg3 arg4 arg5 arg6 arg7 arg8 arg9 arg10 arg11 arg12

With exec-space being:

#!/bin/sh
echo $0
echo $1
echo $2
echo $3
echo $4

Running ./test in fish gives the following error

> ./test
Failed to execute process './test'. Reason:
exec: Exec format error
The file './test' is marked as an executable but could not be run by the operating system.

bash does nothing

> ./test
>

zsh does something

> ./test
/bin/exec-space
arg1 arg2 arg3 arg4 arg5 arg6 arg7 arg8 arg9 a
./test

The result from zsh show that all arguments are concatenated intro $1 but truncated after some number of characters. The morale of the story is then to not use virtualenv in neither paths containing spaces nor long paths. I guess computer software still lives in 1300 BC…

A guy called Steve Smith found out the same thing a couple of years ago: https://stevesmithbytes.blogspot.com/2011/02/handling-spaces-in-shebang-line-in-mac.html

Battle of the PaaSs

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Wed, 15 May 2013 00:00:00 +0000

So what is a PaaS?

PaaS stands for Platform as a Service and means that you pay for a service that provides a platform on which your code can run. That could be a Python platform for running webcrawlers, a node.js stack for running web applications etc.

In this context PaaS is an alternative to the usual web hosting setups:

Regular hosting You have a user with an FTP directory where you upload your PHP files or the like. Very easy but not very customizable or scalable.
Dedicated/virtual server You have a single server over which you have full control (OS, packages). Fully customizable, somewhat scalable (larger server)
Roll your own cloud solution Set up instances and HTTP routing on Amazon Web Services. 100% scalable, very customizable

For the project I’m currently working on I need scalability (multiple instances) and customizability (custom Python packages), which rules out regular hosting (#1) and using a dedicated/virtual server (#2).

Amazon Web Services (#3) of course makes server management pretty easy, but I still don’t want to worry about choosing a Linux distribution, updating packages, and setting up HTTP routing. Therefore using a PaaS is the way to go.

Using a PaaS basically means that you can simply push your code to the service and it will handle deployment, setting up multiple instances and so forth (even installing needed Python packages automatically).

The contenders

Heroku

The grandfather of PaaS: the first widely known and widely used service of this type. It’s high quality, very popular, and is still the trend-setting service in this field. All other services mentioned here are more or less watered-down copies of Heroku. Hopefully we will soon see new ideas coming from the competition but right now Heroku is still the driving force of innovation in the field.

My experience: Very easy to set up and great documentation. It took some time deploying the first time (~4 mins), but the service itself is very responsive both through the command line and the web interface.

Bottom line: Heroku is great (the best), but too expensive.

dotCloud

A relatively new, but also quite popular PaaS.

My experience: Most of the services mentioned here use Amazon EC2 as a backend, including dotCloud. When you need a server on EC2 you “provision” it. The selected OS is transferred to its local storage and keys and DNS is set up. This takes about 8-10 minutes. The other services here have a reserve of instances already booted up and ready so that when you create a new app the service simply need to setup the code and HTTP routing which takes less than five minutes.

BUT dotCloud provisions Amazon instances on demand. And it takes a looooooong time: more than 30 minutes.

30 minute wait time is just something that I don’t want to – and don’t have to – deal with.

This wait problem would also affect zero-downtime deployment where you typically have two different deployments of your site running at the same time (one old, one new) and then change the routing from the old one to the new. This provisioning lag would make zero-downtime deployment very time consuming.

Bottom line: dotCloud provisions instances on demand, which creates long wait times that other services don’t have. No, thank you dotCloud.

Gondor.io

Smaller service focused on Python/Django which seems to fit just my needs.

My experience: For some reason it was very difficult to set up. Whereas the other services only needed a few changes to the Django settings, Gondor needed a lot of tweaking and I never actually succeeded in getting it running.

Bottom line: Never worked.

AppFog vs. cloudControl

That leaves two services: AppFog and cloudControl.

Setting up with cloudControl was extremely easy, thanks to concise and well-written documentation. The AppFog documentation on Django was a bit more sparse, but setup was still pain-free.

The cloudControl command-line is based on Heroku’s command-line tool and is pretty good (Heroku’s own is still better), whereas AppFog’s is based on cloudFoundry’s command-line tool and does have some minor problems.

Category	AppFog	cloudControl
Setup	Okay	Easy
Command-line	Half-baked¹	Better
Web interface	Good	Half-baked
Logs²	Slow and difficult	Okay
Documentation	Worse	Better
Database³	Built-in	N/A - outsourced

af logs and af update are slower than on most other services but not irritatingly so. Also, you cannot run Python/Django commands on the server, e.g. python manage.py syncdb or python manage.py migrate which is a pain (you have to manually do these things).
AppFog has no online interface for viewing logs and the command-line logs are slow. cloudControl has an online log viewing inferface and their command-line logs are much faster to access.
You cannot setup triggers on the AppFog MySQL database. cloudControl doesn’t supply any database themselves but outsources them, for example to ElephantSQL which I found pretty great. Of course you can outsource with AppFog as well, but if they can’t supply a great database solution – they shouldn’t supply one at all – but rather default to outsourcing like cloudControl.

Bottom line

cloudControl is the better service, but their pricing is about double that of AppFog. The flaws of AppFog have so far not been deal-breakers for me.

Therefore my choice of PaaS is – for now – AppFog.

References

PaaS on Wikipedia
Zero-downtime deployment means updating your website with zero downtime for the end users (i.e. without “Website currently undergoing maintenance”). This is usually achieved by setting the updated version up on separate instances while having the old version still running. The HTTP routing is then shifted from pointing to the old version to the new, and the instances with the old version can then be shut down.

Adventures in DNS

malthejorgensen.github@malthejorgensen.github.com (malthejorgensen) — Tue, 09 Apr 2013 00:00:00 +0000

When you start a site on a specific domain you often wanna have some email account associated with that domain, such that users of your site can send mail to an email that is transparently belonging to the site. E.g. support@example.org rather than support-mysite@gmail.com.

An easy and in my opinion very nice solution is setting up a Google Apps for the domain. This way you get the familiar Gmail-interface for your support@example.org mail, and you don’t have to set up and manage some email server like Postfix or exim4. (Which can be HUGE hassle)

To get this email service you need to set up Google Apps MX records, which is the real reason I’m writing this post: I’m getting tired of setting up these records. It’s a tedious and too click-hungry process on my current DNS service GratisDNS.

To recieve mail sent to your domain through Google Apps what you gotta do is have the following MX records setup in the DNS for your domain:

example.org 3600 IN MX 1 aspmx.l.google.com.
example.org 3600 IN MX 5 alt1.aspmx.l.google.com.
example.org 3600 IN MX 5 alt2.aspmx.l.google.com.
example.org 3600 IN MX 10 aspmx2.googlemail.com.
example.org 3600 IN MX 10 aspmx3.googlemail.com.

At GratisDNS you submit a form (no-AJAX) for each MX entry and if you also want to have aliases (.com, .net, .org, .dk) you get to Copy/Paste or type these records in quite a few times. So I wanted to find some alternative DNS service where setting up DNS records would be faster.

So I looked for some danish alternatives and found:

The interface on these sites was weird and I didn’t really understand what was going on so I continued my search internationally.

Note: Dandomain.dk actually has a pretty good interface, and makes it a bit faster to set up the DNS records but still not really as easy as I wanted to be.

Route 53 on AWS (Amazon Web Services) definitely has the features I want but the interface could be better, which is why I ended up choosing gandi.net which has a very slick interface.

Enter the Zone File

All along what I was looking for was the zone file. The zone file is the file the DNS server reads in order to know what to do when a DNS request for your domain is received. It is simply a text file wherein you specify your DNS records. It has a somewhat funny syntax, and different DNS services allow different parts of the syntax in their zone files.

This is the zone file I needed:

@ 3600 IN MX 1 aspmx.l.google.com.
@ 3600 IN MX 5 alt1.aspmx.l.google.com.
@ 3600 IN MX 5 alt2.aspmx.l.google.com.
@ 3600 IN MX 10 aspmx2.googlemail.com.
@ 3600 IN MX 10 aspmx3.googlemail.com.

@ is zone file syntax for “current domain” and in a DNS service this will most likely already be specified when the zone file is loaded. (This is true for gandi.net)

What you might wanna is to read up on:

, which I found to be nice resources.

Gandi.net “Expert mode”

On gandi.net, you can set Edit mode to “Expert” which lets you edit the zone file directly. This is actually the same as what you can do on QuickDNS.dk and pleen.dk, though I didn’t know about zone files at that time. Furthermore on Gandi.net you can make a zone file template that you can use on multiple sites. This is ideal for the MX record setup for Google Apps because it is the same for all sites.

In a template you have to use @ as a placeholder for the domain name (otherwise it would be a template that could be used for other domains) and thus the zone file template for Google Apps MX records is exactly the one listed earlier. QuickDNS.dk actually allows exactly this kind of templating but the interface is very confusing. Pleen.dk allows for editing the zone file directly but not templating.

Gotchas for Gandi.net zone file templates:

You cannot refer to a specific domain name in the zone file – use @ to refer to the domain
You cannot specify the SOA directive – this is set by Gandi.net.
You cannot use parenthesis (for multiline statements)

blagblogblag.com

Is Github’s business model still viable?

The business model v1

The business model v2

Update (May 14th 2026)

Footnotes

Running a script with large input on Heroku (part 2)

Sending multiple files

The long story about sending multiple files

Why do we get base64: invalid input? (BSD vs GNU base64)

Caveats

Notes

Show the time a command was run in fish shell

Notes

Minimal URL encoding

Sample code for UUIDs

Notes

Links

Google Workspace unusual sign-in corporate mobile device

Solution

Finding an AWS IAM user via User Id

AWS S3: Invalid principal in policy

Running a script with large input on Heroku

fish process substitution

Process substitution

Fixing your SSL

Flask exception as push notification

Getting SSH to choose the right key

UNIX tools, spaces in filenames and \0

awk to the rescue?

Afterthought

Fastest way to get your NTFS-drive writeable on OS X

League of Legends on Linux

Thunderbird opening links in Chromium

Macbook Pro backlight on Arch Linux

Editing videos with Blender

Blender to the rescue

Picture-in-picture (PIP)

Saving the movie

Audio sync problem

Thoughts about Blender

Buying things online (scams)

Possible solutions

Sites

Browser plugins

The Unix shebang (#!)

Battle of the PaaSs

The contenders

Heroku

dotCloud

Gondor.io

AppFog vs. cloudControl

Bottom line

References

Adventures in DNS

Enter the Zone File

Gandi.net “Expert mode”

Why do we get `base64: invalid input`? (BSD vs GNU `base64`)

`awk` to the rescue?