Jinked!

I Work At Google Now

Wed, 06 Sep 2017 14:55:29 +0000

Just so we’re clear on my bias and all.

How To Scale Celery

Thu, 15 Jun 2017 12:02:47 +0000

You can’t.

How To Make Elasticsearch Fast

Fri, 12 May 2017 12:11:28 +0000

Add the following to the settings of an index you want to speed up:

{
  "settings": {
    "index": {
      "queries": {
        "cache": {
          "everything": true
        }
      }
    }
  }
}

Why?

// for test purposes only
public static final Setting<Boolean> INDEX_QUERY_CACHE_EVERYTHING_SETTING =
    Setting.boolSetting("index.queries.cache.everything", false, Property.IndexScope);

// This setting is an escape hatch in case not caching term queries would slow some users down
// Do not document.
public static final Setting<Boolean> INDEX_QUERY_CACHE_TERM_QUERIES_SETTING =
    Setting.boolSetting("index.queries.cache.term_queries", false, Property.IndexScope);

Alternatively, if you’re serious about searching, use Solr. Bonus points if you back it with HDFS so you never have to sync data.

Good luck!

Using Go in a Unirepo

Wed, 21 Dec 2016 12:11:28 +0000

I’ve recently started a love affair with Go, much to the disgust of some of my coworkers. One of the best things about using Go is how ridiculously fast your work cycle becomes when compared to other languages, such as Erlang and Elixir. Simple things that require a tremendous amount of work are a non-issue in Go, which ultimately helps you get to market faster.

However, a common complaint about Go is package and dependency management. There are no shortages of community developed package managers for Go, but in general I don’t think a package manager is really needed so long as you structure your work correctly.

Unirepo Folder Layout

Short and simple, here’s what works the best:

code/ <--- This is your GOPATH, but you don't ever need to set it. Keep reading.
  bin/
  pkg/
  src/
    make/
      Makefile
  tools/
  .git/
  .gitignore

Installing Dependencies

Again, keep it simple and just do

GOPATH=/path/to/code go get -u github.com/...

Setting Up Your Code

All new projects should live in code/src/<project_name>. In each project folder, create a Makefile and include the following in it

include ../make/Makefile

Compiling Your Code

This one is a bit tricky, but with the magic of Bash and Make, you can make life pretty easy. In code/src/make/Makefile, paste the following

export GOPATH = $(realpath $(shell dirname $(realpath $(lastword $(MAKEFILE_LIST))))/../../)
export GOBIN = ${GOPATH}/bin
export APP = $(shell basename "$$PWD")

build::
	go build
	go install

run::
	../../bin/$(APP)

test::
	@mkdir -p $(GOPATH)/tests/
	@echo 'mode: set' > coverage.txt && go list $(APP)/... | xargs -n1 -I{} sh -c 'go test -covermode=set 	-coverprofile=coverage.tmp {} && tail -n +2 coverage.tmp >> coverage.txt' && rm coverage.tmp
	@go tool cover -html=coverage.txt -o $(GOPATH)/tests/$(APP).html
	@rm coverage.txt

package::
	GOOS=linux GOARCH=amd64 go build -o ../../bin/$(APP).linux
	GOOS=darwin GOARCH=amd64 go build -o ../../bin/$(APP).mac

%::
		@:
.PHONY: test

Now any project that includes this file (as noted in the section above) will work with Make. For example

cd code/src/someProject
make
make run
make test

It’s important to note this setup automatically sets your GOPATH, you do not need to set it in your bash profile, and is 100% portable across folder locations and OS’s.

This Is Dumb; Why Would You Do This?

Speed and simplicity of management.

Consider working in a large environment where many people will write many different libraries. Every time you want to make a change to a library, you need to commit your code in some other repository, push that code, pull it back down to your Go project, compile, etc. If you have a bug in your new code, you’ll have to do it all over again. Making many small changes? Good luck.

With this setup, you commit the whole tree, and just focus on what you need to do: edit your code. Need to make a change to a dep? Then do it and recompile. The next time you push and merge, your changes will be available to every app within your organization.

As an added bonus, the next time Github is DDOS’d, your work flow won’t be disrupted, assuming you’re not using Github as your primary repository (you’re not, right?).

Enjoy.

How To Start An Elixir App In Production

Sat, 12 Sep 2015 12:56:29 +0000

Elixir has been gaining a lot of traction lately, particularly because of the immensely popular Phoenix web framework. Most of the users of Elixir tend to be smaller shops that have very simple deployment needs, and as such, documentation on large scale/automated deployment of Elixir apps is lacking.

Commands

Elixir programs are run by executing a few different commands at the shell, not too dissimilar from Java, Erlang, etc. Elixir has several commands that do a few different things. The frustrating bit is all of these commands cross over and call each other in obscure and confusing ways. You’ll see what I mean a bit later.

Below is an outline of the two commands we will be using. Note, these aren’t all the commands but only the commands we need to productionize and support Elixir.

elixir

This command evaluates a project file and runs it, but also acts as an interface of sorts to the underlaying Erlang VM. This is ultimately the command that will run Erlang, and is what you will use to launch Elixir in production (sort of, keep reading).

mix

This command also launches projects and code, but is also the compile tool and the main tool you use to launch your app in production.

Configuration

Elixir handles configuration in a way that is incredibly similar to Ruby/Rails. When starting your application, it will read from config/config.exs. At the bottom of that file, you should see something like so:

import_config "#{Mix.env}.exs"

This tells Elixir to include another config file, ie: prod, dev, test. You can specify which environment to load by setting the MIX_ENV environmental variable when starting your application. For example, setting:

MIX_ENV=prod

will load config/prod.exs.

Compiling

Compiling is a one liner:

mix compile

while in your project root. If you like to be explicit, you can do:

mix deps.compile
mix compile

Which will compile deps independently.

Running

Running an Elixir app is a bit tricky depending on your needs. If you’re running distributed nodes, you must run it like so:

MIX_ENV=prod elixir --name app@hostname --cookie "MyErlangCookie" -S mix run --no-compile --no-halt

A few things to note:

You should be using an init system that is event based. That is, use Upstart, systemd, runit, etc. Do not use sysvinit.
elixir sets Erlang parameters, and in this case we’re setting the node name and Erlang cookie.
Replace the app@hostname bit with something more suitable for your environment.
Passing -S mix run tells Elixir to start the app it self using mix. Since mix is the compile tool, we want to instruct mix not to compile our code in production, and to not halt once the code as started running.
The MIX_ENV can be omitted and instead put into some other facility of your init script, ie: Upstart env stanza, etc.

Using the above command, your Elixir app will launch cleanly into the foreground… unless you are using Phoenix! There is one last thing that you must do in order to make a Phoenix app work. Edit your config/prod.exs file and make sure the following is uncommented:

config :phoenix, :serve_endpoints, true

This will start Phoenix when you do mix run. Without the above line, Phoenix won’t be told to start up, resulting in all your code launching, except the web server it self.

That’s about it!

TL;DR

Compile like this:

mix deps.compile
mix compile

If using Phoenix, add this to the following to config/prod.exs

config :phoenix, :serve_endpoints, true

then start your app using Upstart, systemd, runit, and let it handle the foreground process like so

MIX_ENV=prod elixir --name app@hostname --cookie "MyErlangCookie" -S mix run --no-compile --no-halt

Enjoy!

Rendering Engine

Wed, 09 Sep 2015 09:01:02 +0000

Because every game needs a screenshot of their awesome rendering technology…

NodeMUD

Tue, 08 Sep 2015 22:51:50 +0000

I’ve long held a passion for MUD’s; ever since I was 12 years old or so I have been playing and writing code for MUD’s. I’ve spent a lot of time over the past two decades writing various small MUD code bases from the ground up in different languages, including C, C++, VB (I’m so sorry…), Perl, Ruby, et al.

However, after a long time, I’ve finally settled on a language and code base. I present, NodeMUD! NodeMUD is an entire MUD engine for NodeJS (at the time of this writing, 0.12.x should work), written in Coffeescript. Right now, my primary focus is content creation and speed; I want this engine to be fast enough for general purpose MUD’s that support hundreds (thousands?) of players at once. There are some core issues that need to be resolved (such as combat calculations happening on the main loop), but I suppose combat should be implemented before I worry too much about that!

I’ve taken care to write (mostly) functional code. State is only stored in entity objects; all other code, such as interpreters, should be functional. However it’s entirely possible I missed a thing or two.

Anyway, feel free to get the code here.

Moved.

Tue, 08 Sep 2015 22:03:25 +0000

I’ve decided to get off of tumblr for now; mostly it was due to a gripe with the post editor/creator. Ultimately I think I’ve landed on Jekyll. We’ll see how this goes.