Merge pull request #260 from heroku/access-logging

API User Access logging

Merge pull request #258 from heroku/handle-db-timeouts

Captures timeouts when looking up resources.

logplex_redis_writer: Fix base backoff config and make backoff linear (…

…#256)

I messed up the default config (it was set to 5000 seconds). Now it's 5 seconds and the backoff is linear.

Connect redis writers with exponential backoff (#255)

## Rationale
We have seen intermittent connectivity problems with the redis writers during logplex startup that did not automatically recover.

## Changes
- Monitor redis writers from logplex_queue right away
- Connect to log buffers using exponential backoff

Use https instead of plaintext git in rebar.config (#224)

* Use https instead of plaintext git in rebar.config

* Update rebar.config

* Update rebar.lock for new rebar.config

* revert to previous hashes

* test running travis again

* run travis

* missing git->https changes

Adjust redis writer restart intensity (#253)

## Rationale 

The redis writers connectivity appears unreliable.

## Changes
* extend redis writer restart intensity to 15000 restarts within 30 seconds 
* add some housekeeping for redis writers in logplex_queue
* don't let the writer exit normally on unexpected responses from a logplex_queue process

## Details
The prior restart intensity of the redis writer supervisor configuration did allow for 1000 restarts per 1 second. This is problematic for restarting redis writers, for example, when there are 100 shards and 10 writers per shard. On a network connectivity problem the number of restarts gets easily exceeded which forces a redis writer supervisor restart. After a supervisor restart all prior created connection information is lost. Without manual intervention the connection information is not automatically recovered.

The logplex_queue processes hold a list of workers for book keeping. This list doesn't have a function except for introspection. Without the change here this list becomes outdated as writer connections to redis go away.

A redis writer process would exit normally which prevents a automatic restart by the supervisor on unexpected errors when fetching messages from its logplex_queue process.

Connection pool for logs redis reads (#248)

We introduce a connection pool using poolboy that allows us to stabilize redis reconnect attempts using exponential backoffs.

* Remove useless function
* Add poolboy dependency
* Remove weird included file
* Implement logplex logs redis reader

Fix redis read reconnects (#247)

* Make reconnects as bad as before
* Increase reconnect timeout

Add env var for ENABLE_BATCH_REDIS (#246)

Implement batched redis logs writes (#245)

## Rationale

The current implementation of writes to redis buffers uses at least one redis command per log message. This PR optimizes for writing to redis buffers in batches which is supposed to reduce the load on the redis instances holding log buffers. 

## Changes

We have added a control rod `batch_redis`. When enabled the new code path is used. The control rod can be enabled with:
```
bin/control_rods enable_batch_redis
```
and disabled with:
```
bin/control_rods disable_batch_redis
```

The new code path is slightly modified from the old one with pre-processing of messages and redis `LPUSH` command execution as one batch of messages per channel. 

The tests have been designed in a way that they execute both paths with the same assertions. 

We introduced two new metrics:
* `message-batch.processed` which is a counter for the message batches for redis writes
* `message-batch.failed` which is a counter for failed message batch writes to redis