I often have long running tasks, for example, generating a report or performing an analysis of large amounts of data, that run minutes or even hours. Today I encountered a situation where I needed to start such task from a button in a browser window. Since the task takes three to five hours it was impossible to invoke the task directly.

My Internet research on this topic revelead a wide range of different solutions, from hacky to weird. The best I could find was the answer of Mark Biek to a question on Stack Overflow.

In PHP there are several functions to execute a command, for example, exec and shell_exec. However, there are several things to consider when a command should run in the background. But let's first take a look at the most simple example:

shell_exec(sprintf('%s > /dev/null 2>&1 &', $command));

There three important things in this snippet:

• The output (STDOUT) of the script must be directed to a file. In this case > /dev/null indicates that I'm not interested in the output.
• The errors (STDERR) also must be directed to a file. 2>&1 means that STDERR is redirected into STDOUT and therefore into the nirvana.
• The final & tells the command to execute in the background.

Of course it would also be possible to redirect STDOUT and STDERR to a file. In my situation this was not necessary since the command is part of an application with an internal log system.

Since the command will run multiple hours I'm also interested in the current status, that is, is the process still running. We can check if a process is running by using ps, for example:

ps \$PID

In order to be able to look this up, we need the PID of the process. In a shell the PID of the last process is stored in the variable $! and we can pass it to PHP by echoing it.

$pid = shell_exec(sprintf('%s > /dev/null 2>&1 & echo $!', $command));

It is now relatively easy to retrieve the status of the process. The following code is taken directly from the already mentioned Stack Overflow question.

function isRunning($pid)
{
    try {
        $result = shell_exec(sprintf('ps %d', $pid));
        if(count(preg_split("/\n/", $result)) > 2) {
            return true;
        }
    } catch(Exception $e) {}

    return false;
}

At this point I have everything I need and the only thing left to do is pack the code in a class with an easy-to-use interface:

namespace Bc\BackgroundProcess;

class BackgroundProcess
{
    private $command;
    private $pid;

    public function __construct($command)
    {
        $this->command = $command;
    }

    public function run($outputFile = '/dev/null')
    {
        $this->pid = shell_exec(sprintf(
            '%s > %s 2>&1 & echo $!',
            $this->command,
            $outputFile
        ));
    }

    public function isRunning()
    {
        try {
            $result = shell_exec(sprintf('ps %d', $this->pid));
            if(count(preg_split("/\n/", $result)) > 2) {
                return true;
            }
        } catch(Exception $e) {}

        return false;
    }

    public function getPid()
    {
        return $this->pid;
    }
}

It's now relatively easy to execute a command in a background process:

use Bc\BackgroundProcess\BackgroundProcess;

$process = new BackgroundProcess('sleep 5');
$process->run();

echo sprintf('Crunching numbers in process %d', $process->getPid());
while ($process->isRunning()) {
    echo '.';
    sleep(1);
}
echo "\nDone.\n"

In this example the command sleep 5 is executed in the background. As long as the process is running a dot is printed every second.

Please note: If the parent process continues to run while the child process(es) run(s) in the background you should use a more robust solution, for example, the Symfony Process component.

Update: I created a library using the code above. You can find it on Github: BcBackgroundProcess.

It is also possible to install the package through composer:

{ 'require': { 'braincrafted/background-process': 'dev-master' } }

Unlike most other programming and scripting languages PHP does not support threads (there is a third-party library called pthreads) and threads would be required to implement a truly asynchronous, non-blocking message queue. Challenge accepted.

In this article I am going to explain how you can implement a message queue in PHP. I do this mostly for fun and pleasure, because in most production scenarios you probably want to use a real message queue like RabbitMQ.

The message queue typically consists of a server that accepts messages. These messages are sent by a producer and are received by a consumer.

The server

The server is a program that runs in the background and listens to a specific port. Whenever a message arrives at the given port it invokes a consumer with the given message.

Running in the background, listening to a port, reacting when a message arrives. That sounds like a job for ReactPHP. In our case we only need the socket server of React and fortunately there is a subtree split of reacts socket component.

If you are using Composer (if not you really should) you can add react/socket to your project.

// composer.json
{
  "require": {
    "react/socket": "0.3.*"
  }
}

With React we can implement a simple server that listens to a socket and executes code whenever new data arrives at the socket.

// server.php

require_once __DIR__.'/vendor/autoload.php';

// Create loop and socket
$loop   = React\EventLoop\Factory::create();
$socket = new React\Socket\Server($loop);

// New connection is established
$socket->on('connection', function (\React\Socket\ConnectionInterface $conn) {
    // New data arrives at the socket
    $conn->on('data', function ($data) use ($conn) {
        // TODO: Handle message
        echo "$data";
        // Close the connection when we consumed the message
        $conn->close();
    });
});

// The socket should listen to port 4000
$socket->listen(4000);
$loop->run();

In the above script I added a simple echo to output the received data. We could now run php server.phpto start our server and send messages to it by connecting to it via telnet localhost 4000.

Let's assume that our message queue receives messages and needs to perform some time-consuming task with them. We will simulate this with the following code.

function consume($data, \React\Socket\ConnectionInterface $conn)
{
    for ($i = 0; $i < 5; $i++) {
        echo sprintf("%s: Do something with %s\n", date('H:i:s'), $data);
        sleep(1);
    }
}

When we run two telnet simultaneously we will receive the following output:

19:18:07: Do something with foo
19:18:08: Do something with foo
19:18:09: Do something with foo
19:18:10: Do something with foo
19:18:11: Do something with foo
19:18:12: Do something with bar
19:18:13: Do something with bar
19:18:14: Do something with bar
19:18:15: Do something with bar
19:18:16: Do something with bar

Client 1 has an open connection to the server for five seconds and client 2 has an open connection to the server for ten seconds. We want that the clients transfer a message to the server and immediately close the connection while the server accepts new messages and works on those tasks in the background.

Now is a good time to remember the last posting I wrote here: Running background processes in PHP.

We can use BcBackgroundProcess to create a background process and consume the message there.

// composer.json
{
    "require": {
        "react/socket":                     "0.3.*",
        "braincrafted/background-process":  "dev-master"
    }
}

When using the factory provided by BcBackgroundProcess it is quite easy to create a new process and execute it in the background.

// server.php
...
$processFactory = new \Bc\BackgroundProcess\Factory('\Bc\BackgroundProcess\BackgroundProcess');

$socket->on('connection', function (\React\Socket\ConnectionInterface $conn) use ($processFactory) {
    $conn->on('data', function ($data) use ($conn, $processFactory) {
        $command = sprintf('php consumer.php "%s"', addslashes($data));
        $processFactory->newProcess($command)->run();

        $conn->close();
    });
});
...

The command executed in the above code is php consumer.php with the message as first (and only) argument.

The consumer

The consumer is another script which is executed by the server as background process. Its first and only argument is the message. The biggest disadvantage of executing the code in a background process (compared to what we could do if PHP would support threads) is that we can no longer communicate with the server and thus can't output anything there. Instead we will write log messages to a file.

// consumer.php

function consume($message, $filename)
{
    for ($i = 0; $i < 5; $i++) {
        $data = sprintf("%s: Do something with %s\n", date('H:i:s'), $message);
        file_put_contents($filename, $data, FILE_APPEND);
        sleep(1);
    }
}

$message = stripslashes($_SERVER['argv'][1]);
consume($message, "message.log");

Ok, let's try this out. In my experiment I will open four Terminal windows in parallel. The first one will run the server, the second one will watch the message.log log (with tail -f) and the third and forth will be used to write to the message server.

You can see that both messages are consumed in parallel and that the client is started and closes within a second.

The producer

In reality you will probably never send messages to a MQ server using the telnet command line utility, but rather send messages from another script.

When you are using PHP it is extremely simple to write a message to a socket.

// produce.php

function produce($message)
{
    $fp = @stream_socket_client('tcp://localhost:4000', $errno, $errstr, 30);
    if ($fp) {
        fwrite($fp, $message);
        fclose($fp);
    }
}

produce("Hello World!");

Use Cases

As mentioned above this code is not really useful in a production environment. However, I use such a message queue to send real time notifications from a task that runs once a week for a few hours. The task is used only internal and not critical and therefore it would be a huge overkill to install and maintain a real message queue.

BcMq

I created a library from the code described in this article. The principles are the same, but it has a nicer architecture and is tested. You can find it on Github: BcMq.

BcMqBundle

If you want to use the code in a Symfony2 application I made things even easier by creating a bundle that encapsulates BcMq. The bundle uses services to consume messages which makes it quite easy and elegant. Detailed instructions on how to install and use the bundle can be found on Github: BcMqBundle.

Often it can be a little bit tricky to upload files to a web server and things become even more complicated when you want to store the uploaded files on Amazon S3. In this article I am going to explain how you can use Gaufrette to upload photos to S3 from a Symfony2 application.

I will use three different libraries in this article: Gaufrette, KnpGaufretteBundle and AWS SDK for PHP. Gaufrette is an abstraction layer for filesystems. That is, it offers a transparent interface to various types of filesystems, like a local filesystem, FTP, Dropbox, S3 and many others. The library is integrated into Symfony2 by KnpGaufretteBundle which makes it easy to setup and configure Gaufrette. Gaufrette requires the AWS SDK for communication with the S3 service.

Heads up: I have written this tutorial in 2013, since then some of the libraries had major updates, including AWS SDK.

Installation

Installation is easy if you use Composer, we have to add knplabs/gaufrette, knplabs/knp-gaufrette-bundle and amazonwebservices/aws-sdk-for-php to composer.json.

{
  "require": {
    "knplabs/gaufrette": "dev-master",
    "knplabs/knp-gaufrette-bundle": "dev-master",
    "amazonwebservices/aws-sdk-for-php": "dev-master"
  }
}

Using Composer's command line utitility we download the dependencies:

$ composer update

As a last step we need to add KnpGaufretteBundle to our applications app/AppKernel.php:

new Knp\Bundle\GaufretteBundle\KnpGaufretteBundle()

Setup Amazon S3

Before we can upload files to S3 we first have to setup S3 accordingly. This is possible in the AWS Management Console. We want to store all files uploaded from our application in a bucket we first have to create a new bucket for our application. After creating a new bucket we select the Properties tab in the top right. Since our files should be accessible for everyone we add a Bucket Policy. We select Permissions in the right view and click Add bucket policy. The following policy allows everyone to view the files in the bucket. You need to change the name of your bucket (I called mine braincrafted.com).

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Sid": "AddPerm",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::braincrafted.com/*"
    }
  ]
}

We also need to activate Static Website Hosting. To activate this we select Enable website hosting and enter the name of an index and an error document. The documents don't have to exist, but the fields are required.

You can test if your bucket works by using the AWS Console to upload an file to your bucket. The public URL of your bucket can also be found in the Static Website Hosting section of the Properties tab. It should look like this:

http://BUCKET_NAME.s3-website-us-east-1.amazonaws.com

If you have created your bucket in a different region the URL will be slightly different.

When you click your name in the very top you will find a link to Security Credentials. On this page you find your Access Keys and you will need those in a second.

Configure Gaufrette

We can now start by configuring Gaufrette. First we need to setup an adapter to communicate with S3 and then we need to setup a filesystem that uses the adapter performs actions on the filesystem.

Because Gaufrette has two levels we could use, for example, different adapters for different environments. If you have functional tests you probably don't want to upload files to S3 in those but rather use the local filesystem.

# app/config/config.yml

knp_gaufrette:
  adapters:
    photo_storage:
      amazon_s3:
        amazon_s3_id: acme_storage.amazon_s3
        bucket_name: %amazon_s3_bucket_name%
        create: false
        options:
          create: true
  filesystems:
    photo_storage:
      adapter: photo_storage
      alias: photo_storage_filesystem

The previous code creates one adapter photo_storage of the type amazon_s3 and a filesystem photo_storage that uses the photo_storage adapter. Before we continue lets look at the amazon_s3_id option. This option has to refer to a service of class AmazonS3.

We also need to define the bucket name and I like to define such options in my applications parameters.yml.

Configure AWS

I created a bundle for the code concerned with uploading and storing file and called it AcmeStorageBundle. Inside this bundle we need to create a new service for the AmazonS3 class.

# src/Acme/StorageBundle/Resources/config/services.xml

<parameters>
    <parameter key="acme_storage.amazon_s3.class">AmazonS3</parameter>
</parameters>

<services>
    <service id="acme_storage.amazon_s3" class="%acme_storage.amazon_s3.class%">
        <argument type="collection">
            <argument key="key">%acme_storage.amazon_s3.aws_key%</argument>
            <argument key="secret">%acme_storage.amazon_s3.aws_secret_key%</argument>
        </argument>
    </service>
</services>

There are two parameters defined, one for the key and one for the secret key, so we quickly have to define a configuration for AcmeStorageBundle. We also add an option to configure the base URL of the bucket.

# src/Acme/StorageBundle/DependencyInjection/Configuration.php

// ...
public function getConfigTreeBuilder()
{
    // ...
    $rootNode
        ->children()
            ->arrayNode('amazon_s3')
                ->children()
                    ->scalarNode('aws_key')->end()
                    ->scalarNode('aws_secret_key')->end()
                    ->scalarNode('base_url')->end()
                ->end()
            ->end()
        ->end();

    return $treeBuilder;
}

// ...

# src/Acme/StorageBundle/DependencyInjection/AcmeStorageExtension.php

// ...

public function load(array $configs, ContainerBuilder $container)
{
    // ...

    if (!isset($config['amazon_s3']['aws_key'])) {
        throw new \InvalidArgumentException(
            'The option "acme_storage.amazon_s3.aws_key" must be set.'
        );
    }
    $container->setParameter(
        'acme_storage.amazon_s3.aws_key',
        $config['amazon_s3']['aws_key']
    );

    if (!isset($config['amazon_s3']['aws_secret_key'])) {
        throw new \InvalidArgumentException(
            'The option "acme_storage.amazon_s3.aws_secret_key" must be set.'
        );
    }
    $container->setParameter(
        'acme_storage.amazon_s3.aws_secret_key',
        $config['amazon_s3']['aws_secret_key']
    );

    if (!isset($config['amazon_s3']['base_url'])) {
        throw new \InvalidArgumentException(
            'The option "acme_storage.amazon_s3.base_url" must be set.'
        );
    }
    $container->setParameter(
        'acme_storage.amazon_s3.base_url',
        $config['amazon_s3']['base_url']
    );
}

// ...

We can now configure these options in our applications configuration.

# app/config/config.yml

acme_storage:
  amazon_s3:
    aws_key: %amazon_aws_key%
    aws_secret_key: %amazon_aws_secret_key%
    base_url: %amazon_s3_base_url%

We should keep our AWS credentials secret and therefore define them in parameters.yml (which should not be committed to a VCS server).

# app/config/parameters.yml

parameters:
  amazon_aws_key: ~
  amazon_aws_secret_key: ~
  amazon_s3_base_url: ~
  amazon_s3_bucket_name: ~

In the parameters.yml file I also defined amazon_s3_bucket_name, which is used in the Gaufrette configuration to set the bucket name.

Implementing an uploader

Now we have everything configured and we can start writing code. Let's create a class PhotoUploader in AcmeStorageBundle to keep our controllers free of the details on how to upload files.

# src/Acme/StorageBundle/Upload/PhotoUploader.php

namespace Acme\Bundle\StorageBundle\Upload;

use Symfony\Component\HttpFoundation\File\UploadedFile;

use Gaufrette\Filesystem;

class PhotoUploader
{
    private static $allowedMimeTypes = array(
        'image/jpeg',
        'image/png',
        'image/gif'
    );

    private $filesystem;

    public function __construct(Filesystem $filesystem)
    {
        $this->filesystem = $filesystem;
    }

    public function upload(UploadedFile $file)
    {
        // Check if the file's mime type is in the list of allowed mime types.
        if (!in_array($file->getClientMimeType(), self::$allowedMimeTypes)) {
            throw new \InvalidArgumentException(sprintf('Files of type %s are not allowed.', $file->getClientMimeType()));
        }

        // Generate a unique filename based on the date and add file extension of the uploaded file
        $filename = sprintf('%s/%s/%s/%s.%s', date('Y'), date('m'), date('d'), uniqid(), $file->getClientOriginalExtension());

        $adapter = $this->filesystem->getAdapter();
        $adapter->setMetadata($filename, array('contentType' => $file->getClientMimeType()));
        $adapter->write($filename, file_get_contents($file->getPathname()));

        return $filename;
    }
}

The only thing that is a little bit tricky and took me a while to figure out is setting the content type. When we don't set the correct content type manually S3 will assume application/octet-stream and therefore will offer the file as download to the user. If we want to host files that a user can view in a browser (like images) we need to set the correct mime type. Sadly this is not possible on the Filesystem object. However, if you only upload files that should be downloaded you can directly work on the Filesystem object:

$this->filesystem->write($filename, file_get_contents($file->getPathname()));

We create a service for PhotoUploader to access the class in our controllers.

<parameters>
    <parameter key="acme_storage.photo_uploader.class">Acme\StorageBundle\Upload\PhotoUploader</parameter>
</parameters>

<services>
    <service id="acme_storage.photo_uploader" class="%acme_storage.photo_uploader.class%">
        <argument type="service" id="photo_storage_filesystem" />
    </service>
</services>

Setting up the controller and the form

We create an upload form and connect the form with our PhotoUploader class in the controller. In this example we will only create one form field for the uploaded file but it is easily possible to add other fiels (for example, a description).

# src/Acme/PhotoBundle/Form/Type/PhotoType.php

namespace Acme\PhotoBundle\Form\Type;

use Symfony\Component\Form\AbstractType;
use Symfony\Component\Form\FormBuilderInterface;

class PhotoType extends AbstractType
{
    public function buildForm(FormBuilderInterface $builder, array $options)
    {
        $builder->add('photo', 'file', array(
            'label' => 'Photo',
        ));
    }

    public function getName()
    {
        return 'photo';
    }
}

The only thing special in the controller code is the code to call PhotoUploader::upload(). It expects an UploadedFile object which is automically returned by the form.

# src/Acme/PhotoBundle/Controller/PhotoController.php

namespace Acme\PhotoBundle\Controller;

use Symfony\Bundle\FrameworkBundle\Controller\Controller;
use Symfony\Component\HttpFoundation\Request;

use Acme\PhotoBundle\Form\Type\PhotoType;

class PhotoController extends Controller
{
    public function addAction(Request $request)
    {
        $form = $this->createForm(new PhotoType(), array());

        if ($request->isMethod('POST')) {
            $form->bind($request);
            if ($form->isValid()) {
                $data = $form->getData();
                $url = $this->getPhotoUploader()->upload($data['photo']);

                return; // display a response or redirect
            }
        }

        return $this->render(
            'AcmePhotoBundle:Photo:add.html.twig',
            array('form'  => $form->createView())
        );
    }

    /**
     * @return Acme\StorageBundle\Uploader\PhotoUploader
     */
    protected function getPhotoUploader()
    {
        return $this->get('acme_storage.photo_uploader');
    }
}

If we now want to get the full URL to the uploaded file we can use the parameter we defined earlier in the configuration of AcmeStorageBundle:

$url = sprintf(
    '%s/%s',
    $this->container->getParameter('acme_storage.amazon_s3.base_url'),
    $this->getPhotoUploader()->upload($data['photo'])
);

The only thing left to do is create a route for the action and probably add some logic to store the URL returned by the upload() method in a database. However, that is not really the topic of this article and is explained elsewhere.

Whenever writing functional tests I think it is a good idea to use translation keys in the assertions instead of the real text. The translation keys are less likely to change over time and it is often much simpler, because they don't contain dynamic content.

However, Symfony 2 automatically translates all text even in the test environment and I could not find an easy solution on how to deactivate translation in the test environment.

The simplest way to disable translation in functional tests is to create a new translator that returns the message ID instead of the translated message.

# src/Acme/DemoBundle/Translator/NoTranslator.php

namespace Acme\DemoBundle\Translator;

use Symfony\Component\Translation\TranslatorInterface;

class NoTranslator implements TranslatorInterface
{
    public function trans($id, array $parameters = array(),
        $domain = null, $locale = null)
    {
        return $id;
    }

    public function transChoice($id, $number, array $parameters = array(),
        $domain = null, $locale = null)
    {
        return $id;
    }

    public function setLocale($locale)
    {
    }

    public function getLocale()
    {
        return '--';
    }

    public function setFallbackLocale($locale)
    {
    }

    public function addResource($resource)
    {
    }
}

Now we need a way to switch the default translator to our NoTranslator. The easiest way to do this is override the translator.class variable, however, this changes the translation for all environments and I want to change the translator only in the test environment.

Another way to override a service is to write a compiler pass.

# src/Acme/DemoBundle/DependencyInjection/Compiler/TranslatorCompilerPass.php

namespace Acme\DemoBundle\DependencyInjection\Compiler;

use Symfony\Component\DependencyInjection\Compiler\CompilerPassInterface;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\HttpKernel\KernelInterface;

class TranslatorCompilerPass implements CompilerPassInterface
{
    public function process(ContainerBuilder $container)
    {
        $definition = $container->getDefinition('translator.default');
        $definition->setClass('Acme\DemoBundle\Translator\NoTranslator');
    }
}

This would, again, override the translator in all environments. We need an instance of AppKernel to check the environment. At this stage of the boot process of the Symfony 2 kernel we don't have access to the kernel via the dependency injection container. The only place where we have access to the kernel at that early stage is the AppKernel itself and therefore we need to pass it through to our compiler from there.

# app/AppKernel.php

// ...

class AppKernel extends Kernel
{
    public function registerBundles()
    {
        // ...

        if (in_array($this->getEnvironment(), array('dev', 'test'))) {
            $bundles[] = new Acme\DemoBundle\AcmeDemoBundle($this);
            // ...
        }

        // ...
    }
    // ...
}

This is absolutely ok to do. You would have to add your bundle anyways in the AppKernel.php and this is also done by one of the default bundles (JMSDiExtraBundle). Now we passed the kernel to AcmeDemoBundle. There we need to initialise the compiler pass with the kernel.

# src/Acme/DemoBundle/AcmeDemoBundle.php

namespace Acme\DemoBundle;

use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\HttpKernel\Bundle\Bundle;
use Symfony\Component\HttpKernel\KernelInterface;

use Acme\DemoBundle\DependencyInjection\Compiler\TranslatorCompilerPass;

class AcmeDemoBundle extends Bundle
{
    private $kernel;

    public function __construct(KernelInterface $kernel)
    {
        $this->kernel = $kernel;
    }

    public function build(ContainerBuilder $container)
    {
        parent::build($container);

        $container->addCompilerPass(
            new TranslatorCompilerPass($this->kernel)
        );
    }
}

The last thing we have to do is to adapt TranslatorCompilerPass to accept the kernel and check the environment.

# src/Acme/DemoBundle/DependencyInjection/Compiler/TranslatorCompilerPass.php

namespace Acme\TestingBundle\DependencyInjection\Compiler;

use Symfony\Component\DependencyInjection\Compiler\CompilerPassInterface;
use Symfony\Component\DependencyInjection\ContainerBuilder;
use Symfony\Component\HttpKernel\KernelInterface;

class TranslatorCompilerPass implements CompilerPassInterface
{
    private $kernel;

    public function __construct(KernelInterface $kernel)
    {
        $this->kernel = $kernel;
    }

    public function process(ContainerBuilder $container)
    {
        if ('test' === $this->kernel->getEnvironment()) {
            $definition = $container->getDefinition('translator.default');
            $definition->setClass('Acme\DemoBundle\Translator\NoTranslator');
        }
    }
}

Done. We can now write our assertions in functional tests for translation keys instead of translated messages.

I added the code required to disable translation to my BraincraftedTestingBundle, but I wanted to explain what I did. If there is a more convenient solution please let me know.

Update March 19, 2014: Reader Gilles Doge emailed me to add that you can override the translator.class variable only in the test environment:

# app/config/config_test.yml
parameters:
  translator.class: Acme\DemoBundle\Translation\Translator\NoTranslator

In this case you don't need to create an additional compiler pass. However, my solution has the advantage that you can place it in my TestingBundle and have this functionality automatically in all my projects.

Symfony2 Cookbook: How to Override any Part of a Bundle - Services & Configuration Symfony2 Cookbook: How to work with Compiler Passes in Bundles

This blog uses Jekyll to build the HTML code, Compass for the stylesheets and JavaScript is concatenated and minified. Lately I have written more articles and every time I changed the layout or the text I had to manually recompiled using my Makefile. Every time those extra keystrokes hurt me on the inside.

Jekyll and Compass each have a watch command built in and there are tools for the JavaScript part, but I don't want to start three different watchers every time I work on braincrafted.com. Another problem is that after Compass build the CSS files, Jekyll needs to run to put CSS files into the correct directory.

There are numerous tools and libraries available to watch for files changes, but I decided to use Guard pretty quickly because it fulfils all of my requirements:

1. Exclude files and directories
2. Define multiple groups of files
3. Execute a Makefile

Requirement number 1 is important because the builders will place the generated HTML, CSS and JavaScript files in a subdirectory of the watched directory. The second requirement allows me to only execute the builders necessary for a specific file. For example, if a HTML file is changed I only want to run Jekyll. Executing Makefiles allows me to reuse my existing build scripts and saves me some work.

Installation of Guard was pretty easy, at least after I reinstalled my messed up Ruby installation and the syntax for the Guardfile is straight forward.

# ./Guardfile

# Notification via Growl when files change
notification :growl

# Ignore the following diretories and files
ignore %r{^_site/}, %r{/Guardfile/}, %r{/config.rb/}, %r{/code/}, %r{/css(-dev)?/}, %r{/js/}

guard :shell do

    # When HTML and Markdown files are modified run Jekyll
    watch(/^.*\.(md|htm|html|xml)/) do |m|
        `make build-jekyll run-dev`
        n "Built HTML for braincrafted.com"
        m[0]
    end

    # When JavaScript is modified build JS and Jekyll
    watch(/^js-dev\/.*\.js/) do |m|
        `make build-js build-jekyll run-dev`
        n "Built HTML and JavaScript for braincrafted.com"
        m[0]
    end

    # When Sass is modified build CSS and Jekyll
    watch(/^sass\/(.*)\.s[ac]ss/) do |m|
        `make build-css build-jekyll run-dev`
        n "Built HTML and CSS for braincrafted.com"
        m[0]
    end

end

I added a target to my Makefile to make things even more intuitive.

# ./Makefile

watch:
    @echo "Watching for changes..."
    @bundle exec guard

Since I was already at it, I also wanted Safari to automatically reload the page (or open it in a new tab if it is not already open). I inlined some lines of AppleScript in my Makefile to accomplish this.

# ./Makefile

DEV_URL         = http:\/\/braincrafted.com.dev
PROD_URL        = http:\/\/braincrafted.com

define OPEN_SCRIPT
tell application "Safari"
    set alreadyOpen to false
    set reloadURL to "__OPEN_URL__"

    repeat with x from 1 to number of windows
        repeat with y from 1 to number of tabs in window x
            set tabURL to URL of tab y of window x
            if tabURL starts with reloadURL then
                set URL of tab y of window x to tabURL
                set alreadyOpen to true
                set current tab of window x to (tab y of window x)
            end if
        end repeat
    end repeat

    if alreadyOpen is false then
        tell front window
            set current tab to (make new tab with properties {URL:reloadURL})
        end tell
    end if

end tell
endef

# Required because by default $OPEN_SCRIPT is interpretated as a command
export OPEN_SCRIPT

run-dev:
    @echo "Open site in development environment... \c"
    @echo "$$OPEN_SCRIPT" | sed "s/__OPEN_URL__/${DEV_URL}/" | osascript
    @echo "${CHECK} Done"

run:
    @echo "Open site in production environment...  \c"
    @echo "$$OPEN_SCRIPT" | sed "s/__OPEN_URL__/${PROD_URL}/" | osascript
    @echo "${CHECK} Done"

When I now change something in the project, the updated page is already loaded when I switch to Safari.

Resources

• Guard README
• Guard::Shell
• Generate a List of Open Safari Tabs with AppleScript
• Reload Safari tabs in the background via AppleScript
• AppleScript To open a collection of Safari tabsphp?story=20110415065236737
• Is it possible to create a multi-line string variable in a Makefile

I spent the last hour trying to set the easiest and most reliable way to create a keyboard shortcut for Spotifys Add to Queue functionality on OS X. Since I found no solution for this I decided to post it here.

Update June 14, 2013: In the last update Spotify moved Queue in the context menu from the second position to the 6th. I updated the article accordingly (but not the screenshot).

There is no menu bar entry for this functionality in Spotify and therefore no way to use an ApplyScript. I had to simulate mouse clicks to get this done. There exists a range of different tools to set keyboard shortcuts, but I personally prefer BetterTouchToul from Andreas Hegenberg.

In the main BetterTouchTool window you first need to add Spotify to the list of applications. Just press the small plus button in the bottom of the list. I created two shortcuts. One for adding all tracks in the current view to the queue and another one for adding all currently selected tracks to the queue.

Let's first create the shortcut for adding all tracks in the current view to the queue. Manually we would first press Cmd+A to select all tracks, then click the right mouse button and select the second entry from the context menu. We need to create a new shortcut by pressing the Add New Shortcut button in the lower right corner. Define your favourite shortcut to it (I mapped the CAPSLOCK key to Crtl+Option+Shift+Command and used CL+Q) and set Trigger Other Keyboard Shortcut to Cmd+A. Click Attach Another Action and set Trigger Predefined Action to Rightclick. Next we need to attach two six more actions that both all should trigger the Down key. The last action we need to attach is the Enter key.

The shortcut to add only the selected songs to the queue is very similar, just set the shortcut to trigger Right-click instead of Cmd+A.

Please note that the mouse has to be hovered over the selected songs when triggered the shortcut.

A useful Caps Lock key - BrettTerpstra.com

Two years ago I have written a library for a project to generate strings based on a given pattern called ExpExp. ExpExp takes a pattern (a subset of regular expressions) and returns all strings that would match the pattern. For example, the pattern foo(bar|baz) would expand to foobar and foobaz.

Last week I found the library again and I decided, mostly for fun and pleasure, to add new features and to rewrite the whole code. I also moved to code to the Braincrafted namespace. You can find it on Github.

While my goal is to support as many patterns supported by regular expressions as possible, it will ever be only a subset. For example, the star operator in regular expressions matches an infinite amount of infinite long strings. ExpExp can never support such things. Let's stop talking about what's not possible, but about the features that work right now.

Features

ExpExp currently supports the following operators:

• Parentheses
• Disjunction
• Repetition
• Alternation
• Character classes
• Dot operator
• Optional operator

I am now going to discuss these operators in more detail.

Parentheses

Disjunction

Repetition

Alternation

Character classes

Other character classes are

There are also multiple shortcuts available:

Dot operator

The dot operator is replaced by all characters from the word character class.

Optional operator

The optional operator allows you to make a character or the content inside a parentheses optional.

Combination

Of course you can combine all of these operators. For example,

ab(cde|[xyz])

will expand to

• abcde
• abx
• aby
• abz

Use Cases

I don't think there are many use cases in real life for this library, but it was fun and challenging project.

Installation

If you want to play around with ExpExp or find a use cases in one of your project you can use Composer to install it:

{
  "require": {
    "braincrafted/expexp": "dev-master"
  }
}

I already have tagged some versions, so you can use, for example, 0.2.* to get a (relative) stable release.

Usage

Pass the pattern to the expand() method and ExpExp will return an array with all expansions:

use Bc\ExpExp\ExpExp;

$e = new ExpExp();
$result = $e->expand('abc|xyz');

That's it.

First I will assume that we have a fresh installation of Symfony Framwork standard edition and that you have Bower installed. If you don't, do it now:

$ npm install -g bower
$ composer create-project symfony/framework-standard-edition symfony-bower-bootstrap/ 2.4.5
$ cd symfony-bower-bootstrap

We can now add BraincraftedBootstrapBundle to the project and use Bower to download the Bootstrap assets:

$ composer require braincrafted/bootstrap-bundle:@stable
$ bower init
$ bower install bootstrap --save

Add the bundle class to app/AppKernel.php

// app/AppKernel.php

class AppKernel extends Kernel
{
    public function registerBundles()
    {
        $bundles = array(
            // ...
            new Braincrafted\Bundle\BootstrapBundle\BraincraftedBootstrapBundle(),
        );
        // ...
    }
}

In this article I am going to use Assetic to configure and compile the Bootstrap assets. BootstrapBundle comes with a handy auto configure functionality that configures Assetic for you. However, this functionality only works if you install Bootstrap using Composer and therefore we need to disable it.

# app/config/config.yml

braincrafted_bootstrap:
  auto_configure:
    assetic: false

The current stable release of BootstrapBundle supports only the Less version of Bootstrap, but BootstrapBundle v2.1 adds support for the official Sass port of Bootstrap. In this article I am going to use the Less version, but it works pretty similar if for bootstrap-sass.

# app/config/config.yml

assetic:
    filters:
        less:
            node: /usr/local/bin/node
            node_paths: [/usr/local/lib/node_modules]
            apply_to: "\.less$"
        cssrewrite: ~

We also need to tell Assetic where it finds the assets. By default Bower installs assets in the bower_components directory. If you have modified the location you also need to change it in the following snippet:

# app/config/config.yml

assetic:
    ...
    assets:
        bootstrap_css:
            inputs:
                - %kernel.root_dir%/../bower_components/bootstrap/less/bootstrap.less
                - %kernel.root_dir%/../vendor/braincrafted/bootstrap-bundle/Braincrafted/Bundle/BootstrapBundle/Resources/less/form.less
            filters:
                - less
                - cssrewrite
            output: css/bootstrap.css
        bootstrap_js:
            inputs:
                - %kernel.root_dir%/../bower_components/bootstrap/js/transition.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/alert.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/button.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/carousel.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/collapse.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/dropdown.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/modal.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/tooltip.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/popover.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/scrollspy.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/tab.js
                - %kernel.root_dir%/../bower_components/bootstrap/js/affix.js
                - %kernel.root_dir%/../vendor/braincrafted/bootstrap-bundle/Braincrafted/Bundle/BootstrapBundle/Resources/js/bc-bootstrap-collection.js
            output: js/bootstrap.js
        jquery:
            inputs:
                - %kernel.root_dir%/../bower_components/jquery/dist/jquery.js
            output: js/jquery.js

Note that BootstrapBundle includes a less and a js file to provide additional fixes and functionality when using Bootstrap in a Symfony application. You need to include those from the installed bundle.

If you now dump the assets you should three files: css/bootstrap.css, js/bootstrap.js and js/jquery.js.

$ php app/console assetic:dump

Note that Assetic will also create the intermediate files when run in the dev environment.

You can now create a basic layout and include the Bootstrap CSS and JavaScript assets and start building your application.

<!-- src/Acme/DemoBundle/Resources/views/layout.html.twig -->

<!DOCTYPE html>
<html>
<head>

    <title>Bootstrap 101 Template</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <!-- Bootstrap -->
    <link href="{{ asset(&#39;css/bootstrap.css&#39;) }}" rel="stylesheet" media="screen">

    <!-- HTML5 Shim and Respond.js add IE8 support of HTML5 elements and media queries -->
    {% include &#39;BraincraftedBootstrapBundle::ie8-support.html.twig&#39; %}

</head>

<body>
    <div class="container">
        <div class="page-header">
            <h1>Hello, world!</h1>
        </div>
    </div>

    <!-- jQuery (necessary for Bootstraps JavaScript plugins) -->
    <script src="{{ asset(&#39;js/jquery.js&#39;) }}"></script>
    <!-- Include all JavaScripts, compiled by Assetic -->
    <script src="{{ asset(&#39;js/bootstrap.js&#39;) }}"></script>
</body>
</html>

_Please note that the BootstrapBundle includes a basic layout template you can use. However, there has been a bug in v2.0._that causedbootstrap.jsnot to load. We fixed the bug inv2.1.0-alpha1.*

In addition to this article I also created a demo project where you can follow the steps in this article in a full Symfony project. The source code of this demo project resides on Github: florianeckerstorfer/symfony-bower-bootstrap.

I really enjoyed reading the book and I think most of the principles do not only apply to movie companies but to all companies that employ creative people. That includes software companies. It also made me want to watch Pixar movies. Last weekend I already saw Toy Story 3 and Brave. Seven more movies to go.

• Toy Story
• A Bug's Life
• Toy Story 2
• Monsters, Inc.
• Finding Nemo
• The Incredibles
• Cars
• Ratatouille
• WALL-E
• Up
• Toy Story 3
• Cars 2
• Brave
• Monsters University

gm convert -resize 150x100 in.jpg out.jpg

This will resize the image in.jpg to 150x100 and saves the result in out.jpg. You can resize the image by either providing the width or the height:

gm convert -resize 150 in.jpg out.jpg
gm convert -resize x100 in.jpg out.jpg

Note: While I don't intend to write an exhaustive guide to Swift the article in its current form still misses some crucial pieces.

Table of Contents

1. Getting Started
2. Constants & Variables
3. Arrays
4. Collections
5. Loops
6. Conditional Statements
7. Optional Variables
8. Updates

Getting Started

To get started with Swift you need the new Xcode 6 Beta. You can download it from Apples Developer area after you have logged in with your developer account. You need enroll either in the iOS or Mac developer program for $100 per year.

Open Xcode and create a new file (File > New > File) and select the Playground template in iOS > Source:

Next you need to select a folder where to save your playground file and enter a filename. If you have done that you should see a new window where you can start to write some Swift code.

Constants & Variables

Constants and variables can be used to hold a value. The difference between constants and variables is that constants have a constant value, once you assign a value to a constant you cannot change that value. Variables on the other hand have a variable value, you can change the value of a variable any time you want. You can define constants with the let keyword and variables using the var keyword.

// Define a constant
let myFirstConstant = "Hello Constant"

// Define a variable
var myFirstVariable = "Hello Variable"

// Change the value of the variable
myFirstVariable = "Another value"

// This will raise an error
// myFirstConstant = "Another value"

Note: Maybe you have noticed that the semicolon at the end of each line is missing. In Swift you don't need to add a semicolon at the end of a statement if each statement is on its own line. If you want two write two statements on a single line you have to separate them with a semicolon.

let one = 1; var two = 2

Now that we have defined a variable and a constant we need to learn a little bit about types. Swift is a strongly typed language, each variable and constant needs to have a type. It supports all major types like strings, integers, floats, booleans and so own. In a strongly typed language you normally have to define the type everytime you define a variable or constant. As you have seen in the previous example we didn't define a type for myFirstConstant and myFirstVariable. We can omit the type when Swift can infer the type. In the example above Swift was able to infer that the type of myFirstConstant and myFirstVariable is String because we set the value immediately to a string value. However, we could have also provided the type explicetely.

let myFirstConstant: String = "Hello Constant"
var myFirstVariable: String = "Hello Variable"

In this example we can omit the type because Swift can infer it based on the value. If you define a variable and don't immediately assign a value you have to define the type.

var one: Int
var two: Int

Here are a few more example with other types supported by Swift.

var anInteger: Int = 1
var anBoolean: Bool = true
var aString: String = "Hello"
var aDouble: Double = 4.2
var aFloat: Float = 9.9

In general variables also have to have a value. However, Swifts has the concept of Optional Variables that I will cover in a later chapter.

Arrays

Array is a type that can contain multiple values of the same type. For example, if you want to create a list of the houses in Game of Thrones.

var houses = [ "Stark", "Lannister", "Baratheon", "Greyjoy" ]

Swift can infer the type of houses based on the value. We could also explicetly state the type of the array.

var houses: String[] = [ "Stark", "Lannister", "Baratheon", "Greyjoy" ]

Of course we can also create an array of integers.

var favNumbers = [ 3, 7, 42, 69 ]

We can access the element of an array by using the index syntax. Like in most other programming languages the first index of an array is 0.

houses[0]
// "Stark"

houses[1]
// "Lannister"

Unlike in more low-level languages like Objective C or C types in Swift have methods and operations. For example, we can count the elements in an array

houses.count
// 4

Arrays are ordered, therefore each element has an index (must be unique) and there are no gaps in the list of indeces. If you have an array with 5 elements and you remove the element with index 2, the element at index 3 will be moved to index 2 and the element at index 4 will be moved to index 3.

Collections

Collections are somewhat similar to arrays, but while arrays are ordered (each element has an index and the array is orderd by this index) collections are unordered. The index of an collection does not have to be an integer, but can be a string or a double.

var lords = [ "stark": "ned", "lannister": "tywin", "baratheon": "robert" ]

Just like arrays you can access elements in a collections using their index.

lords["stark"]
// "ned"

Loops

One of the most common things you want to do with an array or a collection is that you want to step through each element. Swift offers a number of different types of loops. The probably easiest one to understand is the for-in loop.

for house in houses {
    house
}

Note: If you type this code in a playground in Xcode you will see the number of times the body of the loop is executed. In the example above this would be four times. You can click on the circle icon right to this to view the value of the variable house in each iteration.

If you don't want to iterate through an array or collection but rather iterate to a range of values you can use the range operator. In fact, in Swift there are two range operators. The closed range operator is a...b and returns all numbers between a and b including a and b. The half-closed range operator is a..b and returns the numbers between a and b including a but excluding b.

for i in 1...5 {
    i
}
// 1
// 2
// 3
// 4
// 5

for j in 1..5 {
    j
}
// 1
// 2
// 3
// 4

Swift also supports "normal" for loops, while and do while loops.

Note: If you have already experience in another programming language you might be wondering if I forgot the parentheses in the for statement. I didn't. In Swift you don't use parenthesis when writing Control Flow (for, while,if,switch, …) statements.

Conditional Statements

Conditional statements allow you to execute only when a certain condition is met. The conditional is often also called an expression and we say that an expression is true or is false or is evaluated to true or is evaluated to false.

let num = 5
if num == 5 {
    "Number is 5"
}
// "Number is 5"

Swift supports all major operators known from other programming languages like ==, !=, >, <, >=, <= and so on.

Often you need to execute code when an expression is valid and another piece of code when the expression is not valid. Like any other programming language, Swift has an else statement.

let num = 10
if num == 5 {
    "Number is 5"
} else {
    "Number is not 5"
}
// "Number is not 5"

In the above example this is the same as if you would write two if statements, one with a positive expression, one with the negated expression.

let num = 10
if num == 5 {
    "Number is 5"
}
if num != 5 {
    "Number is not 5"
}
// "Number is not 5"

However this will not always work. For example, consider the following piece of code:

let num = 10
if num > 5 {
    "Number is greater than 5"
}
if num > 3 {
    "Number is grater than 3"
}
// "Number is greater than 5"
// "Number is greater than 3"

If you only want to output one of the two lines you either have to adapt the second conditional to num > 3 && num <= 5 or you need to use else. Using else is a lot more elegant and readable.

let num = 10
if num > 5 {
    "Number is greater than 5"
} else {
    "Number is grater than 3"
}
// "Number is greater than 5"

You can also combine the if and else statements to implement multiple conditionals.

let num = 42
if num == 5 {
    "Number is 42"
} else if num == 42 {
    "You've found the answer"
} else {
    "Number is not 5 and not the answer"
}
// "You've found the answer"

Optional Variables

As mentioned in the Constants & Variables chapter Swift requires you to always set a value for a variable for you can read it. In other programming languages variables have a default value (for example, null in PHP or undefined in JavaScript) when you use a variable before it has a value assigned. When you execute the following code snippet you will get a compiler error.

var foo: Int

if foo == 5 {
    "The value is 5"
}
// Compiler error: "error: variable 'foo' used before being initialized"

This behaviour helps you catch a lot of errors where you forgot to assign a value or assigned a value to the wrong variable. However, there exist situations where you cannot assign a value to variable because we don't know the value yet. For example, when retrieving data using an HTTP request we may not have a value when the request fails. For such cases you can specifically define a variable as optional and assign nil to the variable. If you explicitely state that a variable is optional it is your responsibility to check whether a value has been asigned before using it. You can define an optional variable by adding a question mark ? to the type definition.

var foo: Int?

if foo {
    "Foo is initialized"
} else {
    "Foo is not initialized"
}
// "Foo is not initialized"

In this example the conditional evaluates to true if the variable foo has a value assigned (is not nil) and to false otherwise (is nil). You can also assign nil later to a variable.

var foo: Int?
foo = 5

if foo {
    "Foo is initialized"
} else {
    "Foo is not initialized"
}

foo = nil

if foo {
    "Foo is initialized"
} else {
    "Foo is not initialized"
}
// "Foo is initialized"
// "Foo is not initialized"

It's important to note that only the variable name (foo in the examples above) is only a valid LogicValue that can be used in a conditional if the variable is optional. The following example will throw a compiler error:

var bar: Int = 5
if bar {
    "Bar is initialized"
}
// Compiler error: "type 'Int' does not conform to protocol 'LogicValue'"

It is a common scenario that you have a variable with an optional value and after you check that it has a value it should always have a value. You can use var foo = optionalFoo inside the conditional to assign the optional value to a non-optional value.

var foo: Int? = 5

if var bar = foo {
    bar = 7
    bar = nil
}
// Compiler error: "could not find an overload for '__conversion' that accepts the supplied arguments"

The first assignment bar = 7 goes through, however, the compiler will throw an error when you want to assign nil to bar.

It is worth noting that constants can also have an optional type.

let foo: Int?

if foo {
    "init"
} else {
    "not init"
}

Please note that this article can be considered as a working draft. At this point it includes some of the basic concepts of the Swift programming language, but there exists a lot of stuff that is not covered. If you are eager to learn more please read The Swift Programming Language (iBook Store) from Apple. You can read the book also online. If you want to try things out and don't want to type every example you can download the Swift Tour as a playground file.

Updates

• June 8, 2014
  • Extended and rewritten section on Conditional Statements
  • Added Table of Contents
• June 15, 2014
  • Added chapter on Optional Variables

I am already looking forward to my trips to Stockholm and Hamburg in July.

I use Gulp not only for building my assets but also to run my tests, code coverage and checkstyle and I will tackle all these aspects in these article.

Table of Contents

• Dependencies
• Project Structure
• Gulp
• Stylesheets
  • Stylesheets with Sass
  • Managing Assets with Bower
  • Using Bootstrap
  • Building Stylesheets
  • Gylphicons
• JavaScript
  • Load JavaScript using RequireJS
  • Building JavaScript
• Watching & Reloading
  • Watching
  • Reloading
• Running PHP Commands
  • PHPUnit
  • PHP_CodeSniffer
  • Symfony2 Commands
• Roadmap
• Conclusion
• Updates

Dependencies

First I need to talk about the software and libraries I am using in my project and that are relevant to this article.

• Gulp to run tasks (along with a bunch of Gulp plugins and other Node.js modules)
• Bower to install and manage assets
• Sass
• RequireJS to dynamically load JavaScript files
• Bootstrap as a frontend framework
• jQuery because I am familiar with it and Bootstrap requires it anyway
• PHPUnit to run my tests and code coverage
• PHP_CodeSniffer to check the style of my PHP code

Please note that this article has been updated to reflect the changes in the directory structure in boostrap-sass v3.2

Project Structure

Every Symfony2 project (that uses the standard edition) has the same basic structure. However, since it is a backend framework it gives you little guidance on how to organise assets. Per convention Symfonys assets:install command will copy everything from a bundles Resources/public directory into the web/ folder of the project. I am using this convention to make dealing with assets path in my Sass, JavaScript and Gulp files easier. Here are the parts of my project structure that are relevant to this article:

- src/Bundle/
    - AcmeDemo/
        - Resources/public/
            - js/
            - sass/
    - AcmeFrontendBundle/
        - Resources/public/
            - js/
            - sass/
    - AcmeUserBundle/
        - Resources/public
            - js/
            - sass/
web/
    - bundles/
    - components/
    - css/
    - fonts/
    - js/
Gulpfile.js

If I run the assets:install --symlink command, Symfony will create symlinks from the web/bundles/ directory to the public directory in the corresponding bundle for me. With my structure the web/bundles/ directory looks like this:

- web/bundles/
    - acmedemo/
    - acmefrontend/
    - acmeuser/

I think it is obvious that the web/css/ directory will hold the compiled CSS files, web/fonts/ the font files and web/js/ the JavaScript files. The web/components/ directory stores the assets downloaded by Bower and I will take little bit more about it later.

Gulp

If you have never worked with Gulp and need to know how to install it and learn more about the basic concepts you can either head to the Getting Started guide or the Building with Gulp article on Smashing Magazine. In short, you can install Gulp using NPM.

You need to install Gulp both globally and locally in your project:

$ npm install -g gulp
$ npm install --save-dev gulp

Now you can create a Gulpfile.js and require the gulp module:

var gulp = require('gulp');

gulp.task('default', function () {});

Stylesheets

First I want to talk about managing and building the stylesheets for my project.

Stylesheets with Sass

Sass has great features that make it superior to plain CSS and one of them is the ability to define variables. However, if I compile the different Sass files separately and concatenate them later I can't reference variables from different source files. Because I want a single Sass file anyway I use the import statement to include them into a single master file. I place a master.scss in every bundle and import every .scss file of the bundle. The master.scss of the AcmeFrontendBundle includes the master.scss from every other bundle. It looks like this:

// src/Acme/Bundle/FrontendBundle/Resources/public/sass/master.scss

@import '../../acmeuser/sass/master';
@import '../../acmeother/sass/master';

Managing Assets with Bower

Before I can talk about how I use Bootstrap I need to talk about Bower. The first thing with Bower I did was changing the default download directory by creating a .bowerrc file.

{
  "directory": "web/components"
}

Then I downloaded bootstrap-sass-official (as the name says the official Sass port of Bootstrap) using bower install --save bootstrap-sass-official.

Using Bootstrap

I use my BraincraftedBootstrapBundle to integrate Bootstrap into Symfony. However, I disabled the auto configuration feature for Assetic because I want to configure it myself using Bower and Gulp and therefore it's not relevant if you use the bundle or not. Bower downloads the Sass files into components/bootstrap-sass-official/vendor/assets/stylesheets/bootstrap/ and I just import it from my master.scss in AcmeFrontendBundle. Remember that the assets:install commands copies the files into web/bundles/ and the compilation takes place there.

// src/Acme/Bundle/FrontendBundle/Resources/public/sass/master.scss

@import '../../../components/bootstrap-sass-official/assets/stylesheets/_bootstrap';

@import '../../acmeuser/sass/master';
@import '../../acmeother/sass/master';

Building Stylesheets

Finally I have reached a point where we can talk about building the stylesheets, that is, compiling Sass into CSS code. To actually compile Sass I use gulp-sass and the Node.js port of Sass (it's faster).

$ npm install --save-dev gulp-sass

Because of the way how I import everything I need into my master Sass file the code for the Gulp task remains small and simple.

// Gulpfile.js

var sass = (sass = require('gulp-sass'));

gulp.task('sass', function () {
  gulp
    .src('./web/bundles/acmefrontend/sass/master.scss')
    .pipe(sass({ sourceComments: 'map' }))
    .pipe(gulp.dest('./web/css/'));
});

First I need to symlink the assets into web/bundles/ by running php app/console assets:install --symlink. Executing the task with gulp sass will compile the stylesheets from all bundles and Bootstrap and saves them in web/css/master.css. In my project I have a layout template in my frontend bundle where I can now include this stylesheet.

<!-- src/Acme/Bundle/FrontendBundle/Resources/views/layout.html.twig -->

<link href="{{ asset(&#39;/css/master.css&#39;) }}" rel="stylesheet">

Everything regarding stylesheets should work now; except the Glyphicons provided by Bootstrap.

Gylphicons

Bootstrap includes Glyphicons, an icon font, in its stylesheets by referencing them with bootstrap/. However, the font files are located in web/components/bootstrap-sass-official/vendor/assets/fonts/bootstrap/ and the CSS code in web/css/. I could just change the path to point to web/components, but I am picky when it comes to path and how they look (what if somebody checks the source?) and therefore I copy them into the web/fonts/ directory using a Gulp task. The gulp-copy plugin provides exactly the functionality I require. It has a prefix option that removes the unwanted directories from the beginning of source path.

// Gulpfile.js

var copy = (copy = require('gulp-copy'));

gulp.task('fonts', function () {
  return gulp
    .src('./web/components/bootstrap-sass-official/assets/fonts/bootstrap/*')
    .pipe(copy('./web/fonts', { prefix: 7 }));
});

However, the path is still wrong, instead of bootstrap/ it should be ../fonts/. Luckily Bootstrap uses a variable for this path and I can change it by adding the following line before importing Bootstrap into my Sass.

// src/Acme/Bundle/FrontendBundle/Resources/public/sass/master.scss

$icon-font-path: '../fonts/';

Running gulp fonts sass will execute both the fonts and the sass task and now everything should work fine.

Javascript

I use RequireJS as module and file loader for my project to load only those scripts dynamically that a specific page requires. The entry point of the JavaScript is app.js in AcmeFrontendBundle. Each bundle has it's own main.js that itself requires the files and modules it needs.

Load JavaScript using RequireJS

The app.js also needs to configure jQuery and the jQuery plugins provided by Bootstrap, because these don't have RequireJS modules defined.

// src/Acme/Bundle/FrontendBundle/Resources/public/js/app.js

require.config({
  paths: {
    bootstrap: '../../bootstrap',
    jquery: '../../jquery',
  },
  shim: {
    'bootstrap/affix': { deps: ['jquery'], exports: '$.fn.affix' },
    'bootstrap/alert': { deps: ['jquery'], exports: '$.fn.alert' },
    'bootstrap/button': { deps: ['jquery'], exports: '$.fn.button' },
    'bootstrap/carousel': { deps: ['jquery'], exports: '$.fn.carousel' },
    'bootstrap/collapse': { deps: ['jquery'], exports: '$.fn.collapse' },
    'bootstrap/dropdown': { deps: ['jquery'], exports: '$.fn.dropdown' },
    'bootstrap/modal': { deps: ['jquery'], exports: '$.fn.modal' },
    'bootstrap/popover': { deps: ['jquery'], exports: '$.fn.popover' },
    'bootstrap/scrollspy': { deps: ['jquery'], exports: '$.fn.scrollspy' },
    'bootstrap/tab': { deps: ['jquery'], exports: '$.fn.tab' },
    'bootstrap/tooltip': { deps: ['jquery'], exports: '$.fn.tooltip' },
    'bootstrap/transition': { deps: ['jquery'], exports: '$.fn.transition' },
  },
});

require(['main']);

The interesting part in the above code is the last line: it tells RequireJS to load the main.js file in the same directory. Currently I don't have a lot of JavaScript code in my project, but the following code from main.js enabled Bootstrap alert messages.

// src/Acme/Bundle/FrontendBundle/Resources/public/js/main.js

define(function (require) {
  require(['jquery', 'bootstrap/alert'], function () {
    $('.alert').alert();
  });
});

But what about JavaScript code from other bundles you might ask? I added an entry to the paths option in app.js and then require it at the bottom of the file.

// src/Acme/Bundle/FrontendBundle/Resources/public/js/app.js

require.config({
  paths: {
    // ...
    acmeuser: '../../acmeuser/js',
  },
  shim: {
    // ...
  },
});

require(['main', 'acmeuser/main']);

The main.js of the AcmeUserBundle can now require other modules.

// src/Acme/Bundle/UserBundle/Resources/public/js/main.js

define(function (require) {
  require(['jquery'], function () {
    $('.alert').addClass('hello-world');
  });
});

Building JavaScript

Actually, in my current setup I don't really build JavaScript files. Because I use RequireJS I don't have to concatenate them; I just copy them into web/js/.

// Gulpfile.js

gulp.task('js', function () {
  gulp
    .src([
      './web/bundles/*/js/**/*.js',
      './web/components/bootstrap-sass-official/assets/javascripts/bootstrap/*.js',
      './web/components/jquery/dist/jquery.js',
      './web/components/requirejs/require.js',
    ])
    .pipe(gulp.dest('./web/js'));
});

Watching and Reloading

It can become tedious to execute the sass and js tasks everytime the code of the Sas or JavaScript code changes. Therefore Gulp includes, as most modern build tools do, a watch functionality. In addition I use LiveReload to automatically reload my browser window when the CSS or JavaScript changes.

Watching

Gulp includes the watch() function by default. I use a glob pattern to watch Sass and JavaScript files separately. Of course it would be possible to use only one watcher for both, but then the JavaScript task would run even if only a stylesheets was changed.

// Gulpfile.js

gulp.task('watch', function () {
  var onChange = function (event) {
    console.log('File ' + event.path + ' has been ' + event.type);
  };
  gulp
    .watch('./src/Acme/Bundle/*/Resources/public/sass/**/*.scss', ['sass'])
    .on('change', onChange);
  gulp
    .watch('./src/Acme/Bundle/*/Resources/public/js/**/*.js', ['js'])
    .on('change', onChange);
});

Reloading

LiveReload is a great tool to reload the browser window when a file changes. First I install the gulp-livereload plugin and then I include a snippet in my layout. It's also possible to use a browser extension, but I prefer having everything I need in the repository. After installing the plugin I need to adapt the code of the watch task to inform LiveReload about the changed files.

// Gulpfile.js

var livereload = require('gulp-livereload');

gulp.task('watch', function () {
  var onChange = function (event) {
    console.log('File ' + event.path + ' has been ' + event.type);
    // Tell LiveReload to reload the window
    livereload.changed();
  };
  // Starts the server
  livereload.listen();
  gulp
    .watch('./src/Tvst/Bundle/*/Resources/public/sass/**/*.scss', ['sass'])
    .on('change', onChange);
  gulp
    .watch('./src/Tvst/Bundle/*/Resources/public/js/**/*.js', ['js'])
    .on('change', onChange);
});

In the layout I test if the environment is dev and then include the LiveReload snippet.

// src/Acme/Bundle/FrontendBundle/Resources/views/layout.html.twig

Running PHP Commands

I use Gulp also to run PHP-specific tasks and PHPUnit and PHP_CodeSniffer are just two of them. Because a Gulpfile.js is just a JavaScript file executed using Node.js I can run arbitrary shell commands.

PHPUnit

I use the gulp-phpunit plugin by Mike Erickson to run PHPUnit using Gulp.

$ npm install --save-dev gulp-phpunit

I use the task() function to create a new task with the name test. The src() function allows me to use a glob pattern to select the files I want to test. If you use another directory structure you need to adapt this pattern.

// Gulpfile.js

var phpunit = (phpunit = require('gulp-phpunit'));

gulp.task('test', function () {
  return gulp.src('./src/Acme/Bundle/*/Tests/**/*.php').pipe(
    phpunit('./bin/phpunit', {
      debug: false,
      configurationFile: './app/phpunit.xml',
    })
  );
});

I can execute the task using gulp test. The gulp-phpunit plugin provides a wide range of options. I also create a code coverage report.

// Gulpfile.js

gulp.task('coverage', function () {
  return gulp.src('./src/Tvst/Bundle/*/Tests/**/*.php').pipe(
    phpunit('./bin/phpunit', {
      debug: false,
      configurationFile: './app/phpunit.xml',
      coverageHtml: './build/coverage',
    })
  );
});

PHP_CodeSniffer

To run PHP_CodeSniffer I use the gulp-phpcs plugin by Dmitriy S. Simushev.

// Gulpfile.js

var phpcs = require('gulp-phpcs');

gulp.task('checkstyle', function () {
  return gulp
    .src(['src/Tvst/Bundle/**/*.php'])
    .pipe(phpcs({ bin: './bin/phpcs', standard: 'PSR2', warningSeverity: 0 }))
    .pipe(phpcs.reporter('log'));
});

I also created a shortcut to execute both coverage and checkstyle.

// Gulpfile.js

gulp.task('verify', ['coverage', 'checkstyle']);

Symfony2 Commands

During my build process I also need to run Symfony2 commands. Instead of using a plugin I use the child_process module that comes with Node.js to execute Shell commands. Because the Gulpfile consists of plain JavaScript code there exists no need to use a specific plugin.

// Gulpfile.js

var exec = require('child_process').exec;

gulp.task('installAssets', function () {
  exec('php app/console assets:install --symlink', logStdOutAndErr);
});

// Without this function exec() will not show any output
var logStdOutAndErr = function (err, stdout, stderr) {
  console.log(stdout + stderr);
};

You can see that I created a Gulp task to run the install assets commands. I can integrate this task into other tasks and thus I will not forget to run it when building my assets. In my Gulpfile I also have tasks to load fixtures and update the database schema.

Roadmap

The project I am using this build system for is not yet in production and therefore there are still some things missing. First of all I haven't added minification of CSS and JavaScript to the Gulpfile. Another problem I have with the current setup is that the components/ and bundles/ directories reside in web/ and are accessible by users. Before going into production I want to move these two directories somewhere else.

Another problem that could become relevant when my stylesheets become bigger is that every change in a .scss file will cause a compilation of all Sass code (including Bootstrap). I might need to figure out a way to compile, for examples, bundles separately and concatenate them only for production.

My Gulpfile still misses some other smaller things. For example, I want to use CSSComb to sort the properties in the stylesheets and I want to integrate Autoprefixer to automatically add vendor prefixes to CSS properties. Linting of Sass and JavaScript code is also missing.

Conclusion

I'm satisfied with my current Gulp setup. As mentioned above it lacks some features I need to add before going in to production. The build system is fast, flexible and (hopefully) will scale well. It's not fully automatic, I have to add a little bit of configuration when creating a new bundle, but that happens not often and I prefer control over automation in these cases.

The system is opinionated and it works well in my setup and with my project structure. If you have made different decisions about structure or need some other features there might be problems. Still I am happy about feedback regarding my system. I am new to Gulp and haven't used it in a production project so suggestions are very welcome.

Updates

• 3 November 2014: Since the end of October I use NPM intead of Bower to manage frontend dependencies.
• 25 November 2014: I updated the article to reflect the directory structure changes in bootstrap-sass v3.2.0.

Inspired by the Symfony Standard Edition it contains a basic directory structure, a composer.json with ddeboer/data-import and symfony/console as dependencies and a simple executable.

You can bootstrap an import script project by using Composer and the following command:

$ php composer.phar create-project florianeckerstorfer/console-import-standard-edition path/to/install

Then you only need to change the namespace and adapt the code inside the ImportCommand to fit your needs. The projects README.md contains more information on how to setup a project and customise it.

To get inspirational Andy Dwyer quotes into your terminal you need four things: fortune (to display a quote), cowsay (to make a cow say the quote), lolcat (for the rainbow colouring) and, of course, the quotes. On a Mac you can use Homebrew to install fortune and cowsay:

$ brew install fortune cowsay

lolcat is a Ruby program and can you can install it using RubyGems:

$ sudo gem install lolcat

Lolcat

Let's try these three programs out. lolcat is first.

Cowsay

You can pipe input to cowsay to make it appear in a speak bubble on top of a cow. cowsay has some options to change the eyes and the tongue and even use a different animal, for example, a dragon.

$ echo "Hello World!" | cowsay
 _____________
< Hello World >
 -------------
        \   ^__^
         \  (oo)\_______
            (__)\       )\/\
                ||----w |
                ||     ||

Fortune

If you call fortune without any arguments it will print a random quote.

$ fortune
Felson's Law:
    To steal ideas from one person is plagiarism; to steal from
    many is research.

The default database contains some pretty sexist quotes, as you sadly can expect from something written by white males. But I want to use my own database anyway. The format of a quotes file is pretty simple. You separate the quotes by a single % on a newline.

My whole life is a giant mess and I love it
-- Andy Dwyer
%
I know what things are.
-- Andy Dwyer
%
You're like an angel with no wings.
-- Andy Dwyer

However, the fortune program reads compiled data files and we can use strfile to create these:

$ strfile andy andy.dat

I can now call fortune with this quote file and it will only show quotes from the given file:

$ fortune andy

You can also find these files in my dotfiles repository. There is also a file with Tyrion Lannister quotes.

Piping it All Together

If I now call fortune, pipe the output into cowsay and then pipe its output into lolcat I get the result displayed in the first screenshot.

$ fortune ~/dotfiles/fortune/andy | cowsay -s | lolcat

I placed this line in my .bash_profile file and now every time I open a terminal window I get an inspirational Andy Dwyer quote.

One of the consequences of these developments is that, for example, PHP does no longer mean the full stack of the programming language. You can write code in PHP syntax and interpret it with the official PHP interpreter or you can HHVM`s Just-in-Time compiler. Another example is Dart, a language developed by Google that can run in a VM or you can transpile it to JavaScript to run in a browser. We need to separate the writing of the code (a human using the syntax of a language to create a program) from the execution of the code (a machine reading the code and converting it into a form understood by the hardware). The choice of programming language (or syntax) no longer gives us any indication how and where the code runs.

If you write code in JavaScript you should be aware that this code might execute in a browser, or on a server, or on a desktop computer or in a native phone app.

Yesterday @alicetragedy tweeted the following quote from @patio11 from Frozen Rails.

Ruby's just a tool. You'll never hear a lawyer say: I'm a Microsoft Word lawyer. Check out this contract! Copy, paste, nailed it!

Exactly for this reason you don't see me describing myself as a PHP Developer but as a Web Developer or just Developer. I write most of my code in PHP and JavaScript and I have written code in Ruby, Python, Java, C and other languages before and I am quite confident that I can write any program in any imperative programming language (preferable with C-style syntax). I also did some functional programming, but never enough that I could fully wrap my head around it. The concepts in all these programming languages are basically the same: functions, loops, conditionals and variables.

I find the hard part of learning a new programming language is not the language itself, but the ecosystem. When I write software in PHP and have a specific task I often know the frameworks and libraries to help me, I know exactly where to look for them and if a library is good or bad. The boom of package repositories such as Packagist, NPM, RubyGems, CocoaPod and so on make it easier to find new packages but you need experience with the community, the language and the conventions of the language to judge if a package is useful, well written and well maintained. Connected to this is the fact that knowing a programming language does not enable you to write a "big" application in it. Often it's more effort to learn a new framework than a new language. Full-stack frameworks like Symfony, Rails or Django are monsters with hundreds of APIs, conventions and configuration formats.

Above I said that I am confident that I can write any program in any language. What I didn't say is that I can write any program in the best possible way in any language. You still need years of experience to master the nitty gritty details of a programming language. And often execution environment matters in addition to the syntax of the language. Let me give you a quick example in PHP:

// Merge the following two arrays
$array1 = ['PHP', 'Ruby', 'Python', 'JavaScript'];
$array2 = ['Java', 'C', 'C++'];

// Method 1
$fullArray = array_merge($array1, $array2);

// Method 2
$fullArray = $array1;
foreach ($array2 => $value) {
    $fullArray[] = $value;
}

Which of the two methods executes faster? Well. It depends.

• In PHP 5.3 when the arrays have 100,000 or more elements Method 2 runs faster.
• In PHP 5.5 Method 2 only runs faster when the arrays have more than 10,000,000 elements.

Clearly there have been some optimisations to the array_merge method over the years. I don't have an installation of HHVM ready but the results there may be different again. I don't know such specific performance implications when writing Ruby oder Python code. Years and years of experience, debugging and frustrations gave me such a deep knowledge about PHP. As we can see in the differences between PHP 5.3 and 5.5 the behaviour of the platforms changes in addition to the syntax; if I once knew every optimisation doesn't mean I know them in the next version.

I believe that a programmer becomes a professional programmer by being able to generalise specific solutions in a specific programming language on a specific platform to programming in general. If you can systematically identify problems and solve them systematically (and reproducible) you have become a professional programmer.

I added a Bash function to my Dotfiles that creates an empty PHP file, opens it in Sublime Text and uses Watchman to run the file whenever it changes. At first I thought this should be trivial, but it I had some problems. But let's go through it step-by-step. Creating and opening a file is easy, I just used the subl command. However, to execute the script when it changes I needed to use Watchman and there is no obvious way to display the output of the executed script in the terminal. Therefore I create a wrapper bash script and write the output to a log file that I display using tail -f.

I added some clean-up functionality, the script deletes the watch and trigger from Watchman and removes the wrapper script and log file after quitting tail with Ctrl-C. You can use the script below by adding it to your .bash_profile.

function phppad() {
    TMPNAME="phppad-$RANDOM"
    touch "$@"
    subl "$@"
    echo "(echo '$(date +%n%H:%M:%S) $ php $@';php $@) >> $TMPDIR/$TMPNAME.log" > $TMPDIR/$TMPNAME.sh
    chmod +x $TMPDIR/$TMPNAME.sh
    touch $TMPDIR/$TMPNAME.log
    watchman watch `pwd` > /dev/null
    watchman -- trigger `pwd` run "$@" -- $TMPDIR/$TMPNAME.sh > /dev/null
    tail -f $TMPDIR/$TMPNAME.log

    watchman trigger-del `pwd` run > /dev/null
    watchman watch-del `pwd` > /dev/null
    rm $TMPDIR/$TMPNAME.log
    rm $TMPDIR/$TMPNAME.sh
}

I know there exists at least one REPL, PHPSH, but it's no longer maintained and I want to use Sublime Text to write the code and I want to persist the code, even when quitting the command.

Recently I caught myself pressing Cmd+Tab to get to Safari, then Ctrl+Tab to get to the next tab, Cmd+Tab to get back into PHPStorm and after a few minutes this process starts again. What I needed was a way to temporarily hide the current top-most window to peak into the background.

In this article I am going to create a Peak keyboard shortcut using Keyboard Maestro to do exactly this. When the shortcut is pressed the current top-most window is hidden, pressing the shortcut a second time shows the window again. I am going to share not only the code for the shortcut, but also the process on how to create it.

There exist many different applications to manage keyboard shortcuts, but Keyboard Maestro is probably the most powerful one. Compared to the many free alternatives the price point of €30 for KM looks quite hefty, but it has many unique features, such as variables, loops and conditionals. This means that you can add states to your shortcuts, which is a lot more powerful than it sounds at first glance. Without support for these features, and the great AppleScript support, I would not be able to implement the Peak shortcut.

But let's get started.

The main problem I encountered when I started working on the shortcut was that the first invocation hides the window and I don't have a way to retrieve the previously active window when activating the shortcut the next time. Therefore I need two states, either I am currently "peaking", that is, a window is currently hidden or I am not peaking and no window is hidden. Since the macro editor in Keyboard Maestro is in part visual I am going to post pseudo code for now.

if peakStatus is not "hidden" then
    PeakAppName = name of topmost application
    PeakWindowName = name of topmost window of application PeakAppName
    hide window PeakWindowName of application PeakAppName
    PeakStatus = "hidden"
else
    show window PeakWindowName of application PeakAppName
    PeakStatus = ""
end if

In Keyboard Maestro there exists no way to determine the name of the top-most application or the name of the first window of an application. But I can use AppleScript to get these values and Keyboard Maestro allows me to assign the return value of an AppleScript to a variable and that's what I am going to do.

tell application "System Events"
    set frontApp to first application process whose frontmost is true
    name of frontApp
end tell

Getting the name of the first window of the top-most application is a little bit more complicated. First I get the name of the application and then I tell that application to give me the name of first window.

tell application "System Events"
    set frontApp to first application process whose frontmost is true
    set frontAppName to name of frontApp
end tell

tell application frontAppName
    name of window 1
end tell

However, there are some application that don't return a list of windows (inclding Sublime Text and Spotify) and don't allow to set the visibility of a single window. In this case I am going to hide the complete application instead of just a single window. Therefore I need to replace the name of window 1 to include a fallback. When I can't the name of the window I store an empty string and later will decide between the two cases.

try
    name of window 1
on error errMsg
    ""
end try

I now have the name of the application and window and using the save results to variable option in Keyboard Maestro these values are safely stored in a variable. Currently I am still in the not hidden part of the pseudo code and the last piece still missing is the code to actually hide the window. First I retrieve the names of the application and window from KM and then there is a case distinction wether I have the window name. If I have the name of the window I tell the application to hide the window. Otherwise I use System Events to hide the complete application.

tell application "Keyboard Maestro Engine"
    set frontAppNameVar to make variable with properties {name:"PeakAppName"}
    set frontAppName to value of frontAppNameVar
    set windowNameVar to make variable with properties {name:"PeakWindowName"}
    set windowName to value of windowNameVar
end tell

if windowName is not equal to "" then
    tell application frontAppName
        set visible of window windowName to false
    end tell
else
    tell application "System Events"
        set visible of process frontAppName to false
    end tell
end if

The code in the hidden part of the pseudo code is a little bit easier. The name of the application and window are already stored in a variable and I only have to retrieve these values and show the application. Once again I need to make distinction wether I have the name of the window. If I have the window name I tell the application to show the window, otherwise I use System Events to show the application and then activate it.

tell application "Keyboard Maestro Engine"
    set frontAppNameVar to make variable with properties {name:"PeakAppName"}
    set frontAppName to value of frontAppNameVar
    set windowNameVar to make variable with properties {name:"PeakWindowName"}
    set windowName to value of windowNameVar
end tell

if windowName is not equal to "" then
    tell application frontAppName
        set visible of window windowName to true
    end tell
else
    tell application "System Events"
        set visible of process frontAppName to true
    end tell
    tell application frontAppName to activate
end if

Creating the macro in Keyboard Maestro now is just a matter of arranging several pre-defined actions. You can find the text version of the macro below or take a look at the screenshot of how it should look like in KM.

If All Conditions Met
    The variable ‘PeakStatus’ is not ‘hidden’
    Execute the Following Actions:
        Execute AppleScript
            tell application "System Events"
                set frontApp to first application process whose frontmost is true
                name of frontApp
            end tell
        Save trimmed to variable ‘PeakAppName’.
        Execute AppleScript
            tell application "System Events"
                set frontApp to first application process whose frontmost is true
                set frontAppName to name of frontApp
            end tell

            tell application frontAppName
                try
                    name of window 1
                on error errMsg
                    ""
                end try
            end tell
        Save trimmed to variable ‘PeakWindowName’.
        Set Variable ‘PeakStatus’ to Text
            hidden
        Execute AppleScript
            tell application "Keyboard Maestro Engine"
                set frontAppNameVar to make variable with properties {name:"PeakAppName"}
                set frontAppName to value of frontAppNameVar
                set windowNameVar to make variable with properties {name:"PeakWindowName"}
                set windowName to value of windowNameVar
            end tell
            if windowName is not equal to "" then
                tell application frontAppName
                    set visible of window windowName to false
                end tell
            else
                tell application "System Events"
                    set visible of process frontAppName to false
                end tell
            end if
            Display results in a window.
        Otherwise, Execute the Following Actions:
            Set Variable ‘PeakStatus’ to Text
            Execute AppleScript
                tell application "Keyboard Maestro Engine"
                    set frontAppNameVar to make variable with properties {name:"PeakAppName"}
                    set frontAppName to value of frontAppNameVar
                    set windowNameVar to make variable with properties {name:"PeakWindowName"}
                    set windowName to value of windowNameVar
                end tell

                if windowName is not equal to "" then
                    tell application frontAppName
                        set visible of window windowName to true
                    end tell
                else
                    tell application "System Events"
                        set visible of process frontAppName to true
                    end tell
                    tell application frontAppName to activate
                end if
            Display results in a window.

As I am writing this I realise that it would have been a lot easier to just write one long AppleScript instead of using the pre-defined actions to compose several smaller ones. But since I sunk enough time in writing this macro I leaving it this way for now.

If you don't want to retype everything from this article, you can download the macro and import it into Keyboard Maestro: peak.kmmacros.

Object hydration is the default one and it returns the data as objects. This provides the greatest convenience for the developer, but requires a complex and memory intensive hydration process. In most projects for 95 to 100% of the queries this is a reasonable trade-off, but then there is this single import or export script that deals with 100,000 of objects and object hydration takes too much time or consumes too much memory. For these cases Doctrine supports array hydration which creates arrays instead of objects.

And then there exist ultra-rare cases where you don't import a few hundred thousand rows, but something like 50 million and even array hydration is too slow. In these cases I have fallen back to native SQL queries in the past.

Native SQL queries are considerable faster because no hydration is involved, but since I want to use Doctrine and hydration for the other 99% of my application I have to duplicate code and deal with different ways to invoke queries and so on. Therefore I looked into a feature of Doctrine I haven't really considered before: Custom Hydrators.

Custom Hydration

Custom Hydrators are pretty straight forward, you sub-class the AbstractHydrator class provided by Doctrine and implement the hydrateAllData() method, obtain a reference to a PDOStatement object and returns the result. You can hydrate to whatever you want, array, object or scalar values. Before implementing my first hydrator I looked at the source code of the ObjectHydrator and ArrayHydrator and was a little bit overwhelmed. There was so much code and more loops than I expected. If you know that thousands or even millions of elements will go through this code you get suspicious whenever you see a loop. In retrospect the amount of code makes sense: the hydrator needs to map the column names to the field names used in the code, convert the values into their PHP type, resolve references and so on. Still, too many loops.

Simple Array Hydration

My first idea to speed up the hydration process was to return the row I retrieved from PDO and return it; no mapping or conversion. I loose a lot of convenience compared to object or even array hydration, but when the alternative is to use PDO it makes sense. The result is in the same format as it would be when using PDO directly, but I can continue to use most other features of Doctrine, such as the entity manager, DQL or repositories.

use Doctrine\ORM\Internal\Hydration\AbstractHydrator;

class SimpleArrayHydrator extends AbstractHydrator
{
    protected function hydrateAllData()
    {
        $result = [];
        foreach($this->_stmt->fetchAll(\PDO::FETCH_ASSOC) as $row) {
            $result[] = $row;
        }

        return $result;
    }
}

Benchmarks

I tested my SimpleArrayHydrator with a simple dataset (one table with three columns: a primary key, a string and an integer and about 6,500 rows) and compared the performance to object and array hydration as well as using PDO directly. The results below indicate average values over 1,000 iterations and are measured using Athletic.

We can see in these results that my custom PDO-style hydration is practically as fast as directly using PDO. Object and array hydration can process equally many rows per second for this data set. Side note: the difference between object and array hydration will be bigger for more complex data sets (for examples, references).

Custom Custom Hydrators

While my SimpleArrayHydrator is very fast it does not provide a lot of conveniences for developers. There is definitely room in the middle for a hydrator that is faster than default object and array hydration and provides certain conveniences for the developer. Remember that Doctrine is an extremely popular project and that the default hydrators need to work in a lot of different scenarios. If you develop a custom hydrator for your project you can make certain assumptions to improve the performance while maintaining a certain level of convenience. For example, if your project has a convention that the name of a column in the database and the name of the corresponding property in the object must be equal, you can skip the key mapping during hydration.

Of course, you can also create a hydrator specifically for an entity or even a specific query that causes you performance problems. Please don't create a specific hydrator for every table in your project, that defeats the purpose of using an ORM, but if for a specific entity or query the alternative is switching to PDO code it certainly makes sense to consider it.

Think of it that way, if you use PDO directly you iterate through the result set and save the rows in some data structure you are basically hydrating the result set. In fact, if you use PDO directly you are most likely writing a custom hydrator for every query you execute. Thus it makes sense to consider a custom hydrator.

Therefore I created a small PHP library and CLI script as well as an Alfred workflow to invoke it. I am using it for a few weeks now and it works great for my purposes.

I have set up a page with more information, passwordplea.se (pretty sweet, right?), and the source code of both the Alfred workflow and the PHP library and CLI are on Github.

The Weekly Recommender is a new weekly series that I plan to write for two purposes, giving myself a reason to revisit the content I encountered the past week, collect them in one place and make the highlights accessible to my readers.

Development

• AngularJS Tutorial: A Comprehensive 10,000 Word Guide
• Accessing and Debugging Symfony Form Errors
• CSS Triggers
• wgrep is a command line utility for running selector queries against URLs
• Adaptive Backgrounds: A jQuery plugin for extracting dominant colors from images and applying it to its parent.
• iTerm Replacement Icon

Podcasts

• Inquisitive #8: A Little More Playful, with Shawn Blanc
• 99% Invisible #134: The Straight Line Is A Godless Line
• The Web Ahead #86: Componentizing the Web with Rob Dodson
• Debug #44: Brianna Wu, Amanda Warner, and Revolution 60

Everything Else

• Pete Warden: Why nerd culture must die
• Kinja: What Is Gamegate, and Why? An Explainer for Non-Geeks
• Elle: Lessons From Grimes on How To Be a Boss
• Jo Freeman: The Tyranny of Structurelessness
• Laurie Penny: Why We’re Winning: Social Justice Warriors and the New Culture War
• If Ron Swanson Quotes Were Motivational Posters

The MySQL character encoding utf8 has a maximum of 3 bytes, which contains most characters widely used in the western world. However, Chinese, Japanese or Korean alphabets require 4 bytes and therefore cannot be stored in a MySQL column that is encoded with utf8. MySQL provides an additional character encoding utf8mb4 to support those characters. By the way, Emojis also require 4 bytes and utf8mb4.

In this article I am explaining how to create utf8mb4 databases, tables and columns with plain SQL and with Doctrine ORM.

Update 25th of July 2016 Starting with MySQL 5.7.7 this is no longer a problem. More information can be found here.

MySQL

MySQL lets you set the character encoding and collation on multiple different level. You can set the default encoding for a database, for a table and you can specify the encoding of an indiviual column.

/* Create database */
create database utf8mb4test
    character set utf8mb4
    collate utf8mb4_unicode_ci;

/* Create table */
create table unicode_names
    (
        name varchar(255)
    )
    default character set utf8mb4
    collate utf8mb4_unicode_ci;

/* Create table with mixed encodings */
create table mixed_encoding
    (
        name varchar(255)
            character set utf8mb4
            collate utf8mb4_unicode_ci,
        email varchar(255)
    )
    default character set utf8
    collate utf8_unicode_ci;

When you design your database and decide on encodings for specific columns, please remember that there are TLDs in many languages that require 4 bytes now. IANA has a list with all top-level domains.

If you use pure SQL and create a table without a specific character encoding and collation the table will be created using the default character encoding of the database. The same applies for columns, adding a column without a specific character encoding and collation will use the tables.

Matthias Bynens has a pretty nice overview how to migrate utf8 tables to utf8mb4 and how to set things up correctly on the MySQL side of things.

Doctrine

Most likely you are not creating your schema by hand, but generate it using an ORM. I use Doctrine and the documentation and available resources on this topic are outdated in many places. First of all, the behaviour of Doctrine must have changed recently, because if you Google something like doctrine utf8mb4 you will find resources saying that you have to configure this at a database level.

In fact, the reference included with the source code (as of v2.4.6) says the following

You can't set these values inside the annotations, yml or xml mapping files. To make a database work with the default charset and collation you should configure MySQL to use it as default charset, or create the database with charset and collation details. This way they get inherited to all newly created database tables and columns.

However, it seems that Doctrine now specifically defines the encoding and collation when creating a new table. You can see this if you dump the schema SQL:

$ php app/console doctrine:schema:update --dump-sql

The result is something like this:

CREATE TABLE user
    (
        id INT AUTO_INCREMENT NOT NULL,
        username VARCHAR(255) NOT NULL,
        /* ... */
    )
    DEFAULT CHARACTER SET utf8
    COLLATE utf8_unicode_ci
    ENGINE = InnoDB;

Therefore the settings on the database-level are overwritten by the table-specific configurations set by Doctrine. Unfortunately we need to set the charset and collation for every table since there is no global option and Doctrine doesn't respect the setting defined in MySQL.

The related options are called charset and collate and can be set using all configuration formats, for example, when using annotations:

/**
 * @ORM\Entity()
 * @ORM\Table(name="users", options={"collate"="utf8mb4_unicode_ci", "charset"="utf8mb4"})
 */

Or when using XML:

<entity name="Acme\DemoBundle\Entity\User" table="users">
    <options>
        <option name="charset">utf8mb4</option>
        <option name="collate">utf8mb4_unicode_ci</option>
    </options>
</entity>

If you now dump the SQL to create the schema you should get something like this:

CREATE TABLE user
    (
        id INT AUTO_INCREMENT NOT NULL,
        username VARCHAR(255) NOT NULL,
        /* ... */
    )
    DEFAULT CHARACTER SET utf8mb4
    COLLATE utf8mb4_unicode_ci
    ENGINE = InnoDB;

Doctrine currently doesn't seem to support setting the charset and collation on the column-level. If you want to change the encoding on a per-column basis you need to do this using SQL.

Indexes

When converting your tables from utf8 to utf8mb4 there is one more thing you need to consider. An index in InnoDB always has a maximum size of 767, bytes regardless of the number of bytes used for a single character. Consider a column with a maximum of 255 characters, this would result in 765 bytes when using utf8 but in 1020 bytes when using utf8mb4. If you need an index on a utf8mb4 column the maximum number of characters is 191 instead of 255.

Encoding of the Connection

One last thing you need to take care of is the encoding of the connection. If you are using Doctrine in combination with Symfony you can set this in your config.yml

# app/config/config.yml
doctrine:
  dbal:
    charset: utf8mb4

Or if you don't use Symfony:

$config = new \Doctrine\DBAL\Configuration();
// ...
$connectionParams = array(
    'driver'   => 'pdo_mysql',
    'host'     => 'localhost',
    'user'     => 'user',
    'password' => 'secret',
    'dbname'   => 'mydb',
    'charset'  => 'utf8mb4'
);
$conn = \Doctrine\DBAL\DriverManager::getConnection($connectionParams, $config);

In SQL you would use

SET NAMES='utf8mb4';

Handling Multi-byte Strings in PHP

This has nothing really to do with databases, but just don't forget that when you deal with multi-byte strings (such as Emoji or CJK characters) you need to use the mb_* methods, such as mb_substr() or mb_strlen().

Conclusion

Considering that full UTF8 support is pretty important these days it's probably worth converting tables to utf8mb4 or at least use utf8mb4 in new projects. I heard some evidence that utf8mb4 is slower than utf8, but I couldn't find some definitive performance measurements. Michael Simmons found that joining string columns with different encodings is extremely slower then joining columns with the same encoding (which makes sense). In my opinion this should not keep you from using utf8mb4 now.

And don't forget to treat yourself with a  after dealing with all this encoding stuff.

Development

• Visuzalizing Git Concepts with D3
• Alice is a fixtures generator written in PHP. There exists also a bundle to integrate Alice into Symfony.
• Pixeler can render images in CLI using UTF-8 characters (written in PHP)
• CSShake provides some pure CSS shake effects
• Prism is a syntax highlighter written in JavaScript

Mac OS X

• Airmail 2.0 has been relased and its great.

Podcasts

• The Talk Show #98: ‘Far Less Eloquent As You’, With John Siracusa
• Theory of Everything: Enchanting By Numbers
• Strangers: Falling Slowly

Feminism and Gamergate

• Felicia Day: Crossing the Street
• Jesse Singal: Gamergate Should Stop Lying to Journalists – and Itself
• Jessica Valenti: Gamergate is loud, dangerous and a last grasp at cultural dominance by angry white men
• Allan G. Johnson: A Guy Is A Man Is A Guy

Using one instead of two package managers makes it easier keeping the packages up-to-date and removes clutter from the repositories. I think it is a good idea to keep the dependencies out of the document root and to use a Gulp task to copy only the files that are required into a publicly accessible directory. My task looks like this:

var _ = require('lodash'),
  gulp = require('gulp');

gulp.task('copy-assets', function () {
  var assets = {
    js: [
      './node_modules/html5shiv/dist/html5shiv-printshiv.min.js',
      './node_modules/angular/angular.js',
      './node_modules/angular-resource/angular-resource.js',
      './node_modules/angular-route/angular-route.js',
      './node_modules/picturefill/picturefill.js',
    ],
    css: ['./node_modules/normalize.css/normalize.css'],
  };
  _(assets).forEach(function (assets, type) {
    gulp.src(assets).pipe(gulp.dest('./web/' + type));
  });
});

Now I seriously don't know why I ever used Bower.

This is part tutorial, part reminder for myself on how to install this fucking printer.

You need GCC to compile C code. In my opinion the best way to get GCC is to install Xcode from the Mac App Store. In Xcode you have to install the Command Line Tools (Preferences > Downloads > Components). You also need Homebrew to install some libraries. Open a Terminal window and install wget and gnu-sed.

$ brew install wget
$ brew install gnu-sed
$ brew install ghostscript

Download and install Foomatic RIP and HIPJS for Mac OS X. The page also includes a download for Ghostscript, but I used Homebrew to install it. The links state that Foomatic RIP and HIPJS are for OS X 10.8, but for me they work fine on 10.10 Yosemite.

Next you need to install foo2zjs. Terminal:

$ wget -O foo2zjs.tar.gz http://foo2zjs.rkkda.com/foo2zjs.tar.gz
$ tar zxf foo2zjs.tar.gz
$ cd foo2zjs
$ make
$ ./getweb 1215
$ sudo make install
$ sudo make cups
$ cupsctl WebInterface=yes

Open the CUPS web interface: http://localhost:631

1. Add a printer by clicking on Adding Printers and Classes and then Add Printer. Enter your Mac OS X user name (must be an administrator) and your password.
2. Select the printer (I connected the printer to my AirPort Express and it showed up in the list automatically) and click the submit button.
3. Enter a name and description and submit the form again.
4. Select HP
5. Select HP Color LaserJet CP1215 Foomatic/foo2hp (en)
6. Click on Add Printer
7. Set Color Mode to Color and Bits Per Plane to 2 Bits Per Plane. Submit.

Done.

Please note that I cannot give support on installing this fucking printer on OS X. If you send me emails about this printer I will ignore them.

Gulp did not invent streams, they are part of Node, but it standardises them for build systems in a way that code from different authors can work together without knowing about each other.

Streams in Node.js work a lot like streams work on YouTube. You can start consuming data while the data is not fully loaded, but every chunk of loaded data is given to you immediately after it is loaded and you are not currently processing the previous chunk. Loading does not necessarily mean downloading from the internet, it could also mean reading data from disk or from a database.

Let us consider the following scenario: you have your source code written in CoffeeScript and stored in a bunch of .coffee files. To deploy to production you need to compile CoffeeScript to JavaScript, concatenate them into one big file, minify the source code, and move the file to a dist/ directory.

In a traditional task runner (that would be something like Make or Grunt) each of these steps would need to read the corresponding files from disk at the beginning and write them back to disk when it has done its work.

1. Read src/*.coffee ➞ Compile CoffeeScript ➞ Write tmp/*.js
2. Read tmp/*.js ➞ Concatenate ➞ Write tmp/all.js
3. Read tmp/all.js ➞ Minify ➞ Write tmp/all.min.js
4. Copy tmp/all.min.js to dist/

Streams in Gulp provide us with a way to convert files into object that can flow through the pipeline without having to be written to disk after each step. The same workflow would look like this in Gulp:

1. Start reading src/*.coffee
2. Pipe output from 1 into CoffeeScript
3. Pipe output from 2 into a concatenate function
4. Pipe output from 3 into Minify
5. Write output from 4 to dist/

In code it would look something like this:

// gulpfile.js
var gulp = require('gulp'),
  coffee = require('gulp-coffee'),
  concat = require('gulp-concat'),
  uglify = require('gulp-uglify');

gulp.task('default', function () {
  gulp
    .src('src/*.coffee')
    .pipe(coffee())
    .pipe(concat('all.min.js'))
    .pipe(uglify())
    .pipe(gulp.dest('dist'));
});

(The code above uses, in addition to Gulp, the gulp-coffee, gulp-concat, and gulp-uglify plugins. You can install them using NPM.

I want to show you how these aforementioned file objects flow through the workflow, but to do so we need to take a look at Vinyl. Vinyl is a virtual file format, that is, an abstraction of the filesystem. By abstracting the file system Gulp, and Gulp plugins, doesn't need to know about the underlying filesystem, which in most cases is the local filesystem, but could also be on Github or S3. The other thing Vinyl does is that it provides us with the File object that is passed through the workflow.

As a side note: the src(), dest() and watch() methods implemented by Gulp are barely more than wrapper functions of the corresponding method in the Vinyl adapter.

Ok, let us write a Gulp plugin that visualises the flow of the File object through the pipeline. That way we do not only see how files are processed internally in Gulp, but we also learn how to write a Gulp plugin on our own.

// lib/log.js
var through = require('through2');

module.exports = function (label) {
  function log(file, enc, cb) {
    console.log(label, ':', file.path);
    cb(null, file);
  }

  return through.obj(log);
};

This code is pretty simple and straight-forward. We use the through2 library as a wrapper for Node streams and create a log() function which outputs the filename of each processed file to the console. log() is called by Gulp for each file that it processes with three parameters. The first is a File object created by Vinyl and the third is a callback function. When we are done with our work we pass the file to the callback and through this move it further along in the workflow.

Of course, we don't have to pass the file along, a plugin like concat consumes all files and removes them from the stream. When it has processed the very last file it creates a new File object for the concatenated file and emits it to the workflow.

Now we can use the log plugin in our Gulpfile to display the filename of each processed file alongside a label. We will add the log plugin before every plugin in our workflow to display the plugin name as label. This way we can see which plugin processes which files.

// gulpfile.js
// ...
var log = require('./lib/log');

gulp.task('log', function () {
  gulp
    .src('src/*.coffee')
    .pipe(log('coffee'))
    .pipe(coffee())
    .pipe(log('concat'))
    .pipe(concat('all.min.js'))
    .pipe(log('uglify'))
    .pipe(uglify())
    .pipe(log('dest'))
    .pipe(gulp.dest('dist'));
});

If our src/ directory contains three files, foo.coffee, bar.coffee and qoo.coffee the output will look like this:

coffee : /Users/florian/Projects/slides/gulpjs-streams/src/bar.coffee
concat : /Users/florian/Projects/slides/gulpjs-streams/src/bar.js
coffee : /Users/florian/Projects/slides/gulpjs-streams/src/foo.coffee
concat : /Users/florian/Projects/slides/gulpjs-streams/src/foo.js
coffee : /Users/florian/Projects/slides/gulpjs-streams/src/qoo.coffee
concat : /Users/florian/Projects/slides/gulpjs-streams/src/qoo.js
uglify : /Users/florian/Projects/slides/gulpjs-streams/src/all.min.js
dest : /Users/florian/Projects/slides/gulpjs-streams/src/all.min.js

We can see that bar.coffee is read and pipe through coffee and concat, but is not passed further along because concat consumes the stream. When concat finished consuming all files it emits a new file all.min.js and this one is passed to uglify and dest.

Resources

If you want to learn more about streams in Node and Gulp, take a look at the following resources:

• Stream Handbook by substack
• Gulp (The vision, history, and future of the project)

These have already been linked in the article, but you still might want to take a closer look ant

• Node Stream API
• vinyl
• vinyl-fs
• through2

Slides

This article is the novelisation of my talk at the ViennaJS meet-up on 24 June 2015. The slides are already online. The event was filmed and I plan to update this article when the video is online. The video is now online.

In addition John uses an AppleScript to save the URLs of all pages open in Safari into a text file, grouped by window. He did not post this one, but it was easy enough for me to recreate it.

The script needs to do two things, first gather all the URLs and format them in a string and then save this string to a file named with the current date and time. As you can see working with dates is a little bit messy in AppleScript, but the result works quite well.

--- Gather the URLs of all open tabs
tell application "Safari"
    set _urls to URL of every tab of every window
end tell

--- Create a string that contains all URLs, grouped by window
set _content to ""
repeat with _window_urls in _urls
    repeat with _url in _window_urls
        set _content to _content & "
- " & _url
    end repeat
    set _content to _content & "
"
end repeat

--- Create a string with the current date and time
set the_mon to month of (current date) as integer as string
if (count characters in the_mon) = 1 then set the_mon to "0" & the_mon
set the_day to day of (current date) as string
if (count characters in the_day) = 1 then set the_day to "0" & the_day
set the_hour to time string of (current date) as string
set the_date to (year of (current date) as string) & "-" & the_mon & "-" & the_day & " " & the_hour

--- Save the string to a folder in Dropbox
set _filename to the_date & ".md"
do shell script "/bin/echo -n " & quoted form of _content & " > $HOME/Dropbox/Documents/Tabs/" & quoted form of _filename

I use Keyboard Maestro to assign a hotkey to the script and invoke it whenever I want to save all currently open URLs.

In this post I am going to explore third-party apps that allow scrobbling on OS X and iOS and explain when and what will be scrobbled.

Update (March 20, 2021): Last.fm and Apple Music in 2021

iTunes on OS X

Scrobbling played songs from iTunes always required a third-party application. There is the official Last.fm Mac app, but for the last couple of years I used Simplify. Simplify can scrobble from iTunes, Rdio, Spotify and others and can also display the currently playing song in the menu bar. But does Simplify work with Apple Music? It depends.

It depends whether iTunes reports the currently playing track via AppleScript or not. We can test whether scrobbling works using a small AppleScript

tell application "iTunes"
    tell current track to artist & " - " & name
end tell

First of all, scrobbling works for everything that is in My Music (that is, your library). It does not matter if the song is stored locally, streamed from iTunes Match, streamed from Apple Music, or a song from Apple Music that was made available offline. The same is true for playlists, scrobbling works for you local playlists, as well as Apple Music Playlists. But now things get complicated.

Let's take a look at the For You and New sections.

• If you play a song, album, or playlist directly in the For You section it will not scrobble.
• If you play from the detail view of a playlist that you previously added to My Music it scrobbles.
• If you play a playlist directly from the For You section it will not scrobble.
• If you play a song, or album that you previously added to My Music from the detail view it will not scrobble.
• If you search for a song, album, or artist and play from the search results it will not scrobble.
• Everything in Radio, including Beats 1 will not scrobble.

Basically, only music in My Music and Playlists will scrobble, with the one exception of saved playlists.

Music on iOS

Scrobbling on iOS is a little bit trickier. Again, you need a third-party software. CloudScrob (€0,99) and QuietScrob (free, with €0,99 In-App purchase), both worked in principal for me, but QuietScrob can run in the background, while you have to open CloudScrob to scrobble to Last.fm. Unlike Simplify on OS X, scrobbling with QuietScrob is not immediate but happens in bulk every 30 minutes (or so).

It looks like the same restrictions as for scrobbling with iTunes and Simplify on OS X apply. However, in addition scrobbling does not work when playing an Apple Music Playlist.

CloudScrob has not been updated since November 2013, but QuietScrob latest update is from May 20, so there is hope that developers will find ways to scrobble additional methods of listening on Apple Music.

I am feredir on Last.fm.

Updates

• July 8, 2015, 18:15: Add note about background scrobbling in QuietScrob
• March 20, 2021, 17:54: Add link to new article on this topic: Last.fm and Apple Music in 2021

The other thing about Apple Music is the software. iTunes is a mess, but I argue that it's a better mess than Spotify on OS X. Spotify doesn't have a lot of keyboard shortcuts, you can't configure additional shortcuts and it doesn't support Apple Script. That might be ok for most users, but for power users iTunes is better. The Music app on iOS is a lot better than iTunes, because it is less cluttered. Spotifys iOS app is also better than their OS X app, but is also has plenty of weird UX. In 2015 I kinda expect that the mobile app is better than desktop version.

When it comes to music, the main advantage of Apple Music is that it combines local and cloud music into one library. I can create playlists, even smart playlists, that can contain songs from both. Spotify has support for local files, but it is primitive and I never used it. However, one thing that Apple has to work on is discovery in My Music. Currently you can only view either Recently Added or All. You can sort that view, but the current options are useless for streaming-size libraries. I am not going to rate hundreds of albums. I don't know what Most Played means in Spotify (because the list definitely does not contain my most played stuff), but Apple Music needs something like this. Not an all time most played/best rated, but more like most played/best rated in the last couple of weeks.

Sometimes I like a song from an album that has not yet been released yet and Apple Music let me add unreleased albums to my library. They become playable once released, until then I can play the single. That is a pretty great feature. (Update: However, currently this feature is buggy. For me, it worked for two albums, but for most albums it does not and I have to remove and read the album to get all tracks into my library.)

I like Beats 1 more than expected. Based on the Beats brand I figured that it will be heavy on rap and beat-based music, but I already found one or two artists that I enjoy. It will not become my primary method of music listening, but I will tune in from time to time.

Scrobbling to Last.fm works. Sometimes. Connect is useless, but you can turn it off.

I have updated the article to reflect that automatically adding music once it is released does currently not work as I described it.

Install it by downloading the latest version and double-clicking the alfred-iterm2-tabs.workflow file. Activate it by using the itt keyword.

Run, Lucifer by Adna (iTunes, Apple Music, Spotify, Amazon)

Before The World Was Big by Girlpool (iTunes, Apple Music, Spotify, Amazon)

Apocalypse, Girl by Jenny Hval (iTunes, Apple Music, Spotify, Amazon)

Short Movie by Laura Marling (iTunes, Apple Music, Spotify, Amazon)

Salome by Marriages (iTunes, Apple Music, Spotify, Amazon)

Natalie Prass by Natalie Prass (iTunes, Apple Music, Spotify, Amazon)

Beneath The Skin by Of Monsters And Men (iTunes, Apple Music, Spotify, Amazon)

Pale Honey by Pale Honey (iTunes, Apple Music, Spotify, Amazon)

Wild Nights by PINS (iTunes, Apple Music, Spotify, Amazon)

Currents by Tame Impala (iTunes, Apple Music, Spotify, Amazon)

Blind As Night by Team Me (iTunes, Apple Music, Spotify, Amazon)

Sprinter by Torres (iTunes, Apple Music, Spotify, Amazon)

I am afraid to see my heroes age

— The Exchange by Torres

Ascent to Loosbühelalm from Großarl.

Sundown at Loosbühelalm.

In the morning just after we left Loosbühelalm for Draugstein.

Selfie at Draugstein

Draugstein front and center and Filzmoossattel at the right

After we came back to Loosbühelalm it rained for a while and was foggy, but then the sky started clearing again.

Looking back at Loosbühelalm after we left on the second morning.

We climbed Gründegg (2168m) and got some pretty light and shadow play.

Summit cross at Gründegg

We made some friends on our way down.

I wanted to replicate the functionality in Ionic, a HTML5 framework to create mobile apps using Cordova.

I am going to show you how to create a ion-search-bar directive as well as the required CSS to replicate this functionality with Ionic. Lets first take a look at the markup that describes our interface

<ion-nav-bar>
  <!-- Nav Bar Stuff -->
</ion-nav-bar>
<ion-search-bar placeholder="Search Books"></ion-search-bar>
<ion-content class="has-header has-search-bar">
  <!-- Content Stuff -->
</ion-content>

This is pretty simple. Please note that I added a has-search-bar class to the ion-content element. Now we need a directive to actually create the search bar. The code pretty straight forward (as straight forward as Angular can be), the important stuff happens in the event handler for focus and blur.

angular
  .module('florian.directives', [])
  // Create a `ion-search-bar` directive
  .directive('ionSearchBar', function ($timeout) {
    return {
      restrict: 'E',
      replace: true,
      scope: { search: '=?filter' },
      link: function (scope, element, attrs) {
        scope.placeholder = attrs.placeholder || '';
        scope.search = { value: '', focus: false };
        if (attrs.class) {
          element.addClass(attrs.class);
        }

        // We need the actual input field to detect focus and blur
        var inputElement = element.find('input')[0];

        // This function is triggered when the user presses the `Cancel` button
        scope.cancelSearch = function () {
          // Manually trigger blur
          inputElement.blur();
          scope.search.value = '';
        };

        // When the user focuses the search bar
        angular.element(inputElement).bind('focus', function () {
          // We store the focus status in the model to show/hide the Cancel button
          scope.search.focus = 1;
          // Add a class to indicate focus to the search bar and the content area
          element.addClass('search-bar-focused');
          angular
            .element(document.querySelector('.has-search-bar'))
            .addClass('search-bar-focused');
          // We need to call `$digest()` because we manually changed the model
          scope.$digest();
        });
        // When the user leaves the search bar
        angular.element(inputElement).bind('blur', function () {
          scope.search.focus = 0;
          element.removeClass('search-bar-focused');
          angular
            .element(document.querySelector('.has-search-bar'))
            .removeClass('search-bar-focused');
        });
      },
      template:
        '<div class="search-bar bar bar-header item-input-inset">' +
        '<label class="item-input-wrapper">' +
        '<i class="icon ion-ios-search placeholder-icon"></i>' +
        '<input type="search" placeholder="" ng-model="search.value">' +
        '</label>' +
        '<button class="button button-clear button-positive" ng-show="search.focus" ng-click="cancelSearch()">' +
        'Cancel' +
        '</button>' +
        '</div>',
    };
  });

All the styling and animation is defined in CSS, or in my case in Sass. If you look at the template code in the directive above you can see that I am (misusing) the bar-header to position the search bar. Therefore we need to move the search bar and the content area a few pixels down in the unfocused state and back up in the focused state. One thing in the code below is still bothering me but I couldn't figure out a nicer solution and that is the margin-right right in button.ng-hide. What happens is that Angular hides the cancel button using display: none but this cannot be animated and I could not figure out another way to move the Cancel button off screen in an animatable way. If you have any suggestions please contact me.

.search-bar {
  top: 44px;
  transition: all ease-in-out 250ms;
  z-index: 15;

  button,
  input {
    transition: all ease-in-out 250ms;
  }

  button.ng-hide {
    margin-right: -67px;
  }

  &.search-bar-focused {
    top: 0;
  }
}

.has-search-bar {
  margin-top: -2px;
  top: 88px;
  transition: top ease-in-out 250ms;

  &.search-bar-focused {
    top: 44px;
  }
}

This code works great in the browser (I tested it in Safari, Chrome and Firefox), but on iOS we need to consider the status bar at the top. We need to add 20px to all the top values and remove the top margin of the input field when the search bar is not focused. As you can see in the very first selector, that code is not required if your app runs in fullscreen.

.platform-ios.platform-cordova:not(.fullscreen) {
  .search-bar.bar-header {
    height: 44px;
    top: 64px;

    &.search-bar-focused {
      height: 64px;
      top: 0;
    }
  }

  .search-bar.bar-header.item-input-inset {
    .item-input-wrapper {
      transition: all ease-in-out 250ms;
      margin-top: 0 !important;
    }
  }

  .search-bar.search-bar-focused.bar-header.item-input-inset {
    .item-input-wrapper {
      margin-top: 19px !important;
    }
  }

  .has-search-bar {
    top: 108px;

    &.search-bar-focused {
      top: 64px;
    }
  }
}

In iOS simulator the interaction looks like this:

If you look closely at the animation and the one above with the original iOS behaviour you can see that I am not using a grey background for the search bar. Personally I like the white background better.

There are many ways to track users across the web, and I want to begin with the most basic one. When you open a website in a browser or when an app you are using is talking to a server the data is most likely transferred over the HTTP protocol (or HTTPS if it is a secure connection). If you are reading this article right now in a web browser you leave something like this on my server:

42.42.42.42 - - [17/Sep/2015:18:10:29 -0400] "GET /articles/thoughts-on-apple-music/ HTTP/1.1" 200 3190 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_5) AppleWebKit/600.8.9 (KHTML, like Gecko) Version/8.0.8 Safari/600.8.9"

The interesting parts, from a privacy perspective are the IP address (the first part) and the user agent (the last part). In the example above the IP address is 42.42.42.42 (obviously not my real one), which is the internet address of my computer (or of my home network). The user agent is an identifier of my browser, each browser and each version has its own user agent. There is a massive amount of different user agents. Because user agents are different based on the operating system you are using, the browser, the version of the browser and sometimes even the plugins you have installed it is possibly to identify people solely on their user agent. The Electronic Frontier Foundation (EFF) has done some research on user agent tracking in 2010.

At this point I should mention that by identifying I don't mean that they know your name, it just means that if you access the site a few days or weeks later they can recognise that you are the same person.

Additionally if you are running JavaScript on a site you can extract even more information. For example, the Screen object in JavaScript contains information about your display, which includes the size of your screen and the pixel density. You can also retrieve the width and the height of the browser window. Combining all these types of information, IP address, user agent, screen sizes and so on you can be identified within a pretty large group of people already and we haven't even started with cookies yet.

Cookies are key-value pairs that a site can leave in your browser. When you visit the same site later it can read the value from the cookie it had written before. This is pretty useful, without the concept of logging into a website would not work. But this technique can also be used to track users across a long time span.

All these techniques described above are useful to develop websites, a web server can't send us the page we requested if it does not know our IP address, without cookies it would not be possible to create websites that require a login and the information about our screen and browser are useful to developers to create a better user experience. But the thing is, a website can include arbitrary resources from other servers. This can be either images, styles, or JavaScript code. And these scripts get access to the same information then the main websites you visited.

Well, expect cookies, because for a few years now most browsers block so called third party cookies, which means that now only the website you are visiting can set a cookie, not images, stylesheets, and scripts that are loaded from a different server. However, advertising is a shady business and advertisers and tracking companies always try to find bugs and tricks to circumvent this. Not only the shady ones, even Google does it. Examples here and here.

The problem I have with advertisement and tracking companies is that they are everywhere. I can absolutely live with the fact the website I am visiting is knowing which pages I access, but the problem is that the same ad servers and trackers are included everywhere and if these companies can get enough identifying data from your browser they can track your browsing history quit accurately. They could know which food you like, if you have financial problems, if you are looking for a partner, and, of course, what kind of porn you like. I am also a little bit scared that some governments have deals with these companies and got a backdoor for this information. Ever since Snowden you can never know. Some may just have some pretty weak security and hackers can get into their servers.

Nowadays we rarely type an URL into the address bar of our browser. Instead we click on links on Facebook or Google and thus we often don't know the site we are browsing to and we don't know which type of trackers they include. And most publishers of content include dozens of them. If you look at news sites or blogs you will often find over twenty different third party resources. The Verge, for example, includes 116 resources from 65 different serves on a single page. Not all of these are trackers or advertisement, but still my information is sent to another server.

Using content blockers does not fully solve this problem, but at least it gives users more control over our data. Currently it looks like that whenever an advertiser is coming to a publisher and saying include this JavaScript on your page the publisher says yes, of course. If more people are starting to use ad blockers publishers will need to think about different ways of monetising their content or about different ways to display advertisement.

Update September 24, 2015: Removed the links to both Peace and Crystal from the article. Peace has been pulled from the App Store and Crystal now accepts payments from advertises to whitelists ads. Doesn't feel good to recommend Crystal anymore.

In the following example we generate an array with 10 elements with a value of 20, then we assign to each element a random number between 0 and the elements value (that is, 20) and then filter the array to only contain odd values.

$arr = array_filter(
    array_map(
        function ($v) { return rand(0, $v); },
        array_fill(0, 10, 20)
    ),
    function ($v) { return $v & 1; }
);

How long do you need to look at the following piece of code to understand it?

echo array_sum(array_intersect(
    array_diff([1, 2, 3, 4, 5], [0, 1, 9]),
    array_filter([2, 3, 4], function ($v) { return !($v & 1); })
));

I use the kind of array operations frequently in other languages, but only very seldom in PHP. Today while taking a walk I thought about this problem and a possible solution. What I came up with is Chain, an object-oriented wrapper for arrays that allow chaining of array manipulations.

First let us take a look at what the above examples look like with Chain. The first one is pretty straight forward, create a filled array, map and filter.

$chain = Chain::fill(0, 10, 20)
    ->map(function ($v) { return rand(0, $v); })
    ->filter(function ($v) { return $v & 1; });

The second example is also pretty straight forward I believe

echo (new Chain([1, 2, 3, 4, 5]))
    ->diff([0, 1, 9])
    ->intersect((new Chain([2, 3, 4]))->filter(function ($v) { return !($v & 1); }))
    ->sum();

By using Chain the information is defined in the order we need to understand it. First we define an array, then we create a diff and intersect it with another chain, which is filtered. Lastly we build the sum of the resulting array.

There exist three kind of methods:

1. methods to create the array,
2. methods to manipulate the array,
3. and methods to retrieve the array, a property of the array or reduce the array to a single value.

Currently you can create a Chain by passing an array to the constructor or by using the static ::fill() method which works exactly like array_fill().

Methods that manipulate an array always modify the internal representation of the array and return an instance to the Chain object (that is, they are chainable). When a method accepts an array as parameter it also accepts an Chain object instead.

The last kind of method are those that return some kind of value, not the object. This includes ->count(), ->sum() and ->reduce(). If you want to retrieve the underlaying array you can use the public property ->array.

Take a look at the Chain project on Github; I started with a very basic implementation this evening and implemented some methods. There are still many array functions missing but I hope I to add them very soon.

If you want to help out I would appreciate pull requests very much. You will see that implementing new methods is very easy. I decided to use traits for this project, mostly because the amount of array functions in PHP would make the Chain class and the associated test class huge. By using traits we have a lot of small classes and a lot of small and simple test classes.

Creating a mock is pretty straight forward with PHPUnit:

$mock = $this->getMockForTrait('TraitUnderTest');
$mock->foo();

In addition you can mock methods of the using object that are called by the mock.

$mock->expects($this->any())
     ->method('isBar')
     ->will($this->returnValue(TRUE));

Source:

• PHPUnit: Mocking Traits and Abstract Classes

There are things I could have done better, especially in the question and answer part, but I am actually pretty proud of it.

Together with my friend Karin we bought Interrail tickets and on the first day we started from Linz to Cologne.

On the second day we took a train to Rotterdam and then a ferry to Harwich in England and spent a night in London.

After a few drinks in Hackney and a good nights sleep we continued to Newcastle-upon-Thyne in the north of England.

On the next morning we finally made our way to Scotland and for the first couple of nights we stayed in Edinburgh.

We continued our journey north and stayed for a night in Pitlochery and took a walk to the nearby town of Killicrankie.