Roman Ring on Roman Ring

The Next Generation of Machine Learning Tools

Wed, 22 Jan 2020 00:00:00 +0000

Introduction

Let’s travel back to a simpler time when all everyone talked about in machine learning were SVMs and boosted trees, while Andrew Ng introduced neural networks as a neat party hat trick you would probably never use in practice¹.

The year is 2012, and computer-vision based competition ImageNet is set to be once again won by the newest ensemble of kernel methods. That is, of course, until a couple of researchers unveiled AlexNet², having almost two times lower error rate than the competition, by using what we now commonly refer to as “deep learning.”

Many people point to AlexNet as one of the most important scientific breakthroughs of the decade, certainly one that helped change the landscape of ML research. However, it does not take much to realize that under the hood, it is “just” a combination of prior iterative improvements, many dating back to the early nineties. At its core, AlexNet is “just” a modified LeNet³ with more layers, better weight initialization, activation function, and data augmentation.

Tools in ML Research

So what made AlexNet stand out so much? I believe the answer lies in the tools researchers had at their disposal, enabling them to run artificial neural networks on GPU accelerators, a relatively novel idea at the time. In fact, Alex Krizhevsky’s former colleagues recall that many meetings before the competition consisted of Alex describing his progress with the CUDA quirks and features.

Now let us travel back to 2015 when ML research article submissions started blowing up across the board, including (re-)emergence of many now promising approaches such as generative adversarial learning, deep reinforcement learning, meta-learning, self-supervised learning, federated learning, neural architecture search, neural differential equations, neural graph networks, and many more.

Image via Charrez, D. (2019).

One could claim that this is just a natural outcome of the AI hype. However, I believe a significant factor was the emergence of the second generation of general-purpose ML frameworks such as TensorFlow⁴ and PyTorch⁵, along with NVIDIA going all-in on AI. The frameworks that existed before, such as Caffe⁶ and Theano⁷, were challenging to work with, and awkward to extend, which slowed down the research and development of novel ideas.

A Need for Innovation

TensorFlow and PyTorch were undoubtedly a net positive, and the teams worked hard to improve the libraries. Recently, they delivered TensorFlow 2.0 with a more straightforward interface along with eager mode⁸, and PyTorch 1.0 with JIT compilation of the computation graph⁹ as well as support for XLA¹⁰ based accelerators such as TPUs¹¹. However, these frameworks are also beginning to reach their limits, forcing researchers into some paths while closing doors on others, just like their predecessors.

High-profile DRL projects such as AlphaStar¹² and OpenAI Five¹³ not only utilized large-scale computational clusters but also pushed the limits of deep learning architecture components by combining deep transformers, nested recurrent networks, deep residual towers, among others.

In his interview with The Times newspaper, Demis Hassabis has stated that DeepMind will be focusing on applying AI directly for scientific breakthroughs. We can already see a glimpse of that shift in direction with some of their recent Nature articles on neuroscience¹⁴ and protein folding¹⁵. Even a brief skim through the publications is enough to see that the projects required some unconventional approaches when it comes to engineering.

At NeurIPS 2019, probabilistic programming and bayesian inference were hot topics, especially uncertainty estimation and causal inference. Leading AI researchers presented their visions on what the future of ML might look like. Notably, Yoshua Bengio described transitioning to system 2 deep learning with out-of-distribution generalization, sparse graph networks, and causal reasoning.

To summarize, some of the requirements for next-gen ML tools are:

fine-grained control flow use
non-standard optimization loops
higher-order differentiation as a first-class citizen
probabilistic programming as a first-class citizen
support for multiple heterogeneous accelerators in one model
seamless scalability from a single machine to gigantic clusters

Ideally, the tools should also maintain a clean, straightforward, and extensible API, enabling scientists to research and develop their ideas rapidly.

The Next Generation

The good news is that many candidates already exist today, emerging in response to the needs in scientific computing. From experimental projects like Zygote.jl¹⁶ to even specialized languages, e.g. Halide¹⁷ and DiffTaichi¹⁸. Interestingly, many projects draw inspiration from the fundamental works done by researchers in the auto-diff community¹⁹, which evolved in parallel to ML.

Many of them were featured at the recent NeurIPS 2019 workshop on program transformations. The two I am most excited about are S4TF²⁰ and JAX²¹. They both tackle the task of making differentiable programming into an integral part of the toolchain, but in their own ways, almost orthogonal to each other.

Swift for TensorFlow

As the name suggests, S4TF tightly integrates the TensorFlow ML framework with the Swift programming language. A vote of confidence for the project is that it is led by Chris Lattner, who has authored LLVM²², Clang²³, and Swift itself.

Swift is a compiled programming language, and one of its primary selling points is the powerful type system that is static and inferred. What the last part means in simpler terms is that Swift encompasses ease of use in languages like Python with code validations and transformations at compile-time, e.g., as in C++.

let a: Int = 1
let b = 2
let c = "3"

print(a + b)         // 3
print(b + c)         // compilation (!) error
print(String(b) + c) // 23

Swift features enable the S4TF team to meet quite a few requirements in the next-generation list by having analysis, verification, and optimization of the computation graph executed with efficient algorithms during compilation.

Crucially, the handling of automatic differentiation is off-loaded to the compiler.

struct Linear: Differentiable {
  var w: Float
  var b: Float

  func callAsFunction(_ x: Float) -> Float {
    return w * x + b
  }
}

let f = Linear(w: 1, b: 2)
let 𝛁f = gradient(at: f) { f in f(3.0) }
print(𝛁f) // TangentVector(w: 3.0, b: 1.0)

let 𝛁f2 = gradient(at: f) { f in f([3.0]) } // compilation (!) error
// error: cannot convert value of type '[Float]' to expected argument type 'Float'

Of course, TensorFlow itself is very well supported in this case.

import TensorFlow

struct Model: Layer {
    var conv = Conv2D<Float>(filterShape: (5, 5, 6, 16), activation: relu)
    var pool = MaxPool2D<Float>(poolSize: (2, 2), strides: (2, 2))
    var flatten = Flatten<Float>()
    var dense = Dense<Float>(inputSize: 16 * 5 * 5, outputSize: 100, activation: relu)
    var logits = Dense<Float>(inputSize: 100, outputSize: 10, activation: identity)

    @differentiable
    func callAsFunction(_ input: Tensor<Float>) -> Tensor<Float> {
        return input.sequenced(through: conv, pool, flatten, dense, logits)
    }
}

var model = Model()
let optimizer = RMSProp(for: model, learningRate: 3e-4, decay: 1e-6)

for batch in CIFAR10().trainDataset.batched(128) {
  let (loss, gradients) = valueWithGradient(at: model) { model in
    softmaxCrossEntropy(logits: model(batch.data), labels: batch.label)
  }
  print(loss)
  optimizer.update(&model, along: gradients)
}

On the other hand, if a critical feature is proving to be difficult to implement, having intimate knowledge of the whole pipeline is particularly valuable. For example, the MLIR compiler framework is a direct result of the S4TF efforts.

While differentiable programming is the core goal, S4TF is much more than that with a plan to support the infrastructure for various next-gen ML tools such as debuggers. For example, imagine an IDE warning a user that the custom model computation always results in a zero gradient without even executing it.

Python has an incredible community built around scientific computing and the S4TF team has explicitly taken the time to embrace it via interoperability.

import Python // All that is necessary to enable the interop.

let np = Python.import("numpy") // Can import any Python module.
let plt = Python.import("matplotlib.pyplot") 

let x = np.arange(0, 10, 0.01)
plt.plot(x, np.sin(x)) // Can use the modules as if inside Python.
plt.show() // Will show the sin plot, just as you would expect.

This project is a significant undertaking and still has some ways to go before being ready for production. However, this is a great time to give it a try for both engineers and researchers and potentially contribute to its development.
Work on S4TF has already produced interesting scientific advancements at the intersection of programming language and auto-diff theory²⁴.

One thing that especially stands out for me about S4TF is their approach to community outreach. For example, the core developers hold weekly design sessions, which are open for anyone interested to join and even participate.

To learn more about Swift for TensorFlow, here are some useful resources:

JAX

JAX is a collection of function transformations such as just-in-time compilation and automatic differentiation, implemented as a thin wrapper over XLA with an API that is essentially a drop-in replacement for NumPy and SciPy. In fact, one way to get started with JAX is to think of it as an accelerator backed NumPy.

import jax.numpy as np

# Will be seamlessly executed on an accelerator such as GPU/TPU.
x, w, b = np.ones((3, 1000, 1000))
y = np.dot(w, x) + b

Of course, in reality, JAX is much more than that. To many, it might seem that the project appeared out of thin air, but the truth is that it is an evolution of over five years of research spanning across three projects. Notably, JAX emerged from Autograd – a research endeavor into AD of native program code – generalizing on its core ideas to support arbitrary transformations.

def f(x):
  return np.where(x > 0, x, x / (1 + np.exp(-x)))

# Note: same singular style for the API entry points.
jit_f = jax.jit(f) # Will be 10-100x faster, depending on the accelerator.
grad_f = jax.grad(f) # Will work as expected, handling both branches.

Aside from the grad and jit discussed above, there are two more excellent examples of JAX transformations, helping users to batch-process their data via auto-vectorization of batch dimension (vmap) or across multiple devices (pmap).

a = np.ones((100, 300))

def g(vec):
  return np.dot(a, vec)

# Suppose `z` is a batch of 10 samples of 1 x 300 vectors.
z = np.ones((10, 300))

g(z) # Will not work due to (batch) dimension mismatch (100x300 x 10x300).

vec_g = jax.vmap(g)
vec_g(z) # Will work, efficiently propagating through batch dimension.

# Manual solution requires "playing" with matrix transpositions.
np.dot(a, z.T)

These features might seem confusing at first, but after some practice, they turn into an irreplaceable part of a researcher’s toolbox. They have even inspired recent development of similar functionality in both TensorFlow and PyTorch.

For the time being, JAX authors seem to be sticking to their core competency when it comes to developing new features. Of course, a reasonable approach but is also the cause for one of its main drawbacks: lack of built-in neural network components, aside from the proof-of-concept Stax.

Adding higher-level features is something where end-users can potentially step in and contribute, and given JAX’s solid foundation, the task might be easier than it seems. For example, there are now two “competing” libraries built on top of JAX, both developed by Google researchers, with differing approaches: Trax and Flax.

# Trax approach is functional.
# Note: params are stored outside and `forward` is "pure".

import jax.numpy as np
from trax.layers import base

class Linear(base.Layer):
  def __init__(self, num_units, init_fn):
    super().__init__()
    self.num_units = num_units
    self.init_fn = init_fn

  def forward(self, x, w):
    return np.dot(x, w)

  def new_weights(self, input_signature):
    w = self.init_fn((input_signature.shape, self._num_units))
    return w

# Flax approach is object-oriented, closer to PyTorch style.

import jax.numpy as np
from flax import nn

class Linear(nn.Module):
  def apply(self, x, num_units, init_fn):
    W = self.param('W', (x.shape[-1], num_units), init_fn)
    return np.dot(x, W)

Even though some might prefer a singular way, endorsed by core developers, having a diversity of methods is a good indicator that the technology is sound.

There are also some directions in research where JAX features especially shine. For example, in meta-learning, one common approach to training a meta-learner is by computing the gradients of the inputs. An alternative method for computing gradients – forward-mode auto-differentiation – is necessary to solve this task efficiently, which is supported out-of-the-box in JAX but is either non-existent or an experimental feature in other libraries.

JAX is perhaps more polished and production-ready than its S4TF counter-part and some of the recent developments coming out of Google Research rely on it, such as Reformer – a memory-efficient Transformer model capable of handling context windows of a million words while fitting on a consumer GPU²⁵, and Neural Tangents – a library for complex neural networks of infinite width²⁶.

The library is further embraced by the broader scientific computing community, used for works in molecular dynamics²⁷, probabilistic programming²⁸, and constrained optimization²⁹, among others.

To get started with JAX and for further reading, please review the following:

Conclusion

ML research is starting to hit the limits of the tools we currently have at our disposal, but some new and exciting candidates are right around the corner, such as JAX and S4TF. If you feel yourself to be more of an engineer than a researcher and wonder whether there is even a place for you at the ML table, hopefully, the answer is clear: right now is the perfect time to get into it. Moreover, you have an opportunity to participate on the ground floor of the next generation of ML tools!

Note that this does not mean TensorFlow or PyTorch are going anywhere, not in the near future. There is still much value in these mature, battle-tested libraries. After all, both JAX and S4TF have parts of TensorFlow under their hoods. But if you are about to start a new research project or if you feel that you are working around library limitations more than on your ideas, then maybe give them a try!

References

Ng, A. (2011). Week 4: Neural Networks. COURSERA: Machine Learning. ↑
Krizhevsky, A., Sutskever, I., & Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In advances in Neural Information Processing Systems. ↑
LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE. ↑
Abadi, M., Barham, P., Chen, J., Chen, Z., Davis, A., Dean, J., … & Kudlur, M. (2016). Tensorflow: A system for large-scale machine learning. In 12th USENIX Symposium on Operating Systems Design and Implementation. ↑
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., … & Desmaison, A. (2019). PyTorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems. ↑
Jia, Y., Shelhamer, E., Donahue, J., Karayev, S., Long, J., Girshick, R., … & Darrell, T. (2014). Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia. ↑
Bergstra, J., Breuleux, O., Bastien, F., Lamblin, P., Pascanu, R., Desjardins, G., … & Bengio, Y. (2010). Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for scientific computing conference. ↑
Agrawal, A., Modi, A. N., Passos, A., Lavoie, A., Agarwal, A., Shankar, A., … & Cai, S. (2019). Tensorflow eager: A multi-stage, python-embedded dsl for machine learning. arXiv preprint arXiv:1903.01855. ↑
Contributors, PyTorch. (2018). Torch script. URL https://pytorch.org/docs/stable/jit.html ↑
Leary, C., & Wang, T. (2017). XLA: TensorFlow, compiled. TensorFlow Dev Summit. ↑
Jouppi, N. P., Young, C., Patil, N., Patterson, D., Agrawal, G., Bajwa, R., … & Boyle, R. (2017). In-datacenter performance analysis of a tensor processing unit. In 44th Annual International Symposium on Computer Architecture. ↑
Vinyals, O., Babuschkin, I., Czarnecki, W. M., Mathieu, M., Dudzik, A., Chung, J., … & Silver, D. (2019). Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature. doi:10.1038/s41586-019-1724-z ↑
Berner, C., Brockman, G., Chan, B., Cheung, V., Dębiak, P., Dennison, C., … & Józefowicz, R. (2019). Dota 2 with Large Scale Deep Reinforcement Learning. arXiv preprint arXiv:1912.06680. ↑
Dabney, W., Kurth-Nelson, Z., Uchida, N., Starkweather, C. K., Hassabis, D., Munos, R., & Botvinick, M. (2020). A distributional code for value in dopamine-based reinforcement learning. Nature. doi: 10.1038/s41586-019-1924-6 ↑
Senior, A., Evans, R., Jumper, J., Kirkpatrick, J., Sifre, L., Green, T., … & Penedones, H. (2020). Improved protein structure prediction using potentials from deep learning. Nature. ↑
Innes, M. (2018). Don’t Unroll Adjoint: Differentiating SSA-Form Programs. arXiv preprint arXiv:1810.07951. ↑
Ragan-Kelley, J., Barnes, C., Adams, A., Paris, S., Durand, F., & Amarasinghe, S. (2013). Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. In ACM Sigplan Notices. ↑
Hu, Y., Anderson, L., Li, T. M., Sun, Q., Carr, N., Ragan-Kelley, J., & Durand, F. (2019). DiffTaichi: Differentiable Programming for Physical Simulation. arXiv preprint arXiv:1910.00935. ↑
Baydin, A. G., Pearlmutter, B. A., Radul, A. A., & Siskind, J. M. (2017). Automatic differentiation in machine learning: a survey. The Journal of Machine Learning Research. ↑
Wei, R., & Zheng, D. (2018). Swift for TensorFlow. URL https://github.com/tensorflow/swift. ↑
Bradbury, J., Frostig, R., Hawkins, P., Johnson, M. J., Leary, C., Maclaurin, D., & Wanderman-Milne, S (2018). JAX: composable transformations of Python+ NumPy programs. URL https://github.com/google/jax. ↑
Lattner, C. (2002). LLVM: An infrastructure for multi-stage optimization. Masters thesis, University of Illinois. ↑
Lattner, C. (2008). LLVM and Clang: Next generation compiler technology. In The BSD conference. ↑
Vytiniotis, D., Belov, D., Wei, R., Plotkin, G., & Abadi, M. (2019). The Differentiable Curry. ↑
Kitaev, N., Kaiser, L., and Levskaya, A. (2020). Reformer: The Efficient Transformer. In International Conference on Learning Representations. ↑
Novak, R., Xiao, L., Hron, J., Lee, J., Alemi, A., Sohl-dickstein, J., & Schoenholz, S. (2020). Neural Tangents: Fast and Easy Infinite Neural Networks in Python. In International Conference on Learning Representations. ↑
Schoenholz, S., & Cubuk, E. (2020). JAX, MD End-to-End Differentiable, Hardware Accelerated, Molecular Dynamics in Pure Python. Bulletin of the American Physical Society. ↑
Phan, D., Pradhan, N., & Jankowiak, M. (2019). Composable Effects for Flexible and Accelerated Probabilistic Programming in NumPyro. ↑
Bacon, P. L., Schäfer, F., Gehring, C., Anandkumar, A., & Brunskill, E. A Lagrangian Method for Inverse Problems in Reinforcement Learning. In Advances in Neural Information Processing Systems. ↑

StarCraft II: Next Big Thing in AI

Thu, 24 Jan 2019 00:00:00 +0000

Couple of days ago out of the blue DeepMind announced a StarCraft II related event, with many of the employees being quite excited about it on twitter. In just a few hours we will see what DeepMind has in store for us, but in the meantime let’s take a step back and review how we got here and why it is so important.

Update: the (amazing) event has finished and DeepMind have released a very detailed write-up. See below for my thoughts on the event and the write-up.

DeepMind

At the end of 2013, DeepMind was a small and relatively unknown startup, founded three years prior with one simple goal: “to solve intelligence”. Although not yet famous with the general public, they have made a lot of noise in the world of deep reinforcement learning by introducing DQN - an algorithm, capable of matching or even surpassing humans at playing the Atari video games, while observing the game from raw pixels similar to how a human would. In the early 2014 DeepMind was bought by Google Inc. for $500 million, many speculating that deal was ensured by this one achievement.

Atari game of Breakout, played by DQN agent. Source.

In March of 2016 DeepMind did it again, with their AlphaGo program decisively winning against Go world champion Lee Sedol, who held the title for a great number of years. While in hindsight this seems as a sure thing, three years ago this win came almost out of nowhere, with many researchers at the time were predicting that AI will not match human experts for another 10 years at least.

DeepMind's AlphaGo winning vs Lee Sedol. Source.

StarCraft II

Shortly after DeepMind’s AlphaGo victory, DeepMind were already prepared to announce the next challenge they will be tackling: the StarCraft II video game. StarCraft II is a real-time strategy game that requires making split second decisions on strategical, tactical, and economical levels, with short-term and long-term goals, throughout the whole duration of the game. Together with its predecessor StarCraft: Brood War, it has a rich competitive e-sports history spanning over 20 years, and is actively played by millions to this day.

Serral securing his world championship title at WCS 2018. Source.

With much bigger state and action spaces, real-time component, partial observability, and longer game duration, this is a much more difficult task than any attempted before. In fact many AI researchers have actually attempted to tackle it, notably the StarCraft: Brood War bot OverMind was a collaborative effort of many talented scientists. Yet, when pitted against human players, the AI champion would be easily defeated even by relative novices to the game.

This massive challenge alone wasn’t enough for DeepMind researchers, they decided to go one step further. They wanted to not only challenge human experts, but do so “on their terms”. The AI will be observing the game in a similar way to humans, through image-like features, and will be acting by emulating keyboard and mouse commands as close as possible.

On the right: StarCraft II game as seen by the AI. Source.

Furthermore, the AI is limited in many aspects to match human experiences, e.g. it is allowed to take a limited number of actions per minute (APM) and can only access the same information a human would see on the screen or minimap. This means that the AI must learn to use in-game camera in order to effectively play.

StarCraft II AI emulates actions as close to humans as possible. Source.

The Next Big Thing in AI

A StarCraft II player might wonder what is special about this if there were relatively good AIs built into the game from release. The AI you encounter in-game acts based on some pre-defined set of rules, which means it can only play as well as the person who programmed it, reacting only to foreseen events. This makes the AI inflexible, and easy to exploit, among other things.

In contrast, the AI DeepMind is working on learns the game by itself, essentially from scratch. At each step all it has is the game state it observed, the actions it can take, and some reward stimulus it received for a previous action. In general the approach is called reinforcement learning and here I give a brief overview of the field.

To get a feel for what it’s like for an RL agent to “learn” the game, here is a video of an agent attempting to tackle a set of minigames released by DeepMind. On the left you can see an agent that just started the process and is exploring all the various actions it can take, and on the right is the same agent after a fair bit of such experiments (about 50 million steps on average).

An outside reader might wonder why AI researchers focus so much on games to begin with. The answer is simple: real-world is just too complex for an AI system at this point - one must first learn to walk before he gets to fly. Specifically, games have easily defined rules that can be described to an AI system and are typically easy to run in parallel on a computer, speeding up the learning process.

This however does not mean that current work will be useless in the future. For example, while “playing” with StarCraft II, DeepMind researchers produced a number of general-purpose results such as the population based training, and relational deep reinforcement learning. So while developing an AI capable of winning in StarCraft II with human-like restrictions probably won’t give us AGI, it will definitely bring us an inch closer.

Predictions and Speculations

In this section I will attempt to predict some of the things we will see at the event, along with the nitty-gritty details “under the hood” of the AI, which might get a bit technical. Note that I have no insider information, this is simply my guesses.

For the event itself I believe it will be a Protoss vs Protoss showmatch between DeepMind’s AI and some high level player, either ex-pro or possibly even current pro, however not best of the best tier. I would be really surprised if they are already at a level to challenge world champions.

The core algorithm they will employ will be an actor-critic variant called IMPALA, fused with attention mechanism as described in Relational DRL. Network architecture will have residual + convolutional state encoder component and a number of Conv LSTM blocks generating the policy. It will also have a fair bit of imitation learning based pre-training prior to the DRL loop.

The IMPALA algorithm is essentially a distributed, asyncronous version of A2C I’ve described in this blog with a number of fixes (e.g. importance weighting) to address its “off-policyness” - the fact that samples are gathered with a different policy than the one being currently trained. If you’re familiar with openAI Five then the core idea is similar to their asyncronous PPO: a (massively) distributed advantage actor-critic variant with tricks to correct for the distributional drift.

The attention mechanism has revolutionized the world of NLP, especially after the “Attention is All You Need” article came out. In short, if we’re dealing with sequential input / output then attention mechanism acts as importance weight for items in the given sequence when generating the next output. In NLP this is most often used for machine translation to give attention to words at different parts of the sentence. In StarCraft II this could be used to ensure the agent can quickly switch between tactical and strategical decision making. DeepMind has recently applied it to DRL in their Relational DRL article.

The residual + convolutional architecture makes sense to use given our state space, whereas Conv LSTM cells (NB! different from CNN LSTM) are starting to gain traction in DRL world, especially when observations are rich with spatial information.

During Blizzcon 2018 presentation Oriol Vinyals mentioned that they’ve used imitation learning for their agent - and it of course make sense, given that Blizzard is providing free access (to everybody!) to their massive dataset of replays. Of course it would be impossible to succesfully train a full agent from replays alone, but I think it’s reasonable to use it for NN weights pre-training.

I also wouldn’t be surprised if instead of fully end-to-end approach DeepMind will rely on something modular, e.g. as described here, where individual modules are responsible for specific subsets of the game. Specifically, perhaps DeepMind relies on pre-defined rulesets for scouting and makes use of a simplified game engine for battle simulations that are used in MCTS solver to determine whether they should attack or fall back.

Conclusion

Hopefully you’re now as excited as I am about the upcoming event. Whether the AI is fully end to end or not and whether they will be able to win vs human experts or not, it is still a massive endeavor and should provide for a good show. And if you’d like to get into developing StarCraft II based AIs yourself, join our SC2AI community on discord!

Post-Event Write-Up

The stream was very exciting to watch, both as a player and a researcher. It was funny to see AlphaStar opt for wild strategies and then come out on top. It was also interesting to see it make mistakes such as killing its own units, very AI and human-like behavior at the same time.

To me the most impressive was the micro, and not just due to its ability to make split second decisions. The way AlphaStar knew how to pull back damaged stalkers to regenerate shields, the way it pulled its workers when it saw Oracles - really mind boggling that a single end-to-end neural network is capable of such a rich variety of tactical and somewhat long-term decision-making.

However, AlphaStar is still quite far from conquering StarCraft II universe. First, while Mana is no doubt a great player, he is not quite world champion caliber. Second, this is still a single matchup, whereas any human player would be expected to play vs all three races on the same level. Third, I’m not sure how I feel about having players go against a pool of AlphaStar(s) - I think it definitely makes sense to use for training, but during inference I’d prefer to see a single version used throughout the matches. Overall I would say AlphaStar right now is closer to the AlphaGo version that played vs Fan Hui than the one that won vs Lee Sedol.

Seems that I’ve correctly predicted the match-up and level of play, along with some of the approaches. Specifically, AlphaStar does indeed rely on imitation learning, IMPALA, and attention mechanism, though not quite as described in Relational DRL article. They also indeed use LSTM, but I am not so sure with regards to convolutional layers - there seems to be a bit of confusion as to what interface they ended up using. I’ve also briefly mentioned population based training - seems that DeepMind uses an advanced variant of it, hopefully we will see an article about it soon.

Of the things I’ve missed is the transformer body, which is a state-of-the-art architecture in machine translation. Very surprised to see it applied in DRL. They also use a relatively novel baseline for the advantage function in the PG loss, which they pulled from the Counterfactual Multi-Agent Policy Gradients article. This is noteworthy because for the longest time the value estimate of next state for baseline was the go-to approach of pretty much everybody in DRL.

Finally, they apply pointer network to the policy output, most likely as an efficient way to deal with variable length of action arguments. To me the use of pointer networks was quite surprising and somewhat ironic - I have actually written an essay on this article and while it was an interesting subject, it never crossed my mind it could be applied in such a way to DRL policies. Although in retrospect I guess it makes sense.

During training they also rely on self-imitation and experience replay, which is quite interesting - seems they have finally perfected the combination of actor-critic methods, which are traditionally seen as on-policy, with the benefits of off-policy algorithms. Finally, they use policy distillation which is probably how they were able to fit the final agents into a single machine for inference.

If by now your head is spinning from all the terminology, don’t worry - mine is too. The takeaway message is that it took an impressive amount of very advanced approaches to achieve the level of play we have seen today and I am curious to see what happens next.

Deep Reinforcement Learning With TensorFlow 2.1

Sun, 20 Jan 2019 00:00:00 +0000

Introduction

In this tutorial, I will give an overview of the TensorFlow 2.x features through the lens of deep reinforcement learning (DRL) by implementing an advantage actor-critic (A2C) agent, solving the classic CartPole-v0 environment. While the goal is to showcase TensorFlow 2.x, I will do my best to make DRL approachable as well, including a birds-eye overview of the field.

In fact, since the main focus of the 2.x release is making life easier for the developers, it’s a great time to get into DRL with TensorFlow. For example, the source code for this blog post is under 150 lines, including comments!
Code is available on GitHub here and as a notebook on Google Colab here.

Setup

To follow along, I recommend setting up a separate (virtual) environment.
I prefer Anaconda, so I’ll illustrate with it:

> conda create -n tf2 python=3.7
> conda activate tf2
> pip install tensorflow=2.1

Let us quickly verify everything works as expected:

>>> import tensorflow as tf
>>> print(tf.__version__)
2.1.0

Note that we are now in eager mode by default!

>>> print(tf.executing_eagerly())
True
>>> print("1 + 2 + 3 + 4 + 5 =", tf.reduce_sum([1, 2, 3, 4, 5]))
1 + 2 + 3 + 4 + 5 = tf.Tensor(15, shape=(), dtype=int32)

If you are not yet familiar with eager mode, then, in essence, it means that computation executes at runtime, rather than through a pre-compiled graph. You can find a good overview in the TensorFlow documentation.

GPU Support

One great thing about specifically TensorFlow 2.1 is that there is no more hassle with separate CPU/GPU wheels! TensorFlow now supports both by default and targets appropriate devices at runtime.

The benefits of Anaconda are immediately apparent if you want to use a GPU. Setup for all the necessary CUDA dependencies is just one line:

> conda install cudatoolkit=10.1

You can even install different CUDA toolkit versions in separate environments!

Reinforcement Learning

Generally speaking, reinforcement learning is a high-level framework for solving sequential decision-making problems. An RL agent navigates an environment by taking actions based on some observations, receiving rewards as a result. Most RL algorithms work by maximizing the expected total rewards an agent collects in a trajectory, e.g., during one in-game round.

The output of an RL algorithm is a policy – a function from states to actions.
A valid policy can be as simple as a hard-coded no-op action, but typically it represents a conditional probability distribution of actions given some state.

Figure: A general diagram of the RL training loop.
Image via Stanford CS234 (2019).

RL algorithms are often grouped based on their optimization loss function.

Temporal-Difference methods, such as Q-Learning, reduce the error between predicted and actual state(-action) values.

Policy Gradients directly optimize the policy by adjusting its parameters. Calculating gradients themselves is usually infeasible; instead, they are often estimated via monte-carlo methods.

The most popular approach is a hybrid of the two: actor-critic methods, where policy gradients optimize agent’s policy, and the temporal-difference method is used as a bootstrap for the expected value estimates.

Deep Reinforcement Learning

While much of the fundamental RL theory was developed on the tabular cases, modern RL is almost exclusively done with function approximators, such as artificial neural networks. Specifically, an RL algorithm is considered deep if the policy and value functions are approximated with neural networks.

Figure: DRL implies ANN is used in the agent’s model.
Image via Mohammadi et al (2018).

(Asynchronous) Advantage Actor-Critic

Over the years, several improvements were added to address sample efficiency and stability of the learning process.

First, gradients are weighted with returns: a discounted sum of future rewards, which resolves theoretical issues with infinite timesteps, and mitigates the credit assignment problem – allocate rewards to the correct actions.

Second, an advantage function is used instead of raw returns. Advantage is formed as the difference between the returns and some baseline, which is often the value estimate, and can be thought of as a measure of how good a given action is compared to some average.

Third, an additional entropy maximization term is used in the objective function to ensure the agent sufficiently explores various policies. In essence, entropy measures how random a given probability distribution is. For example, entropy is highest in the uniform distribution.

Finally, multiple workers are used in parallel to speed up sample gathering while helping decorrelate them during training, diversifying the experiences an agent trains on in a given batch.

Incorporating all of these changes with deep neural networks, we arrive at the two of the most popular modern algorithms: (asynchronous) advantage actor critic, or A3C/A2C for short. The difference between the two is more technical than theoretical. As the name suggests, it boils down to how the parallel workers estimate their gradients and propagate them to the model.

Image via Juliani A. (2016).

With this, we wrap up our tour of the DRL methods and move on to the focus of the blog post is more on the TensorFlow 2.x features. Don’t worry if you’re still unsure about the subject; things should become clearer with code examples.
If you want to learn more, one excellent resource is Spinning Up in Deep RL.

Advantage Actor-Critic With TensorFlow 2.1

Now that we are more or less on the same page, let’s see what it takes to implement the basis of many modern DRL algorithms: an actor-critic agent, described in the previous section. Without parallel workers (for simplicity), though most of the code would be the same.

As a testbed, we are going to use the CartPole-v0 environment. Somewhat simplistic, it is still a great option to get started. In fact, I often rely on it as a sanity check when implementing RL algorithms.

Policy & Value Models via Keras API

First, we create the policy and value estimate NNs under a single model class:

import numpy as np
import tensorflow as tf
import tensorflow.keras.layers as kl


class ProbabilityDistribution(tf.keras.Model):
  def call(self, logits, **kwargs):
    # Sample a random categorical action from the given logits.
    return tf.squeeze(tf.random.categorical(logits, 1), axis=-1)


class Model(tf.keras.Model):
  def __init__(self, num_actions):
    super().__init__('mlp_policy')
    # Note: no tf.get_variable(), just simple Keras API!
    self.hidden1 = kl.Dense(128, activation='relu')
    self.hidden2 = kl.Dense(128, activation='relu')
    self.value = kl.Dense(1, name='value')
    # Logits are unnormalized log probabilities.
    self.logits = kl.Dense(num_actions, name='policy_logits')
    self.dist = ProbabilityDistribution()

  def call(self, inputs, **kwargs):
    # Inputs is a numpy array, convert to a tensor.
    x = tf.convert_to_tensor(inputs)
    # Separate hidden layers from the same input tensor.
    hidden_logs = self.hidden1(x)
    hidden_vals = self.hidden2(x)
    return self.logits(hidden_logs), self.value(hidden_vals)

  def action_value(self, obs):
    # Executes `call()` under the hood.
    logits, value = self.predict_on_batch(obs)
    action = self.dist.predict_on_batch(logits)
    # Another way to sample actions:
    #   action = tf.random.categorical(logits, 1)
    # Will become clearer later why we don't use it.
    return np.squeeze(action, axis=-1), np.squeeze(value, axis=-1)

And verify the model works as expected:

import gym

env = gym.make('CartPole-v0')
model = Model(num_actions=env.action_space.n)

obs = env.reset()
# No feed_dict or tf.Session() needed at all!
action, value = model.action_value(obs[None, :])
print(action, value) # [1] [-0.00145713]

Things to note here:

Model layers and execution path are defined separately
There is no “input” layer; model accepts raw numpy arrays
Two computation paths can exist in one model via functional API
A model can contain helper methods such as action sampling
In eager mode, everything works from raw numpy arrays

Agent Interface

Now we can move on to the fun stuff – the agent class. First, we add a test method that runs through a full episode, keeping track of the rewards.

class A2CAgent:
  def __init__(self, model):
    self.model = model

  def test(self, env, render=True):
    obs, done, ep_reward = env.reset(), False, 0
    while not done:
      action, _ = self.model.action_value(obs[None, :])
      obs, reward, done, _ = env.step(action)
      ep_reward += reward
      if render:
        env.render()
    return ep_reward

Now we can check how much the agent scores with randomly initialized weights:

agent = A2CAgent(model)
rewards_sum = agent.test(env)
print("%d out of 200" % rewards_sum) # 18 out of 200

Not even close to optimal, time to get to the training part!

Loss / Objective Function

As I have described in the RL section, an agent improves its policy through gradient descent based on some loss (objective) function. In the A2C algorithm, we train on three objectives: improve policy with advantage weighted gradients, maximize the entropy, and minimize value estimate errors.

import tensorflow.keras.losses as kls
import tensorflow.keras.optimizers as ko


class A2CAgent:
  def __init__(self, model, lr=7e-3, value_c=0.5, entropy_c=1e-4):
    # Coefficients are used for the loss terms.
    self.value_c = value_c
    self.entropy_c = entropy_c

    self.model = model
    self.model.compile(
      optimizer=ko.RMSprop(lr=lr),
      # Define separate losses for policy logits and value estimate.
      loss=[self._logits_loss, self._value_loss])

  def test(self, env, render=False):
    # Unchanged from the previous section.
    ...

  def _value_loss(self, returns, value):
    # Value loss is typically MSE between value estimates and returns.
    return self.value_c * kls.mean_squared_error(returns, value)

  def _logits_loss(self, actions_and_advantages, logits):
    # A trick to input actions and advantages through the same API.
    actions, advantages = tf.split(actions_and_advantages, 2, axis=-1)

    # Sparse categorical CE loss obj that supports sample_weight arg on `call()`.
    # `from_logits` argument ensures transformation into normalized probabilities.
    weighted_sparse_ce = kls.SparseCategoricalCrossentropy(from_logits=True)

    # Policy loss is defined by policy gradients, weighted by advantages.
    # Note: we only calculate the loss on the actions we've actually taken.
    actions = tf.cast(actions, tf.int32)
    policy_loss = weighted_sparse_ce(actions, logits, sample_weight=advantages)

    # Entropy loss can be calculated as cross-entropy over itself.
    probs = tf.nn.softmax(logits)
    entropy_loss = kls.categorical_crossentropy(probs, probs)

    # We want to minimize policy and maximize entropy losses.
    # Here signs are flipped because the optimizer minimizes.
    return policy_loss - self.entropy_c * entropy_loss

And we are done with the objective functions! Note how compact the code is: there are almost more comment lines than code itself.

The Training Loop

Finally, there is the train loop itself. It is relatively long, but fairly straightforward: collect samples, calculate returns and advantages, and train the model on them.

class A2CAgent:
  def __init__(self, model, lr=7e-3, gamma=0.99, value_c=0.5, entropy_c=1e-4):
    # `gamma` is the discount factor
    self.gamma = gamma
    # Unchanged from the previous section.
    ...
  
  def train(self, env, batch_sz=64, updates=250):
    # Storage helpers for a single batch of data.
    actions = np.empty((batch_sz,), dtype=np.int32)
    rewards, dones, values = np.empty((3, batch_sz))
    observations = np.empty((batch_sz,) + env.observation_space.shape)

    # Training loop: collect samples, send to optimizer, repeat updates times.
    ep_rewards = [0.0]
    next_obs = env.reset()
    for update in range(updates):
      for step in range(batch_sz):
        observations[step] = next_obs.copy()
        actions[step], values[step] = self.model.action_value(next_obs[None, :])
        next_obs, rewards[step], dones[step], _ = env.step(actions[step])

        ep_rewards[-1] += rewards[step]
        if dones[step]:
          ep_rewards.append(0.0)
          next_obs = env.reset()
          logging.info("Episode: %03d, Reward: %03d" % (
            len(ep_rewards) - 1, ep_rewards[-2]))

      _, next_value = self.model.action_value(next_obs[None, :])

      returns, advs = self._returns_advantages(rewards, dones, values, next_value)
      # A trick to input actions and advantages through same API.
      acts_and_advs = np.concatenate([actions[:, None], advs[:, None]], axis=-1)

      # Performs a full training step on the collected batch.
      # Note: no need to mess around with gradients, Keras API handles it.
      losses = self.model.train_on_batch(observations, [acts_and_advs, returns])

      logging.debug("[%d/%d] Losses: %s" % (update + 1, updates, losses))

    return ep_rewards

  def _returns_advantages(self, rewards, dones, values, next_value):
    # `next_value` is the bootstrap value estimate of the future state (critic).
    returns = np.append(np.zeros_like(rewards), next_value, axis=-1)

    # Returns are calculated as discounted sum of future rewards.
    for t in reversed(range(rewards.shape[0])):
      returns[t] = rewards[t] + self.gamma * returns[t + 1] * (1 - dones[t])
    returns = returns[:-1]

    # Advantages are equal to returns - baseline (value estimates in our case).
    advantages = returns - values

    return returns, advantages

  def test(self, env, render=False):
    # Unchanged from the previous section.
    ...

  def _value_loss(self, returns, value):
    # Unchanged from the previous section.
    ...

  def _logits_loss(self, actions_and_advantages, logits):
    # Unchanged from the previous section.
    ...

Results

We are now all set to train our single-worker A2C agent on CartPole-v0! The training process should take a couple of minutes. After the training is complete, you should see an agent achieve the target 200 out of 200 score.

rewards_history = agent.train(env)
print("Finished training, testing...")
print("%d out of 200" % agent.test(env)) # 200 out of 200

In the source code, I include some additional helpers that print out running episode rewards and losses, along with basic plotter for the rewards history.

Static Computational Graph

With all of this eager mode excitement, you might wonder if using a static graph is even possible anymore. Of course, it is! And it takes just one line!

with tf.Graph().as_default():
  print(tf.executing_eagerly()) # False

  model = Model(num_actions=env.action_space.n)
  agent = A2CAgent(model)

  rewards_history = agent.train(env)
  print("Finished training, testing...")
  print("%d out of 200" % agent.test(env)) # 200 out of 200

There is one caveat though: during static graph execution, we can not just have Tensors laying around, which is why we needed that trick with the separate ProbabilityDistribution model definition. In fact, while I was looking for a way to execute in static mode, I discovered one interesting low-level detail about models built through the Keras API…

One More Thing…

Remember when I said TensorFlow runs in eager mode by default, even proving it with a code snippet? Well, I lied! Kind of.

If you use Keras API to build and manage your models, then it attempts to compile them as static graphs under the hood. So what you end up with is the performance of static graphs with the flexibility of eager execution.

You can check the status of your model via the model.run_eagerly flag. You can also force eager mode by manually setting it, though most of the times you probably don’t need to – if Keras detects that there is no way around eager mode, it backs off on its own.

To illustrate that it is running as a static graph here is a simple benchmark:

# Generate 100k observations to run benchmarks on.
env = gym.make('CartPole-v0')
obs = np.repeat(env.reset()[None, :], 100000, axis=0)

Eager Benchmark

%%time

model = Model(env.action_space.n)
model.run_eagerly = True

print("Eager Execution:  ", tf.executing_eagerly())
print("Eager Keras Model:", model.run_eagerly)

_ = model(obs)

######## Results #######

Eager Execution:   True
Eager Keras Model: True
CPU times: user 639 ms, sys: 736 ms, total: 1.38 s

Static Benchmark

%%time

with tf.Graph().as_default():
    model = Model(env.action_space.n)

    print("Eager Execution:  ", tf.executing_eagerly())
    print("Eager Keras Model:", model.run_eagerly)

    _ = model.predict_on_batch(obs)

######## Results #######

Eager Execution:   False
Eager Keras Model: False
CPU times: user 793 ms, sys: 79.7 ms, total: 873 ms

Default Benchmark

%%time

model = Model(env.action_space.n)

print("Eager Execution:  ", tf.executing_eagerly())
print("Eager Keras Model:", model.run_eagerly)

_ = model.predict_on_batch(obs)

######## Results #######

Eager Execution:   True
Eager Keras Model: False
CPU times: user 994 ms, sys: 23.1 ms, total: 1.02 s

As you can see, eager mode is behind static, and by default, our model was indeed executed statically, almost matching the explicitly static execution.

Conclusion

Hopefully, this has been an illustrative tour of both DRL and the shiny new things in TensorFlow 2.x. Note that many of the design choice discussions are open to the public, and everything is subject to change. If there is something about TensorFlow, you especially dislike (or like :) ), let the developers know!

A lingering question people might have is if TensorFlow is better than PyTorch? Maybe. Maybe not. Both are excellent libraries, so it is hard to say one way or the other. If you are familiar with PyTorch, you probably noticed that TensorFlow 2.x has caught up and arguably avoided some of the PyTorch API pitfalls.

At the same time, I think it would be fair to say that PyTorch was affected by the design choices of TensorFlow. What is clear is that this “competition” has resulted in a net-positive outcome for both camps!

Reaver: Modular Deep Reinforcement Learning

Sat, 05 Jan 2019 00:00:00 +0000

Starter Agent for Coders Strike Back AI Challenge

Sat, 05 Jan 2019 00:00:00 +0000

Why I Majored in Statistics for a Career in Artificial Intelligence

Sat, 05 Jan 2019 00:00:00 +0000

Undergraduate major is often the first significant career decision a person makes in his life. As artificial intelligence (AI) becomes more and more ingrained in our society, many people begin to consider a career in AI as a viable choice in their life. However, it is still very rare to have an undergraduate degree fully dedicated to AI, so people opt for what they perceive to be the next best thing - computer science. But I believe there is a better alternative: statistics, and in this blog post I will try to explain why, based on my own example.

In the recent years AI and its many subfields like machine learning (ML) have exploded in popularity and are on track to pretty much take over every industry out there. And people have noticed: the introductory course Machine Learning (CS229) at Stanford had over a thousand students enrolled in the fall of 2017!

Stanford's first day of class--record-breaking 1040 people already enrolled for on-campus Machine Learning (CS229). Wow! @danboneh
— Andrew Ng (@AndrewYNg) September 25, 2017

Well, I want to let you in on a little secret: majority of AI/ML, including the hip new trend you’ve probably heard about called deep learning, is just applied statistics in disguise: many ML techniques and algorithms are either fully borrowed from or heavily rely on the theory from statistics.

Unfortunately, statistics is lacking in their PR department, which causes many people to misunderstand the field. When I said I am majoring in statistics, many of my peers reaction was confusion at best, with some even assuming I was joking. In fact I was often looked down upon by both maths and CS majors: mathematicians considered stats to be not “pure” enough, whereas CS people thought it’s not engineering-oriented enough. What’s funny is that I actually agree with both of those camps, but I believe those to be pros rather than cons. So let’s review some of the core subjects I’ve taken during my undergraduate studies and how they have helped me with AI/ML.

Mathematical Analysis. You’ve probably heard of or even taken the “practical” alternative to it - Calculus, which is an okay subject, but in my opinion by not focusing on the theory behind the various theorems and lemmas a student never actually builds an intuitive understanding. And boy does it help in AI/ML. The topics at the heart of MA - continuity and differentiability, are also what is behind most of AI/ML algorithms.

Probability Theory and Statistics. Again, some version of this subject is likely taught in CS degrees as well, but theory is typically avoided. I’ve had no trouble diving head first into reinforcement learning thanks to deep and intuitive understanding of random variables and their estimates, expectations, distributions, and so on.

Numerical Methods. Speaking of what is behind most of AI/ML, this subject tackles the questions of function optimization and approximation. And if function approximation sounds alien to you, then perhaps you’ve heard of its special case - artificial neural networks.

Matrix Calculus. And while we’re on the subject of artificial neural networks, you’ve probably heard that they are represented as a chain of differentiable matrix operations. Well, here is a whole subject dedicated to understanding how to transfer your multivariate differentiation theory into the world of linear algebra.

Monte Carlo Methods. Have you ever wondered how probability theory is applied in practice? How can your computer generate random variables from any distribution? Well, this subject covers this and much more. And if you are into reinforcement learning then this course is probably the most important one to take as it covers a large chunk of the theory behind it. For example, the REINFORCE family of algorithms are built on the monte carlo methods.

Stochastic Processes. Speaking of theory behind RL, here’s a whole subject dedicated to dealing with probability distributions over time. Markov Chains, Renewals, Queues, Brownian Motions, Gaussian Processes, …

Data Analysis. Did I mention that majority of machine learning is actually applied statistics? This course intimately covers the theory behind what people would refer to as classical ML - from simple linear regression to generalized models.

Experimental Design. As we are starting to reach the limitations of our hardware the various ML/RL experiments become increasingly expensive in terms of wall-clock time. More and more people are now looking for ways to extract similar quality of information with less effort. Well, statisticians have been working on this problem for decades and you can learn all about it in this course.

But what about programming, you might ask? Well, with a balanced curriculum you actually get quite a fair share of computer science. In fact, with a couple of good elective choices you can cover most of the fundamental knowledge necessary to work as a software engineer if you ever wanted to switch. For example here are the CS courses I have had in my undergrad:

Intro to Programming (Python)
Object Oriented Programming (Java)
Algorithms and Data Structures (Java)
Database Systems (SQL)
Operating Systems (Python / Java)
Programming Languages (Prolog, Haskell, Scala, OCaml, C)

So where’s the catch? Well, the problem with having a subject that is so widely misunderstood and unpopular is that the recruiters looking at your resume might assume you’re one of those hippies that prefers pen & paper to a keyboard and pass you over for a “safe” computer science guy. Unfortunately to get around this I think it’s inevitable that you still have to get the desired stamp, either via double major or with a computer science focused masters degree.

In conclusion, I believe statistics to be the perfect major for a career in AI. As I am wrapping up my first semester of computer science masters I feel that I am often quite ahead of my peers specifically because of my undergraduate background and hopefully I have persuaded some of you to give it a shot!

Roman Ring on Roman Ring

The Next Generation of Machine Learning Tools

Table of Contents

Introduction

Tools in ML Research

A Need for Innovation

The Next Generation

Swift for TensorFlow

JAX

Conclusion

References

StarCraft II: Next Big Thing in AI

DeepMind

StarCraft II

The Next Big Thing in AI

Predictions and Speculations

Conclusion

Post-Event Write-Up

Deep Reinforcement Learning With TensorFlow 2.1

Table of Contents

Introduction

Setup

GPU Support

Reinforcement Learning

Deep Reinforcement Learning

(Asynchronous) Advantage Actor-Critic

Advantage Actor-Critic With TensorFlow 2.1

Policy & Value Models via Keras API

Agent Interface

Loss / Objective Function

The Training Loop

Results

Static Computational Graph

One More Thing…

Conclusion

Reaver: Modular Deep Reinforcement Learning

Starter Agent for Coders Strike Back AI Challenge

Why I Majored in Statistics for a Career in Artificial Intelligence