triSYCL

triSYCL is a humble implementation test-bed to experiment with the provisional specification of the OpenCL SYCL C++ layer and to give feedback to the Khronos OpenCL SYCL committee.

For legal reasons, the specification used for this open source project is the published current provisional specification and not the last one currently discussed in the Khronos OpenCL SYCL committee. If you are a Khronos member, you can ask for an access to https://github.com/amd/triSYCL-private/tree/future in the https://github.com/amd/triSYCL-private private repository to use a more futuristic version.

This is just the start of a SYCL mock-up, only based on C++1z (2017?) and OpenMP with execution on the CPU right now. So there is nothing related to OpenCL yet. But since in SYCL there is a host fall-back, this can be used as the start of this fall-back...

The parallel kernels can be executed in parallel on the CPU with OpenMP in the first range dimension, if compiled with OpenMP support.

This is provided as is, without any warranty, with the same license as LLVM/Clang.

Main contributor: Ronan Keryell at AMD point cOm

OpenCL SYCL is a single-source C++11-based DSEL (Domain Specific Embedded Language) aimed at facilitating the programming of heterogeneous accelerators by leveraging the OpenCL language and concepts.

OpenCL SYCL is developed inside the Khronos OpenCL SYCL committee and thus, for more information on SYCL, look at http://www.khronos.org/opencl/sycl

SYCL has a lot of interesting advantages compared to plain OpenCL or other approaches:

• SYCL is an open standard from Khronos with a working committee (you can contribute!) and we can expect several implementations (commercial or open source) on many platforms soon, ranging from GPU, APU, FPGA... down to plain CPU;

• it offers a single-source C++ programming model that allows taking advantage of the modern C++11 superpower, unifying both the host and accelerator sides. For example it is possible to write generic accelerated functions on the accelerators in a terse way by using (variadic) templates, meta-programming and lambda expressions;

• SYCL abstracts the concepts behind OpenCL and provides higher-level concepts such as command_group that allows the runtime to take advantage of a more task graph-oriented view of the computations. This allows lazy data transfers between accelerators and host or to use platform capabilities such as OpenCL 2 SVM or HSA for sharing data between host and accelerators;

• the entry cost of the technology is zero since, after all, an existing OpenCL or C++ program is a valid SYCL program;

• the exit cost is low since it is pure C++ without any extension or #pragma. Retargeting the SYCL classes and functions to use other frameworks such as OpenMP 4 or C++AMP is feasible without rewriting a new compiler for example;

• easier debugging

  • since all memory accesses to array parameters in kernels go through accessors, all the memory bound checks can be done in them if needed;
  • since there is a pure host mode, the kernel code can be run also on the host and debugged using the usual tools and use any system (such stdio or iostream...) or data libraries (for nice data visualization);
  • since the kernel code is C++ code even when run on an accelerator, instrumenting the code by using special array classes or overloading some operators allows deep intrusive debugging or code analysis without changing the algorithmic parts of the code;

• SYCL is high-level standard C++11 without any extension, that means that you can use your usual compiler and the host part can use some cool and common extensions such as OpenMP, OpenHMPP, OpenACC,... or libraries such as MPI or PGAS Coarray++, be linked with other parts written in other languages (Fortran...). Thus SYCL is already Exascale-ready!

• even if SYCL hides the OpenCL world by default, it inherits from all the OpenCL world:

  • same interoperability as the OpenCL underlying platform: OpenGL, DirectX...
  • access to all the underlying basic OpenCL objects behind the SYCL abstraction for interoperability and hard-core optimization;
  • construction of SYCL objects from basic OpenCL objects to add some SYCL parts to an existing OpenCL application;
  • so it provides a continuum from higher-level programming à la C++AMP or OpenMP 4 down to low-level OpenCL, according to the optimization needs, from using simple OpenCL intrinsics or vector operation from the cl::sycl namespace down to providing a real OpenCL kernel to be executed without requiring all the cumbersome usual OpenCL host API.

  This OpenCL seamless integration plus the gradual optimization features are perhaps the most compelling arguments for SYCL because it allows high-level programming simplicity without giving-up hard-core performance;

• since the SYCL task graph execution model is asynchronous, this can be used by side effect to overcome some underlying OpenCL implementation limitations. For example, some OpenCL stacks may have only in-order execution queues or even synchronous (blocking) ND-range enqueue, or some weird constrained mapping between OpenCL programmer level queue(s) and the hardware queues.

  In this case, a SYCL implementation can deal with this, relying for example on multiple host CPU threads, multiple thread-local-storage (TLS) queues, its own scheduler, etc. atop the limited OpenCL stack to provide computation and communication overlap in a natural pain-free fashion. This relieves the programmer to reorganize her application to work around these limitation, which can be quite a cumbersome work.

For introduction material on the interest of DSEL in this area, look for example at these articles:

• Domain-specific Languages and Code Synthesis Using Haskell, Andy Gill. May 6, 2014 in ACM Queue and Communications of the ACM.
• Design Exploration through Code-generating DSLs, Bo Joel Svensson, Mary Sheeran and Ryan Newton. May 15, 2014 in ACM Queue and Communications of the ACM.

By reverse chronological order:

• Modern C++, OpenCL SYCL & OpenCL CL2.hpp, Ronan Keryell (AMD & Khronos OpenCL SYCL Working Group Member). November 18, 2014, Presentation at SuperComputing 2014, OpenCL BoF, New Orleans, USA.
• Implementing the OpenCL SYCL Shared Source C++ Programming Model using Clang/LLVM, Gordon Brown. November 17, 2014, Workshop on the LLVM Compiler Infrastructure in HPC, SuperComputing 2014
• SYCL Specification --- SYCL integrates OpenCL devices with modern C++, Khronos OpenCL Working Group — SYCL subgroup. Editors: Lee Howes and Maria Rovatsou. Version 1.2, Revision 2014-09-16.
• OpenCL 2.0, OpenCL SYCL & OpenMP 4, open standards for heterogeneous parallel programming, Ronan Keryell (AMD & Khronos OpenCL Working Group Member). July 3, 2014, Presentation at the Meetup of the High Performance Computing & Supercomputing Group of Paris.
• OpenCL 2.0, OpenCL SYCL & OpenMP 4, open standards for heterogeneous parallel programming, Ronan Keryell (AMD & Khronos OpenCL Working Group Member). July 2, 2014, Presentation at Forum Ter@tec: Calcul scientifique & Open Source : pratiques industrielles des logiciels libres.
• The Future of Accelerator Programming in C++, Sebastian Schaetz, May 18, 2014. Presentation at C++Now 2014.
• SYCL : Abstraction Layer for Leveraging C++ and OpenCL, Maria Rovatsou (Codeplay & Khronos OpenCL Working Group Member). May 12-13, 2014, IWOCL 2014.
• Building the OpenCL ecosystem - SYCL for OpenCL, Lee Howes (Senior Staff Engineer at Qualcomm & Khronos OpenCL Working Group Member). April 21, 2014, HPC & GPU Supercomputing Group of Silicon Valley.
• SYCL 1.2: Unofficial High-Level Overview, AJ Guillon (Khronos OpenCL Working Group Member). March 19, 2014. Video.
• SYCL for OpenCL, Andrew Richards (CEO Codeplay & Chair SYCL Working group). March 19, 2014, GDC 2014.
• Fusing GPU kernels within a novel single-source C++ API, Ralph Potter, Paul Keir, Jan Lucas, Maurico Alvarez-Mesa, Ben Juurlink and Andrew Richards. January 20, 2014, LPGPU Workshop on Power-Efficient GPU and Many-core Computing (PEGPUM 2014).
• Fusing GPU kernels within a novel single-source C++ API, Ralph Potter, Paul Keir, Jan Lucas, Mauricio Alvarez-Mesa, Ben Juurlink, Andrew Richards. November 18, 2013, Intel Compiler, Architecture and Tools Conference.

There are also many interesting articles in the publication list from Codeplay.

Some other known implementations:

• CodePlay has an implementation based on OpenCL SPIR with Clang/LLVM http://codeplay.com/products

• Boost.Compute
• VexCL
• ViennaCL
• C++ Parallelism TS https://github.com/cplusplus/parallelism-ts
• C++ Concurrency TS https://github.com/cplusplus/concurrency_ts
• Bolt
• Thrust
• C++AMP
• Open Source implementation of C++AMP https://bitbucket.org/multicoreware/cppamp-driver-ng
• Intel Lab's iHRC https://github.com/IntelLabs/iHRC
• CUDA

The documentation of the triSYCL implementation itself can be found in http://amd.github.io/triSYCL/Doxygen/triSYCL/html and http://amd.github.io/triSYCL/Doxygen/triSYCL/triSYCL-implementation-refman.pdf

An experimental description of the API generated from triSYCL through Doxygen can be found in http://amd.github.io/triSYCL/Doxygen/SYCL/html and http://amd.github.io/triSYCL/Doxygen/SYCL/SYCL-API-refman.pdf

But since the implementation has moved toward more meta-progamming usage, this API documentation is no longer really descriptive of what is really available, since it is mainly hidden by the meta-programming power. At some point this API documentation will disappear.

Only Clang 3.6+ or GCC 4.9+, Boost.MultiArray (which adds to C++ the nice Fortran array semantics and syntax), Boost.Operators are needed. If you use the debug mode or OpenMP, this works only with GCC 4.9 since current Clang version does not support OpenMP yet or some C++14 with -g.

To install them on latest Linux Debian/unstable (this should work on latest Ubuntu too):

sudo apt-get install clang-3.6 g++4.9 libboost-dev

There is nothing else to do for now to use the include files from include/CL when compiling a program. Just add a -I.../include/CL option when compiling.

There are simple examples and tests in the tests directory. Look at tests/README.rst description.

In the top directory, run

make

that will produce tmp/Doxygen/SYCL with the API documentation and tmp/Doxygen/triSYCL with the documented triSYCL implementation source code.

To publish the documentation on GitHub:

make publish

and finish as explained by the make output.

Some ideas of future developments where you can contribute too: :-)

• update the implementation to stick to the final SYCL 1.2 specification for the IWOCL 2015 tutorial (this is on-going);

• finish implementation of basic classes without any OpenCL support;

• move to CMake for better portability;

• improve the test infrastructure;

• use the official OpenCL SYCL test suite to extend/debug/validate this implementation;

• add first OpenCL support with kernels provided only as strings, thus avoiding the need for a compiler. Could be based on other libraries such as Boost.Compute, VexCL, ViennaCL...;

• make an accelerator version based on OpenMP 4 accelerator target, OpenHMPP or OpenACC;

• make an accelerator version based on wrapper classes for the C++AMP Open Source compiler https://bitbucket.org/multicoreware/cppamp-driver-ng

  Extend the current C++AMP OpenCL HSA or SPIR back-end runtime to expose OpenCL objects needed for the SYCL OpenCL interoperability. This is probably the simpler approach to have a running SYCL compiler working quickly.

  The main issue is that since C++AMP support is not yet integrated in the official trunk, it would take a long time to break things down and be reviewed by the Clang/LLVM community;

• extend runtime and Clang/LLVM to generate OpenCL/SPIR from C++ single-source kernels, by using OpenMP outliner;

• alternatively develop a Clang/LLVM-based version, recycling the outliner which is already present for OpenMP support and modify it to generate SPIR. Then build a specific version of libiomp5 to use the OpenCL C/C++ API to run the offloaded kernels.

  This approach may require more work than the C++AMP version but since it is based on the existing OpenMP infrastructure Intel spent a lot of time to upstream through the official code review process, at the end it would require quite less time for up-streaming, if this is the goal.

  OpenMP4 in Clang/LLVM is getting momentum and making lot of progress backed by Intel, IBM, AMD... so it sounds like the way to go;

• add OpenCL 2.x support with SYCL 2.1;

• since SYCL is a pretty general programming model for heterogeneous computing, if the OpenCL compatibility layer is not required, some other back-end could be written besides the current OpenMP one: CUDA, RenderScript, etc.