PACT'18 tutorial: Tensor Programming with SaC

This tutorial gives a brief introduction to the high-level array language SaC and its attendant tool chain. We show how this technology can be leveraged to conveniently express tensor computations very close to abstract mathematical formulae and, at the same time, how such specifications can be mapped into efficient parallel code for heterogeneous systems that include GPUs.

We intend to combine a quick introduction with live coding session, demonstrating high-productivity and high-performance aspects of SaC by implementing a few neural network algorithms in less than two hours.

Arrays have always been a key data structure in high-performance computing. With an increased interest in neural networks and deep learning, the question of effective abstractions for array programming and their translation into highly efficient parallel codes for heterogeneous systems is freshly invigorated: In the presence of cheap compute power, productivity becomes an increasingly important aspect of high-performance computing. At the same time, performance portability takes centre-stage too as the range of parallel hardware  architectures becomes increasingly heterogeneous.

The SaC approach offers a solution to this Performance-Productivity-Portability  challenge. The functional nature of the language offers very powerful declarative abstractions, yet allowing a compiler to run aggressive optimisations like auto-parallelisation, auto-offloading to accelerators, or loop optimisations which lead to code that performs on par with hand-optimised programs. The design of SaC builds on the psi-calculus as underlying array theory and aligns with a rich history of array languages such as APL, SISAL, Fortran90 or Matlab but provides a more generic substrate, suitable for conveniently expressing high-dimensional tensor operations which lie at the core of deep learning applications. Potentially, this turns SaC into an efficient DSL for deep learning.

We believe PACT provides the ideal audience for this tutorial as it primarily attracts compiler experts with practical expertise at the forefront of modern parallel architectures.

Sven-Bodo Scholz and Artjoms Šinkarovs are computer science researchers working in the area of high-performance compilers and functional languages. Sven-Bodo Scholz is the original author of the SaC compiler and Artjoms Šinkarovs is one of the main contributors.