Intermediate Languages for Automated Spatial Computing

Yi-Hsiang Lai presenting on Wed. 11/20/19-- 

With the pursuit of improving compute performance under strict power constraints, there is an increasing need for deploying applications to heterogeneous hardware architectures with accelerators, such as PIMs and FPGAs. However, although these heterogeneous computing platforms are becoming widely available, they are very difficult to program. As a result, the use of such platforms has been limited to a small subset of programmers with specialized hardware knowledge.

To tackle this challenge, we introduce HeteroCL, a programming infrastructure composed of a Python-based domain-specific language (DSL) and a compilation flow. The HeteroCL DSL provides a clean programming abstraction that decouples algorithm specification from three important types of hardware customization in compute, data types, and memory architectures. HeteroCL further captures the interdependence among these different customization techniques, allowing programmers to explore various performance/area/accuracy trade-offs in a systematic and productive manner. In addition, our framework produces highly efficient hardware implementations for a variety of popular workloads by targeting spatial architecture templates such as systolic arrays and stencil with dataflow architectures. HeteroCL further incorporates the T2S framework developed by Intel Lab. T2S is an intermediate programming model extended from Halide for high-performance systolic architectures. Similar to HetoerCL, T2S cleanly decouples the temporal definition from spatial mapping, which enables productive programming and efficient design space exploration. 

Experimental results show that HeteroCL allows programmers to explore the design space efficiently in both performance and accuracy by combining different types of hardware customization and targeting spatial architectures, while keeping the algorithm code intact.

Exploiting Dynamic Sparsity in Neural Network Accelerators

Yuan Zhou Presenting on Wednesday, 11/13/19 at 1:00PM ET.

Convolutional neural networks (CNNs) have demonstrated human-level performance in many vision-related tasks, including image classification, segmentation, and real-time tasks such as autonomous driving and robotic manipulation. While modern CNNs continue to achieve higher accuracy, they also have larger model sizes and require more computation. As a result, it is challenging to deploy these compute-intensive CNN models to a wider range of applications, especially for embedded and mobile platforms which are area- and power-constrained. 

In this talk, I will introduce our work on reducing the computational costs of CNNs by exploiting dynamic sparsity at run time. I will first present channel gating, a fine-grained dynamic pruning technique for CNN inference. Channel gating identifies the regions in the feature map of each CNN layer that contribute less to the classification result, and turns off a subset of channels for computing the activations in these less important regions. Since channel gating preserves the memory locality in the channel dimension, a CNN with channel gating can be effectively mapped to a slightly modified weight-stationary CNN accelerator. Running a channel gating ResNet-18 model with 2.8x theoretical FLOP reduction, our accelerator achieves 2.3x speedup over the baseline on ImageNet. We further demonstrate that channel gating is suitable for near-memory CNN acceleration by simulating a compute-constrained, high-bandwidth platform. With sufficient memory bandwidth, the actual speedup of the channel gating accelerator scales almost linearly with the theoretical FLOP reduction. 

I will then briefly talk about our ongoing work, precision gating. Compared with channel gating, precision gating reduces computation by exploiting dynamic sparsity in another direction. Rather than pruning away entire channels at non-important pixel locations, precision gating uses low arithmetic precision at these locations while keeping the original precision at important locations. We believe precision gating is also suitable for near-memory CNN acceleration since it can effectively reduce the computational costs of CNN inference. 

MEDAL: Scalable DIMM based Near Data Processing Accelerator for DNA Seeding Algorithm

(Presenting on Wed. 10/30/2019)
Abstract: 
Computational genomics has proven its great potential to support precise and customized health care. However, with the wide adoption of the Next Generation Sequencing (NGS) technology, `DNA Alignment', as the crucial step in computational genomics, is becoming more and more challenging due to the booming bio-data. Consequently, various hardware approaches have been explored to accelerate DNA seeding - the core and most time consuming step in DNA alignment.



Most previous hardware approaches leverage multi-core, GPU, and FPGA to accelerate DNA seeding. However, DNA seeding is bounded by memory and above hardware approaches focus on computation. For this reason, Near Data Processing (NDP) is a better solution for DNA seeding. Unfortunately, existing NDP accelerators for DNA seeding face two grand challenges, i.e., fine-grained random memory access and scalability demand for booming bio-data. To address those challenges, we propose a practical, energy efficient, Dual-Inline Memory Module (DIMM) based, NDP Accelerator for DNA Seeding Algorithm (MEDAL), which is based on off-the-shelf DRAM components. For small databases that can be fitted within a single DRAM rank, we propose the intra-rank design, together with an algorithm-specific address mapping, bandwidth-aware data mapping, and Individual Chip Select (ICS) to address the challenge of fine-grained random memory access, improving parallelism and bandwidth utilization. Furthermore, to tackle the challenge of scalability for large databases, we propose three inter-rank designs (polling-based communication, interrupt-based communication, and Non-Volatile DIMM (NVDIMM)-based solution). In addition, we propose an algorithm-specific data compression technique to reduce memory footprint, introduce more space for the data mapping, and reduce the communication overhead. Experimental results show that for three proposed designs, on average, MEDAL can achieve 30.50x/8.37x/3.43x speedup and 289.91x/6.47x/2.89x energy reduction when compared with a 16-thread CPU baseline and two state-of-the-art NDP accelerators, respectively.

Bio of Wenqin Huangfu: 

Wenqin Huangfu is a fourth year Ph.D. student in University of California at Santa Barbara. His advisor is Professor. Yuan Xie.

Wenqin’s research interests include domain-specific accelerator, Processing-In-Memory (PIM), Near-Data-Processing (NDP), and emerging memory technology. Currently, Wenqin is focusing on hardware acceleration of bioinformatics with PIM and NDP technology.

SubZero: Zero-copy IO for Persistent Main Memory File Systems

Juno Kim, UC San Diego, presenting on Oct 14th

POSIX-style read() and write() have long been the standard interface for accessing file data. However, the data copy their semantics require imposes an unnecessary overhead on accessing files stored in persistent memories attached to the processor memory bus (PMEM). PMEM-aware file systems provide direct-access (DAX)-based mmap() to avoid the copy, but it forces the programmer to manage concurrency control and atomicity guarantees. 

We propose a new IO interface, called SubZero IO, that avoids data movement overheads when accessing persistent memory-backed files while still interacting cleanly with legacy read() and write(). SubZero IO provides two new system calls – peek() and patch() – that allow for data access and modification without a copy. They can improve performance significantly, but they require substantial changes to how an application performs IO. To avoid this, we describe a PMEM-aware implementation of read() and write() that can transparently provide most of the benefits of peek() and patch() for some applications.

Measurements of simple benchmarks show that SubZero outperforms copy-based read() and write() by up to 2× and 6×, respectively. At the application level, peek() improves GET performance of the Apache Web Server by 3.8×, and patch() boosts the SET performance of Kyoto Cabinet up to 2.7×.

FloatPIM: Acceleration of DNN Training in PIM

Presented by Saransh Gupta of UC San Diego on Wednesday, October 9th at 1:00PM ET.

We present PIM methods that perform in-memory computations on digital data and support high precision floating-point operations. First, we introduce an operation-dependent variable voltage application scheme which improves the performance and energy efficiency of existing PIM operations by 2x. Then, we propose an in-memory deep neural networks (DNN) architecture, which not only supports DNN inference but also training completely in-memory. To achieve this, we natively enable, for the first time, high-precision floating-point operations in memory. Our design also enables fast communication between neighboring memory blocks to reduce the internal data movement of the PIM architecture.

Bring Your Own Datatypes to TVM

Gus Smith, University of Washington, presenting on May 22nd.

Deep learning is hungry for computational power, and it seems it will only be satiated through extreme hardware specialization. Google’s TPU and Intel’s Nervana both employ custom hardware to accelerate deep learning. The exploration of new numerical datatypes, which specify how mathematical values are expressed and operated on in hardware, has been key to extracting the best performance from hardware accelerators. Previously, numerical computations used IEEE 754 floating point, a standard which is designed to be general-purpose. However, the general-purpose nature of IEEE floats often leaves a lot of potential performance on the table. As a result, a number of new datatypes have sprung up as competitors to IEEE floating point, including Google's bfloat16, Intel's Flexpoint, Facebook's Deepfloat, and the Posit format from John Gustafson.

By supporting these new custom datatypes in TVM, an extensible deep learning compiler stack developed at the University of Washington, we can enable future workloads which utilize a variety of custom datatypes. In this talk, I discuss how we are taking the first steps towards supporting custom datatypes in TVM, by allowing users to "bring their own datatypes." Many datatype researchers first develop a software-emulated version of their datatype, before developing it in hardware; our framework allows users to plug these software-emulated versions of their datatypes directly into TVM, to compile and test real models.

Data-Free Quantization for Deep Neural Networks

(Ritchie Zhao presenting on Friday, 5/3/19.) 

Quantization is key to improving the execution time and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs. However, industry shows great demand for data-free quantization - techniques that quantize a floating-point model without (re)training. Our talk focuses on this latter topic.

DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS),which duplicates channels containing outliers, then halves the duplicated values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.