Bring Your Own Datatypes to TVM

Gus Smith, University of Washington, presenting on May 22nd.

Deep learning is hungry for computational power, and it seems it will only be satiated through extreme hardware specialization. Google’s TPU and Intel’s Nervana both employ custom hardware to accelerate deep learning. The exploration of new numerical datatypes, which specify how mathematical values are expressed and operated on in hardware, has been key to extracting the best performance from hardware accelerators. Previously, numerical computations used IEEE 754 floating point, a standard which is designed to be general-purpose. However, the general-purpose nature of IEEE floats often leaves a lot of potential performance on the table. As a result, a number of new datatypes have sprung up as competitors to IEEE floating point, including Google's bfloat16, Intel's Flexpoint, Facebook's Deepfloat, and the Posit format from John Gustafson.

By supporting these new custom datatypes in TVM, an extensible deep learning compiler stack developed at the University of Washington, we can enable future workloads which utilize a variety of custom datatypes. In this talk, I discuss how we are taking the first steps towards supporting custom datatypes in TVM, by allowing users to "bring their own datatypes." Many datatype researchers first develop a software-emulated version of their datatype, before developing it in hardware; our framework allows users to plug these software-emulated versions of their datatypes directly into TVM, to compile and test real models.

Data-Free Quantization for Deep Neural Networks

(Ritchie Zhao presenting on Friday, 5/3/19.) 

Quantization is key to improving the execution time and energy efficiency of neural networks on both commodity GPUs and specialized accelerators. The majority of existing literature focuses on training quantized DNNs. However, industry shows great demand for data-free quantization - techniques that quantize a floating-point model without (re)training. Our talk focuses on this latter topic.

DNN weights and activations follow a bell-shaped distribution post-training, while practical hardware uses a linear quantization grid. This leads to challenges in dealing with outliers in the distribution. Prior work has addressed this by clipping the outliers or using specialized hardware. In this work, we propose outlier channel splitting (OCS),which duplicates channels containing outliers, then halves the duplicated values. The network remains functionally identical, but affected outliers are moved toward the center of the distribution. OCS requires no additional training and works on commodity hardware. Experimental evaluation on ImageNet classification and language modeling shows that OCS can outperform state-of-the-art clipping techniques with only minor overhead.

PMTest: A Fast and Flexible Testing Framework for Persistent Memory Programs

(Sihang Liu presenting on Wed. 4/24 Task 2.4 liaison meeting) 

Recent non-volatile memory technologies such as 3D XPoint and NVDIMMs have enabled persistent memory (PM) systems that can manipulate persistent data directly in memory. This advancement of memory technology has spurred the development of a new set of crash-consistent software (CCS) for PM - applications that can recover persistent data from memory in a consistent state in the event of a crash (e.g., power failure). CCS developed for persistent memory ranges from kernel modules to user-space libraries and custom applications. However, ensuring crash consistency in CCS is difficult and error-prone. Programmers typically employ low-level hardware primitives or transactional libraries to enforce ordering and durability guarantees that are required for ensuring crash consistency. Unfortunately, hardware can reorder instructions at runtime, making it difficult for the programmers to test whether the implementation enforces the correct ordering and durability guarantees. 

We believe that there is an urgent need for developing a testing framework that helps programmers identify crash consistency bugs in their CCS. We find that prior testing tools lack generality, i.e., they work only for one specific CCS or memory persistency model and/or introduce significant performance overhead. To overcome these drawbacks, we propose PMTest1 , a crash consistency testing framework that is both flexible and fast. PMTest provides flexibility by providing two basic assertion-like software checkers to test two fundamental characteristics of all CCS: the ordering and durability guarantee. These checkers can also serve as the building blocks of other application-specific, high-level checkers. PMTest enables fast testing by deducing the persist order without exhausting all possible orders. In the evaluation with eight programs, PMTest not only identified 45 synthetic crash consistency bugs, but also detected 3 new bugs in a file system (PMFS) and in applications developed using a transactional library (PMDK), while on average being 7.1x faster than the state-of-the-art tool.

Tuning Applications for Efficient GPU Offloading to In-memory Processing

(Presenting on Wed. 04/17/2019 at 2:00PM ET)

Authors: Yudong Wu, Mingyao Shen, Yi Hui Chen, Yuanyuan Zhou


Data movement between processors and main memory is a critical bottleneck for data-intensive applications. This problem is more severe with Graphics Processing Units (GPUs) due to their massive parallel data processing capability.  Recent research has shown that Processing-in-Memory (PIM) can greatly alleviate data movement bottleneck by reducing traffic between GPUs and memory devices. It offloads relative small execution context, instead of transferring massive data to be processed between memory devices and processors. However,  conventional application code that is highly optimized for locality to execute efficiently in GPU is not a natural match to be offloaded into PIM.   To address this challenge, our project investigates how application code can be restructured to improve the benefit of PIM offloading from GPUs.   In addition, we also study approaches to dynamically determine how much to offload as well as how to leverage all resources including GPUS in case of offloading to achieve the best possible overall performance.   From our experimental evaluations over 14 applications, our approach can averagely improve application offloading performance by 21%.

PIMCloud: Exploring Near-data Processing for Interactive Cloud Services

Shuang Chen from Cornell University
Presenting Wednesday, Apr 10 at 2:00PM EDT

Data centers host latency-critical (LC) as well as best-effort jobs. The former rely critically on an adequate provisioning of hardware resources to meet their QoS target. Many recent industry implementations and research proposals assume a single LC application per node; this is in part to make it easy to carve out resources for LC that one LC application, while allowing best-effort jobs to compete for the rest.

Two big changes in data centers, however, are about to shake up the status quo. First, the micro services model is making the number of LC applications hosted in data centers explode, making it impractical (and inefficient) to assume one LC application per node. That means multiple LC applications competing for resources on a single node, each with their own QoS needs. Second, the arrival of processing-in-memory (PIM) capabilities introduces a complex scheduling challenge (and opportunity).

In this talk, I will present our PIMCloud project, show some initial results, and discuss our ongoing work. First, I will present PARTIES, a novel hardware resource manager that enables successful colocation of multiple LC applications on a single node of a traditional data center. (This work will be presented next week at ASPLOS 2019.) Second, I will discuss how we envision augmenting this framework to accommodate PIM capabilities. Specifically, I will discuss some challenges and opportunities in future nodes where memory channels are themselves compute-capable.

Processing-in-Memory for Energy-efficient NeuralNetwork Training: A Heterogeneous Approach

(Hengyu Zhao Presenting on Wed. 3/20 at 2:00PM ET)

Authors: Hengyu Zhao, Jiawen Liu, Matheus Almeida Ogleari, Dong Li, Jishen Zhao

Abstract: Neural networks (NNs) have been adopted in a widerange of application domains, such as image classification, speechrecognition,   object   detection,   and   computer   vision.   However,training NNs – especially deep neural networks (DNNs) – can beenergy and time consuming, because of frequent data movementbetween  processor  and  memory.  Furthermore,  training  involvesmassive  fine-grained  operations  with  various  computation  andmemory  access  characteristics.  Exploiting  high  parallelism  withsuch  diverse  operations  is  challenging.  To  address  these  chal-lenges,  we  propose  a  software/hardware  co-design  of  heteroge-neous processing-in-memory (PIM) system. Our hardware designincorporates hundreds of fix-function arithmetic units and ARM-based programmable cores on the logic layer of a 3D die-stackedmemory  to  form  a  heterogeneous  PIM  architecture  attached  toCPU.  Our  software  design  offers  a  programming  model  and  aruntime system that program, offload, and schedule various NNtraining  operations  across  compute  resources  provided  by  CPUand heterogeneous PIM. By extending the OpenCL programmingmodel  and  employing  a  hardware  heterogeneity-aware  runtimesystem,  we  enable  high  program  portability  and  easy  programmaintenance  across  various  heterogeneous  hardware,  optimizesystem  energy  efficiency,  and  improve  hardware  utilization.

A Binary Learning Framework for Hyperdimensional Computing

(Presenting Fri. 3/15/19 at 2:00PM ET)

Authors: Mohsen Imani, John Messerly, Fan Wu, Wang Pi, and Tajana S. Rosing

Brain-inspired Hyperdimensional (HD) computing is a computing paradigm emulating a neuron’s activity in highdimensional space. In practice, HD first encodes all data points to high-dimensional vectors, called hypervectors, and then performs the classification task in an efficient way using a well-defined set of operations. In order to provide acceptable classification accuracy, the current HD computing algorithms need to map data points to hypervectors with non-binary elements. However, working with non-binary vectors significantly increases the HD computation cost and the amount of memory requirement for both training and inference. In this paper, we propose BinHD, a novel learning framework which enables HD computing to be trained and tested using binary hypervectors. BinHD encodes data points to binary hypervectors and provides a framework which enables HD to perform the training task with significantly low resources and memory footprint. In inference, BinHD binarizes the model and simplifies the costly Cosine similarity used in existing HD computing algorithms to a hardware-friendly Hamming distance metric. In addition, for the first time, BinHD introduces the concept of learning rate in HD computing which gives an extra knob to the HD in order to control the training efficiency and accuracy. We accordingly design a digital hardware to accelerate BinHD computation. Our evaluations on four practical classification applications show that BinHD in training (inference) can achieve 12.4× and 6.3× (13.8× and 9.9×) energy efficiency and speedup as compared to the state-of-the-art HD computing algorithm while providing the similar classification accuracy.