SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning

 Zihao Ye presenting on Wed. 10/19/22 at 1:00 & 7:00 PM ET

Sparse tensors are rapidly becoming critical components of modern deep learning workloads. However, developing high-performance sparse operators can be difficult and tedious, and existing vendor libraries cannot satisfy the escalating demands from new operators. Sparse tensor compilers simplify the development of operators, but efficient sparse compilation for deep learning remains challenging because a single sparse format cannot maximize hardware efficiency, and single-shot compilers cannot keep up with latest hardware and system advances. We show that the key to addressing both challenges is two forms of composability. In this paper, we propose SparseTIR, a sparse tensor compilation abstraction that offers composable formats and composable transformations for deep learning workloads. SparseTIR constructs a search space over these composable components for performance tuning. With these improvements, SparseTIR obtains consistent performance speedups vs vendor libraries and frameworks on sparse workloads such as Graph Neural Networks, Sparse Convolution, and Network Pruning.

MEMulator & PiMulator: Emerging Memory and Processing-in-Memory Architecture Emulation

(Sergiu Mosanu, UVA, presenting on October 12, 2022.)

Main memory is a crucial SoC and architecture design aspect, affecting system performance, power, and cost. The development of emerging memory technologies, increasingly specialized DRAM memory flavors, and Processing-in-Memory (PiM) architectures introduce the need for system-level modeling and evaluation. However, it is challenging to mimic both software and hardware aspects of emerging memory and PiM architectures using the currently available tools with high performance and fidelity. We develop a system emulation framework that employs a modular, parameterizable, FPGA synthesizable memory and PiM model. Implemented in System Verilog, the memory and PiM model allows users to generate any desired memory configuration on the FPGA fabric with complete control over the structure and distribution of the PiM logic units. We emulate a whole system by interfacing the memory emulation model with CPU soft cores and a soft memory controller. We demonstrate strategies to model several pioneering bitwise-PiM architectures and provide detailed benchmark performance results showing the platform's ability to facilitate design space exploration. We observe an emulation vs. simulation weighted-average speedup of 28x when running a memory benchmark workload. This comprehensive FPGA-based memory emulation enables fast, high-fidelity design-space exploration and evaluation of processing-in-memory architectures as part of a whole system stressed with heavy workloads.

An MLIR-based Intermediate Representation for Accelerator Design with Decoupled Customizations

Hongzheng Chen and Niansong Zhang presenting on Wed. 9/28/22.

The increasing specialized accelerators deployed in data centers and edge devices call for the need of generating high-performance accelerators efficiently. However, the custom processing engines, memory hierarchy, data type, and data communication, complicate the accelerator design. In this talk, we will present our MLIR-based accelerator IR HCL, which decouples the algorithm and hardware customizations at the IR level. We will provide several case studies to demonstrate how our IR can support a wide range of applications, how our IR primitives can be composed with different designs, and how we can achieve high performance and productivity at the same time. Finally, we will discuss the benefits and ongoing efforts of our work.

Accelerating Few Shot Learning with HD Computing in ReRAM

Weihong Xu, UCSD, presenting on Wed. 9/21/22 at 1:00 pm & 7:00 pm ET. 

Hyperdimensional (HD) computing is a lightweight algorithm with fast learning capability that can efficiently realize classification and few-shot learning (FSL) workloads. However, traditional von-Neumann architecture is highly inefficient for HD algorithms due to the limited memory bandwidth and capacity. Processing in-memory (PIM) is an emerging computing paradigm which tries to address these issues by using memories as computing units. Our talk introduces the efficient PIM architecture for HD as well as application of HD on few-shot classification. First, we show the PIM-based HD computing architecture on ReRAM, Tri-HD, to accelerate all phases of the general HD computing pipeline namely, encoding, training, retraining, and inference. Tri-HD is enabled by efficient in-memory logic operations and shows orders of magnitude performance improvements over CPU. However, Tri-HD is not suitable for area and power-constrainted device since it suffers from high HD encoding complexity and complex dataflow. To this end, we present an algorithm and PIM co-design, FSL-HD, to realize energy-efficient FSL. FSL-HD significantly reduces the encoding complexity by >10x and is equipped with an optimized dataflow for practical ReRAM. As a result, FSL-HD shows superior FSL accuracy, flexibility, and hardware efficiency compared to state-of-the-art ReRAM-based FSL accelerators.

How Good is my HTAP System?

 (Elena Milkai, U Wisc-Madison, presenting on April 20th at 1PM & 7PM ET)

Hybrid Transactional and Analytical Processing (HTAP) systems have recently gained popularity as they combine OLAP and OLTP processing to reduce administrative and synchronization costs between dedicated systems. However, there is no precise characterization of the features that distinguish a good HTAP system from a poor one. In this paper, we seek to solve this problem from the perspectives of both performance and freshness. To simultaneously capture the performance of both transactional and analytical processing, we introduce a new concept called throughput frontier, which visualizes both transactional and analytical throughput in a single 2D graph. The throughput frontier can capture information regarding the performance of each engine, the interference between the two engines, and various system design decisions. To capture how well an HTAP system supports real-time analytics, we define a freshness metric which quantifies how recent is the snapshot of the data seen by each analytical query. We also develop a practical way to measure freshness in a real system. We design a new hybrid benchmark called HATtrick which incorporates both throughput frontier and freshness as metrics. Using the benchmark, we evaluate three representative HTAP systems under various data size and system configurations and demonstrate how the metrics reveal important system characteristics and performance information.

Enabling Efficient Large-Scale Deep Learning Training with Cache Coherent Disaggregated Memory Systems

(Zixuan Wang, UCSD, presenting on Wed. 3/16/22 at 1:00 & 7:00 PM ET at our CRISP task-level meeting.) 

Modern deep learning (DL) training is memory consuming, constrained by the memory capacity of each computation component and cross-device communication bandwidth. In response to such constraints, current approaches include increasing parallelism in distributed training and optimizing inter-device communication. However, model parameter
communication is becoming a key performance bottleneck in distributed DL training. To improve parameter communication performance, we propose COARSE, a disaggregated memory extension for distributed DL training. COARSE is built on modern cache-coherent interconnect (CCI) protocols and MPI-like collective communication for synchronization, to allow
low-latency and parallel access to training data and model parameters shared among worker GPUs. To enable high bandwidth transfers between GPUs and the disaggregated memory system, we propose a decentralized parameter communication scheme to decouple and localize parameter synchronization traffic. Furthermore, we propose dynamic tensor routing and
partitioning to fully utilize the non-uniform serial bus bandwidth varied across different cloud computing systems. Finally, we design a deadlock avoidance and dual synchronization to ensure high-performance parameter synchronization. Our evaluation shows that COARSE achieves up to 48.3% faster DL training compared to the state-of-the-art MPI AllReduce
communication.

Contact Zixuan for questions, zxwang@ucsd.edu. 

GraphLily: Near-Memory Acceleration for Graph Linear Algebra

(Yixiao Du, Cornell, presenting on Wednesday, November 10, 2021 at 1:00 p.m. & 7:00 p.m. ET. )

Graph processing is typically memory-bound due to low compute to memory access ratio and irregular data access patterns. The emerging high-bandwidth memory (HBM) delivers exceptional bandwidth and is adopted on FPGAs with near-memory integration and customizable datapaths, which brings the potential to significantly boost the performance of graph processing and relieve the programming burden.
We present GraphLily, a graph linear algebra overlay, to accelerate graph processing on HBM-equipped FPGAs. GraphLily supports a rich set of graph algorithms by adopting the GraphBLAS programming interface, which formulates graph algorithms as sparse linear algebra operations. Operators of different GraphBLAS semirings share the same datapath, which is configurable at run time. GraphLily further builds a middleware to provide runtime support. With this middleware, we can port existing GraphBLAS programs to FPGAs with slight modifications to the original code intended for CPU/GPU execution. GraphLily provides efficient, memory-optimized accelerators for the two widely-used kernels in graph processing, namely, sparse-matrix dense-vector multiplication (SpMV) and sparse-matrix sparse-vector multiplication (SpMSpV). Leveraging the near-memory connectivity, the SpMV accelerator uses a sparse matrix storage format tailored to HBM that enables streaming, vectorized accesses to each channel and concurrent accesses to multiple channels. Besides, the SpMV accelerator exploits data reuse in accesses of the dense vector by introducing a scalable on-chip buffer design. The SpMSpV accelerator complements the SpMV accelerator to handle cases where the input vector has a high sparsity. The evaluation shows that compared with state-of-the-art graph processing frameworks on CPUs and GPUs, GraphLily achieves up to 2.5x and 1.1x higher throughput, while reducing the energy consumption by 8.1x and 2.4x; compared with prior single-purpose graph accelerators on FPGAs, GraphLily achieves 1.2x–1.9x higher throughput. GraphLily is a prototype of near-memory graph processing. We are working to further improve the performance.