End to End AI Use Case-Driven System Design | by Liz Li | Mar, 2024

A radical listing of Applied sciences for finest Efficiency/Watt

The mostly used metric to outline AI efficiency is TOPs (Tera Operations Per Second), which signifies compute functionality however oversimplifies the complexity of AI programs. In relation to actual AI use case system design, many different components must also be thought-about past TOPs, together with reminiscence/cache dimension and bandwidth, knowledge varieties, power effectivity, and so on.

Furthermore, every AI use case has its traits and requires a holistic examination of the entire use case pipeline. This examination delves into its affect on system elements and explores optimization methods to foretell the very best pipeline efficiency.

On this put up, we decide one AI use case — an end-to-end real-time infinite zoom characteristic with a steady diffusion-v2 inpainting mannequin and examine methods to construct a corresponding AI system with the very best efficiency/Watt. This could function a proposal, with each well-established applied sciences and new analysis concepts that may result in potential architectural options.

Background on end-to-end video zoom

• As proven within the beneath diagram, to zoom out video frames (fish picture), we resize and apply a border masks to the frames earlier than feeding them into the steady diffusion inpainting pipeline. Alongside an enter textual content immediate, this pipeline generates frames with new content material to fill the border-masked area. This course of is constantly utilized to every body to realize the continual zoom-out impact. To preserve compute energy, we could sparsely pattern video frames to keep away from inpainting each body(e.g., producing 1 body each 5 frames) if it nonetheless delivers a passable person expertise.

• Stable diffusion-v2 inpainting pipeline is pre-trained on steady diffusion-2 mannequin, which is a text-to-image latent diffusion mannequin created by stability AI and LAION. The blue field in beneath diagram shows every operate block within the inpainting pipeline

• Steady diffusion-2 mannequin generates 768*768 decision pictures, it’s educated to denoise random noise iteratively (50 steps) to get a brand new picture. The denoising course of is applied by Unet and scheduler which is a really sluggish course of and requires numerous compute and reminiscence.

There are 4 fashions used within the pipeline as beneath:

1. VAE (picture encoder). Convert picture into low dimensional latent illustration (64*64)
2. CLIP (textual content encoder). Transformer structure (77*768), 85MP
3. UNet (diffusion course of). Iteratively denoising processing by way of a schedular algorithm, 865M
4. VAE (picture decoder). Transforms the latent illustration again into a picture (512*512)

Most steady Diffusion operations (98% of the autoencoder and textual content encoder fashions and 84% of the U-Internet) are convolutions. The majority of the remaining U-Internet operations (16%) are dense matrix multiplications because of the self-attention blocks. These fashions will be fairly massive (varies with totally different hyperparameters) which requires numerous reminiscence, for cell gadgets with restricted reminiscence, it’s important to discover mannequin compression methods to scale back the mannequin dimension, together with quantization (2–4x mode dimension discount and 2-3x speedup from FP16 to INT4), pruning, sparsity, and so on.

Energy effectivity optimization for AI options like end-to-end video zoom

For AI options like video zoom, energy effectivity is likely one of the high components for profitable deployment on edge/cell gadgets. These battery-operated edge gadgets retailer their power within the battery, with the capability of mW-H (milliWatt Hours, 1200WH means 1200 watts in a single hour earlier than it discharge, if an utility is consuming 2w in a single hour, then the battery can energy the machine for 600h). Energy effectivity is computed as IPS/Watt the place IPS is inferences per second (FPS/Watt for image-based purposes, TOPS/Watt )

It’s essential to scale back energy consumption to get lengthy battery life for cell gadgets, there are many components contributing to excessive energy utilization, together with massive quantities of reminiscence transactions on account of massive mannequin dimension, heavy computation of matrix multiplications, and so on., let’s check out methods to optimize the use case for environment friendly energy utilization.

1. Mannequin optimization.

Past quantization, pruning, and sparsity, there may be additionally weight sharing. There are many redundant weights within the community whereas solely a small variety of weights are helpful, the variety of weights will be lowered by letting a number of connections share the identical weight proven as beneath. the unique 4*4 weight matrix is lowered to 4 shared weights and a 2-bit matrix, complete bits are lowered from 512 bits to 160 bits.

2. Reminiscence optimization.

Reminiscence is a essential part that consumes extra energy in comparison with matrix multiplications. For example, the facility consumption of a DRAM operation will be orders of magnitude greater than that of a multiplication operation. In cell gadgets, accommodating massive fashions inside native machine reminiscence is commonly difficult. This results in quite a few reminiscence transactions between native machine reminiscence and DRAM, leading to greater latency and elevated power consumption.

Optimizing off-chip reminiscence entry is essential for enhancing power effectivity. The article (Optimizing Off-Chip Memory Access for Deep Neural Network Accelerator [4]) launched an adaptive scheduling algorithm designed to reduce DRAM entry. This method demonstrated a considerable power consumption and latency discount, ranging between 34% and 93%.

A brand new technique (ROMANet [5]) is proposed to reduce reminiscence entry for energy saving. The core thought is to optimize the precise block dimension of CNN layer partition to match DRAM/SRAM assets and maximize knowledge reuse, and in addition optimize the tile entry scheduling to reduce the variety of DRAM entry. The information mapping to DRAM focuses on mapping a knowledge tile to totally different columns in the identical row to maximise row buffer hits. For bigger knowledge tiles, identical financial institution in numerous chips will be utilized for chip-level parallelism. Moreover, if the identical row in all chips is crammed, knowledge are mapped within the totally different banks in the identical chip for bank-level parallelism. For SRAM, an identical idea of bank-level parallelism will be utilized. The proposed optimization circulation can save power by 12% for the AlexNet, by 36% for the VGG-16, and by 46% for the MobileNet. A high-level circulation chart of the proposed technique and schematic illustration of DRAM knowledge mapping is proven beneath.

3. Dynamic energy scaling.

The facility of a system will be calculated by P=C*F*V², the place F is the working frequency and V is the working voltage. Methods like DVFS (dynamic voltage frequency scaling) was developed to optimize runtime energy. It scales voltage and frequency relying on workload capability. In deep studying, layer-wise DVFS just isn’t applicable as voltage scaling has lengthy latency. Then again, frequency scaling is quick sufficient to maintain up with every layer. A layer-wise dynamic frequency scaling (DFS)[6] approach is proposed for NPU, with an influence mannequin to foretell energy consumption to find out the very best allowable frequency. It’s demonstrated that DFS improves latency by 33%, and saves power by 14%

4. Devoted low-power AI HW accelerator structure. To speed up deep studying inference, specialised AI accelerators have proven superior energy effectivity, reaching related efficiency with lowered energy consumption. For example, Google’s TPU is tailor-made for accelerating matrix multiplication by reusing enter knowledge a number of occasions for computations, in contrast to CPUs that fetch knowledge for every computation. This method conserves energy and diminishes knowledge switch latency.