Browsing by Subject "Quantization"
Now showing 1 - 4 of 4
Results Per Page
Sort Options
Item Design Approaches for Lightweight Machine Learning Models(2023-03) Chang, YangyangIn this era of data explosion, the diversity and complexity of data are gradually increasing. The corresponding data processing models become massive and complicated. Especially for dealing with high-dimensional data, large-size inputs, and low latency, modern machine learning methods (e.g., deep neural networks) require advanced hardware solutions (e.g., High Power Graphic Processing Units, Tensor Process Units). To run the models efficiently on embedded platforms, designs of lightweight machine learning are important to ensure small computation and memory requirements. This thesis shows six innovative designs of lightweight machine learning models. Specifically, the introduced designs include the quantized vision transformer, optimized binarized neural networks (BNNs), and lightweight convolutional neural networks (CNNs). For high- dimensional multitasking continuous and discrete optimizations, the innovative lightweight designs contain the modified multifactorial and cross-target evolutionary algorithms (EAs). For data compression, this thesis proposes a hybrid compressor for medical electrocardiogram (ECG) data on an embedded system. In future work, the overall lightweight design framework can be integrated by the proposed structures to include full-system compression, optimization, and quantization. Quantization using a small number of bits shows promise for reducing latency and memory usage in DNNs. However, most quantization methods cannot readily handle complicated functions such as exponential and square root, and prior approaches involve complex processes that must interact with floating-point calculations during the quantization pass. The proposed quantized vision transformer in this thesis provides a robust method for the full integer quantization of the vision transformer without requiring any intermediate floating-point computations. The quantization techniques can be applied in various hardware or software implementations, including processor/memory architectures and FPGAs. BNNs have shown promise in low-power embedded systems, but these are typically designed starting from existing architectures that are based on floating-point number representations. It is also hard to meet the classification requirements because the weights and activations are limited to ±1. This thesis applies the efficient genetic algorithm (GA) to optimize a fully connected binarized architecture to increase the BNN performance without changing its basic operators. The simulation results demonstrate the effectiveness of the proposed method to improve the performance of BNNs. Novel design frameworks for lightweight CNNs are proposed for embedded system applications on image classification tasks. Scalable lightweight architectures for CNNs are first proposed. The population-based metaheuristic approaches of the genetic algorithm (GA), cuckoo search (CS), multifactorial evolutionary algorithm (MFEA), and a proposed hybrid evolutionary approach are then used to optimize the proposed CNN architectures. The proposed optimization process uses no assumptions (e.g., weight-sharing) or approximations (e.g., surrogate function). Two encoding methods are proposed related to the most critical computational parts of CNNs, and the metaheuristic approaches are compared for small population sizes. The results from these various metaheuristic approaches are evaluated using the metrics of computation time and classification accuracy. The final architecture obtained, which has a favorable tradeoff between the amount of computation and accuracy, is indicated. On a set of large-dimensional, multitasking, continuous optimization problems, multifactorial optimization has become one of the most promising paradigms for evolutionary multitasking within the field of computational intelligence. This thesis presents an in-depth analysis of this approach by considering several variations of the standard MFEA. By using a simpler structure together with some enhanced operators, two new MFEAs are proposed. In the approach presented, redundant hyperparameters are removed and the operators are simplified. Compared with the traditional MFEA, the proposed two MFEAs produce better results and are suitable for an embedded system implementation. To handle both non-convex continuous and NP-hard discrete optimization problems, this thesis proposes the class algorithm, a new type of evolutionary algorithm. The methodology is inspired by the concepts of division of labor and specialization. Individuals form subpopulations of different classes, and each class has its own characteristics. The entire population evolves through influences among individuals within and between the different subpopulations. The performance of the class algorithm surpasses other evolutionary algorithms for many test functions of single-objective continuous optimization benchmark problems. Compared with mature application software, the class algorithm also shows a competent ability to solve large-scale discrete optimization problems. The computation time is only 0.48 or 0.36 of published GA results when the class algorithm run in series or parallel, respectively, and the class algorithm is very suitable for use either in embedded systems or on a traditional hardware platform. In summary, compared with traditional EAs, the class algorithm not only has better performance but also has a smaller runtime. Cardiovascular diseases are the number one cause of death worldwide. Monitoring patients with heart disease can be done by analyzing the electrocardiogram. However, the large amount of data poses a burden for a system that is implemented as an embedded system with limited memory and computation capabilities. Traditionally, lossless compression methods have been favored to reduce the memory requirements due to the critical nature of the application. However, if the reconstruction of a lossy signal does not significantly affect the diagnosis capability, then those methods may become attractive due to their larger compression ratios. This thesis proposes a hybrid lossy/lossless compression system with good signal fidelity and compression ratio characteristics. The performance is evaluated after decompression using deep neural networks (DNNs) that have been shown to have good classification capabilities. For the CODE (Clinical Outcomes in Digital Electrocardiology) dataset, the proposed hybrid compressor can achieve an average compression ratio of 5.18 with a mean squared error of 0.20, and DNN-based diagnoses of the decompressed waveforms have, on average, only 0.8 additional erroneous diagnoses out of a total of 402 cases compared to using the original ECG data. For the PTB-XL dataset, the hybrid compressor can achieve a high average compression ratio of 4.91 with a mean squared error of 0.01. In addition, the decompressed ECGs have only a 2.46% lower macro averaged area under the receiver operating characteristic curve (AUC) score than when using the original ECGs.Item Frugal sensing and estimation over wireless networks(2014-04) Mehanna, OmarSpectrum sensing and channel estimation are two important examples of background tasks needed for efficient wireless network operations. Channel and spectrum state communication overheads can become a serious burden, unless appropriate sensing and estimation strategies are designed that can do the job well with very limited, judicious feedback. This thesis considers two `frugal' sensing and estimation problems in this regime: crowdsourced power spectrum sensing using a network of low-end sensors broadcasting few bits; and channel estimation and tracking for transmit beamforming in frequency-division duplex (FDD) mode.In the case of spectrum sensing, each sensor is assumed to pass the received signal through a random wideband filter, measure the average power at the output of the filter, and send out a single bit to a fusion center (FC) depending on its measurement. Exploiting linearity with respect to the autocorrelation as well as important non negativity properties in a novel linear programming (LP) formulation, it is shown that adequate power spectrum sensing is possible from few bits, even for dense spectra. The formulation can be viewed as generalizing classical nonparametric spectrum estimation to the case where the data is in the form of inequalities, rather than equalities. Taking into account fading and insufficient sample averaging considerations, a different convex maximum likelihood (ML) formulation is developed, outperforming the LP formulation when the power estimates prior to thresholding are noisy. Assuming availability of a downlink channel that the FC can use to send threshold information, active sensing strategies are developed which quickly narrow down the power spectrum estimate.For the downlink channel tracking problem, the receiver is assumed to send back to the transmitter a coarsely quantized version of the received transmitter-beamformed pilot signal, instead of sending quantized channel information as in codebook-based beamforming. A novel channel tracking approach is proposed that exploits the quantization bits in a maximum a posteriori (MAP) estimation formulation, and closed-form expressions for the channel estimation mean-squared error and the corresponding signal-to-noise ratio are derived under certain conditions.Item Parametric Frugal Sensing of Power Spectra for Moving Average Models(2015-05) Konar, AritraWideband spectrum sensing is a fundamental component of cognitive radio and other applications. A novel frugal sensing scheme was recently proposed as a means of crowdsourcing the task of spectrum sensing. Using a network of scattered low-end sensors transmitting randomly filtered power measurement bits to a fusion center, a non-parametric approach to spectral estimation was adopted to estimate the ambient power spectrum. Here, a parametric spectral estimation approach is considered within the context of frugal sensing. Assuming a Moving-Average (MA) representation for the signal of interest, the problem of estimating admissible MA parameters, and thus the MA power spectrum, from single bit quantized data is formulated. This turns out being a non-convex quadratically constrained quadratic program (QCQP), which is NP--Hard in general. Approximate solutions can be obtained via semi-definite relaxation (SDR) followed by randomization; but this rarely produces a feasible solution for this particular kind of QCQP. A new Sequential Parametric Convex Approximation (SPCA) method is proposed for this purpose, which can be initialized from an infeasible starting point, and yet still produce a feasible point for the QCQP, when one exists, with high probability. Simulations not only reveal the superior performance of the parametric techniques over the globally optimum solutions obtained from the non-parametric formulation, but also the better performance of the SPCA algorithm over the SDR technique.Item R_∞-matrices, triangular L_∞-bialgebras, and quantum_∞ groups(University of Minnesota. Institute for Mathematics and Its Applications, 2014-12) Bashkirov, Denis; Voronov, Alexander A.