All Articles
Summary of Computer Vision AI Application Domains and Their Mainstream Models
This article summarizes the differences among the primary application directions of computer vision in the current AI field (image classification, image localization, object detection, image segmentation, semantic segmentation, instance segmentation) as well as the mainstream models existing in each application direction. In the current field of computer vision, image classification, object detection, and image segmentation are the three most fundamental and rapidly developing domains. Image Classification The problem to be solved by the image classification task in the field of computer vision is: What is the category of the image contained within the image? For example, in the image below, the image recognition task yields a result indicating that the image contains a dog: !image.png Therefore, for traditional image classification tasks, the recognition result corresponds to the category of the most prominent object in the image (such as the dog in the image above), and the entire picture is directly classified according to this most prominent object. Consequently, for image samples in classification tasks, there is generally a very prominent and easily classifiable salient object present. As shown in the sub-figures below, the category corresponding to each image (effectively the most prominent object contained within it) can be determined at a glance: !image.png Of course, it is very likely that an image submitted to the model for classification contains multiple relatively prominent objects to be recognized, such as an image containing both a dog and a cat simultaneously: !image.png For this type of image classification task, a Multi-Label Classifier is required. In fact, from the perspective of classification, machine learning classification algorithms can be divided into three types: - Binary classification: Yes/No, True/False - Multi-class Classification: Returns one from multiple categories - Multi-Label Classification: Returns multiple from multiple categories !image.png For Multi-Class and Multi-Label classification algorithms, the difference in the model's classification and recognition output is that Multi-Class outputs 1 category out of multiple, while Multi-Label outputs multiple categories out of multiple: !image.png Regardless, the image classification task always focuses only on the categories of objects present in the image to be classified, and does not care about the locations or quantities of these objects in the image. Datasets used for model training in the field of image classification include: ImageNet, MNIST, CIFAR, etc. Classic models in the field of image classification include: - VGG, Visual Geometry Group at the University of Oxford - GoogLeNet (Inception), Google - ResNet, Microsoft Research - EfficientNet, Google - DenseNet, Cornell University - MobileNet, Google Object Detection If we need not only to identify the categories of objects in an image, but also to accurately locate their positions within the image, we need to use object detection tasks in computer vision. As mentioned above, image classification applications merely assign a single label to an image through recognition. In contrast, object detection applications provide spatial coordinates (bounding boxes) and classification labels for each detected object in images and videos, making it possible to analyze and process visual data at a more granular level. - The output of an image classification application is simply the label/labels of the objects contained in the image. - The output of an object detection application must include the category labels of the objects contained in the image, as well as rectangular boxes marking the locations of those objects within the image. If an image contains only a single most prominent target, and the application recognizes and locates this target, it is Image Localization. If an image contains many targets and the application must recognize and locate all of them, it is Object Detection: !image.png Classic models in the field of object detection include: !image.png Image Segmentation Image segmentation divides an image into distinct objects at the pixel level and distinguishes them using different color paint. Image Segmentation applications can be further subdivided into Semantic Segmentation and Instance Segmentation. The differences between the two are: - Semantic Segmentation: In the image segmentation process, all objects of the same category form a single class; therefore, all objects of the same category are colored with the same color. - Instance Segmentation: During the image segmentation process, multiple different instances of the same category are treated as separate segments, meaning each object of the same class is considered distinct. Therefore, even if they belong to the same class, each independent object is painted with a different color. The figure below is a typical example of semantic segmentation, where objects of different categories across the entire image are painted with different colors, but objects of the same category share the same color. !image.png The figure below is a typical example of instance segmentation. As can be seen, the three cats recognized in the image are segmented into distinct objects: !image.png Mainstream models in the field of image segmentation: - Mask R-CNN - U-Net - SegNet - Deeplab - PSPNet - Vision Transformers (ViTs) References - What are tasks like image classification, object detection, semantic segmentation, and instance segmentation in artificial intelligence? - Zhihu - Image Classification vs. Object Detection vs. Image Segmentation | by Pulkit Sharma | Analytics Vidhya | Medium - Object Detection Models - GeeksforGeeks - Image Segmentation Models - GeeksforGeeks
Introduction to ONNX Export and Deployment of PyTorch Models
This article provides a detailed summary of the processes and steps for exporting models developed and trained using the PyTorch framework into universal ONNX files, and deploying them using ONNXRuntime with Python/C++ access. The problem this article aims to solve is: How to export the network architecture and its parameters developed and trained in PyTorch into a deployable ONNX file? And how to deploy this ONNX file in other environments? This article uses the convolutional neural network trained in the article [[Implementing a Convolutional Neural Network for Handwritten Digit Recognition Based on PyTorch]] as an example to illustrate this complete process. Exporting PyTorch Models to pt/pth Files In the PyTorch framework, trained or training models and their parameters can be saved as .pt or .pth files. This allows developers to quickly restore the model and its trained parameter state later by loading these files, either to resume training or to directly perform inference on new data. Generally speaking, files with the .pt extension are used to save only the model parameters, while files with the .pth extension save both the model architecture and the parameters. Therefore, the former results in a smaller file size, but requires building the model's framework structure in advance before loading, because the .pt file does not contain the model architecture. The latter is more convenient to use as it contains both the model architecture and parameters internally, allowing direct loading upon use, though it comes with a larger file size. > Of course, in PyTorch, there is no official strict distinction or definition between .pt and .pth file extensions; they are mostly conventions adopted by the developer community. In other words, you can entirely use .pt to save a complete model, or .pth to save only the parameters. Below is a code example for saving a trained model and its parameters in PyTorch, and reloading the model parameters into a new model object: Exporting Models to ONNX Compared to the .pt/.pth file formats used for exporting PyTorch model parameters, the ONNX file format is much more universal for model deployment, sharing, and conversion across different model development frameworks. Therefore, trained models and their parameters can also be exported as ONNX files for model deployment and conversion to other frameworks. The prerequisites for exporting a PyTorch model to an ONNX file and performing inference on new samples using something like ONNXRuntime are installing a few dependency libraries: The code workflow for exporting a trained model and its parameters to an ONNX file is as follows: As indicated by the input_sample variable in the code above, when exporting an ONNX file, you need to explicitly specify the data input format for subsequent inference. After executing the above code, a model.onnx file will be generated in the current directory, which can then be used for ONNX deployment and conversion/usage across different frameworks. Verifying the ONNX File The ONNX file exported above can be checked for validity using the following code: The onnx.checker.check_model interface is used to verify the validity of the ONNX model file, ensuring it conforms to the ONNX specification and can be correctly loaded and executed by ONNX-compatible inference engines. This is a common pre-check step before performing inference with an ONNX model, which helps improve code robustness. Additionally, you can directly use the netron.app tool to inspect the ONNX file. This tool provides a graphical representation displaying very detailed model and parameter information, allowing you to intuitively view the model architecture and the training parameter information of each layer described in the file: !image.png Deploying and Testing the Model via ONNXRuntime The model architecture and parameters stored in an ONNX file are static representations and cannot directly call the eval() and predict() methods available in the PT format. Once the model structure and parameters are loaded from the ONNX file, inference on new sample data must be executed through the ONNXRuntime engine. Therefore, you must first install the onnxruntime package to support the deployment and inference of ONNX files. > As mentioned earlier, using ONNXRuntime to import ONNX files and perform inference on new samples under the PyTorch framework requires installing both the onnxruntime and protobuf packages via pip. Taking the model ONNX file exported earlier as an example, the test code for performing ONNXRuntime inference on a sample from the MNIST dataset is as follows: As can be seen, aside from using torchvision.datasets and torchvision.transforms to access the dataset (in fact, other methods could easily be used to prepare data samples for model inference, as long as the data format matches the requirements of the ONNX input sample), the inference on new sample data in the above code relies solely on numpy and onnxruntime, completely independent of the PyTorch framework used to develop the model. References - Chapter 1: Introduction to Model Deployment — mmdeploy 1.3.1 Documentation - Netron - ONNX Runtime | onnxruntime
Convolutional Neural Network for Handwritten Digit Recognition Based on PyTorch
This article provides a detailed introduction and summary of the code workflow for implementing a Convolutional Neural Network (CNN) based on the PyTorch framework to recognize the MNIST handwritten digit dataset, serving as an introductory foundation for practicing PyTorch and neural network programming. > The code analysis for handwritten digit recognition on the MNIST dataset in this article heavily references the relevant code implementation in Chapter 5 of the book "Deep Learning Principles and PyTorch Practice (Second Edition)". Essentially, it offers a more detailed interpretation of that code combined with my own understanding. The book provides very thorough and brilliant descriptions of the working principles of deep learning and neural networks across various fields, as well as their implementation using the PyTorch framework. It is an exceptional book for beginners in machine learning and neural networks, and I highly recommend it. The MNIST Handwritten Digit Dataset and Its Preparation MNIST is a handwritten digit dataset, which can be said to be the "Hello World" resource repository for entering the fields of machine learning and deep learning. Based on this dataset, whether through classical fully connected neural networks or convolutional neural networks tailored for the image domain, one can very conveniently design, test, train, and validate network structures, thereby gaining a deeper understanding of the relevant theories in deep learning and neural networks through code practice. Each sample in the MNIST dataset consists of a grayscale handwritten digit image (with a resolution of 28x28) and its corresponding digital label. The entire dataset contains 60,000 training samples and 10,000 test samples, so overall it can be divided into a training dataset and a test dataset. !image.png PyTorch has built-in support for various popular open-source training datasets, including MNIST. By executing the following code, you can automatically download the MNIST dataset to the local data subdirectory in the current directory. When executing the following code for the first time, PyTorch will automatically create the data subdirectory and download the dataset into it. Subsequent calls to this code will load the MNIST dataset directly from the data subdirectory. As can be seen, the above code uses train=True/False to distinguish and respectively download/load the training dataset train_set and test dataset test_dataset of the MNIST dataset. In addition, because the raw data in the dataset is in the form of ordinary arrays after loading, the option transform=transforms.ToTensor() is used during the dataset loading process to convert them into PyTorch Tensors, which is the standard data format that PyTorch models can process. PyTorch Data Loaders PyTorch specifically provides DataLoader and sampler to manage the loading and calling of training data, facilitating the efficient management and transfer of training data during subsequent batch training, testing, and validation of networks. Training and testing a neural network generally requires splitting the complete dataset into three completely independent parts to avoid mutual interference: the training set, the validation set, and the test set. > Why do datasets need to be split into independent training, validation, and test sets during use? The training set is relatively easy to understand: training data is fed into the network, the difference between the output results and the training sample labels is compared, and the network parameters are adjusted based on this difference until the difference between the output results and the labels is minimized. The test set is also relatively easy to understand: after the parameters in the neural network are trained, an independent dataset completely unrelated to the training data must be used to test and validate the performance of the trained network. This independent dataset used to test and evaluate the trained network is the test set. The validation set is used during the training of network parameters to periodically test the network parameters currently undergoing training using this independent dataset, allowing real-time insight into the dynamic improvement of network performance during the training process. The following code creates three independent data loaders based PyTorch's DataLoader, corresponding to the training dataset (the complete MNIST training dataset, totaling 60,000 samples), the validation dataset (the first 5,000 samples of the MNIST test dataset), and the test dataset (the last 5,000 samples of the MNIST test dataset). During subsequent simulated parameter training and testing, these three data loader objects can be called directly to access their corresponding datasets. batch_size and Mini-Batch During the definition of the three dataloader variables above, a batch_size parameter is specified. This batch_size parameter is a constant used to specify the number of samples retrieved from the dataset each time. In this case, whether for subsequent training, validation, or testing, the number of samples read from the dataset each time will be batch_size. For example, if batch_size=100 is set, 100 samples can be read each time the dataset is accessed via the dataloader interface. These 100 samples are fed into the network all at once for forward network operations to obtain outputs. The differences between the outputs and labels for these 100 samples are compared, and based on these differences, the parameters in the network are updated using gradient descent and backpropagation. In other words, the training of network parameters is carried out in units of this batch, rather than updating network parameters after calculating each individual sample. This is the concept of mini-batch in neural network training. For the 60,000 samples of the MNIST training dataset, with a batch_size of 100, one epoch of training over the entire training dataset will be divided into 600 mini-batches, meaning the network parameters will be updated 600 times. Of course, generally speaking, training a neural network usually involves many epochs over the same training dataset to achieve satisfactory results. CNN Network Structure Design Based on PyTorch Once the datasets for training, testing, and validation are prepared, the next step is to begin network design. The PyTorch framework provides a torch.nn.Module class as the base class for neural network modules, and the networks we build based on the PyTorch framework should derive from this base class. The main methods to override in this class include __init__ and forward. The former is used to declare the framework required for the network structure, i.e., the specification definitions for each layer; the latter implements the complete process of training samples from input to output, which is the definition of calculations for each layer and the connections between multiple layers. The following code constructs a simple convolutional neural network based on torch.nn.Module. As seen from the code below, this neural network contains two sets of convolution + pooling layers, as well as two fully connected layers, finally outputting the recognition results for 10 digits via the log_softmax activation function. - The first convolutional layer contains 4 convolution kernels with a size of 5x5. padding=2 is used to ensure that the convolution operation does not reduce the spatial dimensions. - The second convolutional layer contains 8 convolution kernels with a size of 5x5 and 4 channels (because the result of the previous convolution + pooling consists of 4 feature maps). Similarly, padding=2 is used to ensure the dimensions do not shrink. - The pooling layers used by both convolutional layers have the same specifications, and pooling calculations do not involve parameter training. Therefore, __init__ contains only one pooling layer, adopting 2x2 max pooling with a stride of 2. - The convolution + pooling layers are followed by two fully connected layers, with the final fully connected layer outputting recognition results via the log_softmax activation function. - A flatten operation must be performed between the last pooling layer and the first fully connected layer to convert multi-dimensional feature maps into the one-dimensional vectors required by the fully connected layer. This conversion is implemented via the x.view operation in the forward function. - The number of neurons in the first fully connected layer is 512, and the second fully connected layer has 10 neurons, corresponding to the 10 digits from 0 to 9. - The activation function for both convolutional layers and the first fully connected layer is ReLU, while the activation function for the final fully connected output layer is log_softmax. In addition to the __init__ and forward functions in the torch.nn.Module base class, the implementation of the convolutional neural network above also includes a custom retrieve_features method. By calling this function with a single sample data as input, the first and second layer feature maps of that sample under the current network parameter state can be retrieved, making it convenient for users to track the computational results of image data in the convolutional layers. PyTorch-Based Network Parameter Training Workflow Once the convolutional neural network defined above is ready, the next step is to create an instance of this network and launch and execute the complete parameter training workflow. First, define a variable based on the previously declared convolutional network class ConvNet, and then specify the loss function (CrossEntropyLoss) and gradient descent algorithm parameters to be used during the subsequent training process. During the training of network parameters, one epoch corresponds to one complete training pass over all 60,000 samples of the training dataset. Training can be repeated for multiple epochs to improve the network's recognition performance, such as setting num_epochs=20 or 30. During training, following the mini-batch training concept introduced above, each access to the training dataset yields a batch of training samples (e.g., 100 samples at a time). This mini-batch of samples is fed into the network all at once for computation to obtain output results. Then, the loss function specified earlier is used to evaluate the difference between the output results and the training sample labels, and based on this difference, gradient descent and backpropagation are used to update the network parameters. The training of one mini-batch of samples corresponds to one update of the network parameters. In addition, during parameter training, you can also periodically (e.g., every 100 mini-batches) use an independent validation set to test and print the recognition performance of the current network parameter state, allowing you to monitor the improvement progress of network performance in real-time during training. As can be seen from the code above, the complete process for the network can be divided into the parameter training process and the data testing/validation process, which correspond to the following code snippets respectively: In addition, during the training and validation/testing process in the above code, a rightness function is defined to calculate the accuracy of the current sample batch by comparing the predicted outputs with the sample labels, returning the number of correct samples and the total number of samples: Model Testing and Validation The final testing phase is relatively simple, largely identical to the periodic testing of network performance using the validation dataset during the training process mentioned above. Similar to the validation step during parameter training, the network enters evaluation mode, reads samples from the test dataset in batches of batch_size, feeds them into the network for inference output, and calculates the prediction accuracy for each batch of samples. At the same time, printing the error rates of the training data and validation data during training allows one to observe the convergence of recognition accuracy throughout the training process. With num_epochs=20 and batch_size=50, training the network and subsequently testing it on the test dataset yields a recognition accuracy as high as 98% or more: !image.png References - Chapter 5 of "Deep Learning Principles and PyTorch Practice (Second Edition)" - [Data Analysis & Machine Learning] Lecture 5.1: Introduction to Convolutional Neural Networks (CNN) - JamesLearningNote - Medium
Introduction to Convolutional Neural Network (CNN) Architecture
Based on online resources, this article provides a detailed analysis and summary of the architecture, data calculation, and transmission process of a simple Convolutional Neural Network (CNN), serving as a foundation for understanding CNN architectures. The most typical characteristic in the design of various network layers in a Convolutional Neural Network (CNN) architecture is the addition of convolutional layers and pooling layers to the classic neural network structure. For the operational logic and workflow of CNN convolution and pooling layers, please refer to the detailed summary in Detailed Explanation of Convolution and Pooling Calculations in Convolutional Neural Networks. Overall Architecture Design of Convolutional Neural Networks (CNN) For artificial intelligence development in fields such as computer vision, image recognition, and image object detection, Convolutional Neural Networks (CNNs) are an almost unavoidable topic. Therefore, to understand AI model algorithms and parameter training in these fields, one must first have a clear understanding of the overall architectural design of CNNs, the functions of each layer, and the data calculation and transmission workflow. The following figure shows a typical image recognition model based on a convolutional neural network: images are input from the input layer on the left, undergo a series of processing steps, and finally output results in the form of recognition probabilities at the output layer. !cnn_banner.png From the overall network structure, it can be seen that the entire network above can be divided into two parts: the feature extraction part and the image classification part. The structure of the feature extraction part is very regular. After 2D images are input from the input layer, they undergo convolution calculation (Convolution), followed by processing the convolution output with the ReLU activation function, and then pooling operations (Pooling). The convolution-ReLU activation-pooling process is repeated multiple times until a Flatten operation converts the multi-dimensional vectors into a one-dimensional vector for subsequent classification operations. The image classification part is based on the 1D feature vectors extracted by the feature extraction part and converted via the Flatten operation. It implements image classification tasks using multiple fully connected layers, ultimately using Softmax as the activation function of the final output layer to output the probabilistic recognition results of the image classification. > Compared to ordinary fully connected neural network structures, using convolutional neural networks for image data has inherent advantages. This is because fully connected neural networks do not consider the spatial structure of image data when inputting and processing data; for example, they treat distant and closely adjacent pixels equally, thus having to infer and learn the spatial structure inherent in the pixels solely from subsequent training data. In contrast, convolutional neural networks directly input and process data as 2D images, naturally accounting for the spatial structure between different pixel locations. Therefore, they are extremely suitable for 2D data structures such as images, enabling faster training and inference. The processing workflows of each layer are summarized in detail below in the order of the complete network structure execution. Convolutional Layer The primary role of the convolutional layer is to extract feature data from images, and subsequent classification layers distinguish and classify different images based on the feature data extracted by the convolutional layer. The operation of the convolutional layer is shown in the animated diagram below. The original image or the feature map from the previous layer is scanned using a convolution kernel as a unit. Depending on whether the input for the convolution calculation is a single-channel image or a multi-channel image/feature map, planar convolution kernels or volumetric convolution kernels are used respectively to perform convolution calculations on multi-dimensional data. The article Detailed Explanation of Convolution and Pooling Calculations in Convolutional Neural Networks provides a very detailed summary of how to perform planar and volumetric convolution calculations and can be referenced for further reading. !image.png A complete scan (convolution calculation) of the input image/feature map using a single convolution kernel yields a new feature map. The number of convolution kernels determines the number of scans executed and the number of new feature maps output for subsequent pooling operations. For example, the figure below uses 16 independent convolution kernels to perform convolution operations on the input image data, generating 16 feature maps: !image.png As mentioned above, the results of the convolution calculation must also be processed using the ReLU activation function before being passed to the next step for pooling operations. By applying the ReLU activation function to every pixel of the convolution calculation result, negative values are filtered out while preserving the shapes of the objects in the image. !image.png Pooling Layer The pooling operation in a convolutional neural network is used to reduce the dimensionality of the feature map data extracted by the convolutional layer. Its purpose is to reduce the amount of data while retaining salient features, thereby lowering the parameter data volume of the CNN as well as the computational cost of training and inference. The calculations performed by pooling are very simple. It only requires scanning the feature maps output by the previous convolutional layer in units of using Max Pooling (selecting and outputting the maximum value) and Average Pooling (averaging all values). The stride after each scan is , and it does not overlap with the previous scan (this is distinctly different from the convolution operation). Therefore, the final scanning result is a downscaled feature map reduced by a factor of in both horizontal and vertical directions. Taking the figure below as an example, the feature map output by the preceding convolutional layer has a resolution of . The pooling operation scans the feature map in blocks, and the final result of the pooling operation is a downscaled feature map. !image.png The pooling operation performs the aforementioned pooling calculations on a single feature map basis. Therefore, however many feature maps the preceding convolutional layer outputs, the pooling layer will output an equal number of downscaled feature maps. For detailed pooling operations, please refer to the article Detailed Explanation of Convolution and Pooling Calculations in Convolutional Neural Networks. Flatten As mentioned above, whether it is the convolutional layer or the pooling layer, their outputs can be considered as multiple 2D feature maps. After convolution and pooling processing, the CNN next converts the feature maps extracted in the previous stage into 1D data to facilitate processing in subsequent stages. This action of converting feature maps into 1D data is called Flatten. The processing in the Flatten stage is very straightforward: it simply unfolds the 2D feature maps from top to bottom and left to right: !image.png Of course, generally speaking, the output of the final pooling layer typically contains multiple feature maps. For such multi-feature-map Flatten operations, the first feature map is converted into a 1D format, and subsequent feature maps are sequentially unfolded right behind this 1D data in the same manner. Regardless, multiple feature maps are ultimately converted into a single 1D data sequence through the Flatten operation for subsequent classification operations. !image.png Recognition Layer As mentioned above, after the Flatten operation, the multi-dimensional feature vectors detected in the convolutional and pooling layers are converted into 1D vectors for image classification tasks. Compared to ordinary artificial neural network structures, the network in this recognition layer has nothing particularly special; it simply consists of multiple fully connected layers stacked with a final Softmax output layer: !image.png The fully connected layer network portion in the CNN architecture takes the feature vectors extracted by the preceding convolution and pooling stages as input, utilizing a network composed of multiple fully connected layers to implement image classification functionality. The final layer of the fully connected layers uses the Softmax activation function to output the recognition results of the image for various categories in the form of probabilities. Looking at the overall architecture of the CNN network above, the primary training parameters contained within the entire network are the convolution kernel weight parameters of the various convolutional layers in the feature extraction part, and the weight and bias parameters between the layers of the fully connected layers in the image classification part. During the training process of image recognition functions using the CNN network structure, based on the output of the network's final Softmax activation function and the training sample labels, the cross-entropy function is used as the loss function to train the convolution kernel weight parameters as well as the fully connected layer weight and bias parameters, until the network can output stable and correct recognition judgment results for input images. - For content regarding deep learning activation functions and loss functions, you can refer to Overview of Common Activation Functions in Deep Learning Neural Network Architectures and Overview of Common Loss Functions in Deep Learning Neural Network Architectures. References - Convolutional Neural Network | Deep Learning | Developers Breach - Flattening CNN layers for Neural Network and basic concepts | by Muhammad Shoaib Ali | Medium - [資料分析&機器學習] 第5.1講: 卷積神經網絡介紹(Convolutional Neural Network) - JamesLearningNote - Medium
Summary of Mathematical Fundamentals for Deep Learning
This article summarizes the mathematical foundations required to learn and understand deep learning, focusing on theories such as differentiation, partial differentiation, and composite functions, along with simple Python code demonstrations. Differentiation The theory of differentiation in mathematics is widely applied to solve various optimization problems in machine learning. Through basic calculus, we can obtain the slope of a function at a specific point, or accurately evaluate the instantaneous rate of change of a function at that point. To understand the concept of differentiation, let's consider the speed of a driving car: - By looking at the total distance traveled by the car over a period of time, we get the average speed during that interval. - By differentiating the relationship between the vehicle's traveled distance and time over that period, we obtain the instantaneous speed of the vehicle at any given point in time. Therefore, the definition of differentiation is the instantaneous rate of change of a function at a specific point, which is essentially the slope of the function at that point. Let us define a function as , and let be a number close to 0. Then the derivative of the function at point (which is effectively the slope at point ) can be calculated using the following expression: The interpretation of the above definition for differentiation is simply: How much change in the value of the function will be caused by a tiny change in the input . When using computer programs to calculate the derivative of a function, is often assigned a fixed value close to 0, and the change of the function at a certain point is then computed. This is known as numerical differentiation (as opposed to analytical derivation based on mathematical theory): For the derivation of mathematical differentiation, there are several commonly used properties that are crucial when studying deep learning theory. The first property is that when , its derivative can be derived as the following expression: In other words, the derivative of a standalone constant is 0 (which is natural, as a constant is a horizontal straight line whose slope is naturally 0), the derivative of is 1, the derivative of is , the derivative of is , and so on. This conclusion also holds for negative exponents; for example, the derivative of is . The second property is the linearity of differentiation. For functions and , and a constant , the following expressions hold: Basically, mastering the above theoretical knowledge is more than enough for calculating and deriving single-variable functions. Partial Differentiation The previous discussion on differentiation focused on single-variable functions with only one variable . However, in practice, the real-world problems we face in deep learning theory are, in the vast majority of cases, multi-variable functions resembling the following: For such multi-variable functions, we need to compute partial derivatives: When calculating a partial derivative, you only need to focus on the variable with respect to which you are differentiating, treating all other variables as constants. For example, let the function be . To find the partial derivative of this function with respect to , we treat as a constant (for instance, fixing at 1). Since the derivative of an independent constant is 0, the partial derivative of with respect to (note the change in the partial derivative symbol) is: Following the same logic to compute the partial derivative of with respect to : The following code example demonstrates the logic of partial differentiation. The process of numerical differentiation remains the same, except that when computing the partial derivative for a particular variable, we ensure that only that variable changes. The following code demonstrates the partial derivatives of the function at the point : Meanwhile, the gradient repeatedly mentioned in machine learning theory is actually the collection of partial derivatives of a function with respect to all its variables: !image.png Composite Functions A composite function is a function formed by combining multiple functions, where one function appears as a variable inside another function. For example: When differentiating composite functions—such as differentiating with respect to variable in the above example—the chain rule can be applied: If a function is expressed as a composite function, the derivative of that composite function can be expressed as the product of the derivatives of each individual function that makes up the composite function. The description of the chain rule above is relatively abstract, which can be better understood through the derivation process of the example below: Deep neural networks typically consist of many layers, and the goal of model training is to minimize the loss function of the final output layer as much as possible by adjusting the parameters of each layer. Therefore, from a broad perspective, the weights, biases, and other parameters of each layer can be viewed as variables of the loss function in the final layer. Minimizing the loss function requires determining the direction of parameter adjustment by calculating the partial derivatives (i.e., derivatives) of each parameter in every layer. The calculation of these partial derivatives relies on the aforementioned composite functions and the chain rule, working backward layer by layer from the output layer. References - Appendix of Math for Machine Learning in Plain English - Chapter 4 of Deep Learning from Scratch: Learning and Implementation with Python - Machine/Deep Learning - Basic Mathematics (Part 2): Gradient Descent | by Tommy Huang | Medium
Summary of Common Loss Functions in Deep Learning Neural Network Architectures
This article provides a detailed study and summary of commonly used loss functions in deep learning neural networks, organizing the characteristics and applicability of various loss functions to facilitate the future understanding of architectural designs in various deep neural networks. Definition of Neural Network Loss Functions The so-called loss function can be regarded as an evaluation metric that characterizes the performance of the current neural network. In other words, it measures to what extent the processing result of the input data by the current neural network is inconsistent with the actual ground-truth result. Therefore, a smaller value of the loss function indicates better performance. The goal of a neural network during the training process is to find the parameters (weights and biases of the neural network) that minimize the loss function. For example, as illustrated in the simple diagram below, we hope to train a neural network to fit a straight line that reflects, to the greatest extent possible, the correspondence between a given sample data input and its corresponding output . Later, based on this line and a new input , we can find its corresponding . Then, how do we evaluate how well this line fits the sample data? The answer is to use a loss function for evaluation. In the figure below, the loss function is the sum of the differences between and the corresponding output of the fitted line across all training sample data. For all training sample data, a larger accumulated sum of differences indicates a larger loss function and a poorer fit; conversely, a smaller loss function indicates a better fit. !image.png > There are two methods to calculate the difference between each sample point and the fitted line in the above figure. One is to use the absolute value of the difference . The other is the squared distance (i.e., using the Mean Squared Error as the loss function). The latter is generally used because during subsequent parameter learning, we need to calculate the derivative of the loss function, and the derivative of a squared term is simpler to compute compared to an absolute value. Why Do We Need Loss Functions? Taking the image classification task as an example, we are often puzzled: since the goal of training neural network parameters is to enable it to classify images more accurately, the most direct indicator for evaluating neural network performance is naturally the success rate of image classification and recognition. Why do we still need to introduce an additional loss function metric? The main reason is that the neural network structures in practical application scenarios are often very complex, with a large number of layers and parameters. In this case, minor modifications to the network weights and parameters during each training iteration will not produce a noticeable change in the number of images correctly classified after being fed into the neural network for learning and training. Consequently, the next step regarding how to adjust the network weights and other parameters would lose its guiding direction. Therefore, in practical neural network parameter learning, a more refined and smooth loss function in terms of output metric evaluation is required to assess the effect produced by each minor parameter adjustment, thereby determining the direction for the next parameter adjustment. Mean Squared Error (MSE) Mean Squared Error (MSE) is one of the most core loss functions in machine learning and statistics, making it particularly suitable for regression tasks (training models based on historical data to predict outputs corresponding to new input data). Mathematically, the definition of Mean Squared Error is as follows: Where is the number of samples, is the true value corresponding to the -th training sample, and is the predicted value calculated by the current model parameters for the input of the -th training sample. Thus, is the prediction error for each sample. By calculating it with the formula above, we obtain the arithmetic mean of the squared prediction errors across all samples, thereby reflecting the average error intensity of the model over the entire training dataset. Example Python implementation code for training Mean Squared Error is shown below: Cross-Entropy Loss Cross-entropy loss is primarily used to evaluate the difference between the true probability distribution provided by the sample data and the predicted probability distribution output by the model. Its calculation formula is as follows: Where: - : Represents the total number of output categories of the model. For example, in handwritten character recognition, the recognition results are digits 0-9, so the total number of output categories is 10. - : Represents the true probability that the current training sample belongs to category , typically represented using one-hot encoding. - : Represents the probability output by the current model that the predicted data belongs to category , where the sum of probabilities across all categories is 1, meaning . > The so-called one-hot encoding is an array where only the label of the correct answer is 1, and all others are 0. For example, the one-hot encoding corresponding to the true probability of a handwritten digit 6 training sample is [0,0,0,0,0,0,1,0,0,0], where only the index position corresponding to the digit 6 has a probability of 1, and all others are 0. Therefore, for multi-classification tasks, if the true probabilities of the current training samples use one-hot encoding, it actually means that during the cross-entropy loss calculation, only the term where the true category contributes to the error calculation, while all other categories are 0. For example, for a certain multi-classification training task, if the true label of a training sample is , and the probability predicted by the model for this sample input is , then the cross-entropy loss corresponding to this sample is calculated as: . In other words, you only need to take the log value of the predicted probability for the term where the true label probability is 1. Example Python implementation code for the cross-entropy loss function is shown below. Note that because is mathematically undefined, cross-entropy implementation codes need to handle this case specially by substituting with a very small positive number. References - "Neural Networks and Deep Learning: A Plain English Introduction" (《深入浅出神经网络与深度学习》) - Neural Network Algorithms - Loss Function_Neural Network Loss Function - CSDN Blog
An Overview of Commonly Used Activation Functions in Deep Learning Neural Network Architectures
This article provides a detailed study and summary of commonly used activation functions in deep learning neural networks, organizing their characteristics and applicability to facilitate the future understanding of various deep neural network architectures. What is an Activation Function? The architecture of a typical neuron in a neural network is shown in the figure below: !image.png - The inputs of the input layer are multiplied by their respective weight parameters , accumulated, and then added to the bias parameter of the neuron. This constitutes the input signal of the neuron. - During internal processing, the neuron takes the above input signal as a parameter and executes its activation function . The output of the activation function serves as the output of the neuron. Therefore, the so-called activation function of a neuron is essentially a non-linear mapping and transformation performed by the ganglion on the weighted sum of input signals. Through the mapping of the activation function, the computational output of the neural network can approximate any complex function (such as curved boundary segmentation or multi-target overlapping detection), solving complex problems in practical applications. Why Are Activation Functions Needed? Without activation functions, regardless of the number of layers in a neural network, its output is merely a linear combination of the inputs. This makes it equivalent to a single-layer perceptron, rendering the number of layers (the so-called depth) meaningless. The problem that activation functions solve is the mapping from a linear space to a non-linear space. > _Without activation functions, no matter how many layers a neural network has, its output is ultimately a linear transformation of the input data, making it incapable of handling complex non-linear problems. However, the problems we need to solve in real life are basically non-linear. By introducing non-linear mappings, activation functions enable neural networks to approximate arbitrarily complex non-linear functions._ Step Function The step function is the activation function used by perceptrons in the early days of neural network development. The step function uses a threshold (usually 0) as a boundary to output binary values: when the input is greater than or equal to 0, the step function outputs 1; otherwise, it outputs 0. Its graph is shown below: !image.png Main characteristics of the step activation function: - Its output jumps in a step-like manner, being discontinuous at and forming an obvious "breakpoint." - Its output range has only two states, 0 and 1 (corresponding to all-or-nothing situations), and cannot express intermediate states or probabilistic information. Therefore, it is relatively suitable for binary classification tasks. - Because it forms a discontinuity at , it cannot be differentiated (calculating derivatives requires the function to change smoothly in a local interval). Consequently, it cannot be used with gradient descent (which relies on calculating partial derivatives of the activation function) or backpropagation to train parameters. > Imagine that when using a step function as an activation function, minor modifications to weights and biases in the neural network might completely flip the output, thereby causing complex and dramatic changes in how the neural network processes other input data. Therefore, in this case, it is difficult to optimize the accuracy of the neural network by gradually modifying the network's weights and bias parameters. Because of characteristics such as having only two values in its output range and being non-differentiable, the step function is no longer used in the hidden layers of mainstream networks. It only finds limited applications in fields like theoretical teaching (perceptrons) and lightweight binary classification systems in embedded systems. The Python implementation of the step function is as follows: Sigmoid Function The Sigmoid function, also known as the Logistic function, is an S-shaped curve function. It is one of the classic activation functions in neural networks, and has been widely used especially in early deep learning and binary classification tasks. The mathematical expression of the Sigmoid function is as follows: Its graph is shown below: !image.png > As seen from the graphical comparison, the Sigmoid activation function is similar to the step activation function; however, when using the Sigmoid activation function, minor changes in weights and biases will only cause minor changes in the output. This point is crucial for neural networks using sigmoid as an activation function to properly learn and adjust their parameters. The main characteristics of the Sigmoid activation function are summarized as follows: - Similar to the step function, the output value ranges between 0 and 1, and it is point-symmetric at the point , making it very suitable for representing probabilities. - Strictly monotonically increasing as the input increases; the output approaches 1 as the input increases, and approaches 0 as it decreases. - The curve is smooth and continuously differentiable everywhere, allowing successive approximation of extreme values through subsequent gradient descent. The Sigmoid activation function was widely used in the early stages of neural network development, but due to numerous drawbacks, it is rarely used in current neural network architectures: - Vanishing gradient problem. As can be seen from the Sigmoid graph above, when the input is greater than 5 or less than -5, the entire graph becomes almost flat, and the derivative approaches 0. This causes parameters to fail to update during the process of seeking the extreme values of the loss function using gradient descent, leading to training stagnation. This is especially pronounced when the network has many layers. - Non-zero-centered problem. The output range of Sigmoid is between 0 and 1, and it is strictly monotonically increasing with the gain of input . Therefore, regardless of the input, its output is always greater than 0. This restricts the direction of subsequent gradient-based parameter adjustments, causing the adjustment direction to oscillate back and forth, resulting in a convergence speed significantly lower than zero-centered activation functions (such as Tanh). - High computational cost. Because it involves exponential operations, its computational cost is too high compared to modern activation functions like ReLU, dragging down training speed. The Python implementation of the Sigmoid function is as follows: Tanh Function The Tanh function, also known as the hyperbolic tangent function, can be considered a variant of the Sigmoid function (essentially, Tanh is a zero-centered improvement of Sigmoid). The biggest difference from the Sigmoid function is that the Tanh function ranges monotonically from -1 to 1 and has a zero mean. The mathematical expression of the Tanh function is: Although its mathematical expression looks complex, its graph is very similar to Sigmoid. The graph of the Tanh function is shown below: !image.png The main characteristics of the Tanh activation function are summarized as follows: - The output range is between -1 and 1, and it is symmetric about the origin (), which avoids the non-zero-centered problem of Sigmoid, making the input distribution of subsequent layers more balanced (no longer constantly positive) and thus accelerating the convergence of gradient descent. - Like Sigmoid, the output increases monotonically and smoothly as the input increases, and the function is differentiable everywhere. - In terms of gradient variation, compared to Sigmoid, the gradient is larger for the same input, which alleviates the vanishing gradient problem to a certain extent. In summary, the Tanh function solves the non-zero-centered problem of Sigmoid, can accelerate the parameter training process based on gradient descent, and alleviates the vanishing gradient problem to some extent (though it still exists when the input deviates too far from 0). However, because it requires exponential calculations, the problem of excessively high computational cost remains. The Python implementation of the Tanh function is as follows (in fact, numpy already has its own implementation of the tanh function): ReLU Function While Sigmoid was a widely used activation function in the early days of neural network development, the most widely used activation function in the deep learning era is ReLU (Rectified Linear Unit). When the input is greater than 0, the ReLU function outputs the original value directly; when the output is less than 0, it returns 0 directly. Its output range is from 0 to positive infinity, and its mathematical expression is: The graph of the ReLU function is shown below: !image.png The main characteristics of the ReLU activation function are summarized as follows: - Piecewise linear. The positive region remains linear with a constant gradient of 1, while the negative region is forced to zero. - Computationally efficient. This is the biggest advantage of ReLU. It does not require any trigonometric or exponential operations, and its calculation speed is more than 10 times faster than Sigmoid and Tanh. - Sparse activation. When the input is negative, the output is 0, at which point the neuron is in an inactive state. This is equivalent to allowing only some critical neurons to be active, forcing the entire network to prioritize the most important features. - Solves the vanishing gradient problem. The gradient is constantly 1 in the positive region, which can be used to support ultra-deep networks. Moreover, the linear positive region avoids saturation, leading to larger gradient update magnitudes and a significantly faster convergence rate. The biggest problem with ReLU is the dying ReLU problem. This refers to the phenomenon where certain neurons may permanently fail when using the ReLU activation function. During training, when a neuron's weight update causes its input to be constantly negative, the output of ReLU is constantly 0, and the gradient is also 0. This causes the neuron to no longer update its parameters and remain in a state of practical permanent failure. When the proportion of dead neurons in the model is large, it will significantly affect the working performance and accuracy of the entire network. The Python implementation of the ReLU function is as follows: Leaky ReLU Leaky ReLU (Leaky Rectified Linear Unit) is an improved version of the ReLU activation function designed to solve the aforementioned dying ReLU problem. Leaky ReLU introduces a small positive slope (typically 0.01) for the negative region: Therefore, in the negative region, Leaky ReLU is no longer constantly 0, but a straight line with a slope of , and its value range also becomes negative infinity to positive infinity. !image.png The design above can effectively solve the dying ReLU problem: the negative-region slope retains a tiny gradient, ensuring that weights can still be updated during backpropagation. In addition, Leaky ReLU maintains the advantage of high computational efficiency. Its calculation requires only a single multiplication and comparison, and its computational complexity is similar to that of ReLU, which is far lower than the exponential operations of Sigmoid/Tanh. Softmax Function The Softmax function is the core activation function used in neural networks to handle multi-class classification tasks (such as recognition classification tasks on the MNIST handwritten digit dataset). Its core role is to convert the raw outputs of a neural network into a probability distribution, making the probability values of each category non-negative and summing up to 1. The primary function of the Softmax function is probability conversion for multi-class classification. Its input is a multi-class vector array (with a total of category outputs), and its corresponding mathematical expression is: In the softmax expression above, the numerator takes the exponent of the input for the -th class to amplify the differences among categories; the denominator takes the sum of the exponents of all category inputs to achieve normalization, ensuring that the sum of the total output probabilities is 1. !image.png The main characteristics of the Softmax activation function are summarized as follows: - Probability normalization. The output values corresponding to all inputs range between 0 and 1, and the sum of all outputs is 1, satisfying the axioms of probability, which makes it very suitable for classification decisions. - Amplification of differences. Through the exponential function in softmax calculations, the output weights of high inputs can be significantly amplified, making the model perform more decisively on high-confidence categories in the output and reducing ambiguous predictions. - The Softmax activation function combined with the cross-entropy loss function is the standard combination for classification tasks, featuring efficient gradient calculation and fast convergence. The Python implementation of the Softmax function is as follows: References - Activation Functions in Neural Networks, Their Respective Pros and Cons, and How to Choose an Activation Function - CSDN Blog - Comprehensive Analysis of Activation Functions - CSDN Blog
Detailed Explanation of Convolution and Pooling Calculations in Convolutional Neural Networks
Based on study materials, this article uses a simple convolutional neural network architecture as an example to provide a detailed summary and explanation of the complete calculation process for convolution and pooling operations in CNNs. > This article is essentially part of the reading notes for Chapter 5 of Deep Learning Principles and PyTorch Practice (2nd Edition), organized in my own words to deepen memory and understanding. This book is a rare and excellent resource for explaining deep learning theory and PyTorch practice. It is very suitable for beginners and is strongly recommended. The figure below shows a typical example of a convolutional neural network structure used for handwritten image target detection: !image.png Looking at the structure of the convolutional neural network above, it can generally be divided into four layers: - Input Layer: The input data is a handwritten image represented by a 2D matrix. Each element in the matrix corresponds to a pixel of the image, and each pixel contains a grayscale value ranging from 0 to 255. The larger the value, the whiter it is; otherwise, the blacker it is. - Convolutional and Pooling Layers: Corresponding to the two sets of interleaved convolutional and pooling layers in the architecture above, they extract features from the input image step-by-step and reduce their dimensionality, retaining only the important features contained in the image and ultimately outputting them to the feature layer to perform classification tasks. - Feature Layer: Also known as the fully connected layer, it primarily performs classification tasks based on the outputs of the convolutional and pooling layers. It flattens the features output by the preceding pooling layer into a 1D vector, integrates these recognized global features, and applies activation functions (such as ReLU, Sigmoid, etc.) for non-linear transformation to obtain classification probabilities or predicted values. - Output Layer: The final output layer contains 10 neurons. For each handwritten image fed into the neural network for recognition, it outputs the recognition probabilities for numbers 0 through 9 across these 10 neurons. The sum of the recognition probabilities for all neurons equals 1. The number corresponding to the highest probability is selected as the output. The purpose of this article is to explain in detail how the calculation operations unfold in the convolutional and pooling layers after an image enters the CNN through the input layer in the above architecture. Calculation of the Convolutional Layer A concept closely related to image convolution is the convolution kernel. A convolution kernel can be thought of as a very small-resolution image. The convolution operation performed on the original image is actually using this small-resolution image (i.e., the convolution kernel) to scan and match pixel by pixel from left to right and top to bottom. The result of the match is a new image, called a feature map. A feature map is a grayscale image where the grayscale value of each pixel represents the degree of match between the original image at that location and the small convolution kernel image, as shown in the figure below: !image.png The following figure illustrates the convolution calculation process between the original image and the small convolution kernel image. Here, the convolution kernel is a 3x3 small image, where each pixel is a real value representing the computational weight of the convolutional neural network. - The weight value corresponding to each pixel in the convolution kernel is the parameter that the subsequent convolutional neural network needs to train. !image.png As shown in the figure above, the first step is to perform convolution calculation using the convolution kernel on the top-left corner of the original image. The convolution calculation performed here multiplies each pixel in the 3x3 area of the top-left corner of the original image (which matches the size of the convolution kernel) by the pixel at the corresponding position in the convolution kernel image. Then, the 9 product results in this 3x3 area are added together to obtain the value of the first pixel in the top-left corner of the feature map. Next, the convolution kernel moves one pixel to the right on the original image and continues the same operation to obtain the value of the second pixel in the first row of the feature map. !image.png Following the calculation logic above, the convolution kernel is moved to the right sequentially, one pixel at a time, until it reaches the far right of the original image. Then it moves down one pixel to start the convolution for the second row. This process repeats in a loop until the convolution kernel moves to the bottom-right corner of the original image, completing the scan of the entire image and yielding a complete feature map. !image.png According to the above convolution calculation logic, when the size of the original image is and the size of the convolution kernel is , the size of the feature map after the convolution calculation is . In other words, the feature map will be slightly smaller than the original image. If you want the feature map size to remain consistent with the original image, you need to pad a border of zeros around the original image before performing the convolution calculation: !image.png The complete convolution operation performed on the original image using a single convolution kernel yields a feature map corresponding to that convolution kernel. In practice, during the execution of a convolutional neural network, multiple different convolution kernels are often used to independently perform convolution operations on the original image, thereby generating multiple feature maps. For example, the figure below shows convolution operations performed on an original image with a resolution of 125x125 using 100 different convolution kernels, generating 100 feature maps with a resolution of 125x125: !image.png This is also the reason why, in the CNN architecture, the 2D input image turns into a cuboid after passing through the convolution calculation of the first convolutional layer. Calculation of the Pooling Layer As shown in the shaded part of the overall CNN architecture below, each convolutional layer is followed by a pooling layer used to simplify the features extracted by the convolutional layer, retaining only the coarse-grained information in the feature map. After convolving the original image with 4 convolution kernels, convolutional layer C1 obtains four 28x28 feature maps. The operation of pooling layer P1 is to perform a pooling operation on each feature map to shrink it, ultimately resulting in four pooled feature maps with a resolution of 14x14. This greatly reduces the data volume while retaining important features. !image.png The pooling operation is very simple. Taking the figure below as an example, the 3x3 pooling operation performed here extracts the pixel with the maximum value from each 3x3 region in the original feature map based on the maximum-value principle, serving as the pooling output for the entire region. In this way, each 3x3 region of the original feature map corresponds to a single pixel in the pooled feature map. !image.png It should be noted that the movement during the pooling operation is non-overlapping, which differs from convolution. Therefore, the output after pooling is much smaller than the original feature map. Consequently, the 9x9 original feature map above, after a 3x3 pooling operation, results in a 3x3 pooled feature map. Similarly, the four original 28x28 resolution feature maps of convolutional layer C1 in the architecture diagram above become four 14x14 resolution pooled feature maps after a 2x2 pooling operation. 3D Convolution Calculation The output of the pooling layer is a 3D cuboid (multiple pooled feature maps). When this pooled data is fed into the next convolutional layer for calculation, its convolution workflow differs from that when the original image is fed into the first convolutional layer (where the original image is a 2D matrix). When the pooling layer's output is fed into the next convolutional layer, the required convolution calculation is a 3D convolution calculation. The figure below shows an example of a 3D convolution kernel. On the left is the pooling layer, and on the right is the next convolutional layer. For the 3D convolution calculation performed on the pooling layer, you must first select a 3D convolution kernel whose depth matches the output of the pooling layer. Then, use this 3D convolution kernel to perform a 3D convolution operation on the multiple pooled feature maps output by the pooling layer. Each operation result (obtained by multiplying corresponding positions across the entire cuboid involved in the 3D convolution operation and then summing them up) corresponds to a single pixel in the convolutional layer. Performing a complete scan of the pooling layer's output using the 3D convolution kernel corresponds to one feature map of the convolutional layer. !image.png Similarly, there can be multiple 3D convolution kernels. Using one 3D convolution kernel to perform one 3D calculation yields one feature map; thus, the operation results of multiple 3D convolution kernels correspond to multiple feature maps of the convolutional layer. Therefore, the output of the convolutional layer is also a cuboid. In summary, for a simple convolutional neural network, the calculation workflow of convolution and pooling is: 2D image output -> first-layer convolution calculation (2D convolution) -> first-layer pooling operation -> second-layer convolution calculation (3D convolution) -> second-layer pooling calculation -> second-layer convolution calculation (3D convolution)... This cycle repeats until the image features calculated by convolution are finally organized and passed to the feature layer for the image classification task. In summary, for a convolutional neural network (CNN), the parameters of its convolutional layers are the parameters within the convolution kernels contained in each convolutional layer. These are the main parameters trained during the CNN training process. Meanwhile, because the pooling layer simply performs 2x2 or 3x3 max-pooling operations, it has no parameters to train. Reference Materials - Deep Learning Principles and PyTorch Practice (2nd Edition), Chapter 5: Handwritten Digit Recognizer - Understanding Convolutional Neural Networks - The Role of Each Layer in Convolutional Neural Networks
Lens Shading Issues in Camera Imaging
This article provides a detailed summary of the pervasive Lens Shading issue in optical imaging, analyzing the root causes and organizing the strategies for resolving it during camera product image tuning. Lens Shading Issues in Lenses By definition, Lens Shading is a phenomenon in optical imaging systems where image brightness or color is uneven due to the physical characteristics of the lens. Depending on its specific manifestation on the image, Lens Shading can be divided into two types: Luma Shading and Color Shading. !image.png In the Luma Shading example on the left, it is clearly observable that the central area of the image formed through the lens is brighter, gradually darkening towards the periphery to form a "vignetting" effect, with the edges of the image being the darkest. Color Shading, shown on the right, occurs when the colors at the center and edges of the image do not match, manifesting, for instance, as reddish or bluish corners. Summary of Causes for Luma Shading The main factors leading to lens Luma Shading include the following. Natural Optical Attenuation A lens is essentially a convex lens whose light-gathering capability at the center is superior to that at the edges, thereby causing the light intensity to attenuate from the center to the surroundings. As shown in the figure below, comparing parallel light rays and oblique light rays : !image.png For a camera without distortion, the illumination attenuation at the periphery of the image follows a attenuation law. For parallel light rays (), since the light originating from point is parallel to the optical axis, its incident angle . At this point, its light intensity follows the formula , which represents an unattenuated state. For oblique light rays (), the light originating from point forms an angle with the optical axis, and its light intensity will thus attenuate to . For example, when , the light intensity emitted at this angle will only be about 56% (), resulting in edge brightness that is significantly lower than the center. Mechanical Obstruction by the Lens Structure The structural components of the lens itself and the lens holder may obstruct some of the obliquely incident light, preventing these light rays from reaching the peripheral areas of the image. Consequently, more light enters and brightens the central area, while less light reaches and darkens the edge areas. As shown in the figure below. !image.png CRA Matching Issues Between Lens and Image Sensor Luma Shading problems caused by CRA mismatches between the lens and the image sensor have been explained in great detail in the article [[Explanation and Matching of CRA Parameters for Image Sensors and Lenses]]. Specifically, when Lens CRA 95% to obtain its RAW format image. - Decompose the RAW image into four channels (R, Gr, Gb, B) according to its Bayer arrangement. If the size of a RAW image is , the size of each channel will be . - Divide the RAW image of each channel into multiple grids horizontally and vertically, and then calculate the average brightness of each grid per channel. - Based on the brightness of the center grid of each channel image as a reference (gain = 1), calculate the compensation gain for the peripheral grids according to the attenuation ratio. The approximate formula is: . - According to the above formula, calculate independent gain tables for each grid of each channel separately (e.g., a grid) and store them in the ISP's LUT (Lookup Table). Subsequent LSC gain compensation for the RAW image will be based on this table. After completing the LSC calibration in ISP Tuning, subsequent LSC correction on the RAW image output by the Sensor applies the corresponding gain tables to the R, Gr, Gb, and B channels of the RAW image separately for gain compensation, avoiding color cross-talk caused by shared parameters. Under high-demand scenarios, calibrations are often performed multiple times under different color temperatures (such as 3000K, 5000K, 7500K) to generate multiple sets of gain tables for different color temperatures. During subsequent LSC correction, these tables are dynamically switched based on the current color temperature to adapt to the ambient light source, achieving a better LSC effect. References - 【ISP】Brief Analysis of Lens Shading - CSDN Blog - Brief Analysis of Shading Correction Principles - Image Shading | ISO 17957 | Image Quality Factors | Image Engineering
Summary of Audio Interfaces in Consumer Audio Codecs
Based on a study and summary of the datasheet for the commonly used TLV320AIC3101 Audio Codec in the market, this article delves into the input/output interfaces of general audio codecs and their internal signal processing flows, establishing a more comprehensive understanding of audio codecs and their audio processing pipelines. The full text is divided into three parts: - Analog Audio Input Interfaces - Analog Audio Output Interfaces - Digital Audio Interfaces Introduction to Analog Audio Input Interfaces Taking the TLV320AIC3101 as an example, this Audio Codec supports 6 single-ended analog audio input pins, including 2 fully differential inputs (offering strong noise immunity, suitable for long-distance microphone connections) and 2 single-ended inputs (simplifying circuit design). Through internal register configurations and external circuitry, it can be set to work in single-ended microphone, differential microphone, and single-ended/differential Line In operating modes. The figure below shows a simplified block diagram of the TLV320AIC3101. For analog audio inputs, the standard approach is: if single-ended microphones are used, two independent left and right channel single-ended microphones can be connected via L2_L and L2_R. If differential microphones are used, two independent left and right channel differential microphones can be connected via L1_L+, L1_L- and L1_R+, L1_R- respectively. !image.png Of course, the differential-mode pins L1_L and L1_R can also be used as single-ended microphone interfaces, allowing up to 6 single-ended microphones to be connected simultaneously. However, it should be noted that the TLV320AIC3101 has only 2 internal ADCs, which can process two analog audio data streams in parallel simultaneously. Therefore, if more analog audio inputs are connected at the same time, they must share the same ADC by multiplexing across multiple audio data channels. Since the audio sampling rate is configured at the ADC, having multiple audio inputs share a single ADC results in a reduced actual sampling rate per channel. For example, a single ADC supports a maximum sampling rate of 96 kHz; if 3 inputs are activated simultaneously, the sampling rate for each analog audio input drops to just 32 kHz. The diagram below illustrates multiple analog audio inputs sharing a single ADC in the TLV320AIC3101, where three analog audio input pins—MIC1L/LINE1L, MIC2L/LINE2L/MICDET, and MIC1R/LINE1R—share the same Left ADC. !image.png Mic In Mic In is the operating mode for connecting analog microphone signals, and it is also the most common analog audio input mode for Audio Codecs. The typical peak-to-peak input signal voltage for electret microphones commonly used in our products generally ranges from 10 mV to 100 mV, whereas the ADC input range is typically designed to be ±1 V (fully differential) or 0–2 V (single-ended). Therefore, before ADC conversion, the microphone output signal must first pass through a preamplifier to boost its signal level, maximizing the dynamic range of the ADC conversion signal level. Additionally, to allow electret microphones to function properly, the Audio Codec must also supply power to the electret microphone via the MICBIAS pin (the typical supply voltage for MICBIAS on the TLV320AIC3101 is 2V–2.5V). The typical connection for supplying power to an electret microphone via MICBIAS involves routing the MICBIAS voltage through a resistor (such as 2.2 kΩ) to the positive terminal of the microphone, forming a current loop (with a typical current of about 0.5 mA). The diagram below shows a single-ended microphone connection: !image.png To further improve the noise-reduction performance of captured microphone signals, a differential connection method can be used to suppress common-mode noise to the greatest extent possible: !image.png Line In The Line In interface (along with the Line Out interface for outputs) is used to interface with other consumer electronic devices or professional audio equipment. The TLV320AIC3101 provides two sets of Line In interfaces, which also support both single-ended and differential input modes, configured via registers. In single-ended input mode, the peak-to-peak voltage () ranges from 0.5 V to 2 V (exceeding the supply voltage must be avoided during interfacing); in differential input mode, the range is 1 V to 4 V (with the common-mode voltage required to be within ). Therefore, when operating in Line In mode, the voltage dynamic range is already close to the full-scale input of the ADC, meaning no preamplification is required. Only appropriate gain fine-tuning within the PGA is necessary—this is what sets Line In mode apart from MIC In mode. Furthermore, when using Line In, there is no need to supply power to the upstream device of the Line In signal via MICBIAS, which is another key difference between the two. The diagram below illustrates the differential connection of Mic In and Line In for the Allwinner H3 platform Audio Codec. !image.png Introduction to Analog Audio Output Interfaces Just as an Audio Codec has analog input interfaces (MIC, Line In), it naturally has analog output interfaces (Line Out, HPOUT). Generally speaking, the analog output interfaces of an Audio Codec are used to amplify the analog audio signals converted by its internal DAC via audio power amplifiers, in order to drive speakers of varying power levels, headphones, or to feed downstream professional audio equipment via Line Out. Most Audio Codecs contain two main types of analog audio output interfaces: - HPOUT: Includes a small power amplifier following the DAC output of the Audio Codec, allowing it to directly drive headphones and low-power speakers. - Line Out: Essentially the raw DAC output of the Audio Codec without internal power amplification, making it suitable for connecting to a subsequent power amplifier stage to drive higher-power speakers. Line Out Regarding analog audio output interfaces, the Line Out interface is suitable for connecting high-impedance equipment, such as professional audio mixing consoles and power amplifiers driving larger speakers. The peak-to-peak output voltage of this interface generally ranges between 1 V and 2 V, characterized by high input impedance, low output impedance, and very low output current (), making it ideal for high-impedance loads (). Taking the TLV320AIC3101 as an example, this Audio Codec contains two sets of independently configurable, fully functional differential Line Out interfaces, both capable of driving 10 kΩ differential loads. The analog audio signals output from the DAC undergo flexible audio signal path control, are mixed with multiple voice signals from different sources in the mixing module, undergo gain adjustment, and are finally output as differential audio signals. Below is the block diagram of the Line Out output structure for the TLV320AIC3101. !image.png - The analog audio signals output by the left and right channel DACs are connected via register configurations to DAC_L1/2/3 and DAC_R1/2/3, respectively. - When mixing functionality is required in the analog audio output, DAC_L1 and DAC_R1 (DAC_L2 and DAC_R2 are used for HPOUT output), along with the PGA-amplified ADC input signals from the analog input side (PGA_L/PGA_R—allowing analog audio signals entering the Audio Codec to bypass the ADC-DAC processing loop after PGA amplification and go directly to the analog audio output stage), are fed into two independent Volume Controls Mixing modules to perform internal mixing. - If mixing functionality is not needed, the mixing module can be bypassed, and DAC_L3 and DAC_R3 can be used directly as the Line Out analog audio output signals. - The output of the mixing module, combined with the audio path control selection of DAC_L3/DAC_R3, undergoes a 0–9 dB gain adjustment and is ultimately converted by the differential circuit at the Line Out interface for differential output. It should be noted that in the circuit block diagram above, the processing path of the analog voice signal output at the Line OUT interface includes a 0–9 dB gain adjustment function. This function is solely used to adjust the signal voltage amplitude from the DAC to the Line Out or mixer, matching the input sensitivity of backend devices (such as power amplifiers or mixing consoles) to avoid excessively weak or overloaded signals. This function can only boost voltage, not current driving capability. Consequently, the Line Out drive circuit capability is only and cannot directly drive low-impedance loads (such as headphones or speakers). If a subsequent high-power speaker needs to be driven, an additional Class-AB or Class-D power amplifier must be added for power amplification. Volume Controls Mixing Module Let us briefly introduce the Volume Controls Mixing module included inside the TLV320AIC3101. This module can mix multiple analog audio signals from different sources with varying amplitudes and gains to produce a mixed audio signal output at the Line Out interface. Supported mixing signal sources include: the left channel ADC analog input audio signal PGA_L, the right channel ADC analog input audio signal PGA_R, the left channel DAC analog audio signal DAC_L1, and the right channel DAC analog audio signal DAC_R1. The diagram below shows the internal working architecture of the mixing module. As can be seen, four independent analog audio signals pass through register-controlled individual gain controls (with a gain control range from 0 to -78 dB) before being summed together and output. !image.png Of course, the mixing module is entirely optional. If mixing is not required in the application, the simplest approach is to directly output DAC_L3 and DAC_R3 via the audio path selection control at the Line Out interface. HPOUT The biggest difference between HPOUT and Line Out is that the HPOUT interface incorporates power amplification, allowing it to directly drive low-power headphones and speaker peripherals. The advantage of using this interface is that for driving low-power peripherals, no external power amplifier is required, resulting in a simpler circuit design and lower cost. The peak-to-peak output voltage of the Audio Codec's HPOUT interface is generally 1V–2V (with adjustable gain), matching the input requirements of headphones (typical headphone sensitivity is 90–110 dB/mW). Its output impedance is (low output impedance ensures flat frequency response without low-frequency attenuation when driving headphones), making it suitable for driving low-impedance loads (16–32 ), with a relatively high output current (30 mA)—a sharp contrast to the Line Out interface, which suits high-impedance loads and has a minuscule output current. HPOUT is specifically designed for driving headphones, and its output power range (15–30 mW) fully matches the requirements of typical headphones (e.g., the power of a 32Ω headphone at 1 Vrms is approximately 30 mW). > The naming of HPOUT can easily cause confusion. For instance, in the TLV320AIC3101, HPOUT stands for High Power Output. However, in reality, the built-in amplifier power of the HPOUT interface is quite small, suitable only for driving headphones or very small speakers, so it can hardly be called "high power." The term "High Power" here is relative to the Line Out interface, because Line Out lacks a built-in amplifier and has a tiny output current. In fact, if this interface is solely used to drive headphone loads, calling it "Head Phone Output" would be much more appropriate. Once you fully understand the Line Out structure diagram above, understanding the HPOUT structure diagram becomes much easier. The DAC output, mixing module, and analog audio signal path management modules are identical to those of Line Out. The biggest difference between the two is that the HPOUT interface includes power amplification prior to the output stage (indicated by the final triangle), enabling HPOUT to drive low-power loads within its power limits. !image.png Aside from the Line Out and HPOUT interfaces mentioned above, some Audio Codecs also feature a built-in analog audio output interface with even higher output power called SPKOUT. The SPKOUT interface can drive 4–8 speakers, achieving output powers around 100–150 mW to meet the driving needs of small-power speakers. In fact, the TLV320AIC3101 combines HPOUT and SPKOUT into a comprehensive HPOUT interface that can drive both headphone loads and low-power speakers. The table below summarizes the application-characteristic differences between the Line Out and HPOUT interfaces: !image.png Introduction to Digital Audio Interfaces In addition to supporting the analog audio input and output interfaces described above, typical Audio Codecs also provide digital audio data transmission interfaces. At a minimum, these digital transmission interfaces are used for: - Transmitting digital audio data—obtained by AD-converting analog audio captured by microphones—via the digital audio interface to a DSP for further processing. - Receiving digital audio data transmitted from a DSP via the digital audio interface, which is then DA-converted and played back on speakers, headphones, or other devices. For Audio Codecs, the most commonly used digital audio interface is the I2S interface. A single I2S interface can be used to transmit dual-channel audio data containing two independent audio streams. To transmit more audio channels simultaneously using the I2S hardware interface, some Audio Codecs also support Time-Division Multiplexing (TDM) over the I2S hardware interface to transmit multi-channel audio data protocols. I2S Interface I2S (Inter-IC Sound) is a synchronous serial interface dedicated to audio data transmission, characterized by simplicity, high efficiency, and strong anti-interference capability. The I2S interface transmits digital audio data, typically represented in PCM (Pulse Code Modulation) format—which essentially corresponds to digital audio data converted from voice captured by microphones in the Audio Codec, or digital audio data received from a DSP that needs to be DA-converted by the Audio Codec and played back through speakers. Parameters associated with digital audio data transmitted via I2S include the sampling rate (e.g., 16 kHz, 32 kHz, 44.1 kHz, 48 kHz, etc.), bit depth (e.g., 16-bit, 24-bit, etc.), and channel count (e.g., stereo dual-channel). During actual transmission, the I2S interface synchronizes audio data transmission using clock signals. It uses multiple clock signals to control data transmission and reception, ensuring accurate audio data transfer. The signal lines of the I2S hardware interface include: - SDIN/SDOUT (Serial Data): Serial data lines used for transmitting digital audio data externally or internally. - WCLK (Word Clock): Word select signal used to indicate whether the current transmission corresponds to the left or right channel data. Typically, WCLK high level indicates left channel data, and low level indicates right channel data. - BCLK (Bit Clock): Serial bit clock line used to control the data sampling rate. Audio data is sampled on the rising or falling edge of BCLK. The diagram below shows the I2S working timing diagram of the TLV320AIC3101. In the I2S standard, data transmission for the MSB data bit of the next channel begins delayed by 1 clock cycle (BCLK) relative to the WCLK edge (rising or falling edge). !image.png TDM Interface As mentioned above, a single I2S interface can only transmit two independent audio data streams using left and right channels, forming a dual-channel stereo audio effect. However, in many scenarios (such as products featuring microphone arrays) where we want to transmit multiple independent audio streams, besides using multiple independent I2S interfaces, a TDM interface can also be used to solve this problem. At the hardware interface level, the TDM interface shares the same set of physical pins (DOUT/DIN, SCK, WS) as the I2S interface, differing only in the data transmission protocol. The I2S mode is used to transmit dual-channel (stereo) audio data; the TDM mode is used to simultaneously transmit multi-channel (2–8 channels) audio data over the same physical interface, multiplexing the same data line through time slot division. In audio codecs, the TDM interface can be used to transmit sample data of multiple independent audio channels over a single data line, with each channel's data transmitted sequentially in pre-allocated time slots. During operation, the TDM interface and protocol divide the data transmission cycle into multiple time slots, with each slot assigned to an independent audio data channel. For example, if TDM supports 8 time slots, it can transmit 8 channels of audio data, with the 8 channels taking turns using their respective time slots to transmit data. The diagram below shows the TDM data transmission timing diagram of the TLV320AIC3101. As can be seen, multi-channel audio data transmission using TDM mode utilizes the same hardware interfaces and physical pins as I2S, except that Word Clock is no longer used to distinguish left and right channels. Instead, transmission time slots for different audio streams are fixedly assigned on Data In and Data Out. Therefore, for upstream and downstream devices using the TDM interface for multi-channel audio transmission, it is necessary to negotiate and configure in advance how many audio streams are transmitted simultaneously, as well as the time slots assigned to each audio stream in the transmission timing. !image.png When using the TDM interface, it should be noted that the TDM interface is still not fully standardized. There is a lack of unified standards regarding the details of its communication protocols (such as the number of time slots, alignment methods, and clock phases), and implementations from different manufacturers may vary to some extent. Consequently, compatibility issues may arise during the debugging and integration of upstream and downstream devices using the TDM interface, requiring the tuning of certain parameters to resolve. PCM Interface In addition to the I2S and TDM interfaces mentioned above, some Audio Codecs also feature a PCM interface used for transmitting digital audio signals (the TLV320AIC3101 does not have a PCM interface). This interface is mostly used in voice communication and multi-channel data transmission scenarios. The PCM interface is generally used to transmit uncompressed multi-channel linear PCM audio data (the ability to more flexibly support multiple channels is the biggest difference between the PCM interface and I2S), supporting various scenarios such as voice and music. During actual use, a typical application scenario for the PCM interface is connecting with single-device multi-channel or multi-device multi-channel audio equipment, enabling the transmission of audio data across multiple independent audio channels via a single set of PCM interfaces. For example: - A single 8-channel ADC chip connects to the Host via a PCM interface, with each time slot transmitting sample data for one channel. - The Host connects to 4 single-channel microphones via a single PCM interface (with each device occupying 1 time slot). !image.png From a hardware definition standpoint, the PCM interface is very similar to the I2S interface and typically includes the following data lines: - PCM_CLK (Bit Clock): Generated and controlled by the master device, with a frequency equal to . For example, at a 16-bit, 8-channel, 48 kHz sampling rate, . - PCM_SYNC (Frame Sync): Generated and controlled by the master device, identifying the start position of an audio frame, with a frequency equal to the sampling rate. - PCM_DIN (Data Input) and PCM_DOUT (Data Output): Transmit multi-channel audio data. As seen from the data lines included in the PCM interface above, the definitions of PCM_CLK, PCM_DIN, and PCM_DOUT are identical to those of I2S. The main difference between the two lies in WCLK for I2S versus PCM_SYNC for PCM. !image.png To thoroughly explain the differences between the two, one must first understand what an "audio frame" means in PCM_SYNC, which serves as the frame synchronization signal in the PCM interface. An audio frame transmitted in the PCM interface is always initiated by a transition of PCM_SYNC (the transition edge is programmable). An audio frame contains one audio sample for each of the multiple channels connected to the PCM interface, with each audio sample occupying a fixed time slot. The time slot occupied by each channel within the audio frame is configured in the registers on both the Host and Device ends. When a PCM interface connects to multi-channel audio equipment in this manner, a single audio frame is divided into multiple time slots, each containing the audio sample data for one channel. The entire audio frame is triggered by the PCM_SYNC signal to start a complete data transmission cycle (i.e., the period of this audio frame). As illustrated in the four-channel audio transmission diagram using a PCM interface below, a single transition of PCM_SYNC (FS) initiates the transmission of one PCM Frame. A PCM Frame contains audio sample data for four independent channels, with each channel's sample data occupying a fixed time slot. !image.png Therefore, essentially, the PCM interface also uses TDM (Time-Division Multiplexing) to transmit audio sample data across multiple channels on a single DIN and DOUT line using different time slots. References - TLV320AIC3101 Datasheet - I2S/PCM - Zhihu
An Analysis of the Secure Boot Workflow Implementation on the Ingenic IPC Platform
This article provides a comprehensive summary and explanation of the Secure Boot function workflow used on the domestic Ingenic IPC platform. Boot Workflow of the Ingenic IPC Platform Similar to the power-on startup, boot image loading, and execution workflow of most embedded hardware platforms, the Ingenic IPC platform also goes through a multi-stage process of image loading and execution after power-on. Reference 2 provides an excellent flowchart explaining the boot image loading process of the Ingenic IPC platform. !image.png 1. Upon chip power-on, the internal hardwired Rom Loader executes first. Based on the detection of the current Power On Pin, the Rom loader determines the boot mode (Flash, SD card, or USB). 2. When booting from Flash, the Rom loader first reads the first-stage bootloader saved at the header of the Flash according to specific rules, which is the SPL Image. 3. The Rom loader loads the SPL image into its internal SRAM and then hands over control to the SPL. The SPL is responsible for performing further initialization, particularly configuring the DDR. Once initialization is complete, it loads the second-stage bootloader from the Flash—namely, U-Boot—into the DDR to start execution. 4. U-Boot serves as the primary bootloader, performing more comprehensive initialization settings for the system, loading the Kernel and Rootfs images into the DDR, and launching the Linux Kernel. At this point, the system is fully booted. The main reason for adding an extra SPL stage between the Romloader and U-Boot is that the internal SRAM space of the Ingenic chip is too small, whereas the U-Boot image is too large for the SRAM to accommodate. Therefore, an intermediate bootloader is required to initialize the larger DDR space and then load the U-Boot image into the DDR for execution. Subsequent images such as the Kernel and Rootfs are all loaded into the DDR for execution. Algorithms Used in the Ingenic IPC Secure Boot Workflow In the Secure Boot implementation provided by the Ingenic IPC solution, two primary algorithms are utilized: - RSA2048: An asymmetric encryption/decryption algorithm. It is primarily used to encrypt/decrypt and then match and compare the SHA256 digest information of the image contents for signature verification. - SHA256: A hash algorithm. It is used to perform digest operations on the SPL Image and the public key of the RSA2048 algorithm. Since the SHA256 hash algorithm does not require key storage, the only key-related information actually stored inside the Ingenic chip's Efuse is the public key digest information of the RSA2048 algorithm. Information Structures in Flash and Efuse Information Structure in the Chip's Efuse Within the EFuse (i.e., OTP) inside the Ingenic chip, the Secure Boot implementation mainly includes two parts: the Secure boot enable configuration information in otp4, and the SHA256 digest information of the RSA algorithm public key in otp9. During chip power-on boot, the Rom loader checks the Secure boot enable configuration written in otp4 to determine whether to boot in Normal boot mode or Secure Boot mode. As mentioned above, since the entire Secure Boot implementation only uses the RSA2048 and SHA256 algorithms—and SHA256 requires no key information to be stored—the OTP only needs to store information related to the RSA public key. !image.png Flash Information Structure Related to Secure Boot The figure below shows the Flash structure corresponding to the SPL Image generated by the PC-side script provided by Ingenic when the Secure Boot mode is enabled. The key points are: - The SPL Image itself is stored starting at offset 0x800 in the Flash, and the SPL Image is not encrypted. - The RSA algorithm public key is stored between Flash 0x300 and 0x400. - When generating the Secure Boot image file based on the raw SPL Image, the PC-side script performs a SHA256 digest operation on the SPL Image content, and then encrypts the result using the RSA private key. The encrypted digest of the SPL Image is stored between Flash 0x200 and 0x300 to be used subsequently for verifying the SPL Image signature. !image.png Secure Boot Startup Workflow on the Ingenic IPC Platform Based on the above information, the following summarizes the startup workflow where the Rom Loader validates the SPL Image under Secure Boot mode after the Ingenic IPC platform is powered on. The complete startup flowchart is shown below: !image.png The above workflow can be divided into two stages: RSA Public Key Verification and SPL Image Digest Signature Verification. RSA Public Key Verification Upon power-on, the Rom Loader detects the Secure Boot configuration written in OTP4. In Secure Boot mode, it reads the SHA256 digest of the RSA public key burned during the production phase from OTP9. Then, the Rom Loader reads the RSA public key from the Flash location 0x300-0x400, performs a SHA256 operation on it, and compares this calculation result with the public key SHA256 digest read from OTP9. If the above comparison matches, it proves that the public key in the OTP matches the one in the Flash, and the boot process continues; otherwise, the boot fails. SPL Image Digest Signature Verification Next, the Rom Loader performs a SHA256 signature operation on the raw contents of the SPL Image to obtain its own calculation result. It then reads the digest—which was encrypted by the PC-side script using the RSA private key—from Flash 0x200-0x300, and decrypts it using the RSA public key read from the Flash to obtain the decrypted SHA256 digest. The two digests are then compared and matched. If they are identical, it indicates that the SPL Image signature verification has passed, and this image was indeed officially released by the hardware vendor (since only the official party possesses the RSA private key used to encrypt and sign the image file). Summary Overall, compared to the relatively complete and comprehensive Secure Boot functionality implemented on the Novatek IPC platform, the Secure Boot feature on the Ingenic platform is much simpler, requiring downstream developers to do a fair amount of work to achieve a fully realized Secure Boot. While the Novatek platform's SDK standardizes full-access protection for the first-stage bootloader, U-Boot, and Kernel images in the boot workflow, the Ingenic platform's Secure Boot protects only the SPL Image by default. Of course, because the RSA verification public key information is already stored in the Ingenic chip's Efuse under Secure Boot mode, downstream developers can easily implement signature verification protection for U-Boot and Kernel images themselves based on the same logic. It is just that this process is not provided by default in Ingenic's SDK and must be implemented by developers independently. Furthermore, in Ingenic's original Secure Boot implementation, only the RSA public key is stored in the Efuse, providing only signature verification protection for the image contents without cryptographic encryption protection. Compared to the Novatek platform, the security level is lower in this regard, as the Novatek platform encrypts the image file contents concurrently with signature verification. Fortunately, Ingenic has opened up a portion of the storage space within the Efuse for downstream developers. Developers can leverage this Efuse storage space to perform customized Secure Boot development, including adding encryption functions specifically for U-Boot, Kernel, and other images. References - Ingenic Secure Boot User Guide - Ingenic Zeratul Development (2) — U-Boot Boot Analysis_Ingenic T41 SDK Package - CSDN Blog
Introduction to Audio 3A Processing: AEC, ANS, and AGC
This series of articles provides a comprehensive summary of the concepts, theories, and working principles of 3A audio processing (AGC, AEC, and ANS) in the field of consumer cameras. It lays the foundation for audio effect tuning and troubleshooting in related products. In the consumer camera domain, similar to the 3A processing in video and imaging—namely AE (Auto Exposure), AWB (Auto White Balance), and AF (Auto Focus)—there is a corresponding 3A processing framework for audio data, known as Audio 3A: AGC (Auto Gain Control), AEC (Acoustic Echo Cancellation), and ANS (Auto Noise Suppression). The purpose of this article is to summarize the theoretical technologies and operational workflows of these audio 3A processes. The following diagram illustrates a typical audio 3A processing workflow for two-way voice communication with a remote end via a server: - The data captured by the microphone is first processed alongside the received remote voice data to identify echoes, and the AEC (Acoustic Echo Cancellation) algorithm is applied to eliminate them. - Next, the echo-cancelled voice signal passes through the ANS (Auto Noise Suppression) module to filter out audio noise and improve the quality of the captured audio. - AGC is then used to dynamically and adaptively adjust the audio gain according to the amplitude of the captured voice, bringing the signal strength transmitted to the remote end to an appropriate level. - The raw audio data is encoded and transmitted over the network and server to the remote end of the voice communication. - Upon receiving the voice information, the remote end organizes the audio data using various buffering and packet-loss retransmission mechanisms. - The remote end decodes the received compressed audio data to restore the raw audio format. - Finally, combined with local mixing and other settings, the raw audio data is played out through the speaker. !image.png Overall, in a full-duplex two-way audio intercom application, the general processing order for audio 3A is: AEC (Acoustic Echo Cancellation) first, followed by ANS (Audio Noise Suppression), and finally AGC (Auto Gain Control). Acoustic Echo Cancellation (AEC) The problem that AEC aims to solve is that during a two-way voice call (such as a conference call), the sound emitted by the speaker at one end (originating from the other end) is captured by the microphone at the same end and transmitted back to the opposite end. This causes the voices of both parties to bounce back and forth, creating a noticeable echo effect. The diagram below illustrates how echoes occur in two-way full-duplex voice calls and conference calls: !image.png As seen from the diagram illustrating the generation principle of two-way call echoes, echo problems only occur during two-way full-duplex voice calls (where both the microphone and speaker at both ends remain active). In half-duplex systems (similar to walkie-talkies), echoes and echo cancellation are non-existent. This is because, in half-duplex voice communication, only one party's microphone is open at any given time. Furthermore, a user always listens to the other party first before pressing the talk button to send their own voice. Consequently, whenever a party's microphone is open, their own speaker is silent, naturally eliminating the presence of echoes and the need for echo cancellation. Of course, this communication mode is relatively inefficient since only one person can speak at a time. In practical echo cancellation application scenarios, echoes are often categorized into two types based on their propagation paths: - Direct Echo: Also known as linear echo. This occurs when the voice signal transmitted from the remote end is played out through the local speaker and directly captured by the local microphone without undergoing any reflection or refraction. This type of direct echo is unaffected by the environment and depends solely on the distance and position between the local microphone and speaker, making the time delay of the echo relatively easy to estimate. - Indirect Echo: Also known as non-linear echo. This occurs when the remote voice signal played through the local speaker reflects and refracts off various obstacles, walls, and floors in the local environment before being captured by the local microphone. Therefore, the magnitude of indirect echo is related to the room environment, layout, and even the sound absorption coefficients of various obstacles. When the communication device itself is moving, the positions of the speaker and microphone relative to the environmental obstacles are constantly changing, resulting in a dynamically varying echo delay. Consequently, this type of echo is relatively difficult to handle. Processing Logic of Echo Cancellation From the perspective of how echoes are generated, consider two communicating parties, A and B. A's voice signal is captured by A's microphone and transmitted to B. After being played out by B's speaker, it is re-captured by B's microphone and mixed with B's own voice signal before being sent back to A. As a result, A hears their own voice coming out of their speaker. Therefore, throughout this process, party A is innocent; the echo problem arises because B's microphone captures both its own voice signal and A's voice signal played by the speaker, mixing them together and sending them to A without any processing, which causes the echo at A's end. !image.png Therefore, solving the echo problem must start at party B's end. After capturing the audio data, B's end must identify and separate its own voice signal from A's voice signal being played by the speaker, eliminate A's voice signal from the mixed audio data, retain only its own voice signal, and then transmit it to A. This is the exact logic behind introducing the AEC echo cancellation module at party B's end. Of course, because party A and party B are engaged in full-duplex two-way voice communication where both sides are symmetrical, in practice, AEC echo cancellation functions and modules need to be implemented simultaneously at both ends. Each module identifies and eliminates the remote voice signal (played by its own speaker) from the mixed signal captured by its local microphone. Working Principle of Echo Cancellation As mentioned above, echo cancellation involves identifying the remote voice signal (played by the local speaker) within the mixed signal received at the microphone and eliminating it, ensuring that only the local raw voice signal is transmitted to the other end. To meet this design requirement, an echo cancellation module generally consists of the following four sub-modules. 1. Delay Estimation Module To eliminate the remote voice signal played by the speaker, the system must first identify this signal within the mixed signal captured by the local microphone and determine the time difference between the remote voice signal played by the speaker and the voice signal captured by the local microphone. With this time difference, subsequent processing modules can accurately locate the remote voice signal that needs to be attenuated within the microphone-captured data. How is the time difference between the two voice signals (the received remote voice signal data and the local microphone-captured voice signal data) determined? The answer is to slice both voice signals, perform a Fourier transform on each sliced audio data frame to obtain their spectral data, and then compare their similarities in the frequency domain using cross-power spectral calculations until the slice where the spectral behavior of both audio data sets matches best is found. This identifies the time difference. 2. Linear Echo Cancellation Module Based on Adaptive Filters This module is the core of the entire echo cancellation system. In the practical implementation of the echo cancellation algorithm, the locally received remote voice data is used as input. Through dynamic iteration and updating of the filter weight parameters of the adaptive filter, its output signal is made as close as possible to the actual echo signal data. Finally, the predicted echo data output by the adaptive filter is subtracted from the mixed signal captured by the microphone, achieving the goal of preserving pure local voice data. 3. Non-linear Residual Echo Suppression Module The linear echo cancellation module implemented using adaptive filters described above can only eliminate the linear echo components of the acoustic path and cannot completely eliminate non-linear distortion. Therefore, after filtering out the linear echoes using the adaptive filter, a non-linear module must be employed to further suppress residual non-linear echoes through techniques such as frequency-domain energy analysis and dynamic gain control. 4. Double-Talk Detection Module This module dynamically monitors the remote voice data and the voice data captured by the local microphone to determine whether only one party is speaking or both parties are speaking simultaneously. If both parties are talking at the same time, the audio data captured by the local microphone will contain the voice signals of both parties. In this case, the local voice signal might be misjudged as residual echo, leading to incorrect updates of the filter parameters. Therefore, when the algorithm identifies that both parties are in a double-talk state via the double-talk detection module, it freezes the updating of the filter weights to prevent the algorithm from diverging due to noise interference. Conversely, when the double-talk detection module detects that only one party is speaking, it can resume updating the adaptive filter weights and increase the echo suppression intensity of the filter to completely eliminate residual echoes, thereby achieving a better intercom effect. Referencing the diagram below helps in better understanding the complete implementation of the echo cancellation algorithm and the roles played by its sub-modules in the overall echo cancellation workflow. !image.png Auto Noise Suppression (ANS) The raw sound signals captured by microphones in commonly used products like smartphones and consumer cameras often contain a significant amount of background noise. This not only degrades the user experience during recording and playback but also reduces the compression efficiency of audio encoding. Therefore, suppressing audio background noise is necessary, which is where the ANS (Auto Noise Suppression) function comes into play. > In addition to the aforementioned ANS technology, audio 3A also includes ANC (Active Noise Cancellation) for controlling environmental noise. The main difference between ANS and ANC lies in their approach: the former adopts a passive noise reduction method, analyzing frequency components in environmental noise to perform targeted filtering, preserving the clear voice part as much as possible and improving sound quality. The latter, commonly found in popular active noise-canceling headphones currently on the market, analyzes the frequency and phase of environmental noise sound waves to generate sound signals with opposite phases to achieve noise cancellation. Therefore, the primary difference is that ANS uses passive filtering for noise reduction, whereas ANC actively generates waveforms with opposite phases to the noise to cancel each other out. Generally speaking, in the consumer camera domain, only ANS is used for processing sound data captured by camera microphones; hence, this article focuses solely on the passive noise reduction mechanism of ANS. Sources of Audio Background Noise Generally speaking, in analog systems, noise can originate from various stages of the system, making analog systems more susceptible to interference and noise generation. For digital devices, anti-interference and noise performance are much better, with the weak links for noise ingress typically being the A/D and D/A conversion sections. In the consumer electronics sector, considering factors such as cost, analog audio components and processing circuits remain mainstream in applications. Consequently, sound quality degradation caused by background noise from various disturbances in these products often presents a major challenge in product development. In general, the audio noise in consumer camera products primarily originates from the following aspects: - Inherent noise during the operation of electronic components. For example, electronic components (capacitors, resistors, etc.) operating at high frequencies inherently generate subtle white noise. These noises are easily picked up by nearby high-sensitivity microphones, amplified, and made clearly audible. - Electromagnetic interference. During the high-frequency switching operation of circuits—especially when wireless communication circuit functions operate within the internal space of the device—high-frequency electromagnetic wave radiation interference leaks into the analog audio signal via audio transmission lines, generating electromagnetic interference noise. - Power supply interference and ground loop noise. Spikes, pulses, and surges in the power supply circuit leak into the audio lines via power paths, and ground potential differences between different circuit sections cause current loops. These power and ground noises have a significant impact on small analog audio signals. As seen from the summary of audio noise sources above, most of these audio noises originate from the design and implementation of hardware devices. Some noise sources can be optimized—for example, by optimizing power grounding design and adjusting layout to improve electromagnetic interference conditions in audio routing modules. However, it is undeniable that structural and electronic design limitations in consumer electronics products can cause some interference factors to not be completely resolved at the hardware level. In such cases, methods must be devised to utilize audio enhancement features within the Audio Codec or even software-level audio noise filtering algorithms to improve the quality of audio capture. Typical Classification and Processing of Noise In general, audio background noise can be divided into two categories: stationary noise and transient noise. The primary characteristic of stationary noise is that its statistical properties (such as mean, variance, and spectral distribution) do not change over time. - In the time domain, stationary noise exhibits small amplitude fluctuations and displays a regular distribution (such as a Gaussian distribution). - In the frequency domain, its spectrum is continuous and stable; for example, white noise covers the entire frequency band. - Typical stationary noise scenarios include: Gaussian white noise, thermal noise of electronic devices, quantization noise, and continuous background environmental noise (such as air conditioner hum or fan noise). Transient noise, on the other hand, is highly bursty and short-lived (ranging from milliseconds to seconds), and its statistical properties generally change drastically over time. - In the time domain, transient noise typically features high amplitude and short-duration impulse characteristics. - In the frequency domain, its energy is generally concentrated in high frequencies or specific frequency bands. - Typical transient noises include impulse noise (keyboard typing, door knocking, switching sounds), mechanical shock sounds, and irregular electromagnetic interference (such as electrostatic discharge). Generally speaking, stationary noise is relatively easy to handle because its statistical properties and spectral distribution are relatively stable and easy to recognize, allowing for targeted suppression based on its spectral characteristics and specific signal features. However, transient noise exhibits strong burstiness in the time domain, and its spectrum almost always overlaps with that of normal speech, making it very difficult to suppress. Consequently, there is currently a lack of good technical solutions for effectively suppressing transient noise. Typical Audio Noise Reduction Algorithms: Spectral Subtraction and Adaptive LMS Filtering The figure below shows the processing flowchart of a typical audio noise reduction algorithm. Whether using traditional signal processing algorithms or AI-enhanced processing approaches, the ultimate goal is to detect the type of audio noise, model the noise, and then perform targeted processing based on the noise model. !image.png Below, two commonly used traditional digital signal processing algorithms for audio noise reduction—Spectral Subtraction and Adaptive LMS Filtering—are explained and illustrated. Spectral Subtraction Spectral subtraction is a frequency-domain-based noise suppression technique. It estimates and models the noise spectrum and subtracts the noise components from the noise-containing mixed signal in the frequency domain. Since it is a frequency-domain-based processing method, the overall processing workflow inevitably involves using the FFT (Fast Fourier Transform) to convert the time-domain audio sample data sequence into a frequency-domain spectral sequence, processing it in the frequency domain, and then converting the processed signal back to the time domain using the IFFT (Inverse Fast Fourier Transform). Such a processing workflow naturally involves a significant amount of computation. The overall working principle of spectral subtraction is to estimate and establish a noise power spectrum template during voice silence periods (i.e., time intervals containing only noise), and then subtract the power spectrum of the noise template from the power spectrum of the noise-containing mixed signal, thereby achieving the goal of retaining only the speech components. The workflow for audio noise cancellation based on spectral subtraction is roughly as follows: - First, slice the time-domain audio sampling data sequence. The slice duration is fixed and matches the number of points for the FFT calculation. For example, with a 16 kHz audio sampling rate and a 512-point FFT calculation, audio data is sliced into units of 32 ms, where each audio slice unit constitutes an audio frame. - Perform an FFT calculation on this audio frame to obtain its spectral data. - Use an independent VAD (Voice Activity Detection) module to detect whether the current period is a speech silence period. When in a silence period, dynamically update the noise power spectrum template based on the spectral sequence currently calculated via FFT, making it consistent with the current noise spectrum state. - Perform spectral subtraction in the frequency domain on the spectral data sequence of the current audio frame: subtract the power spectrum of the current noise power spectrum module at the corresponding frequencies from the spectral data of the current audio frame, yielding frequency-domain data that retains only the speech components. - Finally, convert the processed data back from the frequency domain to the time domain using IFFT, re-combining it into an audio data sequence to restore the continuous time-domain audio sampled signal. Adaptive LMS Filtering Algorithm Unlike spectral subtraction, which operates in the frequency domain, the LMS (Least Mean Squares) filtering algorithm is an adaptive filter algorithm operating in the time domain. It dynamically compares the real-time audio sampled data sequence with the filter's output and iteratively adjusts the filter coefficients based on the calculated error to minimize the mean square error between the desired signal and the filter's output. The adaptive LMS filter primarily has two parameters: - Filter Order (): The length of the noise used for modeling, which matches the time correlation of the noise. A larger means that filtering each audio sample will reference more historical sampling data, resulting in a higher computational load. Typical values range between 64 and 256. - Step Size (): Primarily used to control the convergence speed and stability of noise change tracking, with typical values ranging between 0.001 and 0.01. Execution workflow of the adaptive LMS filtering algorithm: - First, place the current input audio sampling data to be filtered, along with the historical audio sampling data preceding this sample, into a buffer named History. - Based on the historical buffer data and the current parameters of the LMS filter, calculate the output value of the LMS filter: accumulate the products of the historical sampling data History[i] and the filter parameters W[i]. - Calculate the difference between the current audio sampling data input and the calculated LMS filter output . This difference represents the value filtered by the LMS adaptive process. - Update the operating parameters of the filter system based on the difference , the sampling data History[i] in the history buffer, and the step size of the LMS filter. - Return the calculated difference . Below is a reference code snippet for processing with the adaptive LMS filtering algorithm, which can be used to understand the execution workflow above: In practical applications, the LMS adaptive filtering algorithm is often combined with dual microphones to achieve better noise reduction results. The primary microphone is placed close to the human mouth to capture the speaker's voice, while the secondary microphone is placed far from the mouth and close to the noise source (such as the outside of a headset) to capture environmental noise. During operation, the environmental noise data captured by the secondary microphone is used to train and update the parameters of the LMS filter, which then filters the audio sampling data from the primary microphone. This approach yields exceptional suppression effects on periodic noise (such as engine or fan noise). - Of course, if dual microphones are used, the execution workflow and corresponding code of the LMS filtering algorithm above need to incorporate a sampling sequence of a reference audio signal (from the secondary microphone). The calculations for History, the output of the LMS filter, and its parameter updates are based on the sampling data of the secondary microphone, while the filtering of the primary microphone's sampling data is performed by subtracting the LMS filter output from the primary microphone's sampling data. Comparing the implementations of spectral subtraction and adaptive LMS filtering algorithms, spectral subtraction involves two mutual conversions between the frequency domain and the time domain, resulting in a relatively high computational load, though it processes one block of operations per audio frame. On the other hand, the adaptive LMS filtering algorithm performs calculations directly in the time domain, but requires two floating-point operations equal to the filter order for every single sample—one for calculating the filter output and one for updating the filter's operating parameters. Therefore, if the filter order is high, the computational load can also be significant. Overall, however, the latter has lower demands on computing resources and is more suitable for resource-constrained embedded systems. Audio Auto Gain Control (AGC) In audio calls and video conferencing, the primary functions of the Audio Auto Gain Control (AGC) module are: - To stabilize the output level of the audio signal. Regardless of the strength of the signal captured by the microphone (e.g., varying distances of users from the microphone), AGC strives to ensure that the output volume of the audio capture module remains relatively consistent, preventing clipping caused by excessively high volume or inaudibility caused by excessively low volume. - To dynamically adapt to changing environments in volume capture. This includes coping with environmental noise variations, device differences (such as different microphone sensitivities), and differences in user speaking habits, automatically and dynamically adjusting the strength of the voice signal to keep its output at an appropriate level. In the processing chain of typical consumer electronic Audio Codecs, the AGC module generally operates in the digital domain after ADC conversion, implementing the function of dynamically adjusting the gain of voice signal data via DSP algorithms. This processing approach offers better flexibility and allows for collaborative optimization with other DSP algorithms such as noise reduction and AEC. Of course, some low-end Audio Codecs (such as the TLV320AIC3101) lack rich digital audio processing functions internally; in such cases, the AGC module can also directly adjust the PGA gain in the analog domain to achieve auto gain control. The diagram below shows the AGC processing chain of the TLV310AIC3101. !image.png Additionally, AGC dynamic gain adjustment is only used in the input processing chain of the audio signal—namely, the processing of voice signals captured by the microphone—to achieve a basically consistent voice signal strength. In the output processing chain of the audio signal, which is the processing chain for playing voice signals through speakers, AGC is generally unnecessary. The volume of the output voice signal is typically managed manually by the user or controlled via system volume management, requiring no automatic gain. AGC Audio Processing Chain and VAD Module Generally speaking, the sound signal captured from the microphone inevitably contains a certain amount of noise and echoes from two-way intercom states, in addition to the local voice signal we actually want to preserve. Since AGC maintains volume stability by adjusting the gain of audio sampled data, the object of its stabilization must inevitably be the local voice signal with noise and echoes removed as much as possible. This is why AGC should be placed after AEC and ANS in the audio 3A processing chain, as the objects for which we perform auto gain control should not include echoes and noise. !image.png After the aforementioned noise and echoes are processed by the ANS and AEC stages respectively, the AGC auto gain control processing workflow also incorporates a VAD (Voice Activity Detection) module to determine when to initiate gain adjustment for the audio sampled data. The primary objective of the VAD module is to distinguish continuous voice signals captured by the microphone into speech segments and non-speech segments (such as silence or background noise). AGC is activated to dynamically control the gain of audio data only during speech segments, while gain adjustment is suppressed or maintained at a fixed gain during non-speech segments. This prevents amplifying noise and avoids negative impacts of AGC's dynamic gain adjustment on non-speech signals. The VAD module's decision on speech and non-speech segments operates on a per-audio-frame basis. It calculates the root-mean-square (RMS) energy of the audio sampled data within the frame and compares it with the dynamic local noise energy maintained by the VAD module. If the energy of the current frame is higher than the dynamic noise floor plus a fixed offset (e.g., +3 dB), it can be determined as a speech segment candidate. Combined with a state machine control mechanism (such as determining a speech segment when multiple consecutive frames exceed the threshold, and a non-speech segment when multiple consecutive frames fall below it), the final verdict for speech and non-speech segments is rendered. AGC Processing Workflow 1. Preprocessing of Audio Sampled Data As mentioned above, AGC operates in the digital domain of the Audio Codec, executing auto gain processing on the digital sampling sequence captured by the microphone after ADC conversion and other audio processing modules. In this stage, continuous audio sampled data is first sliced into audio frames containing a fixed number of samples, and subsequent AGC processing is performed on a per-audio-frame basis. Frame-based processing in AGC typically employs a 50% overlap method, meaning the first 50% of samples in the current audio frame are identical to the last 50% of samples in the previous audio frame. This smooths out boundary effects between audio frames and prevents sudden jumps caused by different gains in two consecutive audio frames. 2. VAD Detection As noted above, the VAD module determines whether the current audio frame is in a speech segment or a non-speech segment by comparing the frame energy of the current audio with the noise floor energy. AGC gain adjustment is generally triggered only during speech segments within the AGC workflow. 3. Auto Gain Calculation Phase The auto gain calculation phase relies on several preset values of the AGC module: - Target Level: The target volume level that the AGC module expects its output to reach. - Time Constant: The time parameter for gain adjustment, used to control the smoothness of gain regulation and prevent abrupt volume fluctuations. When calculating the gain, the energy of the current audio frame is compared with the target level. If the energy of the current frame is higher than the target level, the gain is appropriately reduced to prevent overloading after the audio sampled data is combined with the gain. Conversely, if the energy of the current frame is lower than the target level, the gain is increased to raise the volume amplitude. For application scenarios with a relatively small signal dynamic range (where differences in signal amplitudes are minor), a linear gain adjustment mode can simply be selected: a gain magnitude is determined based on the difference between the current audio frame's energy and the target level, and the same gain adjustment is applied to all audio samples. For complex signal environments (such as burst noise and multi-band interference), a non-linear gain adjustment strategy must be adopted: - Dynamic Range Control (DRC). DRC is a technique that adjusts the signal dynamic range through non-linear gain adjustment, widely used not only in voice signal processing but also in communication systems and smart devices. DRC operation roughly relies on three parameters: threshold_low, threshold_high, and ratio. Its working logic is generally: when the amplitude and energy of the audio samples are below the threshold_low threshold, DRC sets a relatively low gain to suppress noise; when the amplitude and energy of the audio samples are between threshold_low and threshold_high, DRC sets the gain for the audio samples according to the target gain calculated previously; and when the amplitude and energy of the audio samples are greater than threshold_high, the gain for the audio samples is appropriately compressed according to the ratio setting to prevent clipping. - Multi-band Gain Control. Band splitting is first performed via FFT calculations to divide the signal into low-frequency, mid-frequency, and high-frequency sub-bands. Then, gain is calculated and adjusted independently for each sub-band. This avoids spectral imbalance caused by overall adjustment. The entire processing approach is somewhat similar to the workflow of an equalizer (EQ), setting different gains for different frequency bands to suppress or enhance them. In addition, the time constants for automatic gain calculation and adjustment mainly consist of two types of time parameters used to achieve smooth and natural changes in gain and adjusted voice amplitude: - Attack Time: Rapidly reduces gain (typical value 20–100 ms) when the amplitude of the audio signal suddenly surges, preventing transient overload issues. - Release Time: Slowly restores gain (typical value 100–2000 ms) when the signal amplitude weakens. 4. Gain Application Phase and Post-Processing The final stage of AGC processing applies the gain calculated during the calculation phase to all sample data within the current audio frame. Depending on whether a linear or non-linear gain calculation strategy was adopted in the previous stage, consistent or varying gains are applied to the audio samples accordingly. Additionally, the audio sample sequence data processed by AGC must also have its amplitude limited (e.g., to -1 dBFS) to prevent DAC or speaker overload, and it requires feedback coordination with audio processing modules from previous stages, such as AEC and ANS, to achieve stable control over the audio data. References - Audio Auto Gain Control (AGC) Problems Solved and Principle Analysis - Zhihu - Echo Cancellation (AEC) Principles, Algorithms, and Practice — AEC Background Introduction - CSDN Blog - The Three Musketeers of Audio and Video Processing: AEC — Causes of Echo and Principles of Echo Cancellation - ZEGO Tech Blog, 51CTO Blog - Preliminary Study on Audio 3A Algorithms - The "Noise" Sources and Solutions That Drive Audio Professionals Crazy
Detailed Explanation of WiFi Roaming Protocols - IEEE 802.11 k/v/r
This article explains the concept of wireless roaming in WiFi communications and traditional roaming workflows. It also provides a detailed summary of the three major protocols supporting the WiFi wireless roaming process—IEEE 802.11K/V/R—and elaborates on the specific problems each protocol solves. To support fast and efficient seamless roaming of WiFi devices between different APs, the IEEE 802.11 working group specifically defined three protocols: 802.11K, 802.11V, and 802.11R, collectively referred to as the 802.11 KVR protocols. Together, these three protocols comprehensively solve the problems of "when to initiate a roaming handover," "which AP to roam to," and "how to roam quickly" during the WiFi wireless roaming process. What is Wireless Roaming? Since the IEEE 802.11KVR protocol suite primarily addresses WiFi wireless roaming, we must first define what wireless roaming is and how a WiFi STA interacts with various APs throughout the entire wireless roaming process. WiFi roaming refers to the process where a mobile device (such as a smartphone, laptop, etc.) automatically switches between different WiFi Access Points (APs) to maintain the continuity of its wireless network communication services. This process aims to minimize network interruption time during the switch and avoid noticeable impacts on communication services. Simply put, it is like walking around your house with your phone from the living room to the bedroom. As the WiFi signal in the living room weakens, your phone automatically connects to the router in the bedroom with a stronger WiFi signal, ensuring that your WiFi network stays connected to the AP with the strongest signal while allowing you to use the network continuously, smoothly, and without interruption. As is well known, due to the wireless nature of WiFi communications, wireless signals attenuate very rapidly as they travel through space. Consequently, the coverage area of a single AP is quite limited, typically spanning only a few tens of meters. For a STA that needs to move across a larger space while maintaining a connection to an AP, multiple APs must be deployed at different locations. When the wireless terminal STA moves into the overlapping boundary region of two APs, it associates with the new AP and disassociates from the original AP, maintaining uninterrupted network connectivity throughout the process. This is the general execution process of wireless roaming. !image.png Traditional Wireless Roaming Workflow The complete workflow of traditional wireless roaming is as follows: As the wireless terminal gradually moves away from its currently associated AP, the terminal perceives a gradual decrease in signal strength via RSSI. At this point, the terminal can detect the presence of surrounding APs by probing broadcast frames across multiple channels in the current environment and begin interacting with them. When the signal of the wireless terminal reaches the threshold for roaming handover, the terminal triggers roaming, associates with the new AP, and disconnects from the original AP, thereby completing the AP switch. As can be seen from the above process, in the traditional model, the roaming behavior of the wireless terminal is entirely controlled by the terminal itself. The terminal determines the roaming trigger mechanism and decision logic based on roaming handover thresholds and by independently probing the signal strengths and other parameters of all surrounding WiFi hotspots. Problems with the Traditional Wireless Roaming Model - Severe packet loss during the pre-handover process. During the process where the wireless terminal measures and decides to perform a wireless roaming handover, it must frequently switch between multiple channels to listen to broadcast frames while interacting with multiple other APs. From a communication perspective, this entire process is time-consuming, and packet loss in the normal service data communications maintained in the current connection is inevitable. - Untimely roaming triggers. The trigger condition for traditional wireless roaming is simply the signal threshold of the currently connected AP; switching occurs only when the signal strength drops below this threshold. However, if another AP with better communication quality exists in the current environment before the threshold is reached, a switch will not be triggered. This causes the terminal to remain connected to an AP with a degraded signal, resulting in a poor user communication experience. - The selected roaming AP may not be the optimal AP. According to the workflow described above, for the terminal, the comparison criterion for choosing a new AP is its signal strength. In reality, however, multiple APs existing in the current space have varying workloads. Therefore, when selecting a new AP, its current load should also be comprehensively considered to choose the one with the least load whenever possible, rather than relying solely on signal strength. - Excessively long association times with the new AP. This is particularly true when the AP has complex security authentication modes enabled, such as 802.1X. Due to the requirements of wireless communication security protocols and procedures, completing the full re-association process between the STA and AP takes a relatively long time. This inevitably causes application-layer communication to freeze during this period, degrading the user experience. The 802.11KVR fast roaming protocol suite introduced by IEEE for wireless roaming is designed to solve these exact problems. The difference between the new roaming model supporting 802.11 K/V/R and the traditional roaming model is that the complete workflow and execution logic of K/V/R roaming are determined jointly by the AC, AP, and wireless terminal. 802.11K The full name of the 802.11K protocol is Radio Resource Measurement of Wireless LANs (RRM). The primary working objective of the 802.11K protocol is to collect and share all AP-related information in the current wireless environment (such as signal strength, channel load, workload status, neighbor AP lists, etc.) through mutual communication between the AP and STA, assisting the STA in finding the optimal AP for its next wireless roaming operation. The workflow of the 802.11K protocol is as follows: - The terminal actively requests neighbor report information from the AP. When the terminal STA detects that the signal quality of its currently connected AP is degrading (most typically when the RSSI drops below its preset threshold), the STA sends a Neighbor Report Request frame to the AP. - Upon receiving this request, the AP generates a Neighbor Report and sends it back to the terminal. This neighbor report contains information such as: static information of neighboring APs maintained in real-time by the AP itself (e.g., SSID, BSSID, channel, etc.), as well as dynamic information (current load, signal strength (RSSI), channel interference, etc., of each AP). - The terminal decides and executes the handover. After receiving the neighbor report information, the terminal evaluates it based on comprehensive metrics (such as signal strength, load, interference, etc.) and selects the optimal AP as the roaming target. As seen from the workflow above, the 802.11K protocol focuses on shortening the STA's channel scanning time during roaming. The AP currently connected to the STA provides the candidate AP list and its channel information, while the STA ultimately retains the primary role in deciding the roaming handover logic. 802.11V The full name of the 802.11V protocol is Wireless Network Management (WNM). Similar to the 802.11K protocol, the 802.11V protocol is also used to solve the problem of unreasonable roaming target selection in traditional roaming workflows. However, the main difference between the two lies in whether the AP acts via passive reporting or active guidance within the decision-making mechanism. With the support of the 802.11V protocol, the AP can actively recommend the optimal roaming AP to the terminal based on dynamic network changes, rather than relying solely on the terminal's independent selection. This proactive AP adjustment prevents a single AP in the network from becoming overloaded, thereby improving overall network throughput. Regarding its specific working mechanism: - After obtaining the neighbor AP list information via the 802.11K protocol, the AP can further utilize the BSS Transition Management (BTM) mechanism of 802.11V to send a recommended handover list containing parameters such as the load and interference of each candidate AP to the terminal, guiding it to select the best AP. - Additionally, during normal operation, when an AP detects that its own communication load is too high, it can migrate some terminals to lower-load APs through directed roaming requests (such as sending a BSS Transition Query frame). Therefore, compared to the 802.11K protocol, the decision-making entity in the wireless roaming process shifts to AP-suggested or AP-led under the 802.11V protocol. Its ultimate goal is to achieve load balancing among various APs in the network and optimize overall network performance. 802.11R The main function of the 802.11R protocol is Fast Basic Service Set Transition (FT). Its core objective is to achieve millisecond-level seamless roaming handovers by simplifying the authentication process during AP association and optimizing the key management mechanism, all without degrading security performance. Therefore, the 802.11R protocol specifically addresses the business interruption issues caused by lengthy security authentication procedures between STAs and APs in traditional roaming. In networks supporting WPA2/WPA3 encryption or 802.1X authentication, traditional roaming methods require re-executing a complete 4-Way Handshake and key negotiation. Consequently, this complete association process can take hundreds of milliseconds or even seconds. I have detailed the complete packet interaction process of WPA/WPA2 in another note: Detailed Explanation of WiFi WPA/WPA2 4-Way Handshake Authentication Mechanism | pavelhan.tech. In contrast, the 802.11R protocol utilizes mechanisms such as pre-authentication and key caching to compress the re-association and handover time between the STA and AP to within 50 milliseconds, thereby meeting real-time service requirements such as voice calls (VoWiFi) and AR/VR education. In terms of specific implementation, the core mechanism by which 802.11R achieves fast association handover times is Fast BSS Transition (FT): - Pre-authentication mechanism: Before switching, the STA terminal establishes a secure tunnel with the target AP via the current AP, completing identity verification and key negotiation in advance. This avoids repetitive authentication procedures during the handover. - Hierarchical key management: The 802.11R protocol introduces a two-tier key architecture consisting of PMK-R0 (Master Key) and PMK-R1 (Derived Key). PMK-R0 is centrally distributed by the AC to a group of APs, while PMK-R1 is dynamically generated by each AP based on PMK-R0, enabling cross-AP key sharing. - Four-way interaction optimization: The 8-10 message exchanges of traditional roaming workflows are simplified to 4 (Reassociation Request/Response + FT Action frame exchange), reducing signaling overhead by over 50%. Summary Together, the three protocols—IEEE 802.11K, 802.11V, and 802.11R—provide a relatively complete solution for how WiFi devices achieve fast and efficient seamless roaming between different APs. Among them: - 802.11K focuses on continuously collecting static information about the current wireless environment on the AP. When a STA needs to switch APs, it reads this static information from its currently connected AP and then decides which new AP to switch to. Throughout this process, although the AP provides environmental information related to roaming handovers to the STA, the STA remains the primary decision-maker, with the AP playing a passive role. - 802.11V focuses more on managing dynamic load balancing across the overall network. It relies more heavily on the AP proactively initiating or guiding the STA to switch to a less-loaded new AP. This optimizes overall network communication performance while achieving network-wide load balancing. In this process, the AP adopts a relatively more active role from the perspective of overall network performance and planning. - 802.11R optimizes the association time between the AP and STA, as well as the corresponding key management and authentication mechanisms during the roaming handover process. It shortens the STA's association time with the new AP while ensuring security authentication, preventing communication experience degradation at the user layer caused by frequent handovers. References - Introduction to 802.11K/V/R Protocols - 802.11k/v/r - lsgxeva - Blogs.cnblogs
Explanation and Matching of CRA Parameters for Image Sensors and Lenses
CRA stands for Chief Ray Angle, which is the angle of the chief incoming light ray. Although both lenses and image sensors have CRA parameters, their actual definitions are distinctly different. Lens CRA Parameters The CRA of a lens is determined by its optical design and represents the angle distribution pattern of light rays from the optical center to the image plane. In other words, it is the angle between the chief ray (the ray passing through the center of the entrance pupil) at various positions of the lens after passing through the optical system to the image plane, and the normal line at the intersection point with the image plane. As shown in the figure below, the chief ray angle varies across different positions of the lens. This means that the lens CRA is not a fixed value; rather, it varies depending on the field of view (FOV) position. For example, the figure below shows the chief ray angles corresponding to the 14°, 10°, and 0° fields of view for the Double Gauss 28 Degree Field optical system from the ZEMAX lens library, from top to bottom. !image.png From the optical imaging ray diagram above, it can be seen that the larger the lens field of view, the larger its corresponding CRA angle. At the center position of the lens (the 0-degree field of view), the CRA can be considered as 0. The closer to the edge of the lens, the larger the CRA angle becomes. Generally speaking, when we refer to the lens CRA angle, we usually mean the CRA angle at the edge of the lens. The figure below shows the normalized CRA corresponding to different image height positions for a specific lens model. The CRA at the center of the lens is 0, and the chief ray angles vary at different heights across the image plane, resulting in a CRA curve that changes with image height. !image.png Image Sensor CRA Parameters Compared to lens CRA parameters, sensor CRA parameters are slightly more complex. First, every pixel on an image sensor has a tiny microlens on top of it, which is used to focus as much light entering through the lens as possible onto the photodiode underneath to achieve photoelectric conversion. As seen in the figure below, adding a microlens above the photodiode allows more light to be refracted onto the photosensitive surface, thereby increasing the luminous flux on the photodiode. This is the main function of adding microlenses. !image.png According to the microlens structure described above, if the light entering from the lens is incident perpendicularly onto the microlens (such as in the center region of the image sensor and lens), these rays can all pass through the microlens and enter the light-absorbing region within the pixel array. However, if the light entering from the lens hits the microlens at a certain oblique angle (such as at the edge positions of the image sensor and lens), after being refracted by the microlens, these rays will hit the metal layers inside the pixel (the gray areas in the figure above) instead of reaching the central absorbing region. This causes a decrease in light intensity in the light-absorbing region, leading to reduced sensitivity and increased optical crosstalk at edge pixels, as shown in the figure below. !image.png The above problem leads to the design of microlens shifting. Microlens Shifting Design To address the photosensitivity issue at the edge pixels of the image sensor (caused by oblique light rays from the lens hitting the microlenses), the microlenses need to be shifted by a certain distance. This allows the refracted light to refocus onto the light-absorbing region, recovering the light that would have been lost without the shift. As shown in the figure below, after shifting the position of the microlens toward the center by a certain distance, light incident at an angle is refracted by the microlens into the central light-absorbing region of that pixel. !image.png From the optical theory of lens CRA discussed earlier, the chief ray angle of the light entering the image sensor's photosensitive surface through the lens is related to the positions of both the lens and the sensor's photosensitive surface. In typical optical applications, the optical axis centers of the lens and the sensor coincide. Therefore, at the center of the sensor, the chief ray angle of the light passing through the lens is essentially perpendicular to the microlenses of the central pixels, allowing most of the light to be refracted by the microlenses into the central absorption region. However, as you move outward from the center to the periphery, the chief ray angle of the incoming light from the lens becomes progressively larger. At this point, the microlenses on the pixels in the peripheral areas of the sensor can be processed with the aforementioned shifting technique. The direction and magnitude of the shift should correlate with the chief ray angle of the lens at that specific position. This is the issue of CRA matching between the lens and the sensor. Sensor CRA is Also Related to Pixel Position Following the same logic, for image sensor design, the shift direction and magnitude of the microlenses on the sensor pixels must match the chief ray angle of the incoming light they are expected to receive. This defines the CRA parameter required by the sensor. Similarly, taking the sensor center as the axis, the sensor center has a CRA of 0, and the CRA varies across different sensor regions extending outward, with the CRA increasing closer to the edges of the sensor. Below is the CRA data for the GC4653. The CRA at the center of the sensor is 0, while at the edge it approaches 10 degrees. Generally speaking, the CRA data seen in sensor datasheets refers to the maximum CRA angle at its edges. !image.png Sensor and Lens CRA Matching Selection During the selection process for an image sensor and a lens, special attention must be paid to the matching of their CRA values. The ideal scenario is a perfect match between the lens CRA and the sensor CRA, which yields the highest light collection efficiency. However, in practical applications, a perfect match is difficult to achieve. In such cases, it should be ensured that the difference between the lens CRA and sensor CRA is controlled within 2 to 3 degrees. !image.png If the CRA of the sensor and the lens do not match, refractory image quality issues will arise during later tuning that are difficult to resolve: - When Lens CRA > Sensor CRA, the light entering from the lens will exceed the correction capability of the sensor microlenses, resulting in adjacent pixel crosstalk (Color Shading) in the final image, which manifests as color distortion at the edges of the image. - When Lens CRA
Detailed Analysis of the Secure Boot Workflow on the Novatek IPC Platform
Based on the study of materials related to the Novatek NT98567, this article outlines the complete process of encryption, decryption, and protection across various boot stages in the platform's Secure Boot workflow, providing an excellent practical case study for deeply understanding the concepts and procedures of Secure Boot in embedded systems. Previously, another note detailed the workflow and concepts of Secure Boot: Summary of the Secure Boot Startup Process in Embedded Products. Novatek NT98567 The diagram below shows the block diagram of the Novatek NT98567/98568. Generally, its specifications feature a dual-core ARM A7, up to 2Gb embedded DDR, an ISP and encoding capability of up to 5M at 30fps, support for H.264 and H.265, support for independent dual cameras, and 0.5T of AI computing power. Overall, while the specifications are not exceptionally groundbreaking, a unique feature is its support for EIS (Electronic Image Stabilization), making it suitable for applications such as car DVRs and action cameras. !image.png ARM TrustZone Support on the NT98567 As mentioned above, the CPU configuration of the NT98567/98568 is a dual-core ARM Cortex-A7, which belongs to the ARMv7-A architecture. This architecture fully supports the ARM TrustZone security specifications. Consequently, the system can be divided into two independent subsystems—the Secure World and the Non-Secure World—during application design, which is the implementation of a TEE (Trusted Execution Environment). The SDK for the Novatek NT98567/98568 platform itself also provides OPTEE support under the ARM TrustZone architecture, though OPTEE is optional in the SDK. In other words: - If user applications require privacy and security-sensitive application logic to run in the Secure World, it is best to enable OPTEE, implement these functions within the OPTEE secure environment, and provide interfaces for the non-secure Linux system to access; - If the goal is simply to enable Secure Boot, there is no need to enable the OPTEE option. Enabling OPTEE is essentially independent of the Secure Boot workflow: !image.png For knowledge related to the ARM TrustZone technology domain, you can refer to my other note: An Introduction to TEE and ARM TrustZone Security Technology. EFuse Structure of the NT98567 To enable Secure Boot, keys related to this mechanism must be stored in the chip's secure storage, which can be written only once and cannot be revoked or modified. This is implemented via the chip's built-in EFuse structure. The Secure Boot implementation on the NT98567 primarily utilizes two cryptographic algorithms and their corresponding keys: - AES128: A symmetric key algorithm with a key length of 128 bits, used to provide AES128 encryption protection for the contents of image files. - RSA2048: An asymmetric key algorithm with a key length of 2048 bits, used to verify the signatures of image files. The storage structure of these two keys in the NT98567 EFuse is shown below: !image.png It is worth noting here that the size of a single EFuse block on the NT98567 is 16 bytes. Therefore, the AES128 key is stored in the first block, while the RSA2048 key is too long to fit directly; instead, the SHA256 signature of the corresponding public key is stored in the EFuse, occupying two EFuse blocks. Ideally, if there were no EFuse size limitations, the public key of the RSA2048 algorithm used for signature verification should be stored here. Encryption Protection of Image Files For the image files loaded at various stages during the Secure Boot process, encryption and signature protection must be applied using the two keys stored in the EFuse: - The AES128 key stored in the EFuse is used to encrypt the data content of the image file. - The RSA2048 public key stored in the EFuse (actually, only the SHA256 signature of the public key is stored in the EFuse) is used to provide asymmetric signature protection for the contents of the image file. Through the above data encryption and signing operations, as long as the keys are not compromised, it can be ensured that the contents of the image files required for the normal operation of the Secure Boot process are both encrypted and officially released. In concrete implementation, each image file in the Secure Boot process is prepended with an image header, which stores the public key of the RSA2048 signature algorithm and the signature calculated from the image data content using the RSA2048 private key: !image.png Within the image file header above, the foremost Header is a fixed-size structure containing primarily the RSA2048 public key, the signature, the address offset of the encrypted data within the image file, the size of the encrypted data content, and the checksum of the entire image file. The second part stores the public key of the RSA2048 algorithm. How is the legitimacy of this public key verified? A SHA256 signature operation is performed on this public key, and the result is compared with the public key signature stored in the EFuse. If they match, it indicates that the public key is legitimate. The third part is generated when the PC tool packages the image file: it first calculates a SHA256 signature on the plaintext of the image data content, then encrypts this signature using the RSA2048 private key to obtain the final Signature, which is written into the image header information. The fourth and final part is the ciphertext generated by encrypting the plaintext of the image data content using the same AES128 key stored in the EFuse. This is also the most crucial part of the entire image file. > Therefore, when the PC tool packages and encrypts the original image file, it needs to: prepend a fixed-size Header containing the public key, signature, and positional pointer information for the data content; place the public key portion of the RSA2048 algorithm into the second part of the header; calculate a SHA256 signature based on the plaintext of the original image content, encrypt the signature result with the RSA2048 private key, and place it in the third part of the header; and finally, perform AES128 encryption on the original image content, placing the resulting ciphertext in the fourth part. Secure Boot Startup Workflow As described in the second section regarding the ARM TrustZone mechanism supported by the NT98567, when OPTEE is not enabled, the entire Secure Boot startup workflow is as follows: - Upon powering on, the ROM Loader inside the NT98567 runs first; - The ROM Loader loads the first-stage boot image loader stored in the flash, decrypts it, verifies its signature, and then loads U-Boot; - After loading U-Boot, the Loader similarly decrypts and verifies the signature of U-Boot's contents based on the key in the EFuse and the key, signature, and other information in the U-Boot image header. Once verification passes, U-Boot is executed, which in turn is responsible for loading the Kernel image; - After loading the Kernel, U-Boot similarly decrypts and verifies the signature of the kernel contents based on the key in the EFuse and the key, signature, and other information in the Kernel image header. Once verification passes, the Linux system can boot normally. (Note that the Rootfs image is not protected by Secure Boot here.) The following diagram illustrates how U-Boot decrypts and verifies the encrypted Kernel image during the process of U-Boot booting the Kernel: !image.png References - NT98567 Secure Boot Flow User Guide - NT98567_Secure_boot_User_Guide_en - Information Security: RSA-2048 Performance - Zhihu