All Articles
An Interpretation of the Loss Function and Training Process in YOLOv1
This article summarizes the training positive sample labeling method, loss function, and the complete training process of the YOLOv1 model, laying a solid foundation for understanding the processing workflows of subsequent YOLO versions. Labeling of Positive Training Samples in YOLOv1 As summarized in the article Detailed Interpretation of YOLOv1 Output Information and NMS Filtering Algorithm, the object detection logic of YOLOv1 is that if the center point of an object falls within a specific grid cell, that grid cell is responsible for detecting the object. This means that when constructing training samples for YOLOv1, only the grid cell containing the center point of the target object is considered to contain an object. This is illustrated in the figure below: !image.png Although the bounding box for the kitten in the figure spans as many as 24 grid cells, because the center of this bounding box falls within the pink grid cell, only the pink grid cell is labeled as containing an object when generating the training sample for this image in YOLOv1. The detection confidence for this grid cell is close to 1, while all other grid cells are considered to contain no object, with a detection confidence close to 0. > Grid cells containing objects are labeled as positive sample candidate regions. That is, during the training process, positive training samples are derived exclusively from the predictions at these grid cells, whereas predictions from all other regions serve as negative samples for that target. Interpretation of the YOLOv1 Loss Function Overall, the calculation of the YOLOv1 loss function consists of three main components: - Coordinate Loss (): Evaluates how closely the bounding boxes predicted by the model fit the ground truth boxes of the training samples. - Confidence Loss: Measures the degree of alignment between the confidence value of each predicted box output by the model and whether that box actually contains the target object to be detected. - Classification Loss: Determines whether the category predictions output by the model match the labeled classification of the training samples. The figure below shows the meanings of various parameters used in the loss function calculation: !image.png The figure below shows the calculation formula of the YOLOv1 loss function: !image.png The first two rows of the YOLOv1 loss function represent the loss for the coordinate parameters of the predicted bounding boxes, the third and fourth rows represent the confidence loss for both positive and negative sample bounding boxes, and the last row represents the classification loss at the positive samples. Although the calculation logic of the loss function above looks complex, the actual computation logic is quite intuitive. It can be broken down into three parts for detailed explanation. Coordinate Loss The key to calculating the coordinate loss is understanding the meaning of : - For each ground truth box of a training sample, first locate the grid cell where the center of this ground truth box falls. Then, calculate the IOU between the two predicted boxes output by this grid cell and the ground truth box, and set the value of the predicted box with the larger IOU to 1, while setting the value of the other predicted box to 0. - In this case, effectively, exactly one predicted box with a value of 1 is found for each ground truth box in the training sample. The logic above identifies the responsible predicted box corresponding to each ground truth box in the training sample. Next, following the respective formulas, the loss values for the center coordinates, width, and height between the ground truth box and the responsible predicted box are calculated. The sum of these two loss values gives the total coordinate loss. As seen from the loss function formula above, the coordinate loss value is also multiplied by a coefficient . This increases the weight of the bounding box localization precision loss in the overall loss, prioritizing more accurate localization of the predicted boxes. Confidence Loss The calculation of confidence loss consists of two parts: the loss for responsible predicted boxes and the loss for non-responsible predicted boxes. Loss for responsible predicted boxes: For each ground truth box of the training sample, follow the same steps as the coordinate loss calculation above to find the corresponding responsible predicted box. Each ground truth box has only one responsible predicted box. Then, calculate the confidence loss between the ground truth box and the responsible predicted box. Here, the target confidence of the ground truth box is 1, and the confidence of the responsible predicted box is the model's confidence output value for this predicted box. Loss for non-responsible predicted boxes: The logic for non-responsible predicted boxes is simple. Among the 98 predicted boxes output by the model, apart from the aforementioned responsible predicted box, the remaining 97 boxes are all non-responsible predicted boxes. Calculate the loss values for the responsible and non-responsible predicted boxes separately using the formulas above. Then, multiply the loss of the non-responsible predicted boxes by to suppress the impact of background boxes on the loss values of the true predicted boxes. Summing these two gives the total confidence loss. Classification Loss The classification loss is calculated exclusively for the responsible predicted boxes; the classification loss for all other predicted boxes is 0. For each responsible predicted box, calculate the loss between its 20 class recognition probabilities and the ground truth class of the training sample. It is worth noting that the class of the ground truth box is provided as a 20-dimensional label in one-hot format (only one class is 1, and all others are 0). Therefore, the classification loss is the squared difference between the 20 class recognition probabilities corresponding to each responsible predicted box and the 20-dimensional probability labels of the one-hot ground truth box. References - "YOLO Object Detection" by Jianhua Yang, Ruifeng Li, Chapter 3: YOLOv1 - YOLO Evolution — 1. The previous article mentioned YOLO's approach and how to train it; this article explains YOLO in even more detail... | by Ching-I | Ching-I's Reading Notes | Medium
Detailed Explanation of YOLOv1 Output Information and NMS Filtering Algorithm
The network architecture of the YOLOv1 model has already been explained in great detail in the article [[Detailed Interpretation of the YOLOv1 Model Network Architecture]]. Below is the architecture diagram of the YOLOv1 network: !image.png As can be seen from the architecture diagram above, compared to various classic convolutional neural network models used for image classification, the network structure of YOLOv1 does not introduce particularly groundbreaking innovations. It still extracts features from images through repeated convolution and pooling operations in the first half of the network, and performs category recognition and localization using two consecutive fully connected layers in the second half. However, compared to other networks, the essence of the YOLOv1 architecture lies in its final 7x7x30 output interpretation, the training of model parameters, and the design of its loss function evaluation. YOLOv1 Model Output Information The output of YOLOv1 is a 3D tensor of size 7x7x30. This can be conceptualized as dividing the input image of 448x448 resolution into a 7x7 grid. The model searches for the center point of the target object to be recognized, its bounding box, confidence score, and object category within each grid cell. Each grid cell represents these detected items using a 30-element vector. As shown in the figure below, each grid cell outputs the information for two bounding boxes. The information for each box includes the coordinates of the detected object's center point (relative to the top-left corner of the grid cell), the width and height of the detected object (expressed as ratios relative to the entire image size of 448x448), and the confidence score of this detection box. !image.png It is important to note that YOLOv1's logic for detecting objects in an image is: if the center point of an object falls within a certain grid cell, that grid cell is responsible for detecting and outputting the corresponding bounding box and other information for that object. Therefore, in the box information output by each grid cell, and represent the position of the center point of the target object relative to the top-left corner of the current grid cell, while and represent the width and height of the bounding box predicted by this grid cell relative to the entire input image resolution. Each box has a confidence parameter, which is used to characterize the probability that the current box actually contains the target object. If the box contains the center point of a target object, this confidence parameter will be relatively high, approaching 1. Conversely, if the box does not contain the center point of any target object, its confidence will approach 0, allowing it to be filtered out in subsequent filtering algorithms. The 20 classes are used to represent the final recognition probabilities for various target object categories. For the VOC dataset, it supports the detection and recognition of 20 categories. If the COCO dataset is used, the number of categories output by each grid cell needs to be modified to the recognition probabilities of 80 categories. Two Boxes, One Set of Category Recognition Probabilities? From the output information of each grid cell of the YOLOv1 model summarized above, we can see that the output of each grid cell contains the information of two predicted boxes, along with a single set of recognition probabilities for the target objects. So, if a grid cell happens to contain the centers of two target objects, which object do these category recognition probabilities actually represent? The answer is: in this case, a missed detection will definitely occur. This is also the fundamental reason why YOLOv1's recall rate drops sharply in dense, small-target scenarios, and it is the motivation for subsequent versions (starting from YOLOv2) to introduce anchors, refining the "responsible detection unit" from grid cells to "grid + anchor" or even "pixel" levels. Let's understand this concept based on the figure below: !image.png For the two boxes output by each grid cell, the grid cell will only mark the box that best matches the actual target object (specifically measured by the IOU metric) as a positive sample during output. In the figure above, the bounding box of the actual object is the light gray box, and the matching degree of predicted box A is much better than that of predicted box B. Therefore, in the output of this grid cell, the confidence score of predicted box A will be relatively high, while the confidence score of predicted box B will be suppressed to a lower value. The category information contained in this grid will correspond to the category recognition probability of the image region enclosed by predicted box A. Thus, the entire input image in YOLOv1 is divided into 7x7 grid cells, and each grid cell outputs 2 boxes. At first glance, it seems that YOLOv1 can achieve a maximum detection of objects. However, in reality, each grid cell can output at most one box and its corresponding recognition category, so the entire image can recognize a maximum of only 49 independent objects. In other words: The two boxes output by each grid cell are not intended to detect two objects within a single grid cell; rather, after an object is detected in a grid cell, the two boxes provide an either-or option, allowing the model to select the best-matching box to locate its position. NMS NMS stands for Non-Maximum Suppression. As the name suggests, it is an algorithm that suppresses bounding boxes that are not the maximum, where "suppression" typically means filtering out redundant bounding boxes for the same detected object. As mentioned above, for each image input into the YOLOv1 model for inference and detection, the model outputs predicted boxes via the 7x7x30 3D tensor. These 98 predicted boxes must still go through the NMS algorithm filtering process to truly output the final detection results; otherwise, a single object might have multiple output detection boxes. The basic execution logic of the NMS algorithm: Use bounding boxes with high scores to suppress overlapping bounding boxes with low scores. The execution of the NMS algorithm is processed independently according to the supported categories. Each NMS operation yields the final detection boxes for a single category. Although the calculation logic is identical for different categories, combining the final detection boxes of all categories together at the end yields the final detection results of the model. Step 0: For a specific category, multiply the confidence of the box by the recognition probability of this category to obtain the score of this box for this category. Since the model outputs 98 boxes, each category will have 98 scores. Step 1: Compare the 98 scores of this category with the NMS detection threshold (e.g., 0.2). Boxes with scores lower than this threshold are deleted, and only boxes greater than the threshold are kept in the candidate list. Step 2: Sort all boxes in the candidate list in descending order based on their scores, and directly move the box with the highest score in the candidate list into the detection output for that category. Step 3: Calculate the IOU (Intersection over Union) between the box with the highest score and all other boxes in the candidate list. When the calculated IOU value is greater than the IOU threshold (typically 0.5), it indicates that the two boxes are detecting the same target. In this case, the box with the lower score is removed from the candidate list. Step 4: Remove the box with the highest score from the current candidate list. Repeat steps 2 to 4 above until the candidate list of boxes is empty. At this point, the detection output results for this category are obtained. After performing the above steps for all 20 categories, combining the detection output results of all categories gives the final detection output of the model. !image.png The figure above illustrates the NMS algorithm filtering process for the "dog" category. - First, for the dog category, multiply the confidence score of each box by the recognition probability of the dog category in its corresponding grid cell to obtain 98 scores. - Compare these 98 scores against the threshold of 0.2, delete boxes with scores less than 0.2, and put the remaining boxes into a candidate list. - Sort the boxes in the candidate list in descending order based on their scores. The box with the highest score is directly added to the detection output for the dog category. Simultaneously, execute the NMS algorithm described above: each time, place the box with the highest score into the output list for that category, and eliminate other boxes that heavily overlap with this box using IOU (to prevent multiple detection boxes for a single detected object) until the entire candidate list of boxes is emptied. References - "Deep Learning with PyTorch Object Detection in Action", Section 6.1 Anchor-Free Prediction YOLOv1 - All of the start: YOLOv1. YOLO: you only look once. | by John Shen | Medium - YOLOv1 Detailed Interpretation. Design Concept: | by Steven Meng | Medium
Decoding the YOLOv2 Model Network Architecture
This article provides a detailed summary and study of the YOLOv2 model's BN layer, the Darknet-19 backbone network, and its overall architecture. Introduction of the Batch Normalization (BN) Layer The convolutional structures used in the backbone network of the YOLOv1 model still followed the traditional pattern of linear convolution + non-linear activation function, without incorporating the various normalization layer designs that became widely popular later on. Because Batch Normalization (BN) layers have been applied in an increasing number of models and their performance has been widely recognized by the industry, they have gradually become a standard configuration in convolutional neural networks. Consequently, the architectural design of the YOLOv2 model naturally introduced this thoroughly validated BN layer. Specifically, the convolutional structure within the network architecture was changed from the previous pattern of linear convolution + non-linear activation function to a form consisting of a linear convolution layer + BN layer + non-linear activation function. This structure is also referred to as the CBL (Conv-BN-LeakyReLU) structure. !image.png Darknet-19 Backbone Network As introduced in [[YOLOv1 Model Network Architecture Decoding]], the backbone network of the YOLOv1 model adopted a design similar to GoogLeNet. By the YOLOv2 version, the authors designed a new network structure called Darknet-19 to replace the previous generation's backbone. The Darknet-19 network contains 19 convolutional layers, and every single convolutional layer adopts the CBL structure described above, namely the form of convolutional layer + BN layer + activation function. The figure below shows the configuration parameters for each layer of the Darknet-19 network architecture: - The feature map dimensions corresponding to the "Output" column in the table below correspond to images with an input resolution of 224x224. When Darknet-19 is used as the backbone network for YOLOv2 (with an input image resolution of 416x416), the output feature map dimensions of each layer change accordingly. !image.png > In the architecture diagram above, the Darknet-19 network can be divided into two parts: the first half is the feature extraction part, which, once pre-training is completed, will be used as the backbone network for YOLOv2; the final convolutional layer + global average pooling layer + Softmax portion constitutes the classification part, which is removed in the YOLOv2 model. As can be seen, the Darknet-19 network architecture above contains a total of 19 convolutional layers (3x3 and 1x1 convolutions) and 5 max-pooling layers of 2x2. Finally, drawing inspiration from the NiN network approach (refer to [CNN Classic Network Model Architecture Learning - NiN]), it replaces fully connected layers with global average pooling layers, and uses Softmax in the final layer to output prediction results for 1,000 classification categories. The authors of Darknet-19 trained the network parameters using the ImageNet dataset. In terms of final detection accuracy, the Darknet-19 network achieved levels comparable to the VGG network, but with a much smaller model size (only 25.5M parameters) and floating-point operations amounting to only about 1/5 of VGG, resulting in extremely fast computation speeds. After the Darknet-19 network is pre-trained on the ImageNet dataset, the final layers used for the classification task—including the final convolution, average pooling, and Softmax layers—are removed, and it is then used as the new backbone network for YOLOv2. Passthrough and Reorg In terms of backbone network design, aside from introducing the new Darknet-19 backbone, YOLOv2 also drew inspiration from the SSD model's design of using higher-resolution feature maps. > Typically, different feature maps have different resolutions. The shallower the feature map, the fewer downsampling operations it undergoes, resulting in a higher resolution and finer grid divisions. This obviously helps in extracting more detailed information. Regarding the backbone design of YOLOv2, in addition to the low-resolution feature map output by the final convolutional layer of the Darknet-19 feature extraction section (the 19th convolutional layer, indicated by the red arrow in the figure below), YOLOv2 also extracts a high-resolution feature map from the 13th convolutional layer of the Darknet-19 network (indicated by the green arrow in the figure below). It then merges these two feature maps together to serve as the feature extraction output of the entire backbone network. Under this setup, the feature output of the backbone network combines low-resolution features from the tail of the network and high-resolution features from the middle of the network, making the extracted information more complete. This is the so-called Passthrough architecture of the YOLOv2 model. !image.png Given an input image resolution of 416x416, the dimension of the high-resolution feature map output from the middle section of the backbone network is 26x26x512, while the dimension of the low-resolution feature map output from its tail is 13x13x1024. How can feature maps of different resolutions be merged together for output? The answer is the Reorg operation. The low-resolution feature map at the tail of Darknet-19 has a dimension of 13x13x1024. After passing through a 3x3, stride=1 CBL convolution block, its dimension remains 13x13x1024. Meanwhile, the high-resolution feature map from the middle section of Darknet-19 has a dimension of 26x26x512. It undergoes a reorg operation, where the total data volume remains unchanged, but the resolution is halved and the number of channels is quadrupled. The Reorg operation for a single-channel feature map (which is essentially a data rearrangement) is illustrated in the figure below, where one channel becomes four channels. Through this operation, the dimension of the middle section's high-resolution features becomes 13x13x2048. !image.png After undergoing their respective treatments above, the spatial resolutions of both the tail and middle feature maps become 13x13, making it possible to merge all the feature maps together. Therefore, after this merger, the number of channels in the output feature map is 1024 + 2048 = 3072 channels, resulting in an overall feature map data dimension of 13x13x3072. Overall Architecture of the YOLOv2 Model The overall network architecture of the YOLOv2 model is shown in the figure below: !image.png > It is worth noting that there is a discrepancy between the overall network architecture diagram above and the network architecture diagram provided in the Passthrough section regarding the number of output channels after the backbone network's merge. In the diagram above, the middle high-resolution feature map passes through a CBL convolution block during the Passthrough processing to first reduce its channels to 64, and is then reorged into a 13x13x256 dimension, making the total merged output feature data dimension 13x13x1280. In contrast, the network structure diagram in the previous section does not reduce channels for the high-resolution feature data during Passthrough, directly reorging it into a 13x13x2048 dimension, which makes the total merged output feature data dimension 13x13x3072. However, regardless of whether the output after merging has 1280 or 3072 channels, subsequent steps pass it through a CBL convolution block to reduce the channel count to 1024. The actual feature data dimension ultimately used for final prediction is always 13x13x1024. The input image has an RGB 3-channel resolution of 416x416. After being processed by the Darknet-19 backbone network and merging its tail and middle feature extraction data, the final output feature map has a dimension of 13x13x1024. Predictions for the categories and locations of the objects to be detected in the image are then made based on this 13x13x1024 dimensional feature data. This final prediction process is relatively straightforward: it performs a 1x1 convolution operation on the 13x13x1024 feature extraction data. The final output data dimension is determined by the number of classes of objects to be detected. Each grid cell in YOLOv2 contains 5 prediction boxes (also referred to as anchors in YOLOv2 terminology), and the data contained in each box includes: center coordinates and , width and height and , confidence score, and the probabilities for each class (based on the number of object categories to be recognized). - For the COCO dataset, which supports 80 object classes, the data volume for each anchor is 85. Since each grid cell contains 5 anchors, the total data volume per grid cell is 425. Thus, the data dimension of the model's inference output for the COCO dataset is 13x13x425. - For the VOC dataset, which supports 20 classes, the data volume per grid cell is 125. Thus, the data dimension of the model's inference output for the VOC dataset is 13x13x125. As can be seen from the overall YOLOv2 architecture diagram above, all network layers are convolutional networks. The fully connected layers of YOLOv1 have been completely removed, making YOLOv2 a fully convolutional network. References - "YOLO Object Detection" by Jianhua Yang, Ruifeng Li, Chapter 6: YOLOv2 - DAY28 Deep Learning - Convolutional Neural Networks - YOLO v2 (Part 2) - iT邦幫忙
An Interpretation of the YOLOv1 Model Network Architecture
Overview of the YOLOv1 Architecture The design logic of YOLOv1 is to use a single, complete convolutional neural network to detect objects and their locations in an image end-to-end. Compared to the dominant two-stage paradigm represented by the R-CNN series at the time, YOLOv1 pioneered the architectural design philosophy of using a one-stage approach to achieve both object localization and classification. The network architecture of YOLOv1 draws inspiration from GoogLeNet to design its backbone network. However, instead of adopting GoogLeNet's Inception modules, it uses blocks composed of cascaded 1x1 and 3x3 convolutional layers, making the backbone architecture remarkably simple. The architecture diagram of the YOLOv1 network is shown below: !image.png As seen in the YOLOv1 architecture diagram above, the YOLOv1 model consists of 24 convolutional layers for extracting feature information from the input image, followed by two fully connected layers for predicting the target's location, confidence score, and class probabilities. In the convolutional layers, multiple 1x1 convolutional layers are used to compress the number of feature map channels. As for the activation function, Leaky ReLU is used across all layers except for the final layer, which employs a linear activation. The overall network architecture does not differ significantly from various classical CNN-based image recognition models; the differences primarily lie in the classification and output stages. The final convolutional layer outputs a feature map with 1024 channels and a 7x7 spatial resolution. This feature map is flattened into a 1D vector of size 7x7x1024, which is then processed by two consecutive fully connected layers (containing 4096 and 1470 neurons, respectively). The final fully connected layer with 1470 neurons outputs the final predictions. Because the object detection task performed by YOLOv1 is a spatial task, the 1470 output values of this fully connected layer are reshaped into a 3D tensor of size 7x7x30, where 7x7 is the spatial resolution of this 3D tensor, and 30 is the number of channels. Subsequent sections will explain the object detection information represented by this 7x7x30 tensor in detail. Because detecting and recognizing object locations and classes requires finer-grained visual information, YOLOv1 increased the input image resolution from the standard 224x224 resolution used in various image recognition models to 448x448. Parameter Count of YOLOv1 One obvious issue with the YOLOv1 model is that its parameter count is exceptionally large. Considering only the connections between the output of the final convolutional layer (with dimensions of 7x7x1024) and the first fully connected layer (4096 neurons), the number of parameters reaches: , which is approximately 200 million parameters. This inevitably leads to a very slow inference speed and a large memory footprint during model execution. However, this was unavoidable at the time. After all, back in 2015 when YOLOv1 was introduced, the standard design for convolutional neural networks involved connecting convolutional layers to fully connected layers, each with distinct responsibilities. Consequently, the problem of an excessive number of parameters in fully connected layers was difficult to avoid—until the advent of Global Average Pooling layers, which ultimately resolved this issue. Execution Flow of YOLOv1 The object detection workflow for inference images in YOLOv1 can be broken down into several steps: input image preprocessing, feature extraction via the convolutional neural network, object and location detection via fully connected layers, and final output of the detection results. !image.png The input image preprocessing stage mainly involves two aspects. First, as mentioned above, the input image for YOLOv1 is an RGB image with a resolution of 448x448, so images of varying resolutions must first be resized to this dimensions. Meanwhile, to avoid aspect ratio distortion in the input image, non-1:1 aspect ratio images also need to be processed using techniques like padding. After completing resolution scaling and other steps, pixel normalization is performed to scale each pixel value in the input image from the range to , or further standardized to a distribution with a mean of 0 and a variance of 1. Next, the preprocessed 3-channel RGB image with a resolution of 448x448 passes through 24 consecutive convolutional layers (including their corresponding pooling layers) to progressively extract contained features, outputting a 7x7x1024 feature map. This 7x7x1024 feature map is flattened and then passed through two fully connected layers to detect object categories and locations based on the feature data. The detection results are resized into a 3D tensor of size 7x7x30. Parsing this output 3D tensor yields the object categories and their locations within the image. _The key to the entire execution flow actually lies in the mapping between the information contained in the 7x7x1024 feature map output by the convolutional layers and the 7x7x30 3D tensor output by the model._ Based on this, the entire process can be viewed as dividing the input 448x448 image into a grid. Whether it is the 7x7x1024 feature map output by the convolutional layers or the final 7x7x30 3D tensor output, the 1024 and 30 dimensions both correspond to the feature information and detection results of each grid cell: for each grid cell, processing through 24 convolutional layers yields a feature vector of length 1024, which contains high-level feature information for that grid. Then, the fully connected layer iterates over these grid cells, processes the feature information within them to predict whether there is an object center coordinate at each grid cell, and predicts the corresponding object category, outputting the result as a vector of length 30 to represent the detection information for that grid. !image.png Therefore, as seen from the above process, image input and convolutional layer detection are no different from previous image recognition models like VGG and GoogLeNet. The biggest difference in YOLOv1 lies in the processing results of the fully connected layers, which need to be resized into a 3D tensor for output. The data contained in this 3D tensor corresponds to the object categories and locations detected from the input image. Interpreting YOLOv1 Output Information As described above, the output of YOLOv1 is a 3D tensor of size 7x7x30, corresponding to a grid, meaning the output information contained in each grid cell is a 1D vector of length 30. This 1D vector contains the following information: the parameters for two object detection bounding boxes, BOX1 and BOX2, along with the recognition probabilities for 20 object detection classes. The parameters for each BOX include its center coordinates , width and height , and the confidence score of the detection made by this BOX, totaling 5 parameters per BOX. The 5 parameters for two BOXes, plus the recognition probabilities for 20 target detection classes, combine to form a 1D vector of length 30, as illustrated in the figure below. > Why 20 classes? This is because the original YOLOv1 paper was trained using the VOC dataset. The VOC dataset supports the detection of 20 object classes in total, so the output of the YOLOv1 model defaults to the recognition rates of these 20 classes. If the COCO dataset, which supports 80 classes, were used, the output length for each grid cell would be . !image.png How Does Detection Work? YOLOv1 has a core design regarding the question of "who is responsible for object prediction": Whichever grid cell contains the center point of the object to be detected, that cell is responsible for finding that object. This point is crucial for understanding the output of YOLOv1. Taking the image below as an example, although the kitten spans across multiple grid cells, its center point lies within the pink grid cell. Therefore, in the output of the YOLOv1 model, only the BOX output by the pink grid cell contains the relevant detection information for the kitten, while all other grid cells serve as background boxes. !image.png BOX Coordinate Position Parameters As mentioned above, each grid cell contains detection results for two BOXes, with each BOX representing its position within the grid using . Using the kitten image above to explain the parameters: One of the BOXes output by the pink grid corresponds to the detection box information for the kitten, where represents the offset of the predicted BOX's center relative to the top-left corner of its containing grid cell. Taking the kitten in the image above as an example, the red dot in the pink box is the center point of the kitten. Thus, the corresponding to this kitten detection box (the large red box) is the offset of the red dot relative to the top-left corner of the pink box. and represent the proportions of the width and height of this predicted BOX (the large red box) relative to the width and height of the entire image (448x448 resolution). How to Understand BOX Confidence? The confidence parameter of a BOX bounding box is essentially used to determine whether the BOX contains the center point of an object to be detected. Ideally, according to YOLOv1's design, bounding boxes output by grid cells containing an object's center point will have a high confidence score, while grid cells not containing a center point will output low confidence scores, or even zero. > The BOX confidence directly determines whether a grid cell contains the center point of a target object to be detected. During post-processing, by checking the confidence parameter, one can determine whether the current grid cell contains the center point of a target object. Once confirmed, the recognition probabilities of the 20 object detection classes are checked to determine the category of the target object. What if the Same Grid Cell Contains Center Points of Two Objects? According to YOLOv1's design, the output information of each grid cell can only mark the better of the two prediction boxes as a "positive sample" and assign it only 1 class label. Therefore, when the center points of two different objects happen to fall into the same grid cell, the YOLOv1 network can only: - Select one of the objects (usually the one with a higher IoU with the default box) as the regression/classification target for that grid cell; - Treat the other object as if it does not exist—during training, it incurs neither coordinate loss nor class loss; - During the inference stage, this grid cell can at most output the detection box for the first object, and the second object will inevitably be missed. This is the root cause of YOLOv1's sharp drop in recall rates in scenarios with dense, small objects. References - YOLOv1 詳細解讀. 設計概念: | by Steven Meng | Medium - YOLO Object Detection, Chapter 3: YOLOv1 - YOLO演進 — 1. 在上篇文章有提到過 YOLO 的作法以及如何訓練，在這篇再更詳細的說明YOLO… | by 李謦伊 | 謦伊的閱讀筆記 | Medium
Learning Classical CNN Network Architectures: ResNet
The ResNet network architecture was proposed by Kaiming He and other researchers from Microsoft Research Asia in 2015 in their paper Deep Residual Learning for Image Recognition. It won first place in multiple tasks such as classification, detection, and localization in the 2015 ImageNet image recognition competition. The core contribution of this model lies in solving the training bottleneck of deep networks, making it possible to train networks with hundreds or even thousands of layers, which greatly advanced the development of computer vision. Training Challenges of Deep Networks Before the ResNet network architecture was introduced, researchers discovered during the research and training of deep neural networks that: - Theoretically, the deeper the network structure and the more layers it has, the stronger its expressive power becomes, making it better suited for handling complex problems. - However, once the number of layers reaches a certain threshold, the accuracy of model training and recognition begins to degrade. This drop in accuracy is not caused by overfitting, as the accuracy for both training and validation data gets worse. Increasing the network depth not only fails to optimize network performance, but instead leads to a decline in output accuracy. This phenomenon is known as the degradation problem of deep networks. As shown in the figure below, a deeper network (56 layers vs. 20 layers) not only fails to bring better performance, but actually "degrades" on both the training and test sets. !image.png > It is worth noting that the problem addressed by ResNet is not the vanishing or exploding gradient problem that occurs during the backpropagation of deep networks. Vanishing and exploding gradients happen because, as network depth increases, gradient values tend to approach zero or remain infinite during the forward-backward pass of training, causing network parameters to fail to update or remain in a divergent state. In fact, these two issues have largely been mitigated by introducing BatchNorm into networks. The "degradation" problem tackled by ResNet is an entirely new issue independent of vanishing/exploding gradients. ResNet solved the aforementioned network degradation problem by introducing residual blocks into the model architecture, allowing the depth of deep networks to increase dramatically. As can be seen from the figure below, GoogLeNet, the winner of the 2014 ImageNet dataset image recognition competition, had only 22 layers, whereas ResNet, the winner in 2015, reached a network depth of 152 layers. !image.png Residual Blocks Traditional neural network structures generally connect different convolutional layers, pooling layers, and fully connected layers in a simple sequential order. Each layer in the network only receives information from its immediate predecessor without skipping layers to receive outputs from earlier ones, and each layer's output is only passed to the immediate next layer. The connection pattern of a normal block on the left side of the figure below illustrates this. With such a sequential pattern, as network depth increases, this single connection method leads to network performance degradation. In contrast, ResNet introduces the concept of residual blocks. Building upon traditional network connections, shortcut connections are added so that the model's input data can propagate forward more rapidly through cross-layer connections, as shown on the right side of the figure below. !image.png Why is Residual Mapping Feasible? Suppose our original input is , and the ideal mapping we expect the network to learn is . If a normal connection structure of traditional neural network patterns is used, the network needs to directly fit the mapping from the input of the network block. However, if a residual block pattern is adopted, the block itself does not need to fit ; it only needs to fit the residual mapping . Learning residual mapping is easier to implement and optimize in reality, which is the underlying logic of how residual blocks solve the problem. Let's explain this using the figure below. The mapping that the following network block needs to learn is . If we directly use its input to learn and train the corresponding mapping , this is what traditional network connections do. For the residual block used in ResNet, the input remains , and the target to be learned is still . By routing the input directly to the output via a shortcut connection and adding it to the output of the residual block , we get the output of the entire module . In this way, the learning objective for the internal parameters of the residual block is no longer the direct mapping from to , but rather the mapping from to (i.e., ). _Learning and optimizing residuals in a neural network is much easier than directly learning the final mapping._ !image.png Let's take a simple example. Assume the input is 2.9, and the model's expected output is 3.0, with a difference between them of . When using a normal block with a traditional network structure to perform the above mapping, the error variation from input to output is evaluated based on the expected output of 3.0. If the residual block is used to implement this mapping, the training target for the internal parameters of the residual block is the residual . In other words, the training error variation of this network block is evaluated based on this difference of 0.1. From this, we can find that adopting the residual approach can make the error variations during model training more prominent, making the model easier to train and converge. Structure of Typical Residual Blocks In their paper, the authors proposed two residual block structures, as shown in the figure below: !image.png For networks with fewer layers, the basic residual block is constructed using the sequential convolution - ReLU - convolution structure shown on the left. For networks deeper than 50 layers, considering training depth and time, the authors proposed the bottleneck design shown on the right. This design introduces a convolution to reduce dimensions and then another convolution to restore dimensions. While saving a large number of parameters, it balances the performance and time complexity of deep networks. ResNet Model Architecture ResNet has versions with different numbers of network layers, such as 18, 34, 50, 101, and 152 layers. Among them, the most commonly used is the 50-layer network structure, namely ResNet-50. The overall architecture of the ResNet model can basically be divided into three parts: the input layer, the intermediate sequence of residual blocks, and the final output layer. The input and output layers are identical across all versions of ResNet; the differences among versions lie primarily in the number and structure of the intermediate residual blocks. The input layer of ResNet—namely the first two layers of its network structure—is identical to that introduced in GoogLeNet from the notes Learning Classical CNN Network Architectures: GoogLeNet: The input image first passes through a convolution with a stride of 2, outputting feature maps with 64 channels, followed by a max pooling layer with a stride of 2. The intermediate residual block section of ResNet consists of consecutively connected basic residual blocks. However, for different versions of ResNet models, the configuration and quantity of residual blocks they contain vary. !image.png Taking ResNet-34 as an example, the residual block section contains: three consecutive 64-channel convolutional residual blocks, four consecutive 128-channel convolutional residual blocks, six consecutive 256-channel convolutional residual blocks, and three consecutive 512-channel convolutional residual blocks. Ultimately, it outputs feature maps to the output layer section. The entire ResNet-34 model structure consists of a total of 1 convolutional layer, 32 convolutional layers, and one fully connected layer, totaling 34 parameter layers. The figure below shows the network architecture diagram of the ResNet-34 model: !image.png The feature maps output by the residual block section then pass through a global average pooling layer and are fed into the final fully connected layer with 1,000 neurons. Combined with the Softmax activation function, this achieves the recognition probability output for the 1,000 recognition categories of the ImageNet dataset. References - ResNet: Revolutionizing Deep Learning in Image Recognition - The Annotated ResNet-50 | Towards Data Science - ResNet Detailed Explanation and Practice - CSDN Blog - Intuitive Understanding of ResNet — Introduction, Concepts, and Implementation (Python Keras) | by Chi Ming Lee | Medium
Detailed Explanation of Object Detection Model Evaluation Metrics: mAP, Recall, and Precision
In the field of object detection in computer vision, the primary evaluation metrics are mAP, Recall, and Precision. This article provides a detailed breakdown of the concepts and calculation logic of IoU, Precision, Recall, AP, and mAP. IOU IoU: Intersection Over Union. In the execution of object detection algorithms, every object to be detected in each training image is annotated with a bounding box corresponding to its location. This annotated bounding box is the ground truth detection region for the object. Meanwhile, the object detection algorithm performs detection and inference on the objects in the image, outputting a predicted bounding box. The IoU metric evaluates the detection accuracy of an object detection algorithm for a given object based on the degree of overlap between two bounding boxes. The higher the degree of overlap between the ground truth bounding box and the predicted bounding box, the better the detection accuracy. !image.png The following diagram provides a very intuitive explanation of how IoU is calculated. Specifically, it is the intersection of the object region detected by the system and the ground truth object region (the "Area of Overlap" or "Intersection" part in the diagram below), divided by the union of the two regions (the "Area of Union" or "Union" part in the diagram below). !image.png Therefore, for a given object to be detected in an image, its IoU metric is calculated as the intersection area of the predicted box and the ground truth box divided by the union area of the two boxes. The result of this calculation naturally falls between 0 and 1. A value of 0 indicates that the detection result has no overlap with the actual ground truth box, representing a complete detection failure, while a value of 1 indicates that the predicted box and the ground truth box completely overlap. Naturally, higher values are better for algorithm evaluation. Generally, is used as a baseline to determine recognition rate; a value greater than this threshold indicates a valid detection, while anything lower is considered an invalid detection. Building upon the concept of IoU above, performing object detection on multiple objects in a single image can yield several types of detection results: !image.png - TP (True Positive): The algorithm accurately detects an object in the image, and the IoU overlap between the two bounding boxes is greater than 0.5, making it a correct prediction. - FP (False Positive): The bounding box output by the algorithm has an overlap of less than 0.5 with the ground truth boxes of all objects in the image. A typical example is identifying the background as an object, which represents a false detection. - TN (True Negative): The region is actually background, and the algorithm does not output a corresponding bounding box. This is a normal occurrence and does not need to be considered when calculating object detection metrics. - FN (False Negative): An object that the algorithm was supposed to detect was missed, representing a false negative (missed detection). Categorizing and organizing the above detection results into a table yields: !image.png Recall and Precision Precision: Precision. Recall: Recall. With the above understanding of IoU and the various types of detection results in object detection algorithms, the concepts and calculations of Precision and Recall become quite straightforward. Precision represents the proportion of correct detections—where both the category classification and location detection () are correct—among all detection results output by the object detection algorithm. The formula for Precision is shown in the figure below: !image.png Recall represents the proportion of actual objects in the image that were correctly detected (correct category classification and ) by the object detection algorithm. The formula for Recall is shown in the figure below: !image.png Taking the teacup detection in the image above as an example, Precision represents the proportion of the algorithm's output teacup detections that are actually teacups. Recall represents the proportion of all teacups present in the image that were accurately detected by the algorithm. AP and mAP AP: Average Precision, representing the detection accuracy for a single category. mAP: mean Average Precision, representing the average detection accuracy across all categories. Let's use an example to explain the calculation process of AP in greater detail. Suppose we are detecting dog targets using a certain dataset. There are 8 ground truth dog targets in the dataset, corresponding to GT1–GT8 (where GT stands for Ground Truth, representing the manually annotated locations of the actual dog targets). Meanwhile, the algorithm detects 10 dog targets and their locations, denoted as BB1–BB10 (where BB stands for the Bounding Box of each dog target detected by the algorithm). Some bounding boxes from the algorithm's detection results on the dataset only captured the dog's head or paws, meaning their IoU is very low (e.g., BB3, BB4, BB5, BB7, and BB8 all have relatively low IoU outputs). Furthermore, two ground truth dog targets, GT6 and GT7, failed to be detected (), meaning these two ground truth targets were missed. Based on the above information, the execution results of the object detection algorithm are summarized as follows: It is worth noting that each detection result output by an object detection algorithm contains not only the predicted category and the bounding box coordinates, but also a detection confidence score, which represents the probability that the detected target is correct. !image.png As can be seen from the figure above, for a single ground truth dog target, the detection algorithm may output multiple independent detection results; for example, both BB1 and BB3 correspond to GT1. Generally, we consider an algorithm's output detection result to be a valid TP only when . Therefore, the 10 detection results () output by the algorithm are evaluated for IoU as follows: !image.png Next, according to the procedure, sort them based on their detection confidence (i.e., the probability of being a dog in the figure above): !image.png Then, following the sorted order, calculate the Precision and Recall at each rank: - For Rank 1, , and . - For Rank 2, , and . - For Rank 3, , and . - For Rank 4, , and . - By extension, Rank 5 has , . - Rank 6: , . - Rank 7: , . - Rank 8: , . - Rank 9: , . - Rank 10: , . Thus, based on the 10 detection results output by the object detection algorithm, 10 sets of Precision and Recall data are obtained as follows: !image.png Using these 10 sets of values, we can plot the P-R curve corresponding to the algorithm's output results: !image.png By definition, the AP metric is calculated as the area or integral under the blue P-R curve marked in the P-R plot above. Since both Precision and Recall metrics fall between 0 and 1, the calculated AP metric naturally also falls between 0 and 1. !image.png Calculating the area or integral of the P-R curve directly can be complex, so the two discrete sequence formulas provided in the image above can be used to compute the AP metric. In the first formula, is the precision at the -th rank, and is the difference in recall between the -th rank and the -th rank. Therefore, calculating the AP metric using this formula gives: In the second formula, is the precision at the -th rank, and is defined as 1 if the -th rank is a valid detection (TP), and 0 otherwise. "Number of relevant documents" represents the total number of TPs, which is 8 here (). Therefore, calculating the AP metric based on this formula gives: As we can see, although the two formulas above use different expressions, they lead to the same destination: the final calculated AP value is 0.579 in both cases. Now that the calculation of AP is complete, calculating mAP is relatively simple. The AP calculation performed above was only for the single category of dogs. Generally, object detection algorithms can output detection results for multiple different categories; for example, the COCO dataset contains annotation information for 80 categories. Thus, the mAP metric of an object detection algorithm is obtained by first independently calculating the AP detection metric for each category it supports, and then summing up all the AP values and taking their average to yield the mAP metric of the overall algorithm across all categories. !image.png References - Deep Learning with PyTorch Object Detection in Action, 1.2 - Yy's Program: What do IOU, AP, and mAP commonly used in image recognition mean? - Learning Model: What is AP/mAP/IoU? [Reprint] | by Ryan Lu | AI Anti-Monster Castle | Medium
Working Principles and Applications of Short-Wave Infrared (SWIR) Sensors
This article briefly summarizes the technical concepts of SWIR sensors and imaging systems based on short-wave infrared band imaging, as well as their common applications in industry and daily life, laying a foundation for understanding and applying this technology in practical products. Introduction to SWIR Technology SWIR stands for Short Wave InfraRed Light. The wavelength range is typically between 0.9–1.7 µm (some definitions also cite 0.9–2.5 µm). The figure below shows the position of infrared light in the electromagnetic spectrum. In general, the infrared band can be divided into SWIR (Short-Wave Infrared), MWIR (Mid-Wave Infrared), and LWIR (Long-Wave Infrared). The approximate wavelength ranges corresponding to each part are shown in the diagram below. !image.png From the perspective of exposure and imaging principles, SWIR sensors are similar to the common visible-spectrum image sensors we encounter daily. Both rely on the reflection characteristics of the target material: photons contained in the external illumination are either reflected or absorbed by the object, thereby producing the contrast and shadows required for high-resolution images. - This differs significantly from the imaging principles of the MWIR and LWIR bands, which rely on the blackbody radiation emitted by the object itself, with the radiation volume being temperature-dependent. _Therefore, visible light and SWIR imaging depend on external illumination sources; light shines onto the object, gets reflected, and then enters the lens to form an image. Consequently, their imaging is not all-weather, and supplementary lighting is generally required at night. In contrast, MWIR and LWIR imaging rely on the object's own blackbody radiation, thus supporting 24/7 all-weather imaging._ Therefore, images produced by SWIR cameras are comparable to visible light in terms of sensitivity and detail, featuring distinct shadows and high contrast. Of course, SWIR imaging is monochromatic. Manufacturing Processes and Limitations of SWIR Sensors Common visible-light image sensors are produced based on standard silicon technology. Because the absorption coefficient of silicon drops rapidly near the 1100 nm wavelength, the practical effective detection upper limit of silicon-based optical sensors is generally around 1000 nm. In other words, silicon has extremely weak absorption capacity for photons beyond 1000 nm, resulting in a signal response close to zero in that wavelength band. Therefore, silicon technology is not suitable for SWIR imaging. SWIR imaging sensors typically adopt InGaAs (Indium Gallium Arsenide) technology, which has an upper photon detection wavelength limit of approximately 1700 nm. Consequently, InGaAs sensors can efficiently absorb and convert photons in the 900–1700 nm range, making them the mainstream choice in the field of SWIR imaging. The challenges associated with InGaAs sensors include: - InGaAs cannot be grown directly on silicon wafers; the materials are expensive, the manufacturing process is complex, and yields are low, resulting in high production costs and high prices for the sensors. - Reducing the pixel size of InGaAs sensors negatively impacts image quality, making it difficult to achieve the high integration and large arrays typical of CMOS sensors. Typical Applications of SWIR Sensors Night Vision Imaging Nighttime image monitoring has always been a challenge for ordinary visible-light cameras. The main reason is not that there is no light at night, but rather that the light energy in the visible spectrum is too weak, and too few photons enter the visible-light image sensor. This necessitates the use of external illumination, increased gain (which degrades image quality), and larger pixel sizes or lens apertures (which are too costly) to mitigate the issue. However, in the 900–1700 nm short-wave infrared band, sources such as nightglow, starlight, and 1.5 µm laser reflections remain abundant. SWIR sensors (InGaAs) can convert these invisible-to-the-human-eye lights into high-resolution images. Therefore, in the field of night vision imaging, SWIR offers image brightness 5 to 7 times higher than that of visible light; compared to thermal imagers operating in the LWIR band, its image resolution is significantly higher. The left image below is captured by a visible-light camera, and the right image is captured by a SWIR camera under the same environment: !image.png Fruit Moisture and Agricultural Produce Sorting Water exhibits relatively strong absorption characteristics for light in the 900–1700 nm band, peaking at 1.45 µm, followed by 1.95 µm. Typical agricultural produce and fruits consist of about 70% water. The higher the water content, the lower the object's reflectivity to SWIR-band light, causing it to appear darker in SWIR images. Any physiological changes in produce and fruits (such as ripening, bruising, rotting, or sugar accumulation) alter their internal moisture distribution or sugar concentration, thereby revealing invisible light-and-dark contrasts in SWIR images. !image.png Imaging Through Haze and Fog In foggy or hazy weather conditions, particulate diameters generally range between 0.1 µm and 1 µm. Haze particles in this range cause maximum reflection of visible light, whereas their scattering effect on the SWIR band is much smaller. Consequently, a greater amount of SWIR-band light can penetrate haze particles, drastically reducing the white fog layer in images captured by SWIR cameras. Therefore, in dense fog, haze, and other adverse weather conditions, SWIR imaging systems provide much clearer images than visible-light cameras in monitoring applications such as border security, airports, and coastlines. !image.png High-Temperature Detection When an object's temperature exceeds 140 degrees Celsius, it increasingly emits infrared radiation outward, which can be detected via SWIR technology. The hotter the object, the more infrared radiation it emits, and the brighter it appears in the image. This means SWIR technology can provide a competitive advantage for temperature detection in industrial monitoring and other applications, as it enables non-contact temperature monitoring of materials and products. This is particularly useful in environments where traditional temperature measurement methods are impractical or dangerous. For example, the image below shows a comparison of a soldering iron at 400 degrees Celsius captured by an ordinary visible-light camera (left) and a SWIR camera (right). !image.png References - What is SWIR? | Edmund Optics - What is SWIR? | SWIR Vision Systems - Understanding SWIR: Applications and Benefits - Oxford Instruments - Clear Align Knowledge Center - Optical Engineering Insights & Resources || Why SWIR is Ideal for Border Surveillance & Maritime Imaging
Learning Classical CNN Network Architectures: GoogLeNet
This article provides a comprehensive and detailed summary of the problems inherent in traditional convolutional neural networks and the design principles of the classic GoogLeNet architecture in the field of computer vision and image recognition. Problems in AlexNet and VGGNet In the previous two notes, Learning Classical CNN Network Architectures: AlexNet and Learning Classical CNN Network Architectures: VGGNet, the architectures and implementations of AlexNet and VGGNet—two classic CNN models—were summarized respectively. Relatively speaking, AlexNet has fewer layers and parameters, which limits the network's ability to extract and learn complex patterns from data, resulting in suboptimal ultimate recognition and detection efficiency. VGGNet increased the depth of the CNN neural network, reaching 16 and 19 parameter layers. However, other issues caused by this increased depth naturally arose: - First is the vanishing gradient problem. As is well known, during the training of neural networks, loss functions are used to evaluate model performance. Based on the loss value, backpropagation combined with gradient descent is used to update parameters layer by layer from the output layer backward, thereby reducing the final loss function output. However, as the network depth increases, the backpropagating gradient signal shrinks progressively as it passes through earlier layers, ultimately making it difficult for parameters in the network (especially those in the earlier layers) to receive effective updates during training. - Second, deep models have excessively high computational demands. Taking the VGG16 network model as an example, its parameters reach up to 138M, and the model size exceeds 500MB, requiring substantial computational power for both training and inference. The GoogLeNet architecture redesigns traditional CNN network structures to address these issues and has achieved great application success. GoogLeNet, also known as Inception-v1, was first proposed in 2014 and won first place in the 2014 ILSVRC competition, achieving a Top-5 Error of only 6.67%. _Even more remarkably, GoogLeNet has only 6.8M parameters—9 times fewer than AlexNet and an impressive 20 times fewer than VGG16._ The biggest innovation of GoogLeNet is the introduction of the Inception module, which is why GoogLeNet is also referred to as InceptionNet. Inception Module An Inception module is a building block that allows data to be processed in parallel across multiple scales, enabling the network to capture features at different scales more effectively than previous classic convolutional neural network architectures. An Inception module typically consists of multiple convolutional layers, each using a different kernel size. Unlike previous network models, the multiple convolutional layers in an Inception module operate in parallel. Thus, for the exact same input feature map, the network can compute using convolutional kernels of various sizes. Once these convolutional layers complete their computations, their outputs are concatenated and passed to a pooling layer for further processing. This is the overarching design philosophy of the Inception module. From a structural perspective, the Inception modules proposed in GoogLeNet can be classified into two types: the Native type and the Dimensionality Reduction type. !image.png The common characteristic of both Inception module types is that input data obtained from the previous layer's feature map undergoes parallel processing via 1x1 convolutions, 3x3 convolutions, 5x5 convolutions, and 3x3 max pooling. The outputs from these four paths are concatenated to form the final output of the Inception module. Their difference lies in the fact that: _The dimensionality-reduced version of the Inception module adds a 1x1 convolution operation before the 3x3 and 5x5 convolutions, as well as after the 3x3 max pooling._ Among them: - The 1x1 convolutional block is used to capture local details at the pixel level of the feature map while keeping its spatial resolution unchanged. - The 3x3 convolutional block captures mid-range features from the input feature map. - The 5x5 convolutional block is used to capture broader-range features. - The independent 3x3 max pooling block helps preserve salient responses of the original features (such as edges and textures). As mentioned earlier, the difference between the native module and the dimensionality-reduced module is that the latter incorporates three additional 1x1 convolution operations—placed before the 3x3 and 5x5 convolutions and after the 3x3 max pooling, respectively. _The purpose of the two 1x1 convolutions placed before the 3x3 and 5x5 convolutions is to reduce computational complexity and the number of parameters; whereas the 1x1 convolution placed after the 3x3 max pooling is used to control the number of output channels (otherwise, the pooled channel count would match the input channel count, resulting in an excessively large total channel count). Regardless, the output feature maps from all four paths share the same spatial resolution, making it convenient to concatenate them along the depth dimension._ > Thus, in the dimensionality-reduced version of the Inception module, the 1x1 convolution in the independent pooling path serves a completely different purpose from the extra 1x1 convolutions added in the convolutional paths. How 1x1 Convolutions Reduce Computational Cost In the dimensionality-reduced Inception module, the 1x1 convolutions added before the 3x3 and 5x5 convolutions are primarily intended to reduce the number of parameters and computational cost. Taking the following figure as an example, without using a 1x1 convolution, converting a feature map from to using a 5x5 convolution results in a computational cost as high as: __. !image.png When a 1x1 convolutional layer is inserted between the two feature maps, as shown in the figure below, the computational cost drops dramatically: !image.png Here, the computational cost of the 1x1 convolution is , and the computational cost of the 5x5 convolution is . _Combined, they total only 5.3M—a mere 1/20th of the computation without the 1x1 convolution!_ How the Inception Module Performs Output Concatenation As described above, an Inception module processes input data through four independent paths. After processing, the outputs of these four paths are concatenated at the output stage of the Inception module. The primary prerequisite for concatenating the outputs of the four paths is that their feature map resolutions must be identical. Therefore, the four independent channels are intentionally designed with a stride of 1 and appropriate padding to ensure that the output feature map resolutions match across all channels under the same output data dimensions: - 1x1 convolution path: padding=0 - 3x3 convolution path: padding=1 - 5x5 convolution path: padding=2 - 3x3 max pooling path: padding=1 !image.png With the above settings, the spatial resolutions of the output feature maps from all four channels remain identical after processing the same input data, while only their channel counts (i.e., the number of feature maps in each channel) differ. When the output feature map resolutions across all four channels are identical, concatenating them becomes straightforward: one simply stacks the feature maps of the four channels side by side along the channel dimension. Taking the first Inception module (3a) in the GoogLeNet (Inception V1) paper as an example, the input tensor shape from the previous layer is , produced by preceding convolutional and pooling layers. Processing occurs across the four paths of the Inception module: - 1x1 convolution outputs 64 channels with a resolution (). - 3x3 convolution first reduces dimensions to 96 channels via a 1x1 convolution (), then outputs 128 channels via a 3x3 convolution with padding=1 (). - 5x5 convolution first reduces dimensions to 16 channels via a 1x1 convolution (), then outputs 32 channels via a 5x5 convolution with padding=2 (). - 3x3 max pooling first applies 3x3 max pooling (padding=1) maintaining 192 channels (), then reduces the channel count to 32 via a 1x1 convolution (). Finally, concatenating the outputs of these four independent flows yields a total feature map of channels with a resolution as the final output of the Inception module. Global Average Pooling In traditional convolutional neural network architectures, consecutive fully connected layers are used to classify images based on feature data extracted by convolutional layers. The biggest drawback of fully connected layers is their excessive parameter count, leading to severe overfitting. To address this issue, GoogLeNet adopted the same approach as Network in Network (NiN), using global average pooling to replace fully connected layers. This not only solves the problems of excessive parameters and overfitting in traditional models but also noticeably improves classification accuracy, as demonstrated by practical data testing. The figure below illustrates the different computational logic between a fully connected layer and a global average pooling layer given the same input feature vector. For more details on global average pooling, please refer to Learning Classical CNN Network Architectures: NiN. !image.png Auxiliary Classifier GoogLeNet has a total of 22 layers. In deep networks, during backpropagation and gradient calculation, gradients diminish progressively across intermediate layers moving backward, ultimately leading to the vanishing gradient problem where parameters in the earlier layers of the network cannot be updated effectively. GoogLeNet attempts to solve this problem by introducing an Auxiliary Classifier. The core idea is to attach a small classification sub-network (Auxiliary Classifier) to intermediate parts of the network. This sub-network performs classification based on the feature maps from earlier layers and computes its own loss value. During model training, this loss is added to the model's final output loss to amplify gradient signals and mitigate the vanishing gradient effect. Auxiliary classifiers are used exclusively during the training phase; during inference, only the main network output is retained, meaning inference speed remains unaffected. In the GoogLeNet architecture, an Auxiliary Classifier is added after the 4a and 4d Inception modules, respectively (refer to the overall GoogLeNet network structure section below). _The structure of each Auxiliary Classifier is as follows:_ - The input consists of the feature maps output by the respective Inception module, such as feature maps with 512 channels and a resolution. - 5x5 average pooling: Applies a 5x5 average pooling operation with a stride of 3 to the input feature maps, resulting in 512 channels with a resolution. - 1x1 convolution: Reduces the channel count from 512 to 128 using a 1x1 convolution, outputting 128 channels at a resolution. - Flattens the feature map and connects it to a fully connected layer with 1024 neurons using the ReLU activation function. - Adds a Dropout operation with a dropout rate of 0.7. - Connects to a fully connected layer with 1000 neurons, corresponding to the 1000 categories of the ImageNet dataset. - The 1000-neuron multiply-accumulate output passes through a softmax operation to yield a 1000-dimensional prediction vector. The prediction output of the Auxiliary Classifier is compared against the ground truth labels to compute the auxiliary loss. The loss values generated by the two auxiliary classifiers are multiplied by a weight of 0.3 and added to the main network's classification loss to form the total network loss: During network parameter training, this TotalLoss is backpropagated to update the network parameters. Overall Structure of GoogLeNet With the background information above, understanding the overall architecture of GoogLeNet becomes relatively straightforward: !image.png From the overall architecture, we can observe: - The beginning of GoogLeNet reduces the input image resolution through sequential convolutional and pooling operations. - It then continuously extracts image features through 9 Inception modules, grouped into 3 stages (). Consecutive modules within each Inception stage are directly connected, while stages are bridged by 3x3 max pooling layers. - The output of the final Inception module passes through global average pooling followed by a fully connected layer to perform final image classification. - Additionally, Auxiliary Classifiers are attached to the outputs of Inception 4a and 4d to provide stage-specific losses that assist the training process. The data dimensions and parameter counts for each layer in GoogLeNet are detailed in the table below: !image.png References - GoogLeNet: A Deep Dive into Google’s Neural Network Technology | by Siddhesh Bangar | Medium - Neutrino's Blog: GoogLeNet 簡介與小實驗 - 7.4. Networks with Parallel Concatenations (GoogLeNet) — Dive into Deep Learning 2.0.0 documentation - Understanding GoogLeNet Model - CNN Architecture - GeeksforGeeks - GoogLeNet: Revolutionizing Deep Learning with Inception - Inception-v1 (GoogLeNet) — Winner of ILSVRC 2014 (Image Classification) | by Moris | Computer Vision Note | Medium - 深度学习经典网络解析：6.GoogLeNet-腾讯云开发者社区-腾讯云
Learning Classical CNN Model Architectures: NiN
This article summarizes the design philosophy, overall architecture, and specific implementation details of NiN (Network in Network), a classic model in convolutional neural networks. The design patterns of the three classical convolutional neural network models—LeNet, AlexNet, and VGGNet—are largely identical: they perform feature extraction through a series of convolutional and pooling layer combinations, flatten the extracted feature data, and then feed it into a series of fully connected layers for classification to ultimately output image recognition probabilities. The aforementioned classical convolutional neural network design paradigm presents two major design challenges: - The fully connected layers responsible for image classification based on feature data contain an excessive number of parameters. For instance, in the VGG16 network, the parameter count reaches up to 138MB, and the storage space occupied by the model parameters exceeds 500MB. - It is impractical to add fully connected layers in the first half of the network to increase non-linearity; doing so would disrupt spatial structures while demanding even more parameters and memory. To address these two architectural design issues, Network in Network (NiN), published in 2013 by Min Lin from the National University of Singapore, introduced a novel design perspective to the architectural philosophy of classical convolutional neural networks. The two most significant innovations of the NiN architecture over traditional CNNs are the MLP convolutional layer and the global average pooling layer. MLP Convolutional Layers The convolutional calculation logic of traditional convolutional layers has already been described in detail in the article Detailed Explanation of Convolution and Pooling Calculations in Convolutional Neural Networks and will not be repeated here. It is illustrated simply via a convolutional calculation animation: !CNN%E5%8D%B7%E7%A7%AF%E8%AE%A1%E7%AE%97%E5%8A%A8%E5%9B%BE.gif In summary, the traditional convolutional layer calculation formula is shown in the figure below, where is the index of the pixel position in the input feature map, represents the pixel at the corresponding position, and represents the parameters of the convolution kernel. Thus, convolution calculation is essentially the multiply-accumulate result of the pixels at and surrounding with the corresponding weights of the convolution kernel, which is then processed through the ReLU activation function to yield the feature map of the convolution. !image.png In the implementation of the NiN network, convolution calculations are organized in the manner of an MLP Block. A typical structure of an MLP Block is shown in the figure below: !image.png For each MLP Block, the input is either the original 3-channel RGB image or multiple feature maps output by the previous MLP Block. Inside each MLP Block, there are three consecutive convolutional layers, where the input to each layer is the output of the preceding one. The convolution calculation of the first convolutional layer is identical to standard convolution, whereas the subsequent two layers are convolutional layers. The convolution still calculates the multiply-accumulate of feature map pixel values and weights, followed by non-linear processing using the ReLU activation function. The following code explains the structure and calculation logic of each MLP Block: > Each MLP Block contains 3 convolutional layers: a standard convolutional layer + ReLU activation, followed by two consecutive convolutional layers + ReLU activation. In terms of network architecture, each MLP Block is generally followed by a max-pooling layer. Global Average Pooling Layer In traditional convolutional neural network architectures, multiple fully connected layers are generally used to implement feature classification. The approach involves flattening the feature map produced by the last convolutional layer into a 1D vector, passing it through a succession of fully connected layers, and finally using a softmax layer for logistic regression classification. The primary issues with this approach are: an excessive number of parameters in the fully connected layers, and overfitting caused by multiple fully connected layers. The NiN network replaces fully connected layers with global average pooling operations in an attempt to resolve the issues of excessive parameters and overfitting in the latter half of classical convolutional neural networks. The specific procedure of global average pooling is to make the number of output feature maps from the final convolutional layer equal to the number of classification categories (e.g., 1000 for the ImageNet dataset, or 10 for the MNIST dataset). Then, the average value of each feature map is taken to output a vector, upon which a softmax operation is applied (for the calculation workflow of the softmax operation, refer to Detailed Summary of Softmax Classifier Calculation Workflow), ultimately outputting the recognition probabilities for each category. The workflow of the global average pooling operation is illustrated in the figure below: !image.png Taking the ImageNet dataset (which has 1000 image classification categories) as an example, the global average pooling layer obtains 1000 feature maps from its preceding MLP convolutional layer. During global average pooling, an average calculation is performed on each feature map to obtain its mean value, ultimately yielding a vector composed of 1000 mean values. A softmax operation is then executed on this vector to output the recognition probability for each category. > As can be seen, the calculation logic of the global average pooling layer above is similar to that of ordinary average pooling and thus requires absolutely no parameters. This drastically reduces the number of model parameters (in reality, the majority of parameters in classical CNNs stem from fully connected layers) and avoids the overfitting problems inherent to traditional fully connected layers. Summary of NiN Network Architecture The overall structure of the NiN network is shown in the figure below. The image input is still a resolution RGB 3-channel image. After convolutional processing through four consecutive MLP Blocks, the final MLP Block outputs 1000 feature maps of size . These feature maps are passed through the average calculation of the global average pooling layer and softmax activation to output recognition probabilities for 1000 categories. !image.png You can refer to the figure below to understand the feature map dimensions and calculation types of each layer in the NiN architecture diagram above. Although the NiN network appears to have considerably more layers than the VGG network, its parameter count is much smaller than VGG's due to the omission of fully connected layers. !image.png The complete implementation code for the NiN network, based on the nin_block structure defined above, is as follows: References - 8.3. Network in Network (NiN) — Dive into Deep Learning 1.0.3 documentation - Convolutional Neural Networks - NiN Network (Network In Network) - Tencent Cloud Developer Community - Tencent Cloud - Review: Network In Network. This paper was written by the National University of... | by Fan | Medium
Learning CNN Classical Network Architectures: VGGNet
This article provides a detailed summary of the architecture of the classic VGGNet model in the field of computer vision and image recognition, along with its implementation in PyTorch. Introduction to VGGNet Architecture VGG stands for Visual Geometry Group. The original design concept of VGGNet was to improve model performance by increasing the depth of the convolutional neural network. VGGNet secured second place in the 2024 ImageNet Large Scale Visual Recognition Challenge (ILSVRC) (GoogLeNet took first place that year), achieving a top-5 accuracy (whether the ground truth label is among the top 5 highest probability predictions from the model) of 92.7%. In many subsequent models, VGGNet's network structure is often used as a backbone for image feature extraction. Typical models in the VGGNet architecture family include VGG16, VGG19, and VGG34, where the numbers indicate the count of parameter layers (i.e., convolutional layers and fully connected layers) contained in each model architecture. The following figure illustrates the network architecture of VGG16: !image.png - Throughout the entire VGG16 network architecture, it contains 13 convolutional layers, 5 pooling layers, and 3 fully connected layers, totaling 16 parameter layers (convolutional and fully connected layers combined). - The model takes an RGB 3-channel image with a resolution of 224x224 as input and outputs predicted probabilities for 1000 image classification categories (since VGG16 was designed to predict the ImageNet dataset, which has 1000 classes). - From a structural design standpoint, it remains a standard convolutional neural network. The entire architecture is divided into two parts: the first half is the image feature extraction component composed of convolutional and pooling layers, while the second half is the image classification component composed of fully connected layers. Therefore, the network structure has no fundamental difference from LeNet and AlexNet as summarized in Learning CNN Classical Network Architectures: LeNet and Learning CNN Classical Network Architectures: AlexNet, except that it is much deeper. Because of this significantly increased depth compared to LeNet and AlexNet, the number of parameters in VGG16 reaches 138M. Compared to VGG16, VGG19 has 3 additional convolutional layers—specifically, one extra convolutional layer in each of the conv3, conv4, and conv5 blocks—while all other parts (including kernel sizes, channel counts, pooling strategies, and fully connected layer structures) remain completely identical. In the original VGGNet paper, the authors provided a total of 6 network configurations, labeled A through E. VGG16 corresponds to configuration D, while VGG19 corresponds to configuration E. !image.png As seen in the table above, the network architectures of the 6 configurations (A-E) are largely similar, with the only differences lying in the number of convolutional layers in each block. Parameter Configuration of VGG16 Network Layers The following figure shows the hyperparameter settings for each layer in the VGG16 network architecture: !image.png - As shown, the number of convolutional filters in the conv1, conv2, conv3, and conv4/conv5 sections of the VGG16 network structure is 64, 128, 256, and 512, respectively. The number of filters doubles in each successive convolutional block, which is a crucial design principle of the VGG16 architecture. - The kernel size for all convolutional layers is 3x3 (in sharp contrast to the larger filter sizes used in AlexNet), and the convolution stride is uniformly set to 1. All pooling layers are 2x2 max pooling with a stride of 2. - Regarding activation functions, except for the final output layer which uses the multi-class softmax activation function, all other convolutional and fully connected layers use ReLU. - For the final three fully connected layers, the first two layers both contain 4096 neurons, and the final output layer corresponds to the 1000 categories of the ImageNet dataset, containing 1000 neurons. PyTorch Implementation of VGGNet The PyTorch code implementation of the VGG16 network structure is shown below: References - Understanding VGG: The Backbone of Image Recognition - VGG-Net Architecture Explained. The company Visual Geometry Group… | by Siddhesh Bangar | Medium - VGGNet - HackMD
Detailed Summary of the Calculation Flow of Softmax Classifier
The Softmax activation function is generally used for classifiers in multi-class classification problems. This article provides a detailed introduction and summary of the complete calculation flow for implementing neural network multi-class classifiers based on the Softmax activation function and the cross-entropy loss function. In practical AI projects, multi-class classification problems are frequently encountered. The most typical examples are the MNIST handwritten digit recognition dataset, which classifies 10 handwritten digits from 0 to 9, and the ImageNet image dataset, which classifies 1,000 everyday objects. In the field of machine learning and artificialintelligence, the Softmax classifier is the most commonly used implementation for multi-class classification. Softmax分类器的计算流程详细总结-Softmax多分类器.png Softmax Activation Function For multi-class classification applications in deep neural networks implemented via Softmax, the number of neurons in the fully connected layer at the very end of the network corresponds to the number of classes for the classification operation. The input to Softmax is the output of these neurons (known as logits), and the output of Softmax represents the probabilities of different classification decisions. The calculation flow of the Softmax operation is explained in detail below. Assume there are elements in array : The Softmax calculation flow for this array is as follows. First, take the exponential with base for each element in to obtain a new array : Next, sum all the elements in the array to obtain : Finally, divide each element in array by the calculated above to obtain a new array, which is the result of the Softmax calculation for : Since is the sum of all elements in array , the sum of all elements in the resulting Softmax output is 1, with each element corresponding to the recognition probability of that class for the multi-class classification output. Therefore, the Softmax calculation flow is illustrated in the diagram below: !image.png The implementation code of the Softmax calculation flow in Python is as follows: One-Hot Encoding One-hot encoding represents the probabilities corresponding to classification labels using only the numbers 0 and 1 in multi-class applications. Taking three classification categories (dog, cat, and bird) as an example, these three categories can be represented in the label data as (1, 0, 0), (0, 1, 0), and (0, 0, 1) respectively: !image.png Cross-Entropy Loss Function In the implementation of deep network multi-class classifier applications, the Softmax activation function is generally applied to process the raw outputs of all neurons in the final fully connected layer to obtain the classification probabilities corresponding to each class. The loss function typically used for such applications is the cross-entropy loss function. The expression of the cross-entropy loss function is shown in the diagram below: !image.png Here, represents the one-dimensional vector of the training sample data represented in One-Hot encoding, where only the element corresponding to the correct class is 1, and all other elements are 0. is the one-dimensional vector output after the deep network model's output is processed by Softmax, where each element corresponds to the model's predicted probability for each class. Therefore, the calculation of the cross-entropy loss function is essentially comparing the One-Hot classification results of the training sample labels with the classification prediction results for each class inferred by the model using the same sample. It determines how close the probability of the predicted correct class is to 1; the closer the predicted probability of the correct class is to 1, the smaller the loss function value. The Python code implementation of the cross-entropy function is as follows: As can be seen, the higher the probability corresponding to the correct class 2 in the predicted output (0.8 vs 0.6), the smaller the calculated result of the cross-entropy loss function. Implementation of the Softmax Classifier > Multi-class application: The output of the final fully connected layer is processed using the Softmax activation, the training sample labels are encoded using One-Hot Encoding, and the loss function uses Cross Entropy. The diagram below summarizes the calculation flow of the deep neural network implementing a multi-class classifier based on the Softmax activation function and the cross-entropy loss function. !image.png - The final fully connected layer of the deep neural network receives the input feature variables from the previous fully connected layer, performs a multiply-add operation with the weight coefficients and bias variables of the final layer, and obtains the net outputs of each neuron in the final layer. - The net outputs of the neurons in the final layer, treated as a complete one-dimensional vector, undergo the Softmax operation to output the probability prediction list for each classification inference. - The multi-class prediction output from the deep neural network is compared with the sample labels encoded in One-Hot encoding mode, and the loss is calculated using the Cross Entropy loss function. - Based on the output of the cross-entropy loss function, backpropagation is performed to adjust the parameters at all levels of the entire network. In summary, compared to other activation functions (such as Sigmoid, Tanh, ReLU, etc.), the biggest difference in the calculation flow of the Softmax activation function is that other activation functions perform non-linear transformations on the multiply-add output of each individual neuron, whereas the Softmax activation function performs calculations across the multiply-add outputs of all neurons in the final layer simultaneously. References - "Python Deep Learning and Project Practice", Chapter 3: Softmax Multi-class Classifier - Learning Softmax Regression using TensorFlow | by Leo Chiu | Handwriiten Notes | Medium - Cross Entropy Loss: Intro, Applications, Code
Learning Classical CNN Network Architectures: AlexNet
This article provides a detailed summary of the architecture of the classic AlexNet model in the field of computer vision and image recognition, along with its implementation in PyTorch. In the previous article, Learning Classical CNN Network Architectures: LeNet, we explored Yann LeCun's LeNet-5 network model designed for handwritten digit recognition. Continuing our study of classic model frameworks in computer vision, this article summarizes AlexNet, which stood out and won the ImageNet Large Scale Visual Recognition Challenge. Brief History of AlexNet In the 2012 ImageNet Large Scale Visual Recognition Challenge (ILSVRC), AlexNet—proposed by Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton—won the championship. On the competition's most crucial Top-5 Error Rate metric, AlexNet achieved a landslide victory, outperforming the runner-up by a remarkable 10 percentage points: !image.png It was precisely because of AlexNet's exceptional performance in this competition that it sparked a research boom in deep learning within the field of artificial intelligence and computer vision. AlexNet Network Architecture The network architecture diagram of AlexNet is shown below: !image.png As seen in the figure above, the overall structure of AlexNet is very similar to the LeNet-5 network architecture described in Learning Classical CNN Network Architectures: LeNet. Overall, it can be divided into two parts: - Image Feature Extraction Section: Consisting of layers from the input layer up to the final max-pooling layer, this part outputs a 6x6x256 dimensional image feature vector. - Classification Section: Consisting of the final three fully connected layers. The input to the AlexNet network is an RGB three-channel image with a resolution of 227x227, and the output consists of 1,000 classification probabilities generated via the SoftMax activation function across 1,000 neurons in the output layer. The reason for 1,000 categories is primarily because AlexNet was originally designed to participate in the ImageNet image recognition competition, which contains 1,000 image categories. The overall network structure of AlexNet contains a total of five convolutional layers, three max-pooling layers, and three fully connected layers. - The input data consists of RGB three-channel color image data with a 227x227 resolution. - The first convolutional layer contains 96 kernels of size 11x11x3, with a stride of 4 and the ReLU activation function, ultimately producing 96 feature maps of 55x55 resolution. - The first pooling layer uses 3x3 max pooling with a stride of 2, outputting 96 feature maps of 27x27 resolution. - The second convolutional layer contains 256 kernels of size 5x5x96, with a stride of 1, padding of 2, and the ReLU activation function, ultimately producing 256 feature maps of 27x27 resolution. - The second pooling layer uses 3x3 max pooling with a stride of 2, outputting 256 feature maps of 13x13 resolution. - The third convolutional layer contains 384 kernels of size 3x3x256, with a stride of 1, padding of 1, and the ReLU activation function, ultimately producing 384 feature maps of 13x13 resolution. - The fourth convolutional layer contains 384 kernels of size 3x3x384, with a stride of 1, padding of 1, and the ReLU activation function, ultimately producing 384 feature maps of 13x13 resolution. - The fifth convolutional layer contains 256 kernels of size 3x3x384, with a stride of 1, padding of 1, and the ReLU activation function, ultimately producing 256 feature maps of 13x13 resolution. - The third pooling layer uses 3x3 max pooling with a stride of 2, outputting 256 feature maps of 6x6 resolution. - This concludes the feature extraction section of AlexNet. The final pooling layer flattens the output into a 1D vector of 6x6x256 elements, which serves as the input to the first fully connected layer of the classification network. - The first fully connected layer has 6x6x256 = 9,216 inputs, 4,096 neuron outputs, and uses the ReLU activation function. - The second fully connected layer accepts 4,096 inputs, has 4,096 neuron outputs, and uses the ReLU activation function. - The final fully connected layer has 1,000 neuron outputs, corresponding to the 1,000 recognition categories of the ImageNet dataset. The total number of parameters across the entire network reaches 60 million, primarily concentrated in the final three fully connected layers: . As we can see, the last three fully connected layers account for the vast majority of all parameters. The data dimensions of each layer throughout the network structure are summarized below: !image.png Implementation of AlexNet's Dual-GPU Architecture Additionally, in some reference materials, the architecture diagram of AlexNet appears as shown below: !image.png As can be seen, the above architecture diagram divides the entire network into identical upper and lower halves. The input image is fed into both upper and lower sub-networks simultaneously, and they finally merge at the output layer. This architecture diagram actually reflects the original implementation of AlexNet. Because the GPU configuration and processing power at that time were relatively low, training AlexNet on the ImageNet dataset using a single GPU would have been extremely slow. Therefore, Alex split the network into upper and lower parts, with the two sub-networks running in parallel on two graphics cards and merging their output results at the end. This approach significantly accelerated training and inference speeds. However, regardless of this, from the perspective of network layers and structural design essence, the dual sub-network structure described above is identical to the standard network structure of AlexNet. Summary of AlexNet's Key Features Activation Function In the network architecture design of AlexNet, the activation function was changed from Sigmoid (used in LeNet-5) to ReLU. This modification helps solve the vanishing gradient problem inherent in Sigmoid activation functions when neural networks become too deep or gradients are too small. At the same time, compared to activation functions like Sigmoid and Tanh, ReLU offers advantages such as faster convergence speed, higher training efficiency, and simpler computation. The following figure, demonstrated in the AlexNet paper on the CIFAR-10 dataset, highlights the massive advantage of ReLU over the Tanh activation function in terms of training convergence speed: !image.png Max Pooling Another distinct difference between AlexNet and LeNet-5 is the use of max pooling layers instead of the average pooling layers used in LeNet-5. The 3x3 max pooling layer with stride=2 adopted by AlexNet is better at retaining important features compared to LeNet-5's 2x2 average pooling. Furthermore, because (), the pooling operations overlap, allowing features to be scanned repeatedly and preventing important features from being discarded. Solutions to Overfitting The AlexNet paper mentions several countermeasures to address and optimize model training overfitting. Overfitting refers to a scenario where a model and its trained parameters perform exceptionally well on the training dataset, but poorly on new validation and test datasets. For example, in the implementation of the first two fully connected layers, a dropout=0.5 operation was introduced. During the training process, 50% of the neuron parameters in these two layers are randomly excluded from the forward inference and backward gradient update processes in each training step, thereby improving the generalization capability of the model and its parameters. Another example is Data Augmentation. To improve training efficiency, increasing the training dataset is an effective approach. Simply put, Data Augmentation generates new training data by applying various transformations to the existing training image set (i.e., the ImageNet dataset), rapidly increasing the number of training samples at a low cost. Data augmentation methods here include horizontal image flipping, Gaussian noise, and variations in image color and lighting (PCA). !image.png Another technique is random cropping of input images. The image sizes in the ImageNet dataset are not uniform. During AlexNet training, the shorter side of the input image is first resized to 256 pixels, and then the longer side is extended by 128 pixels from the center of the image to the left and right, with excess parts cropped out, resulting in a 256x256 resolution image as training input. As mentioned earlier, the input image resolution of the AlexNet network is 227x227. Therefore, based on this 256x256 resolution input, it is randomly cropped to a 227x227 resolution during each training iteration. This approach significantly multiplies the number of training images (by a factor of 2,048). !image.png Multi-GPU Training As described in the model structure section above, to accelerate model training and inference execution, the designers of AlexNet split the entire network into two identical sub-networks, utilizing two GTX 580 GPUs running simultaneously to speed up the training process. Because the GTX 580 had only 3GB of VRAM and could not handle heavy computations on its own, splitting the neural network into two independent sub-networks running on separate graphics cards greatly accelerated the training speed. PyTorch Implementation of AlexNet The code implementation of AlexNet based on the PyTorch framework is shown below: The network architecture exported via Netron.app is illustrated below: !image.png References - AlexNet and ImageNet: The Birth of Deep Learning | Pinecone - AlexNet Architecture Explained. The convolutional neural network (CNN)… | by Siddhesh Bangar | Medium - Understanding AlexNet | LearnOpenCV # - 卷積神經網絡 CNN 經典模型 — LeNet、AlexNet、VGG、NiN with Pytorch code | by 李謦伊 | 謦伊的閱讀筆記 | Medium
CNN Classical Network Model Architecture Learning: LeNet
Using LeNet for handwritten digit recognition as an example, this article analyzes the network structure and the overall architecture of a typical Convolutional Neural Network (CNN), establishing a foundational understanding of CNNs used for image recognition and processing. In the article Introduction to CNN Network Architecture Analysis, we briefly summarized the structure and components of a typical convolutional neural network. In this article, we will use a very classic network—LeNet-5, designed for handwritten digit recognition—to explain the structure and execution flow of CNNs in greater detail. Introduction to LeNet LeNet is one of the earliest published convolutional neural networks and also the one with the simplest architecture. It is exceptionally well-suited as an introductory network structure to build a foundational understanding of CNN architecture design and execution flow. The LeNet model was first proposed in 1989 by Yann LeCun, a researcher at Bell Labs, and was continuously optimized. Its design and application were intended for recognizing handwritten digits (i.e., training and recognition on the well-known MNIST dataset). LeNet was widely used in the United States postal service and automated teller machines (ATMs) to help recognize and process handwritten zip codes and numbers on checks. To this day, some ATMs are still running code written by Yann LeCun and his colleague Leon Bottou in the 1990s! !image.png Official LeNet website: MNIST Demos on Yann LeCun's website Explanation of LeNet's Network Architecture The diagram below illustrates the network architecture of LeNet: !image.png Overall, like all convolutional neural networks, the above network architecture consists of two main parts: - Feature Extraction Layer: Contains a combination of two convolutional layers + pooling layers (referred to as pooling layers in the diagram above). - Classification Layer: Contains three fully connected layers. The basic unit within each convolutional block is a convolutional layer, a sigmoid activation function, and an average pooling layer. The output of the final pooling layer in the feature extraction layer is a 16-channel 5x5 feature map. This feature map is flattened into a 1D vector of 16x5x5 elements, which is then fed into the fully connected layers to perform the classification task. The final fully connected layer has 10 neurons, corresponding to the recognition probabilities for the 10 digits from 0 to 9. Detailed information on the layer structures and parameter counts of the LeNet network is summarized as follows: - The input data is a grayscale image with a resolution of 28x28, which is essentially a 28x28 2D array with only a single grayscale channel. - Convolutional layer C1 contains six 5x5x1 convolution kernels with a padding of 2, so the output of this convolutional layer consists of six 28x28 resolution feature maps. The activation function for the convolution calculation results is Sigmoid. - The number of parameters in this layer consists of the weights and biases of the six 5x5x1 convolution kernels, which is 6 x (5 x 5 x 1 + 1). - Pooling layer S2 uses a 2x2 average pooling operation on the 6-channel 28x28 resolution feature maps from convolutional layer C1, outputting 6-channel 14x14 resolution pooled feature maps. - Convolutional layer C3 contains sixteen 5x5x6 volumetric convolution kernels with a padding of 0, so the output of convolutional layer C3 consists of sixteen 10x10 resolution feature maps. The activation function for the convolution calculation results is also Sigmoid. - The number of parameters in this layer consists of the weights and biases of the sixteen 5x5x6 convolution kernels, which is 16 x (5 x 5 x 6 + 1). - Pooling layer S4 still uses a 2x2 average pooling operation on the 16-channel 10x10 resolution feature maps from convolutional layer C3, outputting 16-channel 5x5 resolution pooled feature maps. - The input to the first fully connected layer F5 is the 1D flattened vector of the feature maps output by pooling layer S4. Thus, the input data is a 1D vector of 5x5x16. F5 has 120 neurons, and the activation function is still Sigmoid. - The number of parameters in this layer consists of the input weights and biases for the 120 neurons, which is 120 x (5 x 5 x 16 + 1). - The input to the second fully connected layer F6 comes from the outputs of the 120 neurons in F5. F6 has 84 neurons, and the activation function is Sigmoid. - The number of parameters in this layer consists of the input weights and biases for the 84 neurons, which is 84 x (120 + 1). - The final output layer has 10 neurons, where the output of each neuron corresponds to the recognition probability of a digit from 0 to 9. - The number of parameters in this layer consists ofynn the input weights and biases for the 10 neurons, which is 10 x (84 + 1). Below is a summary of the dimensions of the output vectors at each layer of LeNet during the forward computation process: > Regarding the network structure of LeNet above, note the following: > - In the network structure diagram above, there are a total of 7 layers: two convolutional layers, two pooling layers, and three fully connected layers. However, LeNet in the general sense typically refers to LeNet-5, where the number 5 represents the number of layers containing parameters in the entire network (i.e., two convolutional layers + three fully connected layers). > - In practical CNN implementations, using the ReLU activation function for convolutional layers and max pooling for pooling layers yields better performance. However, neither ReLU nor max pooling existed in the 1980s and 1990s. Therefore, in the original LeNet-5 architecture, the classic Sigmoid activation function was used for convolutional layers, and average pooling was used for pooling layers. PyTorch Implementation of LeNet Based on the network architecture of the LeNet-5 model introduced above, the implementation code for this network under the PyTorch framework is as follows: After implementing the above network and exporting it as an ONNX model file, the network architecture diagram rendered on netron.app appears as follows: !image.png References - 6.6. Convolutional Neural Networks (LeNet) — Dive into Deep Learning 2.0.0 documentation - MNIST Demos on Yann LeCun's website
A Comprehensive Guide to Custom Training, Testing, and Model Export for the YOLOv5 Model
This article provides a detailed, end-to-end summary of preprocessing user-defined training datasets and annotations into the YOLO format, followed by training, testing, and ONNX exporting of the YOLOv5 model. Downloading and Testing the YOLOv5 Model To perform custom training and deployment with the YOLOv5 model, the first step is, of course, to download the YOLOv5 source code. Search for YOLOv5 on GitHub to find the Ultralytics source code repository: ultralytics/yolov5: YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite. As you can see, although YOLOv5 is a relatively early model, its development and maintenance have never stopped and it continues to receive updates to this day: !image.png Use git clone or directly download the latest version of the code locally via HTTPS. Once downloaded, you can run some initial tests using the scripts included with a default pre-trained model. For basic testing and usage of the YOLOv5 model, three primary script files are mainly used: train.py (training), detect.py (inference on new sample data), and export.py (model exporting). Main arguments for train.py: - --weights: Path to the initial weights file, defaults to yolov5s.pt (the official parameter file pre-trained on the COCO dataset, which is automatically downloaded upon first use), used to load the pre-trained model. - --cfg: Path to the model configuration file (model.yaml), which defines the network architecture. Default model configuration files are located in the models directory. - --data: Path to the training dataset configuration file. Default dataset configuration files are located in the data directory. - --batch-size: Number of samples per mini-batch during training, typically 8 or 16. - --epochs: Number of training epochs. - --device: Specifies the CUDA device, such as 0, 0,1,2,3, or cpu. Based on the settings of the --weights and --cfg parameters above, model training can be categorized into the following scenarios: - Training based on official YOLO pre-trained model parameters: Simply pass the .pt file of the pre-trained model via the --weights parameter, which already contains the model configuration information. - Training a YOLO model from scratch: Leave the --weights parameter as an empty string and provide the --cfg parameter to set up the model configuration. - If both --cfg and --weights parameters are provided in the command, the model structure defined by --cfg will take precedence, and it will attempt to load the weight parameters from --weights (if the structures match). Main arguments for detect.py: - --weights: Specifies the path to the model weights file, defaulting to yolov5s.pt. - --source: Input source path for inference data, which can be a single image, video file, directory, URL (rtsp, rtmp, etc.), webcam index (0), screen, etc. - --device: Execution device, can be specified as cpu or a GPU index (e.g., 0, 1), defaulting to automatic selection. - --classes: Specifies a list of class indices to detect, such as --classes 0 2 3, defaulting to None (detect all classes). - --project: Project directory to save inference results, defaulting to runs/detect. - --name: Experiment name (used to create subdirectories), defaulting to exp. - --conf-thres: Confidence threshold. Results are output and marked only if the recognition confidence is greater than this threshold. Main arguments for export.py: - --data: Path to the dataset configuration YAML file used during model training, defaulting to data/coco128.yaml, used to provide metadata such as class information. - --weights: Path to the model weights file to be exported. - --batch-size: Batch size, defaulting to 1. - --include: Specifies the list of model formats to export, defaulting to torchscript. Supported formats include: - torchscript: PyTorch's TorchScript format - onnx: ONNX format - openvino: Intel OpenVINO format - engine: NVIDIA TensorRT format - coreml: Apple CoreML format - saved_model: TensorFlow SavedModel format - pb: TensorFlow GraphDef format - tflite: TensorFlow Lite format - edgetpu: TensorFlow Edge TPU format - tfjs: TensorFlow.js format - paddle: PaddlePaddle format The data subdirectory within the YOLOv5 source code directory contains configuration files for several standard image datasets. You can first test the YOLOv5 model training pipeline using these standard datasets and their configuration files (the coco128 dataset is used here): > Every time a training session is executed using the above scripts, an exp directory is created under runs/train, containing various intermediate files generated during training. Once training is complete, the generated weight parameters file will be weights/best.pt within this directory. Similarly, you can use the following command to run inference tests with detect.py on the sample images included in the YOLOv5 source code: The above script performs inference on all images under data/images, and the generated annotated images are saved in the runs/detect/exp directory. !image.png Preparing the Training Dataset We have previously conducted a quick test on the basic usage of the YOLO model. Next comes preparing a custom dataset for model training. This article uses face mask detection as a learning case. Since there are a large number of such public datasets available online, you can start by searching for public datasets that meet your training requirements. - Websites like Kaggle, Roboflow, and Hugging Face offer a massive collection of public datasets, which you can search according to your needs. This article uses the Face Mask Detection dataset from Kaggle to train the YOLOv5 model: Face Mask Detection. This dataset detects whether people in images are wearing masks. The entire dataset contains a total of 853 images with three detection categories: without mask, with mask, and mask worn incorrectly. The annotation format is Pascal VOC. !image.png After downloading the Face Mask Detection dataset from Kaggle, you first need to preprocess its images and annotation information into the format required by the YOLO model. For the Face Mask Detection dataset, the required preprocessing steps include: - Converting the Pascal VOC annotation format used by the dataset into the YOLO format. - Splitting the images and annotation sets into training, validation, and test sets with an 85:10:5 ratio. - Saving them into three subdirectories (train, valid, and test) according to the directory structure required for YOLO model training. - Preparing the dataset configuration description file FaceMask.yaml for YOLO training. Pascal VOC is a commonly used annotation format for object detection tasks, which differs significantly from the annotation format required for YOLO model training. Therefore, I specifically wrote a script named voc_to_yolo.py to implement steps 1-3 of the above preprocessing pipeline: annotation format conversion, data splitting, and YOLO training directory structure setup. Using this script allows you to convert from a Pascal VOC dataset to a YOLO training dataset: The diagram below illustrates the directory structure of the Face Mask dataset before and after conversion. - In the original raw dataset before conversion, all images are located in the images directory, and all Pascal VOC formatted annotations are located in the annotations directory. - The complete YOLO dataset after conversion is located in the yolo directory, which contains three subdirectories (test, train, and valid) corresponding to the test, train, and validation datasets respectively, each containing subdirectories for image files and YOLO annotation files. !image.png Based on the prepared YOLO-formatted training dataset, compile the corresponding dataset description configuration file FaceMask.yaml as follows. This file will be used subsequently during training to specify the training dataset path, class names, and the number of detection categories. Model Training and Testing Train the model on the YOLOv5 pre-trained weights using the preprocessed dataset above: Training on a CPU takes a relatively long time. With less than 1,000 images, it took over a day to train on my ThinkPad X260 laptop. Once training is complete, the generated model parameter file is best.pt located in the runs\train\exp9\weights directory. Test the test dataset split in the previous phase using the model parameter file generated from the training above: The above script will perform inference on all images in the test dataset specified by datasets\FaceMask\yolo\test\images\. The inference results will be saved in a newly created exp directory under runs/detect, with recognition results having a confidence greater than 0.5 boxed and labeled to indicate whether masks are worn correctly. !image.png Exporting the Model to ONNX Next, you can use export.py in the YOLOv5 source code directory to export the best.pt parameter weights file trained in the previous step into a standard ONNX model file for deployment in standardized environments such as ONNX Runtime: Once the above script executes, a best.onnx file will be generated in the same directory as best.pt. This is the ONNX file exported from the model. References - ultralytics/yolov5: YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite - Face Mask Detection
A Detailed Summary of Gamma Correction in the Image ISP Processing Workflow
This article provides a detailed summary of the Gamma correction module in the image ISP processing workflow, explaining the reasons for including Gamma correction in ISP pipelines, as well as the core concepts and workflows used by mainstream ISPs for Gamma correction. The Significance of Gamma Correction In the ISP processing pipeline for RAW images output by a sensor, the Gamma correction module is a crucial image brightness non-linear transformation module. The primary purpose of the Gamma module is to bridge the gap between the linearity of the image sensor's photosensitive response and the non-linearity of human visual perception and display devices. Specifically, during the exposure process, an image sensor's response to light intensity and exposure duration is largely linear. That is, if the light intensity doubles, the electrical signal output by the image sensor also doubles. The non-linear relationship between the output voltage of a display device and its perceived brightness originates from the legacy CRT monitors. For CRT monitors, the display brightness and input voltage follow a power-law relationship of . If the linear brightness signal captured by the image sensor were fed directly into a CRT monitor for display, the resulting image would suffer from severe loss of shadow details, an overall dark appearance, compressed mid-to-high contrast, and diminished highlight gradation. Therefore, to display the linear brightness images captured by the sensor linearly on a CRT monitor, a Gamma Encode operation must be performed beforehand. This involves applying a Gamma mapping of to the sensor's linear brightness image before sending it to the CRT monitor, allowing the CRT monitor to restore the linear brightness. The overall workflow is illustrated in the figure below: !image.png CRT monitors have long been obsolete, and current mainstream LCD/OLED displays actually feature a nearly linear brightness response. Theoretically, this would mean that raw brightness images captured by image sensors could be displayed directly on liquid crystal displays, eliminating the cumbersome Gamma mapping process described above. However, the problem is that doing so would break compatibility with the massive archive of legacy image files accumulated during the CRT era (which had Gamma Encode operations applied prior to image compression). Consequently, to maintain compatibility with historical content accumulated from the CRT era, modern LCD displays incorporate an internal correction by default. This preserves the closed-loop workflow of linear image -> encoding gamma (0.45) processing -> display gamma (2.2) -> linear image. Human visual perception of light brightness is similarly non-linear, following the Weber-Fechner law. Specifically, the human eye is much more sensitive to dark areas of illumination; thus, under low-light conditions, the human eye can distinguish extremely minute changes in brightness. Conversely, it is much less sensitive to bright areas, meaning that changes in high brightness are often difficult for the human eye to perceive. !image.png Gamma Correction Workflow Based on the Gamma Encode processing performed on the ISP side and the automatic Gamma Decode processing performed on the display side as described above, the gamma encode executed within the ISP pipeline must process the brightness of the RGB three channels as follows: In actual ISP engineering implementations, calculating power functions directly for every pixel is computationally expensive. Therefore, practical engineering applications utilize pre-calculated Look-Up Tables (LUTs) to achieve real-time power mapping for Gamma operations. The input data to the Gamma correction module consists of RGB-domain data processed through Demosaic and CCM. Each pixel contains three channels (RGB). To ensure color balance across the three channels, a unified Gamma mapping table is generally applied to transform all three channels simultaneously. - The processing method mentioned above, where three color channels share the same set of LUTs, is also known as a 1D LUT, which is essentially a Gamma brightness transformation. In some ISPs or image processing workflows, there is also the concept of a 3D LUT, where the RGB channels use independent LUT tables, typically used to implement personalized color style features. To handle complex lighting scenarios (such as backlighting or night scenes) and better adapt to the non-linear characteristics of human visual perception, mainstream ISPs with high image quality requirements often adopt segmented programmable LUT tables to achieve finer-grained control. For instance, a segmented Gamma LUT divides the input brightness range (e.g., 0~255) into three luminance intervals: shadows (0~20%), midtones (20%~70%), and highlights (70%~100%), and then independently configures different Gamma mapping functions for each interval: - Shadows: Uses a low value with a gentle slope to stretch shadow details. - Midtones: Uses a standard value to maintain natural contrast. - Highlights: Uses a high value or logarithmic compression with a steep slope to suppress overexposure. !image.png This segmented LUT design adapts exceptionally well to human visual brightness response, stretching the luminance response in shadows to preserve more detail while compressing the response range in bright areas to suppress overexposure. References - ISP Algorithm — Gamma Correction and Degamma Correction - [Fundamentals] A Comprehensive Guide to Understanding Gamma Correction