All Articles
Detailed Explanation of YOLO Model Configuration Parameters
This article provides a detailed summary and explanation of all parameters contained in the default.yaml configuration file in the ultralytics/YOLO project (excluding visualization and export-related parameters). The path to the configuration parameter file in the ultralytics/YOLO project is ultralytics/cfg/default.yaml. The default.yaml file is the core configuration file of the Ultralytics project. Understanding the meanings of its parameters is crucial for successfully training custom datasets, as well as performing inference, exporting, and deployment. task and mode At the beginning of the default.yaml configuration file, the task and mode parameters are used to set the basic running mode of the model. These two parameters jointly determine how the YOLO model operates. Different tasks load different model architectures and loss functions, while different modes execute corresponding workflows (training, validation, inference, etc.). The task parameter is used to specify the model's task type. Once the task parameter is designated, the project will build the network structure based on the specified task type. The current implementation of the Ultralytics project includes five task types: - detect: Object detection task, used for detecting and localizing target objects in images. detect is the most commonly used task type in this project and is also the default task in default.yaml. - segment: Instance segmentation task, which not only detects object locations but also provides pixel-level segmentation masks. - classify: Image classification task, which classifies the entire image only. - pose: Pose estimation task, used to recognize keypoints/joint positions of humans or animals, providing stick-figure-like recognition and detection results. - obb: Oriented bounding box detection task, used to detect non-axis-aligned targets (such as objects from a drone's perspective). For more details on these different types of tasks, please refer to the article Summary of Computer Vision AI Application Fields and Their Mainstream Models. The mode parameter is used to specify the current running mode: - train: Training mode, used to execute the model training process. - val: Validation mode, used to evaluate the performance of the currently trained model on the validation set and generate validation metric files. - predict: Prediction/inference mode, used to perform inference on designated data using a trained model. - export: Export mode, used to export the model into various deployment formats (such as ONNX, TensorRT, etc.). - track: Enables tracking mode on the detect model, which can assign a unique ID to each detected object in a video and continuously track its motion trajectory. When enabled, the tracking configuration file must be specified via the tracker parameter. - benchmark: Benchmark mode, used to more comprehensively evaluate the model's performance, speed, and accuracy across different hardware and configurations. Training-Related Parameters The parameters related to the model training process in the default.yaml file and their explanations are shown in the table below: !image.png Below is a more detailed explanation and description of some of the parameters above. The save (default True) and save_period (default -1) parameters are used to specify whether to save the trained weight files to the save_dir/weights/ directory. When save is set to true, the current training weights are overwritten and saved in the last.pt file at the end of each epoch. If the current training performance surpasses previous metrics, the weight file is also saved as best.pt. When save_period is set to N (where N is not -1), an intermediate training weight file named epoch_N.pt is additionally saved in the save_dir/weights/ directory every N epochs. The cache parameter determines whether and how the dataset is preloaded and cached at the beginning of training, avoiding time-consuming disk reads and data processing operations repeatedly during training. Setting it to False (the default value) means no caching, loading a new batch of training data from the disk every time; setting it to True or 'ram' means caching the training set data in RAM in advance, fetching data from RAM each time; setting it to 'disk' means that during the first epoch, the training program saves the processed images in a special format (.npy) to a temporary directory on the hard drive, and subsequent epochs read from this cache. The project and name parameters can be combined to customize the save path for model training files. If these two parameters are not set, the files generated by the default training process are saved in a directory like runs/detect/train/exp. Once set, the save path for these files becomes project/name. The exist_ok option determines whether multiple executions should overwrite this project/name or append an execution index suffix to the directory name specified by name to distinguish between multiple execution results. When the pretrained parameter is set to False, model training uses only the model specified by the model parameter, and the weights of each layer are initialized with completely random parameters; when pretrained is set to True, it directly uses the weights in the model file specified by the model parameter; when pretrained is set to the path of a specific weight file, it uses the model specified by the model parameter, but initializes and sets each layer's parameters using the parameter file specified by pretrained. The seed and deterministic parameters are used to control the randomness and reproducibility of the model training process and results. Under the default configuration (seed=0, deterministic=False), the training process and results retain a certain degree of randomness, which is suitable for daily development scenarios. For applications requiring stricter rigor (such as academic paper publication), model training must provide strict determinism and reproducibility. In this case, these parameters should be set to seed=42, deterministic=True. The single_cls parameter is a boolean value with a default of True. When set to True, the model's output forces all objects to be treated as a single class, meaning the model outputs only one class. If your application only cares about whether an object of a certain category exists in the frame, you can set this parameter to True to simplify the complex multi-class detection problem into a more basic object presence detection problem. The rect parameter is used to specify whether to use a fixed square aspect ratio as input for model training images. When rect=False (default value), all training images, regardless of their aspect ratio, are first converted to squares via scaling + padding before being fed into the model for training. This mode is generally targeted at datasets where aspect ratios are inconsistent and vary significantly. When rect=True, YOLO first counts the aspect ratio information of each batch of images to find the most suitable aspect ratio, and then during training, converts all images in this batch to the new aspect ratio before feeding them into the model. For datasets where the aspect ratios of all images are basically the same (such as surveillance cameras, which are typically 4:3 or 16:9), this approach can improve training efficiency. The main purpose of setting the rect parameter is to adjust the input image size according to the dataset's aspect ratio, minimizing GPU computation waste and improving training efficiency and speed. The close_mosaic parameter is used to specify disabling the mosaic data augmentation function for the last N epochs of the training process. Mosaic data augmentation cuts and combines four training images in a random manner per batch to form a new training image for parameter training. This technique enhances training results through the diversity of training data. However, in the late stages of training, we generally hope that parameters stabilize as quickly as possible, whereas mosaic augmentation causes parameters to fluctuate too violently. Therefore, by setting the close_mosaic parameter, you can disable mosaic augmentation N rounds early to ensure parameters stabilize quickly during the final stage of training. The amp parameter is used to control the data types and computational precision used during training. In traditional model training, FP32 precision is generally used throughout the entire workflow. When AMP (Automatic Mixed Precision) mode is enabled, memory-saving FP16 precision is used for forward and backward passes, while higher FP32 precision is reserved only for gradient-related calculations. The advantage of this is that it can drastically reduce memory requirements and accelerate training speed while keeping training metrics basically unchanged. Enabling AMP mode is generally recommended by default, but note that some older hardware does not support AMP. The fraction parameter is used to specify the proportion of training data participating in training relative to the total dataset size. Normally, model training involves feeding 100% of the training set into the model. However, in scenarios requiring rapid validation with very large datasets, the fraction parameter can be set to train the model using only a specific proportion of the training data. In this case, each epoch will randomly extract a fraction of data as specified by the parameter for training. The Profile parameter is used to control whether to enable performance metrics profiling during training. When enabled (Profile=true), the training process performs fine-grained statistics and analysis on every step, ultimately outputting extremely detailed performance analysis data. Consequently, this process requires more memory to store these statistical information and will significantly slow down training speed. Therefore, unless you are adjusting the model framework and need to understand execution efficiency at a very granular level, enabling profile mode is not recommended. The freeze parameter is used to specify which layers' weights should be frozen during training, meaning their parameters will not be updated. When fine-tuning a model on your own dataset using an officially pretrained YOLO model, official parameters have already undergone thorough training on large datasets like COCO and contain extremely powerful feature extraction capabilities. If your dataset is relatively small (within a few thousand images) and the detection task is similar, it is recommended to use the freeze parameter to freeze the backbone network, training only the neck and detection head on your own data. You can control the freezing of the first N layers or freeze specific layer parameters using a list. The multi_scale parameter is used to control the input image size during training. If multi_scale is set to False, the input image size during training is fixed to the imgsz setting of 640x640. When multi_scale is set to True, an image size is randomly chosen for each batch (must be a multiple of 32), and images in that batch are resized to this new dimension before being fed into the model. Since input image sizes vary across batches, this randomly adjusts training data diversity, thereby enhancing the model's adaptability and generalization capabilities. The compile parameter is a new feature introduced in YOLOv8 based on PyTorch 2.0's torch.compile. When enabled, it optimizes the model's computational graph code during training to accelerate code execution efficiency and reduce training time. If your training dataset is relatively large and your hardware specifications are modern, you can enable this option to accelerate the training process. Image Segmentation and Classification Parameters The following three parameters (overlap_mask, mask_ratio, and dropout) are only valid for image segmentation and image classification tasks. Validation and Testing Parameters The table below lists the configuration parameters, default values, and brief descriptions for the Val/Test settings section in the default.yaml configuration file. > Note: Although some parameters in the table below are categorized under the Val/Test settings section of the configuration file, they are also used during the predict process—for example, NMS algorithm-related parameters such as iou, max_det, and conf. !image.png The split parameter is used to specify the dataset to be used when validation is enabled (i.e., val parameter is set to True). Its values range over [val, test, train], representing evaluation using the validation, test, or training set respectively. In the vast majority of cases, it should be set to val to avoid test set data leakage and ensure fair evaluation. The conf, iou, and max_det parameters are filtering parameters for the NMS (Non-Maximum Suppression) algorithm. conf is the object detection confidence threshold, iou is used to filter overlapping boxes, and max_det limits the maximum number of detection results output per image. The half parameter specifies whether FP32 or FP16 precision is used during model forward inference. The dnn parameter specifies whether OpenCV's DNN module or the PyTorch framework is used for model forward inference. Prediction Parameters The table below lists the configuration parameters, default values, and brief descriptions for the Predict settings section in the default.yaml configuration file. !image.png The visualize parameter controls whether to generate visual statistical analysis results, intermediate layer feature maps of image recognition, heatmaps, and other information during validation and inference execution. Viewing these results can assist in debugging and locating issues within the model. When set to True, because these extra files must be generated and written to disk, it will significantly increase memory requirements and affect inference speed. The augment parameter controls whether to perform augmentation operations on input images during inference. When augment=False, the model simply takes the original image and performs a single inference pass to yield results. If set to True, the model applies various augmentation operations (such as flipping, scaling) to the input image, performs multiple inference passes, and finally merges the results into the final output. Consequently, prediction accuracy improves, but processing time increases substantially. The agnostic_nms parameter is a parameter within the NMS algorithm execution flow, determining whether different classes should be distinguished during the duplicate box filtering process. In standard NMS where agnostic_nms=False, the NMS algorithm distinguishes between classes, and detection boxes of different classes do not affect each other. If set to True, NMS processes all classes together during filtering. In this case, two objects of different categories located close to each other might risk being mistakenly suppressed. The classes parameter specifies a list of target category indices to be detected during inference. When this parameter is left unset, it means all target categories supported by the model will be detected and outputted; when set to a specific list, it indicates that during NMS filtering, only target categories in this list are prioritized, and all other targets are filtered out. Hyperparameters The learning rate and optimizer parameters, default values, and explanations during model training are shown below: !image.png The lr0 and lrf parameters are learning rate parameters that adjust parameters during backward passes in model training. The learning rate during training dynamically adjusts according to the chosen optimizer, where lr0 is the initial learning rate at the start of training. Afterward, the learning rate gradually decreases, eventually dropping to lr0 * lrf. The momentum parameter is another optimizer algorithm technique for adjusting model parameters. After introducing the momentum mechanism, each parameter update relies not only on the currently calculated gradient and learning rate but also on historical gradients. The momentum parameter specifies the weighting coefficient of historical gradients in the current parameter adjustment. The higher the value of momentum, the more significant the impact of historical gradients on parameter updates. The weight_decay parameter is used to control the magnitude of weight parameters, preventing overfitting caused by excessively large weights. Specifically, during training, weight_decay and the model's overall weight information are added to the loss calculated by the loss function and the parameter adjustment magnitude, intentionally suppressing parameter sizes if weights become too large. When model training starts, some internal parameters are in a randomly initialized state. Using the maximum learning rate lr0 directly to adjust parameters at this stage can easily cause parameter adjustments to be too large, leading to instability. Therefore, the warmup mechanism of the YOLO model means that during the first few epochs of training, the learning rate is linearly increased from 0 to the value set by lr0. Starting with a very low learning rate and gradually increasing it to lr0 helps model parameters quickly reach a relatively stable state before formal training begins. The warmup_epochs, warmup_momentum, and warmup_bias_lr parameters are all related configuration parameters for the warmup execution flow. Among them, warmup_epochs specifies the number of epochs the mechanism acts for. During the warmup stage, momentum increases linearly from warmup_momentum to the momentum setting, the learning rate (non-bias terms) increases from 0 to lr0, and the bias learning rate increases from 0 to warmup_bias_lr. Once the warmup stage completes, formal training officially commences starting uniformly from the learning rate lr0. The parameters, default values, and explanations related to loss functions are shown below: !image.png The box, cls, and dfl parameters correspond respectively to the loss function weights of the three components in the YOLO model. cls corresponds to the loss value for classification accuracy, while box and dfl together correspond to the loss value for predicted box location accuracy. box primarily evaluates the overlap between predicted and ground-truth boxes from an IoU perspective, whereas dfl evaluates localization accuracy from the perspective of top, bottom, left, and right coordinate offsets. Appropriately tuning box, cls, and dfl parameters allows adjusting the proportion of different loss types in the overall loss, thereby tuning the metrics corresponding to different losses. The nbs parameter stands for nominal batch size. During actual training, due to hardware limitations, the batch size often cannot be set very large and is typically set to 16. When the batch size is relatively small, the model often suffers from high fluctuations and difficulty in convergence during training. An ideal configuration is a batch size like 64. To address hardware-limited batch sizes, the actual image batch fed into the model follows the actual configured batch size (e.g., 16), and the resulting loss values are scaled according to the ratio of nominal batch size to actual batch size. When nbs is set to 64, the difference is 4x, so the loss value is multiplied by 4 before backpropagation for parameter updates. This method solves model training stability issues caused by insufficient actual batch sizes. The parameters, default values, and explanations related to training data color and geometric augmentation are shown below: !image.png The hsv_h, hsv_s, and hsv_v parameters correspond to color augmentation parameters for input training images. The HSV color space contains three dimensions: Hue (H), Saturation (S), and Value (V). The three parameters above define, as scale factors, the random adjustment range applied to the training images' hue, saturation, and brightness relative to the original image during training. Through this dynamic random adjustment across the three dimensions of the HSV space, the model's generalization capability under various lighting and color environments is enhanced. The degree parameter specifies the magnitude of random rotation applied to training set images based on their original angles before being fed into the model. The translate parameter specifies the ratio by which training images are randomly translated horizontally and vertically relative to the original image. The scale parameter specifies the random scaling range applied to the original training images before training. The shear and perspective parameters control geometric distortion for image augmentation, designed to handle motion scenarios such as autonomous driving and drones. shear refers to a stretching action that distorts the original rectangular image into a parallelogram, while perspective refers to a pitching-perspective distortion where the original rectangular image gradually shrinks from top to bottom or bottom to top. The values of these two parameters specify the magnitude of their respective distortions. The parameter names, default values, and explanations for the image transformation augmentation section are shown below: !image.png The above three parameters are relatively straightforward. flipud indicates the probability that training images are flipped vertically before being fed into the model, fliplr indicates the proportion of horizontal flips, and bgr indicates the proportion of training images whose channel order is converted from RGB to BGR prior to training. All three parameters are used to enhance image diversity and model generalization based on existing images. The parameter names, default values, and explanations for the advanced mixed augmentation section are shown below: !image.png The mosaic parameter controls the proportion of the training process that applies mosaic augmentation. Mosaic augmentation scales, crops, and stitches four training images in a 2x2 layout into a single image to be fed into the model for training. Setting this parameter to 1.0 means 100% of the training data uses mosaic-stitched images, 0 means mosaic augmentation is disabled, and a value between 0 and 1 represents the proportion of training data using mosaic augmentation. The mixup parameter controls the proportion of training images that undergo mixup augmentation prior to training. Mixup multiplies the pixel values of two images by randomly sized proportions and overlays their pixels to form a single image (with label confidences processed in the same proportion) before feeding it into the model. The cutmix parameter controls the proportion of training images that undergo cutmix augmentation prior to training. Cutmix randomly crops a rectangular region from one image and pastes it onto another image to form a new training image. The copy_paste and copy_paste_mode parameters are used exclusively for image segmentation models. The auto_augment and erasing parameters are used exclusively for image classification models. Configuration File-Related Parameters If the cfg parameter is set via the command line or here, the parameters in the YAML-format cfg configuration file configured here will be merged into default.yaml when executing various YOLO model tasks. Parameters with the same name will prioritize the values passed in the cfg configuration file. Additionally, as explained earlier for the track mode option under mode, track mode is used to enable tracking on the detect model. In this case, you must specify the tracking configuration file required for running tracking mode using the tracker parameter here: The directory where the configuration file corresponding to the tracker parameter is located is ultralytics/cfg/trackers. The current version includes two tracking modes: botsort and bytetrack. The former is the default tracking mode with higher tracking accuracy; the latter is primarily aimed at high real-time tracking scenarios, offering faster tracking computation speed.
Interpretation of the YOLOv8 Model Network Architecture
This article details and summarizes the similarities and differences in network architecture design between the YOLOv8 and YOLOv5 models by comparing the two, thereby providing a comprehensive and in-depth understanding of the YOLOv8 model structure. Below is the overall architecture diagram of the YOLOv5 model: !image.png Below is the overall network architecture diagram of the YOLOv8 model: !image.png Backbone Network Comparing the Backbone networks on the left side of the YOLOv5 and YOLOv8 architecture diagrams above, it can be seen that the overall structures of the backbone networks of these two models are very similar. The main differences lie in the following two points: - At the P1 layer: YOLOv5 uses either the FOCUS module (older versions) or a 6x6 convolutional layer with a stride of 2 (newer versions) to perform initial downsampling of the input image from 640x640x3 to a 320x320x64 feature map. In contrast, the YOLOv8 model uses a 3x3 convolutional layer with a stride of 2—the same as other dimension-reduction modules in the backbone—to reduce the resolution, taking the same 320x320x64 feature map as input. - The YOLOv8 model replaces the CSPLayer module (i.e., the C3 module) of the YOLOv5 model with the C2f module, where the C2f module is one of the core innovations of YOLOv8. C2f Module As mentioned above, the C2f module in the YOLOv8 backbone network is one of the greatest innovations in its architectural design. First, let's compare the structural differences between the C3 module of YOLOv5 and the C2f module of YOLOv8: !image.png As can be seen, the C3 module adopts a dual-branch parallel structure, splitting the input feature map into two parts processed respectively by the main branch and the shortcut branch: - Main branch: After dimensionality reduction via a 1x1 convolution, features are extracted through multiple Bottleneck residual modules (the number of residual modules can be set via the depth_multiple parameter). - Shortcut branch: Processed directly through a 1x1 convolution to preserve the original feature information. - Final feature fusion: The outputs of the two branches are concatenated directly along the channel dimension and then fused via a 1x1 convolution before outputting. In contrast, the C2f module adopts a multi-stage serial structure, and its design appears somewhat more complex: - Initial processing: The input feature map uniformly passes through a 1x1 convolution to generate intermediate feature maps. - Feature splitting: The intermediate feature map is split into two parts; one part is processed sequentially through multiple Bottleneck modules, while the other part is passed directly to the end. - Multi-stage processing: For a series of subsequent Bottleneck residual modules, the output of each residual module is split into two parts: one part continues to be processed in downstream residual modules, while the other part is passed directly to the end for concatenation. - Multi-scale fusion: Finally, the feature maps from all processing stages are concatenated together and fused via a 1x1 convolution. Comparing the structures of the two, the biggest difference of C2f compared to C3 is that the C2f module concatenates the multi-stage feature maps output by its internal residual modules, thereby fusing feature information of different depths—including intermediate processing stages—resulting in richer features. Furthermore, by introducing more cross-layer connections to form a richer gradient flow design, gradient information can be backpropagated more effectively through the network during training, which alleviates the vanishing gradient problem in deep networks and improves training stability and convergence speed. However, whether it is the C3 module or the C2f module, the dimensions of their input and output feature map information are identical. Similar to the YOLOv5 model, the YOLOv8 module also outputs three streams of feature information with different resolutions to the neck network at 8x, 16x, and 32x scales respectively: 80x80x256, 40x40x512, and 20x20x512 (the default minimum-resolution feature output dimension for YOLOv5 is 20x20x1024). Neck Network There are no significant differences between the two in terms of the neck network. YOLOv8 introduces some simplifications based on the newer version of YOLOv5, but the overall structure is basically consistent: both adopt the PANet structure, which enhances the information flow among the three multi-scale features output by the backbone network through top-down and bottom-up path aggregation, ultimately outputting fully fused feature information to the detection head. The improvements of YOLOv8 over the YOLOv5 neck network mainly include: - Simplifying the network structure by removing two convolutional blocks in the UpSample path. - Just like the backbone network, the C3 modules in the neck network are also replaced by C2f modules. After performing various fusion processes on the three scales of feature information in the neck network, the YOLOv8 model ultimately outputs three streams of multi-scale feature information from the neck network to the detection head: 80x80x256, 40x40x512, and 20x20x512. Detection Head Compared to the preceding backbone and neck networks, the detection head of the YOLOv8 model has undergone revolutionary improvements compared to YOLOv5, mainly reflected in two aspects: the Decoupled Head and the Anchor-Free mechanism. In the detection head design of the YOLOv5 network, a Coupled Head structure is adopted, meaning that the classification task for category determination and the regression task for predicted box coordinates share the same set of convolutional layers, eventually predicting categories and bounding box coordinates through separate output channels at the output stage. In contrast, the YOLOv8 detection head adopts a Decoupled Head design concept, completely separating the classification and regression tasks into two independent network branches. The classification branch is dedicated to target category determination and processes features using independent convolutional layers; the regression branch focuses on bounding box location prediction, outputting coordinate information through specialized convolutional layers, thereby avoiding mutual interference between the two task objectives during training. A comparison of the two is shown below: !image.png As can be seen from the figure above, in the design of the YOLOv8 decoupled detection head, the classification branch and regression branch each contain two independent 3x3 convolutional modules with a stride of 1, followed by dimensionality reduction via a 1x1 convolution according to their respective output information dimensions. For the COCO dataset, the final output data dimension of the model's inference on a single image at the detection head is [1, 8400, 144]. Here, 1 represents the batch size; since a single image is being processed, the batch size is 1. The second value, 8400, represents the sum of all grid points across the three different scales of feature information sent from the neck network to the detection head (). Every grid point on the feature maps outputs a 1D tensor representing the detection results at that grid point. The final 144 can be divided into two parts: the length of the category determination part is 80, corresponding to the 80 categories of the COCO dataset—the higher the class score, the more likely the detected object at that grid point belongs to that category; the length of the bounding box location detection part is 64, which is related to the Anchor-Free mechanism of the YOLOv8 detection head. Starting from the YOLOv2 model, the first few generations of YOLO all used Anchor-based prediction box mechanisms. YOLOv8, however, adopts an Anchor-Free design and no longer relies on aspect ratios clustered from datasets to assist in locating objects. The implementation of the YOLOv8 Anchor-Free mode relies on the DFL (Distribution Focal Loss) mechanism. The so-called DFL mechanism means that each grid output by the prediction box branch path contains the coordinate positions of the object prediction box detected by this grid. Based on the position coordinates of the current grid's center point in the original image resolution (), the target coordinate positions are calculated using four offsets: (left), (top), (right), and (bottom). The top-left corner of the final predicted box coordinate position is , and the bottom-right corner is . Each of the four offsets is represented by a 16-dimensional vector, making the four offsets a 64-dimensional vector, which is the output data for each grid in the prediction box branch path. For example, for the feature output of the P3 layer (), given an anchor grid index of , the coordinates of the center point of this anchor grid in the original image are . Similarly, for the P4 and P5 feature outputs, the stride must be changed to 16 and 32 respectively to calculate the coordinates of their anchor grids in the original image. These anchor coordinate positions plus the four offsets yield the positions of the top-left and bottom-right corners of a detection box in the original image. So, how are the 16-dimensional vectors used to calculate these four offsets ? This involves two parameters: - reg_max: Usually set to 16, meaning the distance range of the offset from 0 to the maximum distance is evenly divided into 16 intervals. - stride: The stride of the feature layer (). Taking the P3 layer as an example to explain the offset calculation logic, its , and the maximum supported offset is pixels. These 128 pixels are divided into 16 segments at intervals of 8 pixels: !image.png The 16-dimensional vector output by the prediction box regression branch for each offset represents the probability that the actual offset falls into each of these 16 segments. The final calculated actual offset is the weighted average of these 16 probabilities and the center points of the 16 segments (rather than simply taking the center point of the segment with the maximum probability). The logic flow for offset calculation is as follows: !image.png Following the above logic, the four offsets can be calculated, which, together with the center point of the anchor grid, allow us to compute the coordinates of the top-left and bottom-right corners of the predicted object bounding box detected by this anchor grid. For the feature outputs of the P4 and P5 layers, the probability output logic and calculation logic of the offsets are completely identical, except that the strides become 16 and 32 respectively. According to the above logic, the YOLOv8 model ultimately outputs 8400 prediction box information entries from the detection head. Such a large number of prediction boxes must subsequently pass through the NMS (Non-Maximum Suppression) algorithm filtering before serving as the final detection output of the model. At this point, the decoupled output of the detection head is basically fully clarified: for the three multi-scale feature outputs of P3, P4, and P5, the detection head decouples them into two branches: the category determination branch and the prediction box coordinate branch. For the COCO dataset, the former outputs 80-dimensional category determination information for each grid; the latter uses a 64-dimensional vector to represent the coordinate position information of the object detected by this prediction box. Therefore, the total output dimension length for each grid is 144. What about the object detection confidence information? The answer is that YOLOv8 does not have an independent confidence prediction branch; instead, it hides the object detection confidence information within the class prediction information. The object detection confidence output by each grid anchor is ultimately the maximum value among the 80-dimensional tensor output by its class prediction branch—that is, the recognition probability of the class with the highest likelihood is treated as the confidence that a target exists at that anchor. References - 【YOLOv8】YOLOv8结构解读-腾讯云开发者社区-腾讯云 - What is YOLOv8? A Complete Guide - YOLOv8 原理和实现全解析 — MMYOLO 0.6.0 文档
Automatic Differentiation and Computation Graphs in PyTorch Explained
Based on the study of PyTorch's fundamental data structure torch.Tensor, this article organizes the theory behind PyTorch's gradient and automatic differentiation features, and uses examples to clearly illustrate the runtime logic of computation graphs and the training workflow within the PyTorch framework. Gradients and Automatic Differentiation in PyTorch torch.Tensor is the core data structure in the PyTorch framework. The core design of this data structure is that it not only carries various data during computation but also has a built-in automatic differentiation mechanism, making the training of deep learning models efficient and flexible. _A deep understanding of the torch.Tensor type, along with its gradient and automatic differentiation mechanisms, forms the foundation for understanding deep learning and developing deep learning models using PyTorch._ A tensor in PyTorch is a multi-dimensional array, conceptually similar to NumPy's ndarray. The key difference between the two is that PyTorch tensors support accelerated computation on GPUs and automatic differentiation. Below is a brief summary of the core attributes contained within the torch.Tensor type in the PyTorch framework. data data stores the actual data contained within the tensor. This is no different from NumPy's ndarray, and even the calculation and declaration APIs are very similar. device torch.Tensor uses the device attribute to indicate which device the tensor is stored on, such as a CPU ('cpu') or a GPU ('cuda'). You can specify the storage location of the tensor data via the device attribute when creating it, and during runtime, you can also transfer the storage location between different devices using methods like to(), cpu(), or cuda(). Note that: All tensors involved in an operation must reside on the same device. Attempting to perform operations on tensors located on different devices (e.g., one on the CPU and one on the GPU) will result in a RuntimeError. dtype The dtype attribute is used to represent the data type of the actual data stored in the tensor, such as torch.float32, torch.int64, torch.int32, etc. The default data type is 32-bit floating-point (torch.float32). Note that because model training typically requires the precision and dynamic range provided by floating-point numbers, only floating-point tensors can compute gradients. Therefore, floating-point tensors are generally used during network construction and training, and after training, they are converted to INT-type tensors via quantization to facilitate model deployment. requires_grad requires_grad is a boolean attribute that determines whether the tensor participates in gradient calculation during computations. When set to True, PyTorch tracks all operations performed on this tensor during runtime, thereby constructing a computation graph and preparing for subsequent gradient calculations. By default, leaf nodes (i.e., Tensors created directly by the user) have their requires_grad attribute set to False, though it can be manually configured via the requires_grad attribute. For non-leaf nodes (i.e., intermediate computation result tensors based on leaf nodes), requires_grad is automatically inferred from the computation process: if any operand requires a gradient, the non-leaf node resulting from the operation will also require a gradient. In summary, torch.Tensor's requires_grad and its related gradient attributes are core to enabling neural networks to perform backpropagation, update parameters, and ultimately complete the model training process. grad The grad attribute is used to store the gradient values obtained during the backpropagation process (i.e., when calling the backward() function) for the current torch.Tensor. When executing a model's backpropagation computation, the gradients of leaf tensors (created directly by the user rather than resulting from operations) accumulate here. In the initial state of a tensor or when requires_grad=False, this attribute is None. Every time backward() is executed on the computation graph, the computed results are accumulated into the tensor's grad attribute rather than replacing it. This is why gradients must be manually zeroed out in the training loop using: optimizer.zero_grad(). When executing a model's backpropagation backward computation, by default, only the grad attribute of leaf tensors is automatically populated. The grad attributes of intermediate tensors generated by operations (non-leaf nodes) remain None by default. This is primarily done to save memory, as they typically do not play a direct role in parameter updates. If you need to retain gradients for intermediate nodes, you can call the retain_grad() method on that tensor. grad_fn The grad_fn attribute points to a Function object that records the operation performed to create this tensor. The grad_fn attribute of each node in the model actually represents an edge in the computation graph, recording the complete computation history of the model from input to output, thereby constructing the dynamic computation graph corresponding to the model. This makes it crucial for the Autograd system to backpropagate gradients. For user-created leaf tensors, grad_fn is always None. For non-leaf nodes, the grad_fn attribute is a specific backpropagation function object, and its function name (such as AddBackward, MulBackward, etc.) indicates the type of forward operation used to create that tensor. Here is a code example: Corresponding printed output: - The UserWarning output above indicates an error/warning when trying to print the gradient information of a non-leaf node. To save memory, gradients of non-leaf nodes are not automatically saved during computations. - The grad_fn attribute of z is a function named PowBackward0, indicating that z originated from a power/exponent function. - Before executing the backward operation, the gradient grad of the leaf node x is always None, and becomes 8.0 after execution. This is because , so the derivative of with respect to is , making the gradient calculation result for node equal to 8.0. For relevant theories on derivative calculations, you can refer to Summary of Mathematical Basics for Deep Learning. Computation Graphs and Automatic Differentiation PyTorch's automatic differentiation feature is its core competitive advantage as a deep learning framework. Therefore, when training and inferring models using PyTorch, you only need to focus on the execution logic of the forward pass; the framework will automatically complete gradient calculations, and then update parameters during the backpropagation process based on the magnitude of each parameter's gradient to achieve gradual convergence, as shown in the figure below. !image.png Based on the forward and backward propagation workflows above, we can see that the core components of a complete model computation graph and PyTorch's automatic differentiation include: - torch.Tensor data structures capable of carrying gradient information. As mentioned above, when requires_grad is set to True, this data structure maintains its own gradient information (grad attribute), and operations performed on it (grad_fn) are automatically tracked by the framework to build a complete computation graph. - Computation Graph. During the forward pass (forward) of model code, the PyTorch framework dynamically generates a directed acyclic computation graph, which records the complete computational process from input to output. - Automatic Differentiation Engine. When the backpropagation function (backward) is called on a model, the internal autograd engine maintained by PyTorch traverses the computation graph in reverse, executing the backward() method of each Function. Its role is to calculate the gradients of each node and then write the computed gradients into the grad attribute of the corresponding node's Tensor. - Optimizer. The optimizer is responsible for updating the parameter values of each node based on the current gradient information of each node, the learning rate set by model hyperparameters, etc. The training process of deep learning models is essentially driven behind the scenes by automatic differentiation for iterative parameter optimization. Below is the complete process during model forward and backward propagation where gradient information at each node is computed and updated based on the computation graph, and the optimizer continuously updates node parameters to achieve convergence: - Forward Pass to Build Computation Graph: When performing operations on tensors with requires_grad=True (usually model parameters and input data), the PyTorch framework dynamically builds a complete computation graph in the background. This graph records all operational steps from input data to the final loss function (a scalar). - Executing Backpropagation (backward): Once the loss value (a scalar) is obtained, call loss.backward(). This operation invokes the Autograd engine, starting from the root node and traversing the computation graph backwards. For each node in the graph, the engine calls its predefined backward() method to compute the local gradient of the operation with respect to its inputs, multiplies this gradient by the gradient coming from the output end (the chain rule), and passes it to the input nodes. - Gradient Accumulation and Update: The final execution result of the aforementioned backpropagation process is to accumulate the computed gradients into the .grad attribute of each leaf tensor (i.e., model parameters). - Because it is cumulative, before each parameter update, you must call optimizer.zero_grad() to clear the gradients of all nodes first, preventing previous gradients from interfering with the current update calculation. - Parameter Update: The final step uses an optimizer (such as torch.optim.SGD) to execute parameter updates based on the .grad attribute of each parameter calculated in the previous step (optimizer.step()), thereby reducing the loss step by step until convergence is reached. Complete Example The following explanation is based on the training workflow of a simple network for recognizing the MNIST dataset, implemented in the article Convolutional Neural Network for Handwritten Digit Recognition Based on PyTorch. From the simple training workflow above, we can clearly see the role and execution steps of PyTorch's automatic differentiation and computation graph features during model training: - output = net(data): Perform a forward pass on the input data within the model to obtain the output. During this forward pass workflow, the computation graph of the entire model is built step by step. - loss = criterion(output, target): Compute an error scalar based on the output from the previous step and the target labels. - optimizer.zero_grad(): Be sure to clear all parameter gradients across the entire network before executing backward computations to avoid gradient accumulation. - loss.backward(): Execute a backward computation based on the error scalar. This backward computation traverses from the output to each leaf node of the model using the computation graph, writing the gradients of each node into its grad attribute. - optimizer.step(): Finally, call the optimizer's step method to update node parameters using parameters such as each node's gradient (grad attribute) and learning rate. The parameter update for the current training step concludes here. References - "PyTorch Deep Learning Application Practice" by Chen Zhaoming and Hong Jinkui, Section 3-3: Automatic Differentiation
YOLOv5 Model Inference Outputs and NMS Algorithm Analysis
This article summarizes the data structure output by the YOLOv5 detection head after forward inference on input image data, as well as the Non-Maximum Suppression (NMS) algorithm workflow executed on these outputs. YOLOv5 Model Output Data The article YOLOv5 Model Network Architecture Explained provides a detailed overview of the model's network architecture. To analyze and summarize the output data of the YOLOv5 model, the most crucial part is the structure of its detection head: !image.png As shown in the figure above, YOLOv5 adopts a multi-scale detection mechanism, outputting feature maps at three different scales from the detection head to detect objects of various sizes. For a standard 640×640 input image, the shapes of the output tensors are: - Large-scale feature map (P3 layer): [batch_size, 3, 80, 80, 5+NC] - Medium-scale feature map (P4 layer): [batch_size, 3, 40, 40, 5+NC] - Small-scale feature map (P5 layer): [batch_size, 3, 20, 20, 5+NC] The dimensions of each output tensor represent the following: - batch_size: The batch size, representing the number of images processed simultaneously. - 3: The number of anchor boxes predicted per grid cell; the default in the YOLOv5 model is 3. - H, W: The height and width of the feature map (e.g., 80×80, 40×40, 20×20). With a 640x640 input, YOLOv5 outputs feature maps at 8x, 16x, and 32x downsampling scales, corresponding to resolutions of 80x80, 40x40, and 20x20, respectively. - 5+NC: The output dimensionality for each prediction box. The information contained in the output of each prediction box is as follows: !image.png In summary, for the P3 layer output, a total of prediction boxes are generated; for the P4 layer output, prediction boxes; and for the P5 layer output, prediction boxes. Each prediction box contains (5+NC) lengths of information describing its attributes. NMS Algorithm As described above, when the YOLOv5 model performs inference on an image, it ultimately outputs over 20,000 prediction boxes across three scale dimensions, typically generating a large number of overlapping prediction boxes for the same object. Therefore, filtering is required to yield the final detection boxes. The process of filtering the model's output tensor data to obtain these final detection boxes is known as the NMS algorithm. The task of the NMS algorithm is to select the most accurate box among overlapping ones, suppress (discard) other redundant boxes, and ultimately produce clear, non-overlapping detection results. The basic execution workflow of the NMS algorithm is as follows: - Sorting: Sort all prediction boxes output by the model in descending order based on their Confidence Score. - Selection: Select the box with the highest confidence score and add it to the list of final output boxes. - Suppression: Based on the highest-confidence prediction box selected in the previous step, calculate the Intersection over Union (IoU) between this box and all remaining boxes. Remove boxes whose IoU exceeds a preset threshold (e.g., 0.5), as they likely point to the same object. - Loop: Repeat steps 2 and 3 until no boxes remain. YOLOv5 introduces several improvements to the basic NMS algorithm: - Unified processing of multi-scale prediction boxes: Since the inference output of the YOLOv5 model includes prediction boxes from three different feature maps (P3, P4, P5), all prediction boxes are concatenated together before NMS processing and then filtered uniformly. In this case, regardless of whether a prediction box comes from the P3 layer (responsible for small objects) or the P5 layer (responsible for large objects), as long as its confidence is high, it has the opportunity to be retained. The final selection is the highest-confidence box for that object. - Support for multi-label and single-label detection: This option is controlled by the multi_label parameter. When multi_label=False (single-label), each prediction box belongs to only a single class with the highest confidence, which is the most common object detection mode. When multi_label=True (multi-label), a prediction box can belong to multiple classes simultaneously (e.g., an object is both a car and a taxi). In implementation, a single raw prediction box is split into multiple detection entries based on the number of classes exceeding the threshold. - Merge-NMS (Weighted Box Fusion): This is an optional post-processing step controlled by the merge parameter. Instead of discarding low-confidence boxes whose IoU exceeds the threshold, it calculates a weighted average between them and the high-confidence box. This theoretically improves localization accuracy by fusing positional information from multiple boxes. Source Code Analysis of non_max_suppression Below, we examine the execution logic and workflow of the NMS filtering algorithm applied to model outputs, referencing the non_max_suppression function in the YOLOv5 source code. > The non_max_suppression function in the YOLOv5 source code is located in utils/general.py. The prediction data obtained from the YOLOv5 model by the non_max_suppression function is a tensor of shape [batch_size, num_boxes, 5 + nc]. Here, batch_size is the number of images fed into the model for inference in a single batch, num_boxes corresponds to the total number of prediction boxes per image inference (25,200 = 19,200 + 4,800 + 1,200), and 5 + nc represents the predicted output data for each box (center_x, center_y, width, height, object_confidence, class_1, ..., class_n). The key procedural code snippets of the non_max_suppression implementation are excerpted and explained below: The above is the declaration of the non_max_suppression function in the YOLOv5 source code, where: - prediction: The raw prediction data output by the model, containing information for all detection boxes across the small, medium, and large scales. - conf_thres: The confidence threshold used to control and filter low-confidence detection boxes during NMS. A detection box is retained for subsequent processing only when its confidence (including object confidence and class confidence) is greater than this threshold. Increasing this value reduces the number of detection boxes but may miss some true objects; lowering it increases the number of boxes but may introduce more false positives. This value is set based on precision and recall requirements. - iou_thres: The IoU (Intersection over Union) threshold used in the NMS algorithm to determine if bounding boxes overlap. When the IoU between two detection boxes exceeds this threshold, NMS keeps the one with higher confidence and removes the lower-confidence one. Increasing this value retains more overlapping detection boxes; lowering it removes overlapping boxes more aggressively. - classes: Specifies that only detection results of particular classes should be retained. If set to a list of class indices, the function returns only detection boxes for those classes, filtering out others. - agnostic: Controls whether class-agnostic NMS processing is performed. - When False (default), NMS is performed independently within each class, and overlapping boxes of different classes do not suppress each other. - When True, NMS treats all classes as a single class, and overlapping boxes suppress each other regardless of their class. - multi_label: Controls whether multi-label detection is allowed (i.e., an object may belong to multiple classes, such as a detection result being both a person and a man). - When set to False (default), each detection box retains only the single class with the highest confidence. - When set to True, a detection box can retain all classes whose confidence exceeds the conf_thres threshold. - labels: Used in auto-labeling mode to pass prior label information. - max_det: Limits the maximum number of detection boxes output per image. By default, a maximum of 300 valid detection boxes are output per image. - nm: Specifies the number of masks, used only for segmentation models. The batch size bs is read from the prediction tensor. The model can predict multiple images at once, and the first dimension of its prediction tensor is the number of images bs. nc is the number of classes output by the model, where nm is only used for segmentation models and is 0 for standard object detection models. xc compares the object confidence of all detection boxes in the prediction output against the confidence threshold conf_thres. Boxes with a confidence greater than the threshold are marked as true, and others as false. In fact, the vast majority of prediction boxes output by prediction are filtered out at this confidence threshold comparison step. An output variable is declared to store the valid detection boxes after subsequent NMS filtering, where the detection information length for each box is 6 (coordinate positions + confidence + class). A for loop then extracts each image frame from the prediction variable for independent processing, and all subsequent operations within this loop target a single image frame. Using x = x[xc[xi]], x retains only the detection boxes of the xi-th image whose confidence is greater than conf_thres, discarding all other boxes. It then checks whether the number of detection boxes in the x list is 0. If it is 0, it means no valid targets were detected in the current image frame, and the loop directly continues to process the next image frame. At this point, the x list holds all detection box information with a confidence detected in the current image frame. The information of all detection boxes in the x list is processed: the classification probability for every class is multiplied by the overall object confidence to obtain the absolute probability for each class; the xywh coordinate information output by the model is converted into the xyxy format required by the NMS algorithm. The code above formats the detection box list needed by the NMS algorithm and performs a preliminary round of filtering using the absolute class probabilities. It is divided into multi-label and single-label cases: - In the multi-label case, the absolute probability of each class is compared with the confidence threshold conf_thres, and all classes with absolute probabilities greater than the threshold are retained. - In the single-label case, the class with the highest absolute probability is found among all classes and compared against the confidence threshold conf_thres. If it exceeds the threshold, it is retained. At this point, each detection box in the x list contains only a single class, formatted as [x1, y1, x2, y2, confidence, class]. If a detection box supports two classes simultaneously (when multi_label is enabled), there will be two separate detection boxes for it in the x list. If the classes parameter is set, only the categories specified by classes are retained in the x list, and all other detection box information is filtered out. The code then checks the number of remaining detection boxes. If it equals 0, it proceeds directly to process the next image frame. This section reaches the core processing steps of the NMS algorithm. First, all detection boxes in the x list are sorted in descending order of confidence, and only the top max_nms prediction boxes are retained. Then, based on the setting of the agnostic parameter, it decides whether subsequent NMS filtering should distinguish between different classes. When agnostic=False, the class index is multiplied by a large number max_wh (default is 7680), ensuring that different classes are computed in separate spatial regions so they do not suppress each other during NMS filtering. Finally, the coordinates boxes and detection confidence scores of each detection box are extracted from the x list, and torchvision.ops.nms is called to perform NMS filtering. torchvision.ops.nms returns a 1D integer tensor containing the indices of the bounding boxes that should be retained after the NMS algorithm, ordered by confidence from highest to lowest. The code above restricts the return result of torchvision.ops.nms to a maximum of the first max_det results as the final output. When the merge feature is enabled, IoU comparisons are used to fine-tune the coordinate positions of the final output detection boxes, enhancing localization accuracy. The final step assigns the NMS-filtered detection box list to output as the final output of the non_max_suppression function. References - yolov5 Non-Maximum Suppression (NMS) Code Detailed Explanation - Jinse Xuguang - CNDS Blog
Detailed Explanation of P, R, PR, and F1 Metrics for Computer Vision Object Detection Models
The four image files F1_curve, P_curve, PR_curve, and R_curve generated in the YOLOv5 model training results correspond to the F1, P, PR, and R metric curves after model training completion, respectively. This article provides a detailed summary and organization of these four metrics. P_Curve The P curve corresponds to the model's Precision, which measures out of all samples predicted by the model as positive, how many are actually true positives. It focuses on the accuracy of the prediction results, answering the question: "How accurate are the detections?" The higher the precision, the fewer false positives and misjudgments (mistaking the background or other objects as the target) the model makes. For example, in security surveillance, high precision means high alarm accuracy, reducing false alarms. The figure below shows a typical P_Curve: !image.png As can be seen, the vertical axis of the P_Curve is Precision, and the horizontal axis is Confidence. As the confidence threshold increases, precision generally shows an upward trend. This is easy to understand: the lower the confidence, the more misjudged targets are counted in, which naturally pulls down the precision metric. As the confidence threshold increases, a large number of low-confidence detection results are filtered out, misjudgments decrease, and the precision metric naturally rises. In an ideal scenario, the P_Curve is a straight line where Precision remains consistently around 1.0. This means that no matter how you set the confidence threshold, the model's predictions are extremely reliable with very few false detections. Of course, this situation is impossible in reality. In most cases, the P_Curve presents a curve where Precision rapidly rises as the Confidence threshold increases, until a certain point where the vertical axis coordinate approaches 1 and remains flat thereafter. However, the larger the area enclosed under the P_Curve (which actually means Precision can approach 1 more quickly), the higher the model's robustness, indicating that it can maintain high precision across different thresholds. If only a few class curves (such as the crib curve in the above image) are significantly lower than the others, it may indicate that the number of samples for that class in the dataset is insufficient (which is often the case) or that the sample quality is poor, making it difficult for the model to accurately learn the features of that class. In this case, you should consider collecting more data or performing data augmentation for that category. Additionally, under normal circumstances, the P_Curve should rise smoothly. If the curve exhibits drastic fluctuations or abnormal sharp drops, it may suggest overfitting or underfitting problems during the training process, requiring targeted treatment. The primary value of the P_Curve lies in assisting with the scientific configuration of the confidence threshold for model predictions, achieving a balance between Precision and Recall to meet project application requirements and goals. For example, in application scenarios where false detections are unacceptable and high precision is pursued (such as high costs for false alarms in security monitoring), a confidence threshold corresponding to high precision (such as 0.95 or higher) should be selected based on the P_Curve. In this case, the model's output will be very conservative, outputting only results it is extremely certain of, at the cost of potentially missing some less certain positive samples, thereby leading to a lower recall. R_Curve Corresponding to the P curve is the R curve, which is the Recall curve. Recall measures out of all true positive samples, how many were successfully predicted by the model. Therefore, the Recall metric focuses on the model's ability to discover positive examples, answering the question: "How complete are the detections?" The higher the recall, the fewer false negatives (failing to detect a true target) the model produces. Similar to the P curve, the horizontal axis of the R curve is also Confidence, and the vertical axis is the model's Recall. Thus, it illustrates how recall changes when the confidence threshold of the model's output varies from 0 to 1. The figure below shows a typical recall curve: !image.png Both the P_Curve and R_Curve use the confidence threshold (Confidence) as the horizontal axis, but the changes in these two metrics relative to the confidence threshold are exactly opposite. In the R_Curve, as the confidence threshold increases, the recall exhibits a downward trend. This is also relatively intuitive to understand: higher confidence leads to more positive case judgments being filtered out (because the confidence does not meet the threshold requirement), which naturally leads to increased missed detections and lower recall. The higher the overall position of the R_Curve in the above image, and the larger the area enclosed by the curve and the two coordinate axes, the more robust the model's performance and the fewer missed detections it has, as it maintains high recall across different confidence thresholds. Similarly, the primary value of the R curve is to help scientifically select and set the confidence threshold to balance Precision (P) and Recall (R) while meeting application requirements. For example, in application scenarios that pursue high recall and completely unacceptable missed detections (such as medical diagnosis, obstacle detection in autonomous driving, etc.), a confidence threshold corresponding to an acceptable high recall (such as 0.95 or higher) should be selected based on the R curve. The figure below is a comparison and summary table of the P curve and the R curve: !image.png PR Curve The PR Curve (Precision-Recall Curve) is a core tool for evaluating the performance of object detection models. It intuitively displays the trade-off relationship between Precision and Recall under different confidence thresholds. The PR curve uses Recall as the horizontal axis and Precision as the vertical axis. Each point on the curve represents the combination of precision and recall achieved by the model at a specific confidence threshold. The figure below is an example of a typical PR curve: !image.png The plotting process for the PR curve of a specific category is as follows: - First, sort all predicted bounding boxes generated by the model on the validation set in descending order of their confidence scores. - Then, iterate through thresholds, starting from the highest confidence (e.g., 0.99) and gradually lowering the threshold (e.g., 0.98, 0.97, ..., 0.01). At each threshold, treat all prediction boxes with a confidence higher than that threshold as positive examples, and calculate the precision and recall at this point. - Finally, connect all calculated (Recall, Precision) points to form the PR curve. Typically, the shape of the PR curve is a curve that starts from the top-left corner and gradually descends towards the bottom-right corner. This means that as recall increases (wanting to detect more true targets), precision will gradually decrease (false detections will increase). Conversely, if you want to increase precision (reduce false detections), recall will decrease (missed detections will increase). When comparing the performance of two models (using different models tested on the same validation dataset, of course), if model A's PR curve completely encloses model B's PR curve, it can be asserted that model A's performance is superior to model B's. If the two curves intersect, it is necessary to compare their pros and cons by calculating the area between each curve and the two axes (which is essentially the AP metric). Relationship Between PR Curve and AP/mAP Metrics While the PR curve is intuitive, it is not a single specific number, making it inconvenient to use—especially when comparing different models. Therefore, the AP and mAP metrics were introduced based on the PR curve. The AP (Average Precision) metric refers to the area between the PR curve and the two coordinate axes. A higher AP value indicates that the model can maintain high precision while maintaining high recall, resulting in better comprehensive performance. Of course, the AP metric is specific to a single detection category. Since most object detection models support detecting targets of multiple different categories, comparing the overall performance of models requires evaluating the average AP metric across all target categories. This average AP metric is the mAP (mean Average Precision) metric. The mAP metric is the average of the AP values for all categories supported by the object detection model after training, and it is currently the most authoritative and commonly used evaluation metric in the field of object detection. F1 Curve Once you fully understand the P and R curves discussed above, the F1 curve becomes quite easy to grasp. The F1 score is the harmonic mean of Precision and Recall, used to comprehensively evaluate model performance. Its calculation formula is: Since both Precision and Recall metrics are decimals between 0 and 1, the F1 score calculated by the above formula also ranges between 0 and 1. A higher F1 score indicates better comprehensive performance of the model in terms of precision and recall. The figure below shows a typical F1 curve: !image.png As can be seen from the above figure, the F1 curve also uses the confidence threshold (Confidence) as its horizontal axis, and the vertical axis represents the F1 score under different confidence levels. Under normal circumstances, the F1 curve typically presents a bell shape that first rises and then falls. At lower confidence thresholds, the model predicts many results (high recall ), but they contain a large number of false detections (low precision ), resulting in a lower F1 score at low confidence levels. As the threshold increases, false detections decrease, precision rises, and the F1 score rises accordingly. After reaching a peak, continuing to increase the threshold filters out some correct predictions (low recall), thereby causing the F1 score to drop. - High and wide curve shape: If the sides of the F1 curve are steep and form a wide plateau at a higher position, it indicates that the model can maintain high performance over a wide range of confidence thresholds, making the model very robust. This is the ideal shape for an F1 curve. - Low and narrow curve shape: If the overall position of the F1 curve is low and the peak is sharp, it means that the model performs moderately well only under very specific thresholds, and its overall generalization ability is poor. Overall, the F1 curve is closely related to the P curve (Precision-Confidence) and R curve (Recall-Confidence), and together they form a complete evaluation system. Combining all three can assist in a more scientific selection of the optimal confidence threshold for model inference based on specific application requirements: - In application scenarios that pursue high precision (avoiding false detections), you can choose a higher threshold to the right of the peak point, sacrificing some recall in exchange for higher precision. - In application scenarios that pursue high recall (avoiding missed detections), you can choose a lower threshold to the left of the peak point, sacrificing some precision in exchange for higher recall.
An Interpretation of YOLO Model Training Output Files
This article interprets the files generated during the YOLO model training process on a custom dataset, helping to build a deeper understanding of the YOLO model training workflow. As summarized in A Comprehensive Guide to Custom Training, Testing, and Model Export for the YOLOv5 Model, after training a YOLOv5 model on a custom dataset using the train.py script, a variety of training result files will be generated under the runs/train/exp directory in the YOLOv5 source code folder (exp is the default directory name; if you train multiple times, it may generate exp2, exp3, etc.). This article provides a detailed explanation of the contents of these files and subdirectories from a functional classification perspective. !image.png weights/ Model Weight File Directory The weights subdirectory under the exp directory is the most important output, as it stores the model weight files generated during the training session. It contains two files: last.pt and best.pt. The best.pt file is the optimal model weight file generated during the training process. The "optimal" here refers to the model weights that perform the best on the validation dataset. During the training process with train.py, the model's performance is evaluated on the validation set using the mAP@0.5 metric at the end of each epoch. If the performance of the current epoch surpasses the previous best record, the best.pt file is updated using the weight parameters of the current epoch. best.pt is also the most commonly used model file in practice because it represents the peak performance achieved during training. In actual production workflows, this model file is typically exported to formats such as ONNX or TensorRT for deployment in production environments. The last.pt file is the model weight file from the final epoch of model training. Regardless of the validation performance on the validation set, the last.pt file is updated with the current model weight information at the end of every training epoch to ensure it always stores the latest training state. The purpose of last.pt is that if training is interrupted unexpectedly or if we wish to resume training based on the currently trained model, we can continue the training process from the current state using the last.pt file. In addition, last.pt can be used to analyze convergence during the training process and observe the final training state. Training Result Metrics The training result metrics after completing YOLOv5 model training correspond to the results.png and results.csv files, which represent the key metric data and charts for each epoch during the training process, respectively. results.png Below is an example of results.png: !image.png As can be seen, results.png displays multiple metrics of the model: - Training set / Dataset losses (box_loss, obj_loss, cls_loss): Corresponding respectively to the bounding box coordinate loss, objectness confidence loss, and classification loss on the training and validation sets. - Ideally, all loss curves steadily decrease and eventually level off at a low level. It is normal for training loss to be slightly lower than validation loss. - Overfitting: If the training loss continues to decrease, but the validation loss starts to increase after a certain point, it indicates that the model is overfitting the training data (the confidence loss in the above image demonstrates this case). In this scenario, you should try increasing the amount of training data, lowering model complexity, or reducing the number of training epochs. - Underfitting: If both training and validation losses remain at a high level and stop decreasing (typically entering a plateau while still high), it suggests the model may be underfitting. This requires a more complex model, longer training time, or appropriate hyperparameter tuning. - Precision metrics (precision, recall, mAP@0.5, mAP@0.5:0.95): - Precision: Indicates what proportion of samples predicted as positive by the model are actually positive. Higher is better. - Recall: Indicates what proportion of all actual positive samples were successfully predicted by the model. Higher is better. - mAP@0.5: The Mean Average Precision calculated at an IoU threshold of 0.5. This is the most commonly used metric; higher is better. - mAP@0.5:0.95: The Mean Average Precision across IoU thresholds ranging from 0.5 to 0.95 (with a step size of 0.05). This is a more stringent metric that better reflects the accuracy of model localization. Among the accuracy metrics mentioned above, mAP@0.5 primarily measures classification capability and basic localization capability. As long as the predicted bounding boxes are roughly in the correct position, a high score can be achieved on this metric. In practical effects, a model might have a high mAP@0.5, but its bounding boxes could be "floaty" and insufficiently precise. By contrast, mAP@0.5:0.95 is much stricter, forcing the model to possess both precise classification and localization capabilities simultaneously. It demands that the model not only correctly identify objects but also accurately outline their boundaries. If a model has a high mAP@0.5:0.95, it means its predicted bounding boxes are exceptionally precise. > The calculation logic for the mAP@0.5:0.95 metric is as follows: First, calculate the mAP across 10 different IoU thresholds ranging from 0.5 to 0.95 with a step size of 0.05 (i.e., 0.5, 0.55, 0.6, ..., 0.95). Then, calculate the mAP metric corresponding to each IoU threshold. Finally, average these 10 mAP values to obtain the final mAP@0.5:0.95 metric. results.csv As shown in the image below, the contents of the results.csv file are basically identical to the charts in results.png, representing a tabular log of loss values and accuracy metrics for each epoch. In addition to the aforementioned data, results.csv also logs three learning rate parameters: x/lr0, x/lr1, and x/lr2. !image.png The three parameters x/lr0, x/lr1, and x/lr2 correspond respectively to the learning rates used during training for the three modules of the YOLOv5 model: the backbone, neck, and detection head (for details, please refer to the article An Interpretation of the YOLOv5 Model Network Architecture). During YOLOv5 model training, the learning rates for these three parts are independent, and none of them remain static. To help the model converge better, a Learning Rate Scheduler is used during training to dynamically adjust the learning rates. The learning rate values for these three parameter groups adjust and change at each training step or epoch according to specific scheduling strategies (such as cosine annealing), and these changes are recorded in the results.csv file. Confusion Matrix (confusion_matrix) Among the YOLOv5 model training result files, there is a confusion_matrix.png file used to visually display the trained model's predictions for each category on the validation dataset, making it primarily useful for evaluating the model's classification performance. Below is a typical confusion_matrix.png file containing 6 categories: !image.png Interpretation of the above file: - Values on the diagonal represent the proportion of samples correctly predicted by the model. Higher values are better (indicated by darker block colors), meaning the category can be correctly recognized. - Values off the diagonal represent the proportion of mispredictions—i.e., the proportion of samples of true class A incorrectly predicted as class B by the model. Lower values are better (indicated by lighter block colors), representing fewer category confusions. As seen from the image above, apart from the crib category which is relatively prone to misclassification, the confusion between other categories is minimal. For categories with high confusion, you need to check which other categories they are easily confused with (in the figure above, crib is mainly confused with background). In this case, you should provide more training images that help distinguish these different categories, or purposely apply data augmentation to simulate scenarios where the two categories are easily confused and retrain the model. F1 / P / R / PR Metric Curves After each training session of the YOLOv5 model, four metric chart files—P_curve, R_curve, PR_curve, and F1_curve—are generated in the exp directory. Detailed explanations of these four metrics can be found in the article Detailed Explanation of P, R, PR, and F1 Metrics for Computer Vision Object Detection Models and will not be repeated here. train_batch and val_batch Image Files During YOLOv5 training, a total of 9 images are generated: three training sample images (train_batch) based on the training dataset, three ground truth label images (val_batch_labels) based on the validation dataset, and three validation prediction result images (val_batch_pred). As shown below: !image.png The three train_batch images display sample images from different batches in the training set. During training, YOLOv5 automatically applies data augmentation operations to the images fed into the model. These operations include scaling, rotation, color adjustment, and mosaic stitching. You can view the effects of data augmentation through these three images, along with the ground truth bounding boxes and category labels on each image, which help understand the diversity of the training data. As seen from the train_batch images above, the YOLOv5 model adopts Mosaic data augmentation technology during training. Its core idea is to combine four different training images into a new, larger training image via random scaling, cropping, and stitching, and then scale it to the 640x640 resolution required by the model before feeding it into the network for training. The three val_batch_labels images show the effects of ground truth labels from different batches in the validation image set, serving as the answer key for testing the model against the validation set. These validation images are directly annotated with the true categories of existing target objects and their precise locations. The content displayed in the three val_batch_pred images corresponds to the val_batch_labels images, illustrating the target categories and positions inferred by the model during validation testing on the exact same batch of validation images. By comparing the ground truth label images (val_batch_labels) with the prediction result images (val_batch_pred), you can identify where the model tends to make mistakes and analyze error patterns such as false positives (mis-detections) and false negatives (missed detections).
YOLOv5 Source Code Analysis: Detect.py
This article provides a detailed analysis of the source code of the detect.py script used to execute inference tasks in YOLOv5, helping you fully understand the entire process from parameter passing and parsing to inference execution and result output. The detect.py script in the YOLOv5 source code is specifically designed to perform inference tasks after a model has been trained. In other words, it utilizes the trained model to make predictions on new data such as images, videos, and real-time streams, identifying and marking targets of interest within them, and saving the results. Typical Ways to Invoke detect.py When using the detect.py script to perform basic inference tasks, the two most important parameters to pass are --weights, which specifies the model's .pt file, and --source, which specifies the target task for inference (whether it's an image, video, or media stream). Detecting images (or a directory) and displaying the results: Detecting a video file, saving the results and detection data: Using a UVC camera and detecting only specific classes: When executing detect.py as shown above, the generated result files will be saved in the runs/detect/exp directory of the YOLOv5 source code (this can be modified via the --project and --name parameters). Parameter List for the detect.py Script Similar to several other execution scripts in the YOLO codebase, detect.py uses the ArgumentParser class from the argparse package to manage its numerous runtime parameters. When calling the script, you first need to pass the corresponding option arguments. The chart below organizes all the parameter options supported by the detect.py script, their default values, and functional explanations for each parameter: !image.png Detailed explanations for some of the less intuitive parameters from the operation list above are as follows: - Note that in the vast majority of cases, there is no need to pass the --data parameter. This is because the model file (.pt) passed via the --weights parameter already contains the classification information necessary for object detection and recognition. If you forcibly use the names list from a specified YAML configuration file via --data, it will override the category information embedded within the model file, which can instead cause confusion. Therefore, the default and recommended approach is to directly use the classification information built into the .pt file. In this case, the model uses its own training-time category definitions, ensuring correct and concise behavior. - The --dnn parameter: In certain scenarios, you may need to use the DNN module in OpenCV (rather than the default PyTorch operators) to execute the inference computation pipeline. This is typically done to integrate with existing C++ applications developed using OpenCV for better execution performance. You can attempt to pass the --dnn parameter during inference so that the underlying calculations use OpenCV's DNN instead of PyTorch. Of course, the prerequisite for passing this parameter is ensuring that the provided model file is in ONNX format; other model file formats do not support the --dnn parameter. - The --visualize parameter: This indicates that during inference, feature maps extracted by various network layers of the YOLOv5 backbone can also be saved. By observing and analyzing these feature maps from different layers, you can diagnose issues during inference and gain a more intuitive understanding of the workflow and principles of convolutional neural networks. - The --augment parameter: This corresponds to the TTA (Test Time Augmentation) operation during YOLOv5 inference. As is well known, augmentation during the training process effectively expands the training dataset, enhances model generalization, and prevents overfitting by applying random scaling, cropping, and color jitter to the original images. During inference, for a given test image, multiple images are generated using a similar logic (such as scaling and flipping), fed into the model for inference to produce multiple detection results, and then mapped back to the original image coordinate space. Finally, steps like NMS are used to integrate these results, yielding more stable and accurate detections. Naturally, enabling this TTA operation involves multiple forward passes for the same image, which will undoubtedly significantly increase the per-image inference time. - The --update parameter: The best.pt file generated after completing custom model training also contains information such as optimizer states, training epochs, and recent evaluation results, which are meaningless for final deployment. By passing the --update parameter, detect.py removes this information during prediction, generating a cleaner model version that retains only model weights, model architecture definitions, and necessary metadata (such as class names). This makes the model file leaner and more suitable for final deployment and sharing. Of course, for a best.pt file you trained yourself, you can also use the strip_optimizer('runs/train/exp/weights/best.pt') command to slim down the model. Note that cleaning optimizer information from the model file using the --update parameter is an irreversible operation, and the processed model file cannot be directly used to resume training or continue interrupted training. Therefore, be sure to keep a backup of the original training weights before executing this. Code Analysis of the Inference Execution Workflow There are mainly only two functions in the detect.py script: parse_opt is used to implement parameter parsing, and run is used to execute the specific inference process. The overall inference process executed by the run function is illustrated in the diagram below: !image.png Below is a summary of the key parts of the detect.py script's execution workflow from a code implementation perspective. Model Loading The code for loading the model is shown below: First, the select_device function is used based on the --device argument to choose the corresponding device type among CPU, CUDA, and Apple's Metal Performance Shaders (MPS). It sets the environment variables for model inference based on the device type and creates a corresponding torch.device object. Subsequent model inference flows will run on this torch.device. DetectMultiBackend handles the model loading process. The reason for using this separate function to load the model is that this inference pipeline supports a variety of different model file formats, not limited to PyTorch-generated .pt files: The DetectMultiBackend function automatically parses the model file type based on the model file name or URL passed via the --weights parameter, and then loads it into the system for use in the subsequent inference pipeline. Creating the Dataloader Based on the Input Source Next, a corresponding dataloader is created based on the type of input source specified by the --source parameter. The subsequent inference process continuously reads frames of images from this dataloader to perform inference and save the results. The input sources passed via --source can generally be divided into three types: - webcam: PC webcams, or media stream URLs specified by prefixes such as http, https, rtmp, and rtsp. - screenshot: Screenshot images of the current computer screen. - others: Ordinary images, video files, and directories. For the three types of sources mentioned above, LoadStreams, LoadScreenshots, and LoadImages are respectively called to return a dataloader class object. The subsequent inference process pulls each frame of image from this object to perform inference: Pay attention to the bs variable in the code above. This variable represents the batch size, which is the number of images fed into the model for inference at a time: - For webcams, the batch size equals the number of simultaneous cameras or video streams being processed (passed as a list of cameras in the --source parameter). - For screenshots, the batch size is fixed to 1, indicating that one frame of screen capture is processed at a time. - For the LoadImages class corresponding to regular file paths, the batch size is also fixed to 1, meaning predictions are made on one image file or one video frame through the model at a time. Inference Workflow The inference for each image frame can be divided into three sub-processes: image preprocessing, inference, and NMS filtering of the inference results. Before executing inference, the model is warmed up, and three Profile objects are created to track the execution time of each sub-process: The model has been loaded in a previous stage. The purpose of this warmup operation is to pass 2–3 dummy images through the model to execute a few forward operations before running the actual inference. This allows the model's parameters and runtime state to reach an optimal condition prior to real inference. Of course, if you are only inferring a single image using detect.py, this warmup holds little significance. However, when inferring continuous video streams, executing a warmup before starting inference puts the hardware in a prime state, which helps accelerate the processing of continuous video streams. Additionally, by creating the dt array containing three Profile(device=device) objects, we can subsequently record the execution times of the three sub-processes in the inference pipeline. Below is the code for the execution flow of the three sub-process steps in inference, each using a Profile to measure its execution time: - Image preprocessing: Primarily converts the image format to a PyTorch tensor, copies it to the torch device, and normalizes all pixel data by dividing by 255. - Inference process: For standard .pt files or ONNX format models, this essentially calls the model() operation to execute a single forward pass. - NMS process: Applies non-max suppression to the raw inference results pred generated in the previous step. Parsing Inference Results The final step of the inference pipeline filters the results through non_max_suppression() and returns the final object detection results pred. The data structure of pred is a PyTorch tensor containing batch size (i.e., the bs variable mentioned above) number of det data structures. A det is also a tensor representing the detection results for a single image. The format of det is (n, 6), indicating that n objects were detected in the image, with each object corresponding to a 6-element tuple described in order as follows: - x1, y1: The top-left coordinates of the detected object. - x2, y2: The bottom-right coordinates of the detected object. - conf: The confidence score of the detection result. - cls: The predicted class. The code for the subsequent post-processing stage is relatively simple. It mainly parses pred after NMS and the det data structures it contains to obtain the coordinate locations, confidence scores, and class predictions of all detected objects across the images, which are then drawn onto the images or saved to external files. The above code first loops through the pred structure, with each iteration handling the detection results det for a single image. For each image's detection results, you can sequentially retrieve the detected object's coordinates, confidence score, and class. Of course, because the model's input resolution is fixed at 640x640, the coordinates output by det are also based on this resolution. To render these detection coordinates onto the original image, scale_boxes must be used to map the model's output coordinates back to the original image dimensions.
Installing PyTorch Development Environment with Nvidia CUDA Support
Operating System: Windows 11 Hardware: HP OMEN Max, Ultra 9 275HX + 5070 Ti Check Nvidia Graphics Driver Version Type nvidia-smi in the terminal: As shown above, the maximum CUDA version supported by the graphics driver is 12.9. - If the above command indicates that the command cannot be found, you need to reinstall the Nvidia graphics driver: Download The Official NVIDIA Drivers | NVIDIA Install the GPU Version of PyTorch Go to the official PyTorch Get Started page and select the appropriate torch version corresponding to your current CUDA version. Since the current CUDA version checked earlier is 12.9, the version to choose here should be the PyTorch version for CUDA 12.8. Copy the installation command below to install the PyTorch library. > CUDA Version Matching Rule: The CUDA version of PyTorch should be ≤ the version supported by the driver. !image.png Verify the PyTorch Environment After completing the installation above, use the following script to test whether the current environment has been successfully set up: Under normal circumstances, the output printed after executing the script above will be: References - PyTorch CUDA Support Errors: Solutions and Installation Guide_Baidu Comate
Detailed Explanation of YOLOv5 Model Configuration Files
Combined with the architecture diagram of the YOLOv5 model, this article provides a detailed summary and explanation of the configuration parameter details for each section of the model's yaml configuration file, helping to gain a more granular understanding of the differences between various scale versions of the model. In versions prior to YOLOv5, .cfg files were generally used to control various configuration parameters. Starting from YOLOv5, the .yaml configuration file format was officially adopted. The configuration files for YOLOv5 are located in the models subdirectory of its source code. By default, this directory contains five model configuration files: yolov5n.yaml, yolov5s.yaml, yolov5m.yaml, yolov5l.yaml, and yolov5x.yaml, corresponding to model structures of various scales respectively. - For details on the differences between various versions of the YOLOv5 model, please refer to Summary of Different Versions of the YOLOv5 Model. !image.png From the structure of this configuration file, the configuration parameters of YOLOv5 can be divided into global parameters and network structure parameters, while the network structure parameters can be further divided into two sub-sections: backbone and head. Global Parameters The global parameter section of the configuration file is shown below: nc The nc (number of classes) parameter is relatively straightforward; it defines the number of target classes that the model needs to detect. Since the YOLOv5 model is trained on the COCO dataset by default, the default value of 80 here corresponds to the number of target classes supported by the COCO dataset. If you want to train a YOLOv5 model from scratch using a custom dataset, nc should correspond to the number of classes supported by your own dataset (set in the dataset configuration file, which overrides the nc setting in the network model configuration file). depth_multiple depth_multiple is a scaling factor used to control the number of residual blocks (Bottleneck modules) contained in each CSPBlock (referred to as the C3 module in YOLOv5 v6.0) within the backbone and neck networks of the network structure. The figure below shows the structure of the CSPBlock in the YOLOv5 network. Here, depth_multiple represents the quantity in the figure below (i.e., the number of DarknetBottleneck residual blocks). Naturally, a greater number of DarknetBottleneck residual blocks contained in each CSPBlock implies a deeper network depth, which is the origin of the term "depth". !image.png The calculation code related to the depth_multiple parameter is as follows: Where is the preset module repetition count in the configuration file, and the max() function is used to ensure that the calculated module count is an integer not less than 1. Taking YOLOv5s as an example, the configuration of the third CSP Layer in the backbone network is [-1, 9, C3, [512]], meaning this CSP Layer contains 9 consecutive DarknetBottleneck residual blocks. The depth_multiple of YOLOv5s is 0.33, so the actual number of residual blocks at this layer in YOLOv5s is . This means that in YOLOv5s, the C3 module originally designed to repeat 9 times at this position is ultimately stacked only 3 times. The depth_multiple parameters for various scale models of YOLOv5 and their corresponding number of CSP Layer residual blocks are shown in the figure below: !image.png width_multiple The logic of width_multiple is similar to that of depth_multiple. It generally controls the output channels (i.e., the number of convolution kernels in feature maps) of all convolutional layers and C3 modules (including both the backbone and neck networks) in the network in the form of a multiplier factor. During the model construction process of YOLOv5, the width_multiple parameter is multiplied by the preset channel count in the configuration file, and then ensured to be a multiple of 8 via the make_divisible function (for GPU computation efficiency): Taking the first convolutional layer of YOLOv5s as an example, the preset channel count in the configuration file is [64, 3, 2] (indicating a default output channel count of 64), and the width_multiple parameter corresponding to YOLOv5s is 0.50. In this case, the actual output channel count of the first convolutional layer is: . This means that in YOLOv5s, the convolutional layer originally designed to output 64 channels actually outputs only 32 channels. The figure below shows the default width_multiple parameters for various scale models of YOLOv5: !image.png anchor Similar to YOLOv3 and v4, YOLOv5 also uses the Anchor mechanism to preset target detection bounding box dimensions, helping the model locate and recognize targets more accurately. The backbone network of the YOLOv5 model outputs feature maps of three different resolutions (large, medium, and small). Each grid cell on each feature map is assigned three anchor boxes of different sizes and aspect ratios: - The anchor configuration information in the configuration file consists of three rows in total, corresponding to feature maps of different scales: P3/8, P4/16, and P5/32, respectively. - Each row contains 3 groups of width and height values, with each group representing the [width, height] of an anchor. - Small feature maps (P3/8) use small anchors to detect small objects, medium feature maps (P4/16) use medium anchors to detect medium objects, and large feature maps (P5/32) use large anchors to detect large objects. Comparing the default anchor parameters provided in the YOLOv5 model configuration file above, you will find that they are completely identical to the anchor box dimensions calculated using the k-means mechanism on the COCO dataset for the YOLOv3 model in Summary of YOLOv3 Output, Anchor, and Loss Function Evolution. In other words, the anchor box dimensions set in this configuration file are based on the COCO dataset. In fact, when training with a custom dataset, the YOLOv5 model first uses an Auto Learning Bounding Box Anchors mechanism to check whether the dimensions of the anchor priors in the custom dataset roughly match the anchor parameter settings provided in the configuration file (which are based on the COCO dataset). If they match, training continues using the default anchor boxes specified in the parameter file; if they do not match, the k-means method is re-executed to calculate the anchors specific to this custom dataset, and these newly calculated anchors are used in subsequent training processes. Conflicts with Dataset Configuration Options In addition to the model configuration files for various YOLOv5 scales, when calling train.py to train the model on custom data, you also need to set the configuration file for the custom training dataset (dataset configuration files are located in the data directory). Since dataset configuration files also have configuration options like nc or anchors, what is the relationship between these configuration options of the dataset and those of the network model? The answer is: The YOLO training script will prioritize the configuration options in the data configuration file. If the nc value in the model configuration file (such as yolov5s.yaml) differs from the nc option in the dataset configuration file, a warning message will be printed in the terminal: Overriding model.yaml nc=80 with nc=5. This indicates that the program is automatically overriding the number of classes in the model output layer from the default value (e.g., 80 classes for the COCO dataset) to the actual number of classes in the training dataset (e.g., 5 classes). In the implementation of the DetectionModel class within the yolo.py source code file, there is the following code. As can be seen, when nc and anchors parameters are provided in the dataset configuration file, YOLO will use the configuration options in the dataset configuration file (such as data/VOC.yaml) to override those in the model configuration file (such as models/yolov5s.yaml): Network Structure Parameters The network structure of YOLOv5 defines each layer of the network in the form of the following list: [from, number, module, args]. Each parameter is explained and summarized in detail below. from is used to specify which layer the input of the current layer comes from: !b7e45d3d-2326-42fd-90b9-a1c3fbec9413.png For the setup of the backbone network (backbone), almost every layer receives data from the previous layer, so from is simply set to -1 here. The neck network structure defined in the head section contains many settings like [-1, 6], which means concatenating the previous layer (usually the result after upsampling) with the 6th layer in the backbone network (a high-resolution feature map) to fuse deep semantic information with shallow positional information. number defines the number of times the current module needs to be repeatedly stacked. For structures that do not require repeated stacking, such as ordinary convolutional layers (Conv) and SPPF, this parameter is set to 1. For the C3 module, this parameter works in coordination with the depth_multiple parameter explained in detail earlier, where the final stacking count = number depth_multiple (the result is rounded to the nearest integer and is at least 1). module specifies the network module used by that network layer. The types of network modules used in defining the YOLOv5 network structure are shown in the figure below: !a84965e0-5355-429a-9b59-8c0836d384e2.png args defines the module parameters required by the module in detail using a list. The specific meaning of the parameters depends on the module type, and their setting format is shown in the figure below. !e57c69f2-d646-461b-b7b1-0a2fb3a08b22.png In the network structure parameter section of the configuration file, the structures of the backbone network and the neck network are defined using the backbone and head sections, respectively. Backbone The configuration of the backbone network structure is as follows: Comparing the above backbone configuration with the backbone architecture diagram makes it much easier to understand. The entire network structure flows from top to bottom: - Convolution block, 6x6, stride=2, 64 channels - Convolution block, 3x3, stride=2, 128 channels - C3 block, 128 channels - Convolution block, 3x3, stride=2, 256 channels - C3 block, 256 channels - Convolution block, 3x3, stride=2, 512 channels - C3 block, 512 channels - Convolution block, 3x3, stride=2, 1024 channels - C3 block, 1024 channels - Finally, an SPPF module, 1024 channels !image.png Head The head section actually defines the structure of the YOLOv5 neck network. Below are the default parameters of the head section in the YOLOv5 model configuration file: This looks slightly harder to understand than the backbone section, but becomes quite clear when combined with the Neck network section in the YOLOv5 model network architecture diagram: !image.png Interpreting the configuration information in the configuration file above, starting from the convolutional block at layer 10 in the lower-left corner of the architecture diagram, the from parameter of this convolutional layer is -1, indicating that it receives input from the final SPPF layer of the backbone network at layer 10 (i.e., the convolutional layer) for convolution processing, and then sends it to the upsampling module Upsample 2 at layer 11 for upsampling... and so on, until the final output at layer 23, which is the last CSP Layer. The last statement in the configuration file, [[17, 20, 23], 1, Detect, [nc, anchors]], indicates that detection information is output from these three layers—17, 20, and 23—to ultimately output the final detection results at the model's detection head. References - Full Analysis of YOLOv5 Principles and Implementation — MMYOLO 0.6.0 Documentation
An Interpretation of the YOLOv5 Model Network Architecture
Based on the P3 version of YOLOv5, this article provides a detailed summary of its overall network architecture and the working logic of each sub-module, laying a solid foundation for further in-depth study of the YOLOv5 model. Note: The YOLOv5 version summarized in this article is YOLOv5l, v6.0, P5 version. For details on the differences between various versions, please refer to Summary of Different Versions of the YOLOv5 Model. Backbone Network Before presenting a complete summary of the entire backbone network structure, the individual sub-modules within the backbone are organized into subsections below. Focus and 6x6 Convolution Early versions of YOLOv5 utilized a Focus structure (a slicing operation) at the beginning of the backbone network to reduce computational load while preserving information. This module is primarily used for image downsampling and feature extraction. By reducing spatial resolution while retaining more original information, the essence of this operation is to convert spatial information (width and height) into the channel dimension. The operation of the Focus module consists of three steps: - Slicing operation: Samples the input image every other pixel along the width and height directions to obtain 4 complementary sub-feature maps. - Channel concatenation: Concatenates the 4 sub-feature maps along the channel dimension, which effectively quadruples the number of channels. - Convolution operation: Applies a standard convolution to the concatenated feature map. Taking the structure of YOLOv5s as an example, an original 608x608x3 image is input into the Focus module. After the slicing operation, it becomes a 304x304x12 feature map, and then through a 1x1 convolution operation with 32 filters, it outputs a 304x304x32 feature map. The processing of the slicing operation is shown in the figure below: As can be seen, this Slice operation is completely identical to the Reorg operation in the YOLOv2 model (refer to Interpretation of the YOLOv2 Model Network Architecture). !image.png The workflow of Focus processing is illustrated in the figure below: !image.png However, in later versions of YOLOv5 (such as after v6.0), the Focus structure was replaced by a standard convolutional layer with a kernel size of 6 and a stride of 2 (Conv2d 6x6, stride=2). This is why some architectural diagrams of different YOLOv5 versions show the Focus module while others show the 6x6 convolution module. Although the Focus module theoretically reduces information loss, in practical applications, a well-designed convolutional layer (using a larger 6x6 kernel) can also effectively capture similar spatial information. Moreover, on hardware such as GPUs, standard convolutions are usually highly optimized, and actual inference speeds may turn out to be advantageous or comparable. Therefore, in newer versions of YOLOv5, replacing the special Focus structure with a general convolutional layer makes the overall network structure more unified and concise, reduces custom modules, and lowers code complexity and maintenance costs. Activation Function In the backbone network of YOLOv5, the SiLU activation function (also known as the Swish function) is universally used to replace the LeakyReLU commonly used in previous versions. Compared to LeakyReLU, the SiLU activation function is smoother around zero, which helps improve gradient flow and the expressive power of the model. The formula for the SiLU activation function is: Where is the standard sigmoid function, with values ranging between 0 and 1. The characteristics of the SiLU function are non-linearity and continuous differentiability. !image.png The figure below shows the structure of the first two convolutional blocks of the backbone network's image input, viewed via Netron from the YOLOv5 ONNX file. The calculation process of the SiLU activation function is clearly visible in this structural diagram, which can be decomposed into two distinct steps: - Sigmoid transformation: First, the output of the convolutional layer is fed into the Sigmoid function. The Sigmoid function maps the value of each element into the interval . - Element-wise multiplication: Next, the original input is multiplied element-wise with the Sigmoid weights obtained in the first step. This step corresponds to the act/Mul operation in the figure below. Its effect is to amplify (weight close to 1), suppress (weight close to 0), or partially retain (weight between 0 and 1) the original features based on the weights calculated by Sigmoid. !image.png However, it should still be noted that different versions of YOLOv5 use different activation functions. Their early versions (such as v1.0-v3.0) still defaulted to LeakyReLU, which is why a large number of CBL (Conv-BN-LeakyReLU) blocks can still be found in the backbones of these early versions. Starting around version v5.0, the official implementation completely switched the default activation function to SiLU, and the CBL blocks consequently became Conv-BN-SiLU. This is the reason why most YOLOv5 model architecture diagrams found online still heavily employ CBL blocks and LeakyReLU activation functions. !image.png CSPLayer Similar to YOLOv4, YOLOv5 also adopts the CSP (Cross Stage Partial) structure in the implementation of its backbone network. In other words, the feature map is split into two parts along the channel dimension; one part undergoes complex convolutional transformations, while the other part takes a shortcut, and finally, the two parts are merged before output. Conceptually, there is no significant difference from YOLOv4. For a more detailed explanation of CSP processing, please refer to Interpretation of the YOLOv4 Model Network Architecture. The structure of the CSP blocks in YOLOv5 is shown in the figure below. As can be seen, there are two types of CSP blocks used in the YOLOv5 model structure: - CSP1 is used in the backbone network, where the convolutional transformation part utilizes ResNet's residual block structure (such as the DarknetBottleneck add=true part in the figure below). - CSP2 is used in the neck network, where the convolutional processing part consists simply of two consecutive convolutional blocks (such as the DarknetBottleneck add=false part in the figure below). !image.png SPPFBottleneck The full name of the SPPF module is Spatial Pyramid Pooling-Fast. It is essentially still an SPP module, but optimized for computational speed and performance. The core objective of both SPPF and SPP modules is to expand the receptive field of feature information and fuse multi-scale context information. The difference between the two is simply that the traditional SPP module processes the input feature map in parallel using multiple large pooling kernels (such as 5x5, 9x9, 13x13) and concatenates the results, whereas the SPPF module applies the same 5x5 pooling kernel repeatedly in a cascading manner multiple times, and then concatenates the pooling results from different stages. - Regarding the working logic of the SPP module, you can refer to another note: Interpretation of the YOLOv4 Model Network Architecture. Compared to the former, the cascading approach of the SPPF module using identical pooling kernels is easier to optimize on hardware, yields faster computation speeds, and can effectively enhance the model's adaptability to objects of different scales. It achieves faster inference speeds while maintaining or even improving model performance. !image.png At this point, the backbone network of the YOLOv5 model becomes clear: assuming an input image resolution of 640x640, the backbone network outputs three sets of feature information with different resolutions and channel counts at scaling factors of 8x, 16x, and 32x respectively, which are then passed to the neck network for feature detection. !image.png In summary, the entire backbone network can be divided into five parts from top to bottom: - Stem Layer: This is the Focus module introduced above, although in newer model versions, the Focus module has been replaced by a standard convolutional network with a 6x6 kernel and a stride of 2. With an input image resolution of 640x640x3, this module outputs a feature map of dimensions 320x320x64. - From Layer 1 to Layer 3, the structure consists of a 3x3 convolution followed by multiple consecutive CSP blocks. The preceding 3x3 convolutional module with a stride of 2 is responsible for halving the resolution of the input feature map while doubling the number of channels. - Layer 1 contains 1 3x3 convolution + 3 consecutive CSP blocks, outputting a feature map of dimensions 160x160x128. - Layer 2 contains 1 3x3 convolution + 6 consecutive CSP blocks, outputting a feature map of dimensions 80x80x256. The output of this layer is fed into the neck network as the large-resolution feature map. - Layer 3 contains 1 3x3 convolution + 9 consecutive CSP blocks, outputting a feature map of dimensions 40x40x512. The output of this layer is fed into the neck network as the medium-resolution feature map. - For the final Layer 4, the first half of the processing remains identical to Layers 1–3, containing 1 3x3 convolution + 3 consecutive CSP blocks, with the addition of an extra SPPFBottleneck module at the end to further expand the receptive field of the output feature map. The output of this layer is a feature map of dimensions 20x20x1024. Finally, this layer's output is fed into the neck network as the small-resolution feature map. Neck Network The neck network is a core component for YOLOv5 to achieve multi-scale object detection. Its objective is to efficiently fuse the features extracted by the backbone network, enabling the model to simultaneously recognize objects of varying sizes. Overall, the architecture and concepts of YOLOv5 and YOLOv4's neck network designs have not changed drastically. However, the core difference is that YOLOv5 introduces improvements such as the CSP2 module (described in the backbone section above) into its neck network, which significantly strengthens feature fusion capabilities and enhances computational efficiency. Regarding the overall network architecture, the neck network of YOLOv5 still adopts the dual-directional structure of FPN (Feature Pyramid Network) + PAN (Path Aggregation Network) shared with YOLOv4. Here, the FPN is responsible for passing strong semantic features top-down, while the PAN is responsible for passing accurate localization features bottom-up, forming a powerful feature pyramid. - FPN processing: Through a Concat operation along the channel dimension, the upsampled deep feature maps are concatenated with the corresponding shallow feature maps, combining the detail information from shallow layers with the semantic information from deep layers. - PAN processing: In the PAN path, convolutional layers with a stride of 2 are used to downsample the fused feature maps once again for further fusion with higher-level features. !image.png Head Compared to previous versions, the detection head does not show significant differences. The three paths of output feature maps undergo independent 1x1 convolution processing to output detection result tensors of various resolutions. !image.png Overall Network Architecture of the YOLOv5 Model Finally, here is the overall network architecture diagram of the YOLOv5 model. Combined with the descriptions of each sub-module above, understanding the overall architecture diagram becomes much easier: !image.png References - 深入浅出Yolo系列之Yolov5核心基础知识完整讲解 - 知乎 - YOLOv5 詳細解讀. 前言 | by Steven Meng | Medium - YOLOv5 原理和实现全解析 — MMYOLO 0.6.0 文档
Summary of Different Versions of the YOLOv5 Model
This article summarizes the different versions of the YOLOv5 model and their differences across various dimensions, including model scale, algorithmic iteration, and output structure. The YOLOv5 model is an open-source model released by Glenn Jocher of Ultralytics. Its GitHub repository is available at: GitHub - ultralytics/yolov5: YOLOv5 🚀 in PyTorch > ONNX > CoreML > TFLite. Model Scale Based on the number of parameters, the YOLOv5 model can be divided into five types: nano (1.9M parameters), small (7.2M parameters), medium (21.2M parameters), large (46.5M parameters), and X-large (86.7M parameters). As the parameter scale increases, the performance naturally improves as well. Basically, if you are deploying to mobile devices or edge computing nodes with embedded processors, the only two viable choices are YOLOv5s and YOLOv5n. - _Model selection depends on the accuracy requirements for image recognition and detection_: For simple object presence detection, YOLOv5s may suffice. However, for complex scenarios that require distinguishing subtle differences or detecting small objects (such as medical imaging), a larger model like YOLOv5l or YOLOv5x is required. !image.png The differences among the aforementioned model versions all stem from two core parameters in the YOLOv5 source code configuration files: depth_multiple and width_multiple. These two parameters generate YOLOv5 models of varying complexities (n, s, m, l, x) by controlling the number of channels (i.e., width) of the convolutional layers in the Backbone and Neck, as well as the stacking count (i.e., depth) of the residual/convolutional layers within the CSP blocks。 - width_multiple: This coefficient is mainly used to control the number of channels in the network feature maps. The larger the coefficient, the more channels each convolutional layer has, resulting in greater model width and parameter capacity, stronger learning capability, but also increased computational cost. - depth_multiple: This coefficient primarily controls the repeated stacking times of certain key modules (such as the C3 modules in the backbone and neck networks). The larger the coefficient, the more network layers there are, deepening the model to extract more complex features, which naturally requires more computational resources. Algorithmic Iteration Unlike the earlier generations of YOLO models, YOLOv5 was directly launched as a PyTorch-based software project. Over the years following its release in 2020, the specific implementation of this project has been continuously updated and optimized, forming an algorithmic iteration sequence from v1.0 to v7.0. To date, the software versions corresponding to the YOLOv5 project, their release timeline, and key update features are illustrated in the figure below. As can be seen, from the perspective of model architecture, the most significant update among these versions is v6.0. This version includes: replacing the Focus module with a standard 6x6 convolutional network with a stride of 2, replacing SPP with SPPF on the small-resolution feature map output at the end of the backbone network, and replacing the previously default LeakyReLU with the SiLU activation function. !image.png For new projects and practical deployments, it is currently recommended to directly use the latest versions such as v6.0 or v7.0. These versions not only incorporate all major architectural optimizations but generally offer better stability and more comprehensive documentation support. Output Structure In addition to the differences in model scale and software project iterations mentioned above, YOLOv5 also provides two output structure configurations (P5 and P6) to further balance detection accuracy, model size, and inference speed, accommodating various application scenarios and hardware conditions. Under conditions with sufficient computing resources (such as intelligent driving and medical imaging), the P6 version can provide ultimate detection accuracy for the detection and recognition of numerous tiny targets. Of course, the P5 version remains the most widely used in industrial sectors today. The differences between the P5 and P6 versions are illustrated in the figure below: !image.png The figure below shows the network architecture diagram of the YOLOv5l P5 output structure: !image.png The figure below shows the network architecture diagram of the YOLOv5l P6 output structure: !fbd0e5d7-8c19-4e31-aba9-961e0077388a.png As seen from the architecture diagrams, the default input for the P5 version is an image with a resolution of 640x640x3, outputting three feature streams downsampled by 8x, 16x, and 32x respectively; whereas the default input for the P6 version is a higher-definition image with a resolution of 1280x1280x3, outputting four feature streams downsampled by 8x, 16x, 32x, and 64x respectively. Version Naming Convention and Task Types Based on the multi-dimensional version differences of the YOLOv5 model summarized above, in practical use, the YOLOv5 model filenames contain information about these dimensions, following a clear naming rule: yolov5{scale}{output_structure}-{task}.pt. For example: - yolov5s.pt: A small-scale (s), P5 output structure model for object detection. - yolov5x6.pt: An extra-large-scale (x), P6 output structure model for object detection. - yolov5m-seg.pt: A medium-scale (m), P5 output structure model for instance segmentation applications. The instance segmentation application mentioned above is supported because, in the latest versions of YOLOv5, in addition to basic object detection, image classification (-cls) and instance segmentation (-seg) are also supported via different task suffixes. These variants are typically developed based on the P5 structure. References - YOLOv5 Principle and Implementation Comprehensive Analysis — MMYOLO 0.6.0 Documentation
An Interpretation of the YOLOv4 Model Network Architecture
The first three versions of the YOLO model—v1 (June 2015), v2 (December 2016), and v3 (April 2018)—were all proposed and developed by Joseph Redmon. However, due to the inevitable application of YOLO models in the military sector, the author decided to quit the computer vision field (hats off to the author's great altruistic spirit). The new YOLOv4 version was jointly released in April 2020 by Alexey Bochkovskiy from Russia and two researchers from Taiwan, China: Hong-Yuan Mark Liao and Chien-Yao Wang. The figure below shows a comparative test result of YOLOv4 against competing models such as YOLOv3 and EfficientDet on the COCO2017 dataset: As can be seen, while maintaining a generally similar detection speed (FPS), the detection accuracy (AP) of the YOLOv4 model has improved by nearly 10% compared to the YOLOv3 version. !image.png Official source code repository address for YOLOv4: GitHub - AlexeyAB/darknet: YOLOv4 / Scaled-YOLOv4 / YOLO - Neural Networks for Object Detection (Windows and Linux version of Darknet ). Similar to previous YOLO versions, the network architecture of YOLOv4 can also be divided into three parts: the backbone, the neck, and the head. - Backbone: The backbone of YOLOv4 is CSP Darknet53, whereas YOLOv3 uses Darknet53. - Neck: The neck of YOLOv4 is PANet, whereas YOLOv3 uses FPN (Feature Pyramid Network). - Head: YOLOv4 retains the YOLO HEAD from YOLOv3. Backbone: CSPDarknet-53 In the design of the backbone network, YOLOv4 borrows the design concept of Cross Stage Partial Network (CSPNet) to build a brand-new, efficient, and powerful backbone network for YOLOv4: CSPDarkNet-53. Compared with Darknet-53 in YOLOv3, CSPDarkNet-53 does not change much in structure; the main differences are the use of CSP residual blocks instead of regular residual blocks, and the replacement of the LeakyReLU activation function with the Mish activation function. CSP block As summarized in the article An Interpretation of the YOLOv3 Model Network Architecture regarding the Darknet-53 backbone structure of YOLOv3, the design of the Darknet-53 network introduced the residual structure popular in ResNet at the time. Therefore, Darknet-53 can be viewed as the stacking of many consecutive residual blocks. CSPDarknet-53, drawing on the concept of CSPNet, replaces regular residual blocks with CSP residual blocks. The figure below shows the structure of the CSP residual block used in CSPDarknet-53: !image.png As can be seen from the structure above, the so-called CSP residual block actually splits the input feature map into two parts, C1 and C2. C1 undergoes no processing, while C2 is processed through a regular residual block and then concatenated and aggregated with C1 for output. The dimensions of the input and output tensors of the entire CSP residual block are completely identical, which is consistent with the processing of regular residual blocks in Darknet-53. The rationale behind the above CSP structure is: There is often a lot of redundant information in the feature maps of convolutional neural networks, especially since information between different channels is likely to be similar. In this case, processing all of this feature information is unnecessary; we only need to process a portion of it while keeping the other portion unchanged. This not only ensures no loss in model performance, but also significantly reduces the computational load and parameter count of the model. Specifically regarding the implementation of the CSP residual block in the CSPDarknet-53 backbone network, refer to the figure below: For the feature map tensor input to each CSP residual block, two independent convolutional blocks are first used to obtain two feature maps with halved channel counts. One path undergoes no processing, while the other passes through a regular residual block. The two parts are then concatenated and combined, and finally processed through a convolution to serve as the output. !image.png As can be seen from the above CSP residual block structure, this structure adds three convolutional blocks at the input and output stages compared to the previous regular residual block. Mish activation function As summarized in the article An Interpretation of the YOLOv2 Model Network Architecture, YOLOv2 introduced the BN (Batch Normalization) layer into the implementation of convolutional layers, making its basic convolutional block the CBL block: Convolution + BN + LeakyReLU activation function. This design was also carried over into YOLOv3. In the backbone network of YOLOv4, the Mish activation function replaces the LeakyReLU activation function commonly used in previous versions. The formula for the Mish activation function is: !image.png The curve of this function is shown in the figure below: !image.png Compared to ReLU, Mish does not set negative values directly to zero; instead, it retains a small negative value, allowing the network to learn richer features. Compared to LeakyReLU, the biggest difference of Mish is its smoother gradient flow, especially around zero. Because its function curve is smooth, it can provide more stable and smoother gradients during backpropagation. This helps alleviate the "dying ReLU" problem where gradients are zero in the negative region, making weight updates smoother, thereby accelerating convergence and enhancing the model's generalization capability. Of course, compared to ReLU and LeakyReLU, the computational cost of the Mish function is significantly higher. However, its application in deep neural networks has shown better results than ReLU, which is why the authors of YOLOv4 introduced it into the backbone network. > Note that YOLOv4 only uses the Mish activation function in the backbone network, while subsequent network parts still adopt the LeakyReLU activation function. The figure below illustrates the network architecture of the YOLOv4 backbone network, CSPDarknet-53. As can be seen, this network architecture is identical to Darknet-53, with the only differences being the replacement of RES blocks with CSP blocks, and the replacement of the LeakyReLU activation function with the Mish activation function. The stacking structure of the CSP residual blocks is still divided into five parts, just like Darknet-53, and the number of CSP blocks in each part remains the classic 1, 2, 8, 8, and 4. Following this stacking structure, the backbone network continues to output feature information at scales of 8x, 16x, and 32x to the subsequent neck network. !image.png Neck: SPP + PAN The neck network in the YOLO model is responsible for collecting feature map information from different stages and resolutions output by the backbone network, fusing and processing them, and then passing them to the head for object location detection and category classification. SPP SPP stands for Spatial Pyramid Pooling. Its core idea is: Using pooling kernels of different sizes to capture features at different scales. In specific implementations, maximum pooling layers of different sizes (typically , , and in YOLOv4) are used to simultaneously process the input feature map, extracting multi-scale features. Finally, these multi-scale features are fused along the channel dimension to form a rich feature representation incorporating multi-scale information, as shown in the figure below: !image.png In the network structure of YOLOv4, the feature information output by the backbone network is further processed through the parallel multi-scale pooling of this SPP module. Without sacrificing speed, this effectively expands the model's receptive field and fuses rich contextual information, thereby significantly enhancing the model's feature representation capability and ultimate object detection performance, making it particularly suitable for large objects and complex scenes. PANet Another improvement in the neck network is the adoption of the Path Aggregation Network (PANet). In the neck network of the YOLOv3 version, the traditional Feature Pyramid Network (FPN) structure adopted only contained a top-down feature fusion path. In other words, it only fused the low-resolution deep feature maps output at the end of the backbone network into the high-resolution shallow feature maps output in the middle stages. In this case, the high-resolution feature maps output from the shallow layers contained deep feature information, but the low-resolution feature maps output from the deep layers of the network did not contain shallow feature information. PANet, introduced into the neck network of YOLOv4, adds a bottom-up feature fusion path on top of the FPN structure. Such a structure allows feature information at different scales to be fused more thoroughly, ensuring that all outputs simultaneously contain both deep and shallow feature information. !image.png Detection Head The detection head of YOLOv4 is identical to that of YOLOv3, utilizing a convolutional layer at the end of each of the three output paths to simultaneously predict three parts of information: coordinates, objectness confidence, and class probabilities. Complete Network Structure of YOLOv4 Based on the detailed summary of the three parts of the network above, understanding the overall YOLOv4 network architecture shown below is quite straightforward: !image.png References - YOLO演進 — 3 — YOLOv4詳細介紹. 之前有介紹過 YOLOv1… | by 李謦伊 | 謦伊的閱讀筆記 | Medium - YOLOv4 中的 Mish 激活函数 - 腾讯云开发者社区 - 腾讯云 - YOLO Object Detection, Yang Jianhua, Li Ruifeng, Chapter 8: YOLOv4
Summary of YOLOv3 Evolution: Output, Anchors, and Loss Functions
This article summarizes the YOLOv3 model, focusing on the organization format of output data, anchor prediction boxes, and the calculation of loss functions. The Output of the YOLOv3 Model As shown in the overall architecture diagram of the YOLOv3 model below, unlike earlier YOLOv1 and YOLOv2 models which output only a single set of 3D tensors and learn/interpret object positions and categories from it, the YOLOv3 model ultimately outputs three independent sets of 3D tensors. These are used to locate and detect objects of different sizes, respectively. !image.png Taking the 13x13x255 3D tensor output in the figure above as an example, the input image is divided into a 13x13 grid. Each grid presets three anchor prediction boxes with different sizes and aspect ratios (for information regarding YOLOv3 anchor boxes, refer to the next section). Then, a 1D vector of length 255 is used to describe the object information detected in that grid (where the center of the object falls within that grid). In other words, this vector of length 255 contains information for 3 prediction boxes, as illustrated below. !image.png The information contained in each prediction box is identical to the output of a single prediction box in YOLOv2: center coordinate positions + width and height + prediction confidence + recognition information for 80 classes (for the COCO dataset). Thus, the information length for each prediction box is 4 + 1 + 80 = 85, and the information length for the 3 prediction boxes contained in a single grid is 255. This is the explanation and origin of the dimensions for the 13x13x255 3D tensor. > If each grid places anchor boxes and the model supports object classes to be detected, the output data length per grid will be . YOLOv3 Anchor Boxes Regarding the design of anchor boxes within the grid, YOLOv3 continues to adopt the prior anchor boxes introduced in the YOLOv2 model. However, while YOLOv2 places 5 anchor boxes in advance per grid, YOLOv3 places 3 anchor boxes per grid (as explained in the tensor output section above). As for the size selection of the anchor priors, YOLOv3 follows the same approach as YOLOv2, using the k-means clustering method on the training dataset to derive the sizes of 9 prior boxes matching that dataset. - For details on how to use the k-means clustering method to obtain dataset anchor priors, please refer to the article Detailed Explanation of the Anchor Mechanism in YOLOv2, which provides a very detailed description of the calculation pipeline for deriving anchor box aspect ratios from a dataset. In the YOLOv3 paper, the authors calculated the sizes of 9 anchor priors based on the COCO dataset. Sorted by area size and divided evenly into 3 parts, they target feature maps of different resolutions: - 3 large boxes, targeting 13×13 resolution feature maps (totaling 13x13x3 = 507 prediction boxes), responsible for large object detection: (116×90), (156×198), (373×326) - 3 medium boxes, targeting 26×26 resolution feature maps (totaling 19x19x3 = 2028 prediction boxes), responsible for medium object detection: (30×61), (62×45), (59×119) - 3 small boxes, targeting 52×52 resolution feature maps (totaling 52x52x3 = 8112 prediction boxes), responsible for small object detection: (10×13), (16×30), (33×23) !image.png YOLOv3 Loss Function Optimization As summarized above, the YOLOv3 model outputs three independent sets of 3D tensors to detect objects of varying sizes. Consequently, its loss function must be applied independently to each of these three outputs. The loss functions used to calculate loss values for the three outputs are completely identical, and their calculation processes run entirely independently without mutual interference. This yields three loss values: Loss_13, Loss_26, and Loss_52. The total loss of the entire model is simply the sum of these three loss values. Regardless of which scale is being evaluated, the YOLOv3 loss function consists of the following three components: - Bounding Box Regression Loss (lbox): Responsible for evaluating the accuracy of the prediction box position (center coordinates ) and size (width and height ). - Confidence Loss (lobj): Responsible for determining whether a target object is contained within the prediction box. - Classification Loss (lcls): Responsible for determining the specific class of the object within the prediction box. Therefore, the calculation pipeline for the overall model loss function is illustrated in the diagram below: !image.png Taking the 13x13 scale feature map as an example, we explain the calculation pipeline for each component of the loss function. First, let's define and explain the sample types during YOLOv3 training. Generally speaking, YOLOv3 categorizes samples into three types when calculating the loss function during training: positive samples, negative samples, and ignored samples. Different types of samples are either used or not used to calculate different types of losses. As mentioned earlier, for the 13x13 resolution model output, a total of 13x13x3 = 507 prediction boxes are outputted. The determination of positive, negative, and ignored samples essentially involves evaluating and classifying these 507 prediction boxes, and then deciding based on the classification results whether they will be used for subsequent loss calculations. - Positive Samples: For each ground truth box annotated in the training samples, the grid containing the center of the ground truth box is first identified. Then, among the three prediction boxes outputted by this grid, the one with the highest IOU with the ground truth box is selected as the positive sample. Thus, each ground truth box corresponds to one positive sample. Positive samples are used for all loss calculation types (bounding box, confidence, and classification). - Ignored Samples: Those whose IOU with any ground truth box is greater than a specified threshold (e.g., 0.5), but which are not the positive sample for that ground truth box. Ignored samples do not participate in any loss calculations and thus have no impact on the final loss value. - Negative Samples: Those whose IOU with all ground truth boxes is less than the specified threshold (e.g., 0.5), which essentially means all other samples excluding positive and ignored samples. Negative samples are used for confidence loss, but do not participate in bounding box and classification loss calculations. Similar to previous YOLO versions, the YOLOv3 loss function also contains three parts: coordinate loss, confidence loss, and class loss. For the calculation of coordinate loss, since only positive samples participate, the coordinate loss value is simply the loss between the coordinates of all positive samples and their corresponding ground truth boxes. Since the center coordinates, widths, and heights for all ground truth boxes and their corresponding positive samples are already known, calculating the coordinate loss is relatively straightforward: !image.png Next is the calculation of confidence loss. YOLOv3 uses Binary Cross-Entropy (BCE) as the calculation function for confidence loss. The model outputs a confidence logit value for each prediction box. This value must be activated via the Sigmoid function, mapping it to the interval to serve as the confidence score representing the probability of containing an object. The confidence loss for a single prediction box is then calculated using the following formula, and summing the confidence losses of all prediction boxes yields the total confidence loss: As mentioned previously, confidence loss calculations consider positive and negative samples, while ignoring ignored samples. - For positive samples (), the confidence loss simplifies to . This means that the closer the model's prediction is to 1, the smaller the loss. - For negative samples (), the loss simplifies to . This means that the closer the model's prediction is to 0, the smaller the loss. Finally, we have the calculation of classification loss. The logic for calculating classification loss is largely identical to that of confidence loss, both employing Binary Cross-Entropy (BCE). However, only positive samples participate in the calculation of classification loss. Similarly, the model outputs a raw logit vector containing information for all classes for each prediction box. The length of this vector equals the number of classes in the dataset (e.g., 80 classes for the COCO dataset). These raw logit values must be activated via the Sigmoid function, mapping them to the interval to serve as the parameter for classification loss calculation. - Note that YOLOv3 allows class predictions where a single box can belong to multiple classes simultaneously (e.g., both "person" and "man" labels). The classification loss for this positive sample across all classes is then calculated using the following formula, and summing the class losses of all positive samples yields the total classification loss: References - 【YOLO系列】YOLOv3论文超详细解读（翻译 ＋学习笔记）-阿里云开发者社区
An Interpretation of the YOLOv3 Model Network Architecture
This article provides a detailed summary of the overall architecture of the YOLOv3 model, including its backbone network Darknet-53, the concept of Feature Pyramid Networks (FPN), and the design principles and implementation theory of multi-scale feature map detection heads. The Brand-New Backbone Network: DarkNet-53 Starting from YOLOv2, the authors of the YOLO series designed a dedicated backbone network named Darknet. The backbone network used in YOLOv2 was DarkNet-19 (refer to Interpretation of YOLOv2 Model Network Architecture for the network architecture design of the YOLOv2 module), meaning the network structure contained a total of 19 convolutional layers. By the time of YOLOv3, the structural design of DarkNet drew inspiration from ResNet's residual network structure—which had already become a mainstream design philosophy in the industry—and introduced DarkNet-53 as the backbone network for YOLOv3. - For details regarding ResNet's residual network structure, please refer to the article Learning Classical CNN Network Model Architectures: ResNet. In the design of DarkNet-53, the convolutional layers still adopt the CBL combination structure used in Darknet-19: Convolution + BN (Batch Normalization) + LeakyReLU activation function. Darknet-53 contains 53 convolutional layers, from which it derives its name. !image.png When performing spatial downsampling on feature maps, unlike DarkNet-19 which uses max-pooling layers, DarkNet-53 universally adopts convolutional layers with a stride of 2 (stride=2). This approach reduces the feature map dimensions while preserving more spatial information. Therefore, the entire Darknet-53 network structure does not contain the MaxPool layers commonly found in traditional convolutional networks; all spatial downsampling operations on feature maps are handled by convolutions with a stride of 2. Furthermore, as mentioned above, the network design of Darknet-53 draws inspiration from the residual block design philosophy of ResNet. The residual block structure used in the Darknet-53 network is illustrated in the figure below. The input and output of the residual block are completely identical in feature map dimensions and channel counts. Internally, the block consists of a sequence of stacked and convolutional blocks, after which the output is added to the input to achieve a residual connection. !image.png Similar to ResNet, the majority of the Darknet-53 network is essentially a stack of residual blocks. The figure below compares the YOLOv2 backbone network (Darknet-19) and the YOLOv3 backbone network (Darknet-53). As can be seen, the downsampling operations in Darknet-19 all use traditional max-pooling layers, whereas those in Darknet-53 exclusively use convolutions with a stride of 2. !image.png The network structure of Darknet-53 is constructed by repeatedly stacking the residual blocks described above. Overall, based on the feature map dimensions at various layers, Darknet-53 can be divided into 5 parts (as circled on the right side of the figure above). The number of residual blocks contained in each of the five parts from first to last are 1, 2, 8, 8, and 4, respectively. The dimensions and channel counts of the input and output feature maps remain consistent within each part, and a convolution with a stride of 2 is used between parts to reduce the feature map dimensions. This "1-2-8-8-4" stacking configuration has also become one of the standard design paradigms for subsequent YOLO frameworks. Darknet-53 is fundamentally still a network for image feature extraction and category classification. Therefore, it ultimately still employs global average pooling combined with fully connected layers and Softmax to output recognition and classification results for the ImageNet dataset. Of course, when used as the backbone network for the YOLOv3 model, the global average pooling and subsequent layers are removed, retaining only the preceding feature extraction portion to output feature maps at different stages for object detection. Below is a comparison of the Darknet-53 network with architectures such as Darknet-19, ResNet-101, and ResNet-152 in terms of metrics like recognition accuracy and running speed. As can be seen, although the speed of Darknet-53 is lower than that of Darknet-19, it still achieves real-time performance, and its recognition accuracy is significantly improved. !image.png Feature Pyramids and Feature Fusion The Feature Pyramid Network (FPN) originates from another object detection network architecture, SSD. The underlying design philosophy is that as the neural network goes deeper and downsampling operations increase, the feature maps output at each layer contain different information: shallow feature maps have higher resolution and thus contain more positional information, but have smaller receptive fields due to fewer downsampling operations, making object category recognition more difficult; conversely, deep feature maps have lower resolution, where object positions and details are lost during progressive downsampling, but because their receptive fields are relatively larger, their performance in identifying and detecting large objects is better. Therefore, to address this inherent contradiction, a natural solution emerges: comprehensively combining shallow and deep features, where shallow features are responsible for detecting smaller objects, and deep features are responsible for detecting larger objects. - The larger the feature map resolution, the richer the positional details, but due to an excessively small receptive field, determining the object category becomes more difficult. - The smaller the feature map resolution, the greater the loss of object positional information during multi-level downsampling, but as the receptive field gradually increases, it becomes easier to determine the category of the target being detected (especially large-scale objects). The following figure illustrates the structure of a feature pyramid: the feature extraction network on the left outputs feature data of three resolutions and dimensions, while the detection network on the right performs detection and recognition for large, medium, and small objects respectively. !image.png To detect large-scale objects, it is sufficient to use only the feature information with the smallest resolution (the top-layer prediction output in the figure above). This is because for larger objects, although the positional information loss in low-resolution feature maps is relatively high, it is sufficient for localizing large objects. Moreover, low-resolution feature maps have larger receptive fields, making it easier to distinguish object categories, thus eliminating the need to rely on high-resolution feature information. For detecting smaller objects, however, one must use not only the higher-resolution feature information output by the feature extraction network (utilizing the rich positional information contained within to accurately localize small objects) but also the lower-resolution feature information (to assist in determining and localizing the object category). Therefore, it is necessary to fuse low-resolution feature information with high-resolution feature information to achieve accurate localization and category determination for small and medium-sized objects. The next question is: how are feature maps of different resolutions fused? The answer is to apply spatial upsampling (Upsample, i.e., up in the figure above) to the low-resolution feature maps output by the deep network to raise their resolution to match that of the shallow features, and then fuse them with the shallow-resolution feature maps. This process primarily involves two operations: spatial upsampling (Upsample) and feature map fusion (Concat). The fusion of the resolution feature map and the resolution feature map in the YOLOv3 model is used below to explain the Upsample and Concat operations: - Upsample operation: Based on the resolution feature map, interpolation is performed using _nearest-neighbor interpolation_. Every resolution feature map is upsampled via this interpolation method into a resolution feature map. - Concat operation: After being upsampled via the Upsample operation, the aforementioned feature map matches the resolution. This yields two sets of resolution feature maps, which are then simply concatenated end-to-end along the channel axis. The fusion processing and output workflow for feature maps of different resolutions are illustrated in the figure below: !image.png The YOLOv3 Architecture Diagram Based on the preceding understanding of the backbone network Darknet-53 and the multi-branch feature map output structure of the feature pyramid, learning the network structure of YOLOv3 becomes much easier: !image.png The top-left corner of the figure above shows Darknet-53, the backbone network of YOLOv3, with the global average pooling and classification fully connected layers removed. Feature maps downsampled by factors of 8, 16, and 32 are output from the third, fourth, and fifth residual block groups of the backbone network, respectively. - For an input image resolution of , the feature map resolutions output by these three parts are , , and , respectively. - For an input image resolution of , which was more common in object detection scenarios at the time, the resolutions of these three feature map branches are , , and , respectively. The right half of the figure above represents the feature detection portion based on FPN theory, taking feature maps of three different resolutions as inputs. Upsampling and Concat operations are performed at different stages, ultimately outputting detection networks with varying grid densities: denser grids are better suited for detecting small and dense objects, while sparser grids are better suited for detecting large and sparse objects. References - "YOLO Object Detection" by Jianhua Yang, Ruifeng Li, Chapter 7: YOLOv3 - YOLOv3 Detailed Interpretation. YOLOv3... | by Steven Meng | Medium - "Deep Learning PyTorch Object Detection in Action" by Hongyi Dong, 6.3 Multi-scale and Feature Fusion YOLOv3 - YOLOv3 — You Only Look Once (Object Detection) | by Moris | Computer Vision Note | Medium
Summary of YOLOv2 Features: Evolution from YOLOv1
This article provides a comprehensive summary of the features evolved in YOLOv2 based on YOLOv1, explaining the characteristics and working logic that improved various model metrics during its evolution. Built upon the design foundation of YOLOv1, YOLOv2 introduced multiple adjustments and optimizations in model design and training. Its mAP on the VOC2007 dataset increased dramatically from 63.4% to 78.6%, while retaining the one-stage design of YOLOv1 for object detection and class recognition. It maintains a very fast inference speed while being capable of recognizing more object categories. - For the concept and calculation logic of the mAP metric, please refer to Detailed Explanation of Evaluation Metrics for Object Detection Models: mAP, Recall, Precision. The following figure summarizes the complete list of improvements made in YOLOv2 compared to YOLOv1. Each of these items will be elaborated on in this article: !image.png Batch Norm YOLOv2 adds batch normalization after every convolutional layer in the model. Thus, the previous configuration of convolution + LeakyReLU activation function has been upgraded to a CBL block consisting of convolution + batch normalization + LeakyReLU activation function. Specific details are thoroughly described in the article Interpretation of YOLOv2 Model Network Architecture. !image.png With the addition of the BN layer, YOLOv2 achieved its first performance boost. On the VOC2007 test set, its mAP improved from the original 63.4% to 65.8%. Hi-res Classifier Because the ImageNet dataset contains a massive amount of annotated image data, most object detection algorithms first pre-train the backbone network based on the ImageNet dataset, using these pre-trained parameters as the initial parameters for the backbone network—a technique known as Image Pretraining. However, during the pre-training process based on the above technique, the input image size received by the backbone network is 224x224. For object detection applications, this small resolution easily leads to the loss of details for small objects, creating difficulties for subsequent detection stages. Therefore, in the design of YOLOv2, a strategy was adopted where the model is first pre-trained on the ImageNet dataset at a 224x224 resolution. Once pre-training is complete, a model fine-tuning stage is added: the network continues training at a 448x448 resolution using the pre-trained parameters to adapt to higher-resolution images. Specifically, the pre-training workflow of the YOLOv2 model is as follows: - First, train the network from scratch on the ImageNet dataset using a 224x224 resolution for approximately 160 epochs. - Then, adjust the model input to 448x448 and continue training on the ImageNet dataset for another 10 epochs. Under this strategy, YOLOv1's performance improved once again: mAP increased from 65.8% to 69.5%. > Why not train the network from scratch directly at the 448x448 resolution, but instead pre-train at a small resolution of 224x224 followed by fine-tuning at a large resolution of 448x448? On one hand, scaling from 224x224 to 448x448 quadruples the resolution, which would significantly prolong model training time given the computing power available at the time. On the other hand, a 224x224 resolution was the standard input for classification networks back then, with a wealth of pre-trained weights and hyperparameter experience available for reference. Therefore, starting with a mature configuration as a warm-up followed by minor fine-tuning is the most reliable approach from an engineering perspective. Fully Convolutional Network and Anchor Prior Box Mechanism In terms of network architecture design, the biggest change in YOLOv2 compared to YOLOv1 is the removal of fully connected layers at the end of the network, replacing them with a fully convolutional network structure. For the specific network structure, refer to Interpretation of YOLOv2 Model Network Architecture. In addition, drawing inspiration from the Faster R-CNN network, YOLOv2 introduces the Anchor prior box mechanism. During the image inference stage, the model only needs to learn the offsets required to map prior boxes to the dimensions of the target bounding boxes, rather than learning the complete dimension information of the entire target box from scratch, which makes training much easier. For a detailed summary of the Anchor prior box mechanism introduced in YOLOv2, refer to Detailed Explanation of the Anchor Mechanism in YOLOv2. Based on the optimization of the model's network structure and the introduction of Anchor boxes, the size of the feature map output by the backbone network is increased from YOLOv1's scale to YOLOv2's 13x13. Anchor boxes are placed in advance at each grid cell (in the YOLOv2 model, the default number of Anchor boxes is 5). This means the entire image can output bounding boxes, and the prediction information for each bounding box is completely independent. Each bounding box has its own center coordinates and , box width and height , prediction confidence, and recognition probabilities for various categories (in the YOLOv1 model, two bounding boxes in a network shared the same class recognition probability, so each bounding box could actually only detect one object). Therefore, the bounding boxes output by the same grid cell in YOLOv2 are mutually independent, which resolves the missed detection problem that occurs when the centers of two objects fall within the same grid cell. The differences in output information between the YOLOv1 and YOLOv2 models are illustrated in the figure below: !image.png Multi-Scale Training Following the modification of the network structure to remove fully connected layers and adopt a fully convolutional network structure—containing only convolutional layers and max-pooling layers—the input resolution of the entire network is no longer restricted to 416x416 images; images of various different resolutions can be used. Consequently, during the training process, the input image size can be continuously changed to adapt to different resolutions and varying sizes of objects to be detected. The backbone network (image feature extraction part) of the YOLOv2 model has a downsampling stride of 32, meaning the size of the feature map extracted by the feature extraction part is of the input image resolution. Therefore, the input resolution of the image needs to be a multiple of 32 (with a minimum size of 320x320 and a maximum size of 608x608). Thus, during the training process of the YOLOv2 model, every 10 epoch iterations, the model randomly selects a new image size from {320, 352, 384, 416, 448, 480, 512, 544, 576, 608} to serve as the image size for the subsequent 10 epochs of training, as shown in the figure below: !image.png Practice has proven that this Multi-Scale Training strategy allows the same network to perform detection with higher accuracy on images of different resolutions, thereby enhancing the detection performance of the network. References - YOLOv2 Detailed Explanation. YOLOv2 is improved based on YOLOv1 using the methods in the red box above, in... | by Steven Meng | Medium - "YOLO Object Detection" by Jianhua Yang, Ruifeng Li, Chapter 6: YOLOv2