All Articles
Notes on Setting Up the Rockchip RK3588 NPU Development Environment
This article details setting up the NPU development and on-board inference environment on the Rockchip platform based on RK3588 + Debian Linux, using the YOLOv5 model from the Model Zoo as an example to summarize the end-to-end deployment workflow. Rockchip NPU Application Development Framework The overall workflow for Rockchip NPU application development is largely consistent with the post-processing workflow of Junchen AI applications. First, models are developed and trained on a PC using PyTorch or TensorFlow. Once training is complete, the model (in ONNX or PT format) is converted into the RKNN format using the RKNN-Toolkit2 tool. This RKNN format is the edge model format capable of running on Rockchip chips, which is then invoked via C/C++ in the production environment for forward inference. The RKNN software architecture is illustrated in the diagram below: !image.png Relationship between RKNN-Toolkit2, RKNN-Toolkit-Lite2, and RKNPU2 C++ API: - RKNN-Toolkit2 is used to convert models trained on a PC or server into the RKNN format runnable on Rockchip NPUs. It runs on the PC side. - RKNN-Toolkit-Lite2 is a tool that allows forward inference based on RKNN-format models directly on Rockchip NPUs. It runs on the Rockchip processor. - RKNN-Toolkit-Lite2 is essentially a Python package running on the Rockchip processor, meaning the forward inference it executes runs within a Python environment on the Rockchip hardware. If higher performance is required, the RKNPU2 C++ API should be used for AI execution instead of RKNN-Toolkit-Lite2. - Therefore: RKNN-Toolkit2 is used for model conversion; RKNN-Toolkit-Lite2 is used for rapid verification and testing of converted models on the board; and the RKNPU2 C++ API is reserved for final project deployment. > Personal opinion: For on-board inference, the efficiency of C/C++ code is vastly superior to Python. Therefore, there is little practical necessity to use Python scripts with RKNN-Toolkit-Lite2 for inference verification on the board. The most mature and direct approach is to use RKNN-Toolkit2 for model conversion, followed by C/C++ inference directly on the board. Of course, the advantage of using RKNN-Toolkit-Lite2 Python scripts for on-board inference verification lies in the support of numerous Python packages and consistency with the PC environment, making it easier to write inference code. List of Chip Platforms Supported by RKNN-Toolkit2: - RK3588 Series - RK3576 Series - RK3566/RK3568 Series - RK3562 Series - RV1103/RV1106 - RV1103B/RV1106B - RV1126B - RK2118 As mentioned above, Rockchip NPU application development relies primarily on the RKNN ecosystem. Application development on Rockchip NPU chips relies mainly on the following two GitHub-maintained repositories: - RKNN-Toolkit2: airockchip/rknn-toolkit2, the model conversion tool. - RK Model Zoo: airockchip/rknn_model_zoo, support examples for running models on-board. Setting Up the RKNN-Toolkit2 Development Environment As of early 2026, the latest version of RKNN-Toolkit2 is 2.3.2. To develop and test applications within the Rockchip RKNN NPU ecosystem, you must first set up the RKNN-Toolkit2 development environment. Clone the RKNN-Toolkit2 and RKModel Zoo repositories from GitHub: Python Environment for Model Conversion Create a dedicated rknn environment (specifying Python version 3.12) and install RKNN-Toolkit2 along with all its dependency packages (rknn-toolkit2-master/rknn-toolkit2/packages/x86_64): > Note: Installing the Python 3.12 environment according to the steps above will result in an error when subsequently converting the YOLOv5 ONNX file to RKNN in the Model Zoo: AttributeError: module 'onnx' has no attribute 'mapping'. This occurs because the onnx library version installed in the Python environment is too high and incompatible with the current rknn-toolkit2 (v2.3.2). You should modify the onnx dependency version in requirements_cp312-2.3.2.txt to onnx==1.16.1 rather than installing the latest version of onnx. Cross-Compilation Environment The cross-compilation tool for board-side C/C++ code (RK3588 uses the aarch64 architecture, so install the 64-bit compilation tool): https://releases.linaro.org/components/toolchain/binaries/6.3-2017.05/aarch64-linux-gnu/gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu.tar.xz. This will be needed later when compiling C/C++ demos for on-board model inference. Board-Side NPU Driver Environment Furthermore, as shown in the RKNN software architecture diagram above, AI inference executed on the RK3588 board also requires support from the board-side NPU environment. This includes the kernel-level NPU driver, the Rockchip RKNN runtime libraries, and other supporting libraries. First, you must enable NPU driver support in the board's kernel and ensure that the kernel's NPU driver version matches the version of RKNN-Toolkit2 and its runtime libraries. You can check the NPU driver version in the kernel boot messages or via dmesg: !image.png In addition to the NPU driver support mentioned above, board-side applications executing inference on the NPU also require the runtime library librknnrt.so. librknnrt.so is located at rknn-toolkit2-master/rknpu2/runtime/Linux/librknn_api/aarch64. When compiling the C/C++ demo for board-side inference, you should link against this dynamic library and place it beforehand in the LD_LIBRARY_PATH of the board's system. Additionally, running the C/C++ demo for on-board model inference may require other multimedia and hardware acceleration libraries in addition to the NPU driver and the librknnrt.so runtime support library: - RGA (Rockchip Graphics Architecture): Used for hardware-accelerated image scaling, cropping, and format conversion (e.g., converting YUV frames from a camera to the RGB format required by the NPU). Requires the librga.so library file and its header files. - MPP (Media Process Platform): Used for hardware video encoding/decoding (H.264/H.265). Requires librockchip_mpp.so. At this point, the Python, C/C++ cross-compilation environment and board-side runtime environment for NPU development on the Rockchip platform are fully prepared. Next, we will use the YOLOv5 demo from the Model Zoo to summarize the environment and NPU development workflow. On-Board Testing Based on YOLOv5 in the Model Zoo The examples directory of the previously downloaded Model Zoo repository (rknn_model_zoo-main) contains multiple model samples already adapted for the Rockchip platform. After minor modifications and compilation, they can be run and tested on the board. The entire workflow can be divided into two steps: model conversion and board-side C/C++ demo compilation. Model Conversion Navigate to the examples/yolov5/model directory and run download_model.sh. The purpose of this script is to download a YOLOv5s ONNX model from a specified network address. Therefore, if you already have your own trained YOLOv5s model, you can completely substitute it here. Next, navigate to the examples/yolov5/python directory and run the convert.sh script to perform model conversion (.onnx -> .rknn). The convert.sh script specifies the dataset description file path DATASET_PATH and the model conversion output path DEFAULT_RKNN_PATH. You can modify these values if necessary. Then, run python convert.py ../model/yolov5s_relu.onnx rk3588 to execute model conversion. The converted model will be saved to the path specified by DEFAULT_RKNN_PATH (with the .rknn extension). At this point, the model file for on-board execution is successfully converted. C/C++ Inference Demo There are two build script files in the rknn_model_zoo-main directory: build-linux.sh and build-android.sh, used to compile C/C++ inference demos for Linux and Android environments on the board, respectively. Taking the RK3588 Debian Linux environment as an example, first modify build-linux.sh by adding the path to the cross-compilation toolchain at the very beginning: !image.png Then, invoke the build-linux.sh script to compile the YOLOv5 on-board inference demo program: ./build-linux.sh -t rk3588 -a aarch64 -d yolov5 -b Release. build-linux.sh will actually navigate to the examples/yolov5/cpp/ directory, locate the CMakeLists.txt file, and invoke cmake to perform the full compilation process. The parameter descriptions for build-linux.sh are as follows: Once build-linux.sh finishes execution, it will generate an install directory under the rknn_model_zoo-main root directory, containing all files required for the demo to run on the board: the demo executable, the runtime dependency library librknnrt.so, the board-side RKNN model file, and test image files. You simply need to package this install directory, upload it to the board, and execute the demo program:
A Comprehensive Guide to the Decoder Architecture and Calculation Workflow of the Transformer Model
This article provides a detailed summary of the overall network architecture of the Decoder part in the Transformer model, along with the data operations and flow processes within it. Previously, in the article A Comprehensive Guide to the Encoder Architecture and Calculation Workflow of the Transformer Model, we thoroughly organized the Encoder architecture of the Transformer. This article continues by examining and analyzing the Decoder structure and the complete calculation workflow of the Transformer. Since the model structure and calculation workflow of the Decoder are very similar to those of the Encoder, it is best to first review A Comprehensive Guide to the Encoder Architecture and Calculation Workflow of the Transformer Model and How to Understand Multi-Head Attention in the Transformer Architecture? to get a clear grasp of the self-attention probabilities and calculation workflow of the Transformer. The following diagram illustrates the complete network architecture of the Transformer Encoder-Decoder: - The left side shows the Transformer Encoder, composed of consecutive Encoder Blocks, which is used to convert input natural language text into vector-represented semantic information. - The right side shows the Transformer Decoder, composed of consecutive Decoder Blocks, which is used to generate natural language outputs (such as machine translation or text generation) based on the semantic vector representations output by the Encoder. !image.png Decoder Inputs and Outputs The input and output structures of the Transformer Decoder are shown in the diagram below. !image.png The Decoder input consists of two parts of data: - Lower Input (Outputs Shifted Right): Used to perform Masked Self-Attention calculation. Its content is the sequence of tokens already generated by the model, allowing the model to know "what I have already said." - Middle Input (Semantic vectors extracted by the Encoder): Used to perform Cross Attention calculation. Its input consists of the and matrices derived from the Encoder's semantic vectors, enabling the model to understand "what the source sentence means." The calculation logic is that the Decoder uses its own (what semantic information I am looking for) to match against the Encoder's and . The output of the Decoder is the "Output Probabilities" located at the top of the above diagram. The dimension of the output content is a vector equal to the vocabulary size of the model. After being processed by the final Linear layer and Softmax layer, this vector represents the probability distribution of each word in the model's vocabulary appearing as the "next output word." Typically, the index with the highest probability is selected, and the corresponding word is looked up in the dictionary to serve as the next output word. Outputs Shifted Right Input and Autoregressive Prediction As mentioned above, the content of the lower Decoder input (Outputs Shifted Right) is the sequence of tokens already generated by the model. How should this be understood during the inference process? In the initial state, the model has no output. At this point, the input content of the Outputs Shifted Right part is merely a start-of-sentence signal (Start of Sentence). Based on the Encoder's semantics and the start signal, the model performs a forward inference pass on the Decoder side to predict and generate the first word (e.g., Hello). Next, this newly generated word Hello is appended to the Decoder's input end. At this moment, the Decoder's input sequence becomes: [, Hello]. The model continues to perform forward inference based on this new input sequence to predict the second word, World. The second word is similarly appended to the input end, making the input sequence [, Hello, World], and so on, until the final output generated by the model is (End of Sentence), at which point inference stops. This is the complete workflow of how the Decoder performs forward inference to predict and generate words one by one. Because every time a word is generated, it is re-appended to the end of the input prediction sequence to infer the next word, this process is known as the autoregressive prediction process. Structure of the Decoder Block The Decoder part in the Transformer consists of multiple identical Decoder Blocks. Its primary function is to generate the final output sequence based on the semantic representation provided by the Encoder (such as the output of the Encoder part in the diagram above) and the previously generated output tokens (such as the Outputs Shifted Right part in the diagram above). !image.png Each Decoder Block consists of three main sub-layers: - Masked Multi-Head Attention: Similar to the encoder's self-attention mechanism, but its primary purpose is to prevent attending to future tokens to preserve the autoregressive property (ensuring no "cheating" occurs during generation). - Multi-Head Attention: This sub-layer allows the decoder to focus on relevant parts of the encoder's output representation. This enables the decoder to concentrate on pertinent segments of the input, which is crucial for tasks like translation. - Feed Forward: A feed-forward neural network that processes the combined output of the masked self-attention and encoder-decoder attention mechanisms. In terms of network structure and inference calculation workflow, the Multi-Head Attention module and the Feed Forward module are identical to their counterparts in the Encoder. The main differences lie in the Masked Multi-Head Attention module and the cross-attention mechanism utilizing the Encoder's output as the Decoder's input. Masked Multi-Head Attention From an engineering implementation perspective, Masked Self-Attention and standard Self-Attention are 90% identical in code structure and calculation workflow. The only difference lies in the field of view (or visibility range) of the two attention mechanisms: - Standard Self-Attention (e.g., used in the Encoder): When this module extracts semantics from a text sequence, it can simultaneously see all words from the beginning to the end. Therefore, it can understand each word based on the context of the entire sentence and extract its semantics. Thus, the field of view of the standard Self-Attention module is global. - Masked Self-Attention (used in the Decoder): When using the masked self-attention module, the output at the current word position can only reference words that have already been generated previously; it cannot reference words that appear after the current word position. In other words, the field of view of the masked self-attention module is historical—it can only see preceding words. The historical viewpoint maintained in the calculation of Masked Self-Attention is crucial for the training process. During the training phase, training samples contain complete, correct answer sequences for a given problem. To improve efficiency, the training process does not run the model times word by word like inference decoding; instead, the entire sequence is fed in all at once, and a single matrix operation is executed to obtain the answer sequence through one forward pass. If a standard self-attention module were used in this scenario, a problem would arise: during text generation by the decoder, preceding words would be able to see subsequent words. Consequently, words further down the sequence would influence those positioned earlier, which is equivalent to the model seeing the answers directly during training—a situation we naturally want to avoid. To prevent this, during the matrix operations of attention, a triangular mask matrix can be applied (as shown in the diagram below) to mathematically force the -th word to only see words from position to , blocking it from seeing subsequent words. This achieves parallelized training while perfectly simulating the causal law of not being able to see into the future. !image.png Accordingly, the calculation formula for the Masked Self-Attention module becomes: Where is a lower triangular matrix. Its logic is as follows: - For all positions where (i.e., words after the current position), set to . - For all positions where (i.e., current and preceding words), set to . > Note: During normal forward inference, the model actually executes one forward pass to generate a single token. In this case, during each forward pass, the text sequence passed to the Decoder module via outputs shifted right naturally contains no future tokens (because future tokens have not yet been generated at that moment). Thus, the mask mechanism is actually unnecessary. However, because the parameters in the Transformer are generated under masked conditions during training, removing this masked self-attention module during forward inference could cause a subtle shift in the numerical distribution of parameters, thereby affecting prediction accuracy. Therefore, even though the masking mechanism is practically no longer needed during the forward inference stage, this masked self-attention module must still be retained to ensure that the model's output during inference is largely consistent in accuracy with its output during training. Encoder Output and Cross-Attention In the implementation of the Encoder's Self-Attention mechanism, , , and all originate from the same input. For the Decoder's multi-head attention module, however, the inputs come from two different sources—this is known as the cross-attention mechanism: - Query () comes from the output of the previous layer of the Decoder, which is actually the output of the masked attention module. Its semantic connotation represents: "I have generated these words; what kind of context information do I need now to translate the next word?" - Key () and Value () come from the final output of the Encoder. Their semantic connotation signifies: "What are the features and meanings of each word in the source text (source language)?" Therefore, the cross-attention mechanism can be conceptualized as the Decoder holding queries () and searching for answers in the Encoder's knowledge base (). As shown in the diagram below, based on the output of the previous Decoder module, Queries are obtained via matrix multiplication with . Based on the Encoder's output, Keys and Values are obtained via matrix multiplication with and respectively. Then, using Queries, Keys, and Values, the standard attention calculation workflow is invoked (refer to How to Understand Multi-Head Attention in the Transformer Architecture?) to yield the computation results of the cross-attention module. !image.png Summary of the Decoder Workflow Finally, let us summarize the workflow of the entire Decoder section: !image.png The first step is inputting the token sequence generated by model inference from the Outputs Shifted Right end. This token sequence undergoes the same preprocessing steps as the Encoder: token vectorization (Embedding) and the addition of positional encoding information. The second step is the processing flow of the Masked Self-Attention module. Its input is the token vector sequence augmented with positional encoding information. This calculation workflow has been explained in great detail in the Masked Multi-Head Attention section above. The output of the masked self-attention module undergoes residual connection and layer normalization via standard Add & Norm. The third step is the processing flow of the Cross-Attention module. This calculation workflow has been detailed in the Encoder Output and Cross-Attention section above. Similarly, its output undergoes residual connection and layer normalization via standard Add & Norm. The fourth step is the Feed Forward network, composed of two fully connected neural network layers. Its role is to perform non-linear transformations and further feature extraction on the complex features captured after passing through the two attention mechanisms (Masked Self-Attention and Cross-Attention). Finally, residual connection and layer normalization are applied via Add & Norm. Steps two through four above constitute a single Decoder Block. The input and output data dimensions of each Decoder Block are identical. The entire Decoder section contains Decoder Blocks, where the output of the previous Decoder Block serves as the input to the next, until all Decoder Blocks have finished processing. The final part is the output structure's conversion layer, which consists of two components: a Linear layer and a Softmax layer. - Linear layer: A fully connected mapping layer used to project the abstract vectors extracted by the Decoder (e.g., 512-dimensional, 768-dimensional, etc.) into a massive space—specifically, the space of the full vocabulary supported by the model's output. If there are 30,000 words in the output dictionary, the output dimension of this layer is 30,000. - Softmax layer: Converts the raw numerical values (Logits) output by the linear layer into a probability distribution via Softmax, ultimately outputting a predicted probability vector. Each value in this vector represents the probability that the corresponding word in the dictionary becomes the next word. The system then selects the word with the highest probability based on this predicted probability vector as the output result, feeds it back to the bottom input end, and begins predicting the next word. References - Architecture and Working of Transformers in Deep Learning - GeeksforGeeks - How Transformers Work: A Detailed Exploration of Transformer Architecture | DataCamp - Cross-Attention Mechanism in Transformers - GeeksforGeeks
Detailed Explanation of Qwen3-VL Model Testing and vLLM Production Deployment Workflow
This article provides a brief overview of the Qwen3-VL vision-language model released by Alibaba in September 2025, and summarizes the workflow for local inference testing of its pretrained model and deployment using vLLM in a production environment. Introduction to the Qwen3-VL Model Alibaba launched the Qwen3-VL series of vision-language models in September 2025. Compared to its predecessors (such as Qwen2-VL), the release of Qwen3-VL marks a formal transition for multimodal large models from simple visual perception to a new stage of deep logical reasoning and autonomous agent interaction. Continuing Alibaba's open-source ethos, Qwen3-VL offers various model scales ranging from edge devices to the cloud. !image.png In terms of overall model architecture, Qwen3-VL continues Alibaba's consistent dual-track parallel strategy of Dense + MoE (Mixture of Experts), with parameter sizes ranging from 2B to 235B. - Dense: All parameters in this model are fully activated and participate in computation when processing each token. - Representative versions include Qwen3-VL-2B / 8B / 32B. - The advantage is the highest knowledge density under the same total parameter size, making deployment relatively simple. However, the disadvantage is that as the parameter size increases, computational overhead grows linearly, and inference becomes very slow. - Therefore, it is suitable for application scenarios where the parameter scale is not excessively large. - MoE: This model structure contains many expert sub-models. When processing specific inputs, a gating network dynamically selects a small subset of them to participate in computation, leaving most parameters inactive. - Representative versions include: Qwen3-VL-30B-A3B (indicating a total of 30B parameters with 3B activated); Qwen3-VL-235B-A22B (indicating a total of 235B parameters with 22B activated). - The advantage is possessing the knowledge base of an ultra-large-scale model while maintaining the inference speed and computational overhead of a small model. The disadvantage is that the VRAM requirement remains very high; although only a small fraction is activated during computation, all parameters must be loaded into VRAM during inference. In addition, Qwen3-VL introduces the Thinking version for the first time: when facing complex STEM problems, math questions, or deep causal analysis, the model no longer relies on fast thinking to provide immediate intuitive answers, but instead breaks down image details step-by-step through a Chain-of-Thought generated via Reinforcement Learning (RL). Therefore, the Qwen3-VL model supports two interaction paradigms: Instruct and Thinking. - Instruct (Instruction-tuned version): Fast thinking; directly generates and outputs the final result upon receiving instructions. - Its characteristics include low latency, fast generation speed, and concise output. - Applicable scenarios include daily conversations, simple image descriptions, basic OCR, real-time UI control, and other tasks with high real-time requirements. - Thinking (Reasoning-enhanced version): Slow thinking; before providing the final answer, the model first generates an invisible (or expandable) Chain-of-Thought (CoT). - Its characteristics include higher accuracy (especially in complex tasks such as STEM, mathematics, and logical reasoning), but it requires longer generation time and consumes more tokens. - Applicable scenarios include complex geometric math problems, logical reasoning, causal analysis of long videos, and scenarios requiring rigorous proof. Inference Testing of Qwen3-VL Below, we download the pretrained model files for the Qwen3-VL-4B-Instruct-FP8 version using Hugging Face Transformers and perform some simple inference tests locally. An inference test for OCR recognition on a handwritten text image is demonstrated below. First, import the necessary packages, use Transformers to download the pretrained model file list from the Hugging Face Hub to local storage, and load them: The above code will automatically download the pretrained model files to the directory pointed to by the HF-HOME environment variable. Since the model weight files are quite large, the download will take a considerable amount of time. The downloaded model files look like this: !image.png Every time Qwen3VLForConditionalGeneration and AutoProcessor start up and load model files, they compare the MD5 hashes of the locally downloaded model files against those in the Hugging Face Hub repository. This process can take a relatively long time. Therefore, once the initial model download is complete, you can use the following code to set the HF Offline mode, avoiding the validation process on every load: Write an interface function to perform OCR recognition on a given image: Pass an image to the interface function above and print the OCR recognition result: Here, an image of handwritten text is passed: !image.png The final OCR recognition result is as follows: As can be seen, even when the handwriting in the image is very sloppy, the model's recognition accuracy remains extremely high. vLLM Production Deployment of Qwen3-VL vLLM is currently the best-supported engine for the Qwen series. It can wrap Transformers models into a high-performance OpenAI-compatible interface, allowing subsequent access to the model directly via the HTTP protocol. First, install vllm and the qwen_vl_utils support package. Because running vLLM has specific requirements for PyTorch and CUDA versions, it is best to install and run vLLM in an independent environment: > Note: Installing vLLM in a Windows environment frequently throws configuration file errors, but this issue does not occur under Ubuntu. Since production deployments typically use Linux, I did not investigate this issue further. Once the vLLM package installation is complete, use the following command to launch the deployment and execution of the Qwen3-VL model in vLLM with a single command: Where: - --model: Specifies the directory where the model files are located. - --served-model-name: Specifies the model name used for subsequent HTTP API access. - --trust-remote-code: Must be included because Qwen3-VL contains custom visual processing logic. - --gpu-memory-utilization 0.75: Instructs vLLM to occupy 75% of the VRAM. vLLM pre-allocates the VRAM to manage PagedAttention. Lower this value if your GPU has limited VRAM. - --max-model-len: Sets the maximum context length. Decrease this value if VRAM is limited. - --limit-mm-per-prompt: Limits the maximum number of images allowed per conversation using JSON format. If you see the following log output, it means the Qwen3-VL model deployed in the vLLM environment is successfully running: Test the locally deployed model using the following code: !image.png For the image specified by the URL parameter above, the response received from the model is as follows, indicating that the model deployment test and validation were successful: References - Introduction to Qwen3-VL
Best Practices for Source Code Management in Open Source Software Projects
When performing secondary development or preliminary research on active and frequently updated open-source software on GitHub (such as Ultralytics), the most frustrating challenge is that upstream code in the source repository is constantly changing, while my own modifications during study and research are scattered everywhere. Eventually, I lose track of what I changed, leading to massive merge conflicts that are extremely inefficient to resolve. Based on online learning and research, this article presents a rational technical path to solve these problems. This workflow is a standard practice adopted by many teams during open-source project research. In short: _By tackling both Git version control and software project code architecture, we use Fork + Branch Isolation + Modular Intrusion to ensure high efficiency in open-source software learning and research_. This article uses the project qiuqiangkong/audioset_tagging_cnn on GitHub to summarize the specific practices of this entire workflow in detail. 1. Isolation of Git Repositories and Branch Architecture First, do not directly clone the original repository from GitHub and push your own modifications simply to the main branch. Instead, leverage Git's Fork and Upstream mechanisms. Fork the original repository while maintaining the relationship between the upstream and downstream repositories. Click the Fork button on the audioset_tagging_cnn project on GitHub to fork it into your own GitHub account. Then create an independent research branch based on its main master branch: !366088a7-9efe-4461-80ff-c4f630499595.png Establish strict branch usage conventions for subsequent study and research: - master branch: Never modify code on this branch! This branch is strictly used to synchronize the latest code from the official repository. - research branch: This is a dedicated workspace for preliminary research and learning. All code experiments and modifications are managed and maintained on this branch. Next, clone the forked audioset_tagging_cnn project from your GitHub account to your local machine: At this point, a crucial step remains: you need to bind the original official repository as the upstream for your master branch, so that the master branch can later pull the latest code from this upstream: 2. Workflow for Synchronizing Upstream Repository Updates Through the process above, the original repository qiuqiangkong/audioset_tagging_cnn has been set as the upstream repository path. Whenever the original repository is updated, you can use the following workflow to synchronize the updated code to your local master and research branches: In the future, whenever there are updates in the upstream original repository, running the above process periodically will merge the upstream updates into the master and research branches of your own repository. 3. Isolation of Directory Structures Even within the research branch, you should avoid casually throwing your test scripts or other files into the root directory or the official source code package. A more sensible approach is to create a folder with a special prefix in the project's root directory (for example, _my_research/; the underscore places it at the top of the directory tree for easy access), and place your test scripts, inference code, and custom configuration files uniformly inside this directory. As shown in the figure below, all files generated during your preliminary research and development should be placed collectively in this special folder to avoid interfering with the code and files in the original repository. !0591d6e3-cf31-4c02-8cb0-3f5d27365a92.png 4. Installing in Editable Mode (for Python Projects) In some Python open-source projects, debugging and development require installing and executing software packages within the current Python environment (for example, the Ultralytics project requires executing the yolo command in the terminal, which means installing the ultralytics package). Such projects generally contain a setup.py or pyproject.toml file in their root directory. To ensure that the code written in _my_research/ can directly reference the modules implemented in the official source code, and that any modifications you make to the official source take effect immediately, you should install the package in editable mode within your virtual environment: The difference between editable mode installation (pip install -e .) and standard installation (pip install .) is as follows: - When performing a standard installation, pip copies all Python files from the current source directory into the site-packages path of the system (or virtual environment). In this case, if you add a line of code print("Hello CV") to the original ultralytics/engine/trainer.py, running the code again will not print this message. This is because the system is executing the previously copied old code version. Therefore, during debugging, every time you modify the code, you must rerun the installation command for the changes to take effect, which is very cumbersome. - On the other hand, if you use the -e flag during package installation, pip does not directly copy the source files. Instead, it creates a shortcut (usually a .egg-link file) in the system's site-packages directory, pointing directly to the current source code directory. In this scenario, any modifications you make to the local source code will take effect immediately the next time you run the code as soon as you save them. There is no need to reinstall repeatedly, making the debugging process much more developer-friendly.
Implementing a Vision Transformer (ViT) Model from Scratch
Implementing a Vision Transformer (ViT) Model from Scratch This article focuses on the task of recognizing the simple MNIST handwritten digit dataset, completing the implementation, training, and validation of a minimal Vision Transformer model to build a comprehensive understanding of the ViT implementation workflow. The MNIST handwritten digit dataset is the simplest computer vision dataset. Implementing a Vision Transformer to recognize handwritten characters on MNIST is not overly difficult, and the data and computational requirements for model training are minimal. Therefore, training a ViT recognition model on the MNIST dataset serves as an excellent introductory experiment for learning Vision Transformers. !image.png Regarding the overall architecture of the Vision Transformer model, you can refer to another note: [[An Introduction to the Architecture of the Vision Transformer (ViT) Model]]. !image.png > _Note: Reference 1 provides a complete Vision Transformer model for recognizing the MNIST handwritten character set. This article is essentially a study record of the aforementioned material, supplemented by my own understanding of the implementation process._ As summarized in the note [[An Introduction to the Architecture of the Vision Transformer (ViT) Model]], the architecture and workflow of the ViT model can be roughly divided into the following stages: image patching, adding the Cls Token, positional encoding, the Transformer Encoder based on multi-head self-attention mechanisms, and the final classification head. Therefore, this article follows the above structure to implement a complete Vision Transformer architecture, training and testing it using the MNIST dataset. 1. Importing Necessary Packages - Import torchvision.transforms to resize the MNIST character set images, ensuring the input resolution is a multiple of the patch resolution, and to convert MNIST character set image data into Tensor format. - Use Adam, included in torch.optim, as the optimizer for the training process. - Automatically download the MNIST dataset from the official PyTorch website using torchvision.datasets.mnist. 2. Patch Embeddings This step implements slicing the input image into fixed-size patches and serializing them. As mentioned in [[An Introduction to the Architecture of the Vision Transformer (ViT) Model]], this functionality is typically implemented using a 2D convolution (Conv2d) where the stride equals the kernel size (kernel_size). The Conv2d operation above is actually a convolutional projection: - kernel_size=patch_size: Extracts each patch using a patch_size × patch_size kernel. - stride=patch_size: Sets the stride equal to the patch size, achieving non-overlapping slicing. - Output channels = d_model: Maps each patch into a d_model-dimensional vector, vectorizing each patch. The complete code for patch embedding is as follows: In the forward operation above, self.linear_project(x) converts the dimensions of an input image from (C, H, W) to (d_model, P_col, P_row); !image.png x.flatten(2) flattens all patches, yielding serialized patches, each with a dimension length of d_model. This is effectively the output result of the patch embedding operation; !image.png The final x = x.transpose operation simply swaps the output data dimensions to fit the standard input format of the Transformer architecture. !image.png 3. Cls Token and Positional Encoding The logic for prepending a Cls Token to the patch list of each image in a batch is relatively simple: create a new cls token with a length equal to the patch vector, and place this token at the very beginning of the patch vectors as the 0-th token. In addition to the Cls Token, positional encodings must be added to the patch vectors. Each positional encoding is unique to the position it represents, enabling the model to recognize the location of each patch. To add positional encodings to the patch vectors, they must share the same dimension, d_model. The formula for constructing the position vectors is as follows: !image.png Thus, broadly speaking, this step consists of two operations: prepending a Cls token to the patch vector list for each sample image in the batch, and adding positional encodings to all vectors. The complete code is as follows: 4. Implementation of Single-Head and Multi-Head Attention Modules The theory and calculation workflow of multi-head attention modules have already been thoroughly summarized and explained in the article [[How to Understand the Multi-Head Attention Mechanism in the Transformer Architecture?]], and will not be repeated here. The calculation formula for the single-head attention mechanism is as follows: You can combine the formula above with the comments in the code to understand the calculation logic and workflow of single-head attention, which is essentially a step-by-step implementation of the formula: Based on the single-head attention module above, implementing the multi-head attention module is relatively straightforward: compute multiple attention heads independently, concatenate their results, and finally pass them through a linear layer to further fuse and output the features from all single-head outputs: The workflow and structure of the single-head and multi-head attention modules are illustrated below: !image.png 5. Transformer Encoder Next is the implementation of the complete Transformer Encoder. The structure of the Transformer Encoder is shown below: !image.png Its corresponding code is: 6. The Complete Vision Transformer Model The architecture of the fully implemented Vision Transformer model is shown below: !image.png - Input images first undergo slicing, vectorization, and serialization in the PatchEmbedding module. - The serialized patch vectors are then supplemented with an extra Cls Token and position vectors in the PositionalEncoding module. - This data is then processed through consecutive TransformerEncoder modules, repeatedly extracting and fusing image features via multi-head attention modules. - Finally, it passes through a classification head and outputs classification probabilities via Softmax. The complete implementation of the Vision Transformer model is as follows: 7. Training and Testing Finally, train the model constructed above, and validate the trained model using the MNIST test dataset. Hyperparameter Settings The hyperparameter settings for training the model are as follows: Preparation of Training and Testing Datasets Training and testing use the MNIST dataset downloaded from the official PyTorch website. Use the following code to download the dataset and prepare the dataloaders: Model Training Next, train the parameters of the constructed model. The entire training execution flow is largely similar to the handwritten digit recognition training process based on Convolutional Neural Networks described in the article [[[Implementing Handwritten Digit Recognition with Convolutional Neural Networks Based on PyTorch]]]: After training for 15 epochs, the printed loss values show that the loss converges steadily downward: Model Testing Finally, use the trained model above to test on the MNIST test dataset: After the aforementioned 15 rounds of testing, the test accuracy of the model on the test set reached 93%. References - Building a Vision Transformer Model From Scratch | by Matt Nguyen | Toward Humanoids | Medium --- Attempting to understand and explain technical problems from the perspective of underlying principles: Audio/Video / Cameras / Smart Home / Bluetooth / WiFi / Wireless Communication / AI. Please follow the WeChat Official Account: Pavel Han.
A Beginner's Guide to the Vision Transformer (ViT) Model Architecture
This article provides a detailed summary of the Vision Transformer (ViT) model architecture and the complete image processing workflow within it, helping to build a foundational understanding of the ViT model. While the Transformer architecture has been widely adopted across various Large Language Models in Natural Language Processing (NLP), Google's milestone 2020 paper "An Image is Worth 16×16 Words" introduced the Transformer to computer vision. By leveraging Patch Embedding and self-attention mechanisms, it has profoundly challenged the traditional dominance of Convolutional Neural Networks (CNNs) in computer vision, signaling a unifying trend across all domains of artificial intelligence. To understand the ViT model architecture and how image data is processed within it, you first need a clear understanding of the Transformer architecture, especially its Encoder section. You can refer to another related note: [[一文彻底搞懂Transformer模型的Encoder结构与计算流程]]. Introduction to the ViT Architecture The Transformer architecture was originally designed for sequential data (1D), whereas images are high-dimensional (Width Height Channels). When feeding image data into a Transformer, the first challenge is converting high-dimensional image data into a 1D sequence that the Transformer can accept. The most intuitive approach would be to treat every individual pixel in the image as a token fed into the model, but this results in an exponential explosion in computational complexity. Therefore, ViT's solution is: patch the image and serialize the resulting patches as inputs to the Transformer model. In essence, the ViT model splits the input image into a sequence of fixed-size patches, serializes these patches, and applies the Transformer's self-attention mechanism to extract image features. This enables the model to capture long-range dependencies between different parts of the image without relying on CNN convolution operations. The figure below illustrates an image classification model based on the ViT architecture. As shown, the input image is divided into multiple fixed-size patches, which are then serialized and vectorized before being fed into the Transformer Encoder for feature extraction. Finally, classification results are output through an MLP Head. Structurally, compared to the Transformer Encoder used for natural language semantic extraction, the only major difference in ViT is the patch-slicing preprocessing applied to the image. !image.png The processing workflow of the ViT-based image classification model can be broken down into several steps, which are analyzed and described below. 1. Image Patching This step is conceptually similar to tokenization and embedding operations performed on text sentences and paragraphs in NLP pipelines. First, the image undergoes a patching operation. For example, if a 224 × 224 image is divided into fixed-size 16 × 16 blocks (where each block is a patch), the entire image will be split into patches. !image.png Since each input image typically contains three RGB channels, the dimensions of each sliced patch are . This patch remains a 3D tensor, whereas the Transformer's self-attention mechanism operates on fixed-length 1D vectors. Therefore, the next step requires flattening this tensor () into a 1D vector. In practical engineering implementations, slicing the input image and flattening each patch are typically performed using a 2D convolution layer whose stride equals its kernel size. For instance, with a patch size of and an output 1D vector length of 768 per flattened patch, you can use the following 2D convolution operation to execute both patching and flattening: Because the kernel size matches the patch size precisely, a single convolution step extracts the features of one patch, which yields higher computational efficiency on GPUs. Through these operations, an input image with a resolution of is divided into 196 patches, with each patch flattened and mapped into a 1D vector of length 768. 2. Adding the Cls Token Before further processing the 196 patch vectors obtained above, an additional classification vector of the same length (768), known as the Cls Token, is appended. This classification token does not represent any specific image patch; its sole task is to interact with the other 196 patches, allowing the final classification result for the entire image to be read from its output representation. Specifically, a learnable vector of shape (1, 768) is manually created and prepended to the front of the 196 patch vectors. Following this step, the input sequence length becomes vectors, with each vector having a length of 768. 3. Positional Encoding The subsequent process follows the same logical flow as natural language text sequences: positional encoding is added to the 197 vectors to preserve the spatial awareness of each patch's location within the original input image. Just as word positions are critical for understanding semantic meaning in natural language sequences, the spatial location of each patch is vital for extracting comprehensive image features. Consequently, explicit positional information must be added to these vector sequences before feeding them into the Transformer. The dimensions of the positional encoding matrix match the input matrix shape (e.g., ) precisely, and this step is accomplished by directly adding the positional embedding matrix to the preceding patch vector matrix. !image.png At this point, after undergoing the aforementioned processing steps, the input image data is fully prepared to enter the Transformer Encoder for self-attention computation. A resolution image has been transformed into a sequence of shape (197, 768) enhanced with positional encodings. 4. Transformer Encoder The processing logic and workflow within the Transformer Encoder are identical to those used for natural language text. The entire Transformer Encoder consists of a consecutive stack of identical Encoder Blocks: !image.png Each Transformer Block contains structures such as Multi-Head Attention modules, Feed-Forward Networks (FFN), and Residual Connections with Layer Normalization. The working logic and calculation workflow of the entire Transformer Encoder can be found in [[一文彻底搞懂Transformer模型的Encoder结构与计算流程]], while the calculation workflow of its Multi-Head Attention module can be referenced in [[如何理解Transformer架构中的多头注意力机制？]]. The input and output tensor dimensions of each Encoder remain unchanged. Therefore, after passing through all Encoder layers, the output is still a matrix of shape (197, 768). 5. MLP Head (Classification Head) As described above, the output of the Transformer Encoder module is a (197, 768) matrix containing the fully extracted feature information of the input image. For classification tasks, however, we only need to extract the CLS Token vector at index 0 (i.e., the extra vector appended during the second step). For image classification tasks, the role of the classification head is essentially information compression and mapping: translating the highly abstract 768-dimensional feature vector into class probability predictions. Therefore, the classification head of a ViT model typically consists of a simple Multi-Layer Perceptron (MLP). For instance, a standard classification head network structure comprises two linear layers sandwiching an activation function to enhance non-linear mapping capabilities, concluded with a Softmax layer to generate probabilities: - Slice and extract the vector at index 0 from the (197, 768) output matrix of the Transformer Encoder. - Linear layer (768 → 768) - Tanh or GELU activation function - Linear layer (768 → K), where is the number of classes - Softmax Testing a Pre-trained ViT Image Classification Model The following code uses Hugging Face's Transformers library to download a pre-trained image classification model based on the ViT architecture and performs forward inference to obtain the final result: Where: - ViTImageProcessor is used to preprocess the input image, including resizing and normalization, ultimately converting the raw image into a format acceptable to the ViT model. This preprocessing routine is somewhat analogous to the tokenization process applied to natural language text before Transformer processing. - ViTForImageClassification is the actual ViT classification model. It receives the image tensor output preprocessed by ViTImageProcessor and outputs the classification logits. Consequently, a Softmax function must ultimately be applied independently to obtain the classification probabilities. References - Vision Transformers (ViT) Tutorial: Architecture and Code Examples | DataCamp - Building a Vision Transformer Model From Scratch | by Matt Nguyen | Toward Humanoids | Medium - Vision Transformer (ViT) Architecture - GeeksforGeeks
How to Understand Multi-Head Attention in the Transformer Architecture?
Introduction to the Self-Attention Mechanism The self-attention mechanism in the Transformer model was first introduced in the 2017 paper "Attention is All You Need" by Google AI researchers. It quickly became the core concept of the Transformer model, which in turn forms the foundation of various Large Language Models (LLMs) and Vision Language Models (VLMs). The brilliance of the self-attention mechanism lies in granting the model a global field of view: it allows every pixel (or feature point) in an image to directly connect with every other pixel (ViT), and every word (Token) in a serialized natural language paragraph to directly connect with all other words in the entire paragraph (LLM). Consider the following example to illustrate this: !image.png When trying to understand this simple sentence, upon seeing the word "sat", our attention naturally focuses on the closely related words "cat" and "mat", while other words hold little significance for understanding "sat" and can basically be ignored. Therefore, so-called attention means: based on semantic or grammatical relationships, helping the model determine the relevance and closeness between other words in the text and the word currently being processed. The closer the relationship between two words, the higher their weight; otherwise, the lower their weight. Preprocessing Of course, before performing self-attention computations, the serialized text data must first be preprocessed. For text data, the preprocessing pipeline mainly consists of three parts: - Tokenization: Breaking down the paragraph and sentences semantically into individual, independent words (tokens). - Embedding: Vectorizing the tokens generated above. Roughly speaking, this is like looking up a dictionary to map each token to a fixed-length vector. - Positional Embedding: Additionally, a positional encoding of the same length is added to each token's vector. This positional encoding reflects the token's position within the entire paragraph or sentence. !image.png At this point, the serialized text of the entire paragraph has been converted into individual tokens, with each token corresponding to a fixed-length vector (e.g., 512, 768) enriched with positional encoding information. These vectors serve as the input data for the subsequent self-attention computation. Q, K, V Matrices The core design of the self-attention mechanism is transforming the input vector into three distinct roles: Query (), Key (), and Value (). We can use a search engine as an analogy for these three roles: - Query (): The "keywords" you type into the search box ("What am I looking for?"). - Key (): The "titles/tags" of videos in the database ("What features do I have to be matched against?"). - Value (): The actual "video content" corresponding to the search results ("What information can I provide if matched?"). From the perspective of mathematical operations, the first step of self-attention computation takes the vectors produced by the aforementioned serialized data as input, and multiplies them separately by three weight matrices to yield the vectors. !image.png Suppose the input paragraph is tokenized into 64 tokens, and the vector length for each token is 512. Then the dimension of the input data is , and the dimensions of the three weight matrices are all . Consequently, the resulting vectors have the same dimension as the input data, which remains . !image.png For the input vector of each token, the QSK calculations are performed separately to obtain three corresponding Q, S, and K vectors. Then, while processing each token, its similarity with other tokens (including itself) is computed through the dot product of its own vector and the vectors of other tokens, thereby measuring the correlation between two elements as . For example, taking the sentence above, we use the query vector of "sat" and compute it against the key vectors of every other word in the sentence (including "sat" itself). The result is a set of similarity scores (which are scalars) telling us how relevant each word is to "sat", and these scores are what we call attention scores. To prevent the calculated attention scores from suffering from vanishing or exploding gradients, they are divided by a scaling factor (where is the feature dimension, e.g., 512). Then, they are converted into a probability distribution via Softmax (where the sum of weights equals 1), and these weights are ultimately multiplied by to produce the final output. Through this complex calculation process, we finally obtain a new vector representation for the word "sat" in its current context. This new vector information comprehensively considers the semantics of all words in the entire sentence, though it is primarily derived from related tokens like "cat" and "mat". This calculation process is executed for every token in the sentence. The semantic extraction of each token simultaneously attends to all other tokens, and even to itself—which is why we call it the self-attention mechanism. !image.png How Do the Attention Formulas Reflect the Relationships of Q, K, and V Matrices? As mentioned earlier when explaining the vectors: - is the Query, representing the specific semantics the current token wants to query from all tokens contained in the paragraph. - is the Key, representing the list of features possessed by all tokens in the paragraph. - is the Value, representing the auxiliary semantics that other tokens can provide to the current token when and match. The computational formula for the QKV matrices is: So how do these two aspects establish a connection? In the formula above, the first step is to compute . When taking the dot product of the vector ("What am I looking for?") and the vector ("What features do I have?"), it essentially calculates: "How well does the requirement described by the current Query match the attributes provided by this Key?" From the perspective of matrix computation, let be the total number of tokens in the text paragraph, and be the vector length of each token (e.g., 512, 768). Then, the matrix corresponding to all tokens in the paragraph is of size , the transpose of () is of size , and the resulting multiplied matrix for all tokens has a size of (this result is actually the attention score matrix among all tokens). Each row represents the matching score table of a specific token against all tokens in the paragraph for precise semantic parsing. The scores calculated via the dot product above can be any real numbers and cannot be used directly to aggregate information. This is where Softmax comes in: Softmax amplifies high scores, suppresses low scores, and then normalizes them, transforming all scores into probabilities between 0 and 1 that sum up to 1. The final step is to multiply the attention score matrix processed above with the vectors of all tokens. Mathematically, this is a weighted summation process. Its output is a new feature vector filtered by weights (where the weights are the attention score matrix) that encapsulates the contextual information of the entire paragraph. > In summary, the essence of the vectors and their attention formula is to extract the precise semantics that the current token represents within the paragraph. The specific calculation process involves first computing the matching degree between the current token's vector and the vectors of all tokens in the paragraph (including the current token itself). The result reflects the relative importance (i.e., attention scores) of all tokens in the paragraph when parsing the precise semantics of the current token. Then, based on this importance, the vectors of all tokens are extracted, and their weighted sum yields the precise semantics of the current token in the current paragraph. With this, the logical concepts and complete computational flow of the self-attention mechanism have been explained. Why Multi-Head? Now let us consider a question: If we only use a single set of (single head), the current token can only extract semantics from a single dimension at a time. However, in complex text semantic processing, a single token element may possess multiple relationships across different dimensions. Example: "Xiao Ming borrowed a thick book from the library." When extracting the precise semantics of the word "book", multiple distinct dimensions are involved: - Grammatical aspect: Attending to "borrowed", identifying that "book" is the object of the verb. - Attributive aspect: Attending to "thick", extracting the physical characteristics of the "book". - Locational aspect: Attending to "library", extracting the scenario where the action takes place. _Therefore, if there is only a single head, the model is forced to encode grammatical, attributive, and locational scenario information simultaneously within the exact same set of weights. This dilutes the attention, making it difficult for the model to capture these multi-dimensional features accurately and concurrently. The introduction of the multi-head mechanism allows the model to independently learn these multi-dimensional relationships across different subspaces._ The operational and computational flow of multi-head attention is actually quite straightforward. The computation steps are identical to the single-head attention mechanism described above, except that the input feature vectors are split into multiple sub-vectors of equal length: - Split: The high-dimensional input feature vector (e.g., 512 dimensions) is split into multiple lower-dimensional sub-vectors (e.g., 8 heads, making each head 64 dimensions). - Parallelism: The 8 split heads independently and in parallel perform the aforementioned computations. - Concat: The computation outputs from each of the 8 heads (each 64 dimensions) are concatenated back together into a 512-dimensional long vector. - Linear: Finally, it is multiplied by a weight matrix to perform a final fusion processing of these multi-dimensional computation results. !image.png References - Understanding Multi-Head Attention in Transformers | DataCamp - What is Attention and Why Do LLMs and Transformers Need It? | DataCamp
A Comprehensive Guide to Understanding the Encoder Structure and Computation Workflow of the Transformer Model
The diagram below illustrates a standard Encoder-Only Transformer architecture: !image.png In general, the execution workflow of an Encoder-Only Transformer architecture proceeds in the following sequence: - Data preprocessing phase, which includes input data vectorization (Embedding) and the addition of positional encodings. - A sequential stack of Encoder Blocks. - The data output phase. For an Encoder-Only Transformer architecture, the data output phase is very straightforward—such as the Linear + Softmax output shown in the figure above. Therefore, the focus of this article will be on the computation workflows of the first two parts. 1. Data Preprocessing For Natural Language Processing (NLP) Transformer architectures, data must undergo at least the following steps before entering the Encoder and Decoder of the Transformer: - Tokenization: Breaking down a complete sentence or paragraph into independent semantic units, known as tokens. This step can be accomplished using currently available, highly efficient tokenization tools. - Embedding: Based on the tokenization results, each independent token is converted into a vector of a fixed length (e.g., 512, 768, etc.). After this vectorization step, each token corresponds to a vector of uniform length. - Taking a vector length of 768 as an example, if the text to be processed contains 100 tokens, it will be converted into a tensor of shape (100, 768) after this step. - Positional Encoding: Adding positional encoding to the vectorized tokens from the previous step. The length of the positional encoding matches the token's vector length, so each token's vector can be directly added to its corresponding positional encoding vector without changing the data dimensions. - Continuing with the previous (100, 768) vector example, the dimensions remain (100, 768) after adding the positional encodings. The diagram below illustrates the complete data preprocessing workflow for the text "How are You": !image.png Following the data preprocessing stage, a piece of text (split into tokens, where each token has a vector length of dimension) is transformed into a tensor of shape (N, dimension), ready for the Encoder processing stage. 2.1 Q, K, and V Matrices in Multi-Head Attention As is well known, the core of the Transformer is the Self-Attention mechanism, whose working logic computes the correlation between each element and all other elements in the input sequence. The key components of this self-attention mechanism are , , and . Self-attention can be conceptualized as an "information retrieval" process: - Query (Q): What am I looking for? (Features of the current patch/element) - Key (K): What do I have? (Feature tags of all other patches/elements) - Value (V): What can I provide? (The actual content information) Just looking at the definitions above might not feel entirely intuitive. How exactly do these , , and function within the data processing workflow? To state the conclusion directly: If the vector length of each token is dimension, then , , and are three independent weight parameter matrices of shape (dimension, dimension). These three weight matrices are the parameters to be trained during the training process. Assuming a token vector has a dimension of , then are three matrices of dimension . The computation process for , , and with respect to this token vector simply multiplies the token vector by each of the three parameter matrices, mapping the input token vector into a new space: As shown in the figure below: The text "Data visualization empowers users to" is split into 6 tokens, and the vector length for each word is 768. In this case, the dimension of each token is (1, 768), while the dimensions of the three weight matrices are each (768, 768). Combined, the dimension of the three matrices together is (768, 2304). !image.png In summary, during this step, the (1, dimension) vector corresponding to each token is transformed via the above computation into three independent (1, dimension) vectors, corresponding to the token's , , and vectors respectively. The combined dimension of these three sets of vectors is (1, 3 x dimension). 2.2 Attention Computation Workflow in Multi-Head Attention After the processing in the previous step, we obtain the , , and vectors for all words in a text sequence. The next computation step is to extract the corresponding vectors based on the and vectors. First, compute the dot product between the vectors and vectors for all tokens in the text sequence. This measures the degree of attention that the -th token pays to the -th token: The closer the vector directions of and , the higher the calculated score. After this computation (assuming the text contains tokens), the resulting score matrix has a dimension of (N, N). To prevent exploding gradients from the above computation and to convert the results into probabilities, two additional steps are required: - Scaling: Divide each score above by . Here, is a scaling factor that ensures the model remains stable during training even when stacked deeply. For a designed Transformer model, is a constant, typically equal to the data dimension divided by the number of heads. - Softmax: Transform the score matrix into a weight matrix ranging between and . The computation workflow for these two steps is highlighted by the red box in the diagram below: The matrix processed by softmax still has a dimension of (N, N). In the illustration below, the text contains only 6 tokens, so this matrix has a dimension of (6, 6). !image.png With the weight matrix obtained above, we can use it to aggregate all the Values (i.e., the matrix): As shown in the red box of the diagram below: Assuming the text contains tokens and each token has a vector length of dimension, this computation is the product of an (N x N) dimensional matrix (the weight matrix calculated from and ) and an (N x dimension) dimensional matrix. The final output structure remains (N x dimension). !image.png At this point, the computation workflow for the self-attention , , and matrices in the Encoder module is complete. The entire Encoder computation takes data of shape (N, dimension) as input and yields an output of shape (N, dimension), meaning the data dimensions remain unchanged. Wait! There is another question: Earlier, during the Embedding transformation process for individual tokens, it was explicitly mentioned that each token's vector length is 768. However, during the final matrix operation above, it is clearly visible that the dimension of the matrix is (6, 64) instead of (6, 768), and the final output is also (6, 64) rather than (6, 768). This brings up the concept of Multi-Head. 2.3 Multi-Head in Multi-Head Attention Let us examine the complete block diagram of the Multi-Head Self-Attention Block: On the left are the , , and matrices for all input tokens, with a data dimension of (N, 3 x dimension); on the right is the processing result of this multi-head attention module, with a dimension of (N, dimension). !image.png The red-boxed notation below indicates that the operations of the entire attention module are divided into 12 independent, parallel computing modules executed simultaneously, after which the results are merged. This is what is known as multi-head attention. For example, in the figure above, there are 6 tokens, and the data dimension for each token is 768. Therefore, for the entire self-attention module, the input consists of three independent (6, 768) matrices—namely the respective , , and vectors for the 6 tokens. In the multi-head processing setup, the computation workflow splits into 12 heads, dividing the input data into 12 independent groups. For each token's , , and vectors, each vector of length 768 is split into 12 segments, where each segment has a length of (which explains the 64 appearing in the matrix calculation above). Thus, for the computation of each head, the input data dimension becomes three independent (6, 64) matrices (for the , , and vectors), the output data dimension for a single head is also (6, 64), and all head outputs are concatenated back together to form (6, 768). > This is the essence of multi-head attention: splitting the input , , and vectors of each token into equal parts, processing these parts simultaneously, and then merging them. The final input and output data dimensions remain consistent. You can refer to the code snippet below for a deeper understanding of the working mechanism of the multi-head self-attention module: 3.0 Feed Forward The feed-forward network in the Encoder module is relatively simple. It is essentially implemented by sandwiching a non-linear activation function between two linear layers to achieve non-linear transformation of the output data, while keeping the input and output data dimensions as (N, dimension): 4.0 The Complete Encoder Block The block diagram of a complete Encoder Block is shown below: !image.png As can be seen, an independent encoder block is essentially a multi-head attention module with its residual connection, followed by a feed-forward network module with its residual connection. Given our understanding of the multi-head attention and feed-forward modules above, comprehending the execution workflow of the encoder block below becomes quite straightforward: At this point, the complete computation workflow for a single Encoder Block is established. Note that for a single encoder block, both the input and output data dimensions are identical—namely (N, dimension)—making it easy to stack multiple encoder blocks sequentially to form the complete Transformer architecture. 5.0 The Complete Transformer Encoder Architecture Revisiting the complete Transformer encoder architecture, it becomes very easy to understand: - First is the data preprocessing part. Input text and other sequence data undergo tokenization and vectorization to convert each token into a fixed-length vector, which is then combined with positional encoding to produce position-encoded vectors of uniform dimension before being fed into the Encoder. The data dimension is (N, dimension). - Next is the core of the Transformer Encoder architecture: a sequential stack of multiple Encoder Blocks, where the input and output data dimensions of each encoder block are both (N, dimension). - Finally, there is the output layer, which is tailored specifically according to application requirements. !image.png References - Transformer Explainer: LLM Transformer Model Visually Explained - How Transformers Work: A Detailed Exploration of Transformer Architecture | DataCamp - Transformer Architecture : Part 1- Encoder | by Abhishek Jain | Medium - Working of Encoders in Transformers - GeeksforGeeks
Summary of YOLOv5 Model Loss Function Calculation
Based on the v6.0/v6.1 version of the YOLOv5 model, this article provides a detailed summary and explanation of the calculation workflow for each part of the loss function. The loss function of the YOLOv5 model consists of three components: - Bounding box localization loss (Box Loss) - Object confidence loss (Obj Loss) - Classification loss (Cls Loss) Among them, the bounding box regression loss adopts CIoU (Complete IoU) loss, while both the object confidence and classification losses use Binary Cross-Entropy (BCE) loss. Object Confidence Loss (Obj Loss) The object confidence loss (Obj Loss) in YOLOv5 is the only loss term that involves both positive and negative (background) samples in its calculation. Its core task is to teach the model to answer the question: "Is there an object in this grid cell? If so, how reliable is this predicted box?" In addition, during the loss calculation process, within the positive sample list returned by build_targets, the confidence label values for all positive samples are not fixed to 1. Instead, they are set to the IoU between the predicted box (acting as a positive sample) and the ground truth box. In other words, the higher the overlap between the positive sample and the ground truth box, the higher its confidence label value. This is the soft label strategy adopted in YOLOv5's loss calculation: - Target () for all negative samples = 0 - Target () for all positive samples = CIoU Score The calculation of the overall object confidence loss can be divided into four steps. 1. Preprocessing of Predicted Values () The forward inference process of the YOLOv5 model outputs prediction tensors at three feature layers (P3, P4, P5). For the confidence channel, the inference output consists of unactivated raw values (logits). Because binary cross-entropy (BCE) is used to compute the confidence loss, the logits output by the model must first be processed through a Sigmoid activation function to map the values into the interval. 2. Constructing Target Values () As summarized in the article Summary of Background Information for YOLOv5 Model Loss Function, the build_targets function within the ComputeLoss class (responsible for calculating the YOLOv5 model loss function) constructs and returns a list of positive samples based on the list of ground truth boxes and the anchor lists of each detection layer. To calculate the object confidence loss here, target values need to be constructed: - First, initialize a zero-tensor with the exact same shape as the predicted feature map. - Then, iterate through all positive samples returned by build_targets: - Calculate the CIoU value between the current predicted box (acting as a positive sample) and the ground truth box (GT Box). - Fill this CIoU value into the corresponding coordinate position in the zero-tensor. - The resulting is a target tensor where most positions are 0 (background) and a few positions are floating-point numbers between (foreground IoU). 3. Calculating Binary Cross-Entropy (BCE Loss) With the predicted values and target values established, the binary cross-entropy loss can be calculated. YOLOv5 uses PyTorch's BCEWithLogitsLoss. For a single predicted box, the calculation formula is: - For negative samples (): The first half vanishes, optimizing only the second half and forcing to approach 0. - For positive samples (): Both halves are present, forcing to approach the CIoU value. Furthermore, as described in Summary of Background Information for YOLOv5 Model Loss Function, if the Focal Loss feature is enabled via hyperparameters, Focal Loss processing must also be applied. The purpose is to suppress the loss from negative background boxes and elevate the loss weight of positive samples. The formula above calculates the confidence loss for a single predicted box. The confidence loss for an entire detection layer is the sum of the confidence losses of all predicted boxes output by that layer: Where is the Batch Size, is the grid size, and is the number of anchors set per grid cell (default is 3). 4. Layer Balancing Because small targets contain fewer feature details, suffer from more background interference, and are harder to regress, the model requires stronger gradient signals to learn how to detect them. Therefore, when summing the confidence losses of the three detection layers to obtain the total confidence loss, weighted summation is applied across layers: a higher weight is multiplied to the P3 layer (which outputs small-target feature information), and a smaller weight is multiplied to the P5 layer (which outputs large-target feature information). This forces the model to pay more attention to prediction results on high-resolution feature maps (P3), thereby improving the recall and accuracy of small targets. To emphasize once again: only in the calculation workflow of the confidence loss are negative samples taken into consideration. The classification loss and bounding box localization loss detailed below do not consider negative samples. Classification Loss (Cls Loss) The goal of classification loss is crystal clear: For predicted boxes identified as positive samples, they must correctly recognize which category the object belongs to. The calculation workflow of the classification loss has the following two key characteristics: - Only calculates positive samples: Similar to the bounding box localization loss , only positive samples assigned with ground truth boxes (i.e., the positive sample list returned by build_targets) participate in the calculation of the classification loss. The classification loss for the more than 20,000 negative sample detection boxes (i.e., background boxes) output by model inference is zero and does not participate in gradient backpropagation. - Multi-class support: Its goal is to correctly identify the specific category of the target from all possible classes. In YOLOv5, the classification loss also adopts standard binary cross-entropy (BCE) loss, but with clever engineering implementations that allow it to be applied to multi-class classification. Its detailed calculation workflow contains four steps. Preprocessing of Predicted Values () Similar to the confidence loss calculation workflow, the first step is to locate positive samples. Based on the output tensor from the model's forward inference and the preset anchor box lists for each prediction layer, the build_targets function is called to extract the list of all positive samples. Subsequent calculations of the classification loss are performed exclusively on these positive samples. To perform BCE calculations, the class prediction logits contained within the positive sample list returned by build_targets are extracted, and Sigmoid activation is applied to these logits. For each predicted box, the model outputs a -dimensional vector, where is the total number of classes (e.g., for the COCO dataset). The predicted values output for each class are passed through Sigmoid activation, independently mapping each class score into the interval to obtain the predicted probability for each class. Constructing Target Values () Construct a target tensor with the same dimensions as the positive sample . Its dimension is . Assign values to the labels within it: - For each positive sample, find the true class annotated by its corresponding ground truth box, assign 1 to the -th position of the vector, and assign 0 to all other positions. The final result is a set of sparse one-hot encodings serving as the target values for the subsequent BCE calculation. In the loss calculation workflow of YOLOv5, to prevent the model from overfitting to the labels, a technique called label smoothing is also used during training. Roughly speaking, this slightly decreases the target value set to for the true class and slightly increases the target values set to . - For example, for a certain positive sample's class label settings, instead of setting , it is set to . The purpose is to encourage the model's predicted probabilities not to be overly absolute (such as 1.0), thereby enhancing the model's generalization ability and robustness. Calculating Binary Cross-Entropy (BCE Loss) relies on BCE, but BCE is calculated independently for each class, followed by summing the losses across all classes. Assume is the index of the positive sample, is the class index, is the probability predicted by the model that the -th sample belongs to class , and is the ground truth label (0 or 1). Then the loss formula for a single class of a single sample is: Since is only 1 at the true class and 0 at the other classes, this calculation can be simplified to: - For the true class : The loss is , forcing the model to increase the predicted probability for the correct class. - For incorrect classes : The loss is , forcing the model to decrease the predicted probabilities for all incorrect classes. Aggregation of Total Classification Loss Finally, the losses of all positive samples across all classes are aggregated (summing the losses across the classes for each positive sample) to obtain the final . Bounding Box Localization Loss (Box Loss) In YOLOv5, the localization loss adopts CIoU Loss (Complete IoU Loss). Its core objective is: For predicted boxes identified as positive samples, accurately adjust their position, size, and aspect ratio so that they overlap with the ground truth (GT) box as much as possible. Compared to standard IoU metrics, CIoU Loss not only focuses on the overlapping area between the predicted box and the ground truth box, but also introduces penalty terms for center point distance and aspect ratio consistency. This provides more comprehensive and stable gradient signals, greatly improving the model's localization accuracy. The complete definition of CIoU Loss is: Where is the penalty term of CIoU, composed of the center point distance term and the aspect ratio consistency term. The calculation workflow of is described below. Filtering Positive Samples and Decoding This step is identical to the classification loss calculation above. First, locate the positive samples—meaning extracting the corresponding predicted boxes (logits) solely from the list of positive samples generated by the build_targets function, and then decoding the offset values output by the model into true normalized coordinates . Similar to the classification loss, the calculation of the localization loss only considers the positive sample list returned by build_targets, ignoring other background negative samples. Calculating IoU and CIoU Based on each positive sample's predicted box, calculate the IoU between this predicted box and the ground truth box, as well as the distance and aspect ratio penalty terms for the CIoU calculation workflow. Finally, substitute all terms into the CIoU Loss formula to compute the CIoU loss for each positive sample. Aggregating Target Bounding Box Loss Values Sum up (and typically average) the values of all positive samples to obtain the total bounding box localization loss . Total Loss After the independent calculation of the above three components, the total bounding box localization loss , total object confidence loss , and total classification loss for a batch are obtained. The overall loss is the weighted sum of these three losses: Where , , and correspond to the weighting weights of the three losses respectively, which are set in the hyperparameter file, corresponding to the box, obj, and cls settings in the hyperparameter file: At this point, the complete loss value for the inference output of a batch of images during the training process has been successfully calculated.
Background Knowledge Summary of the YOLOv5 Loss Function
Based on versions v6.0/v6.1 of the YOLOv5 model, this article provides a detailed summary and explanation of the loss function calculation workflow and the background knowledge involved in its code implementation. As shown in the structure of the YOLOv5 detection head below, similar to previous generations, the bounding box data output by each grid in the YOLOv5 model contains 5 + NC information divided into three parts: the coordinate positions of the predicted bounding box, the detection confidence, and the class probabilities. !image.png Accordingly, the loss function consists of three components: - Bounding box regression loss (Box Loss) - Object confidence loss (Obj Loss) - Classification loss (Cls Loss) Among them, the bounding box regression loss uses CIoU (Complete IoU) loss, while both the object confidence loss and the classification loss use Binary Cross-Entropy (BCE) loss. Binary Cross-Entropy (BCE) First, let's explain the calculation workflow of BCE using the simplest binary classification problem. Assuming the ground truth label of the training data is (0 or 1), and the model's prediction is (a probability value between 0 and 1), the formula for the BCE loss function is: Because taking the logarithm of a decimal between 0 and 1 always yields a result less than 0, the output of the BCE formula above is always positive. For , when , ; when is between 0 and 1, is negative and decreases as approaches 0; when approaches 0, tends toward negative infinity. Therefore: - When the label , the loss function simplifies to . In this case, the higher the predicted probability , the smaller the loss value, getting closer to 0. - When the label , the loss simplifies to . In this case, the lower the predicted probability , the smaller the loss value, getting closer to 0. As shown above, the predicted value used in cross-entropy calculation should be between 0 and 1. Therefore, for the logits output by the three detection heads of the YOLOv5 model, a sigmoid function is applied to the logits before computing the BCE loss. In fact, the YOLOv5 source code uses the BCEWithLogitsLoss function for both object confidence loss and classification loss: The implementation of the BCEWithLogitsLoss function actually corresponds to the following formula: where is the raw logit output by the model, is the sigmoid activation function, and is the target value. Weight Control in Positive Sample Loss Calculation It is worth noting that the BCEWithLogitsLoss function definition above passes two pos_weight parameters via hyperparameters: cls_pw and obj_pw. By default, both parameters are set to 1.0 in the hyperparameter configuration file. In this case, the loss for positive samples () in the BCE formula is multiplied by , while the loss for negative samples () remains unchanged. The actual equivalent formula becomes: Therefore, if you need to adjust the proportion of positive sample loss during model training for class loss and confidence loss, you can tune the cls_pw and obj_pw parameters. Setting these parameters to values greater than 1.0 will strengthen the training penalty for positive samples; otherwise, it will weaken it. - For example, if a high miss rate or lower prediction confidence for positive samples is observed during training, you can try setting obj_pw > 1.0 (e.g., ) to strengthen the positive sample penalty. Conversely, if there are too many false positives (i.e., background incorrectly predicted as positive samples), you can slightly decrease obj_pw (). Construction of the Positive Sample List for Loss Calculation As summarized in the article YOLOv5 Model Prediction Output and Its NMS Algorithm Analysis, YOLOv5 outputs over 20,000 prediction boxes for a single image during inference, the vast majority of which are invalid. To compute the loss during training, we must first find a list of positive samples that closely match the ground truth boxes from these 20,000+ output prediction boxes. This is precisely what the build_targets function does in the YOLOv5 loss calculation code. In summary: The build_targets function is responsible for matching the ground truth boxes of training samples with the model's anchors, thereby filtering out the anchor positions from the model's output tensors that are responsible for detecting specific ground truth boxes. These matched anchor positions serve as positive samples, which are subsequently used to compute the Box Loss, Objectness Loss, and Class Loss. Negative samples (i.e., background) retain a default object confidence value of 0. The construction of the positive sample list can be divided into three steps: anchor matching, grid assignment with neighborhood expansion, and positive sample list generation. Anchor Matching For each ground truth box labeled in the training samples, YOLOv5 uses a shape-matching logic to find suitable positive prediction samples from the prediction boxes output by the three detection layers: for the output of any detection layer, as long as the anchor's shape (i.e., aspect ratio) does not differ too much from the ground truth (GT) shape, that anchor can be used to predict that GT. As a result, a single GT box can be predicted by anchors of multiple scales simultaneously (as long as the shape meets the detection criteria). Specifically for each detection layer, the width and height configurations of the anchor boxes for each level are pre-defined via the anchors parameter in the model configuration file. Each grid cell in every detection layer utilizes three anchor boxes to detect objects in the image according to the aspect ratios listed in the table below: This step processes the three prediction feature layers (P3, P4, and P5) independently. For each GT box, the system calculates its width and height matching status against the three pre-defined anchors on the current feature layer using the following logic. Assuming are the width and height of the GT box, and are the width and height of the anchor, the ratios and are calculated using the following formulas: To determine whether the shapes match, the following condition is checked, where anchor_t is a hyperparameter with a default value of 4.0: The implication of this ground truth-to-anchor matching condition is: if the side length of the GT is between 1/4 and 4 times the side length of the anchor, we consider the anchor to "resemble" the GT, making it a potential match. Otherwise, the anchor is discarded and treated as negative background. Grid Assignment and Neighborhood Expansion After determining which anchor shapes are suitable using the logic above, the next step is to determine which grid cell on the feature map of that layer is responsible for prediction. First, the center point of the GT box is located by calculating its coordinates on the current feature map. Then, taking the floor of these coordinates yields the grid coordinates where the GT center resides: . Additionally, beyond the grid determined by center localization alone, YOLOv5 incorporates a key innovation known as cross-grid neighborhood expansion. That is, after locating the feature map grid containing the GT center, YOLOv5 not only designates this grid as a positive sample, but also includes the four nearest neighboring grid cells (directly above, below, left, and right of the center grid) as positive sample candidates. Which adjacent grids are ultimately selected as positive samples depends on the offset position of the center point within the grid. For example, if the center point is located in the top-left corner of the grid, the current grid + left neighbor + top neighbor will be output as positive samples corresponding to that GT; if the center point is located in the bottom-right corner, the current grid + right neighbor + bottom neighbor will be output. Therefore, generally speaking, if an anchor box's aspect ratio matches a GT box on a specific detection layer, the system will generate 3 positive prediction boxes for that GT box on that layer. Similarly, if a GT box finds two anchor boxes matching its aspect ratio on the current detection layer, the system will generate 6 positive prediction boxes for that GT box on that layer. Generation of the Positive Sample List By performing matching and expansion for each GT box across all three detection layers using the logic above, the complete positive sample list is finally obtained. The build_targets function ultimately outputs the following four parameters for subsequent loss calculation: - tcls: Contains the class indices of the ground truth boxes corresponding to all positive samples. - tbox: Contains the bounding box coordinate information of the ground truth boxes corresponding to all positive samples. - indices: Provides the indices of all positive samples within the model prediction tensor pi, used to precisely extract positive sample predictions from pi (which has a shape of [bs, na, grid_h, grid_w, 5+nc]). - anch: Contains the widths and heights of the matching anchors corresponding to all positive samples. As we can see, for an input image of size , the model's three detection layers output a total of over 20,000 anchors per image during inference. However, even after the neighborhood expansion described above, the final generated positive samples typically number only a dozen or a few tens. In other words, positive samples account for approximately , while negative samples (background) account for over . This is the severe class imbalance problem that all object detection models must face. The Sample Imbalance Problem and Countermeasures To address issues such as the aforementioned positive-negative sample imbalance, training class imbalance, and easy-hard sample imbalance, the YOLOv5 model provides multiple mechanisms. Negative Samples Participate Only in Confidence Loss Calculation As mentioned above, the YOLOv5 loss function consists of three parts: Box Loss (localization), Cls Loss (classification), and Obj Loss (confidence). The total loss is a weighted sum of these three components. Not all loss functions are affected by negative samples during the loss calculation. Box Loss and Cls Loss are computed exclusively using positive samples! Those 20,000+ negative samples (i.e., background) do not participate in coordinate regression or class classification loss calculations at all. The effect is that, regardless of how many negative samples there are, they do not interfere with the model learning "where the object is" and "what the object is." Objectness Loss is the only loss component in which both positive and negative samples participate. For negative samples, the target object confidence is 0, while for positive samples, the target is the CIoU value (typically between 0 and 1). Therefore, it is only in the confidence calculation that we need to combat positive-negative sample imbalance. "Soft Labeling" of Positive Samples (IoU-Awareness) In early YOLO versions, the confidence label for all positive samples was simply 1. In YOLOv5, however, the objectness target for positive samples is set to the CIoU value between the predicted box at that location and the ground truth (GT) box: This means that positive samples predicted more accurately (i.e., with higher IoU) contribute a more rational pulling force to the loss. This mechanism enables the model not just to distinguish between "presence or absence of an object," but to differentiate between "high-quality objects" and "background." To a certain extent, this also smooths out the extreme polarization between positive and negative samples. Focal Loss Although YOLOv5 disables Focal Loss by default (fl_gamma=0.0 in the hyperparameter file), the codebase actually integrates an implementation of Focal Loss, which can be enabled by setting the value of fl_gamma in the hyperparameter file. The working principle of Focal Loss is that it can automatically reduce the weight of easy samples (such as boxes that are obviously background at a glance), allowing the model to focus on difficult samples. Since most background regions are very easy to distinguish (e.g., pure black areas, sky), Focal Loss can drastically reduce the loss generated by this massive volume of negative samples, thereby eliminating the impact of imbalance. > The general idea is: if the model is very confident about a sample (relatively high confidence), its loss weight is scaled down close to 0 during loss calculation; if the model lacks confidence in a sample (relatively low confidence), its loss is preserved during calculation. In terms of specific calculation, Focal Loss multiplies the standard BCE result by a factor of : Here, (gamma, i.e., the value set for the fl_gamma hyperparameter) is a hyperparameter, usually set to 2. - For easy samples, the model's prediction output is very accurate, e.g., . In this case, the Focal Loss factor calculated using the formula above is: . As a result, the original loss is scaled down by 10,000 times, making almost zero contribution to the gradient. - For difficult samples, the model's prediction is poor, e.g., . Its Focal Loss factor is: . Consequently, the loss is only slightly reduced while still maintaining a large value, retaining its guiding significance for the model. Of course, in YOLOv5, Focal Loss is applied only to the calculation of Objectness Loss () and Classification Loss (), and is not used for Box Regression Loss.
SAM3 Model Environment Setup and Image + Video Segmentation Testing
This article provides a detailed record of setting up the environment for SAM3, the open-source image segmentation model released by Meta in November 2025, and walks through the process of performing image and video segmentation based on its official samples. SAM3 (Segment Anything Model 3) is the latest version of the image segmentation model released by Meta. It continues the design philosophy of the previous generation SAM model, further enhancing performance and adaptability in image segmentation tasks. !image.png Downloading Source Code and Weight Files First, download the source code from the SAM3 GitHub repository: facebookresearch/sam3: The repository provides code for running inference and finetuning with the Meta Segment Anything Model 3 (SAM 3), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.. In addition to downloading the source code from GitHub, you also need to download the model weight file sam3.pt (3.2GB) in advance. According to the README in the GitHub repository, the standard procedure requires submitting an application on Hugging Face (facebook/sam3 · Hugging Face). Once approved, you can download it directly from Hugging Face. However, this application is often mysteriously rejected without any given reason—it remains unclear why: !image.png Therefore, alternative methods can be used to download the weight file beforehand. A Zhihu creator provided an alternative download path for the weights: AlienDao Models, which offers extremely fast download speeds. You can download the weight file from this site first and place it in the root directory of the sam3 project. Environment Setup The development environment requirements are as follows: - Python 3.12 or higher - PyTorch 2.7 or higher - NVIDIA GPU with CUDA 12.6 or higher Create and activate the sam3 conda environment according to the project's README file: Next, install PyTorch, navigate to the SAM3 source code directory, and install the sam3 package. (My GPU is an RTX 5070 Ti with CUDA 12.9, so I ran the installation directly using the official PyTorch command below): Please note: When running the static image and video test routines in the sam3 environment, you may encounter a No module named 'triton' error. In this case, you need to install an additional triton-windows package: Image / Video Segmentation Testing Static Image Testing The following code detects the bear target contained in a static image and plots the results. > Note: You need to add a plt.show() statement at the end of the official plot_results function in the source code to actually display the image. The image segmentation result produced by the above code is shown below: !image.png In the code above, the processor executes detection and segmentation operations, returning three variables: masks, boxes, and scores: - masks: Single-channel mask images with the same resolution as the original image. In these mask images, the pixel locations of each detected target are marked with different colors. There is a separate mask image for each segmented target. - boxes: Bounding boxes for the specified target categories. For each target detected in the image, there is a corresponding box containing the coordinates of the top-left and bottom-right corners. - scores: Confidence scores for each detected target. The figure below shows the output information when using "bear" for detection in the bear image. You can combine this information to understand the meaning of the three output variables: Video Testing Further testing of SAM3's video segmentation capabilities was conducted based on the tutorial video by Bilibili UP host "Coding Tea Room" (Reference 3). This test requires relatively high GPU specs. On my laptop with 32GB RAM and an RTX 5070 Ti GPU, processing a 22-second video took a full 6 minutes and 40 seconds. Therefore, it is best to choose relatively short videos for this type of testing; otherwise, waiting during each debugging iteration can be quite tedious. The following script uses SAM3 to perform target detection and segmentation on every frame of a video file based on text prompts. The overall workflow follows the logic of the Coding Tea Room code. However, after completing the image segmentation for each frame, the process of overlaying the resulting mask frame onto the original image frame to generate and save the segmented image was simplified. By using render_masklet_frame to merge the mask with the original frame, the code becomes much easier to understand. In essence, after processing through propogate_video_frames, the code obtains a list of target segmentation masks called sam3_output_frames, which contains all image frames where the specified target was detected. The video_file_frames structure stores the list of all image frames read from the video file. Then, in a loop, the segmentation mask frame is merged with the corresponding original frame to produce a blended frame. For frames where no target is detected, the original frame is used as-is. Finally, all blended frames are saved into an MP4 file. The key code snippet is as follows: References - facebookresearch/sam3: The repository provides code for running inference and finetuning with the Meta Segment Anything Model 3 (SAM 3), links for downloading the trained model checkpoints, and example notebooks that show how to use the model. - The SAM3 Model is Here: Hands-on Guide to Running SAM3 Code, Initial Exploration of SAM3! _Bilibili_bilibili - SAM3 Source Code Practical Part 2: Hands-on Video Segmentation Processing. _Bilibili_bilibili - Serialized Post 1: Meta SAM3 Development Practice — Minimalist Image Segmentation - Zhihu
Gradio: A Python Web UI Framework for Machine Learning
Gradio is currently one of the most popular and user-friendly Python Web UI frameworks in the fields of machine learning and data science. Simply put, its core mission is to let you build a visual demo interface for your machine learning model instantly with a very short amount of Python code (usually just a few lines). It is particularly ideal for rapid prototyping, showcasing model performance to teams/clients, or hosting demos on Hugging Face Spaces. The official website of Gradio is: Gradio !image.png Key Features of Gradio - Pure Python Experience (No HTML/CSS/JS): No front-end knowledge is required. All components (input boxes, buttons, sliders, image uploaders, etc.) can be invoked through Python classes. - Lightning-Fast Construction: For standard "input -> model -> output" workflows, 3-5 lines of code are usually enough to get the job done—making it very simple and easy to pick up. - Rich Component Library: It comes with a large number of specialized components built specifically for machine learning scenarios, such as Image, Audio, Video, Dataframe, and Code. - Shareable Links: After running .launch(share=True), Gradio automatically generates a public temporary URL (valid for 72 hours) that you can directly send to clients or colleagues so they can test your model in their own browsers (while the model actually runs on your computer). _This feature is extremely convenient for temporary demo presentations, eliminating the time wasted on setting up cloud-based demo environments._ - Ecosystem Integration: Acquired and maintained by Hugging Face, it integrates tightly with the Hugging Face ecosystem (Spaces, Transformers). > The one-click sharing feature mentioned above is exceptionally convenient for demonstrating and testing model demos. You just need to pass a share=True parameter when starting the web app, and external networks can directly access it via a generated public URL. There is no longer any need to waste time setting up cloud-based demonstration environments. * Development Environment Setup and Hello World Installing the development environment for Gradio is very straightforward; you can install it directly using pip. If you need to use it within Jupyter notebooks, you also need to install ipykernel: Generally speaking, Gradio offers two main ways to build UI interfaces: - gr.Interface: Provides the highest level of abstraction, making it extremely simple to use and best suited for single-function workflows. - gr.Blocks: More flexible. If you need more complex layouts (such as multi-column arrangements, multiple tabs, or controlling data flow between components), you can use Blocks like building blocks to control every display detail of the page. The following code demonstrates the basic Hello World functionality using both approaches: Once the above code is executed, you can view the Web UI interface in your local browser at http://127.0.0.1:7860. In addition, when the share option in launch is set to True, Gradio automatically generates a public URL accessible from the external network. This allows you to share the URL with colleagues or clients so they can test the model's functionality, while the model and its actual computations remain on your local machine (_in fact, Gradio simply creates a tunnel between the public URL service and your local computer, forwarding all traffic hitting the public URL to your local machine_). This public URL remains valid for 1 week, which is more than enough for demonstration and experiential testing. Below is the UI interface of the Blocks mode shown above: !image.png Main Components of Gradio The use of the Gradio framework is primarily built upon four core concepts: - Blocks/Interface: Responsible for layout management of the entire Web UI architecture. - Component: Over 30 built-in UI components for conveniently handling inputs and outputs of text, audio, images, etc. - Functions: Standard Python functional code that receives data from input components, performs various processing operations, and ultimately outputs and renders the results via output components. - Launch: Kicks off the execution of the entire Gradio Web App. Gradio includes over 30 well-packaged components built-in. These components can generally act as both inputs and outputs, depending on whether they are placed in the inputs parameter or the outputs parameter. They are categorized below by functional type: Text and Chat Classes gr.Textbox: The most commonly used component, primarily for displaying text boxes. - When used in input mode, it provides an interface for users to enter text, supporting options like lines, placeholder, and type="password" (password masking). - When used in output mode, it displays processed text results. gr.Chatbot: Essential for Large Language Models, specifically designed to display conversation history. - Typically used as an output component, receiving a list of (user_message, bot_response) tuples. gr.Markdown: Displays text content supporting Markdown syntax. Often used for introductory notes at the top of pages, formula rendering, or rich text output. gr.Code: Specifically designed for showcasing code snippets, supporting syntax highlighting and a one-click copy button. Multimedia Classes (Video and Audio) gr.Image: For uploading and displaying images. - When used in input mode, it supports uploading static images, taking photos using a webcam, or providing a canvas for users to draw hand-drawn images. It can also automatically convert images into NumPy arrays or PIL objects. - When used in output mode, it displays processed images. gr.Audio: For uploading and playing audio files. - When used in input mode, it supports uploading audio files or recording audio via microphone. - When used in output mode, it plays synthetic speech (TTS) or music generated by functions, while providing a waveform display and download button. gr.Video: Similar to the Image component, it supports uploading or recording videos, as well as playing generated video streams. gr.Gallery: Used to display multiple images simultaneously in a grid layout, making it ideal for image generation models (generating 4 images at once) or image retrieval tasks. Numeric and Control Classes (Parameter Control) These components are typically used as inputs to adjust model parameters or execution options. gr.Slider: A slider component ideal for selecting numeric values within a specific range (such as a confidence threshold between 0 and 1). gr.Dropdown: A drop-down menu component suitable for selecting a single value from a predefined list. gr.Radio: A radio button group. Functions similarly to Dropdown, but all options are directly expanded and displayed. gr.Checkbox / gr.CheckboxGroup: Checkboxes used as boolean switches (True/False) or multi-selection tags. gr.Number: Used as a precise numeric input box. Data and File Classes (Analysis and General Purpose) gr.File: General-purpose file upload/download. If you are handling PDFs, Zip archives, or specific file formats, you can use this component to upload them into your program. Multi-file uploads are supported. gr.Dataframe: - When used in input mode, it acts like an Excel-like table editor where users can manually enter data or paste CSVs. - When used in output mode, it displays Pandas DataFrame data, supporting sorting and filtering. gr.Plot: Specifically designed for displaying charts, offering seamless support for objects generated by plotting libraries like Matplotlib, Plotly, and Bokeh. gr.JSON: Used to display JSON data or Python dictionaries cleanly in a tree structure, making it convenient to inspect complex API response results. Demonstrating OpenCV Image Blurring with Gradio Below is a demonstration of a Web UI that uses multiple Gradio components and OpenCV to blur an input image and display the result. The code is shown below: The final UI rendering effect looks like this: !image.png References - Gradio - Building User Interfaces For AI Applications with Gradio in Python | DataCamp
Open-Source Dataset Management Tool: FiftyOne
This article summarizes the working features, environment setup, and common application scenarios of FiftyOne, an open-source dataset visualization and management engine developed by Voxel51. FiftyOne is an open-source dataset visualization and management engine developed by Voxel51. Its official website is: Maximize your Computer Vision Performance | Try Voxel51. !image.png When it comes to dataset management, FiftyOne's core features include: - Dataset Visualization (The App): FiftyOne provides a web interface that allows you to browse images and their overlaid bounding boxes just like scrolling through a social media feed, enabling you to inspect and audit the quality of annotated data. - Efficient Filtering (Querying): Allows you to quickly filter data using Python expressions. For example, you can use simple code to quickly retrieve all images in a dataset that contain a specified target category and have a resolution higher than a given threshold. - Model Analysis: After training your own model (such as YOLO), you can feed the prediction results back into FiftyOne to directly compare ground truth labels with model predictions. This makes it easy to spot where the model misses or misclassifies objects, helping you improve data quality in a targeted manner. Environment Preparation First, you need to install FiftyOne and its dependencies. It is recommended to perform this in an independent virtual environment. Browsing Datasets Browsing local and online datasets using FiftyOne is very intuitive. By using FiftyOne, you can not only view the images and their annotation information contained in the dataset through its web interface, but also filter and manipulate these visual data just like operating a database using powerful query statements. The following code demonstrates the workflow for accessing an online dataset maintained by FiftyOne and a local YOLOv5-format dataset, respectively. After executing the code above, you can access http://localhost:5151/ locally via a browser to browse and manage the corresponding dataset and its annotations. !7815904f-7a16-4925-8e6a-46ec239303b7.jpg Actually, the code for accessing datasets with FiftyOne is well-suited to run in Jupyter Notebook. As shown in the figure above, after starting FiftyOne via launch_app, you can browse the dataset in its Web UI. Furthermore, launch_app returns a session variable. When running the code above in Jupyter Notebook, you can use this session to access the opened dataset and perform online processing and operations programmatically. For example, we can select low-quality images in the Web UI, and then execute deletion operations on these selected images within the notebook, exporting them to a new dataset: !image.png The session supports many options. Through this approach, you can achieve dynamic interaction with the opened dataset. Downloading and Cleaning Data from Public Datasets FiftyOne has a built-in module called Dataset Zoo. This module contains pre-written download and parsing scripts for dozens of mainstream public datasets such as COCO, Open Images, and VOC. Without FiftyOne, accessing and using these public datasets often requires registering on official websites, downloading dozens of gigabytes of compressed archives, studying their complex JSON or CSV formats, and writing scripts to parse and convert coordinates and annotation information of different formats. With the help of FiftyOne, a single command like foz.load_zoo_dataset("coco-2017", ...) automatically handles filtering, downloading, extracting, and format alignment. By using the code below, you can automatically download the first 1,000 annotated images of people from the COCO-2017 dataset on the FiftyOne website, automatically convert the annotation format to YOLOv5 format, and save them in the ./my_yolo_dataset/coco_subset/train/ subdirectory of the current script's directory. > Note: According to the workflow in the code below, when downloading a specified quantity (1,000 images) of a specific category ("person") from a designated dataset (coco-2017), FiftyOne only streams the subset of images requested by the user (first downloading to C:\Users\\fiftyone, then organizing them into the current script's directory). There is no need to download the entire hundreds-of-gigabytes dataset. This on-demand download approach dramatically saves local disk space and download time. Additionally, during the process of downloading data from online datasets, to improve dataset quality, you can clean the downloaded data to a certain extent before exporting. For example, the following code removes samples without any bounding boxes from the downloaded dataset, as well as annotations with bounding boxes that are too small: After performing the above cleaning operations on the data and then exporting the cleaned data using the export command, you can effectively improve the quality of the training dataset. Assisting YOLOv8 Model Fine-Tuning FiftyOne not only simplifies the processes of downloading training datasets, cleaning data, and browsing datasets, but it can also be combined with model inference to visually display the model's performance on inference data and analyze its failure modes. For example, by combining FiftyOne and YOLO models, after the YOLO model training is complete, you can visualize and evaluate the model's prediction results on the validation dataset. This helps you better understand where the model's predictive capabilities fail and enables targeted improvements to the training data and metrics. The following code demonstrates the workflow of importing validation set data from a YOLO model into FiftyOne, allowing FiftyOne to simultaneously display ground truth boxes and prediction boxes. This enables training personnel to intuitively view the ground truth boxes and model inference boxes on the validation set through FiftyOne's Web UI, and directly understand which data samples cause the current model to fail: Below is the visualization effect of displaying YOLO model validation data, labels, and inference results in FiftyOne: !a95a1162-1e30-459e-8466-b3e44ed840f6.jpg References - Maximize your Computer Vision Performance | Try Voxel51 - Fine-tune YOLOv8 models for custom use cases with the help of FiftyOne — FiftyOne 1.11.0 documentation
A Practical Guide to OpenAI's Lightweight Multi-modal Model: CLIP
Released by OpenAI in early 2021, the CLIP (Contrastive Language-Image Pre-Training) model is a pre-trained neural network designed to match images and text, representing a classic milestone in recent multi-modal research. What Problem Does CLIP Solve? As one of the most milestone models in the field of Computer Vision (CV), CLIP—released by OpenAI in 2021—completely broke the limitations of traditional vision models that relied on fixed category labels and classes. Therefore, it is not just a model, but a paradigm shift that deeply aligns vision with language. Before CLIP, mainstream models in fields such as image recognition and object detection (e.g., ResNet, VGGNet, SSD, YOLO) were typically trained on manually annotated datasets like ImageNet or COCO. If a model was trained as a 1,000-class classifier, it could only recognize those 1,000 categories. To add a new category, one had to recollect data and retrain the model from scratch. In contrast, CLIP leverages massive off-the-shelf internet images combined with descriptive text to train the model's understanding of images/text, establishing a semantic association between the two. Specifically, it maps the visual features of an image and its corresponding textual description in a shared semantic space. This means that once the CLIP model is trained, you do not need to train any parameters for new tasks; as long as something can be described in language, CLIP can recognize it. !image.png Below is a practical case study demonstrating how to build a natural language-based search engine for a local dataset using the CLIP model and Meta's Faiss vector database. This example will reinforce your understanding of the CLIP model's concepts and workflow. The example mainly consists of three parts: - Encoding image files into embedding vectors using CLIP - Saving the image vector data to a local Faiss vector database - Querying and displaying images from the vector database using natural language text commands Image Embedding To encode image files, we use the preprocessing and encoding functions of the CLIP model. First, import the CLIP model: The CLIP model supports different architectural variants, whose primary differences lie in the architecture and scale of the visual encoder. > Different architecture variants of the CLIP model: 'RN50', 'RN101', 'RN50x4', 'RN50x16', 'RN50x64', 'ViT-B/32', 'ViT-B/16', 'ViT-L/14', 'ViT-L/14@336px' Here, models with the RN prefix belong to the ResNet series, while those with the ViT prefix belong to the Vision Transformer series. Generally speaking, ViT architectures consistently outperform ResNet architectures of the same scale, and larger models typically provide better zero-shot classification performance, albeit at the cost of higher computational resources. In this tutorial, we use ViT-B/32, the smallest model in the ViT series, utilizing 32x32 patches. Next, we call the model and preprocess methods of the CLIP model to vectorize images and text respectively: As we can see, the encoding process for images and text in CLIP is quite similar. Both involve encoding followed by normalization. The only differences are: - Encoding images calls the encode_image interface, while encoding text calls the encode_text interface. - Text information must first be tokenized using CLIP's tokenize function before being encoded into vectors. Saving Image Vectors Using Faiss The image vectors obtained via the get_image_embedding interface need to be stored in a vector database. This way, the next time the program starts, it can load and search directly from the database without needing to regenerate the image vectors from scratch. Here, we use Faiss (Facebook AI Similarity Search), a high-performance vector search library developed by Meta. The general idea and workflow are: Use faiss.write_index to save the image vector index into a .index file, while using pickle or json to save the list of all image paths as metadata. During subsequent searches, get_text_embedding is used to generate the vector for the query text, and Faiss's Inner Product search is utilized to perform cosine similarity matching between the image and text vectors. By retrieving the image file ID, the image path can be easily found in the metadata list. Every time the program starts, it attempts to load the existing vector database index file and image path file from the hard drive. On the first run, these files do not yet exist, so a vector database file in IndexFlatIP format needs to be created. Faiss supports various vector indexing modes, and IndexFlatIP is the most fundamental and accurate mode among them. This mode does not compress vectors, nor does it build complex tree or graph structures; instead, it stores all raw vector data (typically float32) as-is. During a query, it computes the query vector (the vector converted from the search text) against every single vector in the database one by one. Although this method is the slowest, its recall rate is 100%, meaning the results are absolutely precise. It is generally sufficient for image libraries containing fewer than 50,000 images. If the image library exceeds 100,000 images and the query speed drops noticeably, you can switch to the IndexIVFFlat mode, which can accelerate query speeds by 10 to 100 times. The following code vectorizes image files from the local hard drive in batches (by calling get_image_embedding), and finally saves the list of image vectors and the list of image file paths into the vector database file specified by index_path and the image path list file specified by image_paths, respectively. Image Search With the previous steps completed, our local image library has been converted into vectors and stored in the vector database. Next, we can use text search commands to query specific images from this vector database. This part is relatively straightforward and can be broken down into three steps: - Perform tokenization, vectorization, and normalization on the text content, which follows the same workflow as the earlier image vectorization process. - Call the search interface of the Faiss vector database using the search text's vector as a parameter. The return values are the similarity scores and the indices of the matched images in the image library. - Display the retrieved list of images. Finally, querying "a boy who is playing football" in my dataset yields the following results: !image.png References - 讓AI幫你穿搭！用CLIP實作一個時尚穿搭資料庫 - Building Image search with OpenAI Clip | by Antti Havanko | Medium
The Open-Source VLM Model MoonDream
This article provides a detailed introduction to MoonDream, a very popular open-source lightweight Vision-Language Model in the industry, summarizing its core feature list, basic testing workflow, and licensing details. Moondream is an open-source, lightweight Vision-Language Model (VLM) initiated and maintained by developer Vikhyat Korrapati. The model is positioned as a high-performance VLM specifically designed for edge devices and resource-constrained environments. Its core advantage is the ability to achieve visual understanding capabilities comparable to—and in specific tasks even surpassing—large models (such as LLaVA or early GPT-4V), all with extremely low energy consumption and memory footprint (running smoothly with less than 4GB VRAM or even on a CPU). MoonDream Official Website: Moondream !image.png By the end of 2025, MoonDream has evolved into its third generation. The parameter scale of its first two generations, Moondream 1/2, was approximately 1.6B - 2B, with a 0.5B parameter version even provided for more resource-constrained applications. The latest Moondream 3 model adopts the MoE (Mixture of Experts) architecture, reaching a total parameter count of 9B, while the activated parameters per single inference are only 2B. What is a VLM? First, to gain a deeper understanding of what a Vision-Language Model (VLM) actually is: simply put, a VLM (Vision Language Model) is a class of AI models capable of simultaneously understanding images and text to achieve cross-modal interaction. To put it more colloquially, a VLM model can be understood from its overall structure and workflow as follows: - At the input end, its input can be considered to consist of two parts: an image, and a natural language command corresponding to that image. - At the output end, the output of a VLM is a piece of text information (such as text content encapsulated in JSON format). Even when outputting numerical content such as position coordinates, the output remains a text string containing numbers. - Model workflow: The model describes and explains the input image, performs object search and localization, etc., according to the natural language command, and ultimately outputs text describing the image content and other retrieval information. - The most typical applications include describing image contents, detecting the positional coordinates of specific objects within an image, and generating OCR detection results for images. Other VLM models include: Alibaba's Qwen2-VL, the open-source LLaVA series, Microsoft's Phi-3.5/Phi-4 Vision and Florence-2, Google's PaliGemma, and InternVL 2 from domestic AI Labs. Environment Setup and Core Feature Demonstration of MoonDream The following describes the environment setup and feature demonstration using MoonDream2, which is the most mature in application and has the most lenient license. First, download the Moondream2 model repository and weight files from Hugging Face. The model repository address is: vikhyatk/moondream2 at main. However, the weight files of Moondream2 are close to 4GB, making direct downloads from Hugging Face very time-consuming. In regions with domestic network restrictions, downloads can be made from the Hugging Face mirror site hf-mirror.com. Below is the download code for the model and its weight files. The related model files will be downloaded to D:\models\moondream2: The core feature matrix of Moondream includes four modes: caption, query, detect, and point. These four modes correspond to different granularities of image understanding (from holistic to local) and output formats (from unstructured text to structured coordinates). The following code demonstrates the inference workflow of these modes using the locally downloaded MoonDream2 model files. First, install the dependency packages required for the demonstration code: The inference code is as follows: The execution result of the code is shown in the figure below: !image.png MoonDream License Different versions of Moondream adopt completely different licensing strategies. !image.png As can be seen, both Version 1 and Version 2 of Moondream adopt the most lenient Apache 2.0 license, allowing free use in commercial fields. However, the use of the latest MoonDream 3 in commercial fields comes with certain additional conditions: - Free-to-use scenarios: Deployment on servers solely for internal company use, embedding as a feature within corporate products, as well as other non-profit and research uses. - Prohibited scenarios (where a separate commercial license must be obtained): Setting up a server to provide paid services externally via APIs (such as HTTP or SDKs), or applications that heavily overlap with official paid services. References - Overview | Moondream Docs