All Articles
Custom Dataset Training of RT-DETR Model and Its Adaptation on RK3588
This article provides a complete record of training the RT-DETR pre-trained model based on the Ultralytics project using a custom dataset, exporting it to ONNX, and adapting it for the RK3588 hardware platform. For the implementation of the RT-DETR model, there are currently two independent implementations: Baidu's PaddleDetection and Ultralytics (YOLOv8/v11 ecosystem). Since I am more familiar with Ultralytics/YOLO, this article uses the Ultralytics/YOLO implementation for experimentation and documentation. !image.png 1. Custom Dataset Training Since this uses the RT-DETR implementation under the Ultralytics project, its training process, organization of intermediate files generated during training, and the tabular format and directory structure of the training data are completely identical to those of YOLO models. You only need to directly copy over the dataset directory and its data.yaml used previously for YOLO model training. Therefore, if you have experience with the YOLO model training workflow, getting started with training RT-DETR in this project will be very straightforward. You can refer to A Comprehensive Guide to Custom Training, Testing, and Model Export of YOLOv5 Models. The training workflow code is also similar to other YOLO models under the Ultralytics project. The training code is as follows: The training workflow is basically the same as regular YOLO, with two points to note: - When training in a Windows environment, the training code must be wrapped inside __main__. This is because Linux systems typically use fork to create processes, which directly clones the current memory state without re-importing modules. Windows, however, does not support fork and must use spawn (cold-starting a new Python interpreter and importing the above script). As a result, when the child process starts, it will run the entire train.py script all over again, triggering a new training request and creating an infinite loop. To protect memory from being exhausted, the system actively throws a RuntimeError to terminate the entire program. By wrapping the training code inside __main__, only the main process executes the train_model function, while child processes are only responsible for executing the training process, thereby avoiding the infinite loop of repeatedly creating new child processes. - Set batch and workers relatively small; otherwise, the program can easily freeze during training. I initially set batch to 16, only to find that the program frequently froze and threw out-of-memory (OOM) errors. Therefore, if you encounter stability issues during training, you can try reducing these two parameters (workers and batch). The above training execution process visibly takes longer than training a YOLOv8 model with the same dataset, likely because the computational complexity of the RT-DETR model is higher than that of YOLOv8. The training results are likewise saved as the best.pt file in the runs directory. 2. The Nightmare of the Grid Sample Operator and Monkey Patch Running AI models on RK3588 hardware generally follows this path: - Complete model training on a PC using frameworks like PyTorch; - Convert the trained .pt model file into the more universal ONNX format; - Use Rockchip's RKNN-Toolkit2 tool to perform on-board model conversion, converting the ONNX model into RKNN format; - Run the model on the RK3588 board using the rknn-toolkit-lite2 tool (Python) or C/C++ language, completing inference testing on the board to obtain results. Since the training process is complete, the next step naturally involves ONNX model conversion. However, this step introduces a major issue caused by the Grid Sample operator: - The Grid Sample operator is supported starting from Opset 16. Therefore, to successfully export to ONNX, the Opset version must be set to 16. Setting it to an Opset version lower than 16 will result in an error during the ONNX model export step. - If Opset is set to 16, the RKNN-Toolkit2 tool (version 2.3.2) can indeed successfully export the RKNN model. However, when this model is run on the RK3588 board along with the librknnrt.so file of version 2.3.2, a segmentation fault (segfault) occurs. The reason is that version 2.3.2 of librknnrt.so does not support the Grid Sample operator at all. In other words, version 2.3.2 of RKNN-Toolkit2 considers the Grid Sample operator supported, so model conversion passes; whereas version 2.3.2 of librknnrt.so does not implement this operator at all, resulting in a segfault during on-board execution. Shouldn't Rockchip's algorithm team synchronize their releases? For the problem above, you can of course modify the model architecture, replace the Grid Sample operator with other more general operators, and retrain it to fundamentally solve the problem. However, if runtime speed on the board is not a primary concern, you can actually bypass this operator using a simple method: Monkey Patching. A Monkey Patch means that for the model you want to export to ONNX, before exporting, you replace any operator unsupported by the board with a more common one, so that the Grid Sample operator is completely absent during ONNX export. Specifically, before exporting the ONNX model, you use the following code to replace the Grid Sample operator (F.grid_sample): 3. ONNX Model Export and RKNN Conversion Having resolved the Grid Sample operator issue, the subsequent ONNX export becomes relatively straightforward. As summarized in the article How to Understand the Opset Version in ONNX Export Process?: The highest Opset version supported by Rockchip NPU's version 2.3.2 model conversion tools and on-board NPU driver is 15. During actual deployment, the optimal Opset version to set is 12 or 13 to avoid stability and compatibility issues that may arise from overly recent versions. The ONNX model exported above is the Opset=13 version with good support on the RK3588. Next, use the RKNN-Toolkit2 tool to perform on-board model conversion, translating the ONNX model into RKNN format: 4. On-Board Inference Execution Finally, running on-board inference on the RK3588 using the rknn-toolkit-lite2 tool allows you to see the execution results. However, the inference execution speed feels relatively slow, which might be related to the manual replacement of the Grid Sample operator: !result_fp16_final.jpg
Detailed Summary of RT-DETR (Real-Time DEtection TRansformer) Network Architecture and Computation Workflow
This article provides a detailed introduction to the network structure of RT-DETR, a Transformer-based image object detection model proposed by Baidu, along with its data computation workflow. Introduction to the RT-DETR Model Previously, in Detailed Summary of DETR (DEtection TRansformer) Network Architecture and Computation Workflow, we organized the network architecture of DETR, a Transformer-based computer vision object detection model. Developed by the Baidu team, RT-DETR (Real-Time DEtection Transformer) aims to resolve the limitation where traditional DETR models fail to meet real-time scenario requirements due to their massive computational overhead. The emergence of RT-DETR marks the official entry of the Transformer architecture into the battlefield of industrial-grade real-time detection. In fact, RT-DETR is the first end-to-end Transformer detector to truly achieve real-time performance. !image.png Similar to the YOLO series which provides model versions of different scales through scaling logic, the RT-DETR model also includes two versions: RT-DETR Large (32M parameters) and RT-DETR Extra-Large (67M parameters). The former offers slightly lower performance but has relatively lower requirements for computing resources, making it suitable for deployment on embedded edge devices. Just like Facebook's DETR model, the overall architecture of the RT-DETR model also consists of three parts: - A Backbone composed of a convolutional network. - A hybrid encoder. - A decoder and detection head. The following sections provide a detailed summary and explanation of the structures of the above three parts and the data computation workflow within them. Backbone Similar to the DETR architecture, the Backbone in the RT-DETR model serves as the feature extraction engine of the entire model, responsible for extracting multi-scale feature maps (standard DETR models only extract feature maps of the smallest resolution) from the input raw images. The official implementation of the RT-DETR Backbone supports two network structure options: HGNetv2 and the ResNet series. - The Backbone network of DETR also uses the same ResNet50 and ResNet101. Unlike DETR, which outputs only a single 1/32-scale feature map from its backbone, RT-DETR's backbone outputs three feature maps of different scales (marked as S3, S4, and S5 in the figure above), corresponding to 1/8, 1/16, and 1/32 of the original image size, respectively. These feature maps contain multi-level information ranging from local textures to global semantics. - HGNetv2 (Higher-order Geometric Network) is a Backbone network structure based on convolutional neural networks developed independently by Baidu. Its core idea is to utilize depthwise separable convolutions and specific structural re-parameterization to enhance accuracy while maintaining low latency. Regardless of whether ResNet or HGNetv2 is used as the backbone network, once the input image data enters the backbone, it goes through 5 stages (Stage 1-5). With each passing Stage, the height and width of the feature map are halved while the number of channels increases (this is identical to the backbone network of YOLO models). Assuming an input image resolution of 640x640, the outputs of the backbone network are: - Stage 3 (S3): Output feature map size is (C_3, 80, 80) — a scale of 1/8. - Stage 4 (S4): Output feature map size is (C_4, 40, 40) — a scale of 1/16. - Stage 5 (S5): Output feature map size is (C_5, 20, 20) — a scale of 1/32. Here, C_3, C_4, and C_5 are the number of channels for the three stages, respectively. Positional Encoding Regarding positional encoding for feature maps, RT-DETR shares the same calculation formulas and workflows as DETR, with both defaulting to Sine-Cosine Positional Encoding. For a point (x, y) on a feature map, its encoding is generated by sin and cos functions of varying frequencies, generating 128-dimensional vectors for each x and y respectively, which are finally concatenated into a single vector of length 256. For specific calculation formulas and workflows, please refer to the relevant section in Detailed Summary of DETR (DEtection TRansformer) Network Architecture and Computation Workflow. However, it should be noted that for the feature maps output by the RT-DETR Backbone, only the S5 feature map is flattened into a long sequence. Positional encoding is applied to all pixel points on this lowest-resolution feature map, which is then fed into the Encoder for self-attention computation. This is because RT-DETR considers performing global attention computation solely on the lowest-resolution feature map to be the most cost-effective. The higher-resolution feature maps S3 and S4 directly enter the CCFM module in the hybrid encoder for fusion computation and do not participate in global attention operations, as described below. Hybrid Encoder: AIFI + CCFM Next comes the hybrid encoder part of RT-DETR, which is the core component of the model. The design of this part enables the model to balance the global modeling capability of Transformers with real-time performance. !image.png As shown in the figure above, the hybrid encoder mainly consists of two core modules: AIFI and CCFM. The S5 feature map output by the backbone is fed into the AIFI module for processing, outputting the F5 feature map of the same dimensions; meanwhile, the F5 feature map, along with the S4 and S5 feature maps output by the backbone, are sent into the CCFM module for more comprehensive feature fusion. However, before processing in either AIFI or CCFM, the S3, S4, and S5 feature maps output by the backbone must first undergo dimension alignment. Dimension Alignment Why is dimension alignment necessary? This is because the number of channels (C_3, C_4, C_5) output by different stages of the backbone network (such as the aforementioned ResNet or HGNetv2) are inconsistent: - Stage S3 outputs 512 channels - Stage S4 outputs 1024 channels - Stage S5 outputs 2048 channels In the RT-DETR architecture, to enable feature maps of different scales to be fused via addition in subsequent processing stages and fed into a decoder of uniform dimension, feature maps of all scales must ultimately be unified to the same number of channels (defaulting to 256 in RT-DETR). Therefore, this step aligns the channel numbers output by different stages to 256. The specific alignment method is straightforward: apply a 1 x 1 convolutional layer to each of the S3, S4, and S5 layers to compress their channel counts from 512/1024/2048 down to 256 channels. After this 1 x 1 convolution, three sets of feature maps with completely identical channel counts (i.e., data dimensions) but different spatial resolutions are obtained. AIFI AIFI: Attention-based Intra-scale Feature Interaction. This module is specifically responsible for processing the feature map from the deepest layer and lowest resolution of the backbone (i.e., the minimum-resolution feature map S5 output by the backbone). > The core essence of AIFI is essentially a single-layer Transformer encoder structure, where the input to this encoder is solely the S5 feature map from the Backbone. Why is self-attention computation unnecessary for S3 and S4? This is because high-resolution feature maps (S3, S4) contain local details, and only the lowest-resolution feature map (S5) contains rich semantic information and global context. Therefore, global interactions only need to be extracted in the low-resolution feature map. Most importantly, the encoder part tends to be the computational bottleneck. If full self-attention computation were performed on feature maps of all scales, the complexity would grow quadratically with the number of pixels. Restricting self-attention computation solely to the lowest-resolution S5 feature map drastically reduces computational cost (FLOPs) while ensuring that the model can extract global relationships among objects in the image. Following the dimension alignment operation above, the dimension of the S5 feature map input to the AIFI module is (256, 20, 20). The subsequent computation workflow matches that of a standard Vision Transformer Encoder: first, flatten this (256, 20, 20) feature map into a vector sequence of length 400 and dimension 256, and then add positional encoding information to this vector sequence. Regarding positional encoding for feature maps, RT-DETR shares the same calculation formulas and workflows as DETR, with both defaulting to Sine-Cosine Positional Encoding. For specific calculation formulas and workflows, please refer to the relevant section in Detailed Summary of DETR (DEtection TRansformer) Network Architecture and Computation Workflow. This is followed by standardized multi-head attention computation (defaulting to 8 heads) and feed-forward network (FFN) processing. Finally, the S5 feature sequence processed by AIFI is reshaped back into a tensor shape of (256, 20, 20), which corresponds to F5 in the architecture diagram above. CCFM In RT-DETR, CCFM discards the computationally complex self-attention calculations of traditional DETR, turning instead to a CNN-based structure to handle the fusion of feature maps across different resolutions—a concept very similar to PANet used in YOLO. Since it fuses multi-scale features, it naturally includes two fusion pathways: - Top-down: Passing high-level semantic features downward. - Bottom-up: Passing low-level localization features upward. The core of CCFM performing bidirectional feature fusion lies in its Fusion module, as shown in the figure below. !image.png Assume the two feature maps input to the Fusion module are F_high (higher-level features) and F_low (lower-level original features), their channel counts have both been aligned to c, and the resolutions of the high-level and low-level feature maps have been transformed to identical resolutions through previous Upsample or Downsample stages. The workflow of the Fusion module is then as follows: - Feature Concatenate: Concatenate the two input feature maps with channel count c along the channel dimension, outputting a feature tensor with a channel count of 2c. - Feature Splitting: The concatenated feature map with 2c channels enters two parallel branches, each processed by a 1 x 1 convolution to compress the channel count back from 2c to c. - For the lower branch, an additional sequence of N consecutive RepBlocks is applied. This RepBlock is essentially a 3 x 3 convolution, extracting local spatial features through successive convolutional layers. - Element-wise Add: Element-wise add the base feature map from the upper branch and the enhanced feature map processed by the RepBlock from the lower branch, keeping both the channel count and resolution unchanged. Therefore, for each Fusion module, the resolution and channel count of its two input feature maps are identical to the resolution and channel count of its output feature map. For the entire CCFM module, its inputs can be divided into three parts: the output F5 of the S5 feature map processed by the AIFI module, and the dimension-reduced feature maps of the S4 and S3 feature maps from the backbone network. The function to be realized by CCFM is to thoroughly fuse the information from feature maps of three different scales. The fusion method is largely similar to YOLO's PANet, utilizing UpSample (yellow squares in the diagram below) and DownSample (blue squares in the diagram below) to transform feature maps of different scales to the same resolution, followed by fusion operations within the Fusion Blocks. !image.png Following bidirectional thorough fusion by the CCFM module, the hybrid encoder finally outputs three fusion-enhanced feature maps from the three Fusion Blocks: - P3 (80 x 80 x 256): Contains rich details, beneficial for detecting small objects. - P4 (40 x 40 x 256): Medium scale. - P5 (20 x 20 x 256): Contains deep semantics, beneficial for detecting large objects. Finally, these three feature maps undergo flattening operations to obtain three serialized vectors of dimension 256 (6400 x 256, 1600 x 256, and 400 x 256). These three sequences are then concatenated (this is the role of operation C in the CCFM module), ultimately outputting a long sequence of 8400 x 256. Decoder and Detection Head The final part of RT-DETR is responsible for end-to-end object detection based on the 8400 x 256 long sequence (for an image input with 640x640 resolution) output by the previous stage (i.e., the hybrid encoder output). This part is mainly composed of the IoU-aware Query Selection module and the Transformer Decoder. That is, the data output by the hybrid encoder is not directly fed into the decoder; instead, an IoU-aware query selection module first screens it to pick the most promising queries from the encoder's output as the decoder's input, thereby reducing the computational load of the decoder. !image.png IoU-Aware Query Selection The overall working logic of the IoU-aware Query Selection part is: Utilizing the output of the hybrid encoder (8400 x 256), select the Top-K (defaulting to top 300) candidates with the highest predicted IoU quality, which are most likely to be true objects, to serve as the queries for the Decoder. First, to achieve IoU awareness, RT-DETR designs a lightweight detection head at the output end of the hybrid encoder. This detection head is not a deep network, but an extremely streamlined set of linear prediction layers. Taking the 8400x256 long sequence output by the hybrid encoder (which can be considered as three feature maps containing 8400 patches) as input, it uses independent linear prediction layers across 3 separate branches to obtain classification predictions, bounding box predictions, and IoU prediction results corresponding to the 8400 patches. All three branches map the input 256-dimensional features to their output target dimensions: - Classification branch: Output dimension is 8400 x num_classes, predicting the confidence of each point belonging to each category. - Regression branch: Output dimension is 8400 x 4, predicting the offset (x, y, w, h) of each point relative to its coordinate origin. - IoU branch: Output dimension is 8400 x 1, predicting the IoU quality score between the predicted bounding box corresponding to this point and the potential ground truth box of the detected object. With the output data from the above three branches, 8400 quality scores can be constructed according to the following formula: The former is the confidence score of the category with the highest confidence value in the classification branch for the current patch, while the latter is the IoU quality score of this patch. Evaluation is performed using this quality score; a point is selected for the subsequent Decoder only when the model is both certain of its category and confident in its ability to box it accurately. Next, the above 8400 quality scores are sorted in descending order by Top-K, and the 300 with the highest quality scores are selected as the output for this stage. Therefore, through the workflow above, it is equivalent to selecting 300 out of 8400 candidate boxes based on quality scores as the basis for further screening, rather than using all 8400 candidate boxes as alternatives, which drastically reduces the computational workload of the Decoder stage. Ultimately, the output of the IoU-aware Query Selection module is not a single tensor, but a set of structured information. For each query selected in the Top-K calculation process above, two pieces of information are output at this stage: - Content Query: The 256-dimensional feature corresponding to each feature point. These vectors are used as input to the self-attention module in the next-stage Decoder, with a dimension of 300 x 256. - Reference Points: The coordinate positions (x, y, w, h) output by the aforementioned regression branch, with a dimension of 300 x 4. Decoder The structure of the Decoder part remains standard: self-attention module + cross-attention module + feed-forward module. However, to balance computational speed, the cross-attention module here does not adopt the full Cross-Attention of standard Transformers (where every query computes against all 8400 points), but instead adopts Deformable Attention. Similar to the architecture of a standard Transformer Decoder, the network structure of the RT-DETR model's Decoder part contains a stacked structure of 6 identical network modules. Each layer contains an independent self-attention computation module + multi-scale sampling cross-attention module + feed-forward network (FFN) module + classification head and regression head composed of linear layers. 1. Self-Attention Module The role of this step is to allow sufficient information exchange among the 300 Content Queries. This self-attention module takes the Content Queries of dimension 300 x 256 as initial input, generates corresponding positional encodings using the Reference Points as references, adds them to the Content Queries, and feeds them into the self-attention computation module. For the computation workflow of the self-attention module, please refer to A Thorough Guide to Understanding the Encoder Structure and Workflow of Transformer Models. The output of the self-attention module consists of the Content Queries after thorough information exchange, maintaining a dimension of 300 x 256. 2. Multi-scale Sampling Cross-Attention Module The input data for the cross-attention module consists of three parts: - The updated Content Queries output by the preceding self-attention module, with a dimension of 300 x 256. - Reference Points representing the position coordinates of 300 prediction boxes, coming from the output of the IoU-aware Query Selection module (for the first decoder layer) or the updated coordinates from the regression head of the previous decoder module (for the subsequent five decoder layers). - Encoder Features, the global feature sequence output by the hybrid encoder part (containing S3, S4, and S5 scales), with a dimension of 8400 x 256. It should be noted that the cross-attention module in the Decoder part of the RT-DETR model differs from the cross-attention operations in standard Decoders. It does not perform matrix multiplication on all 8400 x 256-dimensional data contained in the Encoder Features; instead, it selectively samples near the centers of the 300 prediction boxes and performs cross-attention computations on the sampled points in the vicinity of each reference point (i.e., Deformable Attention). This significantly reduces the computational complexity of cross-attention, thereby achieving a balance between computational efficiency and accuracy. The design of this deformable cross-attention module is one of the keys enabling the RT-DETR model to achieve real-time performance. Regardless, following the computation of this deformable cross-attention module, the output data consists of new Content Query vectors fused with multi-scale local image features, with the dimension remaining 300 x 256. 3. Feed-Forward Network (FFN) Module The purpose of this module is to enhance the non-linear expressive capacity of the features, similar to fully connected layers in CNNs. Just like standard Encoder and Decoder modules, the input data to the FFN module is the Content Queries output by the previous step's cross-attention computation (with a dimension of [300, 256]). The computation workflow of the FFN module follows a standard Linear -> ReLU -> Linear structure, where the intermediate dimension is typically expanded (e.g., to 1024 or 2048). The output data consists of deeply mapped Content Queries, maintaining a dimension of [300, 256]. 4. Classification Head / Regression Head The input data for both is the Content Query output by the previous layer's feed-forward network module, with a dimension of 300 x 256. They respectively map classification prediction values and coordinate offsets through independent linear layers. - Classification head: Mapped to the target category count num_classes via linear layers. The output data is classification probabilities (Pred Logits), with a dimension of [300, num_classes]. - Regression head: Similarly predicts the bounding box coordinate offsets through linear layers, and then updates these coordinate offsets onto the Reference Points input to the current layer. The Reference Points used by the next module are the values updated by these offsets. Model Output The final output of the RT-DETR model comes from the classification head and regression head of the last Decoder block in the Decoder part. These two independent detection heads respectively output the classification judgment values and coordinate values of 300 detection boxes. - Classification head outputs classification prediction scores (Pred Logits): Dimension is [300, num_classes]. Every query gives a score for all categories. - Regression head outputs predicted coordinates (Pred Boxes): Dimension is [300, 4]. These are the final coordinates (x, y, w, h) continuously refined through the 6 Decoder blocks of the Decoder part. Next, how do we obtain the final detection results presented from these 300 detection boxes? First, apply Sigmoid processing to the Pred Logits output by the classification head above, converting them into probability values between 0 and 1. Then, use each query's maximum score across all categories as the confidence score for that box. At this point, we have 300 predicted boxes with category labels and confidence scores. Next, perform threshold filtering. A confidence threshold (e.g., 0.5) is typically set. The system directly discards all boxes with scores lower than this threshold and outputs detection boxes with confidence scores greater than the threshold. - The key point of this part is that because the Transformer's self-attention mechanism has already completed deduplication internally (suppressing duplicate queries pointing to the same target), the remaining detection boxes are theoretically high-quality detection results that do not overlap with each other. References - RT-DETR: Paper Explanation and Inference - lyuwenyu/RT-DETR: [CVPR 2024] Official RT-DETR (RTDETR paddle pytorch), Real-Time DEtection TRansformer, DETRs Beat YOLOs on Real-time Object Detection. 🔥 🔥 🔥
A Detailed Summary of the Google Eddystone BLE Advertising Protocol Specification
Introduction to Eddystone Eddystone is a data format specifically defined for Bluetooth Low Energy (BLE) advertising packets. Introduced by Google as an open-source beacon protocol in 2015, it aims to provide an open and feature-rich advertising standard compared to Apple's iBeacon. !image.png In the Bluetooth protocol stack, Eddystone operates above the GAP (Generic Access Profile) layer. It utilizes the 31-byte payload within BLE advertising packets for communication, essentially defining a standardized data structure based on this 31-byte length constraint. - Because it operates under the BLE advertising mode, communicating devices do not need to establish a pairing connection. During communication, the broadcasting device simply transmits signals outward at regular intervals in a unidirectional manner, and any scanning device within its coverage area is responsible for listening to these messages. - Unlike iBeacon, which only broadcasts a single set of IDs during advertising communications, Eddystone allows devices to alternate broadcasting different types of advertising frames (Frames), which is a major feature of Eddystone. Four Main Frame Types of Eddystone UID (Namespace and Instance ID Frame) This frame type is primarily used to identify the identity of a device. Its frame structure is as follows: - Namespace ID (10 Bytes): Similar to a device's family name, used to identify a specific company, product type, or series. Therefore, generally speaking, devices of the same model produced by the same company share the same Namespace ID. - Instance ID (6 Bytes): Similar to a device's given name, used to identify a specific individual device. Even for devices of the same model, their Instance IDs differ, allowing them to be distinguished from one another. - Custom Space: This portion of space can be customized for specific meanings. TLM (Telemetry Frame) This frame is generally used to broadcast device status, such as battery level, temperature, and uptime. It is the status frame of the device. URL (Uniform Resource Locator Frame) This frame type is Eddystone's most well-known feature. It is used to directly broadcast a compressed URL address (such as https://goo.gl/...). In fact, this is a typical application scenario for Beacon-type Bluetooth devices: - A user's smartphone approaches a bus stop and automatically pops up a webpage containing nearby route information without requiring an app installation. - A user's smartphone approaches a specific area in a supermarket and receives a web URL containing promotional information for goods corresponding to that area. EID (Encrypted ID Frame) This frame type is primarily used to encrypt the ID, changing it at regular intervals to prevent the device from being illegally tracked by others. Eddystone Packet Structure First, let's summarize the overall packet structure of a typical BLE advertising packet. The raw data packet transmitted over the air is typically 8 to 47 bytes in length: !affce66d-09d8-4bec-b2a5-72a72a1fe8d1.png All data structures of Eddystone are contained within its Payload. The Payload portion ranges from 6 to 37 bytes in length, which includes the device's MAC address and the actual advertising data (ADV Data). Excluding the 6-byte device address, the remaining 31 bytes form the user-definable advertising data area. It consists of multiple AD Structures (such as AD 0 to AD N in the diagram above), and the format of each AD Structure is: [Length] [Type] [Data]. For the Eddystone protocol, it typically contains the following three standard AD Structures: - Flags: This part is identical for all Eddystone devices. Used to inform scanners about the physical characteristics of the device (e.g., General Discoverable Mode only, BR/EDR Not Supported). • Byte stream: 02 01 06 ◦ 02: Length of the subsequent data. ◦ 01: Type (Flags). ◦ 06: Value (representing LE General Discoverable Mode + BR/EDR Not Supported). - 16-bit Service UUID: This part is identical for all Eddystone devices. Used to inform scanners that this is an Eddystone-compliant device. • Byte stream: 03 03 AA FE ◦ 03: Length of the subsequent data. ◦ 03: Type (16-bit Service UUID). ◦ AA FE: Eddystone's official registered UUID (0xFEAA), stored in little-endian format. All Eddystone devices must include this UUID. - Service Data (Core Payload): This is where the advertising protocol stores the actual data. • Byte stream example (UID Frame): 17 16 AA FE 00 xx xx xx ◦ 0-17: Length varies depending on the packet content length, up to a maximum of 23 bytes. ◦ 16: Type is Service Data, i.e., 0x16, fixed. ◦ AA FE: UUID 0xFEAA, fixed. ◦ 00/10/20/30: Eddystone Frame Type. ◦ Followed by different frame data. Therefore, it can be simply stated: For all standard Eddystone device packets and advertising packets, the first 7 bytes (02 01 06 03 03 AA FE) are completely identical and fixed. The true differentiated competition and business logic distinction for each device are entirely concentrated within the Service Data payload following type 16. Mapping of Eddystone Frame Types to Service Data Structures As mentioned above, for the advertising packets of all Eddystone devices, the Payload portion contains three AD Structures. The data contents of the first two AD Structures are completely fixed and identical, and only the last AD Structure—the Service Data portion—is truly used to carry differentiated data. The first byte of the Service Data portion is the Length, which corresponds to the length of this section; the second byte is fixed at 0x16, indicating that this section is of the Service Data type, and thus remains unchanged; bytes 3 and 4 are similarly the fixed UUID 0xFEAA. Therefore, the true application-layer differences in Eddystone advertising packets are mainly reflected starting from the fifth byte of the Service Data and onwards. The frame structures are summarized below according to different Eddystone frame types, with subsequent explanations starting from the fifth byte position of the Service Data (i.e., the Eddystone Frame Type). UID Frame (Namespace and Instance ID Frame) - Byte 0, Frame Type, 1 Byte, fixed to 0x00 - Byte 1, TX Power, 1 Byte, calibrated Received Signal Strength Indication (RSSI) at 0 meters, ranging from -100 to +20 dBm - Bytes 2-11, Namespace ID, 10 Bytes, used to distinguish different scopes, organizations, manufacturers, or product types - Bytes 12-17, Instance ID, 6 Bytes, used to identify a specific device, i.e., the device ID, which is unique for every device - Bytes 18-19, Reserved, 2 Bytes, reserved by the official specification, typically padded with 0 URL Frame (Uniform Resource Locator Frame) - Byte 0, Frame Type, 1 Byte, fixed to 0x10 - Byte 1, TX Power, 1 Byte, calibrated RSSI value at 0 meters - Byte 2, URL Scheme, 1 Byte, prefix encoding (e.g., 0x00 represents http://www., 0x03 represents https://). - Bytes 3-19, Encoded URL, 1-17 Bytes, compressed website content (non-ASCII, using specialized suffix compression codes) TLM (Telemetry Frame) - Byte 0, Frame Type, 1 Byte, fixed to 0x20 - Byte 1, TLM Version, 1 Byte, protocol version (currently typically 0) - Bytes 2-3, VBATT, 2 Bytes, battery voltage in millivolts (mV) - Bytes 4-5, Temperature, 2 Bytes, device temperature (8.8 fixed-point format, in degrees Celsius) - Bytes 6-9, ADV_CNT, 4 Bytes, total number of advertising packets sent since reboot - Bytes 10-13, SEC_CNT, 4 Bytes, uptime since reboot (in units of 0.1 seconds) EID (Encrypted ID Frame) - Byte 0, Frame Type, 1 Byte, fixed to 0x30 - Byte 1, TX Power, 1 Byte, calibrated RSSI value at 0 meters - Bytes 2-9, Ephemeral ID, 8 Bytes, time-varying 8-byte cryptographic identifier Supplementary Note 1: What does the aforementioned 8.8 fixed-point format mean? The 8.8 fixed-point format is an efficient method for representing values with fractional parts using 2 bytes (16 bits). It divides the 16 bits into two equal parts: the upper 8 bits represent the integer part, and the lower 8 bits represent the fractional part. A 16-bit field is split into: • Upper 8 bits (Most Significant Byte, MSB): Signed integer part. Uses two's complement form (i.e., values greater than 0x80 are negative, and values less than 0x80 are positive), with a range from -128 to +127. • Lower 8 bits (Least Significant Byte, LSB): Unsigned fractional part. It represents fractions of 256 (i.e., n/256). Supplementary Note 2: What is the compression workflow for the Encoded URL mentioned above? In the Eddystone-URL specification, to pack a URL as long as possible into the limited 31-byte space of a Bluetooth advertisement, Google does not use a complex general-purpose compression algorithm. Instead, it adopts a table-lookup-based byte substitution technique. This method is divided into two parts: prefix compression and suffix/common word compression. Prefix compression: Uses 1 byte to represent the beginning of the URL. Common mapping relationships include: - 0x00: http://www. - 0x01: https://www. - 0x02: http:// - 0x03: https:// Suffix compression: Other regular ASCII characters (such as a-z, 0-9, -) are stored directly as-is. However, for Top-Level Domains (TLDs) and common path symbols, the specification defines a set of special single-byte replacement codes (typically within the range of 0x00 to 0x0D): - 0x00: .com/ - 0x01: .org/ - 0x02: .edu/ - 0x03: .net/ - 0x07: .com - 0x08: .org - 0x0B: .net
How to Understand the Opset Version in the ONNX Model Export Process
When exporting a PyTorch-trained model from a PC or server to the ONNX format, you need to configure the opset version in the export script. Different opset versions heavily impact the subsequent NPU adaptation process. So what exactly is an opset version? What are the differences between opset versions, and how do these differences affect subsequent NPU adaptation? This article attempts to answer these questions. Generally speaking, in the deployment workflow of embedded AI models, Opset (Operator Set) is the version standard for the set of operators defined by ONNX (Open Neural Network Exchange). When exporting a PyTorch model (.pt/.pth) trained on a PC into the ONNX format, setting the correct opset version is crucial because it directly determines whether the exported model can be successfully converted by subsequent NPU edge model conversion tools and run efficiently on the NPU. !image.png To understand how different opsets affect the subsequent NPU model adaptation and deployment workflow, we must first deeply understand what operators and operator sets actually are. What are Operators (Ops) and Operator Sets (Opsets)? In the context of AI models, an operator (op) is the smallest execution unit within a model. A typical deep learning model (such as a YOLO model) is essentially a massive Directed Acyclic Graph (DAG). This graph consists of countless nodes, where each node represents an operator. This is somewhat similar to writing software programs: when implementing a certain functionality, we usually encapsulate it into a function and call it. Thus, an operator is like a standard library function that can be called by an application, and invoking this operator (function) is equivalent to executing a minimum-granularity standardized operation on the NPU. Common operators include: - Conv (Convolution): Extracts features, equivalent to a filtering algorithm in embedded systems. - ReLU (Activation Function): Essentially a simple if(x
Complete Guide to Porting the PANNs Audio Classification Model to the RK3588 Platform
In the previous article Summary of Pre-Research Information on the PANNs Audio Classification Model, I summarized information about the PANNs audio classification model and conducted some preliminary tests on the PC side. The article Training Workflow of the PANNs Speech Classification Model Based on the ESC-50 Dataset recorded the complete workflow of training the ESC-50 dataset using the CNN14 architecture of the PANNs model. This article takes a further step by porting the model trained on the ESC-50 dataset in the previous stage to the Rockchip RK3588 platform and performing on-board inference via Python scripts using Rockchip's rknn-toolkit-lite2 tool. Exporting the Model to an ONNX File Following the model training workflow described in Training Workflow of the PANNs Speech Classification Model Based on the ESC-50 Dataset, we obtain the trained model file cnn14_esc50_best.pth using the ESC-50 dataset. Next, invoke the following script to export this model file into ONNX format, generating cnn14_esc50_rk3588.onnx. > 💡 > The PANNs model actually converts audio data into a spectrogram first, and then performs classification training based on a labeling-aware CNN network. For such models, the initial audio-to-spectrogram conversion part is not well-suited for NPU execution. Therefore, a reasonable approach is: once the audio data is collected, compute the audio-to-spectrogram conversion on the CPU, and then feed the spectrogram into the NPU for CNN classification and recognition. In this case, the model adapted to the NPU should only contain the CNN network part, excluding the preceding spectrogram calculation part. The spectrogram fed into the CNN network for recognition should be generated from 1 second of audio data (corresponding to 101 audio frames), with a Mel scale range of 64, resulting in a spectrogram resolution of 101x64. Preparing Calibration Files for Model Conversion During the model conversion process, 20 audio files need to be extracted from the previous training set (i.e., the ESC-50 dataset) to serve as the calibration dataset during model conversion. Randomly select 20 audio files from the ESC-50 dataset, place them into the dataset_ecs50 subdirectory, and then use a text file named dataset_ecs50.txt to record the path names of all calibration audio files: Use the following script to convert the calibration audio files into .npy format, save them in the dataset_ecs50_npy directory, and automatically update dataset_ecs50.txt with the path names of the generated .npy files. This is because subsequent model conversion steps require .npy format audio files for calibration: At this point, the calibration files for model conversion are ready: - dataset_ecs50_npy: Contains .npy format files for all 20 audio calibration samples. - dataset_ecs50.txt: Contains the path names of all .npy calibration files. Model Conversion Execute the following code to convert the ONNX model exported from the PC into an RKNN model that can run on the RK3588. Once the conversion is complete, the edge model file cnn14_esc50.rknn is generated. Installing the rknn-toolkit-lite2 Environment on the Board The system running on my RK3588 is Debian, which already includes a basic Python 3.12 environment. To perform on-board inference using Python with rknn-toolkit-lite2, you need to install the rknn-toolkit-lite2 environment on the board. First, check the Python version on the RK3588 board, and locate the corresponding rknn-toolkit-lite2 board-side installation package in the rknn-toolkit2-master/rknn-toolkit-lite2/packages directory. Since my on-board Python environment is 3.12.4, the installation package to choose here is rknn_toolkit_lite2-2.3.2-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl. Use the following commands to install the rknn-toolkit-lite2 board environment support package along with the Python packages numpy and librosa required for inference code. Why add the --break-system-packages option? Because the Debian system running on the board introduces the PEP 668 specification. To prevent packages installed directly via pip from conflicting with the system's built-in apt package manager and causing system crashes, the global Python environment is locked. Otherwise, the following error will be reported, and the system recommends installing Python packages within an independent virtual environment. However, since this is just for testing purposes, there is no need to bother setting up a virtual environment; you can simply use the --break-system-packages option to bypass the error above. On-Board Inference At this point, both the on-board rknn-toolkit-lite2 environment and the on-board model (cnn14_esc50.rknn) are ready. Next, write a piece of on-board inference code based on rknn-toolkit-lite2 for testing: > 💡 Note here: Since the WAV files in the ESC-50 dataset have a length of 5 seconds, while the default model input is a spectrogram of a 1-second audio file, when performing inference on WAV files from the ESC-50 dataset on the board, you need to use a sliding window approach (with a sliding overlap time of 0.5 seconds) to split the 5-second WAV file into multiple 1-second audio segments. Then, compute the spectrogram for each 1-second audio segment, run inference to get the results, and finally merge all results to obtain the final recognition output. The inference result of the above code on the audio file 1-977-A-39.wav is as follows: The class ID of the detection result is 39. Checking the ESC-50 ID mapping table, ID 39 corresponds to "Clock Tick", which matches the sound contained in the audio file. !image.png
Running TFLite Models in Android Flutter Projects Using the flutter_litert Plugin
In a previous note (Summary of YOLO Model Workflow in Android Flutter Projects), I detailed the complete workflow of running a YOLOv8 model exported from the Ultralytics project within an Android Flutter project using the flutter_vision plugin. However, using the flutter_vision plugin comes with numerous limitations. For instance, it only supports a few specific YOLO models and is restricted to TFLite model files exported exclusively via Ultralytics' export interface. Consequently, if you want to run models other than YOLO on Android, flutter_vision is no longer a viable option. Therefore, to convert a custom model into the TFLite format and run it within an Android app, a more flexible approach is needed. This process involves two steps: - Converting the custom model's .pt file into the TFLite format. This is covered in detail in How to Convert PT Models to TFLite/LiteRT Format?. - Running the converted TFLite model file within your Android Flutter project while bypassing the limitations imposed by the flutter_vision plugin. This is the problem addressed in this article. flutter_litert vs tflite_flutter When running TFLite models in Android Flutter projects, two plugins are commonly used: flutter_litert and tflite_flutter. - tflite_flutter: This is the most classic and established TFLite plugin in the Flutter community. Most online resources regarding running AI models on Android use this plugin. It directly wraps the C-based TFLite interpreter using Dart FFI. While community-maintained forks still exist (such as tflite_flutter_plus), its configuration and community maintenance are fragmented, making it less recommended for new projects. - flutter_litert: This plugin was introduced as a modern successor in response to Google's official rebranding and upgrade of the TensorFlow Lite brand to LiteRT. It is currently the recommended plugin that aligns with Google's latest AI roadmap, essentially serving as an evolution of tflite_flutter designed to resolve the maintenance pain points of the older plugin. Therefore, it is strongly recommended to use flutter_litert in new projects. !image.png At its core, the flutter_litert plugin acts as a Hardware Abstraction Layer (HAL) wrapper operating at the Flutter (Dart) layer. Its general characteristics include: - Underlying Mechanism: Instead of rewriting the model's inference logic in Dart, the plugin uses Dart FFI (Foreign Function Interface) to very lightly bind Google's pre-compiled C++ LiteRT Native binary library (.so files). The advantage, of course, is that it leverages the maximum execution efficiency of underlying C/C++ languages. - Zero-Copy Philosophy: Its core idea is to allocate a contiguous buffer in memory (such as Float32List), populate data using Dart, and let the C++ engine directly read this physical memory for model matrix operations, avoiding performance overhead caused by cross-language data passing. Overall Architecture and Data Flow of the flutter_litert Plugin The diagram below illustrates the overall architecture and data flow when using the flutter_litert plugin to run AI models in an Android Flutter project: !litert_architecture_high_res.jpg From the overall flow chart above, the data movement during the model's forward inference can be broadly divided into four layers and stages. 1. Acquisition and Scheduling Layer: Flutter Dart Zone (User Space) This part runs inside the Flutter Dart Virtual Machine, acting equivalently to a user-space application in the operating system. Its primary task is to coordinate the timing between frontend UI rendering and model detection results, as well as image preprocessing. Of course, before the model can detect and recognize images, it must first be loaded into memory. Models and their label files are usually bundled within the app's assets, so assets-related APIs are used to load the model: When the phone's camera is open, it pushes high-resolution YUV420 raw stream data to the main thread at a frame rate of 30/60fps and displays it on the screen. Typically, smartphone computing power cannot support detecting every single frame at such a high frame rate—perhaps only completing 10 frames per second. In this scenario, image capture by the camera and image processing by the model are asynchronous. Therefore, they must be processed across two separate threads: the main thread refreshes and displays real-time images according to the camera's capture frame rate, while model image processing must be handled in an independent Isolate (similar to a high-priority background thread) to perform YUV-to-RGB conversion, image scaling, and other preprocessing steps before sending the data to the flutter_litert plugin for recognition and detection. As is well known, before sending images into models like YOLO for forward inference, a series of preprocessing steps must be applied to meet the model's dimensional requirements for input image data. Thus, for each YUV420 frame retrieved from the camera (potentially with a resolution of 1280x720), the preprocessing pipeline must include at least: resizing to 640x640, YUV-to-RGB conversion, image normalization to the 0-1.0 range, and rearranging data format to NCHW (PyTorch models default to NHWC data layouts). Once preprocessing is complete, the input image is ready, and the next step is to call the flutter_litert plugin's API to process it: 2. Cross-Language Function Interface and Memory Zero-Copy Mechanism During the interpreter's forward inference execution on input image data, image data must pass from the app layer's Dart language to the model inference code written in C++. To efficiently transmit large chunks of image memory data across these two languages, it is optimal to use memory pointers for a zero-copy image transfer mechanism. This is where Dart FFI (Foreign Function Interface) shines, allowing Dart to directly share physical memory pointers with underlying C/C++, avoiding repeated copying of large memory blocks between callers and callees. As shown in the diagram above, both the function calls to the interpreter written in Dart and the final return of model execution results from the underlying C++ language back to the user-space Dart layer leverage the memory zero-copy mechanism provided by Dart FFI. 3. Underlying Computing Execution: Native C++ Zone This is where the model actually executes forward inference using C/C++. In the article How to Understand the Core Concepts and Workflow of TFLite/LiteRT?, key concepts and execution workflows of running AI models under the Android TFLite/LiteRT framework were summarized in detail. Google implemented a LiteRT Runtime environment using C++ (essentially the interpreter), which handles the execution and operator scheduling of TFLite/LiteRT format models on Android. During specific operator scheduling and execution, the interpreter further uses delegates to decide which hardware dispatches specific operations: - GpuDelegate (Adreno GPU): Suitable for handling large-scale parallel floating-point matrix multiplications in models. - NnApiDelegate (Hexagon NPU): A dedicated NPU designed for the Android Neural Networks API. If a model is fully quantized to INT8, running it on an NPU is exceptionally fast with extremely low power consumption. - XNNPACK (CPU SIMD): The lowest-level fallback solution. If a specific operator is not supported by the GPU/NPU, execution falls back to the CPU using vector instructions. This is often the root cause of lag due to slow model computation speeds. For specific execution and scheduling logic, please refer to How to Understand the Core Concepts and Workflow of TFLite/LiteRT? and How to Handle the Opset Version and Compatibility Issue in TFLite?. 4. Model Return and Rendering Following the forward inference executed by the LiteRT Runtime, the C++ engine updates the memory state. Through the Dart FFI mechanism, the app-layer Dart code captures the signal that model execution has completed. At this point, the user-layer application can unpack tensor information (such as classes, bounding boxes, and confidence scores) generated by models like YOLO/RT-DETRv2 from the contiguous memory block according to the output data structure and type. After post-processing these tensors (such as applying NMS for YOLO), the results are handed back to Flutter's GPU rendering pipeline to map coordinates based on screen proportions and draw the final target bounding boxes lightweightly onto the canvas. GPU/NPU Acceleration Settings for Model Inference As mentioned above, during forward inference calculations performed by the LiteRT Runtime environment, the interpreter selects corresponding delegate execution options based on operator version compatibility, primarily including GPU, NPU, and CPU SIMD options. So, when running AI models using the flutter_litert plugin within the Flutter framework, how do you configure GPU/NPU hardware acceleration for these operators? This is configured via InterpreterOptions. The specific code is as follows: The configuration logic in the code above is that during operator scheduling, the interpreter prioritizes GpuDelegateV2. If an operator is not supported on the GPU, it falls back to compute on the CPU using 4 threads. In the latest version 2.0.11 of the flutter_litert plugin (as of April 2026), the acceleration modes supported by InterpreterOptions include: - GPU Delegates (Graphics Processing Unit): Suitable for most models, especially CNN-style networks. Advantages: fast speed and relatively low power consumption. Disadvantages: unsupported operators may fall back to CPU. - GpuDelegateV2(): Targeted at the Android platform, backed by OpenGL ES 3.1 / Vulkan APIs. - GpuDelegate(): Targeted at the iOS platform, backed by Apple's Metal API. - Core ML Delegate (iOS): Targeted at iOS devices running iOS 11+ with an A12 chip or higher. Advantage: optimal performance on Apple devices; Disadvantage: iOS only. - On iOS, CoreMLDelegate() should be tried first, falling back to GpuDelegate() if unsupported. - XNNPACK Delegate (CPU Optimization): Used when there is no GPU or the GPU is unsupported. Advantage: good compatibility, faster than standard CPU; Disadvantage: still much slower than a GPU. Note: You can only configure a single acceleration option via options.addDelegate. Calling addDelegate multiple times will result in only the last one taking effect. Therefore, in practice, you should use a try...catch approach to select the most suitable acceleration option:
The Training Process of PANNs Audio Tagging Model Using ESC-50 Dataset
This article documents the complete workflow for custom training of the audio detection and classification model PANNs using the ESC-50 dataset. Introduction to the ESC-50 Dataset In the article Summary of Pre-research Information on PANNs Audio Classification Model, we introduced PANNs, a very popular model in audio classification applications. This model is thoroughly trained on Google's AudioSet dataset, achieving exceptional performance in various types of audio detection, recognition, and classification tasks. Based on the pre-trained file of this model (CNN14 architecture) Cnn14_mAP=0.431.pth, this article fine-tunes the pre-trained model using the ESC-50 audio dataset to adapt it to the detection and classification requirements of the ESC-50 dataset. The ESC-50 dataset consists of 2,000 environmental audio recordings and serves as a benchmark dataset for environmental sound classification methods. Each audio file in the dataset is 5 seconds long and is categorized into 50 semantic classes, with 40 examples per class, loosely grouped into 5 major categories: !image.png ESC-50 Dataset Download URL: karolpiczak/ESC-50: ESC-50: Dataset for Environmental Sound Classification > Note that the original sampling rate of the audio files in the ESC-50 dataset is 44.1kHz, while the pre-trained weights of PANNs Cnn14 are based on 32kHz. Therefore, when reading these audio files for model training, forced resampling from 44.1kHz to 32kHz is required. Training Process Documentation for the ESC-50 Dataset Objectively speaking, the PANNs model does not offer very friendly support for custom dataset training, so you need to design and handle the processing pipelines for each stage of the training process (dataset preparation, model classification head modification, training workflow) on your own. Preparation of the Training Dataset First, download the archive of the ESC-50 dataset from the download URL mentioned above. The entire dataset is around 600 MB. Next, create a custom ESC50 Dataset class inheriting from PyTorch's Dataset class, and implement the __len__ and __getitem__ interfaces within it. It is worth mentioning here that because the PANNs model is trained on 32kHz audio files, the 44.1kHz audio files from the ESC-50 dataset must be converted to 32kHz before loading the training data. In addition, to further standardize the length of audio data in the training dataset, the audio length is uniformly fixed to 5 seconds. Audio files that are too long are truncated directly, while files that are too short are padded with zeros at the end. Modifying the CNN14 Classification Head The PANNs model was designed and trained based on Google's AudioSet dataset, and the pre-trained file Cnn14_mAP=0.431.pth corresponds to a model classification head that supports 527 audio classes. However, the ESC-50 dataset supports 50 audio classes. Therefore, the classification head must be modified before training to meet the requirements of the ESC-50 dataset. The entire process consists of the following steps: - First, instantiate a Cnn14 model based on the PANNs source code and load the pre-trained model file. - Next, modify the classification head of the Cnn14 model, replacing the default linear layer of 527 neurons in the final layer with a linear layer of 50 neurons. - Finally, initialize the weight parameters of the newly modified classification head's linear layer. At this point, the pre-trained dataset and the model architecture to be trained are ready. The next step is to begin training. Model Training The model training workflow is not significantly different from that of a standard convolutional neural network model. It is worth noting that since this is a classification model, the training process uses the standard Cross-Entropy loss function, and the optimizer is Adam. The log information during the training process is as follows: As can be seen, after 30 epochs of training, the recognition accuracy on the validation set reached 93.5%. Once training is complete, the trained model parameters are automatically saved as the cnn14_esc50_best.pth file. Inference Finally, perform inference testing on the trained model using the following code: References - karolpiczak/ESC-50: ESC-50: Dataset for Environmental Sound Classification
Summary of Audio FFT Calculation Workflow and Spectrum Interpretation
This article summarizes the principles, parameters, and calculation workflows for performing FFT on speech signal sequences to obtain their corresponding spectra. It also provides a Python example to compute and interpret the spectrogram of a WAV file. Introduction to Speech Signal FFT Calculation and Parameter Summary As is well known, in the time domain, an audio signal is a pressure value that fluctuates over time (i.e., sound pressure, manifested as amplitude). The principle of Fourier transform proves that any periodic signal can be decomposed into the sum of a series of sine/cosine waves with different amplitudes, frequencies, and phases. Therefore, the fluctuations exhibited by an audio signal in the time domain can also be viewed as being superimposed by countless sine waves with different frequencies, amplitudes, and phases. The FFT (Fast Fourier Transform) calculation performed on a digital audio signal sequence is essentially the extraction of the corresponding combination of various sine waves from this digital audio signal sequence, which is the so-called spectrum. !image.png The FFT calculation performed on the discrete-time sequence of an audio signal in the digital domain has the following characteristics: - The result of performing an FFT calculation on an audio signal sequence is not a continuous frequency analysis. Instead, the total frequency range (from to the audio sampling rate ) is evenly divided into small blocks, with each block called a Bin. The width of each Bin is . - The output result of the FFT calculation is a complex number . To obtain the final spectrogram, it is also necessary to calculate the magnitude of the complex number: . This magnitude represents the energy level of the frequency component in the signal. - For real-valued audio signal sequences, the FFT calculation result is symmetric. That is to say, if 1024 audio sampling points are input for FFT calculation, among the 1024 output complex numbers, the first 512 cover information from 0 to (the Nyquist frequency), and the last 512 points are merely a mirror image of the first half. Therefore, when plotting the spectrogram, usually only the first data points are taken. Correspondence Between FFT Calculation Results and Frequency Domain Bins As mentioned above, for an FFT calculation performed on an audio sequence of length , the calculation result is a complex number sequence of length , where the first half and the second half exhibit mirror symmetry. The entire frequency band corresponding to the FFT calculation result is . Because the entire calculation result sequence is mirror-symmetric from front to back, this frequency range is evenly divided into parts. For each output point of the FFT calculation (called a Bin), the formula for calculating its corresponding frequency point is: where is the index of the complex number in the array (starting from 0). If the sampling rate and the FFT length : - Index 0: (DC component) - Index 1: - Index 2: - ... - Index 512: (Nyquist frequency, i.e., the maximum analysis frequency) The amplitude of each frequency point is obtained by calculating the magnitude of the complex number corresponding to that frequency point: . This magnitude represents the energy size of the frequency component in the signal. Key Parameters of FFT Operation The key parameters for performing FFT operations on audio sequences are as follows: (You can understand these parameters in conjunction with the FFT spectrum calculation workflow described below) - FFT Size / Window Length (): The number of sampling points in each audio sampling frame participating in the calculation, usually a power of 2 (e.g., 512, 1024, 2048, 4096). - Sampling Rate (): The number of samples collected from the analog signal per second. This collection frequency directly determines the bandwidth of the spectrum. According to the Nyquist sampling theorem, the relationship between the maximum frequency that the sampled sample sequence can analyze and the sampling frequency is . Therefore, if you want to analyze high frequencies of , the sampling rate must be greater than . - Window: To reduce spectral leakage, it is necessary to window the signal before performing the FFT calculation. The main types of windows include: Hann, Hamming, and Blackman. - Hop Size: The sliding distance between two adjacent FFT windows. The more consecutive audio frames overlap (the smaller the Hop Size), the smoother the generated spectrogram will be on the time axis, and the less information will be lost (because window functions attenuate edge energy, which can be compensated through overlapping). However, the smaller the Hop Size, the more frames need to be calculated for the same audio sequence, and the computational cost increases linearly. A schematic diagram of the audio sampling frame Window Size, sliding window Hop Size, and Overlap Size between two consecutive frames is shown below: !image.png Speech Signal Spectrum Calculation Workflow To generate a corresponding spectrogram for a speech file, the following steps and workflow are generally required. Speech Signal Acquisition and Framing Audio sampling equipment (MIC) acquires analog speech signals at a fixed sampling rate (e.g., 16K, 32K, 44.1K). According to the Nyquist sampling theorem, the maximum frequency that can be calculated and analyzed via subsequent FFT corresponds to . In addition, a speech signal may be several minutes long, and directly performing an FFT on the entire file is meaningless because the frequency of a speech signal constantly changes over time. Therefore, before performing the FFT calculation, the complete speech signal sequence must first be cut into many small segments, each of which is called an audio frame. Typically, the length of an audio frame is (which is actually the number of points for a single FFT calculation, i.e., the FFT Size), chosen as a power of 2, such as 512, 1024, or 2048. Of course, in terms of framing, two consecutive audio frames will partially overlap. For example, the first audio frame corresponds to audio samples in the 0-1023 range, the second audio frame corresponds to samples in the 320-1352 range, and so on, until the end of the entire audio file. Windowing Every framed audio frame needs to be windowed before performing the FFT calculation. When performing an FFT calculation on an audio sequence without windowing, it defaults to a rectangular window, which leads to spectral leakage and aliasing. The purpose of windowing is to make the amplitude of a signal frame taper off to 0 at both ends, which makes the peaks on the spectrum thinner and less likely to blur together, thereby mitigating the effects of spectral leakage and aliasing. Of course, in this case, the parts at both ends of a signal frame are attenuated and do not receive as much emphasis as the central part. The remedy for this is to have two consecutive audio frames overlap with each other. The time difference between the starting positions of adjacent frames is called the hop size (i.e., Hop Size). The following figure shows the window functions commonly used in speech signal processing: !image.png Windowing refers to pointwise multiplication of the audio frame with a window function frame of the same length: As a result, the beginning and end of this signal frame are smoothly attenuated to 0, which forcibly achieves continuity at the ends, eliminates discontinuities, and avoids jumps caused by inconsistent amplitudes at the start and end points of the signal sequence. FFT Calculation and Spectrogram Next, the windowed digital signal sequence is subjected to FFT calculation. During the calculation, the computational complexity of is reduced to through efficient butterfly computation methods using odd-even decomposition. For an audio digital sequence of length , the result of the FFT calculation is a complex number sequence of the same length . As mentioned earlier, because the audio digital sequence consists entirely of real numbers, this complex number sequence of length exhibits mirror symmetry. Then, calculate the magnitude of each frequency point: , which represents the intensity of that frequency component. Because the energy distributed across various frequency bands of audio spans an extremely large range, it must be converted into decibel values using logarithmic mapping (dB) for representation: . Finally, by arranging the frequency decibel energy distribution calculated for each frame (vertical axis, using color to represent energy magnitude) in chronological order (horizontal axis), a linear spectrogram is obtained. Mel-Scale Spectrogram In the field of digital audio, for tasks like speech recognition or music analysis, ordinary linear frequencies are typically converted into the Mel Scale. This is because the human ear is more sensitive to changes in low frequencies than high frequencies, and the Mel scale better simulates human auditory perception. The frequency distribution on the frequency axis of a linear spectrogram is uniform, whereas the frequency axis of a Mel-scale spectrogram is sparse at the front and dense at the back—its low-frequency portion is very sensitive, while its high-frequency portion is heavily compressed. Human hearing is non-linear and more sensitive to low-frequency signals; we can easily distinguish the pitch difference between and , but if two frequencies are and respectively, although they also differ by , the human ear can hardly tell any difference. > The definition of the Mel scale is precisely designed to make equidistant Mel frequency changes feel equidistant in human auditory perception. The conversion from a linear spectrogram to a Mel spectrogram is not a simple coordinate axis stretching, but rather a re-weighted aggregation of linear frequency energy through a set of Mel Filter Banks. Its general workflow is as follows: 1. First, a series of triangular filters are placed on the spectrum: in the low-frequency region, these triangles are narrow and densely arranged; in the high-frequency region, these triangles become wide and sparsely arranged. 2. Multiply the amplitude data of the entire frequency band of an audio frame with each triangular filter and sum them up: the first filter will collect the energy of the low-frequency region and output a single value; the last wide filter will collect the energy of a large range in the high-frequency region and output a single value after summation. This yields the Mel-scaled energy distribution for this audio frame. 3. Perform the second step processing for all audio frames to obtain the Mel-scaled energy distribution across different frequency bands for all audio frames. Ultimately, the number of frequency points on the Mel spectrogram depends on the number of Mel filter banks. If there are 1024 linear frequency points in the previous linear spectrogram, passing through 40 Mel filters leaves only 40 Mel frequency points. In this way, the horizontal coordinate of the Mel spectrogram remains the same as that of the linear spectrogram, both corresponding to the number of audio frames; its vertical coordinate becomes the number of Mel filter banks, which is 40, ranging from low to high frequencies. Example of Speech Signal Spectrum Calculation Below, taking a phone call speech file R9_ZSCveAHg_7s.wav that comes with the speech classification model (PANNs) code as an example, the Spectrogram and LogmelFilterBank classes from the torchlibrosa library are used respectively to generate the linear spectrogram and Mel-scale spectrogram. The spectrogram plotted by the above code is shown below: !image.png As can be seen from the spectrogram above, whether it is the linear spectrogram at the top or the Mel-scale spectrogram at the bottom, their horizontal coordinates are both the indices of the audio frames. The length of this audio file is 7 seconds, with a sampling rate of 32KHz, single channel, containing a total of 224,000 samples (7s * 32000 = 224000). The window size for FFT spectrum calculation is 1024 audio sampling points, and the number of samples for the audio frame interval (Hop size) is 320. Therefore, the 224,000 audio samples of the entire WAV file are ultimately split into 701 () overlapping audio frames. Consequently, the horizontal coordinates of both spectrograms consist of 701 consecutive audio frames. Regarding the vertical coordinate of the linear spectrogram, because the window size for FFT spectrum calculation is 1024 audio sampling points, the 1024 frequency domain results obtained from the FFT calculation are mirror-symmetric. Therefore, the number of frequency points in the linear spectrogram is . With a sampling rate of 32K, the frequency corresponding to each frequency point is: - Index 0: (DC component) - Index 1: - Index 2: - ... - Index 512: The Mel spectrogram is generated on the basis of the linear spectrogram. In the code, the mel_bins parameter sets the number of Mel filter banks to 64. By filtering and aggregating the energy of various frequency bands of the linear spectrogram in a manner that is dense at low frequencies and sparse at high frequencies, 64 energy density distributions ranging from low to high frequencies are ultimately formed on the vertical coordinate of the Mel-scale spectrogram. References - Window Functions in Speech Signal Processing - Ling Nizhan - Blog園
Summary of Preliminary Research on the PANNs Audio Classification Model
This article organizes the basic information of the classic audio classification model PANNs, its model architecture, the classification of pre-trained parameter files, the setup of the development environment, basic inference testing, and explains the network architecture of its flagship CNN14 model. Introduction to the PANNs Model PANNs stands for Large-scale Pre-trained Audio Neural Networks. Proposed by Qiuqiang Kong and other researchers between 2019 and 2020, its core significance lies in breaking the deadlock of "small samples and difficult training" in the audio field, providing a powerful general-purpose feature extractor for various audio tasks. In terms of the overall workflow, its working principle is equivalent to first generating a corresponding Mel-scale spectrogram for the audio file, and then using a standard convolutional neural network to perform classification tasks based on this spectrogram. The GitHub repository for this model is: qiuqiangkong/audioset_tagging_cnn The PANNs model is powerful because it has been extensively trained on Google's AudioSet dataset, learning to distinguish the spectral features of almost all common sounds in the world. The download link for the model's pre-trained parameter files is: PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition (Pretrained Models). As can be seen from the pre-trained parameter download file list above, the model provides pre-trained parameter files for multiple different network models and parameters: !image.png The naming format of these pre-trained parameter files is generally: ModelArchitecture_SampleRate_EmbeddingSize_PerformanceMetric.pth. - Model Architecture: Supported model architecture versions include CNN14 (recommended model architecture), CNN6/CNN10 (lightweight versions), ResNet/MobileNet, Wavegram-Logmel-Cnn14 (high performance, but computationally intensive), and DecisionLevelMax (suitable for Sound Event Detection inference mode). - Sample Rate: PANNs' official implementation presets support for three sample rates: 8kHz, 16kHz, and 32kHz, with 32kHz as the default sample rate. - Embedding Vector Length: The dimension of the embedding vector entering the detection head, with 2048 as the default. The official weight file repository also provides support for dimensions of 32, 128, and 512. - Performance Metric: The mAP metric tested on Google's AudioSet data validation set. Environment Installation and Basic Inference Testing Environment and dependency installation: As seen from the scripts/0_inference.sh script in the source code directory, the inference mode of this model can be divided into two modes: Audio Tagging and Sound Event Detection. - Audio Tagging: - The problem it solves is: What is the content of this recording. - The model output result is the overall classification probability of the entire audio segment, corresponding to the 527 audio categories in Google's AudioSet dataset. - Its technical judgment logic performs global pooling on the audio file across the entire time dimension, providing only a single overall classification probability judgment. - Sound Event Detection (SED): - The problem it solves is: When did a specific audio event occur. - The model output result is a timeline, marking the start and end times of abnormal sound occurrences on this timeline. - Its technical implementation logic builds upon Audio Tagging by preserving time resolution and then judging audio events across the entire time period frame by frame like a scanner. audio_tagging inference mode test: CNN14 Model Architecture As mentioned above, PANNs is not a single model, but a family of various network architectures. The pre-trained weight files for various models visible on the Zenodo page correspond to different network depths and structures: - CNN Series (CNN6, CNN10, CNN14): This is the most commonly used series in PANNs. CNN14 is the strongest flagship model and the network structure primarily promoted by this project. - ResNet Series (ResNet22, ResNet38, ResNet54): Draws on the residual network structure design, making it suitable for deeper feature extraction. - Lightweight Series (MobileNetV1, MobileNetV2): Designed specifically for mobile phones or embedded devices, trading a small amount of accuracy for extremely fast speed. - Hybrid Series (Wavegram-Logmel-CNN): This network architecture focuses not only on spectrograms but also directly references raw waveforms to capture finer-grained temporal information. The composition of the three network architectures in the CNN series is shown below: !image.png The validation metrics of different architectures in the CNN series combined with different sample rates, Mel-scale filter banks, and data augmentation strategies are shown below: !image.png Regarding the network structure of CNN14: - First, Mel-scale spectrograms corresponding to the audio files are generated based on Spectrogram and LogmelFilterBank from the torchlibrosa library. - This is followed by 6 consecutive convolutional blocks: - The first 5 convolutional blocks consist of: 3x3 convolution + BN layer + ReLU activation function + 3x3 convolution + BN layer + ReLU activation function + 2x2 pooling layer. - The last convolutional block consists of: 3x3 convolution + BN layer + ReLU activation function + 3x3 convolution + BN layer + ReLU activation function + 1x1 pooling layer. - The first convolutional block outputs 64 channels, the second outputs 128 channels, and so on, with the final convolutional block outputting 2048 channels. - The output dimension data of the final convolutional block is [batch_size, channels, time_steps, freq_bins]. By default, channels is 2048; time_steps is the step size in the time dimension, which becomes 1/32 of the audio sampling frames after 5 rounds of 2x2 pooling; freq_bins represents frequency features, which become very small (even 1) after 5 rounds of 2x2 pooling. At this point, each time period (32 consecutive audio sampling frames) corresponds to a 2048-dimensional feature vector, which represents the fingerprint information of the sound at that moment. - Next, global max pooling and average pooling operations are performed simultaneously on the feature vectors extracted by the continuous convolutional blocks along the time dimension, and the two are summed to obtain a 2048-dimensional feature vector. - Finally, a fully connected layer from 2048 to num_classes is used to perform the final classification operation based on the 2048-dimensional feature vector, yielding classification probabilities via sigmoid. References - 【音频分类与检测】PANNs：用于音频模式识别的大规模预训练音频神经网络-CSDN博客
A Detailed Explanation of the Calculation Workflow for Dilated Convolution and Deformable Convolution
The calculation workflow of standard convolution kernels in convolutional neural networks has been comprehensively summarized in the article A Detailed Explanation of Convolution and Pooling Calculations in Convolutional Neural Networks. This article focuses on two relatively specialized convolution operations: Dilated Convolution and Deformable Convolution. Dilated Convolution Simply put, compared to standard convolution, dilated convolution can forcefully expand the receptive field of the convolution operation without increasing the number of parameters. In standard convolution calculations, the elements of the convolution kernel are adjacent to each other: !image.png When performing dilated convolution, holes are inserted between the weight elements of the convolution kernel (i.e., padding with zeros, which are skipped during actual computation). As shown in the animation below, for a 3x3 dilated convolution, holes are inserted between each element during computation. This makes the receptive field equivalent to that of a 5x5 convolution, while the number of parameters and computational cost remain at the 3x3 level. !%E7%A9%BA%E6%B4%9E%E5%8D%B7%E7%A7%AF%E4%B8%8E%E5%8F%AF%E5%8F%98%E5%BD%A2%E5%8D%B7%E7%A7%AF-%E7%A9%BA%E6%B4%9E%E5%8D%B7%E7%A7%AF%E7%9A%84%E8%AE%A1%E7%AE%97.gif The core parameter for dilated convolution shown in the figure above is the Dilation Rate. It specifies the number of holes to be inserted between two consecutive weight elements during the convolution calculation, which represents the spacing between elements: - d=1: Standard convolution. - d=2: Skips one pixel between every two weight elements, and so on. In the dilated convolution calculation workflow illustrated above, a certain number of pixels are skipped between sampling points during computation, determined by the dilation rate parameter. Assume the input feature map is and the 2D convolution kernel is , with a size of . For a value at position on the output feature map, the formula for dilated convolution is: - Here, and represent the dilation effect, altering the indexing step size when reading the input . When , the above formula reduces to the standard convolution calculation formula. - It is important to note that the number of parameters in the convolution kernel remains , and no extra weight parameters are added due to the dilation. In PyTorch, nn.Conv2d already integrates the dilation parameter to specify the dilation rate during convolution. Deformable Convolution In traditional convolutions (including dilated convolutions), regardless of the object's shape outline, or how it is rotated, scaled, or deformed, the sampling shape of the convolution kernel is always a fixed square (e.g., 3x3, 5x5, etc.). The theory of deformable convolution argues that: convolution kernels should possess shape-sensing and adaptive capabilities. In other words, the sampling points of the convolution kernel should not be restricted to a fixed square; instead, they should be offset according to the actual outline and size of the object. By allowing the network to learn this offset itself, precise customization of the receptive field is achieved. The formula for deformable convolution is as follows: The figure below illustrates the calculation workflow of deformable convolution: !image.png Generally speaking, the calculation of deformable convolution is divided into two stages: the first stage is responsible for generating offsets, and the second stage performs the deformable convolution calculation based on these offsets. Let's explain the calculation workflow of the above two stages using a practical example. Assume the input is a 5x5 resolution feature map, and we want to perform a 3x3 deformable convolution. We focus on the center coordinate position of the input feature map below, which is the coordinate point with a value of 13: !image.png Step 1: Offset Generation The calculation in Stage 1 is relatively simple. It is essentially a standard 3x3 convolution. In this case, the output channels must be 18, because the purpose of Stage 1 is to generate two offsets () for every pixel point in the input feature map. For a 3x3 convolution, the corresponding number of output channels is . - Note that this offset calculation involves a separate offset convolution kernel. Therefore, the final output of the offset calculation process is an offset tensor with the same resolution as the input feature map and 18 channels. For the feature point at position , this offset tensor contains 18 offset values: !image.png Thus, for the convolution calculation at the center point , the positions of the 9 surrounding pixels (including itself) correspond to 9 sets of offsets. Step 2: Deformable Sampling Let's take the bottom-left point of the 3x3 region around position to explain the deformable sampling workflow. According to the deformable convolution formula above: The bottom-left point corresponds to . Plugging this into the formula gives . This is a non-integer coordinate (represented by the blue dot in the figure below). We find the four surrounding pixel points on the input feature map (represented by the red box in the figure below): !image.png Next, bilinear interpolation is performed. The closer the distance, the higher the weight. The interpolation weights are (i.e., keeping only the fractional part by removing the integer part). The bilinear interpolation calculation is as follows: This value of 20.476 is merely the deformable convolution calculation result for the bottom-left point . Following the exact same workflow as Step 2, perform the above calculations for all 9 positions to obtain the 9 deformed sampling values: !image.png Step 3: Weighted Summation Using the Main Convolution Kernel Finally, to obtain the final deformable convolution calculation result for position , element-wise multiplication is performed between the 9 values obtained above and the weight parameters of the main convolution kernel: - represents the 9 deformed sampling values obtained in Step 2 above, and represents the weight parameters of the main convolution kernel. Steps 1 through 3 above only calculate the deformable convolution result for the center position of the input feature map. All other pixels must also go through the complete workflow of Steps 1 to 3 to obtain the deformable convolution result for the entire input feature map. As can be seen, the computational overhead of deformable convolution is significantly higher than that of standard convolution. PyTorch does not provide a native implementation of deformable convolution, but torchvision.ops provides DeformConv2d to implement deformable convolution calculations. References - Deformable Convolution - Dilated Convolution
Setting Up the Android Studio + Flutter Full-Stack Development Environment and Key Considerations
This article documents the complete process of setting up a full-stack development environment based on Android Studio and Flutter in a Windows environment. Although the overall setup is not overly complex, several noteworthy issues arise during the initial configuration and testing due to the numerous concepts involved and network connectivity issues when accessing Google servers from mainland China. I have compiled this comprehensive guide for future reference during development. Downloading the Flutter SDK and Configuring Environment Variables Download the Flutter SDK from a mirror source in mainland China. Refer to the Flutter SDK Archive, choose the Stable Channel for Windows, and download it. The download speed is significantly faster than the official default source: !d99ea2d7-2389-4625-b608-a1796c881a53.png No installation is required after downloading. Simply extract the Flutter SDK archive to a permanent directory and add the path of the bin folder inside the SDK to your Path environment variable. Additionally, due to the network environment in mainland China, you need to modify the Flutter community package repositories by adding environment variables. Add the following two environment variables to your user environment variables: PUB_HOSTED_URL and FLUTTER_STORAGE_BASE_URL. !7024f394-d4b1-4a8b-8e8e-9fee7ad6de79.png Now, you should be able to check the current Flutter SDK version in the Windows terminal using flutter --version: Installing and Configuring the Android SDK Download and install the latest version of Android Studio from the Chinese official Android Studio website: Download Android Studio & App Tools - Android Developers | Android Developers. After installing Android Studio, opening it for the first time requires downloading tools like the SDK and NDK online. These toolkits are often several gigabytes in size. Therefore, when running Android Studio for the first time, select the Custom option in the setup wizard to configure an HTTP Proxy before proceeding with the SDK installation. Otherwise, downloading the SDK can be an incredibly frustrating experience. Here is a list of commonly used mirror proxies you can configure: !b5eda9af-5302-4c0f-bcb3-c7c5fa3727d0.png By default, Android Studio does not install the Android SDK Command-line Tools during the SDK setup, but Flutter requires them to manage SDK packages. Therefore, you must manually install this tool. Additionally, since I plan to use Flutter to develop audio/video and AI model packages and libraries that require calling C/C++ code, I also need to install the Android NDK package. Thus, after the basic SDK setup, go to the Settings menu and install the following additional SDK packages: !766a7d54-6774-4cef-a456-16a5faf546bf.png In addition, you need to install two plugins in Android Studio for Flutter development: Flutter and Dart. !4d4dfa01-4df5-4623-b865-6f1a1174c997.png At this point, the Flutter SDK, Android Studio, their respective SDKs, and the Flutter/Dart plugins are all installed. Environment Testing and Verification Before using Flutter for development inside Android Studio, you can run the flutter doctor command in your terminal to verify that your environment is properly set up. Before doing so, run flutter doctor --android-licenses and press y multiple times to accept the Android SDK licenses. The output of the final flutter doctor command will look something like this (since I am not focusing on Web development, I ignored the Chrome error message; if you need Web support, simply install Google Chrome): Changing Gradle Mirrors We are not done yet. When compiling and packaging Android APKs in Android Studio, Gradle is used, which also involves downloading a vast array of dependencies. During the Android app build process, the Gradle build script downloads various dependency packages from Google and MavenCentral by default. In mainland China, this download process is highly prone to getting stuck, so it is necessary to change the repository URL for downloading dependency packages. A quick web search shows that most articles suggest manually modifying the gradle/wrapper/gradle-wrapper.properties file to use a gradle.org mirror, as well as updating allprojects in the android/build.gradle file. While this approach is straightforward, the downside is that you have to manually modify these files every single time you create a new project. If you forget, Gradle will attempt to download dependencies from overseas servers, which can take an extremely long time. Instead, I want to apply these configurations globally so that every newly created Flutter project automatically redirects to domestic mirrors when building Android apps, rather than hitting the global central servers. To achieve this, create an init.d directory inside your Gradle home directory (e.g., mine is located at D:\Android\.gradle, while the default path is typically the .gradle folder under the user directory on the C drive). Inside init.d, create a file named mirror.init.gradle.kts with the following content: This way, every time Gradle loads, it will automatically override the download URLs with the local mirror server addresses, drastically speeding up the build and download process. Debugging At this point, the entire setup process is finally complete. The steps themselves are actually straightforward; the primary challenge lies in the network restrictions in mainland China, where downloading from external servers can be painfully slow. The setup involves several components—Android Studio, the SDK, the NDK, Flutter, and Gradle—each requiring numerous packages. Fortunately, the abundance of local mirror servers helps bridge the gap, and once the initial environment is configured, the subsequent development process runs very smoothly. Next is the classic debugging phase. You can directly use terminal development tools like Gemini CLI or AI IDEs like Trae to modify your code, and then run and debug it using the golden combo of flutter devices + flutter run + r. During normal development, you don't even need to open Android Studio; simply use it to manage and update your SDK. References - Using Flutter in China - Quick Configuration of Domestic Mirror Sources and HTTP Proxy in Android Studio - CSDN Blog
Setting Up the Android Studio + Flutter Full-Stack Development Environment and Important Notes
This article provides a comprehensive guide on setting up a full-stack development environment based on Android Studio and Flutter on Windows. The process itself is not overly complex, but due to the numerous concepts involved and network restrictions in mainland China when accessing Google servers, several points require attention during the initial setup and testing. This record serves as a complete reference for future development. Downloading the Flutter SDK and Setting Environment Variables Download the Flutter SDK from a mirror source in mainland China. Refer to the Flutter SDK Archive List, select the Stable Channel for Windows, and download. The download speed will be much faster than using the official default source: !d99ea2d7-2389-4625-b608-a1796c881a53.png No installation is required after downloading. Simply extract the Flutter SDK zip archive to a fixed directory and add the path of the SDK's bin directory to your system's Path environment variable. Additionally, due to network conditions in China, you need to configure environment variables to modify the address for the Flutter community. Add two user environment variables: PUB_HOSTED_URL and FLUTTER_STORAGE_BASE_URL. !7024f394-d4b1-4a8b-8e8e-9fee7ad6de79.png At this point, you should be able to check the current Flutter SDK version in the Windows terminal using flutter --version: Installing and Configuring the Android SDK Download and install the latest version of Android Studio from the Chinese official website: Download Android Studio and App Tools - Android Developers. After Android Studio is installed, the first launch requires online installation of tools such as the SDK and NDK. These tool packages are often several gigabytes in size. Therefore, during the initial SDK installation in Android Studio, choose the Custom setup option, configure the HTTP Proxy first, and then proceed with the SDK installation. Otherwise, downloading the SDK can be frustratingly slow. A list of commonly usable proxies is provided below: !b5eda9af-5302-4c0f-bcb3-c7c5fa3727d0.png By default, Android Studio does not install the Android SDK Command-line Tools during the SDK installation process, but Flutter requires them to manage mirrors. Therefore, you must install the command-line tools manually. Furthermore, since I plan to develop audio/video and AI model packages/libraries involving C/C++ via Flutter, I also need to install the Android NDK package. Thus, after the SDK installation is complete, go to the Settings page to additionally install the following SDK packages: !766a7d54-6774-4cef-a456-16a5faf546bf.png In addition, you need to install the two essential plugins for Flutter development in Android Studio: Flutter and Dart. !4d4dfa01-4df5-4623-b865-6f1a1174c997.png At this point, the Flutter SDK, Android Studio, its associated SDKs, and the Flutter plugins are all fully installed. Environment Testing and Verification Before you start developing with Flutter in Android Studio, you can use the flutter doctor command in the terminal to verify that your environment is correctly configured. Before doing this, run flutter doctor --android-licenses and press y multiple times to accept the Android licenses. The final output of the flutter doctor command will look similar to this (since I am not focusing on web development, I ignore the error regarding the Chrome browser; if web development is needed, simply install the Chrome browser): Configuring Gradle Mirrors We are not quite done yet. Compiling and packaging Android APKs in Android Studio also requires Gradle, which involves downloading yet another large set of tools. During the Android app building process, the Android build script gradle defaults to downloading various dependency packages from Google and MavenCentral. Domestic networks in China tend to get stuck during this download phase, so you need to change the source address for downloading dependency packages. A quick online search usually reveals solutions like manually modifying the gradle/wrapper/gradle-wrapper.properties file to change the default gradle.org mirror, or adjusting allprojects in the android/build.gradle file. While straightforward, the drawback is that you have to manually edit these two files for every new project you create. If you forget, it will attempt to download dependencies from overseas networks, taking an extremely long time. My goal is to apply these changes globally to the environment so that every newly created Flutter project automatically redirects download requests to domestic mirrors when building an Android app, rather than reaching out to external central servers. The specific approach is to create an init.d directory inside your Gradle installation directory (for example, my Gradle installation path is D:\Android\.gradle, and the default path is usually the .gradle folder under the user directory on drive C), and place a file named mirror.init.gradle.kts inside it with the following content: This way, every time Gradle loads, it will automatically replace the URLs of its related download servers with domestic mirror server addresses, drastically speeding up download times. Debugging With this, the entire setup process is finally complete. The process itself isn't complicated; the main bottleneck is the inconvenience of accessing external networks from within China, as downloading from overseas servers can be maddeningly slow. Furthermore, tools like Android Studio, SDK, NDK, Flutter, and Gradle all require downloading numerous packages to complete the environment setup. Fortunately, thanks to numerous domestic mirror servers acting as relays, once this setup and environment are established, subsequent development becomes much smoother. Next comes the classic debugging phase: modify code directly in the terminal using CLI development tools like Gemini CLI or AI IDEs like Trae, and then use the golden combination of flutter devices + flutter run + r to debug and run your code. During regular development, there is no need to keep Android Studio open; you only need to use Android Studio to manage and update your SDKs. References - Using Flutter in Chinese Networks - Android Studio Quick Configuration of Domestic Mirror Sources and HTTP Proxies (CSDN Blog)
The Adaptation Process of QWen3-VL-4B Multimodal Large Model on RK3588
This article provides a detailed log of the complete adaptation process for the Qwen3-VL-4B-Instruct multimodal large model on Rockchip RK3588, along with key considerations and common pitfalls during adaptation. Download and Testing of the QWen3-VL-4B Model Alibaba released the Qwen3-VL series of vision-language models in September 2025. Compared to its predecessors (such as Qwen2-VL), the release of Qwen3-VL marks a significant milestone where multimodal large models have officially transitioned from simple visual perception to a new phase of deep logical reasoning and autonomous agent interaction. Continuing Alibaba's open-source spirit, Qwen3-VL offers multiple model scales ranging from edge to cloud. !image.png To adapt the Qwen3-VL-4B-Instruct model on the Rockchip platform, you first need to download the model files and perform basic inference testing on a PC to verify the correctness of the downloaded files. This part of the work has been detailed in the article Detailed Explanation of QWen3-VL Model Testing and vLLM Production Environment Deployment Workflow. The HuggingFace link path for the Qwen3-VL-4B-Instruct model is shown below. Follow the aforementioned article to download it using the transformers library and perform local inference verification: !image.png > Note: The local path of the downloaded model files used in the following text is /mnt/d/HuggingFace/hub/models--Qwen--Qwen3-VL-4B-Instruct/snapshots/ebb281ec70b05090aa6165b016eac8ec08e71b17/ Environment Setup Adapting multimodal large models on Rockchip NPU platforms requires both Rockchip's rknn-llm and rknn-toolkit2 toolkits. As of January 2026, the latest versions of these two toolkits on GitHub are: - rknn-llm: 1.2.3 - rknn-toolkit2: 2.3.2 The following work is based on these two versions. Oddly enough, the Python version requirements of the latest versions of these two toolkits conflict with each other. Therefore, it is best to set up separate development environments (rknn and rkllm) based on the two toolkits respectively, and switch between them during different subsequent steps. For the environment setup of rknn-toolkit2, refer to Rockchip RK3588 NPU Development Environment Setup Notes. For the environment setup flow of rkllm, refer to Adaptation Process of Lightweight LLM Model QWen2.5-0.5B on RK3588. _Overall, the adaptation of the QWen3-VL-4B-Instruct model on the RK3588 development board is mainly carried out based on the files contained in the rknn-llm-main/examples/multimodal_model_demo directory within the rknn-llm repository._ - There is a README.md under this directory that can serve as a reference, but it lacks some details. Model Conversion The conversion of multimodal models like Qwen3-VL-4B-Instruct is actually divided into two steps, corresponding to the conversion of the RKNN model and the RKLLM model respectively. The final on-board demo requires loading independent RKNN and RKLLM model files simultaneously, and also depends on their respective runtime libraries: librkllmrt.so and librknnrt.so. RKNN Model Conversion This conversion operation is executed in the rknn environment (i.e., the environment where the rknn-toolkit2 toolkit is installed). This part is divided into two sub-steps: converting the Vision part of the model into an ONNX file, and then converting the ONNX format model file into an RKNN model file for on-board execution. Execute the export_vision.py script in the export sub-directory under rknn-llm-main/examples/multimodal_model_demo to convert the Vision part of the model into an ONNX file: After the execution above is complete, an onnx sub-directory will be generated in the current export directory, containing the converted ONNX file for the Vision part of the model: Next, continue using the export_vision_rknn.py script in the same directory to convert this ONNX file into an RKNN model file that can run on the board: After the execution above is complete, an rknn sub-directory will be generated in the current export directory, containing the on-board RKNN model file for the Vision part of the model: At this point, the model file for the Vision part (qwen3-vl_vision_rk3588.rknn) is ready. RKLLM Model Conversion This conversion operation is executed in the rkllm environment (i.e., the environment where the rknn-llm toolkit is installed). This part can also be divided into two sub-steps: preparation of calibration data and conversion of the RKLLM on-board model. The calibration data preparation work is mainly carried out in the data sub-directory. This directory contains a make_input_embeds_for_quantize.py script used to prepare calibration data, but this script is tailored for Qwen2-VL and will throw an error if used directly on the Qwen3-VL model. Therefore, I modified the script based on the original one to create make_input_embeds_for_quantize_qwen3-vl.py specifically for generating calibration data for the Qwen3-VL model. Then execute under export: After the above execution completes, calibration data will be generated in data/inputs.json and data/inputs_embeds/, which will be used in the subsequent RKLLM on-board model conversion process. Next is the final and most time-consuming step of model conversion (mainly because my graphics card model is relatively new and conflicts with the torch and cuda versions in the rkllm environment, forcing me to perform model conversion using the CPU): converting the RKLLM on-board model file. This primarily uses the export_rkllm.py script under export: Note that the execution of the above script has huge memory requirements. If executed in WSL, it will result in a "Killed" error due to out-of-memory (OOM). At this point, you need to modify the memory and swap configuration of WSL: create a .wslconfig file in your user directory C:\Users\[Username], write the following content into it, restart WSL, and then re-run the above conversion command: Once the execution above is complete, the RKLLM on-board execution file _w8a8_rk3588.rkllm will be generated in the current directory: At this point, the two on-board model files required to run the on-board inference demo—qwen3-vl_vision_rk3588.rknn and _w8a8_rk3588.rkllm—are ready. Compilation of the On-Board Inference Application The general on-board inference application code is located in the deploy directory. Note that RKLLM v1.2.3 requires a cross-compilation GCC version of 10.2 or higher. You need to download the toolchain gcc-arm-10.2-2020.11-x86_64-aarch64-none-linux-gnu first: !image.png In the deploy directory, modify the cross-compilation toolchain path GCC_COMPILER in the build-linux.sh file, and then run the build-linux.sh script to compile the on-board execution files. Once this compilation process is finished, all files required for on-board execution (except for the two model files mentioned above) will be placed in an install directory. On-Board Execution Testing To perform the on-board execution test, upload the install directory and the two on-board model files (qwen3-vl_vision_rk3588.rknn and _w8a8_rk3588.rkllm) to the board's models directory. Then execute: The final running effect of the demo on the RK3588 is shown in the figure below: !image.png References - airockchip/rknn-llm - airockchip/rknn-toolkit2 - 【Hardcore Deployment】Deployment of Qwen3-VL on RK3588 - CSDN Blog --- Attempting to understand and explain technical issues from the perspective of underlying principles: Audio/Video/Cameras/Smart Home/Bluetooth/WiFi/Wireless Communication/AI. Please follow the WeChat Official Account: Pavel Han.
Detailed Summary of DETR (DEtection TRansformer) Network Architecture and Computational Workflow
This article provides a detailed introduction to the network structure and data computational workflow of DETR, an image object detection model based on the Transformer architecture proposed by Facebook in 2020. Introduction to DETR DETR is an AI model for object detection in computer vision proposed by Facebook AI Research (FAIR) in 2020. It introduced the Transformer architecture, widely used in the NLP domain, into computer vision for the first time. It completely revolutionized the traditional design paradigm in object detection, simplifying the task from an accumulation of tedious components and processing pipelines into a genuine end-to-end set prediction problem. Before the advent of DETR, mainstream detectors in the object detection domain (such as Faster R-CNN, YOLO, and SSD) heavily relied on various engineering components: - Anchor Boxes: Presetting different scales and aspect ratios for detecting objects based on the overall distribution of the training dataset. - NMS (Non-Maximum Suppression): Used during post-processing to eliminate duplicate bounding boxes in the model's output data. - Highly complex positive and negative sample matching strategies. The core innovation of DETR lies in leveraging the global modeling capability of the Transformer structure to directly output a fixed-size set of predictions (typically 100), which are then matched one-to-one with the Ground Truth using a Bipartite Matching algorithm. Architecturally, the DETR network mainly consists of three components: a CNN Backbone, a Transformer encoder-decoder, and a detection head. !image.png CNN Backbone The DETR network architecture starts with a convolutional neural network (CNN) backbone. The backbone network employs a standard ResNet (in the original paper, the authors used ResNet-50 and ResNet-101). Acting as a feature extractor for the input image, the backbone processes the raw input image to generate rich visual feature vectors. - For details on the ResNet network architecture, refer to: CNN Classical Network Architecture Learning: ResNet. Regarding the dimensions of image processing, assuming the input image size to the backbone is , the feature extractor processes it through multi-stage ResNet blocks to output a downscaled feature map. In a standard DETR implementation, the feature map downsampling factor is typically 32, resulting in an output feature map dimension of . The channel dimension is usually projected to 256 using a convolution to match the input dimension of the Transformer. - If the input image resolution is , the resolution of the output feature map will be . Referring to the application of Transformers in computer vision within the ViT architecture (An Introduction to Vision in Transformer (ViT) Model Architecture), Transformers accept sequence data rather than matrix-form vectors. Therefore, the feature map of dimensions output by the ResNet backbone is flattened and straightened into a vector sequence of length 625, where each element is a 256-dimensional vector. !image.png Positional Encoding Similarly, while convolutional neural networks inherently perceive spatial information due to translation invariance and local receptive fields, Transformers process sequence data and require additional positional encoding information added to the flattened vector sequences—akin to adding positional information to text sequences in natural language processing. DETR adopts Fixed 2D Sinusoidal Positional Encodings for the positional information of the backbone output feature map sequence. The general approach is to split the encoding information for each pixel position into and directions, encode them separately, and finally concatenate them. As mentioned above, after flattening, each pixel's vector length is 256; hence, 128 dimensions are allocated to encode the -coordinate and another 128 dimensions to encode the -coordinate. Concatenating these two yields the final 256-dimensional positional feature vector. For the coordinates of each pixel on the feature map, each corresponds to a 128-dimensional vector composed of 64 pairs of sine and cosine functions. Their calculation formulas are as follows: The 128-dimensional vector composed of 64 pairs of sine and cosine functions for the -coordinate is calculated using the identical formula. The two 128-dimensional vectors are then concatenated to form a 256-dimensional positional vector. The dimension of the vector obtained in this way precisely aligns with the feature vector dimension after the convolution, and the positional encoding information also forms a vector sequence of length 625, where each element is a 256-dimensional vector. Transformer Encoder-Decoder As described above, after the backbone extracts image features and flattens the feature map, the data fed into the Transformer Encoder has a dimension of a sequence of length 625, with each element in the sequence having a dimension of 256. Note that the data output by the backbone is not directly added to the positional encoding vectors; instead, the positional vectors are injected into the calculations of Query and Key during attention computation, which differs from the standard NLP Transformer computational workflow. The figure below illustrates the processing architecture of the DETR Transformer Encoder and Decoder: !image.png Encoder The multi-head self-attention module of the Encoder uses 8 heads, so the dimension of each head is . The calculation of the , , and matrices for each head is as follows: - Query (): → Dimension: - Key (): → Dimension: - Value (): → Dimension: > Note: When calculating Queries and Keys, the feature vector extracted by the backbone is added to its positional encoding . The calculation of Values does not incorporate positional encodings because Values are responsible for providing content information, while and are responsible for localization and matching. Subsequent calculations for the multi-head attention module follow standard procedures, which are detailed in the article How to Understand Multi-Head Attention Mechanism in Transformer Architecture?. In summary, in terms of input and output data dimensions, the flattened feature vector and positional encoding vector input to each Encoder module (the dashed box on the left of the architecture diagram above) have dimensions of , while their output data dimensions are also . The DETR model defaults to containing 6 consecutive Encoder modules, and the final output data dimension of the Encoder section remains . The working logic of the Encoder is: Utilizing the global attention mechanism to perform context modeling on the local features extracted by the preceding backbone convolutional network, ultimately outputting an enhanced feature sequence of the same size as the input, where each point contains global contextual information and semantic features. Decoder To understand the architecture of the Decoder section, we first define its inputs. The input to the Decoder consists of two parts: - The output of the Encoder, which is intuitively the tensor output by the final Encoder module. - Object Queries: A tensor of dimensions . During the training phase, these are learnable parameters (similar to embeddings). After training is complete, this dimensional tensor serves as fixed parameters for subsequent image inference workflows. These 100 vectors represent 100 candidate bounding boxes. As shown in the Decoder Block architecture below, each Decoder Block sequentially comprises three parts from bottom to top: a Multi-Head Self-Attention module, a Multi-Head Attention (Cross-Attention) module, and a Feed-Forward Network (FFN). !image.png First is the self-attention module (the Multi-Head Self-Attention module in the diagram above). The computational workflow of this self-attention module differs slightly from standard Transformer self-attention: standard Transformer self-attention takes input vectors augmented with positional encoding information and multiplies them with parameter matrices to obtain , , and vectors; whereas the self-attention module here follows this calculation logic: The above self-attention calculation logic effectively uses the input Object Queries simultaneously as both input data and its positional encoding vector, which is why self-addition of Object Queries is performed during the calculation of and in self-attention. > Why perform self-addition on the identical input Object Queries rather than multiplying the input directly by 2? This is because there are actually 6 consecutive stacked Decoder blocks. Only the first layer's decoder uses the self-addition of Object Queries as input when calculating and ; subsequent decoders use the sum of Object Queries (acting as positional vector information) and the output of the previous layer's Decoder as their input during self-attention calculations. After the calculation through the self-attention module above, the output data dimension is . Next is the calculation of the Multi-Head Attention (Cross-Attention) module. In the cross-attention module, the output of the Decoder's self-attention module acts as the active questioner, while the feature map output by the Encoder serves as the knowledge base. The cross-attention module takes three inputs: object_queries output by the Decoder's self-attention module (dimension ), encoder_features output by the Encoder (dimension ), and spatial positional encoding information identical to that of the Encoder (dimension ). Under this setup, the calculation logic and data dimensions of the three tensors are as follows: - Query (): → - Key (): → - Value (): → The subsequent calculation logic of the cross-attention module and the feed-forward network (FFN) module remains identical to the standard Decoder Block workflow (refer to A Comprehensive Guide to the Decoder Structure and Computational Workflow of the Transformer Model for details). Ultimately, the output dimension of the entire Decoder Block is identical to its input dimension, both being . The Decoder section on the right side of the DETR network structure contains a total of 6 consecutive stacked Decoder blocks, where the output of each block serves as the object queries input for the next block. Thus, the final output data dimension of the entire Decoder section remains . Detection Head As mentioned above, after continuous stacking and processing through the 6 Decoder blocks of the entire Decoder section, the final output data dimension of the Decoder is . The task of the final detection head in the DETR model is to map these 100 high-dimensional vectors into human-readable class labels and bounding box coordinates. For the design of the detection head structure, DETR adopts two parallel FFN (Feed-Forward Network) branches that share the Decoder's output but handle distinct tasks. !image.png The Classification Head structure is a simple Linear Layer. The input data dimension of this linear layer is , and the output data is , where is the number of target dataset classes (such as 80 classes in COCO), and represents the background class (no object). The final activation function uses Softmax, which assigns a probability distribution to each query, indicating which class the object corresponding to that query belongs to. The Bounding Box Regression Head structure is a Multi-Layer Perceptron (MLP) with 3 hidden layers, each interspersed with a ReLU activation function. The input data dimension for the entire regression head is , and the output data dimension is . The numerical values represent the normalized center coordinates, width, and height: . With the above output head structure, for each image, the DETR model outputs coordinate information for 100 bounding boxes (from the regression head), with each bounding box corresponding to classification information of length . When filtering background boxes, one simply needs to set a threshold (e.g., 0.5) to filter out prediction boxes whose classification result is the background class (no object) or whose scores are too low, thereby eliminating the need for the extremely complex NMS computation pipeline found in models like YOLO. References - facebookresearch/detr: End-to-End Object Detection with Transformers - Introduction to DETR - Part 1 | DigitalOcean
Adaptation and Deployment Process of the Lightweight LLM Qwen2.5-0.5B on RK3588
This article provides a detailed study and summary of Rockchip's RKLLM tool and its official related guidance documents, documenting the complete process and precautions for adapting the lightweight large language model Qwen2.5-0.5B to the RK3588. Overall, deploying an LLM on the Rockchip NPU follows the same core logic and workflow as deploying YOLO: PC-side model conversion -> pushing the model and libraries to the board -> calling the API to run. However, for large language model adaptation, the RKNN-Toolkit2 toolchain is no longer used; instead, the RKLLM-Toolkit toolchain, optimized specifically for large language models, is employed. - RKNN-Toolkit2: Handles Convolutional Neural Networks (CNNs) and simple Transformers (Vision Transformers). - RKLLM-Toolkit: Specifically handles large language models, supporting weight quantization (W4A16/W8A8) and KV Cache optimization. The following diagram illustrates the framework of Rockchip's RKLLM Toolkit and its software-hardware application ecosystem, which is similar to RKNN: !image.png - RKLLM-Toolkit: Used for model conversion and quantization on the PC. - RKLLM Runtime: Provides C/C++ programming interfaces for Rockchip NPU platforms, helping users deploy RKLLM models on the NPU and accelerate the execution of LLM applications. - RKNPU Kernel Driver: Responsible for interacting with the NPU hardware, which can be found in Rockchip's kernel code. Environment Setup Download the rkllm codebase from the rkllm GitHub repository (airockchip/rknn-llm). PC-Side Environment Create an independent rkllm environment, navigate to the rkllm-toolkit/packages directory within the rknn-llm repository, and install rkllm-toolkit along with its dependencies: Board-Side Environment For running LLMs on the Rockchip NPU, the board-side mainly requires support for the NPU driver and the librkllmrt.so shared library (located under rkllm-runtime/Linux/librkllm_api/aarch64). The NPU driver requires version v0.9.8 or higher. You can check whether your board's driver meets the requirement using the following command; otherwise, you will need to recompile the kernel: Downloading the Model File and Inference Testing on PC Use transformers to download the Qwen2.5-0.5B model files from Hugging Face and perform a local inference test on your PC to verify the integrity of the downloaded files: Slow download speeds from Hugging Face can be resolved by setting a domestic mirror. Set the environment variables before running the download script above: The list of downloaded model files is shown in the figure below: !image.png Model Conversion Using the rkllm-toolkit package installed in the environment setup section, execute the following code to complete model conversion: > Note: Regarding quantization type support, the RK3588 does not support w4a16, but supports four types: w8a8, w8a8_g128, w8a8_g256, and w8a8_g512. After the model conversion finishes, a qwen2_5_0_5b_w8a8.rkllm file of around 800MB will be generated in the current directory. Board-Side Compilation and Inference Once the board-side model conversion is complete, the next step is to prepare the board-side inference code: For cross-compiling the board-side inference code, you can follow standard development methods by compiling on the PC and then uploading to the board for execution. However, for the RKLLM-Toolkit version 1.2.3 that I used, the explicitly recommended cross-compilation tool version in the documentation is GCC 10.2, whereas the compilation toolchain for the Rockchip platform on my PC was gcc-linaro-6.3.1-2017.05-x86_64_aarch64-linux-gnu. Fortunately, the Debian system running on my Rockchip RK3588 board had a compilation tool that met the requirements: Therefore, things became much simpler: you can directly upload the board-side inference source file (main.cpp), header file (rkllm.h), shared library (librkllmrt.so), and the converted board-side model to the board, and compile directly using g++ in the board's Debian system: The print output after execution is as follows: As can be seen, this model is now up and running on the RK3588. However, the content generated by a 0.5B-scale LLM still has some noticeable issues: not only does it casually "solve" the Xi'an-Xianyang integration issue that has troubled Xi'an's development for 30 years, but it also mistakenly attributes Yulin's data to Xianyang. Regarding memory consumption during inference, this model only uses about 680MB of memory. Given that the board-side converted model file itself is already 800MB, this indicates that Rockchip RKLLM does not load the entire model file completely into memory during execution, but rather loads it on-demand. This makes it extremely practical for memory-constrained applications like embedded devices. References - airockchip/rknn-llm - RKLLM SDK User Guide v1.2.3