All Articles
An Analysis of the Workflow and Security Mechanisms of Stripe Online Payments
In the web world, handling user credit card numbers during online payment processes is an extremely high-risk operation. For full-stack developers implementing overseas business scenarios within their applications, they must deal with cross-border credit card network communications, 3D Secure dynamic authentication, and strict PCI-DSS security compliance audits. To address this challenge, Stripe emerged. Its brilliance lies in encapsulating chaotic real-world capital flows into highly predictable, developer-friendly interfaces through exceptionally elegant RESTful APIs and a webhook-based asynchronous event-driven model. Under this architecture, developers' own servers never need to touch, transmit, or store the user's real credit card numbers in plaintext; they simply send instructions to Stripe, and Stripe returns a token indicating the execution result after processing, greatly simplifying the complexity of payment operations. Explanation of the Stripe Online Payment Execution Workflow The flowchart below illustrates the complete workflow of Stripe online payments, using an Android app completing an online payment via Stripe as an example. The entire payment process can be divided into four stages: - Stage 1: Establishing the Payment Session - First, within the Android app, the user initiates the payment process by clicking the button to purchase a product or service. At this point, the app sends a payment purchase request to the developer's backend Spring Boot server. - Upon receiving the request, the backend server creates a new purchase order in its PostgreSQL database. Since the payment has not yet started, the order status is set to Pending. - Next, the backend server sends a request to the Stripe Cloud to create a payment session request. - After receiving this request, the Stripe server returns the session URL and token ID to the backend server. - The backend server then forwards the URL and token ID as-is to the app via a persistent connection between them. - Stage 2: The Payment Interaction Between the Client and the Stripe Server - Following the first stage, the app receives the payment session URL and token from the Stripe server. The app then visits the URL to interact directly with the Stripe server and complete the actual payment process. This step is entirely transparent to both the developer's app and the backend server. - Stage 3: Notifying the Backend Server Upon Payment Completion - When the user finishes the payment process with Stripe in the app, the app UI remains on a page waiting for payment confirmation. Meanwhile, the Stripe server sends a payment confirmation message to the developer's backend server. - Upon receiving the payment confirmation, the backend server updates the purchase order status to PAID in the PostgreSQL database and modifies the user's entitlement information accordingly. - Finally, the backend server sends a purchase completion confirmation back to the app via WebSocket or polling requests. The app then refreshes its UI to display the purchased product or service information to the user. !stripe-architecture-flow.png The Webhook Mechanism for Stripe Server Payment Event Notifications In the workflow described above, after the developer's backend server requests Stripe to create a payment session (Step 3 in the flowchart above), the Stripe server does not open and maintain a persistent connection session with the backend server during subsequent payment sessions (as this would waste too much Stripe server memory). Instead, it uses a webhook mechanism: once the user completes the payment, the Stripe server proactively initiates a request to the Webhook URL registered by the developer's backend server to notify it of the payment completion status. Therefore, to support this webhook mechanism, when developers register their Stripe accounts, they must pre-register their server's public IP address in the Stripe developer dashboard. This allows the Stripe server to know the Webhook URL and send payment completion notifications to it after users finish paying. Consequently, communication between the developer's backend server and the Stripe server always follows a standard RESTful stateless communication model. Each time the developer's backend server sends a request to the Stripe server to create a payment session, the Stripe server returns a payment session URL and token ID, after which the connection is closed. Once the payment is later completed, the Stripe server uses the webhook mechanism to send the event completion notification back to the developer's backend server. Below is a flowchart showing the Stripe server proactively initiating a payment event completion notification to the Webhook URL after a user finishes making a payment. This includes cryptographic authentication information to ensure communication security, while also handling potential webhook communication failures by having the Stripe server retry event notification requests over a certain period to ensure a complete closed-loop process for payments and notifications: !stripe-server-webhook.png Frontend-to-Stripe Communication Modes: Checkout vs. Payment Element For the frontend (app, web), after initiating a payment session request and obtaining the session URL and token ID returned by the Stripe server from the backend server, the actual payment process is completed through communication between the frontend and the Stripe server. Communication modes between the frontend and Stripe primarily fall into two categories: Checkout and Payment Element. Checkout Mode The operational mechanism of Checkout mode is as follows: when the backend server calls the Stripe API to create a payment session, the Stripe server generates a logical order and dynamically renders a single-page application (SPA) webpage dedicated to that order, hosted under Stripe's domain (typically checkout.stripe.com). This Checkout URL is then returned to the developer's backend server and forwarded to the frontend. In this scenario, developers only need to access this Checkout URL within their app or frontend webpage via redirection or by opening a built-in browser. Stripe injects powerful features into this page. If it detects that a user is using Safari, it automatically displays Apple Pay; if Chrome, Google Pay; it even automatically switches languages and local currencies based on the user's IP address (such as automatically showing iDEAL payments in Europe). Developers do not need to write a single line of code for these complex detection logics, as they are entirely handled automatically by the Stripe server. Payment Element Native Mode The operational mechanism of this mode is that the user does not leave the developer's domain or app during the checkout process. Developers are required to design and render a visually appealing checkout page (containing shopping carts, shipping addresses, etc.) on their own web page or app frontend using HTML/CSS or native Android components. In other words, the checkout page is designed and presented to the user by the developer. Of course, to ensure the payment process remains securely isolated, the specific area on the frontend checkout page where users input their credit card numbers must integrate Stripe's frontend SDK to request the rendering of a Payment Element component. On the web, this component is essentially a cross-origin (or a secure view inside an app). With this design, even though the outer input fields of the payment page are designed by the developer, when the user types their bank card number, it is entered directly into Stripe's iframe/secure view and sent straight to Stripe's vault. The developer's own outer JavaScript or Android code cannot read these plaintext credentials, perfectly bypassing PCI compliance audits and ensuring that the entire payment lifecycle and its security are managed entirely by Stripe. The process flowcharts for these two modes are illustrated in the figure below: !stripe-checkout-vs-element.png How does Stripe distinguish between these two modes? The answer is: Stripe does not care which mode the frontend uses at all. It only cares whether the API request initiated by the developer's backend server to the Stripe server uses the Checkout API or the Payment Element API. - When using Checkout mode, the API request sent by the backend server to Stripe is POST /v1/checkout/sessions (refer to https://docs.stripe.com/api/checkout/sessions). At this point, the Stripe server generates a webpage with a payment interface and returns the URL of this webpage to the backend server. - When using Payment Element mode, the API request sent by the backend server to Stripe is POST /v1/payment_intents (refer to https://docs.stripe.com/api/payment_intents). At this point, the Stripe server creates an order record in memory and returns a one-time secret key to the backend server, after which the frontend communicates with the Stripe server based on this key to complete the payment. Stripe's Secure Communication Mechanisms and Keys As shown in the flowcharts above, the entire payment communication process involves three parties: the frontend, the backend, and the Stripe server. To ensure communication security throughout the payment process, Stripe provides three sets of keys for identity authentication and signature verification. These three sets of keys are generated in the Stripe developer dashboard: - Publishable Key (Public key, starting with pk_): Hardcoded into the frontend (app/web) code. Its role is to allow the frontend to prove "whose frontend I am" when communicating with the Stripe server. - Secret Key (Private key, starting with sk_): Must and can only be securely stored in the backend server's environment variables, and must never be leaked to the frontend or committed to a Git repository. Its role is to be placed in the HTTP Authorization: Bearer header when calling Stripe APIs from the backend (such as creating orders, issuing refunds, or querying bills), allowing the Stripe server to identify which merchant the payment request originates from. - In addition to the Secret Key, there is actually a Restricted Key. Its function is similar to the Secret Key, but it is a subset of the Secret Key's permissions, possessing only partial privileges. If only payment session creation capabilities are needed, a Restricted Key can be used to prevent the leakage of higher-privilege Secret Keys. - Webhook Secret (Signing key, starting with whsec_): Also kept exclusively in the backend server's environment variables. Its role is specifically to verify whether Webhook requests sent from Stripe to the backend server have been tampered with by hackers along the way. It should be noted that all communications across the entire Stripe payment workflow take place over HTTPS/TLS connections. The three sets of keys mentioned above are transmitted in plaintext within the encrypted HTTPS channel solely for identity authentication and do not participate in data encryption or decryption operations. As illustrated in the diagram below, three sets of keys are used across the payment workflow for communication role identification and authorization: - Step 1: In the communication where the backend server sends a payment session request to the Stripe server, the backend server must include its SK/RK in the HTTP message. Upon receipt, the Stripe server checks the identity ownership of this SK/RK and verifies whether it has the permission to create sessions. - Step 2: During the actual payment interaction between the frontend and the Stripe server, the frontend must pass its PK to the Stripe server in its HTTP message, allowing the Stripe server to identify the origin of the frontend communicating with it. - Step 3: After the payment process between the frontend and the Stripe server concludes, the Stripe server sends an event notification to the backend server via the webhook mechanism. Upon receiving this message, the backend server must use the Webhook Secret key to verify the message's signature, ensuring that the message genuinely originated from the Stripe server. !68951394-a233-4ed6-9d9b-76fd3a268714.png
Clarifications on Several Issues in the RAG System Workflow
This article briefly summarizes the overall workflow and module decomposition of Retrieval-Augmented Generation (RAG) systems, and clarifies several issues related to the RAG workflow, vector data storage, and vector retrieval based on this foundation. Basic Workflow and Core Module Composition of RAG Systems The following diagram illustrates a typical workflow breakdown of a RAG system: !78e0975c-3248-43c4-a2f6-100c321de106.png From a module decomposition perspective, a RAG system primarily consists of the following independent modules: - Indexing: Cleans unstructured documents, splits them into chunks of varying lengths based on semantics, embeds them (i.e., converts them into vectors), and stores them in a Vector Database (Vector DB). - Retrieval: Calculates the cosine similarity between the user's query text and the vectors in the database based on the input query message, and recalls the Top-K relevant snippets from the vector database. - Augmentation: Concatenates the snippets retrieved from the vector database with the original query text to construct an augmented prompt. - Generation: Feeds the augmented prompt into the LLM, which then generates the final answer based on the augmented context. The above workflow and module breakdown are simple and straightforward. The rest of this article clarifies several basic concepts regarding the operation of RAG systems to ensure a clearer understanding of all aspects of the system's operation. Dimensionality Issues of Stored Vectors in Vector Databases As shown in the diagram above, documents added to the knowledge base are first split into chunks of varying lengths based on semantics (meaning each chunk contains a varying number of tokens). If the number of tokens in each chunk differs, does it mean that their corresponding embedding vectors will also have different data dimensions? _In reality, regardless of whether the input text (including the user's query text and the text chunks decomposed from the RAG knowledge base) is long or short (as long as it falls within the model's maximum context window), the output vector dimension (Dimension) after processing by the same embedding model is always fixed and identical._ Therefore, even if two chunks have different lengths and contain unequal numbers of tokens, the dimensions of the converted vectors remain identical and fixed after passing through the embedding model. This fixed vector dimension matches the token dimension specified by the embedding model. In the concrete transformation pipeline, whether it is Chunk A of the knowledge base text or the Query, after model dimensionality reduction and feature extraction, the output is always a dimensional floating-point array (a 1D vector). For this exact reason, similarity calculations can subsequently be performed between two vectors of completely identical dimensions (i.e., the vector corresponding to the user's query text and the vector corresponding to a specific chunk in the vector database). How do texts of different lengths get transformed into vectors of the same length? This is because the underlying Large Language Model (based on the Transformer architecture) employs a Pooling mechanism in its output layer. The specific implementation consists of the following two steps: - Token Mapping: Texts of varying lengths are first converted into token sequences, and each token is mapped into a hidden state vector within the model. At this stage, this is an matrix, where is the number of tokens and is the token dimension. - Pooling (Dimensionality Reduction and Aggregation): The model compresses this matrix into a vector via a specific strategy to represent the global semantics of the entire text segment. Common approaches for this pooling operation include: - Mean Pooling: Averages the vectors of all tokens across each column. - CLS Token: Extracts the vector corresponding to the special [CLS] (Classification) token at the beginning of the input sequence, which is trained to represent the semantics of the entire sentence. Vector Similarity Calculation Logic As described above, whether it is the user's query text prompt or the vector corresponding to a knowledge base document segment stored in the vector database, their dimensions are identical at . When subsequently calculating the similarity between two vectors, the primary logic utilized is Cosine Similarity and Dot Product. Cosine similarity measures the angle between two vectors in space. Taking vector (Query) and vector (Chunk) as examples, both having a dimension of : > 💡 In natural language processing or audio retrieval, the norm (magnitude) of a feature vector often represents the length of the text or the energy level of the audio segment, whereas the direction of the vector represents the true semantics or feature attributes. Therefore, using the cosine similarity formula above decouples the scale from the abstract semantics. The numerator portion () is actually the dot product between the two vectors, which measures the absolute overlap between the two tensors across all dimensions. The denominator portion () is a scalar acting as a normalization factor to forcibly eliminate scaling errors caused by differences in tensor norms (magnitudes). For two vectors of the same dimension calculated via the formula above, the result ranges between -1 and 1: - A result closer to 1 indicates that the angle between the two vectors is smaller, meaning the vectors nearly overlap in space, their semantics are very similar, and they represent the perfect target for RAG recall. - A result trending toward 0 indicates that the vectors are orthogonal (perpendicular) to each other in space, implying that the two vectors are completely unrelated (i.e., orthogonal), the two text segments discuss entirely different topics, and their feature dimensions do not intersect. - A result close to -1 indicates that the vectors point in opposite directions in space, representing opposite/antonymous semantics. In certain specifically trained embedding spaces, this scenario denotes complete semantic opposition (such as the extremes of good and bad). The vector database then performs the above calculations between all stored vectors and the user's query vector based on this formula (or its approximate efficient algorithms, such as HNSW), recalling the Top-K documents with the matching calculation results. In engineering practice, if all document vectors have undergone normalization prior to storage in the vector database (ensuring their magnitudes equal 1), and the query vector has also undergone the same normalization operation, the denominator in the formula above becomes 1. At this point, the cosine similarity calculation simplifies into a dot product calculation: Vector Database Retrieval Workflow The previous sections outlined the calculation process for finding the vector closest in semantics to the user's query via the cosine similarity formula between two vectors. However, if the RAG database stores a massive volume of content and a large number of records, performing a cosine similarity calculation against every single record in the database for every retrieval operation would result in an astonishing computational load. Below is a detailed summary and explanation of the HNSW (Hierarchical Navigable Small World) algorithm, which is the most mainstream algorithm applied in the field of high-speed vector retrieval. Overall, the HNSW algorithm adopts a multi-layer graph spatial approach to solve the computational and search problems of massive vectors. Under this multi-layer graph structure, nodes in the upper layers are sparse and responsible for quickly locating the general region of the target node, while nodes in the lower layers are dense and responsible for executing fine-grained searches to accurately locate the target node. With this design, the vast majority of nodes are not directly connected, yet any two points can be found in just a few steps. The multi-layer graph structure essentially places each vector in the vector database into different layers during the indexing stage. Upper-layer nodes contain connection information for the lower layers; thus, during vector retrieval, the system first performs a rough localization via upper-layer nodes, then descends to the bottom layer using the lower-layer connections found within those upper nodes, progressively and precisely locating the nearest neighbor node. The information contained within each node can be roughly described using the following data structure: - Layer 2 nodes (i.e., Max Level = 2): Contain connection information for Layer 0 + Layer 1 + Layer 2. - Layer 1 nodes (Max Level = 1): Contain connection information for Layer 0 + Layer 1. - Layer 0 nodes (Max Level = 0): Contain connection information for Layer 0 only. In this scenario, every time a vector search is performed, the system first finds the closest vector Node-Layer-2 in the Layer 2 vector list, retrieves the layer_1_neighbors list saved within Node-Layer-2 to find the closest vector Node-Layer-1 in Layer 1, and subsequently retrieves its corresponding layer_0_neighbors list from Node-Layer-1 to finally identify the Layer 0 vector closest to the target vector from this list. Therefore, the vector search workflow of the HNSW algorithm roughly mirrors the diagram below. Because practical RAG workflows do not merely return a single record, but rather return Top-K records, a candidate list is often added at each layer during the retrieval process, with the final returned result being the list of the top vectors closest to the query text vector: !HNSW_Architecture_Native.png Storage Content of Vector Databases The next question is: for a RAG vector database, vectors corresponding to all chunks after text splitting are stored within it. In engineering practice, what information does each record in these vector databases typically store? Generally speaking, a standard vector database record contains at least the following components: - Primary Key (Primary Key ID): The primary key of the database record, functioning similarly to the primary key in relational databases. The data format is typically Int64 or String (UUID), serving as the unique token for physical storage and in-memory index mapping. - Dense Vector: This is the core content stored in a vector database record. Its data format is a floating-point array (such as Float32, Float16, or INT8), corresponding to the vector content of each text chunk. Vector retrieval involves calculating the cosine similarity between this dense vector and the vector corresponding to the user's input prompt text. - Scalar Metadata: Used to store structural attributes accompanying the vector, generally used for attribute filtering (for instance, using this attribute to distinguish which text file a given text chunk originates from). Data types typically include Int, Varchar, JSON, and others. As we can see, the records in the vector database mentioned above do not contain the raw text chunk data. In this case, when a vector is found via the cosine similarity algorithm during subsequent vector searches, how does the system locate and retrieve the chunk text corresponding to this vector so that it can be concatenated with the user's input query prompt? To address this issue, the most straightforward approach is to store the content of the text chunk directly as a special field (such as a VARCHAR type) within the aforementioned Scalar Metadata portion, keeping it alongside the vector inside the vector database. This way, when the vector is retrieved later, the underlying system can directly return the chunk text string associated with that vector. However, the problem with this scheme is that the essence of vector retrieval computation is intensive floating-point arithmetic, which requires the feature arrays participating in the calculation to reside in the L3 cache or high-speed RAM. Plain text chunk data is entirely useless for the vector computation process. Loading the entire contents of the vector database (including the aforementioned chunk text content) into memory simultaneously results in severe memory fragmentation and cache misses during computation. Consequently, mainstream vector databases (such as FAISS and Qdrant) currently store only the [Primary Key, Dense Vector] portion during vector storage, while massive text information such as raw text chunks and metadata (e.g., source document names, page numbers) is stored in a separate, dedicated Key-Value database (such as RocksDB or Redis) or document database (such as MongoDB or Elasticsearch). In this setup, the Primary Key acts as a foreign key connecting the two heterogeneous databases. Once vector retrieval calculations locate the vector, its corresponding chunk text and other metadata can be retrieved from the other database via the Primary Key.
An Analysis of the Network Structure and Output Data of the YOLOv8-Pose Model
In the article Decoding the Network Architecture of the YOLOv8 Model, I provided a detailed explanation of the network structure of YOLOv8—especially the design and computation process of its detection head—based on the architectural differences between YOLOv8 and YOLOv5. This article focuses on comparing the network structures of YOLOv8 and YOLOv8-Pose, providing a detailed summary and explanation of the detection head, as well as the dimensions and structures of its output data. Reviewing the Detection Head Structure of the Standard YOLOv8 Model Let's state the conclusion first: The Backbone and Neck of YOLOv8-Pose and the standard YOLOv8 model are completely identical. The core architectural differences are 100% concentrated in the Head design, loss function definitions, and corresponding post-processing. Therefore, studying the YOLOv8-Pose model requires a solid understanding of the standard YOLOv8 model, which can be found in Decoding the Network Architecture of the YOLOv8 Model. The detection head structure of the standard YOLOv8 model is illustrated in the figure below. Simply put, the Backbone of the network extracts features from the input image, the Neck thoroughly fuses the feature information contained in feature maps of different scales, and the fully fused features are finally fed into the detection head. The detection head of the YOLOv8 model is divided into two independent processing paths: the regression head uses CIoU + DFL (abandoning the Anchor-Based mechanism used in previous generations) to calculate and output bounding box coordinates, while the classification head uses standard BCE to calculate the class probabilities for each bounding box. Ultimately, the detection head of the YOLOv8 model outputs 8,400 bounding box coordinates and their corresponding class information (with confidence directly taking the maximum value among the class probabilities). During the post-processing stage, valid bounding boxes are filtered out from these 8,400 candidates to form the final detection output. !image.png Detection Head Structure of the YOLOv8-Pose Model As mentioned above, the difference between the YOLOv8-Pose model and the standard YOLOv8 model lies solely in the Head: - Standard YOLOv8 Head: The biggest difference from previous generations like YOLOv5 is the adoption of a Decoupled Head, which splits into two parallel branches: the Box Branch (bounding box regression and DFL) and the Cls Branch (classification). - YOLOv8-Pose Head: Evolves into a Tri-Branch structure based on the standard YOLOv8 Head. Building upon the original Box and Cls branches, it adds an independent keypoint branch, the Keypoint Branch. The figure below is a schematic diagram of the detection head structure of the YOLOv8-Pose model: !e6de4f91-bf0b-4cbf-bb68-b57d9aa15400.png As can be seen, the coordinate regression head (Box Branch) and classification branch (Cls Branch) in the YOLOv8-Pose output head are identical to those of the standard YOLOv8 model. The only difference is that the Pose model introduces an additional, independent, and parallel Keypoint Branch. YOLOv8 Model Output and Filtering To understand the filtering logic and post-processing workflow of the YOLOv8-Pose model's output data, one must first thoroughly understand the post-processing workflow of the standard YOLOv8 model. The YOLOv8 model is pre-trained on the COCO dataset by default. The COCO dataset contains 80 classes; therefore, for the standard YOLOv8 model, the data dimension of the coordinate regression head for each bounding box is 64 (for details, refer to Decoding the Network Architecture of the YOLOv8 Model), the classification head outputs an 80-dimensional vector for each box (due to the 80 classes), and the feature maps across three scales (P3/P4/P5) jointly output 8,400 bounding boxes. Thus, the total output dimension of the detection head is [8400, 144]. Before entering the NMS post-processing stage, the 64-dimensional data for each bounding box from the regression head is converted into 4-dimensional absolute coordinates (typically ) via DFL integration (Softmax weighted summation). Consequently, the raw output tensor fed into the post-processing stage (including NMS) contains data where each bounding box consists of: 4 dimensions for coordinates + 80 dimensions for class probabilities = 84 dimensions. At this point, the output dimension of the YOLOv8 model is [8400, 80 + 4]. Next, based on the 80-dimensional class data from the classification branch, a Max operation is applied to retain only the highest class score and its index for each bounding box, using this maximum score as its confidence. The data dimensions then become: - Coordinates: - Score: - Class: The subsequent post-processing workflow mainly consists of two serial core steps and two thresholds (conf_thres and iou_thres): - Initial Confidence Filtering: A coarse filter based on the aforementioned confidence score to remove all bounding boxes with a confidence lower than conf_thres. This eliminates the vast majority of background boxes using minimal computational overhead. - IoU-based Suppression (IoU-based NMS): A refined filtering process that uses the Intersection over Union (IoU) metric (if the IoU between two boxes exceeds iou_thres, they are considered the same object) to discard redundant boxes for the same object. The bounding boxes surviving this IoU NMS filtering constitute the final detection output of the model. YOLOv8-Pose Model Output and Filtering The YOLOv8-Pose model is pre-trained on the COCO Keypoints (Pose) dataset. In the official annotations of this dataset, only the "person" class has complete annotations for 17 keypoints, while the other 79 classes lack keypoint annotations. Therefore, when Ultralytics officially trains yolov8n-pose.pt, its configuration file (e.g., coco-pose.yaml) hardcodes nc: 1 (Number of Classes = 1, restricting detection to the person class only). This is why the Cls branch dimension in the official pre-trained YOLOv8-Pose model is 1 instead of 80. For the keypoint branch output, when using the COCO Keypoints (Pose) dataset, the annotation data includes 17 human keypoints. Each keypoint requires regressing 3 values: - The coordinate position on the feature map - The coordinate position on the feature map - The visibility confidence, used to determine whether the point is occluded or outside the image field of view Consequently, the keypoint branch outputs dimensions of keypoint data for each bounding box. So, for the YOLOv8-Pose model, given the addition of a keypoint branch, what is the post-processing logic from model output down to the final NMS-filtered output? The answer is that during the post-processing logic of the YOLOv8-Pose model, the post-processing workflow pays no attention to what the keypoints look like; keypoints are simply treated as auxiliary data attached to the bounding box. If the box lives, the points live; if the box dies, the points die. In other words, in the post-processing workflow of the YOLOv8-Pose model, only the outputs of the Box coordinate regression branch and the Cls classification branch participate in the initial confidence filtering and IoU calculation (this part of the workflow is identical to the standard YOLOv8 model). This post-processing procedure ultimately yields the indices of the surviving boxes. When organizing the final output data, these surviving box indices are used to retrieve the coordinates, classes, confidences, and the associated 51-dimensional keypoints of the surviving boxes all at once from the original [8400, 56] tensor. - After filtering through the post-processing workflow, if 3 person bounding boxes are obtained, the positions of each keypoint for those 3 person boxes will be output simultaneously. Fundamentally, therefore, the post-processing workflow of the YOLOv8-Pose model is identical to that of the standard YOLOv8 model, with the keypoint branch data simply being "tacked on" during the final output stage. YOLOv8-Pose Output from a Source Code Perspective Below is a code snippet for performing inference on image data using the YOLOv8-Pose model: The inference result results in the code above is actually a Python list object whose length equals the batch size of the input data. If a single image is fed into the model for inference during each call, the core data of the inference result resides entirely in results[0]. results[0] is an instance of the ultralytics.engine.results.Results class. It acts like a container, packing different sub-objects internally according to the task: - results[0].boxes: Encapsulates all bounding box information filtered by NMS. - results[0].keypoints: Encapsulates all keypoint information accompanying the surviving bounding boxes. - results[0].orig_img: Stores the raw input Numpy image (primarily used when subsequently calling plot to draw bounding boxes and keypoints on the base image). - results[0].names: A mapping dictionary of detection classes (e.g., {0: 'person'}). The primary data information and dimensions contained within the bounding boxes (where is the number of valid detection boxes output by the model) are: - boxes.xyxy: Dimension [N, 4], corresponding to the absolute coordinates of each box: top-left x1, y1, bottom-right x2, y2. - boxes.xywh: Dimension [N, 4], corresponding to the center coordinates, width, and height of each box (cx, cy, w, h). - boxes.conf: Dimension [N], the maximum class confidence score for each of the boxes. - boxes.cls: Dimension [N], the class index corresponding to each of the boxes. - boxes.data: Dimension [N, 6], the raw concatenated tensor arranged sequentially as x1, y1, x2, y2, conf, cls. The primary data information and dimensions contained within the keypoint data (where is the number of valid detection boxes output by the model) are: - keypoints.xy: Dimension [N, 17, 2], representing persons, 17 keypoints per person, with absolute coordinates x, y for each point. - keypoints.xyn: Dimension [N, 17, 2], normalized coordinates ( divided by image width, divided by image height), ranging between [0, 1]. - keypoints.conf: Dimension [N, 17], representing persons, with visibility/confidence for 17 keypoints per person. - keypoints.data: Dimension [N, 17, 3], the concatenated tensor arranged in the order x, y, conf (i.e., the reorganized form of the aforementioned 51-dimensional data).
Summary of KV Cache Workflow in Large Language Model Interactions
By describing the interaction process between agents and large language models (LLMs), this article explains in detail the difference between Input Tokens and Cached Tokens. Understanding this distinction makes it easy to grasp the differences in computing power requirements and pricing between these two types of tokens. The Difference Between Input Tokens and Cached Tokens When purchasing token quotas from various LLM vendors, we often see three different prices: input price (cache hit), input price (cache miss), and output price. While the concepts of input and output tokens are straightforward, what is the difference between cache hit and cache miss here? !b8e61c06-b89d-4e35-ae1a-41f50eca9547.png Similarly, when using various agents such as Claude Code or Gemini CLI, upon exiting a session, the agent outputs usage statistics for input tokens, cached tokens, and output tokens. What exactly is the difference between Input Tokens and Cache Reads here? !28145375-6db3-447c-a161-6ec65851c5d6.png What If There Were No Cached Tokens? To simplify the problem first: without considering any caching mechanisms, communication between an agent and a large language model is always an absolute stateless functional interaction. The LLM has no memory whatsoever, and every interaction and call is a completely brand-new process to it. During a conversational interaction between an agent (or a web page chatting with an LLM) and the LLM, their respective roles and divisions of labor are as follows: - Agent (Intelligent Agent): A piece of code running on a local machine or a cloud server, serving as the business control center for the entire application session. It is responsible for maintaining the dialogue history between both parties, managing tools (API interfaces), and controlling the execution flow. - LLM (Large Language Model): The logical reasoning engine in the cloud. It is responsible for only one thing: receiving a piece of pure text input from the agent via the HTTP protocol, and predicting and outputting the next block of text based on probabilities. Precisely because the LLM lacks any memory throughout the interaction process, no matter how many turns the conversation lasts, the agent must accumulate and send all historical records (both the agent's questions and the LLM's responses) to the LLM at the start of every new conversation turn. Only then can the LLM understand the historical dialogue from the beginning and return a corresponding response to the new question. Consequently, as the number of conversation turns increases, the historical session records sent by the agent to the LLM naturally snowball and grow larger and larger: !%E5%AE%8C%E7%BE%8E%E5%AF%B9%E9%BD%90%E7%89%88_%E4%B8%8A%E4%B8%8B%E6%96%87%E6%BB%9A%E9%9B%AA%E7%90%83%E6%95%88%E5%BA%94.png This introduces two major problems: Problem 1: Bandwidth and Payload Size Explosion: In every interaction, the agent must resend tens of kilobytes or even megabytes of historical conversation records over HTTP. If the session records contain a codebase or PDF files, the required code and memory space after many rounds of accumulation becomes catastrophic. Problem 2: Extreme Waste of Computing Power (The Snowball Effect Causing Input Tokens to Skyrocket): The LLM itself is stateless. By the -th turn of the dialogue, the LLM has already computed the text of the first turns times. However, without a cache, it must mechanically recalculate the matrix multiplication for all this text all over again, wasting computational resources in vain. To address these two issues, it is essential to cache the KV Cache on the LLM server side. How Does KV Cache Solve the Problem? Currently, mainstream large language models adopt the Transformer architecture. For details on the internal calculation process within the Transformer architecture, refer to the two articles: Comprehensive Guide to the Encoder Structure and Calculation Process of the Transformer Model and Comprehensive Guide to the Decoder Structure and Calculation Process of the Transformer Model. To summarize simply: when the vector token corresponding to each word enters the Transformer architecture for attention calculation, its word vector is multiplied by three different weight matrices respectively, yielding the precise semantic vectors of the word in the current context. When the model generates the next word during its operation, it generates the current , and uses this current vector to perform dot-product calculations (calculating correlation scores) against the vectors of all preceding historical tokens. After calculating the correlation scores, it computes a weighted sum of the vectors of all historical tokens based on these scores. In the attention calculation process described above, the of historical tokens is not used, but their and vectors will be reused repeatedly in the generation of subsequent tokens. What the KV Cache stores is: _all_ _and_ _vectors calculated via matrix multiplication when the input text passes through each layer of attention computation._ Thus, when the next new conversation turn arrives, the KV values for all tokens in the historical conversation do not need to be recalculated; they can be read directly from memory. We only need to compute the KV values for the new conversation content, and then merge/concatenate them with the historical KV cache. This eliminates the step of recalculating all tokens of the historical conversation text during LLM processing. The following flowchart explains the workflow of the KV Cache caching mechanism: - Phase 1 performs a full-scale attention calculation (totaling 1,000 tokens) on the entire session context, and saves the resulting KV Cache upon completion. - Phase 2 no longer requires full-scale attention calculation on the contextual text content when generating each word. Instead, it uses the new vector to compute with the cached contents of the KV Cache to obtain the output, and then appends the and vectors corresponding to the newly generated word to the KV Cache for use in subsequent steps. !LLM%E5%8F%8C%E9%98%B6%E6%AE%B5%E6%8E%A8%E7%90%86_Prefill%E4%B8%8EDecode.png This is how caching the KV Cache on the LLM server side can drastically reduce the demand for and waste of GPU computing power. Implicit Caching and Explicit Caching Issues As mentioned above, caching the KV Cache on the server side can effectively solve the problem of wasted computing power (i.e., Problem 2 above). However, one important point to note is that server memory is a limited and extremely expensive resource. Massive requests from massive users require maintaining a large amount of cache on the server side, which means these caches cannot be retained indefinitely and must have a time-to-live (TTL) expiration policy. If the storage duration of the server-side KV Cache exceeds its time limit (for example, if the agent or user has no new requests in the current session for over an hour), the KV Cache will be deleted. Under such circumstances (when the KV Cache is deleted due to expiration), if the user sends a new request to the LLM server, how will the LLM handle it? Mainstream large language models currently have two solutions to this problem: implicit caching and explicit caching. 1. Implicit Caching In this architecture, during every round of interaction between the agent and the LLM, the agent dutifully packages all historical dialogue records, the system prompt, and the latest dialogue request into a giant JSON object, sending it to the LLM via HTTP. Once this giant JSON arrives at the cloud LLM server, its API gateway quickly calculates the prefix hash value of this text. - If it discovers via this prefix that its memory already contains the KV Cache of the historical session (i.e., a cache hit), the LLM will directly skip the Transformer matrix operations for this portion of the text in subsequent calculations, computing only the newly added request content at the tail. In this scenario, only the newly added request portion is counted toward Input Tokens. - If it fails to find the KV Cache for this historical session record in memory via the prefix (i.e., a cache miss), the LLM is forced to perform a full Transformer calculation on the complete historical dialogue records, system prompt, and latest dialogue request contained in the JSON. Therefore, under this mode with a cache hit, the cost of network I/O remains very high (because the complete session history still needs to be sent over every time), but the computing power (GPU calculation time) is saved. Consequently, LLM vendors offer discounted pricing for cache tokens, and the response speed of LLM computations becomes faster. - This mode solves Problem 2 above (the problem of wasted computing power) but fails to solve Problem 1 above (the network bandwidth I/O issue). 2. Explicit Caching The other mode is explicit caching. In this mode, for heavy static context that does not change frequently (such as the software and hardware architecture documentation of an entire project, long API specification documents, technical manuals, or complex system prompts), the agent only needs to upload it once at the very beginning. Once the server receives and finishes computing the KV Cache, it returns a resource identifier (Cache ID). During subsequent recurring dialogues between the agent and the LLM, the agent's JSON payload no longer needs to contain that heavy static context. Instead, it only needs to include a short string: "cached_content": "caches/12345-abcde", along with the session records between both parties. After receiving the request sent by the agent, the LLM server first checks whether the KV Cache corresponding to the Cache ID in the request packet still exists in memory. If there is a cache hit, the KV Cache can be used directly, performing Transformer calculations solely on the session records uploaded by the agent, and then concatenating the KV values to generate the new context. However, if the KV Cache corresponding to the Cache ID has expired and been deleted, the LLM server will immediately return a 404 error, with an attached message stating Cached content not found. At this point, the agent must resend all locally cached static context materials to the LLM, execute a dense Transformer computation across all contexts, and return a new Cache ID to the agent. Therefore, this explicit caching workflow can solve both Problem 1 and Problem 2 simultaneously. Only when the cache expires and misses is it necessary to send the bulky static context to the LLM to execute a full Transformer computation. !%E9%9A%90%E5%BC%8F%E7%BC%93%E5%AD%98vs%E6%98%BE%E5%BC%8F%E7%BC%93%E5%AD%98%E5%AF%B9%E6%AF%94.png
Summary of Learning: Amazon AWS Lambda Service
AWS Lambda is a fully managed serverless compute service. Compared to traditional Java/Python/Node.js services based on EC2 and resident processes, when handling identical application requirements with Lambda, developers only need to upload their code to the server, and AWS automatically provisions the required compute resources, releasing them immediately after execution. This article summarizes the various concepts of the AWS Lambda Service. 1. Introduction to AWS Lambda Service In traditional backend architectures built on AWS EC2 and resident Java/Python/Node.js processes, our development and operations (Ops) model is server-oriented. This means operations teams must constantly monitor operating system security patches, JVM runtime tuning, Application Load Balancer (ALB) configurations, and provision massive amounts of redundant computing power to guard against sudden traffic spikes. These computing resources are often severely underutilized during nighttime hours (which automated load balancing can mitigate but not completely solve), resulting in a massive waste of cloud spend. AWS Lambda is a fully managed serverless compute service introduced by Amazon Web Services. Its emergence has fundamentally transformed the backend computing paradigm: Operations teams simply upload code (supporting Python, Node.js, Java, Go, or even custom runtimes based on container images), and AWS automatically prepares the execution environment with millisecond-level latency, releasing resources immediately after the code finishes executing. This model eliminates tedious operational tasks while automatically handling the spin-up and teardown of computing resources, truly achieving pay-as-you-go pricing. !image.png Core Lambda Concept 1: Event-Driven In the world of Lambda, there are no longer resident daemon processes like Tomcat or Netty that continuously block and listen on HTTP ports. The lifecycle of a Lambda function is entirely driven by events, which are known as triggers in the Lambda context. For example, here are some typical events/triggers in the camera and IoT domains: - An edge security camera sends a boundary-crossing intrusion alert message to AWS IoT Core via the MQTT protocol. - An app client initiates a REST API request to retrieve a list of historical device recordings via Amazon API Gateway. - An edge device successfully uploads a short video clip of anomalous motion trajectories to an Amazon S3 bucket. Lambda is invoked to execute its corresponding code only when these exact events occur, and billing is calculated down to the millisecond. This means that when all devices are in a silent period, the cloud computing node bill drops instantly to zero. Core Lambda Concept 2: Stateless This is the aspect that differs most significantly from traditional backend development models like EC2-based resident Java/Python processes. Since Lambda instances are created on-demand and can be destroyed at any time, you must never store any session or business state data that needs to be reused across requests in the local memory or disk directory (such as /tmp) of a Lambda execution function. All persistent business states must be externalized, for example, by writing them to Amazon DynamoDB (NoSQL key-value store), Amazon RDS (relational databases), or Amazon S3. When designing a serverless network architecture based on AWS Lambda, you must first keep in mind and build your services around certain hard limits established by AWS: - Maximum Execution Time (Timeout): Currently 15 minutes (900 seconds). Therefore, Lambda is entirely unsuited for running long-running backend batch processing tasks; long-running tasks should instead use AWS Fargate or AWS Batch. - Memory and Compute Ratio: The memory configuration for Lambda function execution ranges from 128 MB to 10,240 MB. You cannot directly select the number of vCPU cores; AWS scales vCPU cores and network throughput in direct linear proportion to the allocated memory size, meaning the memory configuration directly dictates storage and computing resources. - Payload: The upper limit for the payload contained within a synchronous invocation request body is 6 MB, and 256 KB for asynchronous invocations. 2. Detailed Workflow of AWS Lambda The implementation of Lambda within the AWS system uses a mechanism similar to Docker containers to create isolated execution spaces for different applications belonging to different users. However, to balance extreme security isolation in a public cloud multi-tenant environment with the ultra-fast startup required by serverless computing, AWS developed and open-sourced a technology called Firecracker written from scratch in Rust. This is a KVM-based microVM (micro-virtual machine). Compared to traditional containers that share the operating system kernel, Firecracker can spin up a micro-sandbox with complete hardware isolation in a fraction of a second. Simply put, it is an advanced, high-performance kind of container. When we deploy code (such as a Python 3.12 device authentication script) to the cloud Lambda environment, its underlying actual lifecycle is illustrated in the diagram below: !be207b75-f9a8-464d-ab69-00110c159d1d.png As shown, the execution process of a Lambda container can be divided into three main phases: - Init (Initialization Phase - Cold Start): When the first request arrives, or if all existing instances are busy, AWS allocates a new Firecracker instance. Upon startup, this instance downloads the deployment package from S3 via the internal network, initializes the Python interpreter, and executes all global code in the Lambda function file outside of def handler(event, context): (such as import boto3 or establishing database connections). This phase is very time-consuming and is known as a Cold Start. For heavy Java JVMs, this phase can even take several seconds. - Invoke (Invocation Phase): AWS passes the JSON payload of the HTTP or MQTT message to the Lambda handler function and executes its business logic. - Freeze & Thaw (Freeze and Reuse): After the handler function returns (return), AWS does not destroy the Firecracker instance. Instead, it suspends the CPU of this micro-virtual machine (while it remains alive in memory). At this point, the instance incurs no charges, but the state in memory (including database connection objects) is fully preserved. If new requests flood in shortly after (typically within 5 to 15 minutes), AWS instantly thaws the instance, skipping the Init phase and jumping straight to Invoke (i.e., directly executing the handler function). This is a responsive Warm Start. Optimizing Cold Starts via the Provisioned Concurrency Mechanism Note that during the warm start process mentioned above, Lambda responds extremely quickly. However, during a cold start, the entire process of container startup, loading, and execution takes longer, which can result in a poor user experience in certain application scenarios. To solve this problem, AWS introduced the Provisioned Concurrency mechanism. Its core logic is to trade space for time. Before traffic actually arrives, you can use API commands to instruct AWS to silently pre-allocate a specified number of micro-virtual machines in the background and pre-run the time-consuming Init phase. This way, when real device concurrent requests arrive, all requests are routed directly to these pre-warmed instances, achieving ultra-fast response times within tens of milliseconds. However, it is worth noting that once provisioned concurrency is enabled, you will be continuously billed over time for these pre-warmed instances, even if there are no requests at all. A typical use case for this mode is identifying the peak traffic windows of your application every day, and then proactively enabling the Provisioned Concurrency mechanism right before those windows arrive to improve user response speeds. 3. Lambda Trigger Message Queue Patterns As noted above, Lambda is merely a computing unit. In real business scenarios, how are request data sent from devices routed smoothly, securely, and reliably to Lambda? Depending on the business scenario, AWS provides three distinct invocation models: !7d870224-20f9-4a02-babe-610fcd072937.png Synchronous Invocation Mode: - Working Mechanism: The client initiates a request and blocks the connection until Lambda finishes execution and returns an HTTP response. - Queue Characteristics: There is no internal queue buffering in this mode. Therefore, if 5,000 requests flood in simultaneously while the current Lambda concurrency quota is only 1,000, the excess 4,000 requests are immediately rejected by the system, returning a 429 Too Many Requests (ThrottlingException) to the frontend device. - Mitigation Strategy: Frontend device requests must implement rigorous exponential backoff and jitter retry algorithms. Asynchronous Invocation Mode: - Working Mechanism: Follows a fire-and-forget principle. After the AWS IoT Rules Engine pushes an event to Lambda's internal gateway, it receives immediate confirmation and places the event into a managed queue inside AWS, from which Lambda asynchronously pulls and executes it. - Queue Retries and DLQ: If an exception is thrown during Lambda code execution or if concurrency throttling is encountered, the internal Lambda queue will retry up to 2 times by default (usually spaced 1 minute and 2 minutes apart). Note: If both retries fail, the message is permanently dropped; thus, this queue mechanism is inherently unreliable. Therefore, to ensure that alerts or critical signaling messages are not lost, you must bind a Dead-Letter Queue (DLQ) to the Lambda, typically designating an Amazon S3/SQS queue to capture ultimately failed messages for subsequent alert analysis or manual intervention and compensation. Stream-Based Queue Triggers: If edge devices emit data at a very high frequency, handling such high-frequency messages directly via Lambda is not an optimal design. - A sound architectural design routes high-frequency messages sent from IoT Core into Amazon Kinesis Data Streams or Amazon SQS first. Lambda does not act as an independent HTTP endpoint; instead, it runs a background poller. Based on configured parameters like BatchSize and MaximumBatchingWindowInSeconds (e.g., wait up to 1 second or gather up to 500 records), this poller bundles scattered stream data into a JSON array and passes it to a Lambda instance for processing. 4. Best Practices for Lambda Development and Deployment In real IoT cloud-native backend development, modifying code manually via the AWS web console or allowing developers to have direct production deployment permissions on their local machines is extremely dangerous. A more rational approach is to promote Infrastructure as Code (IaC) within the team and establish a strict CI/CD automated deployment pipeline. In fact, AWS specifically developed the SAM (Serverless Application Model) tool to handle the development and standardized deployment workflows of serverless cloud services. For Lambda services, the best practice workflow for development and deployment can generally be divided into 5 stages: - Local IDE and Code Definition: Developers write Python business logic (app.py) in their local IDE and maintain a template.yaml configuration file. In this YAML file, all cloud resources must be defined declaratively. This configuration file serves as the system blueprint for cloud deployment, ensuring absolute traceability and immutability of the cloud environment. - Packaging and Building (sam build): Used to generate delivery artifacts. Ultimately, all code and dependencies are archived into a local hidden folder named .aws-sam, ready for the cloud. - Cloud Synchronization (sam deploy): In this step, SAM automatically compresses the packaged code and dependencies into a .zip file and uploads this ZIP package through a secure channel to a hidden bucket in Amazon S3. - Orchestration and Deployment: What gets uploaded in the previous step includes not only the code ZIP package but also the translated template.yaml file. AWS hands this configuration file over to AWS CloudFormation (AWS's infrastructure automation orchestration engine), which performs the actual cloud environment deployment based on template.yaml. - Trigger and Execution: Completing the previous step effectively finalizes the deployment. Next, you can send messages from the client to the server to test and verify functionality. In practical engineering applications, the above workflow should be integrated with CI/CD automated testing and deployment pipelines to make the overall software release process more scientific and efficient: engineers write Python code and template.yaml, push to a code repository (such as GitLab, GitHub, or AWS CodeCommit), triggering an automated CI/CD pipeline where cloud servers automatically execute sam build and sam deploy, followed by rigorous automated testing before production release. !AWS-SAM-CICD-Pipeline.png References - The Ultimate Guide to AWS Lambda | Serverless Framework | Serverless Framework
A Detailed Explanation of the Concept and Workflow of the BN Layer in Convolutional Networks
This article provides a detailed summary of the theoretical significance of Batch Normalization (BN), which is ubiquitous in various convolutional neural network architectures, and the workflow of BN layer calculations in various computer vision network models. The Concept and Significance of the BN Layer Ever since Google proposed Batch Normalization (BN) in 2015, it has become a standard layer in almost all modern CNN architectures (such as ResNet, YOLO, EfficientNet, etc.). !image.png Before the advent of the BN layer, what were the problems associated with training deep neural networks with dozens of layers? - Extreme sensitivity to weight initialization: If the initial weights were set slightly too large or too small, the output values would either explode or vanish after propagating through multiple layers. Consequently, researchers back then had to spend immense effort studying complex parameter initialization methods. - Unusably small learning rates: To prevent network parameters from collapsing during training, learning rates had to be set conservatively (e.g., 0.0001), resulting in extremely long training times and low efficiency. - The vanishing gradient problem: Network designs around 2015 commonly used Sigmoid or Tanh as activation functions. If the input data was too large or too small, it easily fell into the saturation zones of these activation functions (where the curve becomes extremely flat), causing gradients to drop to nearly zero (vanishing gradients), which completely halted network learning. The root cause of these problems is a phenomenon known as Internal Covariate Shift (ICS). While ICS sounds complicated, it describes a simple issue: during training, when parameters in the first layer are updated slightly, this minor change is exponentially amplified as it propagates through the second, third, and all the way to the one-hundredth layer. As a result, for deep networks, the statistical distribution of input data received at each layer varies drastically and irregularly, making it impossible to learn any consistent patterns from these wildly fluctuating values. To address these issues, the BN layer introduced a direct solution: Since the data distribution coming from upstream is always unstable, we insert an extra processing step before each layer to forcefully shuffle and normalize the upstream data into a standard format. Specifically, regardless of the statistical shape of the data passed from the previous layer, the BN layer computes the mean and variance of the current complete training batch (the so-called Mini-batch) and forcibly pulls the input data into a standard normal distribution with a mean of 0 and a variance of 1. Under this mechanism, no matter how deep the network is, every layer can be certain that the data received from the preceding layer conforms to a standard normal distribution. The question is: If we forcefully transform the input data from the previous layer into a normal distribution with a mean of 0 and a variance of 1, wouldn't that destroy the feature discriminability that the preceding network layers worked so hard to learn? To solve this problem, after the normalization calculation, an additional linear transformation step and two learnable parameters, (Scale) and (Shift), are introduced to restore the network's expressive power. This hands the choice of "whether input data needs to be normalized" and "the degree of normalization" back to the network for it to learn and decide on its own. The specific calculation workflow is described below. Another question is: Where is the BN layer typically placed within a CNN network? The answer is that the BN layer is generally sandwiched between linear operations (such as convolutional layers) and non-linear activation functions (such as ReLU), as shown in the diagram below: !image.png This is because the data produced after a convolution operation is often very scattered in its statistical distribution. By using a BN layer to cluster the data around 0 before passing it into the activation function, the data is perfectly positioned in the most sensitive and effective region of the activation function, thereby obtaining maximized and healthy gradient feedback. Calculation Workflow of the BN Layer Generally speaking, the core operations of a BN layer comprise two phases: the Normalization calculation phase and the Affine Transformation phase. In terms of the workflow, for a given feature within a Mini-batch, the BN layer first computes its mean and variance , and then performs normalization. The normalization maps the input data to a zero-centered, unit-variance linear space. Finally, two learnable parameters, (Scale) and (Shift), define a new linear mapping to restore the network's expressive power. > 💡 Therefore, no matter how fancy the BN layer might look during training, both phases of its calculation are ultimately linear transformations. Mathematically, the entire processing of input data by the BN layer remains a simple linear transformation: . The complete formulas and calculation workflow of the BN layer are as follows: the first three formulas represent the normalization process, and the final formula represents the affine mapping. From a data dimensionality perspective, the dimensions of the input and output data remain completely unchanged, and the structural semantics are fully preserved. During normalization, the computation is always performed per channel of the input feature map. In other words, when calculating the mean and standard deviation for a single channel, it aggregates all pixel points across all images in that specific channel. > 💡 Why normalize per channel? Because in a CNN, each channel represents an independent and specific semantic feature. Normalizing each channel independently preserves the physical independence between different features, preventing two distinct semantic features from being forcefully averaged together. The diagram below illustrates the data dimensions at each stage of the BN layer calculation in a standard CNN network: - Stage 1: The input data is the feature map output by the previous convolutional layer, with dimensions . - Stage 2: Taking the channel as the unit, extract the feature maps of a specific channel across images, and compute the mean and variance for all data in these feature maps, resulting in means and variances. - Stage 3: Still operating per channel, normalize the pixels of each feature map in that channel based on its corresponding mean and variance, converting the feature map data of each channel into a standard distribution with a mean of 0 and a variance of 1. - Stage 4: In the affine mapping stage, each channel has a set of learnable parameters and learned during training, which are used to linearly map the feature map of that channel. - Stage 5: The final output data shape remains consistent with the input. !bd0b18312f67600e745c52997774bc43.png To further clarify the calculation workflow, the diagram below illustrates a complete BN layer operation taking a batch of images with 3 channels and a resolution as input: - Throughout the computation, because the input data has 3 channels, the input batch has 3 sets of means and variances, as well as 3 sets of learnable parameters and corresponding to the three channels. - Note that this diagram is for illustrative purposes only. The BN layer in CNNs rarely acts directly on the raw input images. For raw images (), constant values (such as ImageNet's global mean and standard deviation) are typically used directly during the preprocessing stage for mean subtraction and division by standard deviation. The true domain of the BN layer is on deep feature maps within the network, processing the outputs of convolutional layers. !4f65a87e3daa101a9c82653891b7ee26.png Behavior of the BN Layer During Inference It is important to note that the behavior of the BN layer differs completely between training and inference. During the forward pass of training, the code calculates the true statistics and based on the current Mini-batch. Meanwhile, to prepare for inference, the BN layer quietly maintains running averages of the mean (running_mean) and variance (running_var) in the background. During inference (i.e., model.eval() mode), the BN layer no longer computes batch-specific statistics (since inference inputs might consist of just a single image, causing unstable results if batch size varied). Instead, it directly uses the accumulated running_mean and running_var from training for normalization. Similarly, the learnable parameters (Scale) and (Shift) in the affine transformation phase are fixed during inference. Therefore, the linear operations of the BN layer can be mathematically and equivalently fused into the weights () and bias () of the preceding convolutional layer. Consequently, when exporting to ONNX, tools like ONNX-Simplifier are typically used in combination. This process automatically fuses Conv and BN layers, meaning independent BatchNorm nodes will generally no longer appear in the exported *.onnx model graph.
How to Convert PT Models into TFLite/LiteRT Formats?
Previously, when attempting to run Ultralytics' YOLOv8 model on Android phones (refer to Summary of YOLO Model Execution Workflow in Android Flutter Projects), the model could be directly exported into ONNX, TFLite, and other formats using the YOLO model's built-in export interface. However, what if the model we encounter during development lacks an official export interface? How can we convert a model file in .pt format into TFLite for easy deployment on Android devices? Therefore, this article aims to address this question: How can a generic PyTorch model weight file (.pt) be converted into a TFLite model file (.tflite) that can run on Android edge devices? Google's AI Edge Torch / LiteRT To solve the problem stated above, a popular solution involves the following pipeline: - PyTorch (.pt) → ONNX: ONNX is a universal format for model exchange, so this conversion step frees the model file from PyTorch's framework restrictions. - ONNX → TensorFlow (SavedModel): This is a translation process from the ONNX ecosystem language to the TensorFlow ecosystem, ensuring that the target platform (i.e., Google mobile app) correctly recognizes the operators in the model file. - TensorFlow → TFLite: This is somewhat similar to strip and objcopy under Linux. This step performs constant folding, operator streamlining, and most importantly, quantization (converting default FP32 weight parameters into INT8) to accommodate the compute and storage limitations of mobile CPUs/NPUs. However, the traditional PT -> ONNX -> TF -> TFLite path is not a flawless solution. Because PyTorch adheres to NCHW (Channel First) while TensorFlow defaults to NHWC (Channel Last), conversion tools in this traditional pipeline aggressively insert Transpose operators to preserve mathematical equivalence. During execution on mobile CPUs/GPUs, this causes frequent memory rearrangements, severely slowing down inference speed. Consequently, Google introduced the AI Edge Torch tool in April 2024 to enable direct conversion from PT model files to TFLite format models. This tool utilizes torch.export (a core feature of PyTorch 2.x) to directly capture the computational graph and map it to TFLite operators. More importantly, it directly aligns data formats during model conversion, eliminating a massive amount of unnecessary Transpose operators and avoiding the heavy memory rearrangements required during subsequent inference. By 2025, Google rebranded its mobile AI ecosystem: TensorFlow Lite was renamed to LiteRT. This change was primarily driven by the perception that the name "TensorFlow Lite" was too tightly coupled to the TensorFlow framework. The current trend in the AI model ecosystem is that PyTorch dominates research and training, while TFLite dominates mobile deployment. Therefore, to attract PyTorch users, Google emphasizes that the TFLite runtime is a universal edge AI deployment tool, not exclusively for TensorFlow. Thus, the new brand name is LiteRT (Lite Runtime). !image.png Environment Setup To convert a PyTorch model into TFLite/LiteRT format on a PC, you primarily depend on two packages: litert-torch and ai-edge-litert. - litert-torch: Corresponds to the former ai-edge-torch, responsible for mapping PyTorch operators to LiteRT. - ai-edge-litert: Corresponds to the former tflite-runtime, which is the inference engine responsible for executing generated .tflite (LiteRT) files on Android or PC. As of now (March 2026), the APIs for litert-torch and ai-edge-litert are still changing rapidly, making their environmental requirements very strict. Therefore, you must create an independent Python environment and install the specified version of PyTorch in advance; otherwise, package conflicts can easily occur during the installation of these two packages. After careful verification, the currently stable environment setup steps are as follows (installed within a Windows WSL2 environment): Note: The environment for model conversion and PC-side inference must be installed in a Linux environment. When I tried setting it up directly in Windows, I consistently encountered the error below. Judging from the error message,litert-torchdepends onai-edge-tensorflow, andai-edge-tensorflow is currently primarily released in Google's Nightly/Experimental repositories, with native support for the Windows platform lagging far behind. Therefore, it is best to install this conversion tool in Linux or a Windows WSL2 environment. Model Conversion Once the environment is successfully set up, you can begin converting the pre-trained .pt model file under PyTorch into the TFLite format. Here, I directly use the yolov8s.pt file from the Ultralytics/YOLO project for conversion testing. The model export script above successfully exports the FP32 model file. While FP32 offers the highest precision, due to its heavy consumption of memory bandwidth and storage space (for the YOLOv8s model, the converted file size is 44MB), it is certainly not the optimal performance choice for edge CPUs/NPUs. Therefore, quantization processing is required next. Quantization For YOLO models with a built-in export interface, quantization is a straightforward task—you simply configure the export format to fp16, int8, etc. However, after using litert_torch to complete model exportation, the APIs for quantizing TensorFlow Lite model files changed drastically. As a result, almost all online reference code for quantization fails to correctly or successfully complete quantization in the latest litert-torch. Reviewing the README file from https://github.com/google-ai-edge/litert: > > > I have a PyTorch model... > > - Goal: Convert a model from PyTorch to run on LiteRT. > - Path1 (classic models): Use the LiteRT Torch Converter to transform your PyTorch model into the .tflite format, and use AI Edge Quantizer to optimize the model for optimal performance under resource constraints. From there, you can deploy it using the standard LiteRT runtime. > - Path2 (LLMs): Use LiteRT Generative Torch API to reauthor and convert your PyTorch LLMs into Apache format, and deploy it using LiteRT LM. > As seen here, for newer LiteRT implementations, quantization requires a new tool called ai-edge-quantizer (https://github.com/google-ai-edge/ai-edge-quantizer). This tool is a next-generation utility built by Google specifically for INT8/INT4 weight compression on LiteRT models. In modern Edge AI concepts, simple FP16 file size truncation is no longer prevalent; Weight INT8 + Activation FP32 (known as Dynamic Quantization) is the ultimate champion for balancing speed, size, and accuracy on edge devices like mobile phones. The quantization workflow above selects the wi8-afp32 mode (where weight parameters are quantized to INT8 while activation values remain FP32). The resulting quantized model file is only about 11MB, which is 1/4 of the size of the pre-quantization FP32 model. Aside from the wi8-afp32 quantization mode mentioned above, ai-edge-quantizer also supports other quantization modes. As shown in the figure below, Google engineers adopted a very straightforward naming convention to define these quantization modes: - w = Weight (weight parameters, i.e., fixed parameters learned during model training) - a = Activation (activation values, i.e., intermediate feature maps flowing through the model during inference) - i8 / i4 = 8-bit / 4-bit Integer - fp32 = 32-bit Floating-point !8a772873-2af8-4892-8f32-b1998f10d39e.png Inference Testing Now that the LiteRT model file has been successfully converted and quantized, the next step is to perform inference testing on the PC using the LiteRT runtime. As mentioned above, within the LiteRT ecosystem, the ai-edge-litert package corresponds to the former tflite-runtime, acting as the runtime framework responsible for executing forward inference based on LiteRT model files on Android or PC. Therefore, the workflow for forward inference on a PC generally consists of: loading the model and input image file → data preprocessing → executing inference → running NMS post-processing on inference results → overlaying the NMS-filtered results onto the input image. References - AI Edge Torch: High Performance Inference of PyTorch Models on Mobile Devices - Google Developers Blog - Convert Models From Pytorch to TFLite With AI Edge Torch | by David Cochard | ailia Tech BLOG (EN) | Medium - https://github.com/google-ai-edge/litert-torch/blob/main/docs/pytorch_converter/README.md - https://github.com/google-ai-edge/ai-edge-quantizer
How Does TFLite Handle Operator Versions and Compatibility Issues?
When performing model deployment and on-device adaptation on Rockchip NPU platforms, consistency between the Opset version specified during ONNX export and the Opset version supported by the on-device model conversion tool (RKNN-Toolkit2) is crucial. If the conversion tool does not support certain operator types in the ONNX file, the conversion process will fail. So, does TFLite also suffer from Opset version issues—meaning that when exporting a model to the TFLite format, is there an Opset version that must match the version supported by the interpreter, otherwise leading to similar conversion failures as seen on the Rockchip platform? And how does the TFLite workflow handle operator version consistency? This article attempts to summarize the answers to these questions. For related background information, please refer to: - ONNX Opset Version: How to Understand the Opset Version in the ONNX Export Process? - TFLite Framework Core Concepts and Workflow: How to Understand the Core Concepts and Framework Workflow of TFLite/LiteRT? TFLite Operator Version Information Regarding whether the TFLite framework has the same concept of Opset versions and handles this problem similarly to ONNX, the short answer is: TFLite does have the concept of operator versions, but its processing workflow differs significantly from ONNX. A .tflite file is essentially a serialized FlatBuffer binary file. At the header of this file, there is a dedicated OperatorCode array (known as the operator registry). When a model is exported, the TFLite converter registers all operators used in the model into this table, with each operator independently recording its own version number, rather than having a unified operator set version like ONNX. As shown in the striking difference between the two in the figure below: TFLite adopts fine-grained, independent version numbers for individual operators, and the model file itself has no concept of an operator set version; whereas ONNX uses a global operator set version, regulating and managing the entire operator set through the model file. !%E6%97%A0%E6%A0%87%E9%A2%98-2026-03-26-1350.png Why does TFLite adopt this per-operator version management model instead of the global Opset version approach used by ONNX? The primary reason is that operator-level version management provides better forward compatibility, which is crucial for the heavily fragmented software and hardware Android ecosystem. Suppose our model contains only the most basic Conv2D and Relu operators, and we export it using the latest version of the model converter: - In ONNX's Global Opset Mode: The model converter flags this ONNX file as a relatively newer Opset 18. Consequently, even if the Conv2D operator has not changed at all from Opset 1 to 18, an older version of ONNX Runtime encountering Opset 18 might directly throw an error and refuse to load it. - In TFLite's Per-Operator Version Management Mode: During conversion, the converter detects that the model does not use any new features of Conv2D, and thus tags the CONV_2D operator as Version 1 in the TFLite model file. In this case, even if the user's Android phone contains a much older version of the TFLite Interpreter, it can still seamlessly load and run this newly exported model. Operator Version Matching Similar to model exporting, adaptation, and on-device loading and execution on other embedded platforms, running a TFLite model on Android requires attention to three aspects of operator version matching: the operator versions of the model conversion tool (Converter), the operator versions supported by the Interpreter, and the operator versions supported by the Delegate. Only when the operator version information of these three components matches can we ensure that the TFLite model runs successfully on Android. - For concepts related to the TFLite model converter, Interpreter, and Delegate, and their structures within the Android system, please refer to the note How to Understand the Core Concepts and Framework Workflow of TFLite/LiteRT?. Converter The TFLite model Converter tool has an internal operator version probing algorithm to automatically detect the version information of each operator in the model file. For example, for the FullyConnected operator, if the Keep_num_dims feature is not enabled, the Converter will tag this operator as Version 1 during model conversion; if the model's inference calculation requires this new feature, the Converter will tag it as Version 2. This acts as an operator version determination strategy of "use a lower version whenever possible," which is core to ensuring that models and apps achieve the widest possible device compatibility. In summary, for the Converter, no matter how new the tool is, its working logic is: during the TFLite format conversion process, it scans and probes the minimum required version for each operator in the computational graph, and then sets this minimum version as the operator's version. This maximizes the device compatibility range for the operator and the model moving forward. Interpreter The Interpreter also has its own version support matrix. During the loading of the TFLite model file and the construction of the model's computational graph, it similarly scans the version information of each operator and checks its compatibility with its own supported versions. During the process of parsing the FlatBuffer structure of the TFLite model file, the Interpreter extracts the version information of each operator and compares it against its internal OpResolver (operator registry). If an operator requires Version 5 according to the TFLite file, but the Interpreter's codebase only implements Version 4 for that operator, it will immediately throw an Unresolved builtin op exception and refuse to execute. Therefore, whether a model can successfully run on Android heavily depends on the Interpreter's support for operator versions. This is why when developing Android apps using the Flutter framework, you should use the latest version of the tflite_flutter plugin whenever possible. > 💡 The Interpreter itself is just an ordinary C++ dynamic link library (such as libtensorflowlite_c.so) that is hard-packed into the APK rather than provided by the underlying Android system. The advantage of this is that the Interpreter is tightly bound to the app's model. As long as the app passes comprehensive testing, it is guaranteed to run across different hardware platforms and Android system versions. Consequently, if a system contains multiple apps running different models, there will be multiple independent copies of the Interpreter. Of course, TFLite's core interpreter is designed to be extremely lightweight, stripping away all training-related code. A basic C++ Interpreter library is typically only 1MB to 2MB in size, having a negligible impact on modern app sizes. The Interpreter actually contains CPU implementations for all the operators it supports. Thus, if a Delegate does not support a specific operator, the Interpreter automatically falls back to its internally implemented CPU operator. Delegate As for the Delegate, it also maintains its own version information for the operators it supports. During the handshake between the Delegate and the Interpreter, the Interpreter acquires the operator version information supported by the Delegate. In this scenario, during operator scheduling in forward inference, the Interpreter clearly knows which operators are supported by the Delegate and which are not and must therefore be executed on the CPU. Of course, unlike the Interpreter—which is built into the Android app layer—the Delegate's support for operators is fixed within the Android system. This is easy to understand, as the NPU driver implementation beneath the Delegate relies on SoC vendor implementations and cannot be resolved at the app layer. Therefore, unless the entire system is upgraded, the Delegate layer's support for operator versions remains fixed. Operators that the Delegate cannot support must be scheduled by the Interpreter to run on the CPU. Complete Workflow of TFLite Forward Inference and Operator Versions At this point, the complete workflow—covering TFLite operator version management, operator version estimation during TFLite model export, and operator version matching and scheduling between the Interpreter and Delegate on Android—becomes crystal clear: !TFLite_Architecture_Flowchart.png
Summary of the YOLO Model Execution Workflow in Android Flutter Projects
Previously, I implemented image object detection in an Android app by calling a YOLO model using Flutter. This article summarizes the complete execution workflow, covering the TFLite export of the YOLO model and its invocation within an Android Flutter project. Running AI models on Android typically involves exporting the model file to the TFLite format and executing the AI model inference pipeline under the Android TFLite+Interpreter+NCAPI framework. For background knowledge related to this framework, you can refer to the article How to understand the core concept and workflow of TFLite?. TFLite Export of the YOLO Model Since the model export implementation in the Ultralytics project is already mature, you can directly export the tflite model file using its export interface within the ultralytics project. Here, we choose to export in half (float16) format: Additionally, to subsequently invoke the above model file for inference in the Flutter project, you also need to create a corresponding label.txt file containing the category name strings corresponding to the model's detected classes, one per line in sequential order. Since I directly used YOLO's pretrained pt file in my test—which corresponds to the COCO dataset—the category name strings contained in this label.txt consist roughly of the following 80 classes: Dependency Management To add the capability of running AI models to the Android Flutter app, you must first add the relevant dependencies to pubspec.yaml in the project root directory and build.gradle.kts in the android/app directory. The pubspec.yaml file in a Flutter project serves as its metadata center and dependency manager. It primarily contains: basic project configuration information, production environment dependency libraries, development/test stage dependency libraries, resources, and assets. For the dependencies mentioned above, every time the project is built, flutter pub get is automatically invoked to download the source code of the specified dependency libraries from pub.dev to the local machine. This project requires running the YOLO model to detect images from the phone's camera and local photos, so the relevant settings in pubspec.yaml include: Therefore, for running the YOLO model in an Android project, the most critical part of the above configuration file is the inclusion of the flutter_vision plugin. This plugin has a built-in TensorFlow Lite C SDK and includes related processing for specific models like YOLO, such as input/output ByteBuffer, tensor shape conversion, and NMS (Non-Maximum Suppression). For the application layer, it only needs to call the vision.loadYoloModel() interface, and it can create the TFLite Interpreter on the Android native layer and execute inference tasks. Complete Inference Workflow of the Model in Flutter Similar to the complete inference workflows of other edge models, invoking the YOLO model to execute inference tasks in an Android Flutter project also comprises three stages: model loading, forward inference, and post-processing. Model Loading The overall workflow for loading an AI model based on the flutter_vision plugin is as follows: The overall model loading process is straightforward: call the loadYoloModel interface of flutter_vision to create an asynchronous model loading task, pass parameters such as the model file and label file, and then wait for this asynchronous task to finish executing. It is important to note that the flutter_vision plugin only supports YOLOv5, YOLOv8, and YOLOv11 models exported directly in tflite format via the export interface of the ultralytics project, and the model file specified in the parameters along with the modelVersion parameter must be set correctly; otherwise, the subsequent post-processing workflow will fail: !074e2e0e-d2b9-48e6-b0b2-4a1f06769c23.png So, what if we want to run other models in an Android app? In that case, we can only use the tflite_v2 plugin, the flutter_tflite plugin, or the underlying TensorFlow Lite API. Forward Inference and Post-Processing The forward inference process of the flutter_vision plugin mainly involves calling its yoloOnFrame (for camera images) and yoloOnImage (for local images) interfaces. The execution of the above inference interfaces already incorporates the post-processing workflows for different YOLO versions, so their output is an array of . Each item in this array corresponds to the category, coordinates, and confidence of a final detection box, as shown below: After obtaining this array at the application layer, you can parse its contents and draw bounding boxes on the image for labeling. Because the yoloOnFrame and yoloOnImage interfaces already include post-processing steps such as NMS, there is no need to perform post-processing again at the application layer. References - flutter_vision | Flutter package
How to Understand the Core Concepts and Workflow of TFLite/LiteRT?
This article organizes the core concepts of TFLite, compares it with the ONNX framework, and examines the key workflows of its internal components, aiming to build a comprehensive understanding of the overall concept and workflow of this standard AI inference framework on Android. What is TFLite? And a Comparison with ONNX TensorFlow Lite, launched by Google, is an open-source deep learning framework designed specifically for mobile, embedded, and edge devices. It enables developers to run high-performance inference tasks with extremely low latency and a small binary footprint on mobile phones (Android/iOS), microcontrollers (MCUs), and various IoT devices. !image.png As is well known, running various AI models on edge board devices already has a very popular framework in ONNX. So why not use ONNX directly on Android? The reason mainly lies in the ecosystem competition between Facebook's PyTorch/ONNX and Google's TensorFlow. Although ONNX Runtime now supports running on Android, TFLite should still be the preferred choice for running AI models within Android apps for the following reasons: - Compatibility issues: In the implementation of the Android system, Google provides a layer of NNAPI (Neural Networks API) encapsulation. TFLite is Google's primary edge machine learning framework, and its support for NNAPI takes the highest priority. If ONNX is used, various driver incompatibilities or performance degradation may occur when calling NPUs from different SoC manufacturers, making compatibility management a major headache. - Memory and package size issues: To run models using ONNX on Linux or Android systems, ONNX Runtime must be bundled in, and the associated library files are usually large (tens of MBs), which directly increases the app's installation package size and memory footprint. TFLite does not have this problem. The core TFLite interpreter is extremely small (only a few hundred KBs), and the relevant runtime environment is already built into the Android system. Overall Workflow of TFLite Model Deployment Deploying a complex deep learning model to a mobile phone typically involves a complete process ranging from training, pruning/format conversion/quantization, to on-device execution. This workflow is largely identical across all edge AI frameworks. For instance, regarding model adaptation on the Rockchip platform, the overall adaptation process is as follows: - First, complete model training on a PC/AI server and export it as an ONNX model. - Next, convert the exported ONNX model into an RKNN-format on-device model using Rockchip's RKNN-Toolkit. - Within Rockchip's processor, the on-device model cooperates with its runtime library librknnrt.so and the application to complete the on-device inference task. The above workflow is entirely identical to running AI inference using the TFLite framework in Android systems on mobile phones, except that the process is slightly simplified: - Model Conversion Section: First, convert the raw trained PyTorch model (usually .pt or .pth) into a flat buffer format .tflite file using a tool such as TFLite Converter. Similar to the workflow of other on-device model conversion tools, this conversion process includes operations such as operator fusion (combining multiple sequential operations into one to reduce memory access times) and quantization (converting 32-bit floating-point weights in the original model file into 16-bit floating-point FP16 or 8-bit integer INT8, trading a marginal loss in accuracy for multiple times speedup while reducing model size and VRAM requirements). - On-Device Inference Execution Section: While the Android app is running, the TFLite Interpreter built into the Android system is responsible for loading the model, managing memory buffers, and scheduling computing operators. > 💡 Note: In the model conversion phase, models trained under the PyTorch framework generally need to be converted to ONNX models first, and then converted to the tflite format using tools like onnx2tensorflow. However, Google also released an AI Edge Torch tool in 2024 to enable direct conversion. !image.png How Does a TFLite Model Run Under the Android Framework? As seen in the framework diagram above, running a model converted into the TFLite format within an Android app relies on two key components: Interpreter and Android Neural Networks API (NNAPI). The Interpreter is the core control center for running TFLite models in the system. It is responsible for loading models, parsing computational graphs, and managing memory buffers. Throughout the model workflow, the Interpreter does not directly perform complex floating-point and operator calculations; instead, it is responsible for dispatching the computational tasks defined in the model. - In fact, the Interpreter is somewhat similar to Rockchip's librknnrt.so, which assists applications in loading models, parsing computational graphs, and managing memory. Upon receiving input data, it schedules and manages the computational tasks for each layer of the model, assigns them to appropriate processors (CPU/GPU/NPU), and manages memory copies between processors based on specific scheduling conditions. Therefore, its tasks are: load and parse models, dispatch computational tasks, and manage memory across different computational tasks and processors. NNAPI, on the other hand, is a C API provided by the Android system. Its existence is to abstract away the fragmentation of underlying NPU, GPU, and DSP hardware. The NNAPI provided by Google in the Android system is merely an API call interface framework that cooperates with the Interpreter. For SoC manufacturers such as Rockchip, Qualcomm, and MediaTek, their core work to enable Android phones to execute TFLite model inference smoothly lies in implementing the NNAPI interface specifically within their SoC designs. Specifically, these SoC manufacturers need to write corresponding NPU driver instructions based on the NeuralNetworks.h standard provided by the Android NNAPI module. The key tasks of these driver instructions are implementing model operator mapping (translating standard operators defined by Android, such as ANEURALNETWORKS_CONV_2D, into specific instructions that their own NPUs can understand and execute) and memory management (implementing shared control of memory across different types and regions to avoid redundant data copying between the CPU and NPU during inference). In summary, this forms the following calling relationship and workflow: !5a379fcc-72eb-471f-9001-bf892b8a6120.png - First, the application logic code of the app calls the Interpreter to load and parse the model file. - During the process where the Interpreter executes inference on input data, it invokes APIs defined within the Android system's NNAPI framework via the Delegate mechanism. NNAPI defines various standard operators, and during inference, the Interpreter calls the corresponding API interfaces in NNAPI for specific operators. - Beneath NNAPI are the specific operator implementations by various SoCs, which are implemented by individual SoC manufacturers based on the capabilities of the CPU/GPU/NPU within their SoCs. Effectively, NNAPI defines a standard layer of operator interfaces, shielding the differences in how various SoC manufacturers implement these operators on their hardware. Thus, from the perspective of the application and the Interpreter, they only see the standard interfaces defined by NNAPI. The Delegate Mechanism for the Interpreter Calling the NNAPI Framework The calling workflow between the Interpreter and the NNAPI framework described above is merely a simplified explanation; in reality, the Interpreter does not directly call the API interfaces defined by NNAPI. If the Interpreter were to call NNAPI directly using hardcoded logic, two problems would arise: - Some operators may not be supported (Fallback): NNAPI does not necessarily support all operators in a TFLite model file. Direct calls would cause the program to crash when encountering unsupported operators. - Cross-platform compatibility issues: TFLite has grand ambitions—it aims to run not only on Android but also on Windows, Linux, and iOS, none of which have NNAPI (for instance, the iOS app part in the architecture diagram above has no NNAPI at all). Therefore, the Interpreter employs a delegate mechanism when calling the NNAPI framework on Android. In essence, a Delegate is an abstract plugin interface designed by the Interpreter to achieve cross-platform compatibility. - Looking upward (towards the Interpreter), it defines a standard C API via TfLiteDelegate, shielding the differences among various hardware platforms, SoCs, and NPUs through this C API. The Interpreter always needs only to call this C API interface to invoke underlying computing power and complete the inference process. - Looking downward (towards various NPU implementations on different platforms), it acts as a client responsible for invoking deep learning operator acceleration libraries specific to different system and hardware platforms (such as Android's libneuralnetworks.so or iOS's Metal.framework). At this point, the complete structure and calling workflow of the TFLite framework become quite clear: !%E6%97%A0%E6%A0%87%E9%A2%98-2026-03-26-1350.png References - LiteRT in Google Play services C and C++ APIs | Google AI Edge | Google AI for Developers - TFLite Basics - VitoYeah - Blog
Hand Landmark Detection Model: MediaPipe Hand Landmarker
Introduction to MediaPipe Hand Landmarker Google's MediaPipe Hand Landmarker (a hand landmark detection model based on the MediaPipe inference framework) is currently one of the most popular and efficient solutions for real-time hand landmark detection and tracking in the field of computer vision. It can run on mobile devices (Android/iOS), web browsers, and even compute-constrained edge devices, outputting the 3D coordinate positions of 21 hand landmarks within an image at real-time frame rates. The image below shows the indexing of the 21 finger joint coordinates within the hand region that can be detected by this hand landmark model package. The model was trained on approximately 30,000 real-world images alongside multiple rendered, synthetic hand models superimposed onto various backgrounds. !image.png Google provides official implementations of the Hand Landmarker model for four platforms: iOS, Android, Web, and Python. Therefore, if you are using this model on any of these four platforms, you can directly refer to the official sample code to easily invoke the model and fulfill your application requirements. So, what if you want to run this model on other platforms and languages, such as using C/C++ on alternative embedded platforms? The answer is a resounding yes, though it is slightly more complex. Running the Hand Landmarker model depends on Google's open-source MediaPipe inference framework. At its core, the underlying architecture of this framework is a streaming ML framework built around C++, utilizing TensorFlow Lite as the inference backend and managing data streams via graph computation (Graph). MediaPipe itself is open-source and extremely friendly to commercial applications. Its code repository is hosted on GitHub: https://github.com/google-ai-edge/mediapipe. It is licensed under the Apache 2.0 License, allowing you to freely use, modify, distribute, and even integrate it into your own commercial software as closed-source without needing to open-source your business code. Therefore, to run this model using C/C++, you must first cross-compile the MediaPipe framework for your target system, and then invoke the model's inference capabilities on top of this framework. Model Workflow and Output Data Parsing The exceptional performance and detection frame rates achieved by the MediaPipe Hands Landmarker model during runtime stem from its core cascaded two-stage network architecture: 1. Hand Detector: - This model is an extremely lightweight SSD (Single Shot Detector) architecture. Its task is to quickly locate the bounding box of the entire hand in the full image, returning an oriented bounding box. - In video imagery, hand deformation can be extreme (fingers can bend and cross), making direct detection of the entire hand quite challenging. In contrast, the palm is a relatively rigid structure with distinct square-like features that are very easily captured by a network. 2. Hand Landmark Detector: - Once the first-stage Hand Detector model locates the palm region, the system crops, rotates, and aligns that region (using the bounding box returned by the first stage) before feeding it into the second-stage Landmarker model. This model is specialized in precisely detecting 21 3D coordinates within the cropped sub-image. - Because the second-stage model only needs to perform inference on the cropped local region, computational load is drastically reduced while irrelevant background interference is filtered out. This is the fundamental reason why the model runs both fast and accurately. !mediapipe_architecture.png In the interface design and execution flow of the model, the MediaPipe Tasks API includes three main running modes (IMAGE, VIDEO, and LIVE_STREAM), corresponding to single images, image frames in a video file, and real-time streams with callbacks, respectively. These correspond to the following three API call interfaces: - detect(image): Image Mode (IMAGE). Executes without any memory/state. Every input image requires the system to first locate the hand across the entire image before detecting the 21 hand landmarks. This is suitable for processing static photo collections on disk. - detect_for_video(image, timestamp_ms): Video Mode (VIDEO). Maintains memory and executes synchronously with blocking. During inference, it maintains an ROI cache internally. After locating the hand region in the previous frame, it can directly locate that region in the next frame to run the lightweight landmark detection. This is suitable for processing pre-recorded local video files. - detect_async(image, timestamp_ms): Live Stream Mode (LIVE_STREAM). Maintains memory and executes asynchronously without blocking. Tailored specifically for real-time camera feeds. The main thread sends the image to the model inference framework and returns immediately without waiting for the result. Once the underlying C++ thread pool in the inference framework finishes computing, it returns the result to the main thread via a callback function. Therefore, using this mode requires passing a result_callback parameter when initializing HandLandmarkerOptions to specify which function is responsible for receiving the asynchronous results returned by C++. The diagram below illustrates the data structure of the results returned after detecting hands and their landmarks in an image using the above interfaces: !MediaPipe_Result_Structure.png Below is a code example for parsing and utilizing the interface return results: Model Running Example To run this model locally on a PC using Python, you need to install Google's mediapipe support package and download the hand_landmarker.task model file. The download link for the hand_landmarker.task model package is: https://storage.googleapis.com/mediapipe-models/hand_landmarker/hand_landmarker/float16/1/hand_landmarker.task Although the hand_landmarker.task model file uses .task as its extension, it is essentially a compressed archive containing two model files: hand_detector.tflite (palm detection model) and hand_landmarks_detector.tflite (hand landmark detection model). You can simply add a .zip extension to this model file and open it to view its contents: !f15c88bc-9cbc-4f6a-937d-bb545d301cb6.png The following sample code demonstrates how to use OpenCV to perform hand and landmark detection and annotation on a local PC webcam feed: References - https://ai.google.dev/edge/mediapipe/solutions/vision/hand_landmarker - hand_landmarker.ipynb - Colab - mp.tasks.vision.RunningMode | Google AI Edge | Google AI for Developers
Study Notes on Amazon AWS Bedrock Service
Core Features of AWS Bedrock Amazon Bedrock is a fully managed Serverless service provided by AWS that enables unified API access to high-performance foundation models (FMs) from leading AI companies (such as Anthropic, Meta, Mistral, Amazon, etc.), while offering a suite of peripheral features to facilitate the building of generative AI applications. !image.png Below is a list of the core functional modules provided by Amazon Bedrock: - Provides a unified Serverless API interface for Foundation Models (FMs), enabling access to industry-leading multimodal large models (including Anthropic, Meta, Mistral, Cohere, AI21 Labs, Amazon's proprietary Titan and Nova models, as well as domestic models such as DeepSeek, GLM, Qwen, and Kimi). It also allows user applications to dynamically route model inference requests across multiple AWS regions, thereby improving availability and breaking through single-region throughput limits. - Supports native knowledge base applications and fully managed RAG (Retrieval-Augmented Generation) pipelines. It can automatically connect to backend data sources (such as S3, Confluence, SharePoint, etc.), process document chunking, convert them into vectors stored in vector databases like OpenSearch, and automatically perform similarity retrieval and context enhancement when a user asks a question. - Supports model fine-tuning, allowing the use of enterprise proprietary labeled data for supervised fine-tuning of selectively supported models (such as Amazon Titan, Llama, Cohere, etc.) to improve model accuracy in specific domains. - Implements a model-independent safety filtering layer based on Guardrails, supporting custom topic-denial policies to filter harmful information (hate speech, violence, explicit content) from model-generated content, intercept prompt injections, and mask PII (Personally Identifiable Information) to ensure AI outputs comply with enterprise requirements. Relationships and Differences Between Amazon Bedrock, SageMaker, and Q Within the AWS AI ecosystem, the positioning of Amazon Bedrock, Amazon SageMaker, and Amazon Q can easily cause confusion since they all appear to provide support for AI model access on the surface. Below is a detailed explanation of the specific positioning and differences of these three services in actual use. 1. Amazon Q: Positioned as an out-of-the-box, enterprise-customized AI assistant - Essentially a fully built, generative AI assistant tailored for enterprises, directly facing end-users and developers. - Its core feature is that it can be used immediately upon activation without writing any code. It is available in several different editions: - Amazon Q Developer (formerly CodeWhisperer): Primarily embedded in IDEs (such as VS Code) or the AWS Console to assist with writing code, troubleshooting Lambda errors, or explaining complex IoT Rule SQL. - Amazon Q Business: Can be thought of as an enterprise knowledge base for internal employees, allowing companies to integrate internal HR documents, IT guidelines, and other materials. Employees can then ask questions directly via the web interface provided by Q Business, complete with strict access control (i.e., only answering content that the employee has permission to view). - Amazon Q is billed on a per-user basis upon activation; for example, the Amazon Q Business Pro subscription is approximately $20 per user per month. This means its positioning is targeted at internal enterprise use (To-B/To-E) or individual developer efficiency tools, making it unsuitable for direct integration with consumer-facing (To-C) applications. 1. Amazon Bedrock: Positioned as a super gateway for large model APIs (PaaS for GenAI) - This service is a fully managed, serverless generative AI service targeted at application developers (backend/full-stack engineers). - Its core philosophy is that developers do not need to deploy models themselves or manage servers; with just a single line of code (calling an API), they can switch between multiple industry-leading models. Therefore, within the AWS ecosystem, Bedrock serves as the cornerstone for developers to build their own AI applications, providing model services (MaaS) for AI applications. 1. Amazon SageMaker: Positioned as a cloud-based platform for machine learning model development and training (End-to-End MLOps PaaS) - This service is a full-lifecycle machine learning workbench targeted at model developers (data scientists and algorithm engineers). - Using SageMaker for machine learning AI model development and training requires developers to select specific EC2 instance types (such as ml.p4d.24xlarge), configure Jupyter Notebooks, and write model training scripts (PyTorch/TensorFlow). - SageMaker can be used not only for developing and training large generative AI models, but also extensively for traditional predictive AI (such as XGBoost, time series forecasting, etc.). - Therefore, if industry open-source AI models cannot meet specific application requirements, developers can design their own model network architectures, collect massive amounts of training data to train from scratch, or perform deep fine-tuning on existing models. In such cases, SageMaker should be used. The figure below summarizes the positioning differences among the three: !AWS_AI_Architecture_%281%29.png Basic Usage and Testing of Bedrock Models Before writing code to actually invoke models on Bedrock, you first need to install the dependencies in your local environment and configure them properly. Enter the following command in the terminal window of the current system (Windows) to install AWS CLI v2 (after the initial installation, you need to re-enter the terminal to use aws CLI commands): When using the AWS CLI tool for the first time, you need to create a new access key in the AWS Web Console (IAM - Users / [Username] / Security Credentials / Access Keys) and configure this key in the terminal using the aws configure command: To query the list of models supported by a specified AWS Bedrock region and their corresponding model ARNs, use the following command: Below is an example using Python and the AWS boto3 package to access a specified model on Bedrock: As you can see, the process of accessing Bedrock models is very similar to using HTTP to call API endpoints of various large model services, except that it uses the AWS Boto3 bedrock_runtime.converse interface and specifies the modelId.
Summary of Dual-Band Wi-Fi and BT Communication Multiplexing and Concurrent Technologies
During the evolution of Wi-Fi products, Dual Band (2.4GHz and 5GHz) has become a very common and frequently used term. In communication, whether the two frequency bands are multiplexed/switched via time-sharing or can directly achieve dual-band concurrent communication forms two completely different working logics. This article attempts to explain this issue from the perspective of RF link multiplexing and concurrent processing logic. Typical RF Communication Links To understand the RF link multiplexing and concurrency issues of dual-band Wi-Fi and Bluetooth modules during communication, one must first have an overall understanding of the basic composition structure of an RF communication connection. From a simplified perspective, the connection diagram of a typical RF communication link is shown below: !RF_Transceiver_Link_Diagram_%281%29.png As shown in the figure above, this simplified RF communication link consists of the RFIC (also known as the Transceiver), the Radio Frequency Front-End Module (FEM), and matching circuits with antennas. During a specific communication process (taking transmission as an example): - The digital circuit part (the so-called digital baseband part) processes the byte stream 0/1 signals to be transmitted via radio frequency through modulation schemes such as QAM and OFDM to convert them into analog low-frequency baseband signals. This process is essentially a Digital-to-Analog Conversion (DAC) process. - Next, the TX LPF (Low-Pass Filter) shown in the figure filters out the high-frequency digital noise generated during DAC conversion to ensure that the analog baseband signal is smooth and pure. - The PLL/VCO (Local Oscillator) in the figure is responsible for generating high-frequency carrier frequencies in RF communication, such as 2.4GHz and 5GHz. The Up-Mixer is responsible for moving the analog low-frequency baseband signal to the high-frequency carrier frequency band. Therefore, after processing through this stage, the spectrum of the low-frequency baseband signal is shifted to the high-frequency carrier band, and the RF signal output from this stage is a high-frequency carrier RF signal containing the low-frequency baseband signal information. - The power of the high-frequency RF signal output from the above RFIC stage is generally relatively small, so it is further amplified in the PA of the FEM, and then coupled to the antenna for transmission after processing by the RF switch + band-pass filtering circuit + matching network. So, among all stages in the entire RF link above, which ones can be multiplexed during communication for 2.4GHz Wi-Fi, 5GHz Wi-Fi, and 2.4GHz BT? The main principles are as follows: 1. First of all, for wireless communications in the 2.4GHz and 5GHz frequency bands, the FEM and its passive matching circuits in the figure above cannot be cross-band multiplexed due to physical frequency resonance and impedance matching characteristics. This is because the transistor matching circuits inside the PA and LNA are extremely frequency-dependent. The impedance characteristics of 2.4GHz (wavelength approx. 12.5cm) and 5GHz (wavelength approx. 6cm) differ vastly. Meanwhile, the matching network is composed of inductors (L) and capacitors (C), and its reactance is directly affected by the angular frequency . The matching network parameters for different frequency bands need to be tuned independently. Therefore, for 2.4GHz and 5GHz RF communications, independent 2.4G and 5G FEMs and their peripheral matching circuits must be present outside the chip (or within its internal package). 2. As for whether the RFIC part in the figure above can be multiplexed, it depends on the circumstances: 1. Between 2.4GHz Wi-Fi and BT, despite both operating in the 2.4GHz band, because of the huge differences in baseband signal modulation methods and protocol characteristics between the two, various modules inside the RFIC (DAC, ADC, baseband filters, mixers, etc.) are almost two sets of independent physical circuits on the silicon in mainstream Wi-Fi/BT Combo chips. Otherwise, it is difficult to strike a good balance in various physical metrics. 2. Whether 2.4GHz and 5GHz Wi-Fi can share an RFIC depends on whether the device communication requirement is dual-band time-division communication or dual-band concurrent communication. For time-division communication, the two can multiplex the RFIC. At this time, there is only one wideband PLL/VCO inside the chip. When switching frequency bands, the PLL jumps from locking 2.4GHz to locking 5GHz by changing the division ratio. Of course, the cost is a few milliseconds of switching lock time. For concurrent communication, the RFIC cannot be multiplexed. Two completely independent PLLs must be etched on the silicon: one firmly locks onto 2.4G and the other firmly locks onto 5G, interfering with each other not at all. 3. The antenna part can be completely multiplexed. Although electromagnetic waves in the 2.4GHz and 5GHz frequency bands have different frequencies, antenna designs can utilize dual-band resonant antennas (such as PCB inverted-F antennas like PIFA, which contain two vibrators of different lengths), presenting good 50-ohm impedance at both 2.4G and 5G. In specific engineering practices, a Diplexer can be inserted after the matching network and before the antenna in the figure above. This device internally includes a low-pass filter (LPF, allowing 2.4G to pass) and a high-pass filter (HPF, allowing 5G to pass). RF Multiplexing Design for 2.4GHz Wi-Fi and BT Combo Based on the RF link multiplexing design principles summarized above, the following figure shows a typical RF design link for a 2.4GHz Wi-Fi and BT Combo: !Combo_RF_Architecture.png - As mentioned earlier, the baseband signal modulation specifications between Wi-Fi and Bluetooth differ significantly, and Bluetooth's frequency-hopping communication differs greatly from Wi-Fi's OFDM in terms of PLL requirements. Therefore, at the RFIC level, a Combo solution needs a complete and independent implementation for both Bluetooth and Wi-Fi. - As for the FEM, 2.4GHz Wi-Fi and BT both operate in the 2.4GHz band, but because of the huge differences in modulation methods, parameters, and power during communication, they cannot directly multiplex the complete FEM module: - LNA (Low Noise Amplifier): Since weak 2.4GHz signals enter the antenna, the system typically shares the same LNA to amplify the received over-the-air signals. After amplification by the LNA, the signal is split into two via an analog switch and sent to Wi-Fi's Down-Mixer and BT's Down-Mixer respectively. - PA (Power Amplifier): Typically not multiplexed and physically separated. Allowing Bluetooth to directly use Wi-Fi's PA for transmission would cause Bluetooth power consumption to skyrocket exponentially. Therefore, a compact and power-saving Bluetooth-dedicated small PA is usually included within the FEM. - When the FEM shares the communication link (LNA) and antenna during Wi-Fi and BT communication, Wi-Fi transmission and Bluetooth reception cannot be performed simultaneously. At this time, a PTA (Packet Traffic Arbitration) mechanism must be used to schedule and manage the time slots used by Bluetooth and Wi-Fi. - The antenna, matching network, and BPF parts at the RF end can be 100% fully multiplexed. This is because the frequency band is the same and the impedance characteristics are identical; Bluetooth can directly use the 50-ohm perfect matching network tuned for 2.4GHz Wi-Fi. RF Multiplexing Design for 2.4GHz/5GHz Wi-Fi (Time-Division Multiplexing) For most low-cost IoT devices, low-end smartphones, or single-antenna network cards, Wi-Fi chips universally adopt a dual-band selectable or time-division dual-band mechanism to support both frequency bands. Its core characteristic is: one chip with dual bands, but only one of the two can be selected at any given time. Once the Wi-Fi chip and its RF link operate in the 2.4 GHz band, the 5 GHz link remains physically powered down or dormant, and vice versa. In terms of hardware design, to cut costs to the absolute minimum, such dual-band multiplexed chips retain only one MAC, one baseband (PHY), and one frequency synthesizer (Synthesizer/PLL) on the silicon, but the FEM parts must definitely be independent due to the different frequency bands. !TDM_DualBand_WiFi_Architecture_Fixed.png In the dual-band Wi-Fi time-division multiplexing architecture diagram shown above: - Whether it is 2.4GHz or 5GHz Wi-Fi communication, the baseband signal modulation specifications are identical, so the baseband can be completely shared. That means there is only one set of ADC/DAC and baseband filters. - The PLL for high-frequency signals can be multiplexed, retaining only one PLL. When communication needs to be switched between the two frequency bands, the local oscillator signal is provided by switching the phase-locked frequencies to the 2.4G and 5G mixers respectively. Of course, as mentioned above, the cost of this switching is that each switch takes a few milliseconds for the local oscillator signal to stabilize, so the switching frequency cannot be too high. However, it is sufficient for Wi-Fi communication applications that choose between 2.4GHz and 5GHz at any given moment. - Two independent RF front-ends (FEMs) are required for the two frequency bands. In the analog high-frequency band, 2.4G and 5G must have completely independent PAs, LNAs, and T/R switches. - Finally, the RF signals of the two frequency bands are combined using a Diplexer, and the RF signal is coupled to the same dual-band antenna for transmission. RF Multiplexing Design for 2.4GHz/5GHz Wi-Fi (DBDC) Next, the 2.4GHz and 5GHz Wi-Fi RF link in the Dual Frequency Dual Concurrent (DBDC) mode is relatively straightforward. The parts of the entire RF link that can be multiplexed are the fewest, and the cost is the highest: !DBDC_DualBand_WiFi_Architecture.png - Because of dual-band concurrency, the two frequency bands cannot multiplex the PLL. Therefore, the RFIC part consists of two complete and independent implementations. - Due to the huge differences between the 2.4GHz and 5GHz frequency bands, the entire FEM along with its related BPF and matching circuits must also be completely independent implementations. - Finally, the two concurrent RF paths are combined through a Diplexer and transmitted through a single antenna. Of course, they can also be transmitted using their own independent antennas.
A Complete Implementation Workflow of Human Face Recognition Based on InsightFace
This article details the workflow of face recognition in computer vision, and provides a code-level demonstration and explanation using InsightFace, a complete face detection and recognition solution. Complete Workflow of Face Detection + Recognition In industrial deployment practices of face recognition for real-world products, the problem is not solved by a single independent model. Instead, it relies on a cascaded pipeline: Face Detection + Geometric Correction of the Detected Face (Alignment) + Face Recognition (Recognition). In the current industrial landscape—especially on mobile devices (smartphones) and edge computing platforms (such as the Rockchip RK3588)—the standardized face recognition solution is the InsightFace open-source ecosystem. This solution supports real-time, high-precision face recognition on compute-constrained NPUs. Its typical pipeline consists of: SCRFD (lightweight face detection and landmark localization) + affine transformation alignment + MobileFaceNet-ArcFace (lightweight feature extraction). Therefore, the entire face recognition workflow can be divided into three stages: - Phase 1 (Detector - SCRFD): Takes a full high-resolution image as input. The SCRFD model detects face regions and localizes 5 facial landmarks (both eyes, nose tip, and both mouth corners). This stage outputs the bounding boxes of the face regions along with the 5 facial landmarks. - Phase 2 (Aligner - Geometric Transformation): This step performs pure image geometric transformation rather than a deep learning module (OpenCV's warpAffine interface can be used directly for this transformation). It calculates the transformation matrix between the detected landmarks and standard template landmarks. The primary goal is to straighten and normalize the face image detected in the previous stage. - Phase 3 (Recognizer - MobileFaceNet): Takes the cropped and straightened face image as input, outputs its corresponding 512-dimensional feature vector, computes the cosine distance against face feature data stored in a database, performs feature matching, and finds the corresponding face ID. !Face_Recognition_Pipeline.png From the perspective of the data flow diagram: - Detection Stage: Takes a high-resolution image (e.g., 1920 x 1080) as input. Through SCRFD model detection, it finds faces and outputs their corresponding bounding box coordinates () as well as the coordinates of the 5 landmarks for each face (). - Alignment Stage: In this stage, face images of arbitrary sizes and angles are cropped, scaled, and rotated into strictly frontal faces via matrix multiplication. The output dimension for each face image is forcibly fixed to 112x112, resulting in an input tensor dimension of for the next stage. - Recognition Stage: Based on the face images output by the alignment stage, MobileFaceNet extracts a feature vector for each face. Each face outputs a 512-dimensional feature vector, making the output data dimension . Next, based on these 512-dimensional feature vectors, cosine distances are computed against the vector database to perform matching and locate the corresponding face ID. Summary of the Face Recognition and Matching Workflow The process described above—detecting face regions from an input image, aligning the face region images, and extracting feature vectors from the aligned faces—is relatively straightforward. So, once the feature vectors of the face images are extracted, how is face recognition carried out further? In fact, this face recognition process is essentially an image search task within a vector database. This step is identical to the natural language-based image search workflow of OpenAI's CLIP scheme (A Case Study on OpenAI's Lightweight Multi-Modal Model CLIP). - First, a registration library of face feature vectors is required, which pre-stores the feature vectors of known individuals. These known individuals' feature vectors are processed through the same steps of detection-alignment-feature generation to obtain their 512-dimensional feature vectors, which are then stored in this registration library. Note that feature generation must use the same MobileFaceNet model; that is, the enrollment process for known individuals and the subsequent face recognition process must use the exact same MobileFaceNet model for feature vector generation, otherwise the feature vectors will be mismatched. - The subsequent face recognition process is essentially a query operation against the vector registration library. Real-time 512-dimensional face features are captured from the current camera frame and extracted via MobileFaceNet, and cosine distance comparison is performed against the registration library to find and identify the face ID. !Vector_DB_Matching_LR.png If the number of known individuals in the registration library is small, the feature information of all known individuals can be stored and traversed linearly. The time complexity of this traditional linear traversal algorithm is . If the number of known individuals is large, modern vector databases (such as Faiss, Milvus) can be adopted. These databases reduce the search complexity to by introducing Approximate Nearest Neighbor (ANN) algorithms. When storing face feature vectors into the database and matching them using cosine distance, the vector norms must first be uniformly scaled to 1 (a step known as L2 normalization) to project them onto a hypersphere. This is a prerequisite for making cosine similarity equivalent to the vector inner product. Thus, the calculation of cosine similarity simplifies to: Complete Implementation Record of Face Recognition Based on InsightFace The execution workflows for face detection, geometric correction and scaling, face information enrollment, and face recognition have been detailed above. Below is the implementation of the face recognition function using the complete solution provided by InsightFace. First, install the InsightFace package and various dependencies required for the project demo: Face Detection The code workflow for face detection based on InsightFace is as follows: The FaceAnalysis API automatically downloads the buffalo_l model package on its first execution, which contains several ONNX model files for the face detection + recognition pipeline, including: - det_10g.onnx: SCRFD face detection model. - w600k_r50.onnx: ResNet50 face recognition model. - w600k_mbf.onnx: MobileFaceNet recognition model. - genderage.onnx: Predicts gender and age based on face images. - 2d106det.onnx: Dense landmarks module for predicting 106 fine facial contour points (commonly used for beauty filters and stickers). - 1k3d68.onnx: 3D pose module for 3D face reconstruction and deep pitch/yaw angle calculation. When performing face detection, simply call app.get to obtain the coordinates and landmark information of all faces contained within the image frame. Geometric Transformation Having obtained the coordinate positions and landmark information of the face regions in the image via the SCRFD model in the previous step, the next step is to forcibly flatten faces with arbitrary angles (pitch, yaw, roll) into a unified frontal facial topological coordinate system. This ensures that subsequent facial feature extraction steps deal with standardized, forward-facing frontal face images. The basic mathematical principle behind this step is: on a 112x112 resolution canvas, based on the 5 standard frontal benchmark point constants provided by InsightFace and the 5 facial landmark coordinate positions detected in the previous step, a matrix transformation is calculated between the two sets of coordinates. Only rotation, translation, and uniform scaling are permitted (stretching and non-uniform deformation are disallowed), with the objective of minimizing the error between the two sets of landmarks. In practical engineering implementation, OpenCV's cv2.estimateAffinePartial2D can be called directly to execute this geometric transformation. !Face_Affine_Transform.png The code corresponding to this step is as follows: Face Recognition Following the geometric transformation process, each face detected from the original image is transformed into a normalized 112x112 resolution frontal face image. The next step is to extract its feature vector from this standardized face image and query the vector database to retrieve its corresponding ID. For the feature vector extraction task, InsightFace provides two sets of models targeting hardware with varying computing demands: - w600k_r50 (ResNet-50 backbone): Uses the classic ResNet-50 deep residual network with strong feature extraction capabilities, but high computational load, making it suitable for servers and high-performance devices. - w600k_mbf (MobileFaceNet backbone): A lightweight network designed specifically for mobile devices, using Depthwise Separable Convolutions with very few parameters and fast inference speeds, making it suitable for embedded and mobile devices. Regardless of which model is used, a 512-dimensional feature vector is ultimately extracted from the 112x112 face image produced in the previous stage. Now that we have the 512-dimensional feature vector corresponding to the face, the next step is to determine whether they match by calculating the cosine distance between the detected face's feature vector and those stored in the face vector library. Given the feature vectors and of two faces (which are already L2-normalized, i.e., ), the mathematical basis for determining whether they belong to the same person is: If (usually set to 0.45 or 0.5), they can be classified as the same person. Thus, calculating the similarity between two feature vectors is straightforward: you can directly use NumPy's dot product formula and compare it against the threshold:
Training Baidu RT-DETRv2 and Its Deployment on the RK3588 Platform
The official GitHub repository for Baidu's RT-DETRv2 model is: https://github.com/lyuwenyu/RT-DETR. This project contains two independent implementations of the model based on PyTorch and Baidu's PaddlePaddle. This article documents the complete process of custom dataset training, inference testing, ONNX export, and RK3588 development board adaptation for the PyTorch implementation. Source Code Download and Training Dataset Preparation The official GitHub repository for Baidu's RT-DETRv2 model is: https://github.com/lyuwenyu/RT-DETR. First, download the source code from the link above. As shown, the project actually contains both PaddlePaddle and PyTorch implementations for two models: RT-DETR and RT-DETRv2. Here, we only focus on the rtdetrv2_pytorch directory, highlighted in the red box. !331b29d9-52fd-470e-ba95-31042d04f330.png Since the default training data format for the RT-DETR model is COCO, you must convert your own dataset annotation format into the COCO format prior to training. We found a facial feature detection dataset for babies on Roboflow and downloaded it in COCO annotation format. This dataset will be used subsequently to train the RT-DETRv2 model: https://universe.roboflow.com/pavel-txwf8/baby_monitoring-x30nd-1hjmq/dataset/1. Next, begin modifying the training and dataset configuration files. In the source code of RT-DETRv2, the rtdetrv2_pytorch\configs\rtdetrv2 directory contains multiple configuration files for different models and datasets: !image.png The naming conventions of these configuration files follow a typical engineering logic of "Model Scaling + Backbone Network + Training Strategy". - Model Backbone Core Version: - rtdetrv2_r18vd / rtdetrv2_r34vd / rtdetrv2_r50vd：Where r18, r34, and r50 refer to the number of layers in the ResNet backbone network. The vd denotes the use of the ResNet-vd structure. Compared to the original ResNet, it optimizes the downsampling part, which is more beneficial for preserving feature information and is very NPU-friendly. - hgnetv2：A lightweight backbone network developed independently by Baidu. - Model Scaling: - rtdetrv2_s / rtdetrv2_m / rtdetrv2_l：Where s, m, and l are analogous to the Small/Medium/Large scaling in YOLO, typically involving the configuration of hidden layer dimensions and encoder layer counts. - Number of Training Epochs: - 120e / 72e / 30e：Where e stands for Epochs. 120e means training for 120 epochs, generally used for training from scratch or long-cycle fine-tuning to achieve the highest accuracy; 30e means training for 30 epochs, typically used for rapid validation or fine-tuning based on pre-trained models. - Dataset Annotation Format: - coco：Indicates that the configuration adapts to the MS-COCO dataset annotation format by default. Here, rtdetrv2_r18vd_120e_coco.yml is used as the base to modify the configuration file for custom dataset training. The main modification involves appending the configuration information of your own dataset to the end of the aforementioned configuration file as follows: RTDETRv2 Model Training and Local Inference Testing Before training, download the pre-trained weights of the model so that training can be resumed/continued from them. Since we selected rtdetrv2_r18vd_120e_coco as our training configuration file, the corresponding pre-trained weights to download should be rtdetrv2_r18vd_120e_coco.pth. The download link is: https://github.com/lyuwenyu/storage/releases/download/v0.1/rtdetrv2_r18vd_120e_coco.pth. At this point, the training data configuration files and pre-trained weights are ready. The next step is to invoke the rtdetrv2_pytorch\tools\train.py script to execute the training process: Training command: Resuming training after interruption: The files generated during and after the training process are saved in the output_dir specified in rtdetrv2_r18vd_120e_coco.yml, which includes: - checkpoint0xxx.pth：Weight files for each epoch during training. - last.pth: Weight file for the final epoch. - best.pth: The weight file of the epoch with the highest mAP during training, which represents the final training result. This model weight file should be used for inference testing and subsequent model conversion. - log.txt: Records the Loss and AP variations throughout the training process. After training is complete, you can run the following inference test command to verify the effectiveness of the training results. The project contains a local inference script file: rtdetrv2_pytorch\references\deploy\rtdetrv2_torch.py . Issues Encountered During Training: Since some images in the training dataset have excessively high resolutions (e.g., 6000x4000), you need to modify the code in train.py to lift the restriction on input image resolution limits; otherwise, errors will occur during training. Add the following to the beginning of train.py: Exporting the Model to ONNX Baidu's RT-DETRv2 PyTorch implementation provides an ONNX model export script: rtdetrv2_pytorch\tools\export_onnx.py. However, because we intend to deploy it to the RK3588 platform later, we need to modify this script to remove the post-processing logic. This step is crucial for subsequent RK3588 model conversion and board-side inference. Modify the ONNX conversion script based on rtdetrv2_pytorch\tools\export_onnx.py, mainly concerning the model declaration and ONNX export configurations: Additionally, since this model contains the Grid Sample operator—matching the issue encountered in Custom Data Training of the RT-DETR Model and Its Adaptation on RK3588—and the latest version of the RKNN runtime library (v2.3.2) does not support this operator, we follow the same resolution path by using a Monkey Patch to replace the Grid Sample operator prior to exporting the model: Then, execute the script above to export the ONNX format (it is recommended to use onnxsim to further simplify the operators in the model file after each ONNX export): RK3588 Board-Side Model Conversion and Execution The subsequent workflow and steps for converting and running the RKNN model on the RK3588 platform are identical to those described in Custom Data Training of the RT-DETR Model and Its Adaptation on RK3588, with one key exception: the model conversion must be set to the FP16 quantization format. If configured to a quantization mode like W8A8, the board-side execution results are poor, likely because quantization compromises the algorithm's accuracy.