Part 1 of this article series provided a detailed overview of a trigger-based vision system for embedded applications. It also delved into latency measurements of this swift response vision system while explaining latency-related design strategy and measurement methods. Now, Part 2 provides a detailed treatment of optimizing power consumption and wakeup latency of this embedded vision system.
In Linux, power management is a key feature that allows the system to enter various sleep states to conserve energy when the system is idle or in a low-power state. These sleep states are typically categorized into “suspend” (low-power modes) and “hibernate” (suspend to disk) modes that are part of the Advanced Configuration and Power Interface (ACPI) specification. Below are the main Linux sleep states.
Figure 1 Here is a highlight of Linux sleep states. Source: eInfochips
- Wakeup (Idle): System fully active; CPU and components fully powered, used when the device is actively in use; high power consumption, no resume time needed.
- Deep sleep (Suspend-to-RAM): CPU and motherboard components mostly disabled, RAM refreshed, used for deeper low-power states to save energy; low power consumption varying by C-state, fast resume time (milliseconds).
- System sleep (Suspend-to-Idle): CPU frozen, RAM in self-refresh mode, shallow sleep state for low-latency, responsive applications (for example, network requests); low power consumption, higher than hibernate, fast resume time (milliseconds).
- Hibernate (Suspend-to-Disk): Memory saved to disk, system powered off, used for deep power savings over long periods (for instance, laptops); almost zero power consumption, slow resume time (requires reading from disk).
Suspend To Ram (STR) offers a good balance, as it powers down most of the system but keeps RAM active (self-refresh mode) for a quick resume, making it suitable for devices needing quick wakeups and energy savings. Hibernate, on the other hand, saves more power by writing the system’s state to disk and powering down completely, but resulting in slower wakeup times.
Qualcomm’s chips, especially those found in Linux embedded devices, support two power-saving modes to help optimize battery life and improve efficiency. These power-saving modes are typically controlled through the system’s firmware, the operating system, and specific hardware components. Here are the main power-saving modes supported by Qualcomm-based chipsets:
- Suspend to RAM (STR)
- Suspend to Idle (S2Idle)
Triggers suspend mode by writing “mem” or “freeze” in /sys/power/state.
Figure 2 Here is how source flow looks like when device enters sleep and wakes up. Source: eInfochips
As the device goes into suspend modes, it performs the following tasks:
- Check whether the suspend type is valid or not
- Notify user space applications that device is going into sleep state
- Freeze the console logs
- Freeze kernel thread and buses and freeze unwalkable interrupts
- Disable non-bootable CPU (CPU 1-7) and keep RAM into self-refresh mode
- Keep the device into sleep state until any wakeup signal is received
Once the device receives the wakeup interrupt or trigger, it starts resuming the device in reverse order while suspending the device.
While the system is suspended, the current consumption of the Aikri QRB4210 system on module (SoM) comes around to ~7 mA at 3.7-V supply voltage. Below is the waveform of the current drained by the system on module.
Figure 3 Here is how current consumption looks like while Aikri QRB4210 is in suspend mode. Source: eInfochips
Camera sensor power modes
Camera sensors are designed to support multiple power modes such as:
- Streaming mode
- Suspend mode
- Standby mode
Each mode has distinct power consumption and latency. Latency varies by power-saving level and sensor state. Based on use case, ensure the camera uses the most efficient mode for its function, especially while the system is in power saving mode like deep sleep or standby. This ensures balanced performance and power efficiency while maintaining quick reactivation.
In GStreamer, the pipeline manages data flow through various processing stages. These stages align with the GStreamer state machine, marking points in the pipeline’s lifecycle. The four main states are NULL, READY, PAUSED and PLAYING, each indicating the pipeline’s status and controlling data and event flow. Here’s a breakdown of each of the stages (or states) in a GStreamer pipeline:
Figure 4 The above image outlines GStreamer’s pipeline stages. Source: eInfochips
- Null
- This is the initial state of the pipeline, and it represents an inactive or uninitialized state. The pipeline is not doing any work in this state. All elements in the pipeline are in their NULL state as well.
- In this state, the master clock (MCLK) from the processor to the camera sensor is not active; the camera sensor is in reset state and the current consumption by the camera is almost zero.
- Ready
- In this state, the pipeline is ready to be configured but has not yet started processing any media. It’s like a preparation phase before actual playback or processing starts.
- GStreamer performs sanity check and plugin compatibility for the given pipeline.
- Resources can be allocated (for example, memory buffers and device initialization).
- GStreamer entering this state does not impact MCLK’s state or reset signal. If GStreamer enters from the NULL state to the READY state, the MCLK remains inactive. On the other hand, if it enters the READY state from the PLAYING state, the MCLK remains active.
- The current consumption in the READY state depends on the previous state; this behavior can be further optimized.
- Paused
- This state indicates that the pipeline is set up and ready to process media but is not actively playing yet. It’s often used when preparing for playback or streaming while maintaining control over when processing starts.
- All elements in the pipeline are initialized and ready to start processing media.
- Like the READY state, the current consumption in the PAUSED state depends on the previous state, so some optimization in the camera stack can help reduce the power consumption during this state.
- Playing
- The PLAYING state represents the pipeline’s fully active state, where data is being processed and media is either being rendered to the screen, played back through speakers, or streamed to a remote system.
- MCLK is active and the camera sensor is out of reset. The current consumption is highest in this state as all camera sensor data is being captured and passed through the pipeline.
To minimize wakeup latency of the camera sensor while maintaining the lowest sleep current, GStreamer pipeline should be put in the NULL state when the system is suspended. To understand the power consumption due to MCLK and RESET signals assertion, below is the comparison of current consumption between the NULL state of GStreamer pipeline and the READY state of GStreamer pipeline while QRB4210 is in the suspended state.
Figure 5 Current consumption shown while GStreamer is in NULL state and QRB4210 is in suspend mode at ~7 mA. Source: eInfochips
Figure 6 Current consumption shown while GStreamer is in READY state and QRB4210 is in suspend mode at ~30 mA. Source: eInfochips
While the camera is in the NULL state, the QRB4210 system on module draws a current of ~7mA, which is equivalent to the current drawn by the system on module in the suspended state when no camera is connected. When the camera is in the READY state, the QRB4210 system on module draws a current of around ~30 mA. The above oscilloscope snapshot shows the waveforms of the consumed current. All the measured currents are at 3.7-V supply voltage for the QRB4210 system on module.
Latency measurement results
Latency was measured between two trigger events: the first occurs when the device wakes up and receives the interrupt at the application processor, and the second occurs when the first frame becomes available in the DDR after image signal processor (ISP) runs.
As mentioned earlier in Part 1, the scenario is simulated using bash script that keeps the device into the suspend mode and triggers the QRB4210 platform from sleep and wakeup using the RTC wake alarm.
We have collected the camera wakeup latency by changing the camera state from PLAYING to READY and from PLAYING to NULL. In each scenario, three different use cases are followed, which are recording camera stream into eMMC, recording camera stream into SD card, and previewing camera stream to display. The resulting latency is as follows:
- Camera state in READY
Table 1 Latency measurements are shown in READY state. Source: eInfochips
- Camera state in NULL
Table 2 Latency measurements are shown in NULL state. Source: eInfochips
The minimum, maximum, and average values presented in the above tables have been derived by running each scenario for 100 iterations.
Apart from measuring the latency numbers programmatically, below are the results measured using the GPIO toggle operation between two reference events while switching the camera state from READY to PLAYING.
Table 3 Latency measurements are conducted using GPIO. Source: eInfochips
Now refer to the following oscilloscope images for different scenarios used in the GPIO toggle measurement method.
Figure 7 GPIO toggle measurements are conducted while recording into eMMC at 410.641 ms. Source: eInfochips
Figure 8 GPIO toggle measurements are conducted while recording into SD card at 382.037 ms. Source: eInfochips
Figure 9 GPIO toggle measurements are conducted during preview on display at 359.153 ms. Source: eInfochips
Trade-off between current consumption and wakeup latency
Based on the simulated result, we see that current consumption and wakeup latency are dependent on each other.
The consolidated readings show that a camera pipeline in the READY state consumes more current while it takes less time to wake up. On the other hand, if the camera pipeline is in the NULL state, it consumes less current but takes more time to wake up. Refer to the table below for average data readings.
Table 4 The above data shows trade-off between current consumption and wakeup latency. Source: eInfochips
All latency data is measured between the reception of the wakeup IRQ at the application processor and the availability of the frame in DDR after the wakeup. It does not include the time taken by a motion detection sensor to sense and generate an interrupt for the application processor. Generally, the time taken by a motion detection sensor is negligible compared to the numbers mentioned above.
Future scope
To reduce the current consumption of a device in the sleep state optimization, you can follow the steps below:
- Disable redundant peripherals and I/O ports.
- Prevent avoidable wakeups by ensuring that peripherals don’t resume from sleep unnecessarily.
- Disable or mask unwanted wakeup triggers or subsystem that can wake the device from a sleep state.
- Use camera standby (register retaining) mode so that MCLK can be stopped, or its frequency can be reduced.
- Enable LCD display only when preview use case is running.
To optimize wakeup latency, follow the guidelines below:
- Make use of the camera standby mode to further optimize latency to generate the first frame.
- Reduce camera sensor frame size to optimize frame scan time and ISP processing time.
- Disable redundant system services.
- Trigger camera captures from lower-level interface rather than using the GStreamer.
Trigger-based cameras offer an efficient solution for capturing targeted events, reducing unnecessary operation, and managing resources effectively. They are a powerful tool in applications where specific, event-driven image or video capture is needed.
By conducting experiments on the Aikri QRB4210 platform and making minimal optimizations to the Linux operating system, it’s possible to replicate or create a robust trigger-based camera system, achieving ~400-500 ms latency with minimal current consumption.
Jigar Pandya—a solution engineer at eInfochips, an Arrow company—specializes in board bring-up, board support package porting, and optimization.
Priyank Modi—a hardware design engineer at eInfochips, an Arrow company—has worked on various Aikri projects to enhance technical capabilities.
Related content
- The State of Machine Vision
- What Is Machine Vision All About?
- Processors, Sensors Drive Embedded Vision
- Shaping the Scene for Vision Standardization
- Embedded Vision: Giving Machines the Power of Sight
The post Optimize power and wakeup latency in swift response vision systems – Part 2 appeared first on EDN.