Clik here to view.
Most tradeshow experiences tend to be limited to the exhibition floor and a couple of breakout sessions, all housed within the spacious convention center floor plan. However, embedded world North America seemed to diverge from this with a number of companies offering tours of their facilities, one of these companies was NXP. EDN was able to tour their Austin campus with a guide of their “Smart Home Innovation Lab”. This lab is a proving ground for a number of IoT applications and edge computing applications where systems and applications engineers can take the NXP microcontrollers (MCU) and microprocessors (MPU) as well as their RF and sensor tech, and see how they might be able to build a prototype. However, Smart Home Innovation Lab might be a bit of a misleading name since many of proof-of-concept designs fell into the medical and automotive realms where many of the underlying technologies would naturally find use cases that extended well beyond these fields.
The concept and implementation of the internet of things (IoT) has been a very well-discussed topic, especially within the smart home where endless companies have found (and are continuing to find) innovative ways to automate home functions. However, using inference at the edge is relatively nascent, and therefore the use cases where existing IoT applications can be augmented or improved by AI is growing rapidly. In all of these demos, NXP engineers integrated one of their i.MX crossover MCUs for local edge processing capabilities. So, the tour was geared more towards the use cases of TinyML.
The tour spanned over an hour, with Austin-based systems engineers walking the group through each demonstration that took place in a “garage”, “kitchen”, “living room”, media room/theater”, and a “gym”. Many of the demonstrations involved modified appliances that were taken off-the-shelf while some prototypes were co-developed with customers in partnership.
Home mode automations
Many of the solutions were focused on using more unified application-level connectivity standards such as Thread and Matter to simplify integration where smart home devices from different vendors can be used in a single smart home “fabric”. The lab contained two Matter fabrics, including a commercially available Thread border router and an NXP open Thread border router that used the i.MX 93. The NXP open source home automation system that connects many of the IoT devices and acts as a backend to the “dashboard” that appears in Figure 1.
Image may be NSFW.
Clik here to view.
Figure 1 NXP Innovation Lab tour with the home dashboard appearing on the screen and door lock device to the left.
Their proprietary home control system has two main “home mode” automations available: one where the user was away from home and one where they were present at home. The “away from home” demo included automated functions such as the dimming of the lab lights, blinds going down, pausing any audio streaming, and the locking the door. When the user is present, all of these processes are essentially reversed, automating some very basic home functions.
A touch-free lighting experience
The ultra wideband (UWB) technology found in the recent SR150 chip includes a ranging feature that can, for instance, track a person as they are walking through their home. This was another demonstration where a system engineer held the UWB-enabled mobile device and the lights and speakers within the lab essentially followed them, turning on the lights and streaming the radio station through the speakers that were available locally, in the room they were physically occupying and turning off all lights/speakers in the rooms that they had exited. Other use cases for this are in agriculture for locating sprinklers covered in mud, or, in medical applications to kick off automations/check-ups when a nurse walks into a patient’s room. This could also be extended to the automotive space, automatically opening the door that the user is approaching.
Door lock
As with many smart home appliances, smart locks are nothing new. Typically though, these door locks can be remotely engaged with an app, requiring a more manual approach to the solution. The door lock prototype used five different technologies–keypad, fingerprint, face recognition, NFC, and UWB–as well as the i.MX RT1070 MCU/MPU to lock or unlock (Figure 2). The lock used a face recognition algorithm with depth perception while the UWB tech used an angle of attack (AoA) algorithm to ascertain whether the user is approaching the lock from outside the facility or within it. This way, the door lock can be engaged only with multiple forms of identification for building security management; or, in smart home applications, where the door lock will automatically open upon approach from the outside.
Image may be NSFW.
Clik here to view.
Figure 2 Smart door lock using the SR150 and i.MX RT1070 with integrated keypad, fingerprint, face recognition, NFC, and UWB.
The garage: Automotive automations
The “garage” included a model EV where i.MX MCUs are used to run cluster and infotainment systems, demonstrating the graphics capability of the platform. There was also a system that displayed a bird’s eye view of the vehicle where the MCU takes the warp image from the four cameras mounted at different angles, dewarps them, and stitches them together to recreate this inclusive view of the vehicle’s surroundings.
Image may be NSFW.
Clik here to view.
Figure 3 Garage demos showing the EV instrument cluster and infotainment running on i.MX MCUs.
The demo in Figure 4, shows a potential solution to a current potential problem in EVs: a large, singular human-machine interface (HMI) that both the driver and passenger are meant to use. While it does offer a clean, sleek aesthetic, the single screen could be inconvenient when one user needs it as a dashboard, while the other might want it for entertainment purposes. The dual-view display will simultaneously display two entirely different images for users sitting on the right-hand side or left-hand side of the screen. This is made possible by the large viewing angles of the display, so that the driver and passenger are able to view their specific application on the entire screen without impacting the user experience. The technology involves sending two outputs interleaved together where the screen has the ability to deinterleave them and show them on the screen.
This comes with the additional ability to independently control the screen using the entire space available within HMI without impacting the application of the driver or passenger. In other words, a passenger could essentially play Tetris on the screen without messing around with the driver’s map view. This is achieved through electrodes installed under the seat where each electrode is connected to the driver’s, or passenger’s, respective touch controller. Another quite obvious application for this would be in gaming, removing the need for two screens or a split-screen view.
Image may be NSFW.
Clik here to view.
Figure 4 A single dual-view display that simultaneously offers two different views for users sitting to the left or right of it. Electrodes installed under the seats allow one user to independently control the screens via touch without impacting the application of the other user.
Digital intrusion alarm
The digital intrusion alarm prototype seen in Figure 5 can potentially be added on to a consumer access point or router to protect it from malicious traffic such as a faulty IoT device that might jam the network. The design uses the i.MX 8M+ where a ML model is trained with familiar network traffic over a period of time so when unfamiliar traffic is observed, it is flagged as malicious and blocked from the network. The demo showcased a denial of service (DoS) attack that was blocked. If the system detects a known device and blocks it, the user is able to fix the issue, and unblock the device so that it can connect back to the network.
Image may be NSFW.
Clik here to view.
Figure 5 Digital intrusion alarm that is first trained to monitor the traffic specific to the network for a period of time before beginning the process of monitoring network traffic for any potential bad actors.
Smart cooktop, coffee maker, and pantry
A smart cooktop can be seen in Figure 6, the prototype uses face detection to detect whether or not a chef is present, all of this information is processed locally on the device itself. In the event of unsafe conditions, e.g., water boiling over, a burner left on without cookware present, excessive smoke, burning food, the system could potentially detect it and shut off. Once shut off, the home dashboard will show that the cooktop is turned off. Naturally, the entire process can be done without AI, however, it can massively speed up the time it takes for the cooktop to recognize that a cook is present. Other sensors can be integrated to either fine-tune the performance of the system or eliminate the potential intrusion of having a camera.
Image may be NSFW.
Clik here to view.
Figure 6 Smart cooktop demo with facial recognition to sense if a cook is present.
The guide continued to a “smart barista” that uses facial recognition on the i.MX’s eIQ neural processing unit (NPU) to customize the type of coffee delivered from the coffee maker. A pantry classification system also uses the i.MX RT1170 along with a classification and detection model to take streams of the pantry and performance inference to inform the user of the items that are taken out of the pantry. The system could potentially be used in the refrigerator as well to offer the user with recipe recommendations or grocery list recommendations. However, as one member of the tour noted, pantries are generally packed with goods that would not necessarily be within view of this system for vision-based classification.
Current state indicator
Another device was trained, at a very basic level, with knowledge on car maintenance using a GM car manual and used a large language model (LLM) to respond to prompts such as: “How do I use cruise control?” or “Why isn’t my car turning on?” The concept was presented as a potential candidate for the smart home where these smart speakers could potentially be trained on the maintenance of various appliances, e.g., washing machines, dryers, dishwashers, coffee makers, etc., so that the user can ask questions on maintenance or use.
The natural question is how is this concept any different from established smart speakers? Like many of the devices already described, this is all processed locally and where there is no interaction with the cloud to process data and present an answer. This concept can also be expanded for preventive or predictive maintenance in the case where appliances are outfitted with sensors that transmit appliance status information to, for instance, show a continuous record on the service life motor bearings within a CNC machine; or, the estimation life of a drive belt in a washing machine.
An automated media room
The Innovation Lab houses a living room space that experiments with automated media control using UWB, millimeter-wave, vision, and voice activation (Figure 7). In this setup, the multiple mediums will first detect the presence of individuals seated on the couch to trigger a sequence of events including the lights turning on, the blinds going up, and the TV turning on to a preferred program of choice. A system utilizing the i.MX 8M+ and an attached Basler camera as well as another system with an overhead camera will use vision to detect persons and perform personalizations such as changing the channel from a show with more adult content to one catered to a younger audience if a child walks into the area. For those who would find that particular personalization vexing (myself included) the system is meant to be trained towards the preference of the individual.
Another demo in this area included NXP’s “Audio Vista” or sonic optimization. This solution uses a UWB ranging to detect the precise location of the person/people sitting on the couch and communicates with the four speakers located throughout the space to let the user know where/how speakers will have to be moved for an optimal audio setup. This very same underlying UWB technology can be trained to detect heart arrhythmias, breathing, or falls for home health applications. Another media control experiment involved using echo cancellation to extract a voice from a noisy environment so that users do not have to speak over audio to, for instance, ask a smart speaker to pause a TV program.
Image may be NSFW.
Clik here to view.
Figure 7 The living room space that experiments with automated media control using UWB, millimeter-wave, vision, and voice activation. The UWB system can be seen up front, millimeter-wave transmitter and receiver are seated above the speakers, and Basler camera to the far right.
The home theater: Downsizing the AV receiver
In the second-to-last stop, everyone sat in a theater to experience the immersive Dolby Atmos surround sound, an experience provided by the i.MX 8M Mini (Figure 8). The traditional AV receiver design involves a specific audio codec IC as well as an MCU and MPU to handle functions such as the various connectivities and the rendering of video. The multicore i.MX 8M Mini’s Arm Cortex A53s have abundant processing capability so that the audio portion of the processing in a traditional AV receiver takes only ~30% of the entire IC; all this while the 8M Mini handles its own controls, processing, and many other renderings as well.
Dolby Atmos has previously been considered a premium sound function that was not easily provided by products such as soundbars or low- to mid-tier AV receivers. Powerful processors such as the 8M Mini can integrate these functions to lower the barrier of entry for companies, providing not only Dolby Atmos decoding, but MPEG and DTS:X as well. The i.MX also runs a Linux operating system in conjunction with a real-time operating system (RTOS) allowing users to easily integrate Matter, Thread, and other home automation connectivity protocols onto the AV receiver or soundbar.
Image may be NSFW.
Clik here to view.
Figure 8 Theater portion of the Innovation Lab with the Dolby Atmos immersive surround sound experience processed on the i.MX 8M Mini.
The gym: Health and wellbeing demos
The gym showcased a number of medical solutions starting with medical devices with embedded NTAGs so users can scan and commission the device using NFC to, for example, verify the authenticity of the medication that you are injecting into your body. Other medical devices included insulin pouches that utilized NXP’s BLE MCUs that allowed them to be scanned with a phone so that a user could learn the last time they took an insulin shot. Smart watches, fitness trackers, based upon NXP’s RTD boards were also shown that go for up to a week without being charged.
Another embedded device that measured ECG was demonstrated (Figure 9) that has the ability to take ECG data, encrypt it, and send the information to the doctor of choice. There are three main secure aspects of this process:
- Authentication that establishes the OEM credentials
- Verification of insurance details through NFC
- Encryption of health data being sent
The screen in the image shows what a doctor might view on a daily basis to track their patients. This could, for instance, sense a heart attack and call an ambulance. This concept could also be extended to diabetic patients that must track insulin and blood sugar levels as they change through the day.
Image may be NSFW.
Clik here to view.
Figure 9 Tour of health and wellness devices with a monitor displaying patient information for a doctor that has authenticated themselves through an app.
Aalyia Shaukat, associate editor at EDN, has worked in the engineering publishing industry for over 8 years. She holds a Bachelor’s degree in electrical engineering from Rochester Institute of Technology, and has published works in major EE journals as well as trade publications.
Related Content
- An engineer’s playground: A tour of Silicon Labs’ labs
- Analog Devices’ approach to heterogeneous debug
- Ultra-wideband tech gets a boost in capabilities
- New i.MX MCU serves TinyML applications
The post Real examples of the IoT edge: A guide of NXP’s Innovation Lab appeared first on EDN.