I must admit that I have a personal interest in this column. Hmm, that didn’t come out the way I wanted it to. Obviously, I have a personal interest in all my columns, but this one more so than most.
I have a cousin named Graham, who is three years older than your humble narrator (I pride myself on my humility). Graham has always been a bit of a trickster. For example, one day when I was about four and a half years old, he came round to our house to play. My mum made spaghetti Bolognese for lunch. While Mum was out of the room, Graham told me that spaghetti was made from peeled worms. Strangely, I didn’t feel hungry anymore.
It wasn’t until years later that I learned the truth. I think I must have been around six years old. I was watching a children’s program on television. It was some time in April, as I recall. They showed a video of kids in Italy harvesting spaghetti from spaghetti trees. One child would hold a big flat wicker basket. Another kid would use special spaghetti shears to cut the long strands of spaghetti dangling from the tree branches. The severed spaghetti would drop down into the basket. That’s when I realized Graham truly had been tricking me. Spaghetti wasn’t peeled worms—it grew on trees! I felt so silly (but I still don’t like eating spaghetti).
The picture below shows Graham and me sitting in the open boot (trunk) of a car. We didn’t travel this way—we were at a family picnic somewhere. What’s that on my head, you ask? It’s a knotted handkerchief; that is, one of my dad’s handkerchiefs with knots tied in the corners. This was to protect me from the sun. We didn’t have sunscreen or baseball caps back then. This was a common sight for children of both sexes and men of all ages in 1960s England. I only hope it was a clean handkerchief, but we digress…
Me ~3 years old (left) and Graham ~6 years old (right).
The reason I mention all this here is that about 30 or so years ago, Graham lost his sight. This wasn’t instantaneous—it took place over a couple of years—but he ended up unable to see anything at all.
I’m tremendously proud of Graham because he hasn’t let going blind slow him down at all. He danced to the Caleigh band at my wedding (think “barn dance on steroids”) and started hosting his own local radio show.
A few years ago, during a trip to England to visit my dear old mum, I was invited to give a talk on advanced technologies at my alma mater, Sheffield Hallam University. Knowing Graham’s interest in such things, I invited him to attend. Things had changed in the years since I graduated—new buildings, different layouts—and I got lost on the way to the designated lecture theatre. As I was standing at an intersection of corridors in confusion (which is my natural state, so I was playing to my strengths), I heard the tapping of a cane and Graham hove into view. Even though he was as new to the building as I was, it was Graham who guided us to our destination. Please keep all this in mind as we proceed to the next portion of this column.
In 2013, Intel began producing hardware and software that utilized depth tracking, gestures, facial recognition, eye tracking, and other technologies under the Perceptual Computing brand. In 2014, the company rebranded its Perceptual Computing line of technology as Intel RealSense. I recall promises of delight regarding RealSense being embedded in augmented reality (AR) and virtual reality (VR) headsets; however, Intel withdrew from that market around 2021 (sad face).
A little over a month ago, as I pen these words, RealSense completed its spinout from Intel Corporation and began operating as an independent company. The folks at RealSense are no longer actively pursuing mass-market integration into AR/VR headsets. Currently, RealSense specializes in AI-powered computer vision technologies, particularly depth-sensing systems, that enable machines to perceive and understand their environment. Their product lineup includes depth camera systems, System-on-Chip (SoC) vision processors that compute depth and tracking data in real-time, and open-source software development kits (SDKs) that facilitate the integration of RealSense technology into various platforms and applications.
There’s so much to all of this that it’s hard to know where to start. Take a look at the image of the RealSense D415 Depth Camera shown below.
The D415 Depth Camera (Source: RealSense)
There’s much more to this than meets the eye (no pun intended). Fortunately, I was just chatting with Roi Ziss, Chief Product Architect at RealSense, and Guy Halperin, Vice President at Intel and head of R&D at RealSense, and they managed to explain things in a way that even I could understand.
Let’s start with the two RGB+ cameras at the sides (I’ll explain the ‘+’ part of the moniker in a minute). The vision processor SoC inside the D415 uses these to provide stereoscopic machine vision. This works on the same principle as human binocular vision. Each camera captures the scene from a slightly different angle (like your left and right eyes). By comparing these two images, the system can estimate the depth (distance) of objects in the scene on a pixel-by-pixel basis.
“Ah,” you may say, “but what happens if you are in a dark room or if you are looking at a plain white wall, for example.” Well, this is one of the clever bits. CMOS image sensors are naturally sensitive to a broad spectrum—they pick up visible light and near-infrared (NIR) radiation.
If the IR isn’t blocked, it “bleeds” into the red, green, and blue channels, washing out colors and making them look strange. To prevent this, manufacturers add an IR cut filter (a.k.a. IR-blocking filter) that removes most of the NIR light, thereby ensuring accurate color reproduction. In the case of the D415, the two RGB cameras on the sides do not have IR cut filters, which means they also respond to IR (which explains the reasoning behind my use of the ‘+’ in ‘RGB+’ earlier).
Now look at the picture of the D415 again. The big “thing” (I hope I’m not being too technical) we see a little to the left of center is an IR projector. But this projector isn’t simply flooding the scene with IR light. No sirree, Bob! Instead, it projects patterns of ~8,000 “dots” that the vision processor can detect, even in the dark or when facing a uniform, flat surface, such as a white wall, for example.
To put this another way, in order to generate an accurate 3D point cloud, the D415 doesn’t rely solely on stereoscopy. Instead, it uses a combination of stereoscopic disparity (left and right cameras) and active IR projection patterns (to give the system texture for better depth resolution).
This leads us to the third RGB camera located to the right of center in the image above. This camera does include an IR cut filter, which means it provides a true RGB image that can be used by an AI to perform object detection and identification functions. The D415 provides this true RGB image along with the 3D point cloud generated from its depth sensing system—all synchronized on a pixel-by-pixel and frame-by-frame basis—to the customer’s system, which is responsible for providing the AI portion of the process.
Actually, I should point out that, while the vision processor SoC in the D415 doesn’t provide any AI processing, the guys and gals at RealSense recently introduced a new vision processor that does include this capability for users who don’t wish to perform the AI themselves, but that’s a story for another day.
Roi and Guy told me that RealSense has thousands of customers across various fields; they say that they are also constantly amazed by the diverse ways people are utilizing RealSense technology. One major market is robotics; another is agriculture, such as robots picking fruit, which is still a form of robotics, now that I think about it.
However, the application that triggered the writing of this column is the NIIRA audiovisual perception device, which was created by the clever chaps and chapesses at Eyesynth. As shown in the image below, this device utilizes the RealSense D415 subsystem, which is integrated into the frames surrounding the eyepieces.
Meet NIIRA glasses (Source: Eyesynth)
These NIIRA glasses utilize RealSense depth sensing technology to capture 3D spatial data, which is then processed to generate auditory cues for individuals who are visually impaired. This integration allows users to perceive their surroundings through sound, thereby enhancing mobility and spatial awareness.
NIIRA glasses are connected to an AI-powered image processing unit, which is about the size of a Smartphone (Source: Eyesynth)
In slightly more detail, the RealSense SDK delivers the raw perception layer, and Eyesynth’s NIIRA system adds the intelligence. Eyesynth’s proprietary AI interprets the 3D scene, performing object detection, recognition, and contextual analysis (e.g., “chair,” “person,” “doorway”). This semantic information is then translated into soundscapes via the NIIRA’s bone-conduction audio system. The translation into auditory “language” is Eyesynth’s secret sauce.
The reason for using a bone-conduction system is that visually impaired people rely heavily on sound cues from their surroundings, so the last thing they want is to block their ears with headphones or diminish their hearing with earbuds.
As an aside, I often use bone conduction headphones myself while performing tasks like gardening. I’ve seen too many films where some unsuspecting person is listening to music while weeding their flowerbeds, totally oblivious to the zombies approaching from behind (see also my Headphones for a Zombie Apocalypse Cool Beans Blog).
Nothing can replace losing one’s sight, but I think this type of technology will greatly ease the lives of the visually impaired. And this is only the beginning; it won’t be long before the descendants of the NIIRA can learn to recognize family, friends, and colleagues, and perform additional tasks, such as describing the content of written signage (only the information it deems to be important—not every random piece of writing it encounters).
I really wish I could get one of these systems for my cousin Graham. Sad to relate, however, that will have to remain a pipedream until I win the lottery. While we are all waiting for that frabjous day to arrive, I’d love to hear your thoughts on anything you’ve read here.
