feature article
Subscribe Now

Prophesee Image Chip Is Motion Sensitive

Unusual Image Sensor Sees Things in a Different Light

Security cameras and image sensors are pretty stupid, as we’ve explained before. Yet they’re important for a range of applications that have nothing to do with taking selfies at the beach. We use cameras for inspection, for security, for quality control, for medical diagnosis, and for about a hundred other things. 

If you’ve ever dealt with image sensors, you know they’re simple in concept but tricky in practice. You generally get a full image every so often, and maybe a series of partial, interpolated images in between. Your job is to make sense of the images. What changed? What am I looking for? Does the image meet my criteria? How much do I need to filter it to get at the information I really want? 

Then there are all the traditional photographer’s problems, like lenses, lighting, exposure, focal length, chromatic aberration, and optical effects that most of us know little about. But hey, once you’ve dialed all of that in, you’re ready to start processing images. Lots and lots of images. 

With this as background, a French startup has decided to overhaul the whole idea of the embedded camera and create a new type of image sensor — one that doesn’t need so much babysitting because it does much of the work for you. It’s not just a conventional CMOS image sensor with an MCU or an AI accelerator attached. It’s a different type of device that works in a unique way. 

Paris-based Prophesee says its image sensors use “neuromorphic vision that works like the human eye.” That’s a bit confusing and hyperbolic, but basically correct. The key difference is that Prophesee’s image sensors don’t capture images. They look for changes in images and report only that. 

That sounds like a trivial difference, but it changes the whole equation when you’re designing a security camera, inspection system, or industrial monitor. The company believes its Metavision image sensors will make less work for programmers, save energy compared to conventional image sensors, and produce better and more accurate results. A win all the way around. But there are some drawbacks. 

Normal cameras take pictures at a given frame rate. They’re a series of snapshots, really, and finding the differences between one frame and the next is often the first step to analyzing the images. What changed from the previous frame? Do I care? Is it important? 

Frame-by-frame image capture uses either a lot of bandwidth or a lot of processing power to compress the images and save bandwidth. Either way, you’re analyzing frames or comparing frames to each other, looking for significant differences. Higher frame rates produce smoother motion and a more accurate representation of reality, but at the cost of more data, more bandwidth, or more aggressive compression. 

In contrast, each individual pixel in Prophesee’s device is sensitive to motion. It never captures an image but instead uses time-continuous monitoring of light levels in the photodiode to report whether its pixel is changing. If the scene in front of the image sensor is perfectly static, the device produces no output. But if something moves, several hundred pixels will fire off a signal to an external host processor. The result effectively outlines the motion edges

Each pixel in the sensor is asynchronous from all the others; there’s no clock, and there’s no concept of frame rate. They simply report change when they see it, usually within microseconds or milliseconds, which would equate to a frame rate of many thousands of frames/sec. This high equivalent frame rate allows the chip to track very fine motion with high accuracy. By comparison, a conventional camera at 60 fps would miss 99.99% of the motion data. That’s fine if you’re tracking bowling balls, which aren’t very sprightly. But Prophesee expects its devices to be used for medical inspection, vibration monitoring, and small-particle tracking, among other uses. 

The company points to mechanical vibration monitoring and predictive maintenance as ideal use cases. Every pixel operates autonomously, so they can each adjust their own exposure rather than share a single camera-wide setting, which simplifies the task of lighting a greasy old motor. High-frequency vibration would normally require a camera with a high frame rate and lots of buffering, but that’s not an issue here. Each pixel can monitor almost any arbitrary frequency independent of the others, on any visible part of the machinery. 

Prophesee’s event-driven design philosophy seems like a recipe for spiky bursts of data that might be hard to buffer. Luca Verre, company cofounder and CEO, says that’s not the case. “It’s less data, more information. Even the worst-case scenario of shaking the camera up and down” doesn’t produce a ton of information, he says. Walls, floors, backgrounds, and other components of a natural scene are redundant and get ignored. It’s only the edges of motion that get reported, not every infill pixel. That “sparseness,” he says, provides the data you want without the clutter you don’t want. 

The challenge is rewriting already existing code, which is designed for frame-by-frame analysis. To that end, Prophesee is providing an event-based SDK and an ongoing investment in software development. 

For those times when you really do want the full image, Prophesee’s sensor won’t help you. It never captures a photo; it reports only on what’s moved. It also doesn’t detect color, for the same reason. In some ways, Prophesee’s image sensor is closer to a lidar or radar sensor than a conventional CMOS image sensor. 

Verre points out that radar and lidar are active arrays, whereas his company’s sensor is passive (assuming there’s an ambient light source). Lidar sensors need to shoot out frickin’ laser beams (and radar sends radio pulses), which then reflect back and are processed by the array. Both produce a point cloud with distance information, whereas Prophesee’s sensors are “flat,” in the sense that, like cameras, they don’t inherently perceive distance. Radar and lidar are also synchronous and frame based, so detecting motion means analyzing consecutive frames. In the case of an automotive ADAS application, that might take a significant fraction of a second before you can tell if the car ahead is moving, whereas Prophesee’s chip might be 10× quicker.  

As exotic as all of this sounds, Prophesee is already on its third generation of sensor chips, with Gen4 due “soon.” Sounds like there’s a lot of movement inside the development lab.

Leave a Reply

featured blogs
Sep 26, 2021
https://youtu.be/Ivi2dTIcm9E Made at my garden gate (camera Carey Guo) Monday: Ten Lessons from Three Generations of Google TPUs Tuesday: At a Digital Crossroads Wednesday: Announcing Helium, Hybrid... [[ Click on the title to access the full blog on the Cadence Community si...
Sep 24, 2021
Wi-Fi, NB-IoT, Bluetooth, LoRaWAN... This webinar will help you to choose the appropriate connectivity protocol for your IoT application....
Sep 23, 2021
The GIRLS GO Engineering scholarship provides opportunities for women in tech and fosters diversity in STEM; see the winners of our 2021 engineering challenge! The post GIRLS GO Engineering! Empowers Our Next-Gen Women in Tech appeared first on From Silicon To Software....
Sep 23, 2021
The Global Environment Facility Small Grants Programme (GEF SGP), implemented by the United Nations Development Programme, is collaborating with the InnovateFPGA contest. Showcase your  skills with Intel Edge-Centric FPGAs and help develop technical solutions that reduce env...

featured video

Gesture Detection for Automotive In-Cabin Applications

Sponsored by Texas Instruments

See how using 60GHz radar for automotive in-cabin gesture is ideal due to its small size and ability to sense through various materials. Applications using gesture control include changing radio stations, answering phone calls, opening windows, and more.

Click to learn more about gesture detection using 60GHz mmWave radar sensors

featured paper

Ultra Portable IO On The Go

Sponsored by Maxim Integrated (now part of Analog Devices)

The Go-IO programmable logic controller (PLC) reference design (MAXREFDES212) consists of multiple software configurable IOs in a compact form factor (less than 1 cubic inch) to address the needs of industrial automation, building automation, and industrial robotics. Go-IO provides design engineers with the means to rapidly create and prototype new industrial control systems before they are sourced and constructed.

Click to read more

featured chalk talk

Seamless Ethernet to the Edge with 10BASE-T1L Technology

Sponsored by Mouser Electronics and Analog Devices

In order to keep up with the breakneck speed of today’s innovation in Industry 4.0, we need an efficient way to connect a wide variety of edge nodes to the cloud without breaks in our communication networks, and with shorter latency, lower power, and longer reach. In this episode of Chalk Talk, Amelia Dalton chats with Fiona Treacy from Analog Devices about the benefits of seamless ethernet and how seamless ethernet’s twisted single pair design, long reach and power and data over one cable can solve your industrial connectivity woes.

Click here for more information about Analog Devices Inc. ADIN1100 10BASE-T1L Ethernet PHY