feature article
Subscribe Now

Prophesee Image Chip Is Motion Sensitive

Unusual Image Sensor Sees Things in a Different Light

Security cameras and image sensors are pretty stupid, as we’ve explained before. Yet they’re important for a range of applications that have nothing to do with taking selfies at the beach. We use cameras for inspection, for security, for quality control, for medical diagnosis, and for about a hundred other things. 

If you’ve ever dealt with image sensors, you know they’re simple in concept but tricky in practice. You generally get a full image every so often, and maybe a series of partial, interpolated images in between. Your job is to make sense of the images. What changed? What am I looking for? Does the image meet my criteria? How much do I need to filter it to get at the information I really want? 

Then there are all the traditional photographer’s problems, like lenses, lighting, exposure, focal length, chromatic aberration, and optical effects that most of us know little about. But hey, once you’ve dialed all of that in, you’re ready to start processing images. Lots and lots of images. 

With this as background, a French startup has decided to overhaul the whole idea of the embedded camera and create a new type of image sensor — one that doesn’t need so much babysitting because it does much of the work for you. It’s not just a conventional CMOS image sensor with an MCU or an AI accelerator attached. It’s a different type of device that works in a unique way. 

Paris-based Prophesee says its image sensors use “neuromorphic vision that works like the human eye.” That’s a bit confusing and hyperbolic, but basically correct. The key difference is that Prophesee’s image sensors don’t capture images. They look for changes in images and report only that. 

That sounds like a trivial difference, but it changes the whole equation when you’re designing a security camera, inspection system, or industrial monitor. The company believes its Metavision image sensors will make less work for programmers, save energy compared to conventional image sensors, and produce better and more accurate results. A win all the way around. But there are some drawbacks. 

Normal cameras take pictures at a given frame rate. They’re a series of snapshots, really, and finding the differences between one frame and the next is often the first step to analyzing the images. What changed from the previous frame? Do I care? Is it important? 

Frame-by-frame image capture uses either a lot of bandwidth or a lot of processing power to compress the images and save bandwidth. Either way, you’re analyzing frames or comparing frames to each other, looking for significant differences. Higher frame rates produce smoother motion and a more accurate representation of reality, but at the cost of more data, more bandwidth, or more aggressive compression. 

In contrast, each individual pixel in Prophesee’s device is sensitive to motion. It never captures an image but instead uses time-continuous monitoring of light levels in the photodiode to report whether its pixel is changing. If the scene in front of the image sensor is perfectly static, the device produces no output. But if something moves, several hundred pixels will fire off a signal to an external host processor. The result effectively outlines the motion edges

Each pixel in the sensor is asynchronous from all the others; there’s no clock, and there’s no concept of frame rate. They simply report change when they see it, usually within microseconds or milliseconds, which would equate to a frame rate of many thousands of frames/sec. This high equivalent frame rate allows the chip to track very fine motion with high accuracy. By comparison, a conventional camera at 60 fps would miss 99.99% of the motion data. That’s fine if you’re tracking bowling balls, which aren’t very sprightly. But Prophesee expects its devices to be used for medical inspection, vibration monitoring, and small-particle tracking, among other uses. 

The company points to mechanical vibration monitoring and predictive maintenance as ideal use cases. Every pixel operates autonomously, so they can each adjust their own exposure rather than share a single camera-wide setting, which simplifies the task of lighting a greasy old motor. High-frequency vibration would normally require a camera with a high frame rate and lots of buffering, but that’s not an issue here. Each pixel can monitor almost any arbitrary frequency independent of the others, on any visible part of the machinery. 

Prophesee’s event-driven design philosophy seems like a recipe for spiky bursts of data that might be hard to buffer. Luca Verre, company cofounder and CEO, says that’s not the case. “It’s less data, more information. Even the worst-case scenario of shaking the camera up and down” doesn’t produce a ton of information, he says. Walls, floors, backgrounds, and other components of a natural scene are redundant and get ignored. It’s only the edges of motion that get reported, not every infill pixel. That “sparseness,” he says, provides the data you want without the clutter you don’t want. 

The challenge is rewriting already existing code, which is designed for frame-by-frame analysis. To that end, Prophesee is providing an event-based SDK and an ongoing investment in software development. 

For those times when you really do want the full image, Prophesee’s sensor won’t help you. It never captures a photo; it reports only on what’s moved. It also doesn’t detect color, for the same reason. In some ways, Prophesee’s image sensor is closer to a lidar or radar sensor than a conventional CMOS image sensor. 

Verre points out that radar and lidar are active arrays, whereas his company’s sensor is passive (assuming there’s an ambient light source). Lidar sensors need to shoot out frickin’ laser beams (and radar sends radio pulses), which then reflect back and are processed by the array. Both produce a point cloud with distance information, whereas Prophesee’s sensors are “flat,” in the sense that, like cameras, they don’t inherently perceive distance. Radar and lidar are also synchronous and frame based, so detecting motion means analyzing consecutive frames. In the case of an automotive ADAS application, that might take a significant fraction of a second before you can tell if the car ahead is moving, whereas Prophesee’s chip might be 10× quicker.  

As exotic as all of this sounds, Prophesee is already on its third generation of sensor chips, with Gen4 due “soon.” Sounds like there’s a lot of movement inside the development lab.

Leave a Reply

featured blogs
Sep 21, 2023
Wireless communication in workplace wearables protects and boosts the occupational safety and productivity of industrial workers and front-line teams....
Sep 21, 2023
Labforge is a Waterloo, Ontario-based company that designs, builds, and manufactures smart cameras used in industrial automation and defense applications. By bringing artificial intelligence (AI) into their vision systems with Cadence , they can automate tasks that are diffic...
Sep 21, 2023
At Qualcomm AI Research, we are working on applications of generative modelling to embodied AI and robotics, in order to enable more capabilities in robotics....
Sep 21, 2023
Not knowing all the stuff I don't know didn't come easy. I've had to read a lot of books to get where I am....
Sep 21, 2023
See how we're accelerating the multi-die system chip design flow with partner Samsung Foundry, making it easier to meet PPA and time-to-market goals.The post Samsung Foundry and Synopsys Accelerate Multi-Die System Design appeared first on Chip Design....

Featured Video

Chiplet Architecture Accelerates Delivery of Industry-Leading Intel® FPGA Features and Capabilities

Sponsored by Intel

With each generation, packing millions of transistors onto shrinking dies gets more challenging. But we are continuing to change the game with advanced, targeted FPGAs for your needs. In this video, you’ll discover how Intel®’s chiplet-based approach to FPGAs delivers the latest capabilities faster than ever. Find out how we deliver on the promise of Moore’s law and push the boundaries with future innovations such as pathfinding options for chip-to-chip optical communication, exploring new ways to deliver better AI, and adopting UCIe standards in our next-generation FPGAs.

To learn more about chiplet architecture in Intel FPGA devices visit https://intel.ly/45B65Ij

featured paper

An Automated Method for Adding Resiliency to Mission-Critical SoC Designs

Sponsored by Synopsys

Adding safety measures to SoC designs in the form of radiation-hardened elements or redundancy is essential in making mission-critical applications in the A&D, cloud, automotive, robotics, medical, and IoT industries more resilient against random hardware failures that occur. This paper discusses the automated process of implementing the safety mechanisms/measures (SM) in the design to make them more resilient and analyze their effectiveness from design inception to the final product.

Click here to read more

featured chalk talk

Automotive Electronic Seat Control
Sponsored by Infineon
Today’s automotive seat design must keep in mind size, cost, battery life and passing EMC testing. In this episode of Chalk Talk, Amelia Dalton and Rick Browarski from Infineon investigate the newest innovations in automotive electronic seat control. They take a closer look at the anatomy of power seats today, the role that an ECU plays in the control of electronic seats and how Infineon chip set offerings can help you with your next smart power seat design.
Sep 22, 2022