feature article
Subscribe Now

Lattice mVision Stack

Powering Up Low-Power Embedded Vision

We’ve elaborated at length in these pages on the remarkable capabilities FPGAs bring to the world of embedded vision and vision analytics. With enormous numbers of new applications demanding embedded vision capabilities at the endpoint and edge, there’s a huge green field of opportunity out there, just waiting for FPGAs to take over. The ability of FPGAs to accelerate AI tasks on a miserly power budget, perform various video filtering and processing tasks, and adapt to handle basically any interface standard for the myriad sensors, displays, and other related hardware out there is unmatched by any other technology on the market.

But, adoption of FPGAs continues to be limited by the difficulty – both perceived and real – of customizing the devices for each particular application. Historically, FPGA use has been the exclusive purview of engineering teams with expertise in HDL design and FPGA experience. The skill and knowledge required to create Verilog descriptions of custom hardware, simulate and synthesize that code, and complete place and route as well as timing closure – has been a major barrier to entry for the FPGA party for the last three decades. 

FPGA companies are acutely aware of this, of course, and perhaps the most effective weapon they have to combat the problem is the pre-engineered “stack” for specific classes of applications. A stack is an almost-complete sample application – including development boards, reference designs, relevant IP blocks, software, firmware, and whatever else you need to get to your application’s version of “Hello World.” With a stack, you can usually be up and running with something that approximates your project in a matter of hours or even minutes from first powering up your development kit. Then, it’s just a matter of modifying the already-working system to include your own special sauce. While this doesn’t do away with the need for hardware and FPGA expertise altogether, it certainly can dramatically reduce the learning curve.

Lattice Semiconductor has just announced mVision, a new “stack” aimed at machine vision in power-constrained edge and endpoint designs. Lattice has long led the field in ultra-low power FPGAs (think fractions of watts). They have shipped over a billion devices into systems such as smartphones, cameras, and other battery-powered and portable devices. Lattice estimates the machine vision market will grow to over $18B by 2025, and they figure mVision will help their aim to claim a chunk of that loot in the high-volume, low-power end of the spectrum.

mVision supports three different families of Lattice FPGAs – their MIPI-bridging Crosslink and Crosslink-plus families (with up to 6K LUTs, 150 mW power dissipation, and 2.5×2.5 mm packaging), the recently-announced Lattice Nexus Crosslink-NX (built on 28nm FDSOI technology, with 40K LCs, 0.3 W power dissipation, and 6×6 mm packaging) and the larger and more powerful ECP5 family (with 85K LUTs, 1 W power dissipation, and 10×10 mm packaging). This gives us a nice range of capability, power dissipation, and form factor depending on our project’s constraints – all under the same mVision umbrella.

Lattice’s video interface platform (VIP) development boards are clever mix-and-match stacked modules that let you pair your choice of FPGA, camera interface, and IO interfaces. Supported interfaces include MIPI, and LVDS image sensors, MIPI, eDP, DisplayPort, HDMI, GigE Vision, and USB3 IO. Basically, you can connect to most any image sensor and most any display standard with ready-made IP blocks that drop into your design, and the development boards give you the hardware to back it up.

Once you’re connected to your choice of sensors and displays, Lattice has assembled a powerful library of IP blocks to help you tame your video stream. These include image processing functions such as Defect Pixel Correction, HDR, Color Space Conversion, Black-level Correction, 2D Scaler, Gamma Correction, Overlay, AWB, De-Bayer, Video Frame Buffer, Color Filter, and Interpolation; bridging and aggregation functions including CSI-2/DSI D-PHY Receiver, CSI-2/DSI D-PHY Transmitter, Byte to Pixel Converter, Pixel to Byte Converter, FPD-LINK Receiver, and FPD-LINK Transmitter; and connectivity functions including DisplayPort Rx, DisplayPort Tx, GigE, and GigE vision. 

If figuring out how to connect the IP blocks you want to make an application is a little baffling, Lattice helps you get started with a rich set of reference designs that will cover a large gamut of typical applications. These include MIPI DSI/CSI-2 to OpenLDI LVDS Interface Bridge, N to 1 CSI-2/DSI Duplicator, subLVDS to MIPI CSI-2 Image Sensor Bridge, N Input to 1 Output MIPI CSI-2 Camera Aggregator Bridge, 4:1 Image Aggregation, MIPI CSI-2 N to 1 Sideby-side Aggregation, 3D Depth Mapping, and ISP. If what you’re doing resembles one of those applications, your design is mostly done already (don’t tell your boss). 

Lattice’s tool support includes their legacy “Diamond” as well as their newer “Radiant” design tool suites – depending on your FPGA choice. And, if all these reference designs, IP, and tools still seem a little more than your team wants to bite off, Lattice has a range of design services available with experienced engineering talent that is well-versed in this type of design work – from their own Lattice Design Group, as well as third parties including Helion Engineering Vision, Tata Elxsi, and Bitec. These providers can do anything from developing a specialized IP block to implementing entire turnkey solutions.

The notion of the stack is to enable teams with a wide range of engineering skillsets and capabilities to take advantage of the remarkable capabilities that FPGAs bring to the embedded vision market. With the mVision stack, Lattice is largely uncontested at the ultra-low end of the power spectrum. None of the competitors such as Xilinx, Intel, and even Microchip (Microsemi) offer devices with the very-low power consumption of Lattice’s devices (although their FPGAs tend to be much larger and more capable). If your application is heavily power constrained, there don’t appear to be a lot of comparable FPGA alternatives.

It’s been interesting to watch Lattice re-define their mission and strategy after the failed acquisition a couple years ago. What seemed like bad news for the company at the time may have sparked a new wave of energy and innovation that will benefit the market overall. Lattice’s focus on small, inexpensive, low-power programmable logic technology now makes them unique in a market where demand for those qualities is likely to expand rapidly.

One thought on “Lattice mVision Stack”

  1. Hi Kevin — great column — I must admit that I’ve been impressed with the stuff Lattice has been doing over the past couple of years — I also like the “stack” approach, like the mVision stack you talk about here — and I also think their sensAI and Neural Network Compiler (for artificial intelligence (AI) and machine learning (ML) designs) is rather interesting.

Leave a Reply

featured blogs
Jan 21, 2022
Here are a few teasers for what you'll find in this week's round-up of CFD news and notes. How AI can be trained to identify more objects than are in its learning dataset. Will GPUs really... [[ Click on the title to access the full blog on the Cadence Community si...
Jan 20, 2022
High performance computing continues to expand & evolve; our team shares their 2022 HPC predictions including new HPC applications and processor architectures. The post The Future of High-Performance Computing (HPC): Key Predictions for 2022 appeared first on From Silico...
Jan 20, 2022
As Josh Wardle famously said about his creation: "It's not trying to do anything shady with your data or your eyeballs ... It's just a game that's fun.'...

featured video

AI SoC Chats: Understanding Compute Needs for AI SoCs

Sponsored by Synopsys

Will your next system require high performance AI? Learn what the latest systems are using for computation, including AI math, floating point and dot product hardware, and processor IP.

Click here for more information about DesignWare IP for Amazing AI

featured paper

Using the MAX66242 Mobile Application, the Basics

Sponsored by Analog Devices

This application note describes the basics of the near-field communication (NFC)/radio frequency identification (RFID) MAX66242EVKIT board and an application utilizing the NFC capabilities of iOS and Android® based mobile devices to exercise board functionality. It then demonstrates how the application enables the user with the ability to use the memory and secure features of the MAX66242. It also shows how to use the MAX66242 with an onboard I2C temperature sensor which demonstrates the energy harvesting feature of the device.

Click to read more

featured chalk talk

Complete Packaging for IIoT Devices

Sponsored by Mouser Electronics and Phoenix Contact

Industrial Internet of Things (IIoT) design brings a new level of demands to the engineering team, particularly in areas like thermal performance, reliability, and scalability. And, packaging has a key role to play. In this episode of Chalk Talk, Amelia Dalton chats with Joel Boone of Phoenix Contact about challenges and solutions in IIoT design packaging.

Click here for more information about Phoenix Contact ICS 50 Enclosure System