feature article
Subscribe Now

Lattice mVision Stack

Powering Up Low-Power Embedded Vision

We’ve elaborated at length in these pages on the remarkable capabilities FPGAs bring to the world of embedded vision and vision analytics. With enormous numbers of new applications demanding embedded vision capabilities at the endpoint and edge, there’s a huge green field of opportunity out there, just waiting for FPGAs to take over. The ability of FPGAs to accelerate AI tasks on a miserly power budget, perform various video filtering and processing tasks, and adapt to handle basically any interface standard for the myriad sensors, displays, and other related hardware out there is unmatched by any other technology on the market.

But, adoption of FPGAs continues to be limited by the difficulty – both perceived and real – of customizing the devices for each particular application. Historically, FPGA use has been the exclusive purview of engineering teams with expertise in HDL design and FPGA experience. The skill and knowledge required to create Verilog descriptions of custom hardware, simulate and synthesize that code, and complete place and route as well as timing closure – has been a major barrier to entry for the FPGA party for the last three decades. 

FPGA companies are acutely aware of this, of course, and perhaps the most effective weapon they have to combat the problem is the pre-engineered “stack” for specific classes of applications. A stack is an almost-complete sample application – including development boards, reference designs, relevant IP blocks, software, firmware, and whatever else you need to get to your application’s version of “Hello World.” With a stack, you can usually be up and running with something that approximates your project in a matter of hours or even minutes from first powering up your development kit. Then, it’s just a matter of modifying the already-working system to include your own special sauce. While this doesn’t do away with the need for hardware and FPGA expertise altogether, it certainly can dramatically reduce the learning curve.

Lattice Semiconductor has just announced mVision, a new “stack” aimed at machine vision in power-constrained edge and endpoint designs. Lattice has long led the field in ultra-low power FPGAs (think fractions of watts). They have shipped over a billion devices into systems such as smartphones, cameras, and other battery-powered and portable devices. Lattice estimates the machine vision market will grow to over $18B by 2025, and they figure mVision will help their aim to claim a chunk of that loot in the high-volume, low-power end of the spectrum.

mVision supports three different families of Lattice FPGAs – their MIPI-bridging Crosslink and Crosslink-plus families (with up to 6K LUTs, 150 mW power dissipation, and 2.5×2.5 mm packaging), the recently-announced Lattice Nexus Crosslink-NX (built on 28nm FDSOI technology, with 40K LCs, 0.3 W power dissipation, and 6×6 mm packaging) and the larger and more powerful ECP5 family (with 85K LUTs, 1 W power dissipation, and 10×10 mm packaging). This gives us a nice range of capability, power dissipation, and form factor depending on our project’s constraints – all under the same mVision umbrella.

Lattice’s video interface platform (VIP) development boards are clever mix-and-match stacked modules that let you pair your choice of FPGA, camera interface, and IO interfaces. Supported interfaces include MIPI, and LVDS image sensors, MIPI, eDP, DisplayPort, HDMI, GigE Vision, and USB3 IO. Basically, you can connect to most any image sensor and most any display standard with ready-made IP blocks that drop into your design, and the development boards give you the hardware to back it up.

Once you’re connected to your choice of sensors and displays, Lattice has assembled a powerful library of IP blocks to help you tame your video stream. These include image processing functions such as Defect Pixel Correction, HDR, Color Space Conversion, Black-level Correction, 2D Scaler, Gamma Correction, Overlay, AWB, De-Bayer, Video Frame Buffer, Color Filter, and Interpolation; bridging and aggregation functions including CSI-2/DSI D-PHY Receiver, CSI-2/DSI D-PHY Transmitter, Byte to Pixel Converter, Pixel to Byte Converter, FPD-LINK Receiver, and FPD-LINK Transmitter; and connectivity functions including DisplayPort Rx, DisplayPort Tx, GigE, and GigE vision. 

If figuring out how to connect the IP blocks you want to make an application is a little baffling, Lattice helps you get started with a rich set of reference designs that will cover a large gamut of typical applications. These include MIPI DSI/CSI-2 to OpenLDI LVDS Interface Bridge, N to 1 CSI-2/DSI Duplicator, subLVDS to MIPI CSI-2 Image Sensor Bridge, N Input to 1 Output MIPI CSI-2 Camera Aggregator Bridge, 4:1 Image Aggregation, MIPI CSI-2 N to 1 Sideby-side Aggregation, 3D Depth Mapping, and ISP. If what you’re doing resembles one of those applications, your design is mostly done already (don’t tell your boss). 

Lattice’s tool support includes their legacy “Diamond” as well as their newer “Radiant” design tool suites – depending on your FPGA choice. And, if all these reference designs, IP, and tools still seem a little more than your team wants to bite off, Lattice has a range of design services available with experienced engineering talent that is well-versed in this type of design work – from their own Lattice Design Group, as well as third parties including Helion Engineering Vision, Tata Elxsi, and Bitec. These providers can do anything from developing a specialized IP block to implementing entire turnkey solutions.

The notion of the stack is to enable teams with a wide range of engineering skillsets and capabilities to take advantage of the remarkable capabilities that FPGAs bring to the embedded vision market. With the mVision stack, Lattice is largely uncontested at the ultra-low end of the power spectrum. None of the competitors such as Xilinx, Intel, and even Microchip (Microsemi) offer devices with the very-low power consumption of Lattice’s devices (although their FPGAs tend to be much larger and more capable). If your application is heavily power constrained, there don’t appear to be a lot of comparable FPGA alternatives.

It’s been interesting to watch Lattice re-define their mission and strategy after the failed acquisition a couple years ago. What seemed like bad news for the company at the time may have sparked a new wave of energy and innovation that will benefit the market overall. Lattice’s focus on small, inexpensive, low-power programmable logic technology now makes them unique in a market where demand for those qualities is likely to expand rapidly.

One thought on “Lattice mVision Stack”

  1. Hi Kevin — great column — I must admit that I’ve been impressed with the stuff Lattice has been doing over the past couple of years — I also like the “stack” approach, like the mVision stack you talk about here — and I also think their sensAI and Neural Network Compiler (for artificial intelligence (AI) and machine learning (ML) designs) is rather interesting.

Leave a Reply

featured blogs
May 24, 2024
Could these creepy crawly robo-critters be the first step on a slippery road to a robot uprising coupled with an insect uprising?...
May 23, 2024
We're investing in semiconductor workforce development programs in Latin America, including government and academic partnerships to foster engineering talent.The post Building the Semiconductor Workforce in Latin America appeared first on Chip Design....

featured video

Why Wiwynn Energy-Optimized Data Center IT Solutions Use Cadence Optimality Explorer

Sponsored by Cadence Design Systems

In the AI era, as the signal-data rate increases, the signal integrity challenges in server designs also increase. Wiwynn provides hyperscale data centers with innovative cloud IT infrastructure, bringing the best total cost of ownership (TCO), energy, and energy-itemized IT solutions from the cloud to the edge.

Learn more about how Wiwynn is developing a new methodology for PCB designs with Cadence’s Optimality Intelligent System Explorer and Clarity 3D Solver.

featured paper

Altera® FPGAs and SoCs with FPGA AI Suite and OpenVINO™ Toolkit Drive Embedded/Edge AI/Machine Learning Applications

Sponsored by Intel

Describes the emerging use cases of FPGA-based AI inference in edge and custom AI applications, and software and hardware solutions for edge FPGA AI.

Click here to read more

featured chalk talk

Power High-Performance Applications with Renesas RA8 Series MCUs
Sponsored by Mouser Electronics and Renesas
In this episode of Chalk Talk, Amelia Dalton and Kavita Char from Renesas explore the first 32-bit MCUs based on the new Arm® Cortex® -M85 core. They investigate how these new MCUs bridge the gap between MCUs and MPUs, the advanced security features included in this new MCU portfolio, and how you can get started using the Renesas high performance RA8 series in your next design. 
Jan 9, 2024
19,054 views