The age of intelligent video is upon us.  We’ve all played with the new Kinect devices from Microsoft.  We’ve read about lane departure and collision avoidance systems being integrated into cars.  We’ve heard about technologies like facial recognition being used in security applications.  No longer are we content to stream “dumb” video from place to place.  While the master control room with giant arrays of video feeds may make a compelling image for science fiction, the reality will be more like a giant array of cameras – and a very small number of monitors – showing us only the things that actually deserve our attention.  

In security systems, we now have the capability to deploy large numbers of high-quality cameras at a very reasonable cost.  High Definition (HD) video and WIde Dynamic Range (WDR) techniques give us the best quality images we’ve ever had.  Now, the weak links are the human eyes watching all those hours of uneventful image streams.  What we need are intelligent cameras that can monitor their own video and can bring the humans in only when something interesting happens.  

With as much data as all those cameras generate, we end up with a networking and storage problem as well.  There is no need to transmit terabytes of video of the same empty parking lot through your network and use up valuable storage space to accumulate it on massive servers. Ideally, we’d be able to analyze the data at the source – in the camera – and send along only the parts that matter.  Our networks and storage servers would appreciate that very much.

This is not a new idea, of course.  But intelligent, real-time monitoring of all that data, particularly on HD video streams, is a big processing challenge.  Conventional DSP-based solutions burn up too much power to be practically deployed inside cameras, so people look to devices like FPGAs to bring the computation efficiency up to useful levels.  Just this year we’ve done articles on Lattice’s HD/HDR video development kit and Xilinx’s video development platforms and discussed the debut of the Embedded Vision Alliance.  

Now, Altera is attacking another critical part of the video analytics picture – the hardware IP.  While FPGAs can deliver spectacular computation efficiency (in terms of power), the effort and expertise required to build a competent analytics system using an FPGA are considerable.  Altera is announcing this week that they’ve partnered with video analytics firm Eutecus to offer a line of video analytics IP for Altera FPGAs.  The interesting new technology here comes from Eutecus, of course, but equally intriguing is the business model Altera is using to deploy the technology to their customers.

The products consist of “MVE,” a multi-core video analytics engine that can fit in a Cyclone IV FPGA, plus a software GUI that allows you to set up and configure the analytics rules and events.  The IP takes advantage of Altera’s Nios II soft-core embedded processors as well as the capability to create hardware accelerators in the FPGA fabric.  The MVE consists of Eutecus’s InstantVision Embedded software running on Altera’s Nios II core – combined with “C-MVA” (Cellular Multicore Video Analytics) co-processor IP cores implemented in the Cyclone IV fabric.  Of course, since you’re putting all that IP in an FPGA, you can also do other tasks such as managing video inputs and outputs, streaming, conversion, display, and whatever else your system requires.  

One of the cool parts of the Eutecus system is the ease of programming the events.  The provided GUI lets the developer configure each device for the specific scene being monitored and define the events that should trigger an alert.  The overall geometry of the scene can be specified to help the system understand the perspective effects, and the relative sizes of objects of interest – cars, people, etc – can be configured.  Then, events to be monitored can be specified almost like state transitions.  “If an object of this size starts here – and then moves to this other location – we know that was a car making a left turn.”  “If an object of this size is in this area – we know that is a pedestrian in the roadway.”  “If two objects of this size are in this proximity in this area – we have a potential collision.”  The rules are simple to change and edit.

Development teams wanting to deploy intelligent security systems based on HD cameras could clearly save enormous amounts of development time using this analytics kit.  However, one of the most interesting parts of the collaboration is the method by which Altera is deploying and delivering the IP.  Many of the companies developing these types of systems are smaller systems houses with limited development budgets.  Buying a one-time IP license for a large fee would be a stretch for many of them.  To get around that problem, this IP is being made available directly from Altera, without a big, up-front NRE charge. Instead, the IP is licensed on a royalty basis.

Uh, oh.

I bet you are imagining complicated legal documents, verifiable reporting of manufacturing and shipping volumes, and other nightmare scenarios involving lawyers, accountants, and big mahogany negotiating tables, huh?

It’s actually much cooler and easier than that.  

Altera licenses the IP using their normal encrypted IP distribution mechanism.  However, the IP itself is enabled by a security chip that sits alongside the FPGA.  For each device where you want to deploy the IP, you buy a security chip from Altera, and the device unlocks the license on the FPGA-based IP.  No fancy contracts required.  Pretty nifty, huh?  This IP model actually has a lot of potential for higher-end FPGA IP.  Assuming the overhead of connecting the security chip is small, and assuming that the level of security is adequate (for the commercial purposes of Altera and their partners), it’s a very customer-friendly model that could solve a lot of the traditional issues with complex and expensive IP distribution and deployment.

For now, when you license the IP to use in your FPGA design, you’re locked to a specific version.  However, we can see this category of IP being ripe for easy upgrades and enhancements – which we’d be wanting to deploy in systems already in the field.  For that, Altera would need to come up with an infrastructure and a license model to handle all that updating.  It’s not quite as simple as a mobile phone app.  At least not yet.

As for the Orwellian future that technologies like this could enable – we can only try to make sure that the all-seeing-eyes that we are placing on the world use that knowledge for our benefit.  There are myriad applications where video analytics can dramatically enhance safety and security, and we will be better off for that.