editor's blog
Subscribe Now

The Essence of Big Data

iStock_000064716741_Small.jpgThe biggest buzzword that every press release must reference in the title these days is “Internet of Things” (IoT). The second biggest buzzword would appear to be “Big Data”. (Although the IoT uses Big Data, resulting in a re-entrant ranking problem that’s too much for my brain after two conferences this week.)

The question I’ve struggled with, however, is, “What is Big Data?” It’s almost as hard as, “What is the IoT?” One answer might be, “A vague concept that helps make your product sound more sophisticated and leading-edge, if you can pull it off.” But, while possibly true, that’s not particularly helpful.

There could be many nuanced aspects to Big Data, so I’m going to zoom way out and define – OK, maybe not define, but characterize – it via metaphor.

You see… you’ve had this problem, albeit well under control. You think no one has noticed, but we have… we just haven’t said anything. You have an… acquisitive nature. Yeah, we see that UPS truck show up. Again and again. (You were probably bummed that you can’t arrange after-dark delivery.) And over time you ran out of space in your home. So you had to get a storage space for much of your junk. (Yeah, I went there… I called it junk. Am I wrong?)

But, being a person of foresight, you asked the obvious question: If I put this into storage and never get it again, why have it in the first place? That’s a question you probably don’t want to answer honestly, in that it would result in a rather dramatic lifestyle reevaluation. So instead, you ran with the fantasy that you will, in fact, make frequent trips to your little attic-away-from-the-attic to get stuff. And you wanted to be able to do so without rummaging; takes too long and leaves a mess.

So you thought through what would go into the storage space. And you designed and had installed shelving specifically sized for the different things. And you labeled the shelves, numbering positions and levels, and every time you put something in there (which was pretty often, but manageable), you took careful note of where it went.

And anytime you wanted to get something (it did really happen occasionally), you could simply go to the logbook index and see where the item was and retrieve it with nary a bead of sweat raised.

Then came the Difficult Times. Your Mad Uncle Tito passed (tough in its own right), but upon being bestowed the honor of managing his affairs, you discovered that your propensity for Getting and Keeping was genetic. Only yours was diluted as compared to the Mad Uncle.

Not only did he acquire stuff; he acquired houses too. And each of the houses was packed to the gills with stuff. It looked like some of it had value; simply solving the problem with a bulldozer and front loader felt rash. So you rented another very large storage locker and went about trying to move stuff into there.

The problem with your system is that you have to hire professionals to build the shelving and arrange things just so. And it takes people and time to do all the cataloguing and moving. It worked for your own stuff, but for his stuff, well, it just seemed overwhelming.

And that wasn’t even the hard part. When you designed your own shelving, you knew, approximately, what kind of stuff was going to go there. Because it was your stuff. But you had no idea what might be found lurking in Tito’s many closets and under his bed and in his basements. It could be anything. And, for the same reason, you had no idea what you might want to get at in the future.

It’s the fundamental problem of storing Other People’s Stuff. (You down with OPS?)

It hurt you to the very core, but you had to make a strong decision. There was no way to do this in an organized fashion. The houses needed to be emptied and sold faster than you could neatly arrange the contents. So you simply hired some cheap labor to load trucks with stuff. Stuff loaded any old way. And in the storage locker, you simply put it in a pile.

Perhaps you made multiple piles – one for each house, or furniture over here and state plates over there, mixed with other stoneware and flatware and bad hotel art. So you might have created a patina (or illusion) of organization, but that’s it.

And you locked the door and called it good.

And when you wanted to actually find stuff, well, you hired folks that were good at finding stuff. So many people had so much stuff that this had become a new cottage industry, and different companies specialized in finding different kinds of things. Those guys over there were good at finding clothing; that other group was good at finding LPs. (No one had yet cracked the problem of finding remotes.) They joked that they could practically create a market out of the stuff they found, and, in fact, they referred to themselves generically as “marts.”

And that, to me, is the essence of Big Data, as compared to Ye Olde Relaytionale Databayse. It’s a big ol’ pile of Other People’s Stuff. Schema schmema. Perhaps with a few tags and flags here and there so that you can tell which house it came from or which stuff was more likely to be state plates. Other than that, you don’t mess with it, and you damn sure don’t throw anything away. And when you need something, you overlay with a datamart to extract any good bits.

Which, naturally, you use to improve your advertising targeting. Cuz we’re all just dying to receive more advertising – especially if it has our name on it. Makes us feel special.

Leave a Reply

featured blogs
Sep 24, 2018
One of the biggest events in the FPGA/SoC ecosystem is the annual Xilinx Developers Forum (XDF). XDF connects software developers and system designers to the deep expertise of Xilinx engineers, partners, and industry leaders. XDF takes place in three locations this year.  Sa...
Sep 24, 2018
For the second year, the Electronic Design Process Symposium (EDPS) took place in Milpitas, having been at Monterey for many years. This was apparently the 25th year EDPS has run. I find EDPS to be a fascinating conference, and I think it is a shame that more people don'...
Sep 21, 2018
  FPGA luminary David Laws has just published a well-researched blog on the Computer History Museum'€™s Web site titled '€œWho invented the Microprocessor?'€ If you'€™re wildly waving your raised hand right now, going '€œOoo, Ooo, Ooo, Call on me!'€ to get ...
Sep 20, 2018
Last week, NVIDIA announced the release of the Jetson Xavier developer kit. The Jetson Xavier, which was developed in OrCAD, is designed to help developers prototype with robots, drones, and other......