editor's blog
Subscribe Now

The Essence of Big Data

iStock_000064716741_Small.jpgThe biggest buzzword that every press release must reference in the title these days is “Internet of Things” (IoT). The second biggest buzzword would appear to be “Big Data”. (Although the IoT uses Big Data, resulting in a re-entrant ranking problem that’s too much for my brain after two conferences this week.)

The question I’ve struggled with, however, is, “What is Big Data?” It’s almost as hard as, “What is the IoT?” One answer might be, “A vague concept that helps make your product sound more sophisticated and leading-edge, if you can pull it off.” But, while possibly true, that’s not particularly helpful.

There could be many nuanced aspects to Big Data, so I’m going to zoom way out and define – OK, maybe not define, but characterize – it via metaphor.

You see… you’ve had this problem, albeit well under control. You think no one has noticed, but we have… we just haven’t said anything. You have an… acquisitive nature. Yeah, we see that UPS truck show up. Again and again. (You were probably bummed that you can’t arrange after-dark delivery.) And over time you ran out of space in your home. So you had to get a storage space for much of your junk. (Yeah, I went there… I called it junk. Am I wrong?)

But, being a person of foresight, you asked the obvious question: If I put this into storage and never get it again, why have it in the first place? That’s a question you probably don’t want to answer honestly, in that it would result in a rather dramatic lifestyle reevaluation. So instead, you ran with the fantasy that you will, in fact, make frequent trips to your little attic-away-from-the-attic to get stuff. And you wanted to be able to do so without rummaging; takes too long and leaves a mess.

So you thought through what would go into the storage space. And you designed and had installed shelving specifically sized for the different things. And you labeled the shelves, numbering positions and levels, and every time you put something in there (which was pretty often, but manageable), you took careful note of where it went.

And anytime you wanted to get something (it did really happen occasionally), you could simply go to the logbook index and see where the item was and retrieve it with nary a bead of sweat raised.

Then came the Difficult Times. Your Mad Uncle Tito passed (tough in its own right), but upon being bestowed the honor of managing his affairs, you discovered that your propensity for Getting and Keeping was genetic. Only yours was diluted as compared to the Mad Uncle.

Not only did he acquire stuff; he acquired houses too. And each of the houses was packed to the gills with stuff. It looked like some of it had value; simply solving the problem with a bulldozer and front loader felt rash. So you rented another very large storage locker and went about trying to move stuff into there.

The problem with your system is that you have to hire professionals to build the shelving and arrange things just so. And it takes people and time to do all the cataloguing and moving. It worked for your own stuff, but for his stuff, well, it just seemed overwhelming.

And that wasn’t even the hard part. When you designed your own shelving, you knew, approximately, what kind of stuff was going to go there. Because it was your stuff. But you had no idea what might be found lurking in Tito’s many closets and under his bed and in his basements. It could be anything. And, for the same reason, you had no idea what you might want to get at in the future.

It’s the fundamental problem of storing Other People’s Stuff. (You down with OPS?)

It hurt you to the very core, but you had to make a strong decision. There was no way to do this in an organized fashion. The houses needed to be emptied and sold faster than you could neatly arrange the contents. So you simply hired some cheap labor to load trucks with stuff. Stuff loaded any old way. And in the storage locker, you simply put it in a pile.

Perhaps you made multiple piles – one for each house, or furniture over here and state plates over there, mixed with other stoneware and flatware and bad hotel art. So you might have created a patina (or illusion) of organization, but that’s it.

And you locked the door and called it good.

And when you wanted to actually find stuff, well, you hired folks that were good at finding stuff. So many people had so much stuff that this had become a new cottage industry, and different companies specialized in finding different kinds of things. Those guys over there were good at finding clothing; that other group was good at finding LPs. (No one had yet cracked the problem of finding remotes.) They joked that they could practically create a market out of the stuff they found, and, in fact, they referred to themselves generically as “marts.”

And that, to me, is the essence of Big Data, as compared to Ye Olde Relaytionale Databayse. It’s a big ol’ pile of Other People’s Stuff. Schema schmema. Perhaps with a few tags and flags here and there so that you can tell which house it came from or which stuff was more likely to be state plates. Other than that, you don’t mess with it, and you damn sure don’t throw anything away. And when you need something, you overlay with a datamart to extract any good bits.

Which, naturally, you use to improve your advertising targeting. Cuz we’re all just dying to receive more advertising – especially if it has our name on it. Makes us feel special.

Leave a Reply

featured blogs
May 24, 2024
Could these creepy crawly robo-critters be the first step on a slippery road to a robot uprising coupled with an insect uprising?...
May 23, 2024
We're investing in semiconductor workforce development programs in Latin America, including government and academic partnerships to foster engineering talent.The post Building the Semiconductor Workforce in Latin America appeared first on Chip Design....

featured paper

Altera® FPGAs and SoCs with FPGA AI Suite and OpenVINO™ Toolkit Drive Embedded/Edge AI/Machine Learning Applications

Sponsored by Intel

Describes the emerging use cases of FPGA-based AI inference in edge and custom AI applications, and software and hardware solutions for edge FPGA AI.

Click here to read more

featured chalk talk

BMP585: Robust Barometric Pressure Sensor
In this episode of Chalk Talk, Amelia Dalton and Dr. Thomas Block from Bosch Sensortec investigate the benefits of barometric pressure sensors for a variety of electronic designs. They examine how the ultra-low power consumption, excellent accuracy and suitability for use in harsh environments can make Bosch’s BMP585 barometric pressure sensors a great fit for your next design.
Oct 2, 2023
29,604 views