editor's blog
Subscribe Now

The Essence of Big Data

iStock_000064716741_Small.jpgThe biggest buzzword that every press release must reference in the title these days is “Internet of Things” (IoT). The second biggest buzzword would appear to be “Big Data”. (Although the IoT uses Big Data, resulting in a re-entrant ranking problem that’s too much for my brain after two conferences this week.)

The question I’ve struggled with, however, is, “What is Big Data?” It’s almost as hard as, “What is the IoT?” One answer might be, “A vague concept that helps make your product sound more sophisticated and leading-edge, if you can pull it off.” But, while possibly true, that’s not particularly helpful.

There could be many nuanced aspects to Big Data, so I’m going to zoom way out and define – OK, maybe not define, but characterize – it via metaphor.

You see… you’ve had this problem, albeit well under control. You think no one has noticed, but we have… we just haven’t said anything. You have an… acquisitive nature. Yeah, we see that UPS truck show up. Again and again. (You were probably bummed that you can’t arrange after-dark delivery.) And over time you ran out of space in your home. So you had to get a storage space for much of your junk. (Yeah, I went there… I called it junk. Am I wrong?)

But, being a person of foresight, you asked the obvious question: If I put this into storage and never get it again, why have it in the first place? That’s a question you probably don’t want to answer honestly, in that it would result in a rather dramatic lifestyle reevaluation. So instead, you ran with the fantasy that you will, in fact, make frequent trips to your little attic-away-from-the-attic to get stuff. And you wanted to be able to do so without rummaging; takes too long and leaves a mess.

So you thought through what would go into the storage space. And you designed and had installed shelving specifically sized for the different things. And you labeled the shelves, numbering positions and levels, and every time you put something in there (which was pretty often, but manageable), you took careful note of where it went.

And anytime you wanted to get something (it did really happen occasionally), you could simply go to the logbook index and see where the item was and retrieve it with nary a bead of sweat raised.

Then came the Difficult Times. Your Mad Uncle Tito passed (tough in its own right), but upon being bestowed the honor of managing his affairs, you discovered that your propensity for Getting and Keeping was genetic. Only yours was diluted as compared to the Mad Uncle.

Not only did he acquire stuff; he acquired houses too. And each of the houses was packed to the gills with stuff. It looked like some of it had value; simply solving the problem with a bulldozer and front loader felt rash. So you rented another very large storage locker and went about trying to move stuff into there.

The problem with your system is that you have to hire professionals to build the shelving and arrange things just so. And it takes people and time to do all the cataloguing and moving. It worked for your own stuff, but for his stuff, well, it just seemed overwhelming.

And that wasn’t even the hard part. When you designed your own shelving, you knew, approximately, what kind of stuff was going to go there. Because it was your stuff. But you had no idea what might be found lurking in Tito’s many closets and under his bed and in his basements. It could be anything. And, for the same reason, you had no idea what you might want to get at in the future.

It’s the fundamental problem of storing Other People’s Stuff. (You down with OPS?)

It hurt you to the very core, but you had to make a strong decision. There was no way to do this in an organized fashion. The houses needed to be emptied and sold faster than you could neatly arrange the contents. So you simply hired some cheap labor to load trucks with stuff. Stuff loaded any old way. And in the storage locker, you simply put it in a pile.

Perhaps you made multiple piles – one for each house, or furniture over here and state plates over there, mixed with other stoneware and flatware and bad hotel art. So you might have created a patina (or illusion) of organization, but that’s it.

And you locked the door and called it good.

And when you wanted to actually find stuff, well, you hired folks that were good at finding stuff. So many people had so much stuff that this had become a new cottage industry, and different companies specialized in finding different kinds of things. Those guys over there were good at finding clothing; that other group was good at finding LPs. (No one had yet cracked the problem of finding remotes.) They joked that they could practically create a market out of the stuff they found, and, in fact, they referred to themselves generically as “marts.”

And that, to me, is the essence of Big Data, as compared to Ye Olde Relaytionale Databayse. It’s a big ol’ pile of Other People’s Stuff. Schema schmema. Perhaps with a few tags and flags here and there so that you can tell which house it came from or which stuff was more likely to be state plates. Other than that, you don’t mess with it, and you damn sure don’t throw anything away. And when you need something, you overlay with a datamart to extract any good bits.

Which, naturally, you use to improve your advertising targeting. Cuz we’re all just dying to receive more advertising – especially if it has our name on it. Makes us feel special.

Leave a Reply

featured blogs
Mar 28, 2024
'Move fast and break things,' a motto coined by Mark Zuckerberg, captures the ethos of Silicon Valley where creative disruption remakes the world through the invention of new technologies. From social media to autonomous cars, to generative AI, the disruptions have reverberat...
Mar 26, 2024
Learn how GPU acceleration impacts digital chip design implementation, expanding beyond chip simulation to fulfill compute demands of the RTL-to-GDSII process.The post Can GPUs Accelerate Digital Design Implementation? appeared first on Chip Design....
Mar 21, 2024
The awesome thing about these machines is that you are limited only by your imagination, and I've got a GREAT imagination....

featured video

We are Altera. We are for the innovators.

Sponsored by Intel

Today we embark on an exciting journey as we transition to Altera, an Intel Company. In a world of endless opportunities and challenges, we are here to provide the flexibility needed by our ecosystem of customers and partners to pioneer and accelerate innovation. As we leap into the future, we are committed to providing easy-to-design and deploy leadership programmable solutions to innovators to unlock extraordinary possibilities for everyone on the planet.

To learn more about Altera visit: http://intel.com/altera

featured chalk talk

Autonomous Mobile Robots
Sponsored by Mouser Electronics and onsemi
Robotic applications are now commonplace in a variety of segments in society and are growing in number each day. In this episode of Chalk Talk, Amelia Dalton and Alessandro Maggioni from onsemi discuss the details, functions, and benefits of autonomous mobile robots. They also examine the performance parameters of these kinds of robotic designs, the five main subsystems included in autonomous mobile robots, and how onsemi is furthering innovation in this arena.
Jan 24, 2024
9,479 views