Big data

- You see a lot of crossover between AI and big data. The term big data is used to describe a lot of different technologies, but if you go back to the original report, you'll see that the authors weren't thinking of big data as a term. They really used it more as an adjective. It's a way to describe a particular problem. In fact, the first time they used the term, they called it a big data problem. That means that the best way to read this is big data problem, although most people interpret it as big data problem. The important thing is to focus on the problem and not the data. Big data is basically saying that we're collecting more data than we can handle, that it's much easier now to create data than it is to store, analyze, and interpret it. The technology that we had to interpret the data is falling behind the technology we used to create it. You should think of big data as a driver for machine learning. At its core, big data is about managing and analyzing massive data sets. Remember that machine learning needs these massive data sets as a way to find patterns. That being said, it's really easy to get mixed up in the terms. Many database specialists use machine learning and data mining interchangeably. Data mining is a broad term in which you look to the data to find new insights. The big difference in machine learning and data mining is the technology used to find these insights. With machine learning, you typically start with a training set and then use one of the machine learning frameworks. There are a few in Python, R, and other development languages. Data mining typically uses a much broader set of tools. You'll also use a different approach for machine learning. Think of it this way. Data mining is about digging through your data in order to find valuable insights. Machine learning is about training your machine to find patterns. It's almost like a high-tech sorting machine for your data. The upside to mixing all these different terms is that it's not that big of a leap to transition from a big data project to a machine learning project. You'll typically need terabytes or even petabytes of stored data to have a well-functioning big data project. Once you have these massive data sets, it usually doesn't take long for the organization to extract some insights. They want to learn something about their customer or their industry from all this stored data. There's plenty of software out there that helps them do exactly that. So, these big data projects do a pretty good job of preparing your teams for machine learning. They're used to working with large data sets and they're familiar with downloading frameworks in Python to manipulate these sets. Even though the tools are different, they're usually similar enough so that these teams can get started. The big difference is that they just need to think about their data in a different way. Instead of directly mining the data for insights, you'll train the machine or neural network to find patterns. I worked for a few companies that have struggled for years with large big data projects. They thought that the leap to machine learning would be just as difficult as the leap to big data. The reality was is that it was a much smaller step. The teams already knew Python and R and they were very familiar working with large data sets. The big challenge is getting the teams to think differently about the data. They needed to create training sets and readjust the weights on their artificial neural network. This was different from the direct interaction with the data that they were using with these big data tools. If you're an organization that's working with big data, keep in mind that just because you have a new hammer, it doesn't mean that everything is a nail. Organizations that have big data tend to favor machine learning because they spend a lot of time creating this big data shiny new hammer. There are times when smaller AI projects work better with symbolic reasoning, so don't just assume that just because you have the data, machine learning is always the best place to start.