Big data has been in the spotlight for the past few years. This is because it is expected that by analyzing big data, it will be possible to do what was based on experience and intuition rationally. Let’s briefly explain the definition of big data and data mining, which is closely related to big data, and consider the impact of big data analysis on society. We’ll also mention our relationship with edge computing.
What is big data?
What is big data? In general, it means “a generic term for technology for accumulating, analyzing, and processing structured or unstructured data that occurs in large quantities and in real time, or the data itself.”
Structured data is data that was traditionally managed in a database, so-called organized data. Unstructured data, on the other hand, refers to raw data that has not yet been organized.
Before big data analysis, information from the field was analyzed by humans and made company decisions based on it. There, many years of experience and intuition of human beings were required. However, in recent years, the amount and type of data automatically collected from the field due to the development of information equipment has increased at an accelerated rate, and the burden on humans has increased, and the experience and intuition necessary for analysis are not acquired overnight. In other words, the number of people with sufficient experience and intuition is decreasing even though the data from the source of the analysis is increasing.
Therefore, the technology of data mining has developed to support data analysis and decision making. Data mining is a technology that helps to find correlations between data and discover useful relationships among large amounts of data, and is inseparable from big data analysis. In other words, it is a technology that supports making decisions by acting on behalf of processes such as human intuition and experience that have been used to derive analysis so far.
The concept of data mining to analyze big data
Let’s take a closer look at data mining to analyze big data. In general, big data analysis is often based on the following models (DIKW models):
“Data: Data”
This is what we call raw data that remains collected. It’s also unstructured data. It is usually the size to be petapite from terabytes.
“Information: Information”
This is an organized version of the raw data, and it is basically at this stage that data mining can help. It’s also structured data.
Knowledge: Knowledge
It refers to trends and knowledge obtained from information. In recent years, data mining tools have used artificial intelligence and statistical analysis to support the knowledge at this stage.
“Wisdom: Wisdom”
It refers to human judgment from knowledge.
It is important to note that data mining tools can generate information from data, but it is ultimately up to human judgement to generate knowledge and wisdom from information.
Current technology requires correlation between data, but it is necessary for humans to infer actual phenomena from the correlation results. It is a profession called a data scientist that does this, but it is a very difficult job.
The reason why the difficulty is high is that in addition to knowledge for analyzing data (statistical analysis, artificial intelligence, etc.), it is necessary to be familiar with current market trends and human behavior and psychology. In addition to such highly specialized knowledge, insights to infer phenomena from data correlations are also required. It can be said that it is a profession that must balance expertise with knowledge in a wide range of fields. In addition, such human resources cannot be developed overnight, and there is a shortage of human resources. It can be said that it is a problem in the current state of big data analysis.
On the other hand, analysis that relies solely on conventional human experience and intuition may overlook important knowledge and wisdom through the speculation of the person who performs the analysis. There are many cases where objective big data analysis by data mining can obtain knowledge that overturns conventional speculation and common sense. This is a major advantage of big data analysis.
Big Data Analysis Gives Society Impact
As mentioned so far, at first glance, the law can be found from unrelated data, so it is possible to reduce failures due to experience and intuition by big data analysis. Alternatively, we can objectively support human intuition and experience. As there is a famous case as “diapers and beer”, there are examples where common sense and speculation of sales measures have been overturned from the results of big data analysis. By extracting new knowledge through data mining, it is also effective in reviewing sales and marketing strategies.
In the logistics field, data is collected by attaching IC tags to products and cartons, and big data analysis is performed. By analyzing which logistics routes and how much products are concentrated, it is possible to optimize the logistics routes. In addition, optimizing logistics routes reduces costs and transportation time.
In the production sector, you can improve product quality by collecting product inspection data and analyzing trends. In addition, by installing sensors at key points in the production line and analyzing the collected takt time with big data, it is also possible to detect bottlenecks in the entire production line. Sensors can be installed in manufacturing equipment for constant monitoring, and data can be collected and analyzed to predict failures.
In this way, the biggest advantage is that new knowledge can be obtained through big data analysis to improve the efficiency and optimization of operations. It can be said that it is one powerful means which can correspond to the decrease in the working population and the work style reform that the problem is expected to become serious in the future.
In the logistics and production fields, large amounts of data are often generated in real time, and it is difficult to stop each line and collect data. Therefore, when big data analysis is performed in real time without stopping the line, it is considered effective to realize it with edge AI by edge computing.
Big data analysis makes society more meaningful
So far, we have briefly explained big data. There is a problem that there is a shortage of data scientists, but the need for big data analysis will increase more and more in the future. Accordingly, with the help of artificial intelligence and edge computing, big data analysis technology will continue to develop. And that will change society little by little. It is expected to be a meaningful change for human beings.
