(04) 數據的分析與應用.zh-CN.en

Translated from Chinese (Simplified) to English - www.onlinedoctranslator.
com
Data analysis and application
DAeardaptmvn
ceetnf l eecftrirger
doER ic aalti
onCoanntdrol Ai
Lr aCbo.nDiti ia l eEin
ycEtrnigcne
dpenoairntgmaenndtEonfeErgle rng,g
inNeaetrioinnaglNChaitnioYniaUl nCivheursnitgy-oHfsTinech
gnUonloivgeyrsity
Course outline
- data analysis
- Data processing
-ANALYSE information
-Data value
- value loop
- application of data
-market trend
-Industrial application
- Ecosystem development
Ddepartment of Refrigeration and Air Conditioning and Energy Eengineering, National Chin Yi University of Ttechnology 2/76
data processing
- Most of the stored data that have been pre-processed and digitized still cannot be
directly applied, and the corresponding analysis methods must be processed to
effectively capture and convert the data, so that people or applications can more
easily to interpret and process the information, and then analyze and apply it
- Before processing,You must first identify what the problem is, and analyze the data required
to solve the problem and the available data, and then further sort out and adjust the data,
select the corresponding features or perform corresponding calculations according to the
problem object to obtain the required information
- From the information obtained by processing the data, the corresponding analysis actions are carried out according to the
subject of the question, and then it is confirmed whether the hypothesis set for the question needs to be revised or the
answer to the question can be obtained.
data processing
- Data processing can be classified into different processing technologies according to the amount,
diversity (structured or unstructured), processing speed (batch/interactive/real-time), etc.
-batch × large amount of structured data
- Use traditional reporting tools or batch processing jobs
-interactive × large amount of structured data
- Linked database queries for interactive analytics (data warehousing,BItools, online analysis queries)
- Using parallel relational database, memory processing engine/database and other technologies
-Batch ∕ Interactive × unstructured data processing
- Processing of complex unstructured data such as distributed archives, graphic images, and file libraries
- Using technologies such as decentralized file systems, unstructured processing engines and database storage
-immediate × Large amount of unstructured data
- Time-sensitive occasions that require rapid analysis and message delivery or feedback control
(traffic control, boiler control, medical diagnosis, video streaming)
- Use complex event processing, memory data processing, time series database, river computing
(Streams Computing)etc. technology
data processing
- Data processing before data analysis also includes how to import data into analysis
tools for processing
- Data is usually stored in a common access format, such as based on structured query
language (SQL)database, comma-separated values (CSV)files, plain text files, etc.
- For data such as pictures or audio and video, it is necessary to confirm the analysis actions to be performed, and perform analysis operations and obtain
results through corresponding feature extraction.
-Using a standardized general access format, the stored data can be arranged, organized and expanded more
quickly, which is also conducive to exchange and sharing, and the data can be directly imported into
different platforms for application or data integration
- Through the fields of the data form (index feature), heterogeneous data can be aggregated, and it is also
an important basis (index) when introducing analysis tools
-Structured data will be more conducive to batch processing, which can be more convenient and faster to import
data, or generate data required for analysis through calculations such as overlay
ANALYSE information
- After the data is captured and transformed, further processing and analysis are
required to obtain greater value from it
- Data analysis is the most importantunderstanding of the problem, that is, how to correctly find the entry
point of the analysis and avoid falling into a misunderstanding
-Analysis ideas/thinking, analysis process, interpretation of results, and analysis tools used are also
important processes/implementation means
-For ideas/thinking parts, you can use mind maps (mind maps (Mind Map)) to assist in sorting, classify the
problems to be analyzed according to different directions, and then continue to split and refine them, think
about the problems in an all-round way and analyze them
-In the process of analysis, quantitative analysis can usually be carried out based on different quantitative data, and various
mathematical tools or analysis methods can be used to verify arguments or confirm through indicators
- Interpretation of results usually varies from person to person, and in order to better analyze and
express, as well as interpret, communicate or discuss with others (shared with others), data
visualization is a very important means (tool )
type of problem
- Restoration problem:now The purpose is toUndisturbed
- There is a gap between the current state and the past, that is, the bad state may have been revealed, so it is necessary to
find out the problem from the gap and restore the current state to the previous level (the original state)
- It is an occurrence type problem, and the problem point can be clearly seen after it occurs
-Coping methods: find out causality, difference analysis, benchmarking…
- Pursuing ideal type questions:future The purpose is toideal
- The current situation is not up to expectations, so it is necessary to set clear goals and try to achieve them
- It is an exploratory problem, a potential problem that needs to be discovered through analysis
- Coping method: After selecting an ideal, plan an action plan
ideal
- Restoration problem:future The purpose is toUndisturbed Maintain the status pursuit of ideals
Purpose
-Problems that may show bad state in the future occur
Anticipate ahead and develop preventive strategies
Undisturbed
potential crisis
Restore
- It is an exploratory problem prevention
-Coping: try to identify triggers and make plans

now future
defense strategy
Time of occurrence
Six Steps to Problem Solving
1. Identify the problem, identify the problem, and define the problem (narrative problem)
2. Collect data, analyze data, and identify reasons
3. Propose and select solutions (hypothetical)
4. Evaluate and implement the best option
5. Verify facts and evaluate results
6. Identify process variations and negative impacts
1. Identify the problem, identify the problem, and define the problem (narrative problem) 5W1H
who (Who)
- Identify the problem to be solved (the goal to be achieved) what (what)
when (When)
- Analyzing the problem appearance:
where (Where)
What is known? What is unknown? What is required?
how (How)
- by5W1Hto express
mind Mapping
- Mind mapping is a method of organizing a series of thinking thoughts using radiating

branching
-Based on a central idea, expand branches and refine them in many different directions (step by step to
establish keywords and their associations), capture ideas from any direction, and assist in
communication and planning
-It is an effective method to improve understanding and memory through the visual representation of information, and it can also
provide analysis and understanding assistance through the colorization of different thinking nodes (thinking direction/
association level).
-Digital aids evolved from the traditional concept of intelligence analysis boards consisting of paper strips,
pushpins and association lines
- Focus on recording the different ideas (keywords) that emerge at the beginning, and then analyze the
relevance, importance and priority between keywords and themes, as well as keywords, and provide follow-
up discussions
https://online.visual-paradigm.com/
- Identify controllable factors, uncontrollable factors, and possible constraints to select
- analysis methods
- Analyze the results after data processing
- Interpret the results of the analysis and confirm the constraints in the problem (preconditions) Make
- assumptions and confirm whether the constraints are met (feasible)
- Confirm solution scope and make adjustments
- Confirm and revise assumptions and solutions (schemes) through inquiry and doubt, and establish
- scheme evaluation methods (check items). or Judgment Criteria or Evaluation function) Validate candidate
- solutions (execution based on merit)

virtual reality VR (Virtual Reality)
Augmented reality AR (Augmented
Reality) mixed reality MR (Mixed Reality)
- Set up a virtual verification environment to verify points in detail
- Actual input verification (number of test results obtained
- Observation, continuous operation and correction
5. Verify facts and evaluate results while confirming problem solving processes and answers
- Confirm system stability and variability and identify possible negative effects
- Determine if the negative impact is causing persistent adverse consequences (sequelae)
- Analyze the causes and propose corrections, and put into system verification feedback
ANALYSE information
- After clarifying the thinking and grasping the goal of the essential analysis of the problem, the next step is
how to narrow the scope of the analysis and the means and tools of the analysis.
- Before analyzing, determine which information is useless to solve the core problem, and filter the content that needs
to be analyzed and processed
-According to the different forms of data presentation and analysis goals, various analysis methods and tools
from pure manual to intelligent automation begin to flourish and apply
- Different types of analytics maturity (stages of development) can be distinguished depending on the type of analysis,
the depth of analysis and the sophistication of the tools used
ANALYSE information
- Different levels (depths) of analysis methods to meet different purposes
- Directly perform report analysis manually (Mastering the past and present)
- Traditional data report presentation, summarizing the status of analysis objects
- Integrate multi-party data to assist in analysis and diagnosis (Discover behavior patterns)
- In order to further analyze why this state occurs, try to find out the reason
- It is usually necessary to use other additional information (data) to assist in analysis and judgment
- Model estimation and forecast tracking after process flow (predict)

- Find out the occurrence factors of a state through the information provided by the data features
- Continuously monitor the status of each occurrence element, predict whether it occurs and establish a model for estimation, or
confirm the status of each element after occurrence to adjust and correct the estimation model
- An automatic analysis and prediction system that realizes real-time adjustment and correction with artificial intelligence (optimization)
- Add the calculation parameters for adjusting the system characteristics (prediction direction) and the logic and rules for parameter
updating into the estimation model, and design the evaluation function to confirm that the updating direction is correct
- Usually has a certain influence ability, which can interfere with the system state to correct and change the direction, and continue
to keep the system operating in the target direction
data value
- If the data is not properly processed, extracted and analyzed, its value has not yet been developed
and determined, and needs to be discovered.
- Data is processed, analyzed and developed to become a product, and a product may be an analysis report,
a recommendation for a specific decision. A good machine learning model, or an improved decision-
making process, etc.
- The value that the data can generate depends on whether the people who use it have the ability tocorrect
information by the right way used forright scene
-The more common value discussion directions mainly include
- Enterprise value: how to manage and innovate through data and data analysis to increase revenue
- The Economic Value of Information: How to Market Data, Information and Knowledge as Products and Services
- The value of intelligent decision-making: how to enhance the value of decision-making or action with the aid of data and data analysis
Data Value: Enterprise Value
- Big data analysis techniques or methods can not only help manage data problems, but also
actively create profits and develop new business models

Business Monitoring
-Operation monitoring: monitor the current operation status and analyze whether there is any abnormality
- Monitor equipment operation status
Business Insights
-Operational Intelligence: Improve operations using statistics, data mining, predictive analytics, etc.
- Analysis of equipment parts life and possible failure

Business Optimization
-Operational optimization: use forecasting and optimization analysis to improve the operating model
- Analysis of optimal maintenance time and sequence in combination with production or operation schedule (shifts)
Data Monetization
- Generate revenue: Use data to assist product sales or market analysis to gain new sources of profit
- Combining equipment and data, it provides setting adjustment and optimization for different application fields, and provides services at
the same time; or provides monitoring and analysis systems to users independently as commodities
Business Metamorphosis
-Develop new models: create new business models or services to assist in business transformation
Data Value: The Economic Value of Information
- Different from the principle of economic value of traditional materials, equipment or products, information,
knowledge or services such as "information economy (Infonomic)"The main value principles are:
-Quality: whether the information is complete or accurate
- Pay attention to the quality of data acquisition and the aggregation and linking of various data sources
- Relevance: whether the information is easy to interpret or meaningful
- Pay attention to whether the information meets the decision-making needs
-Timeliness: whether the information has decision-making significance in a certain time and space
- Pay attention to the context of the data and the speed of processing and analysis
- Cost: whether the information is easy to obtain or manage the application
- Pay attention to the cost of data sources and technologies used for processing (open source data, cluster processing, etc.)
-Effectiveness: whether the cost-effectiveness of the information meets the needs
- Pay attention to the ratio between the cost of data processing technology and the benefits it brings
- Marketability: whether information can be turned into a service or product to provide revenue benefits
- Emphasis on product service value and business model construction of data and information
- Data, information or knowledge become part of the product, the market characteristics of this emerging
information economy will include
-Increasing marginal rate of return (Increasing marginal revenue)

- In products that rely on information or knowledge, the restrictions and remanufacturing costs for the production of traditional products
such as space, raw materials, and production capacity are relatively low, and the production and remanufacturing speed is relatively
high. Therefore, as the investment in technology and knowledge increases, more, the marginal rate of return of the producer's input
unit will continue to increase
- network effects (Network effects)∕Network externalities (Network externality)

- For products and services in the knowledge economy, the increase in the number of users will not only affect direct revenue,
but also the value of the product or service to users will also increase
- Social software needs to have enough users (users) to continuously attract other potential users to join
through the joint interaction between users
- Accumulating users can also bring some unquantifiable data income, such as user behavior, query /
shopping / usage records, etc.
- Therefore, the Internet service industry often sacrifices short-term costs in the early stage of development in order to seize market share,
in order to accumulate enough users and data to generate future revenue
- Two-sided market (Two-sided market)
- Enterprises develop bilateral networks between different users or business partners by establishing economic platforms, so that
both can benefit from the expansion of the network scale
- Take the example of a credit card company building a two-sided network between merchants and consumers
- Consumers can reduce shopping costs through the credit card company's own bonus rewards or the consumer discounts provided by the
credit card company and the merchant's agreement
- Merchants can stimulate consumption and attract customers through promotions and promotions jointly launched with credit card companies.
They can also use the systems provided by credit card companies to help manage their income, and they can analyze how to adjust the products
and services they provide through data such as consumption records.
- Credit card companies themselves can generate revenue by providing services and platforms, and can also use user
behavior, consumption records and other data on the platform as a basis for analysis to optimize operational efficiency
or expand new businesses, build corporate brands and generate revenue
Data Value: Smart Decision Value
- Through the accumulation and analysis of a large amount of data/data, it can assist people to analyze the law
of a large number of events in the past, thereby making future predictions, and further assisting people in
decision-making, judgment and action.
-There are many elements in the process of people making various decisions and even actions. They mainly obtain
intelligence information through their senses, and analyze and measure based on past experience, understand the
situation, and further measure and judge, so as to generate actions.
-However, human beings lack the ability to grasp a large amount of data, and they are prone to different or hesitant
decisions due to the influence of various environmental factors or psychological factors, and the uncertainty is too high
- Therefore, I hope to use big data analysis to assist in efficient and objective decision-making and judgment, or
automatically make smaller trivial decisions, and assist people in making the final general direction or decision-making
judgment of exceptional events, which can effectively improve people's work. Efficiency, which is equivalent to
increasing the value
Data Value: Smart Decision Value
using big dataAIAided decision-making framework
measure judgment
(Judgment)
data input Analysis forecast action result

(Input) (Prediction) (Action) (Outcome)
training to learn give back
(Training) (Feedback)
- With big data and artificial intelligence (AI) Assisted decision-making or even automated decision-making can reduce the
uncertainty of human monitoring and allow people to focus on special or exceptional events, thereby increasing the
value of the decision or action
- Special or exceptional events may also include small amounts of data on important objects, new issues that cannot be referenced from past
experience, and decisions involving social and moral issues, etc.
value cycle
- The productivity improvement of an enterprise or organization depends on whether it can obtain appropriate
data and have data processing and analysis capabilities to further convert the data into products or
services with commercial value. Generally, the following steps are carried out.
- Digitization and data collection (Digitization and Data Collection)

- Obtain digitized data of various activities through observation, sensing, monitoring, etc.
- Big Data Data Pool (Big Data Pool)

- Put collected data into physical or logical data storage for processing or analysis
-Data processing and analysis (Data Processing and Analyzing)

- Extract, transform and analyze the data in the big data data pool to generate value using various technologies
and tools (connection, contextual correlation and meaningful analysis from data)
- Knowledge Base (Knowledge Base)

- Accumulate analyzed data or information in human brains, information systems or other entities for subsequent learning,
analysis, and decision-making
-Data-driven decision-making (Data Driven Decision Making)

- Based on the aforementioned knowledge and insights, assist people to make decisions and trigger actions to generate final value
price
The evolution of business analytics (data-driven decision-making) technology
- Using data to make decisions is currently the most common application of big data analysis, and decision-
making itself is closely related to business analysis, which has developed along with business and
gradually became physicochemical.
- Since the beginning of taking business intelligence as a competitive advantage with analytics, to gain
higher operational efficiency and create value by making better decisions at critical moments, data
analytics has gradually developed along with business.
-Business analysis is a knowledge that can go beyond intuition and gain an in-depth understanding of business phenomena from a
more objective perspective, and grasp market trends by recording and analyzing production processes, marketing processes
and customer interaction data.
-Data Warehousing (Data Warehouse)and business acumen (Business Intelligence)Other technologies have also begun to develop, but they
are always limited by computing and analysis technologies, and the work is time-consuming and time-consuming.
-The evolution of computer (computer) technology has not only made up for the limitation of insufficient computing power, but the
digitization of data has also made it easier for analysis and processing operations to be carried out in batches
- With the changes of the times, the analysis and decision-making in order to achieve the purpose changes with
the methodology and tools, with the development of technologies such as network, digitization, device
networking and cloud data storage and the focus of analysis technology, the development of For today's
big data analysis technology around IoT technology and artificial intelligence
-Accurate analysis of data within the enterprise has begun to be unable to cope with the ever-changing market. In
order to quickly grasp information from the increasingly large accumulated data, various technologies for direct
analysis of large amounts of data have gradually begun to flourish.
-Data sources have also been expanded to include Internet data, various sensor information, audio and video records,
user/consumer behavior, and diverse living data, which makes the data to be processed and analyzed even larger
and more complex. Diverse, it is difficult to handle such a huge calculation by manual work alone, and it must rely
on computers and analysis tools to assist in a large number of analysis and processing
- Business analysis is also carried out from mastering internal survey dataPrecise analysisandModel building,
transformed into aIdentify trends quicklybyMake predictions and assist in decision making, has
officially entered the era of big data analysis
Business analysis (data-driven decision-making) technology in different eras and technological development backgrounds
time technical name Technical Features
Decision support using data analytics

1970s decision support
(Decision Support)
Use data analysis for decision support and expand the application to high-level
1980s Management support
(Executive Support) Auxiliary support for supervisory and administrative decision-making
Based on multi-dimensional analysis and distributed data tables (multi-dimensional data

online analysis
1990s (Online Analytical Process, table), flexible pivot analysis (Pivot)and related operations to present
OLAP) methods and tools for integrated decision-making information
Build support data-driven with reports and related visual data

1980 – 2000 business intelligence
Tools for decision-making, with a recent focus on data dashboards
(Business Intelligence)
Integrated application of technology and model analysis and forecasting
data analysis Focus on statistics and mathematics for decision analysis, hoping to
2000ssecond half
(Data Analytics) Detailed and complex calculation results come to a conclusion
Big data analysis Focusing on the rapid processing of large amounts of unstructured data, it is hoped that
2010s
(Big Data Analysis) Finding common patterns from large amounts of data to draw conclusions
- With the development of big data analysis technology, more emphasis will be placed on how to strengthen
products through data in the future. It is no longer only information and network-related industries that
can benefit from data analysis, but the digital data from all walks of life brought by data. Transformation
can create goods and services from which to bring value
- The economic value of data brought by the information economy and the final stage of analysis based
on descriptive and predictive analysis: the use of normative analysis will become mainstream
-Analysis of business problems, from the description of "what happened/why happened", to the prediction of
"what will happen", to the guidance specification of "how to make it happen", gradually from understanding
to insight, providing based on prediction suggestion
-In addition to business management, related applications are more in the fields of logistics and transportation and industrial production
management in cooperation with IoT technology, and in cooperation with production equipment for remote production monitoring,
equipment failure analysis and maintenance scheduling, etc.
descriptive (Descriptive) predictive (Predictive) normative (Prescriptive)
what happened? what will happen? what should happen?

what has happened? what could happen? what should happen?
• Improve asset reliability • Predict infrastructure failures • Improve asset utilization
what to do (asset reliability) (Predict infrastructure failures) • Resource utilization scheduling optimization
What to do • Improve labor and inventory costs • Facility space demand forecast
(labor and Inventory Costs) (Forecast facilities space demands)
• Type and number of asset failures • How to Anticipate the Failure of Assets • How to increase the productivity of your assets
what to know • Maintenance costs and details • When to Consolidate Idle Plant Facilities • Technician service path optimization
What to know
• The value of the inventory item • How to Determine Costs by Service Tier • Planning for optimum productivity
• Standard report (Standard Reporting) • prediction model (Predictive Modeling) • optimization (Optimization)
(What happened?) • predict (Forecasting) =>What is the best result?
How to know the answer
• In-depth discussion (Drill down • simulation (Simulation) • Random Variable Optimization
How analytics gets
•
query) (Where exactly is the problem?) (Random Variable Optimization)
answers warning (Alert)
• interim report (Ad Hoc Reporting) =>Best results for specific field bands
(How many / How often / Where) What kind of variability comes?
• business intelligence(Business Intelligence) • prediction model (Predictive Modeling) • business rules (Business Rules)
how to make it happen • Alerts, Reports, Dashboards • predict (Forecasting) • organizational model (Organization models)
what make this
analytics possible
(Alert, Report, Dashboard) • Statistical Analysis (Statistical Analysis) • compare (Comparison)
• Scoring criteria (Scoring) • optimization (Optimization)
- Through data-based business analysis, the optimization of the overall operation will vary with
the change of analysis objects and processing speed requirements
-Specifically, the analysis objects will be divided into precise analysis of individual users, and group analysis of
broad audiences that identify trends through statistical analysis
- And the processing speed varies with the data holding method and update speed, from batch processing that waits
for data input to perform analysis and processing, to real-time processing that updates the data stream at any
time and requires dynamic analysis
-The optimization of the system is also because it is necessary to master the latest data by comparing with the accumulated data in
the past and continuously updating the data, so as to obtain the most real-time information for analysis and decision-making,
so it needs to be used with fast and huge data sources and storage space for application
- Also because the current situation is constantly changing, new variables (controlling factors) that cause the current situation
to change (unsustainable) may continue to appear during the analysis process. Therefore, the correlation analysis and
priority judgment of the factors in the problem analysis are also required. through different ways
- The more common way is to use the characteristic factor analysis diagram (fishbone diagram) to control the factors
individual optimization × Batch processing => accurate discrimination individual optimization × Real-time information => dynamic discrimination
• Information about a specific person or thing • Information about a specific person or thing
extensive collectionand analysis extensive collectionand analysis
• againstPrecise recommendation for specific objectsThe most suitable • againstPrecise recommendation for specific objectsThe most suitable
information/takemost appropriatethe way of handling information/dynamictake the most appropriate treatment
• one-to-one marketing • Behaviorally Targeted Advertising
• Customer Churn Analysis • Send coupons in line with its behavioral characteristics
analysis object
• Predict equipment failure

(optimization)
• Instant product recommendation
Overall optimization × Batch processing => accurate prediction Overall optimization × Real-time information => dynamic forecast
• Collect information on most people or things, and carry out • Collect information about most people or things,
Statistical processingand analyse make statisticsProcessing and Analysis
• provided foroverall usefulinformation/takenmost • provided foroverall usefulinformation/dynamictake the
appropriateprocessing method (no time limit) most appropriate treatment

• Improve the accuracy of search engines or translation software • Predicting road conditions using a dash cam
• bytwitterof tweets predict stock price movements • Predict electricity demand with smart meters
• Improve the usability of the website • Predict if a plane might be delayed
Data processing speed
Feature Factor Analysis Chart (Cause & Effect Analysis)
- Also known as cause and effect diagram, fishbone diagram (Fishbone
Diagram) or Ishikawa diagram (Iskiawa Diagram)
- During the analysis, the main causes of the problem are listed, and the secondary causes are listed
one by one and classified (the classification is attached to the main cause), so that the analysis of
the problem can be effectively controlled.
- If we take the modern industrial production line process as an example, we will generally analyze the six major factors, including
personnel, construction methods, materials, machines (equipment), measurement, and environment.
-It can also carry out countermeasure analysis in which the solution is the main branch, and the implementation details of the plan are
branched to converge to the goal.
(Measure (Material personnel) Cause type Countermeasure
question Target
(surroundings (machine (Work) Law
- After understanding the problem and mastering the reasons for the decision analysis system and the
corresponding changes, the most commonly used content is mainly to analyze and predict the
problem, or to analyze and eliminate exceptions in the production process and optimize the
process. (optimization)
-By tracking the various factors of the production process, through big data analysis, confirm whether there may be
abnormal changes and estimate the magnitude of its impact, and try to find the main factors of the abnormal
changes and adjust them, so as to eliminate the problem as soon as possible , maintain a fixed quality
-Also by tracking the factors of the production process, you can analyze and confirm the impact of each factor on the
production capacity and try to optimize the parameterization settings of the factors, thereby optimizing the production
line process and production schedule (including individual equipment. optimization of production parameters,
processing schedules and maintenance time planning)
-Optimized statistical quality control based on abnormal cause analysis (SQC)is the most common
application in management
Statistical Quality Control (SQC)
- Provide the basis for quality improvement through statistics or analysis charts
-statistics (Statistics): Characterize and quantify qualitative or quantitative items, thereby presenting the overall truth
-Quality (Quality): Analysis of the causal relationship between factors of production and product quality
-control (Control): Optimization of items such as (design + control + improvement + technology)
(Measure (Material ( Original

incidentpeople pregnancymember)because
A There are characteristics to bedue to points Analysis (fishbone diagram))middle not in a fishbone diagram
B nowfield responsibility scope Other departmental responsibilities System negligence
C written in the work standards

question
Not written in the work standards
D Prevention methodwrite clear write no bright
E The worker understands the job content not fully understood human negligence
F have actually implemented not actually implemented
(machine
G The result is normal(surroundings
(Work) Law
market trend
- The demand for big data in the market environment mainly includes technical products such as data services,
infrastructure architecture, application software tools, consulting and integration services
-At present, the industry related to big data is booming, so the infrastructure structure accounts for the majority, while the
consulting and integration services to assist enterprises to introduce related technologies and tools only follow.2020
The overall market size of the year can be roughly estimated at up to400more than $100 million
-Global information technology companies have also invested in related fields. Currently, they mainly focus on structured/unstructured
data processing, storage and hybrid processing, real-time processing and other processing directions, artificial intelligence cognitive
computing services, human-machine dialogue/natural interaction systems and natural language processing (NLP)The combination
of application technology and big data analysis is the main
market trend
- infrastructure
-The related products of the infrastructure architecture of big data mainly provide platform facilities such as data
processing, storage and computing
-Most server providers use a distributed or clustered processing architecture composed of a large number of small and medium-
sized servers for collaborative data processing
-In the current data application, the proportion of unstructured data is getting higher and higher, so the
unstructured processing architecture (such as open sourceHadoop) with a decentralized file system to
speed up data access and processing
-In addition to providing hard disk arrays, data servers or cloud storage for storage, storage equipment suppliers
pay more attention to how to use data more efficiently through application software assistance
- Such as: data management software and interface, data protection and recovery software, data copying, moving and
archiving software, server (storage device) health management software, etc.
market trend
- application software tools
-At present, various electronic information related industries have begun to pay attention to software talents again, and the
related industries of big data analysis are no exception. In addition to the basic application of various application software
and analysis tools, the data model established for the analysis results also needs to be programmed by software before it can
be deployed. The program writing structure will even affect the system operation performance
- Big data software products are mainly used to assist enterprises or application providers in data
collection, sorting, storage, and analysis. There are three main types:
- Data organization and management software: used for data collection, organization and storage
- Such as: data warehousing, database, unstructured data processing, content management, data integration and transformation
- Data analysis and visualization software: use data mining models, statistical analysis, and visualization techniques
techniques to spot patterns, predict the future, and graph
- Other big data application software: application processing, analysis and presentation for specific fields or industries
Data and Analysis Results
-Many manufacturers will build software products on cloud services, pay by subscription or pay by usage, and use
analytics as a service to reduce the cost of purchasing software and hardware for customers
market trend
Data organization and management software
Linked Data Management non-connected data management
(RMDBS, data warehousing...) (NoSQL, file management, graph management...)
data integration software

real-time event processing software
(ETL, data quality, data governance...)
Data analysis and visualization software big data application software
Business Intelligence Analysis Software

financial analysis planning customer demand forecast
(OLAP, Reporting...)
Sales forecast planning Sales Price Analysis
Advanced forecasting and analysis software
(Statistics, Data mining...)

Logistics optimization analysis manpower optimization analysis
visualization software Predictive Maintenance Analysis Web click analysis

(Dashboard, Reporting...)
Product word-of-mouth analysis Brand Loyalty Analysis
special analysis software
(Spatial Analysis, Text Analysis...) Advertising headline benefit analysis …
market trend
- System Integration and Consulting Services
-In addition to consulting services and assistance for application and environment construction, big data consulting and integration
services also include business consulting, information security consulting, system planning and integration services, analysis and
evaluation services, and data security services, etc.
-At present, the consulting services for system planning, network security and information security are more popular
-The application development of some relatively novel application services in specific fields will also be used by system
integration and consulting service providers as a kanban to promote their own products and attract customers'
attention, such as financial analysis, human resources analysis, natural language, voice Assistant, image recognition,
mobile cloud, cognitive computing combined with artificial intelligence, etc.
market trend
Information software and services
• Strengthen existing products (BusinessObject BIplatform) ability to handle large amounts of heterogeneous data
• based onHANAmemory computing platformLeondradoThe platform provides analysis services

SAP
• Strengthen data analysis capabilities and system integration on existing business software
• Develop IoT and other processing, predictive and analytical services for different applications
• strengthen existingSASStatistical analysis, data mining model capabilities, combined with memory computing,Hadoopprocessing technology
SAS • In-depth development of industry-specific and field-specific applications
• Development of Visual Analytics (Visual Analytics)software
• combineHadoopPlatform and real-time event processing technology, buildAzureCloud service, strengthen big data processing service
• Provide big data analysis tool services and analysis service examples in various industries
Microsoft • strengthen existingMicrosoft SQL ServerThe ability to process massive and heterogeneous data on relational databases, combined with
HadoopUnstructured data processing technology
• Develop cognitive services such as facial recognition, voice assistant, language translation, etc.
• strengthen existingOracle DBThe ability to process massive and heterogeneous data on relational databases, combined withHadoopUnstructured data
processing technology
Oracle • Development of database and server integration applications (Appliance)
• Develop real-time event data analysis and processing engine
• Develop big data application services such as finance, human resources and supply chain
market trend
cloud computing services
• Provide basic storage and data pipeline queuing services (Data Pipeline)
• supplyAmazon Redshifstructured data warehousing,Kinesisinstant event handling,Hadoop Elastic MapReduce (EMR)

unstructured data processing,DynamoDBInformation management services such as unstructured databases
Amazon
• supplyAmazon SageMakerMachine Learning Analysis Services
• supplyAmazon Lexnatural language andAmazon Rekognition,DeepLlensImage recognition service
• developingAmazon Echovoice assistant
• supplyCloud Storage, Cloud Datastore, Cloud BigTableEqual structured and unstructured storage services
• Provided for running calculationsGoogle Compute Engine
• Provides a software architecture for parallel computing of large-scale data setsMapReduce (HadoopSchema reference)
Google
• supplyGoogle BigQuery SQLLanguage query service
• supplyCloud Dataflowdata flow,Cloud Pub/SubSubscribe to a messaging service
• developingGoogleVoice Assistant Service
Consultancy service
• Provide big data data analysis, Internet of Things, enterprise digital transformation consulting services
Accenture
• Develop an analytics-as-a-service cloud service platform to provide analytics tools and application services that enterprises pay for according to usage
Industrial application
- The application direction of enterprises is mainly based on the analysis of customer needs and market
segmentation in the market, mainly including the following contents
- Market segment analysis -Operational analysis
- Customer churn analysis -Supply chain analysis
-Customer behavior analysis -Customer service support
-Analysis of community needs - New product strategy
- Brand loyalty analysis -…
- all
- source
- Structured data (linked data, tables, records)
- Complexity data (hierarchical or exclusive source)
- Semi-structured data (XML, JSON)
- web browsing history orWebclick status
- Unstructured data (sound, language, image, video)
-Event data - IoT data (sensors, devices)

- Social media profiles - Scientific information
-Text information -Monitoring data
- Spatial geographic data (maps, hydrology) -…
- data storage technology
- Massively parallel processing relational databases (decentralized)
-Data Application Server (appliance)
- Hadoopfile system
- Stand-alone processing of relational databases
- Field oriented database
- NoSQLdatabase
-…
HadoopClustered Environments and Distributed Solutions
- HadoopIt is a cloud platform architecture that can store and manage a large amount
of data.ApacheAn open source project under the Software Foundation
- Hadoopis a cluster system (Cluster system), which can be expanded from a single server to
thousands of machines for integration, just like a supercomputer for application
- The way data is stored in this cluster is a decentralized file system (Hadoop Distributed File
System (HDFS), by the master node (Master Node)Shred the file into small chunks (usually
starting with64MBunits), and copy all small pieces of data three copies before distributing
them to all child nodes in the cluster (Slave Node)are stored separately asDataNode, and
throughNameNodePerform storage status monitoring
- Mainly used when performing analytical processingMapReduceprogram, by mapping (Map)and

induction (Reduce)The concept of decentralized computing
- Mapping: Process data fragments at each node, decentralize and distribute work
- Reducing: The results of each node's operation are directly sent back to the induction and integration, and the conclusion is obtained.
- Parallel processing of big data on thousands of machines can greatly save processing time
Ha
- data management technology
- HadoopRelated tools -Data virtualization
- CEPcomplex event handling -Database inline analysis
- MapReduceprocessing system - Field oriented database
- Hadoopfile system -Data Application Server (appliance)
- NoSQLdatabase -…
- Streaming Data (& River Computing System)
- In-memoryTemporary memory processing
- In-memorydatabase
Complex event handling (Complex Event Processing (CEP)
- Complex event processing is an analysis technique based on event streams in dynamic environments
- Events represent meaningful state changes
-By analyzing the relationship between events, using filtering, association, aggregation and other technologies,
formulate rules according to time, space, dependencies, constraints and causality, and continuously filter event
sequences from event streams, and finally analyze more complex compound event
-The main processing process includes
- Formatting: Convert the event information obtained by the event acquisition module into the form of internal processing
- Preprocessing: Process the event according to the field content
- Pattern detection: compound several events to find compound events
- Event dispatch: send composite events to the corresponding processing module
- Execute action: The processing model executes the corresponding action according to the event condition
-Mainly used for crime prevention such as identification of online fraud, prevention in banking and other financial industries, as well as risk
aversion and marketing decision-making, etc.
- Data Analysis Technology (Predictive Analysis Algorithms/Models)
- Linear regression - Simple Bayesian rule
- decision tree -Support Vector Machines
-Cluster analysis - Survival analysis
- Time series analysis -Integrated learning
- neural network -…
-Correlation Analysis
- data analysis tool

-Visual presentation tools
- Microsoft PowerBI, Spark Zeppelin, Tableau, …
-Data mining tools
- SPSS Modeler, Azure HDInsight, RapidMiner, KNIME, Rlanguage, …
-Big data application software
- SAP Demand Signal Management (market demand analysis),

SightMachine (Factory big data application)
- planning and implementation
-Business strategy (What is the value of big data? / Where is it used?)
- Where are the sources of information to assist in operation management, market innovation, and valuable information? How to collect? What are the
business processes that can be combined?
-Application situation
- Identify data collection strategies, analytical strategies, analytical models and possible technical architectures from the perspective of
people, customers and contexts
-Technical planning (can be confirmed together when discussing the situation)
- Technology group, technology architecture, technology risk,ITTeam development, data sources, data integration, data
processing, analytical applications
- Implement import
- Small projects can start to expand and develop the technical capabilities of big data processing and analysis
- Problem understanding, data understanding, data preparation, model building, model evaluation, application deployment
-Data management
- Governance structure, data ownership, data privacy, data quality, data security
business strategy
material
Application Scenario #1 Application Scenario #2 ... Application Scenario #N
Governance
technical planning
Implement import Implement import Implement import
Project #1 Project #2 Subproject #2-2
Big Data data processing
Analysis implementation technology implementation
Implement import
Deploy and run Subproject #2-2
Case Brief: Equipment ROI Analysis
- application situation
A wind turbine equipment manufacturer, the customer requires the need to provide data analysis of historical
information such as weather and wind power at the location of wind power generation, in order to evaluate the estimated
power generation output and ROI analysis of equipment investment in this location
- Implementation process
The equipment manufacturer collects15The annual global weather records are converted and
stored in the distributed file system, and the field-oriented data storage system is used to store
product data and set regional weather indexes to retrieve the corresponding weather records. The
industry develops business and R&D data query system, which can correlate the product model, price,
suitable situation and local weather stored in the data warehouse to provide reports for business sales
and R&D evaluation
Case Brief: Equipment ROI Analysis
- Implementation benefits
The equipment manufacturer can provide customers with ROI analysis reports
based on real data analysis, so that it is easier to gain the trust of customers.
R&D engineers can analyze the functional requirements of products based on real weather data, so as
to develop products that customers need and make adjustments according to local conditions.
The analysis query results of the established system can also be quickly responded within a few seconds,
which is convenient for data analysis and invocation.
Case Brief: Retail Pricing Analysis
A chain retail store uses traditional data warehousing andETL (Extract Transform Load)Data conversion
programs help analyze retail sales strategies, pricing strategies, and product strategies. However, with the expansion of
branches and the accumulation of data, the cost of hardware and software for building data warehousing is getting
higher and higher, and more and moreETLThe program also experienced maintenance difficulties, and the analysis
speed gradually decreased, and the analysis took two to three weeks. The retail store hopes to conduct instant price
promotions according to consumers' consumption behavior in each store during the approaching festivals.
Imported by the retail storeHadoopA structured data storage and processing platform that
integrates all POSSales data, product information, website click information, supply chain events, etc.
are stored on average toHadoopcluster computer and useMapReduceAfter processing by the data
processing engine, useDatameterVisual analysis software analysis.
Case Brief: Retail Pricing Analysis
The retail store usesHadoopThe cluster computer system can process hundreds of billions of data at the
same time due to the characteristics of parallel and distributed computing.2~3The pricing analysis work was
completed within hours, and at the same time, the original host system was more than 6,000COBOLlanguage
writing ETLdata, rewritten as approx.400OKMapReduceProgram batch processing, effectively reducing the
mainframe software and hardware procurement, writing and maintenance costs.
After the retail store updates the system, it can further collect and analyze the
consumption behavior of consumers in each store. After mastering more accurate data, it can
more efficiently launch key products.
Case Brief: Analysis of Fraud in the Financial Industry
A financial company provides a payment platform for online transactions and settlements for consumers
to consume through e-commerce websites and limit payments. Platform operators found that the platform receives
e-commerce orders from a wide range of sources and complex. If there are e-commerce shopping promotions, the
system will be overloaded and cannot quickly process the verification that should be completed within seconds,
including the consumer’s name, address, credit card or Information such as financial card numbers, ID numbers,
and past payment records can be used to determine whether there is fraud or fraud.
The financier has imported a complex event processing engine to quickly receive payment
orders from various parties and utilizesIn-memorymemory computing software andHadoop
Cluster computer technology conducts credit review, and stores customer payment information,
purchase records, etc. into field-oriented data storage for subsequent analysis
Case Brief: Analysis of Fraud in the Financial Industry
The financier utilizes distributed memory computing software with relatively low cost
requirements andHadoopCluster computer technology reduces the cost of replacing large data
warehouses. Using the original data warehouse architecture can avoid the cost and complexity of
replacing and updating thousands of applications in the data warehouse. In addition, the introduction
of a complex event processing engine helps to further write applications that can quickly detect
various possible fraudulent behaviors and issue alerts or notifications when they are hit.
Case Brief: E-commerce Browsing Behavior Analysis
With the surge in business of an e-commerce operator's e-commerce platform, the total
amount of data collected on consumers' order information, browsing behavior and click patterns
has reached100PB. Inquiries for analysts who provide products, channels, sales, etc., the industry
uses traditionalTeradataData warehousing for storage and analysis. With the storage and
application needs of unstructured data such as consumer browsing behavior, the industry has
introducedHadoopData processing and analysis tools for analysis and application.
The big data processing technologies used by the e-commerce operators are mainly divided into
three categories: the first category is data integration tools, which are responsible for data capture,
processing and cleaning, including batch and real-time processing; the second category is data storage
tools, including traditional data warehousing,HadoopClustering and other tools for processing various
structured and unstructured data; finally, there are data analysis tools, including various types of analysis
aids, which can be quickly analyzed by analysts.
along withHadoopThe development of the community and the internal development of the
company's engineers, allowHadoopThe tool can further provide interactive analysis of transaction-type and
graph-type data, and can also interact with various data stored in the data warehouse.HadoopThe various
data in the data are compiled, and then various tools are used to conduct interactive inquiries.
The e-commerce operator can not onlyHadoopcluster processing,In-memoryMemory

processing to speed up the processing and analysis of various transaction data and graphic
data, self-developedHadoopInteractive query technology allows analysts to simply useSQL
Grammar can quickly perform mixed analysis of structured and unstructured data to
discover and profit from new trends.
Case Brief: Digital Advertising Bidding Platform
A large-scale digital advertising bidding platform provides advertising matching and bidding
services. On the one hand, it assists customers to place on suitable pages, and on the other hand, it
also assists page owners to find publishers with appropriate bids. The platform needs to quickly
organize the free time and price of the layout to facilitate bidding, and analyze the needs of advertisers
to match the appropriate layout time and click efficiency. The platform aims totwenty four Provides
various analysis reports to advertisers continuously and quickly.
the digital advertiser throughAmazon Hadoop MapReduceThe cloud platform performs data
processing of a large amount of information on the Internet, and uses complex event processing
technology to receive real-time click records from variousAmazon An open-source parallel database
and self-service analysis reports are established on the platform for advertisers to interactively
analyze real-time advertising benefits.
Case Brief: Digital Advertising Bidding Platform
The digital advertiser usesPBThe cloud platform with integrated data processing capabilities
reduces the cost of building and maintaining related software and hardware, and enables advertisers to
quickly query the benefits of various advertisements independently, improving user satisfaction and
adherence. In addition, through automated complex event processing tools to maintaintwenty four
Uninterrupted, detailed and efficient collection of click information and information on each
advertisement page allows users to track and analyze more conveniently.
Ecosystem Development
- Enterprises or individuals rely on various types of software and hardware tools and the assistance of data
suppliers to mine and process potentially valuable data and convert them into valuable data,
information and even knowledge
-The data analysis team in most enterprises that have not undergone a complete digital transformation successfully plays the
role of ITEngineer and Information Management System (Management Information System, MIS) operation management
tasks, responsible for the output of reports or data mining, focusing on business thinking and not familiar with the
company's own business
-However, even professional analysts need various information platforms and analytical processing tools. The data
team needs to quickly combine and schedule various digital services in response to the rapid increase in data
sources, and then implement them into business applications
-In a modern society emphasizing specialization and division of labor, data storage, processing, analysis, and presentation
can be handed over to the services and tools provided by professional external teams, and the company's internal
personnel only need to use this to explore the value of their professional fields. Just work, so that a big data analysis
ecosystem with coordinated development and operation will gradually be formed.
- In the development of the entire big data ecosystem, the following roles may be included
-Consumer or business: end users who use big data, do not necessarily need access to the data
- Such as: product recommendations on shopping websites, maintenance warnings for factory equipment or vehicle engines, etc.
-Value-added supplier: organize and transform the original data and give it to application service providers or end users
- Such as: government open data,weather.comProvides global weather information, etc.
- Application Service Providers: Provide data analysis, models or applications as commodities to customers
- In the information economy, application services may be obtained through subscriptions, and application service providers
may also benefit from advertisements implicit in the services or brand effects that gain market share.
-Software suppliers: provide data processing software, analysis software or data analysis tools, etc.
- In line with the development of cloud computing, many software providers have also set up systems to provide related cloud services
- Well-known suppliers areSASS,SAP,Microsoft,Oracle,IBMWait

-System Integration and Consulting Service Providers: Assisting users and suppliers to make good use of analytical tools and equipment
- Well-known service providers areAccenture,PwC,Orange Business ServicesWait
-Infrastructure suppliers: provide software and hardware products such as storage devices, servers, circuits, etc.
- The above cloud services are also part of the infrastructure supply
- Well-known suppliers areGoogle,Microsoft,Oracle,IBM,Amazon, SAP,

TeradataWait
Big data application users
consumer
enterprise
(personal user)
Data application Integrate with
value conversion Consultancy service
system integration and
Big data application service provider Consultant service provider
Data Processing/Analysis Services Data value-added services
big data software provider Big data value-added provider
Data storage, computing IT architecture services
Big data infrastructure provider
Glossary
- 5W1H
5W1H
- Mind Map / Mind Map

Mind Map
- Characteristic factor analysis diagram/fishbone diagram
Cause and Effect Diagram/Fishbone Diagram
- Statistical Quality Control

Statistics, Quality, Control (SQC)
- HadoopCluster environment
Hadoop cluster environment
- complex event handling

Complex Event Processing (CEP)
problem discussion
- What is data processing

What is data processing?
- Try to list the types of questions

Please list different types of “problem”
- Six Steps to Problem Solving

Six steps of solving problem
- Analysis methods at different levels (depths)

Different levels of analysis
- Common data value exploration directions

Common investigate direction of “data value”
problem discussion
- The main content of enterprise value

The main content of data value in corporation (corporation value)
- The main content of information economic value
The main content of data value in Infonomic (Infonomic value)
- The Market Characteristics of Information Economic Value
Market Characteristics of Infonomic Value
- Steps in the Value Cycle

Steps/process of data value cycling
- Business analysis technology in different eras and technological development backgrounds
Business analysis technology with different age and technology
problem discussion
- Try to list the technical product demand items for big data analysis in the market
Please list the technical product that the markets request for big
data analysis
- Main software product types for big data analysis

Please list the major software product types of big data analysis
- Try to describe the overall technical architecture of industrial applications
Please describe the technical architecture of industrial application
- Try to describe the planning and implementation process of industrial application
Please describe the planning and implementation process of

industrial applications
- Try to describe the various roles and their associations in the development of the big data ecosystem
Please describe the roles and their relationship in the big data
ecosystem
Q&A
Ddepartment of Refrigeration and Air Conditioning and Energy Eengineering, National Chin Yi University of Ttechnology

(04) 數據的分析與應用.zh-CN.en

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

(04) 數據的分析與應用.zh-CN.en

Uploaded by

Copyright:

Available Formats

Translated from Chinese (Simplified) to English - www.onlinedoctranslator.

Data analysis and application

select the corresponding features or perform corresponding calculations according to the

problem object to obtain the required information

answer to the question can be obtained.

diversity (structured or unstructured), processing speed (batch/interactive/real-time), etc.

-batch × large amount of structured data

- Use traditional reporting tools or batch processing jobs

-interactive × large amount of structured data

-Batch ∕ Interactive × unstructured data processing

-immediate × Large amount of unstructured data

results through corresponding feature extraction.

different platforms for application or data integration

an important basis (index) when introducing analysis tools

point of the analysis and avoid falling into a misunderstanding

important processes/implementation means

about the problems in an all-round way and analyze them

- Restoration problem:now The purpose is toUndisturbed

-Coping methods: find out causality, difference analysis, benchmarking…

- Pursuing ideal type questions:future The purpose is toideal

- It is an exploratory problem, a potential problem that needs to be discovered through analysis

- Coping method: After selecting an ideal, plan an action plan

-Coping: try to identify triggers and make plans

2. Collect data, analyze data, and identify reasons

3. Propose and select solutions (hypothetical)

4. Evaluate and implement the best option

5. Verify facts and evaluate results

6. Identify process variations and negative impacts

2. Collect data, analyze data, and identify reasons

3. Propose and select solutions (hypothetical)

4. Evaluate and implement the best option

5. Verify facts and evaluate results

6. Identify process variations and negative impacts

- Mind mapping is a method of organizing a series of thinking thoughts using radiating

communication and planning

pushpins and association lines

2. Collect data, analyze data, and identify reasons

- Identify controllable factors, uncontrollable factors, and possible constraints to select

- Analyze the results after data processing

3. Propose and select solutions (hypothetical)

4. Evaluate and implement the best option

5. Verify facts and evaluate results

6. Identify process variations and negative impacts

2. Collect data, analyze data, and identify reasons

3. Propose and select solutions (hypothetical)

- assumptions and confirm whether the constraints are met (feasible)

- Confirm solution scope and make adjustments

4. Evaluate and implement the best option

5. Verify facts and evaluate results

6. Identify process variations and negative impacts

2. Collect data, analyze data, and identify reasons

3. Propose and select solutions (hypothetical)

4. Evaluate and implement the best option

- solutions (execution based on merit)

5. Verify facts and evaluate results

6. Identify process variations and negative impacts

2. Collect data, analyze data, and identify reasons

3. Propose and select solutions (hypothetical)

- Set up a virtual verification environment to verify points in detail

- Actual input verification (number of test results obtained

- Observation, continuous operation and correction

6. Identify process variations and negative impacts