(03) 數據的收集與存儲.zh-CN.en

Translated from Chinese (Simplified) to English - www.onlinedoctranslator.
com
Data collection and storage
DAeardaptmvn
ceetnf l eecftrirger
doER ic aalti
on Coanntdrol Ai
Lr aCbo.nDiti ia l eEin
ycEtrnigcne
dpenoairntgmaenndtEonfeErgle rng,g
inNeaetrioinnaglNChaitnioYniaUl nCivheursnitgy-oHfsTinech
gnUonloivgeyrsity
Course outline
- What is data
-Data digitization
- Source of information
- How to collect data
-Data collation
-Data characteristics
-Feature Engineering
- How to store data

-Data storage
-Data security
Department of Refrigeration and Air Conditioning and Energy Engineering, National Chin Yi University of Technology 2/88
What is data
- For thinking and communication, human beings have created varioussymbol And use these symbols
to formmaterial And store it, or get it through processingInformation To facilitate analysis and
transmission, resulting fromknowledge And accumulate, gradually formwisdom
-The process of organizing personal knowledge and experience of people, events, and things, using data to
record, store and accumulate, and continue to interpret, understand, and relay the information in the data
through different people, and even reassemble it into a data record. , Allowing knowledge to be exchanged,
passed on and developed
-The accumulation of data and knowledge allows future generations to continue to refine and accumulate personal wisdom based
on previous experience and their own experience and opinions, and then develop into a variety of different thinking and
speculative content, and gradually form disciplines in a common field
-Through the accumulation of knowledge and the development of technology, a series of common concepts, values and
codes of conduct are gradually formed in different life circles, which become different cultures
-Human beings mainly form culture and technology through the exchange, inheritance and development of knowledge, establish a
society with more intensive interaction and exchanges, and gradually improve their lives
What is data
- Symbol (Symbol):
-Representation of a specific meaning/means of expression
-Mainly used for communication, expression or recording
-Sound, light, color, graphics (pattern, icon), shape (geometry, arrangement, abstract figure/
representation), text (character, letter), waveform (sound wave, light wave, electric wave),
action (body language, sign language) Etc. can be symbols
What is data
- material (Data):
-From the relevant context (Context)Facts and observations
-It can be seen as an understanding of people, things, and things
-In a narrow sense, it is a numerical feature obtained through observation
-Usually composed of a series of symbols, which can be a bunch of organized or unorganized text, numbers, files,
etc., which is conducive to sharing, communication and storage
-After measurement, collection, reporting and analysis, different applications can be carried out
-It can also be visualized with charts, images or other analysis tools to assist in the communication of
information, personal interpretation and analysis
What is data
- Information (Information):
-Results obtained by putting data into meaningful situations
-Can be regarded as an abstract concept of understanding of data
-The corresponding meaning can be obtained after storage, analysis and interpretation, but the results may vary from
person to person
-Even the same person may have different understandings of the same information under different time
and space backgrounds, but it is not because the information is biased or divergent, but because of
personal knowledge, opinions and various external factors.
-The method or medium of the record as data will also affect the result of the communication of the information
What is data
- knowledge (Knowledge):
-Things to believe or value based on the content of the information
-It is a known state or fact, which can be regarded as the sum of what is understood, discovered or learned by
humans
-Understanding from the experience of others, people’s common understanding of a certain person, thing, or thing
(consensus)
-The accumulation of others' cognition, experience and experience
What is data
- wisdom (Intelligence):
-Comprehensive ability based on the formation of nerve organs (material basis)
-Includes: perception, knowledge, memory, understanding, association, emotion, logic, discrimination,
calculation, analysis, judgment, culture, moderation, tolerance, decision, etc.
-Able to deeply understand all kinds of people, things and situations, and have the ability to think, analyze, and explore the
truth
-Personal cognition, experience and accumulation of experience
what
Working environment
material Information wisdom
Operational
Data Information Intelligence
Environment
collect Processing and development Analyze and generate

Collection Processing & Exploitation Analysis & Production
Source: Joint Intelligence / Joint Publication 2.0 (Joint Chiefs of Staff)
What is data
- In different fields, data and information may be similar or different in definition

-When the data is reviewed or analyzed in the context, it becomes information
-In the discussion of academic topics, data is the unit of information
-In the computer (computer) field, data will be converted into characters, fields, records,
files and databases for storage or calculation
- After processing the data, the available information is called information
-In the field of data analysis,material Represents a set of qualitative or quantitative variables about one or
more people and things, andInformation Is processed and useful data
- Sorting / processingRefers to the process of correcting the original data by removing outliers and obvious
instrument or logging errors, usually in multiple stages for different processing
- Original dataIt is relative to the data before processing, and the processing/organization of data is usually divided
into multiple stages, so the data sorted in the previous stage will become the relative original data in the next stage
- The biggest difference between data and information is not whether it is "processed", but whether it is "meaningful"
- The process of processing may make the data meaningful
Type of data
- The classification of broad data is shown in the figure below
material
Data
Unstructured data Structured data

Unstructured Structured
Conversion
Qualitative data Quantitative data
Qualitative Quantitative
Categorized information Sequencing data Distance data Fixed ratio data
Nominal Ordinal Interval Ratio
(Categorical)
Data presentation: data visualization/descriptive statistics
Visualize / Descriptive Statistics
- The classification of the digitized data recorded in the computer mainly includes:
-Numerical data: numbers, fixed-point and floating-point numbers or data groups that can perform arithmetic operations
- Audio source, image and video data can be calculated through corresponding methods, usually in this category
-Non-numerical data: data that cannot be executed in arithmetic operations, such as text data
Characteristics of various types of structured data
Descriptive Statistics
grade characteristic Instance chart
statistics Mode median extremum average value Standard deviation
Classify
Discrete dichotomy Frequency/Account Bar graph
(Nominal) O X X X X
Disorder color Mode Pie chart
(Categorical)
Rank
frequency
Bar graph
Mode
O O O X X
Sequencing Ordered categories
degree Pie chart
(Ordinal) compare median
order Stem and leaf diagram
Percentile
frequency
temperature Mode
O O O O O
Fixed distance Number difference
Earthquake intensity median
(Interval) Significant
Class wind average value
Standard deviation
continuous
Amount Histogram
O O O O
Fixed ratio average value
Multiply and divide Sometimes
(Ratio) weight Standard deviation Box plot
Absolute zero
Digitization of data (electronic)
- In modern society, most of the storage, analysis and processing of various data
need to rely on computers, so how to digitize the data is very important
- The characteristics of electronic data
-Fast speed (withMIPS (Million Instruction Per Second)As a unit)

-It can be processed repeatedly with high accuracy
-Ability to store large amounts of data (TB-PB-EB-ZB-YB)(210=1,024Base)

Tera Peta Exa Zetta Yotta
-High quality information
- The units for storing data are in descending order
-Bits-Bytes-Characters-Words-Fields-Records-Files-Database
bit Byte Character Word Field Record File Database
- 1 byte = 8 bits
- 1 Character = 1 byte (Alphanumerics, symbols) ／ 2 bytes (Chinese)
- 1 Word = 2 bytes = 16 bits

- Fields and records are usually the basis for the composition of the database
- Digitization (Digitization)
-Digitalization refers to the process of digitizing (transforming information into digital media),
which can include text, pictures, audio, etc.
-Most of the conversion process is carried out using sensors or scanning equipment, mainly for
the information itself
-Photographs taken by digital cameras are a commonInformation Digitizationmedium
- Digitization/Technology Digitization (Digitalization)

-The main purpose is to use digital technology or methods to improve the business model or
customer experience
-Different from digitization, digitization puts more emphasis on the "process" of digitization, that is, the digitization of processes, including
the interaction between people, organizations and organizations, and the digitization of communication media
-In the organization, digitization often refers to the process of integrating digital technology into existing operating
procedures and operating models, as well as the automation of parts of the organization's operating procedures
-includeEmail, Social media, integrated corporate functionsAPI, And all kinds of software that enable
internal operation processes to be completed on digital channels are common examples
- In various physical systems, what we can observe

The signals observed/measured are all a series of support
The continuing dynamic change is calledAnalog signal
- ThesecontinuousofAnalog signal, After passingSample and
holdAfter the process will becomeDiscrete signal
- Discrete signalPass againQuantifybecomeAfter the

binary value, It becomes the order in a fixed time period
Segment signal changes, that isDigital signal
- Binary digital signal can be directly sent to digital

The circuit performs calculations and is also convenient for storage
- The basic unit of calculation in the computer is binary bit

Yuan as the basis, and then through the instruction stack to achieve each
Complex computing content
- Basic operations of data processing
-Data creation (Create)：Select electronic file type according to file content
-Data conversion (Conversion): Transfer to a different media
-Data sorting (Sort): Arrange in order
-Data merge (Merge): Combine files of the same nature and subject into the same file
-Data distribution (Distribute): Assign data to different files according to conditions and specifications
-Data Search (Search): Find the required data or file according to a certain key value
-Calculations and lists (Compute & Listing): Calculate data or list results according to instructions
-Data update (Update)：Edit, add or delete data

-Data check (Check): Cooperate with search and comparison to confirm the content of the data
Development of big data data (digital data)
- Big data (digital data) originally refers to a collection of structured/

unstructured data from various sources
-The amount of data is so huge that it cannot be managed by a traditional database system, and must be
distributed through multiple servers (possibly up to thousands in parallel) (computer clusters)
- Nowadays, it mainly focuses on the following aspects (a large amount of complex data in a short time)
-quantity(Volume): Data size

-speed(Velocity)：Data input and output speed／ Data volume
Data throughput per unit time (Volume) Correctness
(Veracity)
-changeable(Variety): Diversity of information
value
(Value)
-Authenticity/Correctness (Veracity): The authenticity of the information (Generate/Update)
Diversity speed
(Variety) (Velocity)
Big data data characteristics:
Large, fast, diverse, uncertain
- The processing of big data is different from the sampling and analysis of traditional data statistics. Instead, it
focuses on observing or tracking existing data, and trying to find out its laws and make decision-making
analysis through observation or statistical methods. This has led to many related technologies.
-Large-scale parallel processing (MPP)database
-Data mining
-Distributed file system
-Distributed database
-Cloud computing platform
-Edge Computing Technology
-Extensible storage system based on the Internet
Big Data
source:Teradata, Inc.
Data speed
Data Velocity
immediate
(Real Time)
Near instant
(Near Real Time)
cycle
(Periodic)
batch
(Batch)
Data volume
Social interaction Spreadsheet table

MB GB TB PB Data Volme
(Social)
image database
image Web page
Capital
Why digitize
- All the information in the environment, including the sound we hear, the image we
see, the touch of the skin, etc., are all different signals
-If you want to save this information, you can save the content as data, such as audio tapes,
photos, videos, documents, etc., through recording, photography, description, etc.
-These data need to be stored through physical media, such as tape, paper, etc., but the space occupied and its
maintenance work need to be considered when storing
- After the data is digitized, the main advantages are as follows:
-Convenient to save and record
-Data can be protected through encryption and decryption algorithms
-Convenient for exchange and transmission
-The data capacity can be reduced through compression and decompression algorithms, which is conducive to storage and transmission
-It is convenient to edit and process, and it is not easy to cause damage to the original data
-Easy to analyze, process, debug and debug
Why digitize
- Through the digitization of data, in addition to the use of computers for processing, it also allowsEmbedded
system (chip) Continuously sense the environment to generate cognitive results of the environmental
information to convert the environmental information into digital data storage and analysis
-With the Internet of Things technology, the device can be online, and data can be stored through a remote database
-In addition to facilitating device status monitoring, it can also be remotely controlled, making the application and deployment of the
device more convenient
-Various electrical devices commonly used in daily life, if they are connected in series through the network, equipped with various
environmental sensors to collect environmental information, and then integrated and analyzed with an embedded core with
certain control and calculation capabilities, a variety of life wisdom can be realized Application
-These large amounts of continuously collected data, including environmental information and equipment operating
conditions, are an important basis for big data data
-How to organize, store and analyze the collected data and give it meaning and value is
the most important issue in big data analysis
Embedded Systems
- Embedded Systems(Embedded Systems)

-Generally refers to a microcomputer system that is embedded with controlled targets and designed for specific applications
-The system must be miniaturized in order to integrate and embed the target
-Different from general processors, in addition to the ability to execute and arithmetic programs, more emphasis is
placed on their independence. Usually, required peripheral hardware modules are added to the system chip to
perform tasks with minimal limited hardware resources
-Software programs designed for specific goals are called firmware

-Usually the system will not be an independent system, but as a subsystem of a large device
- However, as technology advances, some embedded systems also require more complex calculation
capabilities
-What is the part of the complex user interface (such as portable mobile devices)
-The microprocessor at the core of a personal computer (CPU)With embedded controller (MCU)The difference between the two is gradually
blurred, and is gradually referred to as microcontrollers.
Embedded Systems
- Microcontroller
(Micro Controller Unit, MCU)
- forEmbedded controllerandmicroprocessor Collective
term for other chip systems
- Independent and effective hardware control
System and data processing
- The basic composition of the microcontroller includes:

- Central processing unit
(Central Processing Unit,CPU)
- Memory
(Memory,IncludeRAM,ROM)
- Input and output unit
(INPUT/OUTPUT,I/O)
- Other module units
- The microcontroller is likeWill have basic skills

The computer unit of energy is concentrated in a single crystal
In the film, So it is often called microelectronics

Brain controller or single chip controller
Embedded Systems
al Interrupt
errupt Timer 1
ontrol ROM RAM
Timer 0
- MPUIt is the basic unit of a computer system CPU

Counter
Integration, includingCPU (ALU+CU),

BUS I/O Serial
Internal memory, bus and output OSC Control Port COM
Inbound etc. Modules TX RX

P0 P1 P2 P3 P4
- andMCUIs based onMPU, In addition, the (MPU)

peripheral circuit and the expanded external
Part of the memory is integrated in the chip
- For computers, there is always only high
IntegratedCPUAs the core, the rest

Are all external peripherals, through the motherboard
Assemble as one
(MCU)
Embedded Unify
[8051] [PIC] MicroChip [FPGA] ALTERA / Terasic [DSP] TI

Explorer 16 De Launchpad (MSP430)
[AVR] ATMEL [ARM] STM32 [ARM] MediaTek [Tensilica] Ai-Thinker

Arduino UNO Nucleo F446-RE LinkIt 7697 (MT7697) ESP8266 / ESP32
(ATmega) (Cortex-M4) (Cortex-M4) (Tensilica Xtensa)
Embedded Systems
Wemos D1 mini
(ESP8266)
Embedded Systems
- along withWafer technologyThe development of current electronic control systems can be achieved through powerful
Embedded computing coreIntegrate the overall control system into a small motherboard
-The computing power of the embedded core is gradually approaching that of a desktop personal computer. At present, the core selection is becoming
more and more important.Low power consumptionas well asSystem cooling
- Many peripheral key modules have also become self-containedCalculus core, Can handle calculations alone andProvide
results directly , Saving a lot of content that needed to be calculated by the computing core on the site in the past,
and equivalently improving the operating efficiency of the overall system
-Continuing, the current electronic control system also often uses this method of processing different sensing analysis
or conversion through different subsystem cores, and finally through the communication between different cores
to gather data to the main core for main control calculations.
-Separate the cores of different subsystems to operate independently, and only use the network connection to integrate and
process the resultsDistributed system architecture, And software to integrate different decentralized subsystems and
manage resourcesDistributed operating system, Also in high-speed, high-bandwidth, low-latency5GThe realization of
the Internet gradually matures
Embedded
Internet of Things Technology
- Internet/Internet (Internet)Refers to the huge and complex network connected through the network
and the network, and provide standardized services, which can be used to process the interaction
and sharing of large amounts of data, and even provide virtual services
- Internet of Things (Internet of Thing, IoT)It is an Internet-based architecture that works with various
wired or wireless sub-network architectures to upload data on items/devices to the Internet, so
that the monitoring and operation of the device can be performed remotely
-It can even connect all the equipment in the factory to the central control servo-management through the
Internet of Things, and cooperate with machine learning or artificial intelligence for production management
to achieve the goal of a smart automated factory
-The simplified conceptual narrative is "Internet of Things"
- The biggest difference between the Internet of Things and the original Internet is the simplified application or reconstruction
of the application layer communication protocol.
-In the initial stage, it was based on the original InternetOSISeven-tier structure to be revised
-With the expansion and improvement of technical content, gradually form its own technical ecosystem
- IoTThe technology ecosystem of is mainly divided into devices, data, connection

capabilities, and technology users for discussion
-Perception/device layer: It is composed of sensors, actuators, hardware, software, connectivity and gateways to
form a device that connects and interacts with the network
-Network／communication layer: When data exchange and transmission, the connection medium of the device (the connection
, Structure
Technology Is the place
- Perception/device layer
-Embedded system: includes hardware and software, and manages specific functions related to large-scale systems,
with a microprocessor (MPU)Or microcontroller (MCU)As the main control core
- Microprocessor unit (MPU):will CPU The functions are loaded on a single or multiple integrated circuits. Although the
microprocessor needs peripheral equipment to complete the work, it only contains CPU, So the processing cost can
be greatly reduced
- Microcontroller unit (MCU): Embedded on a microchip, with the functions expected of a small computer, and includes
CPU,RAM and ROM, Although the microcontroller contains the elements required to perform simple tasks, its power
limit is more than that of the microprocessor (higher power consumption)
-Transducer: A physical device that converts a certain form of energy into other forms of energy
- Sensors: detect changes in the environment and transmit them to the control core through electronic pulse signals
- Actuator: The control object required to act in response to the change recognized by the sensor
-Intelligent devices: devices that have computing capabilities and can actively perceive the environment, usually
including a microcontroller for integrated control. Unit built based on embedded system
-Perception and identification technology: In response to simple control requirements, basic computing capabilities are added to the equipment, so that the
terminal equipment can directly operate independently based on the measurement results, and the data will be uploaded to the cloud separately
- Network/communication layer
- IoTGateway (Getway):fromIoTThe data collected by the device will be moved through the gateway, pre-processed
at the edge node, and thenIoTThe device is connected to the cloud (no need to access the Internet directly),
mainly used for heterogeneous network connections
-Connection medium (connection method): low-power short-distance network, low-power wide area network
- mainly includesWiFi,Bluetooth,ZigBee,Z-Wave,NB-IoT/Cat-M2,4G LTE IoT,

LTE Cat-M1,Cat-0,Cat-1,5G IoT,Sigfox, Narrowband, etc.
-Data exchange method (communication protocol): mutual connection based on open system (OSI) Model, in view of the
simplified five-layer architecture, the data exchange between different layers is adopted, and the corresponding
communication protocol is selected (even cross-platform and cross-system data exchange)
- Due to the Internet of Things architecture, most of the sensor nodes themselves use smallMCUAnd powered by a
battery, soM2M (Machine to Machine)The agreement must take into account the limited hardware capacity and
power consumption conditions, when network transmission, there is a higherThroughput, Low latency, low power
consumption, and even provide differentQoS (Quality of Service)
-Information security technology: Recently, the development of encryption technology for the security of data in the transmission
process has been started, mainly to avoid the simplified communication process, which makes the data easy to be stolen.
- Application layer (IoTplatform)

-againstIoTApplication to provide services, or to establish related platforms
-Common applications include
- Human-computer interaction/monitoring information platform
- Web APPIntegrated applications (front-end, back-end applications)
- Authentication, device management and interfaceAPI
- Cloud gateway, cloud server,Cloud computingAnd service platform
- Predictive maintenance
-Identify the same trend from the current data and predict when the equipment may need to be maintained
- Analysis and decision support platform
-Key related technologies include
- Artificial intelligence - Open source

- Virtual reality - Serverless operation
- Blockchain - Container Cluster Management System
- Quantum computing -Used to automatically deploy, expand and manage containerized applications (Container)
- Kubernetes (K8s)
Advanced message queue protocol
Application layer
Establish an interoperable software layer between communication intermediary software, and an industry-scale standardized communication mechanism
(Advanced Message Queuing Protocol, AMQP)

Between user and device
Interactive interface Restrict application protocol Limited bandwidth and limited network communication protocol, designed for devices with limited connectivity, lightweight
(The Constrained Application Protocol, CoAP) TCP/UDPCommunication protocol, through the server to convert simple information provided by the device intoHTTPAgreement to go online
Information Dissemination Service Multifunctional point-to-point communication protocol can simplify deployment, improve reliability, and reduce complexity
(Data Distribution Service, DDS)
Message queue telemetry transmission For lightweightM2MThe communication protocol designed for communication is mainly used for low-bandwidth connections with
(Message Queueing Telemetry Transport, MQTT) remote locations by the subscriber (Subscriber),announcer(Publisher)And intermediary (Broker)Data exchange structure
Transmission Control Protocol The main communication protocol used by most Internet connections, provides host-to-host communication, splits large
Transport layer
(Transmission Control Protocol, TCP) amounts of data into individual packets, and resends and reassembles packets as needed
Data is transferred between layers
User data packet protocol

When inputting, make sure that the data is connected based onIP (Internet Protocol)The implemented communication protocol can ensure the communication between the process and the
And ensure communication security (User Datagram Protocol, UDP) process, UDPCan be improved throughTCPThe data transmission rate is suitable for data transmission without distortion
Network layer
6LoWPAN Low power versionIPv6, Can shorten the transmission time
Assist device and router IPv6 It can route Internet traffic, identify devices on the network, and locate and address (IP)
communicate
Data link layer IEEE 802.15.4 Radio standards suitable for low-power wireless connections, can be matched Zigbee,6LoWPAN And other standards to use to
build a wireless embedded network

Transmission within the system architecture
Data, find and correct LPWAN The communication distance of this type of network is up to 500 More than meters.LoRaWAN Yes LPWAN A representative of
Error in the physical layer optimization for low power consumption
Bluetooth Low Energy Significantly reduce power consumption and cost, and maintain the range and performance of traditional Bluetooth, low cost, long battery
Physical layer
(Bluetooth Low Energy, BLE) life, the most commonly used technology in consumer electronics products
Establish a communication channel, let
The device can be specified Ethernet Wired connection transmission, low cost, can provide fast data connection and low latency
Intra-environment connection
Long-term evolution technology A wireless broadband communication standard suitable for mobile devices and data terminals, which can increase the capacity and speed of
(Long Term Evolution, LTE) wireless networks, and support multicast and broadcast streaming
Near Field Wireless Communication A communication protocol using electromagnetic fields, which allows the distance4cmInternal (short-distance) communication between two devices is often used in
(Near Field Communication, NFC) applications such as identity recognition door cards, contactless mobile payments, tickets and smart cards
Radio frequency identification Use electromagnetic field to track passive electronic tags without power supply. Compatible hardware can provide power (trigger) and
(Radio Frequency IDentification, RFID) communicate with these tags, while reading their information for identification and verification
Wi-Fi/802.11 Standard options for homes and offices, limited scope, continuous power consumption around the clock
Cloud computing (Cloud Computing)
- Cloud computing is to provide computers with various terminals and other devices on demand through shared software
and hardware resources and informationDynamic and easy to expandandResource virtualizationA kind of
computing method／virtual computing environment
-Mainly emphasize itsDynamic configurationandUse flexibility
-The service characteristics of cloud computing are named after the similarities with the natural cloud and water circle
-The computing resources, storage resources, data resources, and application resources gathered on the Internet are
increasing with the expansion of the Internet. Therefore, the Internet is transforming from a traditional
communication platform toUbiquitous and intelligent computing platform
-Cloud computing mainly relies onResource sharingTo achieveEconomies of scale, The service provider integrates a
large amount of resources for multiple users to use, users can easily request/rent resources, andAdjust usage
according to demand, release unnecessary resources and return to system architecture
Cloud computing (Cloud Compu
- At present, the supplier is mostly responsible for the system User rent
Use, the user does not need to understand the detailed bottom layer Can be used
Implement the architecture of different applications
-Software as a service
(Software as a Service, SaaS)
-Infrastructure as a service
(Infrastructure as a Service, IaaS)
-Platform as a service Hybrid cloud
(Platform as a Service, PaaS) Private Cloud
Public cloud
-Desktop as a service
(Desktop as a Service, DaaS) Data Cloud
Community cloud
Community
Edge computing (Edge Computing)
- Edge computing is mainly in response to the application of big data in the cloud today. If all
information is uploaded to the cloud before monitoring, control, and analysis are performed,
sometimes it will not be anxious, so I hope to collect the data on the on-site terminal／ Gateway(
Gateway) Add calculation skills to make it possibleImmediate supervision,Instant analysis and
reactionAnd other functions, that is, relative to the edge of the data cloud (Edge)Operation and
processing on
- CiscoProposedFog computing (Fog C Enhance by
strengthening the calculation on the field side
The computing power on the point and each section
Computing power and computing resources between points
Sources can be dispatched and borrowed from each other
useSensor / terminal equipment
Communication networkthe concept of
Cloud computing and edge computing
Cloud computing Edge computing
Cloud computing analysis and service Cloud storage, service
Judge whether it is Analysis data storage
Logged-in residents Judgment result storage
Instant event notification Instant event notification

Analysis result
Judge whether it is
Analysis result Logged-in residents
Edge device
Calculation
Do you want Do you want
Trigger an alarm Trigger an alarm
- The application of the Internet of Things in life is closely related to various digital services
-The Internet of Things in life will automatically respond by sensing our behavior (service)
- The more common applications are mainly dialogue robots that are actively contacted by users and provide consultation
- Combined applications related to wearable devices, followed by development of personal health management, exercise
management, digital medical care, etc.
- There are other hard-to-detect social software pushes, shopping website recommendations, online advertisements, etc.
-At the same time, the user's actions (behavior) will be synchronized to the cloud and become a data source for
service providers to continue to develop
- But at the same time, there is also whether these private information can be used by manufacturers without authorization
- In addition, there are also information security issues related to the theft of information through IoT devices
-As various facilities in the living environment realize digital connections and smart devices, the related applications of
the Internet of Things, smart homes, and smart cities will promote the next explosive development of related
technologies and industries
Why digitize
- Digital integration of behavior patterns and management processes
-In addition to the digitization of data, operational processes, marketing operations, etc. can also be digitized through the
integration of various digital technologies (focusing on processes) to achieve efficiency improvements and enhancements.
This step/process is called the enterprise or organization’sDigital transformation
-After establishing standard processes and digital management of behavioral processes, different business management
models can be applied for analysis or adjustment in order to achieve the set goals
- Digital integration of administrative procedures and administrative data
-ThroughElectronic official document sign-off system , With the Internet transmission that is not restricted by space, administrative
efficiency can be improved even more
-The digitization of administrative data, in addition to facilitating the transfer and approval to save time, it can also realize
paperlessness, reduce the waste of resources and implement environmental protection
-Data digitization is helpful to the preservation and access of data at the same time, and it is more compatible with the transparent and
open sharing of data in the administrative system, creating a more complete digital environment
Digital transformation (Digital Transformation) (DX)
- "Digital transformation" is the process of combining digital technology with existing operating processes, that is, organizing a
comprehensive integration of digital technology, including operating processes, marketing services, data analysis, and infrastructure. It is
often discussed together with digitization or digitization.
- Digital transformation also refers to the wider application of digital technology, that is, in
Apply digital technology to streamline the process to make it more rational and flexible
- Operation Process (Operational Process)

- Marketing and Business ((Marketing & Sales)
- Accessibility (Supporting Functions)

- Innovation and R&D (Research & Development)
- Information Technology (Information Technology)
- Digital reinvention (Digital Reinvention)

- An organization that has completed digital reengineering, regardless of industry, can be defined as a technology company, because its operating
process is inseparable from digital technology
- Digital Reengineering can also be said to redefine the value proposition of the entire organization
- In short, through the new technology of digital technology, the industry, life, and the market are greatly changed, and new value or
delivery methods are created. This is digital transformation.
Digital Reengineering and Digital Transformation
- Organizations that have undergone digital reinvention will redefine , And let the organization’s camp
A more complete combination of transportation mode and digital technology
-Whether in R&D, marketing, business, customer service, operations and
other corporate functions, it will focus on customers
Experience and customer value as the core Transport mode Customer interaction and experience
s Model Market Activation

-Enterprises will adopt innovative technology, products,
Service and the experience of each touchpoint, and
Build stronger brand adhesion between customers
-The fundamental transformation of the core value of the organization, from the root
Customer experience Deepening customer insight
I started to discuss the products and services currently available Organizational talent capitalization
Restless Talents erience Actionable Insight
Service, thoroughly examine and gain insight into customers,
Redefine the value proposition and find out what needs to be
Optimize or newly added ring nodes, and finally build
Establish a clear strategy for the most effective execution
Well-planned ecosystem Responsive organizational operations
Orchestrated Ecosystem Responsive Operation
The content of digital transformation at different levels
perational) Sales & Marketing)

exist Increase production and profitability Refined Budget and improve efficiency
• Enterprise • number
• Big • number
• Thing Import • Customer Relationship Management Software
• Produce Digital technology integration

(Customer Relationship Management, CRM)
• Tracking program
can (Support Function) Innovation and R&D (Innovation and R&D)

Tune Structure, reduce expenses and improve performance More meaningful large-scale research and development with more basis and strategy
• all software • Establish the R&D direction on the market mechanism
• Simplify communication channels between departments • Accelerate consumer feedback, product thought and design time
• Partial function outsourcing (entrust a professional platform provider) • Encourage strategic communication between departments
Information Technology (Information Technology) Re-examine and construct the one that best matches the transformationITInfrastructure
construction, speed up decision making, and build strong analytical capabilities to become a more sensitive organization
Five aspects to consider in digital transformation
Group force
• water deal with
• Span ability
• Brave Familiar and
• Sensitive operation process Leader of

• User-centered design thinking • Analytical talents who are sensitive to data
And strategy • Service and design talents who are empathetic to customers
Technology and Tools Data Ecosystem Management Digital transformation strategy
• Introduce technology to different operational streams • Data acquisition • Clearly define the value proposition
Process to drive the business model • Data structure • Business analysis, exploration value
New and changed
• Data governance
• Rapid deployment, frequent testing
• • Scaling & Killing
•
It is necessary to carry out the application and value
Data application
Review and maintain the overall construction and development • Further driving digital development
Balance of • Continuous change management and training
Digital culture and ability building
Digital transformation strategy
- Innovations in products, services, processes, and ideas are all part of digital transformation
-Conversion of products and services
- Through digital technology, various industries have the opportunity to improve and develop diversified smart products and
services to make life more convenient
- example:AIMedical monitoring, mobile payment, smart street lights, logistics drones
-Conversion of production process
- Introduce digital technology to change the original production process and link more internal and external information
- Aiming at a large number of highly flexible customized production
- Through remote monitoring, you can grasp the overview of the production process, and find and solve problems through analysis
- Example: data dashboard, smart order, smart machine
-Conversion of production thinking
- Destructive innovation that breaks the mental framework and establishes a new business model
- The society uses collaboration to produce, share and manage sharing/sharing economy
- In a society with extremely low marginal costs, digital goods and services will tend to be free, but they will be
replaced by competition for market share and brand adhesion
- Example: Sharing economy (shared transportation), cloud streaming, subscription sales of application software
Digital transformation strategy
number
Why digitize
https://theme.ndc.gov.tw/lawout/LawContent.aspx?id=GL000198
Computerized operation specification for document and file management (electronic operation, electronic official document) First,
the purpose
(1) To improve the quality of document production, accelerate document review efficiency, master document processing procedures, improve document exchange efficiency,
enhance document management efficiency, facilitate public application and application, and establish a computerized standard operating process and common guidelines for
document file integration , This specification is specially formulated.
(2) In order to achieve the goal of energy saving and paper reduction, each agency's document and file management related operations, such as document production, document review, process
management, transfer and exchange, and file management, should establish a full-process electronic operating system to clearly specify energy saving and reduction. Paper index, moving
towards paper reduction, and ensuring that documents and files maintain availability according to their retention period.
(3) In order to promote the conversion of document files from paper-based operations to electronic operations, the original operating procedures may be simplified.
Fourth, the definition of terms
(14) Integrated exchange center: refers to the level of electronic exchange and centralized management in the electronic exchange structure, which
is mainly constructed in the competent ministries or municipalities, county (city) governments, and integrates related agencies to form a group
Electronic exchange center with centralized management structure. This center also serves as the core of centralized exchange management,
providing electronic exchange management services for the exchanges under its jurisdiction; for external exchanges, it is a distributed exchange
architecture; information technology followsebXMLExchange standards.
(15) Self-built exchange center: Refers to the internal electronic exchange center for official documents established by some agencies on their own; it is
mainly used for internal exchange, and external exchange needs to be exchanged through a gateway.
Why digitize
(16) Electronic files: refer to text or non-text data that can be processed by a computer and comply with the File Law and relevant
laws and regulations.
(17) Interpretation data (metadata): Refers to relevant information used to describe the background, content,
relevance and data control of the file.
(18) Encapsulated file: refers to the file stored in the form of a package that combines an electronic file with its interpretation data and information
to verify the authenticity and integrity of the file.
(19) Authenticity (authenticity): Refers to the legitimacy of the process of generating, collecting and modifying electronic files that can be
identified and ensured.
(20) Completeness (integrity): Refers to the electronic file management process, which should ensure the integrity of the stored electronic file
content, interpretation data and storage structure.
(21) Accessibility (accessibility): Refers to the maintenance of electronic archives and their management system available for use through the
electronic archives preservation mechanism in line with the legal retention period.
Source of information
- The data sources are complicated and the content is diverse. Some of these data are generated/recorded
by humans, and some are generated/recorded by machines
-Some information is stored in government agencies or enterprises, and it is difficult for ordinary people to visit and access
-Some information belongs to external public sources, everyone can freely access
-There are also specific public research databases established through school research institutions
-Research and survey conducted by individuals and their publicly shared data are also included
- Through the Internet that connects nodes from all over the world together, querying and
accessing these data becomes easier
-But because the acquisition of data has become simple, the confirmation of the credibility and validity of the data
content has also become more important
-Because the dimensions and quantity of data have increased significantly, the difficulty of verifying, sorting and
filtering data has also increased, and various auxiliary tools are required.
- Physical media information
-The most common are all kinds of collections in libraries or museums. Most of them are on paper, but also
contain various cultural relics, exhibits, materials or video discs, etc.
-Retrieval and investigation of data requires a lot of manual labor

-For the data that needs to be excerpted, it will be necessary to create a copy by copying, reproducing, transcribing,
etc., and then aggregate and process it in a digital way
-There is usually a question of whether to allow the creation of a copy or whether to authorize the use
-Even if the collection library has electronic digital archiving of the cultural relics in custody, it is usually
necessary to apply for the authorization to read and back up the relevant electronic data
-Some of the physical cultural relics exhibited have begun to incorporate virtual reality technology to reconstruct the
physical three-dimensional scanning form combined with surface images into the display in the virtual space, so that
visitors can see throughVRorARWatch formally, even interact with it
- Enterprise/Organization Information
-Data accumulated in enterprises or institutions, mainly in database (Data Base)Or data

warehouse (Data warehouse)Accumulate
-Mainly derived from various information systems in enterprises or government agencies
- Including sales system, human resource management system, purchase and sales inventory system, customer relationship management system,
enterprise resource planning system, supply chain management system and public website, etc.
-At present, government agencies have gradually begun to promote the open and transparent operation of information
-Enterprises of the same quality often share relevant data through the formation of strategic alliances in order to jointly
enhance the development of the industry
- Cloud data／Network data

-There are many sources and types of cloud/network data, such as online articles, comments, public personal
data displayed on various social networking sites, and public data on personal/corporate websites, etc.
-When citing network data, pay attention to the authorization of the data or whether the data is publicly usable,
and whether it is properly de-identified to avoid infringement of the privacy of personal data, etc.
-If it is through network exploration, when the records left by network users on the network platform are
automatically retrieved through the program for analysis, you should also pay attention to related issues
-Generally, the collected relevant data will be sorted and converted into statistical data basis (the original data will not
be disclosed), or the relevant quotation will be made after obtaining authorization
- Open information
-Open Information (Open Data)Mainly derived from the process of the scientific community to disclose the research data to
other researchers for follow-up research
-After the Internet began to flourish, contributors began to publish various materials or content on the
Internet, and community-based collaborative writing such as Wikipedia began to appear
- WikiContributors from all over the world co-edit or modify entries, and assist in maintenance and review through the
community to avoid malicious tampering. It can even be regarded as a network system that reflects human knowledge
(Organization for Economic Co-operation and Development, OECD)

- Organisation for Economic Co-operation and Development The member states of China also signed a joint statement, stating that
among the member states, the information collected by public agencies must be publicly shared
- Mainly for the cooperation and development of related materials such as meteorology, ecology, economy, geography, etc., and does not include
military or other confidential matters involving national affairs
-At present, many government agencies and non-profit organizations have successively disclosed various information
in specific fields on the Internet, so as to allow citizens and all human beings to jointly supervise related public
affairs or issues, and help jointly maintain the environment, economy, and culture. issue
- Internet of Things
-Internet of Things (Internet of Things, IoT)The development of technology allows researchers to record various data
through various detection devices, and even combine these sensing devices with equipment to complete research or
ensure production efficiency through continuous monitoring of the environment and equipment status
-The continuous development of technological applications and services corresponding to the Internet of Things, coupled with the evolution
of artificial intelligence technology, the combination of the two presents the Smart Internet of Things (Artificial Intelligence with IoT,
AIoT), So that the Internet of Things can not only receive data and connect devices, but also use artificial intelligence to assist in
analyzing data and assist in decision-making and judgment. This has become the key to continuous expansion towards the Internet of
Everything
-along withAIoTIn addition to the data collection for the environment and equipment, the development of the company has
increased the demand for the collection and analysis of the user’s usage data and habits. The big data analysis results are
transformed into services that are closer to the user’s habits, which can further allow IoT products make more accurate
predictions and judgments to improve the intelligence and efficiency of products
- However, this part is quite easy to accidentally infringe the privacy of users, and there are also problems of infringements
that would be caused if these private data are leaked.
How to collect data
- In addition to collecting data through the Internet and open source databases, it is mainly
achieved through experiments, observations, and setting up data collection nodes.
-The experimental method of collecting data is to design a statistical experiment (experiment) to obtain
information about the influence of these factors on the research variables based on the changes of
certain controllable factors. And keep those conditions other than the control factors unchanged, or
use randomization methods to balance and offset the effects of other factors.
-Observational research is to objectively record the condition of the researched object through observation or
interview without any intervention measures on the research object. Because there is no intervention, it is
also called non-experimental research.
-The data collection node (data collector) collects data directly through the environmental sensor,
by setting a fixed collection time interval, regular measurement and recording
- Use local (Local)Stored offline data collection node

- Use the cloud (Cloud)Stored online data collection node
- Hybrid data collection node
How to collect data
010110000111001011
1110101001010111...
Cloud server
Cloud Server
IoT sensor End user equipment

IoT Sensors Client Devices
How to collect data
- Data collector (Data acquisition, DAQ), Also known as monitor

-A device that samples the signal that measures the physical conditions of the real world and
generates a digital value that can be operated by a computer
-The main components include

- Sensor: Convert physical parameters into electrical signals
- Signal modulation circuit: convert the sensor signal into a retrievable form
- Analog-to-digital converter (A/D converter, ADC):
- material
How to collect data
How to collect data
-Data collection used to be carried out through special equipment. In recent years, due to the prevalence of self-producer activities,
related measurement modules and microcontroller equipment have flourished, and the required sensing modules or
communication modules can be selected by themselves through the microcontroller. Compose control program in the integrated
development environment and construct self-organized data collector
- Modularized various sensing and actuation modules can be used in combination with a basic
microcontroller to form a small embedded system, and can also be used with various current small
and low-power single-board computers.
-The basic unit of the computer is chip-integrated on a single circuit board, so it is called a single-board computer
-Mostly open sourceLinuxOperation Complex processing, even straight
Connect to a small server
how material
http://www.janspace.com/b2evolution/arduino.php/2010/06/26/scooterputer
How to collect data
Data collection and processing Data library Material visualization
Calculation and analysis
Single Board Computer (SBC)

(Single Board Computer)
Data sorting
- In daily life, whether it is work in all walks of life, or when performing various
activities, you will contact and even generate data
-These data sources are diversified, diverse, and large (big data materials)
-In order to correctly interpret the implicit meaning of the data, transform it into efficient decision-
making or produce valuable results, which is the main purpose of big data analysis
-However, some of the data is visible, and some are disorganized, or it may be a fixed large
number of tracking records for a fixed thing
-Therefore, it is necessary to sort out the data before analyzing
- The organization of data is mainly to filter out useful and correct data before storing. On the one
hand, it can save storage space and on the other hand, it is convenient for direct use.
-Before storing, perform classification processing and divide into sub-data clusters for storage, which can be
indexed more quickly when recalling data, or only directly access the required part
Data sorting
- Common data sorting/data sorting (cleaning and amplification) mainly include

-Clear the unreasonable part of the data
- Identify missing values and outliers
- Eliminate invalid (harmful/abnormal) data
- Fill in the missing parts

-Data adjustment
- Data normalization (Normalization)／Standardization (Standardization)

- Data dimension conversion
-Construction of data features (create, add and delete)
Data sorting
- Missing value
-The missing (unanswered) part can be directly confirmed for the data source of the scale type
-Measurement type data may appear suddenly0Jump back to the amount close to the previous one or directly
appear an unrecognizable value (N/A), Then this pen is usually treated as a missing value
- Usually use the mean value of the before and after data for linear interpolation filling operation
- If it is too large or too small, then jump back to the amount close to the previous one. Generally, it is treated as an outlier
-Numerical data can use statistics (mode imputation, mean imputation or regression imputation) to assist in analysis
and data filling; while categorical data is more difficult to perform, and it is generally not recommended to fill in
but directly record the gaps
- If there are too many missing values in the whole data, it will be regarded as invalid data and excluded
Data sorting
- Outliers
-If you can grasp the data type and content, determine the possible range of values and the data type of the answer
content, you can use this to confirm the existence of outliers
- For scale statistics, the scoring option is only1~5, But it appears0(No answer) or6This kind of out-of-range value;
or the occurrence of text in the numerical data; or the logical fallacy of the mutually exclusive items in the
check item at the same time, these all mean that this item is an outlier
- It is necessary to confirm whether the data containing this answer (the subject’s answer) is invalid (harmful/abnormal) data
(a single question is incorrectly filled, a single question cannot be answered, or the entire scale is filled in randomly), and
then confirm Whether the whole pen is not used (chaotic answer part)
-If the data is continuous data, if there is a particularly large error compared to the nearby
numbers, you can observe the entire data and confirm whether it is an abnormal value
- In the measurement data, there is a sudden sudden change that rises to a high place or falls to a low place in an instant
(Peak)Time, it may be an abnormal value caused by interference or measurement error
- According to whether the measurement target will produce such a large change in an instant (such as the temperature will not change rapidly), it
can be judged whether to remove the abnormal value directly, or to refresh the whole data by means of filtering and other methods, thereby
suppressing the measurement noise
Data sorting
- Data filtering (delete invalid data)

-Will first mark suspicious data by checking missing values and outliers
-Assess whether there is a need to delete the percentage of missing values in the whole
- Generally considered to exceed10%Is missing or in the scale10%Missing or failure to answer will affect the deviation of this
sample, so it will be eliminated or separately compared with the statistical results before and after the elimination, and
then confirm whether the results are referential
-Evaluate the location or content of outliers

- If the frequency (proportion) is not high, it can still be included in the use
- Sometimes an assessment question is inserted directly into the scale (for example, directly instructing this question to select a certain value), and
then you can confirm whether to delete the data by screening the question whether it is abnormal
- Reliability and validity testing can also be used to evaluate whether to retain
Data sorting
- When comparing and analyzing the two sets of data, it may be due to the different units (height and
weight), or the different representations of the figures (number of people and proportions), resulting
in different degrees of change, which will affect the results of the statistical analysis.
-Through data normalization (Normalization)And data standardization (Standardization)

Convert raw data into dimensionless (Dimensionless)After the scalar, compare and
analyze the data
-In addition to improving the convergence speed of the model (reducing the convergence time of the gradient descent method), it
can also improve the accuracy of the model (allowing each eigenvalue to make a similar contribution to the result)
-Data standardization, which can convert the original data into a pattern conforming to the standard normal distribution
(average =0, Standard deviation =1)
-Informative Keep it original middle
Feature engineering
- The whole process of feature engineering includes data selection, pre-processing, conversion, data analysis,
interpretation and evaluation, etc. It can be divided into the following steps
-Remove noise and inconsistent data (Data Cleaning)

-Integrate data from different sources (Data Integration)
-Select the data related to the analysis content (Data Selection)
-Convert the data into an appropriate format (Data Transformation)
-Find out rules/patterns with specific calculation methods (Data Mining)
-Determine the usefulness of the rule/mode (Pattern Evaluation)
-Perform visual presentation (Knowledge Presentation)
Feature engineering
- Due to the rapid increase in the amount of data, attributes and dimensions, a large amount of data can be
collected and stored to create a sea of big data
-But the large amount of accumulated data not only takes up storage space, but also cannot directly
generate or add value, so data analysis and extraction are required
-Data mining (Data Mining)It is the rapid development in recent years to find out the trend characteristics
and correlations hidden in the data, which can dig out new facts from the existing data and discover new
relationships that the experts do not yet know.
- Data mining is to discover meaningful patterns or rules, and automatically or semi-automatically explore
and analyze data from a large amount of data, so as to select the effective, new, potentially useful and
ultimately effective data from the data. The process of patterned data
- Data mining is often referred to as mining knowledge in the database (knowledge

mining from databases) , Knowledge extraction (knowledge extraction) , Data rule
analysis (data/pattern analysis), Data Archaeology (data archaeology), Data collection
(data dredging)Wait
Feature engineering
- Feature extraction (Feature extraction)

-Feature extraction is a process of constructing derived values that are informative and non-
redundant from an initial data set
- Feature extraction is a data sorting method that handles the combination of variable groups and maintains sufficient accuracy of data
-Feature extraction is aReduce dimensionalityThe initial data set is reduced to a more manageable
group (characteristics) for analysis, but at the same time the accuracy and completeness of the
original data set can be maintained
- For the processing of more complex image data with a large amount of information, there are many specialized feature
extraction algorithms, and the development of computer graphics and image processing disciplines
-Analysis of the extracted sub-data sets will usually be carried out for analysis propositionsFeature
selectionTo exclude irrelevant, redundant or highly related features
-If the extracted data features are qualitative data, additional data coding may be
required (give corresponding values for processing and analysis)
Feature engineering
- Data dimension conversion
-For a large number of high-dimensional (multi-feature) big data data, dimensionality reduction can reduce the amount of
model calculations and reduce model execution time, and it can also reduce the impact of noise variable information on
the model results (but not all high-dimensional data Data dimensionality reduction is required)
-It can also be structured through the connection between the objects/meanings represented by dimensional variables, and
organized into a hierarchical multi-dimensional matrix/data table that is easy to understand and analyze.
- Online analysis processing analysis is especially commonly used in website analysis and digital
marketing (MDA)To generate a relational database to assist in the analysis of complex data,
which is often used in business intelligence (BI)among
- The original data will be parsed into several dimensions through different indexes
Data box (Cube)To facilitate inspection and analysis

- The associated database will be created by aggregating accumulated information
Fact sheet (Fact Table), And then through the dimension structure and
index key (Key)Create multiple dimension tables associated with them (
Dimension Table), And connected with different structures, while
removing homogeneous redundant data
Feature engineering
- Data dimensionality reduction based on feature selection
- Rule of thumb:
Select or exclude dimensions based on experience on the basic characteristics of the data and the influence on later data
processing and modeling
- Test calculation:
Participate in the calculation by continuously testing multiple dimensions of choice, and use the results to repeatedly
verify and adjust and finally find the best feature to retain
- Statistical Analysis:
Analyze the linear correlation between different dimensions through correlation, and manually remove or filter in
the highly correlated dimensions
- Machine learning:
The feature values or weights of different features are obtained through algorithms, and then larger features are selected according to the
weights, and dimensionality reduction is performed on the basis of retaining the original dimensional features
Feature engineering
- Data dimensionality reduction based on dimensional transformation
-Principal component analysis (PCA)
-Independent component analysis (ICA)
-factor analysis (FA)

-Linear Discriminant Analysis (LDA)
-Nuclear principal component analysis (Kernel PCA)
- Map the data points in the high-dimensional space to the low-latitude space through a mathematical model of a given set of related
variables, and then use the characteristics of the mapped variables to represent the overall characteristics of the original variables
- Mainly divided into two categories: linear dimensionality reduction and nonlinear dimensionality reduction
-Multi-dimensional analysis (MDA)
Feature engineering
- Multi-dimensional analysis (Multi-Dimensional Analysis, MDA)

-Mainly used in relational databases, which is conducive to improving the efficiency of data query and summary
operations, and can provide a variety of views that are conducive to analysis
- First denormalize all data (Denormalization)To create a large data

table (fact data table (Fact Table))
-Through normalization, the amount of data can be simplified and unnecessary updates can be reduced, and the relative query is slow
-Denormalization can merge complex data structures and improve performance, but may cause data duplication and abnormal data
update (there will be a large number of vacancies that must be skipped during update)
- Then through the multi-level structure of dimensional representation, a data point is generated at the intersection of multiple dimensions,
and the corresponding data value is assigned to the point to form a multi-dimensional (three-dimensional) data box (Cube)
- Finally, create the corresponding dimension table (Dimension Table)And the corresponding main
index key (Primary Key),Attributes (Attribute)(Unique and non-repetitive, a narrative field used to
describe this object)
- Each dimension data table is connected to the fact table through the index key, and the fact table is the center,
showing an outwardly expanding star structure or snowflake structure
time
Season one
account January February March Season one
Budget actual Budget actual Budget actual Budget actual
Sales 300 350 300 325 325 325 925 1000

OGS 175 185 175 175 185 190 535 550
gross profit 125 165 125 150 140 135 390 450
Gross margin 42 47 42 46 43 42 42 45
Dimension table
(Dimensional
Table)
Dimension table
(Dimensional
Table)
Dimension table Dimension table
Fact sheet Dimension table
(Dimensional (Dimensional (Dimensional
Table) Table) (Fact Table) Table)
Dimension table
(Dimensional
Table)
Dimension table
Fact sheet Dimension table Dimension table
(Dimensional (Dimensional (Dimensional
Table) (Fact Table) Table) Table)
Dimension table Dimension table Dimension table Dimension table

(Dimensional (Dimensional (Dimensional (Dimensional
Table) Table) Table) Table)
How to store data
- Data storage is the process of using information technology to archive, organize, and share
the large and small digital data you need in your daily life. From applications to network
protocols, documents to media, address books to user preferences, etc. One of the core
elements of data analysis
- Digitized data is mainly stored by converting binary digital electromagnetic data (the presence or absence
of charge) into a storage unit state (the polarity of the magnetic substance / the depth of the dye layer
being etched / the number of electrons in the transistor or the conduction effect).
- The corresponding address/location when data is written and read is through the system for management
operations such as recording, addressing, data partitioning, and cluster allocation
- According to the data type, frequency of use and processing schedule, it is allocated to different spaces or different
media for long-term storage or temporary storage
How to store data
- Data storage has evolved from local storage and mobile storage to connected networks and managed by
software (including virtualization software, storage management and other software-defined storage (
SDS)Technology to separate software and hardware to match them arbitrarily)
-The main implementation of virtualization technology includes networking, resource concentration, user interface, configuration
capabilities, and automatic resource control／Assignment and other tasks require additional management and operating
systems and automation software to create interfaces, monitor, and allocate resources
-Cloud storage is based on virtualization technology, and provides related services after separating the hardware framework. The basic
hardware, management, and operating system software are managed by the service provider, which does not affect the user's
acceptance of services
-Cloud computing is based on cloud storage plus a computing server and related additional functions to
help users directly obtain results from remote servers for analysis and computing
How to store data
- Storage virtualization can improve the performance of storage systems. The main challenge at present is
how to have an open standard and architecture that can integrate storage hardware, software, and
networks.
- The virtual host technology uses physical hardware resources (subjects) through the creation of virtual
machines (VM)(Object), and combine the computing resources of the subject (such as CPU, Memory,
storage space, etc.) as a re-allocable resource library, allowing different objects to dynamically obtain
the required resources according to their needs
- Storage virtualization, virtual hosting and low-latency networks allow service providers to no longer only
provide storage services, but start to cooperate with cloud computing technology to launch various
convenient digital services, making data applications more convenient
How to store data
- Network storage technology uses various technologies including fiber channel, disk array, tape
cabinet, and CD cabinet for physical storage, and then uses management system and software
with virtualization technology.
-Storage Area Network (Storage Area Network, SAN)

- Storage devices connected to the server will be treated as directly connected storage devices by the operating system
- Does not include a sharing mechanism for cache synchronization, which may cause data loss in use
-Network Attached Storage (Network Attached Storage, NAS)

- Provide centralized data storage for heterogeneous network users by providing only operating systems and software
that only provide data storage, data access, and related management functions
- Using file-based communication protocol, the request to access is a section of the abstract file rather than the access
address on the specified storage device
- Can be seen as a private cloud in cloud storage
-Cloud storage (Cloud Storage)

- After the storage space is virtualized and scheduled with management and automation software, the data is uploaded to
the data server via the network
- The resource allocation in the data server is not a quota, but is dynamically managed according to demand
How to store data
- File storage
-Responsible for arranging and reproducing data to users, processing data from top to bottom hierarchical file
directories, usually withNASuse
-Using a specific data transmission protocol to communicate with a block storage system, the data will be
presented in exactly the same style as seen on the client (computer) (visible only)
-Convenient to use, but there may be problems such as excessively long paths, resource consuming, and inefficiency
- Block storage
-Decompose the data into fixed-size data blocks, each block has a unique identifier, a block table needs to be
established for retrieval, and it can be stored in any place, decoupling the data from its storage
environment
-Each block is independent, enjoys complete configuration capabilities, and can be formatted using its own
data transfer protocol and operating system
-No need to process search, index or start applications like file system
-The storage speed is fast and the configuration is flexible, so it is often used as the original server storage or media database
How to store data
- Object storage (Object-based Storage)

-Data is stored in decentralized units called "objects", which can be stored in any place and used in conjunction with
decentralized systems. It belongs to a flat application architecture, which is conducive to large-scale access
-The object contains data and the metadata paired with it (metadata)
-Narrative materials
- Describe the resources used to discover and distinguish meaning, such as titles, abstracts, authors, keywords, etc.
-Structural information
- Describe the content of the data container and indicate how to organize the composite objects in it, such as the order in which the
pages are organized into chapters
-Management information
- Describe information about management resources, such as when and how the data was generated, file types, access permissions,
and other technical information
Data security
- The defense of data security is not limited to the network. The integrity of the storage itself and data backup
are also important issues. In particular, the value of the data itself is becoming more and more important.
The prevention of data damage must also be explored from all aspects.
- Storage-based security protection can be discussed from two aspects
-How to ensure the reliability and availability of data
- Choose appropriate storage media to ensure that the storage environment meets requirements and reduce human interference
- Proper maintenance and use to extend the life of storage media
- To meet the needs and media storage planning, data transfer and backup creation
- Create a remote backup
-The life cycle and confidentiality protection of the data storage process
- Handling, transfer and storage management of physical media or devices
- Digital data encryption, complete access records, storage media and electronic equipment tracking management
- Communication transmission encryption, network information security management and anti-hacking
- Data destruction is internally streamlined, and standard management and processing procedures are established to avoid data leakage
Question discussion
- Glossary
Glossary
- Please explain the difference between data and information
What's the difference between “data” and “information”?
- The connection between symbols, data, information, knowledge and wisdom
Relationships between symbols, data, information, knowledge and

intelligence.
- Broad data classification (category)

Classification of “data”.
- Digitization and Digitization of Data (Technology Digitization)

What's digitization and digitalization?
Question discussion
- Basic operations of data processing

Operations of data processing.
- What are the advantages of data digitization

Why digitize (digitization)?
- What is an embedded system?

What's embedded systems?
- What is the Internet of Things?
What's internet of things (IoT)?
- Please list the three-tier architecture / five-tier architecture of the Internet of Things technology
Please list the three-layer / five-layer architecture of IoT.
Question discussion
- Six Contents of Digital Transformation / Five Aspects to Pay Attention to during Transformation
Six contents of digital transformation.

Five aspects needs to pay attention to when DX.
- Please list five sources of information
Please list five different data sources.
- Try to explain the content of the data

Please explain about the collation of data
- Feature engineering steps

Steps of feature engineering.
- Data storage technology and storage architecture
Data storage technology and storage architecture
Q&A
Department of Refrigeration and Air Conditioning and Energy Engineering, National Chin Yi University of Technology

(03) 數據的收集與存儲.zh-CN.en

Uploaded by

Document Information

Original Description:

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

(03) 數據的收集與存儲.zh-CN.en

Uploaded by

Copyright:

Available Formats

Translated from Chinese (Simplified) to English - www.onlinedoctranslator.

Data collection and storage

- How to collect data

- How to store data

transmission, resulting fromknowledge And accumulate, gradually formwisdom

passed on and developed

speculative content, and gradually form disciplines in a common field

-Mainly used for communication, expression or recording

-It can be seen as an understanding of people, things, and things

-In a narrow sense, it is a numerical feature obtained through observation

etc., which is conducive to sharing, communication and storage

information, personal interpretation and analysis

-Can be regarded as an abstract concept of understanding of data

personal knowledge, opinions and various external factors.

-The accumulation of others' cognition, experience and experience

-Includes: perception, knowledge, memory, understanding, association, emotion, logic, discrimination,

calculation, analysis, judgment, culture, moderation, tolerance, decision, etc.

-Personal cognition, experience and accumulation of experience

collect Processing and development Analyze and generate

Source: Joint Intelligence / Joint Publication 2.0 (Joint Chiefs of Staff)

- In different fields, data and information may be similar or different in definition

-In the discussion of academic topics, data is the unit of information

more people and things, andInformation Is processed and useful data

- The process of processing may make the data meaningful

- The classification of broad data is shown in the figure below

Unstructured data Structured data

Qualitative data Quantitative data

Categorized information Sequencing data Distance data Fixed ratio data

Nominal Ordinal Interval Ratio

Data presentation: data visualization/descriptive statistics

Visualize / Descriptive Statistics

- The characteristics of electronic data

-Fast speed (withMIPS (Million Instruction Per Second)As a unit)

-Ability to store large amounts of data (TB-PB-EB-ZB-YB)(210=1,024Base)

- The units for storing data are in descending order

- 1 Word = 2 bytes = 16 bits

which can include text, pictures, audio, etc.

-Photographs taken by digital cameras are a commonInformation Digitizationmedium

- Digitization/Technology Digitization (Digitalization)

- In various physical systems, what we can observe

The continuing dynamic change is calledAnalog signal

- ThesecontinuousofAnalog signal, After passingSample and

holdAfter the process will becomeDiscrete signal

- Discrete signalPass againQuantifybecomeAfter the

- Binary digital signal can be directly sent to digital

- The basic unit of calculation in the computer is binary bit

Complex computing content

- Basic operations of data processing

-Data creation (Create)：Select electronic file type according to file content

-Data conversion (Conversion): Transfer to a different media

-Data sorting (Sort): Arrange in order

-Data update (Update)：Edit, add or delete data

- Big data (digital data) originally refers to a collection of structured/

distributed through multiple servers (possibly up to thousands in parallel) (computer clusters)

-quantity(Volume): Data size

Data throughput per unit time (Volume) Correctness

Big data data characteristics:

Large, fast, diverse, uncertain

-Large-scale parallel processing (MPP)database

-Cloud computing platform

-Edge Computing Technology

-Extensible storage system based on the Internet