Application of Big Data analytics in the electrical

sector: a real case study

Marco Antonio Bucarelli Francesca Santori
ASM Terni S.p.A. ASM Terni S.p.A. Elissaios Sarmas
Terni, Italy Terni, Italy Decision Support Systems Laboratory,
marco.antonio.bucarelli@asmterni.it francesca.santori@asmterni.it School of Electrical & Computer
Engineering
Salvatore Cipolla Vangelis Marinakis National Technical University of
Engineering Ingegneria Informatica Decision Support Systems Laboratory, Athens
Rome, Italy School of Electrical & Computer Athens, Greece
salvatore.cipolla@eng.it Engineering esarmas@epu.ntua.gr
2023 14th International Conference on Information, Intelligence, Systems & Applications (IISA) | 979-8-3503-1806-7/23/$31.00 ©2023 IEEE | DOI: 10.1109/IISA59645.2023.10345843

National Technical University of

Andrea Natalini Athens Marzia Mammina
Engineering Ingegneria Informatica Athens, Greece Engineering Ingegneria Informatica
Rome, Italy vmarinakis@epu.ntua.gr Rome, Italy
andrea.natalini@eng.it marzia.mammina@eng.it

Abstract — The spread of Artificial Intelligence tools applied
to Big Data will revolutionise many production and
management processes. The electrical sector is approaching this
phase by leveraging these powerful tools in more and more
applications. In this paper, the authors show two applications in
the real case study of ASM Terni developed in the BD4NRG
European Project. The first analytics shown addresses the
problem of optimising the management of energy districts
consisting of renewable energy sources, buildings, electric
vehicle fleets and electrical storage systems using MILP tools.
The second analytics uses electrical, acoustic, thermal and
humidity data from secondary substations to determine
transformer ageing using a stacking regressor.
Keywords — Big data, artificial intelligence, smart grid,
predictive maintenance, optimal management, electric vehicle,
electrical transformers
I. INTRODUCTION
In recent years, the power grid has witnessed a remarkable
transformation driven by advancements in Artificial
Intelligence (AI) and the abundance of Big Data (BD). The
integration of AI techniques and the analysis of vast amounts
of data hold immense potential to revolutionise the
functioning of power systems, enhancing their efficiency,
reliability, and sustainability. This paper aims to delve into the
intricate relationship between AI and the power grid, with a
particular focus on the influence of BD, showing the results of
two tools used in the real environment of Terni electrical
infrastructure.
The power grid, a complex network comprising
generation, transmission, and distribution systems, plays a
pivotal role in ensuring the reliable supply of electricity to
meet the ever-growing demands of modern society. However,
traditional power grid architectures face significant
challenges, including aging infrastructure, intermittent
Renewable Energy Sources (RES), and the increasing
electrification of various sectors. These challenges necessitate
innovative solutions that can adapt and optimise the power
grid's operations in real time [2],[3],[4],[5],[6].
AI, a field encompassing machine learning, deep learning,
and data analytics, has emerged as a powerful tool to address
This work has been partially funded by the European Union’s Horizon
2020 research and innovation programme under the BD4NRG project grant
agreement No. 872613
these challenges. By leveraging AI algorithms, power grid
operators can make data-driven decisions, optimise energy
generation and consumption, improve grid resiliency, and
enable efficient integration of RES. The synergy between AI
and the power grid not only enhances operational efficiency
but also paves the way for the emergence of smarter, self-
healing grid systems capable of autonomously responding to
dynamic conditions.
However, the efficacy of AI techniques in the power grid
heavily relies on the availability and quality of data. The
advent of BD technologies has enabled the collection, storage,
and analysis of vast amounts of information from diverse
sources within the power grid. This wealth of data,
encompassing energy consumption patterns, weather
conditions, equipment health, and customer behaviour, offers
valuable insights to drive intelligent decision-making and
proactive grid management. Using BD analytics in
conjunction with AI algorithms unlocks the true potential of
the power grid, empowering stakeholders to extract
meaningful knowledge and improve system performance
[7],[8],[9],[10].
This paper seeks to explore the multifaceted impact of AI
and BD on the power grid, examining their interplay in various
domains, including load and generation forecasting, non-
intrusive diagnosis, demand side management, energy
optimisation, and asset management. By critically evaluating
existing research and presenting relevant case studies, we aim
to provide a comprehensive understanding of the benefits,
challenges, and prospects of integrating AI and BD in the
power grid.
Ultimately, this research contributes to the ongoing
discourse on the transformation of power systems, providing
valuable insights for policymakers, grid operators,
researchers, and industry professionals alike. The findings
presented in this paper will foster informed decision-making,
facilitate the deployment of AI-based solutions, and guide the
development of strategies that optimize the power grid's
performance while ensuring a sustainable and reliable energy
future.
This research was funded and is based on the activities
carried out in the European Project BD4NRG [1], which
analyses the impact of BD on the energy system, mainly
limited to the Terni pilot.

979-8-3503-1806-7/23/$31.00 ©2023 IEEE

Authorized licensed use limited to: AMITY University. Downloaded on March 23,2024 at 13:54:11 UTC from IEEE Xplore. Restrictions apply.
 The paper is structured as follows: Section II will show
two AI models for the optimal management of a corporate EV
fleet in an energy district and for the predictive maintenance
of electrical transformers. Section III shows the case study of
ASM Terni, which was used as a pilot testing site for the
BD4NRG project, while the results of the applications are
highlighted in Section IV, separate for the two different use
cases. Finally, conclusions are drawn in Section V.
II. MODEL
A. Optimal management of BESS in a microgrid
Energy autonomy in microgrids is a crucial aspect of
ensuring a reliable and resilient source of electricity,
especially in areas with limited or no access to the main grid.
Microgrids can be designed and operated as small-scale,
independent systems, and they are capable of providing
electricity autonomously without relying on the main grid.
Alternatively, grid-connected microgrids are another
application where they can power single buildings, clusters of
buildings, or energy communities, while still having the
ability to draw power from the main grid when their internal
generation is insufficient. The concept of energy autonomy
holds immense value for microgrids, as it empowers an energy
community to generate and utilise its own energy, reducing
dependency on external sources such as the main grid [11]. To
achieve energy autonomy, a microgrid must possess the
capability to generate and store sufficient electricity to meet
the local demand, even when disconnected from the main grid.
RES play a pivotal role in enhancing energy autonomy
levels within microgrids. Adopting RES, such as solar
photovoltaic (PV) panels, wind turbines, or hydroelectric
generators, provides a reliable and sustainable form of energy
[12]. These sources harness the power of nature, reducing
reliance on conventional fossil fuel-based generation
methods. Using RES, microgrids can significantly increase
their energy self-sufficiency and minimise their carbon
footprint. In addition to RES, energy storage systems (ESS)
like batteries and thermal storage technologies play a vital role
in enabling energy autonomy in microgrids. Batteries serve as
an effective means of storing excess electricity generated
during periods of high production and low demand [13]. By
incorporating ESS, microgrids gain the flexibility to balance
energy supply and demand, ensuring uninterrupted power
supply and optimising the utilisation of available resources.
Fig. 1. Methodological Framework for BESS Management
The methodological framework is presented in Figure 1.
The model proposed in this paper aims to optimise the use of
battery ESS (BESS), especially local batteries installed in
microgrids, with the objective of maximising the energy
autonomy levels of the microgrid. The model is based on
Authorized licensed use limited to: AMITY University. Downloaded on March 23,2024 at 13:54:11 UTC from IEEE Xplore. Restrictions apply.
constructing a mixed-integer linear programming (MILP)
problem [14], which incorporates the total load of the
microgrid, the generation of the RES installations, and the
battery's maximum capacity. It should be noted that the model
receives as input forecasts of both the microgrid’s load and the
RES generation. These forecasts are generated by ML and DL
models [15], [16], but this is not within the purpose of this
study. The optimisation function of the model is the number
of hours within the day that the microgrid can operate
autonomously without requiring any electricity from the main.
In these time periods, the microgrid can either directly use the
electricity generated by the RES or use the energy stored in
the ESS by discharging the battery.
The solver used to tackle this problem is Gurobi Optimizer
[17], and the software has been implemented using a Mixed
Integer Programming (MIP) library called Python-MIP [18].
B. Predictive detection of electrical transformers faults
The primary aim of this analytic service is to predict the
ageing of the transformer in order to apply predictive
maintenance and to save costs related to the substitution of the
component. The developed model is applicable to secondary
substation power transformers.
The ageing of the transformer is strictly correlated to the
temperature at the hot point, but since, in most cases, measures
related to temperature are not available, it is necessary to
calculate it by applying a thermal model of the transformer.
Primarily, it’s necessary to calculate apparent power from raw
power by extracting those values from the real-time smart
meter. From apparent power, it is possible, through
mathematical and physical models, to infer the hot-point
temperature of the instrument and, from that, its degradation.
The Python library named Prophet, provided by Meta, allows
the calculation of the ageing [19] [20].
The use case also covers another eventual situation. Power
Smart Meters or Power Quality Analyzers, which are the tools
used to measure power flow in the transformer, are not always
available, but on the other side, acoustic sensors may be more
common than power ones. So, theoretically, is possible to use
audio data to estimate power data, and consequently, the aging
of the transformers.
The tool used for this analytic service is Machine Learning
Regression Model, but first, it is necessary to identify an
appropriate set of points (audio and electric data) that can be
used to train the model. The power that has been inferred in
this way will, then, be used to make a prediction in the same
way as indicated for the normal workflow. All the following
implementation has been done using Python, exploiting useful
and common Data Science and Machine Learning libraries,
such as Sci-Kit Learn [21], Pandas [22], etc.
B.1 Regression Model
The solution implemented in this service consists of three
different components: a Grid Search, a Machine Learning
Layer consisting of three different regressors and one Meta-
Regressor. The Machine Learning Layer consists of three
regression models: a Polynomial Regression model, a Support
Vector Regression, and an Extreme Gradient Boosting
Regressor, commonly renamed XGBoost. The first step is to
find the best parameters for all the models. To avoid arbitrary
setting of the parameters, a Grid Search methodology has been
chosen, using the data to let the model find its best
hyperparameters automatically. Next, each model is trained
separately in order to evaluate the quality of single regressors.
Finally, the Stacking Regressor is the meta-regressor chosen
for this task, and it works by using a model to make a
prediction based on the predictions made by other estimators.
In other words, each of the trained estimators returns a
prediction that will constitute the input of the meta-regressor,
which is then responsible for returning the ultimate estimation.
The described architecture is properly illustrated in Figure
2.

Fig. 2. Regression Model Architecture Fig. 4. Prototype of sensor installed in secondary substations, used in the
second use case

Overall, the sensors used are 14 real time smart meters, 2

III. CASE STUDY
The tools developed in the BD4NRG project and shown in
Section II were tested and validated in the infrastructure of
ASM Terni, which was used as a pilot test site. ASM Terni
S.p.A. is a multi-utility operating in the municipality of Terni,
in central Italy, and provides electricity, gas and water
distribution, and waste collection and management services.
The Terni pilot site is active in research activities on the smart
grid and electric fleet management issues [23], [24].
The infrastructure used to validate the case studies
consists, in the first use case, of a low-voltage electricity grid,
which includes 2 secondary substations, 5 electric vehicle
charging stations, 10 electric vehicles, a 185 kW PV plant and
the ASM headquarters.
AI Analytics
Charging stations
ASM headquarter
Raspberry Pi
E-mobility DSO SCADA
platform System
RES
On Board Device
ESS
EV fleet
Fig. 3. Energy district infrastructure and data flow
The second use case uses data from 10 secondary
substations, in which a prototype of an environmental sensor
has been installed, capable of measuring temperature,
humidity and noise emissions in the cabin and transmitting
them via 4G to ASM's MQTT broker.
power quality analysers, 10 environmental sensors
(temperature, humidity and noise) and 10 onboard devices (for
EVs), corresponding to about 3.75 TB of data.
IV. APPLICATIONS RESULTS
A. Optimal management of BESS in a microgrid
In this section, we provide some indicative results of the
application of the model described in Section II. The results
are presented for a typical day (24-hour span), and they aim to
demonstrate how an instance of the methodology works. It
must be noted that the proposed framework provides a day-
ahead scheduling of the BESS, so the validation process
follows a rolling fashion and is performed day by day.
In Figure 5, we present the output of the application of the
methodology in the ASM Terni use case for a given day. The
blue line describes the production of the PV, while the orange
line describes the consumption of ASM Terni’s building. As
expected, the production is zero during night hours due to a
lack of solar irradiance, while it peaks during the morning
hours. On the opposite, the consumption of the building
follows a more complex motive which is difficult to predict.
It should also be noted that the model takes as input a battery
with a maximum storage capacity of 40 KWh, which is
initially empty.

Fig. 5. Simulated results of the proposed model for one calendar day (24
hours).

Authorized licensed use limited to: AMITY University. Downloaded on March 23,2024 at 13:54:11 UTC from IEEE Xplore. Restrictions apply.
 As seen in Figure 5, the usage of the BESS due to the
optimal scheduling proposed by the MIP models results in
approximately 12 hours of energy autonomy for the ASM
Terni’s microgrid. The hours of energy autonomy can be
considered as the hours that the demand from the grid (red
dotted line) is zero. Such hours are the result of both RES
penetration and BESS usage. Battery starts charging during
the morning hours that PV production exceeds the
consumption of the grid, and thus there is an excess of energy
in the microgrid. On the opposite, the battery starts
discharging during the afternoon hours, and RES production
fails to cover the electricity needs of the microgrid.
Fig. 7. Models Training

B. Predictive detection of electrical transformers faults
Here are reported the results of the architecture described
in Figure 2. Data for this use case are collected from 10
secondary substations provided by ASM Terni, each of them
registering power and noise emitted from the transformer. In
the following analysis, only data from one transformer are
used, but the whole process is applicable to all the remaining
nine secondary substations.
First of all, it’s useful to depict the initial situation,
highlighting the set of points that have been chosen to train the
models.
For reference, in Figure 7, the red line shows the prediction
made by the Polynomial Regression Model, the green line the
Support Vector Regressor Model and the blue line the
XGBoost Model.
The output shows a good overall prediction, and the
quality is confirmed by common metrics for the evaluation of
regressors, which are mean absolute and mean squared errors.
Table 1 reports the metric of all the models illustrated in this
paper.
Lastly, the prediction passes on to the Stacking Regressor,
and the model outputs the final estimated power values. Since
it’s not a “linear” regression model, the depicted line it’s not
smooth as one can expect, but again it shows a good prediction
based on the starting data we have.

Fig. 6. Initial situation. Audio against Power data

Figure 6 shows that there is a correlation between audio Fig. 8. Stacking Regressor Prediction
and power measure. It was obtained by using a single
transformer as a blueprint and then by adding data taken from As we described in the previous section, the final outcome
other sensors to cover all combinations. For a couple of of this analytics service is the ageing prediction, which is
specific and limited intervals of points, proper data depicted in Figure 9. As illustrated, the prediction also has a
engineering techniques (data augmentation) were required in forecasted horizon, the most useful part for a domain expert
order to improve the number of available features, but data that needs to monitor the transformer’s health status.
simulated in this way will not exceed the threshold of 10% of
data, to maintain a sort of balance between augmented data
and real ones. Data has been simulated in a strict interval,
starting from the mean and the standard deviation calculated
on the real power and audio data. Only 100 samples will be
generated out of a total of 636 real and usable samples. The
dataset created in this way, depicted in Figure 6, also describes
a strong positive correlation (Pearson index= 0.75). From that,
it is possible to start the training of each model. Figure 7
reports the predictions made by single regressors.

Fig. 9. Aging Forecasting Prediction

Authorized licensed use limited to: AMITY University. Downloaded on March 23,2024 at 13:54:11 UTC from IEEE Xplore. Restrictions apply.
 TABLE I. METRICS OF THE ML MODELS
Stacking
Metrics Estimators
regressor
Polynomial
SVR XGBoost
Model
MAE 0.221 0.237 0.211 0.0637
MSE 0.061 0.0734 0.567 0.00693
V. CONCLUSION
In conclusion, this scientific paper has explored the
significant potential of applying AI tools to BD in the context
of the electric distribution grid. The integration of AI
algorithms and techniques into the electric distribution grid
has shown immense promise in enhancing grid operations,
improving efficiency, and ensuring reliable power delivery.
Machine learning algorithms have demonstrated their ability
to extract valuable insights from vast amounts of data,
enabling predictive maintenance, fault detection, RES and
load forecasting and grid optimisation.
In the first use case, a MILP problem, combined with
predictive models, is solved to determine the optimal
utilisation of an ESS for an energy district, including RES,
EVs and consumers, to maximise self-consumption and
minimise the carbon footprint. In the second use case, a
stalking regressor model combining three different predictive
models: the polynomial regressor, the support vector
regressor, and the XGBoost regressor, was used to determine
the ageing of electrical transformers at secondary substations
based on electrical, acoustic, and temperature and humidity
data. Both use cases were applied on real data from ASM
Terni within the European BD4NRG project context
[25],[26].
The successful application of AI tools to BD in the electric
distribution grid heavily relies on data quality, availability,
and accessibility. For future research, it is proposed to further
investigate what the applications and benefits are for the use
of AI tools in the power grid by applying it to a higher grid
infrastructure than that used for the pilot test site. In
conclusion, the application of AI tools to big data in the
electric distribution grid has the potential to revolutionise the
way power systems are managed and operated. Through
continued research and collaboration, it is possible to build a
more sustainable and resilient grid for the future.
REFERENCES
[1] Big Data for Next Generation Energy, available online at
https://www.bd4nrg.eu
[2] Malik Ali Judge, Asif Khan, Awais Manzoor, Hasan Ali Khattak,
Overview of smart grid implementation: Frameworks, impact,
performance and challenges, Journal of Energy Storage,Volume 49,
2022, 104056, ISSN 2352-152X,
https://doi.org/10.1016/j.est.2022.104056.
[3] M. Ourahou, W. Ayrir, B. EL Hassouni, A. Haddi, Review on smart
grid control and reliability in presence of renewable energies:
Challenges and prospects, Mathematics and Computers in Simulation,
Volume 167, 2020, Pages 19-31, ISSN 0378-4754,
https://doi.org/10.1016/j.matcom.2018.11.009.
[4] Yeliz Yoldaş, Ahmet Önen, S.M. Muyeen, Athanasios V. Vasilakos,
İrfan Alan, Enhancing smart grid with microgrids: Challenges and
Authorized licensed use limited to: AMITY University. Downloaded on March 23,2024 at 13:54:11 UTC from IEEE Xplore. Restrictions apply.
opportunities, Renewable and Sustainable Energy Reviews, Volume
72, 2017, Pages 205-214, ISSN 1364-0321,
https://doi.org/10.1016/j.rser.2017.01.064.
[5] Dongming Fan, Yi Ren, Qiang Feng, Yiliu Liu, Zili Wang, Jing
Lin,Restoration of smart grids: Current status, challenges, and
opportunities,Renewable and Sustainable Energy Reviews,Volume
143,2021,110909,ISSN 1364-0321,
https://doi.org/10.1016/j.rser.2021.110909.
[6] M. Faheem, S.B.H. Shah, R.A. Butt, B. Raza, M. Anwar, M.W. Ashraf,
Md.A. Ngadi, V.C. Gungor, Smart grid communication and
information technologies in the perspective of Industry 4.0:
Opportunities and challenges, Computer Science Review, Volume 30,
2018, Pages 1-30, ISSN 1574-0137,
https://doi.org/10.1016/j.cosrev.2018.08.001.
[7] Mladen Kezunovic, Pierre Pinson, Zoran Obradovic, Santiago
Grijalva, Tao Hong, Ricardo Bessa, Big data analytics for future
electricity grids, Electric Power Systems Research, Volume 189, 2020,
106788, ISSN 0378-7796, https://doi.org/10.1016/j.epsr.2020.106788.
[8] Dhanya Therese Jose, Jørgen Holme, Antorweep Chakravorty,
Chunming Rong, Integrating big data and blockchain to manage energy
smart grids—TOTEM framework, Blockchain: Research and
Applications, Volume 3, Issue 3, 2022, 100081, ISSN 2096-7209,
https://doi.org/10.1016/j.bcra.2022.100081.
[9] Noha Mostafa, Haitham Saad Mohamed Ramadan, Omar Elfarouk,
Renewable energy management in smart grids by using big data
analytics and machine learning, Machine Learning with Applications,
Volume 9, 2022, 100363, ISSN 2666-8270,
https://doi.org/10.1016/j.mlwa.2022.100363.
[10] Abduallah Ayub Khan, Asif Ali Laghari, Mamoon Rashid, Hang Li,
Abdul Rehman Javed, Thippa Reddy Gadekallu, Artificial intelligence
and blockchain technology for secure smart grid and power distribution
Automation: A State-of-the-Art Review, Sustainable Energy
Technologies and Assessments, Volume 57, 2023, 103282, ISSN 2213-
1388, https://doi.org/10.1016/j.seta.2023.103282.
[11] Debouza, M., Al-Durra, A., EL-Fouly, T. H., & Zeineldin, H. H.
(2022). Survey on microgrids with flexible boundaries: Strategies,
applications, and future trends. Electric Power Systems Research, 205,
107765
[12] Sarmas, E., Spiliotis, E., Marinakis, V., Tzanes, G., Kaldellis, J. K., &
Doukas, H. (2022). ML-based energy management of water pumping
systems for the application of peak shaving in small-scale
islands. Sustainable Cities and Society, 82, 103873
[13] Kerdphol, T., Fuji, K., Mitani, Y., Watanabe, M., & Qudaih, Y. (2016).
Optimization of a battery energy storage system using particle swarm
optimization for stand-alone microgrids. International Journal of
Electrical Power & Energy Systems, 81, 32-39
[14] Sarmas, E., Xidonas, P., & Doukas, H. (2020). Multicriteria portfolio
construction with python. Berlin/Heidelberg, Germany: Springer
[15] Sarmas, E., Strompolas, S., Marinakis, V., Santori, F., Bucarelli, M. A.,
& Doukas, H. (2022). An Incremental Learning Framework for
Photovoltaic Production and Load Forecasting in Energy
Microgrids. Electronics, 11(23), 3962
[16] Sarmas, E., Dimitropoulos, N., Marinakis, V., Mylona, Z., & Doukas,
H. (2022). Transfer learning strategies for solar power forecasting
under data scarcity. Scientific Reports, 12(1), 14643.
[17] https://www.gurobi.com/
[18] https://www.python-mip.com/
[19] Taylor S. J., Letham B. 2017. - Forecasting to Scale. PeerJ 5 Preprints:
e3190v2 https://doi.org/10.7287/peerj.preprints.3190v2
[20] https://facebook.github.io/prophet/
[21] https://scikit-learn.org/stable/
[22] https://pandas.pydata.org
[23] Bragatto, T.; Bucarelli, M.A.; Bucarelli, M.S.; Carere, F.; Geri, A.;
Maccioni, M. False Data Injection Impact on High RES Power Systems
with Centralized Voltage Regulation Architecture. Sensors 2023, 23,
2557. https://doi.org/10.3390/s23052557
[24] Bragatto T., Bucarelli M.A., Carere F., Cavadenti A., Santori F.,
Optimization of an energy district for fuel cell electric vehicles: Cost
scenarios of a real case study on a waste and recycling fleet,
International Journal of Hydrogen Energy, Volume 47, Issue 95, 2022,
Pages 40156-40171, ISSN 0360-3199,
https://doi.org/10.1016/j.ijhydene.2022.08.114.
[25] Sarmas, E., Spiliotis, E., Marinakis, V., Koutselis, T., & Doukas, H. [26] Sarmas, E., Marinakis, V., & Doukas, H. (2022). A data-driven
(2022). A meta-learning classification model for supporting decisions multicriteria decision making tool for assessing investments in energy
on energy efficiency investments. Energy and Buildings, 258, 111836. efficiency. Operational Research, 22(5), 5597-561.

Authorized licensed use limited to: AMITY University. Downloaded on March 23,2024 at 13:54:11 UTC from IEEE Xplore. Restrictions apply.