DiBERT: Short-text Classification in Disaster
Incidents using transformer-based bidirectional
model
I. INTRODUCTION
In recent years, the frequency and severity of major natural
disasters have escalated, drawing global attention to the
catastrophic effects of earthquakes, hurricanes, wildfires, and
floods.
Each disaster type brings its unique challenges and
consequences [1]. The increasing incidence and severity of
natural and artificial disasters worldwide necessitate a robust
and comprehensive approach to disaster management.
Disaster systems embody the structured and coordinated
strategies, technologies, and policies designed to mitigate
the impacts of disasters, enhance resilience, and facilitate
recovery. These systems span preparedness, response,
recovery, and mitigation phases, involving various
stakeholders, including government agencies, non-profit
organizations, communities, and international bodies.
This paper aimed to analyze short texts on disaster-related
incidents, such as earthquakes, hurricanes, and floods. The
dataset came from HumAid Twitter dataset [2], which is a
comprehensive collection of more than 77000 tweets that
have been manually tagged and compiled from 19 significant
natural disasters, including earthquakes, hurricanes, wildfires,
and floods, occurring worldwide between 2016 and 2019.
The dataset is categorized into various humanitarian themes
and contains exclusively English tweets, making it the most
extensive dataset in crisis informatics.
Natural Language Processing (NLP) has emerged as a
trans- formative tool, revolutionizing how information is
processed, interpreted, and acted upon during crises [3]. By
analyzing vast amounts of data from diverse sources such as
social media, news reports, and direct communications, NLP
technologies can identify patterns, trends, and crucial
information that
might otherwise go unnoticed. This capability allows for real-
time monitoring of situations, early detection of emerging
threats, and the dissemination of critical information to both
responders and the affected populations.
The study is focused on the multi-classification of short-
text for disaster management response using humanitarian
categories provided on the HumAid dataset.
The texts will undergo a series of transformations as part
of the preprocessing. Text normalization will be used, such
as removing stop words, punctuation symbols, retweet tags,
HTML tags, URLs, emoticons, and conversion to lowercase
letters. Other techniques for preprocessing will be used if
necessary.
The authors will utilize and improve the DeBERTa model
to embed the words. It integrates absolute position
embeddings of words just before the softmax layer,
enabling the model to decode the masked words by
leveraging the combined contextual embeddings of the
contents and their positions [4].
The preprocessed data and the word embeddings will be
processed using Bidirectional Long Short-Term Memory
(BiL- STM). Hyperparameter tuning will be done to improve
the model. The results will be evaluated using accuracy,
precision, and recall. Other evaluations will be done if
necessary.
The authors will compare the proposed solution with the
baseline models [5] such as Support Vector Machine, Random
Forest, and XGBoost.
This can be used by disaster management organizations to
efficiently identify the need given the situation and accelerate
the response.
 II. METHODOLOGY
A. Architecture
1) Micro BERT: A bi-directional encoder transformer
(BERT) was designed from native PyTorch functionality and
compared to the performance of other pre-trained BERT
models. Let this custom BERT model be called “Micro BERT”
as it is by far the smallest BERT models used to model the
disaster classification task. Micro BERT was trained on both
the normal BERT tasks of masked language modeling (MLM)
and next sentence prediction (NSP) similar to base BERT [6].
To derive the dataset for each of these tasks the first 2 mil-
lion sentences from Google’s Books Corpus was used. Basic
prepossessing was done on these sentences that match the pre-
processing done to the disaster dataset we will be modeling
later. This included casting all characters to lowercase, and
tokenization (used nltk word tokenize function). the specific
tokenizer used was a byte pair encoder that encoded word
pieces founds in the corpus of sentences. The vocabulary size
of this tokenizer was set to 15,000-word pieces (about half the
size of a base BERT model’s vocab of approximately 30,000-
word pieces) [6]. Next, the preparation of the dataset for the
MLM task took place, 15% of the tokens in the sentences
were selected for masking [6]. Of these tokens selected for
masking 80% of them were replaced with the “Mask” token,
10% were replaced with another random token from the
vocabulary, and the final 10% were left unchanged [6]. An
example of this process done on each sentence is shown in
figure 1. Note that in the actual BERT model sub-words
pieces would be used instead of the entire tokens (like shown
in the figure 1 example) [6]. This resulted in many encoded
sentences form the books corpus. The label vectors for this
MLM task were also generated. Label vector has zeros for all
elements that were not masked and has the original word
index at all the masked tokens indices.
Next the dataset was prepared for the NSP task. these
encoded sentences were paired together and labeled if they
followed each other (meaning the first sentences index as
N and the next sentences index was N + 1: labeled as 1)
for the training of the NSP task [6]. Approximately 40,000
sentence pairs were made that followed each other in the
text. Approximately 40,000 sentences pairs were made that
did not followed each other in the text (the first sentences
index as N and the next sentences index does not equal
N + 1, labeled as 0). This pairing of the sentences was
also done on the masked sentence labels. Segment encoding
vectors were also derived for each sentience pairs were these
vectors had a 1 at each index was a token from the first
sentence was present and a 2 at any index were a token
from sentence 2 was present. A consideration is that only
sentence pairs where both sentences token lengths that were
longer than 10 tokens were chosen. This was done in order
to cut down on the total 2 million sentences extracted from
the books corpus and ensure information dense sentences were
selected. It was not feasible to train the model on hundreds
Fig. 1. This figure shows an example of how a tokenized sentences are
transformed into the masked sentences, label sequences, and attention masks.
The BERT model performs masking on .15 of the tokens in each sentence. Of
these tokens selected for masking 80% are replaced with the [MASK] token,
10% are replaced with another random token, and 10% are left unchanged.
The label vector has zeros at all indexes that were not masked and the original
words at masked indices. The attention mask has 1 ones at all indices that are
not a [PAD] token (the example does not include pad tokens therefore it is a
vector of all ones.
of thousands to millions of sentence pairs at the currently
accessible computation resources.
A total of 81908 sentence pairs were created
(approximately 40,000 pairs followed each other while
approximately 40,000 pairs did not) then the CLS tokens was
added at the beginning of the first sentence, the SEP was
added at the end of the first sentence and end of the second
sentence. Attention was added to the attention mask to tell the
attention layers to attend to the added CLS and SEP tokens.
Since the model will never predict the occurrence of a CLS or
SEP tokens, zero were added at these indices in the masked
sentence labels. A vector that labels any token from the first
sentence as 1 and any token from the second sentence as 2
was created. Finally, the mask labels vectors were also
created. All were padded to 512 tokens in length uses the
integer 0 as the pad token in all of the data structures since
this is the size of text our BERT model will be able to input.
Also, attention mask vectors were also created for each
sentence pair in the multi-headed self-attention layers. These
vectors have 1 at all indices that are not a pad token in the
original tokenized sentences.
The final dataset that was input into the BERT transformer
included input ids of the masked sentence pairs, an attention
mask of these original sentences, the segment encoding vector
of these sentence pairs, the labels for the masked tokens in the
original sentence pairs, and the labels for the NSP task if the
sentences logically follow each other for all 81,908 sentence
pairs. This dataset was then partition into 70% train, 15%
validation, and 15% test splits. It was also broken up into 16
sample batches.
Micro BERT employed an embedding layer to embed the
vectorized input ids. This embedding layer had the vocab
size used in the tokenizer (15,000 subword tokens which is
approximately half of the 30,522 subword tokens of BERT

Fig. 2. This figure shows how the masked sentences are combined together
by adding the [CLS] and [SEP] tokens. This is also done for the masked
labels (it adds zeros at all [CLS] and [SEP] indices and the attention mask
(it adds ones at all [CLS] and [SEP] indices
base) and the embedding dimension of each token was kept
very small at 128. This is a significantly smaller embedding
dimension of BERT base at 768. The embedding layer had to-
tal 1,920,000 parameters. The model used another embedding
layer to embed the segment encoding vectors but it had the
same dimension but only a vocab size of 3 since we only had
zeros (padding indices), ones (sentence 1 indices), and twos
(sentence 2 indices) as elements in the segment embedding
vectors. This layer had 384 parameters. Also, a positional
embedding vectors (had the same embedding dim of 128 and
length of 512) were also generated followed the sine and
cosine function [7]. The positional embedding vectors has 0
parameters. The sentence embedding, the segment embedding,
and the positional embedding were all added (via matrix
addition) and a dropout of .2 was applied to this resulting
matrix to produce the input into the encoder blocks [6].
The encoder block took this input and performed multi-
headed self- attention (all tokens can see all other tokens in
the input sequence effectively making this a bi-directional
calculation of attention scores) deriving the key, query, and
value (via the respective key, query, and value weight
matrices) from the embedded representation [7]. The attention
output is calculated from the key (K) query (Q) and value (V),
and dimension of key vector (dk) (via the formula outlined in
the paper” Attention is all you need” [7].
QKT
Attention (Q, K, V ) =softmax √
Vd
k
The multi-headed aspect comes from this process of deriving
attention scores is done multiple times for different parts of
the embedding vectors over the entire input sequence. For
example, attention head one performs the attention calculation
over the entire sequence but only using indices 1 through 8 of
the word embedding [7]. Attention head two performs the
same attention calculation over the entire sequence bu using
instead indices 9 through 16. These outputs from multi head
are then concatenated together after that attention score
calculations
Fig. 3. This figure shows the sine and cosine equations used to
generate the positional embedding vectors at different token positions (pos)
at each different index in the embedding dimension (i) [7]. The variable
dmodel is the dimension of each embedding vector [7].
Fig. 4. The diagram on the right shows how to calculate the output of
self-scaled Dot-Produce attention. The right shows a diagram on how multi-
headed attention which performs attention score calculation of sections of the
embedding (example head ones take all the tokens embedding from index 0
to 9 while head two takes all the tokens embedding from index 10 to 29 [7].
to produce the same dimension output as the input into the
attention layer. Micro BERT used a multi-headed attention
layer with 8 heads. The number of parameters in the attention
layers in Micro BERT was 66,038. A dropout of .2 was
added to the resulting the resulting hidden states (keys,
query’s, values ...) that were calculated in the multi headed
attention). However, the attention masks were applied to the
embedding as to ignore pad embedding in all of the sequences
making this computation more efficient and focused. The
output from attention layer also has .2 dropout applied and is
then added to a residual connection of the input of the
attention layer via matrix addition and then undergoes a layer
normalization layer with 256 parameters (since the output dim
was still 128 resulting in 128 weight and 128 bias in the layer
norm) [6].
The output from the first half of the encoder block was then
passed to a feed forward neural network with a input layer the
size of the embedding dim (128 in Micro BERT), a hidden
layer the size of the feed forward dim (128 in Micro BERT)
with Gaussian Error Linear Unit (GELU) activation, a dropout
of .2 between the hidden and output layer, and a output layer
with the size of the original embedding dimension (128 in
Micro BERT). The feed-forward network had a total 33,024
parameters. This output has .2 dropout applied again and is
then residual connected with the input to the feed-forward
neural network and undergoes another layer normalization
with again 256 parameters. The output of this final layer
normalization is the output from this encoder block [6]. Each
encoder block in Micro BERT has a total of 99574 parameters.

Fig. 5. The box in the diagram surrounds the encoder architecture that Micro
Bert uses [6].
A diagram of the encoder block in it entirely can be seen in
figure 5.
Micro BERT employs 3 different stacked encoder blocks
where the output from the previous block is the input to the
next block. The parameters for these blocks are distinct from
each other but the architectures are identical. The output of the
encoder blocks is then passed to a distinct head based on the
task the model is performing. When the model was
performing MLM the output of the encoder (each specific
embedding of each token in particular) was mapped to a
MLP head with a hidden dim of 128 neurons paired with
ReLU activation and dropout of .2. This hidden state was
them mapped to a output layer the size of the vocab (15,000 in
the case of Micro BERT). This MLP head had a total of
1,951,384 parameters. This output sequence of distributions of
each token was then compared to the masked label vectors
explained above using cross entropy loss function that ignored
pad tokens. This loss was then used to adjust the weights and
biases of the MLM MLP head, encoder, and embedding layers
in the model.
Micro BERT used a different MLP head when training the
model on the NSP tasks. This MLP head only used the CLS
token at the beginning of each sentence as the only embedding
that was passed to the MLP head. This CLS embedding token
was passed to a hidden MLP layer of 128 (with corresponding
relu activation and dropout of .2) and finally to softmax layer
of 2 neurons (a 2 neuron softmax proved to result in higher
accuracy on NSP that using a 1 neuron output layer of
sigmoid activation). This output distribution was again
compared to the label vector for the NSP tasks and the
loss/gradient was
calculated and used to adjust the weight and biases of the
MLP head, encoder, and embedding layers. This MLP head
16,770 parameters.
Finally, Micro BERT included an adjustable MLP head that
performs sequence classification via the CLS token for the
disaster dataset that will be done after training. This MLP
head took the cls embedding and mapped it to a hidden dim
of 128 (with relu and dropout) and an output layer of 10 since
there are 10 classes in the disaster dataset. This MLP head a
parameter count of 17802.
The entire model (including all the different MLP heads)
contained 4,205,476 parameters in total. The Micro BERT
was then trained on MLM and NSP simultaneously on the
dataset derived from the books corpus. The hyper parameters
used was a word piece vocab size of 15,000, embedding
dimensions of 128, 8 attention heads, feed forward
dimension (used in the hidden dimes of the feed forward
layers as well as MLP heads) of 128, dropout of .2, 3
stacked encoder blocks, a validation loss sensitive early
stopper with a patience of 10, and a learning rate of 1e-5.
Note: that the cross-entropy loss function used in each MLP
head were different due to the fact that the MLM function
needed to ignore the PAD indices when calculating the loss
while the other two used basic cross entropy losses functions.
The AdamW optimizer was used. The summary of the
training process can be seen in figure 6. The model achieved a
test loss of 9.1346 which was close to the validation loss at
.91359. Both the non-training partition were a bit higher
than the training loss of 9.0215. The model MLM task test
accuracy of .0140 and a NSP test accuracy of .6341. The
test and validation accuracies were always above the training
accuracy for the kept epochs showing small signs of over-
fitting only on the loss metrics. The total training time for
the training of Micro BERT on the NSP and MLM tasks
simultaneously was 63 minutes. The model’s state at training
epoch 16 was saved as the final state of Micro BERT’s
pretraining. The models train time was 63 minutes with the
kept epoch being 16. Next the pretrained model used to
classify categories form the disaster dataset. Micro BERT
was initialized from the pretraining checkpoint on the book’s
corpus. Like done in NSP only the CLS token was extracted
from the output of the encoder blocks output and mapped to
a MLP head with a hidden layer of 128 (with relu and
dropout of .2) and finally to the output layer of 10 neurons.
Cross entropy loss was again used as the loss function and
Adam as the optimizer. The exact same hyperparameters were
used to train Micro BERT on the disaster classification task as
the books corpora (embedding dimension 128, number of
attention heads 8, number of stacked encoders blocks 3,
feed forward hidden dimension 128, dropout .2, patience of
early stopper 10, learning rate 1e-5). A summary of the final
partitional losses, and accuracies can be found in figure 7.
The results of Micro BERT on the disaster dataset were
losses of .7601, .7900, .8025 on train, validation and test. The
accuracies were .7284, .7236, .7263 on train, validation and
test. The kept validation F1 score was .7162 while the test F1
score was .7179. Again, the training loss was a bit lower
than
Fig. 6. This figure summarizes the training of Micro BERT on the MLM and NSP tasks. The green line represents which epoch the models final parameters
were saved. The red line shows the test value for that respective metric. The text below represents the final states of the saved model.

Fig. 7. This figure summarizes the training of the pretrained Micro BERT on the classification task on the disaster dataset.

the other two partition losses but the accuracies were all close
leading to very small evidence of over-fitting the dataset. The
total train time for this model was 238 minutes and the kept
epoch being 141.
Finally, a non-pretrained Micro Bert (the weights of the
Micro BERT model were not initialized from the training
on the google books corpus) was also fit using the exact
same MLP head (weights were again not initialized from
previous training) and was fit on the disaster classification
dataset (embedding dimension 128, number of attention heads
8, number of stacked encoders blocks 3, feed forward
hidden
dimension 128, dropout .2, patience of early stopper 10,
learning rate 1e-5) to see if pretraining on the books corpus
made a difference. The results from this version of Micro
BERT can be seen in figure 8.
The final losses of the non-pretrained Micro BERT were
train .8477, val .9335 and test .9446. The accuracies were
.7044 train, .6926 validation, and .6947 test. The F1 score for
the validation and test partitions was .677 and .675. The same
trend of small differences between train and val/test losses
continued with small differences in accuracy being present
across partitions (validation and test partitions are about a half
 a percent less accurate). These results show larger signs of
overfitting than the training of the pretrained Micro BERT.
The train time of the non-pretrained Micro BERT was 120
minutes with a kept epcoh of 77.
B. Dataset and Pre-processing
First, the dataset is loaded from a CSV file, skipping the
problematic lines, and the rows with lacking values are
deleted to control the data. integrity. Similarly, the 'class_int'
column is transformed to an integer type, and the 'tweet_text'
rows are converted to strings. One-hot encoding is applied to
classify categorical data. The program removes URLs,
placeholders, and non-word characters using text
preprocessing words, comments, hashtags, and the text is also
tokenized, stemmed and lemmatized using NLTK which has
been downloaded previously. The preprocessed tweets are
stored in a new column, and the boolean values are replaced
with numbers, the step table is all done. Data that can then be
used for further analytical or predictive problem-solving.
# Load dataset while ignoring errors in specific lines df =
pd.read_csv("dataset_final.csv", encoding='latin1', on_bad_lines
= 'skip')
df.isnull().sum()
event 0 tweet_text 0 class_int 0 tweet_text_tokenize 0
tweet_stem 0 tweet_lemma 69 dtype: int64
df = df.dropna(axis=0)
df.isnull().sum()
event 0 tweet_text 0 class_int 0 tweet_text_tokenize 0
tweet_stem 0 tweet_lemma 0 dtype: int64
df.info()
<class 'pandas.core.frame.DataFrame'> Index: 74141 entries, 0 to
74209 Data columns (total 6 columns): # Column Non-Null Count
Dtype --- ------ -------------- ----- 0 event 74141 non-null
object 1 tweet_text 74141 non-null object 2 class_int 74141 non-
null object 3 tweet_text_tokenize 74141 non-null object 4
tweet_stem 74141 non-null object
C. Disaster BERT Model
The code cut flow is the development of disaster text
classification model using a version of the BERT model. The
sentence specifically refers to a compressing model of
TinyBERT that was put together from BERT. This model
feeds on the data about tweets related to disasters as being
trained on the dataset and gets customized through various
metrics including accuracy, F1 score, recall, precision, ROC
curves, and AUC scores. The main task is utilized for training
and fine-tuning the model of the disaster BERT, which is
assigned to categorize tweets into different categories related
to the disaster. This code processes a dataset of tweets for
machine learning by splitting it into training and testing sets
with a 20% test size for model evaluation. It utilizes a BERT
tokenizer to prepare the data for input into a TinyBERT
model, adapting the model's final layer to output ten classes
for multi-class classification. The model is compiled using
the Adam optimizer and categorical cross-entropy loss.
Finally, both the input encodings and labels are converted
into TensorFlow tensors, setting the stage for model training
and evaluation.
D. DeBERTa
The code content does not address the process or use of the
DeBERTa model as it is. DeBERTa is a BERT descendant
that owns some improvements, like the attention mechanism
disentanglement and better parameter sharing strategies. If
you have decided to use the DeBERTa model instead of the
Tiny BERT model, you'll have to modify the code to load the
DeBERTa model and use it for fine-tuning.
E. AlBERT
The supplied code is also impartial of the AlBERT model,
same as DeBERTa. AlBERT is like the second version
of the BERT model architecture, including training the model
in a faster way and better parameter sharing. If you plan to
employ AlBERT rather than TinyBERT, the code also should
be adjusted to incorporate the AlBERT model and tune the
model.
F. TinyBERT
This code is based on heavily relying on the TinyBERT
model. TinyBERT is a lightweight version of the BERT
model with the purpose of providing a compact model
through knowledge distillation techniques that retain the
same levels of performance on downstream tasks as the
original BERT model. The code loads the pre-trained
TinyBERT model which is trained on the disaster tweet
dataset. After training, the model is evaluated for text
classification. This code executes the training and evaluation
of a TinyBERT model, timing the entire training process
using Python's time. Time () to capture start and end times.
Training occurs over 10 epochs with a batch size of 32, using
both training and validation datasets. After training, the script
prints a model summary and calculates the total number of
trainable parameters. It then evaluates the model on the test
set to measure loss and accuracy, also timing this process for
performance analysis. The output includes training and
testing times, the number of epochs, and the total trainable
parameters, offering a comprehensive view of the model's
training performance and efficiency.
G. Evaluation
This code below demonstrates comprehensive evaluation and
visualization processes for a trained TinyBERT model on a
text classification task. Initially, it plots the training and
validation loss and accuracy over epochs, providing visual
insight into the model's learning and generalization
capabilities across training sessions. It then computes
predicted probabilities and applies a softmax function to
transform log its into actual probabilities. The script
calculates the ROC curve and the Area Under the Curve
(AUC) for each class, offering a graphical representation of
model performance across different thresholds. Predicted
probabilities are further used to derive predictions, which are
then compared against the true labels to compute accuracy,
F1 score, recall, and precision, offering a holistic view of the
model's predictive accuracy. A confusion matrix is plotted to
visually assess the model's performance in distinguishing
between different classes. Finally, a detailed classification
report is generated, providing precision, recall, F1-score, and
support for each class, thus summarizing the model’s
performance across various metrics. This series of
evaluations and visualizations helps in understanding the
model's strengths and weaknesses in classifying text into
specific categories.
Fig.8. This figure summarizes the training of the pretrained Tiny BERT on
the classification task on the disaster dataset.
III. RESULTS
The experiments introducing the DiBERT model on short-
text classification in disasters did show up ideal expectations.
The system was found to have a high accuracy rate in the
classification of short text into various categories such as
disaster-related, disaster-related. Measures that were core of
evaluation like accuracy, F1 score, recall and precision were
computed using those parameters to test the effectiveness of
the model in the categorization of disasters. The ROC curve
and corresponding AUC score were also employed to
measure model success at different thresholds. Our
evaluation results have thus indicated that the DiBERT type
of model is capable of handling satisfactorily the short-text
classification tasks in disaster incidents which have become a
real need today. The model's ability to represent contextual
clues and sense of the sentences turned out to be a key ability
to recognizing the texts into correctly disaster-related
categories.
Fig. 9. This figure shows the results of Confusion Matrix for the Model
Fig. 10. Shows the classification report of the final Model
IV. DISCUSSION
The results of the DiBERT model for disaster incidents in
short text classification were quite satisfactory that it allows
the model to perform quite well in determining the exact
class of disaster in many of the classes. The accurateness of
DiBERT together with the metrics like F1 score, recall, and
the precision score are beyond the expectations. It
outperforms the similar models and the conventional
solution. Its capacity to build separate generalizations for
each dataset and its flexibility to different scenarios of such
disasters will surely prove that it is a practical model. Despite
the constraints in the interpretation, these results still hide
many possible futures to explore about combining fine-tuning
and building enhancement technologies to make the model
more suitable for management and response to disasters.
Pretraining is a cornerstone in equipping the model to extract
the sophisticated concepts filiations the essential details from
the large text data, provided by the corpora in the process.
We have found that the training, in particular, plays the vital
role. The results indicate that the higher accuracy to the with the opportunities to utilize the strengths of transformer
unseen data benchmarks are obtained as compared with the models similar to DiBERT on the token level by means of
standard network. The model of the pretrained DiBERT task-specific fine-tuning or modifications of the architecture
clearly proves it being very powerful in the handling of prepared for the token processing specifically Nonetheless,
disaster-related texts; therefore, it is clear how necessary the accounting the drawbacks and dilemmas involved in this
pretraining is for making the models more suitable for the approach, we argue that pretraining is a determinant factor,
specific tasks. aside from the questions of model size optimization, and
directions of future research that may be necessary in order
while achieving this balance of the number of parameters, for transformer-based models to be of high utility in disaster
training time, and model performance is an essential factor, it management and text classification tasks.
remains a challenge to achieve it. Concurrently, models with
fewer parameters could possibly result in an V. CONCLUSION
underperformance. On the other hand, they have the
advantage of being trained within shorter timeframes during In conclusion, the DiBERT model, a short-text classification
which they also require fewer computational resources. For disaster incident model based on a transformer-bidirectional
less complicated problems or for resource-restricted architecture, has introduced a robust tool for disaster
circumstances, simple models may be a more suitable option management and response. The experimenting and evaluation
as the cost-to-benefit ratio does not necessarily lean towards of the model's performance reveals its ability to make sure
the use of complex models. Thus, designing optimized model that the disaster short texts are properly analyzed and
architecture with those parameters as regards resources classified. The experience of transformer-based models like
consumption and model performance is an important issue to DiBERT provides the basis for the development of disaster
be solved in this matter. Also, while scale may be management systems that process information related to
synonymous with more memory consumption, sufficient or disasters better and faster. Such research findings further the
advanced models require you to weigh in factors as development of natural language processing techniques in
computation time against the other. disaster response, thus aid in the extraction of useful
information from a limited number of texts in emergency
situations. This could help pave the way for further research
In addition to that, token tasks might be not so powerful in and development in the area of disaster text classification that
tasks like language masking modeling (MLM), but diBERT
with the same measure will show respectful results in the could be used in real scenarios like emergency response,
short text classification. Even though the overall outcomes of crisis communication, and disaster preparedness.
MLM were not as good as one may expect, the final text Conclusively, the DiBERT model is one of the indispensable
classifier employing the MLM model to approximate a suite devices for disaster management, as it provides quick and
of document-level tasks, such as document classification, was reliable classification of short-texts in emergency situations.
able to resolve complex task. The coming research may deal
VI.REFERENCES
[1] Tamara L Sheldon and Crystal Zhan. The impact of
natural disasters on us home ownership. Journal of the
Association of Environmental and Resource Economists,
6(6):1169–1203, 2019.
[2] Muhammad Imran Ferda Ofli Firoj Alam, Umair Qazi.
Humaid: Human-annotated disaster incidents data from
twitter. In 15th International Conference on Web and
Social Media (ICWSM), 2021.
[3] Shaheen Khatoon, Majed A Alshamari, Amna Asif,
Md Maruf Hasan, Sherif Abdou, Khaled Mostafa Elsayed,
and Mohsen Rashwan. Development of social media
analytics system for emergency event detection and cri-
sismanagement. Comput. Mater. Contin, 68(3), 2021.
[4] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu
Chen. Deberta: Decoding-enhanced BERT with disen-
tangled attention. CoRR, abs/2006.03654, 2020. URL
https://arxiv.org/abs/2006.03654.
[5] Rani Koshy and Sivasankar Elango. Multimodal
tweet classification in disaster response systems using
transformer-based bidirectional attention model. Neural
Computing and Applications, 35(2):1607–1627, 2023.
[6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova. Bert: Pre-training of deep bidirectional trans-
formers for language understanding. arXiv preprint
arXiv:1810.04805, 2018.
[7] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob
Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser,
and Illia Polosukhin. Attention is all you need. Advances
in neural information processing systems, 30, 2017.
Fig. 8. This figure summarizes the training of the non-pretrained Micro BERT on the classification task on the disaster dataset.