ACHARYA INSTITUTE OF GRADUATE STUDIES

(NAAC Re Accredited ‘A+’ Grade & Affiliated to Bengaluru City University)

Soldevanahalli, Bengaluru-560107

DEPARTMENT OF MCA

Faculty Name : B. Manimekala

Semester & Section : III MCA A & B
Course Title : Elective – Machine Learning

Unit 5: Genetic Algorithms

Genetic Algorithms – Representing Hypothesis, Genetic operators and fitness function and
selection, Simple applications of the Genetic algorithm, application of GA in Decision tree, Genetic
algorithm-based clustering, Single objective and Biobjective optimization problems using GA, using
GA to emulate Gradient descent/ascent

GENETIC ALGORITHMS
A genetic algorithm is an adaptive heuristic search algorithm inspired by "Darwin's theory of
evolution in Nature." It is used to solve optimization problems in machine learning. It is one of the
important algorithms as it helps solve complex problems that would take a long time to solve.

Genetic Algorithms are being widely used in different real-world applications, for example,
Designing electronic circuits, code-breaking, image processing, and artificial creativity.

Basic Terminologies: Genetic Algorithm:

➢ Population: Population is the subset of all possible or probable solutions, which can solve
the given problem.
➢ Chromosomes: A chromosome is one of the solutions in the population for the given
problem, and the collection of gene generate a chromosome.
➢ Gene: A chromosome is divided into a different gene, or it is an element of the chromosome.
➢ Allele: Allele is the value provided to the gene within a particular chromosome.
➢ Fitness Function: The fitness function is used to determine the individual's fitness level in
the population. It means the ability of an individual to compete with other individuals. In
every iteration, individuals are evaluated based on their fitness function.
➢ Genetic Operators: In a genetic algorithm, the best individual mate to regenerate offspring
better than parents. Here genetic operators play a role in changing the genetic composition
of the next generation.
➢ Selection: After calculating the fitness of every existent in the population, a selection
process is used to determine which of the individualities in the population will get to
reproduce and produce the seed that will form the coming generation.

Machine Learning – Lecture Notes – Unit 5 | 1

Types of selection styles available
1. Roulette wheel selection
2. Event selection
3. Rank- grounded selection
A genetic algorithm as a heuristic search algorithm to solve optimization problems. It is a subset of
evolutionary algorithms, which is used in computing. A genetic algorithm uses genetic and natural
selection concepts to solve optimization problems.

Working Procedure: Genetic Algorithm

The genetic algorithm works on the evolutionary generational cycle to generate high-quality
solutions. These algorithms use different operations that either enhance or replace the population
to give an improved fit solution. It basically involves five phases to solve the complex optimization
problems, which are given as below:
1. Initialization
2. Fitness Assignment
3. Selection
4. Reproduction
5. Termination

1. Initialization: The process of a genetic algorithm

starts by generating the set of individuals, which is
called population. Here each individual is the solution
for the given problem. An individual contains or is
characterized by a set of parameters called Genes.
Genes are combined into a string and generate
chromosomes, which is the solution to the problem. One
of the most popular techniques for initialization is the
use of random binary strings.
2. Fitness Assignment: Fitness function is used to determine how fit an individual is? It means the
ability of an individual to compete with other individuals. In every iteration, individuals are
evaluated based on their fitness function. The fitness function provides a fitness score to each
individual. This score further determines the probability of being selected for reproduction. The
high the fitness score, the more chances of getting selected for reproduction.

3. Selection: The selection phase involves the selection of individuals for the reproduction of
offspring. All the selected individuals are then arranged in a pair of two to increase reproduction.
Then these individuals transfer their genes to the next generation. There are three types of
Selection methods available, which are: Roulette wheel selection, Tournament selection and
rank based selection.

Machine Learning – Lecture Notes – Unit 5 | 2

4. Reproduction: After the selection process, the creation of a child occurs in the reproduction
step. In this step, the genetic algorithm uses two variation operators that are applied to the parent
population. The two operators involved in the reproduction phase are given below:

➢ Crossover: The crossover plays a most significant role in the reproduction phase of the
genetic algorithm. In this process, a crossover point is selected at random within the genes.
Then the crossover operator swaps genetic information of two parents from the current
generation to produce a new individual representing the offspring.

The genes of parents are exchanged among themselves until the crossover point is met. These
newly generated offspring are added to the population. This process is also called or crossover.
Types of crossover styles available:
1. One point crossover
2. Two-point crossover
3. Livery crossover
4. Inheritable Algorithms crossover

➢ Mutation: The mutation operator inserts random genes in the offspring (new child) to
maintain the diversity in the population. It can be done by flipping some bits in the
chromosomes. Mutation helps in solving the issue of premature convergence and enhances
diversification. Types of mutation styles available,
1. Flip bit mutation
2. Gaussian mutation
3. Exchange/Swap mutation

5. Termination: After the reproduction phase, a stopping criterion is applied as a base for
termination. The algorithm terminates after the threshold fitness solution is reached. It will identify
the final solution as the best solution in the population.

Machine Learning – Lecture Notes – Unit 5 | 3

General Workflow of a Simple Genetic Algorithm

Advantages of Genetic Algorithm

➢ The parallel capabilities of genetic algorithms are best.
➢ It helps in optimizing various problems such as discrete functions, multi-objective
problems, and continuous functions.
➢ It provides a solution for a problem that improves over time.
➢ A genetic algorithm does not need derivative information.

Machine Learning – Lecture Notes – Unit 5 | 4

Limitations of Genetic Algorithms
➢ Genetic algorithms are not efficient algorithms for solving simple problems.
➢ It does not guarantee the quality of the final solution to a problem.
➢ Repetitive calculation of fitness values may generate some computational challenges.

GENETIC OPERATORS
The generation of successors in a GA is determined by a set of operators that recombine and mutate
selected members of the current population. These operators correspond to idealized versions of
the genetic operations found in biological evolution. The two most common operators are
crossover and mutation.

I. Crossover is a genetic operator used to vary the programming of a chromosome or chromosomes

from one generation to the next. Crossover is sexual reproduction. Two strings are picked from the
mating pool at random to crossover in order to produce superior offspring. The method chosen
depends on the Encoding Method.
Crossover mask: The choice of which parent contributes to the bit position ‘fl’ the offspring is given
by an additional string called crossover mask similar to bit masks in unity game engine.
Different types of Crossovers:
1. Single Point Crossover: A crossover point on the parent organism string is selected. All data
beyond that point in the organism string is swapped between the two parent organisms. Strings
are characterized by Positional Bias.

2. Two-Point Crossover: This is a specific case of a N-point Crossover technique. Two random
points are chosen on the individual chromosomes (strings) and the genetic material is exchanged
at these points.

Machine Learning – Lecture Notes – Unit 5 | 5

3. Uniform Crossover: Each gene (bit) is selected randomly from one of the corresponding genes
of the parent chromosomes. Use tossing of a coin as an example technique.

The crossover between two good solutions may not always yield a better or as good a solution.
Since parents are good, the probability of the child being good is high. If offspring is not good (poor
solution), it will be removed in the next iteration during “Selection”.

II. Mutation: In simple terms, mutation may be defined as a small random tweak in the
chromosome, to get a new solution. It is used to maintain and introduce diversity in the genetic
population and is usually applied with a low probability – pm. If the probability is very high, the GA
gets reduced to a random search.
Mutation is the part of the GA which is related to the “exploration” of the search space. It has been
observed that mutation is essential to the convergence of the GA while crossover is not.
Mutation Operators:
1. In this bit flip mutation, we select one or more random bits and flip them. This is used for
binary encoded GAs.

2. Random Resetting is an extension of the bit flip for the integer representation. In this, a random
value from the set of permissible values is assigned to a randomly chosen gene.
3. In swap mutation, we select two positions on the chromosome at random, and interchange the
values. This is common in permutation-based encodings.

4. Scramble mutation is also popular with permutation representations. In this, from the entire
chromosome, a subset of genes is chosen and their values are scrambled or shuffled randomly.

5. In inversion mutation, we select a subset of genes like in scramble mutation, but instead of
shuffling the subset, we merely invert the entire string in the subset.

Machine Learning – Lecture Notes – Unit 5 | 6

 FITNESS FUNCTION AND SELECTION
The fitness function defines the criterion for ranking potential hypotheses and for probabilistically
selecting them for inclusion in the next generation population. If the task is to learn classification
rules, then the fitness function typically has a component that scores the classification accuracy of
the rule over a set of provided training examples. Often other criteria may be included as well, such
as the complexity or generality of the rule. More generally, when the bit-string hypothesis is
interpreted as a complex procedure (e.g., when the bit string represents a collection of if-then rules
that will be chained together to control a robotic device), the fitness function may measure the
overall performance of the resulting procedure rather than performance of individual rules.

The probability that a hypothesis will be selected is given by the ratio of its fitness to the fitness of
other members of the current population. This method is sometimes called fitness proportionate
selection, or roulette wheel selection. Other methods for using fitness to select hypotheses have
also been proposed. In tournament selection, two hypotheses are first chosen at random from the
current population. With some predefined probability ‘p’ the more fit of these two is then selected,
and with probability (1 - p) the less fit hypothesis is selected. Tournament selection often yields a
more diverse population than fitness proportionate selection. In another method called rank
selection, the hypotheses in the current population are first sorted by fitness. The probability that
a hypothesis will be selected is then proportional to its rank in this sorted list, rather than its fitness.

SIMPLE APPLICATIONS OF THE GENETIC ALGORITHM

Genetic algorithms find use in various real-world applications. In this segment, we have elaborated
on some areas that utilize the genetic algorithms in machine learning.
1. Neural networks: Genetic programming in machine learning finds great applications for neural
networks in machine learning. We use it for genetic optimization in neural networks or use cases
like inheriting qualities of neurons, neural network pipeline optimization, finding the best fit set of
parameters for a given neural network, and others.
2. Data mining and clustering: Data mining and clustering use genetic algorithms to find out the
centre point of the clusters with an optimal error rate given to its great searching capability for an
optimal value. It is renowned as an unsupervised learning process in machine learning, where we
categorize the data based on the characteristics of the data points.
3. Image processing: Image processing tasks, such as image segmentation, are one of the major
use cases of genetic optimization. However, genetic algorithms can also be used in different areas
of image analysis to resolve complex optimization problems.
4. Wireless sensor network: A wireless sensor network refers to one which includes dedicated
and especially dispersed centres that maintain the record of the physical conditions of an
environment. It further passes the record created to a central storage system.

Machine Learning – Lecture Notes – Unit 5 | 7

WSN utilizes genetic machine learning to stimulate the sensors. We can optimize and even
customize all the operational stages with the help of the fitness function from genetic algorithms in
wireless sensor networks.
5. Traveling salesman problem (TSP): TSP or traveling salesman problem is one of the real-life
combinatorial optimization problems that were solved using genetic optimization. It helps in
finding an optimal way in a given map with the distance between two points and with the routes to
be covered by the salesman.
Several iterations take place that generate offspring solutions after every iteration to inherit the
qualities of parent solutions. In this way we don’t get the solution only one time that offers you the
opportunity to choose the best route structure. It finds application in real-time processes like
planning, manufacturing and logistics.
6. Vehicle routing problems: One of the generalisations of the travelling salesman problem
discussed above is the vehicle routing problem. Genetic algorithms help in finding the optimal
weight of goods to be delivered through the optimal set of delivery routes. Factors such as depot
points, wait, distance are taken into consideration while solving such problems if they have any
kind of restrictions. Moreover, the genetic algorithm approach is competitive in terms of solution
quality and time with simulated annealing algorithms and tabu search.
7. Mechanical engineering design: Genetic algorithms find application in many designing
procedures of mechanical components. For instance, consider the following genetic algorithm
example where the aircraft wing design is a kind of designing problem that takes multiple
disciplines into consideration. It requires improvement in the ratio of left to drag for a complex
wing.
The fitness function in genetic optimization is flexible to considerations that come as a demand for
a particular design.
8. Manufacturing system: The manufacturing arena includes various examples of the cost
function. Based on the same, the need to find an optimal set of parameters for such functions poses
a problem.
Genetic optimization performs this task of using the optimized set of parameters to minimize the
cost function. It also finds application in product manufacturing to achieve optimum production
plans by considering dynamic conditions like capacity, inventories, or material quality.
9. Financial markets: A variety of issues can be solved using genetic optimization in the financial
market. It helps in finding an optimal combination of parameters that can affect the trades or
market rules. You can also find out the near-optimal value for the optimal set of parameters.
10. Medical science: Medical signs have an array of use cases for genetic optimization. The areas
of predictive analysis include protein prediction, RNA structure prediction, operon prediction, and
others.

Machine Learning – Lecture Notes – Unit 5 | 8

Other processes such as protein folding, gene expression profiling analysis, bioinformatics multiple
sequence alignment, or some of the process alignment that uses genetic optimisation.
11. Task scheduling: Genetic machine learning algorithms are used to derive optimal schedules
that satisfy certain constraints related to a problem.
For instance: assume that you have to prepare the schedule for the semester exams for University.
The genetic algorithm will find the best optimal schedule for the university considering all the
constraints like the number of classrooms, the number of students, the total number of courses and
subjects, and others.
12. Economics: Economics is the study of resource utilization in the production, distribution, and
complete consumption of goods and services over a time period. The genetic algorithm creates
models of demand and supply that derive asset pricing, game theory, and others.
13. Robotics: Robotics comprises the construction, design, and working of the autonomous robot.
Genetic algorithms contribute to the robotics field by providing the necessary insight into the
decisions made by the robot. It generates optimal routes for the robot so that it can use the least
number of resources to get to the desired position.

APPLICATION OF GA IN DECISION TREE

A self-configuring genetic programming algorithm is used to automate the process of constructing
decision trees; it designs decision trees without any imposed restrictions on their structure, in
contrast to standard algorithms. However, decision trees are prone to overfitting regardless of the
algorithm of their construction. This drawback can be overcome if the trees are combined into
ensembles. The construction of ensembles is one of the most powerful methods of machine
learning, which often surpasses other methods in the quality of predictions.

To increase the efficiency of the compositions, it was proposed to combine the trees designed by
the self-configuring genetic programming algorithm in the ensemble.

Decision trees
The decision tree is a method based on the application of various separation functions of the initial
data set, in particular, simple threshold rules.
Decision trees are an effective method, especially in demand in tasks, where interpretation of the
results is required, which is understandable to a layman in the field of data analysis. Decision tree
learning algorithms have a significant drawback; cannot choose the optimal tree structure. To
combat this drawback, it is proposed to use the genetic programming algorithm.

Decision tree design approach

One of the branches of evolutionary algorithms is genetic programming. A characteristic feature of
genetic programming algorithms is the tree structure of the individual, that is, this algorithm
searches for solutions in the space of trees. This is very suitable for designing decision trees. A tree

Machine Learning – Lecture Notes – Unit 5 | 9

is a directed graph in which each subsequent node is associated with one and only previous one.
Vertices can take values from terminal and functional sets.

Ensembles of decision trees

Another problem of decision trees is their high tendency to overfitting. Using ensemble of decision
trees avoids this problem. Usually, when solving problems of data analysis, the structure is selected
and the parameters are adjusted of a separate technology, after which the solution of the problem
is trusted in the best method found. However, a certain set of algorithms when used simultaneously,
according to the selected collective technology, often allows us to get a better solution. Such a set
of algorithms is called an ensemble or composition.

GENETIC ALGORITHM-BASED CLUSTERING

Genetic algorithm-based clustering is an approach in machine learning that leverages the
principles of genetic algorithms (GAs) to perform clustering tasks. Clustering is the process of
partitioning a set of data points into groups (clusters) such that points within the same group are
more similar to each other than to those in other groups.

Basics of Genetic Algorithms (GAs)

Genetic algorithms are a class of optimization algorithms inspired by the process of natural
selection. They work by evolving a population of candidate solutions over several generations to
optimize a given objective function. The main components of GAs are:

Machine Learning – Lecture Notes – Unit 5 | 10

 1. Population: A set of potential solutions.
2. Chromosome: Representation of a potential solution.
3. Fitness Function: A function to evaluate how good a solution (chromosome) is.
4. Selection: Mechanism to choose the better solutions for reproduction.
5. Crossover (Recombination): Combining parts of two parent solutions to create offspring.
6. Mutation: Introducing small random changes to a solution to maintain genetic diversity.

Applying GAs to Clustering

1. Representation (Chromosome Encoding):
➢ Each chromosome represents a potential clustering solution.
➢ A common encoding is to use a string of integers, where each integer represents the
cluster assignment of a data point. For instance, in a dataset with 5 points and 3
clusters, a chromosome might look like [1, 2, 1, 3, 2].
2. Initial Population:
➢ Generate an initial population of random clustering solutions. Each solution is a
chromosome with random cluster assignments.
3. Fitness Function:
➢ The fitness function evaluates the quality of a clustering solution. Common fitness
functions include within-cluster sum of squares (WCSS), silhouette score, or any
other metric that measures the compactness and separation of clusters.
➢ For instance, minimizing the WCSS can be a fitness function:

4. Selection:
➢ Select parent solutions from the population based on their fitness scores.
Techniques like roulette wheel selection, tournament selection, or rank-based
selection can be used.
5. Crossover (Recombination):
➢ Combine parts of two parent solutions to create new offspring. For clustering,
crossover might involve swapping subsets of cluster assignments between two
parent chromosomes.
6. Mutation:
➢ Introduce random changes to offspring solutions to explore the search space. This
could involve randomly reassigning some data points to different clusters.
7. Evolution Process:
➢ Replace less fit solutions in the population with new offspring.

Machine Learning – Lecture Notes – Unit 5 | 11

 ➢ Repeat the selection, crossover, and mutation processes over multiple generations
until a stopping criterion is met (e.g., a maximum number of generations or
convergence to a stable fitness score).

Advantages of GA-based Clustering

➢ Flexibility: Can optimize arbitrary clustering criteria without the need for derivatives.
➢ Global Search Capability: Tends to avoid local minima by maintaining a diverse
population of solutions.
➢ Adaptability: Easily adapts to various data types and clustering objectives.

Challenges and Considerations

➢ Parameter Tuning: Requires careful tuning of GA parameters like population size,
crossover rate, mutation rate, and selection mechanism.
➢ Computational Cost: Can be computationally expensive, especially for large datasets and
many generations.
➢ Convergence Speed: May converge slower than other optimization methods due to the
stochastic nature of GAs.

Applications
Genetic algorithm-based clustering is used in various fields, including:
➢ Market segmentation
➢ Image segmentation
➢ Bioinformatics (e.g., gene expression data analysis)
➢ Document clustering
➢ Anomaly detection

Example Workflow
1. Initialize Population: Randomly generate cluster assignments for an initial set of
solutions.
2. Evaluate Fitness: Calculate the fitness of each solution using a clustering metric.
3. Select Parents: Choose solutions based on their fitness scores for reproduction.
4. Crossover and Mutation: Generate new solutions by combining parts of selected parents
and introducing mutations.
5. Update Population: Replace the less fit solutions with new offspring.
6. Iterate: Repeat the evaluation, selection, crossover, mutation, and updating steps for a
predefined number of generations or until convergence.
The genetic algorithm-based clustering can effectively find high-quality clustering solutions,
leveraging the global search capabilities and flexibility of genetic algorithms.

Machine Learning – Lecture Notes – Unit 5 | 12

 SINGLE OBJECTIVE AND BIOBJECTIVE OPTIMIZATION PROBLEMS USING GA
Single objective optimization involves optimizing a single objective function, which could be
either a maximization or minimization problem.

Workflow for Single Objective Optimization

1. Problem Definition:
➢ Define a single objective function that needs to be optimized. For instance, in a
regression problem, the objective might be to minimize the mean squared error
(MSE) of predictions.
2. Chromosome Encoding:
➢ Encode the potential solutions (e.g., model parameters, feature selections) as
chromosomes. For example, if optimizing the weights of a neural network, each
chromosome could represent a set of weights.
3. Initial Population:
➢ Generate an initial population of random solutions. This diversity helps in exploring
the solution space effectively.
4. Fitness Function:
➢ Evaluate the fitness of each chromosome using the objective function. In the case of
minimizing MSE, the fitness function would calculate the MSE for each set of
weights.
5. Selection:
➢ Select the best-performing chromosomes to act as parents for the next generation.
Techniques like roulette wheel selection, tournament selection, or rank-based
selection can be used.
6. Crossover (Recombination):
➢ Combine parts of two parent solutions to create new offspring. This could involve
combining segments of model parameters or feature subsets.
7. Mutation:
➢ Introduce small random changes to some offspring to maintain genetic diversity
and avoid local minima. For instance, slightly adjust some model parameters.
8. Iteration:
➢ Repeat the selection, crossover, and mutation steps over multiple generations.
Track the best solutions found over iterations.
9. Termination:
➢ Stop when a predefined number of generations is reached, or when improvement
plateaus. The best solution found is taken as the optimal solution.

Machine Learning – Lecture Notes – Unit 5 | 13

Example Application: Hyperparameter Optimization
• Objective: Minimize the validation error of a machine learning model.
• Chromosome Encoding: Each chromosome represents a set of hyperparameters.
• Fitness Function: Evaluate the model’s performance on a validation set.
• Result: The GA finds the set of hyperparameters that minimizes the validation error.

Biobjective (Multi-objective) Optimization using Genetic Algorithms

Multi-objective optimization involves optimizing two or more conflicting objectives
simultaneously. For example, in machine learning, one might want to minimize both the error rate
and the complexity of a model.

Workflow for Multi-objective / Biobjective Optimization

1. Problem Definition:
➢ Define multiple objective functions. For example, in model selection, you might aim
to minimize both the prediction error and the number of features used.
2. Chromosome Encoding:
➢ Similar to single objective optimization, each chromosome represents a potential
solution (e.g., model parameters, feature subsets).
3. Initial Population:
➢ Generate an initial population of random solutions.
4. Fitness Evaluation:
➢ Evaluate the fitness of each chromosome using the objective functions. In
multiobjective optimization, this often involves calculating a fitness vector (one
value for each objective).
➢ Identify a set of non-dominated solutions. A solution is non-dominated if no other
solution is better in all objectives.
5. Selection:
➢ Select parents from the Pareto front or based on a combination of fitness values and
diversity. Techniques like Pareto ranking and crowding distance can be used to
ensure diversity.
6. Crossover and Mutation:
➢ Perform crossover and mutation as in single objective optimization, but ensure that
the offspring also reflect diverse trade-offs between objectives.
7. Iteration:
➢ Repeat the evaluation, selection, crossover, and mutation steps over multiple
generations.
8. Termination:
➢ Stop when a predefined number of generations is reached

Machine Learning – Lecture Notes – Unit 5 | 14

Example Application: Model Selection
• Objectives: Minimize both the prediction error and the model complexity.
• Chromosome Encoding: Each chromosome represents a set of model parameters and
feature subsets.
• Fitness Functions: One function calculates the prediction error, and the other calculates
the complexity (e.g., number of features).
• Result: The GA identifies a set of models that offer the best trade-offs between error and
complexity, providing a Pareto front of solutions.

Single Objective Optimization using GAs focuses on optimizing one goal, such as accuracy or cost,
through iterative evolution of solutions. Biobjective (Multi-objective) Optimization handles
multiple conflicting goals simultaneously, seeking a set of optimal trade-offs. In both cases, GAs
provides a robust, flexible framework for exploring complex solution spaces and finding high-
quality solutions.

GA TO EMULATE GRADIENT DESCENT /ASCENT

Genetic Algorithms (GAs) and Gradient Descent (GD) are both optimization techniques, but they
work quite differently. GAs is based on principles of natural evolution and are suitable for complex,
non-differentiable, or multi-modal functions. Gradient Descent (GD) on the other hand, is an
iterative optimization algorithm that adjusts parameters to minimize a cost function, relying on the
gradient (derivative) of the function. Emulating Gradient Descent with a Genetic Algorithm can be
useful in scenarios where the gradient is not available or the objective function is noisy or
discontinuous.

GA Emulating Gradient Descent/Ascent

To emulate the behaviour of Gradient Descent (GD), GAs can be tailored to iteratively improve
solutions in a manner that mimics the gradient-based optimization's goal-seeking behaviour. The
key idea is to adjust the population evolution mechanisms (selection, crossover, and mutation) to
focus on continuous improvement of the solution, similar to how GD adjusts parameters in the
direction of the gradient.

Workflow
1. Chromosome Encoding:
➢ Each chromosome represents a potential solution, typically a set of model
parameters.
➢ For instance, in neural network optimization, each chromosome might encode the
weights and biases of the network.
2. Initial Population:
➢ Generate an initial population of random solutions (parameter sets).

Machine Learning – Lecture Notes – Unit 5 | 15

 3. Fitness Function:
➢ Define a fitness function that evaluates the quality of each solution. This could be
the objective function used in GD, such as the loss function in a machine learning
model.
➢ For minimization (emulating gradient descent), the fitness function should evaluate
the negative of the loss function to align with GA’s maximization approach.
4. Selection:
➢ Use selection mechanisms that favor better solutions. Tournament selection or
rank-based selection can be used to prefer chromosomes with higher fitness scores.
5. Crossover (Recombination):
➢ Combine parts of two parent solutions to create new offspring. Crossover should
promote exploitation by combining good solutions to produce potentially better
solutions.
➢ Single-point crossover or uniform crossover can be applied to blend parameters of
parent solutions.
6. Mutation:
➢ Introduce small random changes to offspring solutions to maintain diversity and
explore new regions of the solution space.
➢ Mutation should be akin to the learning rate in GD; small mutations can represent
small parameter updates.
7. Elitism:
➢ Preserve a few of the best solutions from one generation to the next to ensure the
quality of solutions does not degrade.
8. Iteration:
➢ Repeat the evaluation, selection, crossover, and mutation steps over multiple
generations.
➢ The goal is to iteratively improve the population, analogous to how GD iteratively
adjusts parameters.
9. Termination:
➢ Stop when a predefined number of generations is reached or when improvement
plateaus.
➢ The best solution found represents the optimized parameter set.

Example Application: Neural Network Training

• Objective: Minimize the loss function (e.g., mean squared error) of a neural network.
• Chromosome Encoding: Each chromosome encodes the weights and biases of the neural
network.

Machine Learning – Lecture Notes – Unit 5 | 16

 • Fitness Function: Negative of the loss function to align with the GA’s maximization process.

Work flow Procedure

1. Initialization: Generate an initial population of neural networks with random weights and
biases.
2. Evaluation: Evaluate each neural network on the training data and calculate the loss. Use the
negative loss as the fitness score.
3. Selection: Select the best-performing networks (based on fitness scores) for reproduction.
4. Crossover: Perform crossover to create new networks by combining weights and biases
from parent networks.
5. Mutation: Mutate the offspring by making small random adjustments to weights and biases.
6. Elitism: Preserve a few top-performing networks to ensure their characteristics are retained
in the next generation.
7. Iteration: Repeat the evaluation, selection, crossover, and mutation steps for several
generations.
8. Termination: Stop after a set number of generations or if the loss function converges to a
satisfactory level.

Benefits and Limitations

Benefits:
• Flexibility: GAs do not require gradient information and can handle non-differentiable,
discontinuous, or noisy objective functions.
• Global Search Capability: GAs can escape local minima due to their stochastic nature and
broad search mechanisms.
Limitations:
• Computational Cost: GAs are generally more computationally intensive than GD,
especially for large populations and complex models.
• Convergence Speed: GAs might converge slower compared to GD, especially on smooth,
convex functions where GD is highly efficient.

While GAs and GD are fundamentally different, GAs can be adapted to emulate the iterative,
improvement-focused nature of GD. By carefully tuning selection, crossover, and mutation
processes, GAs can provide a robust alternative for optimization in scenarios where traditional
gradient-based methods are not applicable.

Machine Learning – Lecture Notes – Unit 5 | 17