Electronic Engineering 3rd Year Individual Project Proposal

Effect of Communication Noise on Multi-agent

Reinforcement Learning and a Noise-tolerant Algorithm
of Learning to Communicate and Act

Student’s Name: Chenhao Xue

Student Number: 19027096

Supervisor: Professor Osvaldo Simeone

Due Date: 04 November 2021

Word Count: 2795

1. Introduction
Future problems many have a distributed nature. Namely, those problems, such as automated driving, large

scale UAV autonomous operations, and coordination of large clusters of robots, involve complex dynamics,

high degrees of coordination, and are extremely suitable to be solved with cooperation (Tuyls, K., & Weiss,

G., 2012). This coincides with our human society. We live in a cooperative social environment that facilitates

learning. So, we not only learn things based on ourselves, but also through teaching, demonstrating, and

discussing, where communication is always involved. Applying this idea of communication in intellectual

human agents, multi-agent systems and multi-agent learning are developed and are widely accepted as an

important fundamental study and one of the most suitable methods to solve those problems with a distributed

nature. However, multi-agent learning is also facing significant challenges: Classification limitations (current

characterizations of MAL, which focus on existing approaches without predicting future development, fail to

fully capture the essence and range of it); Extending the scope (today’s works on MAL is dominated by

reinforcement learning and game theory, which is probably too narrow); and Performance in complex systems

and environments (applications and environments that MAL has often dealt with are relatively simple, and

only involve a few agents).

This project will focus on the third challenge mentioned above, studying performance of MAL in complex

environments. To specify, communication noise will be set between agents in cooperative tasks to simulate the

noisy real-world environment. The performance of both cooperative agents and independent agents under noise

will be studied and a noise-tolerant learning algorithm for communicate and act will be formulated.

Noise is one of the most common difficulties that communication encounters in real-world complex

environments. To find out whether and to what extend MAL, which heavily relies on communication between

agents, can express its value in complex real-world applications, analyzing its performance under noise is one

necessary and valuable starting point. Furthermore, formulating a noise-tolerant algorithm for multi-agent

learning how to communicate and act is a valid basics and will stimulate more profound research on complex

and more realistic applications of MAL, a significant but relatively neglected MAL research direction. Thus,

there is an explicit need for research on the effect of noise on multi-agent learning.
6CCE3EEP Project Proposal Chenhao Xue 19027096

2. Objective, Methodology, and Deliverables

Three current challenges in the field of multi-agent learning are outlined in the introduction above. At the same

time, those challenges are also opportunities and research gaps in the field. Possible opportunities for solving

the first challenge would be inspired by the “multiplied learning, divided learning, and interactive learning”

classification scheme proposed by Weiss and Dillenbourg (Weiss, G., & Dillenbourg, P., 1999). For the second

challenge, there may be several possible solutions: Transfer Learning (Taylor, M. E., & Stone, P., 2009), which

transfers knowledge learned in one task domain to another from agent to another agent, from an agent to a

human, and from a human to an agent; Swarm Intelligence (Dorigo, M., & Stützle, T., 2004), which aims to

develop self-organized and decentralized adaptive algorithms; and Federated Learning (McMahan B. et al.,

2017), which allows multiple agents to build a common, robust learning model without sharing data.

The purpose of this project is to study the third challenge, performance of multi-agent learning in complex

environment, and specific objectives, deliverables, and methodology will be described below.

My project aims to analyze the effect of noise on multi-agent reinforcement learning. To realize the purpose,

the objectives of this project are as follow: firstly, I will determine whether cooperative agents can outperform

independent agents when communication is subject to noise; secondly, I will study the performance of

cooperative agents under different noise conditions; thirdly, I will formulate a noise-tolerant multi-agent

reinforcement learning algorithm to solve a set of communication required multi-agent tasks with built-in noise;

lastly, the performance of the formulated algorithm will be analyzed and compared with traditional methods.

The methods to achieve the above objectives is programmatic: First, study the paper “Multi-agent

Reinforcement Learning: Independent vs. Cooperative Agents” (Tan M., 1993). Second, reproduce an

experiment, which is picked from the three experiments in the above paper with specific reasons. Third, study

the effect of noise by adding white gaussian noise in the communications between agents in the reproduced

experiment, and determine which algorithm performs better. Fourth, study the robustness of the

communication between agents to noise by determining the signal-to-noise ratio at which cooperation between

agents break down. Fifth, quantify the effect of noise by drawing table and diagram of various signal-to-noise

ratio to the performance of cooperative agents, and quantifying potential relationships. Sixth, study the paper

“Learning to Communicate with Deep Multi-agent Reinforcement Learning”, which proposes a set of multi-

3/11/2021 1
6CCE3EEP Project Proposal Chenhao Xue 19027096

agent benchmark tasks that require communication and have built-in noise, and formulates a Reinforced Inter-

Agent Learning (RIAL) and a Differentiable Inter-Agent Learning (DIAL) algorithm (Foerster J.N. et al.,

2016). Seventh, reproduce the set of tasks and formulate a noise-tolerant multi-agent reinforcement learning

algorithm to handle these tasks. Last, analyze the algorithm’s performance.

There will be three deliverables of this project: a project report and codes, and a demonstration video.

The project report will include performance comparison between cooperative agents and independent agent

under communication noise, table and diagram of signal-to-noise ratio to cooperative agents’ performance with

potential relationships, description of the set of multi-agent benchmark tasks, algorithm of noise-tolerant multi-

agent reinforcement learning model, and performance analysis of that model.

The codes will be a reusable Python code that include the reproduced independent vs. cooperative agents

experiment with communication noise, the set of multi-agent benchmark tasks, and the formulated noise-

tolerant multi-agent reinforcement learning algorithm with analysis of its performance.

The demonstration video will demonstrate the background and necessity of this research, the performance of

cooperative agents and its relationship with communication noise, and the noise-tolerant multi-agent

reinforcement learning algorithm, as well as potential future research inspired from this project.

This project will be slightly challenging for an Undergraduate Thesis because reinforcement learning is not

covered in the modules yet. But this will not be a difficulty with self-learning materials on reinforcement

learning and instructions from the project supervisor, as well as the Machine Learning for Engineers module I

will take in the next semester. Also, the innovations in this project will be focused on dealing with

communication noise, which has been covered in the Communication Systems module and Signals and

Systems module. Besides, this project will not involve on campus activities or non-standard equipment, so it

will not be affected by the Covid-19 disruption. Therefore, this will be a feasible project for Undergraduate

Thesis.

3/11/2021 2
6CCE3EEP Project Proposal Chenhao Xue 19027096

3. Explanations of Learning Outcomes

This project fulfills the expected learning outcomes in various aspects. First, this project is heavily research

based in its contextual knowledge of the multi-agent learning field and the broader machine learning field. The

central citation of this project is a relatively old but core research that compares independent and cooperative

agents in multi-agent reinforcement learning. There are many following papers related to it that provide

extensive but slightly scattered knowledge. To conduct effective literature review, I will be required to integrate

knowledge of various multi-agent reinforcement learning algorithms by comparing their structures and critical

functions to comprehend the algorithm and design of experiments in Ming Tan’s research. At the same time,

the innovation of this project mostly focuses on dealing with communication noise, which require

comprehension in signals and communication systems, a purely engineering discipline. This makes the project

to be interdisciplinary and relies strong engineer background to solve it.

Second, to form this project, I am required to evaluate current challenges and opportunities in multi-agent

learning, to compare the focuses and innovations of existing research through vast literature reading, and to

identify the research gap between those either real-world or theoretical innovations. After I have found the

research gap of analyzing the effect of communication noise on cooperative agents, I will be required to

compare existing error detection algorithms and information communication medium by analyzing their

autoencoder structures or design of matched filter demodulator, so that I can properly add noise in

communication channel and design own suitable noise detection and removing algorithm. Since adding

communication noise between cooperative agents is a research gap, I can make sure the approach is my own.

Last, this project will instill me new knowledge and skillsets. Successful outputs of this project include a well

written report and reusable Python code. Therefore, I will have the opportunity to explore knowledge of multi-

agent learning and reinforcement learning, which I have not yet learnt but is popular in engineering research

and business. I will also further develop my Python programming ability by studying advanced machine

learning and signal processing algorithms used in papers, reproducing them, and writing my own algorithm.

Besides, I will also enhance my professional writing skills through paper reading and from King’s Academy.

Through completing the project, I will do weekly reflection of progress and knowledge learned. Those are all

crucial skillsets as a professional engineer, and I will be trained to develop them through this project.

3/11/2021 3
6CCE3EEP Project Proposal Chenhao Xue 19027096

4. Project Schedule
The project objectives and methods will be carefully broken down into tasks and phases and will be planned

in a weekly manner from October 2021 to April 2022 in this section. A Gannt Chart will be included in this

section to demonstrate the planning. Weekly project reflection, where project status will be inspected and

potential update will be made in schedule, and redundancy will also be built in in the Gannt Chart. In my

weekly timetable, on every Monday, I will focus entirely on my week’s project work, and every Friday

afternoon will be my day for project admin, during which I will have supervisor meetings fortnightly, inspect

my whole week’s work, cross off tasks and update my progress on my Gannt Chart.

Breaking down the objectives and methods, I identify the specific tasks of this project as follow:

Semester 1:

1. Literature review on Multi-agent learning, multi-agent reinforcement learning, and reinforcement learning

fields and algorithms.

2. Study the paper “Multi-agent Reinforcement Learning: Independent vs. Cooperative Agents” (Tan M.,

1993).

3. Analyze the three example experiments in the paper by drawing a table to compare their: inputs, outputs,

task of agents, methods and difficulties of adding and detecting noise between communication of agents.

Decide which experiment to reproduce with specific reasons.

4. Write the pseudo code to reproduce the chosen example experiment. Then, based on the pseudo code, write

actual Python code to reproduce the chosen experiment.

5. Literature review on methods of adding noise in communication channel and detecting it at demodulator.

Write pseudo code of adding noise in my experiment. Then, based on the pseudo code, write actual Python

code to add noise in communication.

6. Compare performance of independent agent and cooperative agent under communication noise. Then,

determine the signal-to-noise ratio that cooperation will break down. Compare performance of cooperative

agents under different signal-to-noise ratio with tables and figures and quantify potential relationships

3/11/2021 4
6CCE3EEP Project Proposal Chenhao Xue 19027096

between SNR and performance.

7. Reflect on my project progress. Inspect potential problems in this first part of project and correct them.

Write report on the findings, results, and discussions of the effect of noise on performance of cooperative

agents in multi-agent reinforcement learning experiment, which is the first part of project.

Figure 1: Semester 1 Gannt Chart

C
re
at Effect of Communication Noise on Multi-agent Reinforcement Learning and a Noise-tolerant Algorithm of Learning to Communicate and Act
Student
Enter Name:
Company Name Chenhao Xue
in cell B2. Planned Duration: Completed: Possible Extension: Starting Semester 2: Date of Project Submission:
Student Number: 19027096 Wed, 10/20/2021
Enter
T the name of the Project Lead in cell B3. Enter the Project Start date in cell E3. Pooject Start: label
Chart
is in
Starting
cell C3.
Week: Week 11 Week 12 Week 13 Week 14 Week 15 Week 16 Week 17 Week 18 Week 19 Week 20 Week 21 Week 22 Week 23
he
1 Oct 18, 2021 Oct 25, 2021 Nov 1, 2021 Nov 8, 2021 Nov 15, 2021 Nov 22, 2021 Nov 29, 2021 Dec 6, 2021 Dec 13, 2021 Dec 20, 2021 Dec 27, 2021 Jan 3, 2022 Jan 10, 2022
Di
C Display Week:
ell
T 18 19 20 21 22 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
hi TASK PROGRESS START END M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S
sC
ell
B
Project Tasks (Semester 1)
C
ell
B
Literature Review on Multi-agent Learning and Reinforcement Learning 90% 10/18/21 11/5/21
C
ell
B
Study Ming Tan's Paper 40% 11/2/21 11/7/21
R
o
w
Analyze Three Experiments; Decide One to Reproduce 0% 11/8/21 11/14/21

Write Pseudo Code for Experiment 0% 11/15/21 11/19/21

Write Python Code to Reproduce Experiment 0% 11/20/21 11/28/21

Inspect and Correct Possible Errors 0% 11/26/21 11/30/21

Read Paper on Noise; Write Pseudo Code of Adding, Detecting Noise 0% 11/26/21 12/5/21

Write Python Code to Add Noise in Communication 0% 12/10/21 12/17/21

Compare Performance of Independent vs Cooperative Agents 0% 12/18/21 12/22/21

Calculate Signal-to-Noise Ratio; Draw Table and Figure 0% 12/21/21 12/27/21

Reflect on Current Progress; Inspect and Correct Problems 0% 12/18/21 12/27/21

Write Report on This First Part of Project 0% 12/18/21 12/27/21

T
he
ce
First Semester Courseworks (Some are Still Not Sure)

Computer Vision Assessed Coursework 0% 11/4/21 11/7/21

Communication Systems Coursework 1 0% 11/7/21 11/9/21

Brain-Inspired Computing & Hardware Design Literature Review 0% 11/19/21 11/26/21

Brain-Inspired Computing & Hardware Design Group Project 0% 12/6/21 12/16/21

Communication Ststems Coursework 2 0% N.A. N.A.

First Semester Revisions and Exams (Not Sure Yet)

Sample phase title block

Computer Vision Revision 0% 12/27/21 1/2/22

Communication Systems Revision 0% 1/3/22 1/9/22

Brain-Inspired Computing & Hardware Design Revision 0% 1/10/22 1/16/22

Semester 2:

(Since coursework and exams of Semester 2 are not yet available, the actual time of tasks may be changed.

This will be dtermined at project admin time on 28th January 2022 or 4th February 2022.)

8. Study the paper “Learning to Communicate with Deep Multi-agent Reinforcement Learning” (Foerster

J.N. et al., 2016).

9. Write pseudo code to reproduce the set of multi-agent benchmark tasks that require communication and

have built-in noise. Then, based on the pseudo code, write actual Python code to reproduce those tasks.

10. Literature review on noise-tolerant algorithms in communication systems, and on the design of

communications in multi-agent reinforcement learning algorithms.

11. Design and write pseudo code of the formulated noise-tolerant multi-agent reinforcement learning

algorithm of these benchmark tasks. Then, implement the algorithm with actual Python code.

3/11/2021 5
6CCE3EEP Project Proposal Chenhao Xue 19027096

12. Analyze the performance of the formulated noise-tolerant multi-agent reinforcement learning algorithm.

Compare its performance with the Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent

Learning (DIAL) algorithms formulated in Foerster J.N.’s paper.

13. Write report on the findings, results, and discussions of the noise-tolerant multi-agent reinforcement

learning algorithm, and its performance with potential future improvements and research, which is the

second part of project.

14. Write first draft of project report based on the previously written report of first and second parts of project.

15. Discuss the first draft with supervisor, correct potential errors, and modify the report.

16. Write the final draft of project report, inspect and correct errors again, and submit the report and codes.

17. Record the demonstration video and submit.

Figure 2: Semester 2 Gannt Chart:

C
re
at Effect of Communication Noise on Multi-agent Reinforcement Learning and a Noise-tolerant Algorithm of Learning to Communicate and Act
Student
Enter Name:
Company Name Chenhao Xue
in cell B2. Planned Duration: Completed: Possible Extension: Starting Semester 2: Date of Project Submission:
Student Number: 19027096 Wed, 10/20/2021
Enter the name of the Project Lead in cell B3. Enter the Project Start date in cell E3. Pooject Start: label
T Chart
is in
Starting
cell C3.
Week: Week 11 Week 12 Week 13 Week 14 Week 15 Week 16 Week 17 Week 18 Week 19 Week 20 Week 21 Week 22 Week 23 Week 24 Week 25 Week 26 Week 27 Week 28 Week 29 Week 30 Week 31 Week 32 Week 33 Week 34 Week 35 Week 36
he 1 Oct 18, 2021 Oct 25, 2021 Nov 1, 2021 Nov 8, 2021 Nov 15, 2021 Nov 22, 2021 Nov 29, 2021 Dec 6, 2021 Dec 13, 2021 Dec 20, 2021 Dec 27, 2021 Jan 3, 2022 Jan 10, 2022 Jan 17, 2022 Jan 24, 2022 Jan 31, 2022 Feb 7, 2022 Feb 14, 2022 Feb 21, 2022 Feb 28, 2022 Mar 7, 2022 Mar 14, 2022 Mar 21, 2022 Mar 28, 2022 Apr 4, 2022 Apr 11, 2022
Di
C Display Week:
ell
T 18 19 20 21 22 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17
hi
TASK PROGRESS START END M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S M T W T F S S
sC
ell
B
Project Tasks (Semester 1)
C
ell
B
Literature Review on Multi-agent Learning and Reinforcement Learning 90% 10/18/21 11/5/21
C
ell
B
Study Ming Tan's Paper 40% 11/2/21 11/7/21
R
o
w
Analyze Three Experiments; Decide One to Reproduce 0% 11/8/21 11/14/21

Write Pseudo Code for Experiment 0% 11/15/21 11/19/21

Write Python Code to Reproduce Experiment 0% 11/20/21 11/28/21

Inspect and Correct Possible Errors 0% 11/26/21 11/30/21

Read Paper on Noise; Write Pseudo Code of Adding, Detecting Noise 0% 11/26/21 12/5/21

Write Python Code to Add Noise in Communication 0% 12/10/21 12/17/21

Compare Performance of Independent vs Cooperative Agents 0% 12/18/21 12/22/21

Calculate Signal-to-Noise Ratio; Draw Table and Figure 0% 12/21/21 12/27/21

Reflect on Current Progress; Inspect and Correct Problems 0% 12/18/21 12/27/21

Write Report on This First Part of Project 0% 12/18/21 12/27/21

T
he
ce
First Semester Courseworks (Some are Still Not Sure)

Computer Vision Assessed Coursework 0% 11/4/21 11/7/21

Communication Systems Coursework 1 0% 11/7/21 11/9/21

Brain-Inspired Computing & Hardware Design Literature Review 0% 11/19/21 11/26/21

Brain-Inspired Computing & Hardware Design Group Project 0% 12/6/21 12/16/21

Communication Ststems Coursework 2 0% N.A. N.A.

First Semester Revisions and Exams (Not Sure Yet)

Sample phase title block

Computer Vision Revision 0% 12/27/21 1/2/22

Communication Systems Revision 0% 1/3/22 1/9/22

Brain-Inspired Computing & Hardware Design Revision 0% 1/10/22 1/16/22

C
ell
B
Project Tasks (Semester 2)
C
ell
B
Study Foerster J.N.'s Paper 0% 1/17/22 1/23/22

Write Pseudo Code for Benchmark Tasks 0% 1/24/22 1/30/22

Write Python Code to Reproduce Beenchmark Tasks 0% 1/29/22 2/6/22

Inspect and Correct Possible Errors 0% 1/28/22 2/11/22

Literature Review on Noise-tolerant and Communication 0% 2/7/22 2/16/22

Design and write Pseudo Code for Noise-tolerant Algorithm 0% 2/14/22 2/21/22

Implement the Algorthm with Python Code 0% 2/21/22 3/2/22

Analyze, Compare Performance between Formulated and Original Algorithm 0% 3/3/22 3/14/22

Inspect and Correct Possible Errors 0% 3/14/22 3/20/22

Write Report on This Second Part of Project 0% 3/14/22 3/28/22

Write First Draft of Final Project Report 0% 3/28/22 4/3/22

Discuss First Draft with Supervisor and Modify 0% 4/1/22 4/5/22

Write Final Draft of Project Report, Inspect Again, and Submit 0% 4/5/22 4/12/22

Record Project Demonstration Video and Submit 0% 4/5/22 4/13/22

Second Semester Courseworks (Undecided Yet)

Sample phase title block

Task 1 0% N.A. N.A.

Task 2 0% N.A. N.A.

Task 3 0% N.A. N.A.

Task 4 0% N.A. N.A.

Task 5 0% N.A. N.A.

Second Semester Revisions and Exams (Undecided Yet)

This is an empty row

Hardware Design 0% N.A. N.A.

Machine Learning for Engineers 0% N.A. N.A.

Real Time Systems and Control 0% N.A. N.A.

3/11/2021 6
6CCE3EEP Project Proposal Chenhao Xue 19027096

5. General Project Risks: Ethics, Health and Safety,

GPDR, Cost Considerations
My project feasibility and general risk assessment is shown in the table below. Furthermore, explanations of

why my project will not include Ethics and GPDR considerations are also given below.

Explanations for Ethnics and GPDR:

About the ethnics, this project will not require any sort of human data, because this project will conduct

experiments only based on virtual agents, their cooperation and communication. No data collection from

human participants and analysis of pre-existing human participants data is involved. The research and

experiment process of this project will be carried out through computer programmed environment and no

human data will be involved. Also, the potential outcomes of this project are still at a relatively early stage in

academics and could not be applied to human environments now. Therefore, ethnics will not be a risk in this

project.

About GPDR, this project will not include any sort of personally identifiable data. So, no personally

identifiable data will be collected, analyzed or stored. As explained above, this project has no human

participants involved and is not considering referencing personal data or relating project outcomes to human

applications. Thus, GPDR will not be a risk in this project.

(Project feasibility and general risk assessment is on the next page.)

3/11/2021 7
6CCE3EEP Project Proposal Chenhao Xue 19027096

Table 1: Feasibility and General Risk Assessment

Category Risk Risk Impact
Health & Safety 1. Physical injury during commuting Requiring for recovering will lead to
between dormitory and campus. 2. delay in writing pseudo codes and
Fail ill or infected by COVID19. 3. actual Python codes as well as
Electric shock or fire hazard when supervisor meetings. Significant injury
using electronic devices. or illness may require a change in
participant and update in safety
protocol.
Health & Safety Mental stress from project workload Reduce performance in the individual
and research outcome. project and in other modules. Project
delays and potential extensions caused
by inability to proceed the work.
Ethnics No ethical risks will be in place, N/A N/A
because this project does not
require human data, neither human
participants, nor pre-existing human
data. Specific explanations is
following this table.
GPDR No GPDR risks will be in place, N/A N/A
because this project will not include
personally identifiable data. Specific
explanations is following this table.
Method Data Loss Significant delay to projects, because
the experiments will have to conducted
again and analysis could not proceed
without data.
Cost Unexpected budget requirement May lead to delay because of waiting
for extra money to be transferred.
Technical Inadvertant plagiarism Failure to correstly reference cited
paper/code/research will lead to lower
project mark and even project failure.
Technical Project not aligned to assessment Failure to align project objectives and
criteria and fail fulfill learning deliverables to marking criteria, and to
outcomes fulfill the expected learning outcomes
will result in a low mark of the project.
Technical Project objectives cannot be Failure to achieve project objectives will
achieved lead to failure of outputing desired
project deliverables and will result in a
low mark of the project
Technical Project not including enough If the project is not challenging enough,
challenge. it will provide insufficient knowledge
and skillsets to researcher, and will not
effectively fulfill learning outcomes. This
will result in a lower mark of the project.
Technical Equipment faliure (e.g. laptop is Unable to conduct experiments and
down) write Python code implementations.
Fixing the equipment may have
financial impact. Changing to a new
workstation will cause delay in
reconstructing experiments and
algorithms.
Time Academic workload/injury and Extra issues may negatively affect the
illness/personal issue/extra project project to achieve its outputs, such as
workload missing the deadline, project delay,
extension, and reduce in mark.
Severity (1-5) Probability (1-5) Risk Score (S*P)
4 2
4 3
N/A N/A
N/A N/A
4 2
2 2
5 1
3 3
3 4
2 2
3 2
4 2
Mitigation Action
8 Risk assessment has been
written. Specific attention will
be paid to avoid them from
happening.
12 A project plan that has built-in
redundancy will overcome
potential extra workload.
Meetings with supervisor and
health and wellbeing supports
service will help alleviate the
stress.
N/A
N/A
8 Saving data in multiple
locations such as laptop and
phone. Uploading data to
cloud such as OneDrive,
GoogleDrive, or Baidu
Wangpan, weekly.
4 Main equipment, the laptop,
will be carefully used and
regularly inspected.
Computers are also available
on campus in case laptop
breaks down.
5 The topic of this research and
core references has been
clearly identified. The
contextual knowledge
required are also identified
and researched carefully.
Inspection of project content
and progress will be made
weekly.
9 The close relation of this
project with learning
outcomes has been discussed
in this project proposal. There
will be enough time for
potential changes in project
objectives to make it better
aligned to assessment criteria
and learning outcomes.
12 Frequently communicate with
project supervisor about work
progress and difficulties to
solve encourted problems.
Regularly inspect project work
to ensure I can identify
potential difficulties and
problems early, so that I will
have enough time to solve
them.
4 Since this project is targeting a
research gap in the field, there
will be an extra challenge in
achieving desired outputs.
Also, the result of project may
demonstrate the original
expected outcome is wrong,
which will make this project
much more challenging.
6 Correctly use the equipment
with regular inspection.
Uploading project progress to
the cloud to allow proceeding
of work on computers in
campus.
8 Project management will help
managing the workload and
emergencies. This include an
effective Gant Chart, regular
supervisor meetings and
frequent discussions.
Contingency Action
Project Progress will be uploaded to
OneDrive or GoogleDrive every
week, and two participants
(researcher and supervisor) allows
the other participant to proceed.
If the project becomes too difficult to
proceed, some objectives and
deliverables can be adjusted. Also,
applying for mitigating
circumstances and requesting a
project extension are potential
actions.
N/A
N/A
Repeat experiments to collect data
again. In extreme cases, applying for
mitigating circumstances and
requesting a project extension are
potential actions.
Purchase required equipment with
extra budget.
Work with marker/supervisor to
tackle plagiarism problems. And
investigation records will be provided
as evidence.
Modify the objectives of this project
to better align it with assessment
criteria and learning outcomes.
Adjust project objectives, narrow
down the scope of it, and simplify
the tasks in achieving the objectives.
Also, confirm the feasibility with
supervisor. In extreme
circumstances, applying for
mitigating circumstances and
requesting a project extension are
potential actions.
Adjust the project aim and scope,
and extend the contents in project
objectives to make it more
challenging.
Use up remaining budget. Also,
applying for mitigating
circumstances and requesting a
project extension are potential
actions.
Under extreme circumstances,
applying for mitigating
circumstances and requesting a
project extension are potential
actions.

For risk scores, 1-6 is green, 7-11 is yellow, above 12 is brown.

3/11/2021 8
6CCE3EEP Project Proposal Chenhao Xue 19027096

Though this project is challenging, the deliverables of this project are feasible. The project report will include

the findings, results, and discussions of the whole process, both the relatively straightforward part 1 and the

complicated step 2. So, the project report is guaranteed to be substantial. The codes will include the reproduced

experiments with noise added, the reproduced benchmark tasks, and a noise-tolerant algorithm. Since the

design of experiment and benchmark tasks are provided in reference papers, they are less challenging to

achieve. Also, the demonstration video will be based on the report and literature view, so it will be feasible as

well. Besides, although it is not guaranteed that cooperative agents outperform independent agent under noise,

and that the noise-tolerant algorithm outperforms traditional methods, the process of studying effect of noise

on cooperative agents, and the design of noise-tolerant cooperative agents learning algorithm are feasible to

achieve and are sufficiently valuable for the deliverables of this project.

Lastly, the disruption of Covid-19 on this project is acceptable because the tasks can all be conducted remotely.

The experiments, data acquisition and analysis are carried out on computers and designed virtual environments,

and supervisor meetings can also be held through Teams. Though Covid-19 may negatively affect participants

safety and health conditions, the overall impact on this project will be acceptable with careful self-protection.

3/11/2021 9
6CCE3EEP Project Proposal Chenhao Xue 19027096

Reference
Tuyls, K., & Weiss, G. 2012. Multiagent Learning: Basics, Challenges, and Prospects. AI Magazine, 33(3), 41.

https://doi.org/10.1609/aimag.v33i3.2426

Weiss, G., and Dillenbourg, P. 1999. What Is “Multi” in Multiagent Learning? In Collaborative Learning:

Cognitive and Computational Approaches, ed. P. Dillenbourg, 64–80. Pergamon Press.

Taylor, M. E., and Stone, P. 2009. Transfer Learning for Reinforcement Learning Domains: A Survey. Journal

of Machine Learning Research 10(1): 1633–1685

Dorigo, M., and Stützle, T. 2004. Ant Colony Optimization. Cambridge, MA: The MIT Press/Bradford Books.

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. Arcas 2017. Communication-efficient learning

of deep networks from decentralized data, in Proc. AISTATS, pp. 1273-1282

Tan M. 1993. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the

tenth international conference on machine learning (pp. 330-337).

[https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.84.3701&rep=rep1&type=pdf]

Foerster JN, Assael YM, De Freitas N, Whiteson S. 2016. Learning to communicate with deep multi-agent

reinforcement learning. arXiv preprint arXiv:1605.06676. [https://arxiv.org/abs/1605.06676]

3/11/2021 10