INFRARED THERMAL IMAGING CAMERA

PRESENTED BY:
Ali Riyadh, Abbas Shyal, & Essam Ramadan
2ND STAGE (EVENING STUDY)
DR. OMAR YOUSSEF
 TABLE OF CONTENTS:

INTRODUCTION…………………………………………….2-5

MLX90640……………………………………………………….5-7

PARTS …………………………………………………....…….7 -8

LIBRARIES AND CODE ……….…………………………………....8-11

VISUALIZATION ……………………………………11-12

COMPLETE CIRCUIT, AND RESULTS……………….……….13-14

CODE……………………………………………….…………15-18

1|Page
 INFRARED THERMAL IMAGING CAMERA

INTRODUCTION:

In the last decade with the development of new sensors

and sensor networks the attention of many researches is

focused on providing personalized services for users in their

daily life, human localization or occupancy detection, activities

recognition, and security issues. Vision-based human

localization and actions recognition is under intensive research

in the field of Ambient Assisted Leaving (AAL) and assistive

technologies as the ageing population rapidly growing in a

worldwide scale. However, the traditional cameras are not

suitable for human localization and activities recognition in

home environment mainly because of privacy concerns. This

motivates the research groups to look for other suitable

technologies and sensors and naturally lead to application of

IR array sensors for indoor environment. As mention in despite

of increasing interest just a few manufacturers produce such

devices and typically they have low resolution of 8x8, 16x4 or

2|Page
16x16 pixels. There are many applications of these sensors in

building automation, including fire detection, home

observation, temperature monitoring, hazardous events and so

on. Recently the attention is focused on application of IR array

sensors for human localization and action recognition in AAL

and smart home systems. The limited resolution of the

employed IR arrays has initiated designs with multi-sensor

systems. Additionally, different classification methods like:

Random Forest, Support Vector Machine with Stochastic

Gradient Descent training and k-Nearest Neighbors are

applied. In the reported 97% accuracy in human detection using

low-resolution IR array sensor and complicated algorithms for

noise removal, background estimation and probabilistic

foreground detection. But they recognize that the main

limitation of the proposed method is that it is hard to

distinguish a human presence from other moving heat sources.

In the same time, having a real time infrared array sensor for

detection of human presence in a room, it is very convenient

3|Page
the same sensor to be used for monitoring the state of objects

inside, thus being useful for the smart home automation and

home security systems. For example, the image from a certain

object (kitchen, home appliance, etc.) can be processed and

evaluated in order to determine if there is a potential

overheating, starting a fire or someone has forgotten to switch

off the stove or iron. If such scenario has occurred the system

will notify the user or the fire department, depending on the

developed top applications. In result of our review we came to

the conclusions that: first, for human detection it is necessary

to use IR sensor array with reasonably higher resolution still

keeping the privacy, and second, to integrate this sensor with

modules and services, based on smart home technologies, with

the aim to detect critical conditions or predict them on early

stage, to send alert to caregivers, relatives, etc. and take the

corresponding actions. The purpose of this article is to present

and demonstrate a cost-effective wireless sensor device that

can be applied in the field of health care, AAL and security

4|Page
smart systems in home environment. It is designed as an IoT

solution and it employs a new infrared array sensor. This paper

is organized as follows: section 2 presents the new infrared

array sensor and its main features and capabilities. The

implementation of the IoT solution integrating Melexis

MLX90640 infrared array sensor for monitoring of human

presence and home appliances. The conclusions and the future

work are presented in the last section.

INFRARED ARRAY SENSOR MLX90640:

The sensor that is selected for implementation of the IoT

device is the MLX90640, produced by Melexis [7]. The

MLX90640 is a fully calibrated 32x24 pixels thermal IR array in

an industry standard 4-lead TO39 package with digital

interface. The MLX90640 contains 768 FIR pixels. An ambient

sensor is integrated to measure the ambient temperature of the

chip and supply sensor to measure the VDD. The outputs of all

5|Page
sensors IR, Ta and VDD are stored in internal RAM and are

accessible through I2C interface. The sensor is suitable for

development of our IoT solution with its wide field of view (it

has two options for field of view - 55°x35° and 110°x75°) and

relatively good spatial resolution for low-cost infrared camera.

Its current consumption is less than 23mA, which makes is

suitable even for battery powered solution. The refresh rate is

between 0.5 and 64Hz, meaning that the sensor is capable of

sensing very fast moving objects. The possible applications for

the sensor are: - High precision non-contact temperature

measurements; - Intrusion / Movement detection; - Presence

detection / Person localization; - Temperature sensing element

for intelligent building air conditioning; - Thermal Comfort

sensor in automotive Air Conditioning control system; -

Industrial temperature control of moving parts. Due to the

factory calibrated temperature measurement with calibration

parameters stored in the internal EEPROM it has high precision

of measurement. All these features are a good reason for

6|Page
selection of this sensor to be used in development of solutions

for Indoor Human Detection, Room Occupancy, Tracking

Motion, and continuous monitoring of critical electrical

appliances in smart home.

PARTS FOR THE PROJECT:

1-Melexis MLX90640

2-ESP32

3-ILI9341 320x240 display

4-DuPont wire

7|Page
You can get the MLX90640 for example from sparkfun. There

are two different sensor-types, one with 110°x75° field of view

(MLX90640BAA) and the other with 55°x35° fov

(MLX90640BAB). Then you'll need an ESP32.

To graph the temperatures, you will need a display too. I chose

a 320x240 ILI9341 display. It offers touchscreen and a SD-

card-slot but for the thermal imaging camera you just need the

9 pins for the display and power supply.

At last you need some female-female DuPont wires for the

connections between the ESP, the display and the sensor.

NECESSARY LIBRARIES AND CODE:

Here you can find the Arduino-code from sparkfun for the

MLX90640:

https://github.com/sparkfun/SparkFun_MLX90640_Ardu...

8|Page
I tried to run the code with an Arduino Due but I didn't succeed.

In the web I found several threads from other users who had

similar problems with an Arduino-board. I didn't get

temperatures but the values "not a number" (nan).

Therefore, I switched to an ESP32-board. Installing the ESP32

with your Arduino-IDE is very simple. After a short time you'll

be able to select among many ESP32-boards in the boards

manager of the Arduino-IDE. For my board I choose "ESP32 Dev

Module".

For the display you need two libraries from adafruit:

 The ILI9341

library: https://github.com/adafruit/Adafruit_ILI9341

 the GFX-library: https://github.com/adafruit/Adafruit-

GFX-Library

Then I tried to run the Sparkfun-MLX90640-code now with the

ESP32. Fortunately, I found out that MLX90640-sensor worked

with special values in the memory/register of the sensor. The

9|Page
value at the position 0x800D has to be 1901 (HEX) = 6401

(DEC). I inserted the line

MLX90640_I2CWrite(0x33, 0x800D, 6401);

in the setup-routine of the Arduino-program. But with this

modification I still got error messages while compiling the

code. Another hint was very useful: You must add the

line #include <Arduino.h> ; at the beginning of the

file MLX90640_I2C_Driver.cpp!

10 | P a g e
Now you should get reasonable values from the MLX90640-

sensor. Maybe some pixels are not working or show wrong

temperatures. In this case you have to change the value of the

pixel by hand. I chose a simple linear interpolation with the two

neighbor-pixels. If f.e. the 1*32 + 21 st pixel is faulty the code

will be:

mlx90640To[1*32 + 21] = 0.5 * (mlx90640To[1*32 + 20] +

mlx90640To[1*32 + 22]);

But this only works when the faulty pixel isn't exactly located

on one of the two sides of the picture.

VISUALIZATION OF THE TEMPERATURES:

To visualize the 768 temperatures, I created a simple

conversion of values between [0,180] to RGB-values. The

temperatures of all pixels are being transformed to the interval

[0,180] with the function 180 * (T_pixel - T_minimum) /

(T_maximum - T_minimum). So you get similar colors like

11 | P a g e
those you know from the company flir. below you can see a test

run with random temperatures

12 | P a g e
COMPLETE CIRCUIT, CODE AND RESULTS:

After copying the program you'll have to connect the display

with the ESP32 (look at the picture and code added). During the

upload you first have to press the BOOT-putton on the ESP32.

Afterwards you must press the EN-button. Then if everything

worked fine you'll see the infrared thermal image on the

display.

The whole camera will be powered by one Li-ion battery

followed by a step-up-converter to provide +5V for the ESP32.

Don't go much higher than 5V, otherwise the on board voltage-

13 | P a g e
regulator will get too hot. At the moment I'm just waiting for

the battery holder. Then this project will be finished.

14 | P a g e
Code:

#include <Wire.h> MLX90640 in open 90640,

#include air while (!Serial); &mlx90640);
"MLX90640_API.h" //Wait for user to
#include static float open terminal if (status != 0)
"MLX90640_I2C_D mlx90640To[768]; {
river.h" paramsMLX90640
#include "SPI.h" mlx90640; Serial.println("MLX Serial.println("Para
#include 90640 IR Array meter extraction
"Adafruit_GFX.h" int xPos, yPos; Example"); failed");
#include // Abtastposition Serial.print("
"Adafruit_ILI9341.h int R_colour, if (isConnected() status = ");
" G_colour, B_colour; == false)
// RGB-Farbwert { Serial.println(status
// For the ESP- int i, j; );
WROVER_KIT, // Zählvariable Serial.println("MLX }
these are the float T_max, T_min; 90640 not detected
default. // maximale bzw. at default I2C //Once params
#define TFT_CS minimale address. Please are extracted, we
15 gemessene check wiring. can release
#define TFT_DC 2 Temperatur Freezing."); eeMLX90640 array
#define TFT_MOSI float T_center; while (1);
13 // Temperatur in der }
#define TFT_CLK Bildschirmmitte MLX90640_I2CWrit
14 e(0x33, 0x800D,
#define TFT_RST Serial.println("MLX 6401); // writes the
26 90640 online!"); value 1901 (HEX) =
#define TFT_MISO 6401 (DEC) in the
12 // //Get device register at position
#define TFT_LED ********************** parameters - We 0x800D to enable
27 ***************** only have to do this reading out the
// **************** once temperatures!!!
Adafruit_ILI9341 tft SETUP int status; //
= **************** uint16_t ===============
Adafruit_ILI9341(T // eeMLX90640[832]; ===============
FT_CS, TFT_DC, ********************** ===============
TFT_MOSI, ***************** status = ===============
TFT_CLK, MLX90640_DumpE ===============
TFT_RST, void setup() E(MLX90640_addr ===============
TFT_MISO); { ess, eeMLX90640); ===============
===============
const byte Serial.begin(11520 if (status != 0) ===============
MLX90640_addres 0); ===============
s = 0x33; //Default Serial.println("Faile =========
7-bit unshifted Wire.begin(); d to load system
address of the parameters");
MLX90640 Wire.setClock(4000 //MLX90640_SetRe
00); //Increase I2C status = freshRate(MLX906
#define TA_SHIFT clock speed to MLX90640_Extract 40_address, 0x00);
8 //Default shift for 400kHz Parameters(eeMLX

15 | P a g e
//Set rate to 0.25Hz tft.setRotation(1); _address,
effective - Works tft.setCursor(80, mlx90640Frame);
220);
//MLX90640_SetRe tft.fillScreen(ILI934 if (status < 0)
freshRate(MLX906 1_BLACK); tft.setTextColor(ILI9 {
40_address, 0x01); tft.fillRect(0, 0, 341_WHITE,
//Set rate to 0.5Hz 319, 13, tft.color565(0, 0, Serial.print("GetFra
effective - Works tft.color565(255, 0, 0)); me Error: ");
10)); tft.print("T+ = ");
//MLX90640_SetRe Serial.println(status
freshRate(MLX906 tft.setCursor(100, );
40_address, 0x02); 3); // drawing the }
//Set rate to 1Hz colour-scale
effective - Works tft.setTextSize(1); // float vdd =
=============== MLX90640_GetVdd
//MLX90640_SetRe tft.setTextColor(ILI9 ========= (mlx90640Frame,
freshRate(MLX906 341_YELLOW, &mlx90640);
40_address, 0x03); tft.color565(255, 0, for (i = 0; i < 181; float Ta =
//Set rate to 2Hz 10)); i++) MLX90640_GetTa(
effective - Works { mlx90640Frame,
tft.print("Thermogra //value = &mlx90640);
MLX90640_SetRefr phie - stoppi"); random(180);
eshRate(MLX9064 float tr = Ta -
0_address, 0x04); tft.drawLine(250, getColour(i); TA_SHIFT;
//Set rate to 4Hz 210 - 0, 258, 210 - //Reflected
effective - Works 0, tft.color565(255, tft.drawLine(240, temperature based
255, 255)); 210 - i, 250, 210 - i, on the sensor
//MLX90640_SetRe tft.drawLine(250, tft.color565(R_colo ambient
freshRate(MLX906 210 - 30, 258, 210 - ur, G_colour, temperature
40_address, 0x05); 30, B_colour)); float emissivity
//Set rate to 8Hz tft.color565(255, } = 0.95;
effective - Works at 255, 255));
800kHz tft.drawLine(250, }
210 - 60, 258, 210 - MLX90640_Calcula
//MLX90640_SetRe 60, teTo(mlx90640Fra
freshRate(MLX906 tft.color565(255, me, &mlx90640,
40_address, 0x06); 255, 255)); // emissivity, tr,
//Set rate to 16Hz tft.drawLine(250, ********************** mlx90640To);
effective - Works at 210 - 90, 258, 210 - ************ }
800kHz 90, // **************
tft.color565(255, LOOP **************
//MLX90640_SetRe 255, 255)); // // determine
freshRate(MLX906 tft.drawLine(250, ********************** T_min and T_max
40_address, 0x07); 210 - 120, 258, 210 ************ and eliminate error
//Set rate to 32Hz - 120, pixels
effective - fails tft.color565(255, void loop() //
255, 255)); { ===============
tft.drawLine(250, for (byte x = 0 ; x ===============
210 - 150, 258, 210 < 2 ; x++) //Read ===============
pinMode(TFT_LED, - 150, both subpages =======
OUTPUT); tft.color565(255, {
255, 255)); uint16_t
digitalWrite(TFT_L tft.drawLine(250, mlx90640Frame[83 mlx90640To[1*32 +
ED, HIGH); 210 - 180, 258, 210 4]; 21] = 0.5 *
- 180, int status = (mlx90640To[1*32
tft.begin(); tft.color565(255, MLX90640_GetFra + 20] +
255, 255)); meData(MLX90640 mlx90640To[1*32 +

1|Page
22]); // eliminate // ===== determine
the error-pixels // determine tft.fillRect(260, the colour ====
T_center 25, 37, 10, //
mlx90640To[4*32 + // tft.color565(0, 0, ===============
30] = 0.5 * =============== 0)); ===============
(mlx90640To[4*32 === tft.fillRect(260, =
+ 29] + 205, 37, 10,
mlx90640To[4*32 + T_center = tft.color565(0, 0, void getColour(int j)
31]); // eliminate mlx90640To[11* 32 0)); {
the error-pixels + 15]; tft.fillRect(115, if (j >= 0 && j <
220, 37, 10, 30)
T_min = // drawing the tft.color565(0, 0, {
mlx90640To[0]; picture 0)); R_colour = 0;
T_max = // G_colour = 0;
mlx90640To[0]; =============== B_colour = 20
==== tft.setTextColor(ILI9 + (120.0/30.0) * j;
for (i = 1; i < 768; 341_WHITE, }
i++) for (i = 0 ; i < 24 ; tft.color565(0, 0,
{ i++) 0)); if (j >= 30 && j <
{ 60)
if((mlx90640To[i] > - for (j = 0; j < tft.setCursor(265, {
41) && 32; j++) 25); R_colour =
(mlx90640To[i] < { tft.print(T_max, (120.0 / 30) * (j -
301)) 1); 30.0);
{ mlx90640To[i*32 + G_colour = 0;
j] = 180.0 * tft.setCursor(265, B_colour =
if(mlx90640To[i] < (mlx90640To[i*32 + 205); 140 - (60.0/30.0) * (j
T_min) j] - T_min) / (T_max tft.print(T_min, - 30.0);
{ - T_min); 1); }
T_min =
mlx90640To[i]; tft.setCursor(120, if (j >= 60 && j <
} getColour(mlx9064 220); 90)
0To[i*32 + j]); tft.print(T_center, {
1); R_colour =
if(mlx90640To[i] > 120 + (135.0/30.0) *
T_max) tft.fillRect(217 - j * 7, (j - 60.0);
{ 35 + i * 7, 7, 7, tft.setCursor(300, G_colour = 0;
T_max = tft.color565(R_colo 25); B_colour = 80
mlx90640To[i]; ur, G_colour, tft.print("C"); - (70.0/30.0) * (j -
} B_colour)); 60.0);
} } tft.setCursor(300, }
else if(i > 0) // } 205);
temperature out of tft.print("C"); if (j >= 90 && j <
range tft.drawLine(217 120)
{ - 15*7 + 3.5 - 5, tft.setCursor(155, {
11*7 + 35 + 3.5, 217 220); R_colour =
mlx90640To[i] = - 15*7 + 3.5 + 5, tft.print("C"); 255;
mlx90640To[i-1]; 11*7 + 35 + 3.5, G_colour = 0 +
} tft.color565(255, delay(20); (60.0/30.0) * (j -
else 255, 255)); } 90.0);
{ tft.drawLine(217 B_colour = 10
- 15*7 + 3.5, 11*7 + - (10.0/30.0) * (j -
mlx90640To[i] = 35 + 3.5 - 5, 217 - 90.0);
mlx90640To[i+1]; 15*7 + 3.5, 11*7 + // }
} 35 + 3.5 + 5, ===============
} tft.color565(255, =============== if (j >= 120 && j <
255, 255)); = 150)

2|Page
 { R_colour =
R_colour = 255; //Returns true if the if
255; G_colour = MLX90640 is (Wire.endTransmis
G_colour = 60 235 + (20.0/30.0) * detected on the I2C sion() != 0)
+ (175.0/30.0) * (j - (j - 150.0); bus return (false);
120.0); B_colour = 0 + boolean //Sensor did not
B_colour = 0; 255.0/30.0 * (j - isConnected() ACK
} 150.0); {
} return (true);
if (j >= 150 && j Wire.beginTransmi }
<= 180) } ssion((uint8_t)MLX
{ 90640_address);

3|Page