HSRW EOLab Students Wiki

HSRW AMC Project
Yongyi Wang(27634),Fonyuy Stacy-Kpufotitin(29820),Wai Lin Maung(28690)
1. Introduction
EU The Birds Directive (Directive 79/409/EEC) The European Union adopted the birds directive in 1979 in order to conserve the wild birds in the EU by setting regulations for their protection, management and control. The directive covers the protection of their eggs, nests and habitats. The member countries have the responsibility to monitor, maintain or restore the population of endangered species and ensure the biodiversity of the species of birds in their native ecosystems. Therefore the monitoring systems are important for preservation efforts and monitoring and tracking of bird species and population are considered basic requirements and carry certain ecological, scientific and cultural significance. Because birds, with their fascinating beauty and astounding powers, have long captivated both nature enthusiasts and scientists. Avian species continue to fascinate us with their extraordinary movements and behaviors, from the epic migrations of great raptors across continents to the subtle foraging behaviors of small songbirds in local environments.
Our project is based on the idea to monitor birds in the environment with a simple arduino kit. This main idea of the project is to build an inexpensive monitoring system with simplified components which can be set up in the local ecosystems of the birds non-intrusively. The system automatically takes pictures when triggered by the PIR motion detection sensor at intervals of five minutes. It is powered by a 9V battery which is in turn being charged by PV modules on the roof of housing. The low power design means the camera is in deep sleep mode unless the PIR signal turns it on and it goes right back to sleep after the operation cycle which include the image capture, the wifi and MQTT connection and the upload to local storage(SD card) and MQTT server is completed. The additional features include the access from “Home Assistant” as a video stream and additional control which can be added as necessary.
Fig1 : Bird house design (source-Wang)

2.Material and method
2.1 Material

● ESP 32-Cam ESP 32-Cam is a low cost development board with an onboard camera. It is suitable for our application as it supports WiFi, Bluetooth and SD card. Even though on the datasheet, it supports an SD card of 4GB storage for our part it still functions on 16GB. The specification can be assessed here in the datasheet below.
Fig2: Pins configuration on Esp32cam module (source-digikey)
Esp32 cam can be powered by either 3.3V or 5V but when flashing on 3.3 V, esp32 cam error 0x20004 with camera probe failure was received. Therefore to be on the safe side, it should be powered with 5V. Gpio 3 and 1 are serial pins and transfer data and used to upload codes to the camera module. Esp32 In flashing mode:GPIO 0 is important when uploading codes because to set the camera to flashing mode it has to be grounded. Afte the code is uploaded, remove the connection and unground the GPIO 0, for the camera module to start working. \\ Fig3: Esp32 cam on flashing mode( source: Wai Lin)
One of the disadvantages of the ESP32-cam is that it has no USB connection and cannot be directly connected to the computer. Hence FDTI adapter is used which supports UART serial connection with the camera module. ESP32 cam- Datasheet from digikey data sheet:

● UartSBee V5 UartSBee V5 is FTDI cable compatible USB to Serial adapter equipped with BEE socket(20pin 2.0mm). The integrated FT232RL can be used for programming or communicating with MCUs. UARTSBEE by seeedstudio

● PIR Sensor
Fig 4: PIR sensor description (source: Adafruit )
PIR referred to as “Passive Infrared”, “Pyroelectric”, or “IR motion” sensors, they are small, low power and inexpensive motion sensors. It has a sensitivity range up to 6 meters high and a range of 110° to 70°. There are two IR sensitive slots in 1 PIR sensor, when heat signature such as animals and human passes by, a positive differential change is created, when it leaves the detection area, a negative differential change is then created, which creates an output signal. When motion is detected, the digital output will be high (3V), when there is no motion, digital output will be low again. Since the sensor only detects heat signatures, it is perfect for bird houses, as it will not be triggered by other motions such as falling leaves. Adafruit PIR sensor datasheet

● Solar Panel
● Battery 9V
● Power Control Module
● Female-to-female jumper wires
● Female-to-male jumper wires
● Micro SD card

2.2 Method

ESP32 Library :If you have not installed ESp32 boards library on your ardunio programmer, Go to File> Prerfercences (Clt + Comma) and insert theLink in additional bords URL. It takes some time to download for us 5 minutes. And you are free to continue.

Testing your ESP32-Cam
Before programming the ESP 32 cam for our purposes can be started, the connection and the camera module can be tested. This step takes only a few minutes and it can save a lot of unnecessary troubleshooting along the way. The source code from esp32 library was utilized for this step.
Location of the example code on Esp32 library (source- Wai Lin)
We chose AI thinker camera configuration. If everything goes well you will receive the IP address of your image stream on serial monitor on baud rate 115200.

After inserting the camera Ip address. you will get assess to your camera stream

Camera module and PIR sensor connection
Schematics of the ESP32 cam and pir sensor connection with fdti (source-Wai Lin)
When motion is detected by the PIR sensor, GPIO13 goes high, the board will then wake up, and take a photo through the onboard camera. In addition, the system will take a photo every 5 minutes in order to make sure that the set up is still functioning well. The photos taken will be published to Flespi via MQTT as a message and also saved on the SD card.

Sleeping Modes of ESP3
The ESP32 microcontroller utilized in the project supports a range of general sleeping modes depending on which can reduce the RAM and CPU power and the working Wifi capabilities. Table 1.2 depicts a range of sleeping modes and their features with power supply.

Table 1.2: The sleeping modes supported by ESP32 (source:espressif)

Table 1 source

Wake Up sources : The RTC controller is a built-in timer which can be used to wake up the microcontroller after a predefined amount of time. For our part we are using the external sensor(PIR) to wake up the camera. The Gpio 13 which supports the power on

and off with external signal; the initial level 1 means when the signal is high from the sensor, the waking up of the system takes place.
esp_sleep_enable_ext0_wakeup(gpio_13, int level=1)


MQTT Protocol
MQTT (message queuing telemetry transport) is a messaging protocol that was designed to create a reliable standard for machine-to-machine (m2m) communication. It is a publish-and- subscribe protocol, meaning that instead of communicating with a server, client devices and applications publish and subscribe to topics handled by a broker. It typically uses TCP/IP (Transmission Control Protocol/Internet Protocol) as its transport but can also use other bi- directional transports. It has become the de facto standard for loT communication because of its efficiency and flexibility. u-blox uses this to overlay various radio networks (2G-4G cellular and LoRa) and protocols (USSD, UDP) providing developers with a familiar and simple experience. It allows devices and systems (clients) to communicate by sending messages. Messages are not sent directly from client to client but are published by a client to a topic* stored in an MQTT broker. Clients receive messages by subscribing to one or more topics and will receive messages from that point forwards. Topics are like street addresses - they form a tree which becomes more specific the further down you travel.

3.Result
3.1 Code

Final code.ino

Please refer to the end of the page

3.1.2 Connect to WiFi

Insert your network credentials

<const char* ssid = "iotlab"; 
const char* password = "****"; >
 
**3.1.3 Use Flespi MQTT broker to receive pictures**  
 <file c++ setting up mqtt broker client> 
const char* mqtt_server = "mqtt.flespi.io"; 
const int mqtt_port = 1883; 
const char*  mqtt_user  = 
"Ydl9IsQMA5NqDfGWgQt98ebuHLbm1gsPkDNQTtqho2xP51CbHTaGX9YuIZEz3Xdd"; 
const char* mqtt_password = ""; 
const char* mqtt_TopicName = "esp32/birds"; 
 

3.1.4 Use PIR sensor to trigger wake up the board

Use PIR sensor to trigger wake up the board

   if (SLEEP_DELAY == 0) {
    esp_sleep_enable_ext0_wakeup(GPIO_NUM_13, 1);
    delay(1000);
    esp_deep_sleep_start();
  }
 
  if (SLEEP_DELAY > 0) {
    delay(SLEEP_DELAY);
    delay(30000);
 

3.1.5 Save photos to SD card

saving photos to sd card

EEPROM.begin(EEPROM_SIZE); 
  pictureNumber = EEPROM.read(0) + 1; 
  // Path where new picture will be saved in SD Card 
  String path = "/picture" + String(pictureNumber) +".jpg"; 
  fs::FS &fs = SD_MMC; 
  Serial.printf("Picture file name: %s\n", path.c_str()); 
   File file = fs.open(path.c_str(), FILE_WRITE); 
  if(!file){ 
    Serial.println("Failed to open file in writing mode"); 
  } 
  else { 
    file.write(fb->buf, fb->len); // payload (image), payload length 
    Serial.printf("Saved file to path: %s\n", path.c_str()); 
    EEPROM.write(0, pictureNumber); 
    EEPROM.commit(); 
  } 
  file.close(); 
 

3.1.6 compression of JPEG and convert the picture to UTF-8 bits

Compressiong of jpeg files ino

if (fb->format != PIXFORMAT_JPEG) { 
      Serial.println("Compressing"); 
      bool jpeg_converted = frame2jpg(fb, 80, &_jpg_buf, &frame_size); 
 

3.2 Results on Flespi
We use the platform named as flespi, which allows pictures to be published via MQTT. Once messages flow into flespi, the platform transforms them into the unified JSON format and saves them in the database. Flespi token is a 64-byte randomly generated key-string used to access the data on the flespi platform via API, In MQTT API a token is used as an MQTT connection Username.Token configuration for your project is set up as shown in figure. In order to receive pictures taken in JSON messages, an MQTT channel is needed to receive messages from the broker, then a Device is needed to save the pictures.
(source- Wang)
On flespi, all photos taken can be shown under Messages, together with the time of shooting. This page also includes other useful information such as number of messages received per minute, connection status, and possible error message of MQTT broker.

3.3 Troubleshooting Logic Analyser
In order to understand how the effective communication between our modules and sensors is taking place, a logic analyser used to assess the logic curves of the system. And to test the reliability of our PIR sensor was tested with different condition of motions and calibrated to the desired sensitivity. Fig 7 describes the Universal unsynchronized communication between the esp32 camera and on Channel 2 the PIR sensor state shows high when a motion is detected.
Fig 7: The capture of logic analyser (source -Wai Lin )

3.4 Power Profile
In order to test the prower profile of the system we utilized the Power Profile Kit 2 to test the efficiency of the code and whether the battery can support the system only being charged by the PV module which comprises four cells. The power output on average is measured on average 45mA in direct sunlight conditions and therefore the power consumption is less than that for 2 minutes of 4 image captures and uploading to local SD card and MQTT server. Typically the system would go to sleep for 5 minutes after taking one cycle but here in order to test the reproducibility of the power consumption we run 4 cycles in sequences and they are observed to be similar power curves.

Fig 8: The power curve of 4 photos tanken successively (source - Wang)

3.5 Home assistant integration
Home assistant is an open source home automation that can control a whole host of devices with custom dashboards. It can host a range of add ons such as Node red, Mqtt dockers, microcontrollers and sensors. The Home assistant server can be run on a Raspberry Pi or a local server. In this project home assistant server was hosted on a docker environment namely Oracle.This intergration allows this project to branch out into numerous possibility OpenCV can be hosted on home assistant to object detect the images taken by our camera modules. A whole range of controls can also be made in one go in home assistant environment.
Fig9: ESP Home configuration assistant (source:Wai Lin)
Via ESP Home add-on, our camera module can be integrated into Home assistant environment. The camera can also be controlled by Mqtt protocol(mosquitto) which can be hosted on HA and by publishing on specific topics the commands can be given.

4. Discussion
4.1 Limitations
The flashlight on ESP 32 cam is an indicator for successful photo taking, however, it is too bright for a birdhouse. Animals will be scared away by the flash light. Since birds’ vision can see blue, green, red, and UV light. Replacing the flashlight on board by Infrared LED can solve the problem. With IR LED, the light will not be visible by both birds, and thus taking photos of birds without disturbing them.

Moreover, the PIR sensor is based on sense heat signatures, their sensitivity will decrease under warm environments (especially above 35°C). Therefore, IR sensor is not suitable for hot summers and microwave sensor are recommended for such conditions. Further one problem the esp32 cam has with FDTI adaptors is when flashing the device, it goes into time out mode when not reset in timely manner. In order to avoid this, ESP32-Cam- MB module can be utilized, it allows the board to directly via Micro-USB without a programming device.

5. Conclusion
Schematics of the final setup(source-Wai Lin)
In the end, the surveillance system achieves the set criteria of inexpensive, low power, reliable design which can be set up and operated without much knowledge prior. The system can be utilized for the purposes of monitoring birds and since the images uploaded are time lapsed, a whole host local bird species can be archived in spacial and time. This project has so much potential since esp32 cam can be utilized for many more purposes such as image recognition and tracking systems and can be taken as far as one wishes to go.

Final code.ino

#include "WiFi.h"
#include "esp_camera.h"
#include "esp_timer.h"
#include "img_converters.h"
#include "Arduino.h"
#include "soc/soc.h"           // Disable brownour problems
#include "soc/rtc_cntl_reg.h"  // Disable brownour problems
#include "driver/rtc_io.h"
#include "FS.h"                // SD Card ESP32
#include "SD_MMC.h"            // SD Card ESP32
#include <EEPROM.h>            // read and write from flash memory
#include <PubSubClient.h>
#include <base64.h>
#include <libb64/cencode.h>
 
// define the number of bytes you want to access
#define EEPROM_SIZE 1
 
constexpr int kCameraPin_PWDN   =  32;
constexpr int kCameraPin_RESET  =  -1;  // NC
constexpr int kCameraPin_XCLK   =   0;
constexpr int kCameraPin_SIOD   =  26;
constexpr int kCameraPin_SIOC   =  27;
constexpr int kCameraPin_Y9     =  35;
constexpr int kCameraPin_Y8     =  34;
constexpr int kCameraPin_Y7     =  39;
constexpr int kCameraPin_Y6     =  36;
constexpr int kCameraPin_Y5     =  21;
constexpr int kCameraPin_Y4     =  19;
constexpr int kCameraPin_Y3     =  18;
constexpr int kCameraPin_Y2     =   5;
constexpr int kCameraPin_VSYNC  =  25;
constexpr int kCameraPin_HREF   =  23;
constexpr int kCameraPin_PCLK   =  22;
 
int pictureNumber = 0;
 
 
const char* ssid = "FRITZ!Box 7530 MT";
const char* password = "14649735070293466500";
 
 
const char* mqtt_server = "mqtt.flespi.io";
const int mqtt_port = 1883;
const char* mqtt_user = "Ydl9IsQMA5NqDfGWgQt98ebuHLbm1gsPkDNQTtqho2xP51CbHTaGX9YuIZEz3Xdd";
const char* mqtt_password = "";
 
 
const char* mqtt_TopicName = "esp32/birds";
 
 
framesize_t resolution_ = FRAMESIZE_QVGA;
 
 
#define SLEEP_DELAY 0
#define FILE_PHOTO "/photo.jpg"
 
// OV2640 相机模组的针脚定义
#define PWDN_GPIO_NUM     32
#define RESET_GPIO_NUM    -1
#define XCLK_GPIO_NUM      0
#define SIOD_GPIO_NUM     26
#define SIOC_GPIO_NUM     27
#define Y9_GPIO_NUM       35
#define Y8_GPIO_NUM       34
#define Y7_GPIO_NUM       39
#define Y6_GPIO_NUM       36
#define Y5_GPIO_NUM       21
#define Y4_GPIO_NUM       19
#define Y3_GPIO_NUM       18
#define Y2_GPIO_NUM        5
#define VSYNC_GPIO_NUM    25
#define HREF_GPIO_NUM     23
#define PCLK_GPIO_NUM     22
 
 
#define CAMERA_MODEL_AI_THINKER
 
 
WiFiClient mqttClient;
PubSubClient client(mqttClient);
 
const int LED_BUILTIN = 4;
 
 
void setup_camera() {
 
     // OV2640 camera module
     camera_config_t config;
    config.pin_pwdn     = kCameraPin_PWDN;
    config.pin_reset    = kCameraPin_RESET;
    config.pin_xclk     = kCameraPin_XCLK;
    config.pin_sscb_sda = kCameraPin_SIOD;
    config.pin_sscb_scl = kCameraPin_SIOC;
    config.pin_d7       = kCameraPin_Y9;
    config.pin_d6       = kCameraPin_Y8;
    config.pin_d5       = kCameraPin_Y7;
    config.pin_d4       = kCameraPin_Y6;
    config.pin_d3       = kCameraPin_Y5;
    config.pin_d2       = kCameraPin_Y4;
    config.pin_d1       = kCameraPin_Y3;
    config.pin_d0       = kCameraPin_Y2;
    config.pin_vsync    = kCameraPin_VSYNC;
    config.pin_href     = kCameraPin_HREF;
    config.pin_pclk     = kCameraPin_PCLK;
    config.xclk_freq_hz = 20000000;
    config.ledc_timer   = LEDC_TIMER_0;
    config.ledc_channel = LEDC_CHANNEL_0;
    config.pixel_format = PIXFORMAT_JPEG;
    config.frame_size   = FRAMESIZE_SVGA;
    config.jpeg_quality = 10;
    config.fb_count     = 1;
 
  esp_err_t err = esp_camera_init(&config);
  Serial.printf("esp_camera_init: 0x%x\n", err);
 
  // sensor_t *s = esp_camera_sensor_get();
  // s->set_framesize(s, FRAMESIZE_QVGA);   
  }
 
String msg;
int timeCount = 0;
void getimg(){//拍照分段发送到mqtt
 
    camera_fb_t *fb = esp_camera_fb_get();
    if (fb){
        Serial.printf("width: %d, height: %d, buf: 0x%x, len: %d\n", fb->width, fb->height, fb->buf, fb->len);
        char data[4104];
        //client.publish(mqtt_TopicName, "0");
        for (int i = 0; i < fb->len; i++){
 
            sprintf(data, "%02X", *((fb->buf + i)));
            msg += data;
           // if (msg.length() == 4096){
           //     timeCount += 1;
           //     client.beginPublish(mqtt_TopicName, msg.length(), 0);
           //     client.print(msg);
           //     client.endPublish();
           //     msg = "";
           // }
        }
        if (msg.length() > 0){
            client.beginPublish(mqtt_TopicName, msg.length(), 0);
            client.print(msg);
            client.endPublish();
            msg = "";
        }
        //client.publish(mqtt_TopicName, "1");
        timeCount = 0;
        esp_camera_fb_return(fb);
    }
}
 
void setup_wifi() {
  delay(10);
  Serial.println();
  Serial.print("Connecting to ");
  Serial.println(ssid);
 
  WiFi.begin(ssid, password);
 
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
 
  Serial.println("");
  Serial.println("WiFi connected");
  Serial.print("IP address : ");
  Serial.println(WiFi.localIP());
}
 
void reconnect() {
  while (!client.connected()) {
    Serial.print("Attempting MQTT connection...");
    if (client.connect("ESP32Client", mqtt_user, mqtt_password)) {
      Serial.println("connected");
    } else {
      Serial.print("failed, rc=");
      Serial.print(client.state());
      Serial.println(" try again in 5 seconds");
      delay(5000);
    }
  }
}
 
void setup() {
  Serial.begin(115200);
  Serial.setDebugOutput(true);
  Serial.println();
  setup_camera(); 
  setup_wifi();    
  client.setServer(mqtt_server, mqtt_port); 
  if (client.connect("ESP32Client", mqtt_user, mqtt_password)) {
      Serial.println("mqtt connected");
    } 
}
 
void publishTelemetry(String data) {
  while (WiFi.status() != WL_CONNECTED) {
    delay(500);
    Serial.print(".");
  }
  boolean Status = client.publish_P( mqtt_TopicName, (const uint8_t*) data.c_str(), data.length(), true);
  Serial.println(String(Status ? "Successful" : "Error") );
}
 
void capturePhoto( void ) {
  // Retrieve camera framebuffer
  camera_fb_t * fb = NULL;
  uint8_t* _jpg_buf = NULL;
  esp_err_t res = ESP_OK;
  size_t frame_size = 0;
  Serial.print("Capturing Image...");
 
  digitalWrite(LED_BUILTIN, HIGH);   // turn the LED on
  delay(1000);                       // wait for a second
  fb = esp_camera_fb_get();
  digitalWrite(LED_BUILTIN, LOW);    // turn the LED off
  delay(1000);                       // wait for a second
  if (!fb) {
    Serial.println("Camera capture failed");
    res = ESP_FAIL;
  } else {
    Serial.println("Done!");
    Serial.println(String("Size of the image...") + String(fb->len));
    if (fb->format != PIXFORMAT_JPEG) {
      Serial.println("Compressing");
      bool jpeg_converted = frame2jpg(fb, 80, &_jpg_buf, &frame_size);
      esp_camera_fb_return(fb);
      fb = NULL;
      if (!jpeg_converted) {
        Serial.println("JPEG compression failed");
        res = ESP_FAIL;
      }
    } else {
      frame_size = fb->len;
      _jpg_buf = fb->buf;
      Serial.print("Publish photo...");
 
      publishTelemetry(base64::encode(_jpg_buf, fb->len));
      Serial.println("Done!");
      esp_camera_fb_return(fb);
    }
  }
  if (res != ESP_OK) {
    return;
  }
//Serial.println("Starting SD Card");
  if(!SD_MMC.begin()){
    Serial.println("SD Card Mount Failed");
    return;
  }
 
  uint8_t cardType = SD_MMC.cardType();
  if(cardType == CARD_NONE){
    Serial.println("No SD Card attached");
    return;
  }
 
    // initialize EEPROM with predefined size
  EEPROM.begin(EEPROM_SIZE);
  pictureNumber = EEPROM.read(0) + 1;
  // Path where new picture will be saved in SD Card
  String path = "/picture" + String(pictureNumber) +".jpg";
 
  fs::FS &fs = SD_MMC; 
  Serial.printf("Picture file name: %s\n", path.c_str());
 
  File file = fs.open(path.c_str(), FILE_WRITE);
  if(!file){
    Serial.println("Failed to open file in writing mode");
  } 
  else {
    file.write(fb->buf, fb->len); // payload (image), payload length
    Serial.printf("Saved file to path: %s\n", path.c_str());
    EEPROM.write(0, pictureNumber);
    EEPROM.commit();
  }
  file.close();
  esp_camera_fb_return(fb); 
 
}
 
 
void loop() {
  Serial.println("PSRAM found: " + String(psramFound()));
 
  if (!client.connected()) {
    reconnect();
  }
  if (client.connected()) {
    capturePhoto();
  }
  client.loop();
 
  Serial.println("Going to sleep now");
  if (SLEEP_DELAY == 0) {
    esp_sleep_enable_ext0_wakeup(GPIO_NUM_13, 1);
    delay(1000);
    esp_deep_sleep_start();
  }
 
  if (SLEEP_DELAY > 0) {
    delay(SLEEP_DELAY);
    delay(30000);
  }
}