Raspberry Pi Camera

"Smart" Surveillance Camera

28.04.2019 - 19:30
A while ago I bought a camera module for my Raspberry Pi and I wanted to experiment with it a bit more. Unfortunately there is no official C++ api for the camera, only for python and I like random whitespaces. Originally I tried to write a wrapper for the mmal library that the Pi uses, but it never really worked well, because there is almost no documentation for it. So now I did some reading I stumbled accross mjpg-streamer (link) which has a Raspberry Pi camera plugin. It's very easy to install and use and after a few minutes I was able to watch a livestream of the camera in my browser. But I wanted to process the images so I needed a way to recieve the livestream in C++. Because I thought recieving a livestream would be very complicated I chose OpenCV and with it I was able to read the stream. However, when I tried to use a lower framerate to get higher quality images, OpenCV couldn't read the stream anymore. So I had a look into what it would take to write my own mjpg stream client and it isn't actually that complicated. A mjpg stream is basically a continuous multi part http response which sends jpgs. I adapted my TCP class to send a http request to my Pi and wrote a parser for the response. I got stuck for a while and couldn't figure out what was going wrong and when I did I was like what the f***. When converting a char array to a string on windows the line endings get converted to the platform specific line ending convention, which for window is \r\n. This turned the \r\n line endings of the http response into \r\r\n and under some conditions this would mess up my parsing. After I realized this and changed my parser to operate on the byte array directly it finally worked.

The two use cases I had in mind were a more convinient way to shoot long timelapses and a surveillance camera that automatically detects changes and only records when necessary. Shooting a long timelapse is very easy, just save the images and we are done. A "smart" surveillance camera requires a bit more work though. The naive approach is to compare the current frame to the previous frame and if it changes more than some threshold to account for the noise an event occurred. Doing this indoors works quite well, because the background never changes. However, outside there is wind, clouds, rain, sunshine, etc, so this doesn't work that great. According to my quick google research there isn't a best way to do this, it always depends on the environment. I tried a few different approaches, some better than others, but I am only going to write about what I settled on. First I convert every frame to grayscale, because one color channel is easier to deal with than three. My idea was to create a "pixel movement map", which describes how much a pixel changes/moves by the wind on average over a few frames. Then I use a binary threshold on it and all pixels which change a lot are ignored when comparing two normal frames. I also measure the average difference between two frames and adjust the threshold based on it. If it gets windier the average difference goes up and the threshold to decide whether something happens also gets adjusted up to reduce the number of only wind images, and if it calms down, the threshold gets lowered so smaller details can be caught.

Yesterday and today I tested it for a longer period of time and it works decently, but it still could be improved. I mostly chose the treshold and threshold range too conservatively and there are still too many wind images. And I have to think about some kind of lighting invariance. Today there were some smaller clouds that covered and uncovered the sun, which changed the lighting of the complete image quickly, which was also picked up by my system. I don't mind a few wind images. I rather click through a few of those than miss a small bird. Humans are no problem anyway, if I walk into the frame myself the difference goes up by a couple of orders of magnitudes, so if I just wanted to pick up humans or elefants I could set the threshold very high and have no wind images at all, but I like random animal pictures, so it has to be low ;) Here are some random animals it picked up (some colors may look weird, I have the no IR filtered version which produces slightly off colored images during the day, but it can pick up infra red light for nightvision, but that's a project for another day):

---