1.9 Portfolio Item

https://vimeo.com/254406228

Project Content: 

Introduction:

In its finished form Project Stalker could be an interactive installation, a mixture between animatronics and computer vision. It will take the form of an security camera, which instead of a camera lens will have an eye-ball object. It will be rotatable and always following the faces it sees. That is the reason why the project is called Stalker.

Technology:

Version 2 Raspberry Pi

Software:

  • Raspbian Strech 4.19
  • OpenCV 3.2.0

Hardware:

The development of that version has reached the point of Object Tracking, but it is not fully functioning.

Version 1 Arduino

// Current stage of the code

There is a camera inside the eye object, which sends information to Processing,
The image after that is being processed with OpenCV and the face of the person is being tracked. The eye’s rotation on the horizontal and vertical axis is being controlled by 2 servo motors, which are attached to an Arduino board or a Raspberry Pi.

Tech_Chart

Software:

  • Arduino version used: 1.8.8
  • Processing version used: 3.5.2
  • OpenCV for Processing (Greg Borenstein) version used: 0.5.4

Hardware:

  • Arduino Uno R3
  • Webcam: Logitech C310
  • Servos: 2x Hitec HS-311 Standard

Table of Contents

1.1 Technology 1 – OpenCV
1.1 Technology 2 – Raspberry Pi
1.1 Technology 3
1.1 (optional) Technology 4 – Cloud Vision
1.2 Justification Technology Choice
1.2 Retrospective(week 1.1) & ToDo (week 1.2)
1.2 Development Steps this week #1
1.3 Retrospective(week 1.2) & ToDo(week 1.3)
1.3 Development steps this week. #2
1.4 Retrospective(week 1.3) & ToDo(week 1.4)
1.4 Development Steps this week. #3 #4
1.5 Retrospective(week 1.4) & ToDo(week 1.5)
1.5 Development Steps this week. #5 #6
1.6 Retrospective(week 1.5) & ToDo(week 1.6)
1.6 Development Steps this week. #7 #8
1.7 Retrospective(week 1.6) & ToDo(week 1.7)
1.7 Development Steps this week. #9 #10
1.8 Retrospective(week 1.7) & ToDo(week 1.8)
1.8 Development Steps this week. #11 #12
1.9 Retrospective (whole project)

1.9 Reflection (whole project)

What I have reached/achieved:

I managed to get knowledge and experience in the following directions:

  • Arduino + Processing
  • OpenCV
  • Product design and prototyping
  • Camera streaming and image processing
  • Raspberry Pi
  • Python
  • Deeper understanding in servo motors

What I have learned:

I feel that my biggest mistake is that I had a clear goal in mind – making an animatronic eye, which can track faces. In this way I didn’t give myself enough freedom in experimenting and discovering the broader possibilities brought by the technologies I have chosen. I straight away developed the final prototype, without spending enough time to document my mistakes and struggles, which forced me to have a bit of a reverse development process – starting with the final product and figuring out ways to justify it. Despite that I still have managed to experiment and learn from those experiments.

The possibilities offered by Computer Vision are almost limitless.

Even though, this project is not related to my career path, I do believe that it thought me an important lesson about project management and work process.

1.8 Development Steps this week. #11 #12

Development Step #11 – Controlling Pan/Tilt servos with Raspberry Pi

 

o Approach.

I will try to calculate the Duty Cycle of the servos since they might be different from what the manufacture has claimed. I found a nice instructables tutorial to take me through the process.

o Learning Experience.

I found the manufacturer’s data sheet for the servos:
Annotation 2019-08-25 203537.jpg

The Raspberry Pi has no analog output, but we can simulate this, using a PWM (Pulse Width Modulation) approach. What we will do is to generate a digital signal with a fixed frequency, where we will change the pulse train width, what will be “translated” as an “average” output voltage level.

Note that what matters here is not the frequency itself, but the “Duty Cycle”, that is the relation between the time that the puls is “high” divided by the wave period. For example, suppose that we will generating a 50Hz pulse frequency on one of our Raspberry Pi GPIO. The period (p) will the inverse of frequency or 20ms (1/f). If we want that our LED with a “half” bright, we must have a Duty Cycle of 50%, that means a “pulse” that will be “High” for 10ms.

This principle will be very important for us, to control our servo position, once the “Duty Cycle” will define the servo position.

I am connecting the pan/tilt servos to the following GPIO pins:

  • Tilt Servo – GPIO 17
  • Pan Servo – GPIO 27

Both of them are taking 5V power from Pin#2 and Pin#4, and they are also grounded to the Raspberry Pi.

In theory, the servo will be on its

  • Initial Position (0 degrees) when a pulse of 1ms is applied to its data terminal
  • Neutral Position (90 degrees) when a pulse of 1.5 ms is applied to its data terminal
  • Final Position (180 degrees) when a pulse of 2 ms is applied to its data terminal

To program a servo position using Python will be very important to know the correspondent “Duty Cycle” for the above positions, let’s do some calculation:

  • Initial Position ==> (0 degrees) Pulse width ==> 1ms ==> Duty Cycle = 1ms/20ms ==> 2.0%
  • Neutral Position (90 degrees) Pulse width of 1.5 ms ==> Duty Cycle = 1.5ms/20ms ==> 7.5%
  • Final Position (180 degrees) Pulse width of 2 ms ==> Duty Cycle = 2ms/20ms ==> 10%

So the Duty Cycle should vary on a range of 2 to 10 %.

Now lets find out that for each servo individually.

by typing the following commands in the Terminal

import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
tiltPin = 17

Now, we must specify that this pin will be an “output”

GPIO.setup(tiltPin, GPIO.OUT)

And, what will be the frequency generated on this pin, that for our servo will be 50Hz:

tilt = GPIO.PWM(tiltPin, 50)

Now, let’s start generating a PWM signal on the pin with an initial duty cycle (we will keep it “0”):

tilt.start(0)

Now, you can enter different duty cycle values, observing the movement of your servo. Let’s start with 2% and see what happens (we expect that the servo goes to “zero position”):

tilt.ChangeDutyCycle(2)

…..

  • Result:

Those are the results for both the Pan and the Tilt servos:

GGG02231.JPG

Annotation 2019-08-25 230225.jpg

Annotation 2019-08-25 231646.jpg

GGG02232.JPG

 

The PWM commands to be sent to our servo are in “duty cycles” as we saw on the last step. But usually, we must use “angle” in degrees as a parameter to control a servo. So, we must convert “angle” that is a more natural measurement to us in duty cycle as understandable by our Pi.

I know that duty cycle range goes from 2% to 12% and that this is equivalent to angles that will range from 0 to 180 degrees. Also, we know that those variations are linear.
GGG02233.JPG
So, given an angle, we can have a correspondent duty cycle:

dutycycle = angle/18 + 2

Now we can make a Python script using the formula

Servo5.py

from time import sleep
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

def setServoAngle(servo, angle):
	pwm = GPIO.PWM(servo, 50)
	pwm.start(8)
	dutyCycle = angle / 18. + 3.
	pwm.ChangeDutyCycle(dutyCycle)
	sleep(0.3)
	pwm.stop()

if __name__ == '__main__':
	import sys
	servo = int(sys.argv[1])
	GPIO.setup(servo, GPIO.OUT)
	setServoAngle(servo, int(sys.argv[2]))
	GPIO.cleanup()
sudo python3 angleServoCtrl.py 17 45

The above command will position the servo connected on GPIO 17 with 45 degrees in “elevation”. A similar command could be used for Pan Servo control (position to 45 degrees in “azimuth”):

sudo python servo5.py 27 45

Now lets use the script for controlling both of the servos at the same time:

Servo6.py

 

from time import sleep
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

pan = 27
tilt = 17

GPIO.setup(tilt, GPIO.OUT) # white => TILT
GPIO.setup(pan, GPIO.OUT) # gray ==> PAN

def setServoAngle(servo, angle):
	assert angle >=50 and angle <= 130
	pwm = GPIO.PWM(servo, 50)
	pwm.start(8)
	dutyCycle = angle / 18. + 2.
	pwm.ChangeDutyCycle(dutyCycle)
	sleep(0.3)
	pwm.stop()

if __name__ == '__main__':
	import sys
	if len(sys.argv) == 1:
		setServoAngle(pan, 110)
		setServoAngle(tilt, 50)
	else:
		setServoAngle(pan, int(sys.argv[1])) # 30 ==> 90 (middle point) ==> 150
		setServoAngle(tilt, int(sys.argv[2])) # 30 ==> 90 (middle point) ==> 150
	GPIO.cleanup()

When the script is executed, you must enter as parameters, Pan angle and Tilt angle. For example:

sudo python3 servo6.py 120 50

I found the proper angles for my Servos

Pan Servo:

  • Min – 50
  • Mid – 110
  • Max – 150

Tilt Servo:

  • Min – 50
  • Mid – 60
  • Max – 90

Annotation 2019-08-26 000207.jpg

Now the final test is a loop cycle on the pan and tilt axis:

servo7.py

from time import sleep
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)
GPIO.setwarnings(False)

pan = 27
tilt = 17

GPIO.setup(tilt, GPIO.OUT) # white => TILT
GPIO.setup(pan, GPIO.OUT) # gray ==> PAN

def setServoAngle(servo, angle):
	assert angle >=50 and angle <= 130
	pwm = GPIO.PWM(servo, 50)
	pwm.start(8)
	dutyCycle = angle / 18. + 2.
	pwm.ChangeDutyCycle(dutyCycle)
	sleep(0.3)
	pwm.stop()

if __name__ == '__main__':  
    for i in range (50, 140, 15):
        setServoAngle(pan, i)
        setServoAngle(tilt, i)
    
    for i in range (130, 50, -15):
        setServoAngle(pan, i)
        setServoAngle(tilt, i)
        
    setServoAngle(pan, 110)
    setServoAngle(tilt, 50)    
    GPIO.cleanup()

o Result.

o Sources.

https://www.instructables.com/id/Pan-Tilt-Multi-Servo-Control/

Development Step #12 – Object Tracking with OpenCV and Raspberry Pi

o Approach.

First I will start with some examples using Object Tracking and OpenCV and after that I will proceed with live video and integrating the servo function.

o Learning Experience.

I already have installed the OpenCV library in a previous dev. step.

I will start by analyzing the code by Adrian posted on his page ( https://www.pyimagesearch.com/2015/09/14/ball-tracking-with-opencv/ )

I have to verify that the OpenCV library is installed correctly:
Annotation 2019-08-26 004329.jpg

I want to test if OpenCV is working as before, by running that test python script:

import numpy as np
import cv2
cap = cv2.VideoCapture(0)
while(True):
    ret, frame = cap.read()
    frame = cv2.flip(frame, -1) # Flip camera vertically
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    
    cv2.imshow('frame', frame)
    cv2.imshow('gray', gray)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cap.release()
cv2.destroyAllWindows()

To execute the OpenCV code I need to get into the OpenCV’s virtual environment from the terminal with the following commands:

source ~/.profile
workon cv
python simpleCamTest.py

Unfortunately I bumped into an error “workon cv is not valid”

I started backtracking the issue and reached the conclusion that there could be something wrong with the OpenCV version I am using 3.2.0 and maybe I have missed some packages during my installation, so I decided to look for them.
I headed to this installation guide : https://www.pyimagesearch.com/2017/09/04/raspbian-stretch-install-opencv-3-python-on-your-raspberry-pi/

I started installing all the mentioned libraries, but nothing fixed the issue. So after many tries I decided to completely reinstall the OpenCV library, which is a long process that I wanted to avoid.

Annotation 2019-08-26 033548.jpg

Everything was going smooth until I started getting multiple error messages and the process stopped.

Annotation 2019-08-26 033331.jpg

Annotation 2019-08-26 033400.jpg

Annotation 2019-08-26 033435.jpg

Annotation 2019-08-26 033526.jpg

I tried again, but no effect:

Annotation 2019-08-26 033246.jpg

This time I receive a “free space” error, even though that I use a 32GB sd card.

Going through the comments the only solution to the issue is to make a clean install of Raspbian and start installing OpenCV, VNC and all the other libraries that are needed.

Sadly because of the time scope of this project, I am unable to continue the development.

o Sources.

Ball Tracking with OpenCV

https://www.hackster.io/mjrobot/automatic-vision-object-tracking-5575c4#_=

Color Detection in Python with OpenCV

1.8 Retrospective(week 1.7) & ToDo(week 1.8)

Retrospective 1.7

  • What went well?

I gained some general idea about Python. And I am quite happy that I managed to understand how to identify the pins in the different modes of the GPIO and also managed to connect servos to the Raspberry!

  • What could be better?

I wish I could have experimented with a potmeter or an LED strip.

  • What should be better?

I am pretty satisfied with this week’s progress.

  • Time spent.

18 hours

To Do 1.8

  • Calibrate servo motors according to my prototype.
  • Use object tracking in combination with the servo motors.

1.7 Development Steps this week. #9 #10

Development Step #9 Python

 

o Approach.

For me personally a lynda course will help me out a lot when it comes to learning a completely new programming language:

I will follow this one:
https://www.lynda.com/Python-tutorials/Learning-Python/661773-2.html

As an editor I will be using Atom.
atom-4-logo-png-transparent.png

 

o Learning Experience.

  • Atom setup.

 

 

  • Lynda course.

Topics include:

  • Installing Python
  • Choosing an editor or IDE
  • Working with variables and expressions
  • Writing loops
  • Using the date, time, and datetime classes
  • Reading and writing files
  • Fetching internet data
  • Parsing and processing HTML

When I tried to run the “Helloworld!” example, the console displayed the following error message:

python : The term ‘python’ is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct

After googling it I realized that I need to add manually the python path to the Windows Environment Variables. I followed this tutorial:

 

 

 

o Result.

  • Hello world example.

1.JPG

 

  • Variables and expressions.

2.JPG

 

  • Functions

functions.JPGfunctions2.JPG

 

  • Loops

32.JPG31.JPG

 

  • Conditional structures.

conditions1.JPG

  • Classes

classes1.JPG

  • Date and time classes

dates1.JPG

dates2.JPG

 

  • Formatting time output.

formatting1.JPG

formatting2.JPG

  • Timedelta objects

timedeltas1.JPG

timedeltas2.JPG

  • Calendars

calendar1.JPG

calendar2.JPG

  • Reading and writing files.

files1.JPG

files2.JPG

  • OS path

ospath.JPG

ospath2.JPG

ospath3.JPG

  • File system shell.

shell.JPG

shell2.JPG

o Sources.

https://www.lynda.com/Python-tutorials/Learning-Python/661773-2.html

 

 

 

 

https://stackoverflow.com/questions/11813435/im-trying-to-use-python-in-powershell

https://docs.python.org/3/library/index.html

https://docs.python.org/3/reference/index.html

 

Development Step #10 GPIO & Servo Motors on Raspberry Pi

o Approach.

sdsds

o Learning Experience.

As a starting point I headed to the course called “Raspberry Pi: GPIO” on Lynda.com. The main topics I am interested in are:

  • Locating the GPIO
  • Programming with Python
  • Identifying GPIO pins
  • Using the WiringPi and RPi.GPIO libraries

 

Controlling an LED.

Annotation 2019-08-24 194708.jpg
We will enable the GPIO pin through the terminal. First we need to be super users in order to do that. In stead of typing “sudo” every single line, we can simply enter superuser mode through the following command:

sudo -i

Next step is to change the directory (“cd”) to the directory where the GPIO is located with the following command:

cd /sys/class/gpio

Now we need to assign pin 4.

echo "4" > export
This command takes the value “4” and writes it into a file called export. “echo” stands for print.

Now we need to assign pin 4 as an output pin.

echo "out" > gpio4/direction

To turn the LED on we need to write the following command:

echo "1" > gpio4/value

  • Result
    Annotation 2019-08-24 201028.jpg

    To turn off the LED we can use the reverse command:

echo "0" > gpio4/value

  • Result
    Annotation 2019-08-24 201055.jpg

To turn off any values for pin 4, we can write:

echo "4" > unexport

As a safety measure we can also exit the “sudo” mode by typing:

exit sudo

Annotation 2019-08-24 201541.jpg
Using a Switch / Button

Annotation 2019-08-24 202546.jpg

First I need to assign GPIO pin 4 as input:
sudo -i
cd /sys/class/gpio
echo "4" > export
echo "in" > gpio4/direction

I can check the value of gpio4 with the following command:

cat gpio4/value

At the moment I am not having the button pressed, which means that I should receive a value of “0”.

  • Result.
    Annotation 2019-08-24 211639.jpg

Annotation 2019-08-24 212722.jpg

If I have the button pressed the circuit should become closed and I should receive a value of 1.

  • Result
    Annotation 2019-08-24 211611.jpg
    Annotation 2019-08-24 212802.jpgAnnotation 2019-08-24 213202.jpg

Locating Pins

Annotation 2019-08-24 214727.jpg

There are several different numerations of the GPIO pins.

  • GPIO.BOARD – is the physical location
  • GPIO.BCM – is the numbering according to the default python library for Raspberry (RPi.GPIO)
  • WiringPi – is related to the 3rd party WiringPi python library

Now I will be using a python code to reproduce the previous exercise. First I will import the RPi.GPIO library:

import RPi.GPIO as GPIO

Now I will select the board numbering method. In this case GPIO.BOARD

GPIO.setmode(GPIO.BOARD)

Now I need to select the input pin. In the previous experiment it was pin 4, but according to the GPIO.BCM system, now I am using the GPIO.BOARD system an the corresponding number is 7.

inputpin = 7

The next line should assign the pin as an input pin and indicate that there is a resistor in use.

GPIO.setup(inputpin,GPIO.IN,pull_up_down=GPIO.PUD_UP)

We can now add a if/else statement reflecting the state of the switch/button, if it is closed/pressed or opened/released.

while True:
if GPIO.input(inputpin):
print ("Switch Closed")
else:
print ("Switch Open")inputpin = 7

The final python script should look like that:
Annotation 2019-08-24 220904.jpg

If we were using the GPIO.BCM numbering then it would be:
Annotation 2019-08-24 220920.jpg

We can run the code through the terminal, navigate to and run the script that we just created:

Annotation 2019-08-24 221424.jpg

  • Result.

At the moment the button is not pressed.

Annotation 2019-08-24 221548.jpg

As soon as I press it:
Annotation 2019-08-24 221629.jpg

 

WiringPi Library

To install the WiringPi Library the following commands should be executed one after another:

sudo apt-get install git-core
sudo apt-get update
sudo apt-get upgrade
git clone git://git.drogon.net/wiringPi
cd wiringPi
./build

To test if the installation went smoothly type:
gpio -v

Now to see all the GPIO numbers type in the terminal:
gpio readall

  • Result:

Annotation 2019-08-24 224834.jpg

 

Servo Motors

For the following experiments I will be using the default library called “GpioZero” and HS-311 servo.
Annotation 2019-08-25 161246.jpg
I am attaching the power cable to Pin #2, which is 5v. The ground cable to Pin#9 and the signal cable to Pin#7 (GPIO.4). And for the Raspberry I am using a 5V 4A power supply.

The first script I will use should make the servo move between its minimum, middle and maximum positions with a small delay in-between.

from gpiozero import Servo
from time import sleep

myGPIO=17

servo = Servo(myGPIO)

while True:
servo.mid()
print("mid")
sleep(0.5)
servo.min()
print("min")
sleep(1)
servo.mid()
print("mid")
sleep(0.5)
servo.max()
print("max")
sleep(1)

To execute the script I can do it from the terminal by typing:
cd experiments //change to the experiments folder I store my scripts now
python servo1.py

Or by using a Python editor called Thonny, which turned out to be more convenient.

  • Result
    Annotation 2019-08-25 185010.jpg

first script makes use of the Gpiozero defaults. It assumes the servo uses a signal frame width of 20ms. The pulse width for the minimum and maximum rotation is assumed to be 1ms and 2ms.

I found that with the default settings my servo moves in the diapason from 30 degrees to 150 degrees. With the following script I will try to expand that diapason.

from gpiozero import Servo
from time import sleep
myGPIO=4

myCorrection=0.45
maxPW=(2.0+myCorrection)/1000
minPW=(1.0-myCorrection)/1000

servo = Servo(myGPIO,min_pulse_width=minPW,max_pulse_width=maxPW)

while True:
servo.mid()
print("mid")
sleep(0.5)
servo.min()
print("min")
sleep(1)
servo.mid()
print("mid")
sleep(0.5)
servo.max()
print("max")
sleep(1)

  • Result

The last script I will experiment with generates a range of “value” numbers to sweep the servo between its maximum and minimum position :

from gpiozero import Servo
from time import sleep

myGPIO=4

myCorrection=0
maxPW=(2.0+myCorrection)/1000
minPW=(1.0-myCorrection)/1000

servo = Servo(myGPIO,min_pulse_width=minPW,max_pulse_width=maxPW)

while True:

print("Set value range -1.0 to +1.0")
for value in range(0,21):
value2=(float(value)-10)/10
servo.value=value2
print(value2)
sleep(0.5)

print("Set value range +1.0 to -1.0")
for value in range(20,-1,-1):
value2=(float(value)-10)/10
servo.value=value2
print(value2)
sleep(0.5)

The first “For” loop generates a set of integers between 0 and 20. The value has 10 subtracted from it to give a range -10 to 10. The value is then finally divided by 10 to give a range of -1 to +1. The original set of values contained 20 integers and we still have 20 steps in the sequence. To increase the number of steps to 40 you would replace 20 with 40 and subtract 20 rather than 10.

The second “For” loop does the same thing but generates a sequence from +1 back to -1.

  • Result

 

o Sources.

https://www.raspberrypi-spy.co.uk/2018/02/basic-servo-use-with-the-raspberry-pi/

https://gpiozero.readthedocs.io/en/stable/api_output.html#servo

 

1.7 Retrospective(week 1.6) & ToDo(week 1.7)

Retrospective 1.6

  • What went well?

I managed to take a photo, record a video and play live video from a USB Camera attached to Raspberry Pi. I experimented with the Raspberry Pi Camera module and I succeeded in capturing an image, making a simple script to automatically do it and capturing a video.

I managed to install the OpenCV library on a Raspberry Pi running Raspbian OS. I experimented with Processing for Raspberry Pi.

  • What could be better?

I had major issues with the camera and Processing. I thought it is because I had too much libraries installed, which collide with one another, but even when I invested the time for a clean install of Raspbian with pre-configured Processing, the issues where still there. I wish I could re-adjust my Arduino+Processing code for Raspberry, but unfortunately it didn’t go that way.

  • What should be better?

Instead of trying to figure out a solution to a problem by searching online or writing in forums (slow), I should personally contact local experts from the field, who can help me out.

  • Time spent

18 hours

To Do 1.7

  • Since OpenCV and Processing didn’t work out well, I am going to concentrate on OpenCV and Python. The first step is to get some understanding of the Python basics and syntax.
  • I need to be able to control servo motors via the GPIO interface of the Raspberry Pi.