If you've been following the series of Raspberry Pi articles you'll see we've covered the following topics:
- Installation of ConsoleWD - Weather Display for Raspberry Pi
- Streaming a live video feed from your weather webcam
- Automating the retreival of files from a source such as a webcam
We now want to get those files off to our website, which this article covers. At this point though I'm going to start assuming you know a bit about cron jobs, bash scripts and the like, which we covered in the earlier articles. If you feel like a refresher, then feel free to go back through one or more of the above articles.
Looking at options
There are a number of common ways we could use to send files from our Raspberry Pi, to another computer such as a website. There's
- FTP - a less secure (passwords are sent as readable text) method and many places mentioned it was tricky to set up
- SFTP - a secure method, but again I was warned it was tricky to set up
- SCP - based on SSH and is a secure method - can be setup so that a password is not required to be sent at all
I went wth SCP (Secure Copy). SCP is a means of securley transferring files between two machines on a network. SCP uses SSH for improved security and will prompt you if it needs a password or passphrase for authentication. You can find out more about SCP and some of the options we will be using here.
SSH or Secure Shell is a network communication protocol that enables two computers to communicate (c.f http or hypertext transfer protocol, which is the protocol used to transfer hypertext such as web pages) and share data. An inherent feature of ssh is that the communication between the two computers is encrypted meaning that it is suitable for use on insecure networks. You can find out more about SSH here
Sound complicated? Just think of them as two parts to the puzzle we need to make the connection work.
Getting started
You need to have at least the following setup / in place:
- SSH needs to be enabled on your Raspberry Pi. To do that, from the desktop under the main menu find Preferences - Raspberry Pi Configuration. Open that up and go to the Interfaces Tab and enable SSH. Click on OK
- SSH needs to be enabled on your webserver. This is normally done in the user.conf file for your account. Each hosting setup is different so I'd recommend asking your web host to help you with this if you don't know how. Of course for those hosting their weather sites with us, we can do that for you.
- SCP should also be installed on your webserver. It normally is by default, but if it isn't, check with your webhost.
- Your website will need to have a fixed IP - most have, but you should know what that IP address is
- You should know what your connection port is on your webserver (Computer 2). If it's port 22 then SSH will connect by default on that. If it's not then it needs to be specified.
Passwordless authentication using SSH relies on two keys. Both are generated on the source computer (Computer 1 or your Raspberry Pi) and then one is moved to the destination computer (Computer 2 or your webserver). When a connection is made SSH checks for a match between the two keys, and if there is a match, opens the connection. If not the connection is rejected. So we first need to set up the keys (if they don't exist already).
Generating and copying the SSH keys
There's a good description of the process, that I've based the following on located here if you need additional help along the way. This page also provides lots of details around some of the options you can use when generating keys
On your Raspberry Pi, jump into a Terminal Window (Accessories - Terminal), and type the following:
cd
mkdir -p .ssh
cd .ssh
SSH looks for keys in certian directories so we need to create a .ssh directory and then be in that directory when we generate the keys so they get saved there. Next type
ssh-keygen -t rsa
At the next three prompts just press enter unless you need to change the key name. The two keys will be generated and you will see something similar to this on your screen:
Now on your Raspberry Pi, (Computer 1) browse to the current users .ssh directory and you will find two files:
- id_rsa. This file needs to stay where it is. DO NOT share this file with anyone as if someone else has it they can access your Computer 2 or webserver
- id_rsa.pub - this is the public key file that needs to go on Computer 2 / your webserver
Using something like the Raspberry Pi text editor, or nano if you prefer the command line, to open the file and view the contents. You will see a string of letters and numbers, which is your key.
We now need to copy this onto your webserver, (computer 2) so it's there to perform the key match. If you have access to your sites control panel there will be a way to install SSH from there, and it's probably the easiest way. Copy the key out of the id_rsa.pub file and paste it into the section in your sites control panel. Click on ok and it's done.
Alternatively you can try and copy the file up to the server - I had no luck doing that using step 4 in the steps located here but it is possible just to create the file and paste the key into that.
If you want to copy the keys manually, on your web server, (computer 2) use the file browser to find the .ssh folder in the root of your account (ie not your home or public_html where your website sits, but the next level up. If it's not there you can create it. In that folder either:
- look for a file called authorized_keys. If it doesn't exist, create it and then paste the key from id_rsa.pub into it, and then save the file.
- If the authorized_keys file already exists, open that file in a text editor and paste the key from id_rsa.pub, on the last line of this file. One key must be on one line, and each key must be on seperate lines.
Finally set the permissions on the authorized_keys to 0600, and the .ssh directory to 0700. You can do this in the file manager available through your sites control panel or via using chmod in terminal
chmod 0600 authorized_keys
chmod 0700 /.ssh/There's more information here on file permissions and how to set them if you need it.
Testing the SSH key setup
You should be able to test that you can connect by using the following from the command line:
ssh <websiteusername>@<website IP address>
If that doesn't work you might need to specify the connection port. You can do that this way
ssh -p 23 This email address is being protected from spambots. You need JavaScript enabled to view it. where 23 is the port you want to connect on, huttwe is the user you want to connect with, and 12.34.56.78 is the ip address of your website (Computer 2). Instead of your IP address you can also use a FQDN eg. example.com
NOTE: you will need to try and connect once like this, as the first time you connect SSH will ask for a password to your account. Next time though SSH will store the info for the connection, and just connect without a password.
Setting up the SCP connection
If you have got this far then well done! You are mostly there. Keeping in mind we want to script all this at some point, we're looking for a line or two to get this working.
The scp command syntax take the following form:
scp [OPTION] [user@]SRC_HOST:]file1 [user@]DEST_HOST:]file2OPTION- scp options such as cipher, ssh configuration, ssh port, limit, recursive copy …etc.[user@]SRC_HOST:]file1- Source file.[user@]DEST_HOST:]file2- Destination file
Local files should be specified using an absolute or relative path, while remote file names should include a user and host specification. If you are looking for further options to use with SCP you can find them listed here.
For our purposes we are going to use a script line similar to the following. You will need to change the paths etc. to suit your needs
#Now Use SCP to send to website
# Options: C=compress, v=verbose logging,-P specifies the port if not port 22, -i specifies the location of the local key
# tidied script
scp -C -P23 -i /home/pi/.ssh/id_rsa $IMGFILE This email address is being protected from spambots. You need JavaScript enabled to view it. :/home/huttweat/public_html/data
Where:
- /home/huttweat/public_html/data is the destination file location. Not specifying the actual file name means scp will use the original filename
- $IMGFILE is the sourcefile, which is the variable used by our previous script to pull the file from a webcamera.
Putting everything together
So putting the "go get the webcam image script" and the "send the image to the website" scripts together we now get:
#!/bin/bash
# modifications by weather.net.nz with credit to alvinalexander.com
# a shell script used to download an image from an IP Cam URL and scp to a website.
# this is executed from a crontab entry.
#Next line sets logging detail
#set -x
#Set Download variables
# Put images in this directory (must exist)
IMGDIR=/home/pi/myssd/WeatherD/webcam_images
# wget output animated file
ANIFILE=webcam.`date +"%Y%m%d%H%M"`.jpg
# wget output webcam snapshot
IMGFILE=jpgwebcam.jpg
# Specify wget log files. If not specified these end up in the script output
LOGFILEANI=wgetani.log
LOGFILE=wget.log
# Specify wget download url. NOTE: &characters are escaped
URL=https://192.168.2.119/cgi-bin/api.cgi?cmd=Snap\&channel=0\&rs=wuuPhkmUCeI9WG7C\&user=admin\&password=123456
#Now go and get the file(s) from the webcam
#Options -O is the output file -o is the log output file
cd $IMGDIR
wget $URL -O $ANIFILE -o $LOGFILEANI
wget $URL -O $IMGFILE -o $LOGFILE
#Now Use SCP to send to website
# Options: C=compress, v=verbose logging,-P specifies the port if not port 22, -i specifies the location of the local key
# tidied script
scp -C -P23 -i /home/pi/.ssh/id_rsa $IMGFILE This email address is being protected from spambots. You need JavaScript enabled to view it. :/home/huttweat/public_html/data
Note that not all options are required, however these were the ones I needed to specify to get it to work for me.
One of the really nice things about this script is that it doesn't just work for webcam images. It will pull any file off your local machine / internet and publish it to your website/another machine, just by changing a few of the variables in the script around!
Finally a quick word about the cron job that calls this script every 15 mins that we set up in the previous article. Does it need changing? Well no not really. It's still going to call this script with the scp command in it and send an updated image to the website every 15 mins.
Supporting us and feedback
If you think this article was helpful, there are a number of ways you can thank us. Feel free to add a comment and say thanks in the comments section below. Also you are welcome to register on the site. The more members we have the more it encourages us to write these types of articles. Alternatively if you are into all that social media stuff, feel free to share the article using the buttons below, and tell others about the site. Thanks!!