Just a few days ago I realized that the Raspberry PI I use to control my irrigation system was dead. Could not get to the web interface, pings would time out, could not ssh into it.
The first thing I tried was a simple reboot. The raspberry is in a black box in my backyard, maybe the hot summer days were... too hot? I have a cron job that shuts it down if the temperature goes above 70 degrees. Or maybe the shady wireless card and its driver stopped working? I have another cron job to restart it, so this seems less likely.
So.. I reboot it by phyiscally unplugging it, but still nothing happens. The red led on the board, next to the ethernet plug is on, which means it is getting power. The green led next to it flashes only once. By reading online, this led can flash to report an error, or to indicate that the memory card is being read.
There is no error corresponding to one, single, flash, so I assume it means that it tried to read the flash, and somehow failed. It is supposed to be booting now, so I would expect much more activity from the memory card.
Maybe the card is corrupted, or something bad happened to the file system.
Removed the memory card from the raspberry, inserted it in my laptop. First thing I do is run fsck.
Note that /dev/sdb is the memory card inserted in my laptop! On your computer, it will likely have a different name. Make sure you don't damage one of your real partitions.
Anyway, the command I use is:
# fsck -f /dev/sdb1
First partition is good, let's look at the second one:
# fsck -f /dev/sdb2
TADA! Lot's of problems reported! This is annoying, the file system was corrupted.
Next step is to back it up, just in case. Although there's not much on it, and it took very little to get it running in the first place, a backup may come in handy.
To copy the memory card, all I had to do was:
# dd if=/dev/sdb of=/opt/backup/raspberry-20130730.img
and let it run until completion.
Next step is to fix the file system. Can I really do it? Let's try:
# fsck -f -p /dev/sdb2
Unfortunately, this fails with something like: "fsck failed, please repair manually". Not a good sign. So, let's try once again:
# fsck -y -f /dev/sdb2
This shows several screens of errors. Bad bad sign. Let's try to mount it:
# mount /dev/sdb2 /mnt/tmp
Seems to mount cleanly now. Let's try to put it back, and reboot the system one more time. Still no luck...
Unfortunately, I don't have a backup of the original working memory card. So let's start from scratch, like I did when I first got it.
Downloaded latest raspbian image from: http://downloads.raspberrypi.org/
Installed it, with:
# unzip 2013-07-26-wheezy-raspbian.zip # dd if=2013-07-26-wheezy-raspbian.img of=/dev/sdb
Next step is configure the network on the memory card, so I can put it back in the raspberry, and finish the setup via ssh. To do so, I need to mount the memory card, and modify a few config files:
I need to tell my linux kernel to re-read the partition table of sdb, so it picks up the position of the partitions I just copied into /dev/sdb, with:
# sfdisk -R /dev/sdb
Alternatively, I could just have removed the memory card and re-inserted it. But I'm lazy, and the command is more convenient.
Mount the partition somewhere:
# mkdir -p /mnt/raspberry # mount /dev/sdb2 /mnt/raspberry
Setup the wireless config. This means editing /etc/network/interfaces on
a Debian based system:
# vim /mnt/raspberry/etc/network/interfaces
and added wpa-ssid and wpa-psk, leading to the file looking like this:
auto lo iface lo inet loopback iface eth0 inet dhcp allow-hotplug wlan0 iface wlan0 inet dhcp wpa-ssid "SSID-of-your-wireless-network" wpa-psk "password!" iface default inet dhcp
Save and umount.
# sync # Just in case # umount /mnt/raspberry
Now it is time to try it. Let's remove the memory card from the laptop, and put it back on the raspberry. Reboot, the green leds are blinking happily.
In my home server, responsible for my network, I have dhcpd running configured
to assign the raspberry a static address. I do so with a block like:
subnet 10.1.40.0 netmask 255.255.255.0 { option domain-name-servers 10.1.40.254, 8.8.8.8, 8.8.4.4; option routers 10.1.40.254; range 10.1.40.20 10.1.40.200; group { use-host-decl-names on; host raspberry { fixed-address 10.1.40.9; hardware ethernet 80:1f:02:9a:9d:e6; } } }
In /etc/dhcp/dhcpd.conf. The mac address 80:1f:02:9a:9d:e6 is the one of my raspberry.
You can find it by running the command ifconfig or ip link show on the raspberry itself.
Thanks to that block, the raspberry gets assigned the address 10.1.40.9. If you don't have a similar configuration, or don't know the MAC address of your raspberry, don't despair! It is pretty easy to figure it out.
If you have a dhcp server in your network, you can just look at its logs. Around the time the raspberry is booted, you can probably see a line like:
... Jul 28 07:13:13 yourserver dhcpd: DHCPDISCOVER from 80:1f:02:9a:9d:e6 via eth1 ...
in /var/log/messages.
Alternatively, you can run tcpdump while the raspberry is rebooted, and most likely
see its mac address and assigned ip. To do so, you can use something like:
# tcpdump -v -nei wlan0 port 67 or port 68
In any case, the raspberry boots. My:
$ ping 10.1.40.9
eventually succeeds, and I can login with ssh using username pi, password raspberry, and sudo -s
to become root:
$ ssh pi@10.1.40.9 Password: raspberry $ sudo -s
I use the raspberry as a headless server to control my irrigation system. Unfortunately, raspbian seems to be geared more to desktop users.
Here's what I did to configure it:
Install my ssh keys both for root and pi. This is necessary only if you use ssh-agent and ssh keys.
$ ssh-copy-id pi@10.1.40.9 $ ssh pi@10.1.40.9 $ sudo -s # cp -a ~pi/.ssh ~root/.ssh # chown root -R ~root/.ssh
Disabled password based access. Again, do this only if you use ssh keys.
If you don't though, you should make sure to change the password of user
pi.
# passwd -l pi # passwd -l root
Pruned and installed a few utilities, while updating the system to the latest version:
# apt-get install bootlogd vim mosh screen bsd-mailx postfix # apt-get --purge remove consolekit triggerhappy # apt-get --purge remove cups.* xserver.* x11.* # apt-get update # apt-get dist-upgrade # apt-get autoclean # apt-get autoremove
Configured the language, so it would use my language (and above all, stop apt-get and other tools from complaining about a locale not being set):
# apt-get install locales # dpkg-reconfigure locales
Changed a few settings, in particular, set RAMTMP=yes, to have /tmp in ram, rather than write on ssd, and mounted boot as read only. Both to protect the file systems, in case something else goes wrong with the SSD:
# vim /etc/defaults/tmpfs ... RAMTMP=yes ... # vim /etc/fstab ... /dev/mmcblk0p1 /boot vfat defaults,ro 0 2
Given the corruption problem I had, I was tempted to mark the root file system as using
data=journal, or even sync. Given that I had not found the root cause of the corruption,
in the end I decided to do a back up and leave the setup as is :).
Installed cron jobs. I have two cron jobs on the raspberry pi:
To check the internal temperature, send me an email and reboot the device if it is too hot.
To verify that the wireless is up, and restart it if it is not. I have a tiny USB wireless dongle, an EW-7811Un, which generally works well. However, it does disconnect from time to time, especially if I reboot or poke at the access point :).
The first script is this one:
$ cat ./check-connectivity.sh #!/bin/bash attempts=5 # This is any machine on your network that is always on. The script tries # to ping this machine a few times, if it fails, it restarts the wireless. server=server for n in `seq $attempts`; do logger -t "connectivity-check" "Sending ping request $n to '$server'." ping -c1 "$server" &>/dev/null && { logger -t "connectivity-check" "Server is reachable, nothing to do." exit 0 } done logger -t "connectivity-check" "Server is unreachable, restarting wireless." ( set -x ifdown wlan0 rmmod 8192cu modprobe 8192cu ifup wlan0 ) 2>&1 | (while read line; do logger -t "connectivity-check" "Output: $line"; done;)
While this is the second one:
cat ./check-temperature.sh #!/bin/bash temperature=`vcgencmd measure_temp | sed -e 's/.*=\([^.]*\).*/\1/'` precise=`vcgencmd measure_temp | sed -e "s/.*=\([^\']*\).*/\1/"` email=youremailaddress@whatever.com max=70 logger -t "temperature-check" "Temperature: $precise, max: $max" if [ "$temperature" -ge "$max" ]; then (echo "The temperature is currently $temperature. Greater or equal to $max." echo "" echo "SHUTTING DOWN THE SYSTEM IN 30 SECONDS") | mail $email -s 'Temperature too high - shutting down!' &>/dev/null sync sleep 30 halt fi
To configure cron, I had to add the following lines to /etc/crontab:
* * * * * root /root/utils/check-temperature.sh * * * * * root /root/utils/check-connectivity.sh
Ok, after I install the web interface of my irrigation system, everything seems to be up again.
Now to the backup part. I could remove the memory card again, and do the same I did before. But I am lazy, and would like a backup I can do remotely. So, here's what I did:
Mounted the image I installed on the raspberry locally (remember? one of the first
dds in this blog post). Given that the image contains a few partitions, I had to
use kpartx to make them available, like this:
# kpartx -l 2013-07-26-wheezy-raspbian.img loop0p1 : 0 114688 /dev/loop0 8192 loop0p2 : 0 3665920 /dev/loop0 122880 # mount /dev/mapper/loop0p2 /mnt/raspberry
Make sure to use the loop device created by kpartx, as shown in the output.
Once mounted, used rsync to copy everything from the raspberry to /mnt/raspberry.
I have a terrible memory for the options and flags to use, so just used this
ac-system-backup here,
with something like:
# ac-system-backup 10.1.40.9 /mnt/raspberry
At the end of the sync, unmounted the partitions with:
# umount /mnt/raspberry # kpartx -d 2013-07-26-wheezy-raspbian.img
Finally, renamed the image as:
# mv 2013-07-26-wheezy-raspbian.img backup-raspberry-2013-07-31.img
Next time I have to do a backup, I will first copy this image, and then run rsync.
I still have not found the cause of the file system corruption. In theory, ext4 with journaling should be able to recover cleanly from most states, especially on a system that is hardly (if ever) modified, and the only write activities are for logs.
I spent some time looking at the backup, and the file system was terribly corrupted. Eg, directories turned into files, files with the wrong content, and so on. If it is a software problem, it is a nasty one :)
I checked the power supply, and it is very good, both in terms of voltage and amperage. Given that this is not the first time it happens, I suspect the memory card or the thermal cycle causing issues.
If it happens again, I will probably replace the memory card and see.