Readonly root filesystem for i.MX6 boards running Debian or Ubuntu

Published on April 20, 2015

Archived Notice

This article has been archived and may contain broken links, photos and out-of-date information. If you have any questions, please Contact Us.

Do your customers think of your product more like computer or a toaster?

The software driving our Linux-based systems has roots in mini-computers, where the systems lived in data centers with raised floors, un-interruptible power supplies and staff to handle backups and maintenance.

That's probably not the case for your products. They likely boot directly into your application and have an easily accessible power cord, but not an easily identified "shutdown" button.

In this post, we'll describe a tool (overlayfs) that can help make your systems more reliable in the event of abrupt system power-off.

For the impatient

We've included support for the overlayfs filesystem in our 3.10.53 kernel. One of its' uses is to mount the root filesystem as read-only, while preserving a read-write environment in a RAM disk "overlaid" on top of the rootfs.

To see it in action, you can use this Debian Jessie image:

20150417-nitrogen-3.10.53_1.1.1_ga-jessie-en_US-xfce_armhf.img.gz

Programming the image

The images are slightly-less-than-4GiB image files containing the partition table, so you can copy it to an SD card or SATA drive on /dev/sdc using zcat and dd like so:

~/Downloads$ sudo umount /dev/sdc*
~/Downloads$ zcat 20150417*.img.gz | sudo dd of=/dev/sdc bs=1M
~/Downloads$ sync

Or you can use Alex Page's USB Image Tool under Windows.

If you're using our latest U-Boot and a Nitrogen6 Max board, you can use the new USB Mass Storage Gadget to program the eMMC.

Installing (and selecting) the overlayfs support

The image above, and all future releases of Debian Ubuntu will contain overlayfs support, but they will not be pre-configured to use it. To install and configure it, you'll need to take the following steps:

Make sure you have the kernel 3.10.53 or later installed, and at least ABI >= 7 :
ubuntu@trusty-dev:~$ uname -r 3.10.53-7-boundary-7t5 ubuntu@trusty-dev:~$
If no, upgrade the kernel as follows:
ubuntu@trusty-dev:~$ sudo apt-get update && sudo apt-get dist-upgrade
Install the overlayfs script. This script runs in the initial ramfs disk (initramfs) at boot time, and constructs an overlayfs stack that includes your read-only root and a RAM-disk top level filesystem (more details below).
ubuntu@trusty-dev:~$ sudo apt-get install overlayfs-script1 && sync && sudo reboot Reading package lists... Done Building dependency tree Reading state information... Done The following NEW packages will be installed: overlayfs-script1 0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded. Need to get 3274 B of archives. After this operation, 47.1 kB of additional disk space will be used. Get:1 http://linode.boundarydevices.com/repos/apt/ubuntu-rel/ trusty/main overlayfs-script1 armhf 1.0.0-1 [3274 B] Fetched 3274 B in 0s (16.0 kB/s) Selecting previously unselected package overlayfs-script1. (Reading database ... 52288 files and directories currently installed.) Preparing to unpack .../overlayfs-script1_1.0.0-1_armhf.deb ... Unpacking overlayfs-script1 (1.0.0-1) ... Setting up overlayfs-script1 (1.0.0-1) ... update-initramfs: deferring update (trigger activated) Processing triggers for initramfs-tools (0.103ubuntu4.2) ... update-initramfs: Generating /boot/initrd.img-3.10.53-7-boundary-7t5
Enable your read-only filesystem by setting U-Boot environment variables:
U-Boot > setenv overlayfs yes U-Boot > setenv ofs-size 20 U-Boot > saveenv U-Boot > boot
The first line enables the overlayfs-script. The second line sets the size of the overlafs system as 20% of the available system RAM. If ofs-size is not set, the overlayfs system will default to 50%, which is excessive in most cases.

Note that memory for the RAM disk is not pre-allocated, and will only be used as writes to the filesystem occur.

Poking around after booting

After booting you can check the mounted partitions:

ubuntu@trusty-dev:~$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
udev on /dev type devtmpfs (rw,relatime,size=382632k,nr_inodes=95658,mode=755)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620)
tmpfs on /run type tmpfs (rw,nosuid,noexec,relatime,size=102956k,mode=755)
/dev/disk/by-path/platform-2198000.usdhc-part1 on /mnt/root-RO/mnt/root-RO type ext4 (ro,relatime,data=ordered)
tmpfs-root on /mnt/root-RO/mnt/root-RW type tmpfs (rw,relatime,size=154432k)
overlayfs-root on /mnt/root-RO type overlayfs (rw,relatime,lowerdir=/mnt/root-RO,upperdir=/mnt/root-RW)
tmpfs-root on /mnt/root-RW type tmpfs (rw,relatime,size=154432k)
overlayfs-root on / type overlayfs (rw,relatime,lowerdir=/mnt/root-RO,upperdir=/mnt/root-RW)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,nosuid,nodev,noexec,relatime)
none on /sys/fs/cgroup type tmpfs (rw,relatime,size=4k,mode=755)
none on /sys/fs/fuse/connections type fusectl (rw,relatime)
none on /sys/kernel/debug type debugfs (rw,relatime)
none on /sys/kernel/security type securityfs (rw,relatime)
none on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
none on /run/shm type tmpfs (rw,nosuid,nodev,relatime)
none on /run/user type tmpfs (rw,nosuid,nodev,noexec,relatime,size=102400k,mode=755)
ubuntu@trusty-dev:~$

Note that the root mount point is the overlayfs (line 11:overlayfs-root on /), and that the lowerdir and upperdir options specify the overlay stack.

Also note that we modified the file /etc/fstab to include a reference to the read-only rootfs and some comments about how to re-mount it as read-write:

ubuntu@trusty-dev:~$ sudo cat /etc/fstab
#  This fstab is in RAM, the real one can be found at /mnt/root-RO/etc/fstab
#  The original entry for '/' and all swap files have been removed.  The new
#  entry for the read-only the real root fs follows. Write access can be
#  enabled using:
#    sudo mount -o remount,rw /mnt/root-RO
#  re-mounting it read-only is done using:
#    sudo mount -o remount,ro /mnt/root-RO
#
/dev/disk/by-path/platform-2198000.usdhc-part1 /mnt/root-RO ext4 ro,relatime,data=ordered 0 1
#  remaining entries from the original /mnt/root-RO/etc/fstab follow.
#
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
#
#

The resulting filesystem acts like a regular read-write filesystem. You can add, copy, install, and remove files as you want, your old system appears to work unchanged.

But your changes will not be persistent: they will stay around only until the next reboot!

After reboot you get back the original image, just like if it were an unchangeable CD.

The advantage is obvious: users can't harm to the system by handling it improperly (without graceful shutdown), because actually they don't and they can't write anything to SD card, because the SD card is mounted as read-only.

The resulting image is invincible, (not fireproof or waterproof though, I never said that) it becomes immune to the rough users, and the toaster will always work.

Where the magic happens

The key to making all of this happen comes from the fact that our Debian and Ubuntu images boot first to a RAM disk (the initramfs image). The overlayfs-script1 script we installed above is executed in the context of that RAM disk to locate and create the two-layer stack of writeable top layer and a read-only bottom layer.

To understand the details, you can grab the sources using apt-get source:

ubuntu@trusty-dev:~$ apt-get source overlayfs-script1
ubuntu@trusty-dev:~$ sudo grep -A 10 /proc/cmdline overlayfs-scripts-1.0.0/overlayfs-script1
for CMDLINE_OPT in $(cat /proc/cmdline); do
	case ${CMDLINE_OPT} in
		overlayfs=*)
			OVERLAYFS=${CMDLINE_OPT#overlayfs=}
		;;
		ofs-size=*)
			OFS_SIZE=${CMDLINE_OPT#ofs-size=}
		;;
	esac
done

Modifying the read-only filesystem:

Of course after you made your system read-only, its configuration, updating, upgrading, any kind of persistent modification is not so simple.

You can disable the overlayfs-script temporarily from the U-Boot prompt :

U-Boot > setenv overlayfs no
U-Boot > boot

Note that we didn't use saveenv, so this will only affect a single boot.

After this, you can change anything in the image, and the modifications you make will be persistent.

If you want to make the change permanent, and go back to using a read-write root, you can change the variable using the fw_setenv program as described in this post about accessing the boot flash under Linux.

About overlayfs

OverlayUnion filesystems have a long history in Linux, with Unionfs back in 2004, AUFS in 2006. Their uses varied, but were driven by a desire to run systems off of inherently read-only media like CD-ROMs, and many Linux distributions have used them to distribute Live CDs that allow you to boot completely off of CD.

Overlayfs is the latest in the category and the first to gain acceptance into the Linux kernel source tree.

It can be used with more than two partitions in the stack and in general, updates will occur in the top-most layer of
the stack which contains a given directory and is writeable.

Care must be taken when references to older, read-only inodes are held when updates are made to upper layers of the stack, so it is not fully transparent to applications.