Running ARM Grub on U-boot on Qemu https://www.hellion.org.

uk/blog/posts/grub-on-uboot-on-qemu/

ijc's blog/ posts/ Running ARM Grub on U-boot on Qemu

At the Mini-debconf in Cambridge back in November there was an ARM Sprint (which Hector
wrote up as a Bits from ARM porters mail). During this there a brief discussion about using
GRUB as a standard bootloader, particularly for ARM server devices. This has the advantage of
providing a more "normal" (which in practice means "x86 server-like") as well as flexible solution
compared with the existing flash-kernel tool which is often used on ARM.

On ARMv7 devices this will more than likely involve chain loading from the U-Boot supplied by
the manufacturer. For test and development it would be useful to be able to set up a similar
configuration using Qemu.

Cross-compilers
Although this can be built and run on an ARM system I am using a cross compiler here. I'm using
gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux from Linaro, which can be downloaded
from the linaro-toolchain-binaries page on Launchpad. (It looks like 2013.10 is the latest available
right now, I can't see any reason why that wouldn't be fine).

Once the cross-compiler has been downloaded unpack it somewhere, I will refer to the resulting
gcc-linaro-arm-linux-gnueabihf-4.8-2013.08_linux directory as $CROSSROOT.

Make sure $CROSSROOT/bin (which contains arm-linux-gnueabihf-gcc etc) is in your $PATH.

Qemu
I'm using the version packaged in Jessie, which is 1.7.0+dfsg-2. We need both qemu-system-arm
for running the final system and qemu-user to run some of the tools. I'd previously tried an older
version of qemu (1.6.x?) and had some troubles, although they may have been of my own
making...

Das U-boot for Qemu

First thing to do is to build a suitable u-boot for use in the qemu emulated environment. Since we
need to make some configuration changes we need to build from scratch.

Start by cloning the upstream git tree:

$ git clone git://git.denx.de/u-boot.git
$ cd u-boot

I am working on top of e03c76c30342 "powerpc/mpc85xx: Update

CONFIG_SYS_FSL_TBCLK_DIV for T1040" dated Wed Dec 11 12:49:13 2013 +0530.

We are going to use the Versatile Express Cortex-A9 u-boot but first we need to enable some
additional configuration options:

CONFIG_API -- This enables the u-boot API which Grub uses to access the lowlevel services
provided by u-boot. This means that grub doesn't need to contains dozens of platform

1 of 5 06/05/2019 à 13:54
Running ARM Grub on U-boot on Qemu https://www.hellion.org.uk/blog/posts/grub-on-uboot-on-qemu/

specific flash, mmc, nand, network, console drivers etc and can be completely platform
agnostic.
CONFIG_SYS_MMC_MAX_DEVICE -- Setting CONFIG_API needs this.
CONFIG_CMD_EXT2 -- Useful for accessing EXT2 formatted filesystems. In this example I use
a VFAT /boot for convenience but in a real system we would want to use EXT2 (or even
something more modern)).
CONFIG_CMD_ECHO -- Just useful.

You can add all these to include/configs/vexpress_common.h:

#define CONFIG_API
#define CONFIG_SYS_MMC_MAX_DEVICE 1
#define CONFIG_CMD_EXT2
#define CONFIG_CMD_ECHO

Or you can apply the patch which I sent upstream:

$ wget -O - http://patchwork.ozlabs.org/patch/304786/raw | git apply --index
$ git commit -m "Additional options for grub-on-uboot"

Finally we can build u-boot:

$ make CROSS_COMPILE=arm-linux-gnueabihf- vexpress_ca9x4_config
$ make CROSS_COMPILE=arm-linux-gnueabihf-

The result is a u-boot binary which we can load with qemu.

GRUB for ARM

Next we can build grub. Start by cloning the upstream git tree:
$ git clone git://git.sv.gnu.org/grub.git
$ cd grub

By default grub is built for systems which have RAM at address 0x00000000. However the
Versatile Express platform which we are targeting has RAM starting from 0x60000000 so we need
to make a couple of modifications. First in grub-core/Makefile.core.def we need to change
arm_uboot_ldflags, from:

-Wl,-Ttext=0x08000000

to
-Wl,-Ttext=0x68000000

and second we need make a similar change to include/grub/offsets.h changing

GRUB_KERNEL_ARM_UBOOT_LINK_ADDR from 0x08000000 to 0x68000000.

Now we are ready to build grub:

$ ./autogen.sh
$ ./configure --host arm-linux-gnueabihf
$ make

Now we need to build the final grub "kernel" image, normally this would be taken care of by

2 of 5 06/05/2019 à 13:54
Running ARM Grub on U-boot on Qemu https://www.hellion.org.uk/blog/posts/grub-on-uboot-on-qemu/

grub-install but because we are cross building grub we cannot use this and have to use grub-
mkimage directly. However the version we have just built is for the ARM target and not for host we
are building things on. I've not yet figured out how to build grub for ARM while building the tools
for the host system (I'm sure it is possible somehow...). Luckily we can use qemu to run the ARM
binary:
$ cat load.cfg
set prefix=(hd0)
$ qemu-arm -r 3.11 -L $CROSSROOT/arm-linux-gnueabihf/libc \
./grub-mkimage -c load.cfg -O arm-uboot -o core.img -d grub-core/ \
fat ext2 probe terminal scsi ls linux elf msdospart normal help echo

Here we create load.cfg which is the setup script which will be built into the grub kernel, our
version just sets the root device so that grub can find the rest of its configuration.

Then we use qemu-arm-static to invoke grub-mkimage. The "-r 3.11" option tells qemu to pretend
to be a 3.11 kernel (which is required by the libc used by our cross compiler, without this you will
get a fatal: kernel too old message) and "-L $CROSSROOT/..." tells it where to find the basic
libraries, such as the dynamic linker (luckily grub-mkimage doesn't need much in the way of
libraries so we don't need a full cross library environment.

The grub-mkimage command passes in the load.cfg and requests an output kernel targeting arm-
uboot, core.img is the output file and the modules are in grub-core (because we didn't actually
install grub in the target system, normally these would be found in /boot/grub). Lastly we pass in a
list of default modules to build into the kernel, including filesystem drivers (fat, ext2), disk
drivers (scsi), partition handling (msdos), loaders (linux, elf), the menu system (normal) and
various other bits and bobs.

So after all the we now have our grub kernel in core.img.

Putting it all together

Before we can launch qemu we need to create various disk images.

Firstly we need some images for the 2 64M flash devices:

$ dd if=/dev/zero of=pflash0.img bs=1M count=64
$ dd if=/dev/zero of=pflash1.img bs=1M count=64

We will initialise these later from the u-boot command line.

Secondly we need an image for the root filesystem on an MMC device. I'm using a FAT formatted
image here simply for the convenience of using mtools to update the images during development.
$ dd if=/dev/zero of=mmc.img bs=1M count=16
$ /sbin/mkfs.vfat mmc.img

Thirdly we need a kernel, device tree and grub configuration on our root filesystem. For the first
two I extracted them from the standard armmp kernel flavour package. I used the backports.org
version 3.11-0.bpo.2-armmp version and extracted /boot/vmlinuz-3.11-0.bpo.2-armmp as vmlinuz
and /usr/lib/linux-image-3.11-0.bpo.2-armmp/vexpress-v2p-ca9.dtb as dtb. Then I hand coded
a simple grub.cfg:

3 of 5 06/05/2019 à 13:54
Running ARM Grub on U-boot on Qemu https://www.hellion.org.uk/blog/posts/grub-on-uboot-on-qemu/

menuentry 'Linux' {
echo "Loading vmlinuz"
set root='hd0'
linux /vmlinuz console=ttyAMA0 ro debug
devicetree /dtb
}

In a real system the kernel and dtb would be provided by the kernel packages and grub.cfg would
be generated by update-grub.

Now that we have all the bits we need copy them into the root of mmc.img. Since we are using a
FAT formatted image we can use mcopy from the mtools package.
$ mcopy -v -o -n -i mmc.img core.img dtb vmlinuz grub.cfg ::

Finally after all that we can run qemu passing it our u-boot binary and the mmc and flash images
and requesting a Cortex-A9 based Versatile Express system with 1GB of RAM:
$ qemu-system-arm -M vexpress-a9 -kernel u-boot -m 1024m -sd mmc.img \
-nographic -pflash pflash0.img -pflash pflash1.img

Then at the VExpress# prompt we can configure the default bootcmd to load grub and save the
environment to the flash images. The backslash escapes (\$ and \;) should be included as written
here so that e.g. the variables are only evaluated when bootcmd is evaluated and not immediately
when setting bootcmd and the bootm is set as part of bootcmd instead of executed immediately:
VExpress# setenv bootcmd fatload mmc 0:0 \${loadaddr} core.img \; bootm \${loadaddr}
VExpress# saveenv

Now whenever we boot the system it will automatically load boot grub from the mmc and launch
it. Grub in turn will load the Linux binary and DTB and launch those. I haven't actually configure
Linux with a root filesystem here so it will eventually panic after failing to find root.

Future work
The most pressing issue is the hard coded load address built in to the grub kernel image. This is
something which needs to be discussed with the upstream grub maintainers as well as the Debian
package maintainers.

Now that the ARM packages have hit Debian (in experimental in the 2.02~beta2-1 package) I also
plan to start looking at debian-installer integration as well as updating flash-kernel to setup the
chain load of grub instead of loading a kernel directly.

Forgot to mention...

... that it is possible to debug things using gdb via qemu. First add -s -S to the qemu-system-arm
command line, where -s is short for -gdb tcp::1234 (i.e. start a remote gdb server on TCP port
1234) and -S means wait for gdb to connect before starting.

Then run arm-linux-gnueabihf-gdb grub-core/kernel.exec and run:

(gdb) target remote :1234
(gdb) b *0x68000000
(gdb) c

4 of 5 06/05/2019 à 13:54
Running ARM Grub on U-boot on Qemu https://www.hellion.org.uk/blog/posts/grub-on-uboot-on-qemu/

This will connect to qemu, set a break point at the grub entry point and then continue.

I've also found that you can build grub twice, once for x86 and once for arm-uboot and then use
the x86 version of grub-mkimage to build an arm-boot image. Eventually once the ARM support
hits the Debian archive I presume the installed version of grub-mkimage would be fine but in the
meantime something like this seems to do the trick:
$ ( mkdir obj-arm && cd obj-arm && ./configure --host arm-linux-gnueabihf && make )
$ ( mkdir obj-x86 && cd obj-x86 && ./configure && make )
$ ./obj-x86/grub-mkimage -c load.cfg -O arm-uboot -o core.img -d obj-arm/grub-core \
fat ext2 probe terminal scsi ls linux elf msdospart normal help echo

Lastly I've made some progress on getting grub-mkimage to relocate the arm-uboot flavour of
grub at runtime instead of hardcoding a particular ram address. I don't know if it will be
acceptable upstream but it works for me and I'll send it along once I've cleaned it up a bit.

Comment by ijc — Fri Dec 27 19:45:01 2013

packaged?

Hi,

is this packaged? I see grub-uboot in jessie/armhf, and lots of u-boot flavours, but I
have no idea which one to use for my qemu image (based on Aurélien’s).

Comment by mirabilos — Fri May 29 10:32:18 2015

comment 3

(I really need to figure out how to enable some sort of comment notification).

TTBOMK this is not packaged. After playing with this a bit more I came to the
conclusion that grub on uboot ended up being a bit of a non-starter, partly due to the need to
make the grub kernel relocatable at boot time but mainly because the appl ABI exported by
u-boot isn't really usable as is (not actually nuchanging, discovery mechanism is too ad-hoc etc).
In the end it didn't seem worth investing the amount of effort which would be required to make it
work.

Comment by ijc — Fri Nov 6 10:00:49 2015

Add a comment

Last edited Thu Dec 26 13:20:51 2013

5 of 5 06/05/2019 à 13:54