[Draft] Running NixOS on Banana Pi R4: Building the boot sequence

2024-09-30 14:01:58 +02:00

17 KiB

Raw Blame History

+++ title = 'Running NixOS on the Banana Pi R4: Building the boot sequence' date = 2024-09-29T08:37:23+02:00 draft = true +++

Trigger Warnings

I'm by no mean an expert about U-Boot, neither in ARM CPUs. This is just me trying to port NixOS on the Banana Pi R4, and sharing what I learnt. As such, I may (and certainly will) utter false statements. If I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it !
I can't say really that I'm fluent in English. If you see some grammar or spelling mistakes, please reach me too !
This post's goal is not to be exhaustive, but I want to emphase what I think is important or useful to know. My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material.

Thanks for your comprehension, and have a great time reading this.

How did we get there

It has been a while since I started waiting for the Banana Pi R4. I bought it as soon as a complete bundle with the WiFi 7 module was available. I initially planned to run OpenWRT on it, without bothering too much. But if my machines (including my ROCKPro64) are all running NixOS with my common flake configuration, there is no reason why my router should not. This is because I use a colmena-based deployment system. It allows me to build (and cross-compile) all my machines configurations and to deploy them from my PC.

However, there is two caveats here:

The Nix store usually takes some place on storage devices, so the embedded 8GB eMMC might not be enough. But, since the BPI R4 has an integrated slot for a NVME SSD and I have an empty 500 GB SSD available, it is way more than enough.
There is currently no support or available image of NixOS for the BPI R4.

This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4.

For your information, this is what I had before starting this project:

Some basic Nix knowledge.
I already used U-Boot on some boards with prebuilt Linux images, but never really installed it or tweaked it myself.
Some knowledge about basic Linux utilities used for embedded.
A USB-Serial cable.

For a NixOS SD image generation, some people chose to build it with their Nix configuration already applied to it, and then just flash this image to their board flash device. I will personally just build an image that just serve as an install media, and then manually do all the installation process as I would do on a PC.

Today's post is the first part about building our boot components and getting access to the U-Boot console.

Let's get started !

The ARMv8 boot sequence

Before focusing on the Linux kernel or NixOS, we will have to build the boot sequence for the R4. This includes ARM TrustedFirmware-A and U-Boot.

On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that will be run to init our board) is standardized with the following elements:

BL1: this is a software brick that is already present in the board's Boot ROM. It performs basic hardware initialization and then pass the control to the next step (BL2).
BL2: Another software brick whose goal is to create a secure environment for the software that will be run, and then pass the control to the last step.
BL3: the final element of the boot sequence. It is basically what the board will finally run. In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software), an optionnal BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run.

It seems in reality a bit more complicated than that, dealing with ARM Exception Level, but we will stick to this simple representation.

ARM TrustedFirmware-A provides a standard implementation for BL2, BL31 and BL32. Its documentation is available here. U-Boot is the bootloader which is the most commonly used on ARM boards, and that will act as BL33. It has all the necessary utilities to load and boot the Linux kernel.

Once the board reach U-Boot, it will be able to boot NixOS. Nixpkgs provide two functions to build these two elements as Nix derivations: buildUBoot and buildArmTrustedFirmware. These two functions make the creation of such derivations fairly easy.

Cross compilation on NixOS (for Nix-beginners)

If you're not on an aarch64 machine (or did not enable native build through QEMU on your configuration), you won't be able to build the following derivations. To do so, we have to setup cross-compilation. Everybody knows Nixpkgs as a collection of Nix derivations that contains some utility functions like mkDerivation too. It also provides a cross-compilation system that allows to cross compile each derivation for a compatible architecture !

For example, if we want to build any aarch64 package from any architecture (like hello), we can just run:

nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs/lib>).systems.examples.aarch64-multiplatform' -A hello

This will cross-compile hello for aarch64 in the Nix store. It is possible, because as any package in Nixpkgs, hello is declared through a Nix recipe (a callPackage derivation) that is called by the callPackage function of Nixpkgs. This function setup a bunch of things and among them cross-compilation. It does so by looking at the crossSystem parameter provided to Nixpkgs. It will then cross-compile every dependency of the derivation before finally building our package.

To cross-compile every program we need, we just have to create the following default.nix file:

{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
{
  myProgram = pkgs.callPackage ./myProgram { };
}

Here, we just need to have a myProgram directory with a default.nix file containing a callPackage derivation for our program, and then call nix-build -A myProgram.

Building U-Boot

Most of current Linux images for the BPI R4 uses frank-w's U-Boot fork. This is because, for now, mainline U-Boot does not support the router. His modifications to bring support for our device are just few commits that I'll export as patch, and apply them on top of mainline U-Boot.

Let's create the following default.nix file:

{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
{
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
}

In a u-boot directory, we will create a default.nix file and a patches directory that will contains all the patches that I took from frank-w's U-Boot. Within the u-boot/default.nix file, we will just have to write the following callPackage derivation:

{ buildUBoot, ... }:
buildUBoot {
  defconfig = "mt7988a_bpir4_sd_defconfig";
  extraMeta.platforms = [ "aarch64-linux" ];
  extraPatches = [
    ./patches/0001-pci-mediatek-add-PCIe-controller-support-for-Filogic.patch
    ./patches/0002-Fix-PCIE-on-BPIR4.patch
    ./patches/0003-arm-dts-enable-pcie-in-sd-dts-too.patch
    ./patches/0004-dts-r4-disable-pcie2-in-emmc-dts.patch
    ./patches/0005-defconfig-uEnv-add-defconfigs-and-environment-files.patch
    ./patches/0006-defconfig-r4-update-with-pcie-options.patch
    ./patches/0007-defconfig-r4-add-pstore.patch
    ./patches/0008-defconfig-r4-update-emmc-defconfig.patch
    ./patches/0009-defconfig-r4-fix-duplicates-in-emmc-defconfig.patch
    ./patches/0010-arm64-dts-move-pcie-phy-to-dedicated-xsphy-no-driver.patch
    ./patches/0011-pci-mediatek-print-controller-address-for-card-detec.patch
  ];

  extraConfig = ''
    CONFIG_FIT=n
    CONFIG_USE_DEFAULT_ENV_FILE=n
  '';

  filesToInstall = [ "u-boot.bin" ];
}

As I said previously, Nixpkgs provides a buildUBoot function, to which we have just to pass some arguments. Here, frank-w's U-Boot patches define a new defconfig called mt7988a_bpi_r4_sd_defconfig for the R4. This config enables the generation of a Flattened Image Tree. As far as I understand, this is a blob containing all the necessary boot configurations and files that U-Boot needs to launch our kernel. In my case, at least for the beginning, I chose to stick with the plain-old U-Boot builds (without using FIT) to learn to do it myself instead of using any special packaging format. In the end, the filesToInstall arguments just specifies which final build products we want to keep in the derivation output. For now, we will only keep the u-boot binary and see if we actually need anything else later. That's it for U-Boot ! We can try to build it with nix-build -A ubootBpiR4 and see the u-boot.bin file appear in the Nix store, and in the result folder !

ARM TrustedFirmware-A

As with U-Boot, the mainline version of TrustedFirmware does not support the BPI-R4. For now, distributions are using this fork. I don't really know for sure who is behind this account, but there is a bit too much commits for me to export them as patch to apply on top of mainline. Once again, we will use the Nixpkgs's buildArmTrustedFirmware function. Before that, and as we can see on the frank-w's U-Boot build scripts, we will need to pass to U-Boot binary. It should be included within the final fip.bin file.

To be honest there, I don't really know if TrustedFirmware is mandatory to run U-boot on our SoC, or if U-Boot by itself could have handled everything. But for now, I decided to proceed like the OpenWRT and Debian images. I will surely dig in the TrustedFirmare and ARM documentations later. I'll eventually to a follow-up post later if I achieve to figure out how everything is working.

So, let's create our callPackage derivation in trusted-firmware/default.nix:

{
  buildArmTrustedFirmware,
  fetchFromGitHub,
  dtc,
  ubootBpiR4,
  ubootTools,
  openssl,
  ...
}:
(buildArmTrustedFirmware rec {
  platform = "mt7988";
  extraMeta.platforms = [ "aarch64-linux" ];
  extraMakeFlags = [
    "USE_MKIMAGE=1"
    "BOOT_DEVICE=sdmmc"
    "DRAM_USE_COMB=1"
    "BL33=${ubootBpiR4}/u-boot.bin"
    "all"
    "fip" 
  ];

  filesToInstall = [
    "build/${platform}/release/bl2.img"
    "build/${platform}/release/fip.bin"
  ];
}).overrideAttrs (old: {
  src = fetchFromGitHub {
    owner = "mtk-openwrt";
    repo = "arm-trusted-firmware";
    rev = "bacca82a8cac369470df052a9d801a0ceb9b74ca";
    hash = "sha256-n5D3styntdoKpVH+vpAfDkCciRJjCZf9ivrI9eEdyqw=";
  };
  version = "2.10.0-mtk";
  nativeBuildInputs = old.nativeBuildInputs ++ [ dtc ubootTools openssl ];
}

This is very similar what we have done with U-Boot, we select our platform, then pass the necessary flags for compilation, and then grab the two output files that we need. However here, there is some extra steps. Both buildUBoot and buildArmTrustedFirmware assumes that you're building the mainline U-Boot and TF-A. It was the case with our U-Boot build to which we just add some extra patches. However, with TF-A, we will here use the mtk-openwrt's fork. So we need to override the derivation produced by buildArmTrustedFirmware to pass our own sources and needed dependencies thanks to the overrideAttrs function.

As we can see, this callPackage derivation needs to access our previous ubootBpiR4 derivation. All the other derivations arguments are provided by nixpkgs itself through the callPackage function. However, ubootBpiR4 is itself not present within nixpkgs, we have to manually pass it. There is two methods to do so, the first one is to use a nixpkgs overlay. The other one is to pass it manually as a callPackage argument. That's what we'll do here. This makes our default.nix file looks like that:

{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
rec {
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
  armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
}

Now, we can run nix-build -A armTrustedFirmwareBpiR4 to build everything we need.

Side note on the U-Boot and TF-A combination

If you check how to build the full boot sequence binaries for various board, you will maybe feel like me that it seems to be highly dependant on the SoC manufacturer. In our case, we build U-Boot first and then pass it to TF-A as BL33. For some Allwinner SoCs, TF-A is built first and then passed to U-boot as BL31. For Rockchip based boards there is this whole idbloader.img due to Rockchip's miniloader. At this point, I don't really know if there is a "standard way" to build to whole boot sequence for each SoC. I don't even know what causes these differences in the build process between manufacturers. I guess it's related to the manufacturer implementation, or maybe things that could be done both in U-Boot and in TF-A and then depends about where the manufacturer implemented them ?

Create our boot image

The final step is just to create an empty image and flash what we have built so far at the matching address ranges. This will be done with a simple bash script derivation in the image/default.nix file.

{
  runCommand,
  armTrustedFirmwareBpiR4,
  gptfdisk,
  ...
}:
runCommand "bpi-r4-image" {
  nativeBuildInputs = [
    gptfdisk
  ];
} ''
  IMAGE=$out/nixos-r4-image.img

  mkdir $out

  dd if=/dev/zero of=$IMAGE bs=1M count=4000
  sgdisk -o $IMAGE
  sgdisk -a 1 -n 1:34:8191 -A 1:set:2 -t 1:8300 -c 1:"bl2" $IMAGE
	sgdisk -a 1 -n 2:8192:9215 -A 2:set:63	-t 2:8300 -c 2:"u-boot-env"	$IMAGE
	sgdisk -a 1 -n 3:9216:13311 -A 3:set:63	-t 3:8300 -c 3:"factory" $IMAGE
	sgdisk -a 1 -n 4:13312:17407 -A 4:set:63	-t 4:8300 -c 4:"fip" $IMAGE

  dd if=${armTrustedFirmwareBpiR4}/bl2.img of=$IMAGE seek=34 conv=notrunc,fsync
  dd if=${armTrustedFirmwareBpiR4}/fip.bin of=$IMAGE seek=13312 conv=notrunc,fsync
''

The nixpkgs runCommand function creates a derivation that runs a bash script. This bash script is ran into the nixpkgs standard environment, but we can override this environment with the set passed as the second argument. In this case, we add the gptfdisk package as a dependency to use sgdisk within our bash script. The third argument is the script in itself. We then create the derivation output directory, and generate a zero-filled 4GB image with dd. Then, we generate our partition table with sgdisk with the following partition map:

Block size of 512 bytes

Blocks 34 to 8191: BL2
Blocks 8192 to 9215: Space for U-Boot environment variables
Blocks 9216 to 13311: Factory
Blocks 13312 to 17407: FIP

Finally, we flash our TF-A build outputs into the matching partitions with dd.

We add this callPackage derivation into our default.nix:

{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
rec {
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
  armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
  image = pkgs.callPackage ./image { inherit armTrustedFirmwareBpiR4; };
}

And we can build our boot sequence components with nix-shell -A image to get our nixos-r4-image.img.

Flashing and running

So let's flash the resulting image into an SD card.

dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress

After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot. We get the following logs with minicom:

F0: 102B 0000
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 380E 5800
F5: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 0600 0041 [0000]
G0: 1190 0000
EC: 0000 0000 [3000]
MK: 0000 0000 [0000]
T0: 0000 0221 [0101]
Jump to BL

NOTICE:  BL2: v2.10.0   (release):
NOTICE:  BL2: Built : 00:00:00, Jan  1 1980
NOTICE:  WDT: Cold boot
NOTICE:  WDT: disabled
NOTICE:  CPU: MT7988
NOTICE:  EMI: Using DDR unknown settings
NOTICE:  EMI: Detected DRAM size: 4096 MB
NOTICE:  EMI: complex R/W mem test passed
NOTICE:  BL2: Booting BL31
NOTICE:  BL31: v2.10.0  (release):
NOTICE:  BL31: Built : 00:00:00, Jan  1 1980


U-Boot 2024.04 (Apr 02 2024 - 10:58:58 +0000)

CPU:   MediaTek MT7988
Model: mt7988-rfb
DRAM:  4 GiB
Core:  49 devices, 19 uclasses, devicetree: separate
MMC:   mmc@11230000: 0
Loading Environment from nowhere... OK
In:    serial@11000000
Out:   serial@11000000
Err:   serial@11000000
Net:
Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06
eth0: ethernet@15100000
BPI-R4>

And here we have our initial access to the U-Boot console !

Sources

I succesfully got to this point after myself reading some documentation and a bunch of blog posts. Those where the resources I used:

17 KiB Raw Blame History