dalaran.fr/content/posts/nixos-bpi-r4/2024-09-29-uboot-console.md

+++
title = 'Running NixOS on the Banana Pi R4: Building the boot sequence'
date = 2024-09-29T08:37:23+02:00
draft = true
+++

# Trigger Warnings

- I'm by no mean an expert about U-Boot, neither in ARM CPUs.
This is just me trying to port NixOS on the Banana Pi R4, and sharing what I learned.
As such, I may (and certainly will) utter false statements.
If I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it !
- I can't say really that I'm fluent in English. If you see some grammar or spelling mistakes, please reach me too !
- This post's goal is not to be exhaustive, but I want to emphasize what I think is important or useful to know.
My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material.

Thanks for your comprehension, and have a great time reading this.

# How did we get there

It has been a while since I started waiting for the Banana Pi R4.
I bought it as soon as a complete bundle with the Wi-Fi 7 module was available.
I initially planned to run OpenWrt on it, without bothering too much.
But if my machines (including my ROCKPro64) are all running NixOS with my
[common flake configuration](https://git.dalaran.fr/dala/nixos-config), there is no reason why my router
should not.
This is because I use a [colmena](https://github.com/zhaofengli/colmena)-based deployment system.
It allows me to build (and cross-compile) all my machines configurations and to deploy them from my PC.

However, there are two caveats here:
- The Nix store usually takes some place on storage devices, so the embedded 8 GB eMMC might not be enough.
But, since the BPI R4 has an integrated slot for a NVMe SSD and I have an empty 500 GB SSD available,
it is way more than enough.
- There is currently no support or available image of NixOS for the BPI R4.

This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4.

For your information, this is what I had before starting this project:
- Some basic Nix knowledge.
- I already used U-Boot on some boards with prebuilt Linux images, but never really installed it or tweaked it myself.
- Some knowledge about basic Linux utilities used for embedded.
- A USB-Serial cable.

For a NixOS SD image generation, some people chose to build it with their Nix configuration already
applied to it, and then just flash this image to their board flash device.
I will personally just build an image that just serve as an installation media, and then manually do
all the installation process as I would do on a PC.

Today's post is the first part about building our boot components and getting access to the U-Boot console.

Let's get started !

# The ARMv8 boot sequence

Before focusing on the Linux kernel or NixOS, we will have to build the boot sequence for the R4.
This includes ARM TrustedFirmware-A and U-Boot.

On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that will be run to init our board)
is standardized with the following elements:
- BL1: this is a software brick that is already present in the board's Boot ROM. It performs basic
hardware initialization and then pass the control to the next step (BL2).
- BL2: Another software brick whose goal is to create a secure environment for the software that will be run, and then
pass the control to the last step.
- BL3: the final element of the boot sequence. It is basically what the board will finally run.
In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software),
an optional BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run.

It seems in reality a bit more complicated than that, dealing with ARM Exception Level, but we will stick
to this simple representation.

ARM TrustedFirmware-A provides a standard implementation for BL2, BL31 and BL32. Its documentation is available
[here](https://trustedfirmware-a.readthedocs.io/en/latest/index.html).
U-Boot is the bootloader which is the most commonly used on ARM boards, and that will act as BL33.
It has all the necessary utilities to load and boot the Linux kernel.

Once the board reach U-Boot, it will be able to boot NixOS.
Nixpkgs provide two functions to build these two elements as Nix derivations: `buildUBoot` and `buildArmTrustedFirmware`.
These two functions make the creation of such derivations fairly easy.

# Cross compilation on NixOS (for Nix-beginners)

If you're not on an aarch64 machine (or did not enable native build through QEMU on your configuration),
you won't be able to build the following derivations.
To do so, we have to set up cross-compilation.
Everybody knows nixpkgs as a collection of Nix derivations that contains some utility functions like `mkDerivation` too.
It also provides a cross-compilation system that allows to cross compile each derivation for a compatible architecture !

For example, if we want to build any aarch64 package from any architecture (like `hello`), we can just run:
```bash
nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs/lib>).systems.examples.aarch64-multiplatform' -A hello
```
This will cross-compile `hello` for aarch64 in the Nix store.
It is possible, because as any package in nixpkgs, `hello` is declared through a Nix recipe (a callPackage derivation) that
is called by the `callPackage` function of nixpkgs.
This function setup a bunch of things and among them cross-compilation.
It does so by looking at the `crossSystem` parameter provided to nixpkgs.
It will then cross-compile every dependency of the derivation before finally building our package.

To cross-compile every program we need, we just have to create the following `default.nix` file:
```nix
{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
{
  myProgram = pkgs.callPackage ./myProgram { };
}
```
Here, we just need to have a `myProgram` directory with a `default.nix` file containing a callPackage derivation
for our program, and then call `nix-build -A myProgram`.

# Building U-Boot

Most of current Linux images for the BPI R4 uses [frank-w's U-Boot fork](https://github.com/frank-w/u-boot).
This is because, for now, mainline U-Boot does not support the router.
His modifications to bring support for our device are just few commits that I'll export as patch, and apply
them on top of mainline U-Boot.

Let's create the following `default.nix` file:
```nix
{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
{
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
}
```

In a `u-boot` directory, we will create a `default.nix` file and a `patches` directory that will contain
all the patches that I took from frank-w's U-Boot.
Within the `u-boot/default.nix` file, we will just have to write the following callPackage derivation:
```nix
{ buildUBoot, ... }:
buildUBoot {
  defconfig = "mt7988a_bpir4_sd_defconfig";
  extraMeta.platforms = [ "aarch64-linux" ];
  extraPatches = [
    ./patches/0001-pci-mediatek-add-PCIe-controller-support-for-Filogic.patch
    ./patches/0002-Fix-PCIE-on-BPIR4.patch
    ./patches/0003-arm-dts-enable-pcie-in-sd-dts-too.patch
    ./patches/0004-dts-r4-disable-pcie2-in-emmc-dts.patch
    ./patches/0005-defconfig-uEnv-add-defconfigs-and-environment-files.patch
    ./patches/0006-defconfig-r4-update-with-pcie-options.patch
    ./patches/0007-defconfig-r4-add-pstore.patch
    ./patches/0008-defconfig-r4-update-emmc-defconfig.patch
    ./patches/0009-defconfig-r4-fix-duplicates-in-emmc-defconfig.patch
    ./patches/0010-arm64-dts-move-pcie-phy-to-dedicated-xsphy-no-driver.patch
    ./patches/0011-pci-mediatek-print-controller-address-for-card-detec.patch
  ];

  extraConfig = ''
    CONFIG_FIT=n
    CONFIG_USE_DEFAULT_ENV_FILE=n
  '';

  filesToInstall = [ "u-boot.bin" ];
}
```

As I said previously, nixpkgs provides a `buildUBoot` function, to which we have just to pass some arguments.
Here, frank-w's U-Boot patches define a new defconfig called `mt7988a_bpi_r4_sd_defconfig` for the R4.
This config enables the generation of a Flattened Image Tree.
As far as I understand, this is a blob containing all the necessary boot configurations and files that U-Boot needs to launch our kernel.
In my case, at least for the beginning, I chose to stick with the plain-old U-Boot builds (without using FIT) to learn to do it myself instead of using
any special packaging format.
In the end, the `filesToInstall` arguments just specifies which final build products we want to keep in the derivation output.
For now, we will only keep the u-boot binary and see if we actually need anything else later.
That's it for U-Boot ! We can try to build it with `nix-build -A ubootBpiR4` and see the `u-boot.bin` file
appear in the Nix store, and in the `result` folder !

# ARM TrustedFirmware-A

As with U-Boot, the mainline version of TrustedFirmware does not support the BPI-R4.
For now, distributions are using [this fork](https://github.com/mtk-openwrt/arm-trusted-firmware).
I don't really know for sure who is behind this account, but there is a bit too much commits for me
to export them as patch to apply on top of mainline.
Once again, we will use the nixpkgs's `buildArmTrustedFirmware` function.
Before that, and as we can see on the [frank-w's U-Boot build scripts](https://github.com/frank-w/u-boot/blob/mtk-atf/build.sh#L40),
we will need to pass to U-Boot binary. It should be included within the final `fip.bin` file.

---

To be honest there, I don't really know if TrustedFirmware is mandatory to run U-boot
on our SoC, or if U-Boot by itself could have handled everything.
But for now, I decided to proceed like the OpenWrt and Debian images.
I will surely dig in the TrustedFirmware and ARM documentations later.
I'll eventually to a follow-up post later if I achieve to figure out how everything is working.

---

So, let's create our callPackage derivation in `trusted-firmware/default.nix`:
```nix
{
  buildArmTrustedFirmware,
  fetchFromGitHub,
  dtc,
  ubootBpiR4,
  ubootTools,
  openssl,
  ...
}:
(buildArmTrustedFirmware rec {
  platform = "mt7988";
  extraMeta.platforms = [ "aarch64-linux" ];
  extraMakeFlags = [
    "USE_MKIMAGE=1"
    "BOOT_DEVICE=sdmmc"
    "DRAM_USE_COMB=1"
    "BL33=${ubootBpiR4}/u-boot.bin"
    "all"
    "fip" 
  ];

  filesToInstall = [
    "build/${platform}/release/bl2.img"
    "build/${platform}/release/fip.bin"
  ];
}).overrideAttrs (old: {
  src = fetchFromGitHub {
    owner = "mtk-openwrt";
    repo = "arm-trusted-firmware";
    rev = "bacca82a8cac369470df052a9d801a0ceb9b74ca";
    hash = "sha256-n5D3styntdoKpVH+vpAfDkCciRJjCZf9ivrI9eEdyqw=";
  };
  version = "2.10.0-mtk";
  nativeBuildInputs = old.nativeBuildInputs ++ [ dtc ubootTools openssl ];
}
```

This is very similar what we have done with U-Boot, we select our platform, then pass the necessary flags for compilation,
and then grab the two output files that we need.
However, in our situation, there are some extra steps.
Both `buildUBoot` and `buildArmTrustedFirmware` assumes that you're building the mainline U-Boot and TF-A.
It was the case with our U-Boot build to which we just add some extra patches.
However, with TF-A, we will here use the mtk-openwrt's fork.
So we need to override the derivation produced by `buildArmTrustedFirmware` to pass our own sources and needed dependencies thanks
to the `overrideAttrs` function.

As we can see, this callPackage derivation needs to access our previous `ubootBpiR4` derivation.
All the other derivations arguments are provided by nixpkgs itself through the `callPackage` function.
However, `ubootBpiR4` is itself not present within nixpkgs, we have to manually pass it.
There are two methods to do so, the first one is to use a nixpkgs overlay.
The other one is to pass it manually as a `callPackage` argument.
That's what we'll do here.
This makes our `default.nix` file looks like that:
```nix
{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
rec {
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
  armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
}
```

Now, we can run `nix-build -A armTrustedFirmwareBpiR4` to build everything we need.

# Side note on the U-Boot and TF-A combination

If you check how to build the full boot sequence binaries for various board, you will maybe feel like me
that it seems to be highly dependent on the SoC manufacturer.
In our case, we build U-Boot first and then pass it to TF-A as BL33.
For some Allwinner SoCs, TF-A is built first and then passed to U-boot as BL31.
For Rockchip based boards there is this whole `idbloader.img` due to Rockchip's miniloader.
At this point, I don't really know if there is a "standard way" to build to whole boot sequence for each SoC.
I don't even know what causes these differences in the build process between manufacturers.
I guess it's related to the manufacturer implementation, or maybe things that could be done both in U-Boot and in TF-A
and then depends on where the manufacturer implemented them ?

# Create our boot image

The final step is just to create an empty image and flash what we have built so far at the matching address ranges.
This will be done with a simple bash script derivation in the `image/default.nix` file.
```nix
{
  runCommand,
  armTrustedFirmwareBpiR4,
  gptfdisk,
  ...
}:
runCommand "bpi-r4-image" {
  nativeBuildInputs = [
    gptfdisk
  ];
} ''
  IMAGE=$out/nixos-r4-image.img

  mkdir $out

  dd if=/dev/zero of=$IMAGE bs=1M count=4000
  sgdisk -o $IMAGE
  sgdisk -a 1 -n 1:34:8191     -A 1:set:2   -t 1:8300 -c 1:"bl2"        $IMAGE
	sgdisk -a 1 -n 2:8192:9215   -A 2:set:63	-t 2:8300 -c 2:"u-boot-env"	$IMAGE
	sgdisk -a 1 -n 3:9216:13311  -A 3:set:63	-t 3:8300 -c 3:"factory"    $IMAGE
	sgdisk -a 1 -n 4:13312:17407 -A 4:set:63	-t 4:8300 -c 4:"fip"        $IMAGE

  dd if=${armTrustedFirmwareBpiR4}/bl2.img of=$IMAGE seek=34 conv=notrunc,fsync
  dd if=${armTrustedFirmwareBpiR4}/fip.bin of=$IMAGE seek=13312 conv=notrunc,fsync
''
```

The nixpkgs `runCommand` function creates a derivation that runs a bash script.
This bash script is running into the nixpkgs standard environment, but we can override this environment with the set passed
as the second argument.
In this case, we add the `gptfdisk` package as a dependency to use `sgdisk` within our bash script.
The third argument is the script in itself.
We then create the derivation output directory, and generate a zero-filled 4 GB image with `dd`.
Then, we generate our partition table with `sgdisk` with the following partition map:
```
Block size of 512 bytes

Blocks 34 to 8191: BL2
Blocks 8192 to 9215: Space for U-Boot environment variables
Blocks 9216 to 13311: Factory
Blocks 13312 to 17407: FIP
```
Finally, we flash our TF-A build outputs into the matching partitions with `dd`.

We add this callPackage derivation into our `default.nix`:
```nix
{
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
}:
rec {
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
  armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
  image = pkgs.callPackage ./image { inherit armTrustedFirmwareBpiR4; };
}
```

And we can build our boot sequence components with `nix-shell -A image` to get our `nixos-r4-image.img`.

# Flashing and running

So let's flash the resulting image into an SD card.
```bash
dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress
```

After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot.
We get the following logs with `minicom`:
```
F0: 102B 0000
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 380E 5800
F5: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 0600 0041 [0000]
G0: 1190 0000
EC: 0000 0000 [3000]
MK: 0000 0000 [0000]
T0: 0000 0221 [0101]
Jump to BL

NOTICE:  BL2: v2.10.0   (release):
NOTICE:  BL2: Built : 00:00:00, Jan  1 1980
NOTICE:  WDT: Cold boot
NOTICE:  WDT: disabled
NOTICE:  CPU: MT7988
NOTICE:  EMI: Using DDR unknown settings
NOTICE:  EMI: Detected DRAM size: 4096 MB
NOTICE:  EMI: complex R/W mem test passed
NOTICE:  BL2: Booting BL31
NOTICE:  BL31: v2.10.0  (release):
NOTICE:  BL31: Built : 00:00:00, Jan  1 1980


U-Boot 2024.04 (Apr 02 2024 - 10:58:58 +0000)

CPU:   MediaTek MT7988
Model: mt7988-rfb
DRAM:  4 GiB
Core:  49 devices, 19 uclasses, devicetree: separate
MMC:   mmc@11230000: 0
Loading Environment from nowhere... OK
In:    serial@11000000
Out:   serial@11000000
Err:   serial@11000000
Net:
Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06
eth0: ethernet@15100000
BPI-R4>
```
And here we have our initial access to the U-Boot console !

# Sources

I successfully got to this point after myself reading some documentation and a bunch of blog posts.
Those where the resources I used:
- [frank-w's dokuwiki](https://www.fw-web.de/dokuwiki/doku.php?id=en:bpi-r4:start)
- [Kasper Kondzielski's blog post about running NixOS on the BPI-R3](https://github.com/ghostbuster91/blogposts/blob/main/router2023/main.md)
- [Weijie Gao's post on the Banana Pi Discourse about U-Boot and TF-A on Mediatek SoCs](ihttps://forum.banana-pi.org/t/tutorial-build-customize-and-use-mediatek-open-source-u-boot-and-atf/13785)
- [*Understanding ARM Trusted Firmware using QEMU* by Hemanth Nandish](https://lnxblog.github.io/2020/08/20/qemu-arm-tf.html)