2024-09-29 21:12:12 +02:00
+++
title = 'Running NixOS on the Banana Pi R4: Building the boot sequence'
date = 2024-10-05T08:37:23+02:00
+++
# Before starting
- I'm by no mean an expert about U-Boot, neither in ARM CPUs.
This is just me trying to port NixOS on the Banana Pi R4, and sharing what I learned.
As such, I may (and certainly will) utter false statements.
If I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it !
- I can't say really that I'm fluent in English. If you see some grammar or spelling mistakes, please reach me too !
- This post's goal is not to be exhaustive, but I want to emphasize what I think is important or useful to know.
My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material.
Thanks for your comprehension, and have a great time reading this.
# How did we get there
It has been a while since I started waiting for the Banana Pi R4.
I thought of everything I could do with my own Linux gateway instead of being forced to get along my ISP's.
As soon as the full bundle with the Wi-Fi 7 card was available and the first reviews appeared, my command was already confirmed.
I've been waiting it for nearly a year, and initially though that I would just run OpenWrt on it.
I mean, It is for sure the easiest solution (even if it would without doubt need some tweaks).
However, after a quick thought, I felt like this router would fit even better in my NixOS environment.
I mean, I have a centralized NixOS configuration for all my machines that can be easily built from any of them and be deployed on another
with a single command.
It would make even more sense too cross-compile my router configuration from my PC with its Ryzen 7 5800X and deploy it on the BPI-R4.
However, there are three caveats with this choice:
- The Nix store usually takes some place on storage devices, so the embedded 8
But, since the BPI R4 has an integrated slot for a NVMe SSD and I have an empty 500 GB SSD available,
it is way more than enough.
- I would say (I might be wrong there), that NixOS is not really the best fit for such small home gateways.
Some OpenWrt packages are missing on NixOS, with the most obvious example being LuCI.
I don't think that such packages are mandatory though, they would mainly bring QoL features.
- There is currently no support or available image of NixOS for the BPI R4.
But I just felt that I would still be more comfortable with another machine in my NixOS environment than
with one outside of it that I'll have to manage its own way.
This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4.
For your information, this is what I had before starting this project:
- Some basic Nix knowledge.
- I already used U-Boot on some boards with prebuilt Linux images, but never really installed it or tweaked it myself.
- Some knowledge about basic Linux utilities used for embedded.
- A USB-Serial cable.
- A Banana Pi R4 (of course).
For a NixOS SD image generation on ARM machines, some people prefer to build it with their Nix configuration already
applied to it, and then just flash into their board.
In my case, I prefer the philosophy of having an image that just serve as an installation media, and then manually do
all the installation process as I would do on a PC.
Today's post is the first part about building our boot components and getting access to the U-Boot console.
Let's get started !
# The ARMv8 boot sequence
Before focusing on the Linux kernel or NixOS, we'll have to build the initial boot sequence for the R4.
It would be too easy to just install our bootloader and having an UEFI running it directly **right** ?
The boot phase in ARM-based board is often a bit trickier as it is on x86 hosts.
On our case, we need a firmware setting up the ARM secure environment (which is ARM TrustedFirmware-A)
and a bootloader (U-Boot).
On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that is run to init our board)
is standardized with the following elements:
- BL1: this is a software brick that is already present in the board's Boot ROM. It performs basic
hardware initialization and then pass the control to the next step (BL2).
- BL2: Another software brick whose goal is to create a secure environment for the software that will be run, and then
pass the control to the last step.
- BL3: the final element of the boot sequence. It is basically what the board will finally run.
In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software),
an optional BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run.
It's in fact a bit more complicated than that, but we'll keep this simple representation.
ARM provides a standard and open implementation for BL2, BL31 and BL32 called TrustedFirmware-A.
Its documentation is available [here ](https://trustedfirmware-a.readthedocs.io/en/latest/index.html ).
Finally, our BL33 will be U-Boot, a bootloader widely used for ARM SoCs.
Once the board reach U-Boot, it will be able to boot NixOS.
Luckily, nixpkgs provides two functions to build these two elements as Nix derivations: `buildUBoot` and `buildArmTrustedFirmware` .
# Cross compilation on NixOS (for Nix-beginners)
As I plan to build everything from a x86_64 host, we have to set up a cross-compilation system to build our boot sequence
for ARMv8A.
Once again, nixpkgs make it fairly easy with its embedded cross-compilation system.
For example, if we want to build any aarch64 package from any architecture (like `hello` ), we can just run:
```bash
nix-build '< nixpkgs > ' --arg crossSystem '(import < nixpkgs / lib > ).systems.examples.aarch64-multiplatform' -A hello
```
It is possible, because as any package in nixpkgs, `hello` is declared through a Nix recipe (a callPackage derivation) that
is called by the `callPackage` function.
This function setup a bunch of things and among them cross-compilation by looking at the `crossSystem` parameter provided to nixpkgs.
To cross-compile every program we need, we just have to create the following `default.nix` file:
```nix
{
pkgs ? import < nixpkgs > {
crossSystem = (import < nixpkgs / lib > ).systems.examples.aarch64-multiplatform;
},
}:
{
myProgram = pkgs.callPackage ./myProgram { };
}
```
Here, we just need to have a `myProgram` directory with a `default.nix` file containing a callPackage derivation
for our program, and then call `nix-build -A myProgram` .
# Building U-Boot
At the time I'm writing this post,
most of current Linux images for the BPI R4 uses [frank-w's U-Boot fork ](https://github.com/frank-w/u-boot ).
This is because the BPI-R4 is not supported by mainline U-boot.
The modifications frank-w did to bring this support are just few commits.
We can easily export them as patch and apply them on top of mainline U-Boot.
Let's create the following `default.nix` file:
```nix
{
pkgs ? import < nixpkgs > {
crossSystem = (import < nixpkgs / lib > ).systems.examples.aarch64-multiplatform;
},
}:
{
ubootBpiR4 = pkgs.callPackage ./u-boot { };
}
```
In a `u-boot` directory, we will create a `default.nix` file and a `patches` directory that will contain
all the patches that I grabbed from frank-w's U-Boot.
Within the `u-boot/default.nix` file, we will just have to write the following callPackage derivation:
```nix
{ buildUBoot, ... }:
buildUBoot {
defconfig = "mt7988a_bpir4_sd_defconfig";
extraMeta.platforms = [ "aarch64-linux" ];
extraPatches = [
./patches/0001-pci-mediatek-add-PCIe-controller-support-for-Filogic.patch
./patches/0002-Fix-PCIE-on-BPIR4.patch
./patches/0003-arm-dts-enable-pcie-in-sd-dts-too.patch
./patches/0004-dts-r4-disable-pcie2-in-emmc-dts.patch
./patches/0005-defconfig-uEnv-add-defconfigs-and-environment-files.patch
./patches/0006-defconfig-r4-update-with-pcie-options.patch
./patches/0007-defconfig-r4-add-pstore.patch
./patches/0008-defconfig-r4-update-emmc-defconfig.patch
./patches/0009-defconfig-r4-fix-duplicates-in-emmc-defconfig.patch
./patches/0010-arm64-dts-move-pcie-phy-to-dedicated-xsphy-no-driver.patch
./patches/0011-pci-mediatek-print-controller-address-for-card-detec.patch
];
extraConfig = ''
CONFIG_FIT=n
CONFIG_USE_DEFAULT_ENV_FILE=n
'';
filesToInstall = [ "u-boot.bin" ];
}
```
The `buildUboot` function will actually grab the source from the U-Boot repo and apply each patch
we give it through the `extraPatches` argument.
It will then copy the `defconfig` file from U-Boot config folder, and apply the `extraConfig` content on top.
It will eventually compile it and grab the files provided through `filesToInstall` from the build artifacts
and put it into the Nix store.
# ARM TrustedFirmware-A
Like U-Boot, the mainline version of TrustedFirmware does not currently support the BPI-R4.
For now, Linux distributions are using [this fork ](https://github.com/mtk-openwrt/arm-trusted-firmware ).
I don't really know for sure who is behind this account, and if I should really trust it.
Nonetheless, there is a bit too many commits for me to manage manually, so we'll here just grab this fork's sources.
As I told earlier, we'll use the nixpkgs's `buildArmTrustedFirmware` function.
Before that, and as we can see on the [frank-w's U-Boot build scripts ](https://github.com/frank-w/u-boot/blob/mtk-atf/build.sh#L40 ),
we've to pass the U-Boot binary as input.
It'll be included within the final `fip.bin` file as BL33.
______________________________________________________________________
To be honest there, I don't really know if TrustedFirmware is mandatory to start our board.
Maybe U-Boot could've been enough, at the expense of not setting up the ARM secure environment.
Maybe U-Boot could itself set up this secure environment.
But for now, I decided to proceed like the OpenWrt and Debian images.
To confirm, I'll surely dig in the TrustedFirmware and ARM documentations later.
I'll eventually do a follow-up post later if I achieve to figure out how everything is working.
______________________________________________________________________
So, let's create our callPackage derivation in `trusted-firmware/default.nix` :
```nix
{
buildArmTrustedFirmware,
fetchFromGitHub,
dtc,
ubootBpiR4,
ubootTools,
openssl,
...
}:
(buildArmTrustedFirmware rec {
platform = "mt7988";
extraMeta.platforms = [ "aarch64-linux" ];
extraMakeFlags = [
"USE_MKIMAGE=1"
"BOOT_DEVICE=sdmmc"
"DRAM_USE_COMB=1"
"BL33=${ubootBpiR4}/u-boot.bin"
"all"
"fip"
];
filesToInstall = [
"build/${platform}/release/bl2.img"
"build/${platform}/release/fip.bin"
];
}).overrideAttrs (old: {
src = fetchFromGitHub {
owner = "mtk-openwrt";
repo = "arm-trusted-firmware";
rev = "bacca82a8cac369470df052a9d801a0ceb9b74ca";
hash = "sha256-n5D3styntdoKpVH+vpAfDkCciRJjCZf9ivrI9eEdyqw=";
};
version = "2.10.0-mtk";
nativeBuildInputs = old.nativeBuildInputs ++ [ dtc ubootTools openssl ];
}
```
This is very similar what we have done with U-Boot, we select our platform, then pass the necessary flags for compilation,
and then grab the two output files that we need.
However, in our situation, there are some extra steps.
Both `buildUBoot` and `buildArmTrustedFirmware` assumes that you're building the mainline U-Boot and TF-A.
It was the case with our U-Boot build to which, but not for our TF-A.
So we need to override the derivation produced by `buildArmTrustedFirmware` to set the `src` argument ourselves
and pass supplementary build dependencies in `nativeBuildInputs` .
As we can see, this callPackage derivation needs to access our previous `ubootBpiR4` derivation.
All the other derivations arguments are provided by nixpkgs itself through the `callPackage` function,
but not `ubootBpiR4` that we declared ourselves.
Given that, we can just easily pass it as is in the `callPackage` argument.
This makes our `default.nix` file looks like the following:
```nix
{
pkgs ? import < nixpkgs > {
crossSystem = (import < nixpkgs / lib > ).systems.examples.aarch64-multiplatform;
},
}:
rec {
ubootBpiR4 = pkgs.callPackage ./u-boot { };
armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
}
```
Now, we can run `nix-build -A armTrustedFirmwareBpiR4` to build everything we need !
# Create our boot image
The final step is just to create an empty image and flash what we have built so far at the matching address ranges.
This will be done with a simple bash script derivation in the `image/default.nix` file.
```nix
{
runCommand,
armTrustedFirmwareBpiR4,
gptfdisk,
...
}:
runCommand "bpi-r4-image" {
nativeBuildInputs = [
gptfdisk
];
} ''
IMAGE=$out/nixos-r4-image.img
mkdir $out
dd if=/dev/zero of=$IMAGE bs=1M count=4000
sgdisk -o $IMAGE
sgdisk -a 1 -n 1:34:8191 -A 1:set:2 -t 1:8300 -c 1:"bl2" $IMAGE
2024-10-05 19:47:28 +02:00
sgdisk -a 1 -n 2:8192:9215 -A 2:set:63 -t 2:8300 -c 2:"u-boot-env" $IMAGE
sgdisk -a 1 -n 3:9216:13311 -A 3:set:63 -t 3:8300 -c 3:"factory" $IMAGE
sgdisk -a 1 -n 4:13312:17407 -A 4:set:63 -t 4:8300 -c 4:"fip" $IMAGE
2024-09-29 21:12:12 +02:00
dd if=${armTrustedFirmwareBpiR4}/bl2.img of=$IMAGE seek=34 conv=notrunc,fsync
dd if=${armTrustedFirmwareBpiR4}/fip.bin of=$IMAGE seek=13312 conv=notrunc,fsync
''
```
The nixpkgs `runCommand` function creates a derivation that runs a bash script into nixpkgs's standard environment.
We can add dependencies with the second `runCommand` argument.
In this case, we'll add the `gptfdisk` package as a dependency to use `sgdisk` within our script.
Finally, the third argument is the script in itself.
For the script in itself, we'll create the derivation output directory and generate a zero-filled 4 `dd` .
Then, we want to generate our partition table with `sgdisk` with the following partition map:
```
Block size of 512 bytes
Blocks 34 to 8191: BL2
Blocks 8192 to 9215: Space for U-Boot environment variables
Blocks 9216 to 13311: Factory
Blocks 13312 to 17407: FIP
```
Finally, we flash our TF-A build outputs into the matching partitions with `dd` .
We add this callPackage derivation into our `default.nix` :
```nix
{
pkgs ? import < nixpkgs > {
crossSystem = (import < nixpkgs / lib > ).systems.examples.aarch64-multiplatform;
},
}:
rec {
ubootBpiR4 = pkgs.callPackage ./u-boot { };
armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
image = pkgs.callPackage ./image { inherit armTrustedFirmwareBpiR4; };
}
```
And we can build our boot sequence components with `nix-shell -A image` to get our `nixos-r4-image.img` .
# Flashing and running
The only thing left to do is flashing our image into a SD card.
```bash
dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress
```
After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot.
We get the following logs with `minicom` :
```
F0: 102B 0000
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 380E 5800
F5: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 0600 0041 [0000]
G0: 1190 0000
EC: 0000 0000 [3000]
MK: 0000 0000 [0000]
T0: 0000 0221 [0101]
Jump to BL
NOTICE: BL2: v2.10.0 (release):
NOTICE: BL2: Built : 00:00:00, Jan 1 1980
NOTICE: WDT: Cold boot
NOTICE: WDT: disabled
NOTICE: CPU: MT7988
NOTICE: EMI: Using DDR unknown settings
NOTICE: EMI: Detected DRAM size: 4096 MB
NOTICE: EMI: complex R/W mem test passed
NOTICE: BL2: Booting BL31
NOTICE: BL31: v2.10.0 (release):
NOTICE: BL31: Built : 00:00:00, Jan 1 1980
U-Boot 2024.04 (Apr 02 2024 - 10:58:58 +0000)
CPU: MediaTek MT7988
Model: mt7988-rfb
DRAM: 4 GiB
Core: 49 devices, 19 uclasses, devicetree: separate
MMC: mmc@11230000: 0
Loading Environment from nowhere... OK
In: serial@11000000
Out: serial@11000000
Err: serial@11000000
Net:
Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06
eth0: ethernet@15100000
BPI-R4>
```
And here we have our initial access to the U-Boot console !
# Sources
I successfully got to this point after myself reading some documentation and a bunch of blog posts.
Those where the resources I used:
- [frank-w's dokuwiki ](https://www.fw-web.de/dokuwiki/doku.php?id=en:bpi-r4:start )
- [Kasper Kondzielski's blog post about running NixOS on the BPI-R3 ](https://github.com/ghostbuster91/blogposts/blob/main/router2023/main.md )
- [Weijie Gao's post on the Banana Pi Discourse about U-Boot and TF-A on Mediatek SoCs ](ihttps://forum.banana-pi.org/t/tutorial-build-customize-and-use-mediatek-open-source-u-boot-and-atf/13785 )
- [*Understanding ARM Trusted Firmware using QEMU* by Hemanth Nandish ](https://lnxblog.github.io/2020/08/20/qemu-arm-tf.html )
