20 KiB
+++ title = 'Running NixOS on the Banana Pi R4: Building the boot sequence' date = 2024-09-29T08:37:23+02:00 draft = true +++
Trigger Warnings
- I'm by no mean an expert about U-Boot, neither in ARM CPUs. This is just me trying to accomplish a goal, and sharing what I think I have learned. As such, I may (and certainly will) utter false statements. If you see that I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it !
- I can't say that I'm fluent in English. If you see some grammar or spelling mistakes, please reach me too !
- This post's goal is to be exhaustive. My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material.
Thanks for your comprehension, and have a great time reading this.
How did we get there
It has been a while since I started waiting for the Banana Pi R4. I bought it as soon as a full bundle with the WiFi 7 module was available. The goal initially was to run OpenWRT on it, without bothering too much. But then I tought that if all my machines (including my ROCKPro64) are running NixOS with my common flake configuration, there is no reason why my router should not. This is because I use a colmena-based deployment system. It allows me to build (and cross-compile) all my machines configurations and to deploy them from my PC.
However, there is two caveats here:
- The Nix store usually takes some place on storage devices, so the embedded 8GB eMMC might not be enough. But, as the BPI R4 has an integrated slot for a NVME SSD, and since I had an empty 500 GB SSD available, it is way more than enough.
- There is currently no support or available image of NixOS for the BPI R4.
This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4.
For your information, this is what I had before starting this project:
- Basic Nix knowledge.
- I already used U-Boot on some boards with prebuilt images, but never really installed it or tweaked it myself.
- Some knowledge about basic Linux utilities used for embedded.
- A USB-Serial cable.
I saw that for the image generation, some people chose to build an image with their Nix configuration already applied to it, and then just flash this image to their board flash device. In my case, I would like to stick with an image that just serve as an install media, and then manually do all the installation process as I would do on a PC.
As much as possible, I would like to stick with mainline U-Boot and Linux kernel. I will try to grab any useful patches from R4 specific forks and apply them manually.
Today's post is the first part about building our boot components and getting access to the u-Boot console.
Let's get started !
Initial checks
The BPI-R4 should be shipped with an existing install of OpenWRT on the NAND flash.
If we set the boot-device selection jumper to NAND Flash and connect our USB-Serial cable, we should see boot logs.
So after doing this and starting minicom, I got the following:
ubi0: attaching mtd6
ubi0: scanning is finished
ubi0: attached mtd6 (name "ubi", size 112 MiB)
ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
ubi0: good PEBs: 900, bad PEBs: 0, corrupted PEBs: 0
ubi0: user volume: 3, internal volumes: 1, max. volumes count: 128
ubi0: max/mean erase counter: 2/0, WL threshold: 4096, image sequence number: 1697192243
ubi0: available PEBs: 0, total reserved PEBs: 900, PEBs reserved for bad PEB handling: 19
No size specified -> Using max size (4190208)
Read 4190208 bytes from volume kernel to 0000000046000000
## Loading kernel from FIT Image at 46000000 ...
Using 'config-1' configuration
Trying 'kernel-1' kernel subimage
Description: ARM64 OpenWrt Linux-5.4.246
Type: Kernel Image
Compression: lzma compressed
Data Start: 0x460000e8
Data Size: 4061787 Bytes = 3.9 MiB
Architecture: AArch64
OS: Linux
Load Address: 0x48080000
Entry Point: 0x48080000
Hash algo: crc32
Hash value: ef1d4754
Hash algo: sha1
Hash value: 78589b5e23500626a36c8a2ff523eb8bbcaddabc
Verifying Hash Integrity ... crc32+ sha1+ OK
## Loading fdt from FIT Image at 46000000 ...
Using 'config-1' configuration
Trying 'fdt-1' fdt subimage
Description: ARM64 OpenWrt BPI-R4-NAND device tree blob
Type: Flat Device Tree
Compression: uncompressed
Data Start: 0x463dfc80
Data Size: 34487 Bytes = 33.7 KiB
Architecture: AArch64
Hash algo: crc32
Hash value: 0c548bfa
Hash algo: sha1
Hash value: 0fb7882fc5ef9acba18b6864234dd1aec0b3d2f2
Verifying Hash Integrity ... crc32+ sha1+ OK
Booting using the fdt blob at 0x463dfc80
Working FDT set to 463dfc80
Uncompressing Kernel Image
Loading Device Tree to 00000000ff7ec000, end 00000000ff7f76b6 ... OK
Working FDT set to ff7ec000
Add 'ramoops@42ff0000' node failed: FDT_ERR_EXISTS
Starting kernel ...
Okay, the board is working as expected and theses logs may serve as a reference later. I also quickly checked the hardware revision of my R4. In the first revisions, a resistor on the PCB caused I2C devices to be unusable when using an NVME SSD.0 In my case, the resistor was no longer there, so I should not face any issue.
The ARMv8 boot sequence
Before focusing on the Linux kernel or NixOS, we will have to build the boot sequence for our R4. In our case, this includes ARM TrustedFirmware-A and U-Boot.
On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that will be run to start our board) is standardized with the following elements:
- BL1: this is a software brick that is already present in the board Boot ROM. Its goals are to perform basic hardware initialization and then pass the control to the next step (BL2). To do so, it expects to find the next step at a specific address in RAM.
- BL2: another software brick whose goal is to create a secure environment for the software that will be run, and then pass the control to the last step.
- BL3: the final element of the boot sequence. It is basically what the board will finally run. In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software), an optionnal BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run.
It seems in reality a bit more complicated than that, dealing with ARM Exception Level, but we will stick to this simple representation.
ARM TrustedFirmware-A provides a standard implementation for BL2 and BL31. Its documentation is available here. U-Boot is the bootloader which is the most commonly used on ARM boards, and that correspond here to BL33. It has all the necessary utilities to load and boot the Linux kernel.
This is all the two elements that we will have to build and add to our image to then be able to boot NixOS.
Nixpkgs provide two functions to build these two elements as Nix derivations: buildUBoot and buildArmTrustedFirmware.
This make the creation of such derivations fairly easy.
Cross compilation on NixOS (for Nix-beginners)
If you're not on an aarch64 machine (or did not enable native build through QEMU on your configuration),
you won't be able to build the following derivations.
To do so, we have to setup cross-compilation.
Everybody knows Nixpkgs as a collection of Nix derivations that it contains some utility functions like mkDerivation.
Nixpkgs also provides a cross-compilation system that allows to cross compile each derivation for a compatible architecture !
For example, if we want to build any aarch64 package from any architecture (like GNU hello), we can just do:
nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs/lib>).systems.examples.aarch64-multiplatform' -A hello
This will cross-compile hello for aarch64 in the Nix store, and set a symlink to it in a result directory in the current one.
It is possible, because as any package in Nixpkgs, GNU hello is declared through a Nix recipe (called a callPackage derivation) that
is called by the callPackage function of Nixpkgs.
This function setup a bunch of things, and among them cross-compilation.
It does so by looking at the crossSystem parameters provided to Nixpkgs when importing it.
It will then cross-compile every dependency of the derivation before finally building our package.
So here, to cross-compile each of our program we need, we just have to create the following default.nix file:
{
pkgs ? import <nixpkgs> {
crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
},
}:
{
myProgram = pkgs.callPackage ./myProgram { };
}
Here, we just need to have a myProgram directory with a default.nix file containing a callPackage derivation
for our program, and then call nix-build -A myProgram.
Building U-Boot
We should be ready to build a mainline U-Boot. However, the latter does not support the BPI R4 for now. Most of current Linux images for the BPI R4 uses frank-w's U-Boot fork. His modifications to bring support for our device are just few commits that I'll export as patch, and apply them on top of mainline U-Boot.
Let's create the following default.nix file:
{
pkgs ? import <nixpkgs> {
crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
},
}:
{
ubootBpiR4 = pkgs.callPackage ./u-boot { };
}
In a u-boot directory, we will create a default.nix file and a patches directory that will contains
all the patches that I took from frank-w's U-Boot.
Within the u-boot/default.nix file, we will just have to write the following callPackage derivation:
{ buildUBoot, ... }:
buildUBoot {
defconfig = "mt7988a_bpir4_sd_defconfig";
extraMeta.platforms = [ "aarch64-linux" ];
extraPatches = [
./patches/0001-pci-mediatek-add-PCIe-controller-support-for-Filogic.patch
./patches/0002-Fix-PCIE-on-BPIR4.patch
./patches/0003-arm-dts-enable-pcie-in-sd-dts-too.patch
./patches/0004-dts-r4-disable-pcie2-in-emmc-dts.patch
./patches/0005-defconfig-uEnv-add-defconfigs-and-environment-files.patch
./patches/0006-defconfig-r4-update-with-pcie-options.patch
./patches/0007-defconfig-r4-add-pstore.patch
./patches/0008-defconfig-r4-update-emmc-defconfig.patch
./patches/0009-defconfig-r4-fix-duplicates-in-emmc-defconfig.patch
./patches/0010-arm64-dts-move-pcie-phy-to-dedicated-xsphy-no-driver.patch
./patches/0011-pci-mediatek-print-controller-address-for-card-detec.patch
];
extraConfig = ''
CONFIG_FIT=n
CONFIG_USE_DEFAULT_ENV_FILE=n
'';
filesToInstall = [ "u-boot.bin" ];
}
As I said previously, Nixpkgs provides a buildUBoot function, to which we have just to path some args.
Here, frank-w's U-Boot patches define a new defconfig called mt7988a_bpi_r4_sd_defconfig for the R4.
This config enable by default the use of a FIT (Flattened Image Tree).
As far as I understood, this allows the creation of a blob containing all the necessary boot info that U-Boot needs instead of loading a couple of files.
In my case, at least for the beginning, I chose to stick with the plain-old U-Boot builds to learn to do it myself instead of using
any special packaging format.
In the end, the filesToInstall arguments just tells which final build products we want to get in the derivation output.
For now, we will only keep the u-boot binary and see if we actually need anything else later.
That's it for U-Boot ! We can try to build it with nix-build -A ubootBpiR4 and see the u-boot.bin file
appear in the Nix store, and in the result folder !
ARM TrustedFirmware-A
As with U-Boot, the mainline version of TrustedFirmware does not support the BPI-R4.
For now, distributions are using this fork.
I don't really know who for sure who is behind this account, but there is a bit too much commits for me
to export them as patch.
So even if I told that I would like to stick with mainline, for now we will still using this fork directly.
Once again, we will use the Nixpkgs's buildArmTrustedFirmware function.
Before that, and as we can see on the frank-w's U-Boot build scripts,
we will need to pass to U-Boot binary. It should be included within the final fip.bin file.
To be honest there, I don't really know if TrustedFirmware is mandatory (and I guess it's not) to boot Linux on our SoC, or if U-Boot by itself could have made everything by itself. I don't even know the details of what is an FIP image. But for now, I decided to proceed like the OpenWRT and Debian images. I will surely dig in the TrustedFirmare and ARM documentations later. I'll eventually to a follow-up post later if I achieve to figure out how everything is working.
So, let's create our callPackage derivation in trusted-firmware/default.nix:
{
buildArmTrustedFirmware,
fetchFromGitHub,
dtc,
ubootBpiR4,
ubootTools,
openssl,
...
}:
(buildArmTrustedFirmware rec {
platform = "mt7988";
extraMeta.platforms = [ "aarch64-linux" ];
extraMakeFlags = [
"USE_MKIMAGE=1"
"BOOT_DEVICE=sdmmc"
"DRAM_USE_COMB=1"
"BL33=${ubootBpiR4}/u-boot.bin"
"all"
"fip"
];
filesToInstall = [
"build/${platform}/release/bl2.img"
"build/${platform}/release/fip.bin"
];
}).overrideAttrs (old: {
src = fetchFromGitHub {
owner = "mtk-openwrt";
repo = "arm-trusted-firmware";
rev = "bacca82a8cac369470df052a9d801a0ceb9b74ca";
hash = "sha256-n5D3styntdoKpVH+vpAfDkCciRJjCZf9ivrI9eEdyqw=";
};
version = "2.10.0-mtk";
nativeBuildInputs = old.nativeBuildInputs ++ [ dtc ubootTools openssl ];
}
This looks like what we have done with U-Boot, we select our platform, then pass the necessary flags for compilation,
and then grab the two output files that we need.
However here, there is some extra steps.
Both buildUBoot and buildArmTrustedFirmware assumes that you're building the mainline U-Boot and TF-A.
It was the case with our U-Boot build to which we just add some extra patches.
However, with TF-A, as I said, we will here use the mtk-openwrt's fork.
So we need to override the derivation produced by buildArmTrustedFirmware to pass our own sources and needed dependencies thanks
to the overrideAttrs function.
As we can see, this callPackage derivation needs to access our previous ubootBpiR4 derivation.
All the other derivations arguments are provided by nixpkgs itself through the callPackage function.
However, ubootBpiR4 is itself not present within the nixpkgs we use, we have to manually pass it.
There is two solutions here, the first one is to use an overlay to manually add it to our nixpkgs.
But there is a far further solution, which is to pass it as an argument of the callPackage function.
This make our default.nix file looks like that:
{
pkgs ? import <nixpkgs> {
crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
},
}:
rec {
ubootBpiR4 = pkgs.callPackage ./u-boot { };
armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
}
Now, we can run nix-build -A armTrustedFirmwareBpiR4 to build everything we need.
Side note on the U-Boot and TF-A combination
If you check how to build the full boot sequence binaries for various board, you will be feel like me
that this seems to be highly dependant on SoC manufacturer and the SoC itself.
As we have seen there, we build U-Boot first and then pass it to TF-A as BL33.
For some Allwinner SoCs, TF-A is built first and then passed to U-boot as BL31.
For Rockchip based boardsm there is this whole idbloader.img due to Rockchip's miniloader.
At this point, I don't really know if there is truly a "standard way" to build to whole boot sequence for each SoC.
I don't even know what causes this difference in the build process between manufacturers.
I guess this is related to manufacturers' implementation, or things that could be done both in U-Boot and in TF-A
and then depends about where the manufacturer implemented them ?
Create our boot image
The final step is just to create an empty image and flash what we have built so far at the corrects address ranges.
This will be done with a simple bash script derivation in the image/default.nix file.
I will just dump the callPackage derivation here, and explain everything.
{
runCommand,
armTrustedFirmwareBpiR4,
gptfdisk,
...
}:
runCommand "bpi-r4-image" {
nativeBuildInputs = [
gptfdisk
];
} ''
IMAGE=$out/nixos-r4-image.img
mkdir $out
dd if=/dev/zero of=$IMAGE bs=1M count=4000
sgdisk -o $IMAGE
sgdisk -a 1 -n 1:34:8191 -A 1:set:2 -t 1:8300 -c 1:"bl2" $IMAGE
sgdisk -a 1 -n 2:8192:9215 -A 2:set:63 -t 2:8300 -c 2:"u-boot-env" $IMAGE
sgdisk -a 1 -n 3:9216:13311 -A 3:set:63 -t 3:8300 -c 3:"factory" $IMAGE
sgdisk -a 1 -n 4:13312:17407 -A 4:set:63 -t 4:8300 -c 4:"fip" $IMAGE
dd if=${armTrustedFirmwareBpiR4}/bl2.img of=$IMAGE seek=34 conv=notrunc,fsync
dd if=${armTrustedFirmwareBpiR4}/fip.bin of=$IMAGE seek=13312 conv=notrunc,fsync
''
The nixpkgs's runCommand function create a derivation that runs a bash script.
This bash script is run into the Nixpkgs stdenv, but we can override this environment with the set passed
as its second arguments.
In this case, we add the gptfdisk package as a dependency to use sgdisk within our bash script.
The third argument is the script in itself.
We then create the derivation output directory, and generate a zero-filled 4GB image with dd.
Then, we create our partitions with sgdisk with the following partition map:
Block size of 512 bytes
Blocks 34 to 8191: BL2
Blocks 8192 to 9215: Future space for U-Boot environment variables
Blocks 9216 to 13311: Factory
Blocks 13312 to 17407: FIP
Finally, we flash our TF-A build outputs into the matching partitions with dd.
We add this callPackage derivation into our default.nix:
{
pkgs ? import <nixpkgs> {
crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
},
}:
rec {
ubootBpiR4 = pkgs.callPackage ./u-boot { };
armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
image = pkgs.callPackage ./image { inherit armTrustedFirmwareBpiR4; };
}
And we can build our boot sequence components with nix-shell -A image to get our nixos-r4-image.img.
Flashing and running
So let's flash the resulting image into an SD card.
dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress
After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot.
We get the following logs with minicom:
F0: 102B 0000
FA: 1042 0000
FA: 1042 0000 [0200]
F9: 1041 0000
F3: 1001 0000 [0200]
F3: 1001 0000
F6: 380E 5800
F5: 0000 0000
V0: 0000 0000 [0001]
00: 0000 0000
BP: 0600 0041 [0000]
G0: 1190 0000
EC: 0000 0000 [3000]
MK: 0000 0000 [0000]
T0: 0000 0221 [0101]
Jump to BL
NOTICE: BL2: v2.10.0 (release):
NOTICE: BL2: Built : 00:00:00, Jan 1 1980
NOTICE: WDT: Cold boot
NOTICE: WDT: disabled
NOTICE: CPU: MT7988
NOTICE: EMI: Using DDR unknown settings
NOTICE: EMI: Detected DRAM size: 4096 MB
NOTICE: EMI: complex R/W mem test passed
NOTICE: BL2: Booting BL31
NOTICE: BL31: v2.10.0 (release):
NOTICE: BL31: Built : 00:00:00, Jan 1 1980
U-Boot 2024.04 (Apr 02 2024 - 10:58:58 +0000)
CPU: MediaTek MT7988
Model: mt7988-rfb
DRAM: 4 GiB
Core: 49 devices, 19 uclasses, devicetree: separate
MMC: mmc@11230000: 0
Loading Environment from nowhere... OK
In: serial@11000000
Out: serial@11000000
Err: serial@11000000
Net:
Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06
eth0: ethernet@15100000
BPI-R4>
And here we have our initial access to the U-Boot console !
Sources
I succesfully got to this point after myself reading some documentation and a bunch of blog posts. Those where the resources I used: