408 lines
16 KiB
Markdown
408 lines
16 KiB
Markdown
+++
|
||
title = 'Running NixOS on the Banana Pi R4: Building the boot sequence'
|
||
date = 2024-10-05T08:37:23+02:00
|
||
+++
|
||
|
||
# Before starting
|
||
|
||
- I'm by no mean an expert about U-Boot, neither in ARM CPUs.
|
||
This is just me trying to port NixOS on the Banana Pi R4, and sharing what I learned.
|
||
As such, I may (and certainly will) utter false statements.
|
||
If I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it !
|
||
- I can't say really that I'm fluent in English. If you see some grammar or spelling mistakes, please reach me too !
|
||
- This post's goal is not to be exhaustive, but I want to emphasize what I think is important or useful to know.
|
||
My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material.
|
||
|
||
Thanks for your comprehension, and have a great time reading this.
|
||
|
||
# How did we get there
|
||
|
||
It has been a while since I started waiting for the Banana Pi R4.
|
||
I thought of everything I could do with my own Linux gateway instead of being forced to get along my ISP's.
|
||
As soon as the full bundle with the Wi-Fi 7 card was available and the first reviews appeared, my command was already confirmed.
|
||
I've been waiting it for nearly a year, and initially though that I would just run OpenWrt on it.
|
||
I mean, It is for sure the easiest solution (even if it would without doubt need some tweaks).
|
||
However, after a quick thought, I felt like this router would fit even better in my NixOS environment.
|
||
I mean, I have a centralized NixOS configuration for all my machines that can be easily built from any of them and be deployed on another
|
||
with a single command.
|
||
It would make even more sense too cross-compile my router configuration from my PC with its Ryzen 7 5800X and deploy it on the BPI-R4.
|
||
|
||
However, there are three caveats with this choice:
|
||
- The Nix store usually takes some place on storage devices, so the embedded 8 GB eMMC might not be enough.
|
||
But, since the BPI R4 has an integrated slot for a NVMe SSD and I have an empty 500 GB SSD available,
|
||
it is way more than enough.
|
||
- I would say (I might be wrong there), that NixOS is not really the best fit for such small home gateways.
|
||
Some OpenWrt packages are missing on NixOS, with the most obvious example being LuCI.
|
||
I don't think that such packages are mandatory though, they would mainly bring QoL features.
|
||
- There is currently no support or available image of NixOS for the BPI R4.
|
||
|
||
But I just felt that I would still be more comfortable with another machine in my NixOS environment than
|
||
with one outside of it that I'll have to manage its own way.
|
||
This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4.
|
||
|
||
For your information, this is what I had before starting this project:
|
||
|
||
- Some basic Nix knowledge.
|
||
- I already used U-Boot on some boards with prebuilt Linux images, but never really installed it or tweaked it myself.
|
||
- Some knowledge about basic Linux utilities used for embedded.
|
||
- A USB-Serial cable.
|
||
- A Banana Pi R4 (of course).
|
||
|
||
For a NixOS SD image generation on ARM machines, some people prefer to build it with their Nix configuration already
|
||
applied to it, and then just flash into their board.
|
||
In my case, I prefer the philosophy of having an image that just serve as an installation media, and then manually do
|
||
all the installation process as I would do on a PC.
|
||
|
||
Today's post is the first part about building our boot components and getting access to the U-Boot console.
|
||
|
||
Let's get started !
|
||
|
||
# The ARMv8 boot sequence
|
||
|
||
Before focusing on the Linux kernel or NixOS, we'll have to build the initial boot sequence for the R4.
|
||
It would be too easy to just install our bootloader and having an UEFI running it directly **right** ?
|
||
The boot phase in ARM-based board is often a bit trickier as it is on x86 hosts.
|
||
On our case, we need a firmware setting up the ARM secure environment (which is ARM TrustedFirmware-A)
|
||
and a bootloader (U-Boot).
|
||
|
||
On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that is run to init our board)
|
||
is standardized with the following elements:
|
||
|
||
- BL1: this is a software brick that is already present in the board's Boot ROM. It performs basic
|
||
hardware initialization and then pass the control to the next step (BL2).
|
||
- BL2: Another software brick whose goal is to create a secure environment for the software that will be run, and then
|
||
pass the control to the last step.
|
||
- BL3: the final element of the boot sequence. It is basically what the board will finally run.
|
||
In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software),
|
||
an optional BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run.
|
||
|
||
It's in fact a bit more complicated than that, but we'll keep this simple representation.
|
||
|
||
ARM provides a standard and open implementation for BL2, BL31 and BL32 called TrustedFirmware-A.
|
||
Its documentation is available [here](https://trustedfirmware-a.readthedocs.io/en/latest/index.html).
|
||
Finally, our BL33 will be U-Boot, a bootloader widely used for ARM SoCs.
|
||
Once the board reach U-Boot, it will be able to boot NixOS.
|
||
|
||
Luckily, nixpkgs provides two functions to build these two elements as Nix derivations: `buildUBoot` and `buildArmTrustedFirmware`.
|
||
|
||
# Cross compilation on NixOS (for Nix-beginners)
|
||
|
||
As I plan to build everything from a x86_64 host, we have to set up a cross-compilation system to build our boot sequence
|
||
for ARMv8A.
|
||
Once again, nixpkgs make it fairly easy with its embedded cross-compilation system.
|
||
|
||
For example, if we want to build any aarch64 package from any architecture (like `hello`), we can just run:
|
||
|
||
```bash
|
||
nix-build '<nixpkgs>' \
|
||
--arg crossSystem '(import <nixpkgs/lib>).systems.examples.aarch64-multiplatform' \
|
||
-A hello
|
||
```
|
||
|
||
It is possible, because as any package in nixpkgs, `hello` is declared through a Nix recipe (a callPackage derivation) that
|
||
is called by the `callPackage` function.
|
||
This function setup a bunch of things and among them cross-compilation by looking at the `crossSystem` parameter provided to nixpkgs.
|
||
|
||
To cross-compile every program we need, we just have to create the following `default.nix` file:
|
||
|
||
```nix
|
||
{
|
||
pkgs ? import <nixpkgs> {
|
||
crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
|
||
},
|
||
}:
|
||
{
|
||
myProgram = pkgs.callPackage ./myProgram { };
|
||
}
|
||
```
|
||
|
||
Here, we just need to have a `myProgram` directory with a `default.nix` file containing a callPackage derivation
|
||
for our program, and then call `nix-build -A myProgram`.
|
||
|
||
# Building U-Boot
|
||
|
||
At the time I'm writing this post,
|
||
most of current Linux images for the BPI R4 uses [frank-w's U-Boot fork](https://github.com/frank-w/u-boot).
|
||
This is because the BPI-R4 is not supported by mainline U-boot.
|
||
The modifications frank-w did to bring this support are just few commits.
|
||
We can easily export them as patch and apply them on top of mainline U-Boot.
|
||
|
||
Let's create the following `default.nix` file:
|
||
|
||
```nix
|
||
{
|
||
pkgs ? import <nixpkgs> {
|
||
crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
|
||
},
|
||
}:
|
||
{
|
||
ubootBpiR4 = pkgs.callPackage ./u-boot { };
|
||
}
|
||
```
|
||
|
||
In a `u-boot` directory, we will create a `default.nix` file and a `patches` directory that will contain
|
||
all the patches that I grabbed from frank-w's U-Boot.
|
||
Within the `u-boot/default.nix` file, we will just have to write the following callPackage derivation:
|
||
|
||
```nix
|
||
{ buildUBoot, ... }:
|
||
buildUBoot {
|
||
defconfig = "mt7988a_bpir4_sd_defconfig";
|
||
extraMeta.platforms = [ "aarch64-linux" ];
|
||
extraPatches = [
|
||
./patches/0001-pci-mediatek-add-PCIe-controller-support-for-Filogic.patch
|
||
./patches/0002-Fix-PCIE-on-BPIR4.patch
|
||
./patches/0003-arm-dts-enable-pcie-in-sd-dts-too.patch
|
||
./patches/0004-dts-r4-disable-pcie2-in-emmc-dts.patch
|
||
./patches/0005-defconfig-uEnv-add-defconfigs-and-environment-files.patch
|
||
./patches/0006-defconfig-r4-update-with-pcie-options.patch
|
||
./patches/0007-defconfig-r4-add-pstore.patch
|
||
./patches/0008-defconfig-r4-update-emmc-defconfig.patch
|
||
./patches/0009-defconfig-r4-fix-duplicates-in-emmc-defconfig.patch
|
||
./patches/0010-arm64-dts-move-pcie-phy-to-dedicated-xsphy-no-driver.patch
|
||
./patches/0011-pci-mediatek-print-controller-address-for-card-detec.patch
|
||
];
|
||
|
||
extraConfig = ''
|
||
CONFIG_FIT=n
|
||
CONFIG_USE_DEFAULT_ENV_FILE=n
|
||
'';
|
||
|
||
filesToInstall = [ "u-boot.bin" ];
|
||
}
|
||
```
|
||
|
||
The `buildUboot` function will actually grab the source from the U-Boot repo and apply each patch
|
||
we give it through the `extraPatches` argument.
|
||
It will then copy the `defconfig` file from U-Boot config folder, and apply the `extraConfig` content on top.
|
||
It will eventually compile it and grab the files provided through `filesToInstall` from the build artifacts
|
||
and put it into the Nix store.
|
||
|
||
# ARM TrustedFirmware-A
|
||
|
||
Like U-Boot, the mainline version of TrustedFirmware does not currently support the BPI-R4.
|
||
For now, Linux distributions are using [this fork](https://github.com/mtk-openwrt/arm-trusted-firmware).
|
||
I don't really know for sure who is behind this account, and if I should really trust it.
|
||
Nonetheless, there is a bit too many commits for me to manage manually, so we'll here just grab this fork's sources.
|
||
As I told earlier, we'll use the nixpkgs's `buildArmTrustedFirmware` function.
|
||
Before that, and as we can see on the [frank-w's U-Boot build scripts](https://github.com/frank-w/u-boot/blob/mtk-atf/build.sh#L40),
|
||
we've to pass the U-Boot binary as input.
|
||
It'll be included within the final `fip.bin` file as BL33.
|
||
|
||
______________________________________________________________________
|
||
|
||
To be honest there, I don't really know if TrustedFirmware is mandatory to start our board.
|
||
Maybe U-Boot could've been enough, at the expense of not setting up the ARM secure environment.
|
||
Maybe U-Boot could itself set up this secure environment.
|
||
But for now, I decided to proceed like the OpenWrt and Debian images.
|
||
To confirm, I'll surely dig in the TrustedFirmware and ARM documentations later.
|
||
I'll eventually do a follow-up post later if I achieve to figure out how everything is working.
|
||
|
||
______________________________________________________________________
|
||
|
||
So, let's create our callPackage derivation in `trusted-firmware/default.nix`:
|
||
|
||
```nix
|
||
{
|
||
buildArmTrustedFirmware,
|
||
fetchFromGitHub,
|
||
dtc,
|
||
ubootBpiR4,
|
||
ubootTools,
|
||
openssl,
|
||
...
|
||
}:
|
||
(buildArmTrustedFirmware rec {
|
||
platform = "mt7988";
|
||
extraMeta.platforms = [ "aarch64-linux" ];
|
||
extraMakeFlags = [
|
||
"USE_MKIMAGE=1"
|
||
"BOOT_DEVICE=sdmmc"
|
||
"DRAM_USE_COMB=1"
|
||
"BL33=${ubootBpiR4}/u-boot.bin"
|
||
"all"
|
||
"fip"
|
||
];
|
||
|
||
filesToInstall = [
|
||
"build/${platform}/release/bl2.img"
|
||
"build/${platform}/release/fip.bin"
|
||
];
|
||
}).overrideAttrs (old: {
|
||
src = fetchFromGitHub {
|
||
owner = "mtk-openwrt";
|
||
repo = "arm-trusted-firmware";
|
||
rev = "bacca82a8cac369470df052a9d801a0ceb9b74ca";
|
||
hash = "sha256-n5D3styntdoKpVH+vpAfDkCciRJjCZf9ivrI9eEdyqw=";
|
||
};
|
||
version = "2.10.0-mtk";
|
||
nativeBuildInputs = old.nativeBuildInputs ++ [ dtc ubootTools openssl ];
|
||
}
|
||
```
|
||
|
||
This is very similar what we have done with U-Boot, we select our platform, then pass the necessary flags for compilation,
|
||
and then grab the two output files that we need.
|
||
However, in our situation, there are some extra steps.
|
||
Both `buildUBoot` and `buildArmTrustedFirmware` assumes that you're building the mainline U-Boot and TF-A.
|
||
It was the case with our U-Boot build to which, but not for our TF-A.
|
||
So we need to override the derivation produced by `buildArmTrustedFirmware` to set the `src` argument ourselves
|
||
and pass supplementary build dependencies in `nativeBuildInputs`.
|
||
|
||
As we can see, this callPackage derivation needs to access our previous `ubootBpiR4` derivation.
|
||
All the other derivations arguments are provided by nixpkgs itself through the `callPackage` function,
|
||
but not `ubootBpiR4` that we declared ourselves.
|
||
Given that, we can just easily pass it as is in the `callPackage` argument.
|
||
This makes our `default.nix` file looks like the following:
|
||
|
||
```nix
|
||
{
|
||
pkgs ? import <nixpkgs> {
|
||
crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
|
||
},
|
||
}:
|
||
rec {
|
||
ubootBpiR4 = pkgs.callPackage ./u-boot { };
|
||
armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
|
||
}
|
||
```
|
||
|
||
Now, we can run `nix-build -A armTrustedFirmwareBpiR4` to build everything we need !
|
||
|
||
# Create our boot image
|
||
|
||
The final step is just to create an empty image and flash what we have built so far at the matching address ranges.
|
||
This will be done with a simple bash script derivation in the `image/default.nix` file.
|
||
|
||
```nix
|
||
{
|
||
runCommand,
|
||
armTrustedFirmwareBpiR4,
|
||
gptfdisk,
|
||
...
|
||
}:
|
||
runCommand "bpi-r4-image" {
|
||
nativeBuildInputs = [
|
||
gptfdisk
|
||
];
|
||
} ''
|
||
IMAGE=$out/nixos-r4-image.img
|
||
|
||
mkdir $out
|
||
|
||
dd if=/dev/zero of=$IMAGE bs=1M count=4000
|
||
sgdisk -o $IMAGE
|
||
sgdisk -a 1 -n 1:34:8191 -A 1:set:2 -t 1:8300 -c 1:"bl2" $IMAGE
|
||
sgdisk -a 1 -n 2:8192:9215 -A 2:set:63 -t 2:8300 -c 2:"u-boot-env" $IMAGE
|
||
sgdisk -a 1 -n 3:9216:13311 -A 3:set:63 -t 3:8300 -c 3:"factory" $IMAGE
|
||
sgdisk -a 1 -n 4:13312:17407 -A 4:set:63 -t 4:8300 -c 4:"fip" $IMAGE
|
||
|
||
dd if=${armTrustedFirmwareBpiR4}/bl2.img of=$IMAGE seek=34 conv=notrunc,fsync
|
||
dd if=${armTrustedFirmwareBpiR4}/fip.bin of=$IMAGE seek=13312 conv=notrunc,fsync
|
||
''
|
||
```
|
||
|
||
The nixpkgs `runCommand` function creates a derivation that runs a bash script into nixpkgs's standard environment.
|
||
We can add dependencies with the second `runCommand` argument.
|
||
In this case, we'll add the `gptfdisk` package as a dependency to use `sgdisk` within our script.
|
||
Finally, the third argument is the script in itself.
|
||
For the script in itself, we'll create the derivation output directory and generate a zero-filled 4 GB image with `dd`.
|
||
Then, we want to generate our partition table with `sgdisk` with the following partition map:
|
||
|
||
```
|
||
Block size of 512 bytes
|
||
|
||
Blocks 34 to 8191: BL2
|
||
Blocks 8192 to 9215: Space for U-Boot environment variables
|
||
Blocks 9216 to 13311: Factory
|
||
Blocks 13312 to 17407: FIP
|
||
```
|
||
|
||
Finally, we flash our TF-A build outputs into the matching partitions with `dd`.
|
||
|
||
We add this callPackage derivation into our `default.nix`:
|
||
|
||
```nix
|
||
{
|
||
pkgs ? import <nixpkgs> {
|
||
crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
|
||
},
|
||
}:
|
||
rec {
|
||
ubootBpiR4 = pkgs.callPackage ./u-boot { };
|
||
armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
|
||
image = pkgs.callPackage ./image { inherit armTrustedFirmwareBpiR4; };
|
||
}
|
||
```
|
||
|
||
And we can build our boot sequence components with `nix-shell -A image` to get our `nixos-r4-image.img`.
|
||
|
||
# Flashing and running
|
||
|
||
The only thing left to do is flashing our image into a SD card.
|
||
|
||
```bash
|
||
dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress
|
||
```
|
||
|
||
After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot.
|
||
We get the following logs with `minicom`:
|
||
|
||
```
|
||
F0: 102B 0000
|
||
FA: 1042 0000
|
||
FA: 1042 0000 [0200]
|
||
F9: 1041 0000
|
||
F3: 1001 0000 [0200]
|
||
F3: 1001 0000
|
||
F6: 380E 5800
|
||
F5: 0000 0000
|
||
V0: 0000 0000 [0001]
|
||
00: 0000 0000
|
||
BP: 0600 0041 [0000]
|
||
G0: 1190 0000
|
||
EC: 0000 0000 [3000]
|
||
MK: 0000 0000 [0000]
|
||
T0: 0000 0221 [0101]
|
||
Jump to BL
|
||
|
||
NOTICE: BL2: v2.10.0 (release):
|
||
NOTICE: BL2: Built : 00:00:00, Jan 1 1980
|
||
NOTICE: WDT: Cold boot
|
||
NOTICE: WDT: disabled
|
||
NOTICE: CPU: MT7988
|
||
NOTICE: EMI: Using DDR unknown settings
|
||
NOTICE: EMI: Detected DRAM size: 4096 MB
|
||
NOTICE: EMI: complex R/W mem test passed
|
||
NOTICE: BL2: Booting BL31
|
||
NOTICE: BL31: v2.10.0 (release):
|
||
NOTICE: BL31: Built : 00:00:00, Jan 1 1980
|
||
|
||
|
||
U-Boot 2024.04 (Apr 02 2024 - 10:58:58 +0000)
|
||
|
||
CPU: MediaTek MT7988
|
||
Model: mt7988-rfb
|
||
DRAM: 4 GiB
|
||
Core: 49 devices, 19 uclasses, devicetree: separate
|
||
MMC: mmc@11230000: 0
|
||
Loading Environment from nowhere... OK
|
||
In: serial@11000000
|
||
Out: serial@11000000
|
||
Err: serial@11000000
|
||
Net:
|
||
Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06
|
||
eth0: ethernet@15100000
|
||
BPI-R4>
|
||
```
|
||
|
||
And here we have our initial access to the U-Boot console !
|
||
|
||
# Sources
|
||
|
||
I successfully got to this point after myself reading some documentation and a bunch of blog posts.
|
||
Those where the resources I used:
|
||
|
||
- [frank-w's dokuwiki](https://www.fw-web.de/dokuwiki/doku.php?id=en:bpi-r4:start)
|
||
- [Kasper Kondzielski's blog post about running NixOS on the BPI-R3](https://github.com/ghostbuster91/blogposts/blob/main/router2023/main.md)
|
||
- [Weijie Gao's post on the Banana Pi Discourse about U-Boot and TF-A on Mediatek SoCs](ihttps://forum.banana-pi.org/t/tutorial-build-customize-and-use-mediatek-open-source-u-boot-and-atf/13785)
|
||
- [*Understanding ARM Trusted Firmware using QEMU* by Hemanth Nandish](https://lnxblog.github.io/2020/08/20/qemu-arm-tf.html)
|