[Draft] Running NixOS on Banana Pi R4: Building the boot sequence

.gitignore

@@ -2,4 +2,3 @@ result/
 public/
 resources/
 .hugo_build.lock
-.helix/

content/posts/nixos-bpi-r4/2024-09-29-uboot-console.md

@@ -1,57 +1,49 @@
 +++
 title = 'Running NixOS on the Banana Pi R4: Building the boot sequence'
-date = 2024-10-05T08:37:23+02:00
+date = 2024-09-29T08:37:23+02:00
+draft = true
 +++
 
 # Before starting
 
 - I'm by no mean an expert about U-Boot, neither in ARM CPUs.
-  This is just me trying to port NixOS on the Banana Pi R4, and sharing what I learned.
-  As such, I may (and certainly will) utter false statements.
-  If I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it !
+This is just me trying to port NixOS on the Banana Pi R4, and sharing what I learned.
+As such, I may (and certainly will) utter false statements.
+If I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it !
 - I can't say really that I'm fluent in English. If you see some grammar or spelling mistakes, please reach me too !
 - This post's goal is not to be exhaustive, but I want to emphasize what I think is important or useful to know.
-  My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material.
+My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material.
 
 Thanks for your comprehension, and have a great time reading this.
 
 # How did we get there
 
 It has been a while since I started waiting for the Banana Pi R4.
-I thought of everything I could do with my own Linux gateway instead of being forced to get along my ISP's.
-As soon as the full bundle with the Wi-Fi 7 card was available and the first reviews appeared, my command was already confirmed.
-I've been waiting it for nearly a year, and initially though that I would just run OpenWrt on it.
-I mean, It is for sure the easiest solution (even if it would without doubt need some tweaks).
-However, after a quick thought, I felt like this router would fit even better in my NixOS environment.
-I mean, I have a centralized NixOS configuration for all my machines that can be easily built from any of them and be deployed on another
-with a single command.
-It would make even more sense too cross-compile my router configuration from my PC with its Ryzen 7 5800X and deploy it on the BPI-R4.
-
-However, there are three caveats with this choice:
+I bought it as soon as a complete bundle with the Wi-Fi 7 module was available.
+I initially planned to run OpenWrt on it, without bothering too much.
+But if my machines (including my ROCKPro64) are all running NixOS with my
+[common flake configuration](https://git.dalaran.fr/dala/nixos-config), there is no reason why my router
+should not.
+This is because I use a [colmena](https://github.com/zhaofengli/colmena)-based deployment system.
+It allows me to build (and cross-compile) all my machines configurations and to deploy them from my PC.
 
+However, there are two caveats here:
 - The Nix store usually takes some place on storage devices, so the embedded 8 GB eMMC might not be enough.
-  But, since the BPI R4 has an integrated slot for a NVMe SSD and I have an empty 500 GB SSD available,
-  it is way more than enough.
-- I would say (I might be wrong there), that NixOS is not really the best fit for such small home gateways.
-  Some OpenWrt packages are missing on NixOS, with the most obvious example being LuCI.
-  I don't think that such packages are mandatory though, they would mainly bring QoL features.
+But, since the BPI R4 has an integrated slot for a NVMe SSD and I have an empty 500 GB SSD available,
+it is way more than enough.
 - There is currently no support or available image of NixOS for the BPI R4.
 
-But I just felt that I would still be more comfortable with another machine in my NixOS environment than
-with one outside of it that I'll have to manage its own way.
 This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4.
 
 For your information, this is what I had before starting this project:
-
 - Some basic Nix knowledge.
 - I already used U-Boot on some boards with prebuilt Linux images, but never really installed it or tweaked it myself.
 - Some knowledge about basic Linux utilities used for embedded.
 - A USB-Serial cable.
-- A Banana Pi R4 (of course).
 
-For a NixOS SD image generation on ARM machines, some people prefer to build it with their Nix configuration already
-applied to it, and then just flash into their board.
-In my case, I prefer the philosophy of having an image that just serve as an installation media, and then manually do
+For a NixOS SD image generation, some people chose to build it with their Nix configuration already
+applied to it, and then just flash this image to their board flash device.
+I will personally just build an image that just serve as an installation media, and then manually do
 all the installation process as I would do on a PC.
 
 Today's post is the first part about building our boot components and getting access to the U-Boot console.
@@ -60,54 +52,51 @@ Let's get started !
 
 # The ARMv8 boot sequence
 
-Before focusing on the Linux kernel or NixOS, we'll have to build the initial boot sequence for the R4.
-It would be too easy to just install our bootloader and having an UEFI running it directly **right** ?
-The boot phase in ARM-based board is often a bit trickier as it is on x86 hosts.
-On our case, we need a firmware setting up the ARM secure environment (which is ARM TrustedFirmware-A)
-and a bootloader (U-Boot).
+Before focusing on the Linux kernel or NixOS, we will have to build the boot sequence for the R4.
+This includes ARM TrustedFirmware-A and U-Boot.
 
-On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that is run to init our board)
+On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that will be run to init our board)
 is standardized with the following elements:
-
 - BL1: this is a software brick that is already present in the board's Boot ROM. It performs basic
-  hardware initialization and then pass the control to the next step (BL2).
+hardware initialization and then pass the control to the next step (BL2).
 - BL2: Another software brick whose goal is to create a secure environment for the software that will be run, and then
-  pass the control to the last step.
+pass the control to the last step.
 - BL3: the final element of the boot sequence. It is basically what the board will finally run.
-  In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software),
-  an optional BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run.
+In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software),
+an optional BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run.
 
-It's in fact a bit more complicated than that, but we'll keep this simple representation.
+It seems in reality a bit more complicated than that, dealing with ARM Exception Level, but we will stick
+to this simple representation.
+
+ARM TrustedFirmware-A provides a standard implementation for BL2, BL31 and BL32. Its documentation is available
+[here](https://trustedfirmware-a.readthedocs.io/en/latest/index.html).
+U-Boot is the bootloader which is the most commonly used on ARM boards, and that will act as BL33.
+It has all the necessary utilities to load and boot the Linux kernel.
 
-ARM provides a standard and open implementation for BL2, BL31 and BL32 called TrustedFirmware-A.
-Its documentation is available [here](https://trustedfirmware-a.readthedocs.io/en/latest/index.html).
-Finally, our BL33 will be U-Boot, a bootloader widely used for ARM SoCs.
 Once the board reach U-Boot, it will be able to boot NixOS.
-
-Luckily, nixpkgs provides two functions to build these two elements as Nix derivations: `buildUBoot` and `buildArmTrustedFirmware`.
+Nixpkgs provide two functions to build these two elements as Nix derivations: `buildUBoot` and `buildArmTrustedFirmware`.
+These two functions make the creation of such derivations fairly easy.
 
 # Cross compilation on NixOS (for Nix-beginners)
 
-<!-- If you're not on an aarch64 machine (or did not enable native build through QEMU on your configuration), -->
-
-<!-- you won't be able to build the following derivations. -->
-
-As I plan to build everything from a x86_64 host, we have to set up a cross-compilation system to build our boot sequence
-for ARMv8A.
-Once again, nixpkgs make it fairly easy with its embedded cross-compilation system.
+If you're not on an aarch64 machine (or did not enable native build through QEMU on your configuration),
+you won't be able to build the following derivations.
+To do so, we have to set up cross-compilation.
+Everybody knows nixpkgs as a collection of Nix derivations that contains some utility functions like `mkDerivation` too.
+It also provides a cross-compilation system that allows to cross compile each derivation for a compatible architecture !
 
 For example, if we want to build any aarch64 package from any architecture (like `hello`), we can just run:
-
 ```bash
 nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs/lib>).systems.examples.aarch64-multiplatform' -A hello
 ```
-
+This will cross-compile `hello` for aarch64 in the Nix store.
 It is possible, because as any package in nixpkgs, `hello` is declared through a Nix recipe (a callPackage derivation) that
-is called by the `callPackage` function.
-This function setup a bunch of things and among them cross-compilation by looking at the `crossSystem` parameter provided to nixpkgs.
+is called by the `callPackage` function of nixpkgs.
+This function setup a bunch of things and among them cross-compilation.
+It does so by looking at the `crossSystem` parameter provided to nixpkgs.
+It will then cross-compile every dependency of the derivation before finally building our package.
 
 To cross-compile every program we need, we just have to create the following `default.nix` file:
-
 ```nix
 {
   pkgs ? import <nixpkgs> {
@@ -118,20 +107,17 @@ To cross-compile every program we need, we just have to create the following `de
   myProgram = pkgs.callPackage ./myProgram { };
 }
 ```
-
 Here, we just need to have a `myProgram` directory with a `default.nix` file containing a callPackage derivation
 for our program, and then call `nix-build -A myProgram`.
 
 # Building U-Boot
 
-At the time I'm writing this post,
-most of current Linux images for the BPI R4 uses [frank-w's U-Boot fork](https://github.com/frank-w/u-boot).
-This is because the BPI-R4 is not supported by mainline U-boot.
-The modifications frank-w did to bring this support are just few commits.
-We can easily export them as patch and apply them on top of mainline U-Boot.
+Most of current Linux images for the BPI R4 uses [frank-w's U-Boot fork](https://github.com/frank-w/u-boot).
+This is because, for now, mainline U-Boot does not support the router.
+His modifications to bring support for our device are just few commits that I'll export as patch, and apply
+them on top of mainline U-Boot.
 
 Let's create the following `default.nix` file:
-
 ```nix
 {
   pkgs ? import <nixpkgs> {
@@ -144,9 +130,8 @@ Let's create the following `default.nix` file:
 ```
 
 In a `u-boot` directory, we will create a `default.nix` file and a `patches` directory that will contain
-all the patches that I grabbed from frank-w's U-Boot.
+all the patches that I took from frank-w's U-Boot.
 Within the `u-boot/default.nix` file, we will just have to write the following callPackage derivation:
-
 ```nix
 { buildUBoot, ... }:
 buildUBoot {
@@ -175,36 +160,38 @@ buildUBoot {
 }
 ```
 
-The `buildUboot` function will actually grab the source from the U-Boot repo and apply each patch
-we give it through the `extraPatches` argument.
-It will then copy the `defconfig` file from U-Boot config folder, and apply the `extraConfig` content on top.
-It will eventually compile it and grab the files provided through `filesToInstall` from the build artifacts
-and put it into the Nix store.
+As I said previously, nixpkgs provides a `buildUBoot` function, to which we have just to pass some arguments.
+Here, frank-w's U-Boot patches define a new defconfig called `mt7988a_bpi_r4_sd_defconfig` for the R4.
+This config enables the generation of a Flattened Image Tree.
+As far as I understand, this is a blob containing all the necessary boot configurations and files that U-Boot needs to launch our kernel.
+In my case, at least for the beginning, I chose to stick with the plain-old U-Boot builds (without using FIT) to learn to do it myself instead of using
+any special packaging format.
+In the end, the `filesToInstall` arguments just specifies which final build products we want to keep in the derivation output.
+For now, we will only keep the u-boot binary and see if we actually need anything else later.
+That's it for U-Boot ! We can try to build it with `nix-build -A ubootBpiR4` and see the `u-boot.bin` file
+appear in the Nix store, and in the `result` folder !
 
 # ARM TrustedFirmware-A
 
-Like U-Boot, the mainline version of TrustedFirmware does not currently support the BPI-R4.
-For now, Linux distributions are using [this fork](https://github.com/mtk-openwrt/arm-trusted-firmware).
-I don't really know for sure who is behind this account, and if I should really trust it.
-Nonetheless, there is a bit too many commits for me to manage manually, so we'll here just grab this fork's sources.
-As I told earlier, we'll use the nixpkgs's `buildArmTrustedFirmware` function.
+As with U-Boot, the mainline version of TrustedFirmware does not support the BPI-R4.
+For now, distributions are using [this fork](https://github.com/mtk-openwrt/arm-trusted-firmware).
+I don't really know for sure who is behind this account, but there is a bit too much commits for me
+to export them as patch to apply on top of mainline.
+Once again, we will use the nixpkgs's `buildArmTrustedFirmware` function.
 Before that, and as we can see on the [frank-w's U-Boot build scripts](https://github.com/frank-w/u-boot/blob/mtk-atf/build.sh#L40),
-we've to pass the U-Boot binary as input.
-It'll be included within the final `fip.bin` file as BL33.
+we will need to pass to U-Boot binary. It should be included within the final `fip.bin` file.
 
-______________________________________________________________________
+---
 
-To be honest there, I don't really know if TrustedFirmware is mandatory to start our board.
-Maybe U-Boot could've been enough, at the expense of not setting up the ARM secure environment.
-Maybe U-Boot could itself set up this secure environment.
+To be honest there, I don't really know if TrustedFirmware is mandatory to run U-boot
+on our SoC, or if U-Boot by itself could have handled everything.
 But for now, I decided to proceed like the OpenWrt and Debian images.
-To confirm, I'll surely dig in the TrustedFirmware and ARM documentations later.
-I'll eventually do a follow-up post later if I achieve to figure out how everything is working.
+I will surely dig in the TrustedFirmware and ARM documentations later.
+I'll eventually to a follow-up post later if I achieve to figure out how everything is working.
 
-______________________________________________________________________
+---
 
 So, let's create our callPackage derivation in `trusted-firmware/default.nix`:
-
 ```nix
 {
   buildArmTrustedFirmware,
@@ -247,16 +234,18 @@ This is very similar what we have done with U-Boot, we select our platform, then
 and then grab the two output files that we need.
 However, in our situation, there are some extra steps.
 Both `buildUBoot` and `buildArmTrustedFirmware` assumes that you're building the mainline U-Boot and TF-A.
-It was the case with our U-Boot build to which, but not for our TF-A.
-So we need to override the derivation produced by `buildArmTrustedFirmware` to set the `src` argument ourselves
-and pass supplementary build dependencies in `nativeBuildInputs`.
+It was the case with our U-Boot build to which we just add some extra patches.
+However, with TF-A, we will here use the mtk-openwrt's fork.
+So we need to override the derivation produced by `buildArmTrustedFirmware` to pass our own sources and needed dependencies thanks
+to the `overrideAttrs` function.
 
 As we can see, this callPackage derivation needs to access our previous `ubootBpiR4` derivation.
-All the other derivations arguments are provided by nixpkgs itself through the `callPackage` function,
-but not `ubootBpiR4` that we declared ourselves.
-Given that, we can just easily pass it as is in the `callPackage` argument.
-This makes our `default.nix` file looks like the following:
-
+All the other derivations arguments are provided by nixpkgs itself through the `callPackage` function.
+However, `ubootBpiR4` is itself not present within nixpkgs, we have to manually pass it.
+There are two methods to do so, the first one is to use a nixpkgs overlay.
+The other one is to pass it manually as a `callPackage` argument.
+That's what we'll do here.
+This makes our `default.nix` file looks like that:
 ```nix
 {
   pkgs ? import <nixpkgs> {
@@ -269,13 +258,24 @@ rec {
 }
 ```
 
-Now, we can run `nix-build -A armTrustedFirmwareBpiR4` to build everything we need !
+Now, we can run `nix-build -A armTrustedFirmwareBpiR4` to build everything we need.
+
+# Side note on the U-Boot and TF-A combination
+
+If you check how to build the full boot sequence binaries for various board, you will maybe feel like me
+that it seems to be highly dependent on the SoC manufacturer.
+In our case, we build U-Boot first and then pass it to TF-A as BL33.
+For some Allwinner SoCs, TF-A is built first and then passed to U-boot as BL31.
+For Rockchip based boards there is this whole `idbloader.img` due to Rockchip's miniloader.
+At this point, I don't really know if there is a "standard way" to build to whole boot sequence for each SoC.
+I don't even know what causes these differences in the build process between manufacturers.
+I guess it's related to the manufacturer implementation, or maybe things that could be done both in U-Boot and in TF-A
+and then depends on where the manufacturer implemented them ?
 
 # Create our boot image
 
 The final step is just to create an empty image and flash what we have built so far at the matching address ranges.
 This will be done with a simple bash script derivation in the `image/default.nix` file.
-
 ```nix
 {
   runCommand,
@@ -304,13 +304,13 @@ runCommand "bpi-r4-image" {
 ''
 ```
 
-The nixpkgs `runCommand` function creates a derivation that runs a bash script into nixpkgs's standard environment.
-We can add dependencies with the second `runCommand` argument.
-In this case, we'll add the `gptfdisk` package as a dependency to use `sgdisk` within our script.
-Finally, the third argument is the script in itself.
-For the script in itself, we'll create the derivation output directory and generate a zero-filled 4 GB image with `dd`.
-Then, we want to generate our partition table with `sgdisk` with the following partition map:
-
+The nixpkgs `runCommand` function creates a derivation that runs a bash script.
+This bash script is running into the nixpkgs standard environment, but we can override this environment with the set passed
+as the second argument.
+In this case, we add the `gptfdisk` package as a dependency to use `sgdisk` within our bash script.
+The third argument is the script in itself.
+We then create the derivation output directory, and generate a zero-filled 4 GB image with `dd`.
+Then, we generate our partition table with `sgdisk` with the following partition map:
 ```
 Block size of 512 bytes
 
@@ -319,11 +319,9 @@ Blocks 8192 to 9215: Space for U-Boot environment variables
 Blocks 9216 to 13311: Factory
 Blocks 13312 to 17407: FIP
 ```
-
 Finally, we flash our TF-A build outputs into the matching partitions with `dd`.
 
 We add this callPackage derivation into our `default.nix`:
-
 ```nix
 {
   pkgs ? import <nixpkgs> {
@@ -341,15 +339,13 @@ And we can build our boot sequence components with `nix-shell -A image` to get o
 
 # Flashing and running
 
-The only thing left to do is flashing our image into a SD card.
-
+So let's flash the resulting image into an SD card.
 ```bash
 dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress
 ```
 
 After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot.
 We get the following logs with `minicom`:
-
 ```
 F0: 102B 0000
 FA: 1042 0000
@@ -397,15 +393,14 @@ Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06
 eth0: ethernet@15100000
 BPI-R4>
 ```
-
 And here we have our initial access to the U-Boot console !
 
 # Sources
 
 I successfully got to this point after myself reading some documentation and a bunch of blog posts.
 Those where the resources I used:
-
 - [frank-w's dokuwiki](https://www.fw-web.de/dokuwiki/doku.php?id=en:bpi-r4:start)
 - [Kasper Kondzielski's blog post about running NixOS on the BPI-R3](https://github.com/ghostbuster91/blogposts/blob/main/router2023/main.md)
 - [Weijie Gao's post on the Banana Pi Discourse about U-Boot and TF-A on Mediatek SoCs](ihttps://forum.banana-pi.org/t/tutorial-build-customize-and-use-mediatek-open-source-u-boot-and-atf/13785)
 - [*Understanding ARM Trusted Firmware using QEMU* by Hemanth Nandish](https://lnxblog.github.io/2020/08/20/qemu-arm-tf.html)
+

shell.nix

@@ -5,9 +5,6 @@ pkgs.mkShell {
   packages = with pkgs; [
     nixfmt-rfc-style
     nil
-    ltex-ls
-    proselint
-    mdformat
   ];
 
   nativeBuildInputs = with pkgs; [