[Draft] Running NixOS on Banana Pi R4: Building the boot sequence

remove hugo's build lock and add flake overlay
2024-09-29 21:12:12 +02:00 · 2024-09-29 21:08:46 +02:00
6 changed files with 511 additions and 0 deletions
--- a/.gitignore
+++ b/.gitignore
@ -1,3 +1,4 @@
 result/
 public/
 resources/
 .hugo_build.lock
--- a/.hugo_build.lock
+++ b/.hugo_build.lock
--- a/content/posts/nixos-bpi-r4/2024-09-29-uboot-console.md
+++ b/content/posts/nixos-bpi-r4/2024-09-29-uboot-console.md
@ -0,0 +1,476 @@
 +++
 title = 'Running NixOS on the Banana Pi R4: Building the boot sequence'
 date = 2024-09-29T08:37:23+02:00
 draft = true
 +++
 ## Trigger Warnings
 - I'm by no mean an expert about U-Boot, neither in ARM CPUs.
  This is just me trying to accomplish a goal, and sharing what I think I have learned.
  As such, I may (and certainly will) utter false statements.
  If you see that I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it !
 - I can't say that I'm fluent in English. If you see some grammar or spelling mistakes, please reach me too !
 - This post's goal is to be exhaustive.
  My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material.
 Thanks for your comprehension, and have a great time reading this.
 ## How did we get there
 It has been a while since I started waiting for the Banana Pi R4.
 I bought it as soon as a full bundle with the WiFi 7 module was available.
 The goal initially was to run OpenWRT on it, without bothering too much.
 But then I tought that if all my machines (including my ROCKPro64) are running NixOS with my
 [common flake configuration](https://git.dalaran.fr/dala/nixos-config), there is no reason why my router
 should not.
 This is because I use a [colmena](https://github.com/zhaofengli/colmena)-based deployment system.
 It allows me to build (and cross-compile) all my machines configurations and to deploy them from my PC.
 However, there is two caveats here:
 - The Nix store usually takes some place on storage devices, so the embedded 8GB eMMC might not be enough.
 But, as the BPI R4 has an integrated slot for a NVME SSD, and since I had an empty 500 GB SSD available,
 it is way more than enough.
 - There is currently no support or available image of NixOS for the BPI R4.
 This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4.
 For your information, this is what I had before starting this project:
 - Basic Nix knowledge.
 - I already used U-Boot on some boards with prebuilt images, but never really installed it or tweaked it myself.
 - Some knowledge about basic Linux utilities used for embedded.
 - A USB-Serial cable.
 I saw that for the image generation, some people chose to build an image with their Nix configuration already
 applied to it, and then just flash this image to their board flash device.
 In my case, I would like to stick with an image that just serve as an install media, and then manually do
 all the installation process as I would do on a PC.
 As much as possible, I would like to stick with mainline U-Boot and Linux kernel.
 I will try to grab any useful patches from R4 specific forks and apply them manually.
 Today's post is the first part about building our boot components and getting access to the u-Boot console.
 Let's get started !
 # Initial checks
 The BPI-R4 should be shipped with an existing install of OpenWRT on the NAND flash.
 If we set the boot-device selection jumper to NAND Flash and connect our USB-Serial cable, we should see boot logs.
 So after doing this and starting `minicom`, I got the following:
 ```
 ubi0: attaching mtd6
 ubi0: scanning is finished
 ubi0: attached mtd6 (name "ubi", size 112 MiB)
 ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes
 ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048
 ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096
 ubi0: good PEBs: 900, bad PEBs: 0, corrupted PEBs: 0
 ubi0: user volume: 3, internal volumes: 1, max. volumes count: 128
 ubi0: max/mean erase counter: 2/0, WL threshold: 4096, image sequence number: 1697192243
 ubi0: available PEBs: 0, total reserved PEBs: 900, PEBs reserved for bad PEB handling: 19
 No size specified -> Using max size (4190208)
 Read 4190208 bytes from volume kernel to 0000000046000000
 ## Loading kernel from FIT Image at 46000000 ...
   Using 'config-1' configuration
   Trying 'kernel-1' kernel subimage
     Description:  ARM64 OpenWrt Linux-5.4.246
     Type:         Kernel Image
     Compression:  lzma compressed
     Data Start:   0x460000e8
     Data Size:    4061787 Bytes = 3.9 MiB
     Architecture: AArch64
     OS:           Linux
     Load Address: 0x48080000
     Entry Point:  0x48080000
     Hash algo:    crc32
     Hash value:   ef1d4754
     Hash algo:    sha1
     Hash value:   78589b5e23500626a36c8a2ff523eb8bbcaddabc
   Verifying Hash Integrity ... crc32+ sha1+ OK
 ## Loading fdt from FIT Image at 46000000 ...
   Using 'config-1' configuration
   Trying 'fdt-1' fdt subimage
     Description:  ARM64 OpenWrt BPI-R4-NAND device tree blob
     Type:         Flat Device Tree
     Compression:  uncompressed
     Data Start:   0x463dfc80
     Data Size:    34487 Bytes = 33.7 KiB
     Architecture: AArch64
     Hash algo:    crc32
     Hash value:   0c548bfa
     Hash algo:    sha1
     Hash value:   0fb7882fc5ef9acba18b6864234dd1aec0b3d2f2
   Verifying Hash Integrity ... crc32+ sha1+ OK
   Booting using the fdt blob at 0x463dfc80
 Working FDT set to 463dfc80
   Uncompressing Kernel Image
   Loading Device Tree to 00000000ff7ec000, end 00000000ff7f76b6 ... OK
 Working FDT set to ff7ec000
 Add 'ramoops@42ff0000' node failed: FDT_ERR_EXISTS
 Starting kernel ...
 ```
 Okay, the board is working as expected and theses logs may serve as a reference later.
 I also quickly checked the hardware revision of my R4.
 In the first revisions, a resistor on the PCB caused I2C devices to be unusable when using an NVME SSD.[0]
 In my case, the resistor was no longer there, so I should not face any issue.
 [0]: https://forum.banana-pi.org/t/bpi-r4-nvme-i2c/17152
 ## The ARMv8 boot sequence
 Before focusing on the Linux kernel or NixOS, we will have to build the boot sequence for our R4.
 In our case, this includes ARM TrustedFirmware-A and U-Boot.
 On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that will be run to start our board)
 is standardized with the following elements:
 - BL1: this is a software brick that is already present in the board Boot ROM. Its goals are to perform basic
 hardware initialization and then pass the control to the next step (BL2).
 To do so, it expects to find the next step at a specific address in RAM.
 - BL2: another software brick whose goal is to create a secure environment for the software that will be run, and then
 pass the control to the last step.
 - BL3: the final element of the boot sequence. It is basically what the board will finally run.
 In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software),
 an optionnal BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run.
 It seems in reality a bit more complicated than that, dealing with ARM Exception Level, but we will stick
 to this simple representation.
 ARM TrustedFirmware-A provides a standard implementation for BL2 and BL31. Its documentation is available
 [here](https://trustedfirmware-a.readthedocs.io/en/latest/index.html).
 U-Boot is the bootloader which is the most commonly used on ARM boards, and that correspond here to BL33.
 It has all the necessary utilities to load and boot the Linux kernel.
 This is all the two elements that we will have to build and add to our image to then be able to boot NixOS.
 Nixpkgs provide two functions to build these two elements as Nix derivations: `buildUBoot` and `buildArmTrustedFirmware`.
 This make the creation of such derivations fairly easy.
 ## Cross compilation on NixOS (for Nix-beginners)
 If you're not on an aarch64 machine (or did not enable native build through QEMU on your configuration),
 you won't be able to build the following derivations.
 To do so, we have to setup cross-compilation.
 Everybody knows Nixpkgs as a collection of Nix derivations that it contains some utility functions like `mkDerivation`.
 Nixpkgs also provides a cross-compilation system that allows to cross compile each derivation for a compatible architecture !
 For example, if we want to build any aarch64 package from any architecture (like GNU hello), we can just do:
 ```
 nix-build '<nixpkgs>' --arg crossSystem '(import <nixpkgs/lib>).systems.examples.aarch64-multiplatform' -A hello
 ```
 This will cross-compile hello for aarch64 in the Nix store, and set a symlink to it in a `result` directory in the current one.
 It is possible, because as any package in Nixpkgs, GNU hello is declared through a Nix recipe (called a callPackage derivation) that
 is called by the `callPackage` function of Nixpkgs.
 This function setup a bunch of things, and among them cross-compilation.
 It does so by looking at the `crossSystem` parameters provided to Nixpkgs when importing it.
 It will then cross-compile every dependency of the derivation before finally building our package.
 So here, to cross-compile each of our program we need, we just have to create the following `default.nix` file:
 ``` 
 {
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
 }:
 {
  myProgram = pkgs.callPackage ./myProgram { };
 }
 ```
 Here, we just need to have a `myProgram` directory with a `default.nix` file containing a callPackage derivation
 for our program, and then call `nix-build -A myProgram`.
 ## Building U-Boot
 We should be ready to build a mainline U-Boot. However, the latter does not support the BPI R4 for now.
 Most of current Linux images for the BPI R4 uses [frank-w's U-Boot fork](https://github.com/frank-w/u-boot).
 His modifications to bring support for our device are just few commits that I'll export as patch, and apply
 them on top of mainline U-Boot.
 Let's create the following `default.nix` file:
 ```
 {
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
 }:
 {
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
 }
 ```
 In a `u-boot` directory, we will create a `default.nix` file and a `patches` directory that will contains
 all the patches that I took from frank-w's U-Boot.
 Within the `u-boot/default.nix` file, we will just have to write the following callPackage derivation:
 ```
 { buildUBoot, ... }:
 buildUBoot {
  defconfig = "mt7988a_bpir4_sd_defconfig";
  extraMeta.platforms = [ "aarch64-linux" ];
  extraPatches = [
    ./patches/0001-pci-mediatek-add-PCIe-controller-support-for-Filogic.patch
    ./patches/0002-Fix-PCIE-on-BPIR4.patch
    ./patches/0003-arm-dts-enable-pcie-in-sd-dts-too.patch
    ./patches/0004-dts-r4-disable-pcie2-in-emmc-dts.patch
    ./patches/0005-defconfig-uEnv-add-defconfigs-and-environment-files.patch
    ./patches/0006-defconfig-r4-update-with-pcie-options.patch
    ./patches/0007-defconfig-r4-add-pstore.patch
    ./patches/0008-defconfig-r4-update-emmc-defconfig.patch
    ./patches/0009-defconfig-r4-fix-duplicates-in-emmc-defconfig.patch
    ./patches/0010-arm64-dts-move-pcie-phy-to-dedicated-xsphy-no-driver.patch
    ./patches/0011-pci-mediatek-print-controller-address-for-card-detec.patch
  ];
  extraConfig = ''
    CONFIG_FIT=n
    CONFIG_USE_DEFAULT_ENV_FILE=n
  '';
  filesToInstall = [ "u-boot.bin" ];
 }
 ```
 As I said previously, Nixpkgs provides a `buildUBoot` function, to which we have just to path some args.
 Here, frank-w's U-Boot patches define a new defconfig called `mt7988a_bpi_r4_sd_defconfig` for the R4.
 This config enable by default the use of a FIT (Flattened Image Tree).
 As far as I understood, this allows the creation of a blob containing all the necessary boot info that U-Boot needs instead of loading a couple of files.
 In my case, at least for the beginning, I chose to stick with the plain-old U-Boot builds to learn to do it myself instead of using
 any special packaging format.
 In the end, the `filesToInstall` arguments just tells which final build products we want to get in the derivation output.
 For now, we will only keep the u-boot binary and see if we actually need anything else later.
 That's it for U-Boot ! We can try to build it with `nix-build -A ubootBpiR4` and see the `u-boot.bin` file
 appear in the Nix store, and in the `result` folder !
 ## ARM TrustedFirmware-A
 As with U-Boot, the mainline version of TrustedFirmware does not support the BPI-R4.
 For now, distributions are using [this fork](https://github.com/mtk-openwrt/arm-trusted-firmware).
 I don't really know who for sure who is behind this account, but there is a bit too much commits for me
 to export them as patch.
 So even if I told that I would like to stick with mainline, for now we will still using this fork directly.
 Once again, we will use the Nixpkgs's `buildArmTrustedFirmware` function.
 Before that, and as we can see on the [frank-w's U-Boot build scripts](https://github.com/frank-w/u-boot/blob/mtk-atf/build.sh#L40),
 we will need to pass to U-Boot binary. It should be included within the final `fip.bin` file.
 ---
 To be honest there, I don't really know if TrustedFirmware is mandatory (and I guess it's not) to boot Linux
 on our SoC, or if U-Boot by itself could have made everything by itself.
 I don't even know the details of what is an FIP image.
 But for now, I decided to proceed like the OpenWRT and Debian images.
 I will surely dig in the TrustedFirmare and ARM documentations later.
 I'll eventually to a follow-up post later if I achieve to figure out how everything is working.
 ---
 So, let's create our callPackage derivation in `trusted-firmware/default.nix`:
 ```
 {
  buildArmTrustedFirmware,
  fetchFromGitHub,
  dtc,
  ubootBpiR4,
  ubootTools,
  openssl,
  ...
 }:
 (buildArmTrustedFirmware rec {
  platform = "mt7988";
  extraMeta.platforms = [ "aarch64-linux" ];
  extraMakeFlags = [
    "USE_MKIMAGE=1"
    "BOOT_DEVICE=sdmmc"
    "DRAM_USE_COMB=1"
    "BL33=${ubootBpiR4}/u-boot.bin"
    "all"
    "fip" 
  ];
  filesToInstall = [
    "build/${platform}/release/bl2.img"
    "build/${platform}/release/fip.bin"
  ];
 }).overrideAttrs (old: {
  src = fetchFromGitHub {
    owner = "mtk-openwrt";
    repo = "arm-trusted-firmware";
    rev = "bacca82a8cac369470df052a9d801a0ceb9b74ca";
    hash = "sha256-n5D3styntdoKpVH+vpAfDkCciRJjCZf9ivrI9eEdyqw=";
  };
  version = "2.10.0-mtk";
  nativeBuildInputs = old.nativeBuildInputs ++ [ dtc ubootTools openssl ];
 }
 ```
 This looks like what we have done with U-Boot, we select our platform, then pass the necessary flags for compilation,
 and then grab the two output files that we need.
 However here, there is some extra steps.
 Both `buildUBoot` and `buildArmTrustedFirmware` assumes that you're building the mainline U-Boot and TF-A.
 It was the case with our U-Boot build to which we just add some extra patches.
 However, with TF-A, as I said, we will here use the mtk-openwrt's fork.
 So we need to override the derivation produced by `buildArmTrustedFirmware` to pass our own sources and needed dependencies thanks
 to the `overrideAttrs` function.
 As we can see, this callPackage derivation needs to access our previous `ubootBpiR4` derivation.
 All the other derivations arguments are provided by nixpkgs itself through the `callPackage` function.
 However, `ubootBpiR4` is itself not present within the nixpkgs we use, we have to manually pass it.
 There is two solutions here, the first one is to use an overlay to manually add it to our nixpkgs.
 But there is a far further solution, which is to pass it as an argument of the `callPackage` function.
 This make our `default.nix` file looks like that:
 ```
 {
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
 }:
 rec {
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
  armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
 }
 ```
 Now, we can run `nix-build -A armTrustedFirmwareBpiR4` to build everything we need.
 ## Side note on the U-Boot and TF-A combination
 If you check how to build the full boot sequence binaries for various board, you will be feel like me
 that this seems to be highly dependant on SoC manufacturer and the SoC itself.
 As we have seen there, we build U-Boot first and then pass it to TF-A as BL33.
 For some Allwinner SoCs, TF-A is built first and then passed to U-boot as BL31.
 For Rockchip based boardsm there is this whole `idbloader.img` due to Rockchip's miniloader.
 At this point, I don't really know if there is truly a "standard way" to build to whole boot sequence for each SoC.
 I don't even know what causes this difference in the build process between manufacturers.
 I guess this is related to manufacturers' implementation, or things that could be done both in U-Boot and in TF-A
 and then depends about where the manufacturer implemented them ?
 ## Create our boot image
 The final step is just to create an empty image and flash what we have built so far at the corrects address ranges.
 This will be done with a simple bash script derivation in the `image/default.nix` file.
 I will just dump the callPackage derivation here, and explain everything.
 ```
 {
  runCommand,
  armTrustedFirmwareBpiR4,
  gptfdisk,
  ...
 }:
 runCommand "bpi-r4-image" {
  nativeBuildInputs = [
    gptfdisk
  ];
 } ''
  IMAGE=$out/nixos-r4-image.img
  mkdir $out
  dd if=/dev/zero of=$IMAGE bs=1M count=4000
  sgdisk -o $IMAGE
  sgdisk -a 1 -n 1:34:8191 -A 1:set:2 -t 1:8300 -c 1:"bl2" $IMAGE
 	sgdisk -a 1 -n 2:8192:9215 -A 2:set:63	-t 2:8300 -c 2:"u-boot-env"	$IMAGE
 	sgdisk -a 1 -n 3:9216:13311 -A 3:set:63	-t 3:8300 -c 3:"factory" $IMAGE
 	sgdisk -a 1 -n 4:13312:17407 -A 4:set:63	-t 4:8300 -c 4:"fip" $IMAGE
  dd if=${armTrustedFirmwareBpiR4}/bl2.img of=$IMAGE seek=34 conv=notrunc,fsync
  dd if=${armTrustedFirmwareBpiR4}/fip.bin of=$IMAGE seek=13312 conv=notrunc,fsync
 ''
 ```
 The nixpkgs's `runCommand` function create a derivation that runs a bash script.
 This bash script is run into the Nixpkgs `stdenv`, but we can override this environment with the set passed
 as its second arguments.
 In this case, we add the `gptfdisk` package as a dependency to use `sgdisk` within our bash script.
 The third argument is the script in itself.
 We then create the derivation output directory, and generate a zero-filled 4GB image with `dd`.
 Then, we create our partitions with `sgdisk` with the following partition map:
 ```
 Block size of 512 bytes
 Blocks 34 to 8191: BL2
 Blocks 8192 to 9215: Future space for U-Boot environment variables
 Blocks 9216 to 13311: Factory
 Blocks 13312 to 17407: FIP
 ```
 Finally, we flash our TF-A build outputs into the matching partitions with `dd`.
 We add this callPackage derivation into our `default.nix`:
 ```
 {
  pkgs ? import <nixpkgs> {
    crossSystem = (import <nixpkgs/lib>).systems.examples.aarch64-multiplatform;
  },
 }:
 rec {
  ubootBpiR4 = pkgs.callPackage ./u-boot { };
  armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; };
  image = pkgs.callPackage ./image { inherit armTrustedFirmwareBpiR4; };
 }
 ```
 And we can build our boot sequence components with `nix-shell -A image` to get our `nixos-r4-image.img`.
 ## Flashing and running
 So let's flash the resulting image into an SD card.
 ```
 dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress
 ```
 After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot.
 We get the following logs with `minicom`:
 ```
 F0: 102B 0000
 FA: 1042 0000
 FA: 1042 0000 [0200]
 F9: 1041 0000
 F3: 1001 0000 [0200]
 F3: 1001 0000
 F6: 380E 5800
 F5: 0000 0000
 V0: 0000 0000 [0001]
 00: 0000 0000
 BP: 0600 0041 [0000]
 G0: 1190 0000
 EC: 0000 0000 [3000]
 MK: 0000 0000 [0000]
 T0: 0000 0221 [0101]
 Jump to BL
 NOTICE:  BL2: v2.10.0   (release):
 NOTICE:  BL2: Built : 00:00:00, Jan  1 1980
 NOTICE:  WDT: Cold boot
 NOTICE:  WDT: disabled
 NOTICE:  CPU: MT7988
 NOTICE:  EMI: Using DDR unknown settings
 NOTICE:  EMI: Detected DRAM size: 4096 MB
 NOTICE:  EMI: complex R/W mem test passed
 NOTICE:  BL2: Booting BL31
 NOTICE:  BL31: v2.10.0  (release):
 NOTICE:  BL31: Built : 00:00:00, Jan  1 1980
 U-Boot 2024.04 (Apr 02 2024 - 10:58:58 +0000)
 CPU:   MediaTek MT7988
 Model: mt7988-rfb
 DRAM:  4 GiB
 Core:  49 devices, 19 uclasses, devicetree: separate
 MMC:   mmc@11230000: 0
 Loading Environment from nowhere... OK
 In:    serial@11000000
 Out:   serial@11000000
 Err:   serial@11000000
 Net:
 Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06
 eth0: ethernet@15100000
 BPI-R4>
 ```
 And here we have our initial access to the U-Boot console !
 ## Sources
 I succesfully got to this point after myself reading some documentation and a bunch of blog posts.
 Those where the resources I used:
 - [frank-w's dokuwiki](https://www.fw-web.de/dokuwiki/doku.php?id=en:bpi-r4:start)
 - [Kasper Kondzielski's blog post about running NixOS on the BPI-R3](https://github.com/ghostbuster91/blogposts/blob/main/router2023/main.md)
 - [Weijie Gao's post on the Banana Pi Discource on U-Boot and TF-A on Mediatek SoC](ihttps://forum.banana-pi.org/t/tutorial-build-customize-and-use-mediatek-open-source-u-boot-and-atf/13785)
 - [*Understanding ARM Trusted Firmware using QEMU* by Hemanth Nandish](https://lnxblog.github.io/2020/08/20/qemu-arm-tf.html)
--- a/default.nix
+++ b/default.nix
@ -0,0 +1,18 @@
 { stdenv, lib, hugo, ... }:
 stdenv.mkDerivation {
  name = "dalaran.fr";
  src = lib.cleanSource ./.;
  nativeBuildInputs = [
    hugo
  ];
  buildPhase = ''
    hugo --minify
  '';
  installPhase = ''
    mv public $out
  '';
 }
--- a/flake.nix
+++ b/flake.nix
@ -0,0 +1,11 @@
 {
  description = "Victor Mignot's blog";
  inputs = { };
  outputs = { self, ... }: {
    overlays.default = final: prev: {
      dalaran-fr-blog = prev.callPackage ./default.nix { };
    };
  };
 }
--- a/shell.nix
+++ b/shell.nix
@ -2,6 +2,11 @@
 pkgs.mkShell {
  name = "hugo-shell";
  packages = with pkgs; [
    nixfmt-rfc-style
    nil
  ];
  nativeBuildInputs = with pkgs; [
    hugo
  ];
Author	SHA1	Message	Date
Victor Mignot	dcc46dc4e0	[Draft] Running NixOS on Banana Pi R4: Building the boot sequence	2024-09-29 21:12:12 +02:00
Victor Mignot	3b14e4662e	remove hugo's build lock and add flake overlay	2024-09-29 21:08:46 +02:00