diff --git a/content/posts/nixos-bpi-r4/2024-09-29-uboot-console.md b/content/posts/nixos-bpi-r4/2024-09-29-uboot-console.md new file mode 100644 index 0000000..5ec317d --- /dev/null +++ b/content/posts/nixos-bpi-r4/2024-09-29-uboot-console.md @@ -0,0 +1,406 @@ ++++ +title = 'Running NixOS on the Banana Pi R4: Building the boot sequence' +date = 2024-09-29T08:37:23+02:00 +draft = true ++++ + +# Before starting + +- I'm by no mean an expert about U-Boot, neither in ARM CPUs. +This is just me trying to port NixOS on the Banana Pi R4, and sharing what I learned. +As such, I may (and certainly will) utter false statements. +If I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it ! +- I can't say really that I'm fluent in English. If you see some grammar or spelling mistakes, please reach me too ! +- This post's goal is not to be exhaustive, but I want to emphasize what I think is important or useful to know. +My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material. + +Thanks for your comprehension, and have a great time reading this. + +# How did we get there + +It has been a while since I started waiting for the Banana Pi R4. +I bought it as soon as a complete bundle with the Wi-Fi 7 module was available. +I initially planned to run OpenWrt on it, without bothering too much. +But if my machines (including my ROCKPro64) are all running NixOS with my +[common flake configuration](https://git.dalaran.fr/dala/nixos-config), there is no reason why my router +should not. +This is because I use a [colmena](https://github.com/zhaofengli/colmena)-based deployment system. +It allows me to build (and cross-compile) all my machines configurations and to deploy them from my PC. + +However, there are two caveats here: +- The Nix store usually takes some place on storage devices, so the embedded 8 GB eMMC might not be enough. +But, since the BPI R4 has an integrated slot for a NVMe SSD and I have an empty 500 GB SSD available, +it is way more than enough. +- There is currently no support or available image of NixOS for the BPI R4. + +This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4. + +For your information, this is what I had before starting this project: +- Some basic Nix knowledge. +- I already used U-Boot on some boards with prebuilt Linux images, but never really installed it or tweaked it myself. +- Some knowledge about basic Linux utilities used for embedded. +- A USB-Serial cable. + +For a NixOS SD image generation, some people chose to build it with their Nix configuration already +applied to it, and then just flash this image to their board flash device. +I will personally just build an image that just serve as an installation media, and then manually do +all the installation process as I would do on a PC. + +Today's post is the first part about building our boot components and getting access to the U-Boot console. + +Let's get started ! + +# The ARMv8 boot sequence + +Before focusing on the Linux kernel or NixOS, we will have to build the boot sequence for the R4. +This includes ARM TrustedFirmware-A and U-Boot. + +On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that will be run to init our board) +is standardized with the following elements: +- BL1: this is a software brick that is already present in the board's Boot ROM. It performs basic +hardware initialization and then pass the control to the next step (BL2). +- BL2: Another software brick whose goal is to create a secure environment for the software that will be run, and then +pass the control to the last step. +- BL3: the final element of the boot sequence. It is basically what the board will finally run. +In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software), +an optional BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run. + +It seems in reality a bit more complicated than that, dealing with ARM Exception Level, but we will stick +to this simple representation. + +ARM TrustedFirmware-A provides a standard implementation for BL2, BL31 and BL32. Its documentation is available +[here](https://trustedfirmware-a.readthedocs.io/en/latest/index.html). +U-Boot is the bootloader which is the most commonly used on ARM boards, and that will act as BL33. +It has all the necessary utilities to load and boot the Linux kernel. + +Once the board reach U-Boot, it will be able to boot NixOS. +Nixpkgs provide two functions to build these two elements as Nix derivations: `buildUBoot` and `buildArmTrustedFirmware`. +These two functions make the creation of such derivations fairly easy. + +# Cross compilation on NixOS (for Nix-beginners) + +If you're not on an aarch64 machine (or did not enable native build through QEMU on your configuration), +you won't be able to build the following derivations. +To do so, we have to set up cross-compilation. +Everybody knows nixpkgs as a collection of Nix derivations that contains some utility functions like `mkDerivation` too. +It also provides a cross-compilation system that allows to cross compile each derivation for a compatible architecture ! + +For example, if we want to build any aarch64 package from any architecture (like `hello`), we can just run: +```bash +nix-build '' --arg crossSystem '(import ).systems.examples.aarch64-multiplatform' -A hello +``` +This will cross-compile `hello` for aarch64 in the Nix store. +It is possible, because as any package in nixpkgs, `hello` is declared through a Nix recipe (a callPackage derivation) that +is called by the `callPackage` function of nixpkgs. +This function setup a bunch of things and among them cross-compilation. +It does so by looking at the `crossSystem` parameter provided to nixpkgs. +It will then cross-compile every dependency of the derivation before finally building our package. + +To cross-compile every program we need, we just have to create the following `default.nix` file: +```nix +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +{ + myProgram = pkgs.callPackage ./myProgram { }; +} +``` +Here, we just need to have a `myProgram` directory with a `default.nix` file containing a callPackage derivation +for our program, and then call `nix-build -A myProgram`. + +# Building U-Boot + +Most of current Linux images for the BPI R4 uses [frank-w's U-Boot fork](https://github.com/frank-w/u-boot). +This is because, for now, mainline U-Boot does not support the router. +His modifications to bring support for our device are just few commits that I'll export as patch, and apply +them on top of mainline U-Boot. + +Let's create the following `default.nix` file: +```nix +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +{ + ubootBpiR4 = pkgs.callPackage ./u-boot { }; +} +``` + +In a `u-boot` directory, we will create a `default.nix` file and a `patches` directory that will contain +all the patches that I took from frank-w's U-Boot. +Within the `u-boot/default.nix` file, we will just have to write the following callPackage derivation: +```nix +{ buildUBoot, ... }: +buildUBoot { + defconfig = "mt7988a_bpir4_sd_defconfig"; + extraMeta.platforms = [ "aarch64-linux" ]; + extraPatches = [ + ./patches/0001-pci-mediatek-add-PCIe-controller-support-for-Filogic.patch + ./patches/0002-Fix-PCIE-on-BPIR4.patch + ./patches/0003-arm-dts-enable-pcie-in-sd-dts-too.patch + ./patches/0004-dts-r4-disable-pcie2-in-emmc-dts.patch + ./patches/0005-defconfig-uEnv-add-defconfigs-and-environment-files.patch + ./patches/0006-defconfig-r4-update-with-pcie-options.patch + ./patches/0007-defconfig-r4-add-pstore.patch + ./patches/0008-defconfig-r4-update-emmc-defconfig.patch + ./patches/0009-defconfig-r4-fix-duplicates-in-emmc-defconfig.patch + ./patches/0010-arm64-dts-move-pcie-phy-to-dedicated-xsphy-no-driver.patch + ./patches/0011-pci-mediatek-print-controller-address-for-card-detec.patch + ]; + + extraConfig = '' + CONFIG_FIT=n + CONFIG_USE_DEFAULT_ENV_FILE=n + ''; + + filesToInstall = [ "u-boot.bin" ]; +} +``` + +As I said previously, nixpkgs provides a `buildUBoot` function, to which we have just to pass some arguments. +Here, frank-w's U-Boot patches define a new defconfig called `mt7988a_bpi_r4_sd_defconfig` for the R4. +This config enables the generation of a Flattened Image Tree. +As far as I understand, this is a blob containing all the necessary boot configurations and files that U-Boot needs to launch our kernel. +In my case, at least for the beginning, I chose to stick with the plain-old U-Boot builds (without using FIT) to learn to do it myself instead of using +any special packaging format. +In the end, the `filesToInstall` arguments just specifies which final build products we want to keep in the derivation output. +For now, we will only keep the u-boot binary and see if we actually need anything else later. +That's it for U-Boot ! We can try to build it with `nix-build -A ubootBpiR4` and see the `u-boot.bin` file +appear in the Nix store, and in the `result` folder ! + +# ARM TrustedFirmware-A + +As with U-Boot, the mainline version of TrustedFirmware does not support the BPI-R4. +For now, distributions are using [this fork](https://github.com/mtk-openwrt/arm-trusted-firmware). +I don't really know for sure who is behind this account, but there is a bit too much commits for me +to export them as patch to apply on top of mainline. +Once again, we will use the nixpkgs's `buildArmTrustedFirmware` function. +Before that, and as we can see on the [frank-w's U-Boot build scripts](https://github.com/frank-w/u-boot/blob/mtk-atf/build.sh#L40), +we will need to pass to U-Boot binary. It should be included within the final `fip.bin` file. + +--- + +To be honest there, I don't really know if TrustedFirmware is mandatory to run U-boot +on our SoC, or if U-Boot by itself could have handled everything. +But for now, I decided to proceed like the OpenWrt and Debian images. +I will surely dig in the TrustedFirmware and ARM documentations later. +I'll eventually to a follow-up post later if I achieve to figure out how everything is working. + +--- + +So, let's create our callPackage derivation in `trusted-firmware/default.nix`: +```nix +{ + buildArmTrustedFirmware, + fetchFromGitHub, + dtc, + ubootBpiR4, + ubootTools, + openssl, + ... +}: +(buildArmTrustedFirmware rec { + platform = "mt7988"; + extraMeta.platforms = [ "aarch64-linux" ]; + extraMakeFlags = [ + "USE_MKIMAGE=1" + "BOOT_DEVICE=sdmmc" + "DRAM_USE_COMB=1" + "BL33=${ubootBpiR4}/u-boot.bin" + "all" + "fip" + ]; + + filesToInstall = [ + "build/${platform}/release/bl2.img" + "build/${platform}/release/fip.bin" + ]; +}).overrideAttrs (old: { + src = fetchFromGitHub { + owner = "mtk-openwrt"; + repo = "arm-trusted-firmware"; + rev = "bacca82a8cac369470df052a9d801a0ceb9b74ca"; + hash = "sha256-n5D3styntdoKpVH+vpAfDkCciRJjCZf9ivrI9eEdyqw="; + }; + version = "2.10.0-mtk"; + nativeBuildInputs = old.nativeBuildInputs ++ [ dtc ubootTools openssl ]; +} +``` + +This is very similar what we have done with U-Boot, we select our platform, then pass the necessary flags for compilation, +and then grab the two output files that we need. +However, in our situation, there are some extra steps. +Both `buildUBoot` and `buildArmTrustedFirmware` assumes that you're building the mainline U-Boot and TF-A. +It was the case with our U-Boot build to which we just add some extra patches. +However, with TF-A, we will here use the mtk-openwrt's fork. +So we need to override the derivation produced by `buildArmTrustedFirmware` to pass our own sources and needed dependencies thanks +to the `overrideAttrs` function. + +As we can see, this callPackage derivation needs to access our previous `ubootBpiR4` derivation. +All the other derivations arguments are provided by nixpkgs itself through the `callPackage` function. +However, `ubootBpiR4` is itself not present within nixpkgs, we have to manually pass it. +There are two methods to do so, the first one is to use a nixpkgs overlay. +The other one is to pass it manually as a `callPackage` argument. +That's what we'll do here. +This makes our `default.nix` file looks like that: +```nix +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +rec { + ubootBpiR4 = pkgs.callPackage ./u-boot { }; + armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; }; +} +``` + +Now, we can run `nix-build -A armTrustedFirmwareBpiR4` to build everything we need. + +# Side note on the U-Boot and TF-A combination + +If you check how to build the full boot sequence binaries for various board, you will maybe feel like me +that it seems to be highly dependent on the SoC manufacturer. +In our case, we build U-Boot first and then pass it to TF-A as BL33. +For some Allwinner SoCs, TF-A is built first and then passed to U-boot as BL31. +For Rockchip based boards there is this whole `idbloader.img` due to Rockchip's miniloader. +At this point, I don't really know if there is a "standard way" to build to whole boot sequence for each SoC. +I don't even know what causes these differences in the build process between manufacturers. +I guess it's related to the manufacturer implementation, or maybe things that could be done both in U-Boot and in TF-A +and then depends on where the manufacturer implemented them ? + +# Create our boot image + +The final step is just to create an empty image and flash what we have built so far at the matching address ranges. +This will be done with a simple bash script derivation in the `image/default.nix` file. +```nix +{ + runCommand, + armTrustedFirmwareBpiR4, + gptfdisk, + ... +}: +runCommand "bpi-r4-image" { + nativeBuildInputs = [ + gptfdisk + ]; +} '' + IMAGE=$out/nixos-r4-image.img + + mkdir $out + + dd if=/dev/zero of=$IMAGE bs=1M count=4000 + sgdisk -o $IMAGE + sgdisk -a 1 -n 1:34:8191 -A 1:set:2 -t 1:8300 -c 1:"bl2" $IMAGE + sgdisk -a 1 -n 2:8192:9215 -A 2:set:63 -t 2:8300 -c 2:"u-boot-env" $IMAGE + sgdisk -a 1 -n 3:9216:13311 -A 3:set:63 -t 3:8300 -c 3:"factory" $IMAGE + sgdisk -a 1 -n 4:13312:17407 -A 4:set:63 -t 4:8300 -c 4:"fip" $IMAGE + + dd if=${armTrustedFirmwareBpiR4}/bl2.img of=$IMAGE seek=34 conv=notrunc,fsync + dd if=${armTrustedFirmwareBpiR4}/fip.bin of=$IMAGE seek=13312 conv=notrunc,fsync +'' +``` + +The nixpkgs `runCommand` function creates a derivation that runs a bash script. +This bash script is running into the nixpkgs standard environment, but we can override this environment with the set passed +as the second argument. +In this case, we add the `gptfdisk` package as a dependency to use `sgdisk` within our bash script. +The third argument is the script in itself. +We then create the derivation output directory, and generate a zero-filled 4 GB image with `dd`. +Then, we generate our partition table with `sgdisk` with the following partition map: +``` +Block size of 512 bytes + +Blocks 34 to 8191: BL2 +Blocks 8192 to 9215: Space for U-Boot environment variables +Blocks 9216 to 13311: Factory +Blocks 13312 to 17407: FIP +``` +Finally, we flash our TF-A build outputs into the matching partitions with `dd`. + +We add this callPackage derivation into our `default.nix`: +```nix +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +rec { + ubootBpiR4 = pkgs.callPackage ./u-boot { }; + armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; }; + image = pkgs.callPackage ./image { inherit armTrustedFirmwareBpiR4; }; +} +``` + +And we can build our boot sequence components with `nix-shell -A image` to get our `nixos-r4-image.img`. + +# Flashing and running + +So let's flash the resulting image into an SD card. +```bash +dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress +``` + +After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot. +We get the following logs with `minicom`: +``` +F0: 102B 0000 +FA: 1042 0000 +FA: 1042 0000 [0200] +F9: 1041 0000 +F3: 1001 0000 [0200] +F3: 1001 0000 +F6: 380E 5800 +F5: 0000 0000 +V0: 0000 0000 [0001] +00: 0000 0000 +BP: 0600 0041 [0000] +G0: 1190 0000 +EC: 0000 0000 [3000] +MK: 0000 0000 [0000] +T0: 0000 0221 [0101] +Jump to BL + +NOTICE: BL2: v2.10.0 (release): +NOTICE: BL2: Built : 00:00:00, Jan 1 1980 +NOTICE: WDT: Cold boot +NOTICE: WDT: disabled +NOTICE: CPU: MT7988 +NOTICE: EMI: Using DDR unknown settings +NOTICE: EMI: Detected DRAM size: 4096 MB +NOTICE: EMI: complex R/W mem test passed +NOTICE: BL2: Booting BL31 +NOTICE: BL31: v2.10.0 (release): +NOTICE: BL31: Built : 00:00:00, Jan 1 1980 + + +U-Boot 2024.04 (Apr 02 2024 - 10:58:58 +0000) + +CPU: MediaTek MT7988 +Model: mt7988-rfb +DRAM: 4 GiB +Core: 49 devices, 19 uclasses, devicetree: separate +MMC: mmc@11230000: 0 +Loading Environment from nowhere... OK +In: serial@11000000 +Out: serial@11000000 +Err: serial@11000000 +Net: +Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06 +eth0: ethernet@15100000 +BPI-R4> +``` +And here we have our initial access to the U-Boot console ! + +# Sources + +I successfully got to this point after myself reading some documentation and a bunch of blog posts. +Those where the resources I used: +- [frank-w's dokuwiki](https://www.fw-web.de/dokuwiki/doku.php?id=en:bpi-r4:start) +- [Kasper Kondzielski's blog post about running NixOS on the BPI-R3](https://github.com/ghostbuster91/blogposts/blob/main/router2023/main.md) +- [Weijie Gao's post on the Banana Pi Discourse about U-Boot and TF-A on Mediatek SoCs](ihttps://forum.banana-pi.org/t/tutorial-build-customize-and-use-mediatek-open-source-u-boot-and-atf/13785) +- [*Understanding ARM Trusted Firmware using QEMU* by Hemanth Nandish](https://lnxblog.github.io/2020/08/20/qemu-arm-tf.html) +