diff --git a/content/posts/nixos-bpi-r4/2024-10-05-uboot-console.md b/content/posts/nixos-bpi-r4/2024-10-05-uboot-console.md new file mode 100644 index 0000000..1ff26a4 --- /dev/null +++ b/content/posts/nixos-bpi-r4/2024-10-05-uboot-console.md @@ -0,0 +1,411 @@ ++++ +title = 'Running NixOS on the Banana Pi R4: Building the boot sequence' +date = 2024-10-05T08:37:23+02:00 ++++ + +# Before starting + +- I'm by no mean an expert about U-Boot, neither in ARM CPUs. + This is just me trying to port NixOS on the Banana Pi R4, and sharing what I learned. + As such, I may (and certainly will) utter false statements. + If I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it ! +- I can't say really that I'm fluent in English. If you see some grammar or spelling mistakes, please reach me too ! +- This post's goal is not to be exhaustive, but I want to emphasize what I think is important or useful to know. + My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material. + +Thanks for your comprehension, and have a great time reading this. + +# How did we get there + +It has been a while since I started waiting for the Banana Pi R4. +I thought of everything I could do with my own Linux gateway instead of being forced to get along my ISP's. +As soon as the full bundle with the Wi-Fi 7 card was available and the first reviews appeared, my command was already confirmed. +I've been waiting it for nearly a year, and initially though that I would just run OpenWrt on it. +I mean, It is for sure the easiest solution (even if it would without doubt need some tweaks). +However, after a quick thought, I felt like this router would fit even better in my NixOS environment. +I mean, I have a centralized NixOS configuration for all my machines that can be easily built from any of them and be deployed on another +with a single command. +It would make even more sense too cross-compile my router configuration from my PC with its Ryzen 7 5800X and deploy it on the BPI-R4. + +However, there are three caveats with this choice: + +- The Nix store usually takes some place on storage devices, so the embedded 8 GB eMMC might not be enough. + But, since the BPI R4 has an integrated slot for a NVMe SSD and I have an empty 500 GB SSD available, + it is way more than enough. +- I would say (I might be wrong there), that NixOS is not really the best fit for such small home gateways. + Some OpenWrt packages are missing on NixOS, with the most obvious example being LuCI. + I don't think that such packages are mandatory though, they would mainly bring QoL features. +- There is currently no support or available image of NixOS for the BPI R4. + +But I just felt that I would still be more comfortable with another machine in my NixOS environment than +with one outside of it that I'll have to manage its own way. +This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4. + +For your information, this is what I had before starting this project: + +- Some basic Nix knowledge. +- I already used U-Boot on some boards with prebuilt Linux images, but never really installed it or tweaked it myself. +- Some knowledge about basic Linux utilities used for embedded. +- A USB-Serial cable. +- A Banana Pi R4 (of course). + +For a NixOS SD image generation on ARM machines, some people prefer to build it with their Nix configuration already +applied to it, and then just flash into their board. +In my case, I prefer the philosophy of having an image that just serve as an installation media, and then manually do +all the installation process as I would do on a PC. + +Today's post is the first part about building our boot components and getting access to the U-Boot console. + +Let's get started ! + +# The ARMv8 boot sequence + +Before focusing on the Linux kernel or NixOS, we'll have to build the initial boot sequence for the R4. +It would be too easy to just install our bootloader and having an UEFI running it directly **right** ? +The boot phase in ARM-based board is often a bit trickier as it is on x86 hosts. +On our case, we need a firmware setting up the ARM secure environment (which is ARM TrustedFirmware-A) +and a bootloader (U-Boot). + +On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that is run to init our board) +is standardized with the following elements: + +- BL1: this is a software brick that is already present in the board's Boot ROM. It performs basic + hardware initialization and then pass the control to the next step (BL2). +- BL2: Another software brick whose goal is to create a secure environment for the software that will be run, and then + pass the control to the last step. +- BL3: the final element of the boot sequence. It is basically what the board will finally run. + In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software), + an optional BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run. + +It's in fact a bit more complicated than that, but we'll keep this simple representation. + +ARM provides a standard and open implementation for BL2, BL31 and BL32 called TrustedFirmware-A. +Its documentation is available [here](https://trustedfirmware-a.readthedocs.io/en/latest/index.html). +Finally, our BL33 will be U-Boot, a bootloader widely used for ARM SoCs. +Once the board reach U-Boot, it will be able to boot NixOS. + +Luckily, nixpkgs provides two functions to build these two elements as Nix derivations: `buildUBoot` and `buildArmTrustedFirmware`. + +# Cross compilation on NixOS (for Nix-beginners) + + + + + +As I plan to build everything from a x86_64 host, we have to set up a cross-compilation system to build our boot sequence +for ARMv8A. +Once again, nixpkgs make it fairly easy with its embedded cross-compilation system. + +For example, if we want to build any aarch64 package from any architecture (like `hello`), we can just run: + +```bash +nix-build '' --arg crossSystem '(import ).systems.examples.aarch64-multiplatform' -A hello +``` + +It is possible, because as any package in nixpkgs, `hello` is declared through a Nix recipe (a callPackage derivation) that +is called by the `callPackage` function. +This function setup a bunch of things and among them cross-compilation by looking at the `crossSystem` parameter provided to nixpkgs. + +To cross-compile every program we need, we just have to create the following `default.nix` file: + +```nix +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +{ + myProgram = pkgs.callPackage ./myProgram { }; +} +``` + +Here, we just need to have a `myProgram` directory with a `default.nix` file containing a callPackage derivation +for our program, and then call `nix-build -A myProgram`. + +# Building U-Boot + +At the time I'm writing this post, +most of current Linux images for the BPI R4 uses [frank-w's U-Boot fork](https://github.com/frank-w/u-boot). +This is because the BPI-R4 is not supported by mainline U-boot. +The modifications frank-w did to bring this support are just few commits. +We can easily export them as patch and apply them on top of mainline U-Boot. + +Let's create the following `default.nix` file: + +```nix +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +{ + ubootBpiR4 = pkgs.callPackage ./u-boot { }; +} +``` + +In a `u-boot` directory, we will create a `default.nix` file and a `patches` directory that will contain +all the patches that I grabbed from frank-w's U-Boot. +Within the `u-boot/default.nix` file, we will just have to write the following callPackage derivation: + +```nix +{ buildUBoot, ... }: +buildUBoot { + defconfig = "mt7988a_bpir4_sd_defconfig"; + extraMeta.platforms = [ "aarch64-linux" ]; + extraPatches = [ + ./patches/0001-pci-mediatek-add-PCIe-controller-support-for-Filogic.patch + ./patches/0002-Fix-PCIE-on-BPIR4.patch + ./patches/0003-arm-dts-enable-pcie-in-sd-dts-too.patch + ./patches/0004-dts-r4-disable-pcie2-in-emmc-dts.patch + ./patches/0005-defconfig-uEnv-add-defconfigs-and-environment-files.patch + ./patches/0006-defconfig-r4-update-with-pcie-options.patch + ./patches/0007-defconfig-r4-add-pstore.patch + ./patches/0008-defconfig-r4-update-emmc-defconfig.patch + ./patches/0009-defconfig-r4-fix-duplicates-in-emmc-defconfig.patch + ./patches/0010-arm64-dts-move-pcie-phy-to-dedicated-xsphy-no-driver.patch + ./patches/0011-pci-mediatek-print-controller-address-for-card-detec.patch + ]; + + extraConfig = '' + CONFIG_FIT=n + CONFIG_USE_DEFAULT_ENV_FILE=n + ''; + + filesToInstall = [ "u-boot.bin" ]; +} +``` + +The `buildUboot` function will actually grab the source from the U-Boot repo and apply each patch +we give it through the `extraPatches` argument. +It will then copy the `defconfig` file from U-Boot config folder, and apply the `extraConfig` content on top. +It will eventually compile it and grab the files provided through `filesToInstall` from the build artifacts +and put it into the Nix store. + +# ARM TrustedFirmware-A + +Like U-Boot, the mainline version of TrustedFirmware does not currently support the BPI-R4. +For now, Linux distributions are using [this fork](https://github.com/mtk-openwrt/arm-trusted-firmware). +I don't really know for sure who is behind this account, and if I should really trust it. +Nonetheless, there is a bit too many commits for me to manage manually, so we'll here just grab this fork's sources. +As I told earlier, we'll use the nixpkgs's `buildArmTrustedFirmware` function. +Before that, and as we can see on the [frank-w's U-Boot build scripts](https://github.com/frank-w/u-boot/blob/mtk-atf/build.sh#L40), +we've to pass the U-Boot binary as input. +It'll be included within the final `fip.bin` file as BL33. + +______________________________________________________________________ + +To be honest there, I don't really know if TrustedFirmware is mandatory to start our board. +Maybe U-Boot could've been enough, at the expense of not setting up the ARM secure environment. +Maybe U-Boot could itself set up this secure environment. +But for now, I decided to proceed like the OpenWrt and Debian images. +To confirm, I'll surely dig in the TrustedFirmware and ARM documentations later. +I'll eventually do a follow-up post later if I achieve to figure out how everything is working. + +______________________________________________________________________ + +So, let's create our callPackage derivation in `trusted-firmware/default.nix`: + +```nix +{ + buildArmTrustedFirmware, + fetchFromGitHub, + dtc, + ubootBpiR4, + ubootTools, + openssl, + ... +}: +(buildArmTrustedFirmware rec { + platform = "mt7988"; + extraMeta.platforms = [ "aarch64-linux" ]; + extraMakeFlags = [ + "USE_MKIMAGE=1" + "BOOT_DEVICE=sdmmc" + "DRAM_USE_COMB=1" + "BL33=${ubootBpiR4}/u-boot.bin" + "all" + "fip" + ]; + + filesToInstall = [ + "build/${platform}/release/bl2.img" + "build/${platform}/release/fip.bin" + ]; +}).overrideAttrs (old: { + src = fetchFromGitHub { + owner = "mtk-openwrt"; + repo = "arm-trusted-firmware"; + rev = "bacca82a8cac369470df052a9d801a0ceb9b74ca"; + hash = "sha256-n5D3styntdoKpVH+vpAfDkCciRJjCZf9ivrI9eEdyqw="; + }; + version = "2.10.0-mtk"; + nativeBuildInputs = old.nativeBuildInputs ++ [ dtc ubootTools openssl ]; +} +``` + +This is very similar what we have done with U-Boot, we select our platform, then pass the necessary flags for compilation, +and then grab the two output files that we need. +However, in our situation, there are some extra steps. +Both `buildUBoot` and `buildArmTrustedFirmware` assumes that you're building the mainline U-Boot and TF-A. +It was the case with our U-Boot build to which, but not for our TF-A. +So we need to override the derivation produced by `buildArmTrustedFirmware` to set the `src` argument ourselves +and pass supplementary build dependencies in `nativeBuildInputs`. + +As we can see, this callPackage derivation needs to access our previous `ubootBpiR4` derivation. +All the other derivations arguments are provided by nixpkgs itself through the `callPackage` function, +but not `ubootBpiR4` that we declared ourselves. +Given that, we can just easily pass it as is in the `callPackage` argument. +This makes our `default.nix` file looks like the following: + +```nix +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +rec { + ubootBpiR4 = pkgs.callPackage ./u-boot { }; + armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; }; +} +``` + +Now, we can run `nix-build -A armTrustedFirmwareBpiR4` to build everything we need ! + +# Create our boot image + +The final step is just to create an empty image and flash what we have built so far at the matching address ranges. +This will be done with a simple bash script derivation in the `image/default.nix` file. + +```nix +{ + runCommand, + armTrustedFirmwareBpiR4, + gptfdisk, + ... +}: +runCommand "bpi-r4-image" { + nativeBuildInputs = [ + gptfdisk + ]; +} '' + IMAGE=$out/nixos-r4-image.img + + mkdir $out + + dd if=/dev/zero of=$IMAGE bs=1M count=4000 + sgdisk -o $IMAGE + sgdisk -a 1 -n 1:34:8191 -A 1:set:2 -t 1:8300 -c 1:"bl2" $IMAGE + sgdisk -a 1 -n 2:8192:9215 -A 2:set:63 -t 2:8300 -c 2:"u-boot-env" $IMAGE + sgdisk -a 1 -n 3:9216:13311 -A 3:set:63 -t 3:8300 -c 3:"factory" $IMAGE + sgdisk -a 1 -n 4:13312:17407 -A 4:set:63 -t 4:8300 -c 4:"fip" $IMAGE + + dd if=${armTrustedFirmwareBpiR4}/bl2.img of=$IMAGE seek=34 conv=notrunc,fsync + dd if=${armTrustedFirmwareBpiR4}/fip.bin of=$IMAGE seek=13312 conv=notrunc,fsync +'' +``` + +The nixpkgs `runCommand` function creates a derivation that runs a bash script into nixpkgs's standard environment. +We can add dependencies with the second `runCommand` argument. +In this case, we'll add the `gptfdisk` package as a dependency to use `sgdisk` within our script. +Finally, the third argument is the script in itself. +For the script in itself, we'll create the derivation output directory and generate a zero-filled 4 GB image with `dd`. +Then, we want to generate our partition table with `sgdisk` with the following partition map: + +``` +Block size of 512 bytes + +Blocks 34 to 8191: BL2 +Blocks 8192 to 9215: Space for U-Boot environment variables +Blocks 9216 to 13311: Factory +Blocks 13312 to 17407: FIP +``` + +Finally, we flash our TF-A build outputs into the matching partitions with `dd`. + +We add this callPackage derivation into our `default.nix`: + +```nix +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +rec { + ubootBpiR4 = pkgs.callPackage ./u-boot { }; + armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; }; + image = pkgs.callPackage ./image { inherit armTrustedFirmwareBpiR4; }; +} +``` + +And we can build our boot sequence components with `nix-shell -A image` to get our `nixos-r4-image.img`. + +# Flashing and running + +The only thing left to do is flashing our image into a SD card. + +```bash +dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress +``` + +After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot. +We get the following logs with `minicom`: + +``` +F0: 102B 0000 +FA: 1042 0000 +FA: 1042 0000 [0200] +F9: 1041 0000 +F3: 1001 0000 [0200] +F3: 1001 0000 +F6: 380E 5800 +F5: 0000 0000 +V0: 0000 0000 [0001] +00: 0000 0000 +BP: 0600 0041 [0000] +G0: 1190 0000 +EC: 0000 0000 [3000] +MK: 0000 0000 [0000] +T0: 0000 0221 [0101] +Jump to BL + +NOTICE: BL2: v2.10.0 (release): +NOTICE: BL2: Built : 00:00:00, Jan 1 1980 +NOTICE: WDT: Cold boot +NOTICE: WDT: disabled +NOTICE: CPU: MT7988 +NOTICE: EMI: Using DDR unknown settings +NOTICE: EMI: Detected DRAM size: 4096 MB +NOTICE: EMI: complex R/W mem test passed +NOTICE: BL2: Booting BL31 +NOTICE: BL31: v2.10.0 (release): +NOTICE: BL31: Built : 00:00:00, Jan 1 1980 + + +U-Boot 2024.04 (Apr 02 2024 - 10:58:58 +0000) + +CPU: MediaTek MT7988 +Model: mt7988-rfb +DRAM: 4 GiB +Core: 49 devices, 19 uclasses, devicetree: separate +MMC: mmc@11230000: 0 +Loading Environment from nowhere... OK +In: serial@11000000 +Out: serial@11000000 +Err: serial@11000000 +Net: +Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06 +eth0: ethernet@15100000 +BPI-R4> +``` + +And here we have our initial access to the U-Boot console ! + +# Sources + +I successfully got to this point after myself reading some documentation and a bunch of blog posts. +Those where the resources I used: + +- [frank-w's dokuwiki](https://www.fw-web.de/dokuwiki/doku.php?id=en:bpi-r4:start) +- [Kasper Kondzielski's blog post about running NixOS on the BPI-R3](https://github.com/ghostbuster91/blogposts/blob/main/router2023/main.md) +- [Weijie Gao's post on the Banana Pi Discourse about U-Boot and TF-A on Mediatek SoCs](ihttps://forum.banana-pi.org/t/tutorial-build-customize-and-use-mediatek-open-source-u-boot-and-atf/13785) +- [*Understanding ARM Trusted Firmware using QEMU* by Hemanth Nandish](https://lnxblog.github.io/2020/08/20/qemu-arm-tf.html)