diff --git a/content/posts/nixos-bpi-r4/2024-09-29-uboot-console.md b/content/posts/nixos-bpi-r4/2024-09-29-uboot-console.md new file mode 100644 index 0000000..0c76cbc --- /dev/null +++ b/content/posts/nixos-bpi-r4/2024-09-29-uboot-console.md @@ -0,0 +1,476 @@ ++++ +title = 'Running NixOS on the Banana Pi R4: Building the boot sequence' +date = 2024-09-29T08:37:23+02:00 +draft = true ++++ + +## Trigger Warnings + +- I'm by no mean an expert about U-Boot, neither in ARM CPUs. + This is just me trying to accomplish a goal, and sharing what I think I have learned. + As such, I may (and certainly will) utter false statements. + If you see that I'm wrong about something, you can reach me by mail or on fedi and I will gladly correct it ! +- I can't say that I'm fluent in English. If you see some grammar or spelling mistakes, please reach me too ! +- This post's goal is to be exhaustive. + My goal here is that anyone, even with only basic knowledge about Nix and embedded systems, could use this as material. + +Thanks for your comprehension, and have a great time reading this. + +## How did we get there + +It has been a while since I started waiting for the Banana Pi R4. +I bought it as soon as a full bundle with the WiFi 7 module was available. +The goal initially was to run OpenWRT on it, without bothering too much. +But then I tought that if all my machines (including my ROCKPro64) are running NixOS with my +[common flake configuration](https://git.dalaran.fr/dala/nixos-config), there is no reason why my router +should not. +This is because I use a [colmena](https://github.com/zhaofengli/colmena)-based deployment system. +It allows me to build (and cross-compile) all my machines configurations and to deploy them from my PC. + +However, there is two caveats here: +- The Nix store usually takes some place on storage devices, so the embedded 8GB eMMC might not be enough. +But, as the BPI R4 has an integrated slot for a NVME SSD, and since I had an empty 500 GB SSD available, +it is way more than enough. +- There is currently no support or available image of NixOS for the BPI R4. + +This whole blog post is about me feeling confident enough to create my own NixOS image for the BPI R4. + +For your information, this is what I had before starting this project: +- Basic Nix knowledge. +- I already used U-Boot on some boards with prebuilt images, but never really installed it or tweaked it myself. +- Some knowledge about basic Linux utilities used for embedded. +- A USB-Serial cable. + +I saw that for the image generation, some people chose to build an image with their Nix configuration already +applied to it, and then just flash this image to their board flash device. +In my case, I would like to stick with an image that just serve as an install media, and then manually do +all the installation process as I would do on a PC. + +As much as possible, I would like to stick with mainline U-Boot and Linux kernel. +I will try to grab any useful patches from R4 specific forks and apply them manually. + +Today's post is the first part about building our boot components and getting access to the u-Boot console. + +Let's get started ! + +# Initial checks + +The BPI-R4 should be shipped with an existing install of OpenWRT on the NAND flash. +If we set the boot-device selection jumper to NAND Flash and connect our USB-Serial cable, we should see boot logs. +So after doing this and starting `minicom`, I got the following: +``` +ubi0: attaching mtd6 +ubi0: scanning is finished +ubi0: attached mtd6 (name "ubi", size 112 MiB) +ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes +ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048 +ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096 +ubi0: good PEBs: 900, bad PEBs: 0, corrupted PEBs: 0 +ubi0: user volume: 3, internal volumes: 1, max. volumes count: 128 +ubi0: max/mean erase counter: 2/0, WL threshold: 4096, image sequence number: 1697192243 +ubi0: available PEBs: 0, total reserved PEBs: 900, PEBs reserved for bad PEB handling: 19 +No size specified -> Using max size (4190208) +Read 4190208 bytes from volume kernel to 0000000046000000 +## Loading kernel from FIT Image at 46000000 ... + Using 'config-1' configuration + Trying 'kernel-1' kernel subimage + Description: ARM64 OpenWrt Linux-5.4.246 + Type: Kernel Image + Compression: lzma compressed + Data Start: 0x460000e8 + Data Size: 4061787 Bytes = 3.9 MiB + Architecture: AArch64 + OS: Linux + Load Address: 0x48080000 + Entry Point: 0x48080000 + Hash algo: crc32 + Hash value: ef1d4754 + Hash algo: sha1 + Hash value: 78589b5e23500626a36c8a2ff523eb8bbcaddabc + Verifying Hash Integrity ... crc32+ sha1+ OK +## Loading fdt from FIT Image at 46000000 ... + Using 'config-1' configuration + Trying 'fdt-1' fdt subimage + Description: ARM64 OpenWrt BPI-R4-NAND device tree blob + Type: Flat Device Tree + Compression: uncompressed + Data Start: 0x463dfc80 + Data Size: 34487 Bytes = 33.7 KiB + Architecture: AArch64 + Hash algo: crc32 + Hash value: 0c548bfa + Hash algo: sha1 + Hash value: 0fb7882fc5ef9acba18b6864234dd1aec0b3d2f2 + Verifying Hash Integrity ... crc32+ sha1+ OK + Booting using the fdt blob at 0x463dfc80 +Working FDT set to 463dfc80 + Uncompressing Kernel Image + Loading Device Tree to 00000000ff7ec000, end 00000000ff7f76b6 ... OK +Working FDT set to ff7ec000 +Add 'ramoops@42ff0000' node failed: FDT_ERR_EXISTS + +Starting kernel ... +``` + +Okay, the board is working as expected and theses logs may serve as a reference later. +I also quickly checked the hardware revision of my R4. +In the first revisions, a resistor on the PCB caused I2C devices to be unusable when using an NVME SSD.[0] +In my case, the resistor was no longer there, so I should not face any issue. + +[0]: https://forum.banana-pi.org/t/bpi-r4-nvme-i2c/17152 + +## The ARMv8 boot sequence + +Before focusing on the Linux kernel or NixOS, we will have to build the boot sequence for our R4. +In our case, this includes ARM TrustedFirmware-A and U-Boot. + +On 64-bits Cortex-A based SoCs, the boot sequence (the succession of programs that will be run to start our board) +is standardized with the following elements: +- BL1: this is a software brick that is already present in the board Boot ROM. Its goals are to perform basic +hardware initialization and then pass the control to the next step (BL2). +To do so, it expects to find the next step at a specific address in RAM. +- BL2: another software brick whose goal is to create a secure environment for the software that will be run, and then +pass the control to the last step. +- BL3: the final element of the boot sequence. It is basically what the board will finally run. +In fact, this is a bit more complicated as BL3 itself is subdivided between BL31 (Secure runtime software), +an optionnal BL32 and BL33 (Non-trusted Firmware) which is the final software that will be run. + +It seems in reality a bit more complicated than that, dealing with ARM Exception Level, but we will stick +to this simple representation. + +ARM TrustedFirmware-A provides a standard implementation for BL2 and BL31. Its documentation is available +[here](https://trustedfirmware-a.readthedocs.io/en/latest/index.html). +U-Boot is the bootloader which is the most commonly used on ARM boards, and that correspond here to BL33. +It has all the necessary utilities to load and boot the Linux kernel. + +This is all the two elements that we will have to build and add to our image to then be able to boot NixOS. +Nixpkgs provide two functions to build these two elements as Nix derivations: `buildUBoot` and `buildArmTrustedFirmware`. +This make the creation of such derivations fairly easy. + +## Cross compilation on NixOS (for Nix-beginners) + +If you're not on an aarch64 machine (or did not enable native build through QEMU on your configuration), +you won't be able to build the following derivations. +To do so, we have to setup cross-compilation. +Everybody knows Nixpkgs as a collection of Nix derivations that it contains some utility functions like `mkDerivation`. +Nixpkgs also provides a cross-compilation system that allows to cross compile each derivation for a compatible architecture ! + +For example, if we want to build any aarch64 package from any architecture (like GNU hello), we can just do: +``` +nix-build '' --arg crossSystem '(import ).systems.examples.aarch64-multiplatform' -A hello +``` +This will cross-compile hello for aarch64 in the Nix store, and set a symlink to it in a `result` directory in the current one. +It is possible, because as any package in Nixpkgs, GNU hello is declared through a Nix recipe (called a callPackage derivation) that +is called by the `callPackage` function of Nixpkgs. +This function setup a bunch of things, and among them cross-compilation. +It does so by looking at the `crossSystem` parameters provided to Nixpkgs when importing it. +It will then cross-compile every dependency of the derivation before finally building our package. + +So here, to cross-compile each of our program we need, we just have to create the following `default.nix` file: +``` +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +{ + myProgram = pkgs.callPackage ./myProgram { }; +} +``` +Here, we just need to have a `myProgram` directory with a `default.nix` file containing a callPackage derivation +for our program, and then call `nix-build -A myProgram`. + +## Building U-Boot + +We should be ready to build a mainline U-Boot. However, the latter does not support the BPI R4 for now. +Most of current Linux images for the BPI R4 uses [frank-w's U-Boot fork](https://github.com/frank-w/u-boot). +His modifications to bring support for our device are just few commits that I'll export as patch, and apply +them on top of mainline U-Boot. + +Let's create the following `default.nix` file: +``` +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +{ + ubootBpiR4 = pkgs.callPackage ./u-boot { }; +} +``` + +In a `u-boot` directory, we will create a `default.nix` file and a `patches` directory that will contains +all the patches that I took from frank-w's U-Boot. +Within the `u-boot/default.nix` file, we will just have to write the following callPackage derivation: +``` +{ buildUBoot, ... }: +buildUBoot { + defconfig = "mt7988a_bpir4_sd_defconfig"; + extraMeta.platforms = [ "aarch64-linux" ]; + extraPatches = [ + ./patches/0001-pci-mediatek-add-PCIe-controller-support-for-Filogic.patch + ./patches/0002-Fix-PCIE-on-BPIR4.patch + ./patches/0003-arm-dts-enable-pcie-in-sd-dts-too.patch + ./patches/0004-dts-r4-disable-pcie2-in-emmc-dts.patch + ./patches/0005-defconfig-uEnv-add-defconfigs-and-environment-files.patch + ./patches/0006-defconfig-r4-update-with-pcie-options.patch + ./patches/0007-defconfig-r4-add-pstore.patch + ./patches/0008-defconfig-r4-update-emmc-defconfig.patch + ./patches/0009-defconfig-r4-fix-duplicates-in-emmc-defconfig.patch + ./patches/0010-arm64-dts-move-pcie-phy-to-dedicated-xsphy-no-driver.patch + ./patches/0011-pci-mediatek-print-controller-address-for-card-detec.patch + ]; + + extraConfig = '' + CONFIG_FIT=n + CONFIG_USE_DEFAULT_ENV_FILE=n + ''; + + filesToInstall = [ "u-boot.bin" ]; +} +``` + +As I said previously, Nixpkgs provides a `buildUBoot` function, to which we have just to path some args. +Here, frank-w's U-Boot patches define a new defconfig called `mt7988a_bpi_r4_sd_defconfig` for the R4. +This config enable by default the use of a FIT (Flattened Image Tree). +As far as I understood, this allows the creation of a blob containing all the necessary boot info that U-Boot needs instead of loading a couple of files. +In my case, at least for the beginning, I chose to stick with the plain-old U-Boot builds to learn to do it myself instead of using +any special packaging format. +In the end, the `filesToInstall` arguments just tells which final build products we want to get in the derivation output. +For now, we will only keep the u-boot binary and see if we actually need anything else later. +That's it for U-Boot ! We can try to build it with `nix-build -A ubootBpiR4` and see the `u-boot.bin` file +appear in the Nix store, and in the `result` folder ! + +## ARM TrustedFirmware-A + +As with U-Boot, the mainline version of TrustedFirmware does not support the BPI-R4. +For now, distributions are using [this fork](https://github.com/mtk-openwrt/arm-trusted-firmware). +I don't really know who for sure who is behind this account, but there is a bit too much commits for me +to export them as patch. +So even if I told that I would like to stick with mainline, for now we will still using this fork directly. +Once again, we will use the Nixpkgs's `buildArmTrustedFirmware` function. +Before that, and as we can see on the [frank-w's U-Boot build scripts](https://github.com/frank-w/u-boot/blob/mtk-atf/build.sh#L40), +we will need to pass to U-Boot binary. It should be included within the final `fip.bin` file. + +--- +To be honest there, I don't really know if TrustedFirmware is mandatory (and I guess it's not) to boot Linux +on our SoC, or if U-Boot by itself could have made everything by itself. +I don't even know the details of what is an FIP image. +But for now, I decided to proceed like the OpenWRT and Debian images. +I will surely dig in the TrustedFirmare and ARM documentations later. +I'll eventually to a follow-up post later if I achieve to figure out how everything is working. +--- + +So, let's create our callPackage derivation in `trusted-firmware/default.nix`: +``` +{ + buildArmTrustedFirmware, + fetchFromGitHub, + dtc, + ubootBpiR4, + ubootTools, + openssl, + ... +}: +(buildArmTrustedFirmware rec { + platform = "mt7988"; + extraMeta.platforms = [ "aarch64-linux" ]; + extraMakeFlags = [ + "USE_MKIMAGE=1" + "BOOT_DEVICE=sdmmc" + "DRAM_USE_COMB=1" + "BL33=${ubootBpiR4}/u-boot.bin" + "all" + "fip" + ]; + + filesToInstall = [ + "build/${platform}/release/bl2.img" + "build/${platform}/release/fip.bin" + ]; +}).overrideAttrs (old: { + src = fetchFromGitHub { + owner = "mtk-openwrt"; + repo = "arm-trusted-firmware"; + rev = "bacca82a8cac369470df052a9d801a0ceb9b74ca"; + hash = "sha256-n5D3styntdoKpVH+vpAfDkCciRJjCZf9ivrI9eEdyqw="; + }; + version = "2.10.0-mtk"; + nativeBuildInputs = old.nativeBuildInputs ++ [ dtc ubootTools openssl ]; +} +``` + +This looks like what we have done with U-Boot, we select our platform, then pass the necessary flags for compilation, +and then grab the two output files that we need. +However here, there is some extra steps. +Both `buildUBoot` and `buildArmTrustedFirmware` assumes that you're building the mainline U-Boot and TF-A. +It was the case with our U-Boot build to which we just add some extra patches. +However, with TF-A, as I said, we will here use the mtk-openwrt's fork. +So we need to override the derivation produced by `buildArmTrustedFirmware` to pass our own sources and needed dependencies thanks +to the `overrideAttrs` function. + +As we can see, this callPackage derivation needs to access our previous `ubootBpiR4` derivation. +All the other derivations arguments are provided by nixpkgs itself through the `callPackage` function. +However, `ubootBpiR4` is itself not present within the nixpkgs we use, we have to manually pass it. +There is two solutions here, the first one is to use an overlay to manually add it to our nixpkgs. +But there is a far further solution, which is to pass it as an argument of the `callPackage` function. +This make our `default.nix` file looks like that: +``` +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +rec { + ubootBpiR4 = pkgs.callPackage ./u-boot { }; + armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; }; +} +``` + +Now, we can run `nix-build -A armTrustedFirmwareBpiR4` to build everything we need. + +## Side note on the U-Boot and TF-A combination + +If you check how to build the full boot sequence binaries for various board, you will be feel like me +that this seems to be highly dependant on SoC manufacturer and the SoC itself. +As we have seen there, we build U-Boot first and then pass it to TF-A as BL33. +For some Allwinner SoCs, TF-A is built first and then passed to U-boot as BL31. +For Rockchip based boardsm there is this whole `idbloader.img` due to Rockchip's miniloader. +At this point, I don't really know if there is truly a "standard way" to build to whole boot sequence for each SoC. +I don't even know what causes this difference in the build process between manufacturers. +I guess this is related to manufacturers' implementation, or things that could be done both in U-Boot and in TF-A +and then depends about where the manufacturer implemented them ? + +## Create our boot image + +The final step is just to create an empty image and flash what we have built so far at the corrects address ranges. +This will be done with a simple bash script derivation in the `image/default.nix` file. +I will just dump the callPackage derivation here, and explain everything. +``` +{ + runCommand, + armTrustedFirmwareBpiR4, + gptfdisk, + ... +}: +runCommand "bpi-r4-image" { + nativeBuildInputs = [ + gptfdisk + ]; +} '' + IMAGE=$out/nixos-r4-image.img + + mkdir $out + + dd if=/dev/zero of=$IMAGE bs=1M count=4000 + sgdisk -o $IMAGE + sgdisk -a 1 -n 1:34:8191 -A 1:set:2 -t 1:8300 -c 1:"bl2" $IMAGE + sgdisk -a 1 -n 2:8192:9215 -A 2:set:63 -t 2:8300 -c 2:"u-boot-env" $IMAGE + sgdisk -a 1 -n 3:9216:13311 -A 3:set:63 -t 3:8300 -c 3:"factory" $IMAGE + sgdisk -a 1 -n 4:13312:17407 -A 4:set:63 -t 4:8300 -c 4:"fip" $IMAGE + + dd if=${armTrustedFirmwareBpiR4}/bl2.img of=$IMAGE seek=34 conv=notrunc,fsync + dd if=${armTrustedFirmwareBpiR4}/fip.bin of=$IMAGE seek=13312 conv=notrunc,fsync +'' +``` + +The nixpkgs's `runCommand` function create a derivation that runs a bash script. +This bash script is run into the Nixpkgs `stdenv`, but we can override this environment with the set passed +as its second arguments. +In this case, we add the `gptfdisk` package as a dependency to use `sgdisk` within our bash script. +The third argument is the script in itself. +We then create the derivation output directory, and generate a zero-filled 4GB image with `dd`. +Then, we create our partitions with `sgdisk` with the following partition map: +``` +Block size of 512 bytes + +Blocks 34 to 8191: BL2 +Blocks 8192 to 9215: Future space for U-Boot environment variables +Blocks 9216 to 13311: Factory +Blocks 13312 to 17407: FIP +``` +Finally, we flash our TF-A build outputs into the matching partitions with `dd`. + +We add this callPackage derivation into our `default.nix`: +``` +{ + pkgs ? import { + crossSystem = (import ).systems.examples.aarch64-multiplatform; + }, +}: +rec { + ubootBpiR4 = pkgs.callPackage ./u-boot { }; + armTrustedFirmwareBpiR4 = pkgs.callPackage ./trusted-firmware { inherit ubootBpiR4; }; + image = pkgs.callPackage ./image { inherit armTrustedFirmwareBpiR4; }; +} +``` + +And we can build our boot sequence components with `nix-shell -A image` to get our `nixos-r4-image.img`. + +## Flashing and running + +So let's flash the resulting image into an SD card. +``` +dd if=result/nixos-r4-image.img of={Your SD Card} conv=sync status=progress +``` + +After plugging it in the BPI-R4 SD slot and set the boot device jumper to SD boot. +We get the following logs with `minicom`: +``` +F0: 102B 0000 +FA: 1042 0000 +FA: 1042 0000 [0200] +F9: 1041 0000 +F3: 1001 0000 [0200] +F3: 1001 0000 +F6: 380E 5800 +F5: 0000 0000 +V0: 0000 0000 [0001] +00: 0000 0000 +BP: 0600 0041 [0000] +G0: 1190 0000 +EC: 0000 0000 [3000] +MK: 0000 0000 [0000] +T0: 0000 0221 [0101] +Jump to BL + +NOTICE: BL2: v2.10.0 (release): +NOTICE: BL2: Built : 00:00:00, Jan 1 1980 +NOTICE: WDT: Cold boot +NOTICE: WDT: disabled +NOTICE: CPU: MT7988 +NOTICE: EMI: Using DDR unknown settings +NOTICE: EMI: Detected DRAM size: 4096 MB +NOTICE: EMI: complex R/W mem test passed +NOTICE: BL2: Booting BL31 +NOTICE: BL31: v2.10.0 (release): +NOTICE: BL31: Built : 00:00:00, Jan 1 1980 + + +U-Boot 2024.04 (Apr 02 2024 - 10:58:58 +0000) + +CPU: MediaTek MT7988 +Model: mt7988-rfb +DRAM: 4 GiB +Core: 49 devices, 19 uclasses, devicetree: separate +MMC: mmc@11230000: 0 +Loading Environment from nowhere... OK +In: serial@11000000 +Out: serial@11000000 +Err: serial@11000000 +Net: +Warning: ethernet@15100000 (eth0) using random MAC address - 56:69:11:bc:68:06 +eth0: ethernet@15100000 +BPI-R4> +``` +And here we have our initial access to the U-Boot console ! + +## Sources + +I succesfully got to this point after myself reading some documentation and a bunch of blog posts. +Those where the resources I used: +- [frank-w's dokuwiki](https://www.fw-web.de/dokuwiki/doku.php?id=en:bpi-r4:start) +- [Kasper Kondzielski's blog post about running NixOS on the BPI-R3](https://github.com/ghostbuster91/blogposts/blob/main/router2023/main.md) +- [Weijie Gao's post on the Banana Pi Discource on U-Boot and TF-A on Mediatek SoC](ihttps://forum.banana-pi.org/t/tutorial-build-customize-and-use-mediatek-open-source-u-boot-and-atf/13785) +- [*Understanding ARM Trusted Firmware using QEMU* by Hemanth Nandish](https://lnxblog.github.io/2020/08/20/qemu-arm-tf.html) +