A Beginner's Guide to Installing Gentoo: Part One

March 12th, 2022

This guide will walk you through setting up a base Gentoo system. It is intended for beginners who want to move to an operating system that gives them more control over what is installed. It will produce a reasonably sane Gentoo "daily driver" that will allow you to run TRB and other republican goodies. It will allow you to install xorg and a tiling manager without too much trouble. It will NOT be as strict as a Dulap when it comes to banning all systemdisms and other malware, so use accordingly.1

First of all, a huge thank you to asciilifeform for producing an ultra-hygienic Gentoo and making it available to the world. Dulap remains my go-to for server applications and any box where the need for security is greater than the desire for comfort. The first part of this guide, up to the chroot, is more or less lifted directly from the Dulap construction kit.

Prerequisites

Gather these items before beginning with the guide.

  • Any random laptop or assembled assortment of PC parts2
  • 2 USB thumb drives, at least 2GB
  • A Gentoo Stage 3 tarball: obtained here
  • A Gentoo Minimal Installation CD image: obtained here
Create the Bootable USB Drive

Note the path to your copy of the Gentoo Minimal Installation CD ISO file and check the drive device that corresponds to your USB drive with fdisk -l on Linux or diskutil list on macOS. Create the bootable USB with the following command, adjusted to your ISO file and USB drive:


dd if=install-amd64-minimal-20220308T160629Z.iso of=/dev/sdb bs=4096

Obtain a Stage 3 Tarball

Select a Stage 3 tarball from the Gentoo Downloads page.3 Download it and verify the checksum. Copy it to your other USB drive. If you're working on macOS or Window you'll need to format the drive as FAT so that your Linux build can access it.

Booting From the USB Drive and Formatting the Target Drive

Boot from the bootable install disk you created in the previous step. You'll likely need to enter your BIOS and rearrange the boot order so that USB is tried first before any internal drives.4 Once this option is set, exit the BIOS saving your changes, the machine should boot from the USB drive.

You are now in a mini Gentoo prepared specially for installing Gentoo. You'll first need to format your target drive so that you can proceed with the install. Run fdisk -l to view your drives and verify that your main drive (where you intend to install Gentoo) is in the first position. This will be /dev/sda if it's a SATA drive and /dev/nvme0n1 if it's an NVMe drive. If it is not one of these, reboot and adjust your BIOS boot order again until it is.

Using the correct device, run the following commands:


# Set the partition scheme
parted -s /dev/sda mklabel gpt

# Create the partitions
parted -s -a optimal /dev/sda unit s mkpart boot ext3 64 262143
parted -s -a optimal /dev/sda unit s mkpart primary ext4 262144 100%

# Enable the boot flag on the first partition
parted -s -a optimal /dev/sda set 1 boot on

sync

# Format the partitions
mkfs.ext3 /dev/sda1
mkfs.ext4 /dev/sda2

sync

# First create the mount point if it does not exist
mkdir /mnt/gentoo

# Mount the main partition to a mount point of your choice, here it's '/mnt/gentoo'
mount /dev/sda2 /mnt/gentoo

Now your target drive should be ready. Try running fdisk -l to see a summary of the new partitions.

Next, connect and mount your other USB drive, the one containing the Stage 3 tarball. You can mount this one to /mnt/usb or anywhere else you choose. If you're not sure which device is the USB drive you just plugged in run fdisk -l again and find it among the list.


# Mount the USB drive at /dev/sdb2 to /mnt/usb. Change to your own values if necessary.
mount /dev/sdb2 /mnt/usb

Now run lsmod and dmesg and save the output to your USB drive with the Stage 3 tarball. You will need this information later when configuring the kernel.


lsmod > /mnt/usb/lsmodout.txt
dmesg > /mnt/usb/dmesgout.txt

Extracting the Tarball and Chroot'ing Into the New System

Navigate to the root of your new drive and extract the Stage 3 tarball.


cd /mnt/gentoo
tar xpvf /mnt/usb/stage3-*.tar.xz --xattrs-include='*.*' --numeric-owner

After this completes you can run ls -l and you'll see your new system at home on its new drive.

Use the included net-setup tool to get network access. Then after you have a connection, copy the DNS info to the new environment.


# Run 'net-setup'
net-setup

# Verify that you have a connection
ping thebitcoin.foundation

# Copy the DNS info to the new environment
cp --dereference /etc/resolv.conf /mnt/gentoo/etc/

You are now ready to chroot into the new system.


# Mount the boot partition
mount /dev/sda1 /mnt/gentoo/boot

mount -o bind /dev /mnt/gentoo/dev
mount -o bind /proc /mnt/gentoo/proc
mount -o bind /sys /mnt/gentoo/sys
chroot /mnt/gentoo /bin/bash

You are now "inside" the new system, meaning what was previously /mnt/gentoo is now simply /. For example, ls / will display the contents of the partition at /dev/sda2 that you created earlier, not the contents of the bootable USB drive. Everything you do from here on out will affect the new system.5

Configuring Portage and Obtaining the Kernel Source

After chrooting, run emerge-webrsync to get a snapshot of the latest upstream repo. If you downloaded a Musl Stage 3 tarball then you will want to add the Musl overlay at this point. If not, then skip this step.

Add the Musl Overlay (skip if building a glibc-based system)

Overlays6 are a feature of Portage, Gentoo's package management system, that allow you to add additional packages not tracked in the main upstream repository. They can be thought of as sheets of vellum paper overlaying the default repository. They leave the original package list intact while adding one or more new packages (or versions of packages). This will be covered more in Part Two but for now—if you opted for a musl-based system—you will need to enable at least one overlay. If you opted for a glibc-based system, proceed to the next step.


# Emerge 'eselect-repository' and 'git'
emerge -av app-eselect/eselect-repository
emerge -av dev-vcs/git

# Create the config directory
mkdir -p /etc/portage/repos.conf 

# Enable and sync the 'musl' overlay
eselect repository enable musl
emaint sync -r musl

This will give you an up-to-date musl overlay, which is necessary for emerging7 the correct musl versions of packages for your musl-based build.

Apply the Ban List

Create a file called 'crapolade'8 /etc/portage/package.mask/crapolade and include in it asciilifeform's ban list:


# Create this file at /etc/portage/package.mask/crapolade

# systemdisms
sys-apps/systemd
sys-apps/systemd-tmpfiles
sys-fs/udev
virtual/tmpfiles
sys-auth/consolekit
sys-auth/polkit
sys-apps/dbus

media-video/libav
gnome-base/gconf
>=app-crypt/gnupg-2.0.22
app-accessibility/at-spi2-atk
app-accessibility/at-spi2-core
gnome-extra/polkit-gnome
dev-util/gdbus-codegen
gnome-base/dconf
>=x11-libs/gtk+-3.0.0
#>=sys-devel/gcc-5.0

The way this works is that Portage will check all entries in /etc/portage/package.mask when emerging packages and refuse to install packages that you have "masked". This also applies to packages brought in as dependencies by other packages, so it's a pretty effective way of keeping unwanted cruft out of your machine. You may have noticed that one item, >=sys-devel/gcc-5.0, is commented out. This is because any Stage 3 tarball that you download today will already include a GCC newer than 5.0, so the only result of leaving that line in would be some error spew about "Masked package already installed" every time emerge is run.

Later in this guide you will have to comment out other lines. As I mentioned in the beginning, this guide is not designed to produce a military grade Gentoo, but rather a comfortable daily driver that will run TRB and other republican software.9

In addition to the output of lsmod and dmesg that we saved earlier, there's another utility, lspci, available that I find useful for identifying hardware components. Emerge with:


emerge --ask --verbose sys-apps/pciutils

You will later use the output of lspci and the saved output from lsmod and dmesg to get an idea of what you'll need to enable in the kernel.

If you're on a more recent model laptop you may also need to install the Linux firmware package to enable graphics cards or wifi chips. If your laptop has Intel wifi, for example, you'll probably need this.


emerge -av sys-kernel/linux-firmware

Finally, emerge the Linux kernel source.


emerge -av sys-kernel/gentoo-sources

# Create a symlink to simplify things, change the version/filename to match what you have in /usr/src
ln -s /usr/src/linux-5.15.11-gentoo /usr/src/linux

Configuring and Building the Kernel

If this is your first time configuring a Linux kernel, which would not be unreasonable considering you're reading "A Beginner's Guide to Installing Gentoo", then it may feel a little daunting. Given the nearly endless combinations of possible hardware configurations there's also no way to create a step-by-step guide—you're on your own for this.

The strategy I would recommend going into this for the first time would be: don't overthink it and don't overdo it. There's a non-zero chance that you won't need to make any modifications at all and your kernel will boot with the defaults. Other knobs will be obvious. If you have an NVMe drive, be sure to enable support for NVMe drives. If you have an Nvidia graphics card it's probably ok to disable graphics support for other vendors, etc.10


cd /usr/src/linux

# Launch the config UI, you will make all your changes here
make menuconfig

#Compile the kernel
make && make modules_install

# Copies the kernel and config to /boot
make install

Building the Initramfs

Get genkernel to build the initramfs.


emerge --ask sys-kernel/genkernel

Build the initramfs. Be absolutely sure that the path you pass to --kernel-config points to the same config used to compile the kernel. This config should be present at /boot/config-5.15.11-gentoo (or whatever version kernel you used). If not, adjust accordingly.


genkernel --install --kernel-config=/boot/config-5.15.11-gentoo initramfs

Configure the File Systems Tab (fstab)

Edit /etc/fstab to set boot and primary disks.


# SAMPLE fstab
/dev/sda1       /boot       ext3        defaults        0 2
/dev/sda2       /           ext4        noatime,discard 0 1

Configure the Bootloader

You will have to choose a bootloader. The two main options are LILO and GRUB.11 You should try to get LILO to work first, it's got a much smaller footprint than GRUB and less moving parts means less that can potentially go wrong. If you can't get LILO to work12 or your just don't like LILO for some reason then you can use GRUB.

Option One: LILO

Emerge LILO.


emerge -av sys-boot/lilo

Edit /etc/lilo.conf and adjust the values to match your system. Then run lilo to install the bootloader.


lilo

That should be it.

Option Two: GRUB

Emerge GRUB 2.


emerge -av sys-boot/grub

Exit the chroot and enable EFI if you’re using that. If not, skip to installing GRUB.


# Exit the chroot
exit

# Enable EFI
modprobe efivars
modprobe efivarfs

# Re-enter the chroot
chroot /mnt/gentoo

Install grub.


# For EFI
grub-install --target=x86_64-efi --efi-directory=/boot --debug

# For Non-EFI (adjust device to match your system)
grub-install --target=i386-pc /dev/sda --debug

Generate the GRUB config.


grub-mkconfig -o /boot/grub/grub.cfg

Booting Your New OS

Before rebooting set a root user password. If you do not set this you will be locked out and will have to boot from your USB drive and chroot in all over again.


passwd

If you’re on a machine that only has a wireless connection you’ll also need to grab the wpa_supplicant and a DHCP client before you boot into your new system otherwise you will not have internet connectivity.


emerge -av net-wireless/wpa_supplicant
emerge -av net-misc/dhcpcd

Attempt to reboot to your newly installed Gentoo.


#Exit chroot and reboot
exit
reboot

If everything works you can log in as root with the password you set.

A Few Remaining Basics

Your system now works but before you can start having fun with it there are a few small things to take care of.

Make Your HiDPI Display Usable

If you are on a HiDPI display your console font is likely really tiny, to the point where it is practically illegible. Increase it by choosing a different font.


setfont latarcyrheb-sun32

Make this permanent by editing /etc/conf.d/consolefont and adding to boot services.


rc-update -v show | grep consolefont
rc-update add consolefont boot

See list of available fonts in /usr/share/consolefonts if you want to experiment with different options.

Enable Wireless and Connect to a WiFi Network

The wpa_supplicant replaces that little WiFi icon in the top-right corner of your old OS as the primary means of finding, selecting, and connecting to WiFi networks.


# Add 'wpa_supplicant' to the default and boot runlevels
rc-update add wpa_supplicant default
rc-update add wpa_supplicant boot

# Start the service
rc-service wpa_supplicant start

If this doesn’t work it’s likely because your kernel driver isn’t loaded.


# Load the kernel module
modprobe iwlwifi

# Try starting the service again
rc-service wpa_supplicant start

To make the kernel module load at boot, create the following file:


# /etc/modules-load.d/networking.conf
iwlwifi

You can now use wpa_cli to control your wireless interface. First try scanning for available networks.


# Scan for networks
wpa_cli scan

# Wait a few seconds then view results
wpa_cli scan_results

Note the SSID of the network you would like to connect to. To generate the WPA PSK for the network run the following command with the SSID and password of the network:


wpa_passphrase my-network-5g mynetworkpass

The output will look something like this:


network={
        ssid="my-network-5g"
        #psk="mynetworkpass"
        psk=9f1315f6253481f27965068df528064ae85e6cbb06540090c79e3817cb8f12a7
}

Copy the entire block and add it to the config at /etc/wpa_supplicant/wpa_supplicant.conf. Also add ctrl_interface and update_config knobs to the top of the file. Your file should look something like this:


# WPA Supplicant config at '/etc/wpa_supplicant/wpa_supplicant.conf'

ctrl_interface=DIR=/run/wpa_supplicant GROUP=wheel
update_config=1

network={
        ssid="my-network-5g"
        #psk="mynetworkpass"
        psk=9f1315f6253481f27965068df528064ae85e6cbb06540090c79e3817cb8f12a7
}

Save the config, then use wpa_cli again to load the config and list saved networks.


# Reload the config
wpa_cli reconfigure

# List saved networks
wpa_cli list_networks

# Select the network, where the number is the ID of the network returned by the previous command
wpa_cli select_network 0

You should now have internet access again.

Create a Non-Root User and Install 'sudo'

Now that you have an internet connection you should create a user account and install/configure sudo so that you’re not doing everything as root. First create your new user.


# Create a new user, '-m' creates a home folder for this user
useradd -m mynewuser

# Set the password for this user
passwd mynewuser


emerge -av app-admin/sudo

Configure with visudo


visudo

You can read more elsewhere about this file but an easy way to control who has sudo access is with the "sudo" group. Uncomment the line for this configuration so that it looks like:


%sudo ALL=(ALL) ALL

You can also set a longer timeout if you like (the default is only 5 minutes).


# Include this line to set the default sudo timeout to 30 minutes
Defaults env_reset, timestamp_timeout=30

Create the sudo group and add your user to the group


groupadd sudo
sudo usermod -a -G sudo mynewuser

Now you can log out of the root account with exit and log back in to your new user. Congratulations, you now have a basic working Gentoo linux machine with wireless internet access. You can now sudo emerge any package you like. In part two we’ll set up a minimal graphical environment plus some command line niceties. If you tried using this guide and noticed errors or parts that were unclear please leave feedback so that I can make improvements.

  1. If you intend to build a web server or have no need for a browser or any graphical environment then I suggest following this guide to build a Dulap instead. []
  2. I won't bother with listing "minimum specs" because they aren't high enough to be worth mentioning and because it mostly depends on what you want to run. Gentoo isn't macOS or Windows, you don't need 4GB of RAM just for the OS. Any cheapo laptop from the last 15 years should do. If your goal is to run TRB then you need at least a 1TB SSD. []
  3. At this point in time, if you use a current glibc-based system you will need to create a separate chroot environment later on in order to build TRB with the rotor due to a version incompatibility. If you select a musl-based system then you will be able to build TRB just fine, with a few tweaks, but you may have a harder time running other popular programs that don't yet play nice with musl. []
  4. This varies by machine, depending on your motherboard. Typically F1, F11, or DEL gets you into the BIOS— the post splash screen should say which key, something like "Press F1 to enter setup". From there just keyboard navigate until you find what you're looking for. []
  5. Unless you exit the chroot. []
  6. Now called "ebuild repositories". []
  7. Installing []
  8. You can call it whatever you want but this name fits. []
  9. So why include the ban list in the first place? The idea is to make you think twice about what you're installing, and only unmask the bare minimum that you absolutely need. Often if you are blocked because of a masked package you can get around it by installing an older version the package you want, or manually installing an older version of the specific dependency that's asking to bring in the masked package. Or you can find another program that has similar functionality to what you want but that doesn't depend on a masked package. If you conclude that you really must have a package that is blocked on a certain masked dependency, then you can unmask only that dependency. []
  10. Although don't worry too much about optimization at this point, you can always slim the kernel later on. []
  11. There may be others, I don't know. []
  12. I was unable to get LILO to successfully install the bootloader on my ThinkPad but it's what I use on my FX-8350 desktop. YMMV. []

Bitdash Crawler: Geolocation, Time Series Data Collection, and an IRC Bot

March 7th, 2022

The Bitdash Crawler has been running steadily since August of last year. The first release was minimal—only the basic crawling functionality, and only collecting data returned directly by the probed nodes. This patch adds additional data collection, as well as a new IRC interface for basic status reports.

Geolocation Data

When I first looked into adding this I was reluctant because I thought the only way was to connect to a third-party service via API keys. I didn't want the crawler to have to rely on any third-party service, as those come with the risk that they will change out from under you at any moment and without notice. Thankfully a user in #asciilifeform pointed me to Maxmind's excellent GeoLite2 geolocation database. I was almost surprised at how simple it was and that there was no catch. Yes, you have to provide an email and create an account with them, but after that they just give you their geolocation database1, in CSV format that you can take wherever you like. They even provide clear documentation for creating the schema and importing the CSV data. Overall a very nice offering and I'd recommend it to anyone who needs to add this to their project.

Time Series Data

I'm working on a redesign of the www interface for the crawler2 and it includes charts. Previously the crawler was only storing current snapshots of the data it collected, now it's also generating some aggregate statistics3, taken at set intervals, and storing them indefinitely. This job is handled separately from the main crawler script, via a set of SQL queries run from a cron job.

IRC Bot Interface

With the addition of geolocation and time series data collection the crawler can finally provide some reports its underlying library alone cannot. Soon this data will be available via the www interface but in the meantime I wanted to make it available in the chans via an IRC bot. The bot is based on asciilifeform's logotron bot, with which I've become familiar since using it to power the logger at logs.bitdash.io. It works roughly like this:


   billymg | !c help
crawlerbot | billymg: my valid commands are: src, uptime, help, net-summary, version, trb-status
   billymg | !c net-summary
crawlerbot | Bitcoin Network (IPv4 Nodes Active Within the Last 48 hours) Global: 8166; TRB-Compatible: 61; TRB: 13
crawlerbot | TRB-Compatible by Country: United States: 26; Canada: 4; Singapore: 4; Romania: 4; Russia: 3; France: 2; United Kingdom: 2; Italy: 1; Lithuania: 1; Norway: 1; Australia: 1; Germany: 1; Chile: 1; Belgium: 1; Spain: 1; Ukraine: 1; Netherlands: 1; Finland: 1; Sweden: 1; Switzerland: 1; Bulgaria: 1; Mexico: 1; South Africa: 1;
crawlerbot | TRB by Country: United States: 7; Canada: 1; Romania: 1; Singapore: 1; Lithuania: 1; France: 1; Norway: 1;

Patches and Signatures

I unfortunately had to regrind the genesis patch and the small bug fix patch that followed because there was a typo in the root directory's name. I'll leave the original two patches up for archival purposes but this patch is built on a new tree, with the genesis regrind also including the small fix from the original second patch.

bitdash_crawler_genesis_regrind.vpatch (sig)
bitdash_crawler_add_bot_geo_and_time_series.vpatch (sig)

  1. Presumably a subset of the data included in their full commercial offering, but plenty good enough for my needs. []
  2. Which at the moment is nothing more than a few Sketch mockups. []
  3. Network breakdown by major user agent version, network breakdown by country, TRB breakdown by country, and TRB-compatible breakdown by country. All stats are collected both for recently active nodes (48hrs) and for recently active nodes returning at least one non-self peer (referred to as "participating"). []

Straightening a Bent 1U Server Chassis

February 17th, 2022

I recently acquired a new (used) server which was damaged in shipping.1 Luckily the damage wasn't anything serious, just a slightly bent frame.

In order to have any hope of fixing this I'd first have to remove the motherboard. As you can see, none of the internals appear damaged. It seems like the PCI riser's bracket may have provided some extra sturdiness to the location just at the edge of the motherboard, preventing the case from bending closer to the middle. In any case, buena suerte. Let's get started by first removing the plastic airflow cover and the PCI riser.

Now on to all of the fan and power cables. The fans were no problem but it was a bit tight around the mains, which required some extra finesse. The first thing that got in the way was the zip tie closest to the connectors. It made it difficult to get my fingers underneath the cables and from the other side it looked like it also might have been wrapped around some cables I did not want to disturb.

With a sharp knife I carefully sawed at it until it broke. I also removed one of the heat sinks, since that was getting in the way.2 To help with some of the clips I used a flat head screwdriver to squeeze them open.

With all of the cables unplugged, all that was left was to unscrew and remove the motherboard. There were no surprises here and inspecting the motherboard upon removal further reassured me that it hadn't suffered any damage during transit.

Now with the case (mostly) empty I was ready to move it to the bodega3 for phase two. I scanned the walls and shelves until I found two vices that seemed perfect for the operation.4

I also noticed an old sheet which was being used as a rag, so I made some space on a workbench and laid it flat just to avoid adding any extra scratches to the chassis while working on it. I then attached the vices to either side and started gently tightening the one on the right side to straighten the frame.

As I was doing this I noticed the left side starting to arc up so I moved the left vice inward to prevent adding a new deformity while fixing the original.

I tightened the right side all the way but, as you may have noticed in the photo above, the flat surface isn't going to cut it. In order to fully straighten out the bend I'd need to add some leverage. I once again searched the bodega for something that would do the job. A piece of wood, some cardboard, or a folded piece of paper might do. Instead I found something even better, a small section of 4 AWG wire. I placed it right under the bend and proceeded to slowly tighten the right side again. I again noticed some unwanted bowing so I moved the right vice inward, closer to the wire, and continued tightening.

That was it. The case looked pretty straight to me now so I brought it back inside to take a closer look and reassemble the internals. As you can see from the before and after photos below, the operation seems to have been a success. Perhaps not perfect but hopefully good enough to slide into a rack somewhere.

Before

After

Here it is below with the motherboard back in place and the old thermal paste cleaned off of the CPUs.

And finally all put back together and loaded with 256GB of DDR3 ECC RAM.

I also loaded the SSDs (not visible in the photo) but unfortunately wasn't able to test anything past powering it on since I have no way to connect it to a monitor.5 The next step, after installing Dulap on it (or perhaps Pentacle), is to find a home for it here in Costa Rica. Oh, and I guess I'll also need a hardware RAID controller for it. If anyone has any recommendations I'd much appreciate it as this will be my first time buying one.

  1. Getting things shipped to Costa Rica can be a pain. This item had to be smuggled in a suitcase, and although the impromptu importer went to great lengths to make sure it was packed properly, apparently the baggage handlers went to even greater lengths to ensure that that didn't matter. []
  2. It would've had to have been removed later anyway, as it was attached to the chassis. []
  3. Meaning in this case garage/workshop/storage area, not a convenience store in NYC. []
  4. The bodega in its current state is more like my personal junkyard than anything else. It needs to be thoroughly cleaned and organized, and it will be, just probably not this year. []
  5. I mistakenly ordered an HDMI to VGA cable, rather than the VGA to HDMI cable that I need. []