Booting recent Ubuntu machines with custom built Linux Kernels

One frustration I had after beginning to use Ubuntu 20.04 (LTS) was that from some point, I just could not install a Linux kernel I built and boot the system up. My usual workflow was:

make install -jXX
sudo make modules_install
sudo make install

However my boot would stall while loading the ramdisk – much like this ask-ubuntu question. Fortunately, now, it seems to be that the initramfs size is too large (why does this matter?) causing the stall. The answer to the question above recommends stripping the modules:

sudo make modules_install INSTALL_MOD_STRIP=1

This shall ‘strip’ the modules after they are installed. (And I assume, this would reduce the module size in the initramfs).

Another method that Alireza found was to change the initramfs behavior. Changing the MODULES=dep in the /etc/initramfs-tools/initramfs.conf should reduce the initramfs size down. After changing this, my guess is either

$ sudo make install
$ #or
$ update-initramfs -c -k KERNEL_VERSION

And try booting again 🙂

Enabling vPMU on Guest VMs (KVM)

When running a VM using KVM by means of the qemu-system-x86_64 command, your VMs may not have access to the performance monitoring unit.

To check if you have access to PMU drivers, run the following command in your VMs:

$ dmesg | grep PMU
[ 0.307792] Performance Events: PMU not available due to virtualization, using software events only.

PMU not available!

What you need to do when invoking qemu-system-x86_64 is to add the following argument: -cpu host

$ dmesg | grep PMU
[ 0.314750] Performance Events: Skylake events, full-width counters, Intel PMU driver.

Yes! Now we get access to PMUs on the guest VM, meaning we can use the performance counters offered through perf stat and more tools!

Writing a (Super Simple) Linux Module

Okay, so Today I’ll need the CR3 value given a PID. The Linux kernel does not give this information out to the userspace, so I’ll be building my own module to take care of this. The code is available on my Gitlab.

Two awesome references I’ll be using for this task:

• Building kernel modules
• Getting CR3 given PID (also part of a module)
• Using sysfs kernel objects to communicate with the module

Making the basic module

We need to the actual module that is loadable into the kernel. This can be done by using the tutorial from the first reference. The main module source code needs to have the proper headers (to call proper module MACROs), the proper calls to the macros (description, author, license), and finally actually define which functions of the module are the entry and exit points.

#include 	
#include 	
#include 	

MODULE_DESCRIPTION("Kernel module to translate given PID to CR3 physical address");
MODULE_AUTHOR("Chang Hyun Park");
MODULE_LICENSE("GPL");

static int pid2cr3_init(void)
{
  printk("%s\n", __func__);
  return 0;
}

static void pid2cr3_exit(void)
{
  printk("%s\n", __func__);
}

module_init(pid2cr3_init);
module_exit(pid2cr3_exit);

The Makefile and the Kbuild files are identical to the first reference, except the filename of the code above (pid2cr3.c, add the object as pid2cr3.o​).

The full code can be found at Gitlab here

Actually doing the intended work

Now, given a PID we want to retrieve the CR3 Value! There are a few ways to pass on the PID value to the module.

1. Register the module to listen on a system call [here]
2. Pass on parameters when loading the module (and dynamically change the parameter at runtime) [here]
3. Open a pseudo-file to communicate with the module (sysfs, procfs).

We take the final approach, where we create a new sysfs directory and file which we program to act differently for reads and writes.

For a write to the file, we receive the PID from the userspace, and for the write, we return the CR3 for the priorly received PID.

The code is available here.

That’s it!

Debugging Guest kernel

Getting the vmlinux file of the Ubuntu kernels

I used the answer on superuser.com.

# Add ppas
echo "deb http://ddebs.ubuntu.com $(lsb_release -cs)-updates main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-security main restricted universe multiverse
deb http://ddebs.ubuntu.com $(lsb_release -cs)-proposed main restricted universe multiverse" | \
sudo tee -a /etc/apt/sources.list.d/ddebs.list
sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 428D7C01
# Get the actual vmlinux file
sudo apt-get install linux-image-$(uname -r)-dbgsym
file /usr/lib/debug/boot/vmlinux-$(uname -r) # This is the vmlinux file

Setup qemu to export a serial port for debugging

I referred to the following answer on stackoverflow.

Using this, my resulting qemu code is as follows:

sudo qemu-system-x86_64 -m 512M \
-kernel /boot/vmlinuz-$(uname -r) \
-drive file=ubuntu_14.04.img,index=0,media=disk,format=raw \
-append "root=/dev/sda rw console=ttyS0 kgdboc=ttyS0,115200" \
-netdev user,id=hostnet0,hostfwd=tcp::5556-:22 \
-device virtio-net-pci,netdev=hostnet0,id=net0,bus=pci.0,addr=0x3 \
-serial tcp::1234,server,nowait \
--nographic \
--enable-kvm

Notice the kgdboc=ttyS0,116200 on the kernel commandline, and the -serial tcp::1234,server,nowait on the qemu argument.

Start the debugging process

Startup the VM using the command above. SSH into your machine

ssh localhost -p 5556

trigger a gdb break(?) on the VM

sudo bash -c "echo g > /proc/sysrq-trigger"

Run GDB on your host

$ gdb /usr/lib/debug/boot/vmlinux-$(uname-r)
(gdb) target remote localhost:1234

And your gdb should be connected to the VM’s kernel.

Note: the vmlinux downloaded from ddebs(?) ppas will not include the source code, and so gdb will complain about source code lines not being found.
This should be substitutable, but I haven’t gotten this to work yet. (will update)

Making Simple KVM Image

This tutorial is aimed at quickly making a KVM image.

First off, there are lots of guides available, but I wanted to quickly, and effortlessly make a KVM image using debootstrap. Some guides on the net used nbd for some reason I cannot comprehend. this post uses the more sensible loopback interface to connect to the qemu disk and debootstrap. This guid is based on the tutorial linked above, but has my personal customization.

# Create VM foundations
$ qemu-img create -f raw ubuntu_16.04.img 20G
$ mkfs.ext4 ubuntu_14.04.img
$ mkdir mnt
$ sudo mount -o loop ubuntu_16.04.img mnt
$ sudo debootstrap --arch amd64 --include=ssh,vim xenial mnt

# Setup created environment
$ sudo chroot mnt
$ passwd # and change root password
$ adduser username
$ usermod -aG sudo username

$ sudo umount mnt

$ sudo qemu-system-x86_64 -m 512M \
-kernel /boot/vmlinuz-$(uname -r) \
-drive file=ubuntu_14.04.img,index=0,media=disk,format=raw \
-append "root=/dev/sda rw console=ttyS0" \
-netdev user,id=hostnet0,hostfwd=tcp::5556-:22 \
-device virtio-net-pci,netdev=hostnet0,id=net0,bus=pci.0,addr=0x3 \
--nographic \
--enable-kvm

# Setup DHCP for your network in your VM.
# In the **VM console** (provided by qemu)
# login as root
$ ifconfig -a
# find the interface name other than lo
# In my case it was ens3, it could be eth0, etc.

$ vi /etc/network/interfaces
# add the following two lines:
auto ens3
iface ens3 inet dhcp

$ ifup ens3
# Now you have network connectivity!

# connect via SSH to your VM
# Now, back from your **host**
$ ssh localhost -p 5556 

Also, with Ubuntu 18.04 they got rid of the ifconfig and /etc/network/interfaces and instead, now use netplan.

I used the following sequence of commands to get my interface name, and then setup DHCP

$ ip a
$ vi /etc/netplan/50-cloud-init.yaml # See the clippet below for what goes here
$ netplan try
$ netplan apply

The config file for 50-cloud-init.yaml was as follows

network:
    ethernets:
        ens3:
            dhcp4: true

 

Protected: Research: 20161216

This content is password-protected. To view it, please enter the password below.

Password:

Protected: Research: 20161213

This content is password-protected. To view it, please enter the password below.

Password:

Debugging Gem5 with GDB (SE mode)

In order to debug gem5, we don’t need to hook on via the port that is printed when executing the simulator

0: system.remote_gdb.listener: listening for remote gdb #0 on port 7000

Rather just run with gdb --args ./build/X86/gem5.opt ...

Scala Language

In our FPGA project we’ll be using the UC Berkeley Architecture Research team’s Rocket core. (Most likely we’ll be using it) The core is written in the Scala language and is compiled via Chisel into C++ simulator or Verilog for FPGA/ASIC.

Now I need to know Scala to fully understand the rocket core, and so I’m looking throught the Scala Tutorial, offered at the Scala Language site.

Interesting Features

Method call

Methods taking one argument can be used with an infix syntax.

df format now
df.format(now)

These have the same meaning syntactically. Seems alot like the style of smalltalk (and Objective-C)

Function are Objects (Passing functions as arguments)

object Timer {
  def oncePerSecond(callback: () => Unit) {
    while (true) { callback(); Thread sleep 1000 }
  }
  def timeFlies() {
    println("time flies like an arrow...")
  }
  def main(args: Array[String]) {
    oncePerSecond(timeFlies)
  }
}

Anonymous Functions

object TimerAnonymous {
  def oncePerSecond(callback: () => Unit) {
    while (true) { callback(); Thread sleep 1000 }
  }
  def main(args: Array[String]) {
    oncePerSecond(() =>
      println("time flies like an arrow..."))
  }
}

Notice that the () => denotes an anonymous function. I think it would be a lambda function 😉