Debug a running kernel module with GDB and VirtualBox

In this post We are going to see how to debug a running kernel module of a guest VM from the host.

Setup VirtualBox

First you need to be sure that your kernel have to KGDB support. Then you have to setup your VM in order to add a virtual serial port. This serial port will be used for the remote debugging from the host. The following captures show how to setup it.

You see that a file is used for the serial port on the host. Before you we boot the VM we have to choose how to enable KGDB. Two solution here :
Boot kernel with arguments:

kgdboc=ttyS0,115200

or
Configure kgdboc after the kernel booted:

echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc

Now you have decided you can boot….

After the boot, the debugging…

First you have to sync up the kernel/module binaries and sources between the host and the guest. Personnally I simply use a rsync between the host and the VM, but a VirtualBox or a NFS shared folder should work.

Prepare the debugging on the host

Now we are going to link the virtual serial port created for the VM to a pseudo-termial.

socat -d -d /tmp/serial PTY:

Get the the pseudo-terminal used/returned by socat when starting gdb.

gdb ./vmlinux
(gdb) target remote /dev/pts/6

On the guest

By default gdb will not be able to resolve the symbols of the kernel module. So we need to get the .text, the .bss, the .data addresses.

sudo cat /sys/module/<module_name>/sections/.text
0xffffffffa0371000
sudo cat /sys/module/<module_name>/sections/.bss
0xffffffffa039b440
sudo cat /sys/module/<module_name>/sections/.data
0xffffffffa039a000

With only one line that can be pasted directly to GDB:

export MODULE=vrouter; echo add-symbol-file $MODULE.o $(cat /sys/module/$MODULE/sections/.text) -s .bss $(cat /sys/module/$MODULE/sections/.bss) -s .data $(cat /sys/module/$MODULE/sections/.data)

At this point we have to stop the execution of the kernel in order to load module symbols and to put breakpoints for examples.

echo g > /proc/sysrq-trigger 

Start the debugging

Now that you stop the execution of the kernel you should be able to load the module symbole.

(gdb) add-symbol-file module.o 0xffffffffa0371000 -s .bss 0xffffffffa039b440 -s .data 0xffffffffa039a000
(gdb) break xxxxxx

Your own Docker Registry

A simple post to show how it is simple to deploy your own Docker registry exposed through a HTTPS connection. In order to deploy your own Docker registry you will have to generate a SSL certificate. If you don’t have any access to a CA I suggest you to read the post about how to deploy your own CA. Please note that Docker currently doesn’t support self-signed certificate.

Install docker-registry

git clone https://github.com/dotcloud/docker-registry.git \
/opt/docker-registry

Create your first docker registry config file.

cd /opt/docker-registry
cp config/config_sample.yml config/config.yml

Create some required folders.

mkdir /opt/docker-registry/data

Edit the config, to start we will keep only the common and the dev section, and we will remove a the stuffs related to Google cloud, S3, EC2, to only keep the search backend parameter. So the config should look like this :

common:
    secret_key: _env:SECRET_KEY
    standalone: true

    search_backend: "_env:SEARCH_BACKEND:"
    sqlalchemy_index_database:
        "_env:SQLALCHEMY_INDEX_DATABASE:sqlite://///opt/docker-registry/data/docker-registry.db"

dev:
    storage: local
    storage_path: /opt/docker-registry/data/
    loglevel: debug

Now let’s create protect the registry access by adding a htpasswd file that we will be used by apache2.

htpasswd -c /opt/docker-registry/data/htpasswd <username> <password>

Enable required apache modules.

a2enmod proxy_http
a2enmod ssl
a2enmod headers

The vhost config file of apache2 should be like this :

<VirtualHost *:443>
        ServerAdmin contact@spekii.com
        ServerName docker-registery.spekii.com

        SSLEngine on
        SSLCertificateFile /opt/ssl/docker-registry.crt
        SSLCertificateKeyFile /opt/ssl/docker-registry.key

        CustomLog ${APACHE_LOG_DIR}/access.log combined
        <Location /_ping>
                Satisfy Any
                Allow from all 
        </Location>

        <Location /v1/_ping>
                Satisfy Any
                Allow from all 
        </Location>

        <Location />
                AuthType Basic
                AuthName "Private Docker registery"
                AuthUserFile /opt/docker-registry/data/htpasswd
                Require valid-user
        </Location>

        ProxyPreserveHost  On
        ProxyRequests      Off
        ProxyPass          /  http://127.0.0.1:5000/
        ProxyPassReverse   /  http://127.0.0.1:5000/
        RequestHeader      unset Authorization
</VirtualHost>

First start, first test…

Start the docker-registry processes.

gunicorn --access-logfile /var/log/docker-registry -k gevent \
--max-requests 100 --graceful-timeout 3600 \
-t 3600 -b localhost:5000 -w 8 docker_registry.wsgi:application

Get logged…

docker login https://docker-registry.spekii.com

Username: admin
Password: 
Email: admin@spekii.com
Login Succeeded

Now let’s try to push an image.

docker pull busybox
docker tag busybox docker-registry.spekii.com/busybox
docker push docker-registry.spekii.com/busybox

The push refers to a repository [docker-registry.spekii.com/busybox] (len: 1)
Sending image list
Pushing repository docker-registry.spekii.com/busybox (1 tags)
511136ea3c5a: Image successfully pushed 
6c991eb93460: Image successfully pushed 
9f4e93171ec5: Image successfully pushed 
2d8e5b282c81: Image successfully pushed 
Pushing tag for rev [2d8e5b282c81] on {https://docker-registry.spekii.com/v1/repositories/busybox/tags/latest}

*Be careful of using the same hostname when logging/tagging

Finally we can now add a start script. Few preparation…

useradd -r -s /bin/false docker-registry
chown docker-registry:docker-registry data

… and the script.

#!/bin/sh

export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin

BASE=$(basename $0)

# should be adapted in the default file
DOCKER_REGISTRY_PIDFILE=/var/run/$BASE.pid
DOCKER_REGISTRY_LOGFILE=/var/log/$BASE.log
DOCKER_REGISTRY_USER=docker-registry
DOCKER_REGISTRY_GROUP=docker-registry
DOCKER_REGISTRY_DIR=/opt/docker-registry

if [ -f /etc/default/$BASE ]; then
        . /etc/default/$BASE
fi

case "$1" in
        start)
                gunicorn -D --access-logfile $DOCKER_REGISTRY_LOGFILE -k gevent \
                --max-requests 100 --graceful-timeout 3600 \
                -t 3600 -b localhost:5000 -w 8 docker_registry.wsgi:application \
                -u $DOCKER_REGISTRY_USER -g $DOCKER_REGISTRY_GROUP \
                -p $DOCKER_REGISTRY_PIDFILE --chdir $DOCKER_REGISTRY_DIR
                ;;

        stop)
                PID=$(cat $DOCKER_REGISTRY_PIDFILE)
                kill $PID
                ;;

        restart)
                $0 stop
                $0 start
                ;;

        force-reload)
                $0 restart
                ;;
        *)
                echo "Usage: $0 {start|stop|restart}"
                exit 1
                ;;
esac

exit 0

Your own Certificate Authority

Just a simple tutorial to explain how to create your own CA, how to create certificate using it and how to register it as root certificate on your linux distribution to avoid warning from the web browser.

Create your Root Certificate

The first step to create a root certificate is to create a root private key.

openssl genrsa -out /opt/ssl/ca.key 2048

The second step is to self-sign it. The following command will produce a self-signed certificate with the ca.pem name.

openssl req -x509 -new -nodes -key /opt/ssl/ca.key \
-days 1024 -out /opt/ssl/ca.crt

Fill the formular…

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank 
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:FR
State or Province Name (full name) [Some-State]:Ile de France
Locality Name (eg, city) []:Paris
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Spekii 
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []:spekii.com
Email Address []:contact@spekii.com

Install your root certificate

On Ubuntu, to install a root certificate we need to create a folder here :

/usr/share/ca-certificates/spekii

Copy your certificate in it.

cp /opt/ssl/ca.crt /usr/share/ca-certificates/spekii

Then run the command to update global certificate file.

update-ca-certificates

Create a certification with your CA

Now that you have a CA your are able to generate certificates. The procedure is almost the same as the procedure to create the CA. So first let’s create a key for a http ssl website.

openssl genrsa -out http.key 2048

Now we have to generate a certificate request.

openssl req -new -key http.key -out http.csr

Be careful when filling the common name field, otherwise you will get a “cannot verify authenticity” error.

You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:FR
State or Province Name (full name) [Some-State]:Ile de France
Locality Name (eg, city) []:Paris
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Spekii
Organizational Unit Name (eg, section) []:IT
Common Name (e.g. server FQDN or YOUR name) []: ssl.spekii.com
Email Address []:contact@ssl.spekii.com

Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:

Now we sign the csr with the CA key in order to get the certificate.

openssl x509 -req -in http.csr -CA ca.crt -CAkey ca.key \
-CAcreateserial -out http.crt -days 500

Use this new certificate with Apache

Edit your vhost file.

<VirtualHost *:443>
SSLEngine on
SSLCertificateFile /opt/ssl/http.crt
SSLCertificateKeyFile /opt/ssl/http.key
...
</VirtualHost>

Et voila ! If you want to go a bit further, I added just below some Open source PKI.

• http://phpki.sourceforge.net/
• http://www.ejbca.org/
• https://pki.openca.org/

Network labs with Linux namespaces...

If like me you need to build many network testing environments, and you feel that is a pain to spawn many virtual servers only to realize some little networking tests, namespaces are made for you !

Linux namespaces

The purpose of the namespaces is to wrap system resources in a way that makes that a process associated to the namespaces has its own version of these resources.

 Currently there are six kind of namespaces:

•     network namespaces, we are going to use these ones.
•     User namespaces
•     PID namespaces
•     UTS namespaces
•     IPC namespaces
•     Mount namespaces

Your first lab

First, assume we want to connect two virtual servers to a network. So to achieve this topology we are going to create two network namespaces which will act as two nodes.

The following lines create the two namespaces :

# ip netns add vm1
# ip netns add vm2

Now that we have two namespaces, we need to create the network interfaces for both of them.

Following lines create a virtual Ethernet (veth) pair :

# ip link add name vm1-eth0 type veth peer name vm1-eth0.1
# ip link add name vm2-eth0 type veth peer name vm2-eth0.1

Now we need to add one veth to a namespace for each node :

# ip link set vm1-eth0.1 netns vm1
# ip link set vm2-eth0.1 netns vm2

Every network namespace comes with its own loopback interface. We need to set all interfaces belonging to a namespace to up.

# ip netns exec vm1 ip l set lo up
# ip netns exec vm1 ip l set vm1-eth0.1 up
# ip netns exec vm2 ip l set lo up
# ip netns exec vm2 ip l set vm2-eth0.1 up

By the way, we can see above that to be able to execute a command in a namespace context, we just need to use the following command template :

# ip netns exec <NS> <COMMAND>

Now let's add addresses :

# ip netns exec vm1 ip a add 192.168.0.1 dev vm1-eth0.1
# ip netns exec vm2 ip a add 192.168.0.2 dev vm2-eth0.1

And the routes :

# ip netns exec vm1 ip r add 192.168.0.0/24 dev vm1-eth0.1
# ip netns exec vm2 ip r add 192.168.0.0/24 dev vm2-eth0.1

Now it's the time to connect both nodes together. To achieve this, we can either use a bridge or a virtual switch like Openvswitch.

Let's see with a bridge :

# brctl addbr br1
# ip l set br1 up

# ip l set vm1-eth0 up
# brctl addif br1 vm1-eth0

# ip l set vm2-eth0 up
# brctl addif br1 vm2-eth0

Or with Openvswitch :

# ovs-vsctl add-br br1
# ip l set br1 up

# ip l set vm1-eth0 up
# ovs-vsctl add-port br1 vm1-eth0

# ip l set vm2-eth0 up
# ovs-vsctl add-port br1 vm2-eth0

Finally you should be able to ping a node from the other one.

# ip netns exec vm1 ping 192.168.0.2

Here the full script :

#!/bin/sh

ip netns add vm1
ip netns add vm2
ip link add name vm1-eth0 type veth peer name vm1-eth0.1
ip link add name vm2-eth0 type veth peer name vm2-eth0.1
ip link set vm1-eth0.1 netns vm1
ip link set vm2-eth0.1 netns vm2
ip netns exec vm1 ip l set lo up
ip netns exec vm1 ip l set vm1-eth0.1 up
ip netns exec vm2 ip l set lo up
ip netns exec vm2 ip l set vm2-eth0.1 up
ip netns exec vm1 ip a add 192.168.0.1 dev vm1-eth0.1
ip netns exec vm2 ip a add 192.168.0.2 dev vm2-eth0.1
ip netns exec vm1 ip r add 192.168.0.0/24 dev vm1-eth0.1
ip netns exec vm2 ip r add 192.168.0.0/24 dev vm2-eth0.1
brctl addbr br1
ip l set br1 up
ip l set vm1-eth0 up
brctl addif br1 vm1-eth0
ip l set vm2-eth0 up
brctl addif br1 vm2-eth0

ip netns exec vm1 ping 192.168.0.2 -c 5

This article was just a little introduction of the huge possibilities offered by namespaces.