Incomplete, these steps did not complete a working instance. Will follow up with other attempts.
How It Began:
This journey started as many in the homelab often do, with a “Wouldn’t it be cool if?”. Wells, lets find out. First, lets outline what is the expected outcome, then we can evaluate the hurdles as they come at us.
I want…
A GPU enabled node in my low power K8S homelab cluster
It should…
Run Kubernetes, Run a container provider, meet the requirements of my existing cluster (Storage, Networking, etc should be compatible), Be cheap, meaning I already have one, not require extensive extra upkeep, be able to run pre-trained models, be able to be secured properly
Prep work:
This was pretty straight forward, but as it turns out I had to do a few times. A Fresh flash per the instructions, with a few adjustments. First, I used the docker container SDK manager, because who has an Ubuntu18 machine sitting around anymore? I also had to track down the highest supported version of the SDK and make sure the instructions lined up. SDK manager link, I needed the Docker Image Ubuntu18.04 otherwise it wouldn’t find the correct Board support package. Install it with the command:
docker load -i ./sdkmanager-[version].[build#]-[base_OS]_docker.tar.gz <– in other words the name of the file you downloaded
They also recommend you tag it as the latest, so later you can commit your commands into a new container that will just execute the install. This might end up being a thing later when I start talking about custom kernels… we’ll see
Plug the board’s micro usb port into a handy port on my development machine.
Boot the board into recovery mode by holding rst, pressing power, releasing power, then releasing rst
Wait until I got a “device plugged in” notification, If you’re watching the device list you’re looking for something like this NVIDIA Corp. L4T (Linux for Tegra) running on Tegra to show up.
Fire up the docker container with the following command:
sudo docker run -it –privileged -v /dev/bus/usb:/dev/bus/usb/ sdkmanager –cli –action install –login-type devzone –product Jetson –target-os Linux –version 4.6.4 –host –target JETSON_TX1 –flash –license accept –exit-on-finish
From there it’s mostly just following prompts, at some point the Jetson will reboot and install a few components, if you need to join to wifi, or setup the network this is your time to shine, you might need it before you “install packages”, I haven’t had good luck trying to force the traffic over the usb port, which is the default.
The first attempt:
aka: just installing K8s
The first thing I tried was installing K8s directly on the base image. For the most part is worked. However, the base image is on an onboard memory chip, which is probably fine, but logs, tmp files and all of the IO that K8s would cause would probably not be ideal. That, however is getting ahead of myself. This Install also brought to light an issue that the default kernel had. It wasn’t compiled with the arguments needed to handle Calico networking. Some research led me to discover that it would work fine with Flannel, but I’m not looking to spend a million years migrating my networkin… ok, I spent a million years trying NOT to migrate my CNI. As of this writing, I’m really wondering why.
The Second attempt:
aka: Moving install to attached SSD
Will update as I make this attempt