TL;DR: I setup my home networking with virtualized Opnsense and piHole. Yes, it is overly complex, but I kept my networking knowledge somewhat sharp.

I've always been curious about networking infrustructure, ever since the early days of my prepubescent childhood. I remember when I first got my xbox 360 around my 6th grade birthday, I immediately was curious on how it connected to the internet, how it connected to xbox's servers, and how hostbooters were able to find my "address" called an ip address and kick me offline. For those who don't know, "hostbooting" is gamer slang for a DoS attack on a client's public ip address. Essentially flooding their home network with packets to disconnect them from online gaming services. People would disconnect players on the other team for an easy win.

Anyway, that passion always carried itself with my all the way until my first intership at Datto, where I was pulnged head first into the world of networking. I remember going into the job thinking: I've been programming for over 3 years at this point. I'm pretty comfortable with computers, and have programmed a few things into my raspberry pi. I can handle this internship.

Quickly, I was barraged with terms like router, modem (yes, they are different), switch, and access point. My team was talking about packet routing and firewall rules, how to forward intervlan traffic across a network and QoS services. Needless to say, I felt very overwhelmed during my first week, and I quickly scrapped my idea that this was going to be an easy internship. Nonetheless, I saw this as a challenge. I knew that people -- regular humans -- had created the technology that my coworkers were talking about, and if they had created it, then I could learn it.

Ok, enough of story time.

Setup

One of the first, and most important, things that I had been planning before moving to Boston was getting a Dell r210 II to setup Proxmox on. I wanted to get an enterprise grade server due to three very important reasons:

• I wanted to make sure I could get a background server that was low power. The r210 II seems to run around 60W idle, so that was right around what I anticipated
• I wanted to get some experience with enterprise level server equipment, as I was planning on moving this blog's infrustructure and others to a home hosted setup.
• And last but not least, it's pretty fucking cool.

So, I found a cheap one on Reddit';s /r/homelabsales subreddit, and it was waiting at our apartment the day we arrived to move in.

The next thing I researching was which virtualization platform I was going to use to set all of this up. I looked into VMware's Esxi platform and I had used it before, but I found that their support for older hardware just didn't quite do it for me. I have a super old Chenbro server laying around that I plan on running when I need worker nodes, and I wanted to make sure that I could master one virtualization platform for my home lab instead of just knowing two. They also rolled their own kernel, and while I think that is great for enterprise network hardening, I really felt more comfortable in a debian linux environment. So, with this, I decided to go with Proxmox, as it seemed more "user friendly," had less advanced features enabled by default to make getting up and going easier, and is supported by the open source community.

Proxmox Installation

Now that the virtualization platform has been established, it is time to install that bad boy onto the server. To do this, I simply downloaded the Proxmox VE iso image using qbittorrent and flashed it on a flash drive with dd.

<p>Using the lsblk command, find the disk file for the flashdrive. Using that disk name, run the dd command and copy the ISO over. If everything went well, you should see dd writing the ISO to the usb drive. Wait for this to be done, and then run the sync command. This will ensure the disk write is complete and all changes and saved to disk. The disk shouldn't be mounted, but I ran lsblk again just to double check, then unplugged the drive and put it into the back of the server.

Proxmox has a pretty great installer, so just follow along with the steps until it is running on your system. I don't want to mindlessly rewrite their installation guide, so I will link it here. However, if we don't have a router yet to route packets or run a DHCP server on, how will we reach the proxmox web admin to start our VMs? The computers on the network don't know how to communicate with each other yet. The answer is simple, static routing.

Static Routing on Arch Linux

Most of the time, I use my wireless access card on my laptop to connect to the internet. Using wpa_supplicant and dhcpcd normally suffices in most use cases, but in this case I hadn't gotten my APs up and running yet so I couldn't use the wireless card. Instead, I used a simple USB to ethernet adapter I had lying around and used that interface to connect to proxmox. After plugging the USB adapter in, I used the infamous ip command to setup a static IP and route for my subnet through the USB adapter. But first, we need to look for the new interface name.

Ok, so looking at that output, I could see a new interface named enp0s20f0u. This was the USB interface I was looking for, however if you are following along, yours may be different. Milage may vary. Also, the interface is in the UP state already, so no need to manually bring the interface up, in my case. Should you need to bring the interface up, use the following command: sudo ip link set dev interface_name up

Once we have our interface up, we need to assign an IP address to it so Promox knows who is trying to talk to it and where to send replies back to. This is sorta like a traditional mailing address. If someone decides to send a letter from their house and not put a return address on the letter, the receiver would have no idea where to send the reply, and sending a reply to everyone in the neighborhood isn't an option either if you are discussing your tax returns. The same idea applies to computers. If we need to communicate with a server, we need to sign our packets with a return IP address. In our case, the IP address we choose to put on our interface must be within the network\u2019s subnet. I chose to use 10.10.10.10/24. Not for any particular reason, but because it gave me some room to set up my APs at the beginning of the subnet. To set the interface IP address, use the ip command again, however this time we will use the addr subcommand.

Great! Now we can see our interface has an ipv4 address associated with it, so we have our return address attached to our "letters" to proxmox. However, now our interface has no idea where to send our packets. Going back to the mail analogy, do we use Fedex? UPS? USPS? (The correct answer is support the USPS). Back in computer land, our interface isn't quite sure, so we need to tell it. To do this, we need to setup a static route, which tells our computer which interface to use when sending packets to the 10.10.10.0/24 subnet. Again, let's use the `ip` command.

As you can see, I had no routes setup the first time I ran the command. Then, I added a static route that says "any traffic destined for the 10.10.10.0/24 subnet, send it out the device named enp0s20f0u1". This allows the packets to be sent out the correct interface to get to our switch and then routed to Proxmox. Once everything is finished and if everything went well, you can go to the proxmox machine's static IP address in a web browser, and see the following page:

Now granted, I already have a few VMs running on my machine, otherwise I wouldn't be able to write this blog post now, but it should look relatively close to this. If this page is appearing, the static route worked! Great. Let's move on to the opnsense installation.

Installing OpnSense

Next, I needed to install Opnsense to route packets around on my network, as well as block unwanted connections from the internet. Everywhere I have read online recommends using pfsense as a home opensource router, but I found that Opnsense has a little bit better of a UI experience, and since editing the firewall from the UI tends to be a better experience, I sided with Opnsense. There has also been controversy over pfSense switching their software license from BSD to Apache, but for homelab use like mine, it didn't make any difference.

So first, I downloaded the Opnsense ISO installer in the same fashion as the Proxmox installer, however this time I put the image on a regular USB drive, no dd command involved. This is because I want to mount the installer USB image to my VM and install to the virtual disk, so I need it on the proxmox box raw instead of on a bootable USB. To add the ISO image from the USB, plug the drive in. However, before creating the VM, we need to setup some network configuration for the Opnsense VM so we can connect to the VM through a lan interface and get internet from a WAN interface. Since my r210ii has two physical nic ports, I chose to have the first port (eno0) be the WAN, and the second port (eno1) be the LAN. I created two VLAN aware linux bridges in the network section under "System" of my proxmox node and assigned proxmox to 10.10.10.2 on the LAN bridge. This will give me access to Proxmox admin on LAN.

After the network was ready, I had to actually create the VM. Click the "Create VM" button in the top right of the proxmox window. A popup should appear where the properties for the VM can be defined. Here, I just called the vm "Opnsense", and the OS I picked from the USB drive, and I set up the VM with basic specs, which are listen below:

• 2 vCPU
• 4GB of RAM
• 32 GB of physical ROM storage on a 1TB hard drive on the server
• Default graphics driver
• Added the WAN network to the VM to start, not firewalled.

Proxmox seems to only allow connecting one network adapter to the VM upon creation, so I added the second one later in Hardware properties for the VM. Make sure both network interfaces are not firewalled, as we are installing a firewall on our VM! Having firewall rules across multiple different systems is just asking to bang your head against the metaphorical wall.

Once I had all of that set up, I booted up the VM and was welcomed with the Opnsense installation screen! I followed their instructions, and installed Opnsense on the VM.

** Note: Make sure you pick the virtual hard disk as the installation drive. If you pick the USB stick on accident, opnsense will not be installed and all of your settings will be blown away. Don't make the same mistake I did.

I personally setup my LAN network on pfsense to be 10.10.10.0/24, and the WAN was DHCP from verizon. Once I did that, I went to http://10.10.10.1 in my browser and found the Opnsense admin splash screen. Opnsense has been installed!

Opnsense Home Setup

By default, opnsense comes pretty locked down. Basically, anything coming out of the LAN interface is allowed, but anything coming into the LAN is blocked. Anything traveling from the LAN network out to the internet through the WAN is NAT'd to the WAN IP address. Pretty standard stuff for most internet browsing, but when you are dealing with devices that require direct internet access of some kind, like online gaming services for gaming consoles, we need to allow some explicit external natting that keeps the port the device is requesting. By default, Opnsense randomizes the port when NATing over the WAN interface for privacy reasons, so for my switch, I just made it nat to whichever port it wants and keep that port after NAT. So if the game servers use 1010, the traffic will be kept on that port through the NAT table. To do that, I went to the firewall section, chose NAT and outbound NAT. On that page, I clicked the Hybrid model, since I still wanted Opnsense's default rules, and then added the following rule:

After a quick firewall reload, my switch was able to get a NAT type of B instead of D and connect to Nintendo's online services. As a side note though, a lot of forums tried to guide me to use OpnSense's UPnP plugin, however I installed it and it didn't really seem to have much difference on my switch. The NAT type without the outbound rule, even with UPnP on, was still D and wasn't able to connect to matchmaking services.

Pihole

Ok, now that we have routing setup on the LAN network, I wanted to utilize more of the power of this new hypervisor I just installed for my home router. So, I decided to move some of the network services I keep on a raspberry pi to a VM on the server. This would not only allow for lower latency for DNS requests, but also frees up a raspberry pi for use with Hyperion for ambient lighting around my living room TV (A blog post about that later).

So, I decided to move my PiHole server into a Debian VM. To do this, I downloaded a Debian VM using qbittorrent and transferred it over to Proxmox. I then created a VM called &quot;PiHole&quot; the same way I did above and gave it the following specs:

• 1 vCPU
• 1GB of RAM
• 16 GB of physical ROM storage on a 1TB hard drive on the server
• Default graphics driver
• Added the LAN network to the VM, as only devices on my local network should be able to get to the VM directly

With these settings set for the VM, I started the instance and configured Debian to a very basic install -- No graphical user interface or extra packages, just basic Debian. After I went through the installer and installed it to the virtual hard disk, I stopped the VM, unattached the ISO from the VM, and booted into regular debian under my user. From here, I used the curl command on PiHole's website. This guided me through installing PiHole on the network. I believe I gave the VM a static IP address of 10.10.10.5. Once I could get into the admin console, I moved on to configuring Opnsense.

By default, Opnsense starts Unbound on a default installation, so I disabled this DNS server just to be extra sure that my devices don't use it somehow. Also, I checked to see if a DHCP server was already running on my LAN. When I setup Opnsense, I chose not to start one, but if it is enabled, make sure to disable it before continuing. This could cause DHCP lease conflicts between the servers, which could cause two devices to get the same IP address. Not good.

Once Opnsense was configured to stop doing DHCP and DNS duties, I started the DHCP server for PiHole and told it to hand out addresses in the 10.10.10.0/24 CIDR, starting at 10.10.10.10 and going to 10.10.10.254. This allowed me to have a few static IPs at the beginning of the CIDR for interval services like PiHole and my wireless APs.

Once you do this, you should start to see DNS requests come in if there are other devices on the LAN network. In my case, there were devices connected to my APs, so those devices started to use PiHole as their existing leases expired. Success! Now most ads on all of my devices on my network will be blocked. Good bye smart TV ads!

Conclusion

Overall, I've been really pleased with my new home router setup (minus the noise). Things have been going rather smoothly, and I learned a lot about Proxmox, virtualization, and really forced myself to keep my networking knowledge close to my head. Soon, I plan to setup VLANs on my network to segregate IoT devices like my Chromecast and Nintendo Switch and my Kubernetes VM network. But for now, I think I'm going to leave it as is and focus on some contracting work.

If you liked this article, would like more details, or would like to discuss anything else, please feel free to reach out! As always, I love learning how to improve my setup.