Jan. 20, 2021
TL;DR: I setup my home networking with virtualized Opnsense and piHole. Yes, it is overly complex, but I kept my networking knowledge somewhat sharp.
I've always been curious about networking infrustructure, ever since the early days of my prepubescent childhood. I remember when I first got my xbox 360 around my 6th grade birthday, I immediately was curious on how it connected to the internet, how it connected to xbox's servers, and how hostbooters were able to find my "address" called an ip address and kick me offline. For those who don't know, "hostbooting" is gamer slang for a DoS attack on a client's public ip address. Essentially flooding their home network with packets to disconnect them from online gaming services. People would disconnect players on the other team for an easy win.
Anyway, that passion always carried itself with my all the way until my first intership at Datto, where I was pulnged head first into the world of networking. I remember going into the job thinking: I've been programming for over 3 years at this point. I'm pretty comfortable with computers, and have programmed a few things into my raspberry pi. I can handle this internship.
Quickly, I was barraged with terms like router, modem (yes, they are different), switch, and access point. My team was talking about packet routing and firewall rules, how to forward intervlan traffic across a network and QoS services. Needless to say, I felt very overwhelmed during my first week, and I quickly scrapped my idea that this was going to be an easy internship. Nonetheless, I saw this as a challenge. I knew that people -- regular humans -- had created the technology that my coworkers were talking about, and if they had created it, then I could learn it.
Ok, enough of story time.
One of the first, and most important, things that I had been planning before moving to Boston was getting a Dell r210 II to setup Proxmox on. I wanted to get an enterprise grade server due to three very important reasons:
So, I found a cheap one on Reddit's /r/homelabsales subreddit, and it was waiting at our apartment the day we arrived to move in.
The next thing I researching was which virtualization platform I was going to use to set all of this up. I looked into VMware's Esxi platform and I had used it before, but I found that their support for older hardware just didn't quite do it for me. I have a super old Chenbro server laying around that I plan on running when I need worker nodes, and I wanted to make sure that I could master one virtualization platform for my home lab instead of just knowing two. They also rolled their own kernel, and while I think that is great for enterprise network hardening, I really felt more comfortable in a debian linux environment. So, with this, I decided to go with Proxmox, as it seemed more "user friendly," had less advanced features enabled by default to make getting up and going easier, and is supported by the open source community.
Now that the virtualization platform has been established, it is time to install that bad boy onto the server. To do this, I simply downloaded the Proxmox VE iso image using qbittorrent and flashed it on a flash drive with dd.
$ sudo lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT nvme0n1 259:0 0 476.9G 0 disk ├─nvme0n1p1 259:1 0 512M 0 part ├─nvme0n1p2 259:2 0 8G 0 part [SWAP] ├─nvme0n1p3 259:3 0 20G 0 part / └─nvme0n1p4 259:4 0 448.4G 0 part /home sdb 8:48 1 14.6G 0 disk $ sudo dd if=/path/to/proxmox-ve_6.3-1.iso of=/dev/sdb bs=1M status=progress $ sync
Using the lsblk command, find the disk file for the flashdrive. Using that disk name, run the dd command and copy the ISO over. If everything went well, you should see dd writing the ISO to the usb drive. Wait for this to be done, and then run the
sync command. This will ensure the disk write is complete and all changes and saved to disk. The disk shouldn't be mounted, but I ran lsblk again just to double check, then unplugged the drive and put it into the back of the server.
Proxmox has a pretty great installer, so just follow along with the steps until it is running on your system. I don't want to mindlessly rewrite their installation guide, so I will link it here. However, if we don't have a router yet to route packets or run a DHCP server on, how will we reach the proxmox web admin to start our VMs? The computers on the network don't know how to communicate with each other yet. The answer is simple, static routing.
Most of the time, I use my wireless access card on my laptop to connect to the internet. Using
dhcpcd normally suffices in most use cases, but in this case I hadn't gotten my APs up and running yet so I couldn't use the wireless card. Instead, I used a simple USB to ethernet adapter I had lying around and used that interface to connect to proxmox. After plugging the USB adapter in, I used the infamous
ip command to setup a static IP and route for my subnet through the USB adapter. But first, we need to look for the new interface name.
$ ip link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT group default qlen 1000 link/ether fc:77:74:92:97:08 brd ff:ff:ff:ff:ff:ff 3: enp0s20f0u1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000 link/ether 8c:ae:4c:e1:7a:af brd ff:ff:ff:ff:ff:ff
Ok, so looking at that output, I could see a new interface named
enp0s20f0u1. This was the USB interface I was looking for, however if you are following along, yours may be different. Milage may vary. Also, the interface is in the UP state already, so no need to manually bring the interface up, in my case. Should you need to bring the interface up, use the following command:
sudo ip link set dev <interface_name> up
Once we have our interface up, we need to assign an IP address to it so Promox knows who is trying to talk to it and where to send replies back to. This is sorta like a traditional mailing address. If someone decides to send a letter from their house and not put a return address on the letter, the receiver would have no idea where to send the reply, and sending a reply to everyone in the neighborhood isn't an option either if you are discussing your tax returns. The same idea applies to computers. If we need to communicate with a server, we need to sign our packets with a return IP address. In our case, the IP address we choose to put on our interface must be within the network’s subnet. I chose to use
10.10.10.10/24. Not for any particular reason, but because it gave me some room to set up my APs at the beginning of the subnet. To set the interface IP address, use the
ip command again, however this time we will use the
$ ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: wlp0s20f3: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000 link/ether 8c:ae:4c:e1:7a:af brd ff:ff:ff:ff:ff:ff 3: enp0s20f0u1: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 8c:ae:4c:e1:7a:af brd ff:ff:ff:ff:ff:ff $ sudo ip addr add 10.10.10.10/24 dev enp0s20f0u1 $ ip addr show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: wlp0s20f3: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000 link/ether 8c:ae:4c:e1:7a:af brd ff:ff:ff:ff:ff:ff 3: enp0s20f0u1: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state UP group default qlen 1000 link/ether 8c:ae:4c:e1:7a:af brd ff:ff:ff:ff:ff:ff inet 10.10.10.10/24 brd 10.10.10.255 scope global dynamic noprefixroute enp0s20f0u1 valid_lft 43132sec preferred_lft 37732sec
Great! Now we can see our interface has an ipv4 address associated with it, so we have our return address attached to our "letters" to proxmox. However, now our interface has no idea where to send our packets. Going back to the mail analogy, do we use Fedex? UPS? USPS? (The correct answer is support the USPS). Back in computer land, our interface isn't quite sure, so we need to tell it. To do this, we need to setup a static route, which tells our computer which interface to use when sending packets to the 10.10.10.0/24 subnet. Again, let's use the
$ ip route show $ sudo ip route add 10.10.10.0/24 dev enp0s20f0u1 $ ip route show 10.10.10.0/24 dev enp0s20f0u1 scope link
As you can see, I had no routes setup the first time I ran the command. Then, I added a static route that says "any traffic destined for the 10.10.10.0/24 subnet, send it out the device named enp0s20f0u1". This allows the packets to be sent out the correct interface to get to our switch and then routed to Proxmox. Once everything is finished and if everything went well, you can go to the proxmox machine's static IP address in a web browser, and see the following page:
Now granted, I already have a few VMs running on my machine, otherwise I wouldn't be able to write this blog post now, but it should look relatively close to this. If this page is appearing, the static route worked! Great. Let's move on to the opnsense installation.
Next, I needed to install Opnsense to route packets around on my network, as well as block unwanted connections from the internet. Everywhere I have read online recommends using pfsense as a home opensource router, but I found that Opnsense has a little bit better of a UI experience, and since editing the firewall from the UI tends to be a better experience, I sided with Opnsense. There has also been controversy over pfSense switching their software license from BSD to Apache, but for homelab use like mine, it didn't make any difference.
So first, I downloaded the Opnsense ISO installer in the same fashion as the Proxmox installer, however this time I put the image on a regular USB drive, no dd command involved. This is because I want to mount the installer USB image to my VM and install to the virtual disk, so I need it on the proxmox box raw instead of on a bootable USB.
To add the ISO image from the USB, plug the drive in. However, before creating the VM, we need to setup some network configuration for the Opnsense VM so we can connect to the VM through a lan interface and get internet from a WAN interface. Since my r210ii has two physical nic ports, I chose to have the first port (eno0) be the WAN, and the second port (eno1) be the LAN. I created two VLAN aware linux bridges in the network section under "System" of my proxmox node and assigned proxmox to
10.10.10.2 on the LAN bridge. This will give me access to Proxmox admin on LAN.
After the network was ready, I had to actually create the VM. Click the "Create VM" button in the top right of the proxmox window. A popup should appear where the properties for the VM can be defined. Here, I just called the vm "Opnsense", and the OS I picked from the USB drive, and I set up the VM with basic specs, which are listen below:
Proxmox seems to only allow connecting one network adapter to the VM upon creation, so I added the second one later in Hardware properties for the VM. Make sure both network interfaces are not firewalled, as we are installing a firewall on our VM! Having firewall rules across multiple different systems is just asking to bang your head against the metaphorical wall.
Once I had all of that set up, I booted up the VM and was welcomed with the Opnsense installation screen! I followed their instructions, and installed Opnsense on the VM.
** Note: Make sure you pick the virtual hard disk as the installation drive. If you pick the USB stick on accident, opnsense will not be installed and all of your settings will be blown away. Don't make the same mistake I did.
I personally setup my LAN network on pfsense to be
10.10.10.0/24, and the WAN was DHCP from verizon. Once I did that, I went to
http://10.10.10.1 in my browser and found the Opnsense admin splash screen. Opnsense has been installed!
By default, opnsense comes pretty locked down. Basically, anything coming out of the LAN interface is allowed, but anything coming into the LAN is blocked. Anything traveling from the LAN network out to the internet through the WAN is NAT'd to the WAN IP address. Pretty standard stuff for most internet browsing, but when you are dealing with devices that require direct internet access of some kind, like online gaming services for gaming consoles, we need to allow some explicit external natting that keeps the port the device is requesting. By default, Opnsense randomizes the port when NATing over the WAN interface for privacy reasons, so for my switch, I just made it nat to whichever port it wants and keep that port after NAT. So if the game servers use 1010, the traffic will be kept on that port through the NAT table. To do that, I went to the firewall section, chose NAT and outbound NAT. On that page, I clicked the Hybrid model, since I still wanted Opnsense's default rules, and then added the following rule:
After a quick firewall reload, my switch was able to get a NAT type of B instead of D and connect to Nintendo's online services. As a side note though, a lot of forums tried to guide me to use OpnSense's UPnP plugin, however I installed it and it didn't really seem to have much difference on my switch. The NAT type without the outbound rule, even with UPnP on, was still D and wasn't able to connect to matchmaking services.
Ok, now that we have routing setup on the LAN network, I wanted to utilize more of the power of this new hypervisor I just installed for my home router. So, I decided to move some of the network services I keep on a raspberry pi to a VM on the server. This would not only allow for lower latency for DNS requests, but also frees up a raspberry pi for use with Hyperion for ambient lighting around my living room TV (A blog post about that later).
So, I decided to move my PiHole server into a Debian VM. To do this, I downloaded a Debian VM using qbittorrent and transferred it over to Proxmox. I then created a VM called "PiHole" the same way I did above and gave it the following specs:
With these settings set for the VM, I started the instance and configured Debian to a very basic install -- No graphical user interface or extra packages, just basic Debian. After I went through the installer and installed it to the virtual hard disk, I stopped the VM, unattached the ISO from the VM, and booted into regular debian under my user. From here, I used the curl command on PiHole's website. This guided me through installing PiHole on the network. I believe I gave the VM a static IP address of
10.10.10.5. Once I could get into the admin console, I moved on to configuring Opnsense.
By default, Opnsense starts Unbound on a default installation, so I disabled this DNS server just to be extra sure that my devices don't use it somehow. Also, I checked to see if a DHCP server was already running on my LAN. When I setup Opnsense, I chose not to start one, but if it is enabled, make sure to disable it before continuing. This could cause DHCP lease conflicts between the servers, which could cause two devices to get the same IP address. Not good.
Once Opnsense was configured to stop doing DHCP and DNS duties, I started the DHCP server for PiHole and told it to hand out addresses in the
10.10.10.0/24 CIDR, starting at
10.10.10.10 and going to
10.10.10.254. This allowed me to have a few static IPs at the beginning of the CIDR for interval services like PiHole and my wireless APs.
Once you do this, you should start to see DNS requests come in if there are other devices on the LAN network. In my case, there were devices connected to my APs, so those devices started to use PiHole as their existing leases expired. Success! Now most ads on all of my devices on my network will be blocked. Good bye smart TV ads!
Overall, I've been really pleased with my new home router setup (minus the noise). Things have been going rather smoothly, and I learned a lot about Proxmox, virtualization, and really forced myself to keep my networking knowledge close to my head. Soon, I plan to setup VLANs on my network to segregate IoT devices like my Chromecast and Nintendo Switch and my Kubernetes VM network. But for now, I think I'm going to leave it as is and focus on some contracting work.
If you liked this article, would like more details, or would like to discuss anything else, please feel free to reach out! As always, I love learning how to improve my setup.
April 20, 2022
Hey! I knjow this is kinda off topic but I was wondering which blog platform are youu using for thus site? I'm getting sick and tired of Wordpress because I've haad issues with hackers and I'm looking at optons for another platform. I would be great if you could point mme in the direction off a good platform.
May 7, 2022
Amazing! This blog looks exactly like my oldd one! It's on a completely different subject but it haas pretty much tthe same page layout annd design. Great choice of colors!
binary options daily youtube videos
May 23, 2022
Hi to every , as I am genuinely keen of reading this website's post to be updated regularly. It carries fastidious material.
June 11, 2022
I have beenn absent for a while, but noww I remember why I used tto love this web site. Thanks, I'll ttry and check back more often. How frequently you update your web site?
June 21, 2022
We're a bunch of volunteers and starting a brand new scheme in our community. Your site offered us with useful info to woork on. You've done a formidable process and our whole community shall be thaqnkful to you.
July 2, 2022
Thank you, I have just been looking for info approximately this topic for a whhile and yours is the greatedt I've came upon till now. However, what in regards to the conclusion? Are you certain about the supply?
July 3, 2022
I've recently started a blog, the info you offer on this web site has helpped me greatly. Thank you for all of your time & work.
July 6, 2022
We're a group of volunteers and starting a brand neww scheme iin our community. Your web site offered us with helpful info to work on. You've done an impressive job and our entire group will be grateful tto you.
July 30, 2022
Thanks a lot for sharing this with all people you actually recognise what you're speakking about! Bookmarked. Please additionally seek advice froom my website =). We can have a hyperlink alternate agreement between us!
Aug. 22, 2022
First oof all I want to say terrific blog! I had a quick question which I'd like to ask if you don't mind. I was curious to find oout how you center yourself andd clear your thoughts before writing. I have had a hard time clearing my mind in gettng my thoughts out. I do take pleasure in writing however it just seems like the first 10 to 15 minutes are usually lost simply just trying to figure out how to begin. Any suggestions or tips? Thank you!
Aug. 22, 2022
Very interesting information!Perfect jusst what I was searching for!
Aug. 29, 2022
You made various good points there. I did a search on the theme and found mainly people wull consent with your blog.