Virtualizing OpnSense On My Home Network

Jan. 20, 2021

Kyle Kaniecki

TL;DR: I setup my home networking with virtualized Opnsense and piHole. Yes, it is overly complex, but I kept my networking knowledge somewhat sharp.

I've always been curious about networking infrustructure, ever since the early days of my prepubescent childhood. I remember when I first got my xbox 360 around my 6th grade birthday, I immediately was curious on how it connected to the internet, how it connected to xbox's servers, and how hostbooters were able to find my "address" called an ip address and kick me offline. For those who don't know, "hostbooting" is gamer slang for a DoS attack on a client's public ip address. Essentially flooding their home network with packets to disconnect them from online gaming services. People would disconnect players on the other team for an easy win.

Anyway, that passion always carried itself with my all the way until my first intership at Datto, where I was pulnged head first into the world of networking. I remember going into the job thinking: I've been programming for over 3 years at this point. I'm pretty comfortable with computers, and have programmed a few things into my raspberry pi. I can handle this internship.

"Oh boy, was I wrong..."

Quickly, I was barraged with terms like router, modem (yes, they are different), switch, and access point. My team was talking about packet routing and firewall rules, how to forward intervlan traffic across a network and QoS services. Needless to say, I felt very overwhelmed during my first week, and I quickly scrapped my idea that this was going to be an easy internship. Nonetheless, I saw this as a challenge. I knew that people -- regular humans -- had created the technology that my coworkers were talking about, and if they had created it, then I could learn it.

Ok, enough of story time.

Setup


One of the first, and most important, things that I had been planning before moving to Boston was getting a Dell r210 II to setup Proxmox on. I wanted to get an enterprise grade server due to three very important reasons:

  1. I wanted to make sure I could get a background server that was low power. The r210 II seems to run around 60W idle, so that was right around what I anticipated
  2. I wanted to get some experience with enterprise level server equipment, as I was planning on moving this blog's infrustructure and others to a home hosted setup.
  3. And last but not least, it's pretty fucking cool.

So, I found a cheap one on Reddit's /r/homelabsales subreddit, and it was waiting at our apartment the day we arrived to move in.

The next thing I researching was which virtualization platform I was going to use to set all of this up. I looked into VMware's Esxi platform and I had used it before, but I found that their support for older hardware just didn't quite do it for me. I have a super old Chenbro server laying around that I plan on running when I need worker nodes, and I wanted to make sure that I could master one virtualization platform for my home lab instead of just knowing two. They also rolled their own kernel, and while I think that is great for enterprise network hardening, I really felt more comfortable in a debian linux environment. So, with this, I decided to go with Proxmox, as it seemed more "user friendly," had less advanced features enabled by default to make getting up and going easier, and is supported by the open source community.

Proxmox Installation


Now that the virtualization platform has been established, it is time to install that bad boy onto the server. To do this, I simply downloaded the Proxmox VE iso image using qbittorrent and flashed it on a flash drive with dd.

$ sudo lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
nvme0n1     259:0    0 476.9G  0 disk
├─nvme0n1p1 259:1    0   512M  0 part
├─nvme0n1p2 259:2    0     8G  0 part [SWAP]
├─nvme0n1p3 259:3    0    20G  0 part /
└─nvme0n1p4 259:4    0 448.4G  0 part /home
sdb           8:48   1  14.6G  0 disk
$ sudo dd if=/path/to/proxmox-ve_6.3-1.iso of=/dev/sdb bs=1M status=progress
$ sync

Using the lsblk command, find the disk file for the flashdrive. Using that disk name, run the dd command and copy the ISO over. If everything went well, you should see dd writing the ISO to the usb drive. Wait for this to be done, and then run the sync command. This will ensure the disk write is complete and all changes and saved to disk. The disk shouldn't be mounted, but I ran lsblk again just to double check, then unplugged the drive and put it into the back of the server.

Proxmox has a pretty great installer, so just follow along with the steps until it is running on your system. I don't want to mindlessly rewrite their installation guide, so I will link it here. However, if we don't have a router yet to route packets or run a DHCP server on, how will we reach the proxmox web admin to start our VMs? The computers on the network don't know how to communicate with each other yet. The answer is simple, static routing.

Static Routing on Arch Linux


Most of the time, I use my wireless access card on my laptop to connect to the internet. Using wpa_supplicant and dhcpcd normally suffices in most use cases, but in this case I hadn't gotten my APs up and running yet so I couldn't use the wireless card. Instead, I used a simple USB to ethernet adapter I had lying around and used that interface to connect to proxmox. After plugging the USB adapter in, I used the infamous ip command to setup a static IP and route for my subnet through the USB adapter. But first, we need to look for the new interface name.

$ ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state DOWN mode DORMANT group default qlen 1000
    link/ether fc:77:74:92:97:08 brd ff:ff:ff:ff:ff:ff
3: enp0s20f0u1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether 8c:ae:4c:e1:7a:af brd ff:ff:ff:ff:ff:ff

Ok, so looking at that output, I could see a new interface named enp0s20f0u1. This was the USB interface I was looking for, however if you are following along, yours may be different. Milage may vary. Also, the interface is in the UP state already, so no need to manually bring the interface up, in my case. Should you need to bring the interface up, use the following command: sudo ip link set dev <interface_name> up

Once we have our interface up, we need to assign an IP address to it so Promox knows who is trying to talk to it and where to send replies back to. This is sorta like a traditional mailing address. If someone decides to send a letter from their house and not put a return address on the letter, the receiver would have no idea where to send the reply, and sending a reply to everyone in the neighborhood isn't an option either if you are discussing your tax returns. The same idea applies to computers. If we need to communicate with a server, we need to sign our packets with a return IP address. In our case, the IP address we choose to put on our interface must be within the network’s subnet. I chose to use 10.10.10.10/24. Not for any particular reason, but because it gave me some room to set up my APs at the beginning of the subnet. To set the interface IP address, use the ip command again, however this time we will use the addr subcommand.

$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: wlp0s20f3: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 8c:ae:4c:e1:7a:af brd ff:ff:ff:ff:ff:ff
3: enp0s20f0u1: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 8c:ae:4c:e1:7a:af brd ff:ff:ff:ff:ff:ff
$ sudo ip addr add 10.10.10.10/24 dev enp0s20f0u1
$ ip addr show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: wlp0s20f3: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 8c:ae:4c:e1:7a:af brd ff:ff:ff:ff:ff:ff
3: enp0s20f0u1: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 8c:ae:4c:e1:7a:af brd ff:ff:ff:ff:ff:ff
    inet 10.10.10.10/24 brd 10.10.10.255 scope global dynamic noprefixroute enp0s20f0u1
       valid_lft 43132sec preferred_lft 37732sec

Great! Now we can see our interface has an ipv4 address associated with it, so we have our return address attached to our "letters" to proxmox. However, now our interface has no idea where to send our packets. Going back to the mail analogy, do we use Fedex? UPS? USPS? (The correct answer is support the USPS). Back in computer land, our interface isn't quite sure, so we need to tell it. To do this, we need to setup a static route, which tells our computer which interface to use when sending packets to the 10.10.10.0/24 subnet. Again, let's use the ip command.

$ ip route show
$ sudo ip route add 10.10.10.0/24 dev enp0s20f0u1
$ ip route show
10.10.10.0/24 dev enp0s20f0u1 scope link

As you can see, I had no routes setup the first time I ran the command. Then, I added a static route that says "any traffic destined for the 10.10.10.0/24 subnet, send it out the device named enp0s20f0u1". This allows the packets to be sent out the correct interface to get to our switch and then routed to Proxmox. Once everything is finished and if everything went well, you can go to the proxmox machine's static IP address in a web browser, and see the following page:

Now granted, I already have a few VMs running on my machine, otherwise I wouldn't be able to write this blog post now, but it should look relatively close to this. If this page is appearing, the static route worked! Great. Let's move on to the opnsense installation.

Installing OpnSense


Next, I needed to install Opnsense to route packets around on my network, as well as block unwanted connections from the internet. Everywhere I have read online recommends using pfsense as a home opensource router, but I found that Opnsense has a little bit better of a UI experience, and since editing the firewall from the UI tends to be a better experience, I sided with Opnsense. There has also been controversy over pfSense switching their software license from BSD to Apache, but for homelab use like mine, it didn't make any difference.

So first, I downloaded the Opnsense ISO installer in the same fashion as the Proxmox installer, however this time I put the image on a regular USB drive, no dd command involved. This is because I want to mount the installer USB image to my VM and install to the virtual disk, so I need it on the proxmox box raw instead of on a bootable USB.

To add the ISO image from the USB, plug the drive in. However, before creating the VM, we need to setup some network configuration for the Opnsense VM so we can connect to the VM through a lan interface and get internet from a WAN interface. Since my r210ii has two physical nic ports, I chose to have the first port (eno0) be the WAN, and the second port (eno1) be the LAN. I created two VLAN aware linux bridges in the network section under "System" of my proxmox node and assigned proxmox to 10.10.10.2 on the LAN bridge. This will give me access to Proxmox admin on LAN.

After the network was ready, I had to actually create the VM. Click the "Create VM" button in the top right of the proxmox window. A popup should appear where the properties for the VM can be defined. Here, I just called the vm "Opnsense", and the OS I picked from the USB drive, and I set up the VM with basic specs, which are listen below:

  • 2 vCPU
  • 4GB of RAM
  • 32 GB of physical ROM storage on a 1TB hard drive on the server
  • Default graphics driver
  • Added the WAN network to the VM to start, not firewalled.

Proxmox seems to only allow connecting one network adapter to the VM upon creation, so I added the second one later in Hardware properties for the VM. Make sure both network interfaces are not firewalled, as we are installing a firewall on our VM! Having firewall rules across multiple different systems is just asking to bang your head against the metaphorical wall.

Once I had all of that set up, I booted up the VM and was welcomed with the Opnsense installation screen! I followed their instructions, and installed Opnsense on the VM.

** Note: Make sure you pick the virtual hard disk as the installation drive. If you pick the USB stick on accident, opnsense will not be installed and all of your settings will be blown away. Don't make the same mistake I did.

I personally setup my LAN network on pfsense to be 10.10.10.0/24, and the WAN was DHCP from verizon. Once I did that, I went to http://10.10.10.1 in my browser and found the Opnsense admin splash screen. Opnsense has been installed!

Opnsense Home Setup


By default, opnsense comes pretty locked down. Basically, anything coming out of the LAN interface is allowed, but anything coming into the LAN is blocked. Anything traveling from the LAN network out to the internet through the WAN is NAT'd to the WAN IP address. Pretty standard stuff for most internet browsing, but when you are dealing with devices that require direct internet access of some kind, like online gaming services for gaming consoles, we need to allow some explicit external natting that keeps the port the device is requesting. By default, Opnsense randomizes the port when NATing over the WAN interface for privacy reasons, so for my switch, I just made it nat to whichever port it wants and keep that port after NAT. So if the game servers use 1010, the traffic will be kept on that port through the NAT table. To do that, I went to the firewall section, chose NAT and outbound NAT. On that page, I clicked the Hybrid model, since I still wanted Opnsense's default rules, and then added the following rule:

After a quick firewall reload, my switch was able to get a NAT type of B instead of D and connect to Nintendo's online services. As a side note though, a lot of forums tried to guide me to use OpnSense's UPnP plugin, however I installed it and it didn't really seem to have much difference on my switch. The NAT type without the outbound rule, even with UPnP on, was still D and wasn't able to connect to matchmaking services.

PiHole


Ok, now that we have routing setup on the LAN network, I wanted to utilize more of the power of this new hypervisor I just installed for my home router. So, I decided to move some of the network services I keep on a raspberry pi to a VM on the server. This would not only allow for lower latency for DNS requests, but also frees up a raspberry pi for use with Hyperion for ambient lighting around my living room TV (A blog post about that later).

So, I decided to move my PiHole server into a Debian VM. To do this, I downloaded a Debian VM using qbittorrent and transferred it over to Proxmox. I then created a VM called "PiHole" the same way I did above and gave it the following specs:

  • 1 vCPU
  • 1GB of RAM
  • 16 GB of physical ROM storage on a 1TB hard drive on the server
  • Default graphics driver
  • Added the LAN network to the VM, as only devices on my local network should be able to get to the VM directly

With these settings set for the VM, I started the instance and configured Debian to a very basic install -- No graphical user interface or extra packages, just basic Debian. After I went through the installer and installed it to the virtual hard disk, I stopped the VM, unattached the ISO from the VM, and booted into regular debian under my user. From here, I used the curl command on PiHole's website. This guided me through installing PiHole on the network. I believe I gave the VM a static IP address of 10.10.10.5. Once I could get into the admin console, I moved on to configuring Opnsense.

By default, Opnsense starts Unbound on a default installation, so I disabled this DNS server just to be extra sure that my devices don't use it somehow. Also, I checked to see if a DHCP server was already running on my LAN. When I setup Opnsense, I chose not to start one, but if it is enabled, make sure to disable it before continuing. This could cause DHCP lease conflicts between the servers, which could cause two devices to get the same IP address. Not good.

Once Opnsense was configured to stop doing DHCP and DNS duties, I started the DHCP server for PiHole and told it to hand out addresses in the 10.10.10.0/24 CIDR, starting at 10.10.10.10 and going to 10.10.10.254. This allowed me to have a few static IPs at the beginning of the CIDR for interval services like PiHole and my wireless APs.

Once you do this, you should start to see DNS requests come in if there are other devices on the LAN network. In my case, there were devices connected to my APs, so those devices started to use PiHole as their existing leases expired. Success! Now most ads on all of my devices on my network will be blocked. Good bye smart TV ads!

Conclusion


Overall, I've been really pleased with my new home router setup (minus the noise). Things have been going rather smoothly, and I learned a lot about Proxmox, virtualization, and really forced myself to keep my networking knowledge close to my head. Soon, I plan to setup VLANs on my network to segregate IoT devices like my Chromecast and Nintendo Switch and my Kubernetes VM network. But for now, I think I'm going to leave it as is and focus on some contracting work.

If you liked this article, would like more details, or would like to discuss anything else, please feel free to reach out! As always, I love learning how to improve my setup.


comments

Please enter a valid display name
Please enter a valid email
Your email will not be displayed publicly
Please enter a comment

Please enter the letters into the box

captcha

Carlo

April 20, 2022

Hey! I knjow this is kinda off topic but I was wondering which blog platform are youu using for thus site? I'm getting sick and tired of Wordpress because I've haad issues with hackers and I'm looking at optons for another platform. I would be great if you could point mme in the direction off a good platform.


Yvette

May 7, 2022

Amazing! This blog looks exactly like my oldd one! It's on a completely different subject but it haas pretty much tthe same page layout annd design. Great choice of colors!


binary options daily youtube videos

May 23, 2022

Hi to every , as I am genuinely keen of reading this website's post to be updated regularly. It carries fastidious material.


Vicky

June 11, 2022

I have beenn absent for a while, but noww I remember why I used tto love this web site. Thanks, I'll ttry and check back more often. How frequently you update your web site?


Vera

June 21, 2022

We're a bunch of volunteers and starting a brand new scheme in our community. Your site offered us with useful info to woork on. You've done a formidable process and our whole community shall be thaqnkful to you.


Porter

July 2, 2022

Thank you, I have just been looking for info approximately this topic for a whhile and yours is the greatedt I've came upon till now. However, what in regards to the conclusion? Are you certain about the supply?


Lynette

July 3, 2022

I've recently started a blog, the info you offer on this web site has helpped me greatly. Thank you for all of your time & work.


Milford

July 6, 2022

We're a group of volunteers and starting a brand neww scheme iin our community. Your web site offered us with helpful info to work on. You've done an impressive job and our entire group will be grateful tto you.


answerdev.com

July 30, 2022

Thanks a lot for sharing this with all people you actually recognise what you're speakking about! Bookmarked. Please additionally seek advice froom my website =). We can have a hyperlink alternate agreement between us!


Marvin

Aug. 22, 2022

First oof all I want to say terrific blog! I had a quick question which I'd like to ask if you don't mind. I was curious to find oout how you center yourself andd clear your thoughts before writing. I have had a hard time clearing my mind in gettng my thoughts out. I do take pleasure in writing however it just seems like the first 10 to 15 minutes are usually lost simply just trying to figure out how to begin. Any suggestions or tips? Thank you!


Margie

Aug. 22, 2022

Very interesting information!Perfect jusst what I was searching for!


Jerilyn

Aug. 29, 2022

You made various good points there. I did a search on the theme and found mainly people wull consent with your blog.


link building Images

Oct. 7, 2022

Your mode of explaining all in this paragraph is genuinely pleasant, every one be able to easily know it, Thanks a lot.


Fup

Oct. 10, 2022

Свежее 3D порно на любой вкус! Сочное порно видео, 100% бесплатно. Смотрите на любом устройстве! <a href="https://www.casinozru.com">furry sex</a> <a href="http://www.casinozru.com">porno 24</a> <a href=https://www.casinozru.com>uzb sex</a> <a href=http://www.casinozru.com>porno cast</a>


Best SEO

Oct. 21, 2022

Greetings from Ohio! I'm bored to tears at work so I decided to check out your website on my iphone during lunch break. I love the info you present here and can't wait to take a look when I get home. I'm surprised at how quick your blog loaded on my mobile .. I'm not even using WIFI, just 3G .. Anyways, superb site!


마카오 슬롯 잭팟바카라 깡 배팅

Nov. 3, 2022

Thanks for one's marvelous posting! I certainly enjoyed reading it, you will be a great author. I will ensure that I bookmark your blog and will come back in the foreseeable future. I want to encourage that you continue your great writing, have a nice evening!


Website Speed Test

Nov. 4, 2022

I will right away clutch your rss feed as I can not find your email subscription link or e-newsletter service. Do you've any? Kindly allow me recognise in order that I may subscribe. Thanks.


ดูหนังออนไลน์ 5 แพร่ง

Nov. 16, 2022

Excellent beat ! I would like to apprentice while you amend your website, how can i subscribe for a blog site? The account aided me a acceptable deal. I had been a little bit acquainted of this your broadcast offered bright clear concept


ข่าวฟุตบอลยุโรป

Nov. 16, 2022

When I originally commented I clicked the "Notify me when new comments are added" checkbox and now each time a comment is added I get three emails with the same comment. Is there any way you can remove people from that service? Thank you!


eromangabbs.com

Nov. 16, 2022

Useful information. Fortunate me I found your site by accident, and I'm shocked why this accident did not took place in advance! I bookmarked it.


https://eromangabbs.com

Nov. 24, 2022

This post is genuinely a nice one it helps new web users, who are wishing in favor of blogging.


mangakiss.net

Nov. 24, 2022

Very good article. I absolutely love this site. Keep it up!


https://i-movie.org

Nov. 24, 2022

Hello i am kavin, its my first occasion to commenting anywhere, when i read this post i thought i could also create comment due to this good article.


ознакомься тут

Nov. 24, 2022

Друзья, прошу прощения за оффтоп. Сегодня прочитал в Сети любопытную публикацию про русскую порнушку и открыл кое-что новое: ссылочка на сайт https://promtu.ru/biotualetyi/mr-little-mini.


ข่าวฟุตบอลล่าสุด วันนี้

Nov. 24, 2022

I have read so many posts regarding the blogger lovers except this post is actually a good paragraph, keep it up.


вот эта ссылка

Nov. 25, 2022

Приветствую. Вчера нашел полезную статью про русскую порнуху и открыл много нового для себя - кликай по ссылке - http://popki.pro/znamenitosti/skarlett-yohansson/.


tracfone coupon

Nov. 29, 2022

I just like the valuable information you supply for your articles. I will bookmark your weblog and check once more right here regularly. I'm somewhat sure I'll be told many new stuff proper right here! Good luck for the following!