My homelab is out of control. It’s the main reason that I am starting this blog. Currently when I want to add to my homelab the process looks like this:

  1. Add device / container / vm to the network.
  2. Set it up and configure it.
  3. Forget how I set it up and half remember how it links to the rest of my lab.
  4. ??????
  5. Profit.

Something needs to change. My goal with this blog is to document the cleaning up and organisation of my homelab. Nothing here will be a tutorial, just a place for me to dump my projects so I have a rough idea of how things work. I will be creating a proper system for managing documentation in the future, it will probably be a blog post. By writing out each step in this clean up project, in blog format, I hope that it forces me to remember more about how my own lab operates, and have a public facing whiteboard for all my half thought out ideas of streamlining my setup.

Background

My homelab has grown over the years. It started with a Western Digital My Cloud box, and ever since it has had more additions and changes than I can count. It’s a ship of Theseus at this point, no original part is really the same, with one exception: my subnet. For all intents and purposes my network is still the same flat network I had all those years ago. That was fine when the only infrastructure on the network was an all in one NAS solution, but it’s not fine now. The WD My Cloud is long gone, and so is that simple network. I now have smart devices, VMs, containers, switches, a proper firewall, APs, and a real NAS, not just some random off the shelf solution. All of that is sitting on the same broadcast domain. Given the goal of this blog is to clean up my homelab, I thought starting out with a properly segmented network was a good starting point. The network is the backbone of the entire lab, so where better to start than there?

State of the Network Address

So what does my network currently look like? Three subnets, loosely speaking. My WireGuard server on my firewall hands VPN clients addresses in 10.0.0.0/24. I also have a work-from-home VLAN on a hidden SSID for my work laptop and phone, it lives on 192.168.50.0/24. And everything else, from my APs to my Home Assistant VM sit on one flat 192.168.1.0/24 network. I’m not happy with it. My telly and my workstation shouldn’t have the same level of access to everything on my network. Especially given my TV is made by LG and they haven’t exactly been behaving recently.

Why haven’t I done this already?

Mostly procrastination to be honest, but also just a lack of knowledge. I had read a post on some forum somewhere years ago about how multicast across VLANs wont work. I have Google home speakers at home, and also sometimes cast to my telly. Now had I bothered to actually Google the problem rather than take the post at face value I would have quite quickly come across mDNS reflectors. They seem to solve that issue.

Another reason for not doing this before is actually part of the motivation to do it now. Hardcoded IP addresses. When I first started adding services to my network it was fine. When theres two or three containers or programs running and they need to talk to each other it’s fine to just hard code the IPs. But now that my network is substantially bigger than that it’s become an absolute pain. Any small change means that I have to chase down all the random locations that the IP is hardcoded. Compose files, config files, .env files, random home assistant integrations, firewall rules. That’s not sustainable. In an ideal world I would like to make changes to just one file, and everything else just follows suit. I’m a long way off from that dream. But one day I’ll get there. And having a properly setup network where I can just use hostnames makes that dream inch a little bit closer.

The Plan

So as of now I already have one properly setup VLAN, the WFH_VLAN. That’s going to stay, it will get repurposed as the WFH / Guest VLAN. As for everything else, well it’s getting replaced. I am going to create 4 additional VLANs, as well as keeping the WFH_VLAN.

  • VLAN 10 - INFRA_VLAN
    This will contain all my network infrastructure: firewall, AP, switches.
  • VLAN 20 - TRUSTED_VLAN
    This will be where trusted devices live, clients like personal workstations, laptops, phones, etc.
  • VLAN 30 - SERVER_VLAN
    Pretty self-explanatory, servers, VMs, containers, etc.
  • VLAN 40 - IOT_VLAN
    Anything that’s a smart iot device, along with devices that don’t need to talk to the outside internet, eg. my printer.
  • VLAN 50 - WFH_VLAN
    This is the current WFH_VLAN I already have, I’m going to also double it up as my guest VLAN as well. Basically any device that I don’t have any level of control over gets dumped on this VLAN.

There is also the fact that my DHCP pools are, with the exception of WireGuard clients, all in the 192.168.x.0/24 address space. I want to be a proper grown up with 10.x.x.0/24 addresses. You might notice the complete lack of IPv6. That’s not an oversight, it’s a strategic decision to not have to bother learn how it works. That’s a future project of it’s own.

Getting started

Step 1

The first step was the easiest, I didn’t actually have to do anything. I just double checked that all my hardware - firewall, NICs, switches, and AP - all actually support 802.1Q VLAN tagging. And they do, ticking off step one was a nice easy dopamine boost to start the project with.

Step 2

The first real step was creating the interfaces. Each VLAN needs one. So I went into my firewall (OPNsense) and went to Interfaces → Devices → VLAN. WFH_VLAN was already there with tag 50, the only change I’ll be making to that is it’s DHCP addressing, but that’s for a later step. I created 4 new interfaces, one for each of the new VLANs, then went to Interfaces → Assignments and gave each one of them my LAN side NIC as a parent, enabled the interface, and gave them their gateway IPs.

Step 3

The first problem rears it’s head, honestly I’m surprised I made it all the way to step 3 without something breaking already.

The DHCP decision
I’ve been running OPNsense for quite a while, long enough that ISC was still the default DHCP backend when I first set it up. It’s since been deprecated. So while I’m at it I may as well upgrade to something newer and shinier, Dnsmasq. Dnsmasq is the current default for OPNsense according to it’s documentation so that’s what I’m choosing. Kea was another alternative but it has a bug with DHCP on VLAN interfaces, and given DHCP working across VLANs is part of what I’m after Dnsmasq seemed like the obvious choice. The plan for the moment is to setup the new VLANs on Dnsmasq and leave my current flat network running on ISC. I don’t want my known working DHCP setup to break while I’m migrating everything to the new VLANs foreshadowing. I run Unbound for DNS so I don’t want Dnsmasq to fight Unbound for port 53 so I set Dnsmasq’s DNS port to 53053

Next is creating the DHCP ranges in Dnsmasq for each of the VLANs. In Services → Dnsmasq DNS & DHCP → DHCP Ranges I added a range for each VLAN (10.0.10.100 to 10.0.10.199 for INFRA etc.). That sorts out my dynamic pools, now time for DHCP reservations. I don’t like having clients on the network try to define their own static addresses. My goal here is to make my homelab more manageable. Having all static addresses managed from within the firewall itself keeps things organised. Currently devices are a mix of client side static addresses and DHCP reservations. I want them all to be DHCP reservations. Dnsmasq doesnt like when you assign addresses that are outside the pool so I made a spreadsheet of everything I wanted to be static, and then made columns for device name, hostname, MAC address, current IP, and new IP. I decided on what IP addresses I wanted to give each device on each VLAN, and then added them to the hosts section of Dnsmasq and with their new IPs and hit apply.

Then at my core switch I setup up a trunk port carrying all 5 VLANs, alongside the original flat VLAN on tag 1. Then I setup another 2 trunk ports on the same switch, one to another switch downstream and the other to my AP.

Let’s be real, the steps in this post are completely arbitrary and basically just used as a way to structure this post, but phew I’m glad to be done that step, it was a long one.

Step 4

I’m starting to lose my marbles about now, the very reason I avoided doing this before now was the fact I thought mDNS wouldn’t work. And it won’t, because I haven’t setup the mDNS repeater. So let’s fix that. In System → Firmware → Plugins I searched for os-mdns-repeater in the community plugins and installed it. Then I refreshed the page and went to Services → mDNS Repeater and selected IOT, SERVER, and TRUSTED as the bridged interfaces. I also kept the current LAN network on it to make migration easier. Then I thought I added a floating firewall rule to allow UDP traffic on port 5353 with a destination of 224.0.0.251. This means mDNS traffic should be able to cross between LAN, IOT, TRUSTED, and SERVER.

Step 5

Time to test, in theory everything should work now.

“Theory will take you only so far” - Cillian Murphy 1939

I set up a test IOT SSID on my AP and assigned it VLAN 40. Then I tried to connect to it on my phone. It wouldn’t connect, it was stuck on the Obtaining an IP address stage. What‽‽ Turns out Dnsmasq’s DHCP service never actually started. ISC was still bound to the DHCP socket on the original LAN, I thought they could run side by side as ISC only had the original LAN, and Dnsmasq had all the other VLANs. But no, that didn’t work, so I had to do what I really didn’t want to, change ISC to Dnsmasq on my working production network, that definitely wont cause issues more foreshadowing. I went to Services → ISC DHCPv4 → LAN and disabled ISC. I then gave LAN a range pool in Dnsmasq the same size as my one in ISC.

And? It still didn’t work. Dnsmasq is supposed to automatically create firewall rules on the new interfaces but it didn’t. So I manually added the pass rules for DHCP, DNS, and general internet to the IOT interface. Then I tried connecting to the IOT SSID on my phone. It connected!! It gave my phone an IP address in the correct range 10.0.40.100 - 10.0.40.199.

Step 6

Now that the IOT network is working it’s time to move onto the next VLAN. I chose to go in order of least importance. IOT was a new network and that’s why I chose it as my first one to do. It should be the easiest to setup. I started this after work on a Friday so I knew I wouldn’t need my WFH_VLAN for atleast a couple days, so it’s next on the list.

WFH_VLAN has already been moved over to it’s new address on 10.0.50.0/24. So all I have to do is add some firewall rules. This interface already has existing rules for blocking it from seeing anything on the original LAN network, so I just had to extend the rule to all the new VLANs. While I was at it I also added this rule to my IOT VLAN. There was already a rule in place to drop the connection if trying to access the management interface on the firewall from within the VLAN, so I copied that one over to IOT as well.

Step 7

Onto the next VLAN, I decided to get TRUSTED moved over next. Added the standard set of rules, allow DNS, DHCP, Internet. I also added a broad allow rule from TRUSTED to servers so that TRUSTED devices can access services on my network, Jellyfin, file shares, home assistant etc.. I also added a narrow allow from my workstation into INFRA so that I don’t lose access to my switches management interfaces when I eventually move them onto VLAN 10. I also added a pretty broad allow rule into IOT so that device control and casting would work.

I retagged the port on my switch that connects to my PC to VLAN 20 and after resetting the connection in my network settings I had a new IP, the static one I assigned my workstation 10.0.20.101!!!

Step 8

This was a pain. It was a pain for the exact reasons I started this project. A lack of documentation. I knew my NAS had two NICs, but what I forgot was that both were actually in use. I added an Intel i226-V NIC to my NAS a while back so that it had 2.5g networking. Which has been working great, I can read and write to my spinning rust across the network at gigabit plus speeds. But I forgot that the Home Assistant VM which runs on TrueNAS (I know using TrueNAS as a hypervisor is daft, but splitting up compute and storage is a future problem) uses a macvtap interface sitting on top of the gigabit NIC. Also the gigabit NIC connects to a gigabit switch, which is downstream of the core switch, which the 2.5g NIC is plugged into. So that just made things awkward, two NICs, connected to two switches, needed to move in one go. That hurt my head trying to think about. But that wasnt even the hard part.

A Bug’s Life

Three bugs, all caused by weird and undocumented shenanigans, all committed by past me lead to these bugs.

Hopper

The first bug is a result of whatever nonsense I did when I added the 2.5g NIC to TrueNAS some year(s) ago. I added an alias IP to the 2.5g NIC with it’s new IP of 10.0.30.101, then I flipped the switch port over to VLAN 30 and … I couldn’t access the webui at it’s new IP. Pings work, I can even access my file shares, but the webui just wouldn’t load. Turns out when I added the 2.5g NIC I statically set the webgui interface IP to TrueNAS’ old one, after switching back the port on the switch back to VLAN 1, I could access the webui at it’s old address and I was able to set it to the new one. And after that I could reach it in my browser at the new IP!!!

Flick

Surprise surprise, hardcoding IP’s came back to bite me again. TrueNAS runs some Docker containers, in fact it runs loads(again I know this is silly, I have plans to properly split up compute and storage, ram and ssd prices are holding me back though). Some of the containers are run using the TrueNAS apps catalog, the rest are deployed as docker-compose stacks(the way the lord intended). But my reverse proxy NPMplus is a TrueNAS catalog app. I use NPMplus for handling internal domains and their SSL certs. I have an alias assigned on the 2.5g NIC so that NPMplus can use port 80, 81, and 443 without conflicting with TrueNAS’ own webui port bindings. So into TrueNAS, into apps, edit NPMplus, and then change all the IP assignments, I did leave these static (I know, I know) but I don’t think I’ll be changing my VLANs again soon anyway(which is probably what I thought when I set it up in the first place). After that was working again I just had to go into NPMplus and change all my old IP’s over to the new ones.

Atta

This one was a pretty easy fix, I was tired at this point and just missed a setting on my downstream switch for the VLAN settings for the gigabit link into TrueNAS. I set Allowed VLANs to 30, egress tagging setup correctly, but I left the PVID set to 1 instead of 30. I’m not going to say how long I spent hunting that down(hint: it was a long time).

I’m definitely forgetting a lot when it comes to getting this VLAN setup. It was small issue after small issue, none of which were particularly hard to fix, but I reached my tech nerd limits after already having worked a day in my real job of tech support, and then a couple hours of my accidental second job of fixing my homelab. However here’s some quick fire things that I can remember but couldn’t be bothered to write up properly:

  • Added a firewall rule to allow retrieval of RTSP streams from 2 devices in IOT_VLAN
  • I changed some stale overrides in Unbound
  • Home Assistant had some hardcoded IPs
  • A load of Docker containers were communicating with each other with old hardcoded IPs, changed them out for hostnames
  • Had to fix some Docker networks so containers could see each other

End of Day 1

Trusted, Guest, and Servers are all moved over -ish. Only a couple of smart home devices were added to the IOT VLAN. My brother is the fan of the smart home products, I prefer old fashioned light switches. So I had to wait for him to get home from work and rejoin all the smart items that were under his accounts over to the new IOT SSID. At this point it’s pretty late, I’m finished with thinking about tech for the day, so I met up with some friends and we chatted about everything except tech. In theory there’s only one more thing to move, the scariest, my network infrastructure.

Day 2: Step 9

I started this again on day 2. The network is in a half moved over state. The original VLAN 1 is still there and handling most Wifi clients. I haven’t changed my main SSID over to VLAN 20 yet, it still has a lot of IOT devices on it so I don’t want them to be clients on TRUSTED. So I had to wait for my brother to be free. In the meantime I cleaned up a few loose ends.

The first was pretty easy, my doorbell camera wasn’t showing up in my NVR Frigate. I made sure the doorbell was assigned the right VLAN tag, which it was, so I checked something that has been a bit of a theme, hard-coded IPs. Frigates config still had the old 192.168.1.0/24 address, I changed that for a hostname and it worked !!

Step 10

Pangolin, what I use as an alternative to Cloudflare tunnels, has been giving me issues. I expose my Frigate instance, but when I go to it’s site on my mobile data it won’t load. I double checked that the target IP was correct, but still when I tried to visit the site I just got a page that said, no available server. I was checking logs in Newt and Pangolin, I was inside the newt container doing wgets to see if it could see my Frigate container, and it could. The actual answer was pretty simple, and made me feel a little bit silly when I figured it out. In Pangolin there is a health check. It basically uses a http request to the website to see if it’s still up, well guess what I did? I hardcoded the IP! So I changed that and suddenly Frigate, and my other public facing services were up and running.

Step 11

I think at this point I’ve said a variation of this for each step, but this was the difficult one. I wanted to be careful with this VLAN, INFRA_VLAN. This is where the management interfaces for my switches and AP live. If I mess this up it will become a pain to fix. That’s why I’m moving it last, I wanted to make my mistakes on the easier to troubleshoot VLANs.

First things first, let’s get the firewall rules in place. DHCP, DNS, Internet, narrow rule for my workstation to reflect the rule already in TRUSTED, a rule for NPMplus to reach the admin interfaces for proxying, and a block to everywhere else on the network.

Now to actually move a device over to this VLAN. I thought, let’s start with the AP, it will be the easiest. It wasn’t. I tried to reach the AP. I have an entry for it in NPMplus, so I went to AP.mydomain.tld and I could’t connect, no worries I thought, I’ll try going to it’s IP address, that didn’t work. When I changed over my orginal VLAN to Dnsmasq from ISC I didn’t bother keeping the IP reservations, no need I thought as everything will soon be on the new VLANs. Well the AP picked up a new address from Dnsmasq’s new pool on the original VLAN. I check the leases and see the APs new address. I put the new IP into my browser, and nothing. It wouldn’t load on HTTP or HTTPS. So I pinged it, it responds to pings. My Wifi clients still have internet. I don’t know what’s going on. I try to curl the webpage but I don’t get the expected response. At this point I decided to just factory reset the AP. So I do that, and again, it comes back up, it responds to pings, but I can’t actually get to it’s management interface. So since I factory reset it I knew it would have an IP address of 192.168.0.254. So I got my laptop, plugged an ethernet cable into the AP, and set a static IP in the same subnet. And finally I had access. I set up the AP, added back my SSIDs, and then changed the management VLAN to 10. I plugged it back into the switch and went back to my workstation. And I still couldn’t access the management interface. But I could from my laptop connected to the Wifi. So from my laptop I poked around the settings, the AP needed an update, and in the bug fixes section of the patch notes I saw these words written. Fixed an issue where the device would go offline in certain scenarios after configuring a VLAN id for an SSID. So I updated the AP. After it came back up I still could only reach it on my laptop connected to the Wifi. After some more poking around in the settings I saw a toggle for Layer-3 Accessibility. I can’t believe I didn’t spot this before. After turning that on, I immediately was able to reach the admin interface from my workstation. After I could reach it again I setup my SSIDs with the correct VLAN tags and everything seemed to connect perfectly.

Step 12

With the AP done I was a little scared to tackle the switches. So I started with the gigabit switch. My PC connects into the core switch so I didn’t want to start there. In the gigabit switches management interface I was able to bind the management UI to VLAN 10. I set it’s new IP and gateway, along with a new DNS entry, and it worked first try. I was able to access the switch at it’s new address from my PC which has a narrow firewall rule allowing it access to INFRA_VLAN. That was much more painless than the AP.

Now for the core switch. Turns out my core switch doesn’t have great settings for VLAN management. I can set port access and setup trunks inside of it, but I couldn’t explicitly set a management VLAN. I wasn’t sure if the switch could only be managed over VLAN 1. So I decided to just go for it and changed it’s IP to the new address, and set the gateway as INFRA_VLANs gateway, and then temporarily disabled VLAN 1 in my firewall. And it worked. I have no idea how the switch is handling management VLANs under the hood, but it works, so probably best not to question it too much.

That’s the INFRA_VLAN done, the AP was annoying and stressful, I was expecting it to be the easiest, turns out it was an absolute nightmare. Do I wish I discovered the fact that Wifi clients could access the management page before I factory reset it? Maybe. But it’s working now, no use dwelling on the past.

Step 13

A nice easy one. Printers are normally annoying to deal with, but my trusty Brother printer just kinda worked. It’s connected to Wifi. So I visited it’s management page at it’s IP address, and went through the Wifi setup wizard. I added it to my IOT SSID, and a couple seconds later it spit out a page saying it was connected to the network. I use CUPS so that it’s easy to print from any device on my network, so I logged into CUPS and changed the hardcoded IP to a hostname, and it just worked. Glad to have a nice easy win at this point.

Step 14

I wanted to be able to reach the OPNsense web UI from each VLAN, except IOT and WFH, using a domain. I went with something along the lines of gw.mydomain.tld. I didn’t want to do it using NPMplus as it cant really handle one domain resolving each of the gateway addresses for each VLAN. So doing it through OPNsense seemed like the best route. There isn’t a way to do it through the GUI. After some Googling I saw that some people were adding a config file to /usr/local/etc/unbound.opnsense.d/. So I did the same, and I was able to reach my firewall from all the configured VLANs with gw.mydomain.tld. One small issue, OPNsense didn’t actually let me login. Turns out DNS rebind protection is a thing, so I added gw.mydomain.tld to the Alternate Hostnames allow list in system settings. And then it worked.

Then I tried to reach my dashboard using it’s domain, dash.mydomain.tld and it wouldn’t load. A curl just returned Could not resolve host. Long story short I restarted Unbound a couple of times, double checked my overrides, and still nothing. Turns out in my config file I didn’t add view-first: yes. It defaults to no, so after explicitly making it yes I was able to load my dashboard after flushing the DNS cache on my PC.

Step 15

At this point things are mostly working, I’m still chasing down hardcoded IPs accross the network, but for the most part everything is working. But the flat network is still operating the background. So I need to actually get all the smart devices moved off the main SSID. My brother has the cloud accounts for the random smart stuff. So when he got back from work we went through and reset Google homes, smart plugs, an IP cam, and got them all setup on the new IOT SSID. That was mostly painless, as painless as dealing with random smart product apps can be. I readopted everything in home assistant, my brother readopted everything in the Google home app, and just like that all the smart stuff was on the IOT network.

Now came the real test. What happens now that all the devices I don’t really trust are on the IOT SSID. Time to switch my main SSID over to TRUSTED_VLAN on tag 20. Then I disabled the flat VLAN, and removed VLAN tag 1 from all the switch configs. It worked, I wasn’t really expecting that, at this point I’m conditioned to expect a breakage when I try something. My phone, my laptop, my brothers phone, everything was happy to take a new IP address. Devices on the TRUSTED_VLAN were able to access services living on the SERVER_VLAN. One small problem. I forgot enable the floating firewall rule for mDNS. After that I was able to cast a song from Spotify on my phone, over to the Google home on the IOT VLAN.

Step 16

One last hurdle. Everything else seems to be working perfectly, the network is segmented and it seems like it’s working. Except for Jellyfin on my telly. I use the Jellyfin app from the LG content store. And it would not connect to jellyfin.mydomain.tld, it would just show a blank screen. I tried adding a firewall rule that allowed the TV to see NPMplus so that it could load Jellyfin on it’s domain, but no. Even when trying jellyfin.mydomain.tld in the browser it still wouldn’t load. I tried playing with proxy headers and nothing worked. So I caved, I changed the firewall rule so that the TV could see just Jellyfin, and I just hardcoded the IP on the TV app. You can’t win them all.

So it’s done?

I think so, I’m writing this on a Sunday night after battling the network for the past 2 days. And so far? It’s working. I have tried to test as much as possible, and nothing seems broken. That makes me suspicious. But as you know (well atleast I assume you do if you’re willing to read this far into a networking article) the real test comes with time. Day to day use over the next few weeks will really show what’s working and what isn’t. Actually I do know of atleast one thing that’s broken, my Grafana network dashboards are definitley heavily broken, but that’s something that I’m ignoring for now.

Overall I’d consider the project a success. Online tutorials make it look so easy to setup VLANs on a network, and I’m sure it is when your tutorial enviroment is working under the assumption that the network, firewall, switches etc. are all brand new and never used. But due to the unfortunate reality of the real world, it was significantly more difficult to migrate my living, breathing network over without having any significant down time. I’m happy with how it turned out. I feel relieved to know that random smart devices on the network live in blissful ignorance on their own VLAN.

What’s Next?

I’m not sure, I have more ideas for my lab than I do time or money. Maybe I’ll show the fixing of my Grafana dashboards, but quite frankly that would be very boring. When I get richer, or ram prices become normal again, then I’ll properly split up compute and storage on my network, but neither is happening soon. What I will be doing in the near future is setting up a public Stratum 1 time server, the GPS module is on the way, so maybe that will be the next post on here. But it’s late on a Sunday, I don’t want to think about the next project now. So here is where I’m leaving the blog. Please ignore all the spelling and grammar mistakes, I’m not bothered to proof read.

Thank you for reading,

Luke