Now that I have time for hobbies again I wanna dive right in to building up a home lab. Right now I just have a Netgear router that reached end of life, and a NAS i’d like to make accessible remotely, and a DIY router seems like the correct next step for me. I’m very interested in ad-blocking network wide and someday I’d like to host my own little webpage, but for now I’m really just barely getting started.
I’m information gathering, I don’t know what I don’t know and my networking skills are pretty weak anyway. But even if your needs and desires have nothing to do with mine, I also just like to hear people talk about their set ups. Please? 🌞
The layout I’m designing around is inline: ISP router, then a small dual-ethernet box doing firewall and DNS filtering, then your LAN. For a start, I’d put OPNsense on a cheap dual-NIC mini PC. You get network-wide blocking through a plugin and a VPN to reach your NAS, and VLANs can wait until you need them. I’m biased, since I’m building a box with that exact layout. What I can’t settle: for someone starting out, is an all-in-one box less scary, or does it hide the stuff you’d want to learn?
I usually just use proper network devices. OpenWRT will generally run on anything with Mediatec or Qualcomm based assuming it has support
Another vote for openwrt
I was using an N100 based mini PC and it did fine at 1gbit. But now that I have 3, and it’s realistically closer to 4 or 5 it doesn’t cut the mustard.
Proper hardware based routers are also just so much more efficient. That N100 machine was constantly hot to the touch. My new UniFi router uses less power and does 10 gig easily.
I have 2.5G N100 and it works wonderfully at that rate. I think higher might be tough, but it still pushes that throughput even with IPS enabled in OPNsense.
The complained about heat is right though. I stuck a noctua fan on the top and it’s been happy since. It lives my closet, so noise isn’t a facto anyways
Can I ask what the 10 gig is for?
Gotta go fast. My WiFi can achieve > 1gbit, my internal network is already 10 gig. Why not support faster everywhere?
Sometimes it’s really that simple 🥰
A used Lenovo ThinkCentre Tiny M720q mini PC, with the PCIe riser and an Intel chipset (Huawei) dual-10 Gbit NIC, running a pretty basic install of OpenWrt. Has no issue with 10 Gbit throughput, maxes out at about maybe 12 Gbit if you do a full duplex test, which I think may be a limitation of the limited PCIe lanes in the riser.
I got 10 Gbit internet at home, looked at the prices of 10 Gbit routers and noted right out! Built this for maybe $100 in stuff off of auctions.
A fanless thin client with Opnsense and a four-port Intel NIC.
This is the way!
It’s currently a N100 mini-pc running Debian, but in the past it’s been all kinds of random hardware including a busted old laptop with an expresscard-PCIe riser for the second NIC. Anything can be a router if you know what you’re doing!
thank you, this is a great post! I like your blog!
Thanks!
I run some china box with 5 integrated 2.5GBe NICs, 48GB RAM and 2x 500GB SSD/NVMe. CPU is Celeron® 6305 @ 1.80GHz, two cores or HT, who knows…
On this I run Proxmox. I didn’t like Proxmox integrated firewall, so I rolled my own.
all containers unprivileged
one container is the firewall. it has all up+down stream NICs mapped via bridges, cable internet, backup 5g internet on dedicated ports. one port to the switch with a few VLANs, and one bridge that is local to the Proxmox for the local VLANs.
Every service I operate sits in its own network/segment, usually with a /30 netmask. VMs/containers have a eth0 device in this and the firewall defined as default gw. On the firewall the VLAN interfaces are named as the container/service. So the interfaces are not eth0 but wan, wan5g, frigate etc. Great for readability.
Some networks like “family” or “guest” or the one with the APs/WLAN-controller are /24. Bit big, but what ever.
I have a structured netfilter firewall script, fw.sh calling it’s parts under fw.d/*fw. Each service is managed individually, sometimes widely open (
iptables -A FORWARD -i frigate -o wan* -j ACCEPT) but usually down to the required ports. Database, reverse proxy etc all on separate containers.There is also like 30 VPN configs from 2 VPN providers. I have a home assistant dashboard where I can direct specific devices to use specifig VPNs. My webradio is routed via UK so I can enjoy BBC2.
root@mercury:~# ip rule show 0: from all lookup local 20: from all lookup main suppress_prefixlength 1 300: from XXX lookup wan_table 310: from XXX lookup wan5g_table 1000: from all fwmark 0x1 lookup bbc-ss-lon 1001: from all fwmark 0x1 lookup bbc-ss-man 3000: from XXX lookup pm-gb 4000: from all lookup wan_table 5000: from all lookup wan_table 5100: from all lookup wan5g_table 32766: from all lookup main 32767: from all lookup defaultip rule also does the fail over to wan5g if required. There are some rules where I tried to auto-filter BBC traffic with marked connections via dnsmasq, but BBC knows all my VPNs (not BBC radio it seems).
IPSec VPNs and other OpenVPNs (dialup, site2site) are handled via a dedicated LXC. There is a tor entry node container, for a few services and one of the public WLANs. Firewall makes sure certain traffic can only exit via certain upstream interfaces, vpn*, wan, wan*… very helpful
I didn’t like opensense… don’t know why. Work a lot with firewalls in my job (mainly check point, fortigate) so have some professional background.
Also: client isolated WiFi for all iot shit. No internet for TV (jellyfin), PV controllers and most others.
i had a cheap made in china consumer router with mt7621 on it, porting openwrt was surprisingly easy and its been running well enough for the past 5 years. adguard home runs on it and handles dns adblocking
someone else suggested openwrt on my current EOL router, i might just try that! like, i think eventually I might still want something more robust, but it sounds like a pretty good starting point
even cheap consumer router these days are actually surprisingly capable. mine’s from 2015! what model do you have? might be hard if it’s a realtek/brcm chipset… mediatek(ralink) and qualcomm(atheros) are mostly well supported
Apparently it’s older than I thought and not supported! I could use another project called dd-wrt though
Got a “Dell Wyze 5070 extended”, the extended is important because it lets it have a half height PCIe card. Put a dual NIC Intel card in it. It came with 8 GB of RAM and a 64 GB SATA M.2 drive, plenty. Price out a power brick if it doesn’t come with one, which is a Dell laptop one; it refuses to turn on with a PCIe card in and if the power brick is less than 130W (DA130PE1-00 is a product # of it)!! Very annoying. Note the Intel atom processor in it lacks AVX2 instructions which makes mongoDB past v4 not work on it, so you can’t run the Unifi controller, which has vexed me. But it’s fine for internet routing! Never tested throughput but others have online.
(Note: I would not do this now!) I put proxmox on it then did opnsense (passing through the PCIe NICs) and Debian. I chose opnsense over openwrt because opnsense had an automatic update functionality, while openwrt required you to manually wipe the OS and reinstall - when they fix nasty upgrade path that I’ll come knockin. (Openwrt is like that because that’s how routers do upgrades)
I would not choose proxmox now because its upgrade path is fraught since the free level is rolling and they’ve had critical breaking update errors in the past. No good for your router setup. I want upgrades on and automatic so I don’t get kiddie script hacked.
I’m prepping to do either opnsense only and not in a VM or Debian with Incus (VM manager like promox, but not its own OS) installed and then opnsense in a VM. I’m leaning opnsense direct to just make sure my router works, but I’ve never had trouble with Debian yet so still pondering.
I run a pihole and wireguard on the current Debian VM, and opnsense can do wireguard so most of the way there. In the future I’ll switch that pihole instance (wherever it is) to adguardhome, the local rewrite rules in pihole don’t support wildcards so it’s annoying af to have to load in every local domain I set up.
This has worked well for years and I’ve never measured power but I presume it is low.
Hope this infodump helps!
It really did, thank you :) No one else mentioned updating as far as I know, and that wasn’t something I really thought about at all, coming from a commercial router. Plus I just enjoyed your journey. I think I’ll need wireguard to safely access my NAS remotely so that’s good to know too.
Feel free to ask any Qs if any come up, there is ?likely? better value than the Wyze 5070 extended now a days, I looked and bozos still want like $100 for it. Getting AVX2 would be a solid W for software in the future and worth a minor increase in cost. An ARM board (as long as it has UEFI and can boot regular Debian - not a custom distro for the board) would be a solid move too, but you’ll not get Opnsense then - it’s openwrt only for ARM.
This of course is me thinking of pricing when it wasn’t AI bro “lemme order all your RAM on an IOU” stupid, good luck there!! Just opnsense does not need much RAM - like 3 GB is easily safe (mine is given 2GB right now; uses 600 MB right now but I’ve seen it go past 1.5 GB sometimes). Straight Debian will add a gig or so. So even 4 GB would be fine. Proxmox is needy so it’s not fine there, but cutting it out of the equation would make 4 work if you want to save. And you’d prob get away with a $5 16 GB optane M.2 drive from China (aliexpress) with just opnsense - it’s using 3.2 GB of the 16 I’ve given it. And the Optane drive will never wear out as a bonus.
Edit: I’d estimate stock Debian at ~6 GB used. I think both can fit in 16 GB, even with Debian loaded with Incus to run the Opnsense in a VM.
I build a Linux system with multiple NICs. It’s heavyweight for a router, and probably not very efficient, but it does the job very well and I love it. I’m using foomuuri to manage the NFTables and NAT and it was very easy (although it’s a very obscure piece of software and the documentation and community isn’t excessively comprehensive, you might have better luck with something more common but it works nicely for me). It can pull in all the adblocking lists and all that fun stuff. I find it pretty intuitive to configure, at least compared to the horrors of most networking/routing configuration nonsense.
For DHCP I tried using the ISC’s new “kea” in the hopes it would be modern and clean alternative to the venerable old isc-dhcp-server but it turned out to be a horrible mistake. They made some awful, insane, unjustifiable design decisions in that software. It gets confused about IP addresses already being claimed (by the exact same host/MAC) address because some arbitrary value changed in how it identifies itself, and then does absurd things with static leases, it just randomly decides to give it a dynamic lease instead, and then will fixate on that dynamic lease and never let it have the static lease back, I fought with it for months, it kept breaking my network for absolutely no reason and it was a nightmare.
Back to trusty old isc-dhcp-server for me. And of course bind for DNS, I’m a bit old-school but it’s very comfortable.
dnsmasq might be a lighter-weight and easier alternative you could consider, I think it does DHCP as well, convenient.
And of course, being a full Linux server means you can put all kinds of other stuff on it too. Stuff that needs to be both publicly visible and privately visible on the LAN seems like an obvious fit, but in fact it’s probably the worst place to put those services, because you run into issues with NAT hairpinning and other nonsense. For services that are strictly intended for the private LAN/VPN, or services that are strictly public facing, it can handle those just fine by restricting them to a specific interface. But you should avoid stuff that needs to listen on both sides of the connection, it can be done, but it’s tricky and annoying and not ideal.
If you need both internal and external access, port-forwarding from the outside through the NAT to another dedicated internal host is a much better solution if possible, it’s still a bit tricky (you may have to override the DNS for that domain internally to use the private IP instead of the public one) but it’s quite reliable and functional once you have it set up.
None of this is really for the faint of heart, it’ll likely require a fair bit of trial and error and physical access to the machine to resolve issues. But it’s a fun project and I think the end result is worth it.
Nice. I recently migrated from bind/isc-dhcp to dnsmasq, which has a cleaner 1-line per reservation syntax. PowerDNS is what I manage my domains with, but not in my home-homelab.
Currently running OPNsense on a small Dual-Nic Shuttle PC with an Pentium G4560 and 4 gigs of ram and it’s been handling everything I throw at it perfectly fine. Ive got 16 Interfaces, around 10 Wireguard Peers and a SIT tunnel to HE and Im still getting Line Speeds through the firewall.
I’ve never gotten DIY routers including opnsense to give gigabit line speeds once VLANs are enabled.
a minipc with 2 nics running proxmox with opnsense as a vm and one of the nics passed through. that way I can use it for more stuff.
I used to run openwrt on a Xeon blade in my old C6100. It worked well enough but was a waste. Now it’s just a stock ER605 without the Omada controller.
Router is a bunch of old PCs I frankensteined and put opnsense on.
Ended up i5-7500, 8gb mem, 500gb SSD.
Put in a dual-port SFP+ NIC and a dual port 2.5GbaseT nic.
One of the 2.5gb ports goes to my cable modem.
One of the 10gb ports takes Twinax to my dumb LAN switch, the other to my dumb server switch, which right now is really just my storage network…the link to the firewall isn’t doing anyt
uhing.The dumb switches I have here, though, they’ve got some sort of weird MAC learning disability. It cannot handle macflaps, which I found out about because I had a hard time roaming between APs.
So I got a Netgear GS108T and put OpenWRT on it, and that switches between my APs.
My APs are Linksys MX4200v1’s. Also openwrt. Got four of them…one upstairs, two in the ground floor, one in the basement. Can get very close to 1Gbps anywhere in the house on wifi. Not the best platform for openwrt (ipq807x), as I’ve found out…but it was a hella cheap deal for tri-band AX APs.
I don’t think this is what I’m gonna end up doing, but I LOVE STORIES LIKE THIS. Frankensteining a bunch of shit together just makes me feel so nostalgic for how I got into PCs in the first place. MAC learning disability also made me laugh.
HP Prodesk with some i5 8th gen running proxmox and with a 4x 1gb pcie nic on which a opnsense vm runs.
My DNS was done through pihole. Transitioned now to technitium-dns.
My personal blocklist(s):
https://github.com/Appoxo/Ad-Spam-Blocklist
Contains a list for opnsense (IPs) and technitium-dns (but pihole should be compatible).
Anything else should be mentioned in the readme.In case of questions, feel free to ask.
