Rocket Surgeon
I own 2 OpenWRT routers. Fun little things. Love em.
But running a virtual firewall is a perfectly reasonable goal. OpenWRT doesn’t have the feature set that OPNsense has.
They are not the same sort of product. Lot of common ground, but not the same thing.
There’s a few things we don’t know here.
There’s nothing wrong with your plan, but that’s not how I would do it.
I don’t pass through NICs. I bond them or I bridge them.
In a virtual world, this sort of task is done with virtual switches. OVS switches at my job.
OVS is a lot easier to use than oldskool linux bridges that come installed with Proxmox. There’s already a dropdown in Network where you can build with OVS objects, but you need to add the package.
apt install openvswitch-switch
MGT. For your setup, I might consider (the onboard!) eth3 as my mgt NIC. That might be handy some day if you have to remove that card. Your server will still be online.
DMZ WAN. I would run the WAN line straight to eth1. Add eth1 to a ‘dmz’ virtual switch. Add the OPNsense WAN leg to this dmz virtual switch, so the OPNsense (and nothing else) can directly talk to the upstream router.
LAN Virtual. Create a ‘protected network’ virtual switch. Add the OPNsense LAN leg to this virtual switch. VMs can be a member of this downstream protected network and access any services provided by the OPNsense.
LAN Local. If you need to share the OPNsense protected network back out to other devices, add eth2 to to the protected network switch, and ethernet cable out from eth2 to a dumb switch. Plug other external devices into the dumb switch, and they will be downstream from and protected by your OPNsense, accessing its services.
Feel free to ignore me here. I build a lot of big things, so I use enterprise-scale techniques. There’s nothing wrong with your pass-through plan.
And … you can do this! I have a somewhat similar setup on my laptop with HyperV, so I can distribute wired (work VPN) and wireless (everything else) internet to guest VMs and the main OS. I made two virtual switches in HyperV.
Good luck!
I think because I got into NextCloud to “de-google”, I just accepted some maintenance load.
My intent was to manage my documents. The CORE office suite works sufficiently. Now I update my resume (and manage a few other documents) on my own website, no google.
I uploaded a bunch of music. The music apps are kinda crappy, but they also work sufficiently.
I never got around to setting up email for NextCloud. Looks like maybe I never will.
My searxng instance in my homelab died, and I nuked it. I think when I rebuild, its going on this VPC.
I see NextCloud here quite a bit. What I don’t see discussed is the cloud hosting option.
Hetzner has two NextCloud offerings.
They have a storage service backed by NextCloud. I think that’s new. I don’t use that.
They also have have NextCloud available as an installable app from their service.
This is what I use. https://docs.hetzner.com/cloud/apps/list/nextcloud/
I had to actually follow the directions to get it running.
And I have to patch it occasionally. Mostly, it takes care of itself.
I use the cheapest 1vcpu VPC.
Yes my NextCloud service is a bit sluggish sometimes. But I’m paying like $6/month.
I’m pretty happy with it for storing music and updating my resume.
Maybe if I had the more demanding use cases like some commenters, I might not like it so much.
Installed the Audio Player app. It’s a bit less odd for playing music than the Music app.
… but the Audio Player dashboard widget is worse. So I’m using both apps.
I patched from 31.0.8 to 31.0.12 to 32.0.3. No real problems.
I might get around to fixing these. I tried for a while after I upgraded. No luck.
My API Deploy Damon wont connect and the Websocket looks like I need to install something?
I think I need to do some config on the back end.
The help pages I’ve found for this stuff are all on github.
There’s info there, but they kinda slam through the topic without much explanation.
This is a hetzner default nextcloud setup, so I may well have to refer to their docs too.
My instance works fine; I’m not too concerned about these errors. Any input is very welcome.
I own two GL.inet routers. I liked my Flint so much that I bought an Opal for my office and on the road. These machines are well provisioned. The OpenWRT reviews of them say to just leave the stock bootloader installed. I’ve installed all sorts of packages, multiple subnets, VPN, adblock, etc. GL.inet gear is good stuff.
Cisco offers a whole lot of free online training as well, on several different websites.
Its kind of a pita to get access in the first place, but its definitely free, comprehensive, and starts from the ground up.
And of course, they have paid training options on those same sites.
Well … How much do you want to learn? How serious are you?
If you want to know networking, the authority is Cisco.
I’m scheduled to take my CCST Network exam tomorrow. That’s an entry-level Cisco cert.
I’ve been studying for about 3 months. Wish me luck …
Junior NetAdmin Cert
The CCST training is online and entirely free.
https://www.netacad.com/career-paths/network-technician?courseLang=en-US
Access
You’ve got to jump through some hoops. You need to create an account and go through some verification.
They need to figure out if you are ‘overseas’ and whether you should be able to download encryption products.
I think its probably easiest if you use your work email, that’s what they are really looking for.
Cisco U
There’s a shit-ton of free classes at Cisco U as well.
Most of those are not directly cert-related, but a large amount of them were created for people studying for the CCNA, so they are certainly helpful. There’s all sorts of rando training, keep ya real busy.
Here’s one I’ve started.
https://u.cisco.com/paths/understanding-cisco-data-center-foundations-20705
Lab Environments
The whole study program uses Packet Tracer for the labs, which you download from them.
I also got a copy of Cisco Modeling Labs running. That was a bitch, had to shoe-horn an OVA to run on Proxmox.
And I got an older edu copy of the Cloud Services virtual router, if there’s anything these other lab environments can’t handle. (This version can be freely downloaded … csr1000v-universal9.03.12.00.S.154-2.S-std.iso)
Heh, try posting on the Proxmox forums. Piranhas will eat you up.
Sorry to hear that tho. This comm doesn’t help much. Got about 5 readers here.
I’ve posted here and had it deleted. So I don’t bother.
The instance I’m a member of had an unused selfhosting comm, and I started using it. Other people did too. Thanks for the shout out.
Alright already! I’ll work on my upgrade.
I’m wondering if I should just build a new docker and then migrate the data instead of upgrading in place. I bet that’s the easier thing to do in the end. Sounds safer too. I got backups and all, but …
Netbox is a hell of a package, of which I’ve essentially only touched the IPAM, and I don’t even use it programmatically. I just use the web console to keep track of 4 subnets and about 50 IPs.
It’s got a whole virtualization section that I haven’t touched, although that would make my device mapping more sensible. I just treat em like they are all real, and only map the physical nics on the hypervisor hosts.
I do keep text notes in Netbox entries, but that’s sort of a backup. If its something I’m likely to need to know, I’ll have a note in Proxmox. Usually login links for apps hosted there and the like. And of course I’ve got a folder full of text files with all my deepest secrets.
I’m not real clear what exactly you need to document.
Infrastructure documentation starts with an IPAM.
A good IPAM can help you document all kinds of stuff.
I use NetBox.
https://github.com/netbox-community/netbox?tab=readme-ov-file#getting-started
I’m running it as a Docker container on a Linux VM.
I just looked at their latest screenshots, and it appears they’ve done quite a bit with it since I stood up my copy.
It does even more now. I’ll have to upgrade.
Cool.
Here. SSH key issues. There was a huge forum war.
https://forum.proxmox.com/threads/ssh-keys-in-a-proxmox-cluster-resolving-replication-host-key-verification-failed-errors.138102/
But its still a thing. That still needs to be fixed by a human. Today that’s me.
Regarding CEPH and corosync on the same network … well I’m just getting started with that now. I do have them on different vlans, but its the same 10gb set of nics. I’m hoping if it gets really lousy, my netadmin can prioritize the corosync vlan. I’ll burn that bridge when I come to it.
EDIT …
The linked forum post above leads to the SSH key answer, but its convoluted.
Here’s what I put in my own wiki.
Get the right key from each server.
cat ~/.ssh/id_rsa.pub
Make sure they match in here. Fix em if they don’t.
/etc/pve/priv/authorized_keys
There’s a couple symlinks to fix too, but this should get it.
SSH key management in PVE is handled in a set of secondary files, while the original debian files are replaced with symlinks. Well, that’s still debian. And in some circumstances the symlinks get b0rked or replaced with the original SSH files, the keys get out of sync, and one machine in the cluster can’t talk to another. The really irritating thing about this is that the tools meant to fix it (pvecm updatecerts) don’t work. I’ve got an elaborate set of procedures to gather the certs from the hosts and fix the files when it breaks, but it sux bad enough that I’ve got two clusters I’m putting off fixing.
Corosync is the cluster. It’s a shared file system that immediately replicates any changes to all members. That’s essentially anything under /etc/pve/. Corosync is very sensitive. I believe they ask for 10ms lag or less between hosts, so it can’t work over a WAN connection. Shit like VM restores or vmotion between hosts can flood it out. Looks fukin awful when it goes down. Your whole cluster goes kaput.
All corosync does is push around this set of config files, so a dedicated NIC is overkill, but in busy environments, you might wind up resorting to that. You can put cororsync on its own network, but you obviously need a network for that. And you can establish throttles on various types of host file transfer activities, but that’s a balancing act that I’ve only gotten right in our colos where we only have 1gb networks. I have my systems provisioned on a dedicated corosync vlan and also use a secondary IP on a different physical interface, but corosync is too dumb to fall back to the secondary if the primary is still “up”, regardless of whether its actually communicating, so I get calls on my day off about “the cluster is down!!!1” when people restore backups.
I use PVE professionally. I could spent some time bitching about how it handles ssh keys and the fragile corosync cluster management. I could complain about the sloppy release cycle and the way they move fast and break shit. Or all the janky shit they’ve slapped together in PBS. I could go on.
But I actually pay for a license for my homelab. And ya, it is THE thing at work now.
I’ve often heard it said that Proxmox isn’t a great option. But its the best one.
If you do try it, don’t bother asking questions here.
Go to the source. https://forum.proxmox.com/
Hmm. I used to volunteer with Free Geek in Portland OR. It was essentially that, an e-disposal site and we made refurbs for community organizations. But they did have a store for sale to the public.
I have so much computer junk. I got rid of most of it, but then I got a bunch more when we closed the company office. Got at least 10 monitors, 5 PCs, a mini, couple laptops … and a storage shelf to put it on.
Buy? That’s garbage. Look in garbage places. Used shops of any sort.
I like the college junk store suggestion. I used to do that.
You better watch it. You will shortly have a closet full of junk computer parts.
Ok. Yes, my use case is a private document and media store. I’m ungoogling.
VPN seems like a good place to start. But I’d like a simple answer, and I expect there are none to be had. As you’ve illustrated here, I’ll find a reason to punch holes in the firewall. And then I’m going to need to secure a web server. Life happens. I’ll keep it simple for now while I sort things. Thanks for your perspective.
Cool. Yes, this looks reasonable. It looks logical.
So, my main recommendation is consider the use of virtual bridges to manage the network instead of passthrough. And I recommend installing and using the OVS style virtual bridge.
https://pve.proxmox.com/wiki/Open_vSwitch
This gives you flexibility going forward. Say you want to run something out in the DMZ instead of behind the firewall, well you just attach that VM to the DMZ bridge instead. And it gives you an easy way to provision network for VMs. You just attach them to the LAN bridge.
(RoaS is a terrible name. Router on a Stick. It means your router is on the same switch as its clients, and all the communications go up and down that one port. It’s a perfectly legit way to manage a network, but sorta ugly and not what you are doing with your fancy 3-port rig. :)