SoylentNews is people
SoylentNews
I had asked my employee to set up a server at my place, so a closed user group of us and my customers could access automated builds. Most basic stuff, a Raspi 3 behind a consumer grade DSL/WLAN router with NAT. Port forwarding on 80 and 443. Dynamic name service. Let's encrypt certificates. As simple as it gets. The particular consumer router doesn't "bounce back" the port forwarding. ICMP Pings come back from the external IP, but we can't access the server from the intranet via its external (DNS provided) IP through TCP. Lacking any local naming logic, we simply use the fixed 192.168 address, no biggie.
Now, earlier today, she called me over to show me something was working. At the second glance I noted that the main page of the server was there in a Safari window, together with the domain name in the address bar. At first, I thought maybe something with the router had changed and the bounce was working now. I tried on two other Macs, one with the same 13.1 OS, the other with an older 10.15.7. No go. Dead as usual. We retried the trick on the machine she was working on, and this time it was not connecting, back to usual.
Short of some weird flakiness in the router, the only explanation would be that the site was accessed from the outside, which would mean Safari had tunneled out and got back somehow. There is this "Apple Private Relay" thing, but we're far away from any iCloud subscription. "Privacy" settings must have been more or less "works".
I'm not too fond of the likelihood that my company traffic gets "abducted".
(Score: 2) by RS3 on Wednesday March 15, @06:09PM (19 children)
I had started a good reply and my stupid browser crashed out. I think it's a HW problem in this PC, but it passes all RAM and other diagnostic tests... I'm not going to try to recreate that reply, rather, I'll go a step at a time.
Do you have a static IP from your ISP? (actual routable IP address, not 10.x.x.x or 169.254.x.x or 172.16-31.x.x or 192.168.x.x).
I don't know how to contact you directly (DM) but if this is too slow, maybe the admins here could connect us and I can get you up and running faster. I do server / network admin (and I'm the only IT person for the sites). I sure don't know everything...
(Score: 2) by Rich on Wednesday March 15, @11:42PM (18 children)
I'm exposed to the net with a proper routable IP, the NAT is just the LAN. But don't worry. Everything works as it should. It just happened, that on this single occasion, something worked TOO good.
(Score: 0) by Anonymous Coward on Thursday March 16, @02:13AM
Regardless of the cause, it is a good sign that you need to properly set your firewalls on the host and gateway and to check your allowed routing. I doubt they are properly filtering the traffic the way you should between the various networks and endpoints.
(Score: 2) by RS3 on Thursday March 16, @04:15AM (16 children)
Thanks. Okay, I'll assume you have a proper static IP.
> "The particular consumer router doesn't "bounce back" the port forwarding."
I'm just curious- what do you mean by "bounce back"?
(Score: 1, Informative) by Anonymous Coward on Thursday March 16, @08:48AM (15 children)
They are probably describing "NAT reflection (also referred to as NAT loopback or NAT hairpinning)". This allows machines on your LAN to access the services ports forwarded on your WAN. For example, I have two machines 1.1 and 1.2 connected to a router with a LAN address of 1.0 and a WAN address of 254.254. I set port forwarding on 254.254 so that all incoming packets to port 80 are forwarded to 1.2.
If I have NAT reflection enabled I can get on the machine 1.1 and connect to 1.2 using the 254.254:80 and the traffic will travel from 1.1 to 1.0 to (having its destination changed on entry and source changed on exit) to 1.2 and back (with the addresses swapped in the opposite order, incidently).
Otherwise with NAT reflection disabled, a number of things can happen, the traffic will attempt to go from 1.1 to 1.0 to 254.254 and your router has three choices:
(Score: 2) by Rich on Thursday March 16, @10:47AM (14 children)
Yes. I was describing NAT "reflection" in what not happens.
For the thing that happens, I'd guess 1, with a silent drop. Had a look at the traffic from LAN device to external IP with Wireshark and the TCP SYN show up (double, on two ports) and then it's just TCP retransmissions (double packets, again, with the two source ports), nothing related or forwarded visible. Should rule out 3. The router is provided by the DSL provider, so I'd guess they'd be sane enough to not forward the packets upstream as in 2.
(Score: 2) by RS3 on Thursday March 16, @06:27PM (13 children)
Wow, thank you both. It all makes sense, and I was wondering if routers did that.
In my little hosting situation I'm still using a quite old router (gateway) that I doubt does that intelligent local forwarding (my wording for it). The newer routers they provide won't do proper static NAT. I have a situation where one external WAN IP ports 80 & 443 need to go to server 1, but that same IP port 22 needs to go to server 2, and port 21 needs to go to server 3.
Newer gateway will do static NAT on a port-by-port basis, but every external IP will only map the enabled ports to one internal IP- no matrix map.
I have not tried a very new gateway- maybe they'll work. I got sick of dealing with complete idiots at the ISP. It's a "business" account, but the tech "help" don't know a NAT port map from a pizza.
(Score: 1, Informative) by Anonymous Coward on Friday March 17, @02:56AM (12 children)
This sounds like the router uses a DMZ host rule internally. A kludge you can do is to give a machine on your local network multiple address. Have one set up to act as the local interface; have the other set up to do NAT. Then set the gateway appliance to port forward to the special address. You can then control the forwarding on that local machine to the rest of your hosts using whichever technique you prefer.
I do hope you and Rich have proper firewalls setup if you are hosting machines on a non-dedicated network like this. It can be surprisingly difficult to set up firewalls and other security measures to keep the inevitable breaches contained. Even the big players screw it up from time to time. Unless you really know what you are doing, you should have two routers from two vendors to act as your firewalls. in a nested configuration to separate trusted and untrusted zones.
(Score: 2) by RS3 on Friday March 17, @06:46AM (11 children)
I really appreciate your thoughts and suggestions. I'm not following a few things though. Like I don't understand your first paragraph at all. Well, I understand the scenario you're suggesting, but I don't understand the reason for why I would need to do it.
The system has been working in the current configuration for at least 12 years- up to 15- I just don't remember the exact date we switched from a very expensive T1 to a business fiber provider, but that date is recorded. The system is under constant username / pw guessing attacks, but never been broken into. I use an IP blocker- after a very few guesses, the IP gets blocked for at least a week.
I'm not sure why you're thinking the gateway uses DMZ internally, and I'm not sure why it would matter. I'm aware there've been vulnerabilities in many gateways / routers, but there is no sign that this one has ever been compromised. It might happen to be a good one.
I'm also not sure what you mean by "non-dedicated network"??
I understand your nested firewall idea, but, and I really don't mean to be offensive, and I apologize if it comes out that way, but some IT people are WAY over-cautious, like to preach, scold, and predict doom and destruction, but might be (way) over-zealous. I know some personally.
The sites we're serving are not critical, no-one will be harmed if they ever get hacked. And, I know you'll love this, many are WordPress. Again, I've been running them for 15 years. So, either I'm just very very lucky, or, everything is set up well.
There's one amusing glitch: in one and only one WordPress site, occasionally, maybe once every two or three months, a spam comment will show up in the "waiting to be approved queue". It's so rare I've done little more than to go through and check all settings. It may be a weakness in a plugin, but there are fairly few plugins installed, none have ever been on a CVE or any other problem list, and I keep them updated (as well as WordPress itself).
As I'm reviewing what I've written, it occurs to me that there may be scenarios where people have some kind of Internet modem, but no router / firewall. That is a sad situation. Maybe that scenario spawns the doom and gloom preachers? If so, please know, and check, there are lots of well configured gateways out there. I've always had very well configured gateways with all incoming ports blocked by default, always NAT to 192.168.x.x (or 10.x.x.x) LAN.
(Score: 1, Informative) by Anonymous Coward on Friday March 17, @08:15AM (10 children)
My understanding was that you were only allowed to port forward to a single internal IP address and that you wanted to be able to forward different services on different addresses. An easy kludge around that restriction is to use a dedicated forwarding machine on that single internal IP address. If done correctly, it can actually increase security as a side effect because you can react to attacks spread to all your services at once. The internal DMZ host comment was really a side note because it is an easy and brain-dead way to implement port forwarding in a router with a web interface but only allows a single forwarding address.
A non-dedicated network is a network with multiple uses with different threat models. In your example, I read that as you having a network with both web-accessible servers and internal machines. Those two purposes are fundamentally at odds with each other. Therefore, it can be really hard to properly secure them without defining a good perimeter. The easiest way to do that is to just have two routers in a nested configuration. I understand the proclivity some have to over-secure to the point where it is difficult for users. But, I also understand the sorts of attacks that are out there and that half of those protecting against them are below average (and that many of those below average don't consider themselves below average).
It doesn't really have a problem with WordPress itself. It is secure enough anymore, especially if you stay away from random plugins and themes. It is just the fact that you have people out there that don't appreciate just how dangerous even seemingly innocent things can be (It's just a pretty mouse pointer/screensaver, after all). And that the bad guys only have to succeed once and many have the patience to wait years for the opportunity to make the profit worth it. You don't have to actually have anything of value as long as they think you do and sometimes your value to them may be different that what you think.
As a final note to your "scary" thought. It is completely true. That is one reason why ISPs love giving people a modem/MTA/ONT/etc. with built-in routers and Wi-Fi. The firewall on them can drastically cut down on compromised machines. Ultimately the ISP ends up paying quite a bit to deal with all of them. But the threat is really the same. You leave the gate open for random people to come in and then bad things happen because they manage to unexpectedly get through the door.
(Score: 2) by RS3 on Saturday March 18, @02:58AM (9 children)
Thank you again for all the info- I love to learn. My biggest problem with this discussion is jargon / terminology.
Anyway, my situation is atypical, and most people make a lot of assumptions (it's human nature). For example, the server room is locked, can be alarmed (but not necessary), in small office building which is kept locked and alarmed when empty.
There are NO computers or WiFi on the server LAN- except if I'm there and turn on a workstation. Point is, nobody has access to the LAN.
It's not a full nor part-time job- very tiny operation, very limited budget. I pride myself on keeping it going for almost nothing. Owner is very successful and very busy guy, many other businesses and work. If not for me, he'd shut the operation down.
There's much much more detail, but I'm trying to be as brief as possible.
Sorry, I try to be clear, but I'm always amazed at how people can misunderstand me. Point I was making is that I did a port map matrix 12+ years ago. It is in place right now and has always worked very well. ISP has sent me several newer gateways over the years, but none have allowed me to do an IP matrix. So I'm keeping the 12 or so year old gateway in place. I'm not sure why they're forcing one external IP to map to one internal IP. Any and all ports can be enabled, but not cross-mapped. We own 5 static IPs, but the ISP has 10 set aside in our name. I'm not sure if they're being routed to us- I never tried. My luck is I'd use the IPs and ISP would get wind of it and cut them off.
But 5 is plenty, as long as I can do the IP NAT matrix.
Any ideas about why routers would only map one external IP to one internal IP? Is it considered a security thing? I assumed it was laziness on the programmers who write the code for the gateway's routing function.
Incidentally, this hosting operation was on T1 when I first started helping 15 years ago. Two or three Cisco routers. I have no formal training in Cisco IOS programming, but looked into it. My dad brought home a study guide / course once. So I did some stuff with the Ciscos, but in researching Cisco programming I came across a forum where someone wrote about how there are many ways to write a Cisco config, and you'll get the same input-to-output functionality, but some ways of doing it result in very low (slow) throughput. I found that very discouraging. Cisco had (has?) a java-based browser / GUI tool, but only for some routers. I sort of got it to work, but barely, so I gave up and just used the ISP's provided gateway.
(Score: 1, Informative) by Anonymous Coward on Saturday March 18, @04:56AM (8 children)
Now I understand. I thought you meant that they were trying to force you to upgrade, such that your current solution working solution was no longer allowed.
The reason for mapping the address the way they do is exactly for the reason you describe: laziness. I'm not sure how familiar you are with the packet flow in the Linux firewall and the corresponding pushdown machine. But long story short, the reason why they do it is because they can implement almost all features of the firewall using only 6 types of rules: 1. default deny, 2. outgoing NAT, 3. connection tracking, 4. forwarding internal traffic, 5. port forwarding to a DMZ host, 6. exceptions to the default deny. By manipulating those rules (mostly 1, 5, and 6), which can easily be set up to be handled by your UI, you can get almost complete functionality with relative ease and most people won't notice. But if you know your firewall rules, you'll also know that implementing a complete firewall that includes your "NAT matrix" isn't really that much harder. But if you don't know them, even getting that far can cause you to tear out your hair.
(Score: 2) by RS3 on Saturday March 18, @05:49AM (7 children)
The option I most seriously considered (wish Cisco stuff was more straightforward, no secrets to optimizing) was to build a simple Linux machine to be a firewall / router / NATter, and just use the ISP's as a modem, somewhat firewall.
Is "NATter" okay now? Websters? Oxford? Anyone?
(Score: 0) by Anonymous Coward on Sunday March 19, @12:24AM (6 children)
Look up the cost of Cisco's service offerings and that they make for than 11 billion dollars a year in net income. I think those two facts together more than explain why Cisco sees what you and many others bemoan as a feature.
(Score: 2) by RS3 on Sunday March 19, @03:31AM (5 children)
Yes, it's the most common business model- rape the market. Give away the razor, sell blades. I don't know whether to laugh, cry, or move to a South Pacific Island. Maybe Caribbean, buy a boat, take rich people out on excursions. Yeah, that.
Cisco expert CCNA / CCNEs make pretty good $, and at the time I thought- this is great, I have some Cisco rack routers, I can learn, study the courses my dad gave me, then take official certifications, hopefully on someone else's tab. But that layer of programming complexity, and even more so the idea that I could become expert on those routers, and in 3 years they'll be "deprecated" and my expert knowledge worthless. Kinda happened to me with Novell early 90s. I learned all that on the fly, even wrote some custom .nlm (server modules), liked it, wanted to get into it, but no, MS had to crush Novell, and "business people" making technical decisions forced Windows on us. I had already been very interested in *nix, and had Xenix running at home. Even bought "Coherent", it was cheap, installed and ran well, but I got the feeling it was a toy. Had messed with QNX and wanted to get into that. Fortunately found Linux (SLS) and that solved everything for me. :)
So again, if I ever need to, I'll set up a dedicated Linux firewall / router / load balancer. :)
Thanks!
(Score: 0) by Anonymous Coward on Wednesday March 22, @07:56PM (4 children)
OpenBSD's base system networking software interfaces are actually *really* close to Cisco in most cases, and they have pretty comprehensive coverage of everything you'd want for a network appliance right in the base system with no need to install anything. If you want to learn 90% of Cisco without getting a cert or buying hardware, I'd recommend spinning up a few VMs and playing around with it.
(Score: 0) by Anonymous Coward on Wednesday March 22, @10:06PM (3 children)
Any network stack will get you most of the way there. Cisco bases most of their systems on Linux. Although, if I were to recommend anything, it would be a dedicated router distro. The underlying OS doesn't make a huge difference other than hardware and accelerator support.
(Score: 0) by Anonymous Coward on Wednesday March 22, @11:55PM (2 children)
Of course the functionality is duplicated everywhere, but the actual interface for eg. openbsd ospfd is basically exactly the same as Cisco's, PF is not too different from configuring the real Cisco firewall (but way better), and OpenBSD basically *is* a router distro because it ships everything needed in the base system.
(Score: 0) by Anonymous Coward on Thursday March 23, @02:37AM (1 child)
They are all basically the same, and many of the daemons are literally the same. There are more considerations than what is in the base system or repos. By that measure, Debian basically is a router distro too. Regardless OSPF is overkill for the kind of work that they are talking about. A handful of rules in your firewall would easily handle what they need out of a router.
(Score: 2) by RS3 on Thursday March 23, @04:42AM
You guys are so awesome- thank you. For many reasons I lost interest in learning Cisco IOS and programming, including that the Cisco hardware I have is much too slow for the 80+ Mbit fiber I have now.
Also, it's a little too specialized for my blood. Part of my loss of interest was like you're saying- there are other ways to do the job, including just setting up a Linux machine to do the work. I'd much rather put the time and effort into that. Linux routing seemed much easier, the hardware is super inexpensive, I can spec it to my heart's content, not being beholden to a monster company who will deprecate my router in 2 years, and the replacement (which I could in no way afford nor justify) will bring many unknown changes.
Again, more power to the people who specialize in Cisco- I have great admiration for them (and y'all too!) That stuff is for big "enterprise", not little specialized hosting with only a few servers running a few dozen sites, run by a guy (me) who has too many other interests and talents (music / audio, electronics (analog hardware design), cars, ultrasonics, some programming, ...)
(Score: 0) by Anonymous Coward on Wednesday March 15, @08:02PM (1 child)
> consumer grade DSL/WLAN router with NAT
It's something to do with wlan bridging, common to multiple routers but wasn't observed on wired eth ports. Basically, accessing a LAN IP over WLAN somehow translates to the WAN interface. This typically works even with the WAN cable disconnected, so your traffic probably isn't being routed externally. As to if it's a specific chipset limitation or software NAT bridging issue - no idea.
(Score: 2) by Rich on Wednesday March 15, @11:16PM
The machine this happened on indeed was on WLAN. However, I've got a wild mix of wired (said server, laser printer, old PC, other raspis) and wireless (laptops, phone) devices. I mostly work from a laptop on WLAN, and I have never observed the weird behaviour, in many years, so it would have to be a very weird flakiness in the router to route WLAN through the external IP exactly this one time. Given the likelihood of that fluke, or Apple gathering data for "service quality" purposes, I'd bet on the latter.
(Score: 0) by Anonymous Coward on Wednesday March 15, @08:11PM (8 children)
NAT is not a security feature, it's a fucking hack to get around the limits of 32-bit addressing. Configure a firewall on the router or install a physical one in front of your server.
(Score: 3, Informative) by Rich on Wednesday March 15, @11:00PM (6 children)
To clarify: The router dials into the provider over DSL with PPPoE/LCP/IPCP and gets assigned a proper external IP address. Everything NAT is behind the router, which doubles as firewall. I have activated port forwarding from the external IP address to the server, so the external connection goes to, say 84.123.121.12 (made up, legit worldwide IPv4). The router then NATs connections on 80 and 443 to the server at 192.168.0.100. The router itself also has 192.168.0.1 (aside from that 84.123.121.12), which is the LAN gateway. Completely standard fare for residential LANs with a little serving.
As I have a historic dynamic DNS, the server also polls the external IP and updates the dynamic DNS provider, so e.g. "rich.dnsprovider.net" resolves to that 84.123.121.12. When I access the network from the outside, I can use the 84.123.121.12 IP or the domain name, the router forwards to the server at 192.168.0.100 and everything is jolly fine. It's just that when I access 84.123.121.12 (by means of using the proper "rich.dnsprovider.net" domain name) from INSIDE the LAN (that is from the 192.168.0.0/24 range, it does not port-forward (or better said bounce-back) the connections to 192.168.0.100.
The strange thing is that in that ONE occurence, and never at anytime else observed, one Mac seemed to be able to connect to "rich.dnsprovider.net" from inside the LAN.
(Score: 0) by Anonymous Coward on Wednesday March 15, @11:05PM (4 children)
I see. Barring some kind of local caching resolver or hosts file, if the situation is not reproducible on that or other machines, I'd guess the machine in question was not actually "inside the LAN".
(Score: 2) by Rich on Wednesday March 15, @11:31PM (3 children)
How would it get "outside"? Another poster mentioned a weird thing where WLAN sometimes gets routed through the external IP. Something which has not ever been observed here, in years. The device in question was an entry level ARM MacMini that certainly doesn't have anything WWAN near it. And I think the likelihood of someone doing a drive-by hijacking of my WLAN is also small.
My bet would be on some worldwide Quality-of-Service monitoring for the iCloud VPN, independent of any subscription. (The particular Mini had iCloud off, it's actually being used for testing DISA conformant access hardening of a vertical market application).
(Score: 2, Informative) by Anonymous Coward on Thursday March 16, @12:16AM (1 child)
If it's connected over wireless, it may have been connected over a different network, such as a mobile hotspot or reachable open network. Many systems will automatically connect to any available source. My guess is that someone was messing around trying to make things work and didn't verify correctness of the results before displaying them.
Browser page caching weirdness is also possible, and is not really transparent to the end user even for open source browsers, let alone Safari.
DNS-over-HTTPS is enabled by default in many cases and browsers purposefully ignore system settings and hosts files. I don't know how MacOS does DNS, but it's entirely possible they locally cache results and that could easily lead to confusing situations when testing or moving across networks and the browser and system are not necessarily resolving names using the same methods.
(Score: 0) by Anonymous Coward on Thursday March 16, @07:05PM
Yes, I have seen that. It's why I always recommend to turn off "automatically connect to networks within range"
(Score: 2) by RS3 on Sunday March 19, @03:39AM
Any chance it had something to do with ARP tables / forwarding? You might be able to view the ARP table and see that everything makes sense (IP to MAC).
(Score: 0) by Anonymous Coward on Thursday March 16, @02:29AM
It sounds like you have loopback disabled for the NAT appliance and your ISP router doesn't forwarding downstream packets back to the upstream. The three ways to access that service would be either the Mac was actually using a different connection, like preferring a mobile hot spot someone accidentally had on, or through a tunnel to an outside service, which could be any number of things including a VPN or one of any number of IPv6 transition mechanisms, or the ISP temporarily allowed their router to send downstream packets back on the upstream they came from.
(Score: 0) by Anonymous Coward on Thursday March 16, @02:21AM
Yep. NAT is not a firewall. It can sometimes block you from connecting like a firewall, but then my dog stepping on the power button is a firewall too.
(Score: 2) by fab23 on Wednesday March 15, @09:38PM (2 children)
On macOS 13.x (Ventura) check in the System Settings / Network / Wi-Fi on the connected Wi-Fi (below Known Networks) through the circle with the 3 dots (...) and choose "Network Setting..." in the menu. Then check if this is enabled (not sure if this can be enabled without logged in iCloud / AppleID):
Limit IP address tracking
Limit IP address tracking by hiding your IP address from known trackers in Mail and Safari.
As I am a very atypical macOS user with using Mozilla Firefox and Thunderbird instead of Apple Mail and Safari, I have not seen such problems so far.
(Score: 2) by Rich on Wednesday March 15, @11:39PM (1 child)
This could explain it. It is some kind of external server relay. Maybe Safari wasn't sure whether my domain was some kind of "tracker" and wanted to probe that. However, for a manually entered single domain name that resolves to a single html page that sets no cookies, it would be more than slightly overaggressive.
OTOH, with this scheme, for actual tracked websites, by correlating the trackers "hidden", Apple will get a very good idea where I've been.
(Score: 2) by fab23 on Thursday March 16, @07:39AM
I have no idea why your site matched.
In the document Prepare Your Network or Web Server for iCloud Private Relay [apple.com] they also mention this, what I remember from when this feature was presented:
So there is no information leak in general.
(Score: 0) by Anonymous Coward on Friday March 17, @06:13AM
You sure it's not the Private Relay stuff?
https://whatismyipaddress.com/everything-you-need-to-know-about-apple-private-relay [whatismyipaddress.com]