from the too-much-of-a-good-thing-is-not-so-good dept.
Bandwidth Limits at NWS (National Weather Service) Could Hobble Our Weather Apps:
The National Weather Service's data budget is hitting a ceiling, and it apparently doesn't have the money to fix the situation. Now, the agency is considering throttling users who access its essential forecast data.
In a notice dated Nov. 18, 2020, the NWS said that it is seeking public comment on a proposal to impose data limits on users who access its wide range of public web services. "As demand for data continues to grow across NCEP websites, we are proposing to put new limits into place to safeguard our web services," the memo says. "The frequency of how often these websites are accessed by the public has created limitations and infrastructure constraints."
[...] If nothing else, the data limits could cause services that rely on NWS data, like many smartphone weather apps, to be hours behind the real-time measurements they've been accustomed to accessing. Another possibility is that services switch to gathering information from competing agencies around the globe like the UK Met Office and ECMWF. But services that aggregate data from multiple agencies would be losing out. What's more, it seems that this issue would be really cheap to fix.
The agency held a public hearing on Tuesday (you can watch the archive here, but you have to register). Answering user-submitted questions, officials said that the agency has estimated an expansion of broadband capacity at NWS would only cost $1.5 million.
Democratic Sen. Maria Cantwell sits on the committee that oversees the agency's budget, and she told the Post that she thinks she could find support among her colleagues to make up for the budget shortfall. "Telling people to limit their use of this critical data is not an acceptable answer," Cantwell said.
So, how much does AccuWeather pay for that NWS data?
(Score: 4, Informative) by canopic jug on Sunday December 13 2020, @01:44PM (2 children)
Well, there's the problem. They're loading not just whole pages, but pages filed with cruft. If I hit their observations page, I get 44 objects with 1.3 MB of data transfered and wasting 1.5 MB of resources on the client. About are wasteful graphics like bitmapped headers. That can, and should, be trimmed way down and cached as well.
Money is not free speech. Elections should not be auctions.
(Score: 5, Interesting) by istartedi on Sunday December 13 2020, @07:31PM (1 child)
And a lot of NWS is actually clean compared to the rest of the web!
While technical fat-trimming would certainly help, refactoring takes time and money. The real problem is that NWS is not even a rounding-error in our massive budget, and yet it's just been neglected under the current administration. Either that, or commercial weather services are up to their old lobbying tricks--they've actually tried to kill NWS in the past. Hopefully the incoming administration will realize the value of NWS to almost everybody except these special interests, and act in a responsible manner. Just a few more weeks, problem solved.
Appended to the end of comments you post. Max: 120 chars.
(Score: 0) by Anonymous Coward on Sunday December 13 2020, @09:57PM
So, you're going to wake up Sleepy Joe?
(Score: 1) by pTamok on Sunday December 13 2020, @02:11PM (7 children)
End-users of apps should not be accessing the NWS directly. Each app should take one copy and cache it, serving the app users from the server(s) paid for by the app publisher.
How the NWS operates its web-site is up to the NWS. Screen-scraping for apps should be strongly discouraged.
(Score: 2) by Revek on Sunday December 13 2020, @02:28PM (2 children)
I use their website. No app just noaa.gov and the forecast on the webpage. I use their radar map and hope the html5 replacement is decent.
This page was generated by a Swarm of Roaming Elephants
(Score: 0) by Anonymous Coward on Sunday December 13 2020, @02:38PM (1 child)
Same. Isn't some of the radar imagery using animated GIFs, not Flash?
(Score: 2, Funny) by Anonymous Coward on Sunday December 13 2020, @03:29PM
Better be, come next week or so.
(Score: 2, Insightful) by Anonymous Coward on Sunday December 13 2020, @03:33PM (2 children)
If I'm writing one of the mediocre web scraping apps, like this one for weather, why would I want to stand up my own server, particularly if I'm just some mediocre programm doober who wants to make a name for myself with an app that I hope goes viral? I just need to write a snazzy looking UI on top of a web scraper and try to sell it for 99 cents on the app stores. I'm sure a fuck not going to set up a data server. That is for larger organizations who want to mine your data, not us lone developers.
(Score: 0) by Anonymous Coward on Sunday December 13 2020, @08:00PM (1 child)
And here we have the typical US coder, a parasite, draining everything around them, thinking they are entitled to leech everything.
(Score: 0) by Anonymous Coward on Monday December 14 2020, @01:31AM
Imagine a land where I am allowed access to the data my taxes paid for. You Eurotards would like it over here, but we're still not taking in any of you shithole country people, at least not until at least next month.
(Score: 2) by Runaway1956 on Sunday December 13 2020, @09:58PM
Seems reasonable. End users don't generally access the top DNS servers, and the web works fine. The weather would probably still work if we stopped end users accessing the main web site.
Abortion is the number one killed of children in the United States.
(Score: 3, Informative) by Anonymous Coward on Sunday December 13 2020, @03:44PM (4 children)
Interesting webinar. https://register.gotowebinar.com/register/6331491940012140301 [gotowebinar.com]
Quite a diverse group of users, from general users to heavy users using scripts to pull the immediate output of each model run. The models are getting bigger, so things are expected to get worse and worse. They currently limit each requesting IP address to 1800 requests/minute. This keeps their servers alive, but just after model runs they run out of Internet bandwidth. They are proposing to gradually lower this limit to 60/minute. This should still suport 75% of their users, but the remaining 25% will be affected.
A proposed alternative is to spend money not in the budget to increase B/W. They have two server sites each with 50G b/w. For $1.5 M in equipment, they can upgrade this to 100G which will put off the issue for a little while. Seems like this is useful, but only a stopgap. They fundamentally need another way to distribute their data.
Caching stuff in a CDN is problematic because the data changes frequently with each model run, and each request can request a different subset of the data.
They want to continue to treat all users as equal, and so do not want to have a tiered user system for heavy users with subscriptions.
They are talking about moving to cloud as an alternative.
(Score: 5, Interesting) by Grishnakh on Sunday December 13 2020, @04:23PM
They want to continue to treat all users as equal, and so do not want to have a tiered user system for heavy users with subscriptions.
This sounds like the fundamental problem right here: their philosophical ideals are in conflict with reality.
This looks like a classic "tragedy of the commons" scenario. You can just give out free access to a shared resource and not expect a minority of people to hog it, leading to it not being available to anyone.
A tiered system would solve this problem. People who make heavy use of the resource should be limited, and have to pay to access more. These fees can fund the service for everyone else.
(Score: 2) by Thexalon on Sunday December 13 2020, @07:09PM (2 children)
Rate-limiting single sources makes complete sense.
Another idea I could imagine: If they don't have this already, set up a subscription service, and have the subscription service push the data to a URL of the subscriber's choice, in a randomized order when there is server capacity to handle it, to reduce the polling traffic. It would then be up to the heavy users to work from their copy of the data, thus distributing the load away from NOAA's servers. If that results in faster and more reliable data, the app companies would probably be likely to go for it.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 5, Interesting) by captain normal on Sunday December 13 2020, @08:29PM (1 child)
Why do you think it's called The National Weather SERVICE? It's supposed to be of universal benefit to promote the general welfare. There is a lot of commerce that depends on a decent forecast of the weather, farming, shipping, aviation, transportation, forestry, civil defense, etc... The heaviest users of the band width are the "media" companies like AccuWeather, The Weather Channel, Commodity Weather Group and TropicalTidBits.com which mostly just link to NWS without loading the data on their servers (a lot of them like TropicalTidBits.com are broken unless one enables JS).
"It is easier to fool someone than it is to convince them that they have been fooled" Mark Twain
(Score: 2) by Thexalon on Monday December 14 2020, @09:41PM
I'm not saying drop the current pull-based services, I'm advocating working with the heaviest users of the current pull-based services to come up with ways of reducing their bandwidth while not compromising more than we have to on the quality of the results.
So, for instance, the 4 companies you mentioned have a choice of either pulling so much from NOAA that they overwhelm the server or network capacity and everybody (including them) gets nothing, or they have some conference calls with the CTOs or appropriate tech VPs to sort out what it would take to distribute the load more widely at the cost of maybe being 30 seconds behind on the data. And that leaves occasional users like me still able to go on the NOAA website and get something reasonably up-to-date. It's no different, really, than a completely different branch of NOAA meeting with all the lobstermen in Maine to figure out how to not end up with all the lobsters extinct so they don't all go out of business.
Or, of course, we could raise taxes by $0.005 per person and use the extra money for their capacity increase plan, but my understanding according to libertarian types is that that would be completely unacceptable theft of resources and we should never consider it under any circumstances.
The only thing that stops a bad guy with a compiler is a good guy with a compiler.
(Score: 5, Informative) by Anonymous Coward on Sunday December 13 2020, @04:36PM
I used to like their simple, usually quick, high contrast pages at or near:
https://radar.weather.gov/Conus/northeast.php [weather.gov]
on that same page, they say they are just days away from switching to a low contrast, higher bandwidth webapp
on their new webapp, try resizing the map a little bit and then hold down the back button to see what happens (for every pixel or so of zoom movement, another history item is added to the back button
their new app is less safe and I hope they cancel that UI update -
(Score: -1, Troll) by Anonymous Coward on Sunday December 13 2020, @04:39PM (1 child)
Lt. Cmdr. Geordi La Forge was found bound in chains this aftenoon, arrested after Data discovered the Lt. Cmdr. having sexual intercourse with his beloved pet Spot, his cat.
"You should've seen the visuals I picked up from this visor!" claimed the Lt. Cmdr. "The holodeck has nothing on this!" he sad, finishing with a long, "MEEEEEEOOOOOOOOWWWWWWWW!" as he stomped his feet and wiggled his buttocks in his restraint chair.
Data nor Spot were available for a response. Spot is in stellar health and considered to be "okay."
(Score: 0) by Anonymous Coward on Sunday December 13 2020, @10:08PM
s/he sad/he said/g
(Score: -1, Offtopic) by Anonymous Coward on Sunday December 13 2020, @07:00PM (1 child)
I'm guessing a good share of the cost is in the user data-mining and tracking. Sen. Maria Cant-do-well is from WA state, which is leading the "War on White Males". https://en.wikipedia.org/wiki/McNeil_Island [wikipedia.org]
(Score: 0) by Anonymous Coward on Sunday December 13 2020, @10:11PM
Can she blow you to hell?
(Score: 3, Insightful) by sonamchauhan on Monday December 14 2020, @12:28AM (2 children)
Isn't this is a solved problem? Just setup a torrent. Individual website users access the website as usual. But heavy users replicate data using the torrent (and to share among other torrent users if they're feeling generous).
'Weather Data Torrent' has a nice ring to it anyways :-)
(Score: 4, Insightful) by Rosco P. Coltrane on Monday December 14 2020, @02:35AM (1 child)
There's one big problem with bittorrent: it's decentralized, so Big Data can't track your web usage easily. So Big Data ran a smear campaign for years to equate bittorrent with illegal downloading, and sure enough, in true self-fulfilling fashion, that's what it ended up being used for solely eventually.
Big Data has the means to create their own infrastructure to soak up the load. They don't need bittorrent. But the little guys that could benefit from it - that is, everybody else - don't because it's not well integrated into browsers as a result.
(Score: 2) by sonamchauhan on Wednesday December 16 2020, @02:02AM
You're right of course--support for the Torrent protocol should be baked into browsers by now. :-(
If that was the case, this'd be a non-story that read: 'National Weather Service's uses Torrent to slash data budgets'
For rendering an exquisitely detailed radar graphic, my computer would pull decentralised data feeds from Timbuktu, the neighbours, from the wife's tablet in the kitchen (the latter over local Wifi).
(Score: 0, Troll) by Fuck You Niggers 2 on Monday December 14 2020, @03:24AM (2 children)
Seriously, they host all the radar data back to 1991 [opendata.aws] on AWS so they ought to be able to host current and archived weather observations and forecast models that way, too.
(Score: -1, Troll) by Anonymous Coward on Monday December 14 2020, @05:41AM
The parent is a goatsex link! Do not click!
(Score: 0, Insightful) by Anonymous Coward on Monday December 14 2020, @06:59AM
WHY DO THE ADMINS STILL LET THIS ASSHOLE POST?