SoylentNews is people
SoylentNews
"Come on, I worked so hard on this project! And this is publicly accessible data! There's certainly a way around this, right? Or else, I did all of this for nothing... Sigh..."
Yep - this is what I said to myself, just after realizing that my ambitious data analysis project could get me into hot water. I intended to deploy a large-scale web crawler to collect data from multiple high profile websites. And then I was planning to publish the results of my analysis for the benefit of everybody. Pretty noble, right? Yes, but also pretty risky.
Interestingly, I've been seeing more and more projects like mine lately. And even more tutorials encouraging some form of web scraping or crawling. But what troubles me is the appalling widespread ignorance on the legal aspect of it.
So this is what this post is all about - understanding the possible consequences of web scraping and crawling. Hopefully, this will help you to avoid any potential problem.
Disclaimer: I'm not a lawyer. I'm simply a programmer who happens to be interested in this topic. You should seek out appropriate professional advice regarding your specific situation.
https://benbernardblog.com/web-scraping-and-crawling-are-perfectly-legal-right/
(Score: 0) by Anonymous Coward on Wednesday July 31 2019, @06:53PM (1 child)
No, I think it's because their TOS says limited at their end and wild west at the user's end.
That TOS is the contract that governs the relationship.
The thing is, does that lopsided of a contract indicate that no reasonable person would agree to it if they had a choice?
Is that relevant to the validity of the contract?
(Score: 0) by Anonymous Coward on Thursday August 01 2019, @01:02AM
In a contract of adhesion clauses that are unconscionable or outright illegal aren't supposed to be enforceable, but again, how many politicians and judges do you own? How much can you afford to spend on legal fees?