The latest issue of Wired has an interesting article about a 29 year old mathematician who is using crowd sourced machine learning to manage hedge funds.
Richard Craib is a 29-year-old South African who runs a hedge fund in San Francisco. Or rather, he doesn't run it. He leaves that to an artificially intelligent system built by several thousand data scientists whose names he doesn't know.
Under the banner of a startup called Numerai, Craib and his team have built technology that masks the fund's trading data before sharing it with a vast community of anonymous data scientists. Using a method similar to homomorphic encryption, this tech works to ensure that the scientists can't see the details of the company's proprietary trades, but also organizes the data so that these scientists can build machine learning models that analyze it and, in theory, learn better ways of trading financial securities.
"We give away all our data," says Craib, who studied mathematics at Cornell University in New York before going to work for an asset management firm in South Africa. "But we convert it into this abstract form where people can build machine learning models for the data without really knowing what they're doing."
He doesn't know these data scientists because he recruits them online and pays them for their trouble in a digital currency that can preserve anonymity. "Anyone can submit predictions back to us," he says. "If they work, we pay them in bitcoin."
So, to sum up: They aren't privy to his data. He isn't privy to them. And because they work from encrypted data, they can't use their machine learning models on other data—and neither can he. But Craib believes the blind can lead the blind to a better hedge fund.
(Score: 3, Interesting) by fritsd on Sunday December 18 2016, @03:36PM
<tinfoil-hat rant>
We are EVER so fucked, if we don't implement a Tobin [wikipedia.org] Tax [europa.eu] into the EU financial system as soon as the UK obstacle has finally buggered off [europa.eu].
Are there any data scientists or ML experts who do the final supervision before this system buys half of all the Euros in the world in 250 milliseconds to sell them again 500 milliseconds later? I don't think so; that would be too slow to compete with similar hedge funds.
And how can they work with encrypted data, that implies it must look like random data, which is stupid to feed to any machine learning system or committee [wikipedia.org]?
Maybe they mean that the raw data has had a linear transformation to obfuscate what it really means, so the sub-experts can't try to leach off some of the immense profits this system promises.
"Very well done everybody! Today's winner sub-expert system #322 from "StarvingStudent" from Romania will get 3.12 Bitcoin, while we get the $ 31.2 million profit, half of Zimbabwe, and the cocoa futures of Côte d'Ivoire until 2055. We can always flog off Limpopo province within the next 5 hours without too heavy losses, and I like cocoa."
A nice little earner for poor starving AI students though, for sure. The students of one of my previous employers would have loved such an assignment, especially as it earns "real" Bitcoins. Even better than the Netflix Prize [wikipedia.org].
Madness.. lemme get a sheet, cut 3 holes in it, and walk around (naked underneath) screaming "We're all doomed!!!1!". Hm. it's night and freezing. Think I'll postpone that for another day, and not scare our lovely neighbours on this 4th advent of the "interesting times" year 2016. Please prime minister May, begin the Brexit in March as you promised? Bless you!
</tinfoil-hat rant>
Oh I see I misread the keyword "homomorphic encryption". Interesting. Though I doubt it's really necessary as long as the raw data has no labels and is obfuscated a bit.
(Score: 0) by Anonymous Coward on Sunday December 18 2016, @04:20PM
Why should this one particular organization ("government") receive the benefits of other people's work? I'd rather the money stay in the hands of the actually innovative people, rather than be placed forcibly in the hands of know-nothing, paper-pushing bureaucrats.
(Score: 2) by fritsd on Sunday December 18 2016, @04:28PM
Good question.
Because while the actually innovative people [wikipedia.org] win, everything is hunky dory.
But when they inevitably lose, it's not them that pay back the € 4.9 billion to the taxpaying bank account holders.
(Score: 3, Insightful) by Francis on Sunday December 18 2016, @04:56PM
Because it's not an either or proposition and the current situation is neither. The money mostly gets taken by plutocrats that contribute nothing of value to the country at the expense of the innovators and producers.
Giving it to the government in order to pay for things like infrastructure, education, healthcare and and other things that grow the economy would be a huge improvement over the current situation where we're being held hostage by a group of plutocrats that whine because they're only very wealthy rather than really, really, ridiculously wealthy.
The short sighted measures that they demand in order to keep jobs here cost us huge amounts of money when the taxpayers ultimately have to pay for the negative externalities and consequences of the anti-social behaviors exhibited by most corporations.
The worst of it though is that it impedes other companies that want to go into those markets as they now have to compete with companies that have grown to the point where there's no way of growing large enough fast enough to compete. Many of those companies grew large by violating antitrust laws while the DoJ turned a blind eye. Neither Apple nor Google was taken to task for their antitrust violations. And Intel only got a slap on the wrist after it was really too late to make a meaningful difference.
(Score: 0) by Anonymous Coward on Sunday December 18 2016, @05:08PM
Giving it to the government in order to pay for things like infrastructure, education, healthcare and and other things that grow the economy
Higher salaries and richer pensions for government workers, who are already the most pampered in the entire economy.
group of plutocrats that whine because they're only very wealthy rather than really, really, ridiculously wealthy.
Not arguing the term "plutocrats", but they don't whine about it. This is their calling, what gets them up in the morning.
(Score: 2, Informative) by Francis on Sunday December 18 2016, @05:23PM
Government workers are not the highest paid in the country. I am a government worker and most of us make less than our private sector counterparts do. And by a considerable margin. I'm making less than half what I would be making in the private sector.
I'm not sure where people get the idea that we're overpaid comes from, but it's not true.
(Score: 0) by Anonymous Coward on Sunday December 18 2016, @06:19PM
Have you noticed that voters in red states often elect a Democratic governor, and voters in blue states often elect a Republican?
I think it's because voters in red states want to ensure a certain level of services in their state, that the environment does not get trashed, laws aren't set up to discriminate against gays, etc.
And voters in blue states are tired of state workers taking a ride on their dime with incredibly generous pensions (which they insist are "paid in", but guess by who - the taxpayers again), vacation, and sick leave policy. And nieces and nephews getting fixed up with do-little jobs on the public payroll.
Newsflash: most workers in corporate America don't get to roll over their vacation days anymore, and we never could roll over sick days. And we don't get 12-15 sick days a year or whatever state employees get.
(Score: 3, Informative) by Francis on Sunday December 18 2016, @06:36PM
The issue there isn't the state workers, the issue there is that the politicians keep going out of their way to damage labor's negotiating power.
Allowing plutocrats to keep their spoils is a large part of the problem. Back when tax rates went to above 70% for the top tier income earners there was little incentive to wring every possible cent out of a business.
(Score: 0) by Anonymous Coward on Sunday December 18 2016, @07:38PM
(Score: 0, Flamebait) by khallow on Sunday December 18 2016, @07:13PM
Government workers are not the highest paid in the country. I am a government worker and most of us make less than our private sector counterparts do. And by a considerable margin. I'm making less than half what I would be making in the private sector.
I'm not sure where people get the idea that we're overpaid comes from, but it's not true.
Well, I haven't been given enough information. You may still be very overpaid simply because you don't do much or do what you do very poorly. You might even owe society money because your job inflicts net harm on society.
(Score: 2, Interesting) by khallow on Sunday December 18 2016, @07:40PM
We are EVER so fucked, if we don't implement a Tobin Tax into the EU financial system as soon as the UK obstacle has finally buggered off.
I think it would be better to worry about actual problems. Seriously, from the Wikipedia web page (your first link):
The tax on foreign exchange transactions was devised to cushion exchange rate fluctuations. The idea is very simple: at each exchange of a currency into another a small tax would be levied - let's say, 0.5% of the volume of the transaction. This dissuades speculators as many investors invest their money in foreign exchange on a very short-term basis. If this money is suddenly withdrawn, countries have to drastically increase interest rates for their currency to still be attractive. But high interest is often disastrous for a national economy, as the nineties' crises in Mexico, Southeast Asia and Russia have proven. My tax would return some margin of manoeuvre to issuing banks in small countries and would be a measure of opposition to the dictate of the financial markets.
So what's the problem here that needs to be fixed? If it's a short term situation due to collective decisions of short term speculators, then no one needs to adjust interest rates. The money will come back one way or another and the speculators will be rewarded with a loss in their investments.
The 90s crisis had nothing to with short term speculation and a lot to do with the shenanigans that these countries were pulling at the time. For example, the Mexican government in 1994 suddenly devalued the Peso. In 1997, Thailand, Malaysia, and Indonesia had bubbles develop in their economies coupled with fixed exchange rates that had to be dropped.
A Tobin tax would interfere with the outside world trying to adjust to a country doing such things.
The same lack of justification goes for financial transaction taxes (FTT) advocated in your second link. Not a single concrete reason is given for why FTT is supposed to be a good idea. It's just somehow for "fairness, responsibility, and a stronger Single Market". And the author immediately starts cutting out exceptions for real world activities, market liquidity, or government bonds (government always gets to be excepted from the problems it creates).
Finally, if the EU implements a FTT and an independent UK does not (your Brexit grandstanding), this provides a tremendous opportunity for the UK to provide a work-around for EU silliness. I'm sure the EU won't mind transferring a considerable share of its financial activities to the UK as a result of laws like the FTT.
(Score: 2) by fritsd on Sunday December 18 2016, @09:32PM
Well, I thought there was a problem that the collective decisions of short term speculators could sometimes cause large fluctuations in the stock market (even though statistically you'd think their actions would even out).
This was not a problem in the 1990s because high-frequency trading didn't exist then.
And I read that the algorithmic traders build their offices as close as possible to the physical stock market computer systems, so that their network latency is minimized.
And I read that in the 2010 Flash Crash [wikipedia.org], which occured in the timespan of 36 minutes,
, and
and we're now entering a time of even greater uncertainty in the EU and elsewhere, and with the Brexit the City of London traders will have to invent new plans to Make Money Fast(TM), decoupled from the boring euro banks, so I'd hope that the financial institutions from the EU could somehow construct some kind of flash crash lightning rod Tobin Tax, even though that would make them less competitive in the world of flashy "here today, gone tomorrow" trading.
There would still be room in the world economy for banks that are slightly more expensive and more boring than the hottest, brilliantly bright lightning traders of London and New York. I think. So I agree with your last paragraph that this "EU silliness" would surely cost the EU companies, and would be an incentive for eager EU banks to move to buccaneer London.
The lightning will strike again, guaranteed. But we can't know where and when. Therefore protection has to be generic and in place, and not at the last 250 millisecond human decision to pull the plug out of a computer system (6 may 2010, 2:45:28 p.m. according to the Wiki article. for 5 seconds.).
Granted I only worked at a very low level as a programmer at banks and pensionfunds, so what do I know.
(Score: 2, Informative) by khallow on Monday December 19 2016, @08:50AM
Well, I thought there was a problem that the collective decisions of short term speculators could sometimes cause large fluctuations in the stock market (even though statistically you'd think their actions would even out).
This was not a problem in the 1990s because high-frequency trading didn't exist then.
And I read that the algorithmic traders build their offices as close as possible to the physical stock market computer systems, so that their network latency is minimized.
And I read that in the 2010 Flash Crash, which occured in the timespan of 36 minutes,
The thing is the disease is the cure. If someone is repeatedly causing flash crashes or large fluctuations, they'll eventually run out of money and that's that.
As to your claim that this never happened before, we have Black Monday [wikipedia.org]. The timeline is instructive [wikipedia.org]. A crash still happened despite it occurring on a long enough time frame that humans could react to it. But even in that case, it takes a while to muster large amounts of fluid assets. Supposedly, there was a point where one could have earned 20% over about 24 hours on arbitrage (risk-free trade) which indicates the parties interested had exhausted their means to purchase such things over the relevant period of time.
and we're now entering a time of even greater uncertainty in the EU and elsewhere, and with the Brexit the City of London traders will have to invent new plans to Make Money Fast(TM), decoupled from the boring euro banks, so I'd hope that the financial institutions from the EU could somehow construct some kind of flash crash lightning rod Tobin Tax, even though that would make them less competitive in the world of flashy "here today, gone tomorrow" trading.
Traders != banks. Banks can have traders, but traders can come from a lot of different businesses or even be self-employed as your "36-year-old small-time trader". I think the competitive hit will be bigger than you expect with wider spreads and low volume. Second, all that flashy stuff is a magnet for speculators and market makers. That's profit opportunity that won't be present in the boring market.
The lightning will strike again, guaranteed. But we can't know where and when. Therefore protection has to be generic and in place, and not at the last 250 millisecond human decision to pull the plug out of a computer system (6 may 2010, 2:45:28 p.m. according to the Wiki article. for 5 seconds.).
Or we can just not do anything about it and the problem fixes itself. It's not like Apple is going to stay at $100k and Accenture at a penny forever just because there was a weird market fluctuation. If the stock markets made a habit of never rolling this crap back, it would stop in short order through a combination of the perpetrators losing their shirts and other market participants becoming more proficient in responding to such things.
(Score: 1, Interesting) by Anonymous Coward on Sunday December 18 2016, @04:53PM
whether Numerai has decided not to use their code? Maybe their code is being used, but Numerai told them it wasn't suitable.
That would keep the startup's costs really low.
(Score: 0) by Anonymous Coward on Sunday December 18 2016, @05:19PM
I think it's a safe bet that Numerai is so stuffed full of cash that they purposely request & pay for software development that will never be used. This "red herring" code is just to muddy the waters and keep everyone guessing. Employing 3rd world coders is pretty cheap so it's not like they're throwing their money away - they're just paying a little extra for some "security".
(Score: 2) by fritsd on Sunday December 18 2016, @05:22PM
I think that in these kind of situations, the company with the actual test data outputs (Numerai) holds all the cards.
They can't share the test data outputs together with the training data and test data inputs, because that would defeat the purpose of the machine learning process: the committee members would just train with supervised learning on the training+test data, and you can't tell if it has a low RMS error because they cheated and overfitted [wikipedia.org], or because they did everything honorably and just found an extremely useful algorithm.
That implies that the committee members can never evaluate if they're being rewarded properly either.
Actually I just thought of something: Numerai can put it (=test data outputs) in a public location afterwards; then the committee members can download the (checksummed!!) used-up test data, compare if gives the same score function that they were rewarded with, and have furious phonecalls with data scientists or their students that they know, to ask if they've also been shafted by Numerai, or if the checksums don't match and each committee member gets an individual fake test data set.
If the (public!!) scoring function gives correct results on the test data, AND the test data is identical to your concullega's download of the test data, AND it gives correct results for them, then that proves Numerai is honorable.
Would that work?
The committee members don't have to publish their code or algorithms either: since the purpose is making oodles of money, they might be reluctant to do that anyway.
(Score: 2, Informative) by Anonymous Coward on Sunday December 18 2016, @05:21PM
Many people here cannot read Wired articles. Wired chooses to reveal the articles only if you allow their advertising network to download images and run code on your PC. Think about that for a minute - you have to let random, unknown, people download and run unknown programs on your PC. Yikes!
(Score: 0) by Anonymous Coward on Monday December 19 2016, @03:29AM
After you read a screen/page of the article, do a page-down and when the block comes up, close the tab. Come back to Soylent and open the wired link again. It will open to the screen that you paged-down to
(Score: 0) by Anonymous Coward on Monday December 19 2016, @01:55PM
I am not sure about that - I can read Wired articles just fine using TOR and with most / many things blocked.
(Score: 0) by Anonymous Coward on Sunday December 18 2016, @07:02PM
i think everything that makes more money but isn't something you can touch is just a system to prob (as fast as possible)
as what people might consider valuable.
make money with stocks? naw, make a hedge fund instead. both create nothing of value.
we could have a economy with nothing but apples. we can make lots of things from apples and we also wouldn't starve.
we can, however, make nothing from stocks. a economy based on stocks only would fortunately starve to death in a few weeks.
thus, the value of stocks is just what people perceive them to be.
know and see the future by creating perception (or denying it).
(Score: 1) by khallow on Sunday December 18 2016, @08:00PM
we can, however, make nothing from stocks.
Good thing that nobody is trying to eat stocks then, isn't it?
thus, the value of stocks is just what people perceive them to be.
People are really good at this sort of thing. And perception is more than adequate when the food routinely hits the dinner table.
(Score: 2, Funny) by fritsd on Sunday December 18 2016, @08:47PM
You could make money with stock cubes. You can make bouillon from them, so you wouldn't starve. You might want apples as well though otherwise your diet gets much too salty.
(Score: 2) by goodie on Sunday December 18 2016, @07:03PM
This is somewhat like Kaggle competitions but for financial data?