Being down for an entire trading day is pretty crazy. I can't imagine what's going over there today in the engineering & devops teams. Depending on where the issue is, there might be thousands+ of orders to reconcile (manually?). It will be really interesting to read a post-mortem engineering blog post if they make one.
They don’t seem like serious enough professionals to release a postmortem. Did they do so for the “infinite leverage” bug?
I have a couple grand in play money in Robinhood, because hey, zero commissions, but it’s clearly not sustainable for the long term. I’m glad my real investments are all safely parked at Vanguard, where I likely wouldn’t even notice if the brokerage seized up for a day.
The business model is perfectly sustainable and the same as all the other brokers. Commissions were purely extra income.
The problem is that Robinhood is a young tech company and the "move fast and break things" approach doesn't work when it comes to industries that need reliability and risk management.
I'm surprised they got away unscathed from the lack of proper margin calculations before. That should've led to major fines.
Ah, sure, this is what I meant by it not being sustainable. They keep fucking up in standard SV ways, eventually FINRA or maybe even the SEC will come down hard. The basic brokerage business seems fine.
The biggest impact Robinhood has had is pushing other brokers to go to zero commission models. I joined before that happened, but I will be moving my trading portfolio elsewhere.
I’m surprised more people on here aren’t using something like https://alpaca.markets. It’s primarily for algo trading, but there’s nothing stopping you from using it generally, as it is an API, after all.
Wow, I'm surprised I hadn't found this before. I was looking for something exactly like this at some point in the past 6 months and didn't find much, so thanks for sharing this!
> Being down for an entire trading day is pretty crazy.
"Nines don't matter if the users aren't happy", to quote Charity Majors. No-one will remember all the uptime for the rest of the month, or indeed year. Just the hours they were down for a critical trading day.
It looks like the guy on Twitter doesn't understand that the API is down so you get 503 and a CORS error cause the pre-flight request failed and doesn't have the right headers. It will fail for any date.
The question is really does the front-end normally try to query the next day and what's going on in their order execution system on the backend, which we don't know because this source is just speculation based on console logs in the browser.
> Same here CALL ROBINHOOD ASAP. I spoke with them myself and they are aware. IF YOU NEED TO MAKE TRADES THEY CAN DO IT FOR YOU!
I didn't even know they had a phone number you could call. This should have been emailed out after the first hour of the outage. Or, how about you have a better system down page then just a blank website? If API health checks fail, serve a static site from S3 (route53 can do this). The ops failure of a company that can afford to pay quality DevOps engineers $200-300k a year is beyond belief.
Simple as:
"We are experiencing a technical outage... <corporate blah blah blah here>. If you need to make trades, please do so via phone at 1-800-SHORT-RH"
This person is probing an API which is already down so of course there are errors.
Robinhood would be using a proper date library anyway to handle timezones. The system was fine on the 29th, 1st and even this morning pre-market. Seems more like a capacity issue that cascaded into failure.
The 2nd was a Wednesday four years ago. That wouldn’t make any sense for a leap year error. Why wouldn’t it have gone down on the Monday two days prior?
As somebody who does DevOps consulting I was baffled how they could be down all day if it was a pure operations issue. That tweet is just speculation though. Hopefully more facts and details come out.
My bet is still on it being a database issue. If they indeed are using AWS RDS, I wouldn’t be surprised if they have half of AWS support engineers working on it. Once again, if I had to bet, I say they rolled their own "half-baked" database solution instead of using a AWS native offering.
I recently interviewed there and they said they're still using Postgres master-slave setup most likely on some cloud provider. Could be catastrophic failure with multiple shards going down at the same time. This might also explain why they mentioned their "backups" not working
Why would they get unlimited support from "top tier" AWS engineers?
A company like Robinhood doesn't strike me as a strong candidate for contracting AWS Enterprise support. For the price you get almost nothing out of it.
I doubt it will be possible to just shake it off as there is likely millions of dollars lost for customers. Not in some abstract way like when github goes down but literally because people cannot close out positions and options expire. If it is really just a leap year bug It should be gross negligence. Not sure if their TOS will hold then.
Guess they weren't ready for the massive amounts of requests coming from a world record market correction / recession. That, or they released a bunch of new features over the weekend.
Why we are building highly sensitive monetary performance based applications under the guise of agile and electron is beyond me.
The app market needs a market correction in design. IEEE, military, aerospace, or aviation design standardizing comes to mind.
If they are down for one full day out of every 365, that's still a potential ceiling of 99.726% uptime for the year. My home internet was down 14% of the afternoon on Sunday this week, and on through the evening.
On a month-to-month basis, I can't even seem to crack 99% reliability. (Geez, I should really call them on it, or perhaps get a new modem...)
Point is, how high is Comcast's valuation again exactly?
Your internet goes down, you can’t watch netflix. Boo hoo. Your brokerage goes down, you can’t exit your positions, and, well, just look at the hundreds of thousands of dollars lost on reddit today. People were using their student loan money in their brokerage accounts. It really is a disaster.
Reminder that the financial return maximising function for Robinhood benefits far more from "unsophisticated investors put in their student loans" than "service is up all the time". People who are actively trading, especially in options, may well be losing money.
This is largely my point. I live with <99% internet uptime and I have a Robin Hood account. I only know about this outage because I read about it... feel pretty sure that I am just like the majority of their customer base.
I don't check RH every day and I wouldn't have been compelled to check on it at all yesterday except for the news (I'm losing money, who wants to look at that! LOL)
ISP are pretty much the poster children for the corruption and incompetence that comes from well entrenched oligopolies. Industries with healthy competition, such as retail investing, generally have much higher standards. Comparing RH to Comcast is not a good look for RH.
Ah yes, ISP's; the most beloved of corporate entities, and not in any way aided by regional duopolies.
RH isn't an ISP, and being down for an entire day of trading is a big deal. If I don't like my ISP... welp, I'm pretty much screwed. If I don't like RH I have plenty of options.
If Robinhood wants to target their app towards people who want a fluid interface and neat graphs, compromising on the always-up needs of day traders, I don't see why that's wrong.
It's a trading app that is... down for an entire day of trading. That seems like the wrong sort of compromise to make. The number one priority of anyone who is actively trading is... to be able to trade.
Not even mentioning the other hilarious bugs this company has let out into the wild. They don't seem to have a good grasp on what they're doing, and now many other, more mature brokerages offer zero fee trades. Their time is coming.
I just got an email from Robinhood support on the outage, half an hour before markets close.
I don't actively use RH as my main brokerage so not a big deal to me, but obviously this is unacceptable. Don't recall any other online brokerage suffering an outage like this.
Suffice to say this throws cold water on any mystic around Robinhood having better tech that any incumbent.
This is the problem with startup companies, they think too much about product and too little on communications.
This is very poor communication, sending an outage mail after hours it had occurred, with no ETA for the fix , and absolutely no details on what has happened i.e a hack , or a DDOS, or an infra outage etc.
Don't recall any other online brokerage suffering an outage like this.
Brokerage firms have outages with surprising regularity. I remember Sharebuilder being particularly bad, but Vanguard was down for a bit last week. A whole day though is pretty spectacular.
Other brokerages also support trading over the phone in the event of an online platform failure. Robinhood doesn't, and their email support system was bouncing emails due to high volume of incoming mail.
I don't understand the attraction of Robinhood over the likes of TD or Etrade. TD minimum balance is $50 (the minimum transfer) and Etrade $500. Both are well vetted brokers with additional features and services.
If you are investing, how sexy the app is (if you only use a phone) shouldn't matter. If you are day trading from a phone app (except in emergency), you are, IMHO, delusional.
They have been down basically the entire day. Are they going to face some action? I'm unclear who even punishes them for potentially causing millions of dollar losses for customers.
Probably only if there is some type of class action law suit. If customers actually lost millions due to robinhood’s negligence then there is almost certainly going to be one.
Pardon my ignorance, but what are SEC rules for things like this? Do they expect some reasonable availability for trading platforms or do they consider massive outages like this "negligence"?
My limited understanding is that if you have a license to be a broker in this nature, you have to meet certain requirements. Similar to getting a bank charter - banks have to be open all standard business days during certain hours. Things may happen from time to time that are unexpected - like a massive snow storm that closes the bank for the day or at least part of the day.
I imagine the rules are similar - you must be able to have your clients trade during normal market hours. The regulators understand there can be unexpected events that put systems down unexpectedly (high volume, bug that crashes the system, etc.) and are generally pretty understanding. But I think something like this where they are down the whole day (and STILL down from the looks of their status page, though they claim to have identified the issue), could easily earn them a fine from the regulators for not meeting the license/charter requirements.
The biggest question is whether or not their systems were able to execute margin calls. If they were down this long, that to me suggests data got corrupt and they've been restoring from backups, etc, which means maybe all their services were inactive. If that's the case then could the margin system be active, or are people (and potentially RH) sitting on oversized losses if they were short today? What happens tomorrow if their margins kick in and sell at the open, and then the market drops? Lots of very interesting questions that need to be answered.
I tried to clear all user data for RobinHood off my phone this morning and login fresh. At first I got absolutely nothing, but eventually my stocks and options came up, then my watchlist. However the watchlist was missing everything I added to it yesterday, though if I look up a symbol directly it does show as added. So I think your theory of restoring from backups has some merit. I wonder how they'll reconcile any orders placed/executed after the backup was taken if that's the case.
Aaaand, there's already a twitter account trying to build a class action lawsuit against Robinhood for allowing an outage to happen. Wonder if this'll go anywhere! https://twitter.com/ClassRobinhood
depends on what kind of option contracts they were buying. I loaded up on a bunch of calls last week and am currently sitting here happy as an elephant.
I jumped to Etrade a while back and it's been a relatively positive experience. Power Etrade is garbage though - consistently crashes during spike traffic times
One of the biggest lessons here is not to use tech-forward brokers. I see a lot of people talking about how swapping from robinhood to webull is ridiculous, and I honestly agree. I'm personally thinking of going TD or thinkorswim.
As of 3:44PM EST. It’s still not back. I then signed out of the app and can’t sign back in. I then thought maybe they reset passwords, so I tried resetting. Even that’s not working. Total outage!
You could very well be right, according to this[0] post on Twitter:
> Hey @RobinhoodApp
did someone forget to add isleap() to your python module? Imagine losing thousands of dollars because someone didn’t code for the leap year. It’s looking for 3/03 data and throwing errors out the ass. https://pbs.twimg.com/media/ESIuB4nXYAAYImF?format=jpg
Somewhat related, they ruined their app for me last week by adding the "lists". Instead of I instantly visible scrollable symbols I am interested in trading, the app now presents a bunch of "expand" and "add new list" controls, that when expanded show abridged lists with "click to see more". That totally ruined its main differentiator as an "easy app". Additionally the update was shoddy: due to a memory leak the scrolling eventually becomes very slow.
At Robinhood, you're not the customer. You're the product.[1]
Robinhood makes their money by selling order flow to high-frequency traders. Robinhood doesn't charge commissions, but the price is slightly worse.
This is... not a good analysis. It's directionally correct but not for the reason you say it is.
Virtually all brokers are rebated for routing orders to HFTs. HFTs dominate retail order execution. HFTs pay brokerages for access to their order flow, because that order flow is uninformed and safe to quote narrow spreads for.
Like other brokerages, Robinhood makes money on net interest from cash holdings and stock lending fees (it helps that Robinhood encourages people to buy meme stocks with a lot of short interest). Unlike other brokerages, Robinhood also encourages options trading, which are more lucrative for market makers.
You sort of are the product at Robinhood, but it's not really because Robinhood is doing bog standard order routing that every other broker does too.
> Headlines like Robinhood Gets Almost Half Its Revenue in Controversial Bargain With High-Speed Traders are virtually meaningless, as the above discussion of other monetization options should make clear. At scale, once their userbase actually has material cash in their accounts on any given day, they’ll likely make net interest, but pre-scale they may not even have the treasury function to do the work required to earn it safely. It’s a commentary on how low their revenue is (growth-stage startup optimizing for reach over revenue, film at 11), not a commentary on how beholden they are to Wall Street.
How do you think most brokerages fill orders? I'd rather get filled at the best price possible, which I assume a high frequency firm would be able to do better than some other filling method. Pretty sure most brokerages do this anyway.
Reg NMS prevents HFTs from filling your order outside of the NBBO. if anything you’ll get slightly more favorable prices from a market maker, since they know you’re a retail trader who’s not going to run them over.
Yeah that makes sense then. I have been using Robinhood cause I figured it wouldn't matter anyway the latency differences between click trading and their algos are so large. But I won't be after the outage today.
I was getting ready to buy a bunch of calls this morning via Robinhood. I hit the "submit" button on the trade, and nothing happened. VERY disappointing experience. When it comes to applications like this, downtime is critical. Haven't been able to place a trade all day.
Folks were leveraging up all last week and buying puts, thinking the market was tanking to zero and we're all going to die of "beer virus" as they call it -- and are getting a harsh lesson about timing the market today.
They’ve been giving so much leverage to broke young people, I wouldn’t even be surprised to read that they’re insolvent someday after a big, unexpected move.
I've read some sad stories about, for example, a student betting student loan money in the market and losing it instantly, lured to make risky bets from the other stories of dumb luck success.
It’s a obnoxious to know nothing about the culture of wallstreetbets and then to make declarations as if you know after reading a few posts.
Posts like that are par for the course. Its called “Loss Porn”. The subreddit is absolutely geared towards stock/options trading as gambling and if you think otherwise then you’re wrong. They are very self-aware and do it for fun. And yes some may lose a lot of money but they know exactly what they are getting themselves involved in.
There are people making and losing fortunes every day in the market. These kind of crazy bets aren't new, just more visible.
I remember being in college in 2005 and seeing people going wild with Forex and penny stocks. WSB is just a small subset of traders who like to discuss it on social media.
Marjory if what happens in that sub is humorous. And its pretty clear that there is a lot of satire and it's not the place you go to for in depth analysis 99% of the time.
Loss and gain porn are a normal part of the culture as are the slurs used.
The gains and losses are often real but this is the community you go to the laugh about such things
The person posting it in the subreddit clearly understands the theme of the subreddit. Whoever took the screenshot and shared it somewhere likely doesn't even know what WSB is.
They didn't actually lose that much money, since they bought options. By my mental math they lost about $9000 (give or take a few hundred). Still bad, but better than losing hundreds of thousands.
no... it’s 140 option contracts which provides the option for 14000 shares. they were bought for $66 x 100 x 140 = $924,000 and are now worth $14.43 x 100 x 140 = $202,000.
Risk / reward is proportional. You could have easily made a killing if the markets moved the right way. In fact, anything could happen by Friday. He only needs and 10% move up be in the black
> 5QxjKb7SI2j1d9Zs3jcs · 809 points · 11 hours ago
> As a proud cuck, not gonna lie I get off on being locked out of my account while it gets pummeled and I can do nothing but watch.
> BippityBoppityZop · 156 points · 10 hours ago
> Don’t stop I’m almost there
> SensualKoala · 47 points · 9 hours ago
> "Oh yeah FUCK my savings harder"
> TRAIN_WRECK_0 · 84 points · 9 hours ago
> Anyone feel like barfing? It's almost like my wife's boyfriend stopped fucking her and starting fucking me, and there is nothing I can do to stop it.
@nostromo can't have read very much of that thread. It's one big in-joke.
It's probably one of the most suspenseful market opens in history. The tug of war between bulls and bears across the entire sub has been incredibly entertaining.
Regarding leverage: buying options is a fixed loss against the cash in your account. RH also only allows selling secured options. There's no way to lose more than you started with (at least with the current app, they had problems before).
Retail investors using standard accounts can't buy options on margin, it's only used for faster settlement.
That might be Robinhood's intention but their incompetence has led to plenty of bugs in the past that allow you to rack up losses exceeding your balance.
You thought the market would go up? Specifically a bank? Sure the bankruptcy was a surprise but I really don't know of anyone back then who was expecting a recovery or even a minor bounce at all that year. Hopefully your options were LEAPs.
Were people really expecting the markets to go up today? Futures were mixed pre-market. Bloomberg, at least, was framing the optimism as due to people expecting central banks to intervene but central banks can only do so much when your supply chain evaporates.
I was certainly caught off guard but still managed to make a bit of money (~6%) off of one of those idiotic volatility ETNs today.
You win some you lose some. I exit most of my put positions last week after getting up 13x.
The position I kept, I got fucked hard by this Robinhood downtime. Ended up losing all of it, (4x of my original 13x profit), whereas I was trying to exit it early Monday. If I lose money on the market and it's my fault… fine. But when my broker causes the loss, I'm pissed
Yes, lots of people were. The market closed down 6 days in a row which almost never happens. There's no major collapse like 2008, just a powerful catalyst that led to a fast correction of 10% from all-time highs and released a lot of downward pressure.
Yeah the market is near all-time highs all the more reason there's going to be significant downward pressure. There's been no major economic collapse but China's industrial output is near record lows as evidenced by the air quality.
I see a bunch of comments of people saying I lost $x amount because of the outage, however I wonder if that's actually true. It's much easier to say I was going to do x when doing x was unavailable.
It might be some of that, but folks on that subreddit tend to take super risky positions that can go south on them very fast. It’s possible a few made some really stupid bets during the crash and they couldn’t get out today.
It's impossible to know since the entire platform was down and both current positions and order entry were unavailable. I'm sure there's a lot of exaggeration but today was also the single best day of the DOW right after a 10% decline last week. That volatility can have massive changes in option pricing, especially short-dated expirations (like SPY options expiring today).
>Folks were leveraging up all last week and buying puts, thinking the market was tanking to zero and we're all going to die of "beer virus" as they call it -- and are getting a harsh lesson about timing the market today.
This aged terribly. Anyone who did that received a massive windfall.
Some of us had puts from last month which were well in profit. I personally have lost over $5k from this outage even though I'm technically not negative on my position. Painting all RH users as gamblers is pretty arrogant.
Long term puts (at least 2 weeks out) on the market are the safest investment that you could possibly make. The market recovery today is a dead cat bounce. Mark my words I guarantee you that I will be right about this
I think you’re right that we’ll retest lows but I’m not sure that’s it’s elevator down time. However, we have been on a long run, and the machines are selling to lock in profits. Maybe we do keep going. Guess we’ll find out soon enough!
What are the odds that this was a cyberterrorism attack? If you can predict this kind of market event (the bounce back), and you want to hurt a company and its users, you would simply DDoS at the right time.
Well, all you have to predict is that the market will move - the direction wouldn't matter much, since the brokerage has customers trying to do both calls and puts.
I remember when Schwab went down just as Facebook went public. Couldn't cancel my order, ended up with the stock at a much higher price and the stock had dropped by a lot by the time I had access again. I can't help but be suspicious of events like this.
