Ex-Microsoft, ex-GitHub, laid off nine months ago, not going to violate any agreements, but I will give some context. A lot of this applies to any large engineering org, really. Hilarious that this entire thread is people speculating on things they have no idea about.
If it sounds like I'm defending GitHub a bit, yeah, I am. I can't think of a comparable situation that any other web site has been through. This isn't "oops, we didn't plan for the Black Friday sale", this is a once-in-a-generation event focused on one, important web property that no one would have handled perfectly. (Even us all-knowing commenters here on HN.)
You know that old trope in every submarine movie, where the captain tells the driver to dive lower than they've ever gone, and someone says "I don't know if the boat can take it!" and then they switch the camera to some engineering room where the boat is groaning under the pressure and a bolt comes loose and water starts spraying everywhere but someone runs up with a giant wrench and tightens it and looks around at everyone else in the room like, "whoa, that was close..."?
Or Scotty and Star Trek, "It canna take much more, Captain!"
That's what GitHub has been going through. It's had problems, but it's still here, and usually working well.
- Let me start with: GitHub is ~3,000 talented and really nice people trying to do the right thing, with a great culture. That's the most important thing I want to communicate. While I'm sure no one at GitHub is happy about their uptime, believe me when I tell you that GitHub Engineering is exceptional. They have great leadership, depth at all levels (Distinguished / Principal / Staff / Senior / earlier-in-career) and if you ever get a chance to hire someone from GitHub, you should. I hope all of you get to work with engineers as good as GitHub's.
- GitHub has been improving all aspects of its infrastructure steadily for many years now. The GitHub Engineering blog https://github.blog/engineering/ has been documenting this the entire time. Go point your favorite LLM at it and ask for a summary of all of the major system improvements since 2020. If that work hadn't already been done, GitHub would be a smoking pile of servers at this point. There's a lot I could list, I'm not sure which are already public, but improvements on the order of using thousands fewer CPU's to serve even more traffic than before have been made, and still are, I'm sure.
- GitHub was already serving billions of requests/day before agentic coding hit. Their challenge wasn't scaling a fresh new system with a few users an order of magnitude; it was taking one of the busiest and most important sites on the Internet, and getting hit with 14x traffic in a year, and having to plan for 100x. If you think your systems and infrastructure would have survived that, if you think you would have been able to politically navigate and succeed in getting projects green-lit at a large company to prepare for 100x scaling before it hit, to get those resources for "we might have scaling problems in a year or two" instead of getting them to ramp up on AI coding and other features that were crucial to growth right now, you don't understand large organizational dynamics. That's not a complaint about GitHub or Microsoft; it's an observation about capitalism and how any mature management group prioritizes things in software. I'd expect everyone in the San Francisco/Silicon Valley Reality Distortion Field to understand that.
- We talk in terms of "14x commits" to Git but that's only part of the story. GitHub Actions, webhooks, github.com itself, and other parts of GitHub, have all been under pressure. It's the totality of it, the seams that have been exposed at scale that couldn't have been exposed without that scale, that have caused the instability. That's why architecture gets overhauled.
- Yeah, GitHub has had less consistent uptime since the 2018 Microsoft acquisition. The GitHub that existed before that had much less functionality, an order of magnitude fewer users, and had received very little improvement in the few years before. Microsoft invested and enabled GitHub to grow into something much bigger than it ever could have without them.
- Be grateful that Microsoft - with a 50-year history of shipping developer tools, and more experience operating enterprise software than any other company on the planet - acquired GitHub instead of Google, which was the other major player in contention. Spend a minute or two thinking about the product journey GitHub would have taken under Google, and then think about how many non-search, non-advertising products have succeeded there. Which amazing developer tools from Google do you use regularly? Yeah, I thought so. On behalf of Microsoft, you're welcome.
- Did you notice that GitHub swapped out one of its data centers last year for one 3x larger? No? Maybe that's because they executed it flawlessly, with no downtime. If you've ever done that on a massive web site with as much scrutiny as GitHub receives, you get a gold star.
- Did you notice that over 50% of GitHub read traffic is now being served from Azure instead of GitHub's own data centers, and growing, and that all GitHub Monolith traffic is scheduled to be served from Azure instead of GitHub's data centers by the end of CY26? This massive migration is taking place while traffic is going insane. From https://github.blog/news-insights/company-news/github-availa...: "GitHub can now serve a larger share of customer requests from independent Azure capacity, reducing reliance on any single datacenter while preserving performance. Monolith read traffic served from Azure Central US peaked at 52.75% on July 28—the first time we consistently remained above the halfway line. Git traffic in Azure reached 47%, up from 43% in June, and 29% of all repositories now have a second replica in Central US, making failover less disruptive when a region degrades."
- OpenAI and Anthropic obviously have significant scaling challenges as well, but there's a huge separation between the GPU-based inference part, and the CPU-based front-end systems. Their CPU-based systems are significantly simpler and newer than GitHub's, so I'm not surprised they have fewer outages, but it's still non-zero.
- Ruby on Rails is not the problem (and, for the record, I don't even like Ruby or Rails). Most of the performance-critical systems that were built on Ruby have been migrated to Go or some other language that takes full advantage of multithreading. The web endpoints still served by Ruby are fine.
- Spare me the "Azure sucks" comments. Please. Microsoft Azure is the second-largest computing system on the planet - only AWS is larger in terms of hardware but not in terms of the number of products they ship - and Microsoft's first-party systems that run on it, all at the same time, are among the largest, busiest, most important systems on the planet. Entra ID, Azure SQL, Service Bus, Event Hubs, Office 365, Cosmos DB, OneDrive, etc. all in the tens-to-hundreds-of-billions requests/day. I'm not even counting the massive customer-owned systems that run on it, including almost the entire Fortune 500. Nothing is perfect, everyone has downtime, we all always want more and better features, I want improvements from Azure, too, but, please, grow up. I have opinions, too, I've been programming since the Apple ][+, there are popular technologies that I don't like, but I know that's subjective. "Your favorite technology sucks, mine is better" is not an objective statement about anything.
Anyway, if you haven't shipped at GitHub's scale, in this dynamic of a world of feature churn, and one-of, if not the largest, traffic spikes in the history of the web, have fun saying whatever you're going to say. I've never worked at Google or Amazon, I have opinions about their product design and culture, but there's nothing I can say about their infrastructure and systems because I have no idea about them, and what we all do in this crazy world of programming is harder than it looks.
For some, this won't be the answer to "what's going on?" because I'm not pointing fingers at any one thing. To me, the real answer is: an unprecedented scaling event on what was already one of the busiest sites on the web exposed seams in their systems, including the need to move from their own data centers to a much more scalable cloud provider - that are being addressed as quickly as possible.
I'm not telling you not to look for alternatives, I'm not telling you what to do about it, I'm not saying that all of their problems will magically be solved soon, I have no idea. I'm even launching a new version control system myself very soon to compete with them. But I do know that the thought "GitHub doesn't know what they're doing" is wrong and unhelpful.
Lots of good points in there but when you say "swapped out data center" that sounds very un-cloudy and unreliable to me - workloads in AWS are spread across many buildings unless you try really hard or pay $$$$
14x is a lot commits, but I seriously have doubts that some pushes are putting them over the edge. (as many have suggested)
> GitHub Actions
For me this is probably this biggest culprit along with the general pressure of AI against the hardware. Github actions became really popular. So many of those 14x commits are generating real CPU and network load.
This is a really non trivial problem. They can't just wall it off. They have real customers on that stuff.
GitHub Actions is 'simpler' to scale, it's just a bunch of runners, and Azure has a lot of capacity for VMs (mostly). Git itself is a compute intensive process and is far more interlinked.
Not sure how you see git repos as a compute intensive process. A single git commit takes milliseconds. It's mostly a hash calculation and a small file transaction. They've added some database stuff around the edges, but that is pretty tame as well.
Actions are unbounded, take minutes to resolve - sometimes hours and there are megabytes, gigabytes of network transactions.
Scaling commits is a load balancer and some more instances of a thing you’ve built over and over.
Scaling actions is VMs, networking, imbalances in load, memory, insuring that folks aren’t running VM training against your infrastructure. Oh, and maybe you want to make sure folks aren’t trying to hack that infrastructure as well.
>Spare me the "Azure sucks" comments. Please. Microsoft Azure is the second-largest computing system on the planet
The fact that Azure is big and important has very little to do with the quality of the implementation (and my experience with the services on top of it suggests that quality is not great).
Anyone that has to work with azure knows that the bad stability and quality of the platform impact them. Often they just don't have the choice because only choice available on bigco huge framework contract.
> Spend a minute or two thinking about the product journey GitHub would have taken under Google, and then think about how many non-search, non-advertising products have succeeded there.
Android
Youtube
Waymo
Chrome
Workspace
GCP
>Which amazing developer tools from Google do you use regularly?
3. Click the "Add an Application" button in the address bar to get a Start Menu icon for the page.
4. Delete the in-box Weather app icon.
Now you get the same Weather app in about 130MB of RAM, with no ads. It's not as nice as a native app, of course, but it's 1000% better than the useless ads and MSN feed that you can't block from the built-in Weather icon.
(Also, go into Widget settings and turn off "Discover / Microsoft Start feed". Same crap, different surface. Get rid of it.)
"Now you get the same Weather app in about 130MB of RAM"
What a wonderful world we have created where the fix for a 1GB RAM guzzler reduces it to a mere 130MB.
Now I could start off with my old ZX80 or even my C64 (which is doing fine, thanks, and sports a USB "drive" next to its Quickshot II) but I think my first 80486 is a realistic comparison.
That ran at 25MHz, had a maths co-pro in it had 4MB RAM, a 20MB IDE HDD and I think the Orchid graphics card had 0.5MB RAM. The monitor was of course a 14" CRT VGA. OK so late 1980s!
However, that thing could run Win 3.1 and Word 2 and I think I managed to wedge a dodgy copy of Quark Express on it. I could play F117 and other games on it.
Oh well, lets see what this Linux box has on it:
/usr/bin/inxi - 1.4MB
$ inxi -w Yeovil,UK
Weather:
Report: temperature: 16.05 C (61 F) conditions: scattered clouds
Locale: Yeovil, UK current time: Mon 10 Aug 2026 01:16:05 BST
Source: OpenWeatherMap.org
Obviously, I could install a Flatpak to do that instead and waste far more resources 8)
What inxi commnd shows a map of pollen concentration, weather alerts, and air quality? How do you generate a graph comparing this year's temperatures to the previous yeas' average climate?
Comparing inxi to Windows Weather is like comparing vi to Autodesk. You can make 3d graphics for sure, but it's not exactly comparable.
When I run inxi -w, I get this:
Weather:
Message: Error: Your access to this resource has been blocked. Automated
requests or excessive use are not permitted. This is clearly stated in
both the help menu and man page. If you want a CLI weather tool for
routine use, try wttr.in, or use a weather widget. Remember, you never
had a "right" to this feature, it was just a little easter egg, mostly
for sys admins, which has instead been consistently abused. If you want
to know your local weather, look out your window, there it is.
Looks like something in my powerline config somewhere must be querying too much? I can't find anything on my systems, must be a very old misconfiguration on my end. I still think I prefer the Microsoft tool.
No, but my ISP's IP range once belonged to a business hoster. Wikipedia blocks my IP for that reason too. I'm guessing it has something to do with that, because I can't find inxi references in my config files.
Firefox doesn't support PWAs on desktop and removed their previous self-designed equivalent years ago. It's the reason Brave is still running in the background on my machines.
Are you sure? I still see the "add tab to taskbar" option in the address bar and clicking it open the tab in a new window and shows a popup to pin the tab to the taskbar.
This is a narrative that MS, Google etc want you to believe. They gutted it for ads. That's what's happening. There are workarounds now, that's what the existing ad block options are. They are heavily kneecapped.
Plus, those workarounds will only continue to work (in as much as they currently do) for as long as Google suffers them to. Enshittification, as always, works in baby steps.
uBlock lite is already ineffectual in comparison to uBlock Origin. I understand that it's nonetheless good enough for many people, but it will get worse from here.
MV3 is severely limited in what it can do. One example that blew my mind is that chrome happily renders webpages immediately after application launch, while extensions are still loading.
Firefox, in contrast, doesn't let the network request fire off to the page until every extension is ready. So chrome is essentially screwing over the user, letting ads/ blocked content load, and intentionally degrading their experience, to make things appear slightly faster.
That's nothing to do with MV3 though, that was also true with MV2. The situation actually seems improved in MV3, because static (and dynamic, IIRC) DNR rules are initialized and enforced immediately, right?
> According to Windows Latest, the high memory consumption is due to the fact that Weather is not a fully native Windows application. Instead, it is essentially an MSN Weather web app built on Microsoft's WebView2 framework.
I wonder how this approach is having such an high reduction in memory usage compared to what’s stated in the article. I would assume both use the exact same WebView here.
The only explanation here that I can think of would be that the “Add an application” starts it under an existing Edge process which shares it’s memory instead of completely isolated.
Yes, using an already-running Edge instance saves a bunch of memory. Also, this approach has the advantage of allowing all of your browser extensions to modify your Web-App experience. WebView2 doesn't relay Edge extensions.
For the Windows look and feel, I recommend Mint + Cinnamon. It took me about 15min to not miss anything on Windows and was shocked how I had forgotten how clean a system can be.
Mandatory Windows accounts can still be circumvented with a console command for Win 11 installations, but the overall enshitification package just got worse and worse.
Yeah I love KDE especially because it doesn't force a certain look and feel, it's super configurable out of the box and even more if you install themes and plugins (which I don't even do).
Did it the day before yesterday with a fresh win11 image from Microsoft (no rufus magic or anything). You can press shift + f10 and run a command I don't remember. It restarts the install process and lets you create a local account while it still patches the system before the first start (no fiddling with internet connections required). At least that still works.
> According to Windows Latest, the high memory consumption is due to the fact that Weather is not a fully native Windows application. Instead, it is essentially an MSN Weather web app built on Microsoft's WebView2 framework. Task Manager shows multiple Chromium-based subprocesses running simultaneously, which contributes to the unusually high RAM usage.
If you have Edge or Chrome open (or anything else that similarly uses Chromium) then the incremental increase in RAM usage from the Weather app is likely much smaller than the headline 1GB.
As much as I hate the weather app myself, this isn't an equivalent solution. The weather app shows the temperature in the taskbar itself, without opening anything. Also, I don't keep my browser open 24/7, so it's more than just loading a single extra tab, you need to wait for the browser process itself to start up.
If you want to load weather in a browser from the taskbar instead of having it available without having to open another application, I'd recommend using dillo or some such to load wttr.in or similar. A single http request that loads fast from a comparatively simple application that also loads fast. You might even be able to load it as a .hta file.
A better workaround is to install Linux. My entire operating system, which includes Niri window manager and Dank Linux running a weather app, among many other things takes less around 1 gig of ram.
The most-difficult part of installing any Linux distro is getting the motivation to create the boot media. Luckily for Ubuntu (I'm running v26) you can use an old HD or USB and then [your flavor of LLM] to then walk you through the steps on how to make the media bootable (at least: that's what I did). I am a bluecollar electrician (yes, it took me a few tries; yes, I accidentally fucked up my first few OS's doing stupid not-recommended shit).
----
Highly recommend Ubuntu; I've dabbled in the past, but the fact that AMD and nVIDIA drivers are pre-installed, in a user-friendly GUI installer... is what has helped me make Linux my main desktop (over the past half-year). Bought a Microcenter prebuilt and never even booten Win11 (straight to Ubuntu installer).
Over the past three decades, I have really tried to like several Linux'es, but have always reverted to (e.g:) Win7 or MacOS. Being able to run local LLM-models is what led me to Ubuntu (RX580->VEGA64->5070Ti)... it's been a journey =P
Sidenote: "AMD Drivers", in the sense of "something you have to download and install manually", are not a thing any more, fortunately. It's just nvidia now.
In Debian, even nvidia drivers are part of the OS repositories, albeit in the non-free category which is not enabled by default. I would be surprised if Ubuntu did not have them too.
Agree with this. As much as Ubuntu is derided for several reasons, it just works. I tinkered with different distros for years, and while it was super fun, I wasn't very productive. Ubuntu and Fedora are the only distros where stuff just worked for me.
I've been using Ubuntu since around the time you needed an invite to create a Gmail account. Back then it was a breath of fresh air, because the install was batteries-included and Canonical's help forums were genuinely useful.
Ubuntu is bloated now but still just about OK despite:
* Snooping on users.
* Forcing people to install Snaps.
* Relentlessly promoting systemd, which turned out to be a giant mistake.
* Pointless fiddling about with the desktop in every release.
* Canonical having a universe-class egomaniac for a CEO, even by South African tech-billionaire standards.
But if Ubuntu has been treading water for 25 years, that represents an improvement over the titanic disaster that is Windows 11, and the walled-garden den of developer hostility that is Apple.
Bootable media isn't that difficult. You need the right tool. Not sure what's current on Windows; I used Etchdroid recently. And you need a spare USB drive. That's the most annoying part. And you need to download the OS. And then you use the tool to put the OS on the drive.
I find that Rufus (https://rufus.ie/en/) is excellent for writing USB disks. I believe it's Windows/DOS only, but even as a Linux user for several decades, I boot into Windows just to run it if I need to burn an installer.
I've tried that previously (or sudo dcfldd to give better feedback), but it doesn't always seem to work for ISOs. Ironically, I've had issues writing Linux ISOs onto a USB stick under Linux, so nowadays I just use Rufus which has never failed me.
I've installed a few SBCs and they don't tend to have an issue booting, so I do use dd/dcfldd for those. I wonder if it's because they use SD cards that they don't have an issue.
I too want to shout how it's the Year of the Linux Desktop but I've been fighting mate-clock-applet recently. It has a memory leak, somewhere, and it slowly consumes RAM.
Normally though it's ~65 MiB for both clock & weather.
Also mate-screenshot is leaking RAM/processes. About ~40 MiB leaked per screenshot.
These things do tend to get fixed over time, though.
Doesn't help if in the end most GUI applications even on Linux, are Electron junk like this weather app, as this lack of respect for the user is everywhere.
Super interesting; it's been exactly the opposite in my experience. MacOS/iOS users put up with whatever Apple is telling them they should love - Liquid Glass, right? App Store restrictions, right? Lousy AI, right? - whether it's true or not. Linux users are willing to put up with any amount of work - i.e. bullshit - to make their installations work just to get a UX that's almost as good as Windows from 20 years ago.
As for Windows, I like living in cities. If you live in a city long enough - at least through a couple of major macroeconomic cycles - you'll see it improve in some ways and some neighborhoods, get worse in some ways and some neighborhoods, then improve again, etc. Cycles. Windows found itself in a downturn the last few years, primarily due to the org structure that took Windows shell away from people who could really look after it, and, yes some of those execs have earned my permanent scorn, like a bad local politician would. Now there's a focused team making that neighborhood better. I'm here for the whole ride.
I'll use Linux on computers I don't have to log into very much, so I don't have that UX problem (case-sensitive command lines? Seriously? The 1970's called...). I'll never move to MacOS City; it's like wanting to live in The Villages. Manicured lawns with no soul.
So I live in Windows Town, it's not perfect, no city is, but it's always changing, always moving, going through ups and downs, and now it has a new "city council" that's making changes for the better. And I'm expert enough to make it great for me, and avoid the "neighborhoods" that suck.
This article highlights a problem that Windows has right now, and one which I agree with, which is shipping web apps instead of native. For now, I have a workaround for it:
1. Install uBlock Origin in Edge.
2. Start Edge, browse to MSN Weather.
3. Click the "Add an Application" button in the address bar so I have a Start Menu icon for the page.
4. Delete the in-box Weather app icon.
Now I have the same weather site, no ads, running in about 130MB of RAM.
Enjoy your choice of operating system. Live where you want. Sometimes your city has problems. Move if you want, but don't think you've permanently solved your UX issues if you do, you've just swapped them for different ones.
I agree, KDE Plasma runs circles around every UX available now or at any time in the past. And it's not even close.
I've tried using MacOS, and there are so many frustrating things you just have to put up with. Windows is much worse, but to be fair Windows was okay at one point. KDE is, out of the box, seamless, sensible, fast, and unbelievably feature-packed. But even if the default experience isn't your taste, it's trivial to customize it to work however you want. Whatever panels you want, where ever you want, with whatever widgets you want. WYSIWYG with point-and-click. And the windows, too. Window rules, KWin scripts (which are actually full-blown plugins), whatever you want. I want Steam to always open full screen on start up on my second monitor on the third desktop? Sure, I can do that, again point-and-click.
I truly cannot sign the praises of KDE enough. I think everyone who dislikes KDE has probably not delved deep enough. I can recreate the entire GNOME desktop in KDE in an hour. I can recreate MacOS in 30 minutes.
Liquid Glass needs a lot of cleaning up but it’s fine. App Store restrictions mean very little for iOS users in general and next to nothing for Mac users. Lousy AI also means little to nothing on Mac and earns at max a shrug on iOS—you even have a handy little button to disable all Apple Intelligence shenanigans altogether which is something no Windows user can easily do with Copilot.
No, Apple users are most definitely not having to deal with the same level of bullshit Microsoft’s users are. And it’s not even close.
>> Linux users are willing to put up with any amount of work - i.e. bullshit - to make their installations work just to get a UX that's almost as good as Windows from 20 years ago
I put in the exact same amount of effort to install a Linux distro on my laptop that I would have to install Windows.
1. Download the ISO
2. Write to a USB drive
3. Reboot
4. Run the installer.
5. ???
6. Profit
And all for an experience with way less bloat, none of the bundled spyware, more consistent UI design (depending on which desktop you're using), and better perceived performance.
>>This article highlights a problem that Windows has right now, and one which I agree with, which is shipping web apps instead of native. For now, I have a workaround for it:
This is still a web app, it doesn't solve the problem of MS being too lazy or cheap to make real desktop apps for things. I mean this is the same company who thought taking the Start Menu of all things and re-making it with web crap was a good idea.
I mean, if the only things you can name are Liquid Glass, App Store and not-so-intelligent AI then it seems like Apple users are actually doing quite well?
It's impossible for every idea to be a hit. Liquid Glass was a miss and they backtracked on some changes and fixed some issues that people were vocal about.
I can name lots of things I can't stand about MacOS and iOS and Apple hardware, but that wasn't the point of the comment. Some of the things I can't stand about Apple are things that people used to Apple's stuff don't mind. Cool for you. But I'm still not moving to The Villages, ever.
> things that people used to Apple's stuff don't mind
... but that's what you said about Windows:
> Windows found itself in a downturn the last few years
> I live in Windows Town, it's not perfect, no city is
> [...] avoid the "neighborhoods" that suck
Really the original comment just reads like you wanted to rant about Apple users and MacOS and that's it. You didn't really give any strong arguments for why Windows is better than MacOS - and "it's not perfect" is not an argument when in the same post you say the exact same about your preferred OS.
Linux as an operating system is easily better than macOS. Third party software is lacking though. There’s really nothing that compares to Bear, Things, DEVONthink, Transmit, etc. What I also missed in Linux was the prevalence of what Apple calls “x-callback URLs” [0]. They are incredibly useful and great for glueing different pieces of software together.
With some of Apple’s recent decisions, I do wonder if we’ll see this change in the future. Little Snitch put out a proof of concept for Linux. While it exposes itself as a fairly rough webpage, someone is thinking about it. Linux culture would need to accept paid software, though.
I know that OpenAI has made computing deals with other companies, and as time goes on, the percentage of inference that they run their models on will shift, but I doubt that much, if any, of that has moved from Microsoft Azure data centers yet, so that's not a reason for difference in model performance.
With that said, Microsoft has a different level of responsibility, both to its customers and to its stakeholders, to provide safety than OpenAI or any other frontier provider. That's not a criticism of OpenAI or Anthropic or anyone else, who I believe are all trying their best to provide safe usage. (Well, other than xAI and Grok, for which the lack of safety is a feature, not a bug.)
The risk to Microsoft of getting this wrong is simply higher than it is for other companies, and what's why they have a strong focus on Responsible AI (RAI) [1]. I don't know the details, but I have to assume there's a layer of RAI processing on models through Azure OpenAI that's not there for just using OpenAI models directly through the OpenAI API. That layer is valuable for the companies who choose to run their inference through Azure, who also want to maximize safety.
I wonder if that's where some of the observed changes are coming from. I hope the commenter posts their proof for further inspection. It would help everyone.
I'm a long-time C# dev who got into F# about five years ago. F# is so awesome, I hope that if Grace catches on, that more people will pay some attention to it.
If it sounds like I'm defending GitHub a bit, yeah, I am. I can't think of a comparable situation that any other web site has been through. This isn't "oops, we didn't plan for the Black Friday sale", this is a once-in-a-generation event focused on one, important web property that no one would have handled perfectly. (Even us all-knowing commenters here on HN.)
You know that old trope in every submarine movie, where the captain tells the driver to dive lower than they've ever gone, and someone says "I don't know if the boat can take it!" and then they switch the camera to some engineering room where the boat is groaning under the pressure and a bolt comes loose and water starts spraying everywhere but someone runs up with a giant wrench and tightens it and looks around at everyone else in the room like, "whoa, that was close..."?
Or Scotty and Star Trek, "It canna take much more, Captain!"
That's what GitHub has been going through. It's had problems, but it's still here, and usually working well.
- Let me start with: GitHub is ~3,000 talented and really nice people trying to do the right thing, with a great culture. That's the most important thing I want to communicate. While I'm sure no one at GitHub is happy about their uptime, believe me when I tell you that GitHub Engineering is exceptional. They have great leadership, depth at all levels (Distinguished / Principal / Staff / Senior / earlier-in-career) and if you ever get a chance to hire someone from GitHub, you should. I hope all of you get to work with engineers as good as GitHub's.
- GitHub has been improving all aspects of its infrastructure steadily for many years now. The GitHub Engineering blog https://github.blog/engineering/ has been documenting this the entire time. Go point your favorite LLM at it and ask for a summary of all of the major system improvements since 2020. If that work hadn't already been done, GitHub would be a smoking pile of servers at this point. There's a lot I could list, I'm not sure which are already public, but improvements on the order of using thousands fewer CPU's to serve even more traffic than before have been made, and still are, I'm sure.
- GitHub was already serving billions of requests/day before agentic coding hit. Their challenge wasn't scaling a fresh new system with a few users an order of magnitude; it was taking one of the busiest and most important sites on the Internet, and getting hit with 14x traffic in a year, and having to plan for 100x. If you think your systems and infrastructure would have survived that, if you think you would have been able to politically navigate and succeed in getting projects green-lit at a large company to prepare for 100x scaling before it hit, to get those resources for "we might have scaling problems in a year or two" instead of getting them to ramp up on AI coding and other features that were crucial to growth right now, you don't understand large organizational dynamics. That's not a complaint about GitHub or Microsoft; it's an observation about capitalism and how any mature management group prioritizes things in software. I'd expect everyone in the San Francisco/Silicon Valley Reality Distortion Field to understand that.
- We talk in terms of "14x commits" to Git but that's only part of the story. GitHub Actions, webhooks, github.com itself, and other parts of GitHub, have all been under pressure. It's the totality of it, the seams that have been exposed at scale that couldn't have been exposed without that scale, that have caused the instability. That's why architecture gets overhauled.
- Yeah, GitHub has had less consistent uptime since the 2018 Microsoft acquisition. The GitHub that existed before that had much less functionality, an order of magnitude fewer users, and had received very little improvement in the few years before. Microsoft invested and enabled GitHub to grow into something much bigger than it ever could have without them.
- Be grateful that Microsoft - with a 50-year history of shipping developer tools, and more experience operating enterprise software than any other company on the planet - acquired GitHub instead of Google, which was the other major player in contention. Spend a minute or two thinking about the product journey GitHub would have taken under Google, and then think about how many non-search, non-advertising products have succeeded there. Which amazing developer tools from Google do you use regularly? Yeah, I thought so. On behalf of Microsoft, you're welcome.
- Did you notice that GitHub swapped out one of its data centers last year for one 3x larger? No? Maybe that's because they executed it flawlessly, with no downtime. If you've ever done that on a massive web site with as much scrutiny as GitHub receives, you get a gold star.
- Did you notice that over 50% of GitHub read traffic is now being served from Azure instead of GitHub's own data centers, and growing, and that all GitHub Monolith traffic is scheduled to be served from Azure instead of GitHub's data centers by the end of CY26? This massive migration is taking place while traffic is going insane. From https://github.blog/news-insights/company-news/github-availa...: "GitHub can now serve a larger share of customer requests from independent Azure capacity, reducing reliance on any single datacenter while preserving performance. Monolith read traffic served from Azure Central US peaked at 52.75% on July 28—the first time we consistently remained above the halfway line. Git traffic in Azure reached 47%, up from 43% in June, and 29% of all repositories now have a second replica in Central US, making failover less disruptive when a region degrades."
- OpenAI and Anthropic obviously have significant scaling challenges as well, but there's a huge separation between the GPU-based inference part, and the CPU-based front-end systems. Their CPU-based systems are significantly simpler and newer than GitHub's, so I'm not surprised they have fewer outages, but it's still non-zero.
- Ruby on Rails is not the problem (and, for the record, I don't even like Ruby or Rails). Most of the performance-critical systems that were built on Ruby have been migrated to Go or some other language that takes full advantage of multithreading. The web endpoints still served by Ruby are fine.
- Spare me the "Azure sucks" comments. Please. Microsoft Azure is the second-largest computing system on the planet - only AWS is larger in terms of hardware but not in terms of the number of products they ship - and Microsoft's first-party systems that run on it, all at the same time, are among the largest, busiest, most important systems on the planet. Entra ID, Azure SQL, Service Bus, Event Hubs, Office 365, Cosmos DB, OneDrive, etc. all in the tens-to-hundreds-of-billions requests/day. I'm not even counting the massive customer-owned systems that run on it, including almost the entire Fortune 500. Nothing is perfect, everyone has downtime, we all always want more and better features, I want improvements from Azure, too, but, please, grow up. I have opinions, too, I've been programming since the Apple ][+, there are popular technologies that I don't like, but I know that's subjective. "Your favorite technology sucks, mine is better" is not an objective statement about anything.
Anyway, if you haven't shipped at GitHub's scale, in this dynamic of a world of feature churn, and one-of, if not the largest, traffic spikes in the history of the web, have fun saying whatever you're going to say. I've never worked at Google or Amazon, I have opinions about their product design and culture, but there's nothing I can say about their infrastructure and systems because I have no idea about them, and what we all do in this crazy world of programming is harder than it looks.
For some, this won't be the answer to "what's going on?" because I'm not pointing fingers at any one thing. To me, the real answer is: an unprecedented scaling event on what was already one of the busiest sites on the web exposed seams in their systems, including the need to move from their own data centers to a much more scalable cloud provider - that are being addressed as quickly as possible.
I'm not telling you not to look for alternatives, I'm not telling you what to do about it, I'm not saying that all of their problems will magically be solved soon, I have no idea. I'm even launching a new version control system myself very soon to compete with them. But I do know that the thought "GitHub doesn't know what they're doing" is wrong and unhelpful.