Claude is starting to get really good at coding and autonomously fixing pull requests. It's becoming clear that in a year's time, a large percentage of code will be written by LLMs.
Let me show you what I mean:
Let me show you what I mean:
To start, if you want to see Claude 3.5 Sonnet in action solving a simple pull request, here's a quick demo video we made.
(voiceover by the one and only @sumbhavsethia)
(voiceover by the one and only @sumbhavsethia)
In our internal pull request eval, Claude 3.5 Sonnet passed 64% of our test cases.
To put this in comparison, Claude 3 Opus only passed 38%.
To put this in comparison, Claude 3 Opus only passed 38%.
3.5 Sonnet performed so well that it almost felt like it was playing with us on some of the test cases.
It would find the bug, fix it, and spend the rest of its output tokens going back and updating the repo documentation and code comments.
It would find the bug, fix it, and spend the rest of its output tokens going back and updating the repo documentation and code comments.
Side note: With Claude's coding skills plus Artifacts, I've already stopped using most simple chart, diagram, and visualization software.
I made the chart above in just 2 messages.
I made the chart above in just 2 messages.
Back to PRs, Claude 3.5 Sonnet is the first model I've seen change the timelines of some of the best engineers I know.
This is a real quote from one of our engineers after Claude 3.5 Sonnet fixed a bug in an open source library they were using.
This is a real quote from one of our engineers after Claude 3.5 Sonnet fixed a bug in an open source library they were using.
At Anthropic, everyone from non-technical people with no coding experience to tenured SWEs now use Claude to write code that saves them hours of time.
Claude makes you feel like you have superpowers, suddenly no problem is too ambitious.
The future of programming is here folks.
Claude makes you feel like you have superpowers, suddenly no problem is too ambitious.
The future of programming is here folks.
Loading suggestions...