Tessera

Folgen

Dan Luu

@danluu.com

danluu.com / www.patreon.com/danluu/

55 Posts3738 Follower 14 Folgt

Dan Luu @danluu.com · 4d ago

There's no reason for software to be slow anymore: danluu.com/perf-opt/

The other day, I saw a viral tweet saying that people talking about how LLMs are causing slow, bloated, code are going to eat crow once they re-write everything in super-optimized assembly. We're not quite at the point where we want to write everything in assembly, but some variant of what Nolan Lawson said about testing, you can choose how many bugs you want now, which I less eloquently noted here, is becoming more true for performance.

In response to a comment in my last post that the cost of formerly specialized performance work has dropped by many orders of magnitude and performance work that used to require a person or team that had a rare set of skills can be done by anyone who can type a few sentences1, which means that you can do all sorts of optimizations that used to be too expensive to be worthwhile for all but the largest scale or most lucrative projects, Marc Brooker responded with

Completely agree with your closing point. Dynamic custom software, fitted to a particular workload rather than a class of workloads, seems like a very likely outcome. (Which comes with all kinds of fun risks and opportunities of its own). Kind of reminds me of FFTW. And a ton of weird old demoscene techniques which were all about being super fast and small on a very particular problem (and often very particular hardware). For example, I remember a demo that re-used its code as textures to get great cache locality.

And Michael Malis has noted

There’s been a meme circulating about how AI doesn’t help because “code was never the hard part.” I think that’s true in some domains, but in others, writing the code absolutely was the hard part. JIT compilers are a great example of that. For many pieces of software, a JIT compiler would help a lot with speeding up the code. The rarity of JIT compilers makes me believe that implementing a JIT compiler historically was too difficult for it to be worthwhile. LLMs have lowered the barrier to entry and made it much easier to writ…

Dan Luu @danluu.com · last wk.

The benchmarkpocalypse: danluu.com/benchpocalyp...

There's been a lot of talk about the vulnpocalypse, to which I don't have much to add because I'm not a security person, but I haven't seen much discussion on the closely related (and to be fair, less serious issue), the benchmarkpocalypse.

While it's become easier than ever to make serious performance gains, it's also become easier than ever to reward hack a benchmark and make fake performance gains. The former is probably happening quietly across many different companies, but the latter is something I see at least once a week nowadays. Someone will claim they optimized X and got some huge performance improvement over existing software but, when you look at it, what they did was make some optimization that improves benchmark performance without actually improving real-world performance. This is often some kind of "we re-wrote X in Rust"1 project or a new startup that's looking to either fundraise or sell something, but it happens on other kinds of projects as well2.

Rather than point to someone's bad claim, I'll point to FRE, this regex engine I had an agent build, which I could claim is the world's fastest regex engine because it beats the Rust regex crate at the fairly comprehensive rebar regex benchmark suite. But this was created by putting an agent in a loop for a month with instructions to not overfit to the benchmark but no real supervision. For the most part, getting an LLM to give you a good benchmark score is fairly easy, and this case was no different; it took a couple weaks to roughly match Rust regex crate performance and then another couple weeks to get to 1.4x faster3 on rebar. But, agents are wont to reward hack and ovefit unless you put serious guardrails in place to avoid that, which I didn't do in this case as an experiment.

To check for overfitting, I somewhat arbitrarily4 used the ripgrep benchmark corpus as a benchmark a holdout and instead of being 1.4x faster it was 10x slower on cases where the benchmark didn't take forever due to an alg…

Dan Luu @danluu.com · 2w ago

How do programming languages impact token efficiency and correctness? danluu.com/pl-tokens/

Dan Luu @danluu.com · 3w ago

Earlier this year, the FTC Chair said "you're going to have a hard time keeping up with the number of cases we're gonna be bringing" on privacy. It seems like this is starting; the FTC has filed a case against a company for promising privacy while running ad trackers on their site, which share info

Dan Luu @danluu.com · last mo.

Exercises in benchmarking and evals, part 7: performance napkin math, DeepSWE / Senior SWE-Bench, and winter tires danluu.com/exercise-7/

Screenshot of above the fold content from link.

Graphs of memory latency over time, showing decline until 2005 or so and then a rough levelling off or maybe a small increase.

Dan Luu @danluu.com · 2mo ago

Agentic test processes, LLM benchmarks, and other notes on agentic coding from Galapagos Island: danluu.com/ai-coding/

In general, when I talk to software folks about testing, I'm coming from such a different place that they immediately look at me like I'm an alien, so let's talk about how we tested at this hardware company I worked for, Centaur, which informs my biases about how I like to work. Some of the things that we did that were or are unorthodox in the software world are:

Hired dedicated QA / test engineers, with testing being a first-class career path on par with being a developer
No code review by default
Virtually no hand-written tests
Constant testing via what programmers sometimes called property based testing, randomized testing, fuzzing, etc., although we just called those tests (hand-written tests were called "hand tests").
Large regeression test suite (3 months wall clock to execute on compute farm)
No unit tests

Just to give you an idea of the general structure, when I left (in 2013), we had about 1000 machines generating and running tests at all times for roughly 20 logic designers and 20 test engineers. This was on prem and the machines took up half a floor of the building we were in.

The general structure was that we had maybe 20% of machines running regression tests, and 80% generating and running new tests. Three months of regression tests is too much to gate commits on, so there was a much shorter list of tests that took maybe 10 minutes or so to run that people would run before committing. Those pre-commit tests would run on a special setup to run as quickly as possible, with overclocked machines that were the fastest machines money could buy, as well as a different simulator setup.

Dan Luu @danluu.com · 2y ago

A version of Missile Command for the Commodore 64 where the bottom of your screen is the game state in memory and missiles cause memory corruption: csdb.dk/release/?id=.... In the video below, a missile broke my controls and caused my cursor to get stuck moving down and to the left.

Dan Luu @danluu.com · 2y ago

I find it interesting/surprising that Tesla topped the www.iseecars.com/most-dangero... fatalities per mile ranking from 2018-2022. Fatality rate is strongly negatively correlated with price and weight and Teslas are much more expensive and heavier than average.

Please see link for text version of table

Bar chart showing that Teslas have a higher average selling price than any other tracked car manufacturer

Dan Luu @danluu.com · 2y ago

A funny side effect of the crackdown on "AI" scraping is that I keep getting banned from sites for browsing too quickly. I barely use reddit anymore and I still managed to get IP banned for scraping (the error message indicated that I should get in touch with them if I want to do bulk accesses).

Dan Luu @danluu.com · 2y ago

Steve Ballmer was an underrated CEO danluu.com/ballmer/

There's a common narrative that Microsoft was moribund under Steve Ballmer and then later saved by the miraculous leadership of Satya Nadella. This is the dominant narrative in every online discussion about the topic I've seen and it's a commonly expressed belief "in real life" as well. While I don't have anything negative to say about Nadella's leadership in this post, this narrative underrates Ballmer's role in Microsoft's success. Not only did Microsoft's financials, revenue and profit, look great under Ballmer, Microsoft under Ballmer made deep, long-term bets that set up Microsoft for success in the decades after his reign. At the time, the bets were widely panned, indicating that they weren't necessarily obvious, but we can see in retrospect that the company made very strong bets despite the criticism at the time.

In addition to overseeing deep investments in areas that people would later credit Nadella for, Ballmer set Nadella up for success by clearing out political barriers for any successor. Much like Gary Bernhardt's talk, which was panned because he made the problem statement and solution so obvious that people didn't realize they'd learned something non-trivial, Ballmer set up Microsoft for future success so effectively that it's easy to criticize him for being a bum because his successor is so successful.
Criticisms of Ballmer

For people who weren't around before the turn of the century, in the 90s, Microsoft used to be considered the biggest, baddest, company in town. But it wasn't long before people's opinions on Microsoft changed — by 2007, many people thought of Microsoft as the next IBM and Paul Graham wrote Microsoft is Dead, in which he noted that Microsoft being considered effective was ancient history:

A few days ago I suddenly realized Microsoft was dead. I was talking to a young startup founder about how Google was different from Yahoo. I said that Yahoo had been warped from the start by their fear of Microsoft. That was why they'd po…

Dan Luu @danluu.com · 2y ago

A former Apple engineer discusses Google product culture: > My director wore an Apple Watch and had an iPhone ... my VP too. Nobody was expected to eat the dog food and so few did. This was crazy to me coming from Apple ....

mattnewton 1 hour ago | parent | next [–]

When I worked at Google I got re-orged into the same division as pixel / android.

My director wore an Apple Watch and had an iPhone for personal use, and I am pretty sure I saw an Apple Watch on my VP too. Nobody was expected to eat the dog food and so few did. This was crazy to me coming from Apple- I remember several internal sites would ask you to file a radar (bug report) on why you switched to chrome from safari if you opened them in chrome. So many crazy issues I saw and reported didn’t actually matter to many high ranking members of the pixel team because they didn’t use the devices after 5pm.

There is a lot of incredible talent in that team but I think Google needs a minor culture shift to compete with Apple here.

🌇👃🌃@waxrepli.ca · 2y ago

i can speak with a little authority on this, i work in this field DCs expend water by evaporation, either in cooling towers or evaporative coolers (water becomes vapor, taking latent heat energy with it into the air and leaving cooler liquid water behind) so we expend liquid water resources 1/

Dan Luu @danluu.com · 2y ago

On the 2011-2012 FTC antitrust investigation of Google: danluu.com/ftc-google-a...

This is a summary of the publicly available documents on the 2011-2012 FTC investigation of Google's allegedly antitcompetive actions in search and ads, followed by a tech-focused analysis of the decision from someone who's worked at the two companies that are discussed in the most detail in the memos (Google and Microsoft), worked in search, and worked closely with ads teams on optimizing ads ranking algorithms. I've seen a number of law-focused and economics-focused analyses, but I haven't seen a tech-focused analysis in a level of detail I find satisfying. In particular, a number of key arguments in the memos rely on evidence and inferences that would've seen implausible to someone who was familiar with tech, which I haven't seen discussed.

The law-focused and economics-focused analyses tend to avoid digging into this and, while there have been some articles written about tech errors for a lay audience, they've tended to explain that the inferences that were made were wrong in r...

Laurence Tratt @ltratt.bsky.social · 2y ago

What Factors Explain the Nature of Software? tratt.net/laurie/blog/...

Laurence Tratt: What Factors Explain the Nature of Software?

tratt.net

Dan Luu @danluu.com · 2y ago

Every once in a while, I think about going to work in the game industry.

@ZTGallagher • 1mo ago

I know a guy who was a QA tester for Obsidian Entertainment working on Neverwinter Nights 2. At the end of development, they invited all the QA testers to the parking lot for a celebration party.
There was no party, they disabled all their keys when they got out there, and told them they were all fired. And that was it...

Dan Luu @danluu.com · 2y ago

Interesting comment about SGI leadership knowing about the problems they were facing and still being unable to come up with a way to handle them.

Dan Luu @danluu.com · 2y ago

How web bloat impacts users with slow devices: danluu.com/slow-device/

Table showing that many websites are unusable with devices that are commonly used in non-rich countries.

See link for details.

Dan Luu @danluu.com · 3y ago

We're coming up on a decade since danluu.com/cpu-bugs/. Not only was the prediction valid for a decade, it looks like the next decade will be worse. Today's bug of the day: a significant fraction of high-end consumer CPUs crash if you run CPU-intensive workloads: www.radgametools.com/oodleintel.htm

$Intel Processor Instability Causing Oodle Decompression Failures RAD has become aware of a problem that can cause Oodle Data decompression failures, or crashes in games built with Unreal. We believe that this is a hardware problem which affects primarily Intel 13900K and 14900K processors, less likely 13700, 14700 and other related processors as well. Only a small fraction of those processors will exhibit this behavior. The problem seems to be caused by a combination of BIOS settings and the high clock rates and power usage of these processors, leading to system instability and unpredictable behavior under heavy load. As far as we can tell, there is not any software bug in Oodle or Unreal that is causing this. Due to what seem to be overly optimistic BIOS settings, some small percentage of processors go out of their functional range of clock rate and power draw under high load, and execute instructions incorrectly. This is being seen disproportionately in Oodle Data decompressi...$

Dan Luu @danluu.com · 3y ago

Diseconomies of scale in fraud, spam, support, and moderation: danluu.com/diseconomies...

f I ask myself a question like "I'd like to buy an SD card; who do I trust to sell me a real SD card and not some fake, Amazon or my local Best Buy?", of course the answer is that I trust my local Best Buy1 more than Amazon, which is notorious for selling counterfeit SD cards. And if I ask who do I trust more, my local reputable electronics shop (Memory Express, B&H Photo, etc.), I trust my local reputable electronics shop more. Not only are they less likely to sell me a counterfeit than Best Buy, in the event that they do sell me a counterfeit, the service is likely to be better.

Similarly, let's say I ask myself a question like, "on which platform do I get a higher rate of scams, spam, fraudulent content, etc., [smaller platform] or [larger platform]"? Generally the answer is [larger platform]. Of course, there are more total small platforms out there and they're higher variance, so I could deliberately use a smaller platform that's worse, but I'm choosing good options instead of...

Martin Kleppmann @martin.kleppmann.com · 3y ago

A thoughtful retrospective by @adamwiggins.bsky.social on the business behind Muse, a tool-for-thought/iPad idea-sketching app. I’m sad that the business didn’t work out as planned, but the software still exists and the underlying goals and values remain good. adamwiggins.com/muse-retrosp...

Muse retrospective · Adam Wiggins

The inside story of four years building Muse, a canvas-based thinking tool for iPad and Mac.

adamwiggins.com

Dan Luu @danluu.com · 3y ago

Why it's impossible to agree on what's allowed: danluu.com/impossible-a...

Plot showing that almost everybody disagrees with almost everybody else on a simple question (see link for further explanation).

Dan Luu @danluu.com · 3y ago

Notes on the accident where a Cruise self-driving car dragged a pedestrian under the vehicle for 20 feet after hitting them: danluu.com/cruise-report/

Alt text too long for field. Please see link.

Dan Luu @danluu.com · 3y ago

Why do people post on [bad platform] instead of [good platform]? danluu.com/why-video/

There's a class of comment you often see when someone makes a popular thread on Mastodon/Twitter/Threads/etc., that you also see on videos that's basically "Why would make a Twitter thread? This would be better as a blog post" or "Why make a video? This would be better as a blog post". But, these comments are often stronger in form, such as:

I can't read those tweets that span pages because the users puts 5 words in each reply. I find common internet completely stupid: Twitter, tiktok, Instagram, etc. What a huge waste of energy.

or

When someone chooses to blog on twitter you know it's facile at best, and more likely simply stupid (as in this case)

These kinds of comments are fairly common, e.g., I pulled up Foone's last 10 Twitter threads that scored 200 points or more on HN and 9 out of 10 had comments like this, complaining about the use of Twitter.

People often express bafflement that anyone could have a reason for using [bad platform], such as in "how many tweets a...

Dan Luu @danluu.com · 3y ago

How bad are search results and how do small, independent, search engines compare to Google and Bing? danluu.com/seo-spam/

Table of search result quality on test queries.

From best to worst, we have Marginalia, ChatGPT, Mwmbl, Kagi, Google, and then Bing.

Dan Luu @danluu.com · 3y ago

"The algorithm" has somehow decided that I want to see viral UX/designer content. In every viral UX comparison I've seen so far, the "good" version looks more modern but is also less legible / usable. E.g., below, the contrast is lower and legibility is sacrificed for to make things look clean.

Comparison where "good" version is low contrast and critical information is hard to read.

Dan Luu @danluu.com · 3y ago

Looking back on this 2015 post about how AI will displace humans, I feel like I got the line of reasoning correct, in that the same debate is happening today and this post pre-emptively refuted the incorrect reasons people are now giving for why LLMs aren't a big deal: danluu.com/customer-ser...

There's an ongoing debate over whether "AI" will ever be good enough to displace humans and, if so, when it will happen. In this debate, the optimists tend to focus on how much AI is improving and the pessimists point to all the ways AI isn't as good as an ideal human being. I think this misses one very important factor, which is that the most obvious jobs that are on the potential chopping block, such as first-line customer service, customer service for industries that are either low margin or don't care about the customer, etc., tend to be filled by apathetic humans in a poorly designed system, and humans aren't even very good at simple tasks they care a lot about. When we're apathetic, we're absolutely terrible; it's not going to take a nearly-omniscient sci-fi level AI to displace people in most customer service jobs.

For example, here's a not-too-atypical customer service interaction I had last week with a human who was significantly worse than a mediocre AI. I scheduled an ap...

Dan Luu @danluu.com · 3y ago

Logged on for the first time in months to get some invite cods for folks and I still don't really see much technical content here compared to Mastodon and Twitter. Is it still the case that there isn't critical mass for tech discussions here?

Dan Luu @danluu.com · 3y ago

The difference in the "What's Hot" feed on Bluesky right now vs. Mastodon really sums up why I've found it so much easier to find discussions I like on Mastodon. Bluesky is full of people crowing over their superiority over threads users. Mastodon has less Threads content but, more importantly

#Explore

7h
crowgirl@hachyderm.io
Kim Crawley @crowgirl@hachyderm.io

This is a warning about Meta and Threads.

Please boost this here and outside of Mastodon so that pregnant people in the United States are informed that using Meta/Threads is dangerous.

I'm unlikely to get pregnant but I have ZERO presence on Meta/Facebook/Instagram/Threads as a matter of principle.

#abortionrights #pregnant #threads

5h
protonmail
Proton Mail @protonmail

So #Meta's new #Threads app needs your health and fitness info. It also needs your browsing history and your location, and your purchases, and...well, it seems to need everything. If you want to get fully creeped out, here's the whole #privacy policy: https://privacycenter.instagram.com/policy/.

Dan Luu @danluu.com · 3y ago

how it started how it's going