Very good read—I do think the relevant question is whether automating AI improvement is easier than automating jobs.
Because it does seem automating all jobs is hard (for all the reasons you point out). And so unless automating AI improvement is easier, longer timelines seem reasonable.
"Perhaps AI R&D agents will be very sample efficient, but perhaps this sample efficiency won’t generalize to tasks outside AI R&D itself—kinda like how a professional dancer can perfectly replicate a dance move after one viewing, but wouldn’t be able to do the same with a violin piece."
This bit got me thinking that people don't appreciate how deeply Moravec's paradox might run, even into "knowledge work" where it's easy for humans to understand the deep context of their work environments and what counts as an important consideration or dumb decision, that AIs might often miss (or at least lack the requisite 9s of reliability across many, multi-step tasks).
"It’s easy to underestimate how important [a reliable memory] is. Consider—most large-scale industrial operations are critically dependent on quality management frameworks that emphasize the painstaking prevention and rapid correction of production errors. For example, Six Sigma processes aim to achieve 3.4 defects per million opportunities, or a 99.99966% defect-free rate. Obviously, this exact rate is sometimes not realistically achievable, and there’s something to be said about overinvesting in perfection. However, the general mindset makes total sense. If you’re executing on a highly complex process with hundreds, if not thousands of steps, then even seemingly low levels of error at each step can compound into huge costs. Twenty steps * 99% success rate = 20% of your finished products are bunk."
Very good read—I do think the relevant question is whether automating AI improvement is easier than automating jobs.
Because it does seem automating all jobs is hard (for all the reasons you point out). And so unless automating AI improvement is easier, longer timelines seem reasonable.
Great post, Michelle!
"Perhaps AI R&D agents will be very sample efficient, but perhaps this sample efficiency won’t generalize to tasks outside AI R&D itself—kinda like how a professional dancer can perfectly replicate a dance move after one viewing, but wouldn’t be able to do the same with a violin piece."
This bit got me thinking that people don't appreciate how deeply Moravec's paradox might run, even into "knowledge work" where it's easy for humans to understand the deep context of their work environments and what counts as an important consideration or dumb decision, that AIs might often miss (or at least lack the requisite 9s of reliability across many, multi-step tasks).
Yep. From an old draft:
"It’s easy to underestimate how important [a reliable memory] is. Consider—most large-scale industrial operations are critically dependent on quality management frameworks that emphasize the painstaking prevention and rapid correction of production errors. For example, Six Sigma processes aim to achieve 3.4 defects per million opportunities, or a 99.99966% defect-free rate. Obviously, this exact rate is sometimes not realistically achievable, and there’s something to be said about overinvesting in perfection. However, the general mindset makes total sense. If you’re executing on a highly complex process with hundreds, if not thousands of steps, then even seemingly low levels of error at each step can compound into huge costs. Twenty steps * 99% success rate = 20% of your finished products are bunk."
(The rest is a mess, but the beginning bits & excerpted post touch on the memory aspect of context if you're curious: https://docs.google.com/document/d/1JdcLSlZOXc3B3YDOOFqc19hfC0q_tq8BVMN_HJ5L5XI/edit?usp=sharing)