FRESH

Hacker News

Home

Jeff Dean responds to EDA industry about AlphaChip

248 points by nsoonhui

by vighneshiyer

2 subcomments

I have published an addendum to an article I wrote about AlphaChip (https://vighneshiyer.com/misc/ml-for-placement/) at the very bottom that addresses this rebuttal from Google and the AlphaChip algorithm in general.
In short, I think the Nature authors have made some reasonable criticisms regarding the training methodology employed by the ISPD authors, but the extreme compute cost and runtime of AlphaChip still makes it non-competitive with commercial autofloorplanners and AutoDMP. Regardless, I think the ISPD authors owe the Nature authors an even more rigorous study that addresses all their criticisms. Even if they just try to evaluate the pre-trained checkpoint that Google published, that would be a useful piece of data to add to the debate.

by Keyframe

2 subcomments

At this point in time, why wouldn't we give at least benefit of the doubt to Jeff Dean immediately? His track record is second to none, and he's still going strong. Has something happened that cast a shadow on him? Sometimes it is the messenger that brings in the weight.

by _cs2017_

5 subcomments

Curious why there's so much emotion and unpleasantness in this dispute? How did it evolve from the boring academic argument about benchmarks, significance, etc to a battle of personal attacks?

by oesa

3 subcomments

In the tweet Jeff Dean says that Cheng at al. failed to follow the steps required to replicate the work of the Google researchers.
Specifically:
> In particular the authors did no pre-training (despite pre-training being mentioned 37 times in our Nature article), robbing our learning-based method of its ability to learn from other chip designs
But in the Circuit Training Google repo[1] they specifically say:
> Our results training from scratch are comparable or better than the reported results in the paper (on page 22) which used fine-tuning from a pre-trained model.
I may be misunderstanding something here, but which one is it? Did they mess up when they did not pre-train or they followed the "steps" described in the original repo and tried to get a fair reproduction?
Also, the UCSD group had to reverse-engineer several steps to reproduce the results so it seems like the paper's results weren't reproducible by themselves.
[1]: https://github.com/google-research/circuit_training/blob/mai...

by dang

0 subcomment

Related. Others?
AI Alone Isn't Ready for Chip Design - https://news.ycombinator.com/item?id=42207373 - Nov 2024 (2 comments)
That Chip Has Sailed: Critique of Unfounded Skepticism Around AI for Chip Design - https://news.ycombinator.com/item?id=42172967 - Nov 2024 (9 comments)
Reevaluating Google's Reinforcement Learning for IC Macro Placement (AlphaChip) - https://news.ycombinator.com/item?id=42042046 - Nov 2024 (1 comment)
How AlphaChip transformed computer chip design - https://news.ycombinator.com/item?id=41672110 - Sept 2024 (194 comments)
Tension Inside Google over a Fired AI Researcher’s Conduct - https://news.ycombinator.com/item?id=31576301 - May 2022 (23 comments)
Google is using AI to design chips that will accelerate AI - https://news.ycombinator.com/item?id=22717983 - March 2020 (1 comment)

by nsoonhui

0 subcomment

The context of Jeff Dean's response:
https://news.ycombinator.com/item?id=41673769
https://news.ycombinator.com/item?id=41673808

by segmondy

3 subcomments

It's ridiculous how expensive the wrong hire can be https://www.wired.com/story/google-brain-ai-researcher-fired...

by twothreeone

8 subcomments

I get why Jeff would be pressed to comment on this, given he's credited on basically all of "Google Brain" research output. But saying "they couldn't replicate it because they're idiots, therefore it's replicable" is not a rebuttal, just bullying. Sounds like the critics struck a nerve and there's no good way for him to refute the replication problem his research apparently exhibits.

by 7e

0 subcomment

Google's ultimate proof is to build better chips than the competition. Which, at this point, they cannot.

by LittleTimothy

1 subcomments

"You didn't use enough compute in your reproduction of our methods" is kind of a funny criticism today. Well yeah, you're Google. Sorry guys, no reviewing of our methodology unless you own a cloud. It wouldn't surprise me if it's true that more compute, more pre-training etc. provides a lot of utility, but that does make it difficult to verify the work.
One interesting aspect of this though is vice-versa, whilst Google has oodles of compute, Synopsys has oodles of data to train on (if, and this is a massive if, they can get away with training on customer IP).

by rowanG077

2 subcomments

I don't get. Why isn't the model open if it works? If it isn't this is just a fart in the wind. If it is the findings should be straightforward to replicate.

by gabegobblegoldi

1 subcomments

Additional context: Jeff Dean has been accused of fraud and misconduct in AlphaChip.
https://regmedia.co.uk/2023/03/26/satrajit_vs_google.pdf

by wholehog

1 subcomments

The paper: https://arxiv.org/abs/2411.10053

by AtlasBarfed

4 subcomments

How the hell would you verify an AI-generated silicon design?
Like, for a CPU, you want to be sure it behaves properly for the given inputs. Anyone remember that floating point error in, was it Pentium IIs or Pentium IIIs?
I mean, I guess if the chip is designed for AI, and AIs are inherently nonguaranteed output/responses, then the AI chip design being nonguaranteed isn't any difference in nonguarantees.
Unless it is...

by puff_pastry

1 subcomments

The biggest disappointment is that these discussions are still happening on Twitter/X. Leave that platform already

by bsder

7 subcomments

The fact that the EDA companies are garbage in no way mitigates the fact that Google continues to peddle unsubstantiated snake oil.
This is easy to debunk from the Google side: release a tool. If you don't want to release a tool, then it's unsubstantiated and you don't get to publish. Simple.
That having been said:
1) None of these "AI" tools have yet demonstrated the ability to classify "This is datapath", "This is array logic", "This is random logic". This is the BIG win. And it won't just be a couple of percentage points in area or a couple of days saved when it works--it will be 25%+ in area and months in time.
2) Saving a couple of percentage points in random logic isn't impressive. If I have the compute power to run EDA tools with a couple of different random seeds, at least one run will likely be a couple percentage points better.
3) I really don't understand why they don't do stuff on analog/RF. The patterns are smaller and much better matches to the kind of reinforcement learning that current "AI" is suited for.
I put this snake oil in the same category as "financial advice"--if it worked, they wouldn't be sharing it and would simply be printing money by taking advantage of it.

by lumb63

2 subcomments

I’ve not followed this story at all, and have no idea what is true or not, but generally when people use a boatload of adjectives which serve no purpose but to skew opinion, I assume they are not being honest. Using certain words to describe a situation does not make the situation what the author is saying, and if it is as they say, then the actual content should speak for itself.
For instance:
> Much of this unfounded skepticism is driven by a deeply flawed non-peer-reviewed publication by Cheng et al. that claimed to replicate our approach but failed to follow our methodology in major ways. In particular the authors did no pre-training (despite pre-training being mentioned 37 times in our Nature article),
This could easily be written more succinctly, and with less bias, as:
> Much of this skepticism is driven by a publication by Cheng et al. that claimed to replicate our approach but failed to follow our methodology in major ways. In particular the authors did no pre-training,
Calling the skepticism unfounded or deeply flawed does not make it so, and pointing out that a particular publication is not peer reviewed does not make its contents false. The authors would be better served by maintaining a more neutral tone rather than coming off accusatory and heavily biased.