Like ten years ago now I read this science news article about how machine learning used in a metallurgical lab produced a support structure for bridges that appeared far more durable while using less material. It looked like some nonsensical weird alien blob and no one could explain why it worked better for months.

That's the kind of stuff I think genAI could be cool for: a thing off in the corner of a lab or engineering department that produces "Huh I never thought of that" that experts begin researching from the conceptual stage. That's not how we're using it though, we're using it as a complete answer machine and that's resulting in people implementing these weird alien hallucinations with no critical thought.

To come full circle on the metallurgy thing: as I recall the structure was eventually discarded as ill fit for the purpose because while it had more crush force resistance, it had other weaknesses because the positive/negative conditions on the ask weren't dialed in correctly.

Course I can't find the article now because if you try to search for anything adjacent to genAI you get carted off into the land of whacking off over GPT.