FRESH

Hacker News

Home

New York could prohibit chatbot medical, legal, engineering advice

76 points by bluepeter

by paxys

13 subcomments

"Hey ChatGPT, my NYC landlord is raising my rent by $500, and says I must pay by Monday or leave. What do I do?"
ChatGPT - This is very likely illegal under Housing Stability and Tenant Protection Act of 2019 (HSTPA), specifically New York Real Property Law § 226-c (Notice required for rent increases), RPL § 232-a / § 232-b (Month-to-month termination), RPL § 232-c (Fixed-term lease protections), RPAPL § 711 (Legal eviction procedure) and NYC Admin Code § 26-501+ (Rent stabilization). Here's what you should reply with... And here are some city resources you can contact...
ChatGPT now - IDK, pay a lawyer.
So under the guise of "protection" you are taking away the strongest knowledge tool common people have had at their disposal in a generation, probably ever.

by Esophagus4

2 subcomments

The disclosure requirement is probably a decent thing (you have no idea how many people come into the ER and say, “But ChatGPT told me to do [dumb thing].”) But preventing it from answering at all is absurd.
Make responsible disclosure absolve AI providers of legal responsibility (not legal advice lol).
That way if users ever sue OpenAI for giving them bad advice, OpenAI can say “no way man, you read the disclosure!”
I’m usually in favor of giving people the best info they can and letting them make their own decisions.
This could just be like those terms of service things everyone clicks “agree” to and I’d be fine with that.

by bluepeter

2 subcomments

Whats at least somewhat humorous is the disclaimer requirement that "[t]he text of the notice shall [be] no smaller than the largest font size of other text appearing on the website on which the chatbot is utilized."
H1 hero font size here we come for disclaimers! (Which don't do anything, per the bill, anyway.) But also is the fancy thought that chatbots only appear on websites.

by phishin

3 subcomments

Lawyers protecting lawyers. The one thing AI could help the ordinary people fight back against corporations.

by cm2012

2 subcomments

Bad law. I have gotten better advice from modern llms than from most of the professional categories above.

by aetherspawn

0 subcomment

Great, electrical and mechanical engineers are already underpaid, under appreciated and overworked.
I’ve always found it amusing that lawyers and accountants flash their license around with pride, put it in their email signatures, etc. and it provides authority for them. When people see chartered lawyer or accountant, they respect that person and take their advice.
An engineering license, on the other hand, is so rarely talked about and never quoted in email signatures and the like. And even as a chartered engineer, people really just treat you like a mechanic or a trade and mostly ignore your advice anyway. Yet, it takes the longest to get, and has the most exams/hardest subjects, except for Doctors.
Anything to make an Engineering license worth more is good in my books. Besides, in my experience ChatGPT gives wrong advice for engineering around 50% of the time and therefore probably has no business giving it.

by iamnothere

0 subcomment

Download one of the freely available models and use that, if you have the hardware for it. It’s not a good idea to ask sensitive questions on these nontransparent chatbot platforms.
(FWIW I also think this is a bad law. Why not improve privacy protections instead? Why not allow nonprofessional use with a disclaimer?)

by tzs

0 subcomment

The only penalty for violating this law is that if someone is injured by your chatbot giving bad advice they can sue you for the actual damages they suffered, and if your violation of the law was willful they can also sue for attorney fees.
This only applies to advice that would have illegal for a human to give who is not licensed in the relevant field.

by francisofascii

0 subcomment

The other professions are creating lawful protections for themselves in the upcoming AI revolution. Software engineers have no such protection.

0 subcomment

by sayhisam1

0 subcomment

More slopulist policy.
Why do we care about saving long-tailed distribution of idiots from themselves at the expense of everyone else? And is this even a real demonstrable issue (in terms of percent of harmful responses to total number of responses)

0 subcomment

by gdevenyi

0 subcomment

The medieval guilds will return.

by cybercatgurrl

0 subcomment

they see the writing on the wall and know their time is coming to an end eventually. this feels like the dying last breath of a rigid system unwilling to accept change and acknowledge that the power is now with the people

by arionhardison

2 subcomments

I don't understand how anyone can rationalize this bill in the face of what OpenAI just agreed to with the DoD.
AI can surveil and direct munitions but it cant answer legal questions. Wouldn't this also violate the "no state my limit or restrict the use of AI" that the current administration is pushing?

by thrill

0 subcomment

“Hey Chat, I feel pretty bad, here’s my symptoms. It’s expensive to go to ER, but should I see a doctor?”
Can’t advise you buddy, but here’s some OTC meds that have paid for placement. Been nice knowing you and good luck!

by OutOfHere

1 subcomments

New York residents who opposite this bill can go to https://www.nysenate.gov/legislation/bills/2025/S7263 , register and sign in, click Nay on the bill's page, and submit feedback for their vote.
---
ChatGPT> Before I answer your question, which state are you a resident of?
Human> Not New York. Continue!
ChatGPT> Alrighty then! Here you go...

by htrp

0 subcomment

loopholes wide enough to drive a truck through

by tim-tday

0 subcomment

Now do target acquisition for lethal munitions.

by josefritzishere

0 subcomment

New York really should. AI is literally killing people. This is not OK.

by moomoo11

3 subcomments

[flagged]

by fwip

7 subcomments

Seems like a good bill, at least directionally. If it's a crime to provide advice of this nature without a license, then chat bots shouldn't be dispensing it either.

by mbgerring

1 subcomments

Yes, that’s correct, I do not want a vibe-coded freeway overpass, thanks.
We all need to get serious about the unavoidable, unsolvable fact that these tools produce output of unknowable accuracy. Some things require such accuracy, precision, and, importantly, accountability. LLMs are capable of none of these things. Refusing to be honest about this and take appropriate precautions will lead to disaster.