I know a law firm who creates garbage like this without even using AI

Should have judged the situation better

Creating text is not the only issue, it may be trained from your confidential files.

That's just called common law

Another awesome Regan appendage

This made me laugh. Not wrong though!

from a federal judge in Mississippi

I SO want to love that state, but fuck me, this comes at zero surprise.

Also no, you don't know what it's doing so you could be blindsighted by the latest AI update making unexpected changes. Not only from good-intentioned features but also bugs, or malicious anti-features after the CEO throws their toys out of the Twitter pram.

no malicious update can force you to generate a text and file it in court as your own work.

Consider that a program can edit the file while running at any point, not merely during user input. Like a virus with access to user's files it could even edit a document that's not even being displayed to the user on the screen.

well that would be fucked up for sure. are you suggesting any existing program works like that, or are just speculating what if?

This may be out of date but in this video by Lawful Masses lawyers are concerned that software AI tools which somehow (I don't recall) help them understand a case. This issue is the AI should not use information sourced from another client's confidential case/documents to inform them about another case but they don't know how it works. Responses from Microsoft were not forthcoming.

I would argue they can't know unless they have access to the source code to verify what any (local) AI can do (not personally do it, but a trusted 3rd party audit which isn't behind closed doors).