As with so many parts of modern life, artificial intelligence (AI) is disrupting professions that rely on the written word. Some people think it’s amazing. Some think it’s overhyped. And some think it’s a dangerous job killer that will reduce humans’ ability to think and communicate for themselves.

I’m not sure where I stand yet, but I was wondering the other day what AI itself thinks about using AI for professional writing, editing and research. Like any good journalist, I reached out to multiple sources for comment. Here’s what I learned from the major AI platforms I consulted that use large language models (LLMs).

ChatGPT and Perplexity kind of hemmed and hawed. Eventually I got tired of waiting for them to formulate a response while “still thinking and processing.” Gemini, sort of hid behind the Google Chrome search results. But Claude to its credit answered my questions swiftly and confidently, but not before telling me: “Hank, the irony of your request has not been lost on me.”

According to Claude: “AI writing tools have swept into newsrooms, law firms, academic institutions, and corporate communications departments with remarkable speed. The pitch is seductive: faster drafts, fewer bottlenecks, lower costs.” No surprise there, but then Claude confided: “For professional writing — where accuracy, credibility, and sourcing are non-negotiable — AI tools introduce a set of risks that should give any serious writer or editor pause.”

Hmmm.

Unprompted by me, Claude then addressed the tendency of AI to come up with make-believe-sources.

The Hallucination Problem

“The most damaging flaw of LLMs in professional writing contexts is what researchers and engineers call ‘hallucination,’” asserted Claude. “It’s the tendency of AI systems to generate false information with complete confidence. This isn’t a bug that will be patched in the next update. It is, to a significant degree, structural,” said Claude.

If you’re in a situation or profession that expects you to document the sources for your work, which should give you pause.

As Claude explained during our interview, “language models work by predicting statistically likely sequences of text based on training data. They do not ‘know’ facts the way a database stores records. When asked to produce a citation — a journal article, a court case, a news report — the model generates text that looks like a citation. It follows the correct format, includes plausible author names, and references a real-sounding journal. But the underlying article may not exist at all,” Claude noted.

Personally, I’ve run into the source fabrication problem often and so have many of our clients.

When I asked Claude for an example of AI hallucination, it shared the following story: “This phenomenon gained widespread public attention in 2023 when a New York attorney, Steven Schwartz, submitted a legal brief that cited multiple court cases generated by ChatGPT — none of which were real. The judge was not amused,” noted Claude. “Schwartz faced sanctions, and the case became a widely cited cautionary tale about the dangers of using generative AI in professional legal contexts,” Claude added.

Why Citations Are Especially Vulnerable

According to Claude, “citation requires something AI fundamentally lacks: a reliable, verifiable link between a claim and a specific, retrievable source.” A human researcher, Claude said, “finds a source, reads it, and records it. The citation is downstream of an actual act of retrieval and comprehension. By contrast, AI generates a citation because citations are a pattern that appears in its training data. [AI] has learned that academic sentences tend to be followed by parenthetical author-year references, that legal briefs cite specific case names and dockets, and that journalism includes attributed quotes. So, it produces those patterns — whether or not the underlying source exists.”

To me, it’s amazing the human programmers have figured out a way to mimic academic, legal and journalistic writing style, but you can see how dangerous it can be when mis-used.

As Claude continued: “Even worse Hank, when AI tools do have access to real sources (through retrieval-augmented generation or web search), they can still misrepresent them — paraphrasing inaccurately, attributing quotes to the wrong person, or presenting a source’s argument as the opposite of what it actually says.”

Personally, I’ve frequently been frustrated by inaccurate paraphrasing and misattribution of quotes, and it’s something you should double check before you go live with any material AI has generated for you.

The Problem of Confident Vagueness

If you’re still reading this post, or asking your AI assistant to do so, I thank you for your time.

As Claude noted: “Professional writing demands precision. A medical writer cannot say a drug ‘significantly reduces symptoms’ without citing a specific study with a specific patient population and outcome measure,” Claude said. “An investigative journalist cannot attribute a claim to ‘experts’ without naming them. AI prose tends toward what might be called confident vagueness — it sounds authoritative while eliding the specific details that give professional writing its credibility and accountability,” Claude noted.

Agreed. We’re constantly calling out our clients on that.

“This is partly why AI-generated text often passes a casual read but falls apart under editorial scrutiny,” admitted Claude. “The sentences are grammatical, the tone is appropriate, the structure is sound — but when a fact-checker goes to verify the third paragraph, the sources either don’t exist or don’t say what the text claims.”

In my experience, this is sadly all too accurate.

Accountability

“Beyond sourcing,” continued Claude, “professional writing carries the voice, judgment, and accountability of a named author. A byline is not merely a formality — it represents a human being who can be questioned, who has a reputation at stake, and who made deliberate choices about what to include and what to leave out. AI-generated content erodes that accountability layer entirely,” asserted Claude.

“There is also the question of what gets left out. AI models trained on existing text will reproduce the emphases, omissions, and blind spots of that text,” said Claude. “A professional writer working a beat develops the judgment to know what the data doesn’t show, who hasn’t been heard from, and what a source has an interest in saying. AI has no such judgment — it optimizes for plausibility, not truth,” stated Claude.

A Tool, But Not a Replacement for Professional Judgment

“None of this means AI has no place in a writer’s workflow,” asserted Claude. “Brainstorming, structural outlining, rough drafting, and editing assistance are areas where the stakes of a fabricated citation are low and the efficiency gains are real. But for any writing where sourcing, accuracy, and accountability matter — journalism, legal writing, academic research, medical communication — AI should be treated as a starting point at best, and a liability at worst.”

Even Claude concedes: “The professional writer’s value lies precisely in what AI cannot reliably do: find real sources, read them carefully, represent them accurately, and stake their name on the result.”

More tools and resources related to this post can be found on our website.

Conclusion

AI is great for getting started, especially when you’re just staring at a blank screen and can’t get out of first gear. It can be a great help when it comes to brainstorming, structural outlining, rough drafting, and editing assistance. But turning to your work over to AI blindly is not just lazy; it can irreparably damage your reputation and your firm’s.