Fake AI “podcasters” are reviewing my book and it’s freaking me out

As someone who has been following the growth of generative AI for a while now, I know that the technology can be pretty good (if not quite human-level) at quickly summarizing complex documents into a more digestible form. But I still wasn’t prepared for how disarmingly compelling it would be to listen to Google’s NotebookLM condense my recent book about Minesweeper into a tight, 12.5-minute, podcast-style conversation between two people who don’t exist.

There are still enough notable issues with NotebookLM’s audio output to prevent it from fully replacing professional podcasters any time soon. Even so, the podcast-like format is an incredibly engaging and endearing way to take in complex information and points to a much more personable future for generative AI than the dry back-and-forth of a text-based chatbot.

Hey! Listen!

Listen to NotebookLM’s 12.5-minute summary of my Minesweeper book using the player above.

Google’s NotebookLM launched over a year ago as “a virtual research assistant that can summarize facts, explain complex ideas, and brainstorm new connections—all based on the sources you select.” Just last week, though, Google added the new “Audio Overview” feature that it’s selling as “a new way to turn your documents into engaging audio discussions.”

Google doesn’t use the word “podcast” anywhere in that announcement, instead talking up audio creations that “summarize your material, make connections between topics, and banter back and forth.” But Wharton AI professor Ethan Mollick correctly referred to the style as a “podcast” in a recent social media post sharing a NotebookLM Audio Overview of his book. Mollick called these Audio Summaries “the current best ‘wow this is amazing & useful’ demo of AI” and “unnerving, too,” and we agree on both counts.

Inspired by Mollick’s post, I decided to feed my own book into NotebookLM to see what its virtual “podcasters” would make of 30,000 or so words about ’90s Windows gaming classic Minesweeper (believe it or not, I could have written much more). Just a few minutes later, I was experiencing a reasonable facsimile of what it would be like if I was featured on NPR’s Pop Culture Happy Hour or a similar banter-filled podcast.

Just the facts?

NotebookLM’s summary hits on all the book’s major sections: the pre-history of the games that inspired Minesweeper; the uphill battle for the Windows Entertainment Pack at a business-focused Microsoft of the ’90s; the moral panic over the game’s pre-installation on millions of business and government computers; and the surprising cheating controversies that surrounded the game’s competitive scene.

Sure, I could quibble about which specific bits the summary decided to focus on and/or leave out (maybe feeding different chapters individually would have led to more detail in the collected summaries). But anyone listening to this “podcast” would get the same general overview of my book that they would listening to one of the many actual podcasts that I did after the book launched.

While there weren’t any full-blown, whole-cloth hallucinations in NotebookLM’s summary “podcast,” there were a few points where it got small details wrong or made assumptions that weren’t supported in the text. Discussing Minesweeper predecessor Mined-Out, for instance, NotebookLM’s audio summary says, “So this is where those squares and flags start to come into play…” even though Mined-Out had neither feature.

Then there’s the portion where the summary-cast mentions a senator who called Minesweeper “a menace to the republic,” repeating the quote for emphasis. That definitely captures the spirit of Senator Lauch Faircloth’s tirade against Minesweeper and other games being pre-installed on government computers. In the “podcast” context, though, it sounds like the voices are putting words in Faircloth’s mouth by sharing a direct quote.

Small, overzealous errors like these—and a few key bits of the book left out of the podcast entirely—would give me pause if I were trying to use a NotebookLM summary as the basis for a scholarly article or piece of journalism. But I could see using a summary like this to get some quick Cliff’s Notes-style grounding on a thick tome I didn’t have the time or inclination to read fully. And, unlike poring through Cliff’s Notes, the pithy, podcast-style format would actually make for enjoyable background noise while out on a walk or running errands.