This week, we look into a big issue in academic publishing that highlights the darker side of generative AI: paper mills.
Long before generative AI was even a thing, paper mills profited from intentionally fabricate and sell fraudulent research papers. With the advent of large language models (LLMs) like ChatGPT and many other open source models, the problem has now reached scale, making it easier for bad actors to generate convincingly fake content at scale. Let’s look into a case study.
Focus On: Lessons Learned From Wiley’s Experience
Generative AI has democratised content creation, providing unprecedented opportunities for generating text, images, and even complex research papers. However, this technology’s neutrality makes it very dangerous in the hands of those with bad intents. LLMs are being leveraged by paper mills to fabricate data, generate fake manuscripts, and manipulate images, thereby amplifying the quantity and perceived legitimacy of fraudulent papers.
A recent example of the intersection of paper mills and generative AI involves Wiley, a renowned US publishing house, and its subsidiary Hindawi. Acquired in January 2021, Hindawi quickly became the center of a scandal that changed the entire academic publishing industry. By December 2023, Wiley had to face the staggering reality that multiple journals overseen by Hindawi had been deeply infiltrated by paper mills. These organizations had unleashed a torrent of fraudulent papers, many of which leveraged AI to fake data and manipulate research results.
Ultimately, Wiley retracted over 11,300 papers from its Hindawi portfolio and eventually shut down 19 journals to protect integrity of their publishing operations. The financial repercussions were swift and severe. In its fiscal Q2 2024 earnings report, Wiley noted an $18 million decline in research publishing revenue directly attributed to the Hindawi write off, showcasing just how costly such breaches can be.
Following this shocking experience, Wiley launched an AI-powered Papermill Detection service, designed to counteract future fraudulent publications. This new service incorporates six distinct tools aimed at identifying AI-generated content, problematic phrases, and irregular publishing patterns.
The Implications for Corporates
While one could be tempted to relegate Wiley’s experience to the world of academic publishing, the implications are far broader. The core issues raised by paper mills—content authenticity and integrity—raise questions that all business leaders must address.
Firstly, the globalisation of information and the rapid dissemination facilitated by digital platforms mean that any entity, academic or corporate, is at risk of reputational damage from fraudulent content. Imagine a scenario where a corporation unknowingly cites fraudulent data in its annual report or research paper. The blow to its credibility could be devastating, resulting in a loss of stakeholder trust, market value, and competitive edge.
Moreover, the sophisticated nature of AI-generated content makes it increasingly difficult to differentiate between genuine and fake outputs. Traditional methods of verification may fall short, calling for the adoption of advanced screening tools. Businesses must evolve to incorporate AI tools capable of detecting and mitigating misinformation, ensuring that their internal and external communications remain reliable and trustworthy.
Ethical AI usage emerges as another crucial consideration. As Wiley’s new initiatives demonstrate, guidelines and training programs focused on the responsible use of AI can make a significant difference. In the corporate context, this translates to creating a robust framework that ensures AI tools enhance, rather than compromise, the quality and integrity of outputs. Companies must also remain vigilant of industry trends, like the “death of the black box AI” one we touched on last week.
Transparency and accountability in content creation processes are becoming a must. In an era where trust is a valuable currency, maintaining transparency about AI usage policies and content verification processes can fortify an organization’s reputation. This can be achieved through regular audits, open communication with stakeholders, and adherence to ethical standards. In other words, don’t be afraid to tell your audience your content is partially edited with AI (just like this very newsletter!).
What’s Next? Some actionable insights
1. Implement Robust Screening Tools: Just as Wiley has developed AI tools to detect fraudulent content, businesses should invest in technologies capable of identifying and mitigating misinformation. This includes leveraging AI for content verification and implementing comprehensive internal review processes.
2. Foster Ethical AI Usage: Promoting the ethical use of AI within your organization is crucial. Establish clear guidelines and training programs to ensure that AI tools enhance, rather than compromise, the quality and integrity of your outputs.
3. Stay Vigilant of Industry Trends: Keeping abreast of advancements and recognizing the potential for misuse is essential. An informed leadership team can anticipate threats and implement preventative measures more effectively.
4. Collaborate with Industry Peers: Engage with industry bodies and participate in collective efforts to address AI-enabled threats. Shared knowledge and resources can bolster your defences against sophisticated fraudulent practices.
5. Prioritize Transparency and Accountability: In an era where trust is paramount, maintaining transparency about your content creation processes and AI usage policies can fortify your organization’s reputation and build stakeholder confidence.
Follow me
That’s all for this week. To keep up with the latest in generative AI and its relevance to your digital transformation programs, follow me on LinkedIn or subscribe to this newsletter.
Disclaimer: The views and opinions expressed in Chronicles of Change and on my social media accounts are my own and do not necessarily reflect the official policy or position of S&P Global.