As noted in an excellent article about AI and students,
Why is AI literacy important?
While modern generative AI systems can write out a logical sentence that flows from one point to the next, remember that they don't actually think. Rather than existing as an intelligent being who can think, reason, and make informed decisions with no human involvement, modern AI currently uses a generational model; it simply generates the next word in a sentence to continue the flow of the thought based on the data it has been trained on.
If it generates an incorrect word or statement, it rarely corrects itself. Instead, it goes on to either justify its own claims with made-up facts or lie entirely about things that simply do not exist. This can lead to falsified information being spread using confident language, and if a person is not familiar with AI generative text, this information can be quite convincing.
Generative AI can't tell fact from fiction
Consider the case of Mata v. Avianca, where a legal team used ChatGPT to find, source and quote relevant legal cases. Rather than convey that the requested information was not available or provide a reasonable alternative, the bot created fake examples; names, dates and the cases themselves were completely falsified.
The legal team did not verify the existence of said cases, creating significant problems in the courtroom. The error was simple but severe; the lawyers involved believed that ChatGPT was an advanced search engine that could provide real-time research results, rather than a computer program that used auto-fill to create coherent sentences without real information.
Pressreader Team. (2024, June 3). Building AI literacy: Equipping libraries for the technological shift. https://blog.pressreader.com/libraries-institutions/building-ai-literacy-equipping-libraries-for-the-technological-shift
Remember: AI is a tool, and like any tool, it is not perfect. It has things it does well and things it does poorly. Always check the information generated by AI.
