AI Can't Answer Your Biblical Questions

I have written extensively about the use of computer science and AI techniques in Biblical exegesis, interpretation, and authorship attribution in my academic (MDPI) (Under-Review) and informal work (Blog). Today I would like to write a short post for a non-technical audience explaining in simple terms why AI cannot answer our important questions about the Bible. The overarching reason is that AI systems must be evaluated based on downstream tasks, and there are no such definitive tasks in Biblical exegesis.

Applied AI is, at root, complex mathematical operations performed on a computer. When you create a model to measure the similarity of language or train an LLM, you are creating a system of mathematical relationships. The distinction to keep in mind is that this system of mathematical relationships is sepatate from the utility of those mathematical relationships. This is to say that while we can speak objectively about the mathematical relationships in/from the model, we cannot in isolation speak objectively about its utility. To do that, we must reference a real-world task for evaluation.

A simple example to make this more understandable is cancer image screening. Once you have an AI model, you have no idea how good it is at making predictions just by studying the mathematical properties of the model. You have to test it against a set of known images that are cancer-free and cancerous. This is the downstream task to evaluate the model, and it is rooted in the fact that we are able to identify the test set images correctly. Essentially, there is a definitive verification method to know whether the model’s outputs are correct (probabilistically).

Now, think of the question of whether Paul truly wrote Titus. It is easy to design a model under certain assumptions (i.e., 1 and 2 Timothy are forgeries) that declares Titus to be a forgery. However, if you simply consider 1 and 2 Timothy as authentic, then the same model considers Titus to be authentic! How can we evaluate these conflicting outputs?

In the case of cancer screening, there is an objective verifier: the image is either cancerous or not and thus the model’s prediction is either correct or not. But, in the case of Paul’s letters, the whole point is that there is no way to objectively know which letters are authentic, and thus there is no way to evaluate the outputs. The model is completely left to the creator and interpreters’ prior assumptions. If they already believed 1 and 2 Timothy and Titus were forgeries, then the model proves it! If they believed the inverse, then the model proves it! The model’s usefulness is contingent upon the interpreters beliefs, and thus adds no explanatory value to exegetical questions.

There are many other reasons AI is not useful for Biblical questions, such as complications from scribal use, textual corruption, and small corpus size. The fundamental problem though is that there is no way to objectively evaluate the model. So, the next time you hear someone talk about how this AI method proves X about the Bible, ask them “How could you possible know that’s correct?”