Computing,  Mathematics,  Philosophy,  Research

The Scientific Method – Does it Deliver Truth?

The below is a modified version of a response I wrote recently on Google+ in response to a question about the conflict between reason and faith. The response is also detailed in my recent book Uncommon Wisdom. This essay will argue that the manner in which science has construed the use of reason (and experience) – i.e., the path to discovery – cannot deliver truth. There is, however, another notion about reason which works in conjunction with faith to verify rather than discover the truth. Faith and reason are contradictory when reason is defined as the method of truth discovery. But they are not contradictory when reason is used for verifying the truth. In this case, it would mean that if you found the truth from someone and accepted it on faith to begin the verification, then you could use reason (and experience) to verify, and upon verification you would confirm your faith. If the proposal is false, you should be able to disprove it.

Discovery vs. Verification

In computing theory, there is a well-known distinction between finding a solution vs. verifying the solution. This is called the P=NP problem, where the contention is that finding the solution to a problem requires the same amount of computational complexity as verifying the solution. This is a contention because it has never been proven, and there are many cases where this equality would be disastrous.

For example, finding out your computer’s password is much harder (in case someone doesn’t know the password) than verifying the password (in case you know the password). If these two were the same, then the time taken to verify your password would be the same as the time taken to hack it, and there would be no password security!

If you gave your password to someone, and he or she accepted that password based on faith, they would be able to quickly verify the password (by logging into your computer using that password), even though they could not discover the password so easily all  by themselves. There is, hence, a fundamental difference between finding a solution and verifying it; both require reason, but it is not proven that they are identical. From all practical insights we have so far, it appears that they are not equivalent. Certainly, if this equivalence was proven, there would be far too many problems.

The distinction between discovery and verification is important because reason is used in both cases, although in quite different ways. In the case of verification, we use reason to confirm (or deny) whether a proposed solution actually solves a problem. In the case of discovery, we hypothesize different solutions based on reason, and then use reason for verification. The veridical aspect of reason plays an important role even in discovery, although it also requires a speculative effort to arrive at a proposal to be verified.

There are deeper issues about whether speculation (or creativity) is rational or involves something beyond just reason. It has, for instance, been argued over centuries that reason only helps us derive conclusions from premises, but to come up with a new premise there is no rational method. Empiricists now argue that new ideas are arrived at by the analysis of experience and there are infinitely many distinct ways in which we you can analyse experiences, providing a different explanation for the same experience.

How do we know which of these answers is correct? We have two criteria that both require reason, but only one of them is part of any rational formulation: consistency and completeness. Reason deals in consistency, but completeness is driven by experience: we have to find more and more experiences to check if our theory is complete.

How do we find all the experiences and check against them? Are we sure of our theory because no contradicting experiences have yet been found? If that were the case, even a false theory would be considered true temporarily. The scientific method postulates that the iteration of experiencing, analyzing, postulating and verifying comes to an end. It is not clear to anyone whether it actually does. In this essay, I will try to show why this iterative cycle does not and cannot come to an end in any finite space and time.

The Problem in Discovery

If you have to hack someone’s password, you can try random passwords one after another until you hit upon a correct password. This process will take a finite time only if you can store the list of failed passwords. If you forget the passwords that failed, and retry them, then the process is infinite (and the password can never be hacked).

Alternatively, you can find a method to generate all passwords (which is how passwords are hacked today), but in the case of a natural theory of reality, this means you already know how the universe was created, or could have been created. For instance, you might view the universe as a sequence of 1s and 0s, and use a machine to arrive at a description. This machine would have to be outside the universe, and the amount of data it generates has to be held into some memory store that is far bigger than the universe.

The issue here is that for any sufficiently complex problem, there are many proposed solutions that will likely fail. Each problem – if it is consistent and complete – will have only one solution, although infinitely many wrong solutions are imminently possible. You would have to be incredibly lucky to find the right one quickly, given that there are so many of them. If, however, you are not relying on luck but on reason, then you must have the ability to remember (store) all the failed attempts and carefully avoid them.

Two kinds of problems arise from the above requirement.

First, if you are trying to find the truth about anything, then the number of possible wrong answers are so many that they cannot be stored in the same amount of space that is occupied by that thing. For instance, if you try to describe your computer, there are many incorrect descriptions you can come up with. The sum total of all these incorrect descriptions requires much more space than the computer itself. Colloquially, to know the truth about yourself, you need something bigger than yourself to store all the false answers. To know the truth about the universe, you need something bigger than the universe to store all the false notions about the nature of the universe.

Note that unless you are extremely lucky, you will arrive at the wrong answers initially. As you go after the truth, you have to make sure that you don’t reuse the previous failed answers. To achieve that, you must store all the failed answers. If you are trying to find the truth of anything, you need much more space than that object itself. In theory, therefore, it is impossible to find out the truth of everything because you run out of space to store all the failed proposals. In a simple sense, you must forget that it failed in the past, and retry it over and over – like retrying failed passwords – which makes the problem unsolvable.

Second, even if we assume that we have infinite space available to store all the failed proposals, the problem is still unsolvable due to the time it takes as we iterate over the proposals. This is because, every time you propose a new solution, you must run through the list of proposals that have previously failed, and carefully avoid them. As the failed list becomes longer, it takes longer to run through the failed attempts to even propose a new attempt. In other words, as you fail more, the time it takes to arrive at a new proposal (even a false one), increases. The more you fail, the slower you must fail.

When this is taken to the logical conclusion, the method that relies on reason to solve a problem by speculation becomes infinitely long if the problem is sufficiently hard. The only way out is if you get lucky. For example, you might find the solution in the first attempt or the first few attempts, if you are really lucky. Given that the list of possible failures is enormous, the chances of you getting lucky are indeed very, very small.

The attempt at truth discovery therefore has two kinds of constraints – space and time. You need much bigger space to store the past failed attempts to avoid them in future. As you try newer attempts, each attempt goes slower than before, thereby making this an infinitely long exercise. You cannot use reason to solve this problem. You can only bet on getting lucky for which the chances are minuscule (if the problem is indeed hard).

Can We Discover The Truth?

All inquiry begins in the fundamental question: Can we know the truth? Science converts this into: Can we discover the truth? Knowing and discovering are not the same; e.g., asking someone the password to the computer is not the same as trying to hack it on your own. However, science asserts that we cannot ask anyone, because no one knows. We have to find out the truth using reason and experience. There are profound reasons why reason and experience – when used for discovery – can never yield the truth, and the reason is that we don’t have enough place to keep keep all the wrong answers and not enough time to run through all the wrong answers to eliminate them one by one.

To discover the truth, we must get really, really lucky.

There is, however, another method to know the truth by asking someone who knows. This method works on faith, but not in contradiction to reason and experience: you discover by faith and you verify by reason and experience. Of course, many people today argue against faith claiming that their method of reason and experience will give the results. They only need to look at what it takes to know the truth of anything before asserting this.

The contention is therefore not between reason and faith. It is between faith and luck. The person who is betting on finding the solution through reason and experience is betting on getting really lucky. Conversely, the person who is relying on faith to ask someone the truth is betting that he or she will find the person who knows the truth. You might think that the faithful are lazy, because they don’t make the attempt to discover the truth by themselves, but you need to consider that the method of discovery is futile.

The proponents of reason and experience must recognize the fact that even reason and experience must operate within space and time, and there isn’t enough of it to go around if all the problems have to be solved. If we start solving all the problems using speculation and arrive at false solutions, pretty soon the world will be filled up of matter that mostly represents false ideas. You can’t throw away this false matter because then your method of discovery will produce the same false things over and over again, wasting too much time. If you keep all the false things in place, then you have to scan through all these false ideas before can propose anything new (assuming you are rational), wasting time in a process that can never come to an end, and becomes slower over time.

The conflict between reason and faith is misconstrued and hinges upon the assumption that (a) reason will give the solution in finite time, and (b) reason is only used for discovery. When reason is used for verification, there is no conflict between reason and faith. Additionally, when reason is used for discovery, you need to get lucky.

Summarizing

I can succinctly summarize this essay by saying the following. The scientific method can deliver the truth if P=NP. That proof would cause many problems (such as giving everyone transparent access to all your private information). In other words, if P=NP, then not only we will know the nature of reality, we will also not have any sense of privacy. If everything can be known by reason and experience, then we are open books that can be read by anyone. There is no mental subjectivity or personal privacy. The time it takes for me to know my mental state would be the time it takes you to know the same. This is not to start ringing alarm bells about what science can and cannot do, because I don’t believe P=NP. However, that latter conclusion entails the use of faith in conjunction to reason and experience, because otherwise there isn’t enough space-time to know.