(31th-August-2020)
• Probability forms the basis of information theory. In this section, we give a brief overview of information theory.
• A bit is a binary digit. Because a bit has two possible values, it can be used to distinguish two items. Often the two values are written as 0 and 1, but they can be any two different values.
• Two bits can distinguish four items, each associated with either 00, 01, 10, or 11. Similarly, three bits can distinguish eight items. In general, n bits can distinguish 2n items. Thus, we can distinguish n items with log 2 n bits. It may be surprising, but we can do better than this by taking probabilities into account.
The axioms of probability are very weak and provide few constraints on allowable conditional probabilities. For example, if there are n binary variables, there are 2n-1 numbers to be assigned to give a complete probability distribution from which arbitrary conditional probabilities can be derived. To determine any probability, you may have to start with an enormous database of conditional probabilities or of probabilities of possible worlds.
Two main approaches are used to overcome the need for so many numbers:
Independence
Assume that the knowledge of the truth of one proposition, Y, does not affect the agent's belief in another proposition, X, in the context of other propositions Z. We say that X is independent of Y given Z. This is defined below.
Maximum entropy or random worlds
Given no other knowledge, assume that everything is as random as possible. That is, the probabilities are distributed as uniformly as possible consistent with the available information.
Reducing the Numbers
• The distinction between allowing representations of independence and using maximum entropy or random worlds highlights an important difference between views of a knowledge representation:
• The first view is that a knowledge representation provides a high-level modeling language that lets us model a domain in a reasonably natural way. According to this view, it is expected that knowledge representation designers prescribe how to use the knowledge representation language. It is expected that they provide a user manual on how to describe domains of interest.
• The second view is that a knowledge representation should allow someone to add whatever knowledge they may have about a domain. The knowledge representation should fill in the rest in a commonsense manner. According to this view, it is unreasonable for a knowledge representation designer to specify how particular knowledge should be encoded.
Comments