Bayes Theorem
Chapter 3 - Experiential Growth Method®
Previous page: Model Thinking - Bayes Theorem - Next page: Systems Thinking
Back to Book content or directly to Main Page or EGM
.
Welcome to the Bayes Theorem page
.
Why does the experience of the train passengers never match the punctuality statistics of the train company? Because we update our knowledge about the world around us, not in 'new-formed chunks' which can be counted statistically, but in a Bayesian way, updating existing knowledge.
Bayes' theorem is a fundamental result in probability theory that describes the relationship between prior and posterior probabilities. Probability science would lead us far away from this wiki's core intent. So instead, we limit ourselves to Bayesian statistics because it best describes how we statistically think as humans. Bayes' theorem is a way of figuring out the probability of something happening based on what we already know. It's named after a man named Thomas Bayes, who was a mathematician and philosopher.
.
Core idea
.
Conditional probability
Here's a simple example of how Bayes' theorem works:
Imagine you have a jar of marbles. Some of the marbles are red and some of them are green. You want to figure out the probability that a marble you pick out of the jar will be red.
You know that there are 10 marbles in the jar, and 3 of them are red. So, you can say that the probability of picking a red marble out of the jar is 3/10, or 30%.
Now, imagine that you pick a marble out of the jar and it turns out to be red. You might want to update your estimate of the probability that the next marble you pick out of the jar will be red. Bayes' theorem helps you do this. Using Bayes' theorem, you can take into account the fact that you just picked a red marble and update your estimate of the probability that the next marble you pick will be red. This can be helpful if you want to be more accurate in your predictions: 2/9, or 22%.
Conditional probability means that we - as humans - have and use prior knowledge. (See also Constructivism).
.
.
Description
.
| Bayes' Theorem |
|---|
| Bayes' Theorem is a simple mathematical formula used for calculating conditional probabilities. Bayes' theorem is particularly useful for inferring causes from their effects since it is often fairly easy to discern the probability of an effect given the presence or absence of a cause.
The objective is to discover what sorts of constraints experience tends to impose, and to explain how the person's prior opinions can be used to justify the choice of a posterior probability from among the many that might satisfy a given constraint. This is a kind of "no jumping to conclusions" requirement. We explain it here as a natural result of the idea that rational learners should proportion their beliefs to the strength of the evidence they acquire. |
| Bayes’ Theorem (Stanford Encyclopedia of Philosophy) |
.
.
Basic formula
The probability of a hypothesis - conditional on a given body of data - is
"the ratio of the unconditional probability of the conjunction of the hypothesis with the data to the unconditional probability of the data alone."
.
.
Example
.
Bayes theorem formula
P(Model|Data) = P(Data|Model)*P(Model) / P(Data)
.
P(Model|Data) : What is the probability that the train will run on time | given | my experience?
=
P(Data|Model) : My opinion that the train will run on time expresed in a probability : 60 %
x
P(Model) : The probability that the train will run on time (Railway statistics) : 80%
/
P(Data) : "My personal certainty of my opinion/experience" expresed in a probability : 100% (I am very sure of myself)
The calculation gives us: (60% * 80%) / 100% = 48%
.
In words:
- My believe that only 60% of the trains run on time, and being very sure of this opinion, gives me a feeling of dealing with a railroad where 50% of the trains run on time.
- Even if all trains (100%) would realy run on time, this would only enhance my feeling from 48% to 60%, not to the present railway statistics.
- Only if I would update my opinion and my certainty, I would come very close to the railway statistics.
.
.
Advantages
Bayesian modeling provides a robust framework for handling uncertainty in forecasting by offering dynamic, probabilistic predictions that adapt to new information. This approach allows for more nuanced risk assessment and decision-making.
- Adaptability: Bayesian methods excel in rapidly changing environments, continuously updating predictions as new information becomes available
- Nuanced assessment: by providing probability distributions rather than point estimates, Bayesian models offer a more comprehensive view of potential outcomes and risks
- Improved decision-making: the probabilistic output of Bayesian models aids in quantifying confidence levels, enabling more informed decisions
.
.
Pitfalls
Bayesian reasoning pitfalls can lead to inaccurate conclusions.
- Overconfidence in prior beliefs: we may rely too heavily on our existing knowledge or intuitions, leading to biased priors that don't accurately reflect the true uncertainty in a situation. This can result in overly confident predictions or decisions that don't adequately account for new information
- Bayesian models are only as good as the information they incorporate. Failing to include all available and relevant data can lead to incomplete or inaccurate inferences
- Determining well-reasoned priors that accurately represent existing knowledge can be a time-consuming and challenging task, especially in novel or rapidly changing business environments
.
.
| Bayes theorem, the geometry of changing beliefs |
|---|
| https://www.youtube.com/watch?v=HZGCoVF3YvM |
.
Deep dive
.
Example of Bayesian Thinking in medicine
Bayesian statistics interprets probability as a degree of belief. It uses Bayes' theorem to update prior beliefs with new evidence, resulting in posterior probabilities.
Your medical doctor informs you that you tested positive for a rare disease. You were not feeling well for a while. The disease affects 0.1% of the population and the doctor adds that the test has 99% accuracy. Most people might intuitively think that a positive test result means there's a 99% chance of having the disease. However, this is incorrect due to the base rate fallacy.
Here's why the actual likelihood is around 9%:
1. Prior probability: The disease affects 0.1% of the population, so the prior probability of having the disease is 0.001.
2. Test accuracy: The test is 99% accurate, which means:
- True positive rate (sensitivity): 99% of people with the disease test positive
- False positive rate: 1% of people without the disease test positive
3. Applying Bayes' theorem:
P(Disease | Positive Test) = P(Positive Test | Disease) * P(Disease) / P(Positive Test)
P(Positive Test) = P(Positive Test | Disease) P(Disease) + P(Positive Test | No Disease) P(No Disease)
= 0.99 0.001 + 0.01 0.999
= 0.00099 + 0.00999
= 0.01098
P(Disease | Positive Test) = (0.99 * 0.001) / 0.01098 ≈ 0.0902 or about 9%
This counterintuitive result occurs because the false positive rate (1%) applied to the large population of people without the disease (99.9%) outweighs the true positive rate (99%) applied to the small population with the disease (0.1%).
If a second independent lab confirms the test, the likelihood of having the disease increases significantly:
1. The probability of having the disease after the first test becomes the new prior probability.
2. We apply Bayes' theorem again with this new prior and the second test result.
P(Disease | Two Positive Tests) ≈ 0.9016 or about 90%
.
| David R. Mandel - The psychology of Bayesian reasoning - Frontiers in Psychology - 2014 |
|---|
| Bayesian reasoning also benefits from the use of visual representations of pertinent statistical information, such as Euler circles (Sloman et al., 2003) and frequency grids or trees (Sedlmeier and Gigerenzer, 2001), which further clarify nested-set relations. |
| https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2014.01144/full |
.
.
Examples of Bayesian Thinking in business
Insurance
Insurance companies utilize Bayesian methods to model the probability of various events, such as natural disasters or accidents. They start with prior knowledge about risk factors and historical data, then update their models as new information becomes available. This approach allows them to adjust policy prices and manage risk exposure more accurately over time.
.
Investment
Financial analysts and investors often think in Bayesian terms when making investment decisions. They begin with prior beliefs about a company's performance based on historical data and industry trends. As new information becomes available (e.g., quarterly earnings reports, market conditions), they update their probability estimates for future performance. This continuous updating process helps investors refine their portfolio strategies and make more informed decisions.
.
.
Bayes & AI
.
| Judea Pearl - The book of Why - Basic Books - 2018 |
|---|
| “Bayes’s rule” offers a general solution to the inverse-probability problem. It tells us that if we know the probability of A given B, P(A | B), we ought to be able to figure out the probability of B given A, P(B | A), assuming of course that we know P(A) and P(B). This is perhaps the most important role of Bayes’s rule in statistics: we can estimate the conditional probability directly in one direction, for which our judgment is more reliable, and use mathematics to derive the conditional probability in the other direction, for which our judgment is rather hazy. The equation also plays this role in Bayesian networks; we tell the computer the forward probabilities, and the computer tells us the inverse probabilities when needed. |
The accuracy of the outcome depends entirely on the correctness of the prior probabilities we feed to the program.
.
Artificial Intelligence systems, particularly those employing machine learning techniques, often leverage Bayesian principles in their decision-making processes. This approach offers several advantages over typical human reasoning, making AI thinking potentially superior in many scenarios.
Continuous Learning and Adaptation
AI systems based on Bayesian logic excel at continuous learning. They can update their beliefs (or model parameters) in real time as new data becomes available. This mirrors the Bayesian approach of updating prior probabilities with new evidence to form posterior probabilities. On the other hand, humans often struggle with this continuous updating process. We form opinions based on initial information and find adjusting these beliefs proportionally when presented with new evidence challenging. This cognitive bias, known as anchoring, can lead to suboptimal decision-making.
Handling Uncertainty
Bayesian AI systems are inherently designed to handle uncertainty. They work with probability distributions rather than point estimates, allowing them to express degrees of certainty about different outcomes. Humans often struggle with probabilistic thinking and seek certainty even when it's impossible. We're prone to overconfidence in our predictions and judgments, especially in complex scenarios with multiple variables.
Incorporating Prior Knowledge
Bayesian AI can efficiently incorporate prior knowledge into its decision-making process. This prior knowledge can come from expert opinions, historical data, or previous learning experiences. The system then updates this prior knowledge with new data to form more accurate posterior beliefs. While humans can also incorporate prior knowledge, we often do so inconsistently or with biases. We might give too much weight to recent or vivid experiences (recency bias) or struggle to integrate conflicting information coherently.
Avoiding Overfitting
Bayesian methods in AI naturally guard against overfitting, a common problem in machine learning in which a model fits the training data too closely and fails to generalize well to new data. Bayesian approaches can provide more robust predictions by considering a distribution of possible models rather than a single "best" model. Humans are also susceptible to overfitting, where we might draw overly specific conclusions from limited personal experiences, failing to generalize appropriately to new situations.
Dealing with Small Sample Sizes
Bayesian methods shine when dealing with small sample sizes or sparse data. They can still provide meaningful insights by leveraging prior information and expressing uncertainty appropriately. Humans often struggle to reason correctly with small samples. We're prone to drawing firm conclusions from limited data, a tendency known as the law of small numbers.
Combining Multiple Sources of Information
Bayesian AI systems excel at combining information from multiple sources, weighing each piece of evidence according to its reliability and relevance. This allows for sophisticated multi-modal learning and decision-making. While humans can also integrate various sources of information, we often do so suboptimally. We might give undue weight to specific sources based on emotional factors or struggle to reconcile conflicting information.
Explainable Decision-Making
Bayesian approaches in AI can provide a transparent chain of reasoning for their decisions. Examining how prior probabilities are updated with new evidence makes it possible to understand why the AI system arrived at a particular conclusion (Kinnear & Wilson, 2022). Human decision-making processes, especially in complex scenarios, are often opaque even to ourselves. We make decisions based on intuition or gut feeling without fully articulating our reasoning.
Long-Term Planning Under Uncertainty
Bayesian decision theory allows AI systems to make optimal decisions in complex, uncertain environments by considering the expected utility of different actions. This is particularly powerful for long-term planning scenarios. Humans often struggle with long-term planning under uncertainty. We tend to discount future outcomes inappropriately and have difficulty accurately estimating probabilities of complex future events.
.
| Applications of Bayesian networks in AI |
|---|
| Bayesian networks find applications in a variety of tasks such as:
1. Spam filtering: A spam filter is a program that helps in detecting unsolicited and spam mails. Bayesian spam filters check whether a mail is spam or not. They use filtering to learn from spam and ham messages. 2. Biomonitoring: This involves the use of indicators to quantify the concentration of chemicals in the human body. Blood or urine is used to measure the same. 3. Information retrieval: Bayesian networks assist in information retrieval for research, which is a constant process of extracting information from databases. It works in a loop. Hence, we have to continuously reconsider and redefine our research problem to avoid data overload. 4. Image processing: A form of signal processing, image processing uses mathematical operations to convert images into digital format. Once images are converted, their quality can be enhanced with more operations. The input image doesn’t necessarily have to be in the form of an image; it could be a photograph or a video frame. 5. Gene regulatory network: A Bayesian network is an algorithm that can be applied to gene regulatory networks in order to make predictions about the effects of genetic variations on cellular phenotypes. Gene regulatory networks are a set of mathematical equations that describe the interactions between genes, proteins, and metabolites. They are used to study how genetic variations affect the development of a cell or organism. 6. Turbo code: Turbo codes are a type of error correction code capable of achieving very high data rates and long distances between error correcting nodes in a communications system. They have been used in satellites, space probes, deep-space missions, military communications systems, and civilian wireless communication systems, including WiFi and 4G LTE cellular telephone systems. 7. Document classification: This is a problem often encountered in computer science and information science. Here, the main issue is to assign a document multiple classes. The task can be achieved manually and algorithmically. Since manual effort takes too much time, algorithmic documentation is done to complete it quickly and effectively. |
| https://www.turing.com/kb/an-overview-of-bayesian-networks-in-ai |
.
| How human–AI feedback loops alter human perceptual, emotional and social judgements - Nature Human Behaviour - 2024 |
|---|
| Artificial intelligence (AI) technologies are rapidly advancing, enhancing human capabilities across various fields spanning from finance to medicine. Despite their numerous advantages, AI systems can exhibit biased judgements in domains ranging from perception to emotion. We reveal a feedback loop where human–AI interactions alter processes underlying human perceptual, emotional and social judgements, subsequently amplifying biases in humans. This amplification is significantly greater than that observed in interactions between humans, due to both the tendency of AI systems to amplify biases and the way humans perceive AI systems. Participants are often unaware of the extent of the AI’s influence, rendering them more susceptible to it. |
| https://www.nature.com/articles/s41562-024-02077-2 |
.
| Large Language Models explained briefly |
|---|
| https://www.youtube.com/watch?v=LPZh9BOjkQs&t=2s |
.
Do you want to know more?
.
| Improving Bayesian Reasoning: What Works and Why? |
|---|
| Beyond getting the numbers right: what does it mean to be a “successful” Bayesian reasoner? |
| https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2015.00712/full |
| Rationality, the Bayesian standpoint, and the Monty-Hall problem |
| https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2015.01168/full |
| Effects of visualizing statistical information – an empirical study on tree diagrams and 2 × 2 tables |
| https://www.frontiersin.org/journals/psychology/articles/10.3389/fpsyg.2015.01186/full |
.