Welcome to Tinsight, where Tabbit shares long-form ideas worth your time. Our new series, “Rebuilding Individual Learning,” offers practical ways to rethink learning in the AI era. The complete essay is about 19,000 Chinese characters and will be published in six installments; this is Chapter 3. It explores a new way to internalize probabilistic reasoning, causal reasoning, game theory, and systems thinking: use AI as a demanding reviewer of your real-world judgments, then learn through repeated critique and revision.
Author: Dawei Geng
Source: https://offbook.press/essays/rebuilding-learning/
Chapter 3 · Modeling with AI (1): Thinking Tools as Reviewers
After discussing what questions are worth asking, the next question is: what should be learned in the AI era.
The answer to the question "what to learn" in the AI era is completely different from that in the past. Knowledge is not worth learning (the model knows more than you do), most of the skills are being automated (AI at the execution level is getting better and better), and most of the various "thinking frameworks" are words pretending to be tools (you will feel good after reading it, but you will not be able to use it at all when you encounter specific problems next time).
There are only a few specific things worth learning. Thinking tools are one of the most critical.
What are thinking tools
Thinking tools are specific thinking methods that are loaded in the cognitive system and can be called at any time. Its relationship with knowledge is similar to software and data: knowledge is a pile of information, and thinking tools are programs for processing this information.
The four thinking tools with the widest coverage and the highest returns are:
1. Probability and uncertainty reasoning
Really understand base rates, conditional probabilities, sample bias, selection effects, and calibration. Instead of taking statistics courses, turn them into reflexes of judgment.
2. Causal reasoning
Distinguish between correlation and causation, and understand confounding variables and counterfactual comparisons. Most people equate "A occurs with B" to "A causes B." This is a way to systematically correct this habit.
3. Game Theory and Incentive Structure
There is no need for mathematical depth, what is needed is the reflex of automatically identifying "who is paying for whose decision" in any phenomenon. After installing it, the way you read news, policies, and business phenomena will be completely different.
4. System Dynamics
Understanding nonlinearity, delayed feedback, and emergence. Most misattributions of complex problems are due to failure to simulate feedback loops in the mind.
Once all four tools are internalized, when faced with any phenomenon, the cognitive system will automatically break it down into "what variables are there, what is the direction of cause and effect, how are the incentives distributed among all parties, and where are the feedback loops." This is not a thinking trick, but a reflex.
Whether it is installed or not, there is a very simple test: when faced with a new phenomenon, does the person directly give a conclusion, or does he automatically put it into a certain reasoning framework? The former is poor, and his conclusion may be right or wrong, but he cannot tell the difference; the latter has been internalized, and he will first ask about the base rate, causal direction, incentive distribution, and feedback delay.
Why traditional learning paths fail
When it comes to learning thinking tools, the default path in the past was to read a classic book, read it over and over again, and try to internalize it.
For example, to learn probabilistic reasoning, read "Thinking, Fast and Slow" by Kahneman, to learn cause and effect, read Judea Pearl’s “The Book of Why”, to learn game theory, read Schelling's "The Strategy of Conflict", and to learn systems, read Donella Meadows’s “Thinking in Systems”.
This is not the wrong path, and these books are among the best introductory books in the field. But its failure rate is extremely high.
Specific manifestations of failure: A person carefully reads "Thinking, Fast and Slow" and feels that he has "learned" when he closes the book. Three months later, you ask him, "Which of the judgments you made last week used what you learned from this book?" He can't answer. Six months later, when making decisions, he was still using intuition, still falling into the biases clearly warned in the book, and still unable to identify when he was doing pattern matching and when he was doing real reasoning.
It's not that he's not serious enough. This is a mismatch in the learning method itself: reading is just exposure to information, but thinking tools require repeated use of specific judgments in order to be internalized. Without applying them, reading more will not help.
The remedy in the past was "deliberate practice": after reading the book, write a judgment diary every day, review decisions, and consciously apply tools. This plan is theoretically correct, but in practice most people cannot stick to it. Because writing a diary itself is an extra investment of energy, and your energy is not redundant and will stop within two weeks.
The result is that most people who have carefully read these classic books have not actually installed these tools.
A new path in the AI era: AI as reviewer
AI has completely rewritten this path.
The new path is not to let AI teach you these tools, so what you get is an AI version of the book, which is no better than reading the original book.
The new path is to treat AI as a harsh reviewer who is extremely familiar with this tool, and let it test whether you have internalized your real-world judgment.
Specific operations:
Step One: Create a Tool Portrait
Let AI use a paragraph to clearly summarize the core of this tool, its scope of application, its most common misuse, and how it differs from other tools. This portrait is not for you to read. It is a reference for the AI to maintain consistent review standards.
For example, for causal reasoning, let AI write clearly "what good causal reasoning should and should not do. The most common misuse is correlation as cause and effect, confounding variables are not controlled, and counterfactuals are not thought out clearly." This description will become the standard used by AI in subsequent reviews.
Step 2: Come up with a real-world judgment you made recently
It must be real, specific, and your own judgment: it can be about work, life, or judgment on a certain news.
Must be specific:
Good judgment: "I think company X's next quarter financial report will be worse than expected because Y"
Bad judgment: "I think the market has not been very good recently."
The latter cannot be reviewed because it has no specific content. The former is fine.
Step 3: Let AI use tools to review your judgment
"Please review my just judgment from the perspective of causal reasoning. Did I treat correlation as causation? Does the evidence I mentioned really support the direction of causation? What possible confounding variables did I miss? Is my counterfactual hypothesis reasonable?"
The AI will point out a bunch of questions, and these points are where you actually learn the tool. You're not "reading causal reasoning," you're looking at what causal reasoning reveals about your real-world judgment.
Step 4: Adjust judgment based on feedback and send it back for review
Until the AI has no significant rebuttal. Then change to a new real-world judgment and repeat.
After ten judgments this tool becomes internalized; after thirty judgments it becomes reflexive. Each judgment took 15 minutes, about 7-8 hours for 30 judgments, spread over several months.
Much faster and more effective than chewing through a 500 page book and then trying to internalize it.
Why this method works
The core reason why this method is effective is that it shifts the learning mechanism from “understanding” to “having your reasoning challenged”.
Understanding is an action that occurs within the brain. You read a passage and feel like you “get it.” This "understanding" is a feeling, not an ability. Most "I got it" feelings dissipate after a few days, leaving nothing behind.
Having your reasoning challenged is an experience that occurs in a specific situation. You made a judgment and it was pointed out that there was something wrong with that judgment. This "problem" is concrete, painful, and makes you think "next time it won't happen this way." Every reality check leaves a real trace in your cognitive system.
Traditional reading and learning are mostly about "understanding" and rarely about "being proven wrong", because the book doesn't know what your specific judgment is, and it can't challenge your reasoning directly. It can only give you a bunch of examples and expect you to apply them to yourself (most people can't do this).
The new path in the AI era solves this problem: AI can accurately refute your specific judgments. It knows the tool well enough and your judgments concretely enough that it can do a one-on-one review.
One-on-one reviews used to be something only real tutors or expensive consultants could do, not books. AI makes this a zero-cost, readily available thing. This is a structural advantage that AI has over most learning methods of the past.
A Concrete Example
In order to make this matter more concrete, let's use a fictitious example to demonstrate the mechanism of this method - but remember one thing: you must use real judgment when applying it yourself, hypothetical judgment cannot detect things (will be emphasized again in the "Key Limitations" later).
Suppose you want to learn causal reasoning.
The initial judgment you wrote down (fictitious):
I recently observed that people on my team who liked asking questions were more likely to stay after a year. So I think ‘liking to ask questions’ is an important reason for employee retention. I should pay more attention to candidates’ questioning habits in my next recruitment.
The prompt you give the AI:
Please act as a very strict causal reasoning reviewer. I take this judgment seriously and I will use it in actual hiring decisions. Please point out all possible causal errors in my judgment: correlation being regarded as cause and effect, confounding variables not being controlled, counterfactuals not being thought through, selection effects, survivor bias. Do not hold back—I want to know where my reasoning fails.
The AI's rebuttal might be:
There are several serious causal problems in your judgment:
1. You do not control for confounding variables. People who “like to ask questions” may themselves be smarter, more engaged, and more curious people. What really makes them stay may be these underlying qualities, and "asking questions" is just an external manifestation of these qualities. Paying attention to "questioning habits" when recruiting may be catching the wrong signal. What you really want are the underlying traits.
2. You may confuse the direction of cause and effect. Those who "stay" and those who "leave" are not facing the same environment. Those who stay may be assigned to good projects and good managers, so they have more opportunities and motivations to ask questions. It’s not “asking questions to make them stay”, but “a good environment makes them stay and ask questions”.
3. Survivorship bias. What you observe are "people who are still on the team", you don't see those "people who like to ask questions but have left." If their ratio is also low, your conclusion does not hold.
4. Your counterfactual has not been thought through clearly. "What will happen if I hire someone who doesn't like to ask questions?" You have no data. You only have data about "people who like to ask questions." Unilateral data cannot establish cause and effect.
5. Most importantly, What you observe may be noise. How many people are on your team? With only a dozen, the sample size is simply not large enough to support any causal inferences.
This rebuttal will have a far greater impact on you than reading a causal reasoning book because it targets your specific judgment. You will immediately realize that "there are so many errors hidden in what I thought was a clear judgment."
And the impact will permanently change the way you think the next time you make a similar judgment. The next time you observe a correlation, you automatically start asking, "What are the confounding variables? What about the direction of causation? What about selection effects? Is the sample size sufficient?"
This is not skill improvement. This is a cognitive reflex being installed.
Learning All Four Tools in Parallel
Only causal inference has been discussed previously. The other three tools (probability, game theory, system dynamics) can be learned in exactly the same way.
What’s more interesting is that these four tools can be learned in parallel and will strengthen each other.
Because real-world judgment usually involves more than one tool. Your judgment on "employee retention" can be reviewed at the same time using causal reasoning (causal direction, confounding variables), probabilistic reasoning (sample size, base rate), game theory (employee incentives, manager incentives), and system dynamics (feedback loops, delayed effects).
Therefore, after a judgment is handed over to AI, you can ask it to review it from multiple tool perspectives: "Please review my judgment from the four perspectives of causal reasoning, probability, game theory, and system dynamics." AI will use four lenses to examine the same judgment at the same time, and you will see four different problems with it.
This kind of multi-perspective review is impossible for any book, because a book only talks about one perspective, and your real judgment is multi-dimensional and requires multiple tools to work at the same time.
In terms of time estimate, the four tools are learned in parallel, and the "reflex level" can be reached in 6-12 months. This time span is surprisingly short for most people (much shorter than any degree program), but the rewards are much greater.
A Critical Limitation
This method has a key limitation that must be made clear:
It only works if you actually come up with a real-world judgment.
If all you give to AI are "hypothetical examples", "cases in textbooks" and "other people's judgments", the AI's review will still be accurate, but you won't really internalize it. Because you don't bear the consequences of those judgments, you don't pay for those judgments, your brain doesn't accept them as real.
The essence of internalization is real-world judgment + meaningful rebuttal + real-world correction. All three are indispensable.
There is another underlying thing worth making clear: AI review can point out loopholes in your reasoning, but it cannot replace the verification of your judgment by reality. It speeds up your calibration process, but the real calibration is ultimately done by reality - it is the result of running in the direction you choose, and your judgment of betting being falsified or confirmed by the market. AI just lets you have your reasoning challenged before you come into contact with reality, so that you make fewer avoidable mistakes when you come into contact with reality for the second time.
So this method is not something you can "take time to learn". It must be embedded in your real life: the work you are really doing, the problems you are really thinking about, the situations you are really evaluating. Only in this way can we accumulate enough real judgments to repeatedly polish the tool.
This also goes back to the cycle in Chapter 2: do, observe, ask, learn, and do. The learning of thinking tools occurs in the two steps of "asking" and "learning". If you are not "doing" or "observing", you have no judgment to hand over to the AI for review, and this method has no fuel.
Thinking tools address the methodology of judgment, allowing you to make more accurate judgments in any field. But it doesn't automatically give you knowledge about any specific area. A person who is full of tools can still only make superficial judgments when faced with a field that is completely unfamiliar to him, because he lacks the deep structure of the field.
This is what the next chapter will deal with: cross-disciplinary depth. Moreover, in the AI era, the way in which cross-disciplinary depth is established is completely different from that in the past.
Chapter 1 · Most people use AI to impair cognitive decoupling
Chapter 1 · Most people use AI to impair cognitive decoupling
Chapter 2 · What questions are worth asking?
Chapter 2 · What questions are worth asking?
Chapter 3 · Modeling with AI (1): Thinking Tools as Reviewers
Chapter 4 · Modeling with AI (2): Cross-Disciplinary Learning Starts with the Mirrors You Already Have
Chapter 5 · What AI can and cannot do
Chapter 6 · A starting point: AI programming
Chapter 7 · Closing Thoughts
