I regularly work with GPT-4o, Gemini, and Grok, asking them to debate and cross-check one another. But a study of social-intelligence testing raised a different question for me: how far is a model that scores highly on a paper-and-pencil scenario test from actually understanding the person in front of it?
Social intelligence involves judging emotion, understanding another person’s situation, and choosing an appropriate response in interaction. It is also something I practiced repeatedly during fifteen years in seminary. Pastoral care, counseling conversations, and group dynamics all bring this question into view.
The study is worth reading. We first need to say clearly whom it beat, and where.
Scoring above this group of students does not mean surpassing all psychologists
A 2024 preliminary comparative study in Frontiers in Psychology asked 180 male undergraduate and doctoral psychology-counseling students at King Khalid University in Saudi Arabia, along with ChatGPT-4, Bing, and Google Bard, to answer 64 social-intelligence scenarios. ChatGPT-4 scored 59, Bing 48, and Bard 40. The paper reported that ChatGPT-4’s score was higher than that of all human participants.
“All” here means the participants in that study. This was an all-male student sample from one university. It cannot be extended to every practicing psychologist or psychology expert. It also measured performance on a particular set of questions, not real counseling outcomes.
The study makes me ask what a model can do. It does not justify declaring that AI now understands people better than psychologists do.
Getting a scenario question right is different from accompanying a person in conversation
Scenario questions already organize the problem, options, and scoring method. In a real conversation, even the problem itself may not yet be clear. What someone says and what they are willing to let you know are not necessarily the same.
That gap is what matters most to me in reading the study. Knowing which option is more appropriate and continuing to respond to a person within a relationship are different abilities that need separate evidence. A score on one test cannot vouch for the safety of mental-health services.
Why a model answered well also needs evidence. This study alone cannot tell us how much counseling material the model had encountered, and “it had more training data” does not by itself explain the result.
What does it mean when I feel that it understands me?
When I run a debate engine, Claude or GPT occasionally gives me a response that catches an unstated premise or addresses what I was actually concerned about. The feeling that “it understands me” does occur in collaboration.
But a user’s feeling and whether a model possesses understanding are questions at different levels. I can speak about the former from experience. The latter cannot be settled by one fitting answer.
I still want to ask: if a system can keep hold of context, how should we describe that ability? It is worth discussing. It is not yet a conclusion that this test has answered.
Confirm each task an AI may assist with, one by one
This result does not predict whether psychologists will lose their jobs, nor does it decide the future division of work for professionals.
If AI is to be used in helping work, I would begin with the concrete use: is it organizing text, suggesting questions for discussion, or participating in a judgment that affects someone’s safety? Different uses need different evidence and different arrangements for responsibility. Who checks a response, who can catch an error, and who handles harm cannot be disposed of by a score sheet.
In “Breaking Through the AI Storm”, I wrote about changing divisions of work. Here the point is to state exactly which task is being handed over, then establish whether it is reliable.
The understanding I care about also includes bearing responsibility
In “The Addiction Economy and the Lonely Generation”, I consider how people form connection with others. An AI response can make someone feel heard. But the relationship I value also includes staying, bearing the consequences of one’s response, and facing uncertainty together.
That is my view of understanding. It does not settle whether a model is conscious, and it does not disappear because of a high score on one test.
We can score what an answer gets right. After the answer, who is still willing to stay and deal with what comes next? That is the question I want to keep.


💬 Comments
Loading...