The New York Times has a wonderful daily online game called Pips, which involves placing dominoes on a grid to meet various numeric constraints. It has three levels of difficulty and you’re scored on the time it takes you to solve each puzzle. I found out about it from a Slack group I belong to where we share our daily scores on Times puzzles.
I soon ran into a problem. I was solving the medium and hard problems in 10, 20 minutes. But the person who’d alerted us to this new puzzle was doing them in 2, 3 minutes. Eventually I couldn’t take it any more, and I said, “Dang, how are you doing that???“ And she replied that she did it “by elimination: find the single digit, look what domino it could be, and start there. Same with the equal signs, check what digit that could be, in easy or medium you usually only have 1 or 2 options, which forces some placements.” And I said, “That’s what I’m doing… but apparently you do it 6 times faster.”
(The AI connection is imminent, I promise.)
I scratched my head trying to figure out why I was so much slower at the same strategy. I was approaching this methodically and logically, writing down tables of how many times each number appeared, finding the constraints in order of narrowness, evaluating all possible options, tracking every branch point. Where was I going wrong?
Eventually I realized that my friend was not doing any of the paperwork. She was operating at intuition speed. So I ditched my painstaking process and incorporated more gut feel. Easier said than done. For me, the control freak, it felt reckless and scattershot. But it worked.
I’m still not as fast as her. But I am three times faster than I used to be. If you go fast enough the game gives you a cookie sticker and I’ve gotten a few of those now. This gives me irrational pleasure.
Today the parallel with how we use AI struck me. (Told you I’d get there.) Using generative AI for a complex process like developing a legal brief involves many different stages: ideation, research, structuring, writing, and so on. AI is clearly useful throughout this process. But we’ve seen the results of giving it too much free reign in the legal arena in particular; I’ve drawn attention on my podcast to several occasions where a lawyer got in trouble for submitting a brief created by AI, without any evident vetting, which would have discovered egregious hallucinations.
Finding the balance between human and AI, when to give it free rein and when to check it, edit it, or leave it behind, is the big challenge in many places right now. And I’ve found no better guide to conveying those distinctions than intuition.
We’ve learned how to create detailed, precisely targeted exercises that foster that intuition rapidly (this is in our AI literacy training for schools). Over time this intuition will become more natural and part of the global shared consciousness. The public at large is assimilating AI basics at an amazing rate; when I first started teaching AI literacy in 2018 I was advised to spell out what “AI” stood for before first usage because of the likelihood that there would be attendees who didn’t know. (I wasn’t teaching how to use AI personally at that point, but familiarizing people with its characteristics and how it was affecting their lives.) Today my attendees for the basic intro course arrive with experience of using at least one large language model.
And for all the frameworks and models and diagrams that simplify, codify, and amplify the understanding of AI, none of that is useful for understanding hallucinations. The reality is too complex to take on board from a document. It’s practically axiomatic that understanding of hallucinations—and many of the principles for productive use of generative AI—can only come from experience. (Axiomatic because if it could be formally codified, it could be modeled and integrated by generative AI, thus eliminating the hallucinations in the first place.)
But—good news! Experience will work. Just as learning how to play tennis by reading a book (I actually tried this when I was in school) won’t work, but getting on a court and hitting balls around will. And so the trainings I lead are highly experiential.
This all converges towards a principle of treating AI not like a tool—something you can read an instruction manual for—but a person. (My wife, for instance, did not come with an instruction manual.)
And we do know how to work with people. To be sure, AI is very weird when viewed as a person. But so am I, yet I am still useful. And people can work with me.
The challenge that these principles lead to is that there remains this component of stepping out on a ledge, of trusting in our own intuition, in order to use generative AI. We’ve not going to get the kind of formal proof of reliability that we expect from a tool, like a Material Safety Data Sheet, for AI. This is not the droid you’re looking for.
So while you can blindly trust a tool that comes with a guarantee, that approach won’t work with AI. You need to exercise supervision. Just as when your phone throws up suggestions for the next words to type, you don’t have to take them.
Experience will give you an intuitive understanding of where you can trust AI more, or less. For instance, I place a great deal of trust in it when asking for recipes. I know, AI once suggested someone use glue to stop toppings sliding off a pizza, but that was ancient history in the world of LLMs and I have never gotten a bad recipe from ChatGPT, at least. There’s also an element of safety – I could see if it were incorporating something dangerous, and outside of that possibility, the worst risk is of a cookie batch turning out overdone.
When it comes to a more critical function like generating references for science research, I’ll check them. Plus there are a few other time-saving tricks up my sleeve. At worst the process has turned from a writing problem to a reading problem and thus accelerated.
Many people don’t, of course, do that necessary due diligence—like the aforementioned lawyers—and this is causing much of the friction of AI use in the workplace, creating a false dichotomy of “AI works/AI doesn’t work” when the real function to attend to is the partnership with AI.
Like so many other aspects of AI, this one is particularly acute in education. While some professions have poor use cases for AI, resulting in higher rates of hallucination and greater learning curves for prompting, this is not so for the majority of educational use cases where the goal is to understand something.
Say your goal is to learn about the refraction of light. Generative AI’s ability to help you with this will be superb thanks to an enormous amount of training data—countless physics textbooks, lectures, videos, term papers, past examination sheets, encyclopedias, FAQs—all agreeing closely on the facts that count. That creates tremendous reinforcement within an LLM to converge on correct answers, and to take useful paths in helping explain them.
AI-assisted learning—particularly of the common subjects taught at least through high school—is an exceptionally good use case. One might almost say that this is what AI was born to be good at.
But the price of admission to this is being able to act on and trust your own learned intuition. And this is very hard for some people who are still seeing AI as alien, threatening, and impenetrable. They’re waiting for the day when using AI will be as straightforward and trustworthy as pushing the Lobby button in an elevator. But that is never going to happen (except in very narrow circumstances thanks to the usual caveat that every principle in AI has exceptions).
The reward for trusting that intuition is, like solving Pips faster, greater comfort and accelerated workflows in using generative AI. And as usual, education is the field in which we cannot wait this one out. How have you developed an intuition for using AI?
Leave a Reply