Concept Testing With Images

What is concept testing, and why do rating scales miss visual questions?

Concept testing checks whether an idea works for its audience before anyone builds it, and teams run it to cut risk, drop weak ideas cheaply and get stakeholders behind the one that works. Rating scales miss visual questions because they ask people to put a number on a reaction that happened before they had words for it.

Most concept tests are questionnaires. A respondent reads or looks at a concept, answers scaled questions about appeal, clarity, uniqueness and purchase intent, and the scores get compared across concepts. That works for an idea made of words. People react to images fast and mostly below the level of language, so asking them to rate a brand direction means asking them to translate that reaction into a number, then into a reason, after the fact. What you collect is what they think they should say about it. Our What Is Constellations page puts it plainly: perception often happens before language, and verbal methods assume people can accurately describe their internal reactions.

Figure 1 · The rating scale and the sort

So concept testing with images flips the instrument. People respond to the images directly, by marking what fits the prompt and what doesn't, and nobody has to explain themselves. The explanation comes later, from the pattern across hundreds of responses. This is audience perception mapping: an audience responds to images instead of describing them, and the pattern shows where the group agrees, where it splits, and why. (For the launch side of this, see how to test brand direction with your audience before launch.)

Usability testing gets mixed up with all of this, and it comes at the other end of a project. It watches people use something to see whether it works. Concept testing happens before the thing exists and asks what the idea means to the people it's for.

The concept testing formats, and which one fits visual work

For visual work, the format that holds up best shows every participant the whole range of images at once and asks them to sort it, because a reaction to an image only means something next to the images it was chosen over. The classic formats each solve a different problem, and it helps to know what each one is built to measure.

The last row is what Constellations runs, and it's the format the rest of this page describes. It sits on top of your normal research, after the client interviews, the competitor review and the business questions, and before anyone builds a direction.

Concept boards, mood boards, and what to put in front of people

Put raw design elements in front of people, organized into concept boards: line, form, texture, color and type, large enough to read and numerous enough to produce a pattern. Leave finished logos, full layouts and other people's completed work off the board.

Clients and designers see differently. A client looks at a finished brand and reacts to the whole thing at once. A designer sees the weight of the type, the temperature of the color, the grain of the texture, and recombines those parts into something that doesn't exist yet. If you test a finished piece, the audience tells you whether they like the whole, and you can't trace the reaction back to the part that did the work. If you test the parts, you learn what you're going to build with.

A working set is 24 to 48 images, usually across two or more boards of twelve. You can mix everything on one board to see what pulls when it all competes, or split it into category boards (one for color and texture, one for typography, one for photography) and make people choose inside a category. The Coca-Cola study used two boards of twelve with different images under the same prompt. The pattern had to show up on both boards independently to count, and it did.

Mood boards have a different job, and a good one. (We still use them, and we always ask what people don't like on them.) A mood board turns what's in a stakeholder's head into something everyone can see. When the CEO says “modern,” the room finally knows which decade of modern. For a wedding invitation or an office interior, where that person's taste is the target, the mood board finishes the job. For anything going out to a market, it records what two to twenty people in one room preferred, and the market never sat in that room. The mood board problem goes through where that breaks, and what actually belongs on a concept board covers building a board that can be tested.

One more thing on the board, because it saves arguments later: the CEO's favorite can go in. Put it on the board with everything else and let it compete. If it wins, everybody sees it win, and if it loses, nobody had to be the person who said so.

What questions should a concept test ask?

Ask one question about the images that forces a sort, such as “which of these feel most and least like us,” and let people answer by marking the images instead of describing them.

A question that forces a sort produces two piles. “What do you think of these?” gets you a conversation about them. The prompts that work are short and concrete enough to answer in a glance: which of these make you want a Coke right now, what looks trustworthy, which look most and least interesting.

The half most teams skip is the rejection. A pile of liked images shares a dozen qualities, most of them coincidence, because one person sourced them for one board. The rejected images were on the same board in front of the same people. Whatever separates the two piles is the quality the audience was sorting on, and that quality is what you design from. Ask what they hate goes deeper on building rejection into the test.

How do you test a logo, an ad or a campaign before launch?

Test the pieces before the finished thing: the typography, the shapes and forms, the textures and the color, shown in context, and measure what people associate with each one. The direction gets built from what the audience connected with, and the version you eventually present has been resting on that signal from the start.

A logo collects opinions faster than anything else a brand makes. A social poll or a focus group tells you what your followers will click or what eight people will say while watching the loudest one for cues. Testing the elements tells you what a few hundred people in your market associate with each shape and letterform, and association is the thing that decides whether the brand lands in the right category and at the right price. How to test a logo before you launch it walks through it.

The Coca-Cola study shows what a finished mark does in a test like this. The logo on its own, on concentric red circles, drew 96.67% red. The brand's mascots lost too: the hugging CGI polar bears drew 82% red and the illustrated Santa 79%. People in that study wanted the drink, and anything standing between them and the drink cost points.

Ads follow the same logic with one extra lesson. In that study, a campaign frame of a sweating, ice-filled glass with bold “AHHH” type over it still cleared the bar at 68.75% green, and it was the weakest winner in the study. The plain ice-filled glass with no type scored 97.58%. The typography sitting between the viewer and the drink cost close to 30 points, and a test of the ad's elements finds that before the media buy does.

How many people you need, and how to keep bias out

Patterns in a perception test typically hold from about 30 responses up, and a brand decision is usually tested with 300 to 500 people who match the audience the work is for.

Our own published studies show what different sizes buy you. The Halloween study on spooky or scary ran with 22 people. That's a directional read, and it was reported as one. The study of what reads as American had 57. The Coca-Cola study had 467, and it's the one with enough people to build a decision on. More people past a certain point make the map sharper and the case to a skeptical stakeholder stronger, and they rarely change the direction it points.

The bigger risk is the wrong people. A well-sized group drawn from the wrong audience hands you a clean, confident answer to a question your market never asked. Sometimes the audience you need is hard to reach, and then a proxy has to know the audience well. When InEight rebranded, its buyers were capital construction executives, who are hard to survey at scale, so the test went to the whole company instead, from executives to frontline staff, people who deal with those buyers every week. In general, a test can go to your own audience by link, or to a recruited audience that matches them.

Bias is the first thing a skeptical stakeholder asks about. If you're the one who'll defend a result to a CFO or a board, it's the right question, and you should ask it of any method, ours included. Three kinds of bias do most of the damage in concept testing, and an image test handles each one differently:

• Question-wording bias. A survey question teaches people what kind of answer it wants. An image prompt carries one short line and the answer is a mark on a picture. There's very little wording left to lean on anyone.
• Social desirability bias. People answer surveys in what our home page calls personal PR mode: the safe answer, the smart one. Stefon Harris, Co-Director of Express Newark, described taking a Constellations test this way: “In other surveys, a lot of times you'll give a work answer. Here, when I responded, it was really a personal response.”
• Anchoring and groupthink. In a focus group, the first person to speak sets the frame and the quiet ones defer. In a perception test, everybody responds alone and at the same time, and nobody hears anyone else's answer first.

None of this makes a test immune to a badly built board or the wrong audience, but it does take the room, and everyone performing for it, out of the result.

Reading the results as a perception map

Read the result as a perception map, in three passes: what the images people kept have in common, what separates them from the images people rejected, and whether the contested middle is indifference or a split between two audiences. The output you're after is a sentence about the quality the audience sorted on. That sentence is what the creative team builds from.

First, a word on “heatmap,” because it means three different things. A mouse-tracking heatmap shows where cursors moved on a live page. An eye-tracking heatmap shows where eyes landed in the first instant, and that's mostly reflex. A perception heatmap shows what people connected to an idea when asked. Only the last one tells you what an image means to the people looking at it. What heatmaps actually tell you about your audience covers the difference.

A Constellations map sorts every image into three groups against a consensus threshold, the bar for how much agreement counts:

• Resonance: images most people marked as right for the prompt. Green.
• Resistance: images most people marked as wrong for it. Red.
• Contested: images that split the room, or that nobody felt strongly about.

The bar is set for each study. Our published studies have used anywhere from 59% to 75%, and the Coca-Cola study read its 24 images at 60%. Ten cleared it as craving triggers, twelve crossed it as craving killers, and two split the room. Here are the extremes:

Figure 2 · The three passes

Pass one is the green pile, and the temptation is to pick a favorite from it. Don't. Ask what the winners share. In this study every image in resonance showed visible evidence of cold and none in resistance did. The logo appeared on winners and losers both, and so did people.

Pass two is the red pile, read as a control. The rejected images were on the same boards in front of the same people. Anything the two piles share can be ruled out. What's left is the dividing line, and that line is what you design against for the rest of the project. Rejection carried a second finding here too: the polar bear drew a crowd, and the crowd was red. Lots of dots on an image tells you it got attention and says nothing yet about which way the verdict went.

Pass three is the middle, and it's worth slowing down on. A contested image can mean nobody cared, or it can mean half the room said yes and half said no and the averages cancelled. Those look identical on the map. Segment the results, turn a group on and off, and watch whether the middle resolves. If you're even a little curious whether a group sees it differently, segment, because you can always blend segments back together and you can't recover one you never collected. When the segments disagree covers what to do with a split, and how to read a perception map walks the three passes on more studies.

Reading the map takes about five minutes. Getting the room to stop arguing with it takes a few days.

Preference test, A/B test or perception test: what’s the difference?

A preference test asks which option people like, an A/B test measures which finished option performs better in market, and a perception test shows what a range of images means to an audience and why some are kept and others rejected.

Figure 3 · Where each test sits in a project

They sit at different points in a project. A preference test needs options that already exist and returns a winner without a reason. An A/B test needs two finished assets, so the production money goes out before anything comes back, and the losing version is live in front of customers for the length of the test. For a logo or an identity that's a lot of brand to put at risk. Finished pieces also hide which part did the work, the way the “AHHH” frame above only gave away its cost next to the plain glass. A perception test runs before the direction exists, on elements, and the direction gets built from what it finds.

Concept testing with images next to focus groups, attention prediction and AI panels

Each method answers a different question. Focus groups surface language and the reactions you didn't think to ask about, attention prediction estimates where eyes will land, synthetic AI panels predict answers, and a perception test measures what a crowd of people agreed an image means.

Focus groups have been around since Robert Merton's work at Columbia in the 1940s, and the format has barely moved since. They're good at finding language and at catching the thing you never thought to ask. They struggle with visual direction because of the room: one confident voice steers the session, and the quiet participants, who may look most like your audience, defer to the confident ones. Alternatives to focus groups for creative testing compares the options.

Attention prediction tools build a heatmap too, and that's why they get confused with perception testing. A saliency model estimates where gaze will fall, and that's a good prediction of physiology. It's the right tool once there's a layout to check. It has nothing to say about whether a direction reads as premium or cheap to the people who buy from you, so the order that works is meaning first, then attention. Attention prediction vs. perception testing goes through both.

Synthetic panels are the newest entrant. Blue Pill AI, for one, sells concept testing “from idea to insight in 48 hours” using AI consumer panels and synthetic personas. We've given our own perception test to AI models over the past year. The early results showed no clustering at all, while every human audience we've tested has clustered clearly. Current models do pick images now and show a pattern. On easy prompts with a fairly predictable answer they do okay, and on the abstract, nuanced prompts that branding depends on they're wildly off from the human data, with a fluent, convincing explanation attached, and a confident wrong answer is much harder to catch in a meeting than an obvious non-answer. What happens when you show AI 100 brand directions has the side-by-side.

Brand research is the last comparison people ask about, and the difference is mostly what you're holding at the end. A brand research engagement gives you a report on what your market thinks and says, gathered through interviews and surveys. A perception map gives you the images people connected to an idea and the ones they pushed away, and it sits between that strategy and the finished creative. Most teams that use both run the research first.

Teams that use concept testing with images

Brand teams choosing a direction use it, and so do agencies that have to defend one, universities and public bodies that need to hear past the usual voices, and anyone stuck with a room that can't agree.

InEight, a global capital construction software company, used it at its ten-year rebrand. After the usual research, four perception maps of tonal ranges went out, with visual prompts in place of polished concepts. Many inside the company expected innovation to win. Confident leadership did, in two tones, “Authority & Control” and “Wisdom & Counsel,” and the room picked its direction once without circling back. The InEight rebrand tells the full story. A liberal arts college tested campus imagery with prospective students and learned that giving each map one theme surfaced which classrooms felt welcoming, in the college use case. Express Newark at Rutgers University used it to get past “work answers” from a staff in a year of transition, in how Express Newark built alignment.

Consumer insights research covers each kind of team that uses it, with a published map from each.

How do you run your first concept test with images?

Build a board of elements from what you already know about the business, write one prompt that forces a sort, send it to the audience the work is for, then read the map in three passes and build the direction from the quality it names.

1. Do the business research first. Client interviews, competitors, the questions about market, offering, audience and internal pain points. Nothing here replaces it.
2. Build the boards. Two dozen to four dozen elements, two or more boards of twelve, with the stakeholders' favorites in the mix if they have them.
3. Write the prompt. One line that asks for a sort, with room to reject as easily as to choose.
4. Pick the audience. The people the work has to land with, or the closest proxy that knows them, segmented if any group might see it differently.
5. Read the map. Go through what people kept, what they rejected and where they split, then name the quality and build the direction on top of it.

Results come back in under 48 hours once the test is live. What you walk into the next review with is a map anyone in the room can read, and the conversation moves from whose taste wins to what the audience saw. If you've been the one in that room arguing for the audience with nothing to point at, this is the thing to bring.

It's also how a team avoids what we call the Approval Cliff: the gap between a direction that got approved because nobody blocked it and one the team believes in. A concept test with images closes that gap before launch, because everyone has agreed and they know why. They can see why.

If you'd like us to run one on a live project, pricing lays out three done-for-you studies: one for leadership and project teams, one for the whole company, and one for the audiences outside your walls. Each starts with a 30-minute scoping call to frame the question and settle who takes part, and then we design the boards, field the study and walk you through the map. Teams that would rather run studies themselves can get certified on the method, and universities and public agencies have their own pricing.