The set of questions
The ones a real client in that category would ask: what they need to solve, how they would name it without knowing any brands, what they would ask to compare options. The set is closed before looking at results.
Method
The most common way to assess a brand's visibility in artificial intelligence systems is also the least reliable: open an assistant, type a question, see whether the brand appears and draw a conclusion. That gesture produces an impression, not a data point. And every impression depends on what nobody wrote down: what was asked, in which system, on what date.
Measuring in search engines has a stable reference: a position is a number you can query again. In a generative system that firm ground does not exist. There is an answer produced at a specific moment for a specific question, and getting a different text when you repeat the query is not a flaw in the method: it is the nature of what is being measured.
That does not make research impossible, but it makes it more demanding: it forces you to repeat, to log and to compare. Here we explain how we are doing it, without anticipating results.
Measuring something that does not stay still
A generative system does not return a stored record, but a text built at the moment. That construction varies for three reasons that affect any attempt to measure.
- It varies between runs. The same question, asked twice under similar conditions, can produce different answers in length, in the order of ideas and in the brands that appear.
- It varies over time. Systems are updated and published content changes. What does not appear today may appear later without anyone having intervened at all, and the opposite too.
- It depends on how you ask. An open question, a request for a recommendation and a query that already names a brand trigger different answers. Changing one word can change the entities the system considers relevant.
None of those variations is solved with a technique: they are solved by repeating and logging. If the object of study is unstable, the honest thing is to describe how many times it was observed.
A single query is an anecdote. A set of queries repeated over time, with the same criterion, is a data point.
What is defined before measuring
If the system is unstable, the method has to be stable. Before logging the first answer, four decisions are fixed, so that a later measurement is comparable with the previous one.
The systems that are queried
They do not all retrieve, weigh and cite in the same way, nor do they have the same coverage by language and market. Which ones are included is declared, and it is assumed that the conclusions hold for them, not for "AI" in general.
The frequency
How often the same set is repeated. It defines what kind of change can be detected: comparing captures that are far apart mixes the effect of an intervention with the simple passage of time.
How it is logged
The format is part of the method, not a later formality. A uniform log lets you count proportions over a set; a pile of loose impressions lets you count nothing.
These decisions turn an informal exercise into a baseline. Without comparison there is no way to attribute a change to something that was done.
The Search Intelligence cycle applied to this problem
The cycle Sageo Estudio works with is Research, Measure, Interpret, Intervene and Measure again. Applied to visibility in AI systems, each step has its own content.
Research
Rebuild the situation before saying anything: what the brand says about itself and what others say about it, what entity the systems describe when you ask them about the category without naming it, and what sources are retrieved repeatedly.
Measure
Run the set of questions on the defined systems, with the defined frequency, and log every answer in the same format. The measurement produces the baseline.
Interpret
Look for explanations for what was observed: why the brand appears in some questions and not in others, why it is mentioned without being cited, why the system associates it with an outdated attribute. Every explanation is a testable hypothesis.
Intervene
Work on the hypotheses: consistency of the entity, content that answers clearly and in a way that can be attributed, structured data, technical accessibility and presence in the external sources the systems retrieve.
Measure again
Repeat the same set, on the same systems, with the same log, and compare against the baseline. The comparison is between proportions across several rounds, not between two isolated captures.
What is logged in each answer
Writing down "we appear" or "we do not appear" lets you learn nothing. For every answer, the following is logged.
- Whether the brand was mentioned and in what exact form: the full name, a variant or a product.
- Whether it was cited with a link to a page of its own, and which one. Mention and citation are logged separately.
- Where in the answer it appears —at the start, in the body, in a final list—: the first paragraph is not the same as the end of a long text.
- With what other brands it appears, in what order and in what context: it is the group the system associates the brand with.
- What sources the system used, with the URL when it is available: it is the most direct clue about where it takes what it states.
- What was said about the brand, not only whether it appeared. A mention with wrong information is a problem, not an achievement.
Along with that, the minimum traceability is kept: question, system, date and time. Without those fields, a log cannot be audited or compared.
Why the measurement is repeated
Repeating serves three purposes, and none of them is optional.
The first is to separate the usual from the exceptional. If a brand appears in one out of ten runs of the same question, a capture that shows it does not prove an improvement.
The second is to distinguish your own change from a change in the environment. Systems are updated and competitors publish: with a single measurement before and another after, any difference may be due to the intervention, to an update or to chance.
The third is to make visible what does not repeat. There are questions where a brand appears steadily and others where it appears sporadically: that difference in stability is a finding in itself.
Methodological honesty: what a small set does not let you say
A rigorous method is recognized by what it states and by what it refrains from stating. These are the precautions that accompany this research.
- A set of questions is a sample, not the universe. It describes the behavior of the system towards those questions, under those conditions. Extrapolating to "all AI" or to "the whole market" exceeds what the sample supports.
- Correlation is not cause. An intervention coinciding with a change in the answers does not prove that it caused it. The coincidence proposes a hypothesis; verifying it requires repeating and ruling out alternative explanations.
- The result of one system does not carry over to another. Each system retrieves different sources and weighs them differently. A finding in one is a finding in one.
- The log has a margin of error. Reading a generated answer involves human judgment: what counts as a mention, when a reference amounts to a citation.
- The time window is declared. Every measurement corresponds to a specific period. Without that piece of data, a result is read as if it were permanent, and it is not.
What is described here is the method, not results. Sageo Estudio does not publish visibility figures for AI systems because there are still no finished and authorized measurements to back them. The distinction matters: a documented method can be reviewed and replicated; a figure without a method can only be believed.
Search Intelligence · Research → Measure → Interpret → Intervene → Measure again
Rate this crystal
How many gems do you give it?
Instead of stars we use gems, which is what forms inside a geode. Your rating helps show which crystals are worth polishing further.
Loading ratings…
Comments
A comment can correct, expand or challenge what the article says. It is published after review.
Loading comments…