Published July 2023. updated jan 2024
Disclaimer: translations from Korean provided by Papago and tweaked for better understanding.

For a couple of years, a certain site in Korea publishes a monthly "Brand Reputation Ranking" which claims to show the Brand Value of specific groups and artists (or any product). This longstanding source has been scrutinized so much that, by now, no self-respecting news site or even community publishes their results. In fact, in some social media, the mention of this "ranking" is banned to avoid having to explain over and over why it is nonsense. Examples are Reddit, and even the tabloid-like site Allkpop avoid mentioning it (it also published an article about how useless it is back in 2020). Official news sites like The Korea Herald also pans it out as if it doesn't exist.

But why exactly this is object of so much debate on either its trustworthy or not, and what exactly is it?

A little background

The so called "index" originates from The Korea Institute of Corporate Reputation (한국기업평판연구소), found at reputation.kr and brikorea.com (reputation.kr is a "simplified" and more commercial-oriented version of the main site brikorea.com) which is anything but a Research Institute and is rather a commercial business with copyrights and company secrets that are completely hidden from the public. It is important to keep that in mind because a business have connections, agendas and interference and only with transparency anything a business make will actually have any value. Every Business that generates any ranking or chart must make it transparent and auditable for it to have value, that is how Billboard or Youtube charts work even though they are Businesses. However, there is no transparency in this "Brand Reputation Index". You can't even use it!

In fact, on every single page of the brikorea you will find the same scary warning about the data available:

Just reading through this warning, a few things are clear. They use a system developed inhouse called ALLSNS (which is only mentioned in this warning, nowhere else), it is hard-trademarked (a "trade secret"), and you should be aware you will be punished if you use any of their data. That is not exactly what you would expect from a "Korean Institute" that "researches" brand reputation. In fact, to see data from most of their "reports", you "need" a login (and you guessed it, it is a paid subscription). 

They regard themselves so important, that they actually issue monthly certificates for each Brand that reaches #1 in their analysis. If the image below is cringe, you get it how much they think of themselves:

Official Page for a Girl Group Brand Reputation Ranking

Together with the "Certificate" so that you can, apparently, if you dare, print, laminate, put in a frame and hang in your office:

Checking their credentials, you will also find out that they have no affiliation with any government, official media or research institute, which further clarifies they are an actual business, an aggressive and secretive one at that, and nowhere near a research Institute that they try to pass the image as. But from here, how their ranking works is where things only get more comical. 

First, their highly secretive ALLSNS system mentioned on their warning actually dispenses any introduction, since what it does is on the name. SNS, in Korea, relates to all types of social media. For instance, Youtube is a SNS (its short for Social Networking Service). Therefore, ALLSNS is merely a scrapper of Social Media posts. And if that seems a stretch, that their fancy system is just a scrapper of Social Media Data, all you need to confirm is to hop on their "Analysis" (or "reputation" explanations), as in https://brikorea.com/contents/analysis_1.php:

This explanation remains the same since this Brand Reputation Ranking system was launched years back, and absolutely no further explanation was ever given because they claim it is a trade secret, so don't expect any transparency or actual sources here. But from their site, they explain they evaluate four indexes:

Which is basically merely a counter of how many times people will interact with a social media post, which in most Analytics systems is called "engagement".

Again a very simple concept. This is the amount of "participation" (engagement) that takes form as communication (textual replies, while the above participation index probably includes other forms of engagement such as views and likes) which they recognize it can be positive or negative (but no further explanation on how possibly they can get the subtleties and contextualization of a positive or negative evaluation). This is one of the main points of debate of the validity of their whole system and we will address it later.

This seems to be simply a weighting of what type of media is used in the engagement, and again, mentioned that only the "positive" evaluations from these medias are indexed for the brand. Keep that thought.

Up to now, all of the three first "metrics" are one and the same, with tweaks: How much people interact with their SNS, the "quality" of those interactions in the form of positive and negative evaluations (not really, as we will see), and the importance of the media used towards that interaction. That is only a responsive metric: users responding to SNS posts from the brand, but no coverage of what people are actually talking about the brand outside the brand's own accounts. That is where this seems to enter, this is an index of how much people mention the brand outside of the brand SNS, showing that not only channels from the brand itself are monitored, but from all the SNS covered: if a person, anywhere, mentions the brand (or possibly keywords associated to the brand), this should be picked here. 

At this point another issue is bots and automated responses. There seems to be no mention on being able to detect and filter out those (most likely they don't even bother), therefore purchased followers, mentions and so on are clearly just getting in the mix.

Also notice the lack of how much each of these metrics affect the total index, or how each of those are even calculated. We can only guess, and since mentions and engagement are completely different in nature, its impossible to have a single calculation that fits all types of brands, even inside the same industry such as K-Pop (Groups and Soloists have different user profile interactions, Males and Females too, for instance)

To sum up, this is a system that mashes huge volumes of data from Social Media (SNS), detects the engagement and interaction of users on the brand channels, which types of media get the engagement, and whenever the brand is mentioned outside of official channels, and claims to be able to differentiate positive and negative evaluations of the brand, hopefully filtering out non organic engagement. We only know that this is done on "ALLSNS" platforms (which platforms?, we don't know, but probably the most relevant in Korea that allow scrapping).

This approach of processing large amounts of data is indeed what the company does, as described in their own "Brand Reputation Effect" basic page:

In the mist of all the technical lingo which we won't get into, nothing is actually about ranking and quality interaction, and the only thing we can get it is that they indeed do process large amounts of data - but with no quality or transparency whatsoever. Besides, none of the mentioned technologies have a single thing to do with a "market through corporate management innovation" when all they do is basically what any Analytics system do (like Google Analytics or even Youtube Analytics). Even if you where to argue that, by detecting engagement in different channels, you could conclude which convert more engagement, the fact remains that you would actually need to make a data analysis of each brand consumer profile to evaluate which channel is more important - for instance, Visual Channels are more important for companies where Image is more important, while review and positive/negative contexts are more important for companies where a feedback on their different departments is important. There is no one-size-fit-all, and certainly just a counter of engagement and mentions is not even that.

Engagement on Social Media is important for some companies, but not for others. In fact, today we live in a moment where several brands are toning down their use of Social Media exactly because they realized that, more often than not, they can cause more damage to their brand by posting the wrong thing, than benefiting from anything they can possibly post. Social Media itself is not a one-size-fit-all solution for engagement with your consumers, and some brands are doing just fine without any Social Media presence at all. If Social Media have different importance for each company, for each marketing strategy, a single counter method is even less relevant for individual consumers. If a Brand can be Top-of-mind without social media presence, how can you measure Brand value only by Social Media? This is also true for Korean Artists. What attract investors in their "brand" changes from Artist style to Artist style. Singers will be approached by a different set than Idols, and even among them, the art style of each will affect which companies will look out for them. The most talked about Artist might have no way to be connected with a product and, therefore, have no Brand Value.

They finally end by stating that Brand Reputation is important, and indeed it is, however their methods are completely nonsense, it is just a counter, a basic brainless statistic, a counter that can't cope with context, that can't cope with what type of engagement is actually important to each different brand, and most important, that just a counter of interactions cannot possibly tell you about reputation. If a brand is undergoing a large scandal, it will get a huge amount of "participation", "communication", "media" and "community" interactions, but it will certainly not be a good thing, and detecting if all of those interactions are positive or negative are, as far as 2023, impossible. Even the top technological advancements in predictive and generative AI have difficulty with subtle and context-heavy evaluation. Agents such as ChatGPT or Bard can easily be negatively influenced by those minute characteristics in human speech, and lets not forget that no system today can identify Sarcasm, Irony or Parody.

In short, its just fancy words to fool unknowingly layman, as any statistics must be accompanied by a analysis and result research to be more than crunched numbers. In western terms, the closest this can be is a system for Search Optimization, which does the exact contrary of evaluate the brand value: it generates which terms and keywords should be used to increase the brand click-through on the internet.

What "Brand Reputation Index" really is.

The so called BRI is actually just a measurement of how much a brand is being talked about, but it has in no way the capacity to specify if that is positive or negative as they claim. Processing big data from all SNS is on itself a monumental task, but to actually process that, in real time, with monthly releases, using any form of heuristics to - hypothetically - derivate positive and negative reviews, while simultaneously understanding what is positive or negative to each different brand or marketing approach, would require technologies that are, so far, only starting to emerge and are not even mentioned on their site. Machine Learning, Deep Learning, Heuristics or any form of AI is not used, and their actual algorithms are classed as "trade secret" despite clearly being simple scripts with counters. If they were to even claim that their systems are improving (they don't, and it indeed seems that the results of their rankings reflect similar values for years) it would also raise the prospect that older results were invalidated, and therefore giving out "Awards" and "Certificates" from a method that will soon become obsolete is obnoxious, if not plain stupid.

This measurement can, in no way, be associated with value or prediction of success. It is just about the past, and a poor measurement at that. To better understand how bad it is, think like this: It is as if we measured the value of a Youtube Video by the sum of comments, likes and dislikes. Which would mean the most valued and important video in Youtube are their infamous year Recap videos.

What "Brand Reputation Index" wants to be.

There is a reason why, in most western countries, there is no such thing as "Brand Reputation Indexes". The reason is simple: because there is already one simple metric, easy to understand and universally agreed as valid that do what the so called BRI does, and it doesn't require Big Data, AI or huge processing power: Top-of-mind surveys. 

Top-of-mind surveys are used, and respected across all industries, for decades. It uses real surveys and statistical modeling for sampling that have real meaning, real implication, and are based on actual non-contextual, non-subtle answers. If you have never answered a Top-of-mind survey, they are very easy to do, and very simple to understand how they work. Researchers will ask the sample group what is the first brand that (positively) comes to mind in each subject. For instance, researchers will go in population-dense areas, or neutral websites, and ask "What is the first good K-Pop Girl Group that comes to your mind?". It is easy to understand why that carries a lot more value than simply counting interactions and mentions. 

Every year (or other intervals), Renowned and Reputable (Irony intended) research institutes will conduct, as a trend or per request, Top-of-mind researches on different topics and publish them on open papers with all statistical data available for scrutiny, no trade secret, no suing if you publish them (because, for it to be trustworthy, the source must be clear). Often there will be no "Awards" or "Certificates", just the plain and simple data, but information that carries weight enough that the companies that are on the Top 3 will usually boast about it, and that does mean something. 

Like, Comment and Mention counters? They don't mean anything without context, proper brand-to-brand validation and understanding the nuances of what is being said. Its just garbage. No wonder not a single company in Korea actually "boasts" about results from The Korea Institute of Corporate Reputation.