Refined Topic Modelling: Why Validation of Topic Models is Critical and How to Do it Right

We’ve been discussing a lot about the advantages and “How to’s” on Topic Modelling. Some of its best use cases are seen for Enterprises in the analysis of customer-induced content as well as Marketeers seeking patterns behind huge and jumbled chunks of information sources. However, amid all that automated identification of topics, it becomes critical for the users to carry out a validation – which may come across as a manual exercise until a breakthrough comes up in the direction.

Applying validation to topic models is an essential and advisable practice to ensure the entire proliferation yields productive and useful results. One link gone wrong can defeat the very purpose of topic modelling, otherwise.

What is Topic Modelling?

Topic modelling is a technique that, using various Python-based packages, automates the interpretation of the text to identify the various topics being discussed in a given set of information or text. In a business scenario, this technique promises to find maximum value for enterprises that receive tons of customer reviews or feedback. Finding common patterns within those reviews can effectively help them form their next business strategy for delivering better customer experiences and enduring sustenance. However, when the data input is huge and assorted, do you think, doing this manually could be a possibility?

This is where these Python Packages help in identification of the commonest and prominent topics being discussed in the entire data file – that combines all the reviews, for instance. This automation is called Topic Modelling. Now, the topic obtained must be validated to ensure the packages are successfully delivering accurate results. Here comes the need for Validation.

What is Validation of Topic Models?

Validation is a process or practice that works best for semi-supervised topic modelling results. It is supposed to add value for researchers who join in new or need further sanity on the topics selected. This need is governed by their real purpose behind performing topic modelling. For instance, if a user wants to identify the topic and write something on this further, it is critical that he refines the topic first.

The algorithm used most commonly for the validation or interpretation of topic models entails a CorEx framework. It makes use of anchor words to map them on top of the source file and hence, effectively identify the key concepts.

Also called cross-validation, this method effectively helps determine how many relevant topics are included in a corpus of documents.

What is Interpretation of Topic Models?

This one comes as a subsequent step for validation. This is more like a macro-analysis of the topics derived through the modelling. Right after the validation of the topic models, you perform an interpretation of how many of them can actually be put to use. This exercise is not an easy undertaking. It entails the excellence of the human brain because you now need to make sense out of different words in the data – which may or may not have a similar meaning. So, you need to connect the dots with a far more legitimate knowledge of the context.

Why Applying Validation on Topic Models is Critical?

As engineers, we must admit that topic modelling is, after all, an exercise performed by a machine or tool, and hence, it cannot replicate or compensate for the efficiency of a human brain. It works against certain logics which may be too rigid and robotic, unlike a human brain – that can qualify a logic in real-time to derive better results. With this limitation in machines, there remain possibilities of overcooking in topic models. It may end up overlooking the redundancy and put together similar-looking or similar meaning words under one token – overarching the scope of topic identification.

However, a deeper analysis of overcooked topics may sometimes add value too. For instance, food and foodie can be two different connotations. But they look similar at the outset. Now, during the validation and interpretation, you will likely become more aware of the inter-relation and see these are two different topics under one umbrella. You can talk about healthy food and also mention the ill-habits or risk of being a foodie.

The Significance of Context

No topic will make any sense to anyone performing topic modelling until the context of the topic is fully derived. Everything is verbose until we know the significance – why are we talking or writing about it. This is a limitation with topic modelling at present. The content of topics cannot yet be derived in the present form. However, with effective validation and interpretation, this part can be covered.

How Validation is Done?

You do that by first identifying the anchor words of topics and putting them together at once place. Now, you segregate them as per the results obtained in the package-based topic modelling.

Right after that, you begin to look through the context of all the anchor words and derive one or two topics out of it. If we circle back to the example of food and foodie, we can clearly see that ultimately, we are directing towards the need for healthy food and the disadvantages of being a foodie. Now, the same set of anchor words, if you look through a hawk’s eye, can find multiple contextual topics through them. When you see multiple references to money, you can steer it to savings, wealth management, abundance, income sources, etc.
So, validation demands a practice too. Strategists must be consulted for the purpose of validation.

References for Further Reading:

Articles:

Interpreting and validating topic models
Using LDA Topic Models as a Classification Model Input

Tutorials:

Cross Validation of Topic Models

eBooks and Research Papers:

Topic Model Validation
Validating the Use of Topic Models for Software Evolution

Summing Up

Applying validations to topic models is a highly effective and advisable practice for serving the end purpose of topic modelling – deriving meaningful insights. It can be done using the help of strategists. However, it underlines or rather, questions the efficacy of the topic modelling packages in determining the relevant and countable number of topics in a document corpus.
Engineers need to pull up their socks and add more value to the topic modelling methods – in such a way that the validation becomes an integral automated part of the process – rather than calling out for manual interpretations.

If you would like to add anything, your comments and suggestions are welcome. If you have questions, rather, we’d be glad to address them. Connect with us through the comments section below.