{"id":93751,"date":"2022-04-06T19:58:31","date_gmt":"2022-04-06T19:58:31","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/?p=93751"},"modified":"2022-04-27T21:27:23","modified_gmt":"2022-04-27T21:27:23","slug":"deeper-insights-topic-modeling","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/databases\/sql-server\/bi-sql-server\/deeper-insights-topic-modeling\/","title":{"rendered":"Finding deeper insights with Topic Modeling"},"content":{"rendered":"

Topic modeling is a powerful Natural Language Processing technique for finding relationships among data in text documents. It falls under the category of unsupervised learning and works by representing a text document as a collection of topics (set of keywords) that best represent the prevalent contents of that document. This article will focus on a probabilistic modeling approach called Latent Dirichlet Allocation (LDA), by walking readers through topic modeling using the team health demo dataset<\/a>. Demonstrations will use Python and a Jupyter notebook running on Anaconda. Please follow instructions from the \u201cInitial setup\u201d section of the previous article<\/a> to install Anaconda and set up a Jupyter notebook.<\/p>\n

The second article of this series, Text Mining and Sentiment Analysis: Power BI Visualizations<\/a>, introduced readers to the Word Cloud, a common technique to represent the frequency of keywords in a body of text. Word Cloud is an image composed of keywords found within a body of text, where the size of each word indicates its frequency in that body of text. This technique is limited in its ability to discover underlying topics and themes in the text, because it only relies on the frequency of keywords to determine their popularity. Topic modeling overcomes these limitations and uncovers deeper insights from text data using statistical modeling for discovering the topics (collection of words) that occur in text documents.<\/p>\n

Topic modeling with LDA<\/h2>\n

Text data from surveys, reviews, social media posts, user feedback, customer complaints, etc. and can contain insights valuable to businesses. It can reveal meaningful and actionable findings like top complaints from customers or user feedback for desired features in a product. Manually reading through a large volume of text to compile topics that reveal such valuable insights is neither practical nor scalable. Furthermore, basic tf-idf schemes<\/em><\/a> and techniques like keywords, key phrases, or word cloud, which rely on word frequency, are severely limited in their ability to discover topics.<\/p>\n

Latent Dirichlet Allocation (LDA) is a popular and powerful topic modeling technique that applies generative, probabilistic models over collections of text documents. LDA treats each document as a collection of topics, and each topic is composed of a collection of words based on their probability distribution. Please refer to this paper<\/a> in the Journal of Machine Learning research to learn more about LDA.<\/p>\n

Demo environment setup<\/h2>\n

After following instructions from the \u201cInitial setup\u201d section of the previous article<\/a> to install Anaconda and set up a Jupyter notebook, return to Anaconda Navigator<\/em> and launch CMD.exe Prompt<\/em>.<\/p>\n

\"An<\/p>\n

Figure 1. Launch CMD.exe Prompt <\/strong><\/p>\n

Install spaCy<\/h3>\n

spaCy<\/em><\/a> is a free, open-source library for Natural Language Processing in Python with features for common tasks like tagging, parsing, Named Entity Recognition (NER), lemmatization, etc. This article will use spaCy<\/em> for lemmatization, which is the process of converting words to their root. For example, the lemma of words like \u2018walking\u2019, \u2018walks\u2019 is \u2018walk\u2019. Lemmatization uses contextual vocabulary and morphological analysis to produce better outcomes than stemming. You can learn more about lemmatization and stemming here<\/a>.<\/p>\n

Run the following code in the CMD.exe<\/em> Prompt, to install (or update if already installed) the spaCy<\/em> package, along with its prerequisites.<\/p>\n

\tconda install -c conda-forge spacy<\/pre>\n
 <\/p>\n
\"An<\/p>\n
Figure 2. Install spaCy package <\/strong><\/p>\n
Run the following code in CMD.exe<\/em> Prompt, to download en_core_web_sm<\/code> trained pipeline for English language<\/p>\n
\tpython -m spacy download en_core_web_sm<\/pre>\n
\"An<\/p>\n
Figure 3. Download en_core_web_sm trained pipeline for English<\/strong><\/p>\n
Please refer to spaCy Installation guide<\/a> for detailed instructions and troubleshooting of common issues.<\/p>\n
Install wordcloud<\/h3>\n
The wordcloud package is used to create the word cloud visualization, where the size of each keyword indicates its relative frequency within a particular text document.<\/p>\n
Run the following code in CMD.exe Prompt to install the wordcloud package.<\/p>\n
\tconda install -c conda-forge wordcloud<\/pre>\n
\"An<\/p>\n
Figure 4. Install wordcloud package<\/strong><\/p>\n
Restart the Jupyter notebook Kernel so it can use the newly installed spaCy and wordcloud packages.<\/p>\n
 <\/p>\n
\"An<\/p>\n
Figure 5. Restart Jupyter notebook Kernel<\/strong><\/p>\n
Install Gensim and pyLDAvis<\/h3>\n
Gensim<\/a> is an open-source library for Natural Language Processing focusing on performing unsupervised topic modeling. The demo code in this article uses features specific to genism version 3.8.3 and may not work as expected with other versions of this library.<\/p>\n
pyLDAvis is an open-source package to build interactive web-based visualizations<\/p>\n
Run the following commands in the first cell of the Jupyter notebook to install genism and pyLDAvis<\/p>\n
!pip install gensim==3.8.3\r\n!pip install pyLDAvis<\/pre>\n
\"An<\/p>\n
Figure 6. Install genism and pyLDAvis from Jupyter notebook<\/strong><\/p>\n
Import Natural Language Processing modules<\/h3>\n
This section introduces readers to modules used for Natural Language Processing (NLP) in this article<\/p>\n