Sentiment Analysis Using Snowpark

Introduction

Snowpark now supports Python. It opens opportunities to bring many workforces to Snowflake to use the highly-scalable computations of Snowflake. This article explores how to perform sentiment analysis in Snowpark for Python.

Why Sentiment Analysis?

Sentiment analysis is the process of categorizing opinions expressed in a piece of text, especially to determine if the writer's attitude toward a particular topic, product, etc, is positive, negative, or neutral.

It is very important to clearly understand what customers think about your product and service. It helps immensely to manage the brand's health and brand reputation in the market.

What is Snowpark?

Snowpark provides an API that programmers can use to create Data Frames that are executed seamlessly on Snowflake's data cloud. Using Snowpark, you can write code in a language of your choice, like Python, and can use the Snowflake ecosystem (virtual warehouses) computation to make operations faster and more secure. In the backend, it translates all the Python code into SQL code and executes it.

Libraries for Sentiment Analysis

There are many open-source libraries available in Python to perform sentiment analysis. Text Blob and Vader are two popular libraries that are widely used in many projects. This article explores how to use these two libraries within Snowpark.

Challenges in Using Textblob & Vader with Snowpark

Textblob and Vader libraries are unavailable in Snowflake Snowpark's anaconda repository. Therefore, they cannot be directly imported and used.

Workaround to Use Textblob & Vader

As a workaround, you can download the .whl file from pypi.org and upload it to the Snowflake staging area. Inside the UDF or stored procedure, you can import the wheel file, extract it, and load it to the Snowpark environment. You will also need to import all additional dependencies mentioned in the wheel file.

Steps of Execution

The following demonstrates how to perform sentiment analysis using Textblob.

Connect to Snowflake

import os
from snowflake.snowpark import Session

connection_parameters = {
"account": <snowflake_account>,
"user": <snowflake_user>,
"password": <snowflake_password>,
"role": <snowflake_user_role>,
"warehouse": <snowflake_warehouse>,
"database": <snowflake_database>,
"schema": <snowflake_schema>
}

test_session = Session.builder.configs(connection_parameters).create()


Upload the .whl File

test_session.file.put("textblob.whl", "@my_stage", auto_compress=False, overwrite=True)


Extract and Configure the Wheel File

import fcntl
import os
import sys
import threading
import zipfile
import nltk

class FileLock:
def __enter__(self):
self._lock = threading.Lock()
self._lock.acquire()
self._fd = open('/tmp/lockfile.LOCK', 'w+')
fcntl.lockf(self._fd, fcntl.LOCK_EX)

def __exit__(self, type, value, traceback):
self._fd.close()
self._lock.release()

IMPORT_DIRECTORY_NAME = "snowflake_import_directory"
import_dir = sys._xoptions[IMPORT_DIRECTORY_NAME]

zip_file_path = import_dir + "textblob.whl"
extracted = '/tmp/textblob'

try:
with FileLock():
if not os.path.isdir(extracted):
with zipfile.ZipFile(zip_file_path, 'r') as myzip:
myzip.extractall(extracted)
except:
extracted = path

sys.path.append(extracted)


Perform Sentiment Analysis with TextBlob

from textblob import TextBlob

score = TextBlob(example_string).sentiment.polarity

sentiment = ""
if score < 0:
sentiment = 'Negative'
elif score == 0:
sentiment = 'Neutral'
else:
sentiment = 'Positive'


The polarity score ranges from -1 to 1, where -1 identifies the most negative and 1 identifies the most positive sentiment.

Complete TextBlob UDF

create or replace function find_sentiment(example_string string)
returns string
language python
runtime_version = '3.8'
imports=('@my_stage/textblob.whl')
packages = ('snowflake-snowpark-python', 'pip', 'nltk')
handler = 'find_sentiment'
as
$$
def find_sentiment(example_string):
import fcntl
import os
import sys
import threading
import zipfile
import nltk

class FileLock:
def __enter__(self):
self._lock = threading.Lock()
self._lock.acquire()
self._fd = open('/tmp/lockfile.LOCK', 'w+')
fcntl.lockf(self._fd, fcntl.LOCK_EX)

def __exit__(self, type, value, traceback):
self._fd.close()
self._lock.release()

IMPORT_DIRECTORY_NAME = "snowflake_import_directory"
import_dir = sys._xoptions[IMPORT_DIRECTORY_NAME]

zip_file_path = import_dir + "textblob.whl"
extracted = '/tmp/textblob'

try:
with FileLock():
if not os.path.isdir(extracted):
with zipfile.ZipFile(zip_file_path, 'r') as myzip:
myzip.extractall(extracted)
except:
extracted = path

sys.path.append(extracted)

from textblob import TextBlob

score = TextBlob(example_string).sentiment.polarity

sentiment = ""
if score < 0:
sentiment = 'Negative'
elif score == 0:
sentiment = 'Neutral'
else:
sentiment = 'Positive'

return sentiment

$$;


TextBlob Limitations

TextBlob is simple to use for beginners, but it has limitations. In negative polarity detection, when a negation word is added somewhere in between rather than adjacent to a polarized word, TextBlob doesn't work very well. For instance, it may misinterpret "not best" differently from "not the best and it is an issue."

Using VaderSentiment Instead

The vaderSentiment library handles negative sentiment better than TextBlob. The implementation steps are nearly identical, with only minor code differences:

from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer

sid_obj = SentimentIntensityAnalyzer()
score = sid_obj.polarity_scores(example_string)['compound']

sentiment = ""
if score <= -0.05:
sentiment = 'Negative'
elif score > -0.05 and score < 0.05:
sentiment = 'Neutral'
else:
sentiment = 'Positive'


For positive sentiment, the compound score is ≥ 0.05; for negative sentiment, the compound score is ≤ -0.05; for neutral sentiment, the compound is between -0.05 and 0.05.

End Notes

This article demonstrates how to use TextBlob and Vader to find the sentiment of a sentence using Snowpark for Python. Vader performs better sentiment analysis compared to TextBlob when it comes to negative polarity detection. Your choice between the two depends on your specific use case requirements.

References

• https://docs.snowflake.com/en/developer-guide/udf/python/udf-python-creating.html
• https://github.com/cjhutto/vaderSentiment
• https://textblob.readthedocs.io/en/dev/