Beyond text: Marketing strategy in a world turned upside down

Understanding the consumer is central to marketing. Not least, understanding is central to the acquisition and retention processes. Marketers often face the challenge of gaining access to appropriate data and tools to enhance their understanding. The increasing availability of different data sources has enabled firms to analyze consumer behavior and purchase processes more effectively. Throughout the consumer journey, from before being exposed to the product to making the final purchase decision and beyond, the consumers’ thoughts, feelings, attitudes, and behaviors change with the firm’s marketing mix and their particular needs.

Consumers’ psychological processes are often not explicitly expressed and therefore cannot be easily assessed. Yet, managers can make inferences. Consumers leave traces of their feelings and attitudes in their verbal and non-verbal expressions and consumption behaviors. Managers generally have access to, and could utilize, the data generated through interactions between the firm and its consumers. Consequently, firms could grasp information about the consumer process of “gathering information, making a purchase, receiving the product, making a return, and receiving post-purchase service” (Cui et al., 2021). However, as these processes generate a large body of available data including consumer conversations, transaction data, customer relationship management records, etc., the challenge is to find and utilize the thoughts and feelings hidden within a massive quantity of data.

In addition, consumer touchpoints are not equally accessible across different channels (Du et al., 2021). This introduces notable challenges for marketers aiming to exploit omnichannel information. To help with this, many firms are turning to big data technologies and machine learning techniques to increase task scalability and efficiency.

Data generated outside the direct interaction between the firm and consumer contains valuable information about market trends (Klaus & Maklan, 2013). With online platforms, consumers express their thoughts, and influence others, through word-of-mouth. These information exchange processes impact consumer information searching and decision-making. However, many firms struggle to (a) monitor and influence online exchanges, and (b) strategically respond to current market trends, due to the challenge of collecting and analyzing relevant data. In addition to consumers' and regulators' concerns surrounding data privacy and data security, problems also emerge for those marketers who are deterred by the technological barriers from utilizing appropriate analytical tools or constrained by their budgets (Wedel & Kannan, 2016).

Analyzing consumer data poses significant technical difficulties, especially when integrating across unstructured data types like image, video, and audio. These challenges arise given the different underlying structures (Grewal et al., 2021). Embracing the insights of Balducci and Marinova (2018), we recognize that each data type can be multifaceted and manifest at the same time. This not only highlights the richness of the data but also underscores the challenges of weaving together different encoding layers such as pitch, volume, phoneme, and the words used. Despite these challenges, integrating these data types offers the potential to develop superior models and novel theoretical frameworks. A potential exemplified by the machine learning data fusion approach of Boughanmi and Ansari (2021).

Competitor data is crucial to strategic decision-making. Firms often struggle to directly observe or gather insights on their competitors' underlying marketing strategies which, by their nature, are concealed. They can, however, deduce these strategies from the tangible, observable actions—the marketing behaviors—displayed by competitors. These behaviors, evidenced in aspects like promotional discounts and advertising content, as well as the discernible reactions from consumers, provide a blend of structured and unstructured data that managers can leverage. Such competitor data also offers an opportunity for firms to understand the effects of different marketing strategies without experimenting with every option themselves and absorbing all the risks of such experiments. Managers can learn from their competitors’ successes and failures.

However, it is costly and difficult to monitor and identify successful competitor marketing elements from massive amounts of competitor data and then make predictions based upon this. It is especially difficult to isolate the effect of specific marketing tools. For example, is an advertising video successful because of its acoustics, color palette, a particular image object, or a combination of these? Uncovering insights may require advanced machine learning tools, which deters marketers who are unfamiliar with what is available. While these tasks seemed impossibly difficult before big data and user-friendly machine learning tools, when employing beyond text data and cloud platforms a marketer can now hope to glean helpful clues to such questions.

Marketers can improve their offerings by better understanding consumer feedback and adjusting their strategies given findings about how image, video, and audio messages influence consumer behavior. Villarroel Ordenes et al. (2019) found that the mere presence of images and video in consumer generated content encourages sharing behaviors by other consumers. Product presentation, including design aesthetics, helps marketers to transmit messages, and Liu et al. (2017) proposed an image-based method to quantify such product attributes.

Managers can also quantify and evaluate company logo attributes. For example, Satomura et al. (2014) developed a quantification method to measure consumer confusion about copycat logos. In terms of advertising content, Xiao and Ding (2014) showed that spokesperson faces in print advertisements have a significant impact on viewers’ reactions to the ads, and called for managers to devote efforts to select the right face for a product category. Brickman (1980) established that the voice pitch in TV advisements could be used to predict purchases. These findings have encouraged marketers to customize messages for their audiences.

Managers should also accurately assess their own marketing performance. They may struggle to identify a key product differentiator, as many design elements may be hard to put into words. This may cause miscommunication among departments within the firm and lead to the inability to explain the product and attract consumers. It is challenging for marketers to quantify visual attractiveness and innovativeness and predict market performance of advertising before trials. With increasing adoption of omnichannel marketing, problems of data integration and marketing attribution escalate (Cui et al., 2021). As the consumers are exposed to multiple contacts from the same firm, marketers find it hard to accurately attribute effects and assess the impact of marketing tools at various touchpoints of the consumer journey. This leads to difficulties in evaluating and optimizing marketing spending. See Table 1 for some objectives of marketing, challenges associated with these, and how getting beyond text can help.

Consumers display their meals; marketers share carefully judged product photos; dating app users package themselves to create appealing personas just on the right side of the truth. Images matter in marketing, and automated image analyses seem set to only increase in importance.

There are two major types of image analysis tasks, a.k.a., image mining.

Image classification and object detection are important techniques within “computer vision.” The algorithm used might “look” at a picture, see a pixel grouping, and treat this as a feature. Computers use multiple image features, e.g., pixels, positions, lighting, camera angles, to predict what an image represents using supervised or unsupervised approaches.

Image classification and object detection have been widely applied in marketing research. Researchers might seek a product placement within an image to test a relevant question, e.g., does having a soda in the fore- or background of a picture change engagement. The most obvious benefit of using an algorithm is scale. It is impractical for humans to view thousands of images, never mind have consistent judgement of performance. Nor can we ensure consistency between different human viewers. In comparison, algorithms never tire and are always consistent, reducing noise.

Of course, some uses of beyond text analysis will be controversial. Marketers could predict personal details of consumers from the images they post. The consumers though may not wish to share such details. Individual consumers could be targeted through facial recognition. We see that as a reason for academics to embrace the field. To push back on inappropriate uses, one has to understand what types of image analysis are possible, and how this can be conducted.

One can expect that managers would be able to benefit from the large amount of data that consumers post. Consumers can be seen interacting with the products in much more novel and less constrained ways than might be encountered in a focus group. The manager will also have access to the interactions between the firm and the consumer. For example, images attached to tweets directed at the firm. Many firms classify their products using images to aid consumer selection. Social media platforms, e.g., Instagram, allow tags that can be used to better understand the images. For example, by building a custom model for classifying product offerings, retailers can use social media to identify which influencers wear a firm’s clothing.

Hartmann and colleagues (2021) find that purchase intentions and brand engagement increase as consumers share selfie images holding a branded product. Dynamic monitoring of images posted can be used to analyze engagement and design optimized messages. Commercial APIs have quantified happiness, naturalness, and sadness (Lee, 2021). Improving understanding of images can further consumer engagement, facilitate positive brand exposure, expand the consumer base, and, ultimately, should lead to favorable financial performance.

Consumers can even perform such analyses themselves to get ahead of trends. One might expect those on dating apps, or applying for jobs, to be able to seek advice from an algorithm on what image will help ‘sell’ their profile.

Note much of this beyond text data, here image, will come from outside the direct consumer-firm interaction. Social media monitoring allows the firm to observe images being shared by consumers across the marketplace.

Locating pertinent images in order to comprehend competitors' strategies is relatively straightforward compared to searching for direct information about the competitors’ strategy. Marketers can ascertain the consumer usage and settings highlighted in competitor advertising and deduce the intended target market by analyzing the individuals featured in the advertising. Images in competitors' advertising campaigns can give insight into their focus, and marketers can extrapolate a competitor’s marketing strategy before devising an appropriate response. Marketers can even group competitors based on the themes conveyed in their advertisements.

In addition, marketers can scrutinize product designs to glean insights into their competitors' strategy. Examining the themes portrayed in the design of competitors' products can provide valuable information about their promotional tactics. Various images and related items, such as logos, color schemes, and packaging visuals, are commonly utilized in business. By analyzing these elements, marketers can deduce the relative positioning of their competitors.

Managers can apply similar methods to their own actions. Some intriguing questions include:

Similar ideas apply to all other forms of company image, such as logos. Combining images with text can allow the marketer to identify the relationship between consumer commentary and image use, allowing for a measure of creative effectiveness. Academics have already started working in the area. One study was able to extract perceptions of a brand through user generated data (Klostermann et al., 2018). Researchers have used images to create brand prototypical collages (Dzyabura & Peres, 2021), predict restaurant survival (Zhang & Luo, 2022), and evaluate product aesthetics (Burnap et al., 2023).

Regarding the technical aspects of image analysis, investigating advertising images, product designs, and company logos mostly centers around (1) deviation from the population average to group by visual similarity, (2) design typicality, or (3) facial recognition. To assess similarity, a RGB histogram uses pixel level color distribution; texture histograms describe the pixel level distribution of pre-defined patterns, while Gabor Features use a linear filter to analyze the frequency and orientation of content and recognize patterns (Dzyabura et al., 2023). Photo morphing or principal component analysis can create a prototype average picture (Landwehr et al., 2013). Design typicality may be based on the difference between each product and the average (Landwehr et al., 2011) and used as a component to quantify product design aesthetics (Liu et al., 2017). Studies also use a machine learning facial recognition technique, Eigenface, which creates an average face which can be compared to any other face (Xiao & Ding, 2014), thus improving advertising efficiency.

As a demonstration of how cloud platforms could be used for image analysis, we will give a detailed roadmap on how to train and test the algorithm to classify a specific image, e.g., supervised learning with three platforms: AWS, GCP and Azure. The tested platforms all use CNN as their image classification algorithm, consistent with the recent trends in marketing research. Before analysis one must import the data, e.g., images. (See Appendix 2 for sample code for retrieving images from the web). Note a general rule is that images uploaded should be similar to those later assessed to enhance prediction accuracy. For example, if the model is trained on nighttime images, performance on daytime images will likely disappoint.

Amazon’s image recognition service can be accessed through the AWS console or an API. To begin creating the model, specific preparation of images within a defined domain is required. It is advisable to include a minimum of ten images depicting the object of interest in diverse lighting, backgrounds, and resolutions comparable to those utilized with the model. Amazon Rekognition Custom Labels employs automated machine learning techniques to create a customized model. To accomplish this, the user can generate a project via the console or the API.

Images can be imported from four different locations. For a small dataset, uploading images from a local computer directly to the console is convenient. Otherwise, we recommend sorting the images into folders and importing these into an Amazon S3 bucket. Amazon SageMaker and existing Rekognition Custom Label datasets can also be imported. Images can be labeled through the dashboard while images uploaded through an S3 bucket can have labels assigned through folder names. There are two types of labels for classifying objects: scenes (the environment around an object, e.g., a beach), and concepts (the specific image, e.g., a coke can).

To train a model, select the project and training set. AWS Rekognition Custom Labels offers several ways to create a test set. The easiest approach is to use the built-in option which automatically splits the dataset into 80% for training to learn the associations between labels and images and 20% for testing. The AWS console provides a summary of the training results with metrics to judge model performance. When trained, the model can be applied to new image datasets using the Amazon Resource Name (ARN). To do this, one can start the model in the console with the “Use Model” tab and apply it to other images.

The Google Cloud Platform’s GCP Vertex AI is a machine learning model builder for image classification accessible through the GCP Console or an API. GCP has a similar requirement as Amazon AutoML, image datasets of at least 10 images. In the console, select Vertex AI to create a custom model with a new dataset and select single-label (one label per image) or multi-label classification. Images can be imported either by uploading images from a computer directly into the console or uploading images into GCS (Google Cloud Storage) and supplying Vertex AI with a CSV file containing the image paths. Only 500 images can be imported through the console, so upload via the CSV file route if you have a larger dataset.

Once all the images in the dataset are labelled, GCP automatically splits the images (80/10/10) into three sets, TRAIN, VALIDATION, and TEST. Users can check which images will be used for which purpose and customize this, assigning images to groups by editing the CSV file and reimporting. Vertex AI provides metrics to determine model performance overall and for each label. The first graph shows the trade-off between precision and recall at different confidence thresholds. (The metrics are explained later). The second graph shows how different confidence thresholds would affect the precision and recall metrics along with true and false positive rates. Although Vertex AI does not provide the precision and recall estimates for each label, it provides a confusion matrix that shows the percentage of times each label was predicted correctly. This could be used to calculate these performance metrics.

The third platform is Microsoft’s Azure Custom Vision. This classifies labels based on visual characteristics and allows users to build and deploy customized image classifiers. Azure Custom Vision can be accessed through a web portal or the Custom Vision SDK (software development kit). To use the web portal, create a new project, and select “Classification” under Project Types. Azure distinguishes between “Multilabel”, multiple tags per image—appropriate given different objects in an image, and “Multiclass.” The later allows only a single tag per image. After creating the project, add images and assign labels before uploading and training the model. Azure’s algorithm will train the model and evaluate the classifier. The Custom Vision Service uses the k-fold cross validation process to train the model reporting precision and recall.

Platforms may excel at different tasks. Running an image dataset through its free image analysis is a way to test a platform’s abilities. We used three sets of data downloaded from Kaggle.com to test the image classification performance of the three platforms. The first dataset of 17,760 images showed how platforms recognize images with vehicles in them or not. The second dataset contains 11,385 images with 24 tags as a combination of color (black, blue, brown, green, red, and white) and clothing (dress, pants, shirt, shoes, and shorts). Limitations in storage with the free tier meant both GCP and Azure could only train 5,000 images of the apparel dataset. We, therefore, only looked at images with tags limited to those featuring black apparel. The last dataset requires the platforms to identify real (versus fake) faces in 2,041 images. (See Table 3 and further details at https://​github.​com/​BeyondText/​Beyond_​Text).

What can marketers use these cloud services for? We tested the relative proficiency of the platforms in tasks of varying complexity, encompassing the identification of vehicles, clothing attributes, and facial image authenticity. Marketing uses include basic object detection functions (e.g. detection of vehicles) in user generated content. Marketers can search for insights that involves the interaction of object detection (e.g. apparel) and object visual characteristics (e.g. color). We tested a computationally demanding task, real face detection, which has received significant attention within, and outside, marketing. Knowing an image’s contents allows marketers to investigate consumer preferences relating to, for example, Instagram posts. Marketing researchers studying color’s impact on shopping (Crowley, 1993) can do so at scale. Investigating the different attitudes that consumers exhibit towards human versus artificial agents could also yield novel insights (Miao et al., 2022).

All models report overall precision and recall so we compared these metrics. We also analyzed performance on each tag, e.g., Black Dress, looking at precision, recall and confidence threshold.

Precision measures the fraction of correct predictions (true positives) divided by all true or false positives. Thus, precision measures the percentage of those identified as having a characteristic who actually had the characteristic. \(Precision =\,\frac{\#\, Of\, True\, Positives}{\# Of\, True\, Positives\,+ \#\, of\, False\, Positives}\).

Recall provides insight into the fraction of the test set labels that were predicted correctly. Thus, recall measures the percentage of those who truly have a characteristic correctly identified as having it. \({\text{Recall}}=\frac{\#\, Of\, True\, Positives}{\#\, Of\, True\, Positives\,+ \#\, of\, False\, Negatives}.\)  

F1 score is an amalgam of prediction and recall measuring average model performance for each label or dataset. A higher F1 score indicates that the model performs well for both precision and recall. F1 Score: \({F}_{1}=\frac{\#\, Of\, True\, Positives}{\#\, Of\, True\, Positives\,+\frac{1}{2}(\#\, of\, False\, Positives\,+\#\, of\, False\, Negatives)}.\)  

Confidence score measures the assessed probability of correct prediction. If precision is more important (i.e., that images identified as having a characteristic must actually have it), a confidence threshold should be set higher. The threshold can be set lower to try to identify all images with a characteristic even at the risk of identifying some without the characteristic.

For the vehicle dataset, which is relatively simple, all the cloud platform tools performed very well (see Table 4). AWS performed best with a precision and recall rate of 100%. The three platforms all performed very well with the apparel dataset, with AWS having a small edge.

A clear strength of AWS was that its free tier could handle all tags in the apparel dataset. Appendix 3 shows performance by individual tag—i.e., detecting a specified object. The faces dataset saw meaningful differences emerge. GCP did best, overall precision and recall rate of 97.0% and 98.5% respectively. AWS’ overall precision was 66% but at 86.5% overall recall was better. Azure’s overall precision and recall rate were both only 66.4% but at a higher confidence threshold. That its confidence threshold can be changed gives Azure helpful flexibility.

GCP’s superiority in distinguishing real from fake faces suggests strength in facial recognition. AWS presented the best performance for the vehicle and the apparel categories. Overall, all platforms demonstrated satisfactory results, and the performance differences in the non-face categories are not significant enough for marketers to favor one platform (see Table 5).

Their free tier services differ. AWS offers 5,000 images per month. GCP offers only 1,000. Azure offers 5,000 images per project and two projects per account, so marketers can use Azure’s free service for bigger projects. The storage options offered by these platforms require marketers to consider their particular needs given varied strengths (see Fig. 1). To increase computing capacity, a marketer can always opt for the pay-to-use tiers based on their individual needs.

Two platforms provide a wide range of image analysis beyond image classification. Azure Custom Vision offers object detection to tag pre-trained objects. AWS Rekognition offers an object detection function and many others: facial comparison and search to analyze similarities of a face against another; face detection and analysis to identify a face and its related attributes (i.e., eyes, glasses, and facial hair); celebrity detection to recognize famous faces; labels from pre-trained models (i.e., brand logo, object labels); and text detection. Marketers with complex image analysis needs may find AWS the most versatile.

Video is becoming increasingly prevalent in social media. Analyzing such content will be a significant aspect of marketing, both academic and managerial, in the near future. Furthermore, as consumer data is captured in increasing amounts, video content can aid in more sophisticated eye-tracking type tasks while in-store behavior can also be monitored. For example, affective states have been deciphered using eye movement tracking (Zhou et al., 2021) and variation from neutral emotion facial features (Lu et al., 2016). Kawaf (2019) proposed a screencast videography recording method to allow managers to understand the consumer digital experience. Linking video of behavior to actual sales, can illustrate effective marketing choices.

Social media platforms offer new data sources, such as unboxing videos, where consumers and influencers offer opinions on products and packaging that can appear unfiltered and genuine. Such phenomena of the social media era offer valuable opportunities for greater understanding. As new trends emerge, those who quickly observe them, both academics and managers, will gain fresh insights from novel data sources. Capture of such data does raise public policy privacy concerns, which marketers must be at the forefront of addressing. Understanding the prevalence, and strengths and weaknesses, of methods can inform such work.

Video, including live streaming, poses practical challenges beyond still images. This is, perhaps, why it has received less attention. Analysis of video data starts with basic information including length, speaking rate, and video duration. Sentiment analyses can be applied to text transcripts accompanying videos (Zhou et al., 2021). The overarching objective of video classification is to determine actions within a scene. One approach is to convert the video into a massive dataset of static images. As such, many video classification tools employ the same basis as image classification and object detection e.g., AWS Rekognition. For example, Zhou et al. (2021) analyze online course videos by extracting one frame every ten seconds and applying a Microsoft image emotion recognition model. Other research compared facial expressions in selected frames to emotion categories (Lu et al., 2016). Facial tracking and smile classifier software has been used to code viewer response to advertisements, to evaluate their attractiveness and persuasiveness (Teixeira et al., 2014). Eye tracking technology has assisted marketing researchers to gain a better understanding of attention to advertisements (Brasel & Gips, 2008; Pieters & Wedel, 2012; Pieters et al., 2010).

Another approach is to analyze video holistically, such as, measuring variation between frames. For example, the work of Li et al. (2019) examined how stimulating a music video is using pixel-level distances between frames, to explain the success of pitches for funds. Others have separated foreground (speakers) from background video, applying algorithms, such as OpenCV (computer vision), to calculate motion, magnitude, and direction (Zhou et al., 2021).

Table 2 summarized video analysis tools. Note, GCP Video Intelligence offers custom models, but AWS and Azure do not.

Video analysis has great potential when applied to social media and consumer feedback through user generated content (UGC). Marketers could gain valuable consumer insights by asking questions about the content of social media videos. What do consumers share, what messages do they send within their videos, and how do these differ between and within groups? How consistent are the messages any consumer sends? The vast quantity of UGC gives marketers the opportunity to uncover implicit consumer sentiments reflected through video-based reviews, including YouTube review videos. Marketers can seek to understand (Liu et al., 2020a), and visualize (Dzyabura & Peres, 2021) how their brand is perceived.

Marketers can enhance their persuasiveness with a better understanding of consumers collectively and at the individual level. Evaluating their communications can aid in assessing the accuracy of their perceptions of consumers’ interests and preferences and enable markers to adapt messages to their intended audience. With advanced facial recognition and analysis tools, marketers could access information on recipients’ real time reactions to a transmitted message. For example, Teixeira et al. (2012) measured viewers’ real time emotional responses and focus levels with eye movement tracking and automated facial expression detection. With a better understanding of the impact of its own marketing, a firm’s managers can leverage video to provide better training for their salespeople.

On the competitor side, video advertising can be accessed by all players in the market and consumer reaction is also publicly available online. This has the potential to fundamentally change the data advantage held by marketers compared to that held by their competitors. The ability to mine publicly available video and consumer reaction removes much of the privileged access that a marketer has over their ‘own’ data. As a result, all parties will be on a more equal footing, leading to a new competitive norm. Any shift in market power is not clear yet. A move towards the use of non-proprietary data may induce greater competitive intensity. Alternatively, interrogation of these new data sources might so heavily reward those who can more effectively use the insights that they find it a source of differentiation, and their unique capabilities could then limit the impact of competition.

Amazon Rekognition Video detects objects, scenes, celebrities, text, activities, and inappropriate content from stored and streaming videos. AWS Rekognition does not allow users to build custom models, rather they utilize pre-built operations which include label detection, content moderation, text detection, face detection, and face search. The two major ways to access the Rekognition API are through the AWS CLI (Command Line Interface) and through videos stored in Amazon S3 buckets. The GetLabelDetection operation from AWS Rekognition Video returns a JSON (JavaScript Object Notation) object, useful for data interchange, containing information about the labels created for objects observed. (Appendix 4 gives sample code).

GCP Vertex AI Video has three categories. Video action recognition helps identify the human action or activities within the video. Classification allows users to train a custom model to classify videos, or specific shots and segments. Video Object Tracking detects objects in a video. The API can be accessed through the GCP console or the command line. The first step to uploading videos is to sort the videos into their respective tags. To do this, the analyst should sort these videos into folders with the tags as the folder name. This will make it easier to create a CSV file for the Video Intelligence API console to import the videos. Videos can only be imported in the form of a CSV file through Google Cloud Storage. This involves creating a GCS bucket to house the videos and the CSV files to be imported. Do label the images in the CSV files before importing them.

The Microsoft Azure Video Indexer for Media comes with models trained to extract audio and video data from videos. To start using Azure Video Indexer for Media, a Video Indexer account, Storage account, and Managed Identity need to be set up. The Azure Video Indexer for Media can be accessed at https://​www.​videoindexer.​ai/​. At this console, upload video files either individually or as a project (up to 10 videos at a time). After uploading the videos, the console will output insights for each video including speaker recognition, audio effects, keywords, labels, and sentiment. The pre-built model can also be slightly customized based on the language the video is in, brands, and people.

GCP is the only platform with a custom model offering, so we exclusively test that platform's capabilities. We analyze three publicly available datasets, see Table 6, to assess the detection of video content. The Fight Scenes data contains fighting and non-fighting actions. The other two datasets are randomly drawn from subsets from Hussain et al. (2017) (excluding private and unavailable videos). The source dataset is of video advertisements labeled with persuasive strategies. In particular, this dataset captures whether a video is exciting or not, or funny or not. We draw upon these two commonly recognized advertising strategies. Using these categories, we can test the cloud services’ potential to understand a specific marketing activity using an example of how persuasive elements can be analyzed in advertising messages. (We are grateful to an anonymous reviewer for this suggestion).

Similar to the metrics used in image analysis the overall results of video analysis are measured by precision and recall. (Precision measures the model's accuracy in items classified positively and recall whether the algorithm identifies things that should be classified positively correctly). We analyzed GCP’s performance on three datasets see Table 7. We split training to testing across the three datasets 70:30, 80:20, 80:20 respectively. For all three tasks, GCP accurately predicted the relationship between a video and its associated tag. We then assessed the platform's (GCP) custom model’s performance on three different video analysis tasks—Fight Scenes, Exciting, and Funny.

The overall precision and overall recall metrics show high accuracy across all three datasets, ranging from 68.9% to 97.6% for precision and 70% to 100% for recall, indicating that GCP's custom model offers a highly reliable solution for video content analysis.

The performance disparities among the datasets highlight the complexity of classifying sentiment-related content compared to more straightforward categorizations like Fight or No Fight. Specifically, the Fight Scenes dataset exhibits nearly perfect precision and recall rates, likely because distinguishing fight scenes from non-fight scenes is often a matter of recognizing certain explicit physical actions. These are more easily quantifiable and less open to interpretation, making the classification task more straightforward.

On the other hand, tags like Exciting and Funny involve more nuanced analysis, which can be highly subjective and culturally dependent. For the Exciting tag, a precision of 62.7% versus a recall of 84% suggests that the model is better at correctly identifying truly exciting instances (high recall) but tends to mislabel non-exciting instances as exciting (lower precision). In contrast, for the Not Exciting tag, the model shows higher precision (75.8%) but lower recall (50%), indicating that while the model is careful about labeling a video as not exciting, it fails to catch many actual not exciting cases. Similar nuances can be observed in the Funny dataset. The balanced but less-than-perfect precision and recall indicate that sentiment-based attributes, like humor, are inherently challenging to pin down algorithmically.

In summary, while the GCP model excels at categorizing more explicit content like Fight or No Fight, it faces challenges in capturing the subtleties involved in sentiment-related tags. The false positives and false negatives within these sentiment groups can provide valuable insights for model refinement and underscore the complexities of automated sentiment analysis in video content.

As the only platform offering we considered that had a custom model function, GCP Vertex AI is marketers' only choice in regard to training their own supervised machine learning model. GCP Vertex AI API also provides important features including object detection and tracking, explicit content detection, logo recognition, text detection and extraction, and automated captions and subtitles. In its beta version, the Vertex AI API grants consumers access to celebrity recognition, face detection, and person detection features.

For the general public, AWS Rekognition is a more accessible source for face/person-related features. It provides face search, celebrity recognition, and person pathing features even for its free tier users, along with other features including label detection and content moderation. Microsoft Azure has a unique analysis capacity regarding audio content within the video, including speaker recognition, audio effect, keywords, and a sentiment process function.

The three platforms have different policies for their free tier services: AWS provides 10 training hours plus 4 inference hours per month, and so is suitable for marketers with monthly continuous needs. GCP and Microsoft Azure offer 40 training hours, enabling marketers to complete several intense tasks in a limited time frame. (See Table 8).

To analyze audio data, there are two different approaches: acoustic information analysis and transcribing. The information contained within the sound of audio data, acoustic information, provides marketers with opportunities to infer and understand the implicit messages communicated by the speaker. Researchers have used commercial software to perform this type of analysis, which can be an effective method for understanding real-world marketing applications, but it leaves the researcher reliant upon the software’s capabilities. For instance, Wang et al. (2021) measured focus, stress, and extreme emotion by analyzing variations in vocal tones. Lowe and Haws (2017) manipulated voice pitch and music in video ads using the digital sound engineering software Logic Pro, to test its effect on viewer perception of product size. Hwang et al. (2021) looked at loudness, pitch, and duration of speech in YouTube videos and linked this to whether they were sponsored. Research has also tackled the nature of voices. Chatbots can be effective at sales calls, but consumers may react if their AI nature is revealed (Luo et al., 2019). In general, the commercial applications of AI used have yet to be systematically tested or compared, but researchers can assess validity through additional experiments, for example, to manipulate the attributes of voice (Wang et al., 2021).

Emotion can be detected in music, (Fong et al., 2021). Researchers can find acoustic fingerprints including physical aspects (key, loudness, mode, tempo, and time signature) and listening experience (acousticness, danceability, energy, instrumentalness, liveness, speechiness, and valence) (Boughanmi & Ansari, 2021).

The second broad approach is transcribing the words in audio data before analyzing its content using text-based techniques. This is also a potentially beneficial method. Cloud platforms’ advanced features include speaker identification and live transcription, as well as solutions to technical challenges like isolating words from background noise. In addition to creating their own transcriptions, researchers can obtain existing transcriptions such as movie scripts (Toubia, 2021), video subtitles (Zhou et al., 2021), and YouTube transcripts (Liu et al., 2020b), etc. … which can then be analyzed.

Audio transcription has also been used to analyze verbal cues in communication captured in videos. For instance, text analysis of verbal communications recording consumer interaction has been used to study the effectiveness of salespersons’ query handling (Singh et al., 2020) and frontline problem-solving behaviors (Marinova et al., 2018).

Transcription relies upon speech recognition, and the three primary approaches to this are synchronous recognition, asynchronous recognition, and streaming recognition. Synchronous recognition performs a request on an audio file, typically with a maximum length of one minute. Asynchronous recognition allows multiple requests to be processed simultaneously, making it a more efficient method for handling large volumes of audio data. This approach can be particularly useful in situations where real-time transcription is unnecessary. Streaming recognition provides real time transcription of audio data as it is being recorded. This method is often used in situations where immediate analysis or action is required, such as in live transcription for closed captioning or real-time language translation.

While our “Understanding audio analysis” section offers a balanced exploration of acoustic and transcription techniques, the emphasis on transcription in the coming sections is intentional. This is largely influenced by the cost-efficiency and accessibility of free-tier cloud services that have made transcription an especially attractive method for real-world marketing tasks. Yet, it is crucial to understand that audio analysis spans both dimensions: understanding explicit verbal cues through transcription and capturing implicit tones and emotions through acoustic tools. Together, they offer marketers holistic insight into consumer communication.

Historically, audio transcription was performed by humans, which was slow and prone to error. However, with the rapid advancement of automated text transcription, the process has become much more efficient and affordable. A major application is call center transcriptions, where automated transcription can help analyze customer needs, identify new marketing opportunities, as well as evaluate and improve agents’ performance. Again, a core benefit is the ability to scale.

For academics, automated transcription is simply more practical than research assistants. Larger datasets can be addressed, and the work done much more consistently than could be performed by humans. As for marketing managers, audio data analysis is of great significance in monitoring and adjusting the quality of customer service. It also offers great value to evaluate and suggest modifications to salespeople’s behaviors.

Audio transcription provides a vast amount of textual data that can be analyzed to gain valuable insights into the speech patterns of consumers. Combined with acoustic analysis tools that examine the tone and pitch of consumers’ speech, marketers may be able to detect consumer excitement, fears, and boredom and use this information to craft persuasive messages. However, marketers should always be aware of the potential for consumers to become aware of their tactics and develop countermeasures to combat them, a Red Queen scenario could arise. Where ethically appropriate, marketers can strive to remain ahead of the curve by continually refining their techniques and adapting to new technologies, data sources, and consumer behaviors.

One positive potential result for consumer welfare is that by analyzing audio posted to social media platforms, marketers can gain valuable information on consumer preferences and behaviors to serve them better. This approach is relatively less intrusive, as consumers presumably wanted to be listened to when they posted the audio. This method also allows for the scalability of data collection and analysis. Similarly, consumer reaction to the actions of competitors can be analyzed. By monitoring and analyzing competitors’ audio content, marketers can identify potential threats and opportunities in the market. This analysis can also help track changes in consumer sentiment towards competitors over time.

Internally, the firm can assess its call center operations to see what works and what doesn’t. This could be very useful in employee training and performance evaluation. In addition, automated voices can also be used in call center work, although their effectiveness remains a crucial question. Despite this, the cost-effectiveness of using automated voices over live agents gives marketers a clear incentive to use them. Therefore, it is important to leverage audio data analysis to gain a better understanding of the characteristics of successful automated voice agents and precise circumstances in which these characteristics are effective.

All three platforms allow for the analysis of audio data as shown in Table 1. AWS Transcribe converts speech to text and provides a custom solution for call centers as do the other platforms. Beyond the free tier, AWS Transcribe has more complex products such as building a custom language model that trains Transcribe’s standard models with domain specific text. To use AWS Transcribe, sign up for an AWS free tier account and head to the Transcribe console. This allows users to upload audio files and receive transcriptions directly through the console. After accessing this console, upload the audio file to get the result. Alternatively, an API request can be sent to AWS Transcribe and a JSON response of the result received.

GCP Speech to Text is a service with complex features such as model adaptation and multiple channel audio transcription. Beyond the free tier, opting into Data Logging will decrease the cost of using the models for speech recognition with standard or enhanced models. GCP also has a suite of prebuilt custom solutions. To start, upload audio files to GCS through a bucket. In the GCP console, create a project and import the audio files from the GCS bucket.

The Microsoft Azure Video Indexer for Media standard Audio Indexer provides rich metadata using predefined models. Azure Speech to Text transcribes audio in more than 85 languages/variants and also has batch transcription operations. Customizable options allow addition of specific words to a base vocabulary or custom building a speech to text model. Azure Speech to Text can be accessed in the console. In Azure, each audio file needs to be individually uploaded for transcription.

We next consider the performance of the various platforms. This is a constantly evolving area, so the reader should consider our results as indicative of capabilities at a given point in time. We anticipate the abilities, and ease of use, of all platforms will only improve with time.

Free tiers only provide speech to text analysis, given this we focused on transcription by the cloud platforms that we discuss. We did not build a custom model but tested transcriptions of 3,021 podcast summaries already available online which we collected in the GitHub. The MP3 formats caused a problem for Azure which only accepts WAV files. These were converted to WAV with 16,000 Hz frequency, 16-bit resolution, and Mono audio channel.

Each platform was tasked with transcribing large audio files. The output displayed from all the speech to text analyses are the transcript, confidence level, and alternative words. The format of the results differ as the cloud platforms measure the results in different ways, see Table 9.

AWS provided a transcript of the file with no overall confidence results but gave a confidence score for each word. (AWS’ approach is useful locating words that require a human to review them for accuracy). GCP gives a transcription, splits the audio file up into smaller segments, and provides a confidence level for each segment. Azure provided an overall confidence level. All three platforms gave alternative transcriptions when unsure about words. Given it is most consistently shared, we used the confidence level of the transcription to compare the cloud platforms. Our overall assessment was that all performed well.

For the free tier service, AWS offers 60 min of transcription hours per month for 12 months; GCP offers a total of 60 min; while Azure offers the greatest amount of free capacity at 5 audio hours per month. Aside from transcription features, AWS's other services focus on more specific commercial settings (i.e., custom solutions for call centers, medical transcribing). For medical application of transcription services, Amazon Transcribe Medical can transcribe doctor notes or patient conversations, while GCP Speech to Text allows users to select a medical dictation or medical conversation model. GCP is able to increase domain-specific transcription accuracy with a selection of trained models. Being the only platform that allows users to upload and process multiple audios simultaneously, Microsoft Azure also has the unique ability to perform speaker indexing, caption formatting, and sentiment analysis. Researchers with these text analysis needs may find the Microsoft Azure experience most suitable for their needs (see Table 10).