The Value of Structured and Unstructured Content Analytics of Employees' Opinion Mining

Employees and their knowledge are critical for a company’s success and were extensively covered by literature. Free form of knowledge, as opinions or feedback is analyzed in some companies in order to understand potential areas of improvement or satisfaction. This research focuses on analyzing free format opinions collected with a survey instrument from 586 employees of a bank. Various techniques as text analytics, word embedding, supervised and unsupervised learning were explored, extracting key concepts or entities. Attribution and relational similarity was explored using techniques as supervised text classification or unsupervised, by word vector representation and embedding using word2vdec. The aim was to explain overall satisfaction combining structured data (collected with closed questions survey, using 7 points Likert scale) enhanced with text classification of opinions, related to best and worst applications, along with explanations. The theoretical model used for ontology and taxonomy of text classification was based on Technology Acceptance Model, mapped into the main perceptual constructs, Perceived Ease of Use and Perceived Utility. The scope of research was the banks’ enterprise IT environment, not focused on a specific application. Various quantitative models, including linear models, decision trees and neural nets, were evaluated to capture potential causality of overall employee IT satisfaction level. The results suggest strong influence of perceptual constructs towards satisfaction, within all methods, while unstructured textual data provide additional insights on employees’ perception from concept associations.


Introduction
This paper is the extended version of the Conference Proceeding paper "Employees opinion mining -value of structured and unstructured content analytics in a bank" (Jipa, 2017b) and introduces unsupervised learning methods to explore the text corpora, to complement supervised classification using text analytics and uses an exploratory approach.
Some organizations operate under strict industry regulatory and compliance requirements, dealing with private or confidential data, as the case for the current research paper. In order to fulfill their job, employees are required to follow strict processes and use mandatory systems (as transactional processing systems, customer management systems, claims) where information systems qualities as speed, performance, usability and user experience are critical for employee satisfaction using IT systems and ability to complete tasks. The conceptual framework of this paper is based on Technology Acceptance Model, also referred to as TAM (Davis, 1986;Venkatesh et al., 2003;Shumaila, Foxall and Pallister, 2010) to explain potential causality and factors that affect the behavioral intention to use or technology adoption. TAM (initial and subsequent versions) was adopted in a broad number of research papers and covered by literature reviews but faced also criticism due to its generalization approach (Hwang, Al-Arabiat and Shin, 2015). This paper explores the use of natural language processing to classify concepts on predefined categories, based on TAM model as well as detecting concept similarity with unsupervised learning and relevant keywords extracted. Two different approaches were used: a) Approach A: Supervised method for multi-class classification, based on predefined rules, functions and dictionary based natural language processing in SPSS TA. Specific dictionary was built for the project. b) Approach B: Unsupervised method for similarity check, data driven natural language processing in Python and Gensim using word embedding (vector). Classification, while possible with different methods as Latent Dirichlet Allocation (Alghamdi and Alfalqi, 2015) using Gensim, was not performed as part of this research.
Approach A: TAM factors and modeling was done to derive the conceptual model for text classification using computer software, SPSS Modeler 17 with Text Analytics Module (IBM Knowledge Center -IBM SPSS Modeler V17.0.0 documentation, 2014). Individual sentences were classified in multiple applicable categories (or topics) and computer aided coding was performed using the same scale coding model used for the survey Likert, based on the positive or negative meaning.
Approach B: Alternatively, the free text input (corpus was created based on employees' feedback) is converted into vectors using word2vec model (Mikolov et al., 2013). The technique creates a very sparse vector using "one-hot word encoding" (1 for the word and 0 for the rest) that feeds a shallow, 3 layers neural network that forces the output to a dimensionality reduction. That allows much efficient computations as well as the possibility to capture information from the word neighborhood. Despite the fact that the model is being applicable to large corpus or datasets, there is no minimum recommendation for corpus size in word embedding model. However giving the small dimension of corpus, limited dictionary and bias was expected. Graphic exploration was done using both plotted dimensionality reduced vector data and SPSS Text Analytics capability. These tools provide researchers with easy to use interfaces for training a model from own corpus or loading a pre-trained one using Python. Word embedding used vectors with 100 dimensions. Data Processing was done in Python 3.5 and NLTK libraries (Bird, Klein and Loper, 2009) . Exploration was done using visual plots or similarity detection using Euclidian or cosine distance. We explore the practical and managerial applications and implications of the modeling techniques using the analytical and machine learning tools. For identifying relationships between vectors we used similarity measurement approach (cosine or Euclidian distance) due to its widely usage and practical applicability (text classification, summarization, information retrieval based on mathematical calculation). Cosine distance measurement means measuring the cosine angle between two vectors and works in sparse vector space (Li and Han, 2013). Alternatively, Euclidian distance measurement is available using Gensim (Rehurek and Sojka, 2010) as root of square differences between the respective coordinates of v1 and v2, where v1 and v2 are vectors in the same space.

Literature Review and Hypothesis Development
A simple search in one of the research portals revealed more than 500.000 publications, papers, books or articles indexed that cover TAM, showing model efficiency and popularity despite the inherent criticism (Hwang, Al-Arabiat and Shin, 2015). TAM considers as critically important the evaluation of two constructs: "Perceived Utility" and "Perceived Ease of Use". Nevertheless, since its introduction focused on traditional IT platforms, until nowadays, applicable also to m-commerce and e-commerce (Wei and Bin Ismail, 2009;Ha and Stoel, 2009;Heinhuis, 2013;Nikou and Economides, 2017), TAM presents a solid framework to understand complex interactions between technology qualities and utility through the lens of perception, attitude, behavioral factors, proposing high level interaction patterns that we aim to validate with empirical findings also in this paper. TAM originates from general behavioral and psychology research, inspired by TRA Theory of Reasoned Actions (Ajzen and Fishbein, 1980) and also TPB, Theory of Planned Behavior (Ajzen and Fishbein, 1980) and further revised (Ajzen et al., 2002). TAM was used in many papers and research studies and its current or future applicability is also important (Lee, Kozar and Larsen, 2003). Many TAM theory reviews and meta-analysis suggest that this theory is current and valid to use (Elie-Dit-Cosaque, Pallud and Kalika, 2012), presenting broad views on its usability (Marangunić and Granić, 2015). Some authors focused on analyzing the adoption of various specific services, as m-commerce trying to understand what mobilizes users to use or adopt technologies (Hew et al., 2015), e-commerce, banking services (Luarn and Lin, 2005), learning (Lee, 2010) or Internet usage (Shumaila, Foxall and Pallister, 2010). While not performing a critical literature review, the current paper uses a simplified version of TAM for a more generic IT environment, based on initial model to provide a taxonomy of factors, to define variables contributing to constructs as Perceived Easy of Use and Perceived Usefulness (Davis, Bagozzi and Warshaw, 1989a). It is known that banks' employees cannot use alternative technologies, while operational performance cannot be improved if systems not used (Venkatesh, 2003). Technology brings functional advantages (Rogers, 1995) while in mandatory environments.
The initial model proposed by TAM (Davis, 1986)   opinionating will improve status as a Bank feedback.
H8: The employee satisfaction towards using the entire IT environment can be expressed as a single rank that can be explained by the of Use and are relevant from unstructured __________________________________________________________________________ ______________ data or opinion narrative in both exploratory and quantitative analysis.
In this bank, usage of Informational systems was mandatory (Venkatesh and Davis, 2000); no manual operations were possible outside Information Systems, so measuring behavior or actual system use was not relevant (Venkatesh and Davis, 2000; Hwang, Al-Arabiat and Shin, 2015) from the research perspective, but attitudinal and employee satisfaction was important. Assumption: The system usage measurement for mandatory systems do not bring additional insight for the research focus. Therefore, a specific model was proposed after the exploratory research, considering both positive and negative conflicting statements stated by respondents (evaluation of application received from the same respondent during in depth interviews shown a mixture of negative and positive opinions towards the subject, while improvements were suggested constantly). The main conclusion after exploratory research was hypothesized as Mo Assumptions (MA1-MA2) and one Model While a measurement of the influences between PE PU would help, our goal in this paper is to explain the measured satisfaction rate, confirm and validate the methodology of extracting opinions expressed in free format as open ended answers, classify text data using natural language techniques into Journal of Administrative Sciences and Technology __________________________________________________________________________ data or opinion narrative in both exploratory and quantitative analysis.
usage of Informational (Venkatesh and no manual operations were nformation Systems, so measuring behavior or actual system use (Venkatesh and Davis, Arabiat and Shin, 2015) from the research perspective, but attitudinal and employee satisfaction was Assumption: The system usage systems does not bring additional insight for the research Therefore, a specific model was proposed after the exploratory research, both positive and negative by the same of one core application received from the same respondent during in depth interviews has shown a mixture of negative and positive opinions towards the subject, while were suggested constantly). conclusion after exploratory ed as Model ) and one Model . Employee satisfaction integration in TAM is a key element in this (Wixom and Todd, 2005). While parallel research not covered by this paper was conducted only on analysis of data collected in survey format, the two studies same quantitative analysis methodology or data set related to (Davis, Bagozzi and Warshaw, 1989b).

Research Methodology
Qualitative research techniques were used, as described in the specific literature (Becker and Becker, 1996;Patten, 2007;Creswell, 2013). The Exploratory phase consisted in in-depth interviews with a randomly selected group of 30 Individuals. The interviews were conducted in 6 separate locations, with different branch sizes but similar job responsibilities. All interviews notes were transcribed and indexed.
Qualitative research or Exploratory research was based on series of Techniques including Text Analysis / Unstructured data mining (Clark, Fox and Lappin, 2010). Consequently, there were considered two types of exploratory analysis, manual coding and text mining. In manual coding, indexing of key concepts was performed based on interview transcripts (Saunders, Lewis and Thornhill, 2008a). Concepts were mapped to categories representative to TAM and research focus, from existing literature, as Perceived expected value, Performance Expectancy  and Effort Expectancy (Davis, Bagozzi and Warshaw, 1989b). That was used for constructing survey tool, including Organizational support construct. That was treated as a form of trust from the perspective of Attribution Theory (Kelley, 1973), that states that beliefs and person's cognitive processes should be in agreement, as a result of causal attribution (Krueger, 2007). As such, it was expected that Organizational support construct would have a significant contribution to explaining the result, as formulated by hypothesis H7, H8. In interview transcript, mining approach performed as a parallel semi-automated text analytics approach, using software text classification, parts of speech (POS) identification, concept extraction and type classifications. That was exploratory, aimed to reveal the main factors in a deductive approach. The category classification formed a reduced taxonomy, for building factors influencing positive and negative perception. Interviews were broad, following a semistructured format, allowing respondents to progressively accommodate with the interview team.
Quantitative Research. The tool for data collection to support quantitative research was survey instrument (Rook, 1985;Patten, 2007), aligned with qualitative research methodology (Creswell, 2013). Survey was broader in data collection, but this paper uses a small set of variables in structured format (related to Hypothesis H6, H7, H8, H9), four categorical variables and one ordinal (Employee satisfaction rate towards entire IT environment).   noticed that Organizational support dimension was identified as category but finally discarded due to the very small number of cases. As suggested by many authors, the way that TAM construct U (Perceived Usefulness) can be regarded as a form of total value a user perceives from technology, while focused on task accomplishments , as driver for actual system use. After classification step using TA methodology, a set of variables was created specific to each factor. Classification works on a case text field named document and based on linguistic models apply best-fit categories. Thus, a case (phrase or sentence from respondent, one opinion) could explain multiple factors and is classified into multiple categories. As survey instrument included data represented on Liker 7 points scale with 4 as midpoint (Garland, 1991;Boone and Boone, 2012;Sullivan et al., 2013), the classification rule assigns 7 for positive classification and 1 for negative classification, with 4 as midpoint. Concepts extraction was performed for Positive Opinions, with 286 concepts and 28 types that supported Classification (Category extraction based on 80 descriptors) while Negative Opinion generated 674 concepts, 37 types. Technique named Text Link Analysis (TLA) generated a rich source of information of concepts dependencies, providing ontological support as evaluating reasons for scoring the app as "best" gives some insights on inference, but the paper does not focus on that aspect. That processing choice generates structured data that can be analyzed using quantitative techniques, with 586 valid cases (documents in SPSS Modeler TA).  Opinion mining can reveal much information, reducing variability when focused on specific subject (Vu et al. as shown in unstructured data processing from reviews of products, extracting  Machine learning Text Data Cleaning: minimal manual correction was performed, without typos or spelling and analysis was performed in English 10 __________________________________________________________________________ oncept count and distribution of topics using supervised classification. Source: concepts relevant to subject (Chuttur, and can be used to extract insight or perform sentiment analysis against a specific target in Text performed, without typos or spelling and analysis was performed in English __________________________________________________________________________ ______________ language as part of preprogrammatic removal of punctuation, non-English characters or spaces was performed, generating a corpus of 142129 characters and 22460 tokens. T generated word2vec model was trained from custom corpus in a space with dimensions 100. To avoid inclusion of less used words, a minimum word count 3 was selected (at least three occurrences, including the typos). In order to capture the words proximity, the skip-gram model was selected with a window size of 4. The generated word2vec vocabulary length is 956, being very small, and 40 epochs (iterations) of training were selected to improve the model, due to its size and sparse representation. The model saved to disk for later usage and new data

__________________________________________________________________________
-processing, programmatic removal of punctuation, English characters or spaces was performed, generating a corpus of 142129 characters and 22460 tokens. The generated word2vec model was trained from custom corpus in a space with dimensions 100. To avoid inclusion of less used words, a minimum word count 3 was selected (at least three occurrences, including the typos). In order to capture gram model was selected with a window size of 4. The generated word2vec vocabulary length is 956, being very small, and 40 epochs (iterations) of training were selected to its size and The model was saved to disk for later usage and new data classification (model serving, an economical mode to use trained models in machine learning). Worth noting that a limitation of the word2vec model is that it fails to classify words not present into the corpus, so its generalization capability is reduced. As a result, the corpus is to the research problem. For easy visualization, 2-dimension reduction was performed, using TSNE algorithm used in visualizing high dimensional data, T-Distributed Stochastic Neighbor Embedding t-SNE (Maaten and Hinton, 2008), but comparable result encountered using Principal component Analysis PCA methods. For example, the word "simple" is represented as (X = 18.155998, Y= -10.602283) allowing easy visualization.
corpus representation using TSNE reduction on XY coordinates Source: Author generated using matplotlib.
gram model, word neighborhood is captured in the vector representation. Visual exploration of the corpus can suggest potential term relationships, as in Approach A, but without any effort in coding classification.
Journal of Administrative Sciences and Technology __________________________________________________________________________ classification (model serving, an economical mode to use trained models in machine learning). Worth noting that a limitation of the word2vec model is that it fails to classify words not present into the so its generalization capability is , the corpus is specific For easy dimension reduction was performed, using TSNE algorithm highly used in visualizing high dimensional data, Distributed Stochastic Neighbor (Maaten and Hinton, results are Principal component For example, the word "simple" is represented as (X = -10.602283) allowing easy corpus representation using TSNE reduction on XY coordinates potential term's , as in Approach A, but without any effort in coding classification. It is noted that terms as ["deposits", "accounts", "current" , "monthly"] are very close, while ["balance"] is remote. Similarly ["agreement", "compliance", "risk"] are close. The vector representation can be used to predict the next word in a sentence.   Performing a frequency count without stopwords removal gives comparable Performing a frequency count without stopwords removal gives comparable result with SPSS TA and Wordcloud method. vector representation for "navigation" word. evaluation of three analytical models was performed using GLM, Regression, Neural Net and CHAID. Given the linguistic analysis dependencies (Liddy et al., 2003), the Empirically evaluate TAM , compare models based on Lift and gains curves but maintain theory driven approach and Evaluate Unstructured only model to explain the Satisfaction rate for Theory confirmation.

Results
Approach A: The descriptive results of survey are summarized in the table below. It is interesting to note the generated values from unstructured data extraction. Partition was used with allocation 70% (401) labeled 1_Training and 30% (186) labeled 2_Testing, based on random selection from sample. Descriptive Statistics for both Structured data and extracted with Text Analytics table shows deviation and skewness, but this is expected due to positive / negative model generation: Respondent profile: There was no significant variation between seniority, age or job profile collected as categorical data. However, it was noted that senior age employees might consider better ranting, but decision was not to consider that potential bias. Typical requirements of normal distribution of the data -skewness below 0 and kurtosis below 2.5, recommended -2/2 (Trochim and Donnelly, 2006)-are not necessarily met, in concordance with data extraction expectations.  Target model was evaluated using three scenarios, based on structured and structured data extracted:   While the model intercept is 2.049, we see the positive contribution of Organizational Trust as contributing significantly with 0.322, Helpdesk contribution 0.414/ unit increase and the Negative contribution, that is expected (if the negative aspect decreases by moving from completely dissatisfied to dissatisfied there is unit increase corresponding to 0.267). It is interesting to note the contribution of these non-obvious TAM factors into result explanation. A Decision Tree analysis using CHAID model (expanded on three levels) can bring more insights. Decision Tree's level 1 is split into 4 branches, based on Node 1 values (Helpdesk perception), 4 intervals being identified in the data. Almost 18.9% employees that perceive negative the provided helpdesk support with ratings in the interval of [<4] are grouped into believers and non-believers, split in relative equal groups, based on measurement of attitude that Organization will implement changes based on their feedback, suggesting that there is no dominant a pessimistic and optimistic perception in this cluster. Other important Branch at Node 2 suggests the ambiguity of a group of respondents (120), more than 25% of Training Partition. They are in the interval between midpoint and 5, "somehow satisfied" with the organizational support they receive. This group is further affected by Information quality exposedthey express negative opinion in this regard. Interestingly, this group splits into strong Pessimists and other that might trust that the organization will actively listen to their voice (Node 8 split into Node 9 and 10). Other Level 1 categories of users under Node 3 and 4 will award higher rating, also seen in the descriptive statistics analysis. Decision Tree's Linear Correlation is 0.505 for the Training partition and 0.423 for Test partition.  The Neural network model was performed without partitioning due to the relative small number of cases but presents convergent results with result interpretation. As a guide for evaluation of applicable models, we considered the Gains and Lift curves for the selected model as well as relative error and correlation coefficient for each. As it can be observed, all models provide convergent results.

Approach B:
Full vector space exploration similarity was done using cosine distance measurement model. This aims to uncover potential relationships in the data using no rules, unsupervised learning and listing the closest 5 terms in the vector space we note that expressions as "easy to use " cannot be constructed as in the case of SPSS TA. Some notes are included with some extracted terms.  Interesting finding on client and dissatisfaction, as well as stability issues.

Conclusions and Discussion
This paper aimed to provide an empirical validation of TAM and organizational factors that might be explained as beliefs in organizational support and feedback. We also acknowledge the effort required in SPSS TA for model building. Approach B in analyzing opinions shows alternate ways to explore opinions. Similar approach can be done to perform text classification, similar with Approach A. The table below summarizes the main findings:

Limited
High.
Vocabulary and pre-trained models Yes. Extension of existing models to allow custom models specific to projects.
Load of pre-trained models on large amount of data with high generalization, or project specific corpus and vector space creation.

Implementation extension Limited
High, access to source code.
As conclusion, both approaches generate structured data from textual data and could serve project specific goals. Similarity exploration gives the researcher the possibility to identify missing data, due to the limitation of survey as instrument (Saunders, Lewis and Thornhill, 2008b).
We noticed that in all models this is critically important and significant. In this context by results the entire set of research hypothesis H1-H9 was validated. While analyzing in Scenario C several different models, our aim was explanatory in its effort towards the value of using linguistic models in opinion mining. This source of information is abundant in organizations, and thus not exploited enough due to perceived difficulty in natural language processing and complexity to combine structured data with free format data. We note that people are naturally inclined to associate concepts, in positive or negative attitudinal constructs. It is important to note that full model was based on collecting data in three closed survey questions and 4 open-ended questions. Also, categorical information as job seniority, age or job role, while interesting, was not relevant to the study. That shows that the causal factor is persistent and perceived as such through the organization. Therefore, we appreciate that opinion mining presents a source of value in improving analytics models with proper linguistic models in place, but also brings more reasoning besides the starting hypothesis that can be used to validate also the potential bias induced by survey design and order of questions. Looking and Gains and Lift curves evaluation, we note that convergent scenarios are valuable and can be used for providing empirical validation.  disambiguation. In its nature, text analytics (appreciated by many authors) is not an exact science and the effort in domain adaptation could be significant. Additional value comes from potential use in other studies where a potential rating system is in place and feedback is given in the form of user reviews. By using gamification elements, the adoption of this data collection channel can be accelerated and can increase motivation and engagement, like Organizational support z32.ActioanbleInsightValuesMe.
We appreciate that further research should use confirmatory analysis techniques for measuring latent variables specific to motivation to share opinion, and TAM constructs. All evaluated scenarios present acceptable values for Lift and Gain curves, for both Training and Test partitions. That suggests that while with larger data set for extraction the statistical significance should improve, this approach brings benefits. The current paper did not evaluate the value of link analysis and concept association in the form of ontologies. Therefore, we consider that Hypothesis H1 and H2 originating from TAM are validated. Correlation values between factors contributing to Perceived Easy of use towards Perceived Usefulness (Information as root category in our model) suggest that models are validated. Negative correlation between Positive and Negative suggests also that conceptual model is valid even if the strength is not as expected (as suggested by pair Category_PE_EasyToUse and NegativeCategory_PU_Information negative Pearson Correlation (direction) of -0.108* or pair Category_PE_EasyToUse and NegativeCategory_PU_Information/Applica tion with negative Pearson Correlation -.074* significant at 0.05 level). H3, H4 and H5 are considered empirically validated based on results' interpretation. H6, H7 and H8 prove the direct relationship between structured variables, seen in all models-except the TAS only model, which was deliberately excluded. H9 is evaluatively proved, showing that extracting from natural language key is valuable (Jipa, 2017a). We expected that model strength is lower than survey provided measurements but the paper provides empirical validation of relationships per TAM model while indicating ontological insight by link analysis into complex relationships between factors that contribute to perception constructs. Correlation values inside construct factors suggest that a confirmatory analysis can be performed in a future study. Model specific Hypothesis MH4 regarding the converge of conclusions of analysis, regardless of data collection method, is suggested valid by model fit and stability of results within the limit of the current paper and dataset.