The Gist
- AI boost. 14% productivity surge with AI assistant.
- AI impact. More aid for novices, less for experts.
- AI efficiency. Lower manager calls, higher staff retention.
The advent of generative AI models has ignited an era of fascination and experimentation, yet the tangible economic implications of their widespread implementation have largely remained a mystery.
In a large-scale study of 5,179 customer service agents, researchers discovered an average productivity surge of 14% among those equipped with a conversational AI assistant tool, based on OpenAI’s ChatGPT.
Compiling its results in a report, “Generative AI at Work,“ the National Bureau of Economic Research (NBER) analyzed data from more than 5,000 agents at a Fortune 500 software company who were paired with a digital AI assistant designed to augment, rather than replace, human agents, and the data they collected revealed several key takeaways:
- A significant boost of 13.8 percent in worker productivity, specifically in chat resolution rates per hour.
- The contribution of AI assistance in enhancing performance is more pronounced among less skilled and less experienced workers across all productivity measures, whereas its impact on highly skilled workers remains minimal.
- Positive interactions between customers and agents increased as reflected in the uptick of positive sentiments expressed in their messaging.
- Use of the AI tool reduced requests for managerial intervention and improved employee retention, suggesting it could lead to operational efficiency and better workplace dynamics.
AI Uplift: 13.8% More Chat Resolutions, Same Number of Hours
Researchers find that agents with the AI assistant achieved a significant boost of 13.8 percent in worker productivity, specifically in chat resolution rates per hour.
For the study, agents were split into three groups. The first group didn’t use the AI tool at all. The second group started using the AI tool partway through the study and the third group used the AI tool the entire time. In all, researchers looked at the text and results of 3 million chat conversations handled by 5,179 agents. Out of these, 1.2 million chat conversations were handled by 1,636 agents after they started using the AI tool.
The study discovered that agents who didn’t use the AI tool could typically handle about 1.7 chat conversations every hour. But those who used the AI tool could handle more — about 2.5 chats every hour. Part of this difference might be because the agents who ended up using the AI tool were already a bit quicker to begin with, handling about 2.0 chats every hour before they started using the AI tool, compared to the 1.7 chats every hour for those who never used it.
This pattern was the same when they looked at other things like the number of chats every hour and how often chats were resolved. One interesting thing they noticed was that both the agents who didn’t use the AI tool and the agents who were about to start using it took about the same amount of time on average to handle a chat — around 40 minutes. But after the agents started using the AI tool, they took less time, about 35 minutes on average, to handle a chat.
Mike Micucci, CEO at Automation Anywhere, said the NBER research is a testament to this transformative power of generative AI.
“Imagine customer service representatives freed from tedious, repetitive tasks that can now focus on delivering exceptional service, problem-solving and nurturing customer relationships,” Micucci said. “They can be onboarded quicker and are equipped with tools that turn daunting processes into simple, automated tasks, completed in mere minutes.”
While Micucci said he’s thrilled to see the reported uptick in productivity among customer service agents leveraging a conversational assistant like ChatGPT — he’s not surprised.
“But the true magic lies beyond the numbers and the 13.8% rise in productivity is just the beginning,” Micucci said. “As generative AI continues to evolve and mature, we believe its effects will ripple out, transforming industries and redefining what is possible in the workplace. We’re committed to driving this change and excited for what the future holds.”
Related Article: AI-Enhanced Contact Center Platforms for World-Class Customer Service
AI’s Uneven Impact: Novices Thrive, Experts Barely Blink
The contribution of AI assistance in enhancing performance is more pronounced among less skilled and less experienced workers, whereas its impact on highly skilled workers remains minimal.
Of course, we all know that good customer service is key to a company’s success — we also know productivity among workers can vary greatly depending on skills and experience. NBER’s data shows the productivity impact of AI assistance is most pronounced for workers with the lowest skills — who see a 35% increase in resolutions per hour. However, for highly skilled workers (who are already effective) AI assistance doesn’t seem to boost productivity and (gasp) can even distract them.
With service calls that average 40 minutes, all agents typically start with resolving about two problems per hour. Workers who don’t get AI assistance gradually improved over 8 to 10 months to resolve 2.5 problems per hour. However, workers who got AI assistance right from the start quickly improve to resolve 2.5 problems per hour in just two months — and they keep improving until they can handle over three calls an hour after five months, suggesting that AI assistance helps workers gain experience faster.
Further, NBER found that AI assistance helps lower-skilled workers communicate more like highly skilled agents, closing the language gap. AI-enhanced agents with just two months of experience performed as well as those with more than six months of experience without AI.
By contrast, while there is a positive effect related to calls per hour, AI assistance does not lead to any discernible productivity increase for the most skilled workers with zero effect on average handle time and a small (but statistically significant) decrease in resolution rates and customer satisfaction.
Researchers believe this is because AI can identify patterns of successful customer service from the highly-skilled workers and then share this knowledge with all workers, potentially better than busy managers can. However, “the findings suggest that while lower-skill workers improve from having access to AI recommendations, they may distract the highest-skilled workers, who are already doing their jobs effectively.”
According to Yaad Oren, managing director of SAP Labs and head of the SAP Innovation Center Network, generative AI applications usher in a new era as self-service tools that not only enhance productivity but also empower employees to make informed, data-driven decisions independently, reducing dependence on data and IT experts.
“However, for organizations to leverage these tools successfully, they must have a strategy and guardrails for data usage in place, and prioritize learning and development opportunities for their employees, enabling them to build strong data analysis and management skills,” Oren said. “Once equipped with the necessary strategy, guardrails and proper training, employees can unlock a multitude of benefits beyond productivity, including streamlined decision-making processes, improved data exploration capabilities and the democratization of data access across departments.”
Happy Chatter: AI Boosts Positive Sentiments in Customer-Agent Interactions
Positive interactions between customers and agents increased as reflected in the uptick of positive sentiments expressed in their messaging.
Customers often express their frustrations through verbal abuse or “yelling” at service agents, and agents are tasked with handling these outbursts while keeping their own emotions in check. Emotional labor can lead to stress, burnout and high attrition rates among customer service workers.
In this study, researchers used a language model fine-tuned for sentiment analysis to examine the emotional tone of both the agent’s and customer’s messages in their data. Their findings showed that while customer sentiments had a mean score of 0.14, service agents (who are trained to maintain a highly positive demeanor) had a much higher average sentiment score of 0.89.
The impact of introducing an AI assistant significantly improved customer sentiment, raising the mean score by 0.18 points, while the effect on agent sentiment was negligible due to their already-high scores. In terms of the distribution of sentiment impacts, they found that improvements were most noticeable among agents with lower tenure and skills at the time the AI model was deployed. The greatest effects were seen among workers with 3-6 months of tenure, and a smaller impact was seen only for agents in the highest-skill quintile. This suggests that AI recommendations, designed to promote empathetic responses, could potentially improve agents’ social skills and positively impact customers’ emotions.
AI Tools: Cutting Manager Calls and Curing Employee Churn
The use of the AI tool reduced requests for managerial intervention and improved employee retention.
Gartner predicted contact centers would spend nearly $2 million on conversational enhancements in 2023 — an investment that could come with a big payoff of an estimated $80 billion reduction in labor costs within four years. According to VP analyst Daniel O’Connell, of the approximately 17 million contact centers worldwide, many are plagued by staff shortages, but at the same time, they’re under increasing pressure to reduce labor costs — which can represent up to 95% of their expenses.
NBER estimates that “60 percent of agents in contact centers leave each year, costing firms $10,000 to $20,000 dollars per agent.” And to address these workforce challenges, “the average supervisor spends at least 20 hours per week coaching lower-performing agents.”
In response to this divergent productivity, employee turnover and the substantial cost of training expenses, companies are increasingly adopting AI tools. With its AI assistant in place, NBER found a 25% decline in customer requests to speak to a manager. Further, on average, they found that the likelihood of an AI-assisted worker leaving their job goes down by 8.6%, especially among newer agents (those with less than six months of experience.)
“Progress in machine learning opens up a broad set of economic possibilities,” the NBER study concludes. “In our setting, we find that access to AI-generated recommendations increases worker productivity, improves customer sentiment, and is associated with reductions in employee turnover.”
Rajesh Varrier, EVP and head of digital experience at Infosys, said AI will enable workers of all levels to move beyond repetitive work, unlock value and create exponential impact. But Varrier insists companies will have to shift their thinking on how to use AI to better enable the workforce and how to train the workforce, so that they can use AI to amplify their skills.
“Technological advancements have been focused on improving productivity, but we’re likely to see an increased focus on improving experience and methods of social interaction. The way we use technology has massively changed — enterprises need to ensure that their technology serves the human factor,” Varrier said. “Even with generative AI, there will need to be human intuition always at the end of the process. Ultimately, AI is an enabler of higher potential of human.”
