A lightly edited transcript of my September 19, 2023 conversation with Kristen Hayer on Innovations in Leadership, a Success League Radio production. The audio is on Spotify and Buzzsprout. Short on time? See the highlights post.
The transcript was produced with Whisper and lightly edited for to fix product names, verbal ticks, and mistranscriptions.
Kristen Hayer: Welcome to Innovations in Leadership, a Success League Radio production. This is a podcast focused on customer success and the leaders who are designing and implementing best practices in our field. This podcast is brought to you by the Success League, a consulting and training firm focused on developing customer success programs that drive revenue. My name is Kristen Hayer and I'm the host of Innovations in Leadership and the founder and CEO of the Success League.
Today's guest is Todd Schiller, who's the co-founder and CEO of PixieBrix. They're a company in the artificial intelligence space. I'll let him tell you more about his company, but generative AI is our topic today and I'm super excited to dig in. So Todd, welcome to the show.
Todd Schiller: Thanks for having me, Kristen. Excited to be here.
Kristen: So how did you decide to get into the wild and crazy world of customer success?
Todd: I took a very indirect route to customer success. In college, I studied computer science, then went off and did a PhD in computer science at the University of Washington. One of the highlights there — and it's relevant to our conversation today — is I actually got to watch IBM Watson compete in the original Jeopardy challenge.
Kristen: Oh, awesome.
Todd: After grad school, I shifted over toward building financial systems for banks and hedge funds. That's actually where I got my first taste of customer success. One of my first companies was Numerix. Most people probably haven't heard of them — they build financial analytics software for banks and hedge funds. What's fascinating about them is they actually have L1 customer support folks who have PhDs in finance. So it's a very different kind of customer success than some listeners may be familiar with.
Then, kind of coming full circle from my PhD program, I got to work with the creator of IBM Watson, Dave Ferrucci, when he was at Bridgewater Associates on their AI team.
After the financial world, I shifted over to being an entrepreneur selling B2B software. That's where customer success really became a big part of my life. When you're selling software-as-a-service, it's really not a transactional sale — you're selling an ongoing outcome, and the software just happens to be part of that. In B2B in particular, there's a lot of different personas: analysts, executives, IT, even their own customer support teams, each with different needs.
And then with PixieBrix it's been interesting, because customer support teams are using our software to provide customer support, and we're also using our own software to provide customer support to their teams. So it's a bit of turtles all the way down — but it's also really fun to dogfood our own software as part of sales, marketing, and customer support.
Kristen: That's great. I also am a University of Washington alum.
Todd: Oh yeah, go Huskies.
Kristen: Yeah, go Huskies. Tell us a little bit about PixieBrix. What does your company do, and how does your company approach customer success and support?
Todd: PixieBrix is a browser extension you can use to rapidly add automation, integrations, collaboration, and artificial intelligence to the web apps your teams already use. You can think of us as a cross between, let's say, Grammarly and Zapier.
On the browser extension side, you're probably familiar with things like Grammarly, Guru, or Zingtree. What's powerful about those is you can use them on any site, even sites you don't control and sites you don't have admin access to. So that's the browser extension piece.
And then how familiar are you with tools like Zapier, IFTTT, or Workato? There's a lot of people in the space using those kinds of tools, either internally or with their customers, to help with integrations. We're similar in that you can mix and match the different integrations and the different business logic. So combining that with the browser extension piece lets you do things like add product analytics over your ticketing system, or add response generation based on knowledge-base articles. It really provides those superpowers on any site. You end up with a tailored solution at a fraction of the cost of buying those individual point solutions.
Kristen: That's amazing. I think that sounds really helpful for a lot of things — and I know we're going to talk about some examples today. Let's jump into our main topic. How do you define generative AI, and how is that different from other types of AI that are out there today?
Todd: Generative AI — and it's kind of in the name — is that category of artificial intelligence that generates content. That can be text, images, sound. That's opposed to doing things like classifying information or recognizing entities like product names in text.
The main examples most listeners might've heard of: on the text side, ChatGPT or Google Bard. On the image side, DALL·E, Midjourney, or Stable Diffusion.
What's interesting is that generative AI has actually been around for a while — you can go back five, ten years and find forms of generative AI. The big breakthrough over the last couple of years was with what were called large language models. The idea is you build this massive model, then feed it the entire text of the internet, the entire text of books — all of human communication, you shove it into a model. What you get is something very good at predicting the next word in a sequence of words. Then you just do that on repeat. You give it some text, then say, "What's the most likely word that comes after this?" And it gives you that word. Then you say, "Okay, now that's all the text plus the word you just generated. What's the most likely word that would come next?"
Some people use the analogy of a parrot — it doesn't really understand what it's saying, but it picks up on patterns. In a lot of cases it can pick up patterns in a useful way. When it's hungry, it probably wants a cracker — those sorts of things.
Kristen: Okay, that's a great explanation. It makes a lot of sense. In CS, a lot of AI is being targeted toward digital programs, but I think you're really focused on programs that still have CSMs involved. What do you see as the value of AI when there's already a human kind of leading the charge with a specific customer?
Todd: I'll dig into what makes generative AI unique there, and then we can talk about how we see that.
Where generative AI differs — let's take customer support ticket classification as an example. Previously these systems would work like this: if you want to put things into buckets of "is this an account question?", "is this an item inquiry?", etc., you'd probably have a machine learning model that would look at the words in the case and associate each word with a different kind of support case. For example, if it sees the phrase "cracked screen," that might be positively associated with a broken item inquiry, but negatively associated with an account reset ticket type.
Large language models like ChatGPT are different. Instead, you give it the ticket, and you also give it a prompt with instructions to respond with the category of the ticket. The reason it can categorize the ticket is — if you think about it — if I gave you that information and asked you what the most likely or most probable response would be, responding with a category is definitely more probable than responding with, say, a haiku or a limerick. And it also turns out that responding with the correct category is also the most probable. So if there are words like "cracked screen," it's more probable you'd see a reference to a broken item than a reference to resetting their password.
If you can guide this model appropriately — and some people call this prompt engineering, in terms of what context you're giving to the model — it becomes really great at a lot of different customer success tasks.
Digging in on the human side: where that becomes relevant is that we think AI frees humans to be more human. There are three areas.
First, with AI you can automate a lot of the low-value activities. Things like summarizing a customer conversation history and identifying which issues haven't been resolved yet, finding relevant knowledge-base articles, handling boilerplate text or disclosures, or recording notes in your CRM after an interaction. Those are all low-value activities AI can do. I don't know if you have other low-value activities you end up doing as part of customer success?
Kristen: You know, it's funny — I was teaching a class yesterday on building goal plans with your customers. Two of the people who gave their examples had gone about it in a really traditional way, where they'd had a whole conversation with the customer and talked about what their goals were and set some parameters, and came out with a goal for each item at the end.
Then the third guy used generative AI to create the goal plan. All he did was have a really strategic conversation with the customer and record it. The system just spit out a goal plan for him. He said he made a few little tweaks, but basically he was able to stay really focused on the part he needed to do, which was have the conversation. It created a table for him, which was the goal plan. It was impressive — it actually was a lot better than the two before it that were entirely human-generated. And he didn't have to spend time writing everything down in a table. That was probably the most interesting use of AI I've seen recently in CS.
Todd: And why I love that example is because the second area where we think generative AI really helps in human-in-the-loop environments is that AI can also be your personal trainer or coach. In addition to helping you create that, if you did create something yourself, it could actually evaluate it and say, "Based on how we'd like to write goal plans, or based on how we'd like to write responses, does this match our brand guidelines? Does this match our best practices? Are there some next best actions you might suggest here?" So AI is like having either a digital floor walker or a coach for each of your customer support agents all the time.
Beyond that, there's the pure power to have AI upskill workers. One I love to see our customers use AI for is allowing their agents to converse in foreign languages in email and chat. That lets you be more global — you can have people sitting in the United States or Asia talking to people in all the different languages of Europe. It also lets you provide support from lower-cost geographies if you want to. It's a bit akin to that scene in The Matrix where Keanu plugs in, connects to the desk, and then suddenly downloads all the kung-fu knowledge, or how to fly a helicopter. That's the other big piece of providing those superpowers.
And then it just circles back to what you mentioned: it gives people time to build real relationships and focus on providing real value to the customer.
Kristen: Just to kind of take this to a really practical level — let's say you're a CSM. How do you leverage a tool like ChatGPT? What can you do with it on a day-to-day basis?
Todd: The first thing I'll say is that you actually shouldn't use ChatGPT directly, probably. There is a tool you can go to called ChatGPT and it has a chat interface. If you are going to do that, you need to sign up for a paid account and go into the settings and toggle off the setting that says you allow them to train their model on your conversations. Some companies got into trouble where people were pasting in customer conversations, or their engineers were pasting in source code, and that was finding its way into the model. Pretty much every provider now allows you, if you're on a paid plan or using their API, to opt out of that — but it's something you need to be on the lookout for.
The other reason ChatGPT directly is not the most efficient is because it's a separate tool. You have to copy in and copy out information. You'll also, if you're doing the same task over and over, find yourself repeating yourself. And if you just give it to your customer support team, everyone has different skill levels interacting with it as a tool, so you get inconsistent experiences and inconsistent benefits.
I don't know if you've seen some of those. Then we can dig into what's the best way to actually leverage it.
Kristen: I haven't personally used it a whole lot. I've played with the ChatGPT tool itself just for fun, more than anything else. But I'd love to get your take on how you should leverage it.
Todd: The way I like to frame it is: let's say you were to hire an intern. What would you have them do? I say "intern" because interns require instruction. They're not experts in customer support. And they're not always perfect — sometimes you're going to have to check their work, or you might have to make some corrections.
Common things that come up are: can you have them triage your cases for you? Or take some key responses on how you might respond to a customer and turn that into a full response draft? Or, based on a customer support ticket, go through the tickets and find some knowledge-base articles that might be relevant so that when you're responding to the ticket you have them at hand?
For customer support teams that already have a tool ecosystem, the easiest way to leverage those capabilities is to actually go to your vendors and see if they've started rolling out ChatGPT-like functionality into their tools. Zendesk is an example of a vendor where they've started releasing things like customer sentiment as well as case summarization into the tool. Some of these you'll have to sign up for their early access programs. The other secret is some of these tools aren't actually using ChatGPT under the hood. They might be using a different model — Google PaLM, or Anthropic's model. But it's the same idea on how they're bringing that artificial intelligence into the tool.
Beyond that, I'd recommend you look for ways to introduce some of ChatGPT's more general capabilities across the different tools that haven't already started rolling out those capabilities. Typically these are going to look like browser extensions or desktop tools, because that allows them to work across different applications. For example, we have our offering — and I can talk a little bit more about this later — but we do have some pre-made mods for customer success. One of those is our AI co-pilot mod that allows you to run preset prompts on any page, whether that's Zendesk or Salesforce, Oracle Service Cloud, or a custom in-house tool.
Kristen: Got it. So we've been talking a lot about programs that have CSMs involved — a very human-driven program. What do you see as the biggest benefits of AI in a primarily digital CS environment?
Todd: I'll start with a question to you. When was the last time you used a customer service bot or a phone tree and were very frustrated that it wasn't picking up on what question you were asking, or what you were trying to do?
Kristen: Oh, yeah. It's so fun. You know, you're like, "I said yes. How did you not catch that?"
Todd: Yeah, it's frustrating to be in one of those for sure. The big advantage of ChatGPT and other generative models is that, because they've been trained on so much human communication, they're fluent. They can sort of understand like a human. They can also sound like a human when talking back. So when you think about how you might roll out a chatbot to provide first-line customer support, that's an advantage — you can use ChatGPT-like technology as that fluent layer for receiving the input from the user as well as providing output to the user. But you can still rely on that backbone of logic and decision trees and intents and things like that that would power the original version of the chatbots people were seeing five years ago, during the last chatbot wave. It's really about adding more fluency to those automated interactions.
The other big piece: ChatGPT — generative AI — is a transformational technology. And I mean that sort of figuratively and literally, because it's very good at going between structured interactions and structured data and unstructured information. You can use that to bridge the gap between the natural-language conversations you're having with your customer and all the other initiatives you might be doing as a sales team or a marketing team or a customer success team, where you're trying to build a 360 view of your customers, your products, or what you're trying to do. It's really about bridging those gaps, adding that layer of fluency.
The final thing I'd say is that one of the other benefits of ChatGPT — and this is the other part of the name; that's where the "PT" comes in — is it's pre-trained. You don't have to do a bunch of training. A lot of the problems of these old AI models was you'd go to a vendor and say, "Here's all of my tickets and my customer success thing." And then you'd go through the exercise of labeling these tickets, or labeling where the products are in there, or identifying if the customer's angry, things like that. So you had this massive up-front cost to get set up with AI. But now we have these pre-trained models. They can be good at a variety of tasks just by giving them some examples or just by prompting them. So there's a very quick time to value that you didn't have previously with AI.
Kristen: I do think that's a huge benefit. As we look at how that's showing up for us as customers in our lives, we've seen some of that show up on the consumer side of things. What's interesting to me is, we sort of have this attitude — I think — in customer success that having a person is better than not having a person in business-to-business. But as consumers, we've all been trained that we would actually prefer not to have a person. So I think it'll be interesting to see how this creates engagements with customers that are not human-driven in a way that customers might prefer — because we've all been trained not to want to have to talk to somebody to get things done. There's a huge opportunity to have digital, even as part of a program where maybe there is a CSM who's more of a strategic contact point for the client. I think there's so much that could be done digitally, and that will have a huge benefit for customers in B2B.
Todd: I agree completely. I'm in the same camp as you, where I prefer to talk to a bot if it's a good bot. And then if there's a unique situation that comes up — either because of banking, or because of travel, and I need to solve something — I want that human I'm talking to to be equipped to handle the situation quickly and not have to escalate, or not have to transfer. So it's definitely finding that right balance to get the best of both worlds.
Kristen: So if you're a CS leader, what are various CS tools that are available to you? And how do you decide what to choose and when to implement them?
Todd: For me, I like to look at it through a lens of a capability model versus a maturity model. I'm not sure if those are terms that you've talked about on the show before.
Kristen: If you could briefly explain them, that would be great.
Todd: A maturity model is a model that looks at what sort of level of tools or processes you have in place — level one, two, three, or four. A capability model looks at specific capabilities and outcomes.
The analogy for talking about human development would be: a maturity level would be, are you a toddler? Are you a teenager? Are you an adult? There's a linear progression, and certain capabilities or attributes we associate with each. On the capability side, it's more about asking, well, can you walk? How far can you walk? Do you know how to drive a car? Do you know how to drive a boat? Can you fly a plane?
Translating this to customer success: a maturity model might look at, well, "level one is you have a CRM in place; level four is you have omnichannel communications and it's real-time and it's all automated." The capability model is looking at outcomes and capabilities. So it might be: when an agent picks up a ticket, do they have the context of prior interactions? Or do we have alerting in place to make sure we're actually able to meet our SLAs about response times, even when someone's on vacation? That's why I think it's much more powerful — it's actually talking about specific interactions that are going to drive customer value as well as business outcomes.
In terms of the tool and technology space, there's a bunch of value in looking at Gartner, looking at G2, looking to see how they categorize the space, just so you can see different maps of it. There are categories for contact centers, things like CRM, customer engagement, digital adoption platforms, learning management systems, business process outsourcing. You can look across those spaces, see how the different vendors fit in. Different vendors are going to appear in multiple places.
But really, it always just comes back to: you can understand the space that way, but you still want to come back to business outcomes. For me, it's about how do you find the highest bang-for-the-buck tools? The classic way to do that is you make a two-by-two quadrant of effort versus impact. You say, we want to find the things that are going to drive the most outcome for the smallest amount of effort.
Often it depends on where you're at on your journey. But a big one for me is the capability of being able to collaborate on customer support tickets. Sure, you can do that with a shared email inbox. But then a step up is using a tool like Zendesk or Salesforce, because then you have capabilities around being able to assign cases and ownership, SLA tracking, internal conversations seamlessly, things like that. Does that kind of gel with what you've seen as well?
Kristen: Yeah. I mean, I think in customer success, some teams have support, some CS teams don't. I think there's so much communication, though, in either case that could be handled in a more seamless, automated way. There's lots of room for that.
What do you see as some of the challenges or limitations, or maybe even ethical issues around AI for the field of customer success?
Todd: What's fun about any new emerging technology is that we're learning its boundaries every day. ChatGPT in particular has some unique, interesting challenges.
The first one I'll point out is what people call hallucinations — LLMs can just make up facts, make up information. The reason this happens, going back to the methodology and how these things are trained, is that they're not optimizing for what's accurate or what's actually true. What they're optimizing for is sounding good or sounding fluent. That's where you run into those issues. It can even make up things like citations, make up statistics, make up math.
Kristen: Oh, wow. That's kind of a big problem.
Todd: Yeah. There's a way to solve it — for each of these, we can talk about the way to solve it. For hallucinations in particular: instead of saying "ChatGPT, go generate a response for me," what you instead do is you give it a list of things to choose from, or you give it some content and you say "summarize this content." When you're giving it those sorts of tasks, it's much better able to identify that it's only supposed to be using that as the source material, versus giving it all human knowledge and saying "what's relevant out of this entire space of human knowledge?"
Kristen: Yeah, got it. Okay, cool. What else?
Todd: Another big one we see for customer success teams that deal with real-time communication is response time. If you go try the ChatGPT interface, what you'll see is it streams the output back to you, but the entire response may take anywhere between five and 15 seconds. Depending on how you're using generative AI in a real-time setting, those response times can become an issue if you're dealing with chat. The way to solve that is there are actually different kinds of AI models out there, and there are different trade-offs between how fast they are and how accurate they are. For example, Google PaLM and Google Bard are faster than OpenAI, but they're a little bit less accurate. So depending on what kind of tasks, they might be a better choice.
The other piece is thinking about what kinds of instructions, or what you're actually having the model generate. We often see customers having it generate boilerplate or things that are consistent across all responses — and you can significantly speed things up by instead saying, "Hey, ChatGPT, just put a placeholder here. You don't have to generate the whole legal disclosure. We already know what that's going to look like."
Kristen: Are there any other challenges that you run into in our field?
Todd: The other one — and this applies to more full digital, and this one's kind of fun — is that they can kind of go off the rails. The anecdote here is when Microsoft first released ChatGPT in Bing, there was a story of a New York Times columnist starting to chat with it, and eventually it started telling him to leave his wife or partner, run away with it — they could be happy together. When you're leaving these things unmediated, if there's no guardrails, they can go off the rails.
The route Microsoft took in that case was: they just limited the length of the interaction. They said, "Hey, you can't have more than four or five messages," because they found that with so few messages, it's hard for it to build a relationship — at least strong enough that it would run off and get married, or elope.
The other thing people do is add another layer or another model on top that's basically playing traffic cop. It's just going to start looking for different things that might be happening in the conversation — if it's going toward certain topics, or if certain language is being used — basically saying, "No, no, no, look, let's put this back on the rails."
Kristen: I like that — especially for something I've been thinking about a lot lately, which is regulated industries. There are certain fields where there's a lot of confidential information. I'm thinking in particular healthcare. If you're in the health tech space and you have a CS team, the data you're dealing with can be very confidential. How do you make sure that doesn't end up in the wrong piece of communication, or out in the wrong hands, and put your company at risk? That idea of a layer that traffic-cops things is a good one.
Todd: That's a fantastic point, and a big problem. It goes back to: you shouldn't just use ChatGPT as the interface, because you're encouraging people to copy information out of the other system into that tool. Typically what you want to look for is tools that allow you to just extract specific information from the page — our product lets you do that. And the other piece, as you described, is: can you add a layer for data scrubbing, or data redaction, so that if something does get through — like a credit card number or social security number — it gets scrubbed before it actually hits the AI.
Kristen: So if you were a customer success leader today and you were wondering what to do about AI — I mean, there's a lot of people out there for whom "artificial intelligence" is an intimidating term. They're like, "Oh, that's for the highly technical people, and I'm a CS leader, I'm all about relationships." There's a lot of people who are really intimidated. What recommendation would you have for them, even just starting to understand the technology and think about how to leverage it in their program?
Todd: I tell everyone: sign up for a ChatGPT account and just start using it. The really big innovation ChatGPT had was that they made it a chat interface so anyone can use it. They didn't have to use specialized software. Everyone's familiar with using WhatsApp or iMessage. I just encourage everyone to give it a try.
Often I'll find people who are self-proclaimed AI influencers, or even AI investors, who don't actually use AI in their everyday life. They don't proactively go out and look at it. So really, go out and try it. Just experiment. Ask it to summarize things. Ask it to explain things. Next week I'm going on vacation to Europe — ask it to tell you, "Hey, I'm visiting this city. What should I do?" Ask it to tell you a joke. And then ask it to tell you another joke. What you'll find is that sometimes it tells you the same joke over, or that it's not very funny. By playing around with it, you start to understand: what are the characteristics of it? What is it good at? What is it not so good at?
The other thing I'd recommend is then signing up for things like Google Bard or Anthropic's Claude or these other models, and see how they differ — in speed, in fluency, in accuracy.
The other recommendation I would give — and I gave this a little bit earlier — is go look at your existing tools and see if they're in the process of rolling out AI. A lot of these are going to have early access programs where you can sign up. They'll also have webinars and videos that show you how it works in the context of the tools that you're already familiar with. So I think that's a great way to gradually bring it into your lives.
And then finally, I'd pitch our product for our customer success mod pack, which includes an AI prompt manager, writing assistant, and knowledge base assistant. The idea is it comes with pre-written prompts, so you can see how other people are effectively using AI as part of their customer success role, or marketing role, or sales role. You can also customize things using Google Sheets, and they'll be available to all your agents on the pages they're working on. It puts AI at their fingertips. Really, just find those bite-sized places where you can start playing around with it — that's going to be the best way to understand it.
Kristen: One thing I would add that I found really helpful recently is in the July/August Harvard Business Review — there's a series of really amazing articles about generative AI and human creativity and the intersection there. Some of the ways they explain it were really helpful. I think that's a great tool if you want to go and do a little reading on it. They also bring up some interesting things about artists' rights and copyright, and some of those challenges that I think we probably face less of in customer success, but I think are interesting aspects of AI that we need to be thinking about as a society.
And I think you mentioned before about ethical issues as well. Another fascinating one that comes up in customer success is: when should you disclose that the customer is talking to a bot, instead of a human? Or when should you disclose that a customer support agent — or your customer support manager — is using AI to augment them?
Todd: I kind of fall in the camp of: if it's fully automated, you should definitely be disclosing — not even just for ethical reasons, but for expectation-setting and maintaining trust with your customer. And then really, it's the assistant that becomes more of a gray area. We wouldn't disclose if I was using grammar check to write an email. I wouldn't necessarily disclose if I was using Google Translate to help translate a message to you. So that's kind of my bright line: if there is a human involved, I don't think you necessarily need to disclose. But if it's fully automated, it's probably both ethical and also just best practice to disclose there.
Kristen: That's a really good point. Okay, last question. This is a chance to offer it a little bit if you want to — but what do you see as the biggest trend in customer success right now, and why?
Todd: What we see repeatedly is that teams are being asked to do more with less resources. There are a couple things driving that. In the States, interest rates are high. VC fundraising activity has slowed. So companies are having this renewed focus on profitability.
But at the same time, due to competition and also just generational preference, there's this increasing need to meet the customer on their own terms. We talked about, okay, some customers want to talk to a human; some customers don't want anything to do with the human. It's meeting them on their own terms, their own style of communication. And then also there's this need to deliver value enhancement and not just customer support — not just necessarily solving the problem at hand, but really helping the person get the most out of their relationship with the company or their relationship with the product.
I think success teams have rightly identified AI as a real transformational technology to help reconcile these competing demands of doing more with less. That's why I was really excited to come on the program — because I think this is a very timely conversation.
Kristen: Absolutely. Well, Todd, I really appreciate you taking the time to talk with us about artificial intelligence. It's such a big technology shift in our field. I really appreciate your expertise and your ideas. If someone wanted to reach out to you directly, what's the best way for them to get in touch?
Todd: My email is todd@pixiebrix.com. And then I'm pretty
easy to find on LinkedIn as well.
Kristen: Awesome. Well, thanks again. And I also want to thank our producer, Russell Bourne, and our audio experts at the Auraform audio team. This podcast is a production of Success League Radio. To learn more about the Success League's consulting and training offerings, please visit our website, thesuccessleague.io. For more great customer success content, follow the Success League on LinkedIn or @TSLCustomers on Twitter. You can subscribe to Success League Radio on Apple, Google, Amazon, or anywhere else you get your podcasts. Thanks for listening. We hope you'll join us next time.