Support Stack E14: The Missing Link Between QA, Docs & Fin AI, with Thomas Hils
Most customer support teams treat QA, documentation, and AI as separate systems.
In practice, they are tightly connected — and when one breaks down, the others follow.
In this episode of Support Stack, Conor Pendergrast is joined by Thomas Hils, Head of Support at Wingspan, to explore how to bring these elements together into a single, continuous improvement loop.
The conversation centres on a simple but often overlooked truth: QA cannot be effective without clear, structured process documentation. And AI tools like Fin cannot provide accurate answers if the underlying knowledge does not exist.
Thomas shares how his team uses real customer conversations as the starting point for improving their systems. By analysing tickets against existing documentation, they are able to identify gaps, generate new processes, and refine existing ones. This creates a feedback loop where QA doesn’t just evaluate performance — it actively improves the system itself.
The episode also explores how AI can support this workflow, from analysing conversations to suggesting documentation updates and highlighting inconsistencies. Rather than relying on AI to “figure things out”, the approach focuses on giving AI the structure and context it needs to perform reliably, especially in complex or high-compliance environments.
Key themes include the relationship between QA and documentation quality, the importance of AI-readable content, and how to avoid overwhelming AI systems with too much or poorly structured context. The result is a more resilient support operation, where processes are continuously updated and both human agents and AI tools are better equipped to handle customer queries.
If you are working with Intercom, Fin, or exploring AI in customer support, this episode provides a practical framework for improving resolution rates, reducing reliance on tribal knowledge, and building a more scalable support system.
Join my daily email list to get emails about future episodes.
Episode transcript
Conor Pendergrast (00:00)
Hello, I'm Conor Pendergrast and welcome to episode 14 of Support
So, Thomas Hils, Head of Support for Wingspan, how are you today?
Thomas Hils (00:16)
I'm doing quite well, very excited to be here and to show off some of what I built.
Conor Pendergrast (00:20)
Excellent. So this is our second episode with you. ⁓ We are talking today though about how you were forever AI-pilled from the very start. You were obsessed with AI and you were like, yes, this is the best thing since sliced bread. ⁓
Thomas Hils (00:36)
Yeah, I have been a hard AI skeptic for years in the support context. ⁓ I used to joke that like, it'll come for engineers jobs before it comes for customer support. I don't think I was wrong on the framing. I just didn't quite understand how quickly it would become useful and applicable to complex products, which is kind of my bread and butter support for these kind of technical or otherwise, like very investigative heavy tools. And now
You know, every six months, every two weeks, I have to reevaluate how AI can be helpful. And we've kind of hit this tipping point and now it's in everything that I do.
Conor Pendergrast (01:14)
Yeah, absolutely. last episode, episode 13, we talked about how you use Claude Cowork as a part of your QA process to review conversations. But today is a slight twist on that as well. So set the scene first. Wingspan has a long history of extremely well-documented processes that you have never had to fill in the blanks on. Is that right?
Thomas Hils (01:38)
Yeah, you know,
we've had ups and downs with our process documentation, but largely and very honestly, as many, think small startups will find your documentation, even when you have a best effort is just lacking. And for us, one of the kind of pivotal moments is we went through a banking transition because we are a payroll company back in the summer of 2024, where everything we previously knew about
Conor Pendergrast (01:43)
Yeah.
Bye!
Thomas Hils (02:06)
the tool and how to support it changed. Just we had a hard refresh and due to some of the timing of that, we had to kind of figure it out as we went along. And the result of that was not good documentation. Largely what we did end up making was quickly outdated or it wasn't thorough. And for the folks on the team who were kind of experienced had been there for a while, it was doable. But as we started adding new people to the team, it was very, very clear to us that
This was not sufficient and we needed to do better.
Conor Pendergrast (02:37)
Yeah, absolutely. And I will say, so we referenced this in the last episode in episode 13, where one of the things you're looking for in QA and conversations is compliance because it is utterly critical. And from my experience at Work and Expensify, we were also dealing with bank transactions, ACA transfers, credit cards set up, all very high risk if you don't get it right.
and working through those compliance processes is absolutely critical. But before you can work through compliance processes, you have to have the compliance processes.
Thomas Hils (03:11)
Yeah, absolutely. It's great that, you know, at companies like this, we all go through these compliance trainings through whatever third party company we've contracted with. And I think that, of course, those are helpful and they give you context, but it's important that you're able to adapt them to the on the ground experience that folks are having in support. And then be clear, there's not a lot of room for ambiguity when it comes to these kinds of compliance risks. And so that was like, you know, one of the first places we were most doubling down
making sure that that process documentation exists and that it is applicable.
Conor Pendergrast (03:43)
Yes, and so you, I'm curious, what came first? Did you think of building on top of the existing QA process to improve your processes, your written processes? Or was it the other way around? Were you first tackling trying to improve process documentation first and then got into QA afterwards?
Thomas Hils (04:02)
they were happening simultaneously, but the realization was that these are the same problem there. It's part of ⁓ a similar cycle. And so I combined them pretty early on, ⁓ after, know, getting some pretty poor results from the QA process without this context. ⁓ and then what occurred to me quite quickly is, wow, what we do have is very limited and you can't do QA this way without having these really detailed docs.
Conor Pendergrast (04:30)
because you might intuitively know that someone has followed the right process, but that doesn't work effectively if you are trying to get anyone else, including an LLM like Claude or Claude Cowork, to do those QA processes. They can't follow what's not documented. And it's very similar. I said this the last episode. It's very similar to Fin and Fin's content. Fin cannot give content that doesn't exist.
Vin cannot share information that doesn't exist, so you have to start there.
Thomas Hils (05:00)
Yeah, and it's very surprising how much of, like, I think what we create in traditional documentation does require an element of context that isn't actually written down anywhere. There's these connections between A and B that people assume that an AI can't.
Conor Pendergrast (05:16)
Yes, absolutely. Yeah, yeah, yeah. And often if an AI is inferring connections, it ends up inferring the wrong ones. So let's just avoid that entirely and just document the process well instead. you added it on as part of your QA framework. maybe if someone hasn't seen, again, I'll call back the last episode. If you haven't seen episode 13, go back, take a look at what Thomas was showing us last time.
Thomas Hils (05:24)
Absolutely.
Conor Pendergrast (05:43)
Why don't we dive in and see what it actually does? So as a, like, what's the one sentence for the background for how this works?
Thomas Hils (05:50)
Yeah, one second sentence is tough, but I'll make it work. What it does is it takes real conversations and compares that conversation and what happened in it to a existing database of processes. And it will identify gaps either in a existing doc for this situation or whether a doc exists to handle this type of situation.
Conor Pendergrast (05:53)
Fine, you can go too if you want.
Thomas Hils (06:17)
and then generate one if it doesn't find it.
Conor Pendergrast (06:19)
Perfect, I think that was two sentences. Okay, so ⁓ you're gonna show us how this works in Claude Cowork. And just like with the QA, it is using Claude Cowork, it's using the intercom MCP, which is a type of connection between computers, we'll say. And you've got the skills built into Claude Cowork that goes through this whole process.
Thomas Hils (06:40)
So in this case, it's pulling a specific ticket, ⁓ although it can run this just, you know, grab five tickets and check. ⁓ That's how I've been trying to get like more broad coverage. And then it's going to read the conversation, summarize it to itself, ⁓ then try to get an understanding of the product using our product knowledge base. Like what is being asked for here? What are these elements of the product? And then it's going to go ahead and compare that to our existing process documentation.
Conor Pendergrast (06:42)
Mm-hmm.
Thomas Hils (07:07)
So most of this documentation, what we actually did is we fed it all of this context of existing documentation that we had for our internal support team and some very clear kind of decision tree logic that we had created for a BPO team, which I think is probably the more useful part of this context training to be perfectly onyx. And then gave it this prompt, like, hey, these are documents we want you to be able to reference when doing a ticket QA and. ⁓
when doing this kind of process generation work. So structure it for that. And so that's the bulk of what it is looking at now to see if existing documentation exists. And that is the end
of what it would generate either an update to one of those or a brand new doc that does that. So here it's kind of ⁓ identified the ticket and the core issue that someone was having. ⁓ And it's identified that there's a missing process. So that's like,
A is there's no doc at all, B is something that's not quite covered, or I might have that flipped, so bear with me. ⁓ So there is no troubleshooting doc for this thing, invoice content ⁓ questions. So what in the invoice someone from our product has received, what do they do about it? ⁓ It's got some kind of references to existing docs, but those...
Conor Pendergrast (08:10)
interesting.
Thomas Hils (08:30)
don't actually cover it. And so this is something I'd like that it doesn't just stop at one source. It likes to crawl through everything because stuff is spread out in a way that it understands and speaks to the different ways that these processes might touch each other. ⁓ So it's this web. And now it's kind of offering what it thinks we should do. So in this case, a new doc with a specific title and what it should cover, how it should go through that decision tree, diagnostic steps.
What I think is kind of interesting about how we landed on these docs is because they're ultimately written for the AI where the expectation is that while the humans do interact with them, they interact with them through this kind of interface through Claude. So that means that the docs can be structured very strangely ⁓ to a human, but very clearly to the AI.
So it'll include things like suggested responses, it can include decision trees, and then also these kind of cross-document references. So one of the more important parts of this process for us is that all of these process docs ⁓ carry a list of product areas and product features that tie to docs elsewhere in our kind of database that are AI definitions of those products and features. And...
Conor Pendergrast (09:39)
Okay.
Thomas Hils (09:48)
You know, this is kind of an advanced step, but our docs will watch for changes in those products and feature docs and then assess whether or not the changes that happened there are impact this process. Our goal is to get to this much more fully AI driven process. That is a long-term goal. but just out of the gate, being able to take a conversation, look at your existing content, whether that's written for AI or not.
and determine whether this question could be answered by that content, I think is super powerful. And then of course, getting this out. And it even talks through why it thinks this suggestion matters. ⁓ What's not being covered,
whether that led to a bad answer or whether a human had to have known this to generate the answer that it gave.
Conor Pendergrast (10:37)
Yeah, yeah, absolutely. So what's really interesting about this is that Fin is involved and the teammates are involved. just to make it, after you create this process, is this something that Fin accesses as well or is it just for teammates?
Thomas Hils (10:55)
So right now it's just for teammates. but there is of course another process I've built where this, you know, it's, there's a lot of things looping together here, but these changes to that process documentation, I do, run checks against them and our compare that to our Fin content, which we've got downloaded in local, and it'll suggest improvements there.
Conor Pendergrast (11:00)
Of course, you're running another project.
Thomas Hils (11:20)
So this is like a connector step for us between like our internal AI resources, ⁓ which we're very lucky at Wingspan to have many built out ⁓ and connecting that then ultimately to these customer facing resources. It saves me quite frankly, a considerable amount of time versus me getting ⁓ a product change and then having to manually go through and understand how that would impact that existing guidance or content there.
Conor Pendergrast (11:46)
Yeah. So have you, I'm just curious, have you got to the point where you have found these sort of processes and thought, actually, this is something that Fin should just handle. Has anything spun up from that to say like, okay,
let me turn this into a, either something simple like an internal article or something more complex like a procedure.
Thomas Hils (12:07)
Yeah, we've ⁓ an outcome of this has been quite a few ⁓ new or updated ⁓ articles to be put into fin, whether they are external facing or internal. We have this skill that assesses which piece of fin content is right for the context that the bot suggests here. Again, there's layers of complexity here that I felt that actually aren't required. But this is just how I think in systems. And I think that's been really
I think our fin often trailed in content that the team would know, but fin wouldn't. ⁓ We're all spread thin. We're all trying to do our best to keep all of the different plates spinning. And this has made it a lot easier to get content very quickly into fin and identify where we've got gaps.
Conor Pendergrast (12:56)
Yeah, excellent. Okay, so that's really interesting. And so you include this as part of your QA process now. ⁓ It runs naturally every day and picks up conversations. It compares them to your processes, which you've done a really good job. sounds like you and your CTO have done a really good job of defining processes, writing them down somewhere that Claude Code can access them, which is the really important part.
Thomas Hils (13:11)
Yeah.
Conor Pendergrast (13:24)
And then it suggests gaps in those processes and even goes so far as drafting the changes that that Claude Code should make.
Thomas Hils (13:34)
Yeah,
it queues it up and frankly it's gotten quite good at ⁓ the suggestions. ⁓ Whether that's because it's learning from the human resolution to the ticket. So if it's pulling a ticket where a human has done the thing, ⁓ it'll raise to me, was this right? If it doesn't have the level of confidence it needs to be like, yeah, that makes sense. ⁓
And if it is, the documentation it typically produces is quite accurate to the process that we are doing. Very rarely do I have to tweak it. I think that happened a lot more earlier on when we first started generating these things, but I think that was a result of just how much content was missing.
Conor Pendergrast (14:16)
Yeah, of course. Does this rely mostly on internal notes that your team are leaving on conversations to show their work?
Thomas Hils (14:23)
Yes, so that was a tricky piece. Not every time when someone resolves a ticket, do they clearly articulate what they've done. ⁓ So where possible, it's looking at notes on the ticket. ⁓ What it does a great job of is integrating also with linear, which is how we handle our escalations. And so what we had documented pretty well was like a lot of very basic stuff where there wasn't much diagnostics that happened or those diagnostics were pretty clearly defined.
Conor Pendergrast (14:30)
Yeah, of course.
Thomas Hils (14:52)
⁓ And where that isn't the case often does result in a escalation because we do financial operations and compliance operations. We create a lot of escalations to those particular teams. And ⁓ that helps feed so much context to it because the end of the process is often create the escalation. And part of the work is described in that linear ticket because a good escalation in my mind,
Conor Pendergrast (14:52)
Mm-hmm.
Thomas Hils (15:19)
walks through the exact problem a customer's happening, what you tried to do to resolve that problem and where there's that gap. And that is a perfect training model for ⁓ this type of process documentation generation.
Conor Pendergrast (15:32)
Yeah, okay, that's so interesting. I hadn't thought about the linear aspect of it as well, because there are probably situations where it gets to linear and maybe there are engineering fixes that have to happen to resolve an issue. So it's not a process gap, but it's a bug or it's something that you haven't encountered before. But certainly a lot of the time it will be the fact that there's that Fin and the teammate.
didn't understand the process or didn't know the process or hadn't been documented. And that is a key way of speeding up that resolution for a customer by writing down that process clearly for the teammate to tackle it without having to go through an escalation process later.
Thomas Hils (16:13)
Yeah, absolutely. ⁓ we've had a lot of success just being able to, ⁓ so I've built a version of this for my agents that when they're solving tickets, they can ask for help. Essentially. just the QA process, ⁓ but a little bit adapted to not spit out a score, but to spit out these steps, and also give feedback on the suggested steps as well. So what we've.
really just tried to do is create as many inputs as possible and like kind of areas of correction, whether that's me going over the suggested doc with, with Claude, or it's someone on the team asking for help on a ticket and then being like, no, that's obviously not what we do. And we can adapt that. And I think where folks ⁓ sometimes get frustrated is that their early versions of this are quite bad and it feels like you're never going to make it better.
Conor Pendergrast (16:48)
Mm-hmm.
Yeah.
Thomas Hils (17:07)
⁓ but there is kind of like this tipping point of context where suddenly it just knows enough to fill in blanks, although we got to get rid of those blanks. but it can understand the general motions of what your team does. ⁓ and you end up with kind of a virtuous cycle, right? That's the term.
Conor Pendergrast (17:26)
Yeah, absolutely. Have you found, because I know that there's a risk of not having enough context. I always feel like there's a risk of having too much as well. So these LLMs do have the context windows and if you exceed it, then suddenly it just kind of all falls apart. I don't know if you found this, but certainly with a client ⁓ with fin tasks, there was one point where I had it. I discovered that fin tasks had a limit of 250 examples of
trigger this task and 250 examples that do not trigger this task. And it wasn't documented anywhere, but it's just that I hit that and I wasn't able to add any more. And at the same time, very quickly realized that having 250 examples is considerably worse than having 30 examples of each. so I am like, all of a sudden that task in particular was false, false, positive. It was just firing all the time for totally unrelated things.
Thomas Hils (18:01)
Yeah
Conor Pendergrast (18:23)
So that was that was an example of me giving too much context and it was just totally overwhelming Fin, poor Fin. Is there have you found that there's some things that you have to give it slightly less access to or a lot less access to rather than the whole of notion for example for a particular skill?
Thomas Hils (18:41)
Yes. So what we do is, is kind of have very specific restrictions when it comes to a step in the skill. So there's a step that might look at the, the, have a very, it's called knowledge base, which is just a silly way to do this, but there is a specific knowledge base with all of this AI context. That's actually separate from the process documentation as well. So you're kind of like,
Conor Pendergrast (18:49)
Okay.
Hmm.
Thomas Hils (19:06)
trying to rein in the context there, and they can also inform each other. So it might look at the product knowledge so that it makes sure it understands what's being said in here and how those things work, and then kind of distill that down for what it should search for then in the process documentation. I think it's especially useful when taking kind of customer language or the way that a customer might describe a problem and then matching that up against the content that you, how you describe it.
Conor Pendergrast (19:28)
Mm-hmm.
Thomas Hils (19:34)
That could be very different. I know you had a great episode about teaching Fin that and I applied that here. It was perfect.
Conor Pendergrast (19:39)
hours.
Oh, perfect. for viewers, that's that's Gabriella's episode 11. So I'll put a link up just just there. And that's exactly creating a glossary for Fin to understand customer language. So it was an internal article. I'm really glad that's that's a really great way of applying it into a different different scenario as well. That's great. Oh, how how interesting. So. OK, so what would you recommend as a good like
starting point for someone who maybe they're a little bit confident in Claude and Claude co-work. They've got the intercom MCP set up. Like what's the starting point for getting, for getting this continuous virtuous cycle, virtuous cycle of improvement of process improvement and documentation are written down.
Thomas Hils (20:28)
Yeah, absolutely.
It's an easy start. I think the first and foremost is feeding what you have for your internal team, whatever documentation that might be,
into Claude and give it that kind of explicit instructions or expectations upfront. I've heard people like refer to how you should instruct Claude as like what you would tell to a new hire. And I think that's especially true right at the start of this. So this is what we are aiming to do.
So in this case, hey, I'm looking to set up a process that makes sure that we have documentation for customer support tickets. ⁓ And I want your help with that. The outcome should be, you know, X, Y, and Z, and then feed it that context. And here's the docs that we have for our internal team. It might be helpful to tell it a little bit about.
the quality of your docs, so an accurate assessment if you feel comfortable telling the truth to Claude, like, hey, we know we're missing content or hey, we think this is exhaustive. That might give it some kind of scoping to this particular problem. And I think it's important then to do one big step, which is ask Claude to format that content in a way that is accessible to AI. You need to give it that explicit instruction because like we talked about with the context windows,
Conor Pendergrast (21:44)
Hmm.
Thomas Hils (21:50)
The way that a human needs to read things is very different and we can condense a lot when we give it that explicit expectation that only an AI is going to read this. And then from there, ask it to pull five intercom tickets and see if those processes are covered and iterate, keep doing that. ⁓ And very quickly you find both the gaps in your documentation, but also are on the path to plugging those gaps so fast. my...
Extra recommendation is then set a scheduled task. Every morning I wake up, I sit down on my computer, there are 10 tickets reviewed for process gaps and suggestions for you to do them. Then I go through the 70 other scheduled Claude tasks I have and the workday is over. that is all these behaviors we used to have to teach ourselves to
for good documentation and good process. can automate some element of this. And I think that's really great.
Conor Pendergrast (22:40)
Yeah, fantastic. Well, listen, Thomas, this has been fascinating once again. I was, I was really hopeful that this would be a, a concrete applicable episode and I'm, and it really is like, think, especially in us when we're working in small, smaller organizations, it can be very tempting to just sort of glaze over the documentation a little bit, but I think this is, this is a great way of, of getting those clear steps written down so that when you hire your next hire.
or when you're transferring stuff for Fin to do it, you make my job easier.
Thomas Hils (23:16)
Absolutely.
Conor Pendergrast (23:16)
Super. Okay, Thomas, where, if there's a, is there a particular destination where you'd like us to visit you?
Thomas Hils (23:23)
Yes, you can find me online at my personal website thomas.town. And there will be all of the contact details and anything else you want to know about me.
Conor Pendergrast (23:33)
Excellent. Well, and dear viewer, if you enjoyed this episode, please feed my ravenous ego by pressing the subscribe button and by pressing the like button and sharing it with all of your closest friends and family members. And you can get daily emails from me about Intercom and Fin by going to customersuccess.cx/daily. And by daily, I mean week daily. ⁓ Or you can do a week daily if you really want.
But I will be back for a future episode of Support Stack. Until then, have a wonderful week, month, year, whatever you want to have. And thank you very much, Thomas, again. And toodalooze, and goodbye, everyone.