Can a bot truly replace the warmth of a human reply?

Learnings of implementing AI for customer support at Refrens

“AI won’t be able to answer correctly.”
“It may give wrong answers, which will frustrate the user.”
“It lacks empathy.”

Hello! I’m Shreya Lakhera, a Founder’s Office intern at Refrens, and this was my very first task: integrate AI into our customer support system.

AI has always intrigued me in theory, so this was the perfect opportunity to apply my research-driven mindset in a real business scenario.

You’d think a computer science degree, hundreds of YC blog reads, and hours spent exploring AI tools would prepare you for integrating AI into a company’s workflow. Spoiler alert: it doesn’t.

But I’m grateful to Naman Sarawagi (Founder @Refrens), Vaidik Dalal (Product Manager @Refrens), Shubham Mishra (Support Executive @Refrens) and Mitesh Kariya (Content Marketing Head @Refrens), whose constant guidance made the journey smoother.

So here’s how an intern (yours truly) tackled the massive challenge of using AI to support over 100,000 Refrens customers.

Step A: Why AI in Customer Support Is More Than Just a Buzzword

We already had an efficient customer support system in place.

So the first question we asked ourselves was:
Do we truly need AI in customer support?
Can it actually improve our existing workflow or is it just another trend?

Because let’s face it: the buzz around AI often exceeds its actual value.

Many businesses implement a basic 3-question bot, call themselves “AI-first,” and wonder why their support experience still feels broken.

But the real AI implementation goes beyond your landing page copy. 

It needs to solve the deeper human problems that tech often struggles with—like empathy, context, and customer satisfaction.

Or, in our founder’s lingo, it has to optimise the workflow of our support team, AND continue providing the quality customer experience that Refrens is famous for.

This made our goals clear.

1. Reduce repetitive tasks for our support team.
2. Speed up first-response times for over 100,000 businesses using Refrens.
3. Enhance the human touch in customer interactions, not replace it.

Step B: Laying the Foundation, What Your AI Needs to Be Effective

So, what do you need to make your AI work for your customer support?

Here’s what we learned the hard way:

1. Data is Key, But Relevance > Quantity

We started by feeding CRISP BOT general web data about Refrens and everything that we do at Refrens, thinking it would make the bot a powerhouse.

We were wrong.

CRISP BOT confidently answered questions but often got them wrong. 

Imagine getting a step-by-step guide on how to delete your Refrens business account when all you asked was how to delete an invoice.

That’s when we realised: Breadth ≠ Depth. 

AI needs contextual, relevant data to be effective.

So, we pivoted. We narrowed its focus to internal HelpDesk documentation, the content our support team used every day. We updated it regularly, and suddenly, CRISP BOT became more reliable, showing that AI thrives with relevant, well-curated data.

2. Good AI Starts with Good Prompts

Here’s where the real magic happened: prompt design. 

Our Product Manager, Vaidik, wisely said, “The soul of the bot lies in the prompt.”

It wasn’t about stuffing CRISP BOT with more data; it was about asking it the right questions. Small tweaks in how we framed instructions and clarified intent made a massive difference. 

The bot began to improve in tone, clarity, and confidence. 

We never really intended to make our AI sound human; we aimed to make it helpful, and the human thing just followed.

Step C: Bringing It All Together: Implementing AI in Customer Support

Now that we had the foundation laid, we moved on to the actual implementation. 

1. Setting Clear Objectives

We set clear goals for what we wanted CRISP BOT to achieve:

• Reduce repetitive tickets by automating routine queries (e.g., “How do I download my invoice?”).
• Improve response times for users by getting quicker answers to common questions.
• Ensure valuable assistance by escalating unresolved or sensitive issues to real agents.
• Provide support by filling in for queries and providing an immediate response during off-hours.

2. Real-World Deployment: The Numbers That Surprised Us

In just 105 days (Mar 18 – Jun 30), CRISP BOT transformed the way we approached customer support:

• 3,594 out of 4,476 conversations were automated (that’s 80% of incoming chats).
• 1 out of 4 conversations are resolved by the Crisp Bot. (~26% automation rate)
• 1,024 conversations were fully resolved by CRISP BOT alone (~26% automation rate). 
• Our first response time has reduced by almost 20%
• The human response time dropped to just 2 minutes (~59% faster handover).

Most importantly, our human team could focus on more significant problems: refunds, billing issues, and emotionally sensitive queries, leaving the routine stuff to CRISP BOT.

Step D: The Challenges, Where AI Falls Short 

Of course, nothing is perfect. Here are some cringeworthy moments we encountered:

• A user typed “hi”, and the bot escalated it immediately.
• A user asked to change their billing address, and the bot failed to clarify which address.
• The bot mixed up cancel order with cancel Refrens account, leading to confusion, giving user details on how to remove their account!

AI excels at routine tasks, but it struggles when things get ambiguous. And that’s when humans come in.

From ABC to Z: Humans + AI

Here’s the part where we look ahead and aim for Step Z, the ultimate goal of implementing AI in customer support. 

Step Z isn’t just about automating the most repetitive tasks. It’s about creating a seamless blend of AI and human empathy.

AI doesn’t replace humans. It empowers them.

Step Z is about using AI to understand customer queries, predict needs, and adapt in real-time. 

By refining the training, gathering user feedback, and building systems that learn continuously, we will reach a point where AI acts as a critical partner to your customer support team, enhancing both speed and empathy.

But when things get complex or emotionally charged, humans are still needed. Together, they create a seamless customer support system where AI handles the routine, and humans step in when things need a personal touch.

TL;DR: How to Implement AI in Customer Support

1. Start with a clear goal – Identify what you want AI to solve (e.g., repetitive queries, faster response times).
   
2. Use relevant internal data – Don’t overwhelm the bot with broad content. Train it on helpdesk docs and real support conversations.
   
3. Write better prompts – The quality of instructions matters. Frame prompts clearly to guide tone, structure, and intent.
   
4. Deploy gradually – Test in real conversations, monitor escalations, and keep humans in the loop for complex issues.
   
5. Track performance – Measure automation rate, resolution quality, and response time improvements.
   
6. Keep refining – Continuously update training data and prompts based on feedback and bot performance.

At Refrens, I learned that AI is a tool and when used thoughtfully, it doesn’t replace people. 

It helps them do what only they can do: connect with customers on a deeper, more human level.