How Calendly's CPO & Head of UX Prioritize AI for Real Customer Value

"We found users actually became more critical of features explicitly branded as 'AI-powered"

Steven Shu, Chief Product Officer at Calendly Steven brings over 8 years of AI expertise to Calendly. Previously, he led design teams at LinkedIn where he built recommendation products for product onboarding and developed AI chatbots. His background combines deep product strategy with practical AI implementation experience.

Jess Clark, Head of User Experience at Calendly Jess leads Calendly's user experience initiatives, working closely with Steven to create intuitive scheduling experiences that help people connect more efficiently and achieve their desired outcomes.

4 Unexpected Learnings:

1. Parallel AI experiences often fail: Calendly's experiment with a conversational scheduling chatbot couldn't retain users because the original experience was more efficient. Building separate AI interfaces can create unnecessary tech debt and learning curves.

2. Meeting intensity KPI challenge: Sometimes AI efficiencies can reduce a company's core metrics (like Calendly's "meeting intensity"), requiring leadership to make conscientious decisions about value tradeoffs.

3. Don't fix what isn't broken: When something in your app already works well and delights users, don't force AI into it just for the sake of innovation.

4. The 86% loyalty factor: Companies that provide strong onboarding and continuous educational experiences see 86% higher customer loyalty rates - making AI-powered personalization a critical retention tool.

Navigating the AI Transformation Journey

Calendly is taking a comprehensive approach to AI implementation across its entire customer experience - not just within the product itself. As Steven Shu explains, "AI has the power to transform the customer experience within an app, across apps, and in every way that products overlap with human experience."

The company aims to change customer habits holistically, similar to how they revolutionized meeting scheduling with their signature scheduling links. This approach comes at a critical moment when consumer expectations for simple, intuitive experiences are higher than ever.

"We're seeing a transition from the initial 'delight moment' with AI to focusing on creating a more retentive experience," Shu notes. This shift demands that companies prioritize user experience and adoption while ensuring clear ROI.

The Three Key Pillars of AI Product Development

1. Prioritization: Focus on Real Problems

Effective AI implementation starts with prioritizing problems that genuinely matter to customers while aligning solutions to business goals. For Calendly, this means enhancing user productivity.

"Calendly solves the problem of scheduling meetings to help users close deals, connect with partners, and ultimately close business," says Shu. "Delivering real value is crucial - products should aim to hit the bottom line of productivity for teams and organizations."

The challenge many enterprises face, according to a recent Menlo VC report, is failing to deliver on AI's promise due to a lack of workflows and habits in users' daily lives. As Clark points out, "Beyond creating a delightful experience, it's essential to consider what will stick with users not just in the app but everywhere."

Fast-tracking personalization has become critical, especially for PLG companies. AI can transform experiences beyond initial onboarding, extending personalization to websites, support channels, and other touchpoints to help users discover advanced capabilities.

2. Building Confidence Throughout the Customer Journey

To create the optimal experience, companies must instill confidence across the entire customer journey - both within and outside their applications. Shu and Clark highlight four key approaches:

Make AI visible only when it matters "Research shows people are getting exhausted with the term AI," says Clark. "It should be integrated seamlessly, not as a parallel experience, building on the existing product and customer experience."

Avoid parallel experiences Building separate AI experiences within apps can be costly in terms of tech debt and user cognitive load. "Users have to relearn a different part of the product in a different way to get things done," explains Shu. "Many companies that have built chatbot experiences in their products have failed to get them to stick, with users often preferring the original core product."

Calendly learned this lesson firsthand when their conversational scheduling chatbot experiment failed to retain users. "The original experience was more efficient and better," admits Clark.

Set clear expectations "When delivering an AI product, be clear on what can be done and what can't be done, and make sure the inputs and outputs are clear to the customer," advises Shu.

Build in error handling mechanisms "When in doubt, build in mechanisms to communicate error states or misinformation to continue building confidence and trust throughout the product," says Clark.

3. Respecting Business Guardrails

Implementing AI requires careful consideration of key business tenets:

Value measurement Success must be tracked through clear metrics. Calendly uses "meeting intensity" (frequency of meetings booked) as a KPI, though they recognize that AI efficiencies might sometimes reduce this metric - a tradeoff leadership must consciously accept.

Cost management "Generative AI can be expensive," warns Shu. "Companies must decide whether to pass costs to customers or absorb them in their margins."

Data quality and architecture "Data performance and quality matter more than ever," emphasizes Clark. "Architectural discussions should support the entire company, not just product and engineering."

Unified user experience "Users should have a single, unified engagement experience, regardless of what happens behind the scenes," Shu explains.

Creating a Holistic AI Experience

A cohesive AI strategy requires organization-wide effort and leadership buy-in to avoid "shipping the org chart" and creating disjointed experiences. This includes:

1. Building a robust data architecture where AI agents speak the same language

2. Consolidating help content and streamlining content creation

3. Developing custom chatbots for specific personas

4. Maintaining consistent voice and tone through company-trained AI models

"To uplevel customer experience with AI, focus on making it easy for users to achieve their goals, avoid adding friction, and ensure that AI innovation is actually making the experience smarter and safer," summarizes Clark.

Measuring Success and Adapting

Success measurement should include:

• User engagement metrics

• Task completion rates

• Customer churn and retention

Regular adjustments should be made to AI products through:

• Retooling prompts

• Backend optimizations

• Continuous user testing

• AB testing methodologies

Most importantly, the product must maintain a genuine product-market fit with customers. As Shu concludes, "Gathering feedback early and often is crucial to making informed decisions about investments that yield returns for the customer."

The companies that will win in the AI era are those that prioritize problems that truly matter to customers, maintain consistently high quality experiences, and take a holistic approach to every customer touchpoint with their brand.

4 Things That Didn't Work Well in AI at Calendly

1. Conversational Scheduling Chatbot: Their attempt to implement a conversational interface for scheduling ultimately failed to gain traction. Users consistently preferred the original, more streamlined scheduling experience, demonstrating that sometimes the non-AI solution is actually more efficient and user-friendly.

2. Generic AI Recommendations: Early attempts at providing AI-driven suggestions without sufficient personalization or context awareness led to low adoption rates. As Shu notes, "Customers quickly dismissed features that didn't clearly understand their specific use case or scheduling patterns."

3. Over-branding AI Features: Initially, Calendly prominently labeled and marketed their AI features, which created heightened expectations. "We found users actually became more critical of features explicitly branded as 'AI-powered,'" explains Clark. "The moment we stopped emphasizing the AI and just focused on the value, adoption improved."

4. Cross-channel Consistency: Maintaining a consistent voice, tone and knowledge base across different AI touchpoints proved challenging. "Our first attempts at implementing AI across customer support, product tours, and in-app assistance created disconnected experiences where the AI seemed to have different personalities and knowledge levels depending on where you encountered it," admits Shu. This underscored the importance of a unified data architecture and comprehensive training approach.