
Slack Integration Assessment
Summary
As part of our ongoing commitment to improving user experience, we revisited the Slack integration used alongside PagerDuty. This feature is widely adopted not only by our customers but also by our internal teams, particularly during incident response. The goal of this project was to better understand how the integration is being used and how users feel about its functionality—what works well, what doesn’t, and what could be improved.
Tools
Google Forms
Type & Timeline
Survey
2 weeks
Background
It’s important to constantly revisit and re-evaluate our tools and features to make sure that they are still working well for our users. For this project, we were looking for quick and actionable insights for our Chat Operations team in assessing our Slack app which allows incident response actions to be taken directly from the chat tool. We strongly encourage dog-fooding within our organization so in the interest of moving quickly, we decided that testing with our own internal engineers would be the most efficient way to gather the insights we were looking for.
Goals
We aimed to gather insights on the following:
Common use cases and workflows involving the Slack integration
The roles most actively using the integration
The most frequently performed actions during incident response
User sentiment: likes, dislikes, pain points, and feature requests
Methods
Due to the quick timeline and the kind of data we were looking to collect, we felt as though a survey would be the most appropriate research approach. I collaborated with the product manager of our ChatOps team. I started by outlining what kinds of questions I think we should be asking with the product manager of our ChatOps team. Based on our conversation and their feedback, I created a list of questions that would address all of our goals. We ended up with a 10-question survey with a mix of multiple choice and short-form questions covered these topics:
Role and on-call frequency
Frequency and types of actions performed in the Slack app
Most useful features
Ease of use ratings
Suggestions for improvement and additional functionality
Participants
For this project we wanted to seek out internal users who utilize our Slack integration app while they are on-call or responding to an incident With this in mind we sought out participants who fit the following criteria:
Worked as a DevOps engineer, IT, or infrastructure engineer
Used the PagerDuty Slack integration during incident response
Went on-call at least once every other month
In the end, we had a total of 47 respondents to the survey
Results
Analysis
Our survey gave us both qualitative and quantitative data. This allowed us to gather both the what and the why behind the answers to the survey questions. We began by creating visual representations of our quantitative questions (how often the Slack app is being used, what actions are being taken the most often, difficulty ratings) and then began mapping those answers and connecting them to the answers of our qualitative questions (improvements or changes, missing actions, most useful actions). We created visual representations of our quantitative data and pulled out relevant metrics and statistics. For the quantitative data, we analyzed the written responses and looked for patterns in the responses to find major themes and pulled out quotes to add to our report.
Key Findings
While analyzing our data, we found a few major patterns:
Primary users: Responders, rather than admins or account owners
On-call frequency: Most respondents went on-call about once a month
Slack app usage: The app was used in most incidents by a majority of participants
Top action: Acknowledging incidents was the most common action taken
Chart displaying usage frequency of our Slack app
Chart displaying difficulty ratings of our Slack app
When analyzing written responses, we were surprised with how varied the answers were. We received a wide range of responses with different ideas but managed to gather a few insights that were more universal:
Most helpful features were acknowledging and resolving incidents
“Ack the incident. Most of the time I ack the incident in the Slack app and then use the PD UI for my remaining tasks”
“Acknowledging (during working hours), as I'm already in slack most of the time so it's helpful to be able to take action where I'm already working”
Some participants also discussed their experience around using the Slack app for communication
“I most prefer it for keeping an eye on issues when I'm not on-call in case my teammates need help”
Channels tend to be too noisy and there are too many people involved in incidents, making it hard to follow
“It is very noisy in channels when there are lots of subscribers on a major incident. Makes it harder to quickly scan what's going on”
Outcomes
This survey was done rather quickly and ended up with fewer responses than we would have liked. Given that the responses were so varied, there was an argument that we should perform another survey to get a better picture of how the Slack app is performing. While I no longer work with our ChatOps team, I did push for more research to be done which was completed during that year’s Customer Review Board. We have continued to iterate on our Slack app, especially with our new AI capabilities outlined in the PagerDuty Advance discovery project in this portfolio. Given how heavily used our chat tools are, this is an area within our organization that is constantly being researched and improved upon.