Company IBM Cloud (Billing & Cost Management)
Timeline Spring 2019 - March 2021
My Role Project Design Lead
Collaborators Tracey King, Kristin Holifield, Hannah Moyers, Haley Lucey
[Coming soon]
Project background
The IBM Cloud Billing & Cost Management team had completed our State of Billing UX research project (view case study) the year prior, as well as 2 additional billing-related studies in collaboration with our UX researcher. This project references findings from each of these studies.
- State of Billing UX Research: Research effort to identify most popular use cases and biggest pain points with the IBM Cloud Billing experience. 63 customer interviews, 54 survey respondents, 80 Usabilla & NPS comments. (view case study)
- Billing Baseline 2021: Task analysis survey to understand how successful our Cloud Billing users are in completing 27 distinct billing tasks. 42 user interviews, 12 moderated sessions focused on completing 6 top billing tasks.
- Business Partner Learnings: Moderated interviews with IBM Cloud Business Partner customers. 10 participants including: Value-Add Resellers (VADs), Independent Software Vendors (ISVs), Managed-service Providers, IT managers, Solutions Architects, and buyers from CTO offices.
We were well underway in delivering a vastly improved invoice reconciliation experience for our billing users (view case study). With the invoice improvements underway the team was ready to begin addressing the other key billing-related issues identified in the research studies referenced above.
The problem
Outside of the reconcile invoice experience, the most prominent billing-related item we consistently received negative feedback on was our monitor usage experience.
We found that many of our IBM Cloud Billing users were struggling to understand their spending and usage across the platform.
- ~73% of users unable to successfully locate a specific charge in a given project.
- ~73% of users unable able to successfully locate a prediction for next months spend.
- 100% of users unable to view their PaaS and IaaS charges in a single place (must reference and total charges from multiple pages on and off platform).
- 5 experience breaking pain point themes encountered by the majority of our users.
- System usability score of ~41. Even accounting for a 8 point margin of error, this is 19 points below the industry average SUS score of 68.
Issues elaborated on in further detail below in the Understanding the problem section.
Project overview
Team goal
Increase the overall TCR (task completion rate) and SUS (system usability score) for core billing-related jobs, by addressing the UX gaps and technical issues prevalent on the IBM Cloud Usage dashboard (where usage monitoring occurs).
Design objectives
Understanding the user
This effort is primarily focused around meeting the needs of the Cloud Cost Analyst, one of 3 key Billing personas initially identified in our State of Billing UX research project and later expanded upon in our Business Parter Learnings study.
Who is the user?
While the jobs of the Cloud Cost Leader and Cloud Cost Advocate personas do sometimes overlap, the majority of the responsibility around usage monitoring and cost optimizations typically falls on the Cloud Cost Analyst.
What do they do?
Cloud Cost Analysts work to...
Expand the sections below to see examples and learn more.
Choosing a starting point (determining scope)
It quickly became clear after initial scoping discussions with PM, Engineering and Research that each of the tasks above were large enough to be their own project. Of the 4 billing-related tasks the Cloud Cost Analyst is responsible for we decided to zero in on the first one:
Understanding their organization’s costs
Why? If billing users are unable to do something as fundamental as understand their organization’s costs, then all of the other tasks they’re responsible for essentially become impossible to complete. It made the most sense to start here.
What are their needs (use cases)?
From the 120+ interviews and NPS feedback we received, our IBM Cloud Billing users were not afraid to express their specific needs related to understanding usage and spending details:
Here are a few of the most commonly vocalized needs...
As elaborated on below, the IBM Cloud Usage dashboard experience does not support most of these use cases.
Understanding the problem(s)
As stated above, IBM Cloud billing users struggle to understand their spending and usage across the Platform. When compared to our competitors we lack most of, if not all of the features they include as standard elements of their usage monitoring experiences.
Where is spending and usage tracked today?
Ideally users would be able to track their spending and usage from a single page, the IBM Cloud Usage Dashboard located in Billing & Usage.
IBM Cloud Usage Dashboard.
Unfortunately, this is not often the case...
Due to technical restraints in how we can surface spending and usage data, most users have to toggle through a minimum of 3 different pages to get a complete view of their usage/spending.
IBM Cloud billing users lack a single view of their usage and billing data across their IaaS and PaaS offerings.
Are there other pain points?
Yes, many. 😢
Outside of not being able to slice and dice spending and usage charges from a single page, there are innumerable pain points across the Usage, Billing items, and Invoices pages (many of which are detailed in the State of Billing UX research project).
Pain point themes
Expand the sections below to see examples and learn more.
kUsers seeking to audit or understand their usage have to learn to manually compile the data themselves by going to different pages, asking for help and building spreadsheets, wasting time and risking user error.
“When I go to Invoices, I get redirected. I followed that link, logged in and then had to go through more screens to request access to my invoices. I’ve done that, and while that immediately let me see a charge I don’t understand at all, I got an email message informing me that I will hear from IBM in 3 days. We’re now 2 hours in of my time trying to understand the IBM [billing] systems.”
Users can obtain invoices through different IBM systems. These invoices do not contain common reference points that can be matched to one another. This makes auditing impossible, and creates a lack of user trust in our systems. It also results in more time spent by our support staff.
“I have to be very brave when I [match it up my invoice with every line item] because it is very time consuming.”
“There is one major pain point I do have: I name things so that I can keep track of what they are. So when I get the detailed invoice and I get to file storage, I can see the location but nowhere does it tell me which one it is.”
Predicting overages requires many of man hours. They are hard to discern for IaaS services and difficult at best for PaaS services.
“Usage is blank. Billing items, and Invoices are blank as well in the IBM Cloud portal. I understand that IBM Cloud is separate from IBM Blockchain Services, but as a customer, I don’t really care that they are different within IBM. They are one and the same for me. I buy Cloud services, I put software I purchase from IBM on top of it, I use increasing storage over time and I pay for a managed service through the entire stack. If so, what am I paying for?”
Identifying the source of a usage spike is very tedious in PaaS and not possible with IaaS data. Users are required to navigate through many levels if usage views to better understand the metrics available to them.
“On the usage dash, I need to choose RG/org from the dropdown and switch back and forth. I want to be able to see consolidated usage for multiple resource groups or orgs.”
Notifications for overages don’t seem to work for many users and only allow certain types of products to be monitored. This results in distrust and anger.
“The spending notifications are useless, account details useless until invoice is finalized for the month when I land a $467 bill that I didn't know existed. Unacceptable.”
On top of that, the Usage dashboard was not consistent with IBM Cloud branding standards and had been dinged for many basic visual, layout and page hierarchy issues.
Anything else?
Design explorations
With an understanding of the user, their use cases and existing pain points, I got to work on my design explorations.
What makes a good usage dashboard?
[Coming soon: include quotes, examples]
Sketches & iterations
[Coming soon: Include Brainstorming, Sceanrios]
Filter patterns
[Coming soon]
User testing
[Coming soon]
Finalizing designs
[Coming soon]
Outcomes
[Coming soon]
What I would do different / next
[Coming soon]
Investigate the remaining 3 tasks.
Why?