March 24th marked our first official (and virtual) Release Event. We showed off new features like Teams and our upcoming Productboard Integration, shared great insights and best practices from Shortcut customers, heard our CEO's vision for the future of Product Collaboration, and showed off some of the exciting things that are coming to Shortcut in 2021.
One of the customers we heard from was Shaun VanWeelden, the Field Engineering Manager at Scale AI. He spoke with Joey Shampain, a Senior Product Manager here at Shortcut about how he uses our API to build internal integrations that help ensure the many teams across Scale are using Shortcut in a consistent way that's valuable across the entire org.
You can watch the recording of his 30 minute session below and / or read the key questions and points from his session below that.
Shawn, to kick us off, why don't you tell us a bit about what you and your team do at Scale?
Absolutely. Scale is in the data annotation business. If you imagine that you are a self-driving car company and you have a lot of data to label, to build out a machine learning model, we're going to take all of your video footage and your 3D point clouds around you and get that labeled. Scale provides programmatic access to about 75,000 workers around the world to help you label your data. I lead our field engineering team, which is responsible for the post-sales technical experience of our customers.
Given that we are an API first company with a very technical customer base, there's a lot of engineering challenges that are going to come up just as people are trying to use our product. We help our customer success managers unblock them on any technical needs that are coming up, help our customers debug their scripts, write scripts, look at our API stuff, do our documentation. That is the team that I manage here at Scale. We are a team of seven around the world. We are geographically diverse, not super big, but not tiny either. That's a team in a nutshell.
Can you tell us on how Scale is organizing their teams within the organization?
One of the main ones that's relevant is our EPD team, which is our Engineering Product and Design team. This team is mostly aligned by product verticals. We might have a team for image labeling, video labeling, 3D scene labeling. That's one of our core verticals that really guides the feature sets that we're working on. On our go-to-market side, we also have my team, our customer success managers, the pre-sales engineering team, our, I would say, deal desk, and revenue operational team. The other part of Scale is we got our very operationally intensive business. To do data labels at Scale, I guess, pun somewhat intended, it takes a lot of people to do that.
How do you separate and manage those teams in Shortcut?
When you think about the tools at our disposal projects and workflows and whatnot, I think it's fair to say that we touch every feature. For example, our EPD workflow has about 21 projects in it and they share the same seven workflow states across the backlog, review, and development. It's an important quest stage or just deployed and done, for example. Our finance team has two different projects in Shortcut. We have an analytics team that has a place to go request to analytics asks and things like this. I would say that we would use the same workflow states, probably like the department or at the highest level. Everybody in engineering should have roughly the same workflow of how they move a story from start to finish. Then we use projects as our sub-teams. There might be that sub-team or a project for that image-based team or the video-based team. They each would have their own project and their own stories that they are working through on that side.
How does your team use Shortcut? What are some of your day-to-day rituals within the tool?
First things first, I run our stand ups on top of Shortcut and so we're going in there, and we literally just go filter by filter of each person. "John, what are you working on? Ivan, What are you working on?"
I think that's probably not super unique to us, but working through our board each morning is certainly a key part of how we use Shortcut. I really do like to take advantage of the filters for the team. Given that we have four or five main engineering teams in different product verticals here, the reality is that my team is going to be the one helping our customers and being a bridge to their engineering team, and so being able to track the status of those tickets is important. We definitely link things a lot, so we'll link different stories together to go, "Hey, we're going to follow up on this one."
"Okay. I have to check my card next to it. It's done now. Now everything can be unblocked on my side as well."
That's definitely helpful.
I would say that we have also what we've referred to as an on-call space in the Shortcut. This is a way to get ahead of more urgent issues. If there's an urgent bug or a blocker that's not going to be part of the regular product release cycle, we have a separate space and workflow to track these urgent things. A lot of my team is, frankly, going to be putting stuff on this on-call space to make sure they're getting followed up on. Then we just add ourselves as watchers to those tickets and update our customers from there.
All of our teams use story templates. I'd say maybe half the time a story is created. It's actually going to be created as a request from another team as opposed to myself filing a ticket for myself. When that is the case, I think the story templates really help other people across the company know what to include and know what you are expecting for them to fill out.
For example, we have a lot of tasks to be done on our side that the customer is creating. If you want me to go take an action and review something, maybe something that is stuck, I probably need a list of those task IDs, or at least the way they filter a query for those tasks. Making sure that we ask that in a story template helps me help you.
The reality though is, humans suck at templates and following directions even when we really want to help each other. We've actually been building a lot of Shortcut API integrations to make sure that we have things like clear due dates and labels added to the Stories. We have API integrations that hook across the entire company and make sure that we are getting the information that we need on the story based off of the Shortcut's webhooks there, which is also fun.
Can you elaborate a bit on that integration and how you built it?
First things first. I'm just going to go into the Shortcut's API docs just to make sure we knew what was there. We use the webhooks extensively. For those who aren't as familiar, a webhook is a way to get notified when an action is taken on Shortcut. If a story is created, a comment is added, a label changes, all of those things can let our server and our custom code know that a change has happened in the Shortcut and that we should go look and see if that change is valid or not. We use these webhooks to ensure the Shortcut space is kept clean and organized.
For example, in our on-call space we have different severity ratings of “severity zero” up to what is essentially “severity level shit is hitting the fan” back down to like severity level three which is more along the lines of, "It's nice to do this when we have a moment." If you've put an issue or a story on our on-call board but you did not add a label with the severity, it's actually pretty bad because we've integrated Pager Duty to Shortcut using the API. If you file a story with the Sub-Zero label, we're going to go ping a bunch of engineers and make sure you get the issue resolved as soon as possible. If you have a sev zero issue and you forgot to add your sev zero label on it, well, we're not going to do anything until you realize we're not doing anything. We actually will look at every single issue that's filed on our on-call space, figure out if it has the right labels and if it doesn't, it's fun.
I actually made the bot that comments the story and notifies the owner, as well as the person who filed the ticket that says, "Hey, you just made the story but you forgot to add the severity label. Please do so so we can prioritize it correctly.” At first, this bot was commenting from my name because I was the one that built the API. It's fun. Like, "Shawn, you're so helpful. Thank you. You're replying to everything. This is great." We actually made it a dedicated user now who isn’t me but that everyone knows does sanity checks.
The other one for us is I require a customer label and a due date for every single story on my board. You cannot file a story without getting pinged if you've missed one of these fields. Every time the story changes the workflow state, it's still not correct. You're getting into another email and notification that says, "It's still not right. Please go add the customer label and the due date before I can go close this out."
We also enforce adding the number of points or the estimate for how long the story will take before you close things out on our board. My team cannot move anything to the done state without putting in the number of points, which is actually very helpful for reporting. On our side, a lot of engineer managers want to do this. It was like, "No one's going to fill it out." I was like, "If you literally bombard their inbox until they fill it out you’ll get some nice reporting afterwards.”
Can you share your recommended thought process or recommendations for how people on this call could think about rolling out new processes across multiple teams within a large company like Scale?
I think there's probably two directions to it really. One is the self-serve bottoms-up approach where there's an email that went out saying, "Hey, Shortcut has Teams. When you log in I plugged in this morning, I saw that they may feel like, Hey, there's Teams now," pop up. People will start to use it. We have 25 teams, I'm sure people would come up with their own way to go build a team and organize it there.
But for an org of Scale’s size I would probably argue that we need to take a top-down approach. We actually have a Slack channel dedicated to talking about Shortcut and we would organize as one of our engineering management topics, we are like, "What's our naming convention for teams? How do you want to group things? Is it truly just by product vertical or do we need to do something more than that?" Just to help people out. I think the consistency and the rollout is going to be very helpful for us there. I think that we plan on taking the top-down approach, probably enforcing our best practices and really setting a specific example and an expectation around how we are using Teams as opposed to letting people just discover it on their own. My trust of humans around these sorts of things is very low in general.
Can you talk about how you feel you get the most success collaborating with other teams at Scale and also what are some of the challenges from a company at Scale size when you're collaborating across teams?
I think we need to avoid the black hole that any issue tracker can become. It's very easy to jump into Shortcut and just be like, "I'll just make up a story" for virtually anything. But if no one's going to look at it and prioritize it, I would really encourage folks not to go put it in Shortcut, find a different way to communicate it. Do we need a brainstorming session? Do we need ideas list somewhere else before you go make things in Shortcut? Ideally what's in Shortcut is more than just an idea and can be actioned.
We have a few go-to-market and product folks who keep an eye on things. They review the outstanding issues and requests that are both in Shortcut and not in Shortcut and figure out what needs to be there. What is the relative priority? Are more customers starting to ask for this? They go surprisingly far just in terms of making sure that we have what we need there alongside very clear expectations about how a team should be using labels. Each team to have their own label methodology, but what's not okay is not having the consistency or the clear expectations about how it should work. For me that's solved by two things, one, onboarding and clear internal documentation about how to use Shortcut. We have our own guide about using Shortcut at Scale. There's a clear, good way to use Shortcut and a way not to use Shortcut there. Then as well as we're all human so for us, trying to actually enforce the good behavior when we can programmatically is also helpful. I think, go ahead, Joey.
You talked through your integration work, you talked through some concepts like story templates but can you elaborate on how Scale ensures everybody is able to meet in the middle and work together effectively?
It takes a culture of writing bring about good collaboration. We can talk about information, what we don't want is just, "It's in people's heads or like this team had to bring from a session and now there's a bunch of notes somewhere in Google doc and has access to." You really start to open up the lines of communication about what the organization is working on.
At Scale, we have two different things we do:
For every API change we want to make, we have an email list that goes out to every engineer. You're pretty much required to be on this email list for the proposal about what you want to change on the API. So people know what's coming and hopefully collaborate and say, "Yes, cool idea. That makes a lot of sense."
We also send out these product requirements documents. These are linked from Shortcut to a Google Doc that's three or four pages long and details what the problem is, what the proposed solution is, the timelines, the details, the implementation stuff.
We share these out to a similar group to the engineering team, so people know what's in the works. It's a very short email: "Here's the statement that sounds relevant to you or you want to know about it. Click on Google Docs to get all the details." I think it helps open the door for folks to make sure we're all on the same level playing field and we have access to the same information about what's happening across the company, as opposed to keeping things in very tight silos and only letting people know when it's fully done or already in progress.
Yes, that's great. We use similar practices at Shortcut making sure information is out in the open. Increases cooperation, increases transparency across teams, and ultimately makes teams and the company more effective.