Developer-Effectiveness.md

Developer-Effectiveness

• There’s often lots of talk about how companies can make their developers more productive. But it may be more useful to think about developer effectiveness: how to ensure they’re building the most useful products for their customers. This is about providing an environment that allows developers to be effective.

• How do I make my developers more productive? It's this question. And unfortunately it quickly turns into things like, I'm sure everybody's been through this, measuring lines of code or commits or things like that, very tactical measurements that, in my opinion, are not particularly useful. And actually often are very easy to game, and will drive bad behavior.

• So it's more about how do I make my developers effective? How do I make sure that they're putting their energies and time to building the most useful products for their customers essentially. So a lot of it's about working, not necessarily working longer or harder or hiring better smarter people, it's just working smarter, using your time better and making sure that the company is providing that environment that allows developers to be effective.

  • So, what we did essentially is take a look at what do developers do all day, what is most of their time spent doing? And we did a fair amount of research on that.
  • There's a lot of time spent in meetings, in chasing down problems. Like debugging a production issue, but you don't have the right information, so you're doing trial and error. Let me try this, see if it works. There's lots of trying to find information, so for example, I need API documentation or technical documentation. And then sometimes it's a lot of time spent building software or doing stuff, but those things aren't necessarily correlated or benefiting the customers in the right way. So, there's a lot of that.
  • And then what we also saw is a lot of small amounts of friction, so this is when we start comparing the, what we describe as the high effective versus low effective companies, is it isn't just one thing, it's these small amounts of friction that the high effective companies, they've done a lot of work to make everything fast and friction-free. And it comes down to these small amounts of problems.
  • And what we tend to find is with companies, when they're trying to get better, they focus on something big, so it's normally like, all right, if we move our services to the cloud, or we move to a microservice architecture, or we bring in this platform or whatever, it'll take us two years. But at that point we'll be perfect, everything will be great.
  • Of course, that's not the reality. And actually what happens quite often recently over the last few years is, somebody's jumped on one of these bandwagons and they've come to us after trying to do it for a couple of years, and they've actually got slower. I'm not going to beat up on microservices on this call, but sometimes that's a common situation that you've brought in something complex that does have benefits, but also has made situations more difficult. In the example of microservices, you now have a much more distributed system. And then what's happened is they've not created the environment, the developers are no longer effective.

• So, I can give some quick examples here. So, a typical one obviously is the developer does throughout the day which is, I write some code and in my local development environment, I want to know that it works. And it does what I expect it to do. So this is the most simplest one. And I do that by writing the code, having it loaded into my development app server, look at it. Okay, did it compile, did it work? Does it look like it worked? Does my feature work? I'm also running tests. I have the tests that I wrote, my regression tests, does my code work?

  • That is a feedback loop that, depending on the organization, the engineer does 10 times, a hundred times a day. And of course, everybody knows this, but what we found is interesting is sometimes organizations are not paying enough attention to these small micro feedback loops. They focus on things that are easy to grasp. So for example, I have manual QA, very easy to grasp that those should be automated. Or I'm using a data center instead of the cloud, easy to cross.
  • But some of this hidden friction, these small amounts of friction, don't really get optimized as much. And some of that's because it's hard to measure.
  • And then another key one that I missed was discovering information. I want to know about information that the API depends on, and there's lots of different things there about like, "Okay, am I finding technical documentation? Do I have access to that team? What are the steps? How long does it take to go through that?" Yeah. So that's the basic, the concept of those key feedback loops that we're recommending to companies if they want to be effective to optimize.
  • So an area that we've focused on is deployment velocity. As the number of engineers grows, we found that we needed to invest in tooling and some systemic changes to how we do deployments

• First, it was the context switching. So the context switching was because we had a very fragmented ecosystem. So why did we have a fragmented ecosystem? It has to do with our autonomous culture. So at Spotify, every single team is like a little startup and it's free to charge ahead and reach their mission by themselves. So infrastructure has been built in isolated islands. This is very conducive for speed, but when we grow, that's where stuff starts to break down. So of course, this leads to a lot of cognitive load for our engineers. Hence, the context switching as number one blocker.

• The number two blocker that the engineers brought to us was it's just hard to find things. And you can, of course, see this. It's a fragmented ecosystem. So which service should I be integrating with as an engineer? Should I use the user data service that the customer service team has built? Or should I use the slightly different user data service that the premium team has built? Or should I just go ahead and build my own? This of course, leads to further fragmentation and we're back to problem number one. So those were the two things that we discovered in our user research.

• One, for people deploying new services, we have an architecture advisory group. It's a group of facilitators to whom you can bring technical questions, and they will connect the dots for you across the company. There are representatives on it from all the departments at Etsy, engineering departments.

• The other thing we're doing is our SRE team has put out a production readiness checklist, and they partner with you when you're trying to launch a new service. They will partner with you to make sure you have crossed all the T's and dotted all the I's and sort of give you feedback about what you're doing and what might be a better way of doing it. So those are our two governance practices today.

• We also do one other thing, and we call it the internal fleet management, which is that we try to basically take this pain from our engineers. So how do we do that? I mentioned earlier we have these golden paths. So we measure all systems at Spotify how close they are to the golden paths. If they are on the golden path, we call them they are in the golden state. So if you are not on the golden state, which is like quite a lot of systems, we can try to migrate them automatically. That's what we call fleet management. How we do that is we push out a lot of automatic pull requests, so the teams have the freedom to merge them, of course. Sometimes we also automerge when we are super confident that it wouldn't disturb production, of course.

• The other thing that we're seeing that companies are really moving towards is these ideas of platform teams, right? It's accepting that there are going to be some teams that are developing code for engineers, right? In the past, I think everybody's probably experienced an engineering team just being let loose and gone and build some crazy platform or something, and it's going to solve all the problems. What we're actually seeing now is, yeah, we do want to build these sort of platforms and things that are going to give us economies of scale, but we really need to be careful about what we're building. What we do is we apply the product management techniques that you do for your product to your platforms and your technical capabilities, right?

• So you prioritize in the same way. You talk to your users in the same way. You A/B test. You get feedback, all the same things. It's not an opportunity just to sort of flex your technical muscle. It's we're actually trying to build useful things and also in an incremental way. As product development is working, we want to build it in an incremental way. We want to provide new improvements.