At one of my previous companies, the production system had a series of functional gaps. Manual intervention was required in order to support some exceptional data cases. My engineering manager had been performing almost 100% of this work in an attempt to shield the engineering team from the interrupt-driven distraction of dealing with these support requests. Eventually, that engineering manager was fired. In retrospect, they may have let him go partly because he was neglecting his managerial duties in order to knock out those support requests coming in from other departments. At that time, I had by far the deepest overall business process and system knowledge of anyone on the engineering team. So when that manager was let go, those production support duties fell to me. Right around this time, our company had landed a big contract and was experiencing a 400% increase of production volume and revenue. Those production support duties that once represented maybe 25% of one employee’s day were now my entire existence. After about a month of this, I started training other engineers in the art of production support. We came up with various schedules for rotating engineers into the role. Each week, we had two full time engineers assigned to diagnosing production data problems, running canned SQL scripts and writing ad-hoc reports. It was a training nightmare and a serious drag on morale. The engineers didn’t like the production support work, and frankly, they were quite bad at it.
Things were bleak. The other departments weren’t happy because it was taking us a week to get to their support requests. The engineers dreaded their week-long rotation working the support ticket queue. During this time, I had been made the tech lead on the product, and I honestly had no idea how to fix this problem in the short term. Production volume kept going up, and we couldn’t modify the system fast enough to bridge the functional gaps that were creating these support requests.
It had been about a year since the company’s big volume spike had caused a crisis for the engineering team. A couple of the engineers had valiantly stepped up to take on the production support burden full time. Things were still hectic, but no longer at def-con 5. During that year, one of the customer service leads that we will call Joe, spent a lot of time talking to myself and the two production support engineers. He would talk to us about business flow problems in production and propose some possible solutions. He really knew the business, he had good ideas and he was persistent. I don’t know why it took us so long to come up with the idea, or even whose idea it was, but Joe inspired us. The two production support engineers and I decided to create a new role in engineering for someone like Joe. I had just been promoted from tech lead to manager, and I convinced the powers that be to give me a new job requisition. We interviewed four internal candidates for the position. One of the other candidates excited us to the point that we would have hired her if we had two openings, but in the end, Joe was the obvious choice. He was my first career hire as a manager, and he turned out to be one of my best.
Joe had a Bachelor’s in Philosophy, which he swears was helpful to his transition to software development. Joe had previously worked as a pharmacy technician. He came to our company through a temp agency as a customer support representative, and worked his way up into a lead position. After we hired him into engineering, Joe became proficient in SQL in less than three months. His leadership skills, business knowledge and relationships with the other departments were invaluable to the engineering team. He basically took over the production support responsibilities within a year. Along the way, we hired a couple of other customer service leads with similar stories to get trained up. The two engineers that had been dedicated to production support were able to move back into active development roles on different teams. Two years later, Joe himself was on an engineering team working as a full time developer. He quickly became one of the most effective developers in the department. He was a business process expert that also really knew the code, which was a very powerful combination. Joe dug into the code with a desire to understand that could only come from years of being a frustrated end-user.
In addition to the initial success with Joe, one of the subsequent production support hires from customer service became an effective full time QA engineer. When I left the company, a couple of other hires were showing potential for a future transition as well. A clear technical career path had been established and proven. I have no doubt that if the production support team were larger than 3 or 4 employees at any given time, we would have seen an even more fruitful pipeline of home-grown engineering talent.
Potential engineers are all around the office. Keep your eyes open and let it be known you’re looking. In our case, we lucked into creating an internal development pipeline accidentally. My advice is to be smarter than we were. Actively develop a career path to transition non-engineering internal business experts to your development teams. Advertise it. Get good at interviewing for raw aptitude, curiosity and desire. Only a fraction of application development is knowing programming languages, data structures, etc. In the right environment with the right people, that knowledge can be gained to an effective level in less time than you think. If you can put the future engineers to work doing something critically useful along the way like we did, then it is all upside. Just make sure you give them encouragement, access to education and progressive technical challenges. Your soon-to-be engineers will do the rest and thank you for the opportunity.