What are Productivity Pairing Days?

Productivity Pairing Days (PPD) are sessions that engineering teams can book with the Developer Enablement (DEN) team to deep dive into a particular problem space.

{{divider}}

The DEN team focuses on ensuring our engineering teams can be as productive as they can through tooling to build services. Their work enables engineers to focus on building the GetYourGuide products themselves.

While solving complex problems, PPD sessions allow both teams to learn about one another's processes. Together they identify areas for improvement both technically on the tooling side and on the soft skills side in how teams communicate and collaborate.

The duration can be anywhere from one to four days and involves at least one engineer from both teams. The idea to implement such a process was to provide support to teams in a more structured way. This is essential as engineering as a whole is growing rapidly, yet the number of DEN engineers remains relatively constant.

Malachi Soord, who transitioned from backend engineer at GetYourGuide to development enablement engineer, shares two recent examples of successful pairing sessions.

Streamlining machine learning service bootstrapping

In line with our engineering principles, we wanted to raise the level of automation and reproducibility for Machine Learning (ML) applications. The data scientists within our organization had already done a great job automating their processes and identified the missing component that prevented them from moving faster.

It was a cross team effort involving the DEN team and Recommendation and Relevance team (RnR), who are cross functional and experts in the ML field.

Part of the pairing session included adding building blocks to the existing CI/CD solution, which won't be covered here. The topic covered was how to make new ML applications faster, more approachable, and less error-prone for new data scientists.

We wanted to provide them an easier way of bootstrapping, versioning, and running their new ML models so they can focus on what they do best.

Implementation

Using our existing concept of service templates was the ideal solution to this problem since they are:

• Owned by engineers
• Flexible and parameterized
• Versioned
• Integrated directly into our tooling

A new template is usually added whenever many services share a common structure in case they are written in the same language, are using the same framework, or are of a similar project type.

During the pairing day, RnR paired with DEN on implementing a new service template for future ML applications. In practice, service templates are versioned directories of templated files that can be rendered locally using one command.

They can define generic configuration, scaffolding files for specific language platforms, CI/CD configuration, best practices (testing, code formatting). In the case of ML application adding ML model to model registry, supporting code for running in Databricks (Spark).

One only needs to run one command (where "ml-service" is the name of the template):

gygdev bootstrap -t ml-service [chosen_name]

Once a new service is bootstrapped from this template, it's already a runnable application, ready to be built and deployed to testing/production. Data scientists now only need to worry about implementing the application logic.

This is a view of the template structure, where {{module_name}} would be replaced by the chosen value at bootstrap time.

Variables can be added to any file content or file/folder name in the template and leverage the Jinja templating engine.

It was a great collaboration between teams that are usually working on very different things with a vision to empower our engineers. For the first time, engineers working on ML projects at GetYouGuide will have an automated development environment for their most common development flow.

You might also be interested in: Preventing traffic routing regressions for istio Virtual Services.

Improved delivery quality through canary deployments

During Q2 there was a cross-team initiative around improving the quality of the rollout of services, which led to the implementation of canary deployment strategies for services.

Since we are primarily using Spinnaker as our delivery solution, we added additional pipelines for services to use, including a canary analysis phase and all the relevant scaffolding around for deploying baseline and canary and cleanup.

Unfortunately, the provided Netflix judge didn't give us enough control to provide acceptable results for our requirements of short lived canaries of under 10 minutes. So we investigated alternative solutions and found an undocumented entry point within the Kayenta service to allow us to build and develop our own RemoteJudge solution.

An initial spike was done to get a working prototype working with one of the core customer serving services. This would use Istio 5xx metrics to fail the rollout if it went above the pre-defined thresholds.

Once this spike was achieved, the process was refined into a set of easy to follow steps and templates that could be easily applied and customized for any service.

We then scheduled pairing days with each of the teams that applied to have canaries enabled on their services to help onboard their services to the new process, and explain to them the things to look out for and answer any questions.

We managed to onboard over 10 services within a week, scheduling one-hour sessions to walk through the process. It was a mostly painless process with some improvements found along the way, including:

Customizing the canary phase to work with additional metrics with more flexibility

• Improving the reporting side to give more insights and easier filtering of logs within Kibana
• More granular control of which deployment to run the canary analysis on since not all services consist of a single web deployment.

The whole project gave us a lot of exposure to new tooling and deep-diving into the topic of canary rollouts. At the same time, it enabled us to work closely with many teams with whom we may not have usually had such interactions, which was helpful given the current situation of working from home.

You might also be interested in: How we created a DevOps culture.

Conclusion

As you can see from the above examples of previous productivity pairing days, the topics are highly varied, as is their scope and duration.

If you are interested in joining our team, check out our open positions in engineering.

‍