Luc Shelton

Functioning as a senior software engineer in the "Intelligent Systems Group", my role on the team was to support or develop new services that were mostly consumed or used internally. A lot of the backend services that we provided offered machine learning functionality that was consumed by other teams and products inside of the business, including Builder Studio. The functionality of the products or services we provided were often heavily reliant on our services always being available and working reliably well. The work often involved refactoring code to ensure that services were scalable as distributable and isolated workloads.

Python was often the language of choice during my time working as this role, as that was the language adopted by the data scientists and engineers who were on the team. There were a few instances where I had to use TypeScript with NodeJS.

Goals

• Migrate existing backend services deployed as Docker containers using docker-compose to a scalable orchestration platform such as Kubernetes.
• Refactor existing services to be scalable as distributable, isolated workloads (microservices).

Responsibilities

• Kubernetes Platform Migration
  • Migrated existing services that were deployed as standalone Docker containers using docker-compose to Kubernetes using Azure Kubernetes Service (aks), Helm (helm), and GitLab for CI.
  • Created custom Helm charts for services that needed to be deployed which were not otherwise supported by the Platform or DevOps team (RabbitMQ).
  • Refactored existing projects to support deployment to Kubernetes, including the alteration of GitLab CI build configurations, adding support for health checks, and adjusting Docker container images.
• Security Fixes
  • Added JWT authentication for services that were being consumed by other services internally.
  • Retroactively adding support for HTTPS for all internal services that were originally deployed using docker-compose and as standalone Docker containers. These services used the Alpine Docker container image variant of NGINX.
• Natasha AI Engineering
  • Refactored existing code for Natasha AI, so that it could handle distributed task processing for incoming (transcribed) commands.
  • Integrated the Celery framework for distributed task processing of incoming transcribed commands among independent worker processes.
  • Produced Docker container images for Celery workers which were later deployed to Kubernetes as horizontally scaling pods.
  • Utilised the Pottery library for implementing distributed locks using Redis. This was for ensuring sequential processing of commands for the Natasha AI among independent Celery worker processes that were controlled and scheduled by Kubernetes.
• Build and Automation
  • Fixed issues relating to GitLab CI configurations on internal projects, such as feature branches being erroneously built and deployed.
  • Replaced project-specific duplicates of GitLab CI configurations with ones that were being remotely managed and developed by the Platform or DevOps team.
  • Added unit tests to new or existing projects where applicable, using pytest and coverage.
• Tooling
  • Produced tooling for automating the process for creating and managing Kubernetes cluster deployment environments for existing services that were pending migration (cluster-builder).
  • Produced tooling for creating new projects using the Poetry package and project dependency manager (project-builder).
  • Tools were often developed as command-line accessible and installable PyPi packages, that utilised the likes of click and rich-click for providing a slightly friendlier user experience. It also made them much easier to extend and support due to the nature of these frameworks.