Skip to main content

Migrate existing QA repo to Ops cluster Selenium grid

Overview​

Problem​

Since the QA team is enabling more jobs to the CD QA Repos to run functional tests, the current approach of using a Selenium Grid Server on AWS Spot instances via custom Terraform in each repo might cause issues. We've already seen this happen in some edge cases where teams are trying to run multiple pipelines at once as reported in  PLAT-258 and PLAT-298. The current approach will also result in a much higher AWS cost than utilizing a pre-allocated resource that is more manageable.

Solution​

Platform team has spun up a permanent Selenium grid in our "Ops" cluster where this resource will stay alive. This should work to solve the problems mentioned above as well as provide a more scalable solution for any future use of Selenium. The original documentation will be updated to reflect this change in recommended process, but this document will continue to live independently to specifically address pre-existing QA automation repos that need to be moved over.

Implementation Detail

Below: Selenium specific ops-cluster deployments in our AWS dev account

Each selenium pod in selenium-node-chrome is configured with .5 CPU and 1000 Mi of memory. There are currently 10 pods running with no autoscaling enabled, so you can run up to 10 selenium tests (each under 1 GB of memory) at a time. If any more are attempted, you will need to wait for those tests to be scheduled appropriately.

Updating an existing QA repo​

Pipeline changes in Gitlab​

  1. Go to a QA repo where this GRID_IP CI/CD variable already exists - in this case TestmakerQA repo should have it
  2. Follow instructions to access the project's CI/CD variables by following this GitLab CI/CD variables documentation
  3. Add the GRID_IP variable to the CI/CD settings of your desired repo

Screenshot of GRID_IP variable in CD Settings

Changes required in codebase​

Example of this change can be found here - TestMaker QA repo selenium grid ops cluster vs main

  1. Delete the terraform folder.
  2. Change the .gitlab-ci.yaml file
    1. Fully remove the CI steps for create-infra, destroy-infra, and cleanup steps if they are only used for selenium grid creation. Ensure there are no dependencies that refer to these removed these steps in other parts of the file
    2. NOTE: Do not modify the branch rule condition in the jobs, that should remain the same.
      1. To test in a branch before merging however, you can temporarily modify the branch rule condition to - if: $CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCHthen change those back before merging to "main" branch
  3. Change hubURL in src/main/java/com/wwnorton/TestMaker/utilities/ReadApplicationProperties.java to hubURL = "https://" + hubIpAddress + "/wd/hub";