How to run Playwright tests on a Vercel deployment using GitHub Actions

Prerequisites

For the purposes of the demo we will assume that we have an application hosted on Vercel connected to a GitHub repository. We are also assuming that Playwright is already installed and that some basic smoke tests have been written that we wish to use to test our site. This demo uses the pnpm package manager so all the commands will start with pnpm but this can be substituted out with npm, yarn or any other node package manager you like.

Running the tests post deployment

We can trigger our Playwright tests to execute once a deployment to Vercel is complete by creating a GitHub Actions workflow. This will act as a series of instructions to our CI environment that will automate the execution of jobs we wish to carry out. This is done by configuring a YAML file inside the /.github/workflows directory in our repository.

Here is an example:

name: Run Smoke Tests

on:
  deployment_status:
jobs:
  smoke-tests:
    name: Smoke Tests against ${{github.event.deployment_status.environment}}
    timeout-minutes: 5
    runs-on: ubuntu-latest
    if: github.event.deployment_status.state == 'success' && contains(github.event.deployment_status.environment, 'Production')
    steps:
      - name: Check out code
        uses: actions/checkout@v4
        
      - name: Setup pnpm
        uses: pnpm/action-setup@v2
        with:
          version: latest
          
      - uses: actions/setup-node@v4
        with:
          node-version-file: ".node-version"
          cache: "pnpm"
              
      - name: Install Test Dependencies
        working-directory: tests/smoke-tests
        run: pnpm install

      - name: Install Playwright
        working-directory: tests/smoke-tests
        run: pnpm playwright install --with-deps

      - name: Run Smoke Tests
        run: pnpm test:smoke-ci
        env:
          PLAYWRIGHT_TEST_BASE_URL: ${{ github.event.deployment_status.target_url }}

Let’s walk through it.

The on: keyword in GitHub Actions syntax is used to define the triggers for the workflow.

on: 
  deployment_status:

Utilising deployment_status: here works as a catch-all, triggering the workflow on any deployment status change. This also gives us access to the deployment_status.state object, allowing retrieval of the Vercel deployment URL to pass into Playwright as the endpoint for test execution. As we only want this workflow to run on successful deployments, we will need to refine this trigger to be more specific inside the job itself.

jobs:
  smoke-tests:
    name: Smoke Tests against ${{github.event.deployment_status.environment}}
    timeout-minutes: 5
    runs-on: ubuntu-latest
    if: github.event.deployment_status.state == 'success' && contains(github.event.deployment_status.environment, 'Production')

We run the workflow as a single smoke-tests job. 'timeout-minutes' determines the maximum time before automatic failure of the job. To optimise our Github Actions minutes usage, we should aim to have this as low as possible, preventing wastage on tests that are stuck indefinitely. A 5 minute timeout is sufficient for rudimentary smoke tests but this would quickly be exceeded by a more comprehensive test suite.

The job executes if the following conditions are met:

if: github.event.deployment_status.state == 'success' && contains(github.event.deployment_status.environment, 'Production')

It is here that we are making sure that the tests are only ran on successful deployments. As a nice optional extra we are also making sure tests are only ran on the production environment, saving us GitHub Actions minutes by not running smoke tests unnecessarily on Vercel preview builds.

Now we add our job steps. First we add our boilerplate code to load our repository into the workflow, set up the package manager, install node, install our dependancies, and finally install Playwright.

      - name: Check out code
        uses: actions/checkout@v4
        
      - name: Setup pnpm
        uses: pnpm/action-setup@v2
        with:
          version: latest
          
      - uses: actions/setup-node@v4
        with:
          node-version: latest
          cache: "pnpm"
          
      - name: Install Test Dependencies
        working-directory: tests/smoke-tests
        run: pnpm install
        
      - name: Install Playwright
        working-directory: tests/smoke-tests
        run: pnpm playwright install --with-deps

Now we can finally run our smoke tests.

      - name: Run Smoke Tests
        run: pnpm test:smoke-ci
        env:
          PLAYWRIGHT_TEST_BASE_URL: ${{ github.event.deployment_status.target_url }}

pnpm test:smoke-ci should be substituted with whatever command you have set up in your package.json to run your Playwright tests. The most crucial part of this stage is the final line:

`PLAYWRIGHT_TEST_BASE_URL: ${{ github.event.deployment_status.target_url }}`

This will return the URL of our successful Vercel deployment and pass it into the CI environment as an environment variable.

Playwright configuration

Now that our workflow is complete, we need to configure Playwright to use the Vercel deployment URL. We can now update the baseURL field in our playwright.config.ts file to use the environment variable that we passed in.

use: {
 // Base URL to use in actions like `await page.goto('/')`.
 baseURL: process.env.PLAYWRIGHT_TEST_BASE_URL ?? "http://127.0.0.1:3000",

As it’s a typescript/javascript file we can include logic that allows us to use the Vercel deployment when present but will also revert to using localhost when we are running the tests locally.

And there you have it! That should be all you need to get Playwright tests running on successful Vercel deployments. A nice optional extra could be implementing a final step in the workflow that alerts your team when a smoke test fails through your preferred messaging platform such as Slack or Microsoft Teams, adding an extra layer of proactive error management to your development pipeline.