Last year, we wrote a blog post on how we use Heroku’s Review Apps to create staging apps for every Pull Request submitted to a project’s repository. These apps are created and deployed automatically, completely isolated from the production environment, and contain the changes from the Pull Request. This allows every stakeholder to see how the changes proposed in the Pull Request will actually look when deployed.
But there’s a weak link in the chain: we are relying on developers and stakeholders to be disciplined enough to always click around the Review App linked in the Pull Request before clicking Merge. It has happened more than once that we assumed “surely it’s fine to merge this PR, the Review App built fine and the fix is trivial” only to regret it later when production was down.
To avoid human errors like these we’re now taking things to next level. We have a process in place that spins up a browser and clicks around the Review App to confirm that the Review App wasn’t just built correctly, but that it’s actually possible to log in and do stuff. If there is an error in this test, the Pull Request is marked as failed and merging is prevented.
How do we do it
The entire process is split into two jobs:
- Verify if the Review App is successfully deployed (i.e. both build and release Heroku stages are successful).
- Confirm that the new code didn’t break anything by actually clicking around the Review App.
We are using GitHub’s Actions as CI for the process because it provides easy integration with its API, super helpful open-source Actions, and easy management of credentials via GitHub secrets.
Verifying the Review App deployment
The status displayed (This branch was successfully deployed) on the Pull Request is only the build status. It doesn’t give any details about the release stage or whether the app is live without any exceptions. To make sure that the Review App is successfully deployed, we can verify the HTTP status of the app. Based on this, we came up with two approaches:
- Use the Heroku Review App Deployment Status GitHub Action (best suited for small projects).
- Set up GitHub CI to verify the Review App status during the release phase (for medium to large projects).
The All-in-one Action
We developed a GitHub Action which continuously monitors the entire deployment cycle of a Review App when a Pull Request is created.
Link to the Action: Heroku Review App Deployment Status
Paste the following workflow to a new .github/workflows/<NAME_OF_THE_WORKFLOW>.yml file under the required project:
name: Review App Test
on:
pull_request:
branches:
- master
jobs:
review-app-test:
runs-on: ubuntu-latest
steps:
- name: Verify Review App status
uses: niteoweb/[email protected]
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}Whenever a Pull Request is created for the master branch, the Action automatically verifies the Review App build and HTTP status. If either of them fail, the status-check on the PR is marked as failed.
This approach is best suited for small projects since the entire process is packed into one GitHub Action and you don’t need to follow any complex process to set up.
But, when the project size increases, the build and release time also increase which eventually extends the time taken by the Action to verify the status. 90% of the time the GitHub Action runner simply waits until the phases are completed, which results in you paying for unused compute minutes.
Setting up GitHub CI to verify the Review App status during the release phase
To mitigate the problem of the first approach we came up with an alternative solution which involves Heroku’s Release phase, GitHub Repository Dispatch API, and a couple of Actions.
Instead of running a GitHub workflow for the entire deployment cycle, we only run the workflow to verify the HTTP status of the Review App after the deployment and mark the pull request as either failed or successful.
Prerequisites:
Pull Request merge guard:
Pull Requests can be blocked from merging to the default branch when a status check is failed. This can be done by adding the status check to Branch protection rule (repository settings → branches).
GitHub Status API:
The GitHub Status API allows external services to mark commits with error, failure, pending or success states, which is then reflected in Pull Request the commit is a part of.
GitHub Repository Dispatch event:
We use the GitHub API to trigger a web hook event called repository_dispatch when we want to trigger a workflow for an activity that happens outside of GitHub.
Heroku Release phase:
The release phase enables you to run certain tasks after a new release of a Heroku app is deployed.
Basic Workflow
- Create a pending status on the Pull Request.
- Trigger a GitHub workflow at the end of the Heroku deployment cycle.
- Verify the HTTP status of the Review App.
- Update the status check on the Pull Request based on the Review App HTTP status.
Create a pending status on the Pull Request:
By first creating a pending status for the Pull Request we can prevent it from being merged until the Review App status is received. This can either be created manually using the GitHub statuses API or by using an existing Action. We will use the Pull Request Status Action for this blog post.
Create a <WORKFLOW_NAME>.yml file under .github/workflows and add the following code:
name: Create Pending status on the latest commit
on:
pull_request:
branches: [ master ]
jobs:
set-pr-status:
runs-on: ubuntu-latest
steps:
- name: Set PR status to pending
uses: niteoweb/[email protected]
with:
pr_number: ${{ github.event.pull_request.number }}
state: pending
context: review-app-test
repository: ${{ github.repository }}
description: Waiting for Review App to deploy
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}Every time a Pull Request is created for the default branch the status of the PR is set to pending.
Because of the Pull Request merge guard, the Pull Request can’t be merged until review-app-test is transitioned to state success.
Trigger a GitHub workflow at the end of the deployment cycle:
The next step is to verify the HTTP status of the Review App at the end of the deployment. This is accomplished by sending a repository_dispatch event to the GitHub API during the Heroku Review App release phase (i.e. after build & deploy has completed). The payload contains the PR number and the Review App URL.
Add the following code to .heroku/release.sh:
#!/usr/bin/env bash
set -e
repo=$GITHUB_REPO
token=$GITHUB_TOKEN
app=$HEROKU_APP_NAME
pr=$HEROKU_PR_NUMBER
curl -X POST \
-H "Accept: application/vnd.github.v3+json" \
-H "Authorization: Bearer $token" \
https://api.github.com/repos/$repo/dispatches \
-d '{"event_type":"review-app-test",
"client_payload":{
"review_app_url": "https://$app.herokuapp.com",
"PR_NUM": "$pr"
}
}'
echo "Done."Details of the environment variables used in the above script:
GITHUB_TOKENcan be stored in the application’s pipeline settings tab. It is required for communicating with the GitHub API.GITHUB_REPOcan be taken fromapp.json.HEROKU_PR_NUMBERandHEROKU_APP_NAMEare provided by Heroku.
Unlike the first approach, we are triggering the workflow at the end of the deployment, thus avoiding waiting for the build and release phases to complete inside a GitHub Actions runner, thus saving compute minutes.
Verifying the HTTP status of the Review App:
The received repository_dispatch event triggers the workflow corresponding to the event_type which verifies the HTTP status of a passed URL until the required response code is retrieved or the timeout is met. We can again use an Action that some nice person published on the Actions Marketplace: the wait-for-http-responses Action.
Create a workflow file under .github/workflows and add the following code:
Please note that the workflow needs to be in the master branch because triggering a workflow from outside of GitHub (such as repository_dispatch, issue_comment) requires the workflow file to be on the master branch. Also, these workflows run on repository scope, not on the Pull Request scope.
More information about the Repository dispatch event.
name: Review App Test
on:
repository_dispatch:
types: [review-app-test]
jobs:
review-app-test:
runs-on: ubuntu-latest
steps:
# STEP: 1
- name: Wait For HTTP Response of the Review App
uses: cygnetdigital/[email protected]
with:
url: ${{ github.event.client_payload.review_app_url }}
responseCode: '200,403'
# STEP: 2
- name: Set PR status to failure
if: ${{ failure() }}
uses: niteoweb/[email protected]
with:
pr_number: ${{ github.event.client_payload.pr_num }}
state: failure
context: review-app-test
repository: ${{ github.repository }}
description: Review App deployment has failed
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
# STEP: 3
- name: Set PR status to Success
uses: niteoweb/[email protected]
with:
pr_number: ${{ github.event.client_payload.pr_num }}
state: success
context: review-app-test
repository: ${{ github.repository }}
description: Review App deployment is successful
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}In the first step, we verify the HTTP status of the Review App URL. The URL is obtained from the repository_dispatch event. Parameters from the payload can be accessed by github.event.client_payload.<PARAMETER_NAME>.
Update the Pull Request status based on the HTTP Responses check:
The second step only runs when the first step fails because of if: ${{ failure() }} statement.
If the required response is not received or has timed out, the second step of the workflow updates the PR status check (review-app-test) to failure and because of the Pull Request merge guard, the pull request can’t be merged.
When the required response is received, the second step is skipped and the third step of the workflow updates the PR status (review-app-test) to success.
Since we’re not waiting for the build and release phases, the time spent on the entire process is minimal.
The disadvantage of this process is that it is complex when compared to the previous approach since it involves multiple integrations like GitHub API, GitHub Workflows, GitHub Actions, and Heroku. But if your project takes more than a couple of minutes to deploy its Review Apps, you will save a bunch of compute minutes.
Going the extra mile
Up to this point you have confirmation that the newly deployed Review App does in fact respond to HTTP requests. Fantastic! But you can do even better by automating logging in and clicking around the app by using Selenium browser automation.
We usually use Mozilla Firefox, Gecko driver and Python for browser-testing the Review App because they are readily available in the runner environment.
Create a file .github/scripts/browser_test.py and add the following skeleton code:
from cloudinary.uploader import upload
from splinter import Browser
import os
class TestReviewApp:
"""Test Review App using browser."""
def __init__(self, url: str) -> None:
self.url = url
self.b = Browser(
driver_name="firefox",
headless=True,
)
def test_heading(self) -> None:
"""Test if a heading exisits."
self.b.visit(self.url)
if not self.b.is_text_not_present(
"Example Heading", wait_time=5
):
raise ValueError("Given heading doesn't exists")
def take_screeshot(self) -> str:
"""Take a screenshot, upload it to cloudinary and output the url."""
return upload(self.b.screenshot(full=True))["secure_url"]
def main(url: str) -> None:
t = TestReviewApp(url)
try:
t.test_heading()
print("Successful")
except Exception as e:
print(f"screenshot_url: {t.take_screeshot()}")
raise ValueError(str(e))
if __name__ == "__main__":
"""All the inputs are received from workflow as environment variables."""
url = os.environ["URL"]
main(url)In general, this script varies for every app. The above code is just a basic skeleton for browser testing. We can have tests for every element (element visibility, clicking buttons, etc) in the app’s interface.
Add the following steps after the Review App HTTP verification steps:
name: Review App Test
on:
repository_dispatch:
types: [review-app-test]
jobs:
review-app-test:
runs-on: ubuntu-latest
steps:
<REVIEW APP HTTP VERIFICATION STEPS>
- name: Install Python Dependencies
run: pip install splinter cloudinary
- name: Run Browser Test
run: python .github/scripts/browser_test.py
env:
CLOUDINARY_URL: ${{ secrets.CLOUDINARY_URL }}
URL: ${{ github.event.client_payload.review_app_url }}The first step installs the python dependencies required for the script. We are using Splinter for browser testing and Cloudinary to store the browser state during failure.
The second step simply runs the .github/scripts/browser_test.py script. If there is a failure during the tests, a screenshot of the browser is captured and uploaded to Cloudinary for reference.
Summary
The major part of the process involves in finding out whether and when the Review App is successfully deployed. Once deployed, we can perform any kind of tests on it to check its integrity.
To summarize:
- If your Review Apps take only few minutes to spin up, I suggest using the Heroku Review App Deployment Status GitHub Action. It takes care of all the burden.
- If you have multiple repositories with large applications which have longer deployment times, consider using the second approach. Though it is a complex process, it can be helpful in reducing unnecessary costs.
- Browser scripts can be helpful in verifying whether the core features are intact after introducing new code to your app.
Automate stuff! Save developers’ time.