The bane of automating single-page apps is, ironically, good frontend error handling. Our CodeceptJS-based screen scraper has no idea when a REST error was handled. The frontend team never thought to add a common class to button elements having text like “An error occurred. Please try again.”

Even worse: auto retry. The scraper is unable to distinguish between “no data available” versus a failure. Scraping jobs that run for hours need to pause only as long as necessary. Is an element missing because there is no data, or did an error occur, or did the scraper not wait long enough? It’s impossible to know.

Like everything, you don’t learn this until you try it yourself. If you’re embarking on a new frontend project, I hope you will consider the plight of those who write scrapers and integration tests.

And then there’s infinite scrolling.

Chromium Browser Icon

While scraping or testing websites, sometimes it’s convenient to open a link (e.g., via “control-click”) on a page into a new browser tab, do some work in the new tab, and close it. This avoids the hassle of using breadcrumbs or the Back button. This approach can also result in less work for the website’s backend since closing tabs doesn’t normally request remote data.

Here’s how to automate “control-clicking” a link in Chromium, in order for the link’s contents to appear in a new tab. This example uses CodeceptJS and Puppeteer. This solution has been tested on OSX and Ubuntu. It should work on Windows.

First, create a helper for CodeceptJS and alter CodeceptJS’s configuration to use it:

Then use this in your scenario:

Here’s a Dockerfile that:

  1. Starts with Alpine 3.12 and OpenJDK 15
  2. Installs Python 3.9.1 using pyenv
  3. Updates Alpine packages and removes the ones that are only needed for installing Python
  4. Runs a Python program when the container starts using the user ‘appuser’ instead of root

Most libraries can be unit tested. Most applications can only be integration tested. Apps should be nothing more than mosaics of libraries. But unit tests are never 100% effective. Even with 100% coverage, bugs will be found, especially in the parts that are hard to use. The process of writing documentation — especially with real examples — often exposes usability issues and bugs. Therefore, pick a milestone and add the task that developers typically do last, dislike, and learn from the most: API and user documentation. This will improve your team’s understanding of what and for whom they’re building.

This SLS config file is compatible with Serverless 2.28+ and defines an AWS step function that waits 5 minutes and invokes a lambda. The lambda is granted access to the following RDS resources:

  • read/write to CloudWatch Logs streams
  • read/write to S3
  • read from Systems Manager Parameter Store
  • Connect to an RDS database without a password (mySQL 5.6, hence the use of tls-min-v1.0)

May this serve as a map for lost SLS travelers.

Terris Linenbach

He/him. Coder and gamer since 1980. Lately I’ve been coding in Node and React. Always seeking The Best Way.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store