Last year I created a little npm module called all-the-package-names. It's a simple array of all the existing package names in the npm registry. It can be used to find if a package name is available, or to find all packages with names matching a given pattern.
Here's a one-liner to try it out:
npm i -g trymodule && trymodule all-the-package-names
This module was easy to create, but it is by nature impossible to keep up to date, as hundreds of new packages are added to the npm registry every day.
I wanted to remove myself from the publishing equation, so I set up a scheduled task on Heroku that runs every hour, fetches new names from the registry, commits changes to git, and publishes to npm. To enable this, I had to figure out how to securely perform all these operations within the confines of a Heroku dyno:
- Clone a repository from GitHub
- Run a build task using npm scripts
- Commit changes to the git repository
- Publish new package versions to npm
- Push changes to GitHub
- Keep credentials safe and outside the codebase
This post covers the steps required in setting up such an automation scheme. It is assumed that you have basic working knowledge of Heroku, npm, and GitHub.
heroku create my-npm-bot
Unlike a typical Heroku app, this app doesn't have a
web process. There's
no web server listening on a port and serving requests. It's just an app on
a faceless computer in the cloud that will wake up on occasion to do your bidding.
Heroku has these awesome open source
things called buildpacks that give you complete control of your app's build process.
is a script that is executed on your app's code to prepare it to run on
the Heroku platform. When you deploy a Node.js app, for example, Heroku's
Node buildpack is used
npm install, etc.
In most cases, Heroku users don't need to know about buildpacks at all. Heroku detects the language or framework of your web app and automatically chooses the right buildpack for you. But sometimes you need to exercise control over the app's build process, and that's when buildpacks come in handy.
For the purposes of this app, three buildpacks are required:
- zeke/github-buildpack sets up
.netrcfile to store GitHub credentials.
- zeke/npm-buildpack sets up a
.npmrcfile to store npm credentials.
- heroku/nodejs installs node and npm, which we'll use in our release script.
To add these three buildpacks to your app, run the following commands:
heroku buildpacks:add -i 1 https://github.com/zeke/github-buildpackheroku buildpacks:add -i 2 https://github.com/zeke/npm-buildpackheroku buildpacks:add -i 3 heroku/nodejs
i stands for
index, allowing you to specify the order in which the
buildpacks are executed. In many cases this order is important, but for our
purposes the order probably doesn't matter, as none of these buildpacks are
relying on functionality or build artifacts from another.
heroku config:set GITHUB_AUTH_TOKEN=YOUR_TOKEN
The next time you push your app to Heroku, the GitHub buildpack
you just added will
detect the presence of
GITHUB_AUTH_TOKEN in your app's environment, and use
it to create a
Here's what an entry for GitHub looks like:
machine github.com login YOUR_TOKEN password x-oauth-basic
If you pass the
--netrc flag to
-n for short), it will automatically read credentials
~/.netrc file, allowing you to easily make authenticated requests:
curl -n https://api.heroku.com/appscurl -n https://api.github.com/users/zeke/repos
Now that the GitHub credentials are securely stored,
git commands like
git push will be automatically authenticated.
npm doesn't have a fancy web interface for creating tokens, nor does it support
.netrc format. It is possible, however, to find your personal token by
looking in the
.npmrc file in your home directory.
Here's a one-liner for reading that token:
cat ~/.npmrc | head -1 | sed 's/.*=//g'
Add the token to your Heroku app's config:
heroku config:set NPM_AUTH_TOKEN=YOUR_TOKEN
Like the GitHub buildpack, the npm buildpack
will detect the
NPM_AUTH_TOKEN environment variable and use it to write a
.npmrc file into the app directory. Any
npm commands that require authentication
whoami) will now be authenticated automatically when they
are run within the app directory.
To figure out if everything's working, deploy your app:
git commit --allow-empty "testing npm and github auth"git push heroku master
Then open up a remote bash shell containing your app's code and environment:
heroku run bashcat .npmrccat .netrc
If everything is configured properly, you'll see your credentials in the
.npmrc files, and you should be able to run authenticated
npm whoami and
Tip: If you try to
git push your app to Heroku and get an "Everything up to date"
message from git, it's because nothing in your source code has changed. To
force-push the repo to Heroku, create an empty commit:
git commit --allow-empty -m "empty commit"git push heroku master
Once the authentication is set up, the next step is to write a release script to be executed periodically on Heroku.
mkdir -p scriptstouch scripts/releasechmod +x scripts/release
Next, add the release script to
package.json, so it can be executed as an
Here's a simplified version of the release script used by
all-the-package-names. For the full script, check out scripts/release on the GitHub repo.
#!/usr/bin/env bashset -x # print each command before executionset -o errexit # always exit on errorset -o pipefail # don't ignore exit codes when piping outputset -o nounset # fail on unset variables# set up the repogit clone https://github.com/zeke/all-the-package-namescd all-the-package-namesnpm run buildnpm test# bail if no changes are present|| exitcount=$(cat names.json | wc -l)git add names.jsongit config user.email "email@example.com"git config user.name "Zeke Sikelianos"git commit -m "$count package names"npm version minor -m "bump minor to %s"npm publishgit push origin master --follow-tags# clean upcd ..rm -rf all-the-package-names
Tip: Heroku's node buildpack will install
dependencies from package.json
by default. If your build script requires
devDependencies to be installed too,
set the following in your app environment:
heroku config:set NPM_CONFIG_PRODUCTION=false
There are a few ways to manually test a script on Heroku. You can run it remotely:
heroku run npm run release
or you can shell into your app and run one-off commands inside the shell:
heroku run bashnpm run release
Once you've got a working release script, it's time to automate!
Heroku has a very useful (and free) add-on called Scheduler that allows you to configure one-off tasks to run against your app. There's a web-based GUI for scheduling tasks, and they can be run every ten minutes, every hour, or once a day. It's like cron, but much easier to use.
To install and open Scheduler:
heroku addons:create schedulerheroku addons:open scheduler
The scheduler interface is pretty simple. You specify a command to run and the frequency at which to run it.
From the Heroku docs on Dyno Sleeping:
Free dynos will sleep when a web dyno receives no web traffic for a period of time. In addition, if a free dyno exceeds a quota of 18 hours of activity during a 24 hour window, it will be forced to recharge.
This means you get 18 hours of compute time per day for free! And because
this app doesn't have a
web process, it only needs to be "awake" for a few
minutes a day for the short amount of time required to run the release task
Even for apps with long-running tasks at shorter intervals, the free dyno would still probably be adequate. A task that runs every 10 minutes and takes 5 minutes to complete only amounts to 12 hours of compute time per day, which is still under the 18 hour limit.
If your build process takes a long time or needs to run more frequently, a hobby dyno is adequate for most tasks and will only set you back $7 a month.
Once your automated publishing workflow is finished, you can kick back and let the machines do the work, then watch your contribution graph turn a very dark shade of green...