Introducing PNPM

pnpm much like its name might suggest is a private npm registry server. What follows is something I might put in a readme for pnpm, but a bit more verbose.

Before we get too far into this post, if you want to play with a demo of pnpm, you can find it here; hopefully it's still alive, sorry if I take it down before you find this.

npm adduser --registry="https://pnpm.herokuapp.com/"
npm publish --registry="https://pnpm.herokuapp.com/"

If you don't have a random package to test publish, you can always use pnpm itself.

At my job I have been trying to get my team into node.js which I have had previous experience with and really enjoyed. One thing I really liked about using node was node_modules and npm. Breaking up our applications into small and simple modules has some useful properties: reuse through composition and encapsulating complexity behind a simple api.

One issue with using npm to publish modules is that it doesn't work for internal private code. There are a lot of ways around this: use git urls for dependencies or a paid private npm server. Git urls don't easily integrate with the rest of npm semantic versioning (plus sadly we don't use git internally) and a paid private server adds another dependency on external resources.

Given that, I decided on looking into a self-hosted npm registry server. One of the first things I found was using a couchdb mirror of registry.npmjs.org. The issue with is that the public npm registry is very large and we don't use most of it, so we would just be wasting a lot of hard disk. Next, I happened upon sinopia and it worked really well at home when I was on a Linux machine. When I got to work, primarily Windows machines, I ran into several issues. The file locking mechanism the was working in Linux, didn't work in windows and took me a while to find. The source for sinopia was a bit disorganized and I didn't feel like making any drastic changes, so I decided to start my own project.

Another more personal reason was tinkering with npm to get a better feel for how it worked. I had always been interested in how npm worked, but now at least I know what http requests it makes to the registry and have gone through the docs a few times to learn all the cool things it could do.

What made this project so fun was how little code I actually had to write. The npm registry is written in js and level is a cross platform embedded json store (I tried nedb at first because its written entirely in js but had issues with json documents with perionds in the keys). All I had to due was to glue them together and setup the correct routes in express. Now of course I'm not done yet and there there is room for improvement, but already npm search, npm install and npm publish work without a fuss.

NOTE: many of the tests are directly from npm registry as well; also its an admin party so go crazy with the demo.

Here is a quick layout of the project:

.
├── app       - web app code (for npm and web client)
├── lib       - data access code (mostly leveldb puts/gets)
├── npm       - npm registry js code (modified to work in node)
├── public    - web ui code (modeled after npmjs)
├── views     - html views  (also modeled after npmjs)
└── test
    ├── npm   - npmjs tests
    └── unit  - basic unit tests

The code lacks comments but most of it is standard express.js/mocha and is very clean and minimal.

To actually get this darn thing working on your system, you really should have just read the readme. Just kidding, moving along; pull down the repo as such:

git clone https://github.com/djblue/pnpm.git; cd pnpm

If you are on windows and have visual studio 2013 (the case of my system at work):

npm install --msvs_version=2013

If you have a different version, just switch it up to your version.

If you are using a system with python 3 (the case of my system at home) as the default python:

npm install --python=python2

The reason you need to specify this is the level is a node c extension which means it need to be compiled. On Linux that would be gcc and on Windows that would be visual studio. Make sure you have those installed on their respective systems.

After the install, go:

npm start

Now you should be good to go. Just make sure the registry flag is specified to http://localhost:3000 or whatever domain/port you are running it on.

Another useful feature, also present in sinopia, is the ability to install all your private/public packages in one command. If pnpm doesn't find your package locally published, it will ask npmjs if it has it. If npmjs has it, pnpm will return that result and store it locally. This is handy if you don't want to store your node_modules directory in your revision control system.

NOTE: I haven't tried to optimize database access for scaling. It's pretty much all in memory manipulations and doesn't use stream like it should.

I have recently gotten into docker, so I will probably add that to the project soon which would make pnpm even easier to play with or deploy. Let me know if this interests you.

So I hope if you've made it this far and that you spend some time playing with pnpm, maybe even contribute.

Thanks for reading.