About

This site collects and displays results of regression tests for distributions (i.e. plugins, installed into the database), published at PostgreSQL eXtension Network (aka PGXN). It's inspired by the official PostgreSQL buildfarm system, which does about the same thing for PostgreSQL itself. There are some important differences, though.

The ultimate goal of this is empowering the distribution authors with feedback about failing tests, causes of the failures etc. Because who runs all the regression tests on all PostgreSQL version / platform combination, supported by the distribution, for every change? Or whenever a new major PostgreSQL version is released? I certainly don't. Assuming you have written the tests.

Hopefully, this will result in more reliable, less fragile distributions, available to PGX users.

Limitations

The system is not perfect and has limitations that sometimes cause false positives or false negatives. This comes from the variety of distributions, platforms etc. So take the current results with a grain of salt, and it will take some time to fix this.

Also, currently only extensions published on PGXN are tested (because that's where we get them using pgxnclient). However, it's possible that in the future this limitation will be lifted.

API

The system provides a simple JSON-based API. It's inspired by RESTful principles (although it's not RESTful and never will be), and PGXN API.

All the data that you see on this website are served purely through the API - try rewriting http://pgxn-tester.org/{...} to http://api.pgxn-tester.org/{...} and you'll see the actual JSON data returned by the API.

The API is also used by the clients (requesting info what to test, submitting results back), again using HTTP/JSON. And of course, the collected data is stored in PostgreSQL.

The API documentation is available here. It's far from perfect, so let me know if you have questions.

HTTPS

Both the API and web UI is available by both HTTP and HTTPS. There's nothing really secret here, so the HTTPS support may seem pointless, but it allows you to query the API from other HTTPS sites easily.

The certificates are issues by CAcert, a community-driven Certificate Authority issuing certificates for free. The downside is that the CA is not included in common browsers, so you have to install the root certificate on your own.

Running the tests

The best way to participate is to run the testing client, which gives us more data (especially if you're running a unique machine (e.g. operating system different from the other clients). To do that, all you need to do is submit a new machine for approval (TODO), and then configure and run the pgxn-tester-client (LINK).

The machines are identified by names of famous fictional computers (books, movies, ...), and you may choose names for machines you run. Machines running regression tests go crazy from time to time (probably due to the quality of the tested code), this list might be a good inspiration. Also, there's a longer list on wikipedia.

The only name that's unacceptable is "skynet" - we don't want to be the ones responsible for the rise of terminators and destruction of humanity, right?

Security

The one major difference compared to running regression tests of PostgreSQL itself is that while the code commited to PostgreSQL is closely reviewed, running regression tests of distributions essentially means running arbitrary C code downloaded from the Internet.

While I'm not aware of any "evil" distributions doing nasty things, it's probably only a matter of time. So you really need to isolate the tests as much as you can - run them with a user with minimum privileges, with no access to important data etc. If you can run the tests in a VM (LXC, kvm, vmware, ...), that's even better.

Generally, the best isolation is provided by HVM virtualisation solutions (e.g. KVM-based solutions like qemu-kvm). Recently, there's a lot of fuss about LXC, and it's often promoted as a "lightweight virtualization" - be aware that while it's maturing quickly, it relies on kernel namespaces and not all resources are properly namespaced in the kernel (especially true for production deploymeent, frequently using older kernels and LXC versions). Also, it's "same kernel" solution which means the container (essentially, just a named group of processes) uses the same kernel as the rest of the system, so if there's a local exploit (e.g. a broken syscall) ...

Contributing code

The other thing you can do is contributing code, both to the client or server. Just send me a pull request on github.

Donations

Of course, while running this is not extremely expensive, is not free. And with some additional money, we could for example for an AWS instances with Windows (difficult to get otherwise) and run the tests on it. We don't have any "donation button" right now, so contact me directly.

Feedback

If you need to discuss something about this site, post a message to the PGXN group. If you need to reach me directly, send me an e-mail at tomas@pgaddict.com.