Please, security test our code!

Since Oracle was so nice as to remind everyone what software security is really like with closed-source software, I wanted to remind people how finding and reporting security issues works in PostgreSQL:

1. Feel free to "reverse engineer" the code.  In fact, here it is on github if you want to scrutinize it.
2. We generally credit security researchers who find real security holes (with limitations for duplicates, etc.).  Over the last few years, some of the most critical bugs in PostgreSQL were found by professional security researchers doing things like fuzz testing.
3. If you think you've found a security issue, please report it to security@postgresql.org. If it turns out to be a non-issue, we'll tell you, and you can report it as a regular bug.
4. Be prepared to answer questions about your report.  We showed you our code, you can show us yours.

Our open approach to security is the reason why PostgreSQL was rated by the Database Hacker's Handbook as "the most secure by default": more secure than Oracle.  And why for five Defcons in a row, security hackers have been unable to crack PostgreSQL in the annual Schemaverse competition.

And please ... update your servers! We send out those incremental updates for a reason, and often as not, there are security patches.   More information on our Security Page.

Note: the above is my personal opinion and is not the opinion of The PostgreSQL Project or any other organization.

Using Docker to isolate VPN clients

As a consultant, I need to use VPNs to log into many of our client environments.  As one of our data security policies, our staff are not allowed to connect to multiple customer VPNs from the same machine at the same time; plus many VPNs do not permit this.  However, this causes some work efficiency issues whenever we need to run a job for a customer which involves a lot of wait time, such as building an index.

I've used VirtualBox VMs as isolation environments, but VBox consumes a lot of memory and system resources.  So I started thinking: could I use Docker containers to supply some network/file isolation while connected to multiple VPNs?  The answer is yes.

Let's take the difficult case first: a Cisco IPSec VPN.  First, I exported the VPN configuration from Gnome's network manager, which creates a PCF file.  Next, I start Docker container based on the official Ubuntu Trusty images, giving it the net_admin Linux cap to allow the VPN to work.  I also need to mount the new PCF file into a directory in the container so I can access it.

     docker run -it --cap-add net_admin \
        --volume /home/josh/vpns:/mnt/vpn ubuntu:trusty

After this I need to install the connection software:

     apt-get update
     apt-get install vpnc ssh

Once vpnc is installed, I need to convert that PCF file to a vpnc file and install it:

     cd /mnt/vpn
     pcf2vpnc vpn.pcf /etc/vpnc/vpnc.conf
     chmod 600 /etc/vpnc/vpnc.conf

Once that's all set up, I can start the VPN:

     root@c5b7802ca120:/# vpnc vpnc
     Enter password for jberkus@10.1.1.15:
     VPNC started in background (pid: 78)...

Success!  Sometimes, though, you'll see this error message:

         Cannot open "/proc/sys/net/ipv4/route/flush"

I haven't pinned down what causes that yet, but it seems to have no effect on actual ability to use the vpnc VPN.  Probably I need one more --cap-add, but I don't yet know what it is.

Anyway, once you have that running, docker commit the image and you have a saved VPN image for that particular VPN (just remember not to upload it to Docker Hub).  Then you can run the image whenever you need that VPN.

OpenVPN is actually more complicated, because device access as well as net access is required.  On the up side, you only need the config file to actually connect.  See this project on how to set up your container, and then run:

       openvpn --config /mnt/vpn/config.ovpn

Now, the caveat:  Docker containers do not supply nearly the levels of isolation which a full VM would, especially since we have to grant the "net_admin" cap to get the VPN to work.  While containers prevent accidental network cross-connection, they would not block targeted malware which attacks the network stack. In our case, that means I can use this technique for our regular clients, but not for PCI-compliant customers or other clients with strong regulated data isolation requirements.

But for the others, I can now easily run 2 VPNs at a time, while protecting them from cross-connection, and me from accidentally targeting the wrong customer's server.

Thanks to Rikaard Hosein, programmerq, and other helpful folks on IRC channel #docker for advice on this.

Some notes on today's update release

We released a security and cumulative bugfix release to all supported versions today.  That means it's update time.  What follows is my personal advice on the update.

For the first time in a while, we have a bunch of "low-risk" security fixes in this release, but no "critical" security fixes.  The reason I put those terms in quotes is that it doesn't matter how critical the fixes are in general; it matters how critical they are to you.  So you should definitely read over the release notes and the CVE notices to check how they affect you.

All five of the security holes patched require prior authentication.  Four of the five have not been proven to have an actual privilege escalation vector; they may be only denial-of-service attacks.  And the fifth security issue only affects you if you are using per-column privileges for columns with constraints on them.  That's why I regard these issues as relatively "low-risk".

There are also some important fixes to performance and replication for versions 9.4 and 9.3, so users of those versions should apply the update soon.  For other users, unless you live in the Fiji Islands or other places affected by timezone changes, you can probably wait for your next scheduled maintenance window.  You do have scheduled maintenance windows, yes?

Other people who might care to apply this update sooner rather than later include:

• Users who have already had issues with autovacuum
• People using the new logical decoding
• Users who have a single archive which is shared between master and replicas.
• Folks who create a bunch of tablespaces.
• Developers who use tsquery, xpath(), and/or complex regular expression searches
• JSONB users.
• Norwegians who use Postgres on Windows
• Users who have reported bugs with explicit locking and deadlocking in the last few months.

Again, though, read the release notes.  Because it's always possible that we fixed a bug that already affects you.

Another reason to set pg_hba.conf restrictively

pg_hba.conf can be a pain in the neck.  You're expecting some burst traffic, you deploy 8 new virtual appservers, fire them up, and ... they can't connect to Postgres.  Gah!

That kind of experience makes it tempting to set everything to:

   host  all  all   0.0.0.0/0   md5

But I just encountered another case this month why you shouldn't.  While everyone thinks about malefactors as the main reason to have tight restrictions on which hosts can connect via pg_hba.conf, one should remember that accidents are far more common than malice.  As was the case this time.

I set up a benchmark test for a client which involved replaying 8 hours of activity logs against a test copy of their database.  What they didn't tell me (because they didn't know due to poor internal documentation) was that their database used DBlink to write data to another database several times a minute.

Thank goodness they had exercised good pg_hba.conf practices!  All I got was a bunch of DBlink errors, instead of scribbling all over production data.   Whew!

Recap of last week's security release

Rather than writing up a blow-by-blow of last week's security release here, I've written a full article for LWN.   Subscription required, I'm afraid (or you can wait 2 weeks).

Now we find out how many Postgres servers there really are

Seven years ago, the MySQL worm spread throughout the internet by taking advantage of a critical vulnerability in MySQL authentication on Windows.  Before the worm, I hadn't really believed Marten Mickos' claims about the number of MySQL installations.  Once over 30,000 servers were infected, though, I believed him.

Well, given the vulnerability announced and patched today, I think we'll have a chance to find out how many PostgreSQL servers there are.  You have updated your servers, right?

In the PostgreSQL world, we're used to thinking in terms of a few thousand users because that's the number which is active in the community and that we hear from regularly.  It's likely that today's vulnerability will show us how many PostgreSQL users there really are.  For example, this security scanner finds over 120,000 PostgreSQL servers which are listening on port 5432 on public IPs (and if your server is on that list, you'd better patch it!).

No idea why 40% of these servers are in Poland.  I had no idea that Depesz was that busy.  A word to the wise, Depesz: firewalls?

Today's Security Update: XML vulnerabilities

If you're subscribed to the PostgreSQL News feed (and if not, why aren't you?) you already know that we released another security patch today.  This update release patches two security holes having to do with XML functionality, which can be used by any authenticated user for privilege escalation.  Everyone should apply the updates at the next reasonable downtime, and users of contrib/xml2 should schedule a downtime this weekend.

You'll notice that this security update release is on a Friday.  This is not our normal practice, since it makes it difficult for IT departments to respond in a timely fashion (especially the folks in Australia/Japan, who will be getting the announcement on Saturday).  However, as our number of packagers for different platforms has grown, it's become increasingly difficult to schedule coordinated releases, especially during the summer and the India/Pakistan holiday season.  As it is, the Solaris folks will be getting their binaries late.

Anyway, about these vulnerabilities:  The first one is a vulnerability in the built-in xmlparse() function.  Among other things, this function used to allow users to validate the XML against a DTD contained in an external file.  However, a creative user would be able to call any file which the "postgres" user has permissions on, and read parts of that file in the error messages from the DTD reader.  Since "files the postgres user has access to" includes pg_hba.conf and .pgpass (if defined), this is a dangerous capability.  As such, we have had to disable the DTD-validation feature.  This validation feature may return at a later date if we can figure out a reasonable way to make it safe.

The second security hole is in the outdated extension (or "contrib module"), xml2.  Users still use xml2 rather than the built-in XML because of its XSLT support.  The problem is that xslt_process() had a documented "feature" which allowed the function to transform XML documents fetched from external resources, such as URLs -- without requiring superuser permissions.  Not only could this be used to read local files on the PostgreSQL server, it could be used to write them as well, making this a much worse security hole if you have xml2 installed.   As such, the xslt_process() feature has been disabled, and will probably not return.

We've been sitting on both of these patches for an embarassingly long time (for our project, at least), because we were looking for a solution which didn't involve disabling functionality which is potentially valuable to some users.  Sadly, we were not able to.