Last Week in Kubernetes Development: Week ending January 21

Community Meeting Summary

The demo for this week's meeting was Kubernetes running on Docker For Mac. The Docker.com folks have been hard at work enabling this, and the demo now looks pretty polished. Developers with Mac desktops should be able to easily use Kubernetes with their existing Docker for Mac workflows.

Jaice DuMars explained the delay of the alpha release and gave a 1.10 stats update (see below). Dan Williams updated folks on SIG-Network, who have been making a lot of changes, including adding IPv6 support. Phillip Wittrock updated everyone on what the Steering Committee is currently working on, especially creating template SIG charters so that all SIGs can create their own charters. If you have opinions about the organization and leadership of your SIG, please take the survey on SIG governance.

Kubernetes will be participating in Google Summer of Code with the CNCF this year. Please contact Ihor Dvoretskyi if you are interested in mentoring or know a student. SIG Intros and Deep Dives at KubeCon Europe will be announced soon. The project will have another "Meet Our Contributors" on February 7th, this one focused on helping out new contributors (contact Paris Pittman to participate).

The format for the Community meeting will also be changing slightly in the future. SIGs will be scheduled for updates per release cycle instead of ad hoc, and demo speakers will be asked to rehearse before the meeting.

Release Schedule

This was week 3 of version 1.10 development. This week should have included an early alpha release, mainly as a dry run for release packaging. However, it's been delayed because Branch Manager Caleb Miles had a painful bike accident and has been offline. An alpha release is expected this week.

Feature Freeze, which was supposed to be January 22nd, has also been delayed by one week because the Features Lead is still waiting for status clarification on some features from several SIGs. Final Feature Freeze deadline will now be on the 29th. Many SIGs have updated their features, though, and Ihor has created the Feature Tracking Board for version 1.10.

Feature Work

While 148 patches were merged last week, most of them were minor bug fixes (including at least ten for GCE support), cherry-picks for copying fixes across releases, typo fixes, and some doc and release note corrections. Among the interesting feature work was:

• Kubeadm now supports overriding flags for the api-server, controller, and scheduler during kubeadm init.
• RBAC now returns reasons for allowed authorizations, making them auditable.
• The way that Secrets, ConfigMaps, and the downwardAPI update their data has been changed. If you work with one of these features, you may want to check your code against the refactor.
• The api-server now supports custom cipher suites for connections.
• The api-server and controller now run in privileged containers, enabling correct installation on OpenStack.
• The kubelet and controller manager have been refactored to use cobra CLI command communication, which is part of ongoing work to clean up component configuration in Kubernetes and HyperKube.
• CRDs can now be created using YAML as well as JSON.

 

Deprecated

• Dockershim no longer re-implements the CRI API, and instead communicates with the Kubelet using pure gRPC.

 

Version Updates

• kube-dns for Kubeadm is now 1.14.8

 

Other Merges

• We now have automated tests using the vSphere Cloud Provider simulator
• The implicit disabling of admission plugins that are not in the enablement list for kube-apiserver has been restored.
• A major issue with using custom Admission Control config files has been fixed.
• Keepalives for the etcd server have been adjusted so that they actually close the socket if the client can't connect.
• The Pod describer now includes HostPort in its output.

 

Graph of the Week

 

This week's graph, brought to you by Jorge Castro, is Approvers and the Approvers Histogram This graph shows you the number of pull request approvals in each repository, and the accompanying histogram shows you who did those approvals. Looks like @cblecker is our leading approver for last month.

The great JSONB tradeoff

One thing we're waiting on to get 9.4 out the door is making a decision on JSONB storage.  Currently under debate are patches to the JSONB format which would, for users with large numbers (as in 150+) of top-level keys, cause JSONB to take up to 60% less space on disk in exchange for an up to 80% increase (that is slowdown) in response times on key extraction.  We need your feedback as potential users of the feature to make a good decision about what's best.

At stake is the question of whether the key locations should be saved as successive offsets, or as lengths.  The advantage of the former is that it speeds up extraction by making it only two lookups to locate a top-level key regardless of the number of keys you have.  The disadvantage is that a series of increasing offsets isn't very compressible.

For JSONB fields which consist of a few top-level keys and large values, this question makes no difference at all.  However, for the worst case ... 150+ top-level keys with short (under 10 bytes) values, the difference is quite dramatic.  For example, I constructed a test with 183 keys, of which 175 were NUMERIC.  I checked both table size and time to extract key #160 from 100,000 rows:

Metric	Offsets	Lengths
Table Size	1147 MB	541 MB
Extract 100K	2.8s	5.2s

This is a "worst case" scenario for the difference between these two designs.  Note that the extraction slowdown affects only retrieving the value to the client; it does not affect index lookups of JSONB rows, which are speedy no matter which patch is employed.

However, we're undecided on this "fix" because we don't know a couple things:

• How likely are users to have 150+ top-level keys in one field (or keys on any single level together) with short values?
• Is up to 60% space savings in return for up to 80% extraction slowdown a good tradeoff?  Or a bad one?

As such, I am turning to you, the users, to help us decide what tradeoff makes sense for 9.4.  Let me know in the comments.

More on 9.4 percentiles

Some of you may be following the discussion on JSONB compression on -hackers, which is the remaining major issue for 9.4.  As part of this, I needed to run some statistics on average column sizes for JSON vs JSONB tables. This makes a good example of how incredibly useful WITHIN GROUP will be in 9.4.

First, I created two versions of the data, one called "jsonic" which has text-JSON data, and one called "jsonbish" which has the JSONB version.  Then I ran some comparisons using pg_column_size on the JSON columns.

with colsizes_b as (
    select pg_column_size(jsonbish.jcol) as colsize
    from jsonbish
),
distrib_b as (
    select percentile_cont(array [0,0.25,0.5,0.75,1])
    within group (order by colsize) as colsize_distribution
    from colsizes_b
),
colsizes_j as (
    select pg_column_size(jsonic.jcol) as colsize
    from jsonic
),
distrib_j as (
    select percentile_cont(array [0,0.25,0.5,0.75,1])
    within group (order by colsize) as colsize_distribution
    from colsizes_j
)
select 'json' as thetype,
    colsize_distribution from distrib_j
union all
select 'jsonb',
    colsize_distribution
    from distrib_b;

So I'm taking the column size of each row, then sorting them by size, and then doing a percentile distribution using the new WITHIN GROUP.  There's no group there because the group is actually the whole table.  I've chosen the usual box plot percentages: minimum, 25%, median, 75%, and maximum.

And the results:

 thetype |    colsize_distribution   
---------+----------------------------
 json    | {1741,1767,1854,1904,2292}
 jsonb   | {3551,5866,5910,5958,6168}

... which demonstrates that we do indeed have a major issue with compressability.

Why HStore2/jsonb is the most important patch of 9.4

There are a bunch of features which are pending for 9.4, still, and a bunch of features which are already committed.  Given how interesting some of those are: SET PERSISTENT, Logical Changeset Extraction, Materialized Views, etc., I think a lot of people will be surprised that I think Hstore2/jsonb is the single most important patch -- important enough that I think we shouldn't release 9.4 unless it goes in.  Why would I make this wild assertion?  Let me explain.

Open source databases rise and fall on the popularity of the programming languages which use those databases.  MySQL largely rose on the success of PHP, and it fell as PHP became marginalized.  Our current PostgreSQL salad days are based largely on the current hegemony of Python, Ruby, and Rails.  While other events have fed into changes in adoption, where the numbers of developers come from is really "what is the default database for popular language X".

While the future is unpredictable, the current momentum in programming languages is behind two platforms: Node.js and Go.  PostgreSQL already enjoys good support and adoption among Go users.  However, our adoption in the Node.js community is less encouraging.

I was given a set of statistics I'm not allowed to publish, but I can summarize them.  Two of them are fairly alarming:

• PostgreSQL is the database for fewer than 1 out of 8 Node.js deployments which use a database.
• The rise in popularity of MongoDB almost exactly parallels the rise in usage of Node.js.

If you've watched database adoption trends for the last 20 years like I have, this is alarming.  We are in danger of being sidelined.  If we want PostgreSQL 10.5 to enjoy the same level of adoption that version 9.3 does, then we need to appeal to Node.js users and whatever comes after them.

What do Node.js users want that we don't have?  There's three main things that I've been able to identify:

1. A better, faster, driver which fully supports asynchronous querying.
2. Relatively painless multi-node scaling
3. Full, indexed support for jsonish hierarchical data and queries.

The first two points need to happen outside the core PostgreSQL project, at least for 9.4.  However, the last point is very much on the table; we have the HStore2/jsonb patch pending.  If that goes in, the PostgreSQL project will be seen as still making progress and still relevant to Node.js users and to other people who like document databases.  If it gets booted to 9.5, and there is no discernable progress on JSON features in 9.4, I believe that we will have permanently conceded the bulk of database market to the new databases for the forseeable future.

Oh, and if anyone wants to work on our Node driver ... please pitch in!

Adjusting defaults for PostgreSQL.conf

For PostgreSQL 9.4, we've been discussing adjusting defaults for postgresql.conf to give users a better out-of-box experience.  Our existing defaults were set at a time when we used SysV memory, and when the average developer laptop or low-end server had 512MB of RAM.   So, a survey.

Here's some things contributors have already adjusted:

• shared_buffers now goes up to 128MB automatically depending on available RAM reservation (we did this for 9.3)
• effective_cache_size is automatically set to shared_buffers X 3 (might be X 4 by the time we release)

Now we are discussing increasing the defaults and/or autotuning for work_mem and maintenance_work_mem.  Suggested new defaults are 4X the current: 4MB and 64MB.  These should allow PostgreSQL to run on any machine of 1GB or greater memory a bit better.

However, one thing which came up during this discussion is that many people may be running PostgreSQL on virtual machines which have less than 1GB of RAM available to PostgreSQL.  I don't think they are, but we lack data.  So I put up a survey to see what amounts of RAM our users really have on their smallest machines/VMs.   Please fill out the survey so that we can have reasonable input for development, and if you have comments, please put them below.

Free VMs for Reviewers

In case you somehow missed it, the second CommitFest for PostgreSQL 9.4 development is underway.  While we don't have quite as many patches in the queue as we had last time, we have quite a few, and more than we've had for any CF2 before.  This means we need as many reviewers as possible!

In order to make sure that nobody is prevented from reviewing due to not having a suitable development environment available, the PostgreSQL community is offering free virtual machines for reviewing and testing patches for this CommitFest.

If you want a VM for this purpose, please email me the following at any point during the CF:

1. Your public SSH key.
2. Which patch(es) you intend to review/test.
3. Preference for CentOS or Ubuntu.
4. When you expect to do the review.

Since these are EC2 virtual servers, they will not be suitable for performance testing.  However, if you want to review a replication patch, we can make multiple VMs available.  All VMs will be terminated at the end of the CF.

VMs will come equipped with all tools required to build PostgreSQL, plus a checkout of the PostgreSQL code from the beginning of the CF.  You may need to install additional requirements yourself if you are reviewing extensions or special features with dependencies.

Note that I may take a day to respond with your VM access.

9.4 Commitfest 1 Wrap-up

The first CommitFest of the 9.4 development cycle involved an unprecedented number of patch submissions.  As of today, 9.4CF1 is still open because of one patch which apparently only Tom Lane can commit.  However, everything else out of a brutal commitfest has been done, and it's 16 days after the CF was supposed to finish, so I think I can write the wrap-up now.

First, let's talk numbers: this CF started with 106 patches, peaked at 108 patches, and finished with 102 patches (6 were moved to September).  This is over 50% more patches than we've had for the first CommitFest of the annual development cycle and more patches than we've had in any single CF before. Last year, CF1 only involved 59 patches.  So if you are a committer and you feel exhausted, that's why.  Some graphs:

As you can see, this year is a jump forward in the number of patches being submitted to the first commitfest, and if it's representative of the rest of the year, we're all going to spend a lot more time reviewing patches.

Why so many patches?  Well, I can think of a few reasons:

• contributions from the staffs of SalesForce and Huawei
• beginning of 2ndQuadrant's work on streaming logical replication
• a bunch of regression test patches
• some new major contributors, including one from Google Summer of Code
• more people submitting WIP patches to get spec feedback

All of the above are unquestionably good things, and not anything we want to deter, so we need to find ways to shoulder the load.  On the other hand, I'm not going to pay much attention to anyone who tells me we're "making things difficult for submitters".

So what did we get out of all this patch activity?  Some highlight features:

• new pgbench options
• ALTER TABLE ... ALTER CONSTRAINT
• improvements in compression of BLOBs and large rows
• a bunch more regression tests
• REFRESH MATERIALIZED VIEW CONCURRENTLY
• create background workers at runtime
• FILTER option for aggregates
• Part 1 of streaming logical replication
• WITH ORDINALITY to supply "row numbers" for set-returning functions

All in all, 49 patches were committed (I'm expecting the remaining pending patch to make a round 50).  However, 46 were sent back and an additional 6 were moved to the September CommitFest, so we need to plan for that one to be busy too.

So, what are we going to do to handle more patches?  I have some thoughts on that in my next blog post ...

9.4 Commitfest 1 Has Begun!

The first CommitFest for version 9.4 of PostgreSQL has started, and we have an inspiring list of new features, in 98 patches, ready to be reviewed.  Which means we need you to help review them!  Among the pending patches are:

• Logical Changeset Generation, the basis for the new replication;
• Setting postgresql.conf variables from the SQL command line;
• Transaction Log size autotuning;
• Transforms between custom PostgreSQL datatypes and PL/Python, Perl and Javascript datatypes;
• Row-Level Security;
• GIN Fast Scan, and other GIN index improvements;
• Block-based range indexes for very large tables;
• Concurrent update of materialized views;
• Multiple pgbench enhancements;
• Many performance improvements;
• A bunch of additional regression tests.

Given that that's nearly 100 patches, and the PostgreSQL project has only 20 committers, it means we need you to help us review patches.  Yes, you!  You don't have to be able to hack PostgreSQL code to review a patch; if you can apply a patch to source code and build PostgreSQL using "make", you can help.  More information here in How To Review A Patch.

If you want to help, but aren't sure what to review, please join the pgsql-rrreviewers mailing list and announce yourself there.  I will assign you a patch to review.

For those of you who are experienced reviewers or who have submitted a patch to this CommitFest, please note the following:

1. If you submitted a patch to this CommitFest, you are expected to review at least one other person's patch.  Should you choose not to do so, the community will not be obligated to guarantee you a review of your patch.
2. If you put your name down as a reviewer on a patch, and you don't post anything for 5 days, we will send you a reminder and take your name off the patch, to prevent you from blocking other reviewers.
3. If your patch goes to "waiting on author" and you don't respond for 5 days, the patch will be Returned with Feedback.

Rules 2 and 3 are new per this year's developer meeting.  Thanks, and lets have a fast and productive CommitFest!

PostgreSQL New Development Priorities 5: New User Experience

So, I started this looking for our five major goals for future PostgreSQL develoment.  The last goal is more nebulous, but I think equally important with the other goals.  It's this: improve the "new user experience".

This is not a new goal, in some ways.  Improving installation, one of our previous 5 goals, was really about improving the experience for new users.  But the new user experience goes beyond installation now, and competition has "raised the bar".  That is, we matched MySQL, but now that's not good enough; we need to match the new databases.   It should be as easy to get started on a dev database with PostgreSQL as it is with, for example, Redis.  Let me give you a summary of the steps to get up, running, and developing an application in the two platforms:

Redis:

1. install Redis, either from packages or multiplatform binaries.  No root access is required for the binaries.
2. read a 1-page tutorial
3. run redis-server
4. run redis-cli or install drivers for your programming language
5. start developing
6. when your app works, deploy to production
7. in production, tune how much RAM Redis gets.

PostgreSQL:

1. install PostgreSQL from packages or the one-click installer.  Root/Admin access is usually required.
2. search the documentation to figure out how to get started. 
3. figure out whether or not your packages automatically start Postgres.  If not, figure out how to start it.  This may require root access.
4. Install drivers for your programming language.
5. Figure out how to connect to PostgreSQL.  This may require making changes to configuration files.
6. Read more pages of documentation to learn the basics of PostgreSQL's variety of SQL, or how to program an ORM which works with PostgreSQL.
7. Start developing.
8. Deploy to production.
9. Read 20 pages of documentation, plus numerous blogs, wiki pages and online presentations in order to figure out how to tune PostgreSQL.
10. Tune PostgreSQL for production workload.  Be unsure if you've done it right.

The unfortunate reality is that a new user will hit a lot of points in the "getting to know PostgreSQL" where they can be stuck, confused, and at a loss.  At those points, they may decide to try something else, and never come back.  I've seen it happen; just last SFPUG I was talking to a guy who started on Postgres, ran into a shared memory issue, switched to Mongo, and didn't come back to Postgres for 2 years.

So, what can we do about it?  Well, a few things:

• better new user tutorials, such as the ones on postgresguide.org
• better autotuning, made a lot easier to implement as of version 9.3.
• a "developer mode PostgreSQL"

The last would be a version of PostgreSQL which starts when the developer opens a psql prompt, shuts down when they exit, starts with minimal processes and crash safety turned off, and above all with a security configuration which allows that user to immediately connect to PostgreSQL without figuring anything else out.  With some of the work on recovery mode supplying a single-user Postgres, this should become easier, but it needs a lot more work.

Those are the five things I can see which would greatly expand the market for PostgreSQL and keep us competitive against the new databases.  Yes, I'm talking really big features, but any two out of the five would still make a big difference for us.  There may be others; now that you've seen the kind of big feature I'm talking about, put your suggestions below.

PostgreSQL New Development Priorities 4: Parallel Query

Parallel query is the first priority from those suggested in the comments that I agree should be a major PostgreSQL development priority.  I think that Joel Jacobson summarized it neatly: Bring Back Moore's Law.  Vertical scaling has always been one of PostgreSQL's strengths, but we're running into hard limits as servers are getting more cores but not faster cores.  We need to be able to use a server's full CPU capacity.

(note: this series of articles is my personal opinion as a PostgreSQL core team member)

The benefits to having some kind of parallel query are obvious to most users and developers today.  Mostly, people tend to think of analytics and parallel query across terabyte-sized tables, and that's definitely one of the reasons we need parallel query.  But possibly a stronger reason, which isn't much talked about, is CPU-heavy extensions -- chief among them, PostGIS.  All of those spatial queries are very processor-heavy; a location search takes a lot of math, a spatial JOIN more so.  While most users of large databases would like parallel query in order to do things a bit faster, PostGIS users need parallism yesterday.

Fortunately, work on parallelism has already started.  Even more fortunately, parallel query isn't a single monumental thing which has to be done as one big chunk; we can add parallelism piecemeal over the next few versions of Postgres.  Rougly, parallel query breaks down into parallelizing all of the following operations:

• Table scan
• Index scan
• Bitmap scan
• In-memory sort
• On-disk sort
• Hashing
• Merge Join
• Nested loop join
• Aggregation
• Framework for parallel functions

Most of these features can be worked on independently, in any order -- dare I say, developed in parallel?  Joins probably need to be done after sorts and scans, but that's pretty much it.  Noah Misch has chosen to start with parallel in-memory sort, so you can probably expect that for version 9.4.

PostgreSQL New Development Priorities 3: Pluggable Parser

Really, when you look at the long-term viability of any platform, pluggability is where it's at.  A lot of the success of PostgreSQL to date has been built on extensions and portability, just as the success of AWS has been built on their comprehensive API.   Our future success will be built on taking pluggability even further. 

In addition to pluggable storage, a second thing we really need is a pluggable parser interface.  That is, it should be possible to generate a statement structure, in binary form, and hand that off to libpq for execution.  There was recently some discussion about this on -hackers. 

If there were a way to hand off expression trees directly to the planner, then this would allow creating extensions which actually had additional syntax, without having to fork PostgreSQL.  This would support most of those "compatibility" extensions, as well as potentially allowing extensions like SKYLINE OF which change SQL behavior.

It would also help support PostgreSQL-based clustered databases, by allowing all of the parsing for a particular client to happen on a remote node and get passed to the clustered backends.  The pgPool2 project has asked for this for several years for that reason.

More intriguingly, it would allow for potentially creating an "ORM" which doesn't have to serialize everything to SQL, but can instead build expression trees directly based on client code.  This would both improve response times, and encourage developers to use a lot of PostgreSQL's more sophisticated features since they could access them directly in their code.

Taking things a step further, we could extend this to allow users to hand a plan tree directly to the executor.  This would fix things for all of the users who actually need query hints (as opposed to those who think they need them), as well as taking efficiency a step beyond cached plans.

There are a lot of reasons this would be just as difficult to do as pluggable storage.  Currently parsing depends on a context-dependant knowledge of system catalogs, including things like search_path.  So I have no idea what it would even look like.  But a parser API is something that people who hack on Postgres and fork it will continue to ask for.

PostgreSQL New Development Priorities 2: Pluggable Storage

Over the last decade, Greenplum, Vertica, Everest, Paraccel, and a number of non-public projects all forked off of PostgreSQL.  In each case, one of the major changes to the forks was to radically change data storage structures in order to enable new functionality or much better performance on large data.  In general, once a Postgres fork goes through the storage change, they stop contributing back to the main project because their codebase is then different enough to make merging very difficult.

Considering the amount of venture capital money poured into these forks, that's a big loss of feature contributions from the community.  Especially when the startup in question gets bought out by a company who buries it or loots it for IP and then kills the product.

More importantly, we have a number of people who would like to do something interesting and substantially different with PostgreSQL storage, and will likely be forced to fork PostgreSQL to get their ideas to work.  Index-organized tables, fractal trees, JSON trees, EAV-optimized storage, non-MVCC tables, column stores, hash-distributed tables and graphs all require changes to storage which can't currently be fit into the model of index classes and blobs we offer for extensibility of data storage.  Transactional RAM and Persistent RAM in the future may urge other incompatible storage changes.

As a community, we want to capture these innovations and make them part of mainstream Postgres, and their users part of the PostgreSQL community.  The only way to do this is to have some form of pluggable storage, just like we have pluggable function languages  and pluggable index types.

The direct way to do this would be to refactor our code to replace all direct manipulation of storage and data pages with a well-defined API.  This would be extremely difficult, and would produce large performance issues in the first few versions.  It would, however, also have the advantage of allowing us to completely solve the binary upgrade of page format changes issue.

A second approach would be to do a MySQL, and build up Foreign Data Wrappers (FDWs) to the point where they could perform and behave like local tables.  This may be the more feasible route because the work could be done incrementally, and FDWs are already a well-defined API.  However, having Postgres run administration and maintenance of foreign tables would be a big step and is conceptually difficult to imagine.

Either way, this is a problem we need to solve long-term in order to continue expanding the places people can use PostgreSQL.

PostgreSQL New Development Priorities: Scale It Now

The comments on my introductory post on this topic mentioned a lot of the major features which users would like to see in PostgreSQL.  Among those mentioned were:

• Improvements to replication
• Parallel query
• Index-organized tables
• Better partitioning
• Better full-text search
• Logical streaming replication

However, as with other projects, our perennial temptation is to listen to current users rather than potential users.  We can focus on making PostgreSQL better for the people who already use it.  It's attractive, but that way lies obsolescence.

What we need to focus on is the reasons why people don't use PostgreSQL at all.  Only by exploiting new markets -- by pushing Postgres into places which never had a database before -- do we grow the future PostgreSQL community.  And there's a bunch of ways we are failing new users.

For example, listen to Nelson Elhage, engineer at Stripe.com:

"I love Mongo's HA story.  Out of the box I can build a 3-node Mongo cluster with a full replica set.  I can add nodes, I can fail over, without losing data."

Wouldn't it be nice if we could say the same thing about Postgres?  But we can't.

If we're looking for a #1 PostgreSQL development priority, this is it:

We need a "scale it now" button.

This is where we're losing ground to the new databases.  In every other way we are superior: durability, pluggability, standards-compliance, query sophistication, everything.  But when a PostgreSQL user outstrips the throughput of a single server or a single EC2 instance, our process for scaling out sucks.  It's complicated.  It has weird limitations.  And most of all, our scale-out requires advanced database expertise, which is expensive and in short supply.


We need some way for users to go smoothly and easily from one Postgres node to three and then to ten.  Until we do that, we will continue to lose ground to databases which do a better job at scaling, even if they suck at everything else.

I don't know what our answer to scaling out will be.  Streaming logical replication, PostgresXC, Translattice, Hadapt, something else ... I don't know.  But if we don't answer this vital question, all our incremental improvements will amount to naught.

PostgreSQL's New Development Priorities: Part I

Six years ago, I came to the first Developer Meeting for PostgreSQL in Ottawa -- it wasn't even called that at the time, we just sort of pulled a bunch of people together -- with a list of the five big features which PostgreSQL was missing.  We'd just eliminated a big obstacle to Postgres adoption by creating the native Windows port, and these were the remaining five technical issues which most frequently caused people to evaluate PostgreSQL and decide not to use it.  These were:

1. Difficulty of install
2. Lack of built-in simple replication
3. Performance for web applications
4. Driver quality
5. Difficult configuration

As you can tell, a lot of the above was taken from interviews with MySQL users, but they were all legitimate issues.  Thing is, the PostgreSQL community in versions 8.4 through 9.2 pretty much eliminated the first three issues, the fourth was addressed by Rails, PHP and Python without any help from us, and the last item is well on its way to being addressed in 9.3.    There's still stuff left to be done; better autoconfiguration utilities, better OSX install, LinQ driver for .NET, etc., but as top priorities the above issues are largely under control or in the process of being fixed.

So what's next for PostgreSQL development?  What do we need to do, technically, to continue as the #1 open source database?  I plan to address this in a series of upcoming blog posts.


(note: I'm aware that a large chunk of database adoption is driven by non-techical issues.  However, that's not what I'm addressing at this time.)

PostgreSQL 9.3: Current Feature Status

So we're finally beginning the final playoffs for 9.3 features, otherwise known as "CommitFest 4".  Here's where we are on the various features which have been discussed, proposed, submitted, or thought about for 9.3.  The "likely/uncertain" is my own judgement based on the activity I've seen on the feature so far, and may not reflect the project consensus (although it does largely reflect the "needs review/ready for committer" split on those patches).

I've also probably missed stuff. Mostly this is by not seeing some patches as significant, which actually are.

As you can see, what's already committed would make a respectable release for any other DBMS.  But here at PostgreSQL, we don't believe that there's any such thing as "enough features".   If your database can't process 16TB of data with six nines of uptime, parallelize all your application logic, make breakfast and drive the kids to school without waking you up, we still have work to do!

Already Committed:

• Use POSIX shared memory: no more SHMMAX
• LATERAL subqueries
• Streaming-only Cascading and Remastering
• extend large object access to 4TB (was 2GB)
• Improve pg_upgrade performance
• COPY FREEZE mode
• Automatically updatable views
• Create your own background workers
• configuration file directories
• additional ALTER, IF NOT EXISTS objects
• pg_basebackup generates replica configuration
• pg_terminate_backend your own queries
• SP-GiST indexing for Range Types

Pending CF4, Looking Likely:

• Writeable Foreign Tables
• PostgreSQL database federation (pgsql_fdw)
• LOCK TIMEOUT
• Event Triggers
• pg_retainxlog
• Improved Foreign Key locking (FOR KEY SHARE)
• Recursive VIEWs
• Improved performance for SP-GiST and Range Types
• Merging recovery.conf into postgresql.conf
• pg_ping utility
• Additional JSON features and built-in functions
• Updating postgresql.conf from SQL command-line

In CF4, Uncertain:

• Logical Streaming Replication, and its many dependant patches.
• Data page checksums
• Auto-updated Materialized Views
• Reducing Hint Bit I/O
• Storing counts, other data in GIN indexes
• Row-Level Security and dependant SEPgSQL features.
• xlogreader
• use pg_receivexlog for promotion
• REINDEX CONCURRENTLY
• Parallel pg_dump

Already Bounced to 9.4:

• pg_dump file-per-object format ("split")
• Trim Trailing NULLs optimization
• Automatic audit logging ("timing events")
• TABLESAMPLE
• pg_stats_lwlocks view
• array foreign keys

Never Got Worked On in 9.3:

• Query parallelization
• UPSERT/MERGE
• Merge joins for Ranges
• Global temp tables
• Temp tables on Replicas
• Reordering database columns
• Spatial JOIN
• special JSON GiST Indexes

If your favorite feature is in the "never got worked on" or the "already bounced", it's a bit too late to do anything about it. However, if your favorite feature is in CF4 -- especially if it's on the "uncertain" list -- now's the time to get involved.  Download the patch, test the functionality, test performance, find bugs, submit a review, help writing docs (since new features don't get accepted without docs).  After February 15th, it'll be too late, and you'll have to wait for another year.

Regardless of what does or doesn't make it in at this point, we have a great 9.3 expected.