Monday, April 30, 2012

Call for Lightning Talks for pgCon

Hackers, users, pgCon attendees:

You want to give a lightning talk at pgCon!

Yes, you do.  The fun, the glory, the laughter, the everlasting fame!  These can all be yours.

Be one of the ten "brave and true" who put together five minutes about PostgreSQL tools, experiences, forks, ideas, websites, or even (especially) jokes.    Anything from "hacking wal files" to "the PostgreSQL drinking game" is an acceptable topic for the lighting talks.   A short schedule:
  • Right Now: send your lightning talk idea to  I'll need a title, speaker full name, speaker cell phone number, and brief (one sentence) description.
  • This Friday: I'll get back to you with acceptance (or not)
  • Friday, May 11th (or sooner) you get me your PDF slides for the talk.
  • Thursday, May 17, 5:15 PM: meet Magnus Hagander in the plenary presentation room for your order of speaking.
  • Thursday, May 17, 5:30PM to 6:30PM: you and 9 others deliver your talks
Fine print: Lightning talks are strictly five (5) minutes in length, and speakers who run over will be cut off.  PDF slides or browser access only, which will be presented on the conference laptop, so no demos, animations, private network access, or installed software.  Lightning talks are subject to pgCon's anti-harassment policy.

Friday, April 27, 2012

Sharding Postgres with Instagram

On Tuesday last week we had a terrific SFPUG meeting at which Mike Kreiger of Instagram explained how they grew and eventually sharded their 2TB of Postgres data to support 27 million users.  It's a great presentation which explains the growth process of a successful web/mobile startup, as well as horizontally scaling PostgreSQL.

Yes, you too can use PostgreSQL to make One Billion Dollars!

Video is on UStream.   Sorry you can't see the slides on the video; we had technical issues with the camera.  Slides are here, you can click along with Mike talking on your own.

Tuesday, April 24, 2012

Red Hat Kernel cache clearing issue

Recently, mega-real-estate sales site Tigerlead called us with a very strange problem.  One of their dedicated PostgreSQL servers refused to use most of its available RAM, forcing the system to read from disk.  Given that the database was 60GB in size and the server had 96GB of RAM, this was a painful performance degradation.

Output of free -m:

             total       used       free     shared    buffers     cached
Mem:         96741      50318      46422          0         21      44160
-/+ buffers/cache:       6136      90605
Swap:        90111          3      90107
As you can see here, the system is only using half the free memory for cache, and leaving the other half free.  This would be normal behavior if only half the cache were needed, but IOstat also showed  numerous and frequent reads from disk, resulting in IOwaits for user queries.  Still, there could be other explanations for that.

So, I tried forcing a cache fill by doing a pgdump.  This caused the cache to mostly fill free memory -- but then Linux aggressively cleared the cache, again getting it down to around 40GB of cache within a few minutes.  This seemed to be the case no matter what we did, including tinkering with the vm parameters, increasing the size of the swap file, and changing shared_buffers.  This was highly peculiar; it was as if Linux was convinced that we had half as much RAM as we did.

What fixed the problem was changing the kernel version.  It turns out that kernel
2.6.32-71.29.1.el6.x86_64, released by Red Hat during a routine update, has some kind of cache management issue which can't be fixed in user space.  Fortunately, they now have a later kernel version out as an update.


[root ~]# free -g
             total       used       free     shared    buffers     cached
Mem:            94         24         70          0          0         19
[root ~]# uname -a
Linux 2.6.32-71.29.1.el6.x86_64 #1 SMP Mon
Jun 27 19:49:27 BST 2011 x86_64 x86_64 x86_64 GNU/Linux


[root ~]# free -g
             total       used       free     shared    buffers     cached
Mem:            94         87          6          0          0         83
[root ~]# uname -a
Linux 2.6.32-220.4.2.el6.x86_64 #1 SMP Tue
Feb 14 04:00:16 GMT 2012 x86_64 x86_64 x86_64 GNU/Linux

That's more like it!   Thanks to Andrew Kerr of Tigerlead for helping figure this issue out.

I don't know if other Linux distributors released the same kernel with any routine update.  I haven't seen this behavior (yet) with Ubuntu, Debian, or SuSE.  If you see it, please report it in the comments, or better to the appropriate mailing list.

Monday, April 23, 2012

GSOC Begins

We have accepted the projects for PostgreSQL's participation in the 2012 Google Summer of Code.  They are:
  • JDBC Foreign Data Wrapper, by Atri, mentored by Merlin Moncure
  • Document Collection Foreign Data Wrapper, by Zheng Yang (a returning student), mentored by Satoshi Nagayasu
  • Implementing TABLESAMPLE, by Qi, mentored by Stephen Frost
  • Better Indexing for Ranges, by Alexander, mentored by Heikki Linnakangas
  • xReader Streaming xLog Reader, by Aakash, mentored by Kevin Grittner
So a bunch of exciting cool stuff.  We're looking forward to great things from these students, not the least of which is becoming ongoing contributors.  It's two fewer projects than last time, due mostly to having fewer mentors available.

When you see these students on the PostgreSQL mailing lists, please be friendly and help them out.

Saturday, April 14, 2012

Of Booze and Brogrammers

There's been a little bit of noise about the culture of "hard partying" at programmer conferences.   Ryan complains about binge drinking, and Kevin complains about parties so loud you can't talk.  While I think both of these things are undesirable, I don't see that either is on any dramatic increase overall, actually, and certainly not more than it's on the increase in American society in general (you wanna see real binge drinking?  Watch Mad Men or the Food Network).  However, I do agree that there is too much emphasis on high-decibel boozing at many current tech conferences, and that it should be changed.

Since I'm often a conference organizer, I wanted to approach this issue from a conference organizer's perspective.  Prepare yourself for a long, rambling post about drunken parties, brogrammers, conference organizing, teetotallers, pgCon, SCALE, and middle age.


Why Limit Drunken Partying at Conferences?

I drink.  I like beer and wine, a lot.  I enjoy going to parties, and have been known to attend parties at tech conferences where I got more than a little buzzed.  More than once I missed Friday morning sessions at OSCON entirely.  The fact that I don't do this anymore is almost entirely due to changes in my life: I'm now 41, married, and CEO of a company.

So, Ryan's feelings aside, what's wrong with loud, drunken parties at conferences?  Ryan doesn't have to attend them if he doesn't want to.

Well, first, Ryan isn't alone.  The career of programming in general appeals to people who don't like parties, and I think if you did a survey of any large, general developer conference you'd find that 25% to 50% of the attendees either didn't drink, didn't like parties, or both.  So if boozy parties are the only form of evening entertainment at your conference, you're deliberately excluding a quarter to half of your attendees.  You're encouraging them to go home in the evening, and if they do, they're liable not to come back to the conference the next day.

Speaking of the next day, large numbers of hung-over conferencegoers make for very poor attendance at morning sessions at the conference, which makes scheduling hell.  Do you schedule unpopular talks first thing Sunday morning and basically write them off?  Or do you schedule popular talks in hopes that it'll get people out of bed, and risk offending an important speaker?

Secondly, the parties have taken the place of the BOFs we used to have.  Many conferences used to have great Birds-Of-a-Feather informal sessions in the evenings.  These offered a great opportunity for projects and vendors who couldn't get on the regular session schedule to reach their users, and for group discussions as a follow-on for the sessions during the day.  However, I've generally given up on trying to hold PostgreSQL BOFs at most conferences now, since I'm always head-to-head against some party with free food and beer.  The last time I had a BOF at OSCON, for example, eight people attended.

Finally, there's the liability.  Whether or not your conference is officially sponsoring or hosting the party, if you put it on your schedule and announce it, you are liable in the eyes of the court for bad things which happen there: property damage, alcohol poisoning, accidental injury, and sexual assault.  Frankly, I'm surprised that there hasn't been an alcohol-related death or felony arrest at a major tech conference yet; it's only a matter of time.


Why Have Boozy Bashes?

Of course, there's quite a few reasons why we have loud parties at conferences.  Among them:
  • Unmarried 25-Year-Olds are a large minority of the conference population, and do a lot of partying.  It's part of being 25 years old.
  • The Brogrammers are desperately trying to prove to themselves that, while they may be programmers, they're not geeks.  Drunken parties are part of this self-deception.
  • People Want To Blow Off Steam after having their brains crammed full all day.  Many/most don't want to extend 9 hours of hacking into 14.
  • Some Vendors/Sponsors Prefer Parties as their way of reaching your attendees, and are willing to pay for it. 
  • It's Easier For Overworked Conference Organizers to arrange a party than other evening activities which actually require planning.
For the first four reasons, hard partying at conferences is not going away.  If there isn't something officially scheduled with the conference, someone will create something unofficial.  So you should make the assumption that any large conference will have at least one or two parties with booze and music.  With moderation, this does not have to be a bad thing.

Where this becomes a problem, though, is when the high-octane parties are the only things to do in the evening, or when they are emphasized as the main point of going to the conference.  This is where the last reason above is important.

It's certainly easier, as a conference organizer, to get a vendor to sponsor a bash near the conference center than it is to plan other activities.  All you have to do is connect the vendor's PR person with a nearby restaurant or hotel, and your work is done.  Sometimes you don't even have to do that much.  However, that's really not good enough; if you're too overworked to plan the conference activities after 5:30pm, then recruit a new volunteer who likes planning social things.   You'll be pleasantly surprised to discover that your community does, indeed, have such people, and that they're thrilled to be able to contribute something meaningful.


Some Alternatives to Loud Parties

Not all conferences are selling Party-Only tickets like JSConf is.  In fact, some conferences have done a very good job of providing interesting alternatives to loud, hard-drinking parties for evening activities.  Let me provide a few as examples you can emulate:


Bring Back The BOFs

Unlike the O'Reilly conferences, Usenix LISA has kept their BOFs, going all evening each evening of the conference.  Rather than help vendors sponsor parties, they encourage vendors to sponsor BOFs with refreshments.  As a result, the BOFs at LISA are very well attended; I'd estimate that 2/3 of the conference attendees stay all evening for the BOFs.

Due to my lack of academic credentials, I usually can't get a PostgreSQL talk into the main program at LISA, but this doesn't bother me because the BOFs are so awesome.  Last year there were more than 60 people in the PostgreSQL BOF, and I was able to present all about 9.1.


Parties Don't Need to Be All About Drinking

LinuxCon 2011 decided to have a big "20th Anniversary of Linux" party.  So they rented a dance hall and had a 1920's-themed party with a swing band.  Further, they encouraged attendees to bring period party clothes, and had a costumer on hand for those who didn't own any.  The party had ersatz gambling, a photo booth, a raffle, and dance lessons.  It was terrific, and I don't say that just because I won the raffle.   For the first two hours, the music was low enough you could easily talk.

More importantly, the party was fun even if you didn't drink.  Compare that with the common conference party, which has a bad DJ in a big barren hotel ballroom with all the free booze you can manage.   At parties like that there's nothing to do but drink; it's too loud to talk and there's nothing else to do.  Even as a drinker, that fits my definition of a sucky party and I'll go elsewhere.


Shall We Play A Game?

The most incredibly awesome party alternative of the year was provided by SCALE10x, who booked a Saturday night "games night" after the Weakest Geek and the end of the BOFs.  This included head-to-head hackable FPS games (for the hackers), vintage arcade games, board games and RPGs, including a brand-new board game there for beta testing, Nerf weapons, and a Lazertag arena.  Local attendees brought their whole families, including kids to the event.  It was the best time I've had after dark at a conference in years.

Sure, there was alcohol at the event.   But since it was a no-host bar and there were so many distractions, nobody drank very much.  I had one beer; any more would have gotten me creamed by the 12-year-olds in the Lazertag arena.  And even the 25-year-olds and the brogrammers had a good time.

SCALE has, if anything, led the way in party alternatives.  Friday night they have Ignite talks, and Saturday they've always had The Weakest Geek, a pseudo-game show on stage.   While their Games Night may be a bit elaborate for smaller conferences, you can reasonably plan fun entertainment which doesn't require getting soused.


Learning Moderation

So this is my call to conference organizers: the professional world of programming is not a frat house.  We can provide evening activities at conferences which are either an alternative to alcohol and loud music, or which are still fun even if you don't drink.  Yes, it's a bit more work, but it's worth it.

I'm not saying that we shouldn't have loud parties.  Parties are fun, and in demand by a large portion of your attendees.  Just that we shouldn't only have boozy bashes to the exclusion of all other evening activities.  It's called "moderation", and a good thing to practice in all portions of life.

And then we'll all have more fun.

Friday, April 13, 2012

An open "Dear United" letter

Dear United:

I'm afraid that we'll no longer be seeing each other regularly.   I've been seeing some new airlines, and they've made me realize what an abusive partner you've been.  You've taken me for granted, cheated and abused me, and I'm not putting up with it anymore.

I fly a lot for business.  As a result, I've had frequent flyer status on United/Star Alliance for years ... generally either Gold or Silver, depending on how many international trips I make.  The benefits and perks you gave me kept me coming back to you, even as you fell further and further behind technologically, and have been treating your customers with steadily declining courtesy.

This year, you stripped away all of my useful benefits as a Silver member.  I no longer get Economy Plus seats, free checked baggage, free upgrades, or really anything else which makes spending time with you less unpleasant.  I'm in the back of the plane with the proles, and you've made the back of the plane into a pretty nasty place. 

The new airlines I've been seeing (VirginAmerica, JetBlue, and Southwest), are all younger, hipper,  better looking, and have much better senses of humor than you.  More importantly, they appreciate me and treat me like a valued person.  Jetblue and Virgin have more leg room in coach, onboard entertainment systems, wireless internet, and decent quality food and booze on sale.  Southwest is plainer, but makes up for it by being "fast" (if you know what I mean), and checking my luggage for free.  All three of my new airlines' web sites are an entire generation of technology ahead of yours.

We've both known this was coming since you acquired Continental.  We knew that being bigger and fatter wasn't going to make you more sensitive to my needs.  You've treated me shabbily, and it's time to end it.

A former Mileage Plus member

P.S.: I have a suggestion for a new United motto:

"We know you don't have a choice when you fly.  That's why you're on United."

Thursday, April 12, 2012

FOSS4G-NA 2012

I just spent the week in DC speaking at FOSS4G-NA, or as I like to call it, "Fossforgna".  I came out at the invitation of the conference committee, and did a talk on "PostgreSQL 9.2 and Beyond" and a keynote on "Firehose Engineering", slides for both of which are located on our presentations page.  My "BOF" on Postgres peformance was less successful, because the attendees brought really hard problems.  Here's a heads-up: PostGIS desperately needs parallel query support.

The big news, of course, is the release of PostGIS 2.0.  I also saw excellent talks by: CartoDB (Maps-as-a-service, running on PostGIS); Metropolitan Airports Commission (flight tracking using PostGIS and PL/R); the British Anarctic Survey (tracking pack ice with PostGIS); NOAA's mapping system for international emergencies (especially poignant on Tuesday).  There were multiple talks about using opengeo stacks with PostGIS for various forms of ecological and biological tracking, including ocean floor diversity, deforestation, and endangered species tracking.  Just overall, really cool stuff which makes me proud to have done a very small part in helping enable it.

The hallway track was also very busy and I met a lot of people I never knew were using PostGIS before.

There were somewhere between 300 and 400 people at FOSS4GNA, and with only a couple of exceptions, every single one was using PostGIS.  The conference really brought home to me that PostGIS users probably now make up the majority of the PostgreSQL user base, and us folks in "mainstream" PostgreSQL ought to be paying more attention to that.  While PostgreSQL is a highly competitive relational database, PostGIS is the unquestioned first choice for GIS applications.

In any case, a terrific conference for open source geospatial application developers of all kinds, and a great opportunity to make business contacts in the opengeo world.  I recommend attending the next Fossforgna next year.

Wednesday, April 4, 2012

Reminder: GSOC applications close tommorrow

Just a quick reminder that you have less than 40 hours to apply for Google Summer of Code.   If you're a student, time to get moving!

Monday, April 2, 2012

Baby you can drive my CAR

Photo courtesy K Lars Lohn.  Used with permission.

 Yesterday my colleague Robert Hodges made a post disputing the CAP theorem. While I have some issues with his logic -- and after all, it was a beery post put up on April 1 -- his post does bring up an issue I've meant to blog about for a while, so here goes.

CAP is an exclusionary triad: it consists of three elements, any pair of which excludes the other.   Another exclusionary triad all my readers will be familiar with is PQT: Price, Quality, Time.  Thing is, CAP is only one of several possible triads in distributed system design, and is not the most interesting triad for clustered database designers.

Most of the time you're ignoring Partition Tolerance for a clustered database design since it's limited to a single data center and partial network failure is not considered that great of a risk.  This isn't universally true; for example, CAP rears its ugly headgear whenever you contemplate having automated failover for a failover/pooling proxy.  However, there are more substantial tradeoffs to be made even in a single-network system.

There are several other possible combinations which fit the exclusionary triad model and relate more closely to database design.  For example, CRT (Complexity, Response Time, Throughput) is a triad for read requests in a distributed database.  I may write more about CRT later.

Today, though, I'm going to write about one of the triads which govern writes to a clustered database system.  CAR: Consistency, Availability, or Response Time.   You can choose any two of the three but not all three.  To explain:

Consistency refers to having writes succeed or fail, and produce the same results, regardless of to which node you connect.  Ideal Consistency would be the consistency of a single node.

Availability is the clustered database's ability to survive the failure of one or more nodes while still servicing client requests.  Ideal Availability would support any number of failed nodes as long as one node was still running.

Response Time is the speed of service of a single write request.  Ideal Response Time is the response time of a single node.

While real systems make some compromises on all elements of the triad, we can show this as a valid triad by defining the three extreme cases:

CR: a traditional, single-node transactional database is Consistent and has good Response Times. Obviously, it cannot survive node failure.

AR: a non-transactional multi-node key-value database with fully replicated data on each node, which accepts writes to any node and asynchronously attempts to copy the writes to other nodes would have perfect Availability and single-node Response Times, while completely abandoning consistency.

CA: on the other hand, consider a multi-node database in which each node has duplicate copies of the data and all transactions are written to all nodes using 2-phase commit (2PC).  This database would have perfect Consistency and maximum Availability, at the cost of Response Times which get progressively slower as the cluster grows.

If you look at today's open source clustered databases, you'll find that they have all chosen one side or another of the above triangles, with some compromising in one direction or another.  For example:

CR: mainstream PostgreSQL and MySQL.

AR: MongoDB, CouchDB, Bucardo.

CA: Continuent, VoltDB, Synchronous Replication, PostgresXC.

Obviously there are more design differences between the databases above than where they fall on the CAR triangle.  But it's a good way to look at the tradeoffs you need to make.  Probably more than CAP.