Freezing Your Tuples Off, Part 3

First, read Part 1 and Part 2 of this series.

 

vacuum_freeze_table_age

To understand the purpose of this parameter, you have to understand how vacuum has worked since version 8.4.  Vacuum no longer usually scans the entire table; instead, it has a bitmap (called the visibility_map) which tells it which data pages in the table have reclaimable space, so that it scans only the portion of the table which is "dirty".  This is a considerable optimization for large tables, where only 20% of the rows might have been updated or deleted since the last vacuum cycle.

This does, however, introduce a problem. Since old, untouched data pages aren't being vacuumed (since they don't have any reclaimable rows anymore), they are not getting frozen regardless of the setting of vacuum_freeze_min_age.  This makes it much more likely you'll hit an eventual wraparound vac when you least expect it.

The idea of vacuum_freeze_table_age is to compliment autovacuum_max_freeze_age by telling the database: "If it's almost vacuum_freeze time for this table,  and you were vacuuming the table anyway, then scan the whole table and to freeze out tuples."  In theory, this should allow you to hold off wraparound vacuum by running a regular, cancellable vacuum instead.  In practice, though, that's a dubious benefit for large tables, since either way you get a full-table scan and a vacuum which runs for a long time.  Small, heavily updated tables will tend to get most of their pages vacuumed most of the time anyway, and thus be unaffected by this setting.

As such, there's no point in setting it to a particularly low or creative level.  Simply set it to 80% of whatever autovacuum_max_freeze_age is set to (recommended: 800 million).

 

Flexible Freeze

By now it should have occurred to you that Postgres has a serious problem with vacuum freeze behavior on large tables.  It's the unfortunate bad side-effect of optimizations in several other areas.  You can tweak the settings as I recommended, but you'll still be in danger of having a wraparound vacuum kick in and saturate your IO at some unexpected time.  So what can you do to behave better and proactively freeze tables before they get to that point?

Well, you have one piece of information PostgreSQL doesn't.  You know when the slow periods in your application usage are.  Try following this "flexible freeze" program in a cron script:
During expected "slow periods", run the query from Part I to get a list of tables, and then:

1. Set PostgreSQL to "soft vacuum" with a high vacuum_cost_delay (such as 50ms).
2. Set PostgreSQL to aggressively freeze tuples, with vacuum_freeze_table_age at 50% of autovacuum_freeze_max_age, and vacuum_freeze_min_age set to 10% of its usual value.
3. Until the slow period is almost over, loop through the list, vacuuming each table.
4. This will help you avoid wraparound vacuum when you least expect it.

Or, you can use our Flexible Freeze python script.

 

Further Work

Obviously the suggestions in this article are workarounds.  Where PostgreSQL really needs to go is to find some way to avoid needing to vacuum old, cold data ever.  The obstacle to doing this is that nobody has figured out how.

One of the first methods suggested was to have an 8-byte XID, which would postpone wraparound by a billion cycles.  However, since there are two XIDs on every row header in the database, this would dramatically increase database sizes for many users, especially the users who need more XIDs in the first place.  It would also increase memory requirements for a lot of Postgres operations.  A second method I suggested during the 9.2 cycle was to have an "XID cycle" counter in the page header of each data page.  This would have the advantage of not increasing the size of rows.  However, it would have the drawback of failing in cases where there were rows from more than one XID cycle in the page.

Further, both of these methods hit a major performance problem: the CLOG.  PostgreSQL's CLOG tracks which XIDs committed and which rolledback, and thus which rows are valid and which are orphaned.  Currently, checking the CLOG is quite rapid because it's kept small.  Any patch which involves increasing the size of XIDs or keeping more of them will cause the CLOG to bloat by multiples, substantially affecting response times on transaction-processing workloads.  That's unacceptable.  We've discussed radically restructuring the CLOG, but that's a major enough change that it would need to come with other benefits than just avoiding freeze.

An alternate approach to never freezing old data at all would be to develop some kind of daemon which did background freezes of chunks of large tables.  This daemon would need to be able to tell when the system was "busy" and stop work.  It would also need to be able to track what old data pages it had visited and which it hadn't, and which ones need to be revisited because they've been dirtied.

Anyone wanting to take up the above -- either by coding or by funding -- let me know!  Until then, happy freezing!

Random Page Cost Revisited

Because the planner has limited information about the host system PostgreSQL is running on, we offer several planner configuration variables, which can be set either dynamically or in the postgresql.conf file.  One of these, which has been a source of much contention in the past, is Random Page Cost (or random_page_cost, hereafter RPC).  For the last couple of years RPC was not a variable you needed to worry about, but now, thanks to advances in storage technology, it's become one of the essential parameters again.

Random Page Cost is meant to represent the coefficient between the cost of looking up one row (out of many) via sequential reads, vs. the cost of looking up a single row individually using random access (disk seeks).  This factor strongly influences the planner's decisions to use indexes vs. table scan, and composite vs. simple indexes, in executing queries.

For many generations of spinning hard drives, this ratio remained about the same, around 4.0. 

Things have changed recently though.  In fact, there are five now-common types of storage where you want to change RPC in order to match a different scan/seek ratio for different hardware.

1. High-End NAS/SAN: when working well, your large storage box combines a large volatile cache with many many disks, improving the number of concurrent seeks you can do.  However, if storage is not direct-attach, you usually have limited throughput.  This means that a random_page_cost of 2.5 or 3.0 is more appropriate.

2. Amazon EBS and Heroku:  EBS, especially when "RAIDed", is an extreme example of the NAS/SAN case.  Seeks are fast with virtually unlimited concurrency, while storage bandwitdh is erratic and slow, limiting full table scans.  As such, we often use a random_page_cost of 1.1 to 2.0.

3. Bad SANs: the advent of iSCSI seems to have inspired a large variety of very poor-performing inexpensive storage units, such as the lower-end Dell Equallogics and various SATA-based Promise units.  These boxes are cheaper for a reason, folks!  Anyway, their quirks often give these boxes unique performance characteristics which strongly affect database access and prompt you to modify planner parameters.  For example, we discovered through testing that one Promise box had decent throughput, but seek rates slower than a single drive due to a combination of architecture and driver issues.  As such, we set RPC to 6.0.  You'll have to trust the results of bonnie++ and fio in order to figure out where you should set things, though; each inferior unit is inferior in a different way.

4. SSDs: in general, SSDs have not lived up to their early hype for performance; random writes and sequential scans on most SSDs are only incrementally better than on HDDs.  One area where SSDs shine, though, is random page access; a good array of SSDs can make even large index scans faster than sequential scans.  Therefore, use a random_page_cost of 1.5 to 2.5.

5. NvRAM (or NAND): durable memory, such as FusionIO and Virident drives, do deliver on most of the performance promises originally made about SSDs.  Everything is faster, sometimes orders of magnitude faster, than HDDs, most of all random access.  While the concept of NvRAM might make you inclined to set RPC to 1, forcing almost 100% index access, that's going a little far because PostgreSQL still has a disk page structure.  Therefore, I suggest 1.0 to 1.5.

Note that two other variables, effective_cache_size and effective_io_concurrency also affect seek/scan behavior.  Possibly I will blog about them at a different time.  Further, I have only been able to test with relatively few models and configurations of the above types of storage, so test, monitor and check your results before committing to a long-term configuration change.