pg_waldump does, and instead lets you move a cursor through the records one at a time in your terminal, which is the whole point of it. It is written in Rust.
pg_waldump can filter in most of the ways you would want. -x for a transaction ID, -r for a resource manager, -R and -B for a relation and a block. But once you filter, you lose sight of what was going on around the records you kept. I kept running it with -x to find the transaction I cared about, then running it again without -x because I wanted to see what else was going on around those records, and after doing that a few too many times I thought it would be nicer to load the whole segment up front and change the view in place instead.
The screen is split into three panes. Tab moves the focus between them.
A list of the records. LSN, XID, record length, whether the record carries a full-page image, resource manager, a description.
The graph line down the left edge is the part I wanted most. The records sharing the transaction ID of the record under the cursor are connected by it.
0/100 13840 171 Heap UPDATE
┏━ 0/100 13841 8135 Heap LOCK <- first record of this XID
┃ 0/100 13839 8135 Heap LOCK
┣━ 0/100 13841 171 Heap UPDATE <- cursor
┃ 0/100 13840 8135 Heap LOCK
┃ 0/100 13839 171 Heap UPDATE
┣━ 0/100 13841 64 Btree INSERT_LEAF
┗━ 0/100 13841 34 Transact COMMIT <- last record of this XID
The records of other transactions stay where they are, so you can follow one transaction and still see everything that happened in between. s and r jump to the next and previous record with the same XID1.
The colour of the line comes from the last record of that XID. That is, from how the transaction ended.
| State | Condition |
|---|---|
| Committed | ends with COMMIT or COMMIT_PREPARED |
| Aborted | ends with ABORT or ABORT_PREPARED |
| Pending | ends with a Transaction record that is neither a commit nor an abort, such as PREPARE |
| Unfinished | does not end with a Transaction record at all |
That last one turns out to be quietly useful. It means the transaction does not finish inside this WAL file, the rest of it is in the next segment, and so you know whether you need to open another file before you go looking for a commit that is not there.
Only the 16 ANSI colours are used, by the way. What they look like is up to your terminal theme. NO_COLOR turns them off.
The details of the selected record. The XLogRecord header, the block references (RelFileLocator, fork, block number, flags), and the fields decoded per resource manager.
It is an accordion, and Enter expands or collapses a block. For a Heap UPDATE it expands t_infomask and t_infomask2 of xl_heap_header into flag names, so checking whether something like HEAP_XMAX_INVALID is set no longer means leaving the tool and looking the bits up. Handy.
The bytes of the whole WAL segment. Each line is addressed twice, by its LSN and by its offset in the file, which means you can read a record and see which of the 8kB pages it happens to be sitting on without doing the arithmetic yourself.
Only the bytes of the selected record get colour. And they are coloured by what they are: the XLogRecord header, the descriptors, a full-page image, block data, main data. A record crossing a page boundary is coloured as the two or more pieces it really occupies. The page header that splits it is not part of the record, so it stays uncoloured.
Whatever the DETAILS cursor sits on is picked out in the dump too, so the accordion doubles as a navigator over the bytes. Having the decoded fields and the raw bytes next to each other also helped while writing the *desc.c equivalents2.
If pg_config is in your PATH, the build needs nothing else. The definitions of XLogRecord and friends come from the server headers at build time, so those headers have to be there3.
git clone https://github.com/MasahikoSawada/pg_walview.git
cd pg_walview
cargo build --release
# If PostgreSQL is installed somewhere of your own
PG_INCLUDE_DIR=/path/to/pgsql/include/server cargo build --release
Then pass the path to a WAL file. It does not connect to a server, so the instance does not have to be running, and a file you copied off a machine somewhere is enough.
pg_walview /path/to/pg_wal/000000010000000000000001
The main keybindings:
| Key | Action |
|---|---|
j / k |
Next / previous record |
g / G |
First / last record |
s / r |
Next / previous record with the same XID |
Space / - |
Page down / page up |
Tab |
Switch pane |
q |
Quit |
--wal-segsize. The segment size itself is read from the long page header, so a non-default one is read correctly.pg_waldump and pg_walinspect are both there for looking inside WAL, and they have more features. pg_walview does one thing. It lets you move around.
In my case I spend a fair amount of time on logical replication debugging. Where did this transaction commit? What came between these two records? For that kind of question it has been worth writing.
The source is on GitHub, under the MIT License. Contributions are welcome!
The jump is disabled when the XID is 0, 1 or 2. Those are InvalidTransactionId, BootstrapTransactionId and FrozenTransactionId, and since a lot of records carry no XID at all, connecting them would not mean anything. ↩
The per-resource-manager decoding does the same job as the *desc.c files under src/backend/access/rmgrdesc/ in PostgreSQL itself. I kept the structure close to that, on the assumption that it should be easier to follow upstream changes later. ↩
Which also means a binary only reads WAL from the version it was built against. Opening a segment from another version reports a page magic mismatch rather than misreading it. --version prints which PostgreSQL a given binary was built for. ↩
When you compare them, the difference is obvious:
=# select uuidv4(), uuidv7() from generate_series(1, 5);
uuidv4 | uuidv7
--------------------------------------+--------------------------------------
216aae4c-6b02-4bea-bd84-7fea9617e0cc | 01991632-e3a0-7468-ac42-26afbc51df65
55372a64-0351-40a1-a32c-318e581f4561 | 01991632-e3a0-748a-bb4c-9882ddaf0721
e154c7a7-4a6b-4446-96be-6a1cb84b773e | 01991632-e3a0-749f-b429-d2ef56b9683e
59adba82-8a88-4f4f-b859-036430d09045 | 01991632-e3a0-74b5-9a86-8107f1851200
be096cb4-ee48-4bf8-ae30-3ca6d09af786 | 01991632-e3a0-74c9-a08a-1ec588e5a60d
(5 rows)
UUIDv7 offers many advantages when used as a primary key in databases. PostgreSQL uses Btree indexes for primary keys. Therefore, even when loading large amounts of data, the UUID values being inserted are always in ascending order, which provides high locality for index updates and leads to better performance. Additionally, in PostgreSQL, it helps reduce Full Page Writes (FPW).
Here are the results of inserting 5 million rows using SERIAL type (sequence) and UUID type (UUIDv4 and UUIDv7) as primary keys (verified on PostgreSQL 18 Beta):
| SERIAL | UUIDv4 | UUIDv7 | |
|---|---|---|---|
| Druation | 8.452 s | 42.24 s | 16.922 s |
PostgreSQL has a native uuid data type, and (as of PostgreSQL 17) there are two main ways to generate UUIDs:
The first method is to use the built-in gen_random_uuid() SQL function. This generates a version 4 UUID (more details about UUID versions will be discussed later). It’s PostgreSQL’s own implementation.
The second method is to use the uuid-ossp contrib module. This performs UUID generation using an external library, which varies depending on the platform. On Linux and macOS, it uses libuuid. This method can generate versions 1 through 5.
As mentioned above, PostgreSQL 17 currently doesn’t support UUIDv7 generation, so if you want to use UUIDv7, you need to either use a published extension or implement it yourself. A search on GitHub reveals the following extensions:
There’s also a blog post about creating a UUIDv7 generation function using pg_tle + PL/Rust:
Since PL/Rust has limitations on crates can be used, the Rust uuid crate is not available. If you want to create a UUIDv7 generation function using a custom extension, using pgrx1 is probably the easiest way. For just generating UUIDv7, you only need the following code:
use pgrx::prelude::*;
use uuid::Uuid;
::pgrx::pg_module_magic!(name, version);
#[pg_extern]
fn pgrx_uuidv7() -> pgrx::Uuid {
let uuid = Uuid::now_v7();
pgrx::Uuid::from_bytes(uuid.into_bytes())
}
Great news is that upcomoing PostgreSQL 18 will introduce the uuidv7() SQL function, making UUIDv7 available to all PostgreSQL users (commit log)!
For backward compatibility, gen_random_uuid() will continue to exist as a function that generates UUIDv4. Along with uuidv7(), uuidv4() has also been added, but it’s just an alias for gen_random_uuid().
As per RFC, the data format of UUIDv7 is:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| unix_ts_ms |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| unix_ts_ms | ver | rand_a |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|var| rand_b |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| rand_b |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Every UUID data contains the number representing the version of UUID:
Version
|
v
01937a3a-3d34-74d0-a1b7-e1f1b53064d8
In UUIDv7, the rough format is: millisecond-precision timestamp followed by the version number, and then random data (plus variant, to be precise) after the version.
timestamp random data(+var)
|-----------| |-------------------|
01937a3a-3d34-74d0-a1b7-e1f1b53064d8
For use cases where millisecond precision isn’t sufficient, the RFC allows implementations to use the rand_a (12 bits) portion (and optionally rand_b) as additional data to maintain monotonicity of generated values. The RFC introduces several methods for how this can be done.
Different UUIDv7 generation function implementations use different approaches, but how rand_a and rand_b are used is crucial for maintaining data monotonicity in environments with high-frequency UUID generation. For example, in the simplest format of “millisecond timestamp + all random data,” if more than 1000 UUIDs are generated per second, all leading timestamps will have the same value, so monotonicity of the generated UUIDv7 data isn’t guaranteed. This means that in systems that might generate more than 1000 UUIDs per second, using such a UUIDv7 generation function won’t fully leverage UUIDv7’s advantages. It’s important to choose the right UUIDv7 generation function based on your use case2.
PostgreSQL’s UUIDv7 implementation adopts Method 3 (Replace Leftmost Random Bits with Increased Clock Precision) from the RFC. Specifically, it uses the rand_a portion for sub-millisecond timestamps, using 60 bits (=48+12) total for the timestamp. This allows it to handle about 4 million UUID generations per second. Furthermore, within the same process, it’s guaranteed that the rand_a portion is increased by a certain step for each UUID generation, so UUIDv7 data generated from a single process is guaranteed to be monotonically increasing even at higher frequencies.
Additionally, it’s possible to specify an interval value as an argument, allowing you to offset the timestamp stored in the UUID data by a specified period.
The source code can be found here.
A framework for creating PostgreSQL extensions in Rust ↩
For example, looking at pg_uuidv7’s implementation, you can see it uses “millisecond timestamp + random data” ↩
A comparison with glibc can be found here.
This post is a note on how to build PostgreSQL using musl libc.
The source code of musl libc can be downloaded here. It seems that there is also a place called musl.cc where you can download pre-built binaries, but this time I built it from source code:
$ wget https://musl.libc.org/releases/musl-1.2.5.tar.gz
$ tar zxf musl-1.2.5.tar.gz
$ cd musl-1.2.5
$ ./configure --prefix=/home/masahiko/musl --syslibdir=/home/masahiko/musl-lib/
$ make
$ make install
By specifying --prefix and --syslibdir options, you can specify the directories where musl libc and the dynamic linker will be installed, respectively. The build completed in about 20 seconds in my environment.
You can find a program called musl-gcc in the bin directory of the installation destination:
$ ls /home/masahiko/musl/bin
musl-gcc
musl-gcc is a wrapper program (it’s actually a shell script), and it seems that when you compile programs using this, they will be linked against musl libc.
As preparation, include the musl-gcc installed above in the PATH:
$ export PATH=/home/masahiko/musl/bin:$PATH
Then, copy the directories /usr/include/linux, /usr/include/asm, and /usr/include/asm-generic to the location where musl libc was installed:
$ cd /home/masahiko/musl/include
$ cp -rs /usr/include/linux linux/
$ cp -rs /usr/include/asm asm/
$ cp -rs /usr/include/asm-generic asm-generic/
The reason for doing this will be explained later.
Download the PostgreSQL source code and built it:
$ git clone git://git.postgresql.org/git/postgresql.git
$ cd postgresql
$ ./configure --prefix=/home/masahiko/pgsql CC=musl-cc --without-readline --without-icu --witout-zlib
$ make
$ make install
CC=musl-ccYou can specify the compiler to use with CC.
--without-XXXBy default, PostgreSQL builds with readline, zlib, and icu enabled (in the case of using configure script). While the readline header files are in usr/include/readline, /usr/include also contains the header files of glibc. Therefore, if it’s configured to search /usr/include directory when compiling programs, the build using musl-gcc wouldn’t work. So I disabled these libraries for the build.
The reason I copied /usr/include/linux etc. directories in the preparation step was to deal with this problem. This seems to be the recommended way, but I went with it for now. I think readline etc. could also be built in a similar way. Not sure.
While PostgreSQL can be built without readline etc. /usr/include/linux etc. are essential for building PostgreSQL as PostgreSQL’s built-in programs are using it. For instance, pg_combinebackup requires /usr/include/linux. Without these steps, I got the following error in my environment:
musl-gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Werror=vla -Wendif-labels -Wmissing-format-attribute -Wimplicit-fallthrough=3 -Wcast-function-type -Wshadow=compatible-
local -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -Wno-format-truncation -Wno-stringop-truncation -O2 -I../../../src/interfaces/libpq -I../../../src/include -D_GNU_SOURCE
-c -o pg_combinebackup.o pg_combinebackup.c
pg_combinebackup.c:24:10: fatal error: linux/fs.h: No such file or directory
24 | #include <linux/fs.h>
| ^~~~~~~~~~~~
In my environment, I got the following WARNING during the build, but the build itself succeeded, so no problem:
pg_get_line.c: In function _pg_get_line_append_:
pg_get_line.c:129:27: warning: _({anonymous})_ may be used uninitialized [-Wmaybe-uninitialized]
129 | if (prompt_ctx && sigsetjmp(*((sigjmp_buf *) prompt_ctx->jmpbuf), 1) != 0)
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
After building PostgreSQL from source code, verify that it is indeed linked against musl libc:
$ ldd bin/postgres
linux-vdso.so.1 (0x00007ffd0bdf6000)
libc.so => /home/masahiko/musl/lib/libc.so (0x00007f21be639000)
musl libc is not completely compatible with glibc, and there are some behavioral differences, make check-world also passed.
While musl libc is smaller as a library than glibc, glibc is faster in terms of performance, so I’d also like to compare the performance as the next step.
]]>road to PostgreSQL. ROAD stands for “Read Only Archived Data” and is used to convert existing tables that are accessed infrequently but cannot be dropped into compact, read-only tables.
CAUTION: since pgroad is still in development and a hobby project, it’s not production ready
You can register pgroad in the database using CREATE EXTENSION command (you also need to add pgroad to shared_preload_libraries first):
=# CREATE EXTENSION pgroad;
CREATE EXTENSION
Create sample tables using pgbench:
$ pgbench -i -s 100
dropping old tables...
creating tables...
generating data (client-side)...
vacuuming...
creating primary keys...
done in 10.43 s (drop tables 0.00 s, create tables 0.01 s, client-side generate 7.41 s, vacuum 0.16 s, primary keys 2.86 s).
$ psql
=# \dt+ pgbench_accounts
List of relations
Schema | Name | Type | Owner | Persistence | Access method | Size | Description
--------+------------------+-------+----------+-------------+---------------+---------+-------------
public | pgbench_accounts | table | masahiko | permanent | heap | 1281 MB |
(1 row)
The pgbench_accounts table is a heap table, and we convert it into a road table:
=# ALTER TABLE pgbench_accounts SET ACCESS METHOD road;
ALTER TABLE
=# \dt+ pgbench_accounts
List of relations
Schema | Name | Type | Owner | Persistence | Access method | Size | Description
--------+------------------+-------+----------+-------------+---------------+--------+-------------
public | pgbench_accounts | table | masahiko | permanent | road | 118 MB |
(1 row)
You can see that the table has shrunk from 1281MB to 118MB. Once converted into a road table, you cannot modify the table:
=# DELETE FROM pgbench_accounts ;
ERROR: road_tuple_delete is not supported
=# INSERT INTO pgbench_accounts (aid) VALUES (1);
ERROR: cannot insert tuple directly into a ROAD table
HINT: Use ALTER TABLE ... SET ACCESS METHOD or CREATE TABLE ... AS to insert tuples
The implementation is very simple. It scans the existing table, stores tuples into 16kB chunks (in memory), compresses it, and writes them to the road table.
+-------------------+ +------------+ compression +---------+
| | --> | chunk | -------------> |xxxxxxxxx| \
| | | (16kB) | +---------+ \
| | +------------+ \ +--------------------+
| original table | \ | |
| (heap, 8kB pages) | +------------+ compression +---------+ `-> | new table |
| | --> | chunk | -------------> |xxxxxxxxx| --------> | (road, 8kB pages) |
| | | (16kB) | +---------+ .-> | |
| | +------------+ / | |
| | / +--------------------+
| | +------------+ compression +---------+ /
| | --> | chunk | -------------> |xxxxxxxxx| /
| | | (16kB) | +---------+
| | +------------+
+-------------------+
Currently the road table internally utilizes heap tuples. However, heap tuple headers contain some data like xmin and xmax that road doesn’t need, so I would like to add support for a custom (more compact) tuple format some day.
Chunk pages are compressed using pglz by default, but lz4 can also be chosen if enabled in the PostgreSQL.
I’ve implemented basic features for now:
Since pgroad focuses specifically on archiving existing data, road tables can only be created in the following two ways:
ALTER TABLE ... SET ACCESS METHOD roadCREATE TABLE ... USING road AS ...Also, you cannot create a road table inside a transaction block.
PostgreSQL provides hook points that extensions can tap into by registering their own function. ProcessUtility_hook is one such hook point that gets called when DDL statements are executed. pgroad uses ProcessUtility_hook to detect if the SQL statement was CREATE TABLE AS or ALTER TABLE ... SET ACCESS METHOD road:
static void
road_ProcessUtility(PlannedStmt *pstmt, const char *queryString,
bool readOnlyTree,
ProcessUtilityContext context, ParamListInfo params,
QueryEnvironment *queryEnv,
DestReceiver *dest, QueryCompletion *qc)
{
NodeTag tag = nodeTag(pstmt->utilityStmt);
if (tag == T_CreateTableAsStmt)
{
RoadInsertState.called_in_ctas = true;
Assert(!RoadInsertState.called_in_atsam);
}
else if (tag == T_AlterTableStmt)
{
AlterTableStmt *atstmt = (AlterTableStmt *) pstmt->utilityStmt;
ListCell *cell;
foreach(cell, atstmt->cmds)
{
AlterTableCmd *cmd = (AlterTableCmd *) lfirst(cell);
if (cmd->subtype == AT_SetAccessMethod)
{
Relation rel = relation_openrv(atstmt->relation, ShareLock);
/*
* Are we about to change the access method of the relation to
* ROAD table AM?
*/
if (strcmp(cmd->name, "road") == 0)
{
/* Remember the original table's OID */
RoadInsertState.atsam_relid = RelationGetRelid(rel);
RoadInsertState.called_in_atsam = true;
}
RelationClose(rel);
break;
}
}
Assert(!RoadInsertState.called_in_ctas);
}
prev_ProcessUtility(pstmt, queryString, false, context,
params, queryEnv, dest, qc);
}
We remember if either of the two DDL is executed, and raise an error in the insert callback if necessary:
static void
road_tuple_insert(Relation relation, TupleTableSlot *slot,
CommandId cid, int options, BulkInsertState bistate)
{
RoadInsertStateData *state;
ItemPointerData tid;
RowNumber rownum;
bool shouldFree;
HeapTuple tuple = ExecFetchSlotHeapTuple(slot, true, &shouldFree);
state = road_get_insert_state(relation);
if (!(state->called_in_atsam || state->called_in_ctas))
ereport(ERROR,
(errmsg("cannot insert tuple directly into a ROAD table"),
errhint("Use %s or %s to insert tuples",
"ALTER TABLE ... SET ACCESS METHOD",
"CREATE TABLE ... AS")));
The Table AM is actually a collection of callbacks. Table AM developer implements callbacks that get invoked for functionality like scans, index creation etc. that the table AM wants to support. Table AMs are nicely abstracted from other PostgreSQL components, so you can implement yours fairly independently. PostgreSQL provides transaction manager, buffer manager etc. so AMs can choose whether or not to leverage those facilities. For example, using the buffer manager provided by PostgreSQL core allows Table AM developers to implement their access method without having to consider the lower levels than the shared buffer.
That said, there is a lot more to consider when implementing a table AM:
While designing an idea, a robust Table AM is great, for hobby projects I recommend constraining the use cases as much as possible. More constraints means less cases to handle an dsimpler implementation. Start with a “minimally viable” table AM - something that works for a narrow use case, even if it’s not fully practical. Getting something working will be motivating you! The learning is in the process more than the end product.
pgroad has the following limitations that made its implementation very straightforward:
road tables can only be made with an exclusive lock on the source table.
road tables in transaction blocks
While pgorad focuses on archival, some other fun Table AM ideas:
I encourage you to try developing your own PostgreSQL Table AM extension!
]]>To prevent this problem, PostgreSQL has a safety mechanis called “aggressive vacuum”, runs automatically before the wraparound happens (specifically, before consumign approx. 2^31 XIDs). This clears old XIDs so new ones can continue to be consumed. In recent years, I’ve been working on improvements around this safety mechanism, so I often needed to test the mechanism itself, which requires consuming a massive number of XIDs and takes time.
For example, a poor-man’s approach to complete 2 billion wirte transactions. Moreover, when issuing new XIDs, status data accociated eich each XID, like CLOG and Commit Timestamp, also need to expand. When testing those behaviors, actually consuming XIDs is important.
So in this post, I’ll introduce various methods to consume XIDs quickly.
Comsuming 1 billion XIDs using this use-defined function that internally generates 1 billion subtransactions.
CREATE PROCEDURE consume_xids(cnt int)
AS $$
DECLARE i int;
BEGIN
FOR i in 1..cnt LOOP
EXECUTE 'SELECT txid_current()';
COMMIT;
END LOOP;
END;
$$
LANGUAGE plpgsql;
=# select pg_current_xact_id();
=# call consume_xid(100_000_000);
Time: 797958.576 ms (13:17.959)
=# select pg_current_xact_id();
Took 13 minutes, quite slow. XID generation takes exclusive locks so parallelization doesn’t help much.
The pg_resetwal resets PostgreSQL’s internal data. We can force the next XID like this. Since it just overwrite the internal data so pg_resetwal should complete instanly.
$ psql -c "SELECT pg_current_xact_id()"
$ pg_ctl stop
$ pg_resetwal -x 2000027648
$ pg_ctl start
$ psql -c "SELECT pg_current_xact_id()"
Why is the next XID an odd number, 2000027648? This is because we need to make sure the next XID lang on a CLOG page (8kB) boundary. Otherwise on startup, we’d get erros that transaction status (e.g. committed, aborted etc) is inaccessible. This happens because on startup, if the current XID isn’t at CLOG page boundary, statuses for unused XIDs in that page are initialied to 0 (see TrimCLOG() for details). This applies to all pg_resetwal use case, although it’s normally not an issue as XID skips are small.
This quickly and easily skips XIDs but not a “real” use case. We need to stop and start server, choose the next XID carefully, and it depends on page size.
PG_FUNCTION_INFO_V1(set_next_xid);
Datum
set_next_xid(PG_FUNCTION_ARGS)
{
TransactionId next_xid = PG_GETARG_TRANSACTIONID(0);
TransactionId xid;
uint32 epoch;
if (!TransactionIdIsNormal(next_xid))
elog(ERROR, "cannot set invalid transaction id");
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
if (TransactionIdPrecedes(next_xid,
XidFromFullTransactionId(TransamVariables->nextXid)))
{
LWLockRelease(XidGenLock);
elog(ERROR, "cannot set transaction id older than the current transaction id");
}
/*
* If the new XID is past xidVacLimit, start trying to force autovacuum
* cycles.
*/
if (TransactionIdFollowsOrEquals(next_xid, TransamVariables->xidVacLimit))
{
/* For safety, we release XidGenLock while sending signal */
LWLockRelease(XidGenLock);
SendPostmasterSignal(PMSIGNAL_START_AUTOVAC_LAUNCHER);
LWLockAcquire(XidGenLock, LW_EXCLUSIVE);
}
ExtendCLOG(next_xid);
ExtendCommitTs(next_xid);
ExtendSUBTRANS(next_xid);
/* Construct the new XID */
epoch = EpochFromFullTransactionId(TransamVariables->nextXid);
xid = XidFromFullTransactionId(TransamVariables->nextXid);
if (unlikely(xid > next_xid))
++epoch;
TransamVariables->nextXid =
FullTransactionIdFromEpochAndXid(epoch, next_xid);
LWLockRelease(XidGenLock);
PG_RETURN_VOID();
}
It can avoid serer restart unlike previous method. Expands CLOG and CommitTS etc near new XID. It can trigger autovacuum launcher for aggressive vacuum if new XID is old enough. Also, it should complete instantly.
=# select txid_current();
txid_current
--------------
737
(1 row)
Time: 0.850 ms
=# select set_next_xid('999981056'::xid);
set_next_xid
--------------
(1 row)
Time: 0.483 ms
=# select txid_current();
txid_current
--------------
999981056
(1 row)
Time: 0.926 ms
However, ExtendCLOG() only create pages if the passed XID lands on a page boundary, so the next XID must be calculated carefully same as before. This jumps XIDs more than incrementing them. Expands CLOG near new XID but does nothing for preceding XIDs.
Finally, the approach used by the xid_wraparound testing extension I recently pushed to the PostgreSQL source code.
Tough written for regression tests, the SQL functions in the exntesion can work if you installed it to your system.
=# CREATE EXTENSION xid_wraparound;
CREATE EXTENSION
=# \dx xid_wraparound
List of installed extensions
Name | Version | Schema | Description
----------------+---------+--------+--------------------------
xid_wraparound | 1.0 | public | Tests for XID wraparound
(1 row)
=# \dx+ xid_wraparound
Objects in extension "xid_wraparound"
Object description
-----------------------------------
function consume_xids(bigint)
function consume_xids_until(xid8)
(2 rows)
The key point of this method that differs from previous method is to move XIDs forward by “skiping while consuming” XIDs; it consume XIDs normally near “intereting XIDs” (those needing CLOG expansion etc) and skip XIDs internally elsewhere.
=# select txid_current();
txid_current
--------------
737
(1 row)
=# select consume_xids('1000000000');
NOTICE: consumed 10000071 / 1000000000 XIDs, latest 0:10000809
NOTICE: consumed 20000880 / 1000000000 XIDs, latest 0:20001618
NOTICE: consumed 30001689 / 1000000000 XIDs, latest 0:30002427
NOTICE: consumed 40002498 / 1000000000 XIDs, latest 0:40003236
NOTICE: consumed 50003230 / 1000000000 XIDs, latest 0:50003968
NOTICE: consumed 60003297 / 1000000000 XIDs, latest 0:60004035
NOTICE: consumed 70003998 / 1000000000 XIDs, latest 0:70004736
NOTICE: consumed 80004096 / 1000000000 XIDs, latest 0:80004834
NOTICE: consumed 90004766 / 1000000000 XIDs, latest 0:90005504
NOTICE: consumed 100004895 / 1000000000 XIDs, latest 0:100005633
NOTICE: consumed 110005534 / 1000000000 XIDs, latest 0:110006272
NOTICE: consumed 120005694 / 1000000000 XIDs, latest 0:120006432
NOTICE: consumed 130006302 / 1000000000 XIDs, latest 0:130007040
NOTICE: consumed 140006493 / 1000000000 XIDs, latest 0:140007231
NOTICE: consumed 150007070 / 1000000000 XIDs, latest 0:150007808
NOTICE: consumed 160007292 / 1000000000 XIDs, latest 0:160008030
NOTICE: consumed 170007838 / 1000000000 XIDs, latest 0:170008576
NOTICE: consumed 180008091 / 1000000000 XIDs, latest 0:180008829
NOTICE: consumed 190008606 / 1000000000 XIDs, latest 0:190009344
NOTICE: consumed 200008890 / 1000000000 XIDs, latest 0:200009628
:
:
NOTICE: consumed 960038174 / 1000000000 XIDs, latest 0:960038912
NOTICE: consumed 970038423 / 1000000000 XIDs, latest 0:970039161
NOTICE: consumed 980038942 / 1000000000 XIDs, latest 0:980039680
NOTICE: consumed 990039222 / 1000000000 XIDs, latest 0:990039960
consume_xids
--------------
1000000738
(1 row)
Time: 2893.244 ms (00:02.893)
=# select txid_current();
txid_current
--------------
1000000739
(1 row)
Good, reasonably fast. It can simulate more “real” XID consumption by doing normal consuming and related processing.
Note that it doesn’t trigger autovacuum launcher for aggressive vacuum so timing depends on the new XID and autovacuum_naptime. This may change in the future.
Thank you for reading.
]]>