2019-03-13 02:43:20 +01:00
|
|
|
/*-------------------------------------------------------------------------
|
|
|
|
*
|
|
|
|
* pg_checksums.c
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
* Checks, enables or disables page level checksums for an offline
|
|
|
|
* cluster
|
2018-04-05 21:57:26 +02:00
|
|
|
*
|
2020-01-01 18:21:45 +01:00
|
|
|
* Copyright (c) 2010-2020, PostgreSQL Global Development Group
|
2018-04-05 21:57:26 +02:00
|
|
|
*
|
2019-03-13 02:43:20 +01:00
|
|
|
* IDENTIFICATION
|
|
|
|
* src/bin/pg_checksums/pg_checksums.c
|
2018-04-05 21:57:26 +02:00
|
|
|
*
|
2019-03-13 02:43:20 +01:00
|
|
|
*-------------------------------------------------------------------------
|
2018-04-05 21:57:26 +02:00
|
|
|
*/
|
2019-03-13 02:43:20 +01:00
|
|
|
|
2018-08-31 19:42:18 +02:00
|
|
|
#include "postgres_fe.h"
|
2018-04-05 21:57:26 +02:00
|
|
|
|
2018-08-31 19:42:18 +02:00
|
|
|
#include <dirent.h>
|
2019-04-02 03:58:07 +02:00
|
|
|
#include <time.h>
|
2018-08-31 19:42:18 +02:00
|
|
|
#include <sys/stat.h>
|
|
|
|
#include <unistd.h>
|
2018-04-05 21:57:26 +02:00
|
|
|
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
#include "access/xlog_internal.h"
|
2018-04-05 21:57:26 +02:00
|
|
|
#include "common/controldata_utils.h"
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
#include "common/file_perm.h"
|
|
|
|
#include "common/file_utils.h"
|
2019-05-14 20:19:49 +02:00
|
|
|
#include "common/logging.h"
|
2018-06-20 18:19:16 +02:00
|
|
|
#include "getopt_long.h"
|
2018-08-31 19:42:18 +02:00
|
|
|
#include "pg_getopt.h"
|
2018-04-05 21:57:26 +02:00
|
|
|
#include "storage/bufpage.h"
|
|
|
|
#include "storage/checksum.h"
|
|
|
|
#include "storage/checksum_impl.h"
|
|
|
|
|
|
|
|
|
|
|
|
static int64 files = 0;
|
|
|
|
static int64 blocks = 0;
|
|
|
|
static int64 badblocks = 0;
|
|
|
|
static ControlFileData *ControlFile;
|
|
|
|
|
2019-05-30 22:58:17 +02:00
|
|
|
static char *only_filenode = NULL;
|
2019-03-23 00:37:36 +01:00
|
|
|
static bool do_sync = true;
|
2018-08-30 11:31:05 +02:00
|
|
|
static bool verbose = false;
|
2019-04-02 03:58:07 +02:00
|
|
|
static bool showprogress = false;
|
2018-04-05 21:57:26 +02:00
|
|
|
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
typedef enum
|
|
|
|
{
|
|
|
|
PG_MODE_CHECK,
|
|
|
|
PG_MODE_DISABLE,
|
|
|
|
PG_MODE_ENABLE
|
|
|
|
} PgChecksumMode;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Filename components.
|
|
|
|
*
|
|
|
|
* XXX: fd.h is not declared here as frontend side code is not able to
|
|
|
|
* interact with the backend-side definitions for the various fsync
|
|
|
|
* wrappers.
|
|
|
|
*/
|
|
|
|
#define PG_TEMP_FILES_DIR "pgsql_tmp"
|
|
|
|
#define PG_TEMP_FILE_PREFIX "pgsql_tmp"
|
|
|
|
|
|
|
|
static PgChecksumMode mode = PG_MODE_CHECK;
|
|
|
|
|
2018-04-05 21:57:26 +02:00
|
|
|
static const char *progname;
|
|
|
|
|
2019-04-02 03:58:07 +02:00
|
|
|
/*
|
|
|
|
* Progress status information.
|
|
|
|
*/
|
|
|
|
int64 total_size = 0;
|
|
|
|
int64 current_size = 0;
|
|
|
|
static pg_time_t last_progress_report = 0;
|
|
|
|
|
2018-04-05 21:57:26 +02:00
|
|
|
static void
|
2018-08-31 19:42:18 +02:00
|
|
|
usage(void)
|
2018-04-05 21:57:26 +02:00
|
|
|
{
|
2019-07-01 13:07:14 +02:00
|
|
|
printf(_("%s enables, disables, or verifies data checksums in a PostgreSQL database cluster.\n\n"), progname);
|
2018-04-05 21:57:26 +02:00
|
|
|
printf(_("Usage:\n"));
|
2018-08-23 20:32:56 +02:00
|
|
|
printf(_(" %s [OPTION]... [DATADIR]\n"), progname);
|
2018-04-05 21:57:26 +02:00
|
|
|
printf(_("\nOptions:\n"));
|
2019-05-30 22:58:17 +02:00
|
|
|
printf(_(" [-D, --pgdata=]DATADIR data directory\n"));
|
|
|
|
printf(_(" -c, --check check data checksums (default)\n"));
|
|
|
|
printf(_(" -d, --disable disable data checksums\n"));
|
|
|
|
printf(_(" -e, --enable enable data checksums\n"));
|
|
|
|
printf(_(" -f, --filenode=FILENODE check only relation with specified filenode\n"));
|
|
|
|
printf(_(" -N, --no-sync do not wait for changes to be written safely to disk\n"));
|
|
|
|
printf(_(" -P, --progress show progress information\n"));
|
|
|
|
printf(_(" -v, --verbose output verbose messages\n"));
|
|
|
|
printf(_(" -V, --version output version information, then exit\n"));
|
|
|
|
printf(_(" -?, --help show this help, then exit\n"));
|
2018-04-05 21:57:26 +02:00
|
|
|
printf(_("\nIf no data directory (DATADIR) is specified, "
|
|
|
|
"the environment variable PGDATA\nis used.\n\n"));
|
2020-02-28 08:54:49 +01:00
|
|
|
printf(_("Report bugs to <%s>.\n"), PACKAGE_BUGREPORT);
|
2020-02-28 08:54:49 +01:00
|
|
|
printf(_("%s home page: <%s>\n"), PACKAGE_NAME, PACKAGE_URL);
|
2018-04-05 21:57:26 +02:00
|
|
|
}
|
|
|
|
|
2020-02-24 10:13:25 +01:00
|
|
|
/*
|
|
|
|
* Definition of one element part of an exclusion list, used for files
|
|
|
|
* to exclude from checksum validation. "name" is the name of the file
|
|
|
|
* or path to check for exclusion. If "match_prefix" is true, any items
|
|
|
|
* matching the name as prefix are excluded.
|
|
|
|
*/
|
|
|
|
struct exclude_list_item
|
|
|
|
{
|
|
|
|
const char *name;
|
|
|
|
bool match_prefix;
|
|
|
|
};
|
|
|
|
|
Fix various checksum check problems for pg_verify_checksums and base backups
Three issues are fixed in this patch:
- Base backups forgot to ignore files specific to EXEC_BACKEND, leading
to spurious warnings when checksums are enabled, per analysis from me.
- pg_verify_checksums forgot about files specific to EXEC_BACKEND,
leading to failures of the tool on any such build, particularly Windows.
This error was originally found by newly-introduced TAP tests in various
buildfarm members using EXEC_BACKEND.
- pg_verify_checksums forgot to count for temporary files and temporary
paths, which could be valid relation files, without checksums, per
report from Andres Freund. More tests are added to cover this case.
A new test case which emulates corruption for a file in a different
tablespace is added, coming from from Michael Banck, while I have coded
the main code and refactored the test code.
Author: Michael Banck, Michael Paquier
Reviewed-by: Stephen Frost, David Steele
Discussion: https://postgr.es/m/20181021134206.GA14282@paquier.xyz
2018-11-30 02:34:45 +01:00
|
|
|
/*
|
|
|
|
* List of files excluded from checksum validation.
|
|
|
|
*
|
|
|
|
* Note: this list should be kept in sync with what basebackup.c includes.
|
|
|
|
*/
|
2020-02-24 10:13:25 +01:00
|
|
|
static const struct exclude_list_item skip[] = {
|
|
|
|
{"pg_control", false},
|
|
|
|
{"pg_filenode.map", false},
|
|
|
|
{"pg_internal.init", true},
|
|
|
|
{"PG_VERSION", false},
|
Fix various checksum check problems for pg_verify_checksums and base backups
Three issues are fixed in this patch:
- Base backups forgot to ignore files specific to EXEC_BACKEND, leading
to spurious warnings when checksums are enabled, per analysis from me.
- pg_verify_checksums forgot about files specific to EXEC_BACKEND,
leading to failures of the tool on any such build, particularly Windows.
This error was originally found by newly-introduced TAP tests in various
buildfarm members using EXEC_BACKEND.
- pg_verify_checksums forgot to count for temporary files and temporary
paths, which could be valid relation files, without checksums, per
report from Andres Freund. More tests are added to cover this case.
A new test case which emulates corruption for a file in a different
tablespace is added, coming from from Michael Banck, while I have coded
the main code and refactored the test code.
Author: Michael Banck, Michael Paquier
Reviewed-by: Stephen Frost, David Steele
Discussion: https://postgr.es/m/20181021134206.GA14282@paquier.xyz
2018-11-30 02:34:45 +01:00
|
|
|
#ifdef EXEC_BACKEND
|
2020-02-24 10:13:25 +01:00
|
|
|
{"config_exec_params", true},
|
Fix various checksum check problems for pg_verify_checksums and base backups
Three issues are fixed in this patch:
- Base backups forgot to ignore files specific to EXEC_BACKEND, leading
to spurious warnings when checksums are enabled, per analysis from me.
- pg_verify_checksums forgot about files specific to EXEC_BACKEND,
leading to failures of the tool on any such build, particularly Windows.
This error was originally found by newly-introduced TAP tests in various
buildfarm members using EXEC_BACKEND.
- pg_verify_checksums forgot to count for temporary files and temporary
paths, which could be valid relation files, without checksums, per
report from Andres Freund. More tests are added to cover this case.
A new test case which emulates corruption for a file in a different
tablespace is added, coming from from Michael Banck, while I have coded
the main code and refactored the test code.
Author: Michael Banck, Michael Paquier
Reviewed-by: Stephen Frost, David Steele
Discussion: https://postgr.es/m/20181021134206.GA14282@paquier.xyz
2018-11-30 02:34:45 +01:00
|
|
|
#endif
|
2020-02-24 10:13:25 +01:00
|
|
|
{NULL, false}
|
2018-11-30 02:14:58 +01:00
|
|
|
};
|
|
|
|
|
2019-04-02 03:58:07 +02:00
|
|
|
/*
|
|
|
|
* Report current progress status. Parts borrowed from
|
2019-04-25 16:49:37 +02:00
|
|
|
* src/bin/pg_basebackup/pg_basebackup.c.
|
2019-04-02 03:58:07 +02:00
|
|
|
*/
|
|
|
|
static void
|
|
|
|
progress_report(bool force)
|
|
|
|
{
|
|
|
|
int percent;
|
|
|
|
char total_size_str[32];
|
|
|
|
char current_size_str[32];
|
|
|
|
pg_time_t now;
|
|
|
|
|
|
|
|
Assert(showprogress);
|
|
|
|
|
|
|
|
now = time(NULL);
|
|
|
|
if (now == last_progress_report && !force)
|
|
|
|
return; /* Max once per second */
|
|
|
|
|
|
|
|
/* Save current time */
|
|
|
|
last_progress_report = now;
|
|
|
|
|
|
|
|
/* Adjust total size if current_size is larger */
|
|
|
|
if (current_size > total_size)
|
|
|
|
total_size = current_size;
|
|
|
|
|
|
|
|
/* Calculate current percentage of size done */
|
|
|
|
percent = total_size ? (int) ((current_size) * 100 / total_size) : 0;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Separate step to keep platform-dependent format code out of
|
|
|
|
* translatable strings. And we only test for INT64_FORMAT availability
|
|
|
|
* in snprintf, not fprintf.
|
|
|
|
*/
|
2019-05-28 14:26:24 +02:00
|
|
|
snprintf(total_size_str, sizeof(total_size_str), INT64_FORMAT,
|
|
|
|
total_size / (1024 * 1024));
|
|
|
|
snprintf(current_size_str, sizeof(current_size_str), INT64_FORMAT,
|
|
|
|
current_size / (1024 * 1024));
|
|
|
|
|
2019-06-06 15:10:52 +02:00
|
|
|
fprintf(stderr, _("%*s/%s MB (%d%%) computed"),
|
2019-04-02 03:58:07 +02:00
|
|
|
(int) strlen(current_size_str), current_size_str, total_size_str,
|
|
|
|
percent);
|
|
|
|
|
|
|
|
/* Stay on the same line if reporting to a terminal */
|
|
|
|
fprintf(stderr, isatty(fileno(stderr)) ? "\r" : "\n");
|
|
|
|
}
|
|
|
|
|
2018-04-05 21:57:26 +02:00
|
|
|
static bool
|
2018-11-30 02:14:58 +01:00
|
|
|
skipfile(const char *fn)
|
2018-04-05 21:57:26 +02:00
|
|
|
{
|
2020-02-24 10:13:25 +01:00
|
|
|
int excludeIdx;
|
|
|
|
|
|
|
|
for (excludeIdx = 0; skip[excludeIdx].name != NULL; excludeIdx++)
|
|
|
|
{
|
|
|
|
int cmplen = strlen(skip[excludeIdx].name);
|
2018-10-19 15:44:12 +02:00
|
|
|
|
2020-02-24 10:13:25 +01:00
|
|
|
if (!skip[excludeIdx].match_prefix)
|
|
|
|
cmplen++;
|
|
|
|
if (strncmp(skip[excludeIdx].name, fn, cmplen) == 0)
|
2018-11-30 02:14:58 +01:00
|
|
|
return true;
|
2020-02-24 10:13:25 +01:00
|
|
|
}
|
Fix various checksum check problems for pg_verify_checksums and base backups
Three issues are fixed in this patch:
- Base backups forgot to ignore files specific to EXEC_BACKEND, leading
to spurious warnings when checksums are enabled, per analysis from me.
- pg_verify_checksums forgot about files specific to EXEC_BACKEND,
leading to failures of the tool on any such build, particularly Windows.
This error was originally found by newly-introduced TAP tests in various
buildfarm members using EXEC_BACKEND.
- pg_verify_checksums forgot to count for temporary files and temporary
paths, which could be valid relation files, without checksums, per
report from Andres Freund. More tests are added to cover this case.
A new test case which emulates corruption for a file in a different
tablespace is added, coming from from Michael Banck, while I have coded
the main code and refactored the test code.
Author: Michael Banck, Michael Paquier
Reviewed-by: Stephen Frost, David Steele
Discussion: https://postgr.es/m/20181021134206.GA14282@paquier.xyz
2018-11-30 02:34:45 +01:00
|
|
|
|
2018-11-30 02:14:58 +01:00
|
|
|
return false;
|
2018-04-05 21:57:26 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
static void
|
2018-08-31 19:42:18 +02:00
|
|
|
scan_file(const char *fn, BlockNumber segmentno)
|
2018-04-05 21:57:26 +02:00
|
|
|
{
|
2018-09-01 21:27:12 +02:00
|
|
|
PGAlignedBlock buf;
|
|
|
|
PageHeader header = (PageHeader) buf.data;
|
2018-04-05 21:57:26 +02:00
|
|
|
int f;
|
2018-08-31 19:42:18 +02:00
|
|
|
BlockNumber blockno;
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
int flags;
|
|
|
|
|
|
|
|
Assert(mode == PG_MODE_ENABLE ||
|
|
|
|
mode == PG_MODE_CHECK);
|
|
|
|
|
|
|
|
flags = (mode == PG_MODE_ENABLE) ? O_RDWR : O_RDONLY;
|
|
|
|
f = open(fn, PG_BINARY | flags, 0);
|
2018-04-05 21:57:26 +02:00
|
|
|
|
|
|
|
if (f < 0)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("could not open file \"%s\": %m", fn);
|
2018-04-05 21:57:26 +02:00
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
|
|
|
files++;
|
|
|
|
|
|
|
|
for (blockno = 0;; blockno++)
|
|
|
|
{
|
|
|
|
uint16 csum;
|
2018-09-01 21:27:12 +02:00
|
|
|
int r = read(f, buf.data, BLCKSZ);
|
2018-04-05 21:57:26 +02:00
|
|
|
|
|
|
|
if (r == 0)
|
|
|
|
break;
|
|
|
|
if (r != BLCKSZ)
|
|
|
|
{
|
2019-09-03 08:26:55 +02:00
|
|
|
if (r < 0)
|
|
|
|
pg_log_error("could not read block %u in file \"%s\": %m",
|
|
|
|
blockno, fn);
|
|
|
|
else
|
|
|
|
pg_log_error("could not read block %u in file \"%s\": read %d of %d",
|
|
|
|
blockno, fn, r, BLCKSZ);
|
2018-04-05 21:57:26 +02:00
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
blocks++;
|
|
|
|
|
2018-04-15 14:05:56 +02:00
|
|
|
/* New pages have no checksum yet */
|
2018-08-31 19:42:18 +02:00
|
|
|
if (PageIsNew(header))
|
2018-04-15 14:05:56 +02:00
|
|
|
continue;
|
|
|
|
|
2018-09-01 21:27:12 +02:00
|
|
|
csum = pg_checksum_page(buf.data, blockno + segmentno * RELSEG_SIZE);
|
2019-04-02 03:58:07 +02:00
|
|
|
current_size += r;
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
if (mode == PG_MODE_CHECK)
|
2018-04-05 21:57:26 +02:00
|
|
|
{
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
if (csum != header->pd_checksum)
|
|
|
|
{
|
|
|
|
if (ControlFile->data_checksum_version == PG_DATA_CHECKSUM_VERSION)
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("checksum verification failed in file \"%s\", block %u: calculated checksum %X but block contains %X",
|
|
|
|
fn, blockno, csum, header->pd_checksum);
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
badblocks++;
|
|
|
|
}
|
|
|
|
}
|
|
|
|
else if (mode == PG_MODE_ENABLE)
|
|
|
|
{
|
2019-09-03 08:26:55 +02:00
|
|
|
int w;
|
|
|
|
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
/* Set checksum in page header */
|
|
|
|
header->pd_checksum = csum;
|
|
|
|
|
|
|
|
/* Seek back to beginning of block */
|
|
|
|
if (lseek(f, -BLCKSZ, SEEK_CUR) < 0)
|
|
|
|
{
|
2019-04-29 16:05:07 +02:00
|
|
|
pg_log_error("seek failed for block %u in file \"%s\": %m", blockno, fn);
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Write block with checksum */
|
2019-09-03 08:26:55 +02:00
|
|
|
w = write(f, buf.data, BLCKSZ);
|
|
|
|
if (w != BLCKSZ)
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
{
|
2019-09-03 08:26:55 +02:00
|
|
|
if (w < 0)
|
|
|
|
pg_log_error("could not write block %u in file \"%s\": %m",
|
|
|
|
blockno, fn);
|
|
|
|
else
|
|
|
|
pg_log_error("could not write block %u in file \"%s\": wrote %d of %d",
|
|
|
|
blockno, fn, w, BLCKSZ);
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
exit(1);
|
|
|
|
}
|
2018-04-05 21:57:26 +02:00
|
|
|
}
|
2019-04-02 03:58:07 +02:00
|
|
|
|
|
|
|
if (showprogress)
|
|
|
|
progress_report(false);
|
2018-04-05 21:57:26 +02:00
|
|
|
}
|
|
|
|
|
2018-08-30 11:31:05 +02:00
|
|
|
if (verbose)
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
{
|
|
|
|
if (mode == PG_MODE_CHECK)
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_info("checksums verified in file \"%s\"", fn);
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
if (mode == PG_MODE_ENABLE)
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_info("checksums enabled in file \"%s\"", fn);
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
}
|
2018-08-30 11:31:05 +02:00
|
|
|
|
2018-04-05 21:57:26 +02:00
|
|
|
close(f);
|
|
|
|
}
|
|
|
|
|
2019-04-02 03:58:07 +02:00
|
|
|
/*
|
|
|
|
* Scan the given directory for items which can be checksummed and
|
|
|
|
* operate on each one of them. If "sizeonly" is true, the size of
|
|
|
|
* all the items which have checksums is computed and returned back
|
|
|
|
* to the caller without operating on the files. This is used to compile
|
|
|
|
* the total size of the data directory for progress reports.
|
|
|
|
*/
|
|
|
|
static int64
|
|
|
|
scan_directory(const char *basedir, const char *subdir, bool sizeonly)
|
2018-04-05 21:57:26 +02:00
|
|
|
{
|
2019-04-02 03:58:07 +02:00
|
|
|
int64 dirsize = 0;
|
2018-04-05 21:57:26 +02:00
|
|
|
char path[MAXPGPATH];
|
|
|
|
DIR *dir;
|
|
|
|
struct dirent *de;
|
|
|
|
|
2018-04-06 14:43:50 +02:00
|
|
|
snprintf(path, sizeof(path), "%s/%s", basedir, subdir);
|
2018-04-05 21:57:26 +02:00
|
|
|
dir = opendir(path);
|
|
|
|
if (!dir)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("could not open directory \"%s\": %m", path);
|
2018-04-05 21:57:26 +02:00
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
while ((de = readdir(dir)) != NULL)
|
|
|
|
{
|
2018-08-31 19:42:18 +02:00
|
|
|
char fn[MAXPGPATH];
|
2018-04-05 21:57:26 +02:00
|
|
|
struct stat st;
|
|
|
|
|
Fix various checksum check problems for pg_verify_checksums and base backups
Three issues are fixed in this patch:
- Base backups forgot to ignore files specific to EXEC_BACKEND, leading
to spurious warnings when checksums are enabled, per analysis from me.
- pg_verify_checksums forgot about files specific to EXEC_BACKEND,
leading to failures of the tool on any such build, particularly Windows.
This error was originally found by newly-introduced TAP tests in various
buildfarm members using EXEC_BACKEND.
- pg_verify_checksums forgot to count for temporary files and temporary
paths, which could be valid relation files, without checksums, per
report from Andres Freund. More tests are added to cover this case.
A new test case which emulates corruption for a file in a different
tablespace is added, coming from from Michael Banck, while I have coded
the main code and refactored the test code.
Author: Michael Banck, Michael Paquier
Reviewed-by: Stephen Frost, David Steele
Discussion: https://postgr.es/m/20181021134206.GA14282@paquier.xyz
2018-11-30 02:34:45 +01:00
|
|
|
if (strcmp(de->d_name, ".") == 0 ||
|
|
|
|
strcmp(de->d_name, "..") == 0)
|
2018-04-05 21:57:26 +02:00
|
|
|
continue;
|
|
|
|
|
Fix various checksum check problems for pg_verify_checksums and base backups
Three issues are fixed in this patch:
- Base backups forgot to ignore files specific to EXEC_BACKEND, leading
to spurious warnings when checksums are enabled, per analysis from me.
- pg_verify_checksums forgot about files specific to EXEC_BACKEND,
leading to failures of the tool on any such build, particularly Windows.
This error was originally found by newly-introduced TAP tests in various
buildfarm members using EXEC_BACKEND.
- pg_verify_checksums forgot to count for temporary files and temporary
paths, which could be valid relation files, without checksums, per
report from Andres Freund. More tests are added to cover this case.
A new test case which emulates corruption for a file in a different
tablespace is added, coming from from Michael Banck, while I have coded
the main code and refactored the test code.
Author: Michael Banck, Michael Paquier
Reviewed-by: Stephen Frost, David Steele
Discussion: https://postgr.es/m/20181021134206.GA14282@paquier.xyz
2018-11-30 02:34:45 +01:00
|
|
|
/* Skip temporary files */
|
|
|
|
if (strncmp(de->d_name,
|
|
|
|
PG_TEMP_FILE_PREFIX,
|
|
|
|
strlen(PG_TEMP_FILE_PREFIX)) == 0)
|
|
|
|
continue;
|
|
|
|
|
|
|
|
/* Skip temporary folders */
|
|
|
|
if (strncmp(de->d_name,
|
|
|
|
PG_TEMP_FILES_DIR,
|
|
|
|
strlen(PG_TEMP_FILES_DIR)) == 0)
|
2019-03-14 06:14:49 +01:00
|
|
|
continue;
|
Fix various checksum check problems for pg_verify_checksums and base backups
Three issues are fixed in this patch:
- Base backups forgot to ignore files specific to EXEC_BACKEND, leading
to spurious warnings when checksums are enabled, per analysis from me.
- pg_verify_checksums forgot about files specific to EXEC_BACKEND,
leading to failures of the tool on any such build, particularly Windows.
This error was originally found by newly-introduced TAP tests in various
buildfarm members using EXEC_BACKEND.
- pg_verify_checksums forgot to count for temporary files and temporary
paths, which could be valid relation files, without checksums, per
report from Andres Freund. More tests are added to cover this case.
A new test case which emulates corruption for a file in a different
tablespace is added, coming from from Michael Banck, while I have coded
the main code and refactored the test code.
Author: Michael Banck, Michael Paquier
Reviewed-by: Stephen Frost, David Steele
Discussion: https://postgr.es/m/20181021134206.GA14282@paquier.xyz
2018-11-30 02:34:45 +01:00
|
|
|
|
2018-04-06 14:43:50 +02:00
|
|
|
snprintf(fn, sizeof(fn), "%s/%s", path, de->d_name);
|
2018-04-05 21:57:26 +02:00
|
|
|
if (lstat(fn, &st) < 0)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("could not stat file \"%s\": %m", fn);
|
2018-04-05 21:57:26 +02:00
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
if (S_ISREG(st.st_mode))
|
|
|
|
{
|
2018-08-31 19:42:18 +02:00
|
|
|
char fnonly[MAXPGPATH];
|
2018-04-05 21:57:26 +02:00
|
|
|
char *forkpath,
|
|
|
|
*segmentpath;
|
2018-08-31 19:42:18 +02:00
|
|
|
BlockNumber segmentno = 0;
|
2018-04-05 21:57:26 +02:00
|
|
|
|
Fix various checksum check problems for pg_verify_checksums and base backups
Three issues are fixed in this patch:
- Base backups forgot to ignore files specific to EXEC_BACKEND, leading
to spurious warnings when checksums are enabled, per analysis from me.
- pg_verify_checksums forgot about files specific to EXEC_BACKEND,
leading to failures of the tool on any such build, particularly Windows.
This error was originally found by newly-introduced TAP tests in various
buildfarm members using EXEC_BACKEND.
- pg_verify_checksums forgot to count for temporary files and temporary
paths, which could be valid relation files, without checksums, per
report from Andres Freund. More tests are added to cover this case.
A new test case which emulates corruption for a file in a different
tablespace is added, coming from from Michael Banck, while I have coded
the main code and refactored the test code.
Author: Michael Banck, Michael Paquier
Reviewed-by: Stephen Frost, David Steele
Discussion: https://postgr.es/m/20181021134206.GA14282@paquier.xyz
2018-11-30 02:34:45 +01:00
|
|
|
if (skipfile(de->d_name))
|
|
|
|
continue;
|
|
|
|
|
2018-04-05 21:57:26 +02:00
|
|
|
/*
|
|
|
|
* Cut off at the segment boundary (".") to get the segment number
|
|
|
|
* in order to mix it into the checksum. Then also cut off at the
|
2019-05-30 22:58:17 +02:00
|
|
|
* fork boundary, to get the filenode the file belongs to for
|
2018-04-05 21:57:26 +02:00
|
|
|
* filtering.
|
|
|
|
*/
|
2018-08-31 19:42:18 +02:00
|
|
|
strlcpy(fnonly, de->d_name, sizeof(fnonly));
|
|
|
|
segmentpath = strchr(fnonly, '.');
|
2018-04-05 21:57:26 +02:00
|
|
|
if (segmentpath != NULL)
|
|
|
|
{
|
|
|
|
*segmentpath++ = '\0';
|
|
|
|
segmentno = atoi(segmentpath);
|
|
|
|
if (segmentno == 0)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("invalid segment number %d in file name \"%s\"",
|
|
|
|
segmentno, fn);
|
2018-04-05 21:57:26 +02:00
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2018-08-31 19:42:18 +02:00
|
|
|
forkpath = strchr(fnonly, '_');
|
2018-04-05 21:57:26 +02:00
|
|
|
if (forkpath != NULL)
|
|
|
|
*forkpath++ = '\0';
|
|
|
|
|
2019-05-30 22:58:17 +02:00
|
|
|
if (only_filenode && strcmp(only_filenode, fnonly) != 0)
|
|
|
|
/* filenode not to be included */
|
2018-04-05 21:57:26 +02:00
|
|
|
continue;
|
|
|
|
|
2019-04-02 03:58:07 +02:00
|
|
|
dirsize += st.st_size;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* No need to work on the file when calculating only the size of
|
|
|
|
* the items in the data folder.
|
|
|
|
*/
|
|
|
|
if (!sizeonly)
|
|
|
|
scan_file(fn, segmentno);
|
2018-04-05 21:57:26 +02:00
|
|
|
}
|
2018-04-05 22:38:03 +02:00
|
|
|
#ifndef WIN32
|
2018-04-05 21:57:26 +02:00
|
|
|
else if (S_ISDIR(st.st_mode) || S_ISLNK(st.st_mode))
|
2018-04-05 22:38:03 +02:00
|
|
|
#else
|
|
|
|
else if (S_ISDIR(st.st_mode) || pgwin32_is_junction(fn))
|
|
|
|
#endif
|
2020-02-27 07:31:27 +01:00
|
|
|
{
|
|
|
|
/*
|
|
|
|
* If going through the entries of pg_tblspc, we assume to operate
|
|
|
|
* on tablespace locations where only TABLESPACE_VERSION_DIRECTORY
|
|
|
|
* is valid, resolving the linked locations and dive into them
|
|
|
|
* directly.
|
|
|
|
*/
|
|
|
|
if (strncmp("pg_tblspc", subdir, strlen("pg_tblspc")) == 0)
|
|
|
|
{
|
|
|
|
char tblspc_path[MAXPGPATH];
|
|
|
|
struct stat tblspc_st;
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Resolve tablespace location path and check whether
|
|
|
|
* TABLESPACE_VERSION_DIRECTORY exists. Not finding a valid
|
|
|
|
* location is unexpected, since there should be no orphaned
|
|
|
|
* links and no links pointing to something else than a
|
|
|
|
* directory.
|
|
|
|
*/
|
|
|
|
snprintf(tblspc_path, sizeof(tblspc_path), "%s/%s/%s",
|
|
|
|
path, de->d_name, TABLESPACE_VERSION_DIRECTORY);
|
|
|
|
|
|
|
|
if (lstat(tblspc_path, &tblspc_st) < 0)
|
|
|
|
{
|
|
|
|
pg_log_error("could not stat file \"%s\": %m",
|
|
|
|
tblspc_path);
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Move backwards once as the scan needs to happen for the
|
|
|
|
* contents of TABLESPACE_VERSION_DIRECTORY.
|
|
|
|
*/
|
|
|
|
snprintf(tblspc_path, sizeof(tblspc_path), "%s/%s",
|
|
|
|
path, de->d_name);
|
|
|
|
|
|
|
|
/* Looks like a valid tablespace location */
|
|
|
|
dirsize += scan_directory(tblspc_path,
|
|
|
|
TABLESPACE_VERSION_DIRECTORY,
|
|
|
|
sizeonly);
|
|
|
|
}
|
|
|
|
else
|
|
|
|
{
|
|
|
|
dirsize += scan_directory(path, de->d_name, sizeonly);
|
|
|
|
}
|
|
|
|
}
|
2018-04-05 21:57:26 +02:00
|
|
|
}
|
|
|
|
closedir(dir);
|
2019-04-02 03:58:07 +02:00
|
|
|
return dirsize;
|
2018-04-05 21:57:26 +02:00
|
|
|
}
|
|
|
|
|
|
|
|
int
|
|
|
|
main(int argc, char *argv[])
|
|
|
|
{
|
2018-06-20 14:33:48 +02:00
|
|
|
static struct option long_options[] = {
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
{"check", no_argument, NULL, 'c'},
|
2018-06-20 14:33:48 +02:00
|
|
|
{"pgdata", required_argument, NULL, 'D'},
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
{"disable", no_argument, NULL, 'd'},
|
|
|
|
{"enable", no_argument, NULL, 'e'},
|
2019-05-30 22:58:17 +02:00
|
|
|
{"filenode", required_argument, NULL, 'f'},
|
2019-03-23 00:37:36 +01:00
|
|
|
{"no-sync", no_argument, NULL, 'N'},
|
2019-04-02 03:58:07 +02:00
|
|
|
{"progress", no_argument, NULL, 'P'},
|
2018-08-30 11:31:05 +02:00
|
|
|
{"verbose", no_argument, NULL, 'v'},
|
2018-06-20 14:33:48 +02:00
|
|
|
{NULL, 0, NULL, 0}
|
|
|
|
};
|
|
|
|
|
2018-04-05 21:57:26 +02:00
|
|
|
char *DataDir = NULL;
|
|
|
|
int c;
|
2018-06-20 14:33:48 +02:00
|
|
|
int option_index;
|
2018-04-05 21:57:26 +02:00
|
|
|
bool crc_ok;
|
|
|
|
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_logging_init(argv[0]);
|
2019-03-13 02:43:20 +01:00
|
|
|
set_pglocale_pgservice(argv[0], PG_TEXTDOMAIN("pg_checksums"));
|
2018-04-05 21:57:26 +02:00
|
|
|
progname = get_progname(argv[0]);
|
|
|
|
|
|
|
|
if (argc > 1)
|
|
|
|
{
|
|
|
|
if (strcmp(argv[1], "--help") == 0 || strcmp(argv[1], "-?") == 0)
|
|
|
|
{
|
|
|
|
usage();
|
|
|
|
exit(0);
|
|
|
|
}
|
|
|
|
if (strcmp(argv[1], "--version") == 0 || strcmp(argv[1], "-V") == 0)
|
|
|
|
{
|
2019-03-13 02:43:20 +01:00
|
|
|
puts("pg_checksums (PostgreSQL) " PG_VERSION);
|
2018-04-05 21:57:26 +02:00
|
|
|
exit(0);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
2019-05-30 22:58:17 +02:00
|
|
|
while ((c = getopt_long(argc, argv, "cD:deNPf:v", long_options, &option_index)) != -1)
|
2018-04-05 21:57:26 +02:00
|
|
|
{
|
|
|
|
switch (c)
|
|
|
|
{
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
case 'c':
|
|
|
|
mode = PG_MODE_CHECK;
|
|
|
|
break;
|
|
|
|
case 'd':
|
|
|
|
mode = PG_MODE_DISABLE;
|
|
|
|
break;
|
|
|
|
case 'e':
|
|
|
|
mode = PG_MODE_ENABLE;
|
|
|
|
break;
|
2019-05-30 22:58:17 +02:00
|
|
|
case 'f':
|
|
|
|
if (atoi(optarg) == 0)
|
|
|
|
{
|
|
|
|
pg_log_error("invalid filenode specification, must be numeric: %s", optarg);
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
only_filenode = pstrdup(optarg);
|
|
|
|
break;
|
2019-03-23 00:37:36 +01:00
|
|
|
case 'N':
|
|
|
|
do_sync = false;
|
|
|
|
break;
|
2018-08-30 11:31:05 +02:00
|
|
|
case 'v':
|
|
|
|
verbose = true;
|
2018-04-05 21:57:26 +02:00
|
|
|
break;
|
|
|
|
case 'D':
|
|
|
|
DataDir = optarg;
|
|
|
|
break;
|
2019-04-02 03:58:07 +02:00
|
|
|
case 'P':
|
|
|
|
showprogress = true;
|
|
|
|
break;
|
2018-04-05 21:57:26 +02:00
|
|
|
default:
|
|
|
|
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
if (DataDir == NULL)
|
|
|
|
{
|
|
|
|
if (optind < argc)
|
|
|
|
DataDir = argv[optind++];
|
|
|
|
else
|
|
|
|
DataDir = getenv("PGDATA");
|
|
|
|
|
|
|
|
/* If no DataDir was specified, and none could be found, error out */
|
|
|
|
if (DataDir == NULL)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("no data directory specified");
|
2018-04-05 21:57:26 +02:00
|
|
|
fprintf(stderr, _("Try \"%s --help\" for more information.\n"), progname);
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Complain if any arguments remain */
|
|
|
|
if (optind < argc)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("too many command-line arguments (first is \"%s\")",
|
|
|
|
argv[optind]);
|
2018-04-05 21:57:26 +02:00
|
|
|
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
|
|
|
|
progname);
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
2019-05-30 22:58:17 +02:00
|
|
|
/* filenode checking only works in --check mode */
|
|
|
|
if (mode != PG_MODE_CHECK && only_filenode)
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
{
|
2019-07-09 15:47:09 +02:00
|
|
|
pg_log_error("option -f/--filenode can only be used with --check");
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
fprintf(stderr, _("Try \"%s --help\" for more information.\n"),
|
|
|
|
progname);
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
2019-06-07 13:48:39 +02:00
|
|
|
/* Read the control file and check compatibility */
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
ControlFile = get_controlfile(DataDir, &crc_ok);
|
2018-04-05 21:57:26 +02:00
|
|
|
if (!crc_ok)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("pg_control CRC value is incorrect");
|
2018-04-05 21:57:26 +02:00
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
2019-03-13 01:51:02 +01:00
|
|
|
if (ControlFile->pg_control_version != PG_CONTROL_VERSION)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("cluster is not compatible with this version of pg_checksums");
|
2019-03-13 01:51:02 +01:00
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
2019-03-18 01:11:52 +01:00
|
|
|
if (ControlFile->blcksz != BLCKSZ)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("database cluster is not compatible");
|
2019-03-18 01:11:52 +01:00
|
|
|
fprintf(stderr, _("The database cluster was initialized with block size %u, but pg_checksums was compiled with block size %u.\n"),
|
|
|
|
ControlFile->blcksz, BLCKSZ);
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
2019-06-07 13:48:39 +02:00
|
|
|
/*
|
|
|
|
* Check if cluster is running. A clean shutdown is required to avoid
|
|
|
|
* random checksum failures caused by torn pages. Note that this doesn't
|
|
|
|
* guard against someone starting the cluster concurrently.
|
|
|
|
*/
|
2018-04-05 21:57:26 +02:00
|
|
|
if (ControlFile->state != DB_SHUTDOWNED &&
|
|
|
|
ControlFile->state != DB_SHUTDOWNED_IN_RECOVERY)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("cluster must be shut down");
|
2018-04-05 21:57:26 +02:00
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
if (ControlFile->data_checksum_version == 0 &&
|
|
|
|
mode == PG_MODE_CHECK)
|
2018-04-05 21:57:26 +02:00
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("data checksums are not enabled in cluster");
|
2018-04-05 21:57:26 +02:00
|
|
|
exit(1);
|
|
|
|
}
|
2019-03-23 13:56:43 +01:00
|
|
|
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
if (ControlFile->data_checksum_version == 0 &&
|
|
|
|
mode == PG_MODE_DISABLE)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("data checksums are already disabled in cluster");
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
exit(1);
|
|
|
|
}
|
2019-03-23 13:56:43 +01:00
|
|
|
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
if (ControlFile->data_checksum_version > 0 &&
|
|
|
|
mode == PG_MODE_ENABLE)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_error("data checksums are already enabled in cluster");
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
|
|
|
|
/* Operate on all files if checking or enabling checksums */
|
|
|
|
if (mode == PG_MODE_CHECK || mode == PG_MODE_ENABLE)
|
|
|
|
{
|
2019-04-02 03:58:07 +02:00
|
|
|
/*
|
|
|
|
* If progress status information is requested, we need to scan the
|
|
|
|
* directory tree twice: once to know how much total data needs to be
|
|
|
|
* processed and once to do the real work.
|
|
|
|
*/
|
|
|
|
if (showprogress)
|
|
|
|
{
|
|
|
|
total_size = scan_directory(DataDir, "global", true);
|
|
|
|
total_size += scan_directory(DataDir, "base", true);
|
|
|
|
total_size += scan_directory(DataDir, "pg_tblspc", true);
|
|
|
|
}
|
|
|
|
|
|
|
|
(void) scan_directory(DataDir, "global", false);
|
|
|
|
(void) scan_directory(DataDir, "base", false);
|
|
|
|
(void) scan_directory(DataDir, "pg_tblspc", false);
|
|
|
|
|
|
|
|
if (showprogress)
|
|
|
|
{
|
|
|
|
progress_report(true);
|
|
|
|
fprintf(stderr, "\n"); /* Need to move to next line */
|
|
|
|
}
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
|
|
|
|
printf(_("Checksum operation completed\n"));
|
|
|
|
printf(_("Files scanned: %s\n"), psprintf(INT64_FORMAT, files));
|
|
|
|
printf(_("Blocks scanned: %s\n"), psprintf(INT64_FORMAT, blocks));
|
|
|
|
if (mode == PG_MODE_CHECK)
|
|
|
|
{
|
|
|
|
printf(_("Bad checksums: %s\n"), psprintf(INT64_FORMAT, badblocks));
|
|
|
|
printf(_("Data checksum version: %d\n"), ControlFile->data_checksum_version);
|
|
|
|
|
|
|
|
if (badblocks > 0)
|
|
|
|
exit(1);
|
|
|
|
}
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Finally make the data durable on disk if enabling or disabling
|
|
|
|
* checksums. Flush first the data directory for safety, and then update
|
|
|
|
* the control file to keep the switch consistent.
|
|
|
|
*/
|
|
|
|
if (mode == PG_MODE_ENABLE || mode == PG_MODE_DISABLE)
|
|
|
|
{
|
|
|
|
ControlFile->data_checksum_version =
|
|
|
|
(mode == PG_MODE_ENABLE) ? PG_DATA_CHECKSUM_VERSION : 0;
|
2018-04-05 21:57:26 +02:00
|
|
|
|
2019-03-23 00:37:36 +01:00
|
|
|
if (do_sync)
|
|
|
|
{
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_info("syncing data directory");
|
|
|
|
fsync_pgdata(DataDir, PG_VERSION_NUM);
|
2019-03-23 00:37:36 +01:00
|
|
|
}
|
2018-04-05 21:57:26 +02:00
|
|
|
|
Unified logging system for command-line programs
This unifies the various ad hoc logging (message printing, error
printing) systems used throughout the command-line programs.
Features:
- Program name is automatically prefixed.
- Message string does not end with newline. This removes a common
source of inconsistencies and omissions.
- Additionally, a final newline is automatically stripped, simplifying
use of PQerrorMessage() etc., another common source of mistakes.
- I converted error message strings to use %m where possible.
- As a result of the above several points, more translatable message
strings can be shared between different components and between
frontends and backend, without gratuitous punctuation or whitespace
differences.
- There is support for setting a "log level". This is not meant to be
user-facing, but can be used internally to implement debug or
verbose modes.
- Lazy argument evaluation, so no significant overhead if logging at
some level is disabled.
- Some color in the messages, similar to gcc and clang. Set
PG_COLOR=auto to try it out. Some colors are predefined, but can be
customized by setting PG_COLORS.
- Common files (common/, fe_utils/, etc.) can handle logging much more
simply by just using one API without worrying too much about the
context of the calling program, requiring callbacks, or having to
pass "progname" around everywhere.
- Some programs called setvbuf() to make sure that stderr is
unbuffered, even on Windows. But not all programs did that. This
is now done centrally.
Soft goals:
- Reduces vertical space use and visual complexity of error reporting
in the source code.
- Encourages more deliberate classification of messages. For example,
in some cases it wasn't clear without analyzing the surrounding code
whether a message was meant as an error or just an info.
- Concepts and terms are vaguely aligned with popular logging
frameworks such as log4j and Python logging.
This is all just about printing stuff out. Nothing affects program
flow (e.g., fatal exits). The uses are just too varied to do that.
Some existing code had wrappers that do some kind of print-and-exit,
and I adapted those.
I tried to keep the output mostly the same, but there is a lot of
historical baggage to unwind and special cases to consider, and I
might not always have succeeded. One significant change is that
pg_rewind used to write all error messages to stdout. That is now
changed to stderr.
Reviewed-by: Donald Dong <xdong@csumb.edu>
Reviewed-by: Arthur Zakirov <a.zakirov@postgrespro.ru>
Discussion: https://www.postgresql.org/message-id/flat/6a609b43-4f57-7348-6480-bd022f924310@2ndquadrant.com
2019-04-01 14:24:37 +02:00
|
|
|
pg_log_info("updating control file");
|
|
|
|
update_controlfile(DataDir, ControlFile, do_sync);
|
2018-04-05 21:57:26 +02:00
|
|
|
|
Add options to enable and disable checksums in pg_checksums
An offline cluster can now work with more modes in pg_checksums:
- --enable enables checksums in a cluster, updating all blocks with a
correct checksum, and updating the control file at the end.
- --disable disables checksums in a cluster, updating only the control
file.
- --check is an extra option able to verify checksums for a cluster, and
the default used if no mode is specified.
When running --enable or --disable, the data folder gets fsync'd for
durability, and then it is followed by a control file update and flush
to keep the operation consistent should the tool be interrupted, killed
or the host unplugged. If no mode is specified in the options, then
--check is used for compatibility with older versions of pg_checksums
(named pg_verify_checksums in v11 where it was introduced).
Author: Michael Banck, Michael Paquier
Reviewed-by: Fabien Coelho, Magnus Hagander, Sergei Kornilov
Discussion: https://postgr.es/m/20181221201616.GD4974@nighthawk.caipicrew.dd-dns.de
2019-03-23 00:12:55 +01:00
|
|
|
if (verbose)
|
|
|
|
printf(_("Data checksum version: %d\n"), ControlFile->data_checksum_version);
|
|
|
|
if (mode == PG_MODE_ENABLE)
|
|
|
|
printf(_("Checksums enabled in cluster\n"));
|
|
|
|
else
|
|
|
|
printf(_("Checksums disabled in cluster\n"));
|
|
|
|
}
|
2018-04-05 21:57:26 +02:00
|
|
|
|
|
|
|
return 0;
|
|
|
|
}
|