archive-push problem

[hunterhuang8810]

I want to use archive-push to move pg wal files from archive folder to backup wal folder, i.e. copy all archived wal files from
/home/service/var/pgdata/archivedir to pg_probackup wal archive folder , but I can't run success:

pg_probackup-12 archive-push -B /home/postgres/pg_backup --instance pgsql --wal-file-name=000000010000000000000020 --wal-file-path=/home/service/var/pgdata/archivedir

ERROR: Could not open file "/home/service/var/pgdata/archivedir/global/pg_control" for reading: No such file or directory

How do I use archive-push command to move all archived wal files from one folder to pg_probackup base folder?

[gsmolk]

Hello!

ERROR: Could not open file "/home/service/var/pgdata/archivedir/global/pg_control" for reading: No such file or directory

We are trying to get system ID here before pushing WAL to make sure that we are pushing to the correct repo.
And currently this behaviour cannot be disabled.
It is a known issue and we will fix it in the upcoming -release:
#265

Currently I can only recommend to use cp to solve this problem =(

[hunterhuang8810]

Hello!

ERROR: Could not open file "/home/service/var/pgdata/archivedir/global/pg_control" for reading: No such file or directory

We are trying to get system ID here before pushing WAL to make sure that we are pushing to the correct repo.
And currently this behaviour cannot be disabled.
It is a known issue and we will fix it in the upcoming -release:
#265

Currently I can only recommend to use cp to solve this problem =(

OK , noted

[hunterhuang8810]

BTW,

annother question, I have changed postgres instance archive_timeout to 4min, but the backup still failed.
As no data change, I found wal archive will not triggerred even when after set archive_timeout , so the backup fail. any solution for this?

[postgres@10-176-2-190.rds-app-test.dgdev01 pgsql]$ pg_probackup-12 backup -B /home/postgres/pg_backup --instance pgsql -b full
INFO: Backup start, pg_probackup version: 2.4.9, instance: pgsql, backup ID: QOPQ0B, backup mode: FULL, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: Current PostgreSQL role is superuser. It is not recommended to run backup or checkdb as superuser.
INFO: Backup QOPQ0B is going to be taken from standby
INFO: Wait for WAL segment /home/postgres/pg_backup/wal/pgsql/000000010000000000000024 to be archived
WARNING: By default pg_probackup assume WAL delivery method to be ARCHIVE. If continuous archiving is not set up, use '--stream' option to make autonomous backup. Otherwise check that continuous archiving works correctly.
ERROR: WAL segment 000000010000000000000024 could not be archived in 300 seconds
WARNING: backup in progress, stop backup
WARNING: Backup QOPQ0B is running, setting its status to ERROR
[postgres@10-176-2-190.rds-app-test.dgdev01 pgsql]$

[gsmolk]

Did you set archive_timeout on master?

[hunterhuang8810]

Did you set archive_timeout on master?

yes, I set on all instance, including master and standby

[hunterhuang8810]

how to solve this? the root cause is , if no any data change, pg will not switch wal files, so as for archive_timeout.

postgres=# show archive_timeout ;
archive_timeout

4min
(1 row)

postgres=# select pg_switch_wal();
pg_switch_wal

0/2A000000
(1 row)

postgres=# select pg_switch_wal();
pg_switch_wal

0/2A000000
(1 row)

postgres=#

--I tried stream mode, still need archive. I don't understand why must archive before backup db? And what's the solution for this scenario?
[postgres@10-176-2-190.rds-app-test.dgdev01 pgsql]$ pg_probackup-12 backup -B /home/postgres/pg_backup --instance pgsql -b full --stream
INFO: Backup start, pg_probackup version: 2.4.9, instance: pgsql, backup ID: QOPS81, backup mode: FULL, wal mode: STREAM, remote: false, compress-algorithm: none, compress-level: 1
WARNING: Current PostgreSQL role is superuser. It is not recommended to run backup or checkdb as superuser.
INFO: Backup QOPS81 is going to be taken from standby
INFO: PGDATA size: 142MB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 0
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Wait for WAL segment /home/postgres/pg_backup/backups/pgsql/QOPS81/database/pg_wal/00000001000000000000002A to be streamed
WARNING: WAL segment 00000001000000000000002A could not be streamed in 300 seconds

[gsmolk]

Can you please rerun this backup with --log-level-file=VERBOSE and then attach file here?

[hunterhuang8810]

Can you please rerun this backup with --log-level-file=VERBOSE and then attach file here?

-------backup on standby, all cluster instance archive_timeout=4min, pg_probackup archive_timeout=5min------
[postgres@10-176-2-190.rds-app-test.dgdev01 pgsql]$ pg_probackup-12 show-config -B /home/postgres/pg_backup --instance pgsql

Backup instance information

pgdata = /home/service/var/pgdata
system-identifier = 6926437883201436364
xlog-seg-size = 16777216

Connection parameters

pgdatabase = postgres

Replica parameters

replica-timeout = 5min

Archive parameters

archive-timeout = 5min

Logging parameters

log-level-console = INFO
log-level-file = OFF
log-filename = pg_probackup.log
log-rotation-size = 0TB
log-rotation-age = 0d

Retention parameters

retention-redundancy = 0
retention-window = 0
wal-depth = 0

Compression parameters

compress-algorithm = none
compress-level = 1

Remote access parameters

remote-proto = ssh
[postgres@10-176-2-190.rds-app-test.dgdev01 pgsql]$

[postgres@10-176-2-190.rds-app-test.dgdev01 pgsql]$ pg_probackup-12 backup -B /home/postgres/pg_backup --instance pgsql -b full --log-level-file=VERBOSE
INFO: Backup start, pg_probackup version: 2.4.9, instance: pgsql, backup ID: QOPSTD, backup mode: FULL, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: Current PostgreSQL role is superuser. It is not recommended to run backup or checkdb as superuser.
INFO: Backup QOPSTD is going to be taken from standby
INFO: Wait for WAL segment /home/postgres/pg_backup/wal/pgsql/000000010000000000000029 to be archived
WARNING: By default pg_probackup assume WAL delivery method to be ARCHIVE. If continuous archiving is not set up, use '--stream' option to make autonomous backup. Otherwise check that continuous archiving works correctly.
ERROR: WAL segment 000000010000000000000029 could not be archived in 300 seconds
WARNING: backup in progress, stop backup
WARNING: Backup QOPSTD is running, setting its status to ERROR
[postgres@10-176-2-190.rds-app-test.dgdev01 pgsql]$

[gsmolk]

Please attach /home/postgres/pg_backup/log/pg_probackup.log file.

[hunterhuang8810]

[postgres@10-176-2-190.rds-app-test.dgdev01 ~]$ pg_probackup-12 backup -B /home/postgres/pg_backup --instance pgsql -b full --log-level-file=VERBOSE
INFO: Backup start, pg_probackup version: 2.4.9, instance: pgsql, backup ID: QOPTVM, backup mode: FULL, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
WARNING: Current PostgreSQL role is superuser. It is not recommended to run backup or checkdb as superuser.
INFO: Backup QOPTVM is going to be taken from standby
INFO: Wait for WAL segment /home/postgres/pg_backup/wal/pgsql/00000001000000000000002C to be archived
WARNING: By default pg_probackup assume WAL delivery method to be ARCHIVE. If continuous archiving is not set up, use '--stream' option to make autonomous backup. Otherwise check that continuous archiving works correctly.
ERROR: WAL segment 00000001000000000000002C could not be archived in 300 seconds
WARNING: backup in progress, stop backup
WARNING: Backup QOPTVM is running, setting its status to ERROR
[postgres@10-176-2-190.rds-app-test.dgdev01 ~]$

[postgres@10-176-2-190.rds-app-test.dgdev01 log]$ cat pg_probackup.log
2021-02-18 16:21:22 CST [292130]: INFO: command: pg_probackup-12 backup -B /home/postgres/pg_backup --instance pgsql -b full --log-level-file=VERBOSE
2021-02-18 16:21:22 CST [292130]: INFO: Backup start, pg_probackup version: 2.4.9, instance: pgsql, backup ID: QOPTVM, backup mode: FULL, wal mode: ARCHIVE, remote: false, compress-algorithm: none, compress-level: 1
2021-02-18 16:21:22 CST [292130]: LOG: Backup destination is initialized
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SELECT pg_catalog.current_setting($1)
2021-02-18 16:21:22 CST [292130]: VERBOSE: (param:0) = block_size
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SELECT pg_catalog.current_setting($1)
2021-02-18 16:21:22 CST [292130]: VERBOSE: (param:0) = wal_block_size
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SELECT pg_catalog.current_setting('is_superuser')
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SELECT proname FROM pg_proc WHERE proname='pgpro_edition'
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SELECT pg_catalog.pg_is_in_recovery()
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SHOW data_checksums
2021-02-18 16:21:22 CST [292130]: LOG: This PostgreSQL instance was initialized with data block checksums. Data block corruption will be detected
2021-02-18 16:21:22 CST [292130]: WARNING: Current PostgreSQL role is superuser. It is not recommended to run backup or checkdb as superuser.
2021-02-18 16:21:22 CST [292130]: INFO: Backup QOPTVM is going to be taken from standby
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SELECT system_identifier FROM pg_catalog.pg_control_system()
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SELECT extnamespace::regnamespace, extversion FROM pg_catalog.pg_extension WHERE extname = 'ptrack'
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SELECT proname FROM pg_proc WHERE proname='ptrack_version'
2021-02-18 16:21:22 CST [292130]: LOG: Database backup start
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SELECT pg_catalog.pg_start_backup($1, $2, false)
2021-02-18 16:21:22 CST [292130]: VERBOSE: (param:0) = 2021-02-18 16:21:22+08 with pg_probackup
2021-02-18 16:21:22 CST [292130]: VERBOSE: (param:1) = true
2021-02-18 16:21:22 CST [292130]: VERBOSE: (query) SELECT timeline_id FROM pg_control_checkpoint()
2021-02-18 16:21:22 CST [292130]: LOG: Looking for segment: 00000001000000000000002C
2021-02-18 16:21:23 CST [292130]: INFO: Wait for WAL segment /home/postgres/pg_backup/wal/pgsql/00000001000000000000002C to be archived
2021-02-18 16:21:52 CST [292130]: WARNING: By default pg_probackup assume WAL delivery method to be ARCHIVE. If continuous archiving is not set up, use '--stream' option to make autonomous backup. Otherwise check that continuous archiving works correctly.
2021-02-18 16:26:23 CST [292130]: ERROR: WAL segment 00000001000000000000002C could not be archived in 300 seconds
2021-02-18 16:26:23 CST [292130]: WARNING: backup in progress, stop backup
2021-02-18 16:26:23 CST [292130]: VERBOSE: (query) SET client_min_messages = warning;
2021-02-18 16:26:23 CST [292130]: VERBOSE: (query) SET datestyle = 'ISO, DMY';
2021-02-18 16:26:23 CST [292130]: VERBOSE: (query) SELECT pg_catalog.txid_snapshot_xmax(pg_catalog.txid_current_snapshot()), current_timestamp(0)::timestamptz, pg_catalog.pg_last_wal_replay_lsn(), labelfile, spcmapfile FROM pg_catalog.pg_stop_backup(false, false)
2021-02-18 16:26:23 CST [292130]: WARNING: Backup QOPTVM is running, setting its status to ERROR
[postgres@10-176-2-190.rds-app-test.dgdev01 log]$

[hunterhuang8810]

I hope provide a solution to backup success no dependent on wal archive, backup sucess first.

backup sucess first.

[gsmolk]

Please try this one and attach the file /home/postgres/pg_backup/log/pg_probackup.log:

pg_probackup-12 backup -B /home/postgres/pg_backup --instance pgsql -b full --log-level-file=VERBOSE --stream

[hunterhuang8810]

seems --stream will finally success, though still wait archive failed.

as below, log file contents is long, I not find where to upload file

[postgres@10-176-2-190.rds-app-test.dgdev01 ~]$ pg_probackup-12 backup -B /home/postgres/pg_backup --instance pgsql -b full --log-level-file=VERBOSE --stream
INFO: Backup start, pg_probackup version: 2.4.9, instance: pgsql, backup ID: QOPWNX, backup mode: FULL, wal mode: STREAM, remote: false, compress-algorithm: none, compress-level: 1
WARNING: Current PostgreSQL role is superuser. It is not recommended to run backup or checkdb as superuser.
INFO: Backup QOPWNX is going to be taken from standby
INFO: PGDATA size: 158MB
INFO: Start transferring data files
INFO: Data files are transferred, time elapsed: 0
INFO: wait for pg_stop_backup()
INFO: pg_stop backup() successfully executed
INFO: Wait for WAL segment /home/postgres/pg_backup/backups/pgsql/QOPWNX/database/pg_wal/00000001000000000000002D to be streamed
WARNING: WAL segment 00000001000000000000002D could not be streamed in 300 seconds
WARNING: Failed to get next WAL record after 0/2D000000, looking for previous WAL record
INFO: Syncing backup files to disk
INFO: Backup files are synced, time elapsed: 0
INFO: Validating backup QOPWNX
INFO: Backup QOPWNX data files are valid
INFO: Backup QOPWNX resident size: 174MB
INFO: Backup QOPWNX completed
[postgres@10-176-2-190.rds-app-test.dgdev01 ~]$