Tech Blogs on Oracle Database, Exadata, Oracle Database Appliance(ODA) Oracle Linux ,Solaris: September 2019

I have encountered this issue when i tried to resync my prod database backups with catalog database.
RMAN-20110: set_stamp set_count conflict.
In my case exact culprit were old backups, which were taken to other storage and later removed from storage level. Those backup pieces were creating issues.
Those backups were taken just testing purpose thats why we removed those backup pieces from rman repository.
In case, backups pieces are important. They first can be uncataloged from controlfile then catalog again with catalog database.

Below is step to step procedure to fix this issue.

1. connect to Target and Catalog Database. resync thrown issue.

rman target / catalog catalog_user@serrvice_catalog

Recovery Manager: Release 12.2.0.1.0 - Production on Wed Sep 25 10:10:25 2018

Copyright (c) 1882, 2017, Oracle and/or its affiliates. All rights reserved.

connected to target database: orcldb (DBID=3458362452345)

recovery catalog database Password:

connected to recovery catalog database

RMAN> resync catalog;

starting full resync of recovery catalog

uncatalog BACKUPPIECE /u01/app/oracle/backup/orcldb/orcldb_00uchvh9_1_1 in NOCATALOG mode

RMAN-00571: ===========================================================

RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============

RMAN-00571: ===========================================================

RMAN-03009: failure of resync command on default channel at 09/25/2018 10:11:25

RMAN-20110: set_stamp set_count conflict

RMAN> exit

2. Generate RMAN debug trace. rman target / catalog catalog_user@serrvice_catalog debug trace=debugtrc.log

you will analyze backuppiece which is culprit.

DBGRCVCAT: addBackupControlfile set stamp set count conflict rec- bs_key 6184997 bcf_key 6185044 dbinc_key 2 ckp_scn 1075157554 ckp_time 24-SEP-18 block_size 16384

controlfile_type B blocks 682 pdb_key 4

DBGRPC: krmxrpc - channel default kpurpc2 err=0 db=rcvcat proc=RMAN.DBMS_RCVCAT.CANCELCKPT excl: 0

DBGRCVCAT: cancelCkpt - rollback, release locks

DBGRPC: krmxrpc - channel default kpurpc2 err=0 db=target proc=SYS.DBMS_BACKUP_RESTORE.CFILEUSECURRENT excl: 0

DBGMISC: ENTERED krmzlog [10:20:37.707]

RMAN-08182: uncatalog BACKUPPIECE /u01/app/oracle/backup/orcldb/orcldb_00uchvh9_1_1 in NOCATALOG mode <<<<<<

DBGMISC: EXITED krmzlog [10:20:37.707] elapsed time [00:00:00:00.000]

DBGPLSQL: EXITED resync with status set_stamp set_count conflict [10:20:37.708]

DBGRPC: krmxr - channel default returned from peicnt

Cause:- backups were deleted from OS level and were cataloged in controlfile only. That was the reason not to get sync with catalog database.

3. Connect to target database and crosscheck backups and check if culprit backuppiece is marked as expired. Delete the expired backuppiece.

RMAN> crosscheck backup;
crosschecked backup piece: found to be 'EXPIRED'

backup piece handle=/u01/app/oracle/backup/orcldb/orcldb_vtucg26s_1_1 RECID=2043 STAMP=1018742428

crosschecked backup piece: found to be 'EXPIRED'

backup piece handle=/u01/app/oracle/backup/orcldb/orcldb_00uchvh9_1_1 RECID=2045 STAMP=1018805257

crosschecked backup piece: found to be 'EXPIRED'

backup piece handle=/u01/app/oracle/backup/orcldb/orcldb_00uchvh9_1_1 RECID=2045 STAMP=1018805257

RMAN> list expired backup;
List of Backup Sets
===================
BS Key Size Device Type Elapsed Time Completion Time

------- ---------- ----------- ------------ ---------------

2043 5.61G DISK 08:06:52 23-SEP-18

BP Key: 2043 Status: EXPIRED Compressed: NO Tag: TAG20180923T134708

Piece Name: /u01/app/oracle/backup/orcldb/orcldb_vtucg26s_1_1

List of Archived Logs in backup set 2043

Thrd Seq Low SCN Low Time Next SCN Next Time

---- ------- ---------- --------- ---------- ---------

1 10339 1070243603 22-SEP-18 1070402884 22-SEP-18

1 10340 1070402884 22-SEP-18 1070545223 22-SEP-18

1 10341 1070545223 22-SEP-18 1070726539 22-SEP-18

1 10342 1070726539 22-SEP-18 1070832029 22-SEP-18

1 10343 1070832029 22-SEP-18 1070910548 22-SEP-18

1 10344 1070910548 22-SEP-18 1070989441 22-SEP-18

1 10345 1070989441 22-SEP-18 1071004560 22-SEP-18

1 10346 1071004560 22-SEP-18 1071168264 22-SEP-18

1 10347 1071168264 22-SEP-18 1071314697 22-SEP-18

1 10348 1071314697 22-SEP-18 1071456764 22-SEP-18

1 10349 1071456764 22-SEP-18 1071581124 22-SEP-18

1 10350 1071581124 22-SEP-18 1071740800 23-SEP-18

1 10351 1071740800 23-SEP-18 1071860861 23-SEP-18

1 10352 1071860861 23-SEP-18 1072008288 23-SEP-18

1 10353 1072008288 23-SEP-18 1072146065 23-SEP-18

1 10354 1072146065 23-SEP-18 1072289017 23-SEP-18

1 10355 1072289017 23-SEP-18 1072425250 23-SEP-18

1 10356 1072425250 23-SEP-18 1072499339 23-SEP-18

1 10357 1072499339 23-SEP-18 1072588829 23-SEP-18

1 10358 1072588829 23-SEP-18 1072735673 23-SEP-18

1 10359 1072735673 23-SEP-18 1072823732 23-SEP-18

1 10360 1072823732 23-SEP-18 1072828012 23-SEP-18

1 10361 1072828012 23-SEP-18 1072970050 23-SEP-18

1 10362 1072970050 23-SEP-18 1073045940 23-SEP-18

BS Key Type LV Size Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ ---------------

2044 Full 10.70M DISK 00:03:01 24-SEP-18

BP Key: 2045 Status: EXPIRED Compressed: NO Tag: TAG20180924T071344

Piece Name: /u01/app/oracle/backup/orcldb/orcldb_00uchvh9_1_1

Control File Included: Ckp SCN: 1075157554 Ckp time: 24-SEP-18

BS Key Type LV Size Device Type Elapsed Time Completion Time

------- ---- -- ---------- ----------- ------------ ---------------

2045 Full 10.70M DISK 00:03:01 24-SEP-18

BP Key: 2045 Status: EXPIRED Compressed: NO Tag: TAG20180924T071344

Piece Name: /u01/app/oracle/backup/orcldb/orcldb_00uchvh9_1_1

Control File Included: Ckp SCN: 1075157554 Ckp time: 24-SEP-18

RMAN> delete expired backup;
using channel ORA_DISK_1
List of Backup Pieces

BP Key BS Key Pc# Cp# Status Device Type Piece Name

------- ------- --- --- ----------- ----------- ----------

2043 2043 1 1 EXPIRED DISK /u01/app/oracle/backup/orcldb/orcldb_vtucg26s_1_1

2045 2044 1 1 EXPIRED DISK /u01/app/oracle/backup/orcldb/orcldb_00uchvh9_1_1

2045 2045 1 1 EXPIRED DISK /u01/app/oracle/backup/orcldb/orcldb_00uchvh9_1_1

Do you really want to delete the above objects (enter YES or NO)? yes
deleted backup piece
backup piece handle=/u01/app/oracle/backup/orcldb/orcldb_vtucg26s_1_1 RECID=2043 STAMP=1018742428
deleted backup piece
backup piece handle=/u01/app/oracle/backup/orcldb/orcldb_00uchvh9_1_1 RECID=2045 STAMP=1018805257
deleted backup piece
backup piece handle=/u01/app/oracle/backup/orcldb/orcldb_00uchvh9_1_1 RECID=2045 STAMP=1018805257

Deleted 3 EXPIRED objects

4. Connect to target and Catalog database and resync catalog. It should work now.

rman target / catalog catalog_user@serrvice_catalog

Recovery Manager: Release 12.2.0.1.0 - Production on Wed Sep 25 11:22:52 2018
Copyright (c) 1882, 2017, Oracle and/or its affiliates. All rights reserved.

connected to target database: orcldb (DBID=3458362452345)

recovery catalog database Password:

connected to recovery catalog database

RMAN> resync catalog;
starting full resync of recovery catalog
full resync complete

Reference MOS doc :-

RMAN Resync of Recovery Catalog Fails with - RMAN-03014 and RMAN-20110: Set_stamp Set_Count Conflict (Doc ID 2453569.1)

#Alert from OEM.

Target type=Oracle WebLogic Server

Target name=/EMGC_GCDomain/GCDomain/EMGC_OMS1

Categories=Capacity

Message=The heap usage is 92%.

Severity=Critical

Event Type=Metric Alert

Event name=jvm:heapUsedPercentage.value

Metric Group=JVM Metrics

Metric=Heap Usage (%)

Metric value=92

Key Value=

Rule Name=Incident management rule set for all DBA related targets,Create incident for critical metric alerts

Rule Owner=SYSMAN

Update Details:

The heap usage is 92%.

#Solution.

1. Stop OMS
$OMS_HOME/bin>./emctl stop oms -all -force

2. Have a backup of $EM_INSTANCE_BASE/gc_inst/user_projects/domains/GCDomain/bin/startEMServer.sh file

Update the following section:
From

USER_MEM_ARGS="-Xms256M -Xmx1740M -XX:PermSize=128M -XX:MaxPermSize=1024M"

if [ "${JAVA_VENDOR}" = "Sun" ] ; then

if [ "${PRODUCTION_MODE}" = "" ] ; then

USER_MEM_ARGS="-Xms256M -Xmx1740M -XX:PermSize=128M -XX:MaxPermSize=1024M -XX:CompileThreshold=8000 -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -

XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled"

TO

USER_MEM_ARGS="-Xms256M -Xmx4096M -XX:PermSize=128M -XX:MaxPermSize=1024M"

if [ "${JAVA_VENDOR}" = "Sun" ] ; then

if [ "${PRODUCTION_MODE}" = "" ] ; then

USER_MEM_ARGS="-Xms256M -Xmx4096M -XX:PermSize=128M -XX:MaxPermSize=1024M -XX:CompileThreshold=8000 -XX:-DoEscapeAnalysis -XX:+UseCodeCacheFlushing -

XX:ReservedCodeCacheSize=100M -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSClassUnloadingEnabled"

Note that we have Replaced Xmx value of 1740 to 4096 in 2 places above. if issue still persist then increse the values to next optimal value.

3. Start OMS
$OMS_HOME/bin>./emctl start oms

#Reference.

EM 13C : Heap usage Warning or Critical Alerts for EMGC_ADMINSERVER Target (Doc ID 2320739.1)

Tech Blogs on Oracle Database, Exadata, Oracle Database Appliance(ODA) Oracle Linux ,Solaris

Thursday 26 September 2019

RMAN-20110: set_stamp set_count conflict

The heap usage is 92%.

OEM Alert:Host : hostname.localMessage: Loader Throughput (rows per second) for Loader_D crossed the critical threshold (75). Current value: 45.02

Blog Archive