Concurrent program to bounce Mobile Warehouse Applications (MWA)

Scope: Create a Concurrent program to bounce Mobile Warehouse Applications (MWA)

Shell Program for MWA Bounce on the Server

First, create a shell script on the Application server.

 #!/bin/bash
# mwa_bounce.sh 03/29/2016 RG

echo “Following are System Parameters”
echo “~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~”
p0=$0
p1=$1
p2=$2
p3=$3
p4=$4
echo “1st System Parameter :”$p0
#echo “2nd System Parameter :”$p1
echo “3rd System Parameter :”$p2
echo “4th System Parameter :”$p3
echo “5th System Parameter :”$p4
echo “~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~”
echo “~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~”
echo “Following are User Parameters “
echo “~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~”
#u1=$5
#echo “1st User Parameter :”$u1
echo “~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~”

echo “Stopping MWA Server via “$p0
echo
nohup $ADMIN_SCRIPTS_HOME/mwactlwrpr.sh stop $p1
echo $ADMIN_SCRIPTS_HOME/mwactlwrpr.sh stop APPS

mail -s “Concurrent Request “$p4” Submitted to Bounce MWA by “$p3 ramesh.ganesan@mycompany.com< /dev/null

sleep 30

# Kill MWADIS

for pids in `ps -fu appldev|grep MWADIS|grep -v “grep”|awk ‘{print $2}’`
do
echo “forcing kill on MWADIS PID ${pids}”
kill -9 ${pids}
done

# Kill telnet
for pids in `ps -fu appldev|grep telnet|grep -v “grep”|awk ‘{print $2}’`
do
#DTL=`ps -fu appldev|grep telnet|grep -v “grep”|awk ‘{print $15}’` # 11i
DTL=`ps -fu appldev|grep telnet|grep -v “grep”|awk ‘{print $17}’` # R12.2
echo “forcing kill on PID ${pids} for ${DTL}”
kill -9 ${pids}
done

sleep 30

echo “Starting MWA Server via “$p0
echo $ADMIN_SCRIPTS_HOME/mwactlwrpr.sh start APPS
echo
rm -rf /tmp/mwa_alert.txt
touch /tmp/mwa_alert.txt
nohup $ADMIN_SCRIPTS_HOME/mwactlwrpr.sh start $p1 |tee /tmp/mwa_alert.txt

# if the dispatcher isn’t free, email me

MWAERROR=`cat /tmp/mwa_alert.txt | grep “is not free to start the dispatcher”`

if [ ! -z “$MWAERROR” ]
then
echo $MWAERROR | mail -s “MWA Error” ramesh.ganesan@brakepartsinc.com
fi

Copy the program to $CUSTOM_TOP/bin and create Soft link

cd $XXBPI_TOP/bin
[appldev@uswodapp013 bin]$ ln -s /ua4001/appl/fnd/12.0.0/bin/fndcpesr mwa_bounce

Create Concurrent Program Executable

As SYSADMIN, define the Executable via Concurrent->Program->Executable

mwa1

Create Concurrent Program Definition

sysadmin, define the Executable via Concurrent->Program->Define

mwa2

Assign a Request Group

Security->Responsibility->Request

mwa3

Submit the Request and view Log file and verify Email

mwa4

 

Log file

mwa5

Email output

mwa6

Program Definition and Request Group can be downloaded and uploaded to other EBS instances with MWA

Program Definition Download

FNDLOAD apps/apps O Y DOWNLOAD $FND_TOP/patch/115/import/afcpprog.lct xxbpi_mwa_bounce_prgm.ldt PROGRAM APPLICATION_SHORT_NAME=”XXBPI_MWA_BOUNCE” CONCURRENT_PROGRAM_NAME=”XXBPI_MWA_BOUNCE”

Upload

FNDLOAD apps/apps O Y UPLOAD $FND_TOP/patch/115/import/afcpprog.lct xxbpi_mwa_bounce_prgm.ldt

Request Group Download

FNDLOAD apps/apps O Y DOWNLOAD $FND_TOP/patch/115/import/afcpreqg.lct xxbpi_system_admin_reports_reqgrp.ldt REQUEST_GROUP REQUEST_GROUP_NAME=”System Administrator Reports” APPLICATION_SHORT_NAME=”FND”

Upload

FNDLOAD apps/apps O Y UPLOAD $FND_TOP/patch/115/import/afcpreqg.lct xxbpi_system_admin_reports_reqgrp.ldt
Advertisements

R12-Increase Java Heap Size and JVM

Current Settings

[applmgr@uswodapq03 scripts]$ ./adoacorectl.sh status | grep “OC4J:oacore” | awk -F”|” ‘{print $3}’ | while read L ; do ps -fp $L | grep -v PID ; done | awk -F”-X” ‘{print $2 $3}’
mx512M ms128M

Shows we have 1 JVM with 512 as MX and 128 as MS.

Scope is to increase the JVM to 2 and Increase the Memory size.

Steps – Modify $Context_File

[applmgr@uswodapq03 ~]$ grep oacore_nprocs $CONTEXT_FILE
<oacore_nprocs oa_var=”s_oacore_nprocs”>1</oacore_nprocs>

[applmgr@uswodapq03 ~]$ grep s_oacore_jvm_start_options $CONTEXT_FILE
<oacore_jvm_start_options oa_var=”s_oacore_jvm_start_options”>-server -verbose:gc -Xmx512M -Xms128M -XX:MaxPermSize=160M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Dcom.sun.management.jmxremote -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/ebsqa/applmgr/inst/apps/ebsqa_uswodapq03/ora/10.1.3/j2ee/oacore/config/jazn.xml</oacore_jvm_start_options>

—–change the heap size as below and increase jvm to 2.
-Xmx1024M -Xms512M -XX:MaxPermSize=256M

[applmgr@uswodapq03 admin]$ grep s_oacore_jvm_start_options $CONTEXT_FILE
<oacore_jvm_start_options oa_var=”s_oacore_jvm_start_options”>-server -verbose:gc -Xmx1024M -Xms512M -XX:MaxPermSize=256M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+UseTLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Dcom.sun.management.jmxremote -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/ebsqa/applmgr/inst/apps/ebsqa_uswodapq03/ora/10.1.3/j2ee/oacore/config/jazn.xml</oacore_jvm_start_options>

[applmgr@uswodapq03 admin]$ grep oacore_nprocs $CONTEXT_FILE
<oacore_nprocs oa_var=”s_oacore_nprocs”>2</oacore_nprocs>

Stop and Start Application Services

cd $ADMIN_SCRIPTS_HOME
./adautocfg.sh

adopmnctl.sh stopall
adopmnctl.sh startall

After Changes- increased Heap memory with 2 JVM
[applmgr@uswodapq03 scripts]$ ./adoacorectl.sh status | grep “OC4J:oacore” | awk -F”|” ‘{print $3}’ | while read L ; do ps -fp $L | grep -v PID ; done | awk -F”-X” ‘{print $2 $3}’
mx1024M ms512M
mx1024M ms512M
Emergency fix – quick solution
Just in case you need to fix an issue quickly, you may want to edit $INST_TOP/ora/10.1.3/opmn/conf/opmn.xml file directly and restart OPMN processes to enable new parameters.

<data id=”java-options” value=”-server -verbose:gc -Xmx1024M -Xms512M -XX:MaxPermSize=256M -XX:NewRatio=2 -XX:+PrintGCTimeStamps -XX:+Use
TLAB -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Dcom.sun.management.jmxremote -Djava.security.policy=$ORACLE_HOME/j2ee/oacore/config/java2.policy -Djava.awt
.headless=true -Dhttp.webdir.enable=false -Doracle.security.jazn.config=/ebsqa/applmgr/inst/apps/ebsqa_uswodapq03/ora/10.1.3/j2ee/oacore/config/jazn.xml -Dhttp
.cookie.ignoreCommaInCookiesNamed=X_NoMatchingCookies”/>

<process-type id=”oacore” module-id=”OC4J” status=”enabled” working-dir=”$ORACLE_HOME/j2ee/home”>
<process-set id=”default_group” numprocs=”2″/>
Make sure to edit $CONTEXT_FILE to keep it in sync.

EBS R12 – OPP Out of Memory Error

Cause: Oracle EBS Reports are ending in Warning.
Concurrent Logs having an OPP error:

+—————————————————————–+
Executing request completion options…
Output file size: 4323
+————- 1) PUBLISH ————-+
Unable to find an Output Post Processor service to post-process request 10225428. Check that the Output Post Processor service is running.
+————————————–+

+————- 2) PRINT ————-+
Not printing the output of this request because post-processing failed.
+————————————–+
Finished executing request completion options.
+—————————————————————–+

Check the Concurrent OPP Logs.

NOTE: If 2 threads enabled, then there will be 2 log files.

Logfile Location:

ls -ltr $APPLCSF/$APPLLOG/FNDOPP*
FNDOPP74655.txt
FNDOPP74654.txt

OR

ps -ef | grep OPP

applebs 15024 6778 17 13:00 ? 00:04:32 /us1001/product/10.1.3/appsutil/jdk/bin/java -DCLIENT_PROCESSID=15024 -server -Xmx384m -XX:NewRatio=2 -XX:+UseSerialGC -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc -mx2048m -Ddbcfile=/ua1002/inst/apps/bpiprd_uswodapp002/appl/fnd/12.0.0/secure/bpiprd.dbc -Dcpid=74654 -Dconc_queue_id=6269 -Dqueue_appl_id=0 –Dlogfile=/ua1002/inst/apps/bpiprd_shared/logs/appl/conc/log/FNDOPP74654.txt –DLONG_RUNNING_JVM=true -DOVERRIDE_DBC=true -DFND_JDBC_BUFFER_MIN=1 -DFND_JDBC_BUFFER_MAX=2 oracle.apps.fnd.cp.gsf.GSMServiceController

applebs 15026 6778 4 13:00 ? 00:01:15 /us1001/product/10.1.3/appsutil/jdk/bin/java -DCLIENT_PROCESSID=15026 -server -Xmx384m -XX:NewRatio=2 -XX:+UseSerialGC -Doracle.apps.fnd.common.Pool.leak.mode=stderr:off -verbose:gc -mx2048m dbcfile=/ua1002/inst/apps/bpiprd_uswodapp002/appl/fnd/12.0.0/secure/bpiprd.dbc -Dcpid=74655 -Dconc_queue_id=6269 -Dqueue_appl_id=0 –Dlogfile=/ua1002/inst/apps/bpiprd_shared/logs/appl/conc/log/FNDOPP74655.txt -DLONG_RUNNING_JVM=true -DOVERRIDE_DBC=true -DFND_JDBC_BUFFER_MIN=1 -DFND_JDBC_BUFFER_MAX=2 oracle.apps.fnd.cp.gsf.GSMServiceController

Log has following errors.

[5/18/16 12:57:04 PM] [UNEXPECTED] [74607:RT10225399] java.lang.OutOfMemoryError: Java heap space
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
at java.lang.Class.getDeclaredMethod(Class.java:1935)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1382)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:52)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:438)
at java.security.AccessController.doPrivileged(Native Method)
at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:413)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:310)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1106)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1474)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1392)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1150)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:326)
at oracle.ias.cache.group.HttpPacket.<init>(Unknown Source)
at oracle.ias.cache.group.StreamHandler.write(Unknown Source)
at oracle.ias.cache.group.EndPoint.write(Unknown Source)
at oracle.ias.cache.group.Transport.syncSend(Unknown Source)
at oracle.ias.cache.group.Sender.execute(Unknown Source)
at oracle.ias.cache.group.Sender.run(Unknown Source)

Troubleshooting

Option 1, patching

12i Environments:
1. Please download,review and apply Patch 7687414 in a test environment.
3. Please retest the issue.
4. If the issue is resolved, please migrate the solution as appropriate to other environments.

Option 2, Increase the maximum Java heap size for the OPP to 1024MB (1GB):

Determine what the heap size per OPP process is currently:

select DEVELOPER_PARAMETERS from FND_CP_SERVICES
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = ‘FNDCPOPP’);

-The default should be:

J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx512m

3. Increase the Heap Space per Process to 1024:

update FND_CP_SERVICES
set DEVELOPER_PARAMETERS =
‘J:oracle.apps.fnd.cp.gsf.GSMServiceController:-mx1024m
where SERVICE_ID = (select MANAGER_TYPE from FND_CONCURRENT_QUEUES
where CONCURRENT_QUEUE_NAME = ‘FNDCPOPP’);
OR
1. Log into applications with the System Administrator responsibility.
2. Navigate to Concurrent -> Program -> Define
3. Query the XML Publisher Template Re-Generator program
4. Set the following value for the Executable Options : -Xmx1024m
5. Save changes.
6. Retest the program.
The OPP queue can be Recreated the using $FND_TOP/patch/115/sql/afopp002.sql file as ‘APPLSYS’ user. On running the script you will be prompted for username and password.
Option 3, Check timeout settings.
There are two new profiles options that can be used to control the timeouts:
Profile Option : Concurrent:OPP Response Timeout Internal Name : CONC_PP_RESPONSE_TIMEOUT Description : Specifies the amount of time a manager waits for OPP to respond to its request for post processing.
Profile Option : Concurrent:OPP Process Timeout Internal Name : CONC_PP_PROCESS_TIMEOUT Description : Specifies the amount of time the manager waits for the OPP to actually process the request.
Option 4,  Configure the XML Publisher Administrator Configuration settings.
Note: Reports built with large images will require additional space.  So please insure there is adequate temp space and your code is optimized when running large reports/templates.
1. As XML Publisher Administrator navigate to Administration->Configuration.
2. Under Temporary Directory pick a temporary file location on your concurrent processing node. This should be at least 5GB or 20x larger than largest XML data file you generate
3.  Under FO Processing, set:
o Use XML Publisher’s XSLT processor set to True
o Enable scalable feature of XSLT processor set to False
o Enable XSLT runtime optimization set to True
Bounce OPP processes.
Additional Info: Review below Meta link Documents.
Doc ID 1268217.1
Doc ID 862644.1]
Doc ID 1266368.1
Doc ID 427233.1
Doc ID 352518.1
Doc ID 978495.1

Unable to start OAFM service (time out while waiting for a managed process to start) exiting with status 152

Review below MOS Docs

OC4J Fails to startup with “ERROR Failed to set the internal configuration of the OC4J JMS Server” (Doc ID 372412.1)
OAFM OC4J is Not Starting. Error: “time out while waiting for a managed process to start” (Doc ID 952583.1)

Check Log files under ($LOG_HOME/ora/10.1.3/opmn)
and below logs.

$LOG_HOME/ora/10.1.3/j2ee/oafm/oafm_default_group_1/application.log
$LOG_HOME/ora/10.1.3/j2ee/oafm/oafm_default_group_1/server.log
$LOG_HOME/ora/10.1.3/opmn/default_group~oafm~default_group~1.log
$LOG_HOME/ora/10.1.3/opmn/oafm_default_group_1/oafmstd.out
$LOG_HOME/ora/10.1.3/opmn/oafm_default_group_1/oafmstd.err
$LOG_HOME/appl/admin/log/adoafmctl.txt
2017-09-14_111520

2017-09-14_111850

Issue resolved after doing below actions:

Clear the TLD cache:

  1. Stop all middle tier services
  2. Delete/backup all the files under directory:

$COMMON_TOP/_TldCache
$ORA_CONFIG_HOME/10.1.3/j2ee/oafm/persistence

cd $COMMON_TOP
mv $COMMON_TOP/_TldCache /home/applprd/oafm_log_backup041116/

  1.      cd $ORA_CONFIG_HOME/10.1.3/j2ee/oafm/
    mv persistence /home/applprd/oafm_log_backup041116/

3. start all middle tier services

cd $ADMIN_SCRIPTS_HOME
$ adoafmctl.sh stop
$ adoafmctl.sh start
$ adoacorectl.sh stop
$ adoacorectl.sh start

$ adopmnctl.sh status” You should see all as “Alive”

AWS Cloud and Oracle Database

Amazon Web Services (AWS) provides 2 types of services.

  1. Amazon Relational Database Service (RDS) which is fully managed service. Few clicks you got the Database up and running and scalable with few clicks. All the backup’s, patching, storage and high availability are managed by AWS.
  2. Amazon Elastic Compute Cloud (EC2) allows full control over setup of infrastructure and database. So we will be in charge of backup’s, patches, storage provisioning…etc.

There are 2 options of pricing, one with Amazon’s Oracle license included and the other with Bring your own license. Prices are reasonable and it’s per hour usage. https://aws.amazon.com/rds/oracle/pricing/

Amazon RDS and EC2 Available for Oracle, MySql, Sql server. OS, Linux, RHEL, SLES, Windows, SQL are available.

Amazon Relational Database Service (RDS) is a managed database service. RDS makes it easy to set up, operate, and scale Oracle Database deployments in the cloud. With Amazon RDS, you can deploy multiple editions of Oracle Database in minutes with cost-efficient and re-sizable hardware capacity. Amazon RDS frees you up to focus on application development by managing time-consuming database administration tasks including provisioning, backups, software patching, monitoring, and hardware scaling.

Amazon EC2, has 3 types of pricing involved depending on the type of instance you choose (On-demand, Reserved, and Spot Instances) and all are pay for what you use method.

AWS free usage FAQ – https://aws.amazon.com/free/faqs/

Attached are 2 examples of Oracle DB setup using EC2 and RDS. Also attached few PDF for reference.

Create instance in EC2 in AWSCreate Oracle RDS instance in Amazon

Create Oracle RDS instance in Amazon

adautocfg.sh and adcfglonce.pl dbconfig – fails with ORA-01157 Lock Datafile

When executed Auto config  and Clone Config after RMAN Duplicate, following error occured.

ERROR: InDbCtxFile.uploadCtx() : Exception : Error executng BEGIN fnd_gsm_util.append_ctx_fragment(:1,:2,:3); END;: 1; Oracle error -1157: ORA-01157: cannot identify/lock data file 515 – see DBWR trace file
ORA-01110: data file 515: ‘+USUAT_DATA’ has been detected in FND_GSM_UTIL.APPEND_CTX_FRAGMENT.oracle.apps.ad.autoconfig.oam.InDbCtxFileException: Error executng BEGIN fnd_gsm_util.append_ctx_fragment(:1,:2,:3); END;: 1; Oracle error -1157: ORA-01157: cannot identify/lock data file 515 – see DBWR trace file
ORA-01110: data file 515: ‘+USUAT_DATA’ has been detected in FND_GSM_UTIL.APPEND_CTX_FRAGMENT.

 

[orajzuat@usuatsrvdbd1bin]$ export PATH=/usr/bin:$PATH
[orajzuat@usuatsrvdbd1bin]$ perl adcfgclone.pl dbconfig /us9001/oracle/11.2.0/appsutil/bpijzuat_usuatsrvdbd1.xml

Beginning dbconfig Apply – Fri Jan 29 14:53:12 2016
…….
  |     30% completed       ERROR: InDbCtxFile.uploadCtx() : Exception : Error executng BEGIN fnd_gsm_util.append_ctx_fragment(:1,:2,:3); END;: 1; Oracle error -1157: ORA-01157: cannot identify/lock data file 515 – see DBWR trace file
ORA-01110: data file 515: ‘+USUAT_DATA’ has been detected in FND_GSM_UTIL.APPEND_CTX_FRAGMENT.
oracle.apps.ad.autoconfig.oam.InDbCtxFileException: Error executng BEGIN fnd_gsm_util.append_ctx_fragment(:1,:2,:3); END;: 1; Oracle error -1157: ORA-01157: cannot identify/lock data file 515 – see DBWR trace file
ORA-01110: data file 515: ‘+USUAT_DATA’ has been detected in FND_GSM_UTIL.APPEND_CTX_FRAGMENT.
        at oracle.apps.ad.autoconfig.oam.InDbCtxFile.uploadCtx(InDbCtxFile.java:249)
        at oracle.apps.ad.autoconfig.oam.CtxSynchronizer.uploadToDb(CtxSynchronizer.java:328)
        at oracle.apps.ad.tools.configuration.FileSysDBCtxMerge.updateDBCtx(FileSysDBCtxMerge.java:721)
        at oracle.apps.ad.tools.configuration.FileSysDBCtxMerge.updateDBFiles(FileSysDBCtxMerge.java:226)
        at oracle.apps.ad.context.CtxValueMgt.processCtxFile(CtxValueMgt.java:1690)
        at oracle.apps.ad.clone.ApplyDatabase.runCVM(ApplyDatabase.java:3058)
        at oracle.apps.ad.clone.ApplyDatabase.runCVMAndAutoConfig(ApplyDatabase.java:3003)
        at oracle.apps.ad.clone.ApplyDatabase.doConf(ApplyDatabase.java:649)
        at oracle.apps.ad.clone.ApplyDatabase.doApply(ApplyDatabase.java:473)
        at oracle.apps.ad.clone.ApplyDatabase.<init>(ApplyDatabase.java:366)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:501)
        at oracle.apps.ad.clone.util.CloneProcessor.run(CloneProcessor.java:67)
        at java.lang.Thread.run(Thread.java:637)
  –     30% completed
Completed Apply…
Fri Jan 29 14:54:23 2016
Starting database listener for bpijzuat:
Running:
/us9001/oracle/11.2.0/appsutil/scripts/bpijzuat_usuatsrvdbd1/addlnctl.sh start bpijzuat
Logfile: /us9001/oracle/11.2.0/appsutil/log/bpijzuat_usuatsrvdbd1/addlnctl.txt
You are running addlnctl.sh version 120.1.12010000.4
Starting listener process bpijzuat …
Listener bpijzuat has already been started.
addlnctl.sh: exiting with status 0
addlnctl.sh: check the logfile /us9001/oracle/11.2.0/appsutil/log/bpijzuat_usuatsrvdbd1/addlnctl.txt for more information …
[orajzuat@usuatsrvdbd1bin]$

Cause: This error occurs when there is missing or corrupted TEMP tablespace tempfile

Solution:  Drop and Re-create Tempfile.

Current Temp Tablespace:

select file#,name,ts# from v$tempfile;

FILE# NAME  TS#
1     +USUAT_TMP/usuat/tempfile/temp3.257.902419859   292
2     +USUAT_TMP/usuat/tempfile/temp4.256.902419677   393

select name from v$tablespace where ts# in (292,393)

NAME
Temp1
Temp2

We can see the same error when trying to select the size of the tempfile

select file_name,bytes/1024/1024/1024 from dba_temp_files where tablespace_name=’TEMP1′;

select file_name,bytes/1024/1024/1024 from dba_temp_files where tablespace_name=’TEMP2′;

–ORA-01157: cannot identify/lock data file 513 – see DBWR trace file
–ORA-01110: data file 513: ‘+USUAT_DATA’

 Note: Same error as the adconfig

Create temporary tablespace TEMP3 tempfile ‘+USUAT_TMP’ size 20G;

ALTER DATABASE DEFAULT TEMPORARY TABLESPACE temp3;

drop tablespace TEMP1;

drop tablespace TEMP2;

Now run adautocfg.sh….SUCCESS…. 

 

 

RMAN-11003 and ORA-32010 during cloning

During a recent clone using RMAN Duplicate, RMAN failed with RMAN-11093 and ERROR: ORA-32010: cannot find entry to delete in SPFILE .

–Tail from RMAN Duplicate logfile with the Errors1

Cause:

Looking at MOS Doc ORA-32010: cannot find entry to delete in SPFILE (Doc ID 1384262.1), identifies the cause to “Due to some comments in the pfile, and/or invalid Entries for example missing a single quote to enclose values for parameters as set in pfile/spfile.

I did make a change to PFILE to convert the files from ASM to EXT on the source node. (As below).                *.db_file_name_convert=’+EBSPROD_ASM/’,’/u10/orc/data/’,’+EBSPROD_ASM/ebsprd/datafile/’,/u10/orc/data/’

Looking at it again, clearly shows the extra tick.

        SQL>  show parameter db_file_name_convert
                    NAME                                 TYPE            VALUE
                   ——————————
                   db_file_name_convert   string     +EBSPRD_ASM/, /u20/oracle/data/,                                                        +EBSPRD_ASM/ebsprd/datafile/, /u20/oracle/data/’

          SQL>

Solution: Manually complete the RMAN Duplicate from this point.

1) BACKUP CONTROL FILE FROM PROD

SQL> alter database backup control file to trace as ‘/HOME/ORACLE/RG_cntrfile.sql’;

2) EDIT THE CONTROL FILE (“SET DATABASE”, change the data files with path of target( you can obtain path from the failed RMAN log file when it laid out the files or manually modifying it, change the DB name). Save the file on clone database node (/home/affebsdev/rg_cntrfile.sql)

–SEE BELOW – contents of rg_cntrfile.sql

 

CREATE CONTROLFILE REUSE SET DATABASE “AFFEBSDEV” RESETLOGS FORCE LOGGING ARCHIVELOG
MAXLOGFILES 32
MAXLOGMEMBERS 5
MAXDATAFILES 512
MAXINSTANCES 8
MAXLOGHISTORY 14607
LOGFILE
GROUP 1 (‘/u10/oracle/data/log01a.dbf’,’/u10/oracle/data/log01b.dbf’) size 1048576512,
GROUP 2 (‘/u10/oracle/data/log02a.dbf’,’/u10/oracle/data/log02b.dbf’) size 1048576512
DATAFILE
‘/u10/oracle/data/system01.dbf’,
‘/u10/oracle/data/system02.dbf’,
‘/u10/oracle/data/system03.dbf’,
‘/u10/oracle/data/system04.dbf’,
‘/u10/oracle/data/system05.dbf’,
‘/u10/oracle/data/ctxd01.dbf’,
‘/u10/oracle/data/owad01.dbf’,
‘/u10/oracle/data/a_queue02.dbf’,
‘/u10/oracle/data/odm.dbf’,
‘/u10/oracle/data/olap.dbf’,
‘/u10/oracle/data/sysaux01.dbf’,
….
….

CHARACTER SET UTF8
;

3) SHUTDOWN THE CLONE DATABASE INSTANCE

 SQL>Shutdown Immediate;

4) FIX THE PFILE WITH CORRECT VALUE, then create SPFILE from PFILE.

Vi initaffebdev.ora
Fix the value and save the init file.

5) CREATE SPFILE FROM PFILE

  SQL>sqlplus / as sysdba
            SQL>create spfile from pfile=’ initaffebdev.ora’;

6) STARTUP NOMOUNT;
7) Execute the Controlfile on the Target Node as SYSDBA

SQL> sqlplus / as sysdba @rg_cntrfile.sql

       Control file created.

8) Mount the database

SQL> alter database Mount;
ORA-01100: database already mounted

9) Database can now be opened with reset logs.

SQL> alter database open resetlogs;
Database altered.

11) Re-create temp table space as below

ALTER TABLESPACE TEMP1 ADD TEMPFILE ‘/u10/oracle/data/temp01.dbf’ SIZE 6120M REUSE AUTOEXTEND OFF;

ALTER TABLESPACE TEMP1 ADD TEMPFILE ‘/u10/oracle/data/temp01a.dbf’ SIZE 6120M REUSE AUTOEXTEND OFF;

ALTER TABLESPACE TEMP2 ADD TEMPFILE ‘/u10/oracle/data/temp02.dbf’ SIZE 6120M REUSE AUTOEXTEND OFF;

ALTER TABLESPACE TEMP2 ADD TEMPFILE ‘/u10/oracle/data/temp02a.dbf’ SIZE 6120M REUSE AUTOEXTEND OFF;

12) Verify the status

sqlplus ” / as sysdba”
SQL> select open_mode from v$database;

OPEN_MODE
——————–
READ WRITE

Now continue with cloning/post clone steps J