Power off : DB => Storage => RACK with RoCE switches
crsctl stop cluster -all
dcli -l root -g <db_group> shu -h now
dcli -l root -g <cell_group> shu -h nowPower on : RoCE switches => Storage => DB
Power on RACK and wait for few minutes => Check cell and disks status => Start cluster and database
After replacing new disks by replacing old disks in exadata, the status changed to “importForceRequired”
CellCLI> list celldisk
CD_CELL001_xstcell importForceRequired
CD_CELL002_xstcell importForceRequired
CD_CELL003_xstcell importForceRequired
CD_CELL004_xstcell importForceRequired
CD_CELL005_xstcell importForceRequired
CD_CELL006_xstcell importForceRequired
CD_CELL007_xstcell importForceRequired
CD_CELL008_xstcell importForceRequired
CD_CELL009_xstcell importForceRequired
CD_CELL010_xstcell importForceRequired
CD_CELL011_xstcell importForceRequired
CD_CELL012_xstcell importForceRequired
CD_CELL013_xstcell importForceRequired
CD_CELL014_xstcell importForceRequired
CD_CELL015_xstcell importForceRequired
CD_CELL016_xstcell importForceRequired
CD_CELL017_xstcell importForceRequired
CD_CELL018_xstcell importForceRequired
CD_CELL019_xstcell importForceRequired
CD_CELL020_xstcell importForceRequired
CD_CELL021_xstcell importForceRequired
CD_CELL022_xstcell importForceRequired
CD_CELL023_xstcell importForceRequired
FD_00_xstcell importForceRequired
FD_01_xstcell importForceRequired
FD_02_xstcell importForceRequired
FD_03_xstcell importForceRequired
FD_04_xstcell importForceRequired
FD_05_xstcell importForceRequired
Check if all disks are printed from OS using a shell script.
[root@x3cel00 raw]# for disks in `cellcli -e list celldisk|awk '{print $1}'`;do echo $disks;done
CD_CELL001_xstcell
CD_CELL002_xstcell
CD_CELL003_xstcell
CD_CELL004_xstcell
CD_CELL005_xstcell
CD_CELL006_xstcell
CD_CELL007_xstcell
CD_CELL008_xstcell
CD_CELL009_xstcell
CD_CELL010_xstcell
CD_CELL011_xstcell
CD_CELL012_xstcell
CD_CELL013_xstcell
CD_CELL014_xstcell
CD_CELL015_xstcell
CD_CELL016_xstcell
CD_CELL017_xstcell
CD_CELL018_xstcell
CD_CELL019_xstcell
CD_CELL020_xstcell
CD_CELL021_xstcell
CD_CELL022_xstcell
CD_CELL023_xstcell
FD_00_xstcell
FD_01_xstcell
FD_02_xstcell
FD_03_xstcell
FD_04_xstcell
FD_05_xstcell
Use the command “import celldisk cell_disk_name force” to force the import of the disks.
[root@x3cel00 raw]# for disk in `cellcli -e list celldisk|awk '{print $1}'`;do cellcli -e import celldisk $disk force;done
CellDisk CD_CELL001_xstcell successfully imported
CellDisk CD_CELL002_xstcell successfully imported
CellDisk CD_CELL003_xstcell successfully imported
CellDisk CD_CELL004_xstcell successfully imported
CellDisk CD_CELL005_xstcell successfully imported
CellDisk CD_CELL006_xstcell successfully imported
CellDisk CD_CELL007_xstcell successfully imported
CellDisk CD_CELL008_xstcell successfully imported
CellDisk CD_CELL009_xstcell successfully imported
CellDisk CD_CELL010_xstcell successfully imported
CellDisk CD_CELL011_xstcell successfully imported
CellDisk CD_CELL012_xstcell successfully imported
CellDisk CD_CELL013_xstcell successfully imported
CellDisk CD_CELL014_xstcell successfully imported
CellDisk CD_CELL015_xstcell successfully imported
CellDisk CD_CELL016_xstcell successfully imported
CellDisk CD_CELL017_xstcell successfully imported
CellDisk CD_CELL018_xstcell successfully imported
CellDisk CD_CELL019_xstcell successfully imported
CellDisk CD_CELL020_xstcell successfully imported
CellDisk CD_CELL021_xstcell successfully imported
CellDisk CD_CELL022_xstcell successfully imported
CellDisk CD_CELL023_xstcell successfully imported
CellDisk FD_00_xstcell successfully imported
CellDisk FD_01_xstcell successfully imported
CellDisk FD_02_xstcell successfully imported
CellDisk FD_03_xstcell successfully imported
CellDisk FD_04_xstcell successfully imported
CellDisk FD_05_xstcell successfully imported
After force import, the celldisk status changed to normal.
[root@x3cel00 raw]# cellcli
CellCLI: Release 11.2.3.2.1 - Production on Tue Jul 18 16:20:48 GMT+05:30 2023
Copyright (c) 2007, 2012, Oracle. All rights reserved.
Cell Efficiency Ratio: 658
CellCLI> list celldisk
CD_CELL001_xstcell normal
CD_CELL002_xstcell normal
CD_CELL003_xstcell normal
CD_CELL004_xstcell normal
CD_CELL005_xstcell normal
CD_CELL006_xstcell normal
CD_CELL007_xstcell normal
CD_CELL008_xstcell normal
CD_CELL009_xstcell normal
CD_CELL010_xstcell normal
CD_CELL011_xstcell normal
CD_CELL012_xstcell normal
CD_CELL013_xstcell normal
CD_CELL014_xstcell normal
CD_CELL015_xstcell normal
CD_CELL016_xstcell normal
CD_CELL017_xstcell normal
CD_CELL018_xstcell normal
CD_CELL019_xstcell normal
CD_CELL020_xstcell normal
CD_CELL021_xstcell normal
CD_CELL022_xstcell normal
CD_CELL023_xstcell normal
FD_00_xstcell normal
FD_01_xstcell normal
FD_02_xstcell normal
FD_03_xstcell normal
FD_04_xstcell normal
FD_05_xstcell normal
All griddisks are also active.
CellCLI> list griddisk
XS_CD_CELL001_xstcell active
XS_CD_CELL002_xstcell active
XS_CD_CELL003_xstcell active
XS_CD_CELL004_xstcell active
XS_CD_CELL005_xstcell active
XS_CD_CELL006_xstcell active
XS_CD_CELL007_xstcell active
XS_CD_CELL008_xstcell active
XS_CD_CELL009_xstcell active
XS_CD_CELL010_xstcell active
XS_CD_CELL011_xstcell active
XS_CD_CELL012_xstcell active
XS_CD_CELL013_xstcell active
XS_CD_CELL014_xstcell active
XS_CD_CELL015_xstcell active
XS_CD_CELL016_xstcell active
XS_CD_CELL017_xstcell active
XS_CD_CELL018_xstcell active
XS_CD_CELL019_xstcell active
XS_CD_CELL020_xstcell active
XS_CD_CELL021_xstcell active
XS_CD_CELL022_xstcell active
XS_CD_CELL023_xstcell active
CellCLI> list griddisk attributes name,asmmodestatus
XS_CD_CELL001_xstcell UNUSED
XS_CD_CELL002_xstcell UNUSED
XS_CD_CELL003_xstcell UNUSED
XS_CD_CELL004_xstcell UNUSED
XS_CD_CELL005_xstcell UNUSED
XS_CD_CELL006_xstcell UNUSED
XS_CD_CELL007_xstcell UNUSED
XS_CD_CELL008_xstcell UNUSED
XS_CD_CELL009_xstcell UNUSED
XS_CD_CELL010_xstcell UNUSED
XS_CD_CELL011_xstcell UNUSED
XS_CD_CELL012_xstcell UNUSED
XS_CD_CELL013_xstcell UNUSED
XS_CD_CELL014_xstcell UNUSED
XS_CD_CELL015_xstcell UNUSED
XS_CD_CELL016_xstcell UNUSED
XS_CD_CELL017_xstcell UNUSED
XS_CD_CELL018_xstcell UNUSED
XS_CD_CELL019_xstcell UNUSED
XS_CD_CELL020_xstcell UNUSED
XS_CD_CELL021_xstcell UNUSED
XS_CD_CELL022_xstcell UNUSED
XS_CD_CELL023_xstcell UNUSED
CellCLI> list alerthistory
8_1 2022-02-28T17:12:44+05:30 warning "Hugepage allocation failure in service cellsrv. Number of Hugepages allocated is 0, failed to allocate 3000"
51 2023-07-18T12:50:56+05:30 critical "ORA-00700: soft internal error, arguments: [main_6a], [3], [Invalid IP addresses in cellinit.ora file], [], [], [], [], [], [], [], [], []"
52 2023-07-18T13:03:37+05:30 critical "ORA-00700: soft internal error, arguments: [main_6a], [3], [Invalid IP addresses in cellinit.ora file], [], [], [], [], [], [], [], [], []"
53 2023-07-18T13:07:14+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL001 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL001_xstcel Grid Disk : XS_CD_CELL001_xstcel"
54 2023-07-18T13:07:14+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL002 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL002_xstcel Grid Disk : XS_CD_CELL002_xstcel"
55 2023-07-18T13:07:15+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL003 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL003_xstcel Grid Disk : XS_CD_CELL003_xstcel"
56 2023-07-18T13:07:15+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL004 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL004_xstcel Grid Disk : XS_CD_CELL004_xstcel"
57 2023-07-18T13:07:16+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL005 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL005_xstcel Grid Disk : XS_CD_CELL005_xstcel"
58 2023-07-18T13:07:17+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL006 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL006_xstcel Grid Disk : XS_CD_CELL006_xstcel"
59 2023-07-18T13:07:17+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL007 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL007_xstcel Grid Disk : XS_CD_CELL007_xstcel"
60 2023-07-18T13:07:18+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL008 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL008_xstcel Grid Disk : XS_CD_CELL008_xstcel"
61 2023-07-18T13:07:19+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL009 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL009_xstcel Grid Disk : XS_CD_CELL009_xstcel"
62 2023-07-18T13:07:19+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL010 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL010_xstcel Grid Disk : XS_CD_CELL010_xstcel"
63 2023-07-18T13:07:20+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL011 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL011_xstcel Grid Disk : XS_CD_CELL011_xstcel"
64 2023-07-18T13:07:20+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL012 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL012_xstcel Grid Disk : XS_CD_CELL012_xstcel"
65 2023-07-18T13:07:21+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL013 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL013_xstcel Grid Disk : XS_CD_CELL013_xstcel"
66 2023-07-18T13:07:22+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL014 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL014_xstcel Grid Disk : XS_CD_CELL014_xstcel"
67 2023-07-18T13:07:22+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL015 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL015_xstcel Grid Disk : XS_CD_CELL015_xstcel"
68 2023-07-18T13:07:23+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL016 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL016_xstcel Grid Disk : XS_CD_CELL016_xstcel"
69 2023-07-18T13:07:24+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL017 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL017_xstcel Grid Disk : XS_CD_CELL017_xstcel"
70 2023-07-18T13:07:24+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL018 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL018_xstcel Grid Disk : XS_CD_CELL018_xstcel"
71 2023-07-18T13:07:25+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL019 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL019_xstcel Grid Disk : XS_CD_CELL019_xstcel"
72 2023-07-18T13:07:25+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL020 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL020_xstcel Grid Disk : XS_CD_CELL020_xstcel"
73 2023-07-18T13:07:26+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL021 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL021_xstcel Grid Disk : XS_CD_CELL021_xstcel"
74 2023-07-18T13:07:27+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL022 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL022_xstcel Grid Disk : XS_CD_CELL022_xstcel"
75 2023-07-18T13:07:27+05:30 clear "Hard disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/CELL023 Firmware : Unknown Slot Number : Unknown Cell Disk : CD_CELL023_xstcel Grid Disk : XS_CD_CELL023_xstcel"
76 2023-07-18T13:07:28+05:30 clear "Flash disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/FLASH01 Firmware : Unknown Slot Number : Unknown Cell Disk : FD_00_xstcel Grid Disk : Not configured Flash Cache : Not configured Flash Log : Present"
77 2023-07-18T13:07:29+05:30 clear "Flash disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/FLASH02 Firmware : Unknown Slot Number : Unknown Cell Disk : FD_01_xstcel Grid Disk : Not configured Flash Cache : Not configured Flash Log : Present"
78 2023-07-18T13:07:29+05:30 clear "Flash disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/FLASH03 Firmware : Unknown Slot Number : Unknown Cell Disk : FD_02_xstcel Grid Disk : Not configured Flash Cache : Not configured Flash Log : Present"
79 2023-07-18T13:07:30+05:30 clear "Flash disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/FLASH04 Firmware : Unknown Slot Number : Unknown Cell Disk : FD_03_xstcel Grid Disk : Not configured Flash Cache : Not configured Flash Log : Present"
80 2023-07-18T13:07:30+05:30 clear "Flash disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/FLASH05 Firmware : Unknown Slot Number : Unknown Cell Disk : FD_04_xstcel Grid Disk : Not configured Flash Cache : Not configured Flash Log : Present"
81 2023-07-18T13:07:31+05:30 clear "Flash disk was replaced. Status : NORMAL Manufacturer : Unknown Model Number : Unknown Size : 0M Serial Number : /opt/oracle/cell11.2.3.2.1_LINUX.X64_130109/disks/raw/FLASH06 Firmware : Unknown Slot Number : Unknown Cell Disk : FD_05_xstcel Grid Disk : Not configured Flash Cache : Not configured Flash Log : Present"
82 2023-07-18T13:38:00+05:30 critical "ORA-00700: soft internal error, arguments: [main_21], [11], [Not enough open file descriptors], [], [], [], [], [], [], [], [], []"
It is not possible to create multiple straight forward or visible index in the same column of the table.
SQL> BEGIN
for x in 1 .. 10 loop
execute immediate 'CREATE index ind_'||x||' on bline(name)';
END loop;
END;
/ 2 3 4 5 6
BEGIN
*
ERROR at line 1:
ORA-01408: such column list already indexed
ORA-06512: at line 3
But invisible indexes can be created with different structure.
2 b-tree cannot be created on the same column.
One bitmap invisible and b-tree can be created on same column.
SQL> CREATE index ind_2 on bline(name) invisible;
CREATE index ind_2 on bline(name) invisible
*
ERROR at line 1:
ORA-00955: name is already used by an existing object
SQL> CREATE bitmap index ind_2 on bline(name) invisible;
Index created.
Cause:
SQL> EXEC dbms_stats.gather_database_stats();
BEGIN dbms_stats.gather_database_stats(); END;
*
ERROR at line 1:
ORA-06550: line 1, column 7:
PLS-00201: identifier 'DBMS_STATS.GATHER_DATABASE_STATS' must be declared
ORA-06550: line 1, column 7:
PL/SQL: Statement ignored
Solution:
If it is multitenant, then open the pdb.
SQL> alter pluggable database updb open;
Pluggable database altered.
SQL> alter pluggable database updb save state;
Pluggable database altered.
SQL> alter session set container=updb;
Session altered.
SQL> set timing on
SQL> EXEC dbms_stats.gather_database_stats();
While opening of a new pdb which was created from PDB$SEED, error ORA-959 occurred due to missing temp tablespace.
SQL> show pdbs
CON_ID CON_NAME OPEN MODE RESTRICTED
---------- ------------------------------ ---------- ----------
2 PDB$SEED READ ONLY NO
3 UPDB MOUNTED
SQL> alter pluggable database UPDB open;
Warning: PDB altered with errors.
Plugin violations shows these failures which need action to open the PDB clean.
SQL> col NAME for a20
SQL> col CAUSE for a20
SQL> col MESSAGE for a20
SQL> col ACTION for a20
SQL> set lines 200 pages 1000
SQL> SELECT name,cause,message,action from PDB_PLUG_IN_VIOLATIONS;
NAME CAUSE MESSAGE ACTION
-------------------- -------------------- -------------------- --------------------
UPDB Sync Failure Sync PDB failed with
ORA-959 during 'alt
er user c##hydra tem
porary tablespace te
mp1'
UPDB Sync Failure Sync PDB failed with
ORA-959 during 'alt
er user system tempo
rary tablespace temp
1'
Workaround the problem by manually creating the missing temporary tablespace.
SQL> alter session set container=updb;
Session altered.
SQL> CREATE temporary tablespace TEMP1 tempfile '/u01/app/oracle/oradata/hydrupgd/updb/temp1.dbf' size 1G;
Tablespace created.
PDB is opened without any restricted mode.SQL> alter pluggable database updb close;
Pluggable database altered.
SQL> alter pluggable database updb open;
Pluggable database altered.