Grid Infrastructure redundant interconnect in 11.2.0.3

For many years Oracle Clusterware have suffered from lack of native support of cluster interconnect redundancy. In the same time we still could configure more than one network for RAC cache fusion. To prevent interconnects being single point of failure, we could use 3rd party clustering solutions for older versions of oracle, or just use solutions provided by OS like network cards teaming, this often brought additional cost and complexity to the solution.

Starting from Oracle Grid Infrastructure 11.2.0.2 few feature called HAIP  (High Availability Interconnect) been introduced, allowing us to configure up to 4 private networks for interconnect use. New crs resource ‘ora.cluster_interconnect.haip’ have appeared crs ‘kitchenware’ which uses multicast protocol for initial bootstrap and rac cache fusion. This puts additional requirement to the network because often the protocol is turned off at switch level. As an another requirement, each private network must sit in its own subnet, otherwise unplugging the cable will cause node eviction. 

We can configure up to 4 networks for interconnect, if we set more than one private network, lets say 2, oracle will assign and use 4 addresses anyway attached to 2 physical network interfaces. In following example I’ve created 3 interconnects initially, and then removed first (eth1) and third (eth3), leaving only eth2. Here we see all 4 addresses attached to the single net interface.

eth2      Link encap:Ethernet  HWaddr 08:00:27:AD:90:C2
          inet addr:10.0.1.1  Bcast:10.0.1.255  Mask:255.255.255.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
          RX packets:2334553 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2305159 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:2497624256 (2.3 GiB)  TX bytes:2474843374 (2.3 GiB)

eth2:1    Link encap:Ethernet  HWaddr 08:00:27:AD:90:C2
          inet addr:169.254.20.220  Bcast:169.254.63.255  Mask:255.255.192.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth2:2    Link encap:Ethernet  HWaddr 08:00:27:AD:90:C2
          inet addr:169.254.91.47  Bcast:169.254.127.255  Mask:255.255.192.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth2:3    Link encap:Ethernet  HWaddr 08:00:27:AD:90:C2
          inet addr:169.254.169.41  Bcast:169.254.191.255  Mask:255.255.192.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

eth2:4    Link encap:Ethernet  HWaddr 08:00:27:AD:90:C2
          inet addr:169.254.231.159  Bcast:169.254.255.255  Mask:255.255.192.0
          UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1

In terms of RAC interconnect configuration, we will see all 4 addresses configured at database level. 

SQL>  select * from GV$CLUSTER_INTERCONNECTS where inst_id=1;

   INST_ID NAME            IP_ADDRESS       IS_ SOURCE
———- ————— —————- — ———————–
         1 eth1:1          169.254.20.220   NO
         1 eth2:1          169.254.91.47    NO
         1 eth3:1          169.254.169.41   NO
         1 eth1:2          169.254.231.159  NO


|The HAIP feature allow us to change IP of interconnect on the fly without impacting the service, as it was in previous versions (we needed to stop crs on both nodes to change interconnects). I’ve created small test case which generates lots of messages and block flying over interconnect. This was done by creating small table and updating same blocks, even rows from 1st instance and odd rows from 2nd instance. The test generated several hundred messages and blocks per second travelling over the interconnect network,  with load of about 4MB/sec. 

While the load was present, I removed 2 networks, leaving only one, then I added 2nd network and dropped first one. The interfaces wasn’t dropped immediately, few minutes passed until the relocation has occurred. According to  ‘gc blocks lost ‘/’gc current lost’ and other stats, no blocks dropped during the operation.

End of part I

Signal 11 failure on installing 11gR2 grid infrastructure 32 bit

Recent few weeks I’ve been installing oracle 11g r2 infrastructure (clusterware) 32 bit on vmware workstation. The installation process finished successfully and root.sh is launched, the following error appears:
“Failure with signal 11 from command: /u01/app/grid/bin/ocrconfig -local -upgrade oracle oinstall
Failed to create or upgrade OLR”

I have checked out many solutions, but with no success.

As a last chance, I’ve decided to install 64 grid infrastructure. Installation finished successfully with no errors. So, I believe, that ‘signal 11’ error comes from 32bit version.

Hope, I save some time for someone, trying to resolve problem like this.

Oracle, uncaught Java exception: java.lang.UnsatisfiedLinkError

This is very common error in java environment. In most cases the problem will be in LD_LIBRARY_PATH variable which is not set properly. Lets show how we can perform diagnostics:

  1. Check trace for more info. The trace can be found in udump directory
  2. Check LD_LIBRARY_PATH set properly i.e. set to $ORACLE_HOME/lib
  3. Check how LD_LIBRARY_PATH variable set inside java
  4. create or replace function get_java_property(p_property varchar2) return varchar2
    as language java name ‘java.lang.System.getProperty(java.lang.String) return java.lang.String’;
    select get_java_property(‘java.library.path’) from dual;

  5. Check that you library actually located in $OH/lib and has proper access rights
  6. Check that your lib bit szie compares to bit size of oracle db. If oracle db is 64 bit than lib must be 64 as well.

Raw devices support in oracle db 12

According to some info from oracle support. Block devices will not be supported from next version of oracle – 12. Seems that oracle completely moves to ASM.

At this moment, oracle RAC 11.2 installer does not allow to use block/raw devices at installation phase. Later one can change location of voting disks and ocr to block devices. This will work on linux only.
note: it’s not possible to store voting files on both ASM and block devices

To move voting files use:
crsctl replace votedisk {+asm_disk_group | path_to_voting_disk}

For ocr:
ocrconfig -replace current_OCR_location -replacement new_OCR_location

Character set conversion problem

Recently we have changed codepage of one of oracle 8.1.7 database from NEE8ISO8859P4 to LV8RST104090. After the change, we have noticed that one in-house application failed with error message “ORA-12703: this character set conversion is not supported”

The problem was that this database used by another UTF8 database through dblink. The application from UTF8 is calling procedure from LV8RST104090 and output parameters cannot be converted. But IN parameters are converted normally.

How we solve the problem. The Simplest method is to use RAW data type which is not automatically converted between databases with different codepages. We have used UTL_RAW package. Change parameter type to RAW and use UTL_RAW.CAST_TO_RAW to convert to raw and
UTL_RAW.CAST_TO_NVARCHAR2,
UTL_RAW.CAST_TO_NUMBER to convert back the parameters.

Signal 11 failure on installing 11gR2 grid infrastructure 32 bit

Recent few weeks I’ve been installing oracle 11g r2 infrastructure (clusterware) 32 bit on vmware workstation. The installation process finished successfully and root.sh is launched, the following error appears:
“Failure with signal 11 from command: /u01/app/grid/bin/ocrconfig -local -upgrade oracle oinstall
Failed to create or upgrade OLR”

I have checked out many solutions, but with no success.

As a last chance, I’ve decided to install 64 grid infrastructure. Installation finished successfully with no errors. So, I believe, that ‘signal 11’ error comes from 32bit version.

Hope, I save some time for someone, trying to resolve problem like this.