Unlike a cluster where only a few nodes are interconnected, Massively Parallel Processing (MPP) systems allow a large number of nodes to be interconnected. There are MPP systems with more than a thousand nodes. Each of these nodes is a separate system that has its own CPU, disks, controllers, memory, and internal system buses to form a “loosely coupled” architecture that shares nothing. All of these nodes are connected via a high-speed, high-bandwidth interconnect. Each node has a separate copy of the operating system. In the MPP architecture, Oracle is installed in RAC mode.
Each Oracle instance on each node is responsible for all the resources it contains, has a view of the entire database, and can find out which node has a lock on any part of the database. If you need something that is locked by another node, it would require an inter-instance ping, where the other instance would have to write to disk all the changes it has made. A typical database query would go against one node; the data would be collected from the memory or disks of any appropriate node and would travel across the interconnect to return the data to the requesting node. Each node controls its own set of disks and can take over another set if one node fails. Therefore, all nodes can be configured to be primary or a combination of primary and secondary. The database files are placed on the primary nodes, while the secondary nodes provide the necessary redundancy to take over and replace the primary nodes, should the primary nodes fail.
The biggest disadvantage of MPP architectures is not the architecture itself, but rather the application design flaws that exist in a typical implementation. Most application designers are well-versed in deploying applications to SMP machines. However, MPP architectures require a total paradigm shift in terms of analyzing which data should be placed on which node to reduce data exchange and inter-instance ping between nodes. Since each node is very independent in an architecture that does not share anything, any situation that causes nodes to exchange resources on a large scale due to data exchange generates high traffic across the interconnect. This situation is causing serious performance problems in Oracle RAC databases. Oracle RAC database performance is primarily dependent on incorrect hardware architecture configuration. You must consider the hardware architecture and operating system implementation before processing the Oracle Real Application Cluster configuration. Otherwise, you would have a lot of problems after deployment and changing hardware configuration requires downtime of your high availability databases.