Oracle Database 19c on Oracle Direct NFS Using FA File

With Pure Storage Purity 6.1, we saw the arrival of file support to the family of Pure Storage^® FlashArray™ systems, providing a unified block and file storage platform. 

In this blog post, I’ll share how to get an Oracle Database 19c up and running on a Pure Storage FlashArray using Oracle Direct NFS (dNFS).

Oracle Direct NFS (dNFS)

Oracle dNFS is a mature Oracle Database feature and has been available since Oracle Database 11g Release 1. Oracle dNFS addresses many of the challenges of running Oracle on NFS, providing many benefits. These include:

• Eliminating operating system write-ordering locks
• Improving high availability (HA) of NAS storage 
• Bypassing operating system caches to improve database performance on NAS storage
• Supporting up to four parallel network paths to storage
• Automatically load balancing across all network paths
• Optimising scalability of NAS storage 
• Displaying linear scaling over both 1GbE, 10GbE, 40GbE, and 100GbE paths
• Vastly reducing system CPU utilisation 
• Reducing system CPU by bypassing file system stack 
• Also available for Microsoft Windows 

Before configuring Oracle dNFS, I always recommend confirming that the Linux NFS exports can be mounted and used OK, ruling out any storage array, network, operating system, and access permission issues.

For RAC, consult: Oracle MOS Doc ID 359515.1 Mount Options for Oracle files for RAC databases and Clusterware when used with NFS on NAS devices.

Getting Started

Log on to the FlashArray, click Storage from the left-hand panel, and select File Systems.

Create File System

Navigate to the File Systems panel and click +.

Provide File System Name, for example, oracle.

Figure 1: Create File System.

Create Directory

Click on the newly created File System oracle.

Navigate to the Directories panel and click +.

Provide Directory Name and Path, for example, cdb2-u02 and /cdb2-u02.

Figure 2: Create Directory.

Repeat for any other required file systems, for example, cdb2-u03.

Figure 3: Create Directory.

Add Export Policies

I have pre-created a policy called nfs-no-root-squash with the following settings:

• Rules
  • Clients: Only the Oracle Database nodes should be added as clients 
  • Access: no-root-squash 
  • Permission: rw
  • Version: NFSv3
• Details
  • Type: NFS
  • Enabled: True
  • User mapping: false
  • Version: NFSv3

Select the newly created Directory, and from the Policies panel, click the three vertical pips and select Add Export Policies.

Figure 4: Add Export Policies.

That’s the FlashArray complete, now over to the Oracle database server.

For detailed information, review: Oracle MOS Doc ID 762374.1: Step by Step – Configure Direct NFS Client (DNFS) on Linux.

Oracle Database 19c

If your Oracle Home has been previously linked with dnfs, you may be surprised to see that Oracle dNFS is already in use. Query the v$dnfs_servers view to see if dNFS is in use, for example:

select svrname, dirname FROM v$dnfs_servers;

Server Export Name

---------------------------------------- -----------

z-x70-a-fastfile /cdb2-u02z-x70-a-fastfile /cdb2-u03

In the example above, Oracle has automatically configured dNFS using the NFS mount information in the /etc/fstab file.

If not enabled, you will need to shut down your database and link Oracle. 

Change the directory to $ORACLE_HOME/rdbms/lib and perform the following:

cd $ORACLE_HOME/rdbms/libmake -f ins_rdbms.mk dnfs_on

Example output:

[oracle@z-rac2 lib]$ make -f ins_rdbms.mk dnfs_on
rm -f /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/lib/odm/libnfsodm19.so; \
cp /u01/app/oracle/product/19.0.0/dbhome_1/lib/libnfsodm19.so /u01/app/oracle/product/19.0.0/dbhome_1/rdbms/lib/odm/libnfsodm19.so

How to Create an oranfstab File

Change the directory to $ORACLE_HOME/dbs and create/edit a file called oranfstab.

The parameters used in my oranfstab are described below:

server
The NFS server name. This can be anything and is used for information and does not need to be resolvable by DNS.

local
Up to four paths on the database host, specified by IP address or by name.

path
Up to four network paths to the NFS server, specified either by IP address or by name.

export
The exported path from the NFS server.

mount
The corresponding local mount point for the exported volume.

nfs_version
Specifies the NFS protocol version used by Direct NFS Client. Possible values are NFSv3, NFSv4, NFSv4.1, and pNFS. 

The default version is NFSv3. If you select NFSv4.x, then you must configure the value in oranfstab for nfs_version. Specify nfs_version as pNFS if you want to use Direct NFS with Parallel NFS.

Below is an extract for my FA Files NFS exports on my database lab server.

#x70a

server: x70a

local: 192.168.90.26: path:192.168.90.40

nfs_version: nfsv3

export: /cdb2-u02 mount: /u02

export: /cdb2-u03 mount: /u03

Now bounce your Oracle database, and confirm the Oracle database alert log file includes an Oracle Direct NFS ODM Library entry.

Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 6.0.

If the $ORACLE_HOME/dbs/oranfstab file has been updated correctly, we should now see the server name defined within the oranfstab file rather than /etc/fstab name, for example:

Server Export Name

---------------------------------------- -----------

x70a /cdb2-u02

x70a /cdb2-u03

Calibrate IO

Introduced in Oracle 11g, the Oracle provided DBMS_RESOURCE_MANAGER.CALIBRATE_IO procedure can be used to provide an indication of the storage subsystem performance.

You need to run calibrate_io from within the container database as it is not an allowed operation from within a pluggable database.

SET LINESIZE 200SET SERVEROUTPUT ONDECLARE  lat INTEGER;  iops INTEGER;  mbps INTEGER;BEGIN-- DBMS_RESOURCE_MANAGER.CALIBRATE_IO (<DISKS>, <MAX_LATENCY>, iops, mbps, lat);   DBMS_RESOURCE_MANAGER.CALIBRATE_IO (1000, 10, iops, mbps, lat);   DBMS_OUTPUT.PUT_LINE ('max_iops = ' || iops);   DBMS_OUTPUT.PUT_LINE ('latency = ' || lat);   DBMS_OUTPUT.PUT_LINE ('max_mbps = ' || mbps);end;/

Confirm timed_statistics are enabled and run the example SQL above.

SELECT NAME,VALUE FROM V$PARAMETER WHERE NAME=’timed_statistics’

For example:

SQL> @calibrate_io

Database Details

===============================================

timed_statistics should be TRUE

filesystemio_options should be SETALL

Name   Value

------------------------- ----------

timed_statistics   TRUE

Name   Value

------------------------- ----------

filesystemio_options   SETALL

max_iops = 229657

latency  = .443

max_mbps = 5565

max_iops = 229657

latency = 0

max_mbps = 5565

PL/SQL procedure successfully completed.

Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

Version 19.21.0.0.0

Calibration is extremely disruptive to the database performance. It is strongly recommended to run calibration only when database users can tolerate severe deterioration to database performance. For optimal calibration results, no other database workloads should be running.

Oracle dNFS Views

Oracle 19c provides the following database views to assist with dNFS management:

• v$dnfs_channels: Displays information about the Oracle process connections (channels) open to NFS servers
• v$dnfs_files: Displays information about the Oracle process files open through Direct NFS
• v$dnfs_servers: Displays information about the Direct NFS servers accessed by Direct NFS
• v$dnfs_stats: Displays information about the Oracle process NFS operation statistics issued by Direct NFS

Oracle also provides a database package to monitor dNFS performance and throughput. Read my previous blog on how to configure and use MONDNFS.