{"id":73147,"date":"2015-09-30T11:36:06","date_gmt":"2015-09-30T11:36:06","guid":{"rendered":"https:\/\/www.red-gate.com\/simple-talk\/uncategorized\/oracle-12c-cluster-acfs-leverages-flex-asm\/"},"modified":"2021-07-14T13:07:20","modified_gmt":"2021-07-14T13:07:20","slug":"oracle-12c-cluster-acfs-leverages-flex-asm","status":"publish","type":"post","link":"https:\/\/www.red-gate.com\/simple-talk\/databases\/oracle-databases\/oracle-12c-cluster-acfs-leverages-flex-asm\/","title":{"rendered":"Oracle 12c Cluster: ACFS Leverages Flex ASM"},"content":{"rendered":"

In Oracle Database Releases earlier than 12c, an Automatic Storage Management (ASM) instance runs on every node in the cluster, and ASM Cluster File System (ACFS) Service on a node connects to the local ASM instance running on the same host. If the ASM instance on a node were to fail, then the shared Disk Groups and hence ACFS file systems can no longer be accessed on that node.<\/p>\n

With introduction of Flex ASM in Oracle 12c, the hard dependency between ASM and its clients has been relaxed and a smaller number of ASM instances need run on a subset of servers in a cluster. In this scenario, when there might be nodes without an ASM instance, a new instance type has been introduced by Flex ASM – ASM proxy instance <\/em>which gets metadata information from ASM instance on behalf of ACFS. If ASM instance is not available locally, ASM proxy instance connects to other ASM instances over the network to fetch the metadata. Moreover, if the local ASM instance fails, then ASM proxy instance can failover to another surviving ASM instance on a different server resulting in uninterrupted availability of shared storage and ACFS file systems. However, I\/O to the underlying storage does not go through Oracle ASM, but goes directly through the Oracle ASM proxy instance. An ASM proxy instance must be running on a node which needs to provide ACFS service.<\/p>\n

Figure 1<\/strong> shows that as an ASM instance is not running on Hub Node 2<\/strong>, ADVM \/ ACFS services on Hub Node 2<\/strong>l utilize the ASM proxy instance<\/em> (+APX2) to access the metadata from the remote ASM instance (+ASM1) running on Hub Node 1.<\/strong><\/p>\n

\"\"<\/p>\n

Figure 1<\/strong>.
\nFlex ASM can be configured on either a standard cluster or a Flex Cluster. When Flex ASM runs on a standard cluster, ASM services can run on a subset of cluster nodes servicing clients across the cluster. When Flex ASM runs on a Flex Cluster, ASM services can run on a subset of Hub Nodes servicing clients across all of the Hub Nodes in the Flex Cluster. Besides, in a flex cluster, only hub nodes can host the ACFS services because hub nodes have direct access to storage.<\/p>\n

A Demonstration<\/h2>\n

To explore how ADVM \/ ACFS can leverage Flex ASM features, we will use an ASM Flex Cluster that is configured with two hub nodes (host01<\/strong> and host02<\/strong>). Our first set of tasks is to create an ACFS file system resource.<\/p>\n

Check Prerequisites<\/h2>\n

First, let’s verify that all kernel modules needed for ACFS and ADVM are loaded on both the nodes.<\/p>\n

 [root@host01 ~]# lsmod |grep oracle\r\noracleacfs<\/span>           2837904  1 \r\noracleadvm<\/span>            342512  1 \r\noracleoks<\/span>             409560  2 oracleacfs,oracleadvm\r\noracleasm<\/span>              84136  1 \r\n\r\n[root@host02 ~]# lsmod |grep oracle\r\noracleacfs<\/span>           2837904  1 \r\noracleadvm<\/span>            342512  1 \r\noracleoks<\/span>             409560  2 oracleacfs,oracleadvm\r\noracleasm              84136  1 <\/pre>\n
The ASM Dynamic Volume Manager (ADVM) proxy instance is a special Oracle instance which enables ADVM to connect to Flex ASM and is required to run on the same node as ADVM and ACFS. For a volume device to be visible on a node, an ASM proxy instance must<\/em> be running on that node. Let’s verify that an ASM proxy instance is running on both the nodes.<\/p>\n
 [root@host02 ~]# crsctl stat res ora.proxy_advm<\/span> -t\r\n----------------------------------------------------------------\r\nName           Target  State        Server                   State details\r\n----------------------------------------------------------------\r\nLocal Resources\r\n----------------------------------------------------------------\r\nora.proxy_advm\r\n               ONLINE  ONLINE<\/span>       host01<\/span>                   STABLE\r\n               ONLINE  ONLINE<\/span>       host02<\/span>                   STABLE\r\n<\/pre>\n
Create the ADVM Volume<\/h2>\n
We’ll next modify the compatible.advm<\/strong> attribute of the DATA<\/strong> ASM disk group to enable all the new ASM Dynamic Volume (ADVM) features included in release 12.1, and then create a new volume – VOL1<\/strong> within the DATA<\/strong> disk group with a volume size of 300 MB.<\/p>\n
 [grid@host01 root]$ asmcmd setattr -G DATA compatible.advm 12.1.0.0.0\r\n\r\n[grid@host01 root]$ asmcmd volcreate -G DATA -s 300m VOL1<\/span>\r\n\r\n[grid@host01 root]$ asmcmd volinfo -G DATA VOL1\r\n\r\nDiskgroup Name: DATA\r\n         Volume Name: VOL1<\/span>\r\n         Volume Device: \/dev\/asm\/vol1-106<\/span>\r\n         State: ENABLED\r\n         Size (MB): 320\r\n         Resize Unit (MB): 32\r\n         Redundancy: MIRROR\r\n         Stripe Columns: 4\r\n         Stripe Width (K): 128\r\n         Usage: ACFS\r\n         Mountpath:<\/pre>\n
Create ACFS File System and Corresponding Mount Point<\/h2>\n
Now let’s construct an ACFS file system on the newly-created volume VOL1<\/strong> and also create a mount point on both the nodes to mount the ACFS file system.<\/p>\n
[root@host01 ~]# mkfs -t acfs \/dev\/asm\/vol1-106 \r\n[root@host01 ~]# mkdir -p \/mnt\/acfsmounts\/acfs1\r\n[root@host02 ~]# mkdir -p \/mnt\/acfsmounts\/acfs1<\/pre>\n
Configure Cloud File System Resource For ACFS File System<\/h2>\n
Now using the srvctl<\/strong> commands, we will create an Oracle Cloud File System resource on the volume device VOL1<\/strong> with the mount point \/mnt\/acfsmounts\/acfs1<\/strong>:<\/p>\n
[root@host01 ~]# srvctl add filesystem -m \/mnt\/acfsmounts\/acfs1 -d \/dev\/asm\/vol1-106 \r\n\r\n[root@host01 ~]# srvctl status filesystem -d \/dev\/asm\/vol1-106\r\n\r\nACFS file system \/mnt\/acfsmounts\/acfs2 is not mounted<\/span>\r\n\r\n[root@host01 ~]# mount | grep vol1\r\n\r\n[root@host02 ~]# mount | grep vol1<\/pre>\n
We need to start the file system resource to mount the ACFS file system.<\/p>\n
[root@host01 ~]# srvctl start  filesystem -d \/dev\/asm\/vol1-106\r\n\r\n[root@host01 ~]# srvctl status  filesystem -d \/dev\/asm\/vol1-106\r\n\r\nACFS file system \/mnt\/acfsmounts\/acfs2 is mounted on nodes host01,host02<\/span>\r\n\r\n[root@host01 ~]# mount | grep vol1\r\n\r\n\/dev\/asm\/vol1-106 on \/mnt\/acfsmounts\/acfs1 type acfs (rw)<\/span>\r\n\r\n[root@host02 ~]# mount | grep vol1\r\n\r\n\/dev\/asm\/vol1-106 on \/mnt\/acfsmounts\/acfs1 type acfs (rw)<\/span>\r\n<\/pre>\n
Verify The Cloud File System Resource<\/h2>\n
It can be verified that the new Cloud File System is indeed working properly as a small text file created on it from host01<\/strong> can be successfully accessed from host02<\/strong>.<\/p>\n
[root@host01 ~]# echo "Test File on ACFS"<\/strong> > \/mnt\/acfsmounts\/acfs1\/testfile.txt\r\n\r\n[root@host02 asm]#  cat \/mnt\/acfsmounts\/acfs1\/testfile.txt\r\n\r\nTest File on ACFS<\/strong><\/pre>\n
Leveraging Flex ASM<\/h2>\n
To demonstrate how ACFS \/ ADVM leverages these Flex ASM capabilities, let us first have a look at the current state of various cluster resources in our two node cluster.<\/p>\n
View Baseline Status of Various Cluster Resources<\/h2>\n
It can be seen that currently:<\/p>\n
    \n
  • An ASM instance is running on both the nodes<\/li>\n
  • DATA disk group, volume VOL1 on DATA diskgroup and ACFS file system on VOL1 are all online on both the nodes.<\/li>\n<\/ul>\n
     [root@host01 ~]# crsctl stat res ora.asm -t\r\n----------------------------------------------------------------\r\nName           Target  State        Server         State details       \r\n----------------------------------------------------------------\r\nCluster Resources\r\n----------------------------------------------------------------\r\nora.asm<\/span>\r\n      1        ONLINE  ONLINE<\/span>       host01          STABLE\r\n      1        ONLINE  ONLINE<\/span>       host02          STABLE\r\n----------------------------------------------------------------\r\n\r\n[root@host01 ~]# crsctl stat res ora.DATA.dg -t\r\n----------------------------------------------------------------\r\nName           Target  State        Server         State details       \r\n----------------------------------------------------------------\r\nLocal Resources\r\n----------------------------------------------------------------\r\nora.DATA.dg<\/span>\r\n               ONLINE  ONLINE       host01<\/span>         STABLE\r\n               ONLINE  ONLINE       host02<\/span>         STABLE\r\n----------------------------------------------------------------\r\n\r\n\r\n[root@host01 ~]# crsctl stat res ora.DATA.VOL1.advm -t\r\n----------------------------------------------------------------\r\nName           Target  State        Server   State details       \r\n----------------------------------------------------------------\r\nLocal Resources\r\n----------------------------------------------------------------\r\nora.DATA.VOL1.advm<\/span>\r\n               ONLINE  ONLINE       host01<\/span>  Volume device \/dev\/a\r\n                                            sm\/vol1-106 is \r\n                                            online,STABLE\r\n               ONLINE  ONLINE       host02<\/span>  Volume device \/dev\/a\r\n                                            sm\/vol1-106 is \r\n                                            online,STABLE\r\n\r\n\r\n[root@host01 ~]# crsctl stat res ora.data.vol1.acfs\r\n\r\nNAME=ora.data.vol1.acfs<\/span>\r\nTYPE=ora.acfs.type\r\nTARGET=ONLINE          , ONLINE\r\nSTATE=ONLINE on host01, ONLINE on host02<\/span><\/pre>\n
    Confirm Flex ASM Proxy Instances<\/h2>\n
    Let’s also verify that ASM proxy instances +APX1<\/strong> running on host01<\/strong> and +APX2<\/strong> running on host02<\/strong> are using ASM instances +ASM1<\/strong> and +ASM2<\/strong> running locally on their corresponding nodes.<\/p>\n
    SQL> col client_instance_name format a20\r\n     col asm_host_name for a20\r\n\r\nSELECT DISTINCT \r\n    i.instance_name asm_instance_name, \r\n    i.host_name asm_host_name, \r\n    c.instance_name client_instance_name, \r\n    c.status\r\n  FROM gv$instance i, gv$asm_client c\r\n WHERE i.inst_id = c.inst_id;<\/pre>\n
    ASM_INSTANCE_NAM ASM_HOST_NAME        CLIENT_INSTANCE_NAME STATUS\r\n---------------- -------------------- -------------------- ------------\r\n+ASM1            host01.example.com   +APX1<\/span>                CONNECTED\r\n+ASM1            host01.example.com   +ASM1                CONNECTED\r\n+ASM2            host02.example.com   +APX2<\/span>                CONNECTED<\/pre>\n
    To make ACFS \/ ADVM leverage Flex ASM capabilities, we can perform a simple test: On stopping the ASM instance on node host02<\/strong> we should observe that ACFS \/ ADVM services will continue to run on host02<\/strong> while utilizing an ASM proxy instance to satisfy metadata requests to a remote ASM instance \u2013 in this case, +ASM1<\/strong> running on host01<\/strong>.<\/p>\n
    Halt ASM Instance On host02<\/h2>\n
    Let’s halt ASM instance +ASM2<\/strong> running on node host02<\/strong>.<\/p>\n
    [root@host01 ~]# srvctl stop asm -n host02\r\n\r\nPRCR-1014 : Failed to stop resource ora.asm\r\nPRCR-1065 : Failed to stop resource ora.asm\r\nCRS-2529: Unable to act on 'ora.asm' because that would require stopping or relocating 'ora.DATA.dg', but the force option was not specified\r\n\r\n[root@host01 ~]# srvctl stop asm -n host02 -f\r\n\r\n[root@host01 ~]# srvctl status asm\r\nASM is running on host01<\/pre>\n
    Verify Availability of VOL1 On host02<\/h2>\n
    If we now check we’ll find that even though the +ASM2<\/strong> instance and the DATA<\/strong> disk group are not available on host02<\/strong>, volume VOL1<\/strong> created on the DATA<\/strong> disk group is still mounted on host02<\/strong> because of Flex ASM. Node host02<\/strong> connects to the remote ASM instance +ASM1<\/strong> running on host01<\/strong> using its ASM proxy instance +APX2<\/strong> to access the metadata, and that keeps volume VOL1<\/strong> online.<\/p>\n
     [root@host01 ~]# crsctl stat res ora.asm -t\r\n----------------------------------------------------------------\r\nName           Target  State        Server         State details       \r\n----------------------------------------------------------------\r\nCluster Resources\r\n----------------------------------------------------------------\r\nora.asm<\/span>\r\n      1        ONLINE  ONLINE       host01          STABLE\r\n      2        ONLINE  OFFLINE      host02<\/span>          STABLE\r\n----------------------------------------------------------------\r\n\r\n[root@host01 ~]# crsctl stat res ora.DATA.dg -t\r\n----------------------------------------------------------------\r\nName           Target  State        Server         State details       \r\n----------------------------------------------------------------\r\nLocal Resources\r\n----------------------------------------------------------------\r\nora.DATA.dg<\/span>\r\n               ONLINE  ONLINE       host01         STABLE\r\n               OFFLINE OFFLINE      host02<\/span>         STABLE\r\n----------------------------------------------------------------\r\n\r\n\r\n[root@host01 ~]# crsctl stat res ora.DATA.VOL1.advm -t\r\n----------------------------------------------------------------\r\nName           Target  State        Server   State details       \r\n----------------------------------------------------------------\r\nLocal Resources\r\n----------------------------------------------------------------\r\nora.DATA.VOL1.advm<\/span>\r\n               ONLINE  ONLINE       host01  Volume device \/dev\/a\r\n                                            sm\/vol1-106 is \r\n                                            online,STABLE\r\n               ONLINE  ONLINE       host02  Volume device \/dev\/a<\/span>\r\n                                            sm\/vol1-106 is<\/span> \r\n                                            online,STABLE<\/span><\/pre>\n
    Verify Availability of Cloud File System acfs1 On host02<\/h2>\n
    We can also verify that the file on cloud file system acfs1<\/strong> is visible from both host01<\/strong> and host02<\/strong>.<\/p>\n
    [root@host01 acfs1]# cat \/mnt\/acfsmounts\/acfs1\/testfile.txt\r\n\r\n\u201cTest File on ACFS\u201d<\/strong>\r\n\r\n[root@host02 asm]# cat \/mnt\/acfsmounts\/acfs1\/testfile.txt\r\n\r\n\u201cTest File on ACFS\u201d<\/strong><\/pre>\n
    Confirm Flex ASM Proxy Instances<\/h2>\n
    We can also verify that since ASM instance +ASM2<\/strong> running on host02<\/strong> is no longer available, ACFS \/ ADVM leverages Flex ASM and ASM proxy instance +APX2<\/strong> running on host02<\/strong> accesses the metadata from remote ASM instance +ASM1<\/strong> that is running on host01<\/strong> to access volume VOL1<\/strong> present on the DATA<\/strong> disk group.<\/p>\n
    SQL> col client_instance_name format a20\r\n     col asm_host_name for a20\r\n\r\nSELECT DISTINCT \r\n    i.instance_name asm_instance_name, \r\n    i.host_name asm_host_name, \r\n    c.instance_name client_instance_name, \r\n    c.status\r\n  FROM gv$instance i, gv$asm_client c\r\n WHERE i.inst_id = c.inst_id; \r\n\r\n\r\nASM_INSTANCE_NAM ASM_HOST_NAME        CLIENT_INSTANCE_NAME STATUS\r\n---------------- -------------------- -------------------- ------------\r\n+ASM1            host01.example.com   +APX1                CONNECTED\r\n+ASM1            host01.example.com   +ASM1                CONNECTED\r\n+ASM1            host01.example.com   +APX2<\/span>                CONNECTED<\/pre>\n
    Summary<\/h2>\n
      \n
    • A new instance type has been introduced by Flex ASM – ASM proxy instance<\/em>, which gets metadata information from ASM instance on behalf of ACFS.<\/li>\n
    • If an ASM instance is not available locally, ASM proxy instance<\/em> connects to other ASM instances over the network resulting in uninterrupted availability of shared storage and ACFS file systems.<\/li>\n
    • This provides a much higher degree of flexibility, scalability and availability for file services to clients.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
      In Oracle Database Releases earlier than 12c, an Automatic Storage Management (ASM) instance runs on every node in the cluster, and ASM Cluster File System (ACFS) Service on a node connects to the local ASM instance running on the same host. If the ASM instance on a node were to fail, then the shared Disk Groups and hence ACFS f…<\/p>\n","protected":false},"author":316217,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[143533],"tags":[],"coauthors":[],"class_list":["post-73147","post","type-post","status-publish","format-standard","hentry","category-oracle-databases"],"acf":[],"_links":{"self":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/73147","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/users\/316217"}],"replies":[{"embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/comments?post=73147"}],"version-history":[{"count":1,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/73147\/revisions"}],"predecessor-version":[{"id":91643,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/posts\/73147\/revisions\/91643"}],"wp:attachment":[{"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/media?parent=73147"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/categories?post=73147"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/tags?post=73147"},{"taxonomy":"author","embeddable":true,"href":"https:\/\/www.red-gate.com\/simple-talk\/wp-json\/wp\/v2\/coauthors?post=73147"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}