CRS-4640
We observed that osysmond.bin was taking too much memory and the usage of SWAP was increasing. At this moment, to prevent the server from crashing due to full SWAP, we decide to stop all cluster ready services (CRS) in this node.
[root@primary02 ~]# . /home/grid/.bash_profile
[root@primary02 ~]# crsctl stop crs
...
While SWAP was freed after all cluster processes stopped, which includes osysmond.bin, we may try to start CRS as the following.
[root@primary02 ~]# crsctl start crs
CRS-4640: Oracle High Availability Services is already active
CRS-4000: Command Start failed, or completed with errors.
Unfortunately, it failed with CRS-4640.
CRS-4640 means that Oracle High Availability Services (OHAS) is still running, there's no way to start it again. You should inspect your server condition, it could be something stopping you from doing this.
Solution
To stop all cluster services including OHAS cleanly, we may take it down forcibly.
[root@primary02 ~]# crsctl stop crs -f
If the above command cannot resolve the problem, we may kill its process at OS level like this:
[root@primary02 ~]# ps -ef | grep "ohasd"
root 8389000 1 0 Dec 18 - 95:00 /u01/app/19.0.0/grid/bin/ohasd.bin reboot
...
[root@primary02 ~]# kill -9 8389000
A system reboot on this node can also helps.
Now we can start the CRS now.
[root@primary02 ~]# crsctl start crs
CRS-4123: Oracle High Availability Services has been started.
After several minutes, you may check the status of all resources.
[root@primary02 ~]# crsctl status resource -t