How to Resolve ORA-16055: FAL request rejected

In Oracle MAA, data guard plays a very important role to secure our data on the remote location. Since it involves network and interconnection between two databases, it is pretty complicated to do troubleshooting. That's why you might see ORA-16055: FAL request rejected at times.

The network problem could include host, router, firewall, LAN, VPN, IDS, IPS and leased line, etc. Any node on the route could cost your time to find out the problem.

The reaction from the standby database is also important, it's not purely accepters. Don't expect it that can digest everything transported from FAL_SERVER.

Now, let's see how I found ORA-16055: FAL request rejected and how I handled it.

ORA-16055: FAL request rejected

Found an error in the alert log:

From the aspect of primary database, this message implies that I tried to communicate with the standby database, but it does not respond to my message.

Let's check the error code ORA-16055 first:

"This may have various causes". Yes, it's right. The possible causes could be very dependent. The above message do not offer much useful information.

This error appeared in the alert log of the primary database, but we should take appropriate actions on the standby database.

If ORA-16055 occurred only once in alert log, you don't have to worry about it, FAL request could have been back to normal. If ORA-16055: FAL request rejected repeated itself in the alert log, you should solve the problems, although the problem might not be critical at this moment.

Doing Some Tests

The error means that some problems occurred between the primary database and the standby database. In the first place, I thought it's a network problem, but the standby database can be accessed via tnsping:

As we know, tnsping can test only some degrees of connection health, such as host and listener, but not service and database. For more information, you may check the post below:
TNSPING Errors Collections

Several Possibilities

In the above test, the result shows that the listener of the standby database is online and working at this moment, but the transportation from the primary database stopped. There could be several possibilities:

The standby listener hanged or was told not to accept new connections by PMON.

If your standby database opened as read-only, it could hit the maximum processes. Or, one possibility, the listener hanged.

Local archived log file is missing

You should make sure the archived log files required by the standby database are still there (primary database).

Poor network condition, temporary traffic jam over leased lines.

Intermittent network could result the error, you should gauge the network condition.

The standby database needs to be recovered.

There could be a damaged data file needed to be recovered, or the broker is dysfunctional.

The space used by the standby database was full, either data or archived logs destination.

The standby database will accept no more redo changes from the primary database.

Log receiving or switching was too busy for the standby database to respond.

If the standby server is working with not only the database, but also other services which could make OS exhausted. You should watch them closely.

One of user system limits has been reached.

The reached resource limit is usually the maximum number of open files or maximum size of file created.

Cleaning Archived Logs

Finally, we found the space of archived logs of the standby database was full, so we released the space of archived logs, the transportation service began to work automatically. You will see the working message of FAL in the alert log like this:

ARCX: Begin FAL archive ...

ARCX: Complete FAL archive ...

No more ORA-16055: FAL request rejected.