Today I hit a problem were Enterprise Manager was reporting managed server BI_server1 was down. That was a surprise as I was pretty sure it had just started with no problems!

Refresh EM. Still reported as down.

Restart everything. Still “down”.

Okay, back to the WLS Console. Nope, BI_server1 is definitely running. So why doesn’t EM agree?

After a bit of digging, it turned out that the Node Manager was unreachable, according to WLS Console. And the node manager was logging:

<WARNING> <There was a problem initializing the domain <domain> at 'D:\OBI\user_projects\domains\domainName'. Please make sure that this domainName: <domain> is registered and is fully enrolled for this nodemanager at: 'D:\OBI\user_projects\domains\domainName'.>
<WARNING> <I/O error while reading domain directory>
java.io.IOException: Invalid state file format.

I followed a few guides to re-register the domain with Node Manager, and each time I tried it reported back a successful registration but the issue remained unresolved.

For server bi_server1, the Node Manager associated with machine MyServer is not reachable.

A search on Oracle support for the above turned up Doc ID 1293552.1, NodeManager not Reachable: java.io.IOException: Invalid State File Format.

I opted for option 2, stopping the node manager and admin server then deleting the following files:

…user_projects\domains\\servers\bi_server1\data\nodemanager\bi_server1.state …user_projects\domains\\servers\bi_server1\data\nodemanager\bi_server1.lck …user_projects\domains\\servers\bi_server1\data\nodemanager\bi_server1.pid

After starting up Node Manager, Admin Server, and BI_Server1, then restarting all the managed services in EM, the environment was back up.