Today I hit a problem were Enterprise Manager was reporting managed server BI_server1 was down. That was a surprise as I was pretty sure it had just started with no problems!
Refresh EM. Still reported as down.
Restart everything. Still “down”.
Okay, back to the WLS Console. Nope, BI_server1 is definitely running. So why doesn’t EM agree?
After a bit of digging, it turned out that the Node Manager was unreachable, according to WLS Console. And the node manager was logging:
<WARNING> <There was a problem initializing the domain <domain> at 'D:\OBI\user_projects\domains\domainName'. Please make sure that this domainName: <domain> is registered and is fully enrolled for this nodemanager at: 'D:\OBI\user_projects\domains\domainName'.> <WARNING> <I/O error while reading domain directory> java.io.IOException: Invalid state file format.
I followed a few guides to re-register the domain with Node Manager, and each time I tried it reported back a successful registration but the issue remained unresolved.
For server bi_server1, the Node Manager associated with machine MyServer is not reachable.
A search on Oracle support for the above turned up Doc ID 1293552.1, NodeManager not Reachable: java.io.IOException: Invalid State File Format.
I opted for option 2, stopping the node manager and admin server then deleting the following files:
After starting up Node Manager, Admin Server, and BI_Server1, then restarting all the managed services in EM, the environment was back up.