Edge Microgateway Error during Load Testing

[kkleva]

Hello,

We've been running a series of performance tests on an array of Edge Microgateway instances load balanced using Amazon ELB.

One very consistent error I've found causing the gateways to crash appears to be be caused by the number of open file connections. Here is the error we see in the logs:

Error: EMFILE: too many open files, uv_resident_set_memory
    at Error (native)
    at /home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:186:25
    at Array.forEach (native)
    at Object.serializeLogRecord (/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:149:36)
    at Object.logger.stats (/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:111:61)
    at null._repeat (/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:50:18)
    at wrapper [as _onTimeout] (timers.js:264:19)
    at Timer.unrefdHandle (timers.js:301:14)
/home/ec2-user/apigee-edge-micro-1.1.0/gateway/lib/logging.js:186
      var mem = process.memoryUsage();

The issue can be expected when running a load test and monitoring a list of open files:

lsof -i -n -P | grep node

What we find is that something with in Egde Micro appears to increase the number of open file handles until the ulimit is reached. Then this error occurs crashing the server and restarting. It appears to happen consistently when testing over 100 TPS.

Is this something unique to my installation? What can we do to prevent this kind of issue in production?

I'll attempt to increase the users ulimit -a (open files) to a higher amount than the default 1024, however is seems this would count more as a work around than a solution.