This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Router restart error

Posted on 03-23-2025 02:14 AM
abdullah_m
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

Hello there, 
I'm trying to restart edge-router on my on-prem apigee installation, 


I ran the command 

 

 

 /opt/apigee/apigee-service/bin/apigee-service edge-router restart

 

 

and i get the status as running. after several seconds, the status become NOT RUNNING(DEAD). 

Here's the log stream : 

 

 

2025-03-23 11:57:56,801  main INFO  REGISTRATION - ZKServerRegistrationAdapter.storeServerRegistrationPath() : Registering the server uuid e7f4bc75-5d0a-43bb-b265-fb99a533a487 with region dc-1 and pod gateway information
2025-03-23 11:57:56,806  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ServerRegistrationService
2025-03-23 11:57:56,806  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : ServerBindService has these dependencies [ [NameRegistration{name=ServerRegistrationService}] ]
2025-03-23 11:57:56,807  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ServerBindService
2025-03-23 11:57:56,807  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : ConfigStoreService has these dependencies [ [NameRegistration{name=ServerBindService}, NameRegistration{name=RepositoryService}, NameRegistration{name=HttpClientService}, NameRegistration{name=ComponentConfigurationService}] ]
2025-03-23 11:57:56,813  main INFO  SERVICES.CONFIGSTORE - ConfigStoreClientService.start() : Starting ConfigStoreClientServiceImpl with setting configstoreEnabled: false
2025-03-23 11:57:56,814  main WARN  SERVICES.CONFIGSTORE - ConfigStoreClientService.start() : Cache configuration ignored
2025-03-23 11:57:56,818  main INFO  SERVICES.CONFIGSTORE - ConfigStoreClientService.start() : Start complete
2025-03-23 11:57:56,818  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ConfigStoreService
2025-03-23 11:57:56,818  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : ServerBindObserverService has these dependencies [ [NameRegistration{name=ServerRegistrationService}] ]
2025-03-23 11:57:56,820  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ServerBindObserverService
2025-03-23 11:57:56,820  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : ClusterService has these dependencies [ [NameRegistration{name=ServerRegistrationService}, NameRegistration{name=CommunicationService}, NameRegistration{name=ServerBindObserverService}, NameRegistration{name=ZooKeeperService}, NameRegistration{name=MetricsService}] ]
2025-03-23 11:57:56,823  main INFO  CLUSTER - ClusterServiceImpl.start() : Registering to join a cluster with servers of type router
2025-03-23 11:57:56,823  main INFO  CLUSTER - ClusterServiceImpl.start() : Registering to join a cluster with servers of type message-processor
2025-03-23 11:57:56,831  main INFO  ZOOKEEPER - ZooKeeperServiceImpl.registerPathWatcher() : attaching watcher on path /regions/dc-1/pods/gateway/types/router/{uuid}, async is true
2025-03-23 11:57:56,991  main INFO  ZOOKEEPER - ZooKeeperServiceImpl.registerPathWatcher() : attaching watcher on path /regions/dc-1/pods/gateway/bindings/servers/{uuid}/organizations/{org}/environments/{env}, async is true
2025-03-23 11:57:57,011  main INFO  ZOOKEEPER - ZooKeeperServiceImpl.registerPathWatcher() : attaching watcher on path /regions/dc-1/pods/gateway/bindings/orgs/{org}, async is true
2025-03-23 11:57:57,020  main INFO  ZOOKEEPER - ZooKeeperServiceImpl.registerPathWatcher() : attaching watcher on path /regions/dc-1/pods/gateway/bindings/orgenvs/{org}/environments/{env}, async is true
2025-03-23 11:57:57,048  main INFO  SERVER.BINDING - ServerBindServiceImpl.internalGetBindings() : Could not fetch bindings from cache for uuid f42b4c40-7ae0-457a-8538-c536569edbf3. Fetching from repo and adding to cache
2025-03-23 11:57:57,070  pool-6-thread-1 INFO  c.a.i.InputValidator - InputValidator.getEnforcementTimestamp() : checkmarx enforcement timestamp set to 0
2025-03-23 11:57:57,079  main INFO  ZOOKEEPER - ZooKeeperServiceImpl.registerPathWatcher() : attaching watcher on path /regions/dc-1/pods/gateway/types/message-processor/{uuid}, async is true
2025-03-23 11:57:57,102  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of f42b4c40-7ae0-457a-8538-c536569edbf3 is now HB_FAILED. handle = 10.21.11.80 at 1742720277102
2025-03-23 11:57:57,143  main INFO  SERVER.BINDING - ServerBindServiceImpl.internalGetBindings() : Could not fetch bindings from cache for uuid 3c3de726-01ab-42cf-9777-82b771f8a5ec. Fetching from repo and adding to cache
2025-03-23 11:57:57,146  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of 3c3de726-01ab-42cf-9777-82b771f8a5ec is now HB_FAILED. handle = 10.21.11.80 at 1742720277146
2025-03-23 11:57:57,166  main INFO  SERVER.BINDING - ServerBindServiceImpl.internalGetBindings() : Could not fetch bindings from cache for uuid 8a9c6976-4d31-47f0-ac15-2c94d67b7ed8. Fetching from repo and adding to cache
2025-03-23 11:57:57,168  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of 8a9c6976-4d31-47f0-ac15-2c94d67b7ed8 is now HB_FAILED. handle = 10.21.11.81 at 1742720277168
2025-03-23 11:57:57,217  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : ClusterService
2025-03-23 11:57:57,218  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : OrganizationService has these dependencies [ [NameRegistration{name=CommunicationService}, NameRegistration{name=RepositoryService}] ]
2025-03-23 11:57:57,218  main INFO  ORGANIZATIONS - OrganizationServiceImpl.start() : Starting OrganizationService w/ http client enabled = false
2025-03-23 11:57:57,223  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of 3c3de726-01ab-42cf-9777-82b771f8a5ec is now CONNECTED. handle = 10.21.11.80 at 1742720277223
2025-03-23 11:57:57,224  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of 8a9c6976-4d31-47f0-ac15-2c94d67b7ed8 is now CONNECTED. handle = 10.21.11.81 at 1742720277224
2025-03-23 11:57:57,225  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of f42b4c40-7ae0-457a-8538-c536569edbf3 is now CONNECTED. handle = 10.21.11.80 at 1742720277225
2025-03-23 11:57:57,237  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : OrganizationService
2025-03-23 11:57:57,237  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : FeatureService has these dependencies [ [NameRegistration{name=OrganizationService}] ]
2025-03-23 11:57:57,249  main INFO  FeatureService - SystemDefaultConfiguration.setupXSLDoctypeFlag() : Allow XSL External entities system default is false. Trial mode is false.
2025-03-23 11:57:57,253  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Successfully started the service : FeatureService
2025-03-23 11:57:57,253  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : RuntimeConfigurationService has these dependencies [ [NameRegistration{name=ComponentConfigurationService}, NameRegistration{name=CommunicationService}, NameRegistration{name=LoadBalancingManagementService}, NameRegistration{name=LogService}, NameRegistration{name=ConfigStoreService}] ]
2025-03-23 11:57:57,254  main INFO  KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : LoadBalancingManagementService has these dependencies [ [NameRegistration{name=CommunicationService}, NameRegistration{name=ComponentConfigurationService}] ]
2025-03-23 11:57:57,260  main INFO  MESSAGING.RUNTIME - LoadBalancingManagementServiceImpl.initRelevantEventsSet() : Events supported by this server are [Organization, KeyStore, VirtualHost, Environment, ResourceReference, KeyStoreResource]
2025-03-23 11:57:57,268  main INFO  NGINX_ADAPTOR - NginxLoadBalancingAdaptor.initialize() : Initializing Nginx Adaptor
2025-03-23 11:57:57,273  main INFO  NGINX_ADAPTOR - Nginx.canWriteFile() : Testing if conf file /opt/nginx/conf.d/__apigee_____test_____testvh__.conf can be written
2025-03-23 11:57:57,274  main INFO  NGINX_ADAPTOR - Nginx.canWriteFile() : Testing if conf file /opt/nginx/conf.d/__apigee__-__test__-__testvh__.key can be written
2025-03-23 11:57:57,274  main INFO  NGINX_ADAPTOR - Nginx.canWriteFile() : Testing if conf file /opt/nginx/conf.d/__apigee__-__test__-__testvh__.cert can be written
2025-03-23 11:57:57,275  main INFO  NGINX_ADAPTOR - Nginx.canWriteFile() : Testing if conf file /opt/nginx/conf.d/__apigee__-__test__-__testvh__-client.pem can be written
2025-03-23 11:57:57,275  main INFO  NGINX_ADAPTOR - Nginx.isRunningAlready() : Finding status of Nginx on this machine using command /opt/nginx/scripts/apigee-nginx status
2025-03-23 11:57:57,301  main INFO  NGINX_ADAPTOR - Nginx.isRunningAlready() :  exit code of the command 3
2025-03-23 11:57:57,304  main INFO  NGINX_ADAPTOR - NginxLoadBalancingAdaptor.clearExistingConfigs() : Attempting to clear existing conf files at /opt/nginx/conf.d
2025-03-23 11:57:57,306  main WARN  NGINX_ADAPTOR - FileWriter$1.postVisitDirectory() : dir found under conf.d
2025-03-23 11:57:57,306  main WARN  NGINX_ADAPTOR - Nginx.checkConfig() : config testing with all files
2025-03-23 11:57:57,306  main INFO  NGINX_ADAPTOR - Nginx.checkConfig() : Testing config of Nginx on this machine using command [sh, -c, /opt/nginx/scripts/apigee-nginx configtest ]
2025-03-23 11:57:57,337  main INFO  NGINX_ADAPTOR - Nginx.checkConfig() :  exit code of the command [sh, -c, /opt/nginx/scripts/apigee-nginx configtest ] 0
2025-03-23 11:57:57,338  main INFO  NGINX_ADAPTOR - NginxLoadBalancingAdaptor.initialize() : Nginx config test passed
2025-03-23 11:57:57,338  main INFO  NGINX_ADAPTOR - NginxLoadBalancingAdaptor.initialize() : Nginx is not running trying to get service
2025-03-23 11:57:57,339  main INFO  NGINX_ADAPTOR - Nginx.start() : Trying to start Nginx using command /opt/nginx/scripts/apigee-nginx start
2025-03-23 11:57:59,881  main INFO  NGINX_ADAPTOR - NginxLoadBalancingAdaptor.initialize() : Unable to start Nginx
2025-03-23 11:57:59,883  main INFO  NGINX_ADAPTOR - Nginx.stop() : stopping Nginx using command /opt/nginx/scripts/apigee-nginx stop
2025-03-23 11:57:59,905  main INFO  NGINX_ADAPTOR - Nginx.stop() : Nginx stop complete
2025-03-23 11:57:59,906  main INFO  NGINX_ADAPTOR - NginxLoadBalancingAdaptor.initialize() : Initialized Nginx Adaptor status Failed
2025-03-23 11:57:59,907  main ERROR LOAD-BALANCER - LoadBalancingManagementServiceImpl.start() : Could not Initialize nginx adaptor
2025-03-23 11:57:59,912  main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.startService() : ServiceDeployer.deploy() : Got a life cycle exception while starting service [LoadBalancingManagementService, Unable to initialize load balancing adaptor] : {}
com.apigee.kernel.exceptions.spi.UncheckedException: Unable to initialize load balancing adaptor
        at com.apigee.lb.service.LoadBalancingManagementServiceImpl.start(LoadBalancingManagementServiceImpl.java:85)
        at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:229)
        at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:77)
        at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:233)
        at com.apigee.kernel.MicroKernel.start(MicroKernel.java:145)
        at com.apigee.kernel.MicroKernel.start(MicroKernel.java:140)
        at com.apigee.kernel.MicroKernel.main(MicroKernel.java:90)
2025-03-23 11:57:59,913  main ERROR KERNEL - MicroKernel.deployAll() : Error in deploying the deployment : LoadBalancingManagementService. Exception: {}
com.apigee.kernel.exceptions.spi.UncheckedException: Unable to initialize load balancing adaptor
        at com.apigee.lb.service.LoadBalancingManagementServiceImpl.start(LoadBalancingManagementServiceImpl.java:85)
        at com.apigee.kernel.service.deployment.ServiceDeployer.startService(ServiceDeployer.java:229)
        at com.apigee.kernel.service.deployment.ServiceDeployer.deploy(ServiceDeployer.java:77)
        at com.apigee.kernel.MicroKernel.deployAll(MicroKernel.java:233)
        at com.apigee.kernel.MicroKernel.start(MicroKernel.java:145)
        at com.apigee.kernel.MicroKernel.start(MicroKernel.java:140)
        at com.apigee.kernel.MicroKernel.main(MicroKernel.java:90)
2025-03-23 11:57:59,914  Thread-1 INFO  KERNEL - ShutdownHook.run() : ShutdownHook.run : System shutdown in progress...
2025-03-23 11:57:59,916  Thread-1 INFO  KERNEL.DEPLOYMENT - ServiceDeployer.stopService() : ServiceDeployer.unDeploy() : Successfully stopped the service : ServerBindService
2025-03-23 11:57:59,916  Thread-1 INFO  KERNEL.DEPLOYMENT - ServiceDeployer.stopService() : ServiceDeployer.unDeploy() : Successfully stopped the service : ServerRegistrationService
2025-03-23 11:57:59,917  Thread-1 INFO  KERNEL.DEPLOYMENT - ServiceDeployer.stopService() : ServiceDeployer.unDeploy() : Successfully stopped the service : RepositoryService
2025-03-23 11:57:59,918  Curator-Framework-0 INFO  o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.backgroundOperationsLoop() : backgroundOperationsLoop exiting
2025-03-23 11:57:59,923  Thread-1 INFO  KERNEL.DEPLOYMENT - ServiceDeployer.stopService() : ServiceDeployer.unDeploy() : Successfully stopped the service : ZooKeeperService
2025-03-23 11:57:59,923  Thread-1 INFO  KERNEL.DEPLOYMENT - ServiceDeployer.stopService() : ServiceDeployer.unDeploy() : Successfully stopped the service : MachineKeyService
2025-03-23 11:57:59,924  Thread-1 INFO  KERNEL.DEPLOYMENT - ServiceDeployer.stopService() : ServiceDeployer.unDeploy() : Successfully stopped the service : EventService
2025-03-23 11:57:59,931  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of f42b4c40-7ae0-457a-8538-c536569edbf3 is now DISCONNECTED. handle = 10.21.11.80 at 1742720279931
2025-03-23 11:57:59,933  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of 3c3de726-01ab-42cf-9777-82b771f8a5ec is now DISCONNECTED. handle = 10.21.11.80 at 1742720279933
2025-03-23 11:57:59,933  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of 8a9c6976-4d31-47f0-ac15-2c94d67b7ed8 is now DISCONNECTED. handle = 10.21.11.81 at 1742720279933
2025-03-23 11:57:59,938  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of f42b4c40-7ae0-457a-8538-c536569edbf3 is now HB_FAILED. handle = 10.21.11.80 at 1742720279938
2025-03-23 11:57:59,940  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of 8a9c6976-4d31-47f0-ac15-2c94d67b7ed8 is now HB_FAILED. handle = 10.21.11.81 at 1742720279940
2025-03-23 11:57:59,940  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of f42b4c40-7ae0-457a-8538-c536569edbf3 is now CONNECTED. handle = 10.21.11.80 at 1742720279940
2025-03-23 11:57:59,942  RPCClientClientProtocolChildGroup-RPC-0 INFO  CLUSTER - ServerState.setState() : State of 8a9c6976-4d31-47f0-ac15-2c94d67b7ed8 is now CONNECTED. handle = 10.21.11.81 at 1742720279942
2025-03-23 11:58:00,939  Thread-1 INFO  KERNEL.DEPLOYMENT - ServiceDeployer.stopService() : ServiceDeployer.unDeploy() : Successfully stopped the service : CommunicationService
2025-03-23 11:58:00,940  Thread-1 INFO  KERNEL.DEPLOYMENT - ServiceDeployer.stopService() : ServiceDeployer.unDeploy() : Successfully stopped the service : WebService

 

 

0 3 225
0 Likes
3 REPLIES 3

Posted on --/--/---- --:-- AM
AlexET
Staff
Reply posted on --/--/---- --:-- AM

Hello @abdullah_m,

We saw your question and wanted to let you know we’re keeping it on our radar. We’ll also invite others in the community to pitch in and share their thoughts.

0 Likes

Posted on --/--/---- --:-- AM
troseman
Staff
Reply posted on --/--/---- --:-- AM

The "edge-router" service is comprised of two independent servers, one of which starts the other. In the logs you have shared, edge-router is undergoing what we refer to as bootstrapping (fetching information from zookeeper and other source to build configuration files for nginx) then attempts to start the nginx server but fails to do so. Unfortunately there is not much in terms of context here that explains why nginx failed to start successfully. You may be able to find more clues elsewhere in nginx logs be it port bind failure or other status. If possible. I would recommend reaching out to Google Support for more direct assistance as they are better suited to provide more specific guidance.

However a few suggestions on where to look next. 

Within the folder `/opt/apigee/var/log/edge-router` you can find beyond the usual suspects the edge-router.log. While normally containing a subset of events this may hold other clues about the nginx start up failure. And `/opt/apigee/var/log/edge-router/nginx` may also have some additional context.

0 Likes

Posted on --/--/---- --:-- AM
FabiBo
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

Hello

We have had the same issue. 
In the output you see, that apigee tries to start the nginx:

2025-03-23 11:57:57,339  main INFO  NGINX_ADAPTOR - Nginx.start() : Trying to start Nginx using command /opt/nginx/scripts/apigee-nginx start

I tried to start the apigee-nginx directly, which was not possible, as the port is already in use:

# /opt/nginx/scripts/apigee-nginx start
Starting nginx: nginx: [emerg] bind() to 127.0.0.1:9080 failed (98: Address already in use)
nginx: [emerg] bind() to 127.0.0.1:9080 failed (98: Address already in use)
nginx: [emerg] bind() to 127.0.0.1:9080 failed (98: Address already in use)
nginx: [emerg] bind() to 127.0.0.1:9080 failed (98: Address already in use)
nginx: [emerg] bind() to 127.0.0.1:9080 failed (98: Address already in use)
nginx: [emerg] still could not bind()

After killing the process, the edge-router starts without problems.

# netstat -tulnp | grep :9080
tcp 0 0 127.0.0.1:9080 0.0.0.0:* LISTEN 1704/nginx: master 

# kill 1704

 

Top Solution Authors
View all