No transactions are getting recorded in Apigee On-Premise/Private Cloud after installing Monetization

If you have just installed Monetization in a 16.0.9 private cloud environment, and on-boarded an org but after setting up all monetization artifacts you do not see any transactions getting records, please check the following:

0. A quick way to check if transactions are being recording is to check in the postgresql database that was specified during the setup of Monetization:

psql -h /opt/apigee/var/run/apigee-postgresql/ -U apigee -d apigee
apigee=# select * from rkms.transaction;

If this returns 0 rows no transactions are getting recorded.

1. Make sure the transaction recording policy has been set up correctly. When you trace a request that you believe should result in a transaction getting recorded. You should see at the end of the trace, after clicking on the box labeled AX that the following Variables have the following values:

mint.tx.status: 200

mint.tx.ignoreMessage: false

If you do not have 200 or ok for status, and ignore is set to true then most likely something is wrong with the transaction recording policy, and you should revise the documentation here

4363-screen-shot-2017-02-16-at-62813-pm.png

2. Check if messages are making it into the mxgroup qpid queue on the qpid severs used in your setup using the qpid-stat -q command on all available qpid nodes.

If the ax-q-mxgroup and ax-q-mxgroup-dl have 0 messages, this suggests that the MPs are not sending any monetization messages. If that is the case you need to investigate the MPs or for a quick verification restart the MPs, and see if afterwards the queues are getting messages when you send requests that should trigger a transaction recording.

3. If you have messages in ax-q-mxgroup-dl but not in ax-q-mxgroup in the msgOut column, this suggests that the messages are NOT getting pushed from qpidd queue to postgres. The first thing to check is if there are edge-qpid-server log entries in /opt/apigee/var/log/edge-qpid-server/logs/system.log.

One issue Support has seen are errors which suggest that the postgresql database information for the mxgroup queue was not available to qpid-server:

2017-02-15 14:39:37,836  ax-q-mxgroup-persistpool-thread-1 WARN  c.a.a.m.MessageConsumer - MessageConsumer.process() : Sending message batch to the DLQ.com.apigee.analytics.persistence.api.PersistenceException: No Sink found  for qualifier MsgQualifier{queueName='ax-q-mxgroup', groupName='mxgroup', cg=null}. Rejecting messages        at com.apigee.analytics.persistence.api.AbstractPersistenceService.persist(AbstractPersistenceService.java:58) ~[persistence-service-api-1.0.0.jar:na]        at com.apigee.analytics.persistence.api.PersistenceDelegateService.persist(PersistenceDelegateService.java:86) ~[persistence-service-api-1.0.0.jar:na]        at com.apigee.analytics.messageconsumer.MessageConsumer.process(MessageConsumer.java:303) ~[message-consumer-service-1.0.0.jar:na]        at com.apigee.analytics.messageconsumer.MessageConsumer$PersistTaskExecutor.run(MessageConsumer.java:256) [message-consumer-service-1.0.0.jar:na]        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_111]        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_111]        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_111]
2017-02-15 14:41:42,841  ax-q-mxgroup-persistpool-thread-1 WARN  c.a.a.m.MessageConsumer - MessageConsumer.process() : Sending message batch to the DLQ.

When this error occurs, please restart the edge-qpid-server component to resolve this issue.

Comments
rajanish
New Member

just wanted to highlight "please restart the edge-qpid-server component to resolve this issue."after monetization is installation is completed.


@karl.kalckstein - Thanks for the post! This is very detailed and helpful steps to troubleshoot, not just qpid / group configuration issues but any monetization specific issues as well.

sam-was
New Member

We have a situation where we do see the data in ax-q-mxgroup but its not going out anywhere.

{quote}

Queues
queue dur autoDel excl msg msgIn msgOut bytes bytesIn bytesOut cons bind
==========================================================================================================================
ax-q-axgroup001 Y 0 1.03k 1.03k 0 2.51m 2.51m 12 2
ax-q-axgroup001-dl Y 0 0 0 0 0 0 0 2
ax-q-mxgroup Y 1.04k 1.04k 0 2.52m 2.52m 0 0 2

{quote}

sam-was
New Member

4875-apigee-logs-monetization-error.png

Errors we see in logs

rajanish
New Member
Few things i would check is
  1. check if consumer connection exists. The connection stat command should help qpid-stat -c
  2. check if qpid is reachable to mgmt server
    From mgmt host : telnet qpid_host:5636
  3. check to see if mgmt server timed out trying to connect. Below is the info log which indicates mgmt is no more trying to reconnect.
    log : "Gate keeper has 0 permits, waiting in queue" Management server restart will help in this case.
  4. If 1 to 3 looks good then need to check if monetization is installed correctly on management server. Simplest way to check is if mint api does not return 404 ${MGMT_HOST}/v1/mint/organizations/{ORG}
sam-was
New Member

Hi Thanks @rajanish@apigee.com

It was the firewall problem in our case which was causing the issue.

Version history
Last update:
‎02-16-2017 02:58 AM
Updated by: