Calculating total API traffic for an Organization ...

kurtkanaskie · ‎12-19-2022

Use Case

As a customer with annual API usage limits, I need to monitor total API usage during the year to avoid overage charges.

Problem

Apigee Edge or X does not currently offer customers an out-of-the box solution to determine overall API traffic usage for organizations to compare against entitlements.

Solution

Calculate API call volume using Apigee Metrics API for an organization and all of its environments. Combine the results for each organization for the grand total.

Here are a couple options.

Using Metrics API in a Bash script (supports both Edge and X).
Using apigeecli from command line (Apigee X only).

Metrics API

Multiple Metrics API calls are used to return a count of API traffic for an organization. For each environment a call is made for each month and the results calculated for the year. Monthly calls are used because the Metrics API time range in Apigee X is limited to 92 days, it's 365 days in Apigee Edge. Using a monthly time unit works for both and provides monthly details.

Metrics APIs have limits but they are not prohibitive, see analytics API limits for Apigee Edge and Apigee X.

Pseudo Code:

For an organization:

Get list of environments
For each environment
    For each month
        Get API usage stats
    Combine to get total usage for the environment
Consolidate to get total usage for all environments in the org

Using a Bash Script

Example Usage and Result:

$ api-usage-org.sh -o apigeex-org-name -t $(gcloud auth print-access-token) | jq
ORG: apigeex-org-name
ENVS: [ "prod", "test" ]
ENV_USAGE: {"name":"prod","months":[271,78,610,175,184,44,37,266,295,455,127,38],"total":2580},
ENV_USAGE: {"name":"test","months":[9503,6626,1951,6742,31077,28310,27198,12376,18358,107216,1446,427],"total":251230},
ORG_TOTAL: 253810
{
  "datetime": "Mon Dec 12 13:55:51 EST 2022",
  "name": "apigeex-org-name",
  "usage": [
    {
      "name": "prod",
      "months": [
        271,
        78,
        610,
        175,
        184,
        44,
        37,
        266,
        295,
        455,
        127,
        38
      ],
      "total": 2580
    },
    {
      "name": "test",
      "months": [
        9503,
        6626,
        1951,
        6742,
        31077,
        28310,
        27198,
        12376,
        18358,
        107216,
        1446,
        427
      ],
      "total": 251230
    }
  ],
  "total": 253810
}

Script to Calculate Year-to-date API traffic for an organization.

#! /bin/bash
# Works on GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20)
# Author: Kurt Kanaskie, Google (PII Removed by Staff)
# Updated: 2022-12-14

Help()
{
   # Display Help
   echo
   echo "Reports total API traffic for a specific year or for current year to date."
   echo "Defaults to report on an Apigee X organization."
   echo "Uses environment variables for ORG and TOKEN if set."
   echo
   echo "Syntax: $0 [-h|-l|-m|-o|-t|-y|-v"
   echo "options:"
   echo "    -h    Print this help"
   echo "    -l    Legacy Apigee Edge"
   echo "    -o    Organization name"
   echo "    -q    Quiet, don't show progress output"
   echo "    -s    Summary, don't show monthly details"
   echo "    -t    Access token"
   echo "    -y    Year"
   echo
   echo 'Usage X:     ./api-usage-org.sh -o apigeex-org-name -t $(gcloud auth print-access-token) | jq'
   echo 'Usage Edge:  ./api-usage-org.sh -l -o edge-org-name -t $(get_token) | jq'
   echo
   echo 'Save output: ./api-usage-org.sh -o x-org-name -t $(gcloud auth print-access-token) | jq > x-org-name.json'
   echo
}

ShowProgress()
{
	if [ $QUIET == false ]
	then
		>&2 echo $1
	fi
}

YEAR=`date "+%Y"`
MONTH=`date "+%m"`
APIGEE_API=apigee.googleapis.com
SHOW_MONTH_COUNT=true
QUIET=false

while getopts "o:y:t:hlsq" flag
do
	case "${flag}" in
		h) Help
			exit;;
		l) APIGEE_API="api.enterprise.apigee.com";;
		o) ORG=${OPTARG};;
		s) SHOW_MONTH_COUNT=false;;
		t) TOKEN=${OPTARG};;
		y) YEAR=${OPTARG}; MONTH=12;;
		q) QUIET=true;;
			\?) # Invalid option
			echo "Error: Invalid option"
			Help
		exit;;
	esac
done

if [ "$ORG" == "" ]
then
	echo
	echo "ERROR: Organization is required."
	Help
	exit
fi

DATE=`date`
AUTH="Authorization: Bearer $TOKEN"
ShowProgress "ORG: $ORG"
ENVS=$(curl -s -H "$AUTH" https://$APIGEE_API/v1/organizations/$ORG/environments | jq -r .)
if [ "$ENVS" == "" ] || [[ "$ENVS" == *'"error":'* ]]
then
	echo
	echo "ERROR: No environments or unauthorized."
	echo "ENVS: $ENVS"
	Help
	exit
fi

ShowProgress "ENVS: $ENVS"
ORG_TOTAL=0
ORG_USAGE='{"datetime":"'"$DATE"'","name":"'"$ORG"'","environments":['

for E in $ENVS
do
	# ENVS: [ "config", "dev", "prod", "test" ]
	# Skip square brackets
	if [ "$E" != "[" ] && [ "$E" != "]" ]
	then
		# E: "config",
		# remove any " or , characters
		temp1="${E#\"}"; temp2="${temp1%\,}"; ENV="${temp2%\"}"
		
		# echo ENV: $ENV
		ENV_TOTAL=0
		ENV_USAGE='"months":['
		for M in {1..12}
		do
			if [ "$M" -le "$MONTH" ]
			then
				if [ $M -lt "12" ]
				then
					TR=$M/01/$YEAR%2000:00~$(($M+1))/01/$YEAR%2000:00
				else
					TR=$M/01/$YEAR%2000:00~$M/31/$YEAR%2023:59
				fi

				MONTH_VALUE=$(curl -s -H "$AUTH" "https://$APIGEE_API/v1/organizations/$ORG/environments/$ENV/stats?select=sum(message_count)&timeRange=$TR" | jq -r .environments[0].metrics[0].values[0])
				# Bug Fix start: Edge returns exponential value so pipe to jq to convert to an integer, drop any ".0" or convert 4.7025005E7 to 47025005
				if [ "$MONTH_VALUE" == "null" ]
				then
					MONTH_VALUE=0
				else
					MONTH_VALUE=$(echo $MONTH_VALUE | jq)
				fi
				# Bug Fix end
				ENV_TOTAL=$(( $ENV_TOTAL + $MONTH_VALUE ))
				ENV_USAGE="${ENV_USAGE}${MONTH_VALUE},"
			fi
		done
		# Remove trailing , from last run through the loop
		ENV_USAGE="${ENV_USAGE%?}"
		if [ $SHOW_MONTH_COUNT == true ]
		then
			ENV_USAGE="{\"name\":\"${ENV}\",${ENV_USAGE}],\"total\":${ENV_TOTAL}}"
		else
			ENV_USAGE="{\"name\":\"${ENV}\",\"total\":${ENV_TOTAL}}"
		fi
		ShowProgress "ENV_USAGE: $ENV_USAGE"
		ORG_USAGE="${ORG_USAGE}${ENV_USAGE},"
		ORG_TOTAL=$(( $ORG_TOTAL + $ENV_TOTAL ))
	fi
done
# Output to stderr to show progress
ShowProgress "ORG_TOTAL: $ORG_TOTAL"
# Remove trailing , from last run through the loop
ORG_USAGE="${ORG_USAGE%?}"
ORG_USAGE="${ORG_USAGE}],\"total\":${ORG_TOTAL}}"
echo $ORG_USAGE

Using apigeecli

The apigeecli tool can be used to generate an on demand summary view of API usage for the entire organization over a year with or without monthly details.

Example year-to-date summary

apigeecli -t $TOKEN organizations reports yearly -o apigeex-org-name -y 2022
ORGANIZATION              YEAR                      API CALLS
apigeex-org-name          2022                      254143

Example year-to-date summary with monthly details

apigeecli -t $TOKEN organizations reports yearly -o apigeex-org-name -y 2022 --env-details
ENVIRONMENT               MONTH                     API CALLS
prod                      1/2022                    271
test                      1/2022                    9503
prod                      2/2022                    78
test                      2/2022                    6626
prod                      3/2022                    610
test                      3/2022                    1951
prod                      4/2022                    175
test                      4/2022                    6742
prod                      5/2022                    184
test                      5/2022                    31077
prod                      6/2022                    44
test                      6/2022                    28310
prod                      7/2022                    37
test                      7/2022                    27198
prod                      8/2022                    266
test                      8/2022                    12376
prod                      9/2022                    295
test                      9/2022                    18358
prod                      10/2022                   455
test                      10/2022                   107216
prod                      11/2022                   127
test                      11/2022                   1446
prod                      12/2022                   49
test                      12/2022                   749

Summary

ORGANIZATION              YEAR                      API CALLS
apigeex-org-name          2022                      254143

Hope this is helpful.

greatdevaks · ‎01-25-2023

@kurtkanaskie Amazing article.

For apigeecli, I believe the yearly reports would only be available for Apigee Enterprise Plus Subscription, right? Or does it work regardless of the Apigee Subscription type?

kurtkanaskie · ‎01-25-2023

Thanks, glad you found it useful.

Apigeecli and the Apigee API works for any subscription type, both are dependent on the data retention associated with each type.

kurtkanaskie · ‎03-15-2023

A bug has been identified and fixed, for Apigee Edge orgs that have traffic in the millions. The response from Edge API may include an exponential value (e.g. 4.7025005E7) which is valid JSON, but wasn't properly processed in the bash script.

The gist of the change is to convert values that include ".0" and convert exponential values to an integer using:

MONTH_VALUE=$(echo $MONTH_VALUE | jq)

I've updated the script above and noted the "Bug Fix" change.

OliverJacob12 · ‎03-26-2023

@kurtkanaskie wrote:
Use Case
As a customer with annual API usage limits, I need to monitor total API usage during the year to avoid overage charges.
Problem
Apigee Edge or X does not currently offer customers an out-of-the box solution to determine overall API traffic usage for organizations to compare against entitlements.
Solution
Calculate API call volume using Apigee Metrics API for an organization and all of its environments. Combine the results for each organization for the grand total.
Here are a couple options.
Using Metrics API in a Bash script (supports both Edge and X).
Using apigeecli from command line (Apigee X only).
Metrics API
Multiple Metrics API calls are used to return a count of API traffic for an organization. For each environment a call is made for each month and the results calculated for the year. Monthly calls are used because the Metrics API time range in Apigee X is limited to 92 days, it's 365 days in Apigee Edge. Using a monthly time unit works for both and provides monthly details.
Metrics APIs have limits but they are not prohibitive, see analytics API limits for Apigee Edge and Maintenance calorie calculator James Smith .
Pseudo Code:
For an organization:

Get list of environments
For each environment
    For each month
        Get API usage stats
    Combine to get total usage for the environment
Consolidate to get total usage for all environments in the org