As a customer with annual API usage limits, I need to monitor total API usage during the year to avoid overage charges.
Apigee Edge or X does not currently offer customers an out-of-the box solution to determine overall API traffic usage for organizations to compare against entitlements.
Calculate API call volume using Apigee Metrics API for an organization and all of its environments. Combine the results for each organization for the grand total.
Here are a couple options.
Multiple Metrics API calls are used to return a count of API traffic for an organization. For each environment a call is made for each month and the results calculated for the year. Monthly calls are used because the Metrics API time range in Apigee X is limited to 92 days, it's 365 days in Apigee Edge. Using a monthly time unit works for both and provides monthly details.
Metrics APIs have limits but they are not prohibitive, see analytics API limits for Apigee Edge and Apigee X.
For an organization:
Get list of environments
For each environment
For each month
Get API usage stats
Combine to get total usage for the environment
Consolidate to get total usage for all environments in the org
Example Usage and Result:
$ api-usage-org.sh -o apigeex-org-name -t $(gcloud auth print-access-token) | jq
ORG: apigeex-org-name
ENVS: [ "prod", "test" ]
ENV_USAGE: {"name":"prod","months":[271,78,610,175,184,44,37,266,295,455,127,38],"total":2580},
ENV_USAGE: {"name":"test","months":[9503,6626,1951,6742,31077,28310,27198,12376,18358,107216,1446,427],"total":251230},
ORG_TOTAL: 253810
{
"datetime": "Mon Dec 12 13:55:51 EST 2022",
"name": "apigeex-org-name",
"usage": [
{
"name": "prod",
"months": [
271,
78,
610,
175,
184,
44,
37,
266,
295,
455,
127,
38
],
"total": 2580
},
{
"name": "test",
"months": [
9503,
6626,
1951,
6742,
31077,
28310,
27198,
12376,
18358,
107216,
1446,
427
],
"total": 251230
}
],
"total": 253810
}
Script to Calculate Year-to-date API traffic for an organization.
#! /bin/bash
# Works on GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20)
# Author: Kurt Kanaskie, Google (PII Removed by Staff)
# Updated: 2022-12-14
Help()
{
# Display Help
echo
echo "Reports total API traffic for a specific year or for current year to date."
echo "Defaults to report on an Apigee X organization."
echo "Uses environment variables for ORG and TOKEN if set."
echo
echo "Syntax: $0 [-h|-l|-m|-o|-t|-y|-v"
echo "options:"
echo " -h Print this help"
echo " -l Legacy Apigee Edge"
echo " -o Organization name"
echo " -q Quiet, don't show progress output"
echo " -s Summary, don't show monthly details"
echo " -t Access token"
echo " -y Year"
echo
echo 'Usage X: ./api-usage-org.sh -o apigeex-org-name -t $(gcloud auth print-access-token) | jq'
echo 'Usage Edge: ./api-usage-org.sh -l -o edge-org-name -t $(get_token) | jq'
echo
echo 'Save output: ./api-usage-org.sh -o x-org-name -t $(gcloud auth print-access-token) | jq > x-org-name.json'
echo
}
ShowProgress()
{
if [ $QUIET == false ]
then
>&2 echo $1
fi
}
YEAR=`date "+%Y"`
MONTH=`date "+%m"`
APIGEE_API=apigee.googleapis.com
SHOW_MONTH_COUNT=true
QUIET=false
while getopts "o:y:t:hlsq" flag
do
case "${flag}" in
h) Help
exit;;
l) APIGEE_API="api.enterprise.apigee.com";;
o) ORG=${OPTARG};;
s) SHOW_MONTH_COUNT=false;;
t) TOKEN=${OPTARG};;
y) YEAR=${OPTARG}; MONTH=12;;
q) QUIET=true;;
\?) # Invalid option
echo "Error: Invalid option"
Help
exit;;
esac
done
if [ "$ORG" == "" ]
then
echo
echo "ERROR: Organization is required."
Help
exit
fi
DATE=`date`
AUTH="Authorization: Bearer $TOKEN"
ShowProgress "ORG: $ORG"
ENVS=$(curl -s -H "$AUTH" https://$APIGEE_API/v1/organizations/$ORG/environments | jq -r .)
if [ "$ENVS" == "" ] || [[ "$ENVS" == *'"error":'* ]]
then
echo
echo "ERROR: No environments or unauthorized."
echo "ENVS: $ENVS"
Help
exit
fi
ShowProgress "ENVS: $ENVS"
ORG_TOTAL=0
ORG_USAGE='{"datetime":"'"$DATE"'","name":"'"$ORG"'","environments":['
for E in $ENVS
do
# ENVS: [ "config", "dev", "prod", "test" ]
# Skip square brackets
if [ "$E" != "[" ] && [ "$E" != "]" ]
then
# E: "config",
# remove any " or , characters
temp1="${E#\"}"; temp2="${temp1%\,}"; ENV="${temp2%\"}"
# echo ENV: $ENV
ENV_TOTAL=0
ENV_USAGE='"months":['
for M in {1..12}
do
if [ "$M" -le "$MONTH" ]
then
if [ $M -lt "12" ]
then
TR=$M/01/$YEAR%2000:00~$(($M+1))/01/$YEAR%2000:00
else
TR=$M/01/$YEAR%2000:00~$M/31/$YEAR%2023:59
fi
MONTH_VALUE=$(curl -s -H "$AUTH" "https://$APIGEE_API/v1/organizations/$ORG/environments/$ENV/stats?select=sum(message_count)&timeRange=$TR" | jq -r .environments[0].metrics[0].values[0])
# Bug Fix start: Edge returns exponential value so pipe to jq to convert to an integer, drop any ".0" or convert 4.7025005E7 to 47025005
if [ "$MONTH_VALUE" == "null" ]
then
MONTH_VALUE=0
else
MONTH_VALUE=$(echo $MONTH_VALUE | jq)
fi
# Bug Fix end
ENV_TOTAL=$(( $ENV_TOTAL + $MONTH_VALUE ))
ENV_USAGE="${ENV_USAGE}${MONTH_VALUE},"
fi
done
# Remove trailing , from last run through the loop
ENV_USAGE="${ENV_USAGE%?}"
if [ $SHOW_MONTH_COUNT == true ]
then
ENV_USAGE="{\"name\":\"${ENV}\",${ENV_USAGE}],\"total\":${ENV_TOTAL}}"
else
ENV_USAGE="{\"name\":\"${ENV}\",\"total\":${ENV_TOTAL}}"
fi
ShowProgress "ENV_USAGE: $ENV_USAGE"
ORG_USAGE="${ORG_USAGE}${ENV_USAGE},"
ORG_TOTAL=$(( $ORG_TOTAL + $ENV_TOTAL ))
fi
done
# Output to stderr to show progress
ShowProgress "ORG_TOTAL: $ORG_TOTAL"
# Remove trailing , from last run through the loop
ORG_USAGE="${ORG_USAGE%?}"
ORG_USAGE="${ORG_USAGE}],\"total\":${ORG_TOTAL}}"
echo $ORG_USAGE
The apigeecli tool can be used to generate an on demand summary view of API usage for the entire organization over a year with or without monthly details.
Example year-to-date summary
apigeecli -t $TOKEN organizations reports yearly -o apigeex-org-name -y 2022
ORGANIZATION YEAR API CALLS
apigeex-org-name 2022 254143
Example year-to-date summary with monthly details
apigeecli -t $TOKEN organizations reports yearly -o apigeex-org-name -y 2022 --env-details
ENVIRONMENT MONTH API CALLS
prod 1/2022 271
test 1/2022 9503
prod 2/2022 78
test 2/2022 6626
prod 3/2022 610
test 3/2022 1951
prod 4/2022 175
test 4/2022 6742
prod 5/2022 184
test 5/2022 31077
prod 6/2022 44
test 6/2022 28310
prod 7/2022 37
test 7/2022 27198
prod 8/2022 266
test 8/2022 12376
prod 9/2022 295
test 9/2022 18358
prod 10/2022 455
test 10/2022 107216
prod 11/2022 127
test 11/2022 1446
prod 12/2022 49
test 12/2022 749
Summary
ORGANIZATION YEAR API CALLS
apigeex-org-name 2022 254143
Hope this is helpful.
@kurtkanaskie Amazing article.
For apigeecli, I believe the yearly reports would only be available for Apigee Enterprise Plus Subscription, right? Or does it work regardless of the Apigee Subscription type?
Thanks, glad you found it useful.
Apigeecli and the Apigee API works for any subscription type, both are dependent on the data retention associated with each type.
A bug has been identified and fixed, for Apigee Edge orgs that have traffic in the millions. The response from Edge API may include an exponential value (e.g. 4.7025005E7) which is valid JSON, but wasn't properly processed in the bash script.
The gist of the change is to convert values that include ".0" and convert exponential values to an integer using:
MONTH_VALUE=$(echo $MONTH_VALUE | jq)
I've updated the script above and noted the "Bug Fix" change.
@kurtkanaskie wrote:Use Case
As a customer with annual API usage limits, I need to monitor total API usage during the year to avoid overage charges.
Problem
Apigee Edge or X does not currently offer customers an out-of-the box solution to determine overall API traffic usage for organizations to compare against entitlements.
Solution
Calculate API call volume using Apigee Metrics API for an organization and all of its environments. Combine the results for each organization for the grand total.
Here are a couple options.
- Using Metrics API in a Bash script (supports both Edge and X).
- Using apigeecli from command line (Apigee X only).
Metrics API
Multiple Metrics API calls are used to return a count of API traffic for an organization. For each environment a call is made for each month and the results calculated for the year. Monthly calls are used because the Metrics API time range in Apigee X is limited to 92 days, it's 365 days in Apigee Edge. Using a monthly time unit works for both and provides monthly details.
Metrics APIs have limits but they are not prohibitive, see analytics API limits for Apigee Edge and Maintenance calorie calculator James Smith .
Pseudo Code:
For an organization:
Get list of environments
For each environment
For each month
Get API usage stats
Combine to get total usage for the environment
Consolidate to get total usage for all environments in the org
Example Usage and Result:
$ api-usage-org.sh -o apigeex-org-name -t $(gcloud auth print-access-token) | jq
ORG: apigeex-org-name
ENVS: [ "prod", "test" ]
ENV_USAGE: {"name":"prod","months":[271,78,610,175,184,44,37,266,295,455,127,38],"total":2580},
ENV_USAGE: {"name":"test","months":[9503,6626,1951,6742,31077,28310,27198,12376,18358,107216,1446,427],"total":251230},
ORG_TOTAL: 253810
{
"datetime": "Mon Dec 12 13:55:51 EST 2022",
"name": "apigeex-org-name",
"usage": [
{
"name": "prod",
"months": [
271,
78,
610,
175,
184,
44,
37,
266,
295,
455,
127,
38
],
"total": 2580
},
{
"name": "test",
"months": [
9503,
6626,
1951,
6742,
31077,
28310,
27198,
12376,
18358,
107216,
1446,
427
],
"total": 251230
}
],
"total": 253810
}
Script to Calculate Year-to-date API traffic for an organization.
#! /bin/bash
# Works on GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin20)
# Author: Kurt Kanaskie, Google (PII Removed by Staff)
# Updated: 2022-12-14
Help()
{
# Display Help
echo
echo "Reports total API traffic for a specific year or for current year to date."
echo "Defaults to report on an Apigee X organization."
echo "Uses environment variables for ORG and TOKEN if set."
echo
echo "Syntax: $0 [-h|-l|-m|-o|-t|-y|-v"
echo "options:"
echo " -h Print this help"
echo " -l Legacy Apigee Edge"
echo " -o Organization name"
echo " -q Quiet, don't show progress output"
echo " -s Summary, don't show monthly details"
echo " -t Access token"
echo " -y Year"
echo
echo 'Usage X: ./api-usage-org.sh -o apigeex-org-name -t $(gcloud auth print-access-token) | jq'
echo 'Usage Edge: ./api-usage-org.sh -l -o edge-org-name -t $(get_token) | jq'
echo
echo 'Save output: ./api-usage-org.sh -o x-org-name -t $(gcloud auth print-access-token) | jq > x-org-name.json'
echo
}
ShowProgress()
{
if [ $QUIET == false ]
then
>&2 echo $1
fi
}
YEAR=`date "+%Y"`
MONTH=`date "+%m"`
APIGEE_API=apigee.googleapis.com
SHOW_MONTH_COUNT=true
QUIET=false
while getopts "o:y:t:hlsq" flag
do
case "${flag}" in
h) Help
exit;;
l) APIGEE_API="api.enterprise.apigee.com";;
o) ORG=${OPTARG};;
s) SHOW_MONTH_COUNT=false;;
t) TOKEN=${OPTARG};;
y) YEAR=${OPTARG}; MONTH=12;;
q) QUIET=true;;
\?) # Invalid option
echo "Error: Invalid option"
Help
exit;;
esac
done
if [ "$ORG" == "" ]
then
echo
echo "ERROR: Organization is required."
Help
exit
fi
DATE=`date`
AUTH="Authorization: Bearer $TOKEN"
ShowProgress "ORG: $ORG"
ENVS=$(curl -s -H "$AUTH" https://$APIGEE_API/v1/organizations/$ORG/environments | jq -r .)
if [ "$ENVS" == "" ] || [[ "$ENVS" == *'"error":'* ]]
then
echo
echo "ERROR: No environments or unauthorized."
echo "ENVS: $ENVS"
Help
exit
fi
ShowProgress "ENVS: $ENVS"
ORG_TOTAL=0
ORG_USAGE='{"datetime":"'"$DATE"'","name":"'"$ORG"'","environments":['
for E in $ENVS
do
# ENVS: [ "config", "dev", "prod", "test" ]
# Skip square brackets
if [ "$E" != "[" ] && [ "$E" != "]" ]
then
# E: "config",
# remove any " or , characters
temp1="${E#\"}"; temp2="${temp1%\,}"; ENV="${temp2%\"}"
# echo ENV: $ENV
ENV_TOTAL=0
ENV_USAGE='"months":['
for M in {1..12}
do
if [ "$M" -le "$MONTH" ]
then
if [ $M -lt "12" ]
then
TR=$M/01/$YEAR%2000:00~$(($M+1))/01/$YEAR%2000:00
else
TR=$M/01/$YEAR%2000:00~$M/31/$YEAR%2023:59
fi
MONTH_VALUE=$(curl -s -H "$AUTH" "https://$APIGEE_API/v1/organizations/$ORG/environments/$ENV/stats?select=sum(message_count)&timeRange=$TR" | jq -r .environments[0].metrics[0].values[0])
# Bug Fix start: Edge returns exponetial value so pipe to jq to convert to an integer, drop any ".0" or convert 4.7025005E7 to 47025005
if [ "$MONTH_VALUE" == "null" ]
then
MONTH_VALUE=0
else
MONTH_VALUE=$(echo $MONTH_VALUE | jq)
fi
# Bug Fix end
ENV_TOTAL=$(( $ENV_TOTAL + $MONTH_VALUE ))
ENV_USAGE="${ENV_USAGE}${MONTH_VALUE},"
fi
done
# Remove trailing , from last run through the loop
ENV_USAGE="${ENV_USAGE%?}"
if [ $SHOW_MONTH_COUNT == true ]
then
ENV_USAGE="{\"name\":\"${ENV}\",${ENV_USAGE}],\"total\":${ENV_TOTAL}}"
else
ENV_USAGE="{\"name\":\"${ENV}\",\"total\":${ENV_TOTAL}}"
fi
ShowProgress "ENV_USAGE: $ENV_USAGE"
ORG_USAGE="${ORG_USAGE}${ENV_USAGE},"
ORG_TOTAL=$(( $ORG_TOTAL + $ENV_TOTAL ))
fi
done
# Output to stderr to show progress
ShowProgress "ORG_TOTAL: $ORG_TOTAL"
# Remove trailing , from last run through the loop
ORG_USAGE="${ORG_USAGE%?}"
ORG_USAGE="${ORG_USAGE}],\"total\":${ORG_TOTAL}}"
echo $ORG_USAGE
The apigeecli tool can be used to generate an on demand summary view of API usage for the entire organization over a year with or without monthly details.
Example year-to-date summary
apigeecli -t $TOKEN organizations reports yearly -o apigeex-org-name -y 2022
ORGANIZATION YEAR API CALLS
apigeex-org-name 2022 254143
Example year-to-date summary with monthly details
apigeecli -t $TOKEN organizations reports yearly -o apigeex-org-name -y 2022 --env-details
ENVIRONMENT MONTH API CALLS
prod 1/2022 271
test 1/2022 9503
prod 2/2022 78
test 2/2022 6626
prod 3/2022 610
test 3/2022 1951
prod 4/2022 175
test 4/2022 6742
prod 5/2022 184
test 5/2022 31077
prod 6/2022 44
test 6/2022 28310
prod 7/2022 37
test 7/2022 27198
prod 8/2022 266
test 8/2022 12376
prod 9/2022 295
test 9/2022 18358
prod 10/2022 455
test 10/2022 107216
prod 11/2022 127
test 11/2022 1446
prod 12/2022 49
test 12/2022 749
Summary
ORGANIZATION YEAR API CALLS
apigeex-org-name 2022 254143
Hope this is helpful.
For each environment a call is made for each month and the results calculated for the year. Monthly calls are used because the Metrics API time range in Apigee X is limited to 92 days, it's 365 days in Apigee Edge. Using a monthly time unit works for both and provides monthly details.