customizing apigee-templater to include FlowCallouts & existing org parms for ApigeeX

Team , we are trying to see if we could customize apigee templater (https://github.com/apigee/apigee-templater) to include FlowCallout to use our existing reusable shared flows (for logging, error, security) & update parameters for our orgs & environments. we dont see any samples to do that even though the github link talks about samples. instructions/direction for including FlowCallout & updating parameters like org name, environments etc would help.

We wanted to expose the web client available through apigee-templater to our client application developer community which has very minimal knowledge on apigee but still use the web client of apigee-templater to build & deploy the api proxy with the standard instructions & steps to reduce the dependency on api management team within our org.

@tyayers @kurtkanaskie @dchiesa1 

 

1 5 334
5 REPLIES 5

I don't know that much about the apigee-templater project, but ... I understand that it can be used as a generator tool, to build API proxy bundles from the command line, based on input you provide.

From the README, I understand that there is a fixed set of template options the tool offers. For example, you can choose one of 3 different target types: generic HTTP Urls, BigQuery Queries, or BigQuery Tables. And there are options for APIKey or token validation, and you can include or exclude quotas and spike arrest policies. That all seems like a pretty cool set of features. But I guess most people would want a little more control and flexibility. And for that the rEADME says there's an extensibility model.

I looked briefly into that, and ... it seems like you need to learn a novel plugin framework, a new mental model, to extend the tool to allow it to build from a different template. It seems like you'd need to explore that and learn how to do that. I'm sorry I don't have any experience with that framework and I'm not clear on whether flowcallouts and the like would be easy to include in the thing. Seems like you'd need to write some Typescript and so on to make that happen, but I don't know for sure.

Facing the same challenge, I built an alternative tool, called apigee-proxy-generator, which uses a different approach for extensibility. Rather than defining a new JS or Typescript programming model for the proxy templates, I just used the filesystem. Basically you define your template in terms of an existing API proxy, with placeholders in it. So if you want a FlowCallout, you just insert that FlowCallout into the ProxyEndpoint or TargetEndpoint, and it's there. It's just like authoring an API proxy, except that you have the opportunity to insert placeholders that get filled in with data at the time you run the generator. This seems like a more approachable way to define templates for your proxies, because you can include whatever you like in the template, and there's no need to learn a new model, or build a thing like "FlowCalloutPlugin()" or whatever, in Typescript.

This latter generator tool is not "interactive" - it doesn't have an execution mode in which it asks a bunch of questions interactively, and then does it's generation work. Instead you need to provide all of the input data to the tool in a file. But it seems to me this latter generator tool is more flexible, with the templates as well as the data that you combine with the templates. You can define an arbitrary number of conditional flows. You can have arbitrary policies including FlowCallouts, ExternalCallout, Dialogflow, etc... You can include resources like properties files or OASpec files into your generated proxy. You can supply arbitrary custom data to embed into the generated API proxy.  

The trickiest thing with this latter tool is... the template itself is a lodash template. lodash is a nodejs module that is pretty broad and wide, but ONE of the things it offers is a template engine. Using a template like "Hello {{= name}}" , you combine it with a dataset like { "name" : "Raghu" } and get an output like "Hello Raghu". Obviously it gets more involved than that simple example, but that's the basic idea. And because Apigee proxy definitions are just text files, applying this template approach to a proxy bundle is easy. The trick is, when authoring new proxy templates, for example, for a proxy that invokes a FlowCallout and so on, you probably need to know the lodash template syntax. You need to know how to specify {{= flow.name}} in the template to provide the name for a conditional flow. And so on. There are examples you can follow, and to ME it's not too difficult, but for other people who don't know lodash, it might be an obstacle. Can you decipher this:? 

  <Flows>

    <!-- there is one flow for each item in the config -->
    {{
    flows.forEach( (flow,ix) => {
      let amPolicyName = 'AM-PreparedQuery-' + (ix+1);
      let d = path.dirname(sourceFilename),
          fqAssignMessagePolicyFileName = path.resolve(d, '..', 'policies', amPolicyName + '.xml');
      let avContent1 =
          "  <AssignVariable>\n"+
          "    <Name>bq_query</Name>\n" +
          `    <Template>${flow.query}</Template>\n`+
          "  </AssignVariable>\n";
      let avContent2 = ''; /* will optionally hold another AssignVariable */
    }}

    <Flow name="{{= flow.name}}">
      <Request>
        <Step>
          <Name>VerifyAPIKey-1</Name>
        </Step>
    {{
        let pathPattern = '';
        // the query may have named perameters. Extract them. No possibility for SQL injection here.
        let re = new RegExp('{[_a-zA-Z][_a-zA-Z0-9]+}','g');
        let m = flow.path.match(re);
        if (m) {
          ....

Bottom line, between the apigee-templater project and this latter generator tool, you have some options.

Hi @dchiesa1 , seeing below error when we try to generate code using apigee-proxy-generator. could you please help.

% node ./genProxyFromTemplate.js \

    --generateonly \

    --source ../templates/bq-simple-proxy-template \

    --config ../data/config-bq-flights.json

node:internal/modules/cjs/loader:1042

  throw err;

  ^

 

Error: Cannot find module 'apigee-edge-js'

Require stack:

- /Users/xman/tools/apigee-proxy-generator-main/tools/genProxyFromTemplate.js

    at Module._resolveFilename (node:internal/modules/cjs/loader:1039:15)

    at Module._load (node:internal/modules/cjs/loader:885:27)

    at Module.require (node:internal/modules/cjs/loader:1105:19)

    at require (node:internal/modules/cjs/helpers:103:18)

    at Object.<anonymous> (/Users/xman/tools/apigee-proxy-generator-main/tools/genProxyFromTemplate.js:26:18)

    at Module._compile (node:internal/modules/cjs/loader:1218:14)

    at Module._extensions..js (node:internal/modules/cjs/loader:1272:10)

    at Module.load (node:internal/modules/cjs/loader:1081:32)

    at Module._load (node:internal/modules/cjs/loader:922:12)

    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) {

  code: 'MODULE_NOT_FOUND',

  requireStack: [

    '/Users/xman/tools/apigee-proxy-generator-main/tools/genProxyFromTemplate.js'

  ]

}

 

Node.js v18.13.0

raghunatha.palle@JNVKJ34QMX tools % npm install apigee-edge-js      

npm WARN skipping integrity check for git dependency ssh://git@github.com/tuxpoldo/node-getopt.git

npm WARN deprecated har-validator@5.1.5: this library is no longer supported

npm WARN deprecated uuid@3.4.0: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.

npm WARN deprecated request@2.88.0: request has been deprecated, see https://github.com/request/request/issues/3142

npm ERR! code UNABLE_TO_GET_ISSUER_CERT_LOCALLY

npm ERR! errno UNABLE_TO_GET_ISSUER_CERT_LOCALLY

npm ERR! request to https://codeload.github.com/tuxpoldo/node-getopt/tar.gz/05e498731c14b648fa332ca78d3a301c5e4be440 failed, reason: unable to get local issuer certificate

 

npm ERR! A complete log of this run can be found in:

npm ERR!     /Users/xman/.npm/_logs/2023-08-17T13_59_11_507Z-debug-0.log

A couple things.

  1. Before you try

    node ./genProxyFromTemplate.js ...

    ... you need to run

    npm install

    You don't want to install just apigee-edge-js, you want to install all the dependencies. The npm install command will install all dependencies.

  2. The error complaning about "unable to get local issuer certificate" has to do with your list of trusted certificates and possibly your corporate proxy. Basically npm is connecting to that hostname and it does not trust the certificate. Why codeload.github.com might be omitted from your trusted CA certs list is something you can talk to your network security people about. But you may be able to add the required cert to your list of trusted certs, if you follow this advice. Or, one of the other options within that Stackoverflow answer. If you choose to disable TLS, you should re-enable it immediately after running npm install.

That seems like it should be straight forward, but I don't have experience with the apigee-templater project or Dino's tool (which I could not easily decipher).

I've seen these types of requests before, I guess the first task is to survey your API designs to identify "standard" patterns. Then create the requisite Shared Flows to support all of the patterns. Once you have that, which I think you already do, you're ready to consider the effort to use either the apigee-templater or the generator tool. But it's not all about the tool to create the proxy, there's more. Let's take a step back and look at the broader picture.

An alternate approach would be to use Git Repos to hold the various templated proxy configurations. Then your developers merely browse the repos, clone the one they want and build/deploy using CI/CD (e.g. maven). Maven can easily be configured to perform templated replacement of values (maven replacer plugin does this). The benefit of this approach is that you can now incorporate full lifecycle CI/CD for your developers, not just API Proxy creation. By full lifecycle CI/CD I mean, a phased approach starting with static linting (including Apigeelint which can also be extended to ensure specific Shared Flows are used), unit testing (for JS), integration testing (Apickli / Gherkin based BDD) and Open API Spec publishing. The maven based full lifecycle CI/CD supports both proxy deployment and configuration management. In addition to creating and deploying the proxy, it can create KVMs, Target Servers, API Products, test developers, test apps and even download the keys to test the APIs automatically.

Using this approach, you not only get repeatable / consistent designs, you also get a mature Git based approach to API Proxy development, testing and deployment.

Just saw this thread, in case any questions are still open to apigee-templater, just reply here or message me, happy to help.