Solved: MessageLogging Policy - Logstash - Page 2

Report Inappropriate Content · 10-13-2015 06:56 AM

Hello,

Right the issue is that I'm not seeing messages from the MessageLogging policy on a basic ELK stack that is running on a remote machine. (Using Apigee Edge Cloud FYI)

I've set up a new API Proxy, which is essentially a 'Hello World' nodejs example which I've then attached a MessageLogging policy to it's Proxy Endpoint, Post Flow Response.

As follows:

    <PostFlow name="PostFlow">
        <Request></Request>
        <Response>
            <Step>
                <Name>Message-Logging-LogStash</Name>
            </Step>
        </Response>
    </PostFlow>

And the Policy is set up as:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-LogStash">
    <DisplayName>Message Logging-LogStash</DisplayName>
    <Syslog>
        <Message>Message.id = {request.header.id}</Message>
        <Host>aremotehost</Host>
        <Port>443</Port>
        <Protocol>TCP</Protocol>
    </Syslog>
</MessageLogging>

(Note I've masked the remote host and the ootb default message)

On the listening remote host a proxy is set up within the webserver that is listening and sends the call to the ELK stack; this has been tested and works as expected; I can make calls to it and see the message's displayed via Kibana. This has been tested internally and externally.

When I trace the sending a request it seems to report that everything is fine and that the MessageLogging policy is getting a 200 response

Note: that I have also modified the host to something that does not resolve and had no luck, in fact Apigee reports 200 through out.

Looking around, I've seen that there have been a few successful attempts at this, such as here

Now I've tried the following Logstash input types:

syslog
tcp
http

All work as expected when making calls NOT via Apigee; but I'm unable to see anything from Apigee.

Is there anything obvious I'm missing?
Is there a way to see a more verbose trace/output/logging when using Apigee Edge Cloud?
How can I be certain that the MessageLogging policy is sending message as expected?

Thanks

Alex

Report Inappropriate Content

Right sorry for the radio silence... firstly thank you all for your help. My PoC works; so let me out line what's going on.

(Rather crude setup outline)

   Int           Firewall       Ext         
                    +                       
+---------------+   |                       
|               |   |                       
| Local Machine |   |                       
|               |   |     +----------------+
| +-----------+ |   |     |                |
| |           | |   |     | Apigee Message |
| |  haproxy  +-----------+ Logging Policy |
| |           | |   |     |                |
| +----+------+ |   |     +----------------+
|      |        |   |                       
| +----+------+ |   |                       
| |           | |   |                       
| | ELK Stack | |   |                       
| |           | |   |                       
| +-----------+ |   |                       
|               |   |                       
+---------------+   +

The ELK stack is essentially OOTB, the only thing to note is the Logstash instance config:

input {
    syslog    {
        type=>"syslog"
        port=>5544
        }
    }
output {
      elasticsearch { host => localhost }
        stdout { codec => rubydebug }
}

The Apigee Message Logging Policy is as follows:

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<MessageLogging async="false" continueOnError="false" enabled="true" name="Message-Logging-LogStash">
    <DisplayName>Message Logging-LogStash</DisplayName>
    <Syslog>
        <Message>message = {request.header.id:unknown_apigee_request_header_id}</Message>
        <Host>apigee-logger.domain.com</Host>
        <Port>443</Port>
        <Protocol>TCP</Protocol>
        <SSLInfo>
            <Enabled>true</Enabled>
        </SSLInfo>
    </Syslog>
</MessageLogging>

HAProxy was installed, and certs created (as this was just a PoC dummy certs were made and used)

listen logstash
   bind *:443 ssl crt /etc/ssl/domain.com/domain.com.pem
 
   mode   tcp
   option tcplog
 
   server log1 127.0.0.1:5544

Note that 'apigee-logger.domain.com' 443 traffic was routed to the local machine on 443.

Then using the 'trace' function we can make the calls and see:

haproxy logging:

Oct 14 16:25:57 localhost haproxy[7187]: 54.208.24.86:52526 [14/Oct/2015:16:25:52.881] logstash~ logstash/log1 199/1/4276 0 -- 0/0/0/0/0 0/0

Logstash Logging:

{
           "message" => "<14>Wed Oct 14 15:37:43 UTC 2015Info: message = unknown_apigee_request_header_id\u0000\n",
          "@version" => "1",
        "@timestamp" => "2015-10-14T15:37:44.002Z",
              "type" => "syslog",
              "host" => "127.0.0.1",
              "tags" => [
        [0] "_grokparsefailure_sysloginput"
    ],
          "priority" => 0,
          "severity" => 0,
          "facility" => 0,
    "facility_label" => "kernel",
    "severity_label" => "Emergency"
}

So that's kinda good I guess :)

View solution in original post