How ExaVault Tracks Account Performance with Distributed Tracing

Published on 17 Oct 2019

Tutorials

Our customers rely heavily on us to provide a consistently stable service for their business-critical transactions. In this blog post, we’ll walk you through how we achieve this through distributed tracing, and with the help of our friends at Instana.

The ExaVault API

The ExaVault API powers all ExaVault services. Both our web interface and our FTP servers make API calls to fulfill requests. Our customers, too, utilize our API to integrate our platform into their systems. Therefore, the performance of our API is critical, and any issues with it can ripple and affect other layers of the platform. Our API on average handles 35K requests per minute and over 50 MM calls daily.

Drilling Down to Account Level Performance

Often a customer will report a problem that we can’t reproduce on our test accounts, so we need the ability to drill down into the performance of their account only. Fortunately, with Instana’s SDK, this is not only possible but very easy to set up and use. Having the ability to do so can help reduce investigation time drastically.

As requests come into our API, Instana records the endpoint that was accessed and automatically measures useful metrics such as the latency of each call and how many calls are made to other layers in the stack, like the database or caching layer.

Utilizing the Instana SDK

Instana’s SDK supports tagging each request with extra metadata as they are processed via our API. These pieces of metadata are extremely useful to help us investigate issues. The metadata that we associate with the calls are entirely up to us:

account – the account name of the client accessing the service
user – the name of the account’s user
IP address – IP address of the user
parameters – a JSON encoded list of parameters that were passed

Here’s what we needed to do to get the Instana SDK set up to track this metadata for us.

Step 1. Download and install the monitor

At ExaVault, we utilize Chef to handle DevOps automation. Installing Instana is a piece of cake; we just downloaded and installed the ‘instana-agent’ cookbook to our application server run list. This way, the agent is installed automatically across all of our application servers.

If you’re not running Chef, installing the Instana agent is very simple. See the agent setup instructions for more info.

Step 2. Set up code to utilize the SDK

ExaVault is primarily powered by PHP, so the examples below will be PHP specific. However, the concepts in the examples should be able to be easily applied to other languages and frameworks.

The ExaVault API is built on the Lumen framework, a modern lightweight framework designed for APIs. This presented a couple of framework-specific challenges with the Instana SDK that I’ll discuss in a little bit.

We have a front controller that processes all incoming requests for our API. In this controller, we have the following code to begin and annotate a new Instana trace:

// Check for the Instana tracer before proceeding
if (class_exists('\Instana\Tracer')) {
  // Create a new tracer object and set the tracking 
  // span named for the endpoint 
  $tracer = new \Instana\Tracer();
  $span = $tracer->createSpan($method);

  // If we know who this user is, annotate the trace
  // with the user info.
  if (!is_null($user)) {
    $span->annotate('username', $user->getUsername());
    $span->annotate('account', $user->getAccountName());
  }
  // We should always have an IP address to annotate
  $span->annotate("ipAddress", $this->_ipAddress);

  // Now loop through all of the parameters
  foreach ($params as $key=>$value) {
    // sanitize data here (code removed)

    // If a parameter is an array or object, convert
    // it to text before annotating
    if (is_array($value) || is_object($value)) {
      $value = print_r($value, true);
    }                    
    $span->annotate("" . $key, "" . $value);
  }
}

The basic idea in the above code is that we create a new Instana tracer object, start a trace, and add metadata to track using the “annotate” function, which accepts a string key/value pair.

At some point in the code, we must call a function on the tracer to stop the trace. Because we’re using a framework which passes the request object between middlewares, we must add an additional middleware to stop the trace after the call has been executed. The span is returned to the next middleware so it can be accessed later:

 $request->merge(['internal_vars' => [
	       /// ...
            'instanaSpan' => $span,
            'user' => $user,
            'ipAddress'=> $this->_ipAddress,
            'method' => $method
        ]]);
        
        return $next($request);

In a separate middleware that is configured to run last, the span is stopped which sets the latency and records the data to Instana:

class InstanaStop
{
    public function handle($request, Closure $next)
    {

        $internalVars = $request->input('internal_vars');
        $span = $internalVars['instanaSpan'];

        $response = $next($request);

        if(class_exists('\Instana\Tracer')) {
            $span->stop();
        }

        return $response;
    }
}

Step 3: Configure Instana to filter by custom metadata

Instana’s web interface features powerful filtering tools that allow us to drill down into various metrics. In order to filter by our custom metadata that we set up via the SDK, we need to do the following:

In the Instana web UI, go to the Analyze tab.
Under the filters subtab, select “more”.
In the Add Filters dialog box that appears, choose “call.tag”.
In the key field, enter the key you want to filter on. In our example, we would enter “account” to filter on the account name that we associated calls with.
In the value field, enter the value of the key you want to filter. In our case, we’ve chosen “ftptest”, one of our production test accounts.
Hit “Add Filter”.

The analyze tab should now only show calls for that specific account. Additionally, if you select the “show graph” button, you’ll get a handy visual breakdown of the calls made by that account. In this graph, you can see that there are regular latency spikes for one of the API calls approximately every 15 minutes.

With the ability to drill down and analyze specific user elements, we are now able to effectively pinpoint what may be causing an issue in a fraction of the time. Filtering by custom metadata through the Instana SDK answered our needs. Though the decision to go with Instana had already been made, after getting everything set up and configured to make the correct calls through our API, we could not be happier with our choice of Application Performance Monitoring (APM).

You can learn more about how ExaVault optimizes customer experience through automatic application monitoring. Watch our webinar co-hosted with Instana – How ExaVault Reduces Downtime and Improves Customer Experience. Happy Monitoring!