Tapping into Logs

In today’s world of cloud-hosted service-oriented architectures, infrastructures are becoming increasingly complex and dynamic. Companies need a way to tell the full story of how their systems are behaving.

Managing logs and making them discoverable for an infrastructure is not a trivial task. Sometimes the effort of setting up logging solutions outweighs the benefits. Sumo Logic is one of the many log management platforms that gives organizations enhanced operational insight into their systems by leveraging log data.

We use their tools to deep dive into service outages and diagnose problems more efficiently. We can wire up Sumo Logic to our whole infrastructure and query very specific information across a large set of logs. While debugging, it's useful to track down an unhandled error message that only shows up in the logs of a specific VM in a production environment. Another benefit is being able to track the behavior of an application without maintaining a new database table.

There are several logging solutions out there with different limitations and pricing tiers. Most of them solve the same type of problems. Please do not consider this a marketing plug for Sumo Logic (who by the way, have been awesome to us). Hopefully you can take away some capabilities that a logging solution could unlock for your organization.

One of the main uses of a log management system is to take unformatted data and normalize it into a report. We can achieve this in Sumo Logic by logging rows with a distinct pattern that we can look for when trying to extract this data. This pattern can be some unique string, or a top level property of a JSON object. We will look at extracting information with the latter.

Let’s say our logs have JSON objects that look like this:

{
  "log_this_user":{
    "guid": "ada9fe4c-fbb5-4eec-b665-9cbfdd72e58b",
    "isActive": true,
    "age": 39,
    "eyeColor": "blue",
    "name": "Sue Washington"
  }
}
{
  "log_this_user":{
    "guid": "447634c8-59b4-4ea1-8554-60595d83c4c6",
    "isActive": true,
    "age": 26,
    "eyeColor": "blue",
    "name": "Rosemary Dotson"
  }
}
{
  "log_this_user":{
    "guid": "af66cf66-558c-44fa-91a8-77576071d173",
    "isActive": true,
    "age": 31,
    "eyeColor": "blue",
    "name": "Pitts Greer"
  }
}

We want to capture the following properties: guid, isActive, age, eyeColor, and name. We also know that these logs only come from user_service servers. Our query in Sumo Logic would look something like this:

First we search for any messages that contain the string ‘log_this_user’ within the user_service’s logs.

We are then pulling out the inner json object using regex into a field called ‘user_row’. Multi allows us to parse multiple occurrences of { “log_this_user”:{...}} within one message.

We then define each field we want to extract from the parsed JSON

The result messages would look something like:

There are several other ways that we can leverage log management platforms. Hopefully this example gives you an idea about some basic capabilities and prompts you to dive deeper into what these products have to offer.