Stack Your Logs In One Place With Loggly

Garrett Wilkin, April 13th, 2011

LogglyIf you’re writing a serious application, on the web or otherwise, you’ve got to have good logging.  In the context of application development, reading logs is my personal favorite method for tracking bugs and rooting them out.  A service like Loggly is perfect for programmers like me.   It provides an easy to use web interface with searching capabilities that allows you to see what your application is logging at any time that you’d like to check in.  Debugging with Loggly is nice, but it’s just the tip of the iceberg, thanks in part to its Loggly API.

Imagine that you’re a web programmer implementing your own “passion project.”  You start with one server and one application instance, but as the internet starts to show some traffic-love to your idea expansion becomes a pressing need.  There are plenty of cloud services out there that can replicate your application as needed, but that replication doesn’t solve the problems of coordination or analysis.  Once you have multiple instances of your application, how do you track what’s going on in any efficient way?  Loggly is your answer.  Consolidate your logs in one place and write your analytics to crunch that one data store.

I recently spoke with Kord Campbell, Loggly’s CEO and founder about his vision for the company.  When I asked him what he saw as Loggly’s ideal use case scenario he told me that he could see Loggly in literally hundreds of thousands of different applications.  Loggly’s aim will be to keep the service simple enough that it can be relevant to as many use cases as possible.  Certainly that’s the thing to do when you’re building a utility for the web.

Loggly is currently humming along with average daily load from a typical account hitting 8GB worth of log messages, with the big ones coming in at 20GB, and some outliers pushing the envelope with 100GB days here and there.  That’s a lot of data and it adds up quickly, which is part of Loggly’s value proposition: let Loggly take care of log retention so you don’t need to.  The next step beyond that is analysis.  Loggly presents a nice UI for searching and analyzing your logs over a period of time and there are more data crunching features in the works.

The Loggly RESTful API serves up JSON responses and offers two modes:   administration and dealing with logs.   To use either, you’ll have to gain clearance through OAuth.  This makes it a little harder to test the API from the command-line, but controlled access really is what you want, isn’t it?

In administration mode you can control both devices and inputs.  What are devices and inputs?  Two essential concepts that you need in order to understand loggly!  Devices send information to inputs.  This is a many to many relationship.  Under this model, you can have multiple devices sending data to a single input.  You can also have devices that send logs to multiple inputs.  Reflecting on this concept, one potential configuration could be a web application using one loggly input and one device to start.  If the application needs to be replicated due to high traffic, the new application instance could use the API to set up a new device on the same input.

With the API you can add, update, or delete inputs. You can even manage an input’s devices.  When it comes to devices, you can retrieve a list of all devices, and add or remove a device from an input.  Support for this type of administration through the API is a key to enabling new application instances to contact loggly to initiate new devices on existing inputs.

If you’re ready to dive in, check out the API docs, the NodeJS implementation of the API in Winston, or the node-loggly package, as well as the loggly team’s page on github.  Loggly is enjoying lots of interest from the development community as shown by the ever increasing amount of open source API implementations.

Both comments and pings are currently closed.

4 Responses to “Stack Your Logs In One Place With Loggly”

April 27th, 2011
at 8:01 am
Comment by: Time Is Money: Cashboard’s Invoicing API

[...] Let the experts handle yourdata center needs, the analysis and reporting of your data, or even the processing of your log files.  Now, let CashBoard provide you with time tracking software.  You can even write your own [...]

May 26th, 2011
at 12:56 pm
Comment by: Loggly Introduces User Defined, Variable Pricing | API Evangelist

[...] Stack Your Logs In One Place With Loggly (programmableweb.com) [...]

November 16th, 2011
at 8:03 am
Comment by: Librato Hits Sweet Spot: Metrics and Management Without Refactoring

[...] backend as a service providers, and much earlier we profiled niche companies like ReportGrid and Loggly.  All in all it adds up to a lower barrier to entry when creating a high performance web [...]

July 17th, 2012
at 11:10 am
Comment by: Loggly Announces Double-Down Investment Round

[...] I spoke to Loggly just over a year ago, and was excited and inspired by their vision and passion for creating value out of a technological by-product: log files.  In my recent conversation with the Loggly team they advocated for JSON formatted log data because structured data lends itself to indexing and analysis much more naturally. [...]

Follow the PW team on Twitter

ProgrammableWeb
APIs, mashups and code. Because the world's your programmable oyster.

John Musser
Founder, ProgrammableWeb

Adam DuVander
Executive Editor, ProgrammableWeb. Author, Map Scripting 101. Lover, APIs.