The Art of Data Logging

when it comes to analyzing the effectiveness and usability of your applications, gathering the data you need in order to make decisions about what can be improved is essential. however, this data has to be retrieved and stored somewhere in order for meaningful reports to be generated. this information gathering is done through a process called data logging.

what is data logging?

data logging is the recording of data over time by either a specific standalone device or a computer system set up to capture data. the type of data recorded depends on the type of environment in which the data logger resides. for example, standalone data loggers typically gather information on things such as wildlife research, weather conditions, traffic conditions, actions of a vehicle such as a car or plane, and many others. computer systems set up for logging tend to gather data on the events that occur on a computer, network, server, or even an entire it system.

once recorded, the data can be obtained and analyzed either manually or using specific software that assists in the data analysis. such software can typically offer a high-level overview of the collected data for quick review as well as more detailed reports that allow for a deeper understanding of key metrics that the entity recording the data desires to capture.

logging data

to log data, you need to have a device or computer system that can be configured to capture the type of data required. the key advantage of data loggers is that they allow for information to be recorded 24 hours a day, seven days a week, which frees the researcher or analyst to perform tasks other than recording data. rather than needing to be at a specific location to collect data, someone can be analyzing previously recorded data from an office or lab.

for example, when performing wildlife research, wildlife may behave differently when one or more humans are present. automatically logging data while humans are not present can provide a great deal of insight that otherwise may not be discovered by direct observation. another example where this can be advantageous is logging weather related data, as most people do not want to carry measuring instruments into particularly bad storms in order to take specific weather measurements directly.

for the purposes of application monitoring, data logging records information pertinent to the maintenance of the infrastructure that is required for an application to run. this could be many types of data, such as actions performed by a user on a computer or software system, traffic to and from a server, or actions taken on an entire it network. there are many possibilities for what can be recorded, and all of this information can be helpful to systems administrators as they work to improve the operation of their specific systems.

for example, knowing the ip addresses of visitors to a server can help identify potential or ongoing attacks. many times, an overabundance of traffic from a particular ip range within a short time span could indicate that there is an attempted attack of some kind (such as attempting to quickly load pages with various injections to try to gain sensitive information). if that range of addresses can be effectively blocked, it may prevent the attack from succeeding, thus potentially saving a company quite a bit of time and money dealing with the aftermath of such an attack.

log data

when data is logged, it is typically stored either on the recording device itself or at another site to which it sends the data. more often than not, the data is stored in some type of textual format, which can then be parsed to organize the information into useful groups for statistics and analysis.

for example, a report from a weather station data logger can produce helpful reports about the status of numerous weather conditions over a period of time. these, of course, can help to determine if a warning should be issued for thunderstorms, wind or other conditions that might arise based on the acquired data and reports.

when it comes to logging data for apps and it systems, log files can be stored for numerous metrics. your organization can track visits, logins, usage statistics, and other useful information. for instance, if you are logging data on your website, you can track metrics such as page load times, number of visitors, what pages visitors viewed most, some indication of where the user is from, what site referred the visitor to your site (e.g., google search, a link on another site, social media), how long visitors stay on your site, and what page is most often used to leave the site (often called an exit page). these type of metrics can really assist your organization in adjusting various aspects of the site in order to improve the usability of the site for visitors and customers.

notice how there are some lines describing the log file, followed by numerous lines that contain particular information in the same order on each line. among other data, each line shows the time of the event, the type of request made (get, post, etc.), the file that was requested, and more. from a file such as this, numerous reports can be created which can be extremely helpful in analyzing how to make improvements to the website that can help future visitors by providing a faster and easier experience for them.

when it comes to logging data on apps, this could be expanded to log information not only on the website, but also on other pieces such as databases, application servers, load balancers, networks, and other parts of the it system related to any number of apps.

data logging systems

there are a number of data logging systems you can use. the options range from setting up your own logging on each of your servers to full-featured logging provided by app monitoring services which allow you to configure the type of events you wish to log and allow you to customize reports on all of the logged information.

setting up logging for all the parts of an app can be a tedious task since information will need to be logged from a number of different machines. depending on how your infrastructure is set up, there could be quite a lot of logging to set up and log files to parse. the more apps you add, the more work you will need to do to keep up with all of the necessary logging and the creation of meaningful reports.

for a smaller infrastructure, logging the data yourself may work fairly well. however, you will need to acquire or write software that can provide reports on the logged data. this could mean having several software systems in place from which key metrics will need to be obtained. over time, and especially with significant growth, it can become tedious to continue setting up the logging for each machine and getting various reporting software integrated for each new app.

ultimately, if you are running one or more apps in an environment that uses more than on a server, it is probably best to use a service that provides the data logging for you. this is often a feature of website/application monitoring services, as well as paas and other types of cloud services.

with an app monitoring service, you can set up the service to point to your servers and keep a log of when things are up and running, when things are in a warning state, and when they are down. alerts are also provided based on the settings you choose for when to be alerted to issues. depending on the service, there may also be additional information logged.

with a paas and many cloud-based services, you can add your entire app infrastructure to the service, which can then provide logging for you. in such cases, you can usually provision copies of particular services rather easily, simplifying the setup of any data logging and ensuring all the necessary data is captured. some of these services even allow you to have on-site servers, cloud servers, or a mixture of these (hybrid cloud). in addition, some may offer the monitoring services as well, giving you the ability to grab the key metrics of all your apps easily, while also ensuring that downtime is minimized.