Long Tests: Saving All App’s Debug Logs and Writing Your Own Logs

Let's imagine we have an app installed on a Linux server in the cloud. This app uses a list of user proxies to establish an internet connection through them and perform operations with online resources.

The Problem

Sometimes, the app has connection errors. These errors are common, but it's unclear whether they stem from a bug in the app, issues with the proxies, network/OS conditions on the server (where the app is running), or just specific cases that don't generate a particular error message. These errors only occur sometimes and not with every proxy but with many different ones (SSH, SOCKS, HTTP(s), with and without UDP), providing no direct clues that the proxies are the cause. Additionally, it happens at a specific time of day (but this might be a coincidence). The only information available is a brief report from a user, lacking details. 

Short tests across different environments with various proxies and network conditions haven’t reproduced the problem, but the user claims it still occurs.

The Solution

• Rent the same server with the same configuration.
• Install the same version of the app.
• Run tests for 24+ hours to emulate the user's actions.
• Gather as much information as possible (all logs – app logs, user (test) action logs, used proxies, etc.) in a way that makes it possible to match IDs and obtain technical details in case of errors.

The Task

1. Write some tests with logs.
2. Find a way to save all the log data.

To make it more challenging, I'll introduce a couple of additional obstacles and assume limited resources and a deadline. By the way, this scenario is based on a real-world experience of mine, with slight twists and some details omitted (which are not important for the point).

Testing Scripts and Your Logs

I'll start with the simplest, most intuitive method for beginner programmers: when you perform actions in your scripts, you need to log specific information:

output_file_path = "output_test_script.txt"


def start():
#  your function logic

print(f'start: {response.content}')

with open(output_file_path, "a") as file:
	file.write(f'uuid is {uuid} -- {response.content} \n')

    
def stop():
	#  your function logic

print(local_api_data_stop, local_api_stop_response.content)

with open(output_file_path, "a") as file:
		file.write(f'{uuid} -- {response.content} \n')

    
#  your other functions and logic

if __name__ == "__main__":
with open(output_file_path, "w") as file:
		pass

Continuing, you can use print statements and save information on actions, responses, IDs, counts, etc. This approach is straightforward, simple, and direct, and it will work in many cases.

However, logging everything in this manner is not considered best practice. Instead, you can utilize the built-in logging module for a more structured and efficient logging approach.

import logging

# logger object
logger = logging.getLogger('example')
logger.setLevel(logging.DEBUG)

# file handler
fh = logging.FileHandler('example.log')
fh.setLevel(logging.DEBUG)

# formatter, set it for the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
fh.setFormatter(formatter)
ch.setFormatter(formatter)

# add the handlers to the logger
logger.addHandler(fh)
logger.addHandler(ch)

# logging messages
logger.debug('a debug message')
logger.info('an info message')
logger.warning('a warning message')
logger.error('an error message')

Details.

The first task is DONE!

Let's consider a case where your application has a debug log feature — it rotates with 3 files, each capped at 1MB. Typically, this config is sufficient. But during extended testing sessions lasting 24hrs+, with heavy activity, you may find yourself losing valuable logs due to this configuration.

To deal with this issue, you might modify the application to have larger debug log files. However, this would necessitate a new build and various other adjustments. This solution may indeed be optimal, yet there are instances where such a straightforward option isn’t available. For example, you might use a server setup with restricted access, etc. In such cases, you may need to find alternative approaches or workarounds.

Using Python, you can write a script to transfer information from the debug log to a log file without size restrictions or rotation limitations. A basic implementation could be as follows:

import time


def read_new_lines(log_file, last_position):
   with open(log_file, 'r') as file:
       file.seek(last_position)
       new_data = file.read()
       new_position = file.tell()
   return new_data, new_position


def copy_new_logs(log_file, output_log):
   last_position = 0


   while True:
       new_data, last_position = read_new_lines(log_file, last_position)
       if new_data:
           with open(output_log, 'a') as output_file:
               output_file.write(new_data)
       time.sleep(1)


source_log_file = 'debug.log'
output_log = 'combined_log.txt'


copy_new_logs(source_log_file, output_log)

Now let's assume that Python isn't an option on the server for some reason — perhaps installation isn't possible due to time constraints, permission limitations, or conflicts with the operating system and you don’t know how to fix it. In such cases, using bash is the right choice:

#!/bin/bash

source_log_file="debug.log"
output_log="combined_log.txt"

copy_new_logs() {
   while true; do
       tail -F -n +1 "$source_log_file" >> "$output_log"
       sleep 1
   done
}

trap "echo 'Interrupted! Exiting...' && exit" SIGINT
copy_new_logs

The second task is DONE!

With your detailed logs combined with the app's logs, you now have comprehensive debug information to understand the sequence of events. This includes IDs, proxies, test data, etc along with the actions taken and the used proxies. You can run your scripts for long hours without constant supervision. The only task remaining is to analyze the debug logs to get statistics and potential info on the root cause of any issues, if they even can be replicated according to user reports.

Some issues required thorough testing and detailed logging. By replicating the users’ setup and running extensive tests, we can gather important data for pinpointing bugs. Whether using Python or bash scripts (or any other PL), our focus on capturing detailed logs enables us to identify the root causes of errors and troubleshoot effectively. This highlights the importance of detailed logging in reproducing complex technical bugs and issues.