It could rightfully be said that logs are one of the most underestimated and underutilized tools at a freelance php developer’s disposal. Despite the wealth of information they can offer, it is not uncommon for logs to be the last place a developer looks when trying to resolve a problem. Show
In truth, PHP log files should in many cases be the first place to look for clues when problems occur. Often, the information they contain could significantly reduce the amount of time spent pulling out your hair trying to track down a gnarly bug. But perhaps even more importantly, with a bit of creativity and forethought, your logs files can be leveraged to serve as a valuable source of usage information and analytics. Creative use of log files can help answer questions such as: What browsers are most commonly being used to visit my site? What’s the average response time from my server? What was the percentage of requests to the site root? How has usage changed since we deployed the latest updates? And much, much more. This article provides a number of tips on how to configure your log files, as well as how to process the information that they contain, in order to maximize the benefit that they provide.
Note: This article presumes basic familiarity with the Unix shell. For those lacking this knowledge, an Appendix is provided that introduces some of the commands needed for accessing and reading log files on a Unix system. Our PHP Log File Example ProjectAs an example project for discussion purposes in this article, we will take Symfony Standard as a working project and we’ll set it up on Debian 7 Wheezy with
This quickly gives us a working test project with a nice UI. Tips for Configuring Your Log FilesHere are some pointers on how to configure your log files to help maximize their value. Error Log ConfgurationError logs represent the most basic form of logging; i.e., capturing additional information and detail when problems occur. So, in an ideal world, you would want there to be no errors and for your error logs to be empty. But when problems do occur (as they invariably do), your error logs should be one of the first stops you make on your debugging trail. Error logs are typically quite easy to configure. For one thing, all error and crash messages can be logged in the error log in exactly the same format in which they would otherwise be presented to a user. With some simple configuration, the end user will never need to see those ugly error traces on your site, while devops will be still able to monitor the system and review these error messages in all their gory detail. Here’s how to setup this kind of logging in PHP:
Another two lines that are important to include in a log file for a live site, to preclude gory levels of error detail from being to presented to users, are:
System Log (syslog) ConfgurationThere are many generally compatible implementations of the
(Note: In this article, we’ll be using The basic syslog configuration is generally adequate for capturing your log messages in a system-wide log file (normally The system log provides several log facilities,
eight of which (
Now, whenever you write a log message to One additional important note; The In order to shut down a less important level of logging (e.g., on a production server), you may simply redirect related messages to
One specific customization that is useful, especially to support some of the PHP log file parsing we’ll be discussing later
in this article, is to use tab as the delimiter character in log messages. This can easily be done by adding the following file in
And finally, don’t forget to restart the syslog daemon after you make any configuration changes in order for them to take effect:
Server Log ConfgurationUnlike application logs and error logs that you can write to, server logs are exclusively written to by the corresponding server daemons (e.g., web server, database server, etc.) on each request. The only “control” you have over these logs is to the extent that the server allows you to configure its logging functionality. Though there can be a lot to sift through in these files, they are often the only way to get a clear sense of what’s going on “under the hood” with your server. Let’s deploy our Symfony Standard example application on nginx environment with MySQL storage backend. Here’s the nginx host config we will be using:
With regard to the last two directives above: Note: This is an intentionally oversimplified nginx config file that is provided for example purposes only. It pays almost no attention to security and performance and shouldn’t be used as-is in any “real” environment. This is what we get in
This shows that, for serving one page, the browser actually performs 9 HTTP calls. 7 of those, however, are requests to static content, which are plain and lightweight. However, they still take network resources and this is what can be optimized by using various sprites and minification techniques. While
those optimisations are to be discussed in another article, what’s relavant here is that we can log requests to static contents separately by using another
Remember that nginx Parsing such logs is no different than parsing application logs. Other Logs Worth MentioningTwo other PHP logs worth mentioning are the debug log and data storage log. The Debug LogAnother convenient thing about nginx logs is the debug log. We can turn it on by replacing the
The same setting applies for Apache or whatever other webserver you use. And incidentally, debug logs are not related to error
logs, even though they are configured in the Although the debug log can indeed be verbose (a single nginx request, for example, generated 127KB of log data!), it can still be very useful. Wading through a log file may be cumbersome and tedious, but it can often quickly provide clues and information that greatly help accelerate the debugging process. In particular, the debug log can really help with debugging nginx configurations, especially the most complicated
parts, like Of course, debug logs should never be enabled in a production environment. The amount of space they use also and the amount of information that they store means a lot of I/O load on your server, which can degrade the whole system’s performance significantly. Data Storage LogsAnother type of server log (useful for debugging) is data storage logs. In MySQL, you can turn them on by adding these lines:
These logs simply contain a list of queries run by the system while serving database requests in chronological order, which can be helpful for various debugging and tracing needs. However, they should not stay enabled on production systems, since they will generate extra unnecessary I/O load, which affects performance. Writing to Your Log FilesPHP itself provides functions for opening, writing to, and closing log files ( There are also numerous logging libraries for the PHP developer, such as Monolog (popular among
Symfony and Laravel users), as well as various framework-specific implementations, such as the logging capabilities incorporated into CakePHP. Generally, libraries like Monolog not only wrap Here’s a simple example of how to write to the log:
Our call here to
Here’s what the contents of the log file would look like after running the above code:
Maximizing the Value of Your PHP Log FilesNow that we’re all good with theory and basics, let’s see how much we can get from logs making as few changes as possible to our sample Symfony Standard project. First, let’s create the scripts
And let’s require these scripts in
For
the development environment, we want to require these scripts in We also want to track what controllers are being invoked, so let’s add one more line in
Note that these changes total a mere 15 extra lines of code, but can collectively yield a wealth of information. Analyzing the Data in Your Log FilesFor starters, let’s see how many HTTP requests are required to serve each page load. Here’s the info in the logs for one request, based on the way we’ve configured our logging:
So now we know that each page load is actually served with two HTTP requests. Actually there are two points worth mentioning here. First, the two requests per page load is for using Symfony in
dev mode (which I have done throughout this article). You can identify dev mode calls by searching for OK, now let’s surf a bit on the demo site (to build up the data in the log files) and let’s see what else we can learn from these logs.
We see that the number of
This tells us that there were 2 page loads of the site root. Since we previously learned that (a) there are 2 requests to the app per page load and (b) there were a total of 10 HTTP requests, the percentage of requests to the site root was 40% (i.e., 2x2/10).
Here we used the unique ID of a request to check all log messages related to that
single request. We thereby were able to determine that the controller class responsible for serving requests to site root is
As expected in this simple test case, only my host computer has accessed the site. This is of course a very simplistic example, but the capability that it demonstrates (of being able to analyse the sources of the traffic to your site) is obviously quite powerful and important.
Having
Answer: 80% (i.e., 8/10)
For purposes of this example, we’ll define “slow” as taking more than 5 seconds to provide a response. Accordingly:
Answer: 20% (i.e., 2/10)
No, Symfony standard uses only URL slugs, so this also tells us here that no one has attempted to hack the site. These are just a handful of relatively rudimentary examples of the ways in which logs files can be creatively leveraged to yield valuable usage information and even basic analytics. Other Things to Keep in MindKeeping Things SecureAnother heads-up is for security. You might think that logging requests is a good idea, in most cases it indeed is. However, it’s important to be extremely careful about removing any potentially sensitive user information before storing it in the log. Fighting Log File BloatSince log files are text files to which you always append information, they are constantly growing. Since this is a well-known issue, there are some fairly standard approaches to controlling log file growth. The easiest is to rotate the logs. Rotating logs means:
The most common
solution for this is
Another, more advanced approach is to make
This way,
Remote LogsAs the project grows, parsing information from logs gets more and more resource hungry. This not only means creating extra server load; it also means creating peak load on the CPU and disk drives at the times when you parse logs, which can degrade server response time for users (or in a worst case can even bring the site down). To solve this, consider setting up a centralized logging server. All you need for this is another box with UDP port 514 (default) open. To make
Having this, setting up the client is then as easy as:
(where ConclusionWhile this article has demonstrated some of the creative ways in which log files can offer way more valuable information than you may have previously imagined, it’s important to emphasize that we’ve only scratched the surface of what’s possible. The extent, scope, and format of what you can log is almost limitless. This means that – if there’s usage or analytics data you want to extract from your logs – you simply need to log it in a way that will make it easy to subsequently parse and
analyze. Moreover, that analysis can often be performed with standard Linux command line tools like Indeed, PHP log files are a most powerful tool that can be of tremendous benefit. ResourcesCode on GitHub: https://github.com/isanosyan/toptal-blog-logs-post-example Appendix: Reading and Manipulating Log Files in the Unix ShellHere is a brief intro to some of the more common *nix command line tools that you’ll want to be familiar with for reading and manipulating your log files.
Here we first match lines against
regular expression “starts with 2014”, then cut months. Finally, we use combination of Where does PHP CLI log?There are two possible values for error_log: a custom log file and the syslog. If the syslog is used, then all PHP errors will be sent directly to the default system log file—in Linux, this is typically /var/log/syslog.
Where is Apache PHP log?If you have build Apache and PHP from source, then the error logs by default is generated at your ${Apache install dir}/logs/error_log , i.e., generally /usr/local/apache2/logs/error_log . Else, if you have installed it from the repository, you will find it at /var/log/apache2/error_log .
How do I check website logs?Go to Websites & Domains > the site's domain name > click File Manager. Alternatively, go to Files. Select Logs in the left frame and click error_log in the list.
Where can I find error logs?Access Windows Error Logs via the Run command
Launch the Run dialog box by simultaneously pressing the Windows key and the R key from your keyboard; In the newly launched Run windows, type in eventvwr; Hit Enter to run the command; And the Event Viewer window should automatically pop up right after that.
|