I keep on coming back to it time & time again. I give it to testers but usually I hand a copy to the developers. They complain at first and after they read it they go: “Hey, this is really a good idea….”.

Maybe you should re-work this into something more prominent & permanent on your site. I think it adds a lot of value.

I also find logging especially interesting from an automation perspective. I can have watchers on a log, that alert me if certain conditions that I want to know about occur (i.e. see if a defect has actually been resolved if there is doubt about the fix).

[James' Reply: Burp Proxy.]

Sebi

David

The benefit is much bigger for progress/status reports (that are seen in the screen as actions happen), but it is the same for logs:
There are times where an application will crash, but all the data the users/testers have is what was done until now. The developer can (easily or not) deduce what the app was doing when it crashed because he knows the flow — but the users are left with guessing.

Without this little aid, if function E crashes when called after D, but E is *not always* called, a log for crash would look like “A, F, B, D, ” while a log without crash would look “A, F, B, D, G, A”. Hard to see the reason, right? Had the testers know that function E was starting, the ‘sporadic’ issue would have been isolated much more easily.

Well, I guess the point is made, will stop repeating .
Thanks!!

As you say, not only it is hard for one routine to know what will be the next routine called, but doing that would wreck modularity cohesion and low-coupling .

But for most part (perceiving delays, or analyzing hangs for example) logging at the beginning of each routine would be enough.

[James reply: Okay, so logging "I'm about to try this... if you don't hear from me, tell my wife I love her..."]

If I may, let me add a 14th point, an insight I learned from a tester in my team:
For functions the software does by itself, in addition to logging the result of each step after it happens (imported data; generated report; exported graph…) it is very useful to log *what the next step will be*. This way, when a failure occurs, it is easy to know what the software was trying to do.

Since I learned that, I am paying attention to it, and impressed by how much common logs or progress reports lack this.
It takes a lot of time for an operation to happen, but you need to wait until the end of it to know which it was.
Once my computer froze during Windows’ shutdown with the “playing logoff sound” screen. I’m pretty sure the sound playing was over, and was left wondering what caused the OS freeze.

Hope you approve the addition

[James' Reply: Let's say I'm a programmer. How would I actually implement this? Are you just saying you want me to log the start of events, instead of just logging when things end?

Obviously, I can't know, from within a specific routine, what the next thing will be.]

One thing that was nice was that the logs were not written to files, rather they were submitted via XML objects to a logging service which greatly enhanced the efficiency and was less of a drain on the system (as writing to disk is slower and more of a resource hog). Also, there was a configuration file which allowed us to set different logging levels for each environment (info, error, debug, etc…). The data/variable pairs would align with a global variable for whatever environment in which the code was deployed.

[James' Reply: Cool. I have also used SQL Server transaction logs as log files, and you get those for free in database apps.]

I hesitate to say this, but we were a scripted organization I hope we’re allowed to post here.

[James' Reply: You can post it. But what do you mean by it? Detailed procedural scripts? If so, why? You had logging, which is like having a GPS track of where you were patrolling, but you still felt that letting your drivers patrol as they saw fit was bad?]

While we may differ on certain aspects of testing, I agree wholeheartedly that logging (or the absence of) is one of the most overlooked gems for a tester. Monitoring and rigid examination of logs is something I try to instill into each member of my testing team. When engaged in a project in the planning and analysis phases, it’s also something I urge the architects to consider.

[James' Reply: We don't yet know if we differ. I do plenty of planning in my testing, when the situation calls for it. To be exploratory means to not STOP planning once you have a "plan."]

[James' Reply: Excellent point.]

Tell me if something is unclear.=)

/Sigge

[James' Reply: I read your comment. What specific wording change would you suggest?]

I especially like your initial discussion on how documentation gets to people, so they end up asking for something they often really doesn’t have any use for. I’ve seen it multiple times, and it’s a sad waste of everybodys time.

This also reminds me of back some twelve-thirteen years ago, when I was programming heavy calculations in SQL. After each step I created a complete table as a copy of all the data I was working on, so I could debug step-by-step. It made life so much easier.

Thinking about it makes it seem pretty ridicolous that we have gotten ourselves used to saying: what happened ? and not questioning why it couldn’t be answered… Just because it’s our job to play detectives in order to reconstruct what lead to a particular failure, it doesn’t mean that we should do it without the proper tools.

Logging is not only a friend – it’s a great friend!

are you trying to suggest that ET is better because only ET uses logging help ?

i would say logging is a damn good support tool ( i would even say it is sometimes necessary) for any sort of testing that you are doing ? whether it is scripted,exploratory,automated ? and even during debugging

[James' Reply: I quite agree. Logging is the friend of all.]

[James' Reply: It depends what you mean by "recorder." When I do web testing I routinely use Burp Proxy to record everything that goes from the browser to the server. (I have also tried using the Selenium recorder but found it disappointing. It doesn't record very much.) I use line sniffers when and where feasible. I have used screen recorders, key loggers, and even a little digital camera. There are pros and cons to each approach.

What rarely works, except in a stateless command-line environment, is record-playback automation. There are so many things that can mess that up, you end up fiddling with it all the time.]

I just came off of a project that had logging that actually hurt the product. The project was on Windows and the ETW logging framework was what we used. It was supposed to just have “2% overhead” (yeah, I hear you “2% of what?”, I asked that too…). In practice the overhead was significantly more. Also, some logging statements had bugs in them that lead to program instability. This isn’t a counter-case against what you are arguing for, just a cautionary tale.

[James' Reply: Excellent point. The first time I tested very heavily with logging we spent a week chasing a bug whereby a process on the server started (as noted in the log) but never finished (the event end was not logged). Turns out it was a bug in the logging code triggered when events happened so quickly that the thread tried to open the log file while it was still in the process of being closed.]

Are you using a logging framework, or just using built-in facilities in the language/platform you are developing for?

[James' Reply: I don't know how the developer will do it. I don't know much about logging libraries, but I suppose they must exist and have some cool features. All we need is a formatted print to a file. The challenge comes in when many processes are trying to write to the same log, or when speed is a big concern. Maybe an established logging library would help with that.]

Do you think you are “lucky” to be working in a regulated environment where this type of proposal might have a better chance than with less safety critical software?

[James' Reply: Yes.]

Was there any feedback from the devs on the proposal?

[James' Reply: Not yet.]

Do you have any more to say about the level of detail in the logs?

[James' Reply: I love details. I want data about the events, not just the events themselves. In testing a search system I had the developers dump the SQL of for each search, and from this I built an automatic search test matrix. Of course, there has to be a way to deal with the huge size of the logs. That's why I like being able to set debug levels.]

One conflict that came up between dev and test on my project was that some of the devs wanted a log that would be detailed enough to use to debug with. Testers just wanted “important” events. The “debug” level logs could be 100′s of MB for just a few minutes of product use. It was a major discussion when I made a proposal to adjust what gets logged at what level later in the project.

[James' Reply: Yeah, that's an issue. Of course, you can just filter the log. So it's not that big a deal unless the log is super duper huge. I guess 100's of MB would qualify as that. At that point the issue is intrusiveness-- the log may actually reduce testability by causing a major performance problem in the product. Obviously, you'd have to get smarter about logging or else build a special infrastructure to handle it. Seems to me that super detailed logging might be built on some sort of event trigger, so that it's not going all the time, but rather only when you hit a panic button or when certain warning events occur.]

This was extremely powerful when investigating bugs and tracking the actual location in the code where this error/warning was generated.

We actually came to an agreement with our development team that when a specific error number was used more than once in the source code, the full SW package to be tested was returned to development immediately.