I've spent a lot of time recently doing a deep dive into all the academic papers in a particular field of neurobiology, and I have a different approach:

1. Read the title and abstract 2. Read the methodology 3. If the methodology isn't totally asinine, then read the rest

I discovered after reading hundreds of papers that most of them are total nonsense.

I found myself reading them and updating my "knowledge," then getting to the methodology section and realizing it was bullshit. I felt like I couldn't fully "un-update" my model when that happened, the damage was sort of done.

They make claims and they have "evidence" that sounds compelling enough to slip by some filter, but then the methodology is totally bunk. The n is way too small, or limited in some other fundamental way, the experiment design is idiotic cargo cult stuff. Obviously just going through the motions of publishing because they have to, rather than having some valuable insight. Then papers like that cite each other, and build this whole wobbly network, full of sound and fury, and signifying nothing.

I've learned to totally ignore any paper that I haven't read the methodology for first.

Critiquing the methodology section is the best part of reading scientific papers. However, it soon becomes bittersweet as when you implement your own experiments, you soon become aware of how difficult it is to design a good experiment. That being said, if journalists read methodology sections (or anyone, really) then the world would be a much better (and less sensationalised) place.

Those steps don't quite map well to computer science academic papers - or, at least, to systems papers, which are what I tend to read and write.

Conclusions in CS papers are generally worthless. The contribution of the work is typically not new knowledge, but a thing - a system, a technique, an algorithm, a language, a framework, etc. So the conclusions in CS papers tend to be "We presented a foo for a blah blah blah." I joke that conclusions in CS papers are mostly vestigial, and exist only because they're expected to exist. (Please note that this bothers me, and I try to include actual conclusions in my papers, which will take the form of general principles we can learn from this new technique or system that we're presenting. But such discussion is not needed, and is often cut due to space limitations.) So, skip the conclusions sections in CS papers.

The introduction can also usually be skipped, if it's an area that you are already familiar with. CS papers tend to have to tell a "story", and the introduction sets the scene for why the work is relevant and who cares about this sort of thing. These introductions can start out very broad and general, trying to motivate the work from larger trends in industry and society. (Yes, even CS systems papers will appeal to larger trends in society to motivate their work. There are a lot of CS systems papers which will appeal to the prevalence of, say, online social networks to motivate their work on the systems required to support such things. Such as graph processing or real time data management.)

If you are familiar with the field, you can typically skip all this. You already know it. Please note that introductions are great if it's a field you're not as familiar with.

A CS systems paper typically has the structure similar to: 1. Introduction; 2. Background; 3. Design; 4. Results; 5. Related Work; 6. Conclusions.

If I want to quickly assess if a CS paper is worth my time, I typically read the title, the abstract and then skim the Design section and skim the Results section. (Of course, if it's my field, I often can't resist just skipping to the Related Work to see if they cited me.)

If the paper passes my first look, then I'll typically start by reading the Design. If I hit any difficulty in understanding the Design section, then I'll jump back to the beginning to get the proper context. I don't bother reading the Results section until I understand what they did, and think it's reasonable. (If your Design section tells me how you implemented the best 3-wheel car in the world, I'm not going to bother reading your Results. I don't care to try to reason about the performance of your 3-wheel car, because 4-wheel cars exist - unless you can convince me of situations where 4-wheel cars are not an option. That's a hard sell, though.)

I suppose quantum computing presents one of the few exceptions? And on the other end, distributed systems? Where else is new ground being explored? Cross functional work?

This is, by far, the most appealing approach.

I have to dig through a lot of research papers for my blog, most of the papers are dead-boring, but I feel obligated to read them with a fine-tooth comb to better understand what's being reported.

Filtering irrelevant or poor documentation by following your process seems like a no-brainer. Thanks for sharing!

When we were taught how to write papers in 6th grade or so, it seemed very repetitive. I have to summarize the research first, then go into detail, then another summary? But it's really more like making an easy-to-scan index of your paper.