Understanding and troubleshooting Blue Screen Errors

Classroom Deck - XPS

Dell Confidential

Class Outline
1. Understanding a Crash 2. Tools You Can Use 3. Sysinternals Suite of Tools

Dell Confidential - DRAFT

Expert Tools

What is a Blue Screen Error?

Blue Screen Error messages

are caused when a condition arises that requires the entire system to halt In order to troubleshoot, you need two critical pieces of information from this error

The first highlighted section is the type of error (0x8E) The second section is the driver/file that called the error (ati2dvag.dll)

NOTE: The driver/file is not always listed unfortunately

At this point the system must be restarted as all processing has ended Notice the section at the bottom Beginning dump of physical memory What does that mean? Important Information about the creation of a dump file:

The process SMSS.exe must be running The physical dump (RAM) must not be larger than the size of the paging file
Dell Confidential - DRAFT
3

Expert Tools

Why Does Windows Crash?

Microsofts analysis of crash root causes indicates:

~70% caused by third-party driver code ~15% caused by unknown (memory is too corrupted to tell) ~10% caused by hardware issues ~5% caused by Microsoft code There are lots of third-party drivers! From online crash analysis database: 55,000 unique drivers 24 new/day (28,000 in 2004) 220,000 total drivers 98 revised/day (130,000 in 2004) Many Devices Over 1,263,300 distinct Plug and Play (PnP) IDs (680,000 in 2004) 1,600 PnP IDs added every day
4

Expert Tools

Random Reboots IMPORTANT NOTE

Your customers may be experiencing a Blue
Screen Error but not know it as the system restarts too quickly for them to see it

Before troubleshooting the Random Reboot

symptom, be sure to follow these directions:

Open My computer properties Click Advanced tab Click Settings under Startup and Recovery Ensure the box Automatically Restart is NOT CHECKED

This will ensure that the customer has time to

alert you that a BSOD has just occurred instead of an instant reboot

You do not want to confuse random reboots

with Blue Screen errors as they are often troubleshoot differently

Dell Confidential - DRAFT

Expert Tools

Physical Memory Dumps

Most Blue Screen error messages are actually requested by
a driver or application to prevent the system from locking up or losing more data

Physical Memory Dumps are an attempt by Windows XP to

store the conditions that caused the system to give the error

This data can be very useful for you to determine what driver
or application most likely called for the crash

Once you know what direction to start looking, there are

tools available to help you resolve this

Notice that the previous Blue Screen already told us it was

called by the file ATI2Dvag.dllsometimes were not that lucky

Dell Confidential - DRAFT

Expert Tools

BSOD Troubleshooting: Get some details!

Just like when you are troubleshooting a No POST, its important to know all the background details from the customer
What is the issue?

When did the issue start? (CRITICAL)

When does the issue occur? (CRITICAL) What changes have been made to the system? (CRITICAL) What has already been done to try and resolve the problem? What type of Internet connection is available/active?

If no Internet Connection is available or the primary system crashes before being able to access the Internetis there a second one available Internet Connectivity is very important for using the Online Crash Analysis Tool, downloading the Debug Tool, and updating drivers that may cause the crash (such as the ATI BSOD seen on the previous slide)
7

Dell Confidential - DRAFT

Expert Tools

What is the issue?

The initial step in the troubleshooting process is finding what is or is
not occurring in the system. Once you make contact with the customer you must determine the problem that is being encountered

Preferred: Connect to the customers system using DellConnect and witness it first hand If Internet or other issues are keeping DellConnect from working, ask the customer for as many details as possible

With this information in hand, we are able to perform three basic steps
to get started:

Run MSCONFIG to disable any unnecessary startup items and services Open the Event Viewer to check for a history of these issues (covered on the next slide)

Check the Device Manager for any drivers with a yellow (!) or red (x)

Dell Confidential - DRAFT

Expert Tools

What is the Issue Opening the Event Viewer

The event viewer contains a historical record of all events, errors and
crashes on the customers system IMPORTANT: This tool is most important when the customer can only remember the date/time of the error and not the specifics of the BSOD error It is accessible by:

Open the Control Panel (switch to Classic Mode) Select Administrative Tools and click Event Viewer

You will begin your search by looking under both the Application and
System section

Dell Confidential - DRAFT

Expert Tools

What is the issue Using the Event Viewer

The Event Viewer will always contain much more information than is
necessary for our purposes

Procedure:

Ask the Customer what time and date the error last occurred on Click Applications first Look for any red and white X that says ERROR next to it, that occurred on or near the date/time the customer stated Double click that item and use DellConnect or the customer to get the information the gray box Repeat this process with the System section until you feel you have the information neceesary

Dell Confidential - DRAFT

10

Expert Tools

What is the issue Event Log sample

Scenario: The customer tells you that the system hung on the 11th
around 3:15pm when he tried to do something, cant remember but the system is working fine now. This customer wants to know why his system is hanging

Question 1: Why is this a good time to use the Event Viewer?

Upon opening the Event Viewer, you see this:

Question 2: Which items should we double click?

Dell Confidential - DRAFT

11

Expert Tools

What is the issue Event Log sample

As the Application Hang items seem most relevant to our issue of a
frozen computer, you double click it and see this:

QUESTION 1: What application should

we begin troubleshooting? QUESTION 2: If you are unsure what file caused this hang, what tools can you use to determine what program this file belongs to?

NOTE: Blue Screens and other errors

will still have the red x notification.

Dell Confidential - DRAFT

12

Expert Tools

When did the issue start?

There are two key reasons to ask the customer When did this issue
start

A customer may actually tell you what is causing the error message

Example:
Customer: It started right after I got my DSL installed, actually You: What model is your DSL modem, and does it use USB or Networking Customer: Oh I connected both Question for the class: Why was this discussion useful?

Over time the availability of system restore points decreases

System Restore Points may become corrupted or deleted Example:

Customer: This has been happening on and off for six months now, figured Id call in You: Do you remember any changes you made during that timeframe Customer: Not really You: What do you have on your system Customer: The tower, monitor, and my USB Camera You and the customer try a system restore, but there are no available restore points Question for the class: What tool can you use here to get more information about this error?
13

Dell Confidential - DRAFT

Expert Tools

When does the problem occur?

Many issues require a specific circumstance to occur.
In order to troubleshoot effectively you need to be able to consistently re-create the issue

Use your available Lab Create the issue when using DellConnect Search the Groups.google.com and other online resources for others having the same issue

Many problems will only occur on the Internet, with a particular browser
or only with select web pages. The important step is taking ownership and attempting to duplicating the issue.

EXAMPLE: I recently troubleshot an issue with Outlook Express. PROBLEM: A map attachment in an E-Mail was selected to print. The Photo Printing Wizard opened normally; you click next and are offered an option to select the picture you want to print. At this point the customer was offered several pornographic photos to print in the Picture Selection window. DISCUSSION: DellConnect was used to confirm the issue was as described by the customer. Multiple searches had been made of the hard drive without any success identifying the problem files. RESOLUTION: After confirming the steps I had previously recommended I configured a system in the lab using the same application & created a couple of emails with attached photos. When I printed a photo & discovered that the Wizard scanned the temporary internet files for photos. Deleting the temporary internet files & cookies from Internet Options removed the undesirable files from the Photo Printing Wizard & the customer was off to discuss web viewing habits with his son.

The moral of this story is that by re-creating the issue on another

system we were able to validate what was actually happening, not what appeared to be happening.
14

Dell Confidential - DRAFT

Expert Tools

What changes were made prior to this happening?

Everyone makes changes to their systems, often several times a day. What is different from
when the system was running normally and the current situation?

Changes most likely to cause OS Errors

Was hardware installed prior to the problem?

New printer USB keyboard, mouse, game pad, joystick, photo reader Cameras and their associated software Use the Driver Rollback feature in XP Uninstall the driver, software and hardware added Check Add/Remove programs for recent XP Updates Attempt to perform a system restore to a point before the last update Are there new items in add/remove programs? Are two versions of the same application installed in add/remove programs?

Was a new driver installed?

Was a Windows Update installed?

Was a program installed?

Changes are not limited to new software or hardware

The customer switched to a new Internet Service Provider

Dell Confidential - DRAFT

15

Expert Tools

What has already been done to try to resolve the issue

This step is important for two reasons:

Efficiency: Do not repeat steps already tried Results: Previous attempts to fix the system that were NOT successful can tell you what IS NOT the issue

Example: A customer receives a Blue Screen error when attempting to play one game, however they have already tried patching the game, updating video drivers and reinstalling the game. Next Step: Since it does not appear the be the game or video drivers, you can safely skip those steps and start troubleshooting using MSCONFIG, Event Viewer, updating the sound card drivers and running DXDiags.

Resources to check on previous steps:

Put the customer on hold if necessary and read all relevant case logs Ask the customer what specific steps they have performed and their results Use DellConnect to see if items such as MSCONFIG, or Anti-Virus have been installed by the customer

Dell Confidential - DRAFT

16

Expert Tools

What type of Internet Connection is present

As stated earlier, this information is mainly useful to knowing what tools are
available

DellConnect only operates properly using high speed DSL/Cable Internet

Dial-up connections limit your ability to download updated drivers, the Win Debug Tool and SysInternals tools (covered later)

High Speed connections have a higher likelihood of spyware or virus infestation

Question: Why is this?

High Speed connections make it easier to update Anti-Virus and Anti-Spyware

applications

Naturally it is easier to troubleshoot a system using DellConnect on a high

speed connection than relying on the customers interpretation of the error

Dell Confidential - DRAFT

17

Expert Tools

Fundamental Troubleshooting
There are three basic scenarios for OS Errors or crashes

During boot

Removal ALL peripherals except for keyboard, mouse and monitor Press F8 between the BIOS and XP Screens, select Last known good configuration Press F8 between the BIOS and XP Screens, select Safe mode with Networking
If it still crashes: Potentially a memory, hard drive or video card failure use Dell Diags If it boots: Open the Event Viewer in Safe Mode to determine what was causing the error, as well as running MSCONFIG and uninstalling all third party software/drivers

Upon using a device or opening an application

Recreate the issue and record the information in the Blue Screen Error Research DSN, Support.Microsoft.com and www.google.com Update the driver or application Run MSCONFIG to remove potential conflicting applications Visit the website of the manufacturer to check for
Known issues similar to yours Verify it is compatible with the customers OS and system configuration such as compatible sound cards

Randomly

Ask the customer for the approximate time/date it happened Open the Event Viewer, and check both Applications and System sections for Error that fits the description of the customer
Expert Tools

Dell Confidential - DRAFT

18

Advanced Troubleshooting Windows Debugger

Everyone in this class should be familiar with the Online Crash
Analysis tool as it is part of the BSOD Tree in DSN

Therefore we will skip this tool and use the more advanced Windows
Debugger application if you were not able to resolve the issue with the OCA

Windows Debugger is a tool provided by Microsoft for advanced

troubleshooting of issues (bugs) and software developers

Instructions are located on the next slide

Dell Confidential - DRAFT

19

Expert Tools

Advanced Troubleshooting Install/Setup of Debugger

The Windows Debugger is not included by default on Windows XP You must visit this location to download it ONTO THE CUSTOMERS
COMPUTER

http://www.microsoft.com/whdc/devtools/debugging/default.mspx

Ensure that you select the version appropriate for the system, as there
are 32-bit and 64-bit versions

Click either of the links in the middle of the page starting Install Debugging tools for windows On the next page, choose the latest version available NOTE: The file size is approximately 15MB, be prepared to wait

Run the installer application Click Start / All Programs / Debugging Tools for Windows / WinDBG Click File / Symbol File Path / type

srv*c:\symbols*http://msdl.microsoft.com/download/symbols without the quotes

Expert Tools

Dell Confidential - DRAFT

20

Advanced Troubleshooting Using Debugger

Every time Windows crashes with a Blue Screen, it will attempt to save
a dump file into C:\Windows\minidump These dump files are named by the date and the order of crash. For example Mini080306-1.dmp

08 = Month of August 03 = 3rd day 06 = 2006 -1 = First crash of this particular day File > Open Crash Dump Browse to C:\Windows\minidump Open the file closest to the date the customer claims the system crashed Wait for the application to run Note the file listed after Probably caused by:

In the debugger

Dell Confidential - DRAFT

21

Expert Tools

Advanced Troubleshooting Using !Analyze v

From the previous slide, the debugger said:

probably caused by: bcmw15.sys Spend 5 minutes checking various site results It appears to be related to wireless driver in the TCP/IP stack

A Google search for bcmw15.sys file is not very conclusive

Therefore perform the more advanced debugging by typing !Analyze
v into the command line at the bottom of the debugger
This provides a LOT of more information However were looking at one field: Process_Name

This field tells us what process/program asked for the Blue Screen

The answer is ccEvtMgr which we know to be Norton Anti-Virus

Conclusion:
We now know that the Norton ccEvtMgr application caused a Blue Screen error due to the bcmw15.sys

A google search for CCEvtMgr gives more results

Lets perform a lab to test this ourselves

Dell Confidential - DRAFT
22

Expert Tools

Questions?
Up until now we have only discussed the preparation and questions
necessary to begin the debug process

The next few slides will outline labs that we will guide you through in
order to actually USE the debugger with sample crashes

Dell Confidential - DRAFT

23

Expert Tools

Lab: Viewing a crash dump file

PURPOSE: In this Lab we will use the debugger to open and view a
minidump file

Key Points:


Memory dumps from previous crashes are stored in C:\Windows\minidump Event Viewer is useful but not always helpful A customer complains their system crashed on the 12th of November, 2006 but they cannot recall the error Open the Event Viewer, can you find the error? Does the Event Viewer tell us what caused the BSOD? Use the Debug Tool to open the appropriate dump file and determine the case of the BSOD

Lab Procedure

Dell Confidential - DRAFT

24

Expert Tools

Driver Verifier

Taking it a step further: Driver Verifier

Dell Confidential - DRAFT

25

Expert Tools

Driver Verifier What is it? WHAT: Driver Verifier is a tool created by Microsoft meant for hardware
creators to test their drivers, determining that piece of softwares ability to operate reliably

WHEN TO USE: Use this tool when Debug fails to help you for either of
these reasons:

The BSOD errors are different every time <OR> The Debugger tool lists a Windows file (such as NTOSKRNL.EXE) as the cause

HOW TO USE: Run the Verifier tool to force the system to crash, and
then run the WinDBG program on the latest dump file

IMPORTANT NOTE: It is very likely this tool will cause the system to crash, this is on purpose so the tool can determine which driver caused the crash
Dell Confidential - DRAFT
26

Expert Tools

Driver Verifier How to start How to run:

Click Start / Run Type Verifier Click Create Custom Settings Click Next

Dell Confidential - DRAFT

27

Expert Tools

Driver Verifier - Setup If you are unsure what driver

caused the crash:

Select Individual settings from a full list, click Next Check all options EXCEPT Low resources sim, click Next Choose Automatically Select Unsigned drivers, click next Click Finish and reboot See if the system Blue Screens

If you know what driver caused

the crash:


Select Individual settings from a full list Check all options EXCEPT Low resources sim, click Next Choose Select Driver Names from a list, click Next Scroll through the list and check the driver you suspect, click next Click Finish and reboot See if the system Blue Screens

Dell Confidential - DRAFT

28

Expert Tools

Driver Verifier How to use it

Run Driver Verifier until you can force the system to Blue Screen

Either using All Unsigned Drivers or selecting one manually from a list

Reboot after the Blue Screen Disable Driver Verifier

Open Verifier.exe from the command line Choose Delete Existing Settings and click Finish Reboot to finalize settings

Open the WinDBG Application Load the latest dump file Perform your analysis again

Dell Confidential - DRAFT

29

Expert Tools

SysInternals - AutoRuns
WHAT IS IT: AutoRuns is a tool developed by Sysinternals (now owned
by Microsoft), available at www.sysinternals.com, that allows you to enable/disable ANY startup items

Many items are not listed in MSCONFIG Use this when you absolutely cannot get rid of a startup application or service causing your Blue Screen errors

WHERE IS IT: http://download.sysinternals.com/Files/Autoruns.zip HOW TO USE IT:

Download and run the application Allow time for it to scan

Click Options / Verify Digital Signatures, and then Options / Hide Microsoft entries Click the refresh button renew the list Uncheck any items you feel may be causing the errors
Expert Tools

Dell Confidential - DRAFT

30

Autoruns

Dell Confidential - DRAFT

31

Expert Tools

Dell Crash Analysis Tool

What is the Dell Crash Analysis Tool (CAT) ? CAT scans systems for suspect drivers and suggests a list of
driver files that may need to be updated, repaired, or replaced

For what kind of errors can I use CAT ? The Crash Analysis Tool (CAT) helps determine why you are
receiving either a blue screen error or a system crash error

32

Expert Tools

33

Expert Tools