Smart Media, Compact Flash, Memory Stick Data Recovery

SmartMedia, CompactFlash & Memory Stick Data Recovery

The recovery of data from Smartmedia, CompactFlash and Memory Stick is very much like a normal data recovery job. In this serie, we'll examine the basic data structures involved, how they can become corrupted and, last but not least, what can be done to recover them. Our goal isn't to provide an encyclopedic programmer's reference on the topic, rather we strive to provide the curious user with some background. As developers of PhotoRescue, the award winning, risk free, data-recovery solution for digital media recovery, we have some experience to share.

What can go wrong ?

Hard disks blow electronic components, crash heads, loose their logical structure to viruses, but what usually happens to solid state media ?

In the camera...

Digital Cameras contain a full operating system that supports classical Input/Output operations on a FAT based file system. Due to memory constraints in early cameras (and possibly also to inexperienced developers), the file system implementations are sometimes a bit shaky. It is not uncommon to see cameras "wrap their writes around" when they don't realize they have reached the end of the card, erasing critical system structures located at the beginning of the media. Some cameras become confused when structures different from what they expect such as non-picture directories are present on a card; others are unable to switch safely from FAT-12 to FAT-16. Sometimes, a minor logical inconsistency leads to a major corruption of the card through what is known as the "avalanche effect" in datarecovery circles.

In the readers...

Card Readers usually interface with the OS's file system drivers through drivers that act as bridges between the file system handler and (for example) the USB mass storage driver. While some of these drivers do not respond properly to standard system calls, the majority of problems encountered by the users after they have manipulated a card in a USB reader are in fact camera related. The camera's simpler operating system just cannot handle the perfectly valid situations created by a sequence of file deletion, creation and moves under the computer's operating system. That's why it is always a good idea to simply try to read a card that a camera will not accept in an external reader. Some recoveries are as easy as that.

Peculiarities

NAND - Flash based media uses a complex low level sector virtualization called "wear leveling algorithm" to distribute the wear evenly across the memory array and maximize the number of write cycles it can sustain. After analyzing hundreds of failed cards we have determined that this algorithm can sometimes partially kick in. Another thing to keep in mind is that when a memory card begins to fail physically, total failure is not very far - backup at once.

In the real world...

Card fall into lakes, bake in the desert sun, wear down, are mishandled. This can lead to partial or complete hardware failure or virtually any kind of software corruption one can imagine. To better understand those failures, let's now have a look at a few card data structures.

The Card's Organization and Data Structures

At the card level : implementation varies a bit from vendor to vendor and type of media to type of media. The cards usually use FAT-12 (up to 64MB) and FAT-16 (above 64MB). In the future, as card capacity grows, they'll probably start using FAT-32.

The Card Information System (CIS)

The CIS is a hidden area located outside of the emulated logical or physical drive. It contains information about the card manufacturer, many of the device characteristics and, is sometimes used to "hide" manufacturer specific features (such as the Olympus Panoramic mode). We'll ignore this area for the moment.

SmartMedia, CompactFlash & Memory Stick Data Recovery (c) DataRescue Inc, 2001 - 2003 The recovery of data from Smartmedia, CompactFlash and Memory Stick is very much like a normal data recovery job. In this serie, we'll examine the basic data structures involved, how they can become corrupted and, last but not least, what can be done to recover them. Our goal isn't to provide an encyclopedic programmer's reference on the topic, rather we strive to provide the curious user with some background. As developers of PhotoRescue, the award winning, risk free, data-recovery solution for digital media recovery, we have some experience to share. What can go wrong ? Hard disks blow electronic components, crash heads, loose their logical structure to viruses, but what usually happens to solid state media ? In the camera... Digital Cameras contain a full operating system that supports classical Input/Output operations on a FAT based file system. Due to memory constraints in early cameras (and possibly also to inexperienced developers), the file system implementations are sometimes a bit shaky. It is not uncommon to see cameras "wrap their writes around" when they don't realize they have reached the end of the card, erasing critical system structures located at the beginning of the media. Some cameras become confused when structures different from what they expect such as non-picture directories are present on a card; others are unable to switch safely from FAT-12 to FAT-16. Sometimes, a minor logical inconsistency leads to a major corruption of the card through what is known as the "avalanche effect" in datarecovery circles. In the readers... Card Readers usually interface with the OS's file system drivers through drivers that act as bridges between the file system handler and (for example) the USB mass storage driver. While some of these drivers do not respond properly to standard system calls, the majority of problems encountered by the users after they have manipulated a card in a USB reader are in fact camera related. The camera's simpler operating system just cannot handle the perfectly valid situations created by a sequence of file deletion, creation and moves under the computer's operating system. That's why it is always a good idea to simply try to read a card that a camera will not accept in an external reader. Some recoveries are as easy as that. Peculiarities NAND - Flash based media uses a complex low level sector virtualization called "wear leveling algorithm" to distribute the wear evenly across the memory array and maximize the number of write cycles it can sustain. After analyzing hundreds of failed cards we have determined that this algorithm can sometimes partially kick in. Another thing to keep in mind is that when a memory card begins to fail physically, total failure is not very far - backup at once. In the real world... Card fall into lakes, bake in the desert sun, wear down, are mishandled. This can lead to partial or complete hardware failure or virtually any kind of software corruption one can imagine. To better understand those failures, let's now have a look at a few card data structures. The Card's Organization and Data Structures At the card level : implementation varies a bit from vendor to vendor and type of media to type of media. The cards usually use FAT-12 (up to 64MB) and FAT-16 (above 64MB). In the future, as card capacity grows, they'll probably start using FAT-32. The Card Information System (CIS) The CIS is a hidden area located outside of the emulated logical or physical drive. It contains information about the card manufacturer, many of the device characteristics and, is sometimes used to "hide" manufacturer specific features (such as the Olympus Panoramic mode). We'll ignore this area for the moment.
The master boot record In hard disks this sector contains the system start up code that is responsible for selecting the active partition and loading the boot sector. It also contains a data structure (the partition table) that describes the hard disk partitioning. In cards, this sector is present mostly for "compatibility" reasons, as cards are rarely used (today!) to boot operating systems. Its corruption can cause the media not to be accessible as a logical device ( ie visible as a drive letter ). Below is a typical smartmedia MBR.		<click to enlarge>
<click to enlarge>	The boot sector is used with the master boot record to set up the "geometry" for the card. Its corruption can lead to generic messages about the media not being formatted or being unreadable. Note the FAT-12 indicator for this 64MB smart media card. Note that each cluster (or allocation unit) contains 32 sectors (or 16 kilobytes)
The FAT contains linked list describing the clusters occupied by the files. Its corruption can lead to any kind of trouble: sometimes one or more files are unreadable, sometimes "seek errors" are generated, etc.... In digital cameras, one single corruption in the FAT often causes the camera to ignore valid data that follows or to generally misbehave, for example by simply shutting down or locking up. There are usually two FAT copies on a drive (see the above screen capture). The FAT-12 pictured below contains entries for seven files (color coded). the 056000 means cluster 05 and cluster 06 the 078000 means cluster 07 and cluster 08 the 09A000 means cluser 09 and A0 ( 10 in hexadecimal ) ... FFF also means chain termination, or last cluster of the run. File 5 for example begins cluster 92 (5D in hexadecimal) FAT-12 can address at most 4096 clusters, ( in practice slightly less than that because numbering doesn't begin at 0). 4095 times 16 kb gives 64 MB.		<click to enlarge>
<click to enlarge>	The root directory If the root directory is corrupted or if the boot sector points to a wrong root directory, the files and subdirectories can become lost. Pictures are usually not stored in the root directory because a root directory is a structure with a well defined size : therefore it can only contain a limited number of files ( 256 entries in this case). Below is a root directory that only contains a pointer to the sub-directory the camera actually uses to store pictures. The classical symptom of a root directory pointing to a bad location is a number of files with strange names and outrageous sizes.
Sub-directories contain file descriptors. Corruption of these entries can lead to wrong file content being read from the media. This is especially a problem after a failed data recovery attempt ( for example with chkdsk ) that rewrites correct file entries pointing to wrong locations. This directory contains 7 files. In this example, we'll interest ourselves in file number 5.		<click to enlarge>
	The data area contains the actual file data. In most cases, the file data remains on the card when files are erased. This is what makes data recovery possible. Here is the content of our fifth file. It can be clearly recognized by its exif header.

The screenshots on this page were made with Winhex
Coming next : data preservation tips.

SmartMedia, CompactFlash & Memory Stick Data Recovery

The master boot record

The boot sector

The FAT

The root directory

Sub-directories

The data area

[Go Back to DataRescue Home Page ] DataRescue 45 quai de la Dérivation 4020 Liège (Belgium) tel 32-4-3446510 fax 32-4-3446514 Please send us your questions or comments.