Salvaging photos off SD card

I would have to agree with the user experience of even trying to
locate files in Win7 being remarkably difficult.

However working as we do, often in both linux and windows camps, a
windows program that is remarkably good at finding and restoring files
is made by piriform and is free. Its called recuva.

Post by cr
Can anyone give any pointers to a Linux method of salvaging pics off a
corrupted SD card?
(No panic - my problem has been solved for now but I had to use Windows 7 to
do it).
The SD card was in my Panasonic TZ3. I copied the files across to my
computer (running Debian Lenny) using a card reader and just copy and paste
as I usually do, the files numbered p1230800 to 1230999 copied fine, but
1240001 to 1240200 weren't visible. Nor did using the TZ3 with its USB
connector help.
The TZ3, on the other hand, was displaying all the pics perfectly well.
1230800 to 1230999 were in a directory Pana_123, the invisible files were in
Pana_124 which was corrupted (ls -l just showed blanks for its listing).
I tried Photorec which found all the 123's but not the 124's. dd didn't
seem to be able to read the whole card. I tried using Windows XP but that
wouldn't even show Pana_124.
I reasoned that since the TZ3 could obviously read the card, maybe its
PictBridge printer output function would work, I tried that with Linux and
WinXP but they didn't seem to recognise it.
So in desperation I got out a laptop that has Windows 7** on it, set the TZ3
to PictBridge output, used the Windows 'get pictures' function and it found
them all (very slowly). Presumably it was emulating a Pictbridge enabled
printer.
It's only by chance I had the Win7 laptop, I'd really like to know if there's
a Linux solution. I *think* some bit of software which imitates Pictbridge
would do it.
(** As an aside, how did MS manage to screw up Win7 so spectacularly? I'm so
not a MS fan but I will concede that XP is a good reasonably intuitive
operating system - roughly as useable as Gnome, other than lacking multiple
workspaces. But Win7 is the worst OS I've ever encountered, if they'd
intended to make it obscure, cryptic and difficult to use they couldn't have
done a better job. Every time I've tried to use it I end up cursing it.
And its translucent windows look really naff.)
cr
_______________________________________________
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

2011-06-22 06:29:48 UTC

Thanks. I see it's said to work under XP. I'll download it for my Windows
partition.

cr

Post by Chris Gavin-Egan
I would have to agree with the user experience of even trying to
locate files in Win7 being remarkably difficult.
However working as we do, often in both linux and windows camps, a
windows program that is remarkably good at finding and restoring files
is made by piriform and is free. Its called recuva.

Post by cr
Can anyone give any pointers to a Linux method of salvaging pics off a
corrupted SD card?
(No panic - my problem has been solved for now but I had to use Windows 7
to do it).
The SD card was in my Panasonic TZ3. I copied the files across to my
computer (running Debian Lenny) using a card reader and just copy and
paste as I usually do, the files numbered p1230800 to 1230999 copied
fine, but 1240001 to 1240200 weren't visible. Nor did using the TZ3
with its USB connector help.
The TZ3, on the other hand, was displaying all the pics perfectly well.
1230800 to 1230999 were in a directory Pana_123, the invisible files were
in Pana_124 which was corrupted (ls -l just showed blanks for its
listing).
I tried Photorec which found all the 123's but not the 124's. dd
didn't seem to be able to read the whole card. I tried using Windows
XP but that wouldn't even show Pana_124.
I reasoned that since the TZ3 could obviously read the card, maybe its
PictBridge printer output function would work, I tried that with Linux
and WinXP but they didn't seem to recognise it.
So in desperation I got out a laptop that has Windows 7** on it, set the
TZ3 to PictBridge output, used the Windows 'get pictures' function and it
found them all (very slowly). Presumably it was emulating a Pictbridge
enabled printer.
It's only by chance I had the Win7 laptop, I'd really like to know if
there's a Linux solution. I *think* some bit of software which imitates
Pictbridge would do it.
(** As an aside, how did MS manage to screw up Win7 so spectacularly?
I'm so not a MS fan but I will concede that XP is a good reasonably
intuitive operating system - roughly as useable as Gnome, other than
lacking multiple workspaces. But Win7 is the worst OS I've ever
encountered, if they'd intended to make it obscure, cryptic and difficult
to use they couldn't have done a better job. Every time I've tried to
use it I end up cursing it. And its translucent windows look really
naff.)
cr

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

Atom Smasher

2011-06-21 11:27:47 UTC

Post by cr
Can anyone give any pointers to a Linux method of salvaging pics off a
corrupted SD card?

============

the first thing i'd try is "testdisk". if that doesn't work i'd try
"foremost".

there's also autopsy/TSK.

in any case i'd mount the faulty card read-only (or better still, not
mount it at all) and use dd to mirror it. then work from the mirrored
file.

http://www.cgsecurity.org/wiki/TestDisk
http://foremost.sourceforge.net/
http://www.sleuthkit.org/
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"These numbers have nothing to do with the technology of
the devices; they are the maximum that thermodynamics will
allow. And they strongly imply that brute-force attacks
against 256-bit keys will be infeasible until computers
are built from something other than matter and occupy
something other than space."
-- Bruce Schneier, Applied Cryptography

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

2011-06-22 06:51:29 UTC

Post by Atom Smasher

Post by cr
Can anyone give any pointers to a Linux method of salvaging pics off a
corrupted SD card?

Thanks. TestDisk seems to include PhotoRec. I haven't seen mention
of 'foremost' or 'sleuthkit' before though.

I did take the precaution of setting the 'write protect' tab on the SD card
before messing with it.

cr

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

Robin Sheat

2011-06-21 12:59:39 UTC

Post by cr
Can anyone give any pointers to a Linux method of salvaging pics off a
corrupted SD card?

I was sent one of these a while ago.

The first thing I did was dd it to an image using conv=noerrors (which got as
much data as it could.)

I then ran recoverjpeg over it. In my case, it got all but about two photos
(which was expected as the owner of the camera had taken a couple of photos
after the card died but before he realised what was happening.)

This was after he'd asked a photography place and they'd quoted him a
substantial amount of money to recover them, then they tried and couldn't get
anything.

Of course, every failure condition is different, so you may have different
results.

Robin.

2011-06-22 06:13:32 UTC

Post by Robin Sheat

Post by cr
Can anyone give any pointers to a Linux method of salvaging pics off a
corrupted SD card?

I was sent one of these a while ago.
The first thing I did was dd it to an image using conv=noerrors (which got
as much data as it could.)
I then ran recoverjpeg over it. In my case, it got all but about two photos
(which was expected as the owner of the camera had taken a couple of photos
after the card died but before he realised what was happening.)
This was after he'd asked a photography place and they'd quoted him a
substantial amount of money to recover them, then they tried and couldn't
get anything.
Of course, every failure condition is different, so you may have different
results.
Robin.

Thanks. I didn't try dd with the noerrs switch. I might try that just to
see if it works.

cr

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

Simon Bridge

2011-06-21 12:42:39 UTC

PhotoRec?
http://blog.mypapit.net/2007/10/how-to-recover-photo-files-from-sd-card-mmc-with-photorec.html

Post by cr
Can anyone give any pointers to a Linux method of salvaging pics off a
corrupted SD card?

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

2011-06-22 06:22:48 UTC

Thanks, but Photorec was the first thing I tried, several times, with
different size settings for the card. (It's included in the Debian distro).
But it failed to find the files in the corrupted '124' directory. (The
interesting question is how the TZ3's firmware managed to read them all).

Cheers

cr

Post by Simon Bridge
PhotoRec?
http://blog.mypapit.net/2007/10/how-to-recover-photo-files-from-sd-card-mmc
-with-photorec.html

Post by cr
Can anyone give any pointers to a Linux method of salvaging pics off a
corrupted SD card?

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

Simon Bridge

2011-06-22 10:38:45 UTC

Post by cr
Thanks, but Photorec was the first thing I tried, several times, with
different size settings for the card. (It's included in the Debian distro).
But it failed to find the files in the corrupted '124' directory. (The
interesting question is how the TZ3's firmware managed to read them all).us

That and recent windows could do this, but not old windows or dd - it's
suspicious.

Presumably you have formatted a new card on that camera, taken some
photos, and had a go accessing all the directories in gnu/linux?

I had a go looking to see if there was any pict printer emulation for
linux (eg getting the computer to pretend to be a printer, but "print"
to a file) and got swamped by drivers for pict output. Dunno how to
filter properly ...

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

2011-06-23 06:58:50 UTC

Post by cr
Thanks, but Photorec was the first thing I tried, several times, with
different size settings for the card. (It's included in the Debian
distro). But it failed to find the files in the corrupted '124'
directory. (The interesting question is how the TZ3's firmware managed
to read them all).us

That and recent windows could do this, but not old windows or dd - it's
suspicious.
Presumably you have formatted a new card on that camera, taken some
photos, and had a go accessing all the directories in gnu/linux?
I had a go looking to see if there was any pict printer emulation for
linux (eg getting the computer to pretend to be a printer, but "print"
to a file) and got swamped by drivers for pict output. Dunno how to
filter properly ...

To clarify: What I *think* happened (and this is partly guesswork): I
suspect the TZ3's firmware was the only thing that could read the corrupted
directory on the card. When I selected 'Printer output' on the camera
dial, I suspect it engaged Pictbridge mode and extracted each picture from
the card and fed it to the computer. Win 7 was the only OS that could
emulate a Pictbridge device. I don't think Win7 was reading the card
directly.

I know from experience that normally all directories on the SD card are
visible to Linux, I've taken thousands of photos with my TZ3 and read them
all using a card reader. (I've also done it the other way, copied files
from the computer to an SD card in a card reader). It's just this
particular card got a corrupted directory, I'll probably retire it rather
than risk it happening again.

cr

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

Volker Kuhlmann

2011-06-23 07:52:25 UTC

Post by cr
To clarify: What I *think* happened (and this is partly guesswork): I
suspect the TZ3's firmware was the only thing that could read the corrupted
directory on the card. When I selected 'Printer output' on the camera
dial, I suspect it engaged Pictbridge mode and extracted each picture from
the card and fed it to the computer. Win 7 was the only OS that could
emulate a Pictbridge device. I don't think Win7 was reading the card
directly.

Sounds like a reasonable explanation to me.

Post by cr
from the computer to an SD card in a card reader). It's just this
particular card got a corrupted directory, I'll probably retire it rather
than risk it happening again.

Wise move. The fact that dd couldn't read the card indicates the card is
having massive hardware problems. If it was a hard disk you'd junk it
just the same.

Volker

--
Volker Kuhlmann is list0570 with the domain in header.
http://volker.dnsalias.net/ Please do not CC list postings to me.

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

Robin Sheat

2011-06-23 09:55:19 UTC

Post by Volker Kuhlmann
Sounds like a reasonable explanation to me.

Me too. It's possible that the fat32 implementation in the camera is
pretty basic/weird, and so while certain corruption upsets real drivers
for it, its own one just blithely goes ahead ignoring the dodgy data.

Robin.

Simon Bridge

2011-06-23 14:32:37 UTC

Post by Robin Sheat

Post by Volker Kuhlmann
Sounds like a reasonable explanation to me.

... since this is a camera and so it expects only a couple of actual
file formats perhaps it ignores the FS completely and just looks for the
bites that tell it this is a jpeg .... eof. In which case the FS is a
sop to computer users?

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

Simon Bridge

2011-06-23 14:21:11 UTC

Sounds like a reasonable explanation to me.

Me too.

Post by cr
from the computer to an SD card in a card reader). It's just this
particular card got a corrupted directory, I'll probably retire it rather
than risk it happening again.

Wise move. The fact that dd couldn't read the card indicates the card is
having massive hardware problems. If it was a hard disk you'd junk it
just the same.

You'd think a low-level reading of the card would have done the
trick ... ah, but dd works on the block-special-device which is produced
by the file system module, so this would mean that the file system was
fubared so the driver did not read bits of it ... but the camera
firmware operates primitively enough to ignore the fault.

So to read this card under gnu/linux you want to go for some low-level
forensics that ignores the file system details.

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

Volker Kuhlmann

2011-06-23 21:17:49 UTC

Post by Simon Bridge
You'd think a low-level reading of the card would have done the
trick ... ah, but dd works on the block-special-device which is produced
by the file system module,

Sorry? The filesystem sits on top of the block layer. dd operates at the
block layer and bypasses the filesystem. The filesystem is optional and
exists for your convenience. The block layer is required to communicate
with the storage media. That's why you use dd to copy block devices with
filesystem corruption.

Volker

Robin Sheat

2011-06-23 21:38:02 UTC

Post by Volker Kuhlmann
Sorry? The filesystem sits on top of the block layer. dd operates at the
block layer and bypasses the filesystem. The filesystem is optional and
exists for your convenience. The block layer is required to communicate
with the storage media. That's why you use dd to copy block devices with
filesystem corruption.

It's a bit different with SD cards, but your point is still effectively
correct.

They have firmware that manipulates the blocks on the card to present it as a
single block device, even though behind the scenes the blocks may not be
allocated linearly (e.g. blocks may be moved to allow wear levelling.)

However, as far as I know, it's not possible to bypass this and look directly
at the flash blocks (on an SD card, other flash media may be different) so the
camera will be seeing the same thing that dd sees. Which is why you're still
right :)

I've heard that they do things like understand that the FAT area will be
written to a lot more than the regular file area, and so wear level that at a
different rate, but that's just hearsay, I'm not sure if it's true.

Robin.

Simon Bridge

2011-06-25 05:57:07 UTC

Post by Simon Bridge
You'd think a low-level reading of the card would have done the
trick ... ah, but dd works on the block-special-device which is produced
by the file system module,

So why didn't dd work?

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

2011-06-25 08:37:28 UTC

Post by Simon Bridge
You'd think a low-level reading of the card would have done the
trick ... ah, but dd works on the block-special-device which is
produced by the file system module,

So why didn't dd work?

Well, photorec didn't work either, and photorec claims to do a low-level read
and ignore filing systems.

I tried photorec on the card in a USB card reader, and on the card in the
camera in its output-to-PC-via-USB mode. I tried dd with the card reader,
and (I think, though I can't recall for sure) on the card in the camera also.

So I can't say for sure what effect the hardware or firmware of the card
reader or the camera may have had.

cr

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

Steve Holdoway

2011-06-25 09:51:44 UTC

Post by cr

Post by Simon Bridge
You'd think a low-level reading of the card would have done the
trick ... ah, but dd works on the block-special-device which is
produced by the file system module,

So why didn't dd work?

Well, photorec didn't work either, and photorec claims to do a low-level read
and ignore filing systems.
I tried photorec on the card in a USB card reader, and on the card in the
camera in its output-to-PC-via-USB mode. I tried dd with the card reader,
and (I think, though I can't recall for sure) on the card in the camera also.
So I can't say for sure what effect the hardware or firmware of the card
reader or the camera may have had.
cr

ddrescue is probably what you're after...
--
Steve Holdoway <***@greengecko.co.nz>
http://www.greengecko.co.nz
MSN: ***@greengecko.co.nz
Skype: sholdowa

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

2011-06-28 07:27:42 UTC

Post by cr

Post by Simon Bridge
You'd think a low-level reading of the card would have done the
trick ... ah, but dd works on the block-special-device which is
produced by the file system module,

Sorry? The filesystem sits on top of the block layer. dd operates at
the block layer and bypasses the filesystem. The filesystem is optional
and exists for your convenience. The block layer is required to
communicate with the storage media. That's why you use dd to copy block
devices with filesystem corruption.

So why didn't dd work?

Well, photorec didn't work either, and photorec claims to do a low-level
read and ignore filing systems.
I tried photorec on the card in a USB card reader, and on the card in the
camera in its output-to-PC-via-USB mode. I tried dd with the card
reader, and (I think, though I can't recall for sure) on the card in the
camera also.
So I can't say for sure what effect the hardware or firmware of the card
reader or the camera may have had.
cr

(When i said 'didn't work', I mean it got half the images but failed to find
those in the corrupted directory.)

Well, I had another go at it, with the card in the camera, used photorec
direct on the card (it's a Kingston, may be genuine, showed up as Matsushita
in Photorec) - and it got the lot. I'm not sure what, if anything, I was
doing wrong before.

Thanks everybody for the informative discussion.

cr

_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

Robin Sheat

2011-06-25 12:28:20 UTC

Post by Simon Bridge
So why didn't dd work?

Probably due to not telling it to ignore read errors. So hitting a bad block
caused an abort.

Robin.

Volker Kuhlmann

2011-06-25 23:38:37 UTC

So why didn't dd work?

Because the storage media is physically damaged. Akin to a hard disk
with bad blocks.

Some points have already been raised, but I'll elaborate.

The flash card consists of a huge flash memory storage area, a
microcontroller, and optionally a USB interface (aka another
microcontroller). The microcontroller sits between the card pins and the
flash memory. Its job is to organise the memory to make it look like a
hard disk block device, and to perform defect management and wear
levelling. Defect management works the same as on hard disks, when
blocks are detected as bad they're marked as "don't use". This includes
blocks with manufacturing defects; it's cheaper to mark a few as bad and
reuse the chip as a whole then to throw it away altogether. You notice
the total capacity is less than a power of two, the difference is
reserved for defect management. Wear levelling means that the controller
uses some strategy to allocate memory blocks to disk blocks, and that
this allocation changes over time. You can write 100x to the same
numbered disk block, but the controller may choose to store each time on
different memory blocks. When you read a disk block, the controller
reads the memory last used to store that block. (This btw is the reason
you can NOT erase a flash card securely. You have to physically destroy
it.) This holds true for all flash cards as far as I know. It is
probable that the levelling strategy assumes a fat filesystem is being
used, but that would depend on the card manufacturer.

Additionally, SD cards have another memory area to which access is
resctricted and it's encrypted by the card's microcontroller. That's why
it's called *secure* digital. Also remember that when the content mafia
sells you something as "secure", you DON'T WANT IT. The bastards don't
even publish approximate specs for the cards, not even the
non-encryption parts, you can't even get an official pinout. All you get
is third-hand info. If you want better, you have to join the club ($$$$)
and sign NDAs. It's annoying this stuff has become so popular. A big
part of the reason for this is that the card interface of SD is
significantly easier and chaper than that of CF (but you can plug a CF
card in as a hard disk as long as you are able to make the connector
fit).

dd accesses the block interface of the card. By definition it always
works; if you get I/O errors you have dead hardware. If it's not the
reader it's the card or a bad contact. To recover from a dud card you
employ the same techniques as for hard disks, with some changes that
make it easier. Never EVER write. Read all blocks and ignore errors. The
order is irrelevant (unlike with hard disks), ddrescue is no advantage
here. Make sure the block size is not larger than 512 bytes. There is no
advantage is keeping on trying with solid state memory. If your life
depends on it, try at different temperatures of the chip (not its case).
Try the lower ones first, re-try in steps down to -10degC. No
condensation. (Have fun...) You could write into a sparse file with dd,
which makes sure any unreadable card blocks are read from the image copy
as zeros. Don't forget to nail the card to the wall afterwards.

If you are succesfully getting your photos when the card is read by the
camera or some pictprinter then these devices might employ a reading
strategy which does not assume reading through a fat filesystem,
although the camera writes to the card in a format that passes as fatfs
at least in the absence of read errors.

You can't say fatback or photorec failed to do the best you can get out
of them unless you run them on the best copy of the card image you can
get. I would't trust their ability to ignore read errors when needed.
Besides, best practice is to copy as much info as you possibly can
before messing around with a copy of the copy.

Without having tried, I'd use these commands:

truncate -s 2G /tmp/cardimage
dd bs=512 conv=notrunc,noerror iflag=direct seek=0 \
if=/dev/sdX of=/tmp/cardimage

truncate is in modern coreutils >6.12 <=8.9. There are scripts around
too (one in my scriptutils). Adjust size to whatever the card is.

If you have a lot of read errors or a lot of empty card space copy with
cp --sparse=always to save disk space.

Another interesting property of some flash memory is that uninitialised
memory reads back as random values, meaning you can't md5sum an unused
flash stick. You have to clear it with cat /dev/zero >/dev/sdX once
first. (Save the partition table first with fdisk -l, and re-create it
with the same settings.)

Volker

Greg Trounson

2011-06-28 02:36:27 UTC