ubuntu - disk sync

[Atom Smasher]

this is annoying... if i use rsync or pv to copy stuff to an external
drive, i see it going up to 100% done and then the command exits... but
the light on the drive still indicates that it's writing... and when i run
"sync" it doesn't return until the disk light turns off, which tells me
that this has nothing to do with disk cache.

why is the progress indicator getting ahead of what's really happening?
what can i do to fix it?

thanks...
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"Today every inhabitant of this planet must contemplate the
day when this planet may no longer be habitable. Every man,
woman and child lives under a nuclear sword of Damocles,
hanging by the slenderest of threads, capable of being cut
at any moment by accident or miscalculation or madness."
-- John F. Kennedy


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Steve Holdoway]

Post by Atom Smasher
this is annoying... if i use rsync or pv to copy stuff to an external
drive, i see it going up to 100% done and then the command exits... but
the light on the drive still indicates that it's writing... and when i run
"sync" it doesn't return until the disk light turns off, which tells me
that this has nothing to do with disk cache.
why is the progress indicator getting ahead of what's really happening?
what can i do to fix it?
thanks...

Spare memory is used as disk cache. The rsync exits when all data is
written to that cache, sync then flushes that cache to disk.

Linux is making best use of the available resources. It doesn't really
need fixing...

hth,

Steve
--
Steve Holdoway <***@greengecko.co.nz>
http://www.greengecko.co.nz
MSN: ***@greengecko.co.nz
Skype: sholdowa


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Atom Smasher]

Post by Steve Holdoway
Spare memory is used as disk cache. The rsync exits when all data is
written to that cache, sync then flushes that cache to disk.
Linux is making best use of the available resources. It doesn't really
need fixing...

======================

i don't care when the data is done writing to spare RAM... i care when the
data is done writing to its destination.

on bsd, the disk light goes out just when rsync reaches 100%. i would say
that bsd is representing the data transfer accurately, and linux is not.
thus, i'd like to "fix" it on linux so that it accurately represents how
the data is being written to its destination.
--
...atom

________________________
http://atom.smasher.org/
762A 3B98 A3C3 96C9 C6B7 582A B88D 52E4 D9F5 7808
-------------------------------------------------

"There can be no greater good than the quest for peace,
and no finer purpose than the preservation of freedom."
-- U.S. President Ronald Reagan


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Steve Holdoway]

Post by Atom Smasher

Post by Steve Holdoway
Spare memory is used as disk cache. The rsync exits when all data is
written to that cache, sync then flushes that cache to disk.
Linux is making best use of the available resources. It doesn't really
need fixing...

======================
i don't care when the data is done writing to spare RAM... i care when the
data is done writing to its destination.
on bsd, the disk light goes out just when rsync reaches 100%. i would say
that bsd is representing the data transfer accurately, and linux is not.
thus, i'd like to "fix" it on linux so that it accurately represents how
the data is being written to its destination.

write a wrapper?
--
Steve Holdoway <***@greengecko.co.nz>
http://www.greengecko.co.nz
MSN: ***@greengecko.co.nz
Skype: sholdowa


_______________________________________________
NZLUG mailing list ***@linux.net.nz
http://www.linux.net.nz/cgi-bin/mailman/listinfo/nzlug

[Daniel Pittman]

Post by Atom Smasher

Post by Steve Holdoway
Spare memory is used as disk cache. The rsync exits when all data is
written to that cache, sync then flushes that cache to disk.
Linux is making best use of the available resources. It doesn't really need
fixing...

======================
i don't care when the data is done writing to spare RAM... i care when the
data is done writing to its destination.

You can, as someone suggested, mount with 'sync' to enforce that every write
is synchronous. Then you will learn in short order that you *do* care about
writes going to cache and background flushing - the performance cost is pretty
huge.

Post by Atom Smasher
on bsd, the disk light goes out just when rsync reaches 100%.

That seems hard to credit, given that it is such a performance drain.
Certainly FreeBSD and OpenBSD both do write-behind flushing when the software
doesn't demand immediacy of writes.

Post by Atom Smasher
i would say that bsd is representing the data transfer accurately, and linux
is not.

Did you know that the kernels (and POSIX) provide a mechanism for the
application software to decide if it wants to know when the data hits disk or
not?

Post by Atom Smasher
thus, i'd like to "fix" it on linux so that it accurately represents how the
data is being written to its destination.

Nothing is broken, so there is nothing to "fix" other than your misassumptions
about how things best work, I fear.


Anyway, if you really want this behaviour everywhere just mount 'sync' and
accept the stupidly huge performance cost.

If you want it from specific applications then either use, or add, an option
to fsync the content after write.

Regards,
Daniel

[Robin Sheat]

Post by Atom Smasher
this is annoying... if i use rsync or pv to copy stuff to an external
drive

Mount it with the 'sync' option, perhaps. I think this might (have been|be)
default for vfat devices, but I know it's not for ext* USB drives.

However, it will make things appear slower, because you lose some of the
benefit that write caching gives you. But depending on your use-case, it might
be the right thing to do.

Robin.