[mythtv-users] Two files after transcoding

[Daryl McDonald]

My system turns off and on as needed. I ran the orphans.py and it only
found one third of the .old files I could see in the recordings file.
I executed the four suggestions, including rescan, and it showed no
further orphans. It was not until the next day, at least six-teen
hours later that I rounded up and deleted 36 remaining .old files. So
with what I'm reading recently, I'm confused. If .old files are not in
the database and orphans.py finds files not in the database, why did
it only find one third of the 262 GB's of them?

On Tue, Mar 18, 2014 at 10:25 AM, Michael T. Dean
<mtdean at thirdcontact.com> wrote:
> On 03/17/2014 04:42 PM, Mark Perkins wrote:
>>>
>>> The successive runs that still showed orphans were simply showing files
>>> that the backend had not yet finished deleting.  Had you just waited, they
>>> would have disappeared, eventually, with only the one run.
>>
>> Mike is making quite an important point here that took me few goes to
>> understand when I first used find_orphans.py. (IIUC) find_orphans.py still
>> uses the slow delete function that is built into Mythtv, so even though the
>> script returns back to the mini-menu or appears complete the delete process
>> is just starting and could still run for hours if there are a lot if files.
>
>
> Yes, and that's an important clarification, too--it could take hours (and
> many, many of them), depending on your system configuration and the size of
> the files being deleted.
>
>
>> On my first go I just assumed it would be instantaneous like 'rm
>> xxxx_xxxxxxxxx.mpg'.
>>
>> I'm not sure but I assume that it would be safe for the system to shutdown
>> and the delete would just continue at next restart?
>
>
> If you have slow deletes enabled, shutting down the system would result in
> all the deletes completing "instantaneously" (read "as quickly as your file
> system can handle"--which may mean several seconds to many minutes,
> depending on size of files being deleted, the file system, and the file
> system options in use).  Generally in this situation, system start/stop
> scripts will time out waiting for the kernel's process cleanup to complete
> (note that mythbackend is shut down already, and we're simply waiting on the
> I/O of the deletion process to complete, so it's all in the kernel's hands,
> now) and report an error shutting down mythbackend.  If you wait, patiently
> (more so than your start/stop script) until the process completes shutdown
> (when it's no longer visible with ps -efw | grep mythbackend), then all is
> well, and you can restart mythbackend without issues.
>
> If, instead, you kill -KILL <pid> (or do the same thing, but incorrectly,
> with kill -9 <pid>), you may be creating inconsistencies in your file
> system.  Similarly, this could occur if you shut down the system (or, worse,
> hard power it off) before the "as fast as your file system allows" deletion
> occurs (depending on your system stop scripts and how they deal with a
> slow-exiting process).  In theory, a good journaling file system will ensure
> the deletions complete properly, even if you do any of the above, but at
> this point, it's completely out of MythTV's hands (from the point you
> attempted to shut down mythbackend in the first place).
>
> Note, too, that until the process cleanup completes, system start/stop
> scripts could have issues restarting mythbackend.  These issues could result
> in either error messages from the start script (which would be good) or
> "smart" start scripts could do things to try to work around issues when they
> notice the old process is still there--possibly even issuing a kill -KILL.
>
> And, finally, if you do not have slow deletes enabled and you delete a
> copious amount of data all at once through mythbackend then "quickly" shut
> down mythbackend, it may result in the files not being deleted.  In the case
> of orphaned files that were being deleted, you'd be no worse off than you
> were before.  In the case of actual recording files (with valid recording
> metadata in the database), you could actually get some "stuck" recordings
> that are in a limbo between deleted and not deleted/between orphaned and not
> orphaned, but still take up gigabytes of space on the hard drive.  My
> recommendation is actually to enable slow deletes, regardless of your file
> system (and how 1337 it is at deleting large files), just to prevent this
> race--at least until I have a workaround in place that would clean up the
> mess created by shutting down a "fast deletes" mythbackend at exactly the
> wrong time after deleting a huge number of recordings.  (Note, though, that
> this race condition is pretty hard to trigger, as there's a very small
> window during which you'd need to shut down mythbackend and--TTBOMK--it has
> to be a "hard" shutdown, with kill -KILL or hard power off.)
>
> Mike
>
> _______________________________________________
> mythtv-users mailing list
> mythtv-users at mythtv.org
> http://www.mythtv.org/mailman/listinfo/mythtv-users
> http://wiki.mythtv.org/Mailing_List_etiquette
> MythTV Forums: https://forum.mythtv.org