Incomplete files from scheduled ftp transfer.

Everything about TurboFTP Sync Service.

Moderators: idler, Saturn

Incomplete files from scheduled ftp transfer.

Postby kwijibot » Thu Jul 27, 2006 4:39 am

If files are copying to a monitored ftp folder from outside and the scheduled cycle starts does Turboftp detect the file still building? What does it do?

We are trialing Turboftp to drop pdf pages into our server hotfolders from client folders on our ftp site. The files in the hotfolders are then removed automatically by the software.

Some files are erroring due to the pdf files being incomplete.

Is there a way to copy only files that are complete?

Thanks, kwijibot.
kwijibot
 
Posts: 11
Joined: Thu Jul 27, 2006 4:35 am

Postby idler » Fri Jul 28, 2006 6:08 pm

Hello,

Incomplete files can download because the new file appears in the directory before it is completely saved to disk. Any data in buffers (or still to be generated) never downloads.

We wondered if file locks could resolve this situation? But unless both the FTP server and the application savings files into your monitored directories apply exclusive file locks, there is no guarantee that an incomplete file will not download.

I have verified this by setting a file lock on a file on my server. If I attempt to edit the same file while requesting a lock, my session pauses; if I just start in editing without referencing the lock, I can edit the file notwithstanding a lock. Sadly, that is how file locking operates: any participant that ignores the lock can bypass the synchronization imposed by the locking mechanism.

Onto TurboFTP: I then attempted to download the locked file through TurboFTP to a PC client in my lab through my FTP server. Since the file downloads without incident, I conclude that this FTP server (vsftpd on Linux) ignores file locks.

We could advise you to implement file locking in your applicadtion, but without a FTP server that respects file locks a secure solution can not be achieved. Moreover, you may not have authority to install another FTP server (that may be your ISP's perogative) and unless you have source to your application, implementing file locks may not be possible.

Nevertheless, there may be a satisfactory solution: If you save these files into another directory and then move (or rename) them into your download directory, then only complete files should appear for download. It is vital that you move these files because if you copy them the same phenomenon can occur: namely, the file name enters the directory while the system is copying the file contents. However, if you move (or rename) the file, then its location on disk never changes - only links within directories. Therefore, the download should be complete as soon as a file appears in the download directory.

Please let us know if this information has been helpful.

Alan
idler
Support Staff
 
Posts: 5
Joined: Thu Jul 13, 2006 6:54 am

Postby kwijibot » Sun Jul 30, 2006 9:37 pm

[quote="idler"]Nevertheless, there may be a satisfactory solution: If you save these files into another directory and then move (or rename) them into your download directory, then only complete files should appear for download. It is vital that you move these files because if you copy them the same phenomenon can occur: namely, the file name enters the directory while the system is copying the file contents. However, if you move (or rename) the file, then its location on disk never changes - only links within directories. Therefore, the download should be complete as soon as a file appears in the download directory.[/quote]

I looked around for an option to rename the files on the way through but was unable to find the option within the schedule task settings.

I was going to do as you say and copy to a 'temp' directory then when completed 'rename' to a new directory - which sounds like it would work.

But not having the option rules it out.

Am i missing something?
kwijibot
 
Posts: 11
Joined: Thu Jul 27, 2006 4:35 am

Postby admin » Mon Jul 31, 2006 7:07 am

>Some files are erroring due to the pdf files being incomplete.

What errors you actually refer to?

Leon
admin
Support Staff
 
Posts: 267
Joined: Mon Jul 11, 2005 1:24 am

Postby idler » Mon Jul 31, 2006 4:19 pm

Leon,

I believe this user hopes to use TurboFTP to upload PDF files to their server. He observes incomplete (and consequently unusable) PDF files on the server.

These incomplete files may occur if an upload aborts or requires a restart. And during any upload there is always a time when the PDF file is allegedly available because it appears in the upload directory, but all the buffers have not been transmitted and saved on the server.

I believe an upload should be directed into a hidden directory where files are not published until all the data has been saved. Then, the files can be moved into the published directory without exposing an incomplete file.

I know I can rename files within command line FTP clients. Can I do the same within TurboFTP? I believe I can because I uploaded a file into a directory test1 on my test server with TurboFTP and then moved the file into a directory test2 using TurboFTP. If I select a file in the server pane, the popup menu (use the right mouse button) offers options to "Move to ...." or "Rename ...". The commands logged are those I would type to perform such a task in a command line FTP client.

These commands would appear to be what our user wants for this part of the task. Can he script this through TurboFTP's task Schedule? Otherwise, he would have to manually move files after their upload.

Alan
idler
Support Staff
 
Posts: 5
Joined: Thu Jul 13, 2006 6:54 am

Postby admin » Tue Aug 01, 2006 3:02 am

I need to see the log to know what errors the user is referring to and I need more details that relate to the user description "Some files are erroring due to the pdf files being incomplete. " Does "incomplete" mean the transferred file is corrupted in transit?

One way to prevent corrupted file in upload or download is to enable
Options | Advanced Con't | Check transferred file integrity

When enabled, the program will issue XCRC command after each file transfer to verify if the transferred file matches the original. If not, the file will be retransferred until it passes the CRC check.

Again, please forward us the task log showing the error so that we can better understand the problem.

Thanks.

Leon
admin
Support Staff
 
Posts: 267
Joined: Mon Jul 11, 2005 1:24 am

Postby kwijibot » Wed Aug 02, 2006 2:03 am

Leon, i should probably have made it more clear - the errors are not in turbo ftp but are in our Prepress software.

The software is erroring when it trys to process the pdf files which turbo ftp is copying from the ftp site because they are incomplete.

I am trying to work out a way of ensuring the pdf files are never left incomplete after a sync cycle has passed.

I have the 'Check file integrity already activated'

As the other admin said i really would prefer it if turbo ftp built the file in a temp directory first and then copied it completely over into the main directory. Is there any way to configure this using the scheduled sync?
kwijibot
 
Posts: 11
Joined: Thu Jul 27, 2006 4:35 am

Postby idler » Wed Aug 02, 2006 3:42 pm

Hello,

Let's suppose the first download was always a known file (say I_AM_DOWNLOADING.txt). Then, any downloads would follow. And finally, the last file to download would be another special file (say I_AM_DONE.txt). With appropriate choice of filenames or timestamps, I believe this could be accomplished.

Could you script on your system to delay any processing if the first of the special files is in the destination directory, but the final one hasn't arrived yet? After processing, the special files would need to be cleared.

Alan
idler
Support Staff
 
Posts: 5
Joined: Thu Jul 13, 2006 6:54 am

Postby kwijibot » Fri Aug 04, 2006 3:45 am

Unfortunately the files do not come down in any particular order.

As they are made and checked they are sent to our ftp site.
kwijibot
 
Posts: 11
Joined: Thu Jul 27, 2006 4:35 am

Postby kwijibot » Sat Sep 02, 2006 6:34 am

I have come to the conclusion that there should be an option to build and check the file integrity in a temp folder before copying to the destination.

This would be better in a commercial situation like ours.
kwijibot
 
Posts: 11
Joined: Thu Jul 27, 2006 4:35 am

Postby admin » Mon Sep 04, 2006 4:38 pm

Hi Kwijibot,

I am suspecting it is caused by another problem. Do you have the file corruption when running multiple upload tasks at the same time? If so upgrade to 5.00.540 and see if it still persists.

Leon
admin
Support Staff
 
Posts: 267
Joined: Mon Jul 11, 2005 1:24 am

Postby kwijibot » Sun Sep 10, 2006 9:38 pm

[quote="admin"]Hi Kwijibot,
I am suspecting it is caused by another problem. Do you have the file corruption when running multiple upload tasks at the same time? If so upgrade to 5.00.540 and see if it still persists.
Leon[/quote]

Yes it sounds like my problem but i do not upload only download.

I have updated and will see if it is better.

Kwijibot.
kwijibot
 
Posts: 11
Joined: Thu Jul 27, 2006 4:35 am

Postby admin » Wed Sep 27, 2006 5:38 pm

Hi,

How does it work out for you?

Leon
admin
Support Staff
 
Posts: 267
Joined: Mon Jul 11, 2005 1:24 am

Postby admin » Tue Feb 13, 2007 3:55 am

Hi,

A fix is on the way you will get rid of the problem with 5.30.

Leon
admin
Support Staff
 
Posts: 267
Joined: Mon Jul 11, 2005 1:24 am


Return to TurboFTP Sync Service

Who is online

Users browsing this forum: No registered users and 1 guest

cron