S3 Sync - Documentation

SprightlySoft S3 Sync synchronizes a folder on your computer with a folder on Amazon S3.

Requirements

The application requires a Microsoft Windows computer running the Microsoft .NET Framework 2.0 or later. You will also need an Amazon AWS account with S3 enabled.

How to Run the Program

Open a command prompt window where S3Sync.exe is located and type S3Sync.exe with the command line parameters you require to synchronize your files. Here is an example of what you would type:

S3Sync.exe -AWSAccessKeyId xxxxxxxxxxxx -AWSSecretAccessKey xxxxxxxxxxxxxx -LicenseKey xxxxxxxxxxxx -SyncDirection upload -LocalFolderPath "C:\myfolder" -BucketName mybucket -S3FolderKeyName "myfolder/"

To simplify the process of building and running a command line, you can use the S3 Sync Command-Line Wizard application included in the installation package.

Command Line Parameters

The following describes the parameters of S3 Sync.

-SyncDirection (required) - The direction of the synchronization.

• Upload - Upload files from your computer to Amazon S3. Your computer will be the source.
• Download - Download files from Amazon S3 to your computer. Amazon S3 will be the source.
• Bidirectional - Upload files from your computer to S3 and download files from S3 to your computer. A cache files is used to determine what has changed on your computer and what has changed on S3 since the last sync.

-AWSAccessKeyId (required) - Your AWS access key ID provided by Amazon.

-AWSSecretAccessKey (required) - Your AWS secret access key provided by Amazon.

-LicenseKey (required) - Your license key for S3 Sync. You can purchase a license key or request a free trial from http://sprightlysoft.com/S3Sync/.

-UseSSL (optional) - Specifies whether SSL should be used when communicating with Amazon. The default value is true.

• True - Request URLs to Amazon will begin with https://.
• False - Request URLs will begin with http://.

-IgnoreSSLCertificateErrors (optional) - When true the application will not exit with an error if the SSL certificate of Amazon has an error. You would get an SSL error if your bucket name has a period in it. The default value is false.

• True - The program will not exit with an error if an SSL certificate error exists.
• False - The program will exit with an error if an SSL certificate error exists.

-RequestEndpoint (optional) - The endpoint to use in request URLs to Amazon. If no value is specified the application will determine which endpoint to use based on the location of your bucket. You can use this parameter to connect to other providers. Set this parameter to s3-us-gov-west-1.amazonaws.com to connect to GovCloud or set it to s.greenqloud.com to connect to GreenQloud. To view the list of Amazon S3 endpoints visit http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region

-BucketName (required) - The name of the Amazon S3 bucket which you will be synchronizing files with.

-S3FolderKeyName (optional) - The key name of the Amazon S3 folder you will synchronize files with. A folder key name look like "rootfolder/subfolder/". This folder will be created if it does not exist. Use an empty value if you want to synchronize to the root of the bucket.

-LocalFolderPath (required) - The folder on your computer that you will be synchronizing with. This can be any valid path. The path will look like "C:\MyFolder".

-LogOnlyMode (optional) - Specifies whether changes should be applied to the destination or if changes should only be logged and no actions will be performed. The default value is false.

• True - the program will run, check what changes are required, and report these changes. The program will not make any changes to the destination.
• False - The program will check which files need to be deleted and transferred, and then it will delete and transfer the required files.

-OutputLevel (optional) - Specifies what level of logging the program should output. The default value is 1.

• 1 - Minimal logging. Only show what will be deleted and what will be transferred.
• 2 - Detailed logging. Show all files that exist between both systems and show debug information in the event of an error.
• 3 - HTTP logging. Show all information of detailed logging and all HTTP requests to and from Amazon.

-LogFilePath (optional) - The path and file name where a log file will be generated. The log file contains all information that was written to the console window. Leave the value empty if you do not want to create a log file. The path will look like "C:\Temp\S3SyncLog.txt".

You can substitute parameters names surrounded with the < > characters with parameter values from the command line. For example if your BucketName is MyBucket you can set your LogFilePath to "C:\Temp\S3Sync<BucketName>Log.txt" and the application will create a log file with the name C:\Temp\S3SyncMyBucketLog.txt. If you set your LogFilePath in the S3Sync.exe.config file you will need to encode the < and > signs. The value would look like C:\Temp\S3Sync&lt;BucketName&gt;Log.txt.

You can include <<yyyy-MM-dd HH.mm.ss>> in the LogFilePath and the application will replace it with the current date and time.

-OverwriteExistingLogFile (optional) - Specifies whether an existing log file should be overwritten when the program runs. The default value is true.

• True - Delete the existing log file when the program starts. The log file will only contain data about the current execution of the program.
• False - Append to the existing log file and do not overwrite data.

-LogEncodingCodePage (optional) - The format the log file is encoded in. To see a fill list of values see the .NET Name column at https://msdn.microsoft.com/en-us/library/windows/desktop/dd317756(v=vs.85).aspx. The default value is utf-16.

-CompareFilesBy (optional) - When a file exist on the source and destination with the same name different methods can be used to determine if the data in the files are different and if the source file needs to be copied over the destination file. The methods to determine if files are different are listed below. The default value is Timestamp.

• ETag - Compares the source file's ETag hash with the destination file's ETag hash. This is the most accurate method but it takes time to calculate the ETag of a local file. If there is no ETag for a S3 file that was uploaded with multipart upload, the application compares files by timestamp.
• Timestamp - Compares files by the date modified timestamp and size. The file on S3 needs the date modified timestamp in its metadata. If you uploaded the file with S3 Sync and set SaveTimestampsInMetadata to true the timestamp was saved in the metadata. If the timestamp and size of the local file and the timestamp and size of the file on S3 are the same, the files are considered the same. If a timestamp does not exist for a file on S3, files are compared by size only. When the SyncDirection is upload or download the application save the remote timestamps to a file under the CachePath. The application will look for this file when doing a synchronization to avoid getting the timestamps from Amazon.
• Size - Compare the size of the source file to the size of the destination file. If both files are the same size, the files are considered the same. This method is quicker that ETag but is not as accurate. Two files may have the identical size but not the identical content. This method does not require a value in the metadata like the Timestamp method.

-CacheRemoteTimestamps (optional) - During an upload sync or download sync when the CompareFilesBy method is timestamp, the application can store the timestamp values from S3 files in a local file. By caching these values successive syncs will not need to retrieve the timestamp value for each file from S3. This makes the sync process quicker. The cache file is stored under CachePath value. The default value is true.

-UploadThreads (optional) - Sets the number of concurrent uploads. Use this to upload multiple files at the same time which will improve your bandwidth utilization. The default value is 4.

-DownloadThreads (optional) - Sets the number of concurrent downloads. The default value is 4.

-ThreadUploadSpeedLimitKBps (optional) - Sets the maximum upload transfer speed in kilobytes per second. Set this value to 0 if you do not wish to limit the speed of an upload. This setting applies to each upload thread and is not a global setting. The default value is 0.

-ThreadDownloadSpeedLimitKBps (optional) - Sets the maximum download transfer speed in kilobytes per second. Set this value to 0 if you do not wish to limit the speed of a download. This setting applies to each download thread and is not a global setting. The default value is 0.

-ShowTransferProgress (optional) - Show upload and download progress while uploads and downloads are taking place. If you are calling S3 Sync from another application and that application reads the output from the console window, that application may throw an error when the progress is updated. Set this value to false to prevent S3 Sync from reporting the progress when files are being transferred. The default is true.

• True - Show progress during a transfer.
• False - Do not show progress during a transfer.

-S3ErrorRetries (optional) - The number of times a failure communicating with S3 should be retried. The delay between retries increases as the number of retries increases. Only certain errors are retried such as connection timeout errors and 503 responses from Amazon. The default value is 5.

-SignatureVersion (optional) - The signature version used when generating an authentication header for a request. See http://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html for more information. The default is 4.

• 2 - Signature version 2 will be used.
• 4 - Signature version 4 will be used.

-Region (optional) - The region used when creating a signature version 4 authentication header. If no value is specified the application gets the region from the bucket location.

-RequestHeaders (optional) - A list of headers to add in each request to Amazon. Multiple headers must be separated by the vertical bar character. Header name and header value must be separated by the colon character. Example; the value will look like "x-amz-security-token:Gg6etqDaVwN89gYTcR7sgNRm7GGqyD71NWsBQ==" when using temporary security credentials.

The following parameters only apply when the SyncDirection is "Upload" or "Bidirectional":

-UploadHeaders (optional) - A list of headers to add in the request when uploading files. Multiple headers must be separated by the vertical bar character. Header name and header value must be separated by the colon character. Example; use "x-amz-acl:public-read|x-amz-storage-class:REDUCED_REDUNDANCY" to make uploaded files public and stored with reduced redundancy.

-OverrideContentTypes (optional) - The content type of a file is set when uploading a file to S3. The program determines the content type of a file by it's file name extension. You can overwrite one of these values or specify a new value here. Separate multiple entries with the vertical bar character. Separate extensions and values with the colon character. Example; use ".zip:application/x-zip-compressed|.aspx:text/plain" to set the content type to application/x-zip-compressed for .zip files and text/plain for .aspx files.

-UseMultipartUpload (optional) - Specifies if Amazon's multipart upload feature should be used when uploading large files to S3. Multipart upload is beneficial since you can resume an upload if it gets interrupted. If a file is smaller than the MultipartPartSizeBytes value, multipart upload will not be used on that file regardless of this setting. The default value is true.

• True - Use multipart upload.
• False - Do not use multipart upload.

-MultipartPartSizeBytes (optional) - The size of a part in bytes when using multipart upload. The smallest size a part can be is 5242880. The default value is 5242880.

-SaveETagInMetadataForMultipart (optional) - When a file is uploaded to S3 using multipart upload, Amazon does not save the ETag in a standard way. If you are comparing file content by ETag, use this setting to make the application calculate the ETag of file and save it in metadata during the upload. The metadata ETag will be used when the application compares file content by ETag. The default value is true.

-SaveTimestampsInMetadata (optional) - The application can save the last write time and creation time of files you upload to metadata in S3. These values can be used when restoring files locally and you would like to set file timestamps to their original values. Timestamps in metadata are required if you are comparing files by Timestamp. The default value is true.

-IncludeLocalSubFolders (optional) - Specifies whether sub folders of your LocalFolderPath should be synchronized to S3. The default value is true.

• True - Synchronize sub folders to S3.
• False - Do not synchronize sub folders to S3. Only files in the LocalFolderPath path will be synchronized.

-ExcludeLocalFolders (optional) - A list of local folder paths that will be excluded from synchronization. Multiple paths should be separated with the vertical bar character. Example; use "C:\MySync\DoNotSyncA|C:\MySync\DoNotSyncB" to exclude the folders C:\MySync\DoNotSyncA and C:\MySync\DoNotSyncB.

-ExcludeLocalFilesRegularExpression (optional) - Do not synchronize files that match the regular expression. Example; use ".*\.jpg$|.*\.gif$" to exclude files that end with . jpg or .gif. Use ".*\\My\ Music\\.*|.*\\Downloads\\.*" to exclude files where \My Music\ or \Downloads\ is in the path.

-ExcludeLocalFoldersRegularExpression (optional) - Do not synchronize folders that match the regular expression. Example; use ".*My\ Documents\\My\ Music.*" to exclude folders that contain My Documents\My Music in the path.

-IncludeOnlyLocalFilesRegularExpression (optional) - Only synchronize files where the file names match the regular expression. Example; use ".*\.txt$" to only synchronize files that end with .txt.

-IncludeOnlyLocalFoldersRegularExpression (optional) - Only synchronize files where the folder path matches the regular expression.

-DeleteS3Items (optional) - Delete files on S3 that do not exist in the local list of files. Any extra files that do not exist on your computer will be deleted from S3. The default value is true.

• True - Delete files on S3 that do not exist on your local system.
• False - Leave extra files on S3. Do not delete them.

The following parameters only apply when the SyncDirection is "Download" or "Bidirectional":

-GetETagFromMetadataForMultipart (optional) - When comparing files by ETag, the application calculates the ETag of the local file and gets the ETag of the file on S3. If the file on S3 was upload with the multipart upload technique, the ETag value will be a nonstandard value and it cannot be used to match against the local file. If the standard ETag value was stored in S3 metadata this option can be used to get the ETag from metadata. If no ETag is found in metadata, files are compared by size. The default value is true.

• True - Get the ETag of files upload with multipart upload from S3 metadata.
• False - Do not check metadata for the standard format ETag. Multipart uploaded files will be compared by size.

-RestoreTimestampsFromMetadata (optional) - If the last write time and creation time of files were stored in S3 metadata this option will set the time stamp of files downloaded to the values in metadata. The default value is true.

-ExcludeS3FilesRegularExpression (optional) - Do not synchronize files that match the regular expression. Example; use ".*\.jpg$|.*\.gif$" to exclude files that end with . jpg or .gif from synchronization.

-ExcludeS3FoldersRegularExpression (optional) - Do not synchronize files where the folder matches the regular expression.

-IncludeOnlyS3FilesRegularExpression (optional) - Only synchronize files where the file names match the regular expression. Example; use ".*\.txt$" to only synchronize files that end with .txt. Use ".*file1\.txt$|.*file2\.txt$" to download file1.txt and file2.txt.

-IncludeOnlyS3FoldersRegularExpression (optional) - Only synchronize files where the folder name match the regular expression.

-DeleteLocalItems (optional) - Delete local files that do not exist in S3. Any extra files that exist on your computer but do not exist on S3 will be deleted. The default value is true.

• True - Delete local files that do not exist on S3.
• False - Leave extra local files. Do not delete them.

-DeleteLocalItemsToRecycleBin (optional) - When deleting a local file, send it to the recycle bin. If a file path is longer than 260 characters the file will not be placed in the recycle bin regardless of this value. The default value is true.

• True - Send deleted files to the recycle bin.
• False - Permanently delete local files. Do not use the recycle bin.

-GlacierRecheckWaitSeconds (optional) - Specifies how long to wait between checks of completed Glacier restore operations. If a file is on the Glacier storage class and you would like to download it, the application submits a request to restore the file. Amazon takes up to 5 hours to process the request and make the file available. The GlacierRecheckWaitSeconds parameter indicates how long to wait between checks to see if the file is available. The default value is 300.

The following parameters only apply when the SyncDirection "Bidirectional":

-FirstTimeBidirectionalSync (optional) - This setting tells the application what to do when no cache file exists and a bidirectional sync will be run for the first time. In a bidirectional sync local files are upload to S3 and S3 files are downloaded. The default value is Merge.

• Merge - Combine local files and S3 files. If a file with the same name exist on S3 and locally, the ConflictWinner parameter will determine which file wins the conflict. The ConflictHandling parameter determines what to do with the looser file.
• Upload - Make S3 look like your local system. Your files will be uploaded and any extra files on S3 will be deleted.
• Download - Make your local system look like S3. Files on S3 will be downloaded and any extra local files will be deleted.

-ConflictWinner (optional) - When a file is changed on S3 and locally, this parameter indicated which file should be kept. The default value is LatestTimestamp.

• LatestTimestamp - The file with the most recent timestamp parameter is the winner. The timestamp of a file on S3 is kept in a x-amz-meta-local-date-modified header value. This value is set by S3 Sync.
• Local - The local file is the winner.
• S3 - The file on S3 is the winner.

-NoTimestampConflictWinner (optional) - This parameter specifies what to do when ConflictWinner is LatestTimestamp and no timestamp exists for the file on S3. The default value is Error.

• Local - The local file is the winner.
• S3 - The file on S3 is the winner.
• Error - An error is raised and no changes take place. The conflict will need to be manually resolved by deleting the file or renaming it.

-ConflictHandling (optional) - This parameter tells the application what to do with the conflict looser file. The default value is Rename.

• Rename - The looser file is renamed. The new file name will contain a timestamp and the name of the location where the file came from.
• Overwrite - The looser file is deleted and the winner will take it's place.
• Error - An error is raised and no changes take place. The conflict will need to be manually resolved by deleting the file or renaming it.

-CachePath (optional) - The folder where the cache files are stored. The cache file contains a list of files that existed when the last sync took place. A cache file is generated for local files and for files on S3. The default value on Windows 2000 and Windows XP is %USERPROFILE%\Local Settings\Application Data\SprightlySoft\S3Sync. The default value on Windows Vista and later is %USERPROFILE%\AppData\Local\SprightlySoft\S3Sync.

-VerifyCacheOnExit (optional) - This parameter tells the application if it should verify the cache file when a sync is complete. A verification is done by listing files on S3 and listing files in the S3 cache. If the 2 lists are the same the cache is valid. If the cache is not valid a problem occurred and an error is raised. The default value is true.

-EmailLogCondition (optional) - Specifies under what conditions an email should be sent when the program completes. The email will contain the log of the program's operation. The default value is 0.

• 0 - Never send an email.
• 1 - Send an email when there is an error.
• 2 - Always send an email when the program completes.

-EmailSmtpHost (optional) - The SMTP server used to send the email.

-EmailSmtpPort (optional) - The port of the SMTP server.

-EmailSmtpEnableSsl (optional) - If SSL should be used with the SMTP server. The default value is false.

• True - Use a SSL connection.
• False - Do not use a SSL connection.

-EmailSmtpUserName (optional) - The user name to log into the SMTP server. Keep this value empty is your SMTP server does not require authentication.

-EmailSmtpPassword (optional) - The password used to log into the SMTP server.

-EmailFromAddress (optional) - The from address to be used in the email.

-EmailRecipients (optional) - The email addresses the email will be sent to. Separate multiple addresses with a comma.

-EmailSubject (optional) - The subject of the email that will be sent. Use <ExecuteResult> to show Success, Failed, or Warning regarding the execution of the program. Use <MachineName> to show the computer name the program is currently running on. Enter parameter names surrounded by < and > to show the value of those parameters. The default value is "S3 Sync <ExecuteResult> <LocalFolderPath>".

Exit Codes

The application returns an exit code when it completes. The exit code is visible at the bottom of the log file, in the console window, and from other programs calling the application. The following is a list of exit codes the application can return.

• 0 - The application completed successfully.
• -1 - An error occurred and no further actions were performed.
• -2 - An error occurred and the application continued. This error is generated when a file or folder cannot be accessed. Possible reasons may be the file is in use, you do not have permission to the folder, or the file is invalid. Check the log file for the full error message.

Running the Application As a Scheduled Task

If you would like to run the application regularly you can use Window's built in Task Scheduler. Read the following article on how to create a scheduled task. How to Create a Scheduled Task to Automate S3 Sync