You can use a batch file to execute commands after S3 Sync completes. Say you want to upload files to S3 then delete the local files if the upload was successful. The following batch file is an example of this process.

Upload files to S3 then delete local files.

C:\Program Files (x86)\SprightlySoft\S3 Sync\S3Sync.exe" -AWSAccessKeyId xxxxxxxxxxxx -AWSSecretAccessKey xxxxxxxxxxxx -BucketName mybucket -S3FolderKeyName "myfolder/" -SyncDirection Upload -LocalFolderPath "C:\myfolder" -DeleteS3Items true -LogOnlyMode false -OutputLevel 1 -CompareFilesBy Timestamp -LicenseKey xxxxxxxxxxxx -LogFilePath "C:\Temp\S3 Sync <BucketName> <S3FolderKeyName> <<yyyy-MM-dd HH.mm.ss>> <SyncDirection>.txt"

@if %ERRORLEVEL% neq 0 goto error

@echo S3 upload successful.
RMDIR "C:\myfolder" /Q /S
REM pause
exit 0

:error
@echo S3 upload error. Local delete not executed.
REM pause
exit 1
  • The first line in the script above uploads files with S3 Sync.
  • The %ERRORLEVEL% line checks the error level returned from S3 Sync. If the error level is not 0, the application skips the next block of code and goes to the :error section.
  • The @echo line writes a message to the console window.
  • The line with RMDIR deletes the local folder. This line will only be executed if error level from S3 Sync was 0 which means success.
  • The REM pause line would wait for the use to press a key to continue. The REM statement comments out the line and it will not be executed. Remove REM if you would like to test the script and want to be notified before continuing.
  • The exit 0 line exits the batch script with a success code. The remainder of the script will not be executed.
  • The :error line is a section marker of the script.
  • The @echo line writes a message to the console window.
  • The REM pause line can be uncommented during testing.
  • The exit 1 line exits the batch script with an error code.

The following is an example of downloading files with S3 Sync then deleting those files with S3 Delete. You can get S3 Delete for free at http://sprightlysoft.com/S3Delete/

Download files from S3 then delete files on S3.

"C:\Program Files (x86)\SprightlySoft\S3 Sync\S3Sync.exe" -AWSAccessKeyId xxxxxxxxxxxx -AWSSecretAccessKey xxxxxxxxxxxx -BucketName mybucket -S3FolderKeyName "myfolder/" -SyncDirection Download -LocalFolderPath "C:\myfolder" -DeleteLocalItems true -LogOnlyMode false -OutputLevel 1 -CompareFilesBy Timestamp -LicenseKey xxxxxxxxxxxx -LogFilePath "C:\Temp\S3 Sync <BucketName> <S3FolderKeyName> <<yyyy-MM-dd HH.mm.ss>> <SyncDirection>.txt"

@if %ERRORLEVEL% neq 0 goto error

@echo S3 download successful.
"C:\Program Files (x86)\SprightlySoft\S3 Delete\S3Delete.exe" -AWSAccessKeyId xxxxxxxxxxxx -AWSSecretAccessKey xxxxxxxxxxxx -BucketName mybucket -S3KeyName "myfolder/" -LogFilePath "C:\Temp\S3 Sync delete.txt" -OutputLevel 2
REM pause
exit 0

:error
@echo S3 download error. S3 delete not executed.
REM pause
exit 1
  • The first line in the script above downloads files with S3 Sync.
  • The %ERRORLEVEL% line checks the error level returned from S3 Sync. If the error level is not 0, the application skips the next block of code and goes to the :error section.
  • The @echo line writes a message to the console window.
  • The line with S3Delete.exe deletes the files from S3. This line will only be executed if error level from S3 Sync was 0 which means success.
  • The REM pause line would wait for the use to press a key to continue. The REM statement comments out the line and it will not be executed. Remove REM if you would like to test the script and want to be notified before continuing.
  • The exit 0 line exits the batch script with a success code. The remainder of the script will not be executed.
  • The :error line is a section marker of the script.
  • The @echo line writes a message to the console window.
  • The REM pause line can be uncommented during testing.
  • The exit 1 line exits the batch script with an error code.

To use the batch files above copy the code an place them in a text file using a tool such as Notepad. Change the parameters such as AWSAccessKeyId to your values. Save the file with a .bat extension. An example of the file name is “DownloadThenDelete.bat”. You can double click the bat file to execute it.

{ Comments on this entry are closed }

S3 Sync and Server-Side Encryption

by Anton on March 30, 2015

S3 Sync now supports server-side encryption using Amazon KMS-Managed Keys and Customer-Provided Keys. Server-side encryption is the process where Amazon encrypts files after you upload them. If you provide the correct credentials when retrieving a file, Amazon decrypts the file and returns it to you. Server-side encryption makes storing files on Amazon more secure.

Server-Side Encryption using AWS KMS-Managed Keys (SSE-KMS)

Amazon allows you to generate encryption keys within Amazon and specify which encryption key to use when uploading a file. This method is called AWS KMS-Managed Keys. You specify the ID of the encryption key to use when uploading a file. When downloading a file you must specify the same ID of the encryption key. You generate and manage encryption keys through the Identity and Access Management service which is accessible through the AWS web management console. If someone gets access to your S3 bucket they will not be able to retrieve a file unless they specify the correct ID of the encryption key. When managing encryption keys through the Identity and Access Management service you can delegate encryption key administrators and  encryption key users. These are the users that will be able to encrypt data, decrypt data, and manage encryption keys. When using SSE-KMS, AWS CloudTrail is available. CloudTrail keeps and audit history of who used an encryption key and when.

To use SSE-KMS in S3 Sync you will need to use the UploadHeaders parameter. The following is an example of this parameter.

-UploadHeaders “x-amz-server-side-encryption:aws:kms|x-amz-server-side-encryption-aws-kms-key-id:c112fc98-2e5d-4eab-aebc-0f565aa7e6fc”

The x-amz-server-side-encryption section tells S3 you will be using SSE-KMS. The x-amz-server-side-encryption-aws-kms-key-id section tells S3 which encryption key to use for the operation. You can get the key ID for an encryption key you generated in the AWS web management console under the Identity and Access Management service.

Note: The UploadHeaders parameter applies to upload, download, and bidirectional syncs in S3 Sync.

Server-Side Encryption using Customer-Provided Keys (SSE-C)

Amazon allows you to generate your own encryption key and send that key to S3 when uploading a file. Amazon will encrypt the file with your encryption key after it receives it. Amazon will then discard the key. When requesting the file you must include the same encryption key that was sent when uploading the file. Amazon will decrypt the file and return it to you. The benefit here is that Amazon does not store the encryption key so there is no way for someone with access to your account to retrieve a file. The drawback is you need to securely store your encryption keys locally. If you loose your keys you cannot retrieve your files.

The following is an example of the S3 Sync UploadHeaders parameter for using SSE-C.

-UploadHeaders “x-amz-server-side-encryption-customer-algorithm:AES256|x-amz-server-side-encryption-customer-key:Qi1sHpQnppeJAo5WyM8w/BGJXJmTO/LBg6dcjThY6nM=|x-amz-server-side-encryption-customer-key-MD5:IcpiSupe46fQ0fb5AGQ2RQ==”

The x-amz-server-side-encryption-customer-algorithm section says you will be using the AES 256 encryption method on S3. The x-amz-server-side-encryption-customer-key section is the encryption key. The x-amz-server-side-encryption-customer-key-MD5 section is the MD5 hash of the encryption key. Amazon will store the x-amz-server-side-encryption-customer-key-MD5 value with the file but not the x-amz-server-side-encryption-customer-key value.

The following is code to generate an encryption key and it’s MD5 hash in C#.

System.Security.Cryptography.AesManaged MyAesManaged = new System.Security.Cryptography.AesManaged();
MyAesManaged.GenerateKey();
String CustomerKey = Convert.ToBase64String(MyAesManaged.Key);

System.Security.Cryptography.MD5 MyMD5 = System.Security.Cryptography.MD5.Create();
Byte[] MD5Bytes = MyMD5.ComputeHash(MyAesManaged.Key);
String CustomerKeyMD5 = Convert.ToBase64String(MD5Bytes);

To read more about server-side encryption see http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html

{ Comments on this entry are closed }

How to Create a Scheduled Task to Automate S3 Sync

March 20, 2014

Microsoft Windows has a built in task scheduler. It allows you to automatically run a program at a certain time. Task scheduler is great for automating backups. You can configure a task to send files to Amazon S3 using SprightlySoft S3 Sync. The following article will take you step by step through the process of […]

Read the full article →

How to use Amazon Glacier in S3

February 24, 2014

Amazon Glacier is a service that allows you to archive your files on Amazon’s cloud infrastructure. It is designed for large files that are accessed infrequently such as video and picture backups. The main advantage of Glacier is the price. It costs $1 for 100 GB of storage space per month. This is 88% cheaper […]

Read the full article →

Restricting an IAM User to a Sub Folder in Amazon S3

September 13, 2013

Do you want to use multiple IAM users with a single S3 bucket but don’t want the users to access each other’s files? You can craft a S3 bucket policy to limit a user to a specific S3 sub folder. The following will show you how to create a bucket policy and use SprightlySoft S3 […]

Read the full article →

S3 Sync v4 Supports Restore from Glacier

January 24, 2013

S3 Sync version 4 has been released. The latest version supports downloading files that have been moved to the Glacier storage class. Amazon Glacier is a file archiving service. Storing a file in Glacier costs 1/10th the cost of storing it in S3. The drawback of Glacier is that if you want to retrieve a […]

Read the full article →

SprightlySoft Releases Glacier Sync

December 19, 2012

SprightlySoft is pleased to announce the release of Glacier Sync. Glacier Sync is a Windows application that allows you to synchronize a folder on your computer with a vault on Amazon Glacier. Amazon Glacier is an extremely inexpensive file archiving service. It is designed to store files for a long period of time where the […]

Read the full article →

Uploading Files to S3 with Plupload and ASP.NET

November 16, 2012

Plupload is a component that makes it easy to upload files through a web browser. When you add Plupload  to a web page, users can select multiple files to upload and users can see the progress while an upload is taking place. To read more about Plupload see http://www.plupload.com/. Plupload can be configured to upload files to […]

Read the full article →

S3 Sync Now Supports AWS GovCloud, GreenQloud, and Seeweb

May 11, 2012

SprightlySoft is pleased to announce S3 Sync now supports Amzon’s AWS GovCloud service. GovCloud provides compute and storage services for US government agencies. Data in this environment is segregated and complies with specific requirements for hosting government information.  To read more about GovCloud see http://aws.amazon.com/govcloud-us/. To use S3 Sync with GovCloud you must first have a […]

Read the full article →

SprightlySoft Releases S3 Delete

May 17, 2011

SprightlySoft is pleased to announce the release of S3 Delete. S3 Delete is a command line program that allows Windows users to delete files from Amazon S3. Amazon S3 is a web service that allows you to store and retrieve files. You can upload as many files you’d like and they can be made available […]

Read the full article →