CSEG Cloud APIs for Amazon S3 and IBM Cloud

WebService Interface Detail – API

CSEG for Amazon Simple Storage Service (Amazon S3) and IBM Cloud supports WebServices to store, retrieve, delete and update files on cloud storage; and AES and RSA non-versioned key generation.

The cloud service provider allows its users to protect their data using either server-side encryption or client-side encryption. Using Amazon S3/IBM Cloud client-side encryption, the data is encrypted before it is sent to the cloud being used.

In server-side mode, the cryptographic operations are performed on the cloud service provider’s server. Only Amazon S3 is supported for the server-side encryption. To perform this operation the user provides the key fetched from Key Manager. This mode supports only AES algorithm and AES keys of size 256. The operations performed through this mode are fast, as all the information is on Amazon S3 server and the operation is also performed on the same. This may involve some security issues as all the data is sent over network and is in unencrypted format.

In client side mode, the cryptographic operations are performed on the server where CSEG resides. In this mode the Amazon S3/IBM Cloud client generates a random key of type AES. Amazon S3 or IBM Cloud client then performs the crypto operations on the given data using the randomly generated key. Once the operation is performed the given NAE key of type AES or RSA (based on the transformation tag of the request) is fetched from Key Manager server and is used to wrap the keys generated by Amazon S3 or IBM Cloud client. After this, the encrypted data and the wrapped key are sent to the Amazon S3 or IBM Cloud server for storage. The AES or RSA keys used for wrapping can be rotated.

The AWS SDK for Java uses envelope encryption. In envelope encryption, the encryption key is provided to the Amazon S3 encryption client and the client handles the rest of the process.

Important Notes

1. Ensure the temp folder exists on the webserver else, the download of file will fail.

2. The certAlias (client certificate alias for making SSL connections) and certPassword (password for the provided certificate alias) parameters are optional for all APIs.

3. Global session is not supported as it requires authentication using valid username and password.

Session Caching

CSEG supports caching of NAE session-pool and AWS/IBM Cloud session-pool for a user. In session-pool, a session is cached for a user so that for additional requests no new session is created for the particular user till the time the user-session is cached. User has to maintain the parameters for both NAE session-pool and AWS/IBM Cloud session-pool.

Maintaining NAE Session

To maintain NAE session for a user, session-pool is created using the following parameters. The parameters are maintained in dispatcher-servlet.xml file in the \%CATALINA_HOME%\webapps\safenetcloud\WEB-INF directory. User can modify the default values, as required.

• Size of cache: The number of different user sessions that can be cached. Default value is 2000. Use the following snippet to change the default value.

  <!-- size of cache -->
  <constructor-arg type = "int" value = "2000"/>
  

• Cache expiry time: The time after which the session expires from the last use of the particular cached session. Default value is 3600000 millisecond. Use the following snippet to change the default value.

      <!-- cache expiry time in milli sec -->
      <constructor-arg type = "int" value = "3600000"/>
      <constructor-arg ref="closeSessionListener"/>
  

• Salt: The string used along with the message digest algorithm parameter to create the hash of the key to store the user session in the cache. Default string used is ThisIsIt@123. Use the following snippet to change the default value.

      <!-- salt -->
      <value> ThisIsIt@123</value>
  

• Message digest algorithm: The algorithm used to calculate the hash of the key used in creating the user session. Default algorithm used is SHA-256. Possible values can be SHA-1, SHA-256, and SHA-512. Use the following snippet to change the default value.

      <!-- message digest algorithm -->
      <value>SHA-256</value>
  

Maintaining AWS Session

To maintain AWS session for a user, session-pool is created using the following parameters. The parameters are maintained in dispatcher-servlet.xml file in the \%CATALINA_HOME%\webapps\safenetcloud\WEB-INF directory. User can modify the default values, as required.

• Size of cache: The number of different user sessions that can be cached. Default value is 2000. Use the following snippet to change the default value.

  <!-- size of cache -->
  <constructor-arg type = "int" value = "2000"/>
  

• Cache expiry time: The time after which the session expires from the last use of the particular cached session. Default value is 3600000 millisecond. Use the following snippet to change the default value.

      <!-- cache expiry time in milli sec -->
      <constructor-arg type = "int" value = "3600000"/>
      <constructor-arg ref=" closeAWSSessionListener"/>
  

  In addition, user can configure the client configuration settings to modify the AWS session properties. The default values of three properties are modified and are maintained in dispatcher-servlet.xml file in the \%CATALINA_HOME%\webapps\safenetcloud\WEB-INF directory. You may configure other properties as required. Refer the Amazon S3 Java documentation for available properties. The properties are:

  • ConnectionMaxIdleMillis: The maximum time in millisecond for which the connection in the AWS session can remain idle. Use the following snippet to change the default value.

        <!-- connection idle time in msec -->
        <property name="ConnectionMaxIdleMillis" value="3600000"/>
    

  • ConnectionTTL: The maximum time in millisecond for which the connection remains alive in the AWS session. Use the following snippet to change the default value.

        <!-- connection time to live in msec -->
        <property name="ConnectionTTL" value="3600000"/>
    

  • maxConnections: The maximum number of threads that can make simultaneous request in the AWS session. Use the following snippet to change the default value.

        <!-- max thread simultaneously request -->
        <property name="maxConnections" value="100"/>
    

  The cache expiry time of both NAE Session and AWS/IBM Cloud Session should be greater than the time a file can take for any operation.

Form Variables

The following list provides the various parameters used in the operations (Upload, Download, Update, and Delete):

• accessKey – AWS Identity and Access Management (IAM) access Key ID used to access the Amazon services/resources. Or the IBM Cloud APIn key ID used to access the IBM Cloud services/resources.

• secretKey – AWS IAM secret access key used to access the Amazon services/resources. The access key ID and access secret key are created by the root user of IAM. Or the IBM Cloud API key used to access the IBM Cloud services/resources.

  The root user creates the access key ID and access secret key to grant access to other users to buckets. For more information on IAM access keys, refer to http://docs.aws.amazon.com/general/latest/gr/managing-aws-access-keys.html. Refer to the IBM Cloud website for the IBM Cloud API key details.

• bucketName – bucket name (created in Amazon S3 or IBM Cloud) where the file is to be uploaded/updated or from which the file is to be deleted.

• region/endpoint – name of the region (from approved list) where the bucket is to be stored in case of Amazon S3 or a string of the endpoint where the bucket is to be stored in case of IBM Cloud.

• ksUserName – Key Manager user name.

• ksPassword – Key Manager password.

• keyName – Key used for encryption/decryption.

• name – name of the file to be uploaded/downloaded to/from the bucket.

• fileName – name of the file to be deleted from the bucket.

• file – directory path of the file including the file name to be uploaded.

Upload File

This service uploads the file onto the AWS server or the IBM Cloud server and uses http POST method for it.

URL: <http/https>://<ip address>/safenetcloud/storage/upload

Input Parameters

• accessKey – Must

• secretKey – Must

• bucketName –Must

• region/endpoint – Must (use region in case of Amazon S3 or endpoint in case of IBM).

• ksUserName – Must

• ksPassword – Must

• keyName – Must (for server side it should be AES of key size 256, for client side it should be AES or RSA of any size). The server side encryption is supported only with Amazon S3.

• name – Must (File will be uploaded with this name)

• file – Must (file path)

• isClientSide - Optional, true for client side else false. Default is true.

• transformation - Valid only when isClientSide is true. Value must be either AES or RSA. Default is AES. For Amazon S3, if AES is passed in this parameter, AESWrap is used as key wrapping algorithm and if RSA is passed then RSA/ECB/OAEPWithSHA-256AndMGF1Padding is used as key wrapping algorithm.

• canKeyRotate – Optional, used to enable rotation of wrapping key for client-side encryption. Value must be either true or false. Default is false. The already uploaded encrypted files without this parameter needs to be downloaded and then uploaded again with canKeyRotate option as true if key rotation is required for them.

For client-side encryption with Amazon S3, file size of maximum 64GB can be uploaded.

For client-side encryption with Amazon S3, AES/GCM/NoPadding is used as the content encryption algorithm, previously it was AES/CBC. So any new file uploaded will have the AES/GCM/NoPadding as the content encryption algorithm. For files, uploaded previously using CSEG 8.10 2 or lower version will continue to have AES/CBC as content encryption algorithm, even if rotateWrapKey is performed. To have the relatively more secure AES/GCM/NoPadding as content encryption algorithm, user has to download the file (uploaded using CSEG 8.10.2 or lower version) and upload it again with the latest CSEG release. The method to upload a file to Amazon S3 remains the same and this detail is provided for information purpose only.

The file to be uploaded should not end with an extension .instruction. Any file uploaded to Amazon S3/IBM Cloud with the canKeyRotate parameter as true generates another file with same name and extension .instruction. Suppose a file abc is uploaded, then additional file abc.instruction will be created in the Amazon S3 or IBM Cloud.User is required to make a note of the files that have been uploaded with canKeyRotate parameter as true.

Response

Successful file upload returns the message “File uploaded successfully” else error response is returned.

Notes

• Upload request can’t be given in XML/JSON or any payload format. It should be uploaded using relevant parameters as well as using "Multipart_Form_Data" content type.

• For any file greater than 100MB multipart upload is used. Each part upload is equal to one POST request.

• To upload the file make sure that form variables are provided as per list in Form Variables section.

• Temporary file is uploaded at temp folder of webserver. It is removed once upload is complete (either successful or fail due to any reason).

• Version key is not supported.

• If, a user adds a metadata ("x-amz-meta-x-amz-meta-clientsideencryption") for client side encryption and an AES256 for customer SSE algorithm for server side encryption, then overwriting of file can be avoided while calling FileUpdate Service API.

The figure below explains encrypting a file and uploading it to AWS or IBM server.

With CSEG, upload file process on client mode for Amazon S3/IBM is as follows:

1. User provides the key name, data, and AWS/IBM login credentials to the core solution.

2. Core solution connects to Key Manager, locates the key name on Key Manager and fetches the key.

3. Core solution initializes the Amazon S3/IBM Cloud encryption client object with the NAEKey.

4. Sends the data and the key to the Amazon S3/IBM Cloud encryption client.

5. Amazon S3/IBM Cloud encryption client generates a one-time-use symmetric key. This key is called the envelope symmetric key.

6. Amazon S3/IBM Cloud encryption client encrypts the data using the envelope symmetric key.

7. The cloud encryption client wraps the envelope symmetric key using the NAEKey.

If the private encryption key is lost, the encrypted data can’t be decrypted.

1. The cloud encryption client uploads the wrapped envelope key and the encrypted data to the cloud (Amazon or IBM) under a specified bucket.

For server mode (only for Amazon S3), the upload file process, steps 1 to 4 remain same as client mode. Steps 5 and 6 are as follows:

1. User provides the key name, data, and AWS/IBM login credentials to the core solution.

2. Core solution connects to Key Manager, locates the key name on Key Manager and fetches the key.

3. Core solution initializes the Amazon S3/IBM Cloud encryption client object with the NAEKey.

4. Sends the data and the key to the Amazon S3/IBM Cloud encryption client.

5. Amazon S3 encryption client sends the data, key to Amazon S3 server for performing encryption and storage.

6. Amazon S3 encryption server performs encryption and store in specified bucket.

Rotate Wrap Key

This service rotates the key (AES/RSA), which was used to wrap the encryption key for the client-side encryption, and uses PUT method for it.

URL: :///safenetcloud/storage/rotateWrapKey

Input Parameters

accessKey – Must

• secretKey – Must

• bucketName –Must

• region/endpoint – Must (use region in case of Amazon S3 or endpoint in case of IBM)

• ksUserName – Must

• ksPassword – Must

• keyName – Must. AES or RSA key used for wrapping during the file upload process.

• fileName – Must (File for which key rotation is to be performed).

• newKeyName – Must. Name of the new key (AES or RSA) to be used for wrapping.

• oldVersion – No value required for first time key rotation. For subsequent key rotation, it is must. The XML example provided here shows the key rotation with no value in the oldVersion parameter.

• newVersion – Must. User needs to make a note of it. This newVersion value is to be used in the oldVersion parameter when the key is further rotated. Also, this newVersion parameter value is required for all download request for the file.

• certAlias – Optional

• certPassword – Optional

The newVersion should be noted as it is required for all download requests for the file from Amazon S3/IBM Cloud. The newVersion value is to be used in the oldVersion parameter during next key rotation.

XML

<WrapKeyRotationRequest>
<accessKey>abcdefghijtfngn</accessKey>
<accessKey>AKIAJZ6R6X54A5RC56DA</accessKey>
<secretKey>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</secretKey>
<bucketName>demosafenettest</bucketName>
<region>US_EAST_1</region>
<ksUserName>testuser</ksUserName>
<ksPassword>asdf1234</ksPassword>
<keyName>aws_rsa_2048_nv1</keyName>
<fileName>rfile1</fileName>
<newKeyName>aws_aes_256_nv</newKeyName>
<oldVersion></oldVersion>
<newVersion>2</newVersion>
</WrapKeyRotationRequest>

Response

Successful key rotation returns the message " wrapped with new key", else error response is returned.

Download File

This service fetches the file from the AWS or IBM server and use http GET method for it. This returns the stream; it is users' responsibility to fetch the file content from the stream and store at required location. Users can also use html client or any other GUI client.

URL: :///safenetcloud/storage/download

Input Parameters

• accessKey – Must

• secretKey – Must

• region/endpoint – Must (use region in case of Amazon S3 or endpoint in case of IBM).

• bucketName – Must

• ksUserName – Must

• ksPassword – Must

• keyName – Must (should be same as used for upload)

• name – Must (name of the file used to fetch from the Amazon S3/IBM Cloud)

• isClientSide - optional, true for client side else false. Default is true

• transformation - valid only when isClientSide is true. Value must be either AES or RSA. Default is AES.

• version - valid only for those files on which wrap key rotation was performed. The version provided during the last wrap key rotation request is to be used here. destination – the absolute path where the file will be downloaded on the client’s system. For example, C:\test\testfile.txt.

Response

Successful file download returns the file content; else, an error is returned.

To download the file make sure the form variables are provided as per the list, in Form Variables section.

The figure below explains decrypting a file and downloading it from AWS/IBM server.

With CSEG, the download file process on client side for Amazon S3/IBM Cloud is as follows:

1. User provides the key name, data, and Amazon S3/IBM Cloud login credentials to the core solution.

2. Core solution connects to Key Manager, locates the key name on Key Manager and fetches the key.

3. Core solution initializes the Amazon S3/IBM Cloud encryption client object with the NAEKey.

4. The cloud (Amazon or IBM) encryption client requests the data from Amazon S3/IBM Cloud.

5. The cloud encryption client extracts the metadata and decrypts the encrypted envelope key by using the master encryption key provided by the user.

6. The cloud encryption client decrypts the encrypted data using the envelope key.

7. The cloud encryption client sends the decrypted data to the user.

For Amazon S3 server mode, the decryption and download file process, steps 1 to 4 remain same as client mode. Steps 5 and 6 are as follows:

User provides the key name, data, and Amazon S3/IBM Cloud login credentials to the core solution.

1. Core solution connects to Key Manager, locates the key name on Key Manager and fetches the key.

2. Core solution initializes the Amazon S3/IBM Cloud encryption client object with the NAEKey.

3. The cloud (Amazon or IBM) encryption client requests the data from Amazon S3/IBM Cloud.

4. Amazon S3 encryption client sends request to the server for downloading the specific file in a given bucket.

5. Amazon S3 encryption client stores the downloaded file in temp folder of WebServer and then sends the file to the user.

Delete File

This service deletes the file from the AWS or IBM server and use http DELETE method for it. To access this service user needs to send an XML or a JSON request.

URL: :///safenetcloud/storage/deleteFile

Input Parameters

• accessKey – Must

• secretKey – Must

• region/endpoint – Must (use region in case of Amazon S3 or endpoint in case of IBM).

• bucketName – Must

• fileName – Must

XML

<FileDeleteRequest>
<accessKey>abcdefghijtfngn</accessKey>
<secretKey> skfntrnsjfvfdvdsd</secretKey>
<bucketName>abc</bucketName>
<region>AP_SOUTHEAST_2</region>
<fileName>xyz</fileName>
</FileDeleteRequest>

JSON

{
"FileDeleteRequest": {
"accessKey": "abcdefghijtfngn",
"secretKey": "skfntrnsjfvfdvdsd",
"bucketName": "abc",
"region": "AP_SOUTHEAST_2",
"fileName": "xyz"
  }
}

Response

AWS or IBM returns successful message even if file is not found. This means, if a user requests to delete a file then in every case the result is displayed as “file deleted successfully” except for the cases where an exception may occur.

With CSEG, delete file process for Amazon S3 or IBM Cloud is as follows:

1. User provides the key name, data, Amazon S3 or IBM Cloud login credentials, region/endpoint, bucketName and file name to the core solution.

2. CSEG requests the delete data request from Amazon S3 or IBM Cloud.

FileUpdate Service

This service encrypts the unencrypted files available in the bucket. To access this service user needs to send an XML or a JSON request. This is a PUT request.

URL: :///safenetcloud/storage/encryptFiles

Input Parameters

• accessKey – Must

• secretKey – Must

• region/endpoint – Must (region in case of Amazon S3 or endpoint in case of IBM).

• bucketName – Must

• ksUserName – Must

• ksPassword – Must

• keyName – Must (for server side should be AES of key size 256, for client side should be AES or RSA of any size). The server side encryption is supported only with Amazon S3.

• canKeyRotate – Optional. Value must be either true or false. Default value is false. If this parameter is true then the AES/RSA wrap key used in client-side operation can be rotated.

• isClientSide - optional, true for client side else false. Default is true.

• transformation - valid only when isClientSide is true. Value must be either AES or RSA. Default is AES.

XML

<FileUpdateRequest>
<accessKey>abcdrhfksdfgsdfg123</accessKey>
<secretKey> fsdfsgfesfgvdfv23234r2vxvxv365cxxcfvv</secretKey>
<bucketName>abc-bucket</bucketName>
<region>AP_SOUTHEAST_2</region>
<ksUserName>abc</ksUserName>
<ksPassword>safnt@123</ksPassword>
<keyName>AES256Key</keyName>
<isClientSide>true</isClientSide>
<transformation>AES</transformation>
</FileUpdateRequest>

JSON

{
"FileUpdateRequest": {
"accessKey": "abcdrhfksdfgsdfg123",
"secretKey": "fsdfsgfesfgvdfv23234r2vxvxv365cxxcfvv",
"bucketName": "abc-bucket",
"region": "AP_SOUTHEAST_2",
"ksUserName": "abc",
"ksPassword": "safnt@123",
"keyName": "AES256Key",
"isClientSide": "true",
"transformation": "AES"
}
}

Response

Successful operation will return the file list encrypted during this operation. Else error will occur. In case of no unencrypted file available in the bucket, blank response occurs.

• For a particular region and a particular bucket, we cannot send multiple requests at same time, with different keys.

• For client side encryption, files are downloaded to the webserver for encryption and uploaded back after encryption.

• If the file on server is already encrypted (using either server or client side) then no encryption is performed.

• If any file present on the server has been uploaded without using CSEG API then it gets corrupted or is replaced with new one. To avoid this, user must first download the file and then upload again using CSEG API.

• If user adds metadata ("x-amz-meta-x-amz-meta-clientsideencryption") for client and AES256 for customer SSE algorithm for server then overwriting of this file can be avoided.

Key Generation

This service creates AES and RSA key on Key Manager server. To access this service, user needs to send an XML or a JSON request. This is a POST request.

URL: :///safenetcloud/storage/keyGenerate

Input Parameters

• ksUserName – Must

• ksPassword – Must

• keyName – Must

• algorithm – Must, valid values: RSA-512, RSA-1024, RSA-2048, RSA-3072, RSA-4096, AES-128, AES-192, AES-256.

XML

<KeyGenerationRequest>
<ksUserName>abc</ksUserName>
<ksPassword>safnt@123</ksPassword>
<keyName>123425</keyName>
<isDeletable>false</isDeletable>
<isExportable>false</isExportable>
<algorithm>AES-128</algorithm>
</KeyGenerationRequest>

JSON

{
"KeyGenerationRequest": {
"ksUserName": "abc",
"ksPassword": "safnt@123",
"keyName": "123425",
"isDeletable": "false",
"isExportable": "false",
"algorithm": "AES-128"
}
}

Response

Successful operation will return successful key creation message else error will come. With CSEG, key generation process for Amazon S3 or IBM Cloud is as follows:

1. User provides the key name, Key Manager Credentials, key algorithms and size.

2. CSEG connects to the Key Manager and generates the key.

List Files

This service lists all the files available in the given bucket in request format. This is a GET request.

URL: :///safenetcloud/storage/bucketInfo?bucketName=?&region=?

Input Parameters

• accessKey – Must, Header Parameter

• secretKey – Must, Header Parameter

• region/endpoint – Must (region in case of Amazon S3 or endpoint in case of IBM). Query Parameter

• bucketName – Must, Query Parameter

Response

XML

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<FilesInBucket>
<file>test123.docx</file>
<file>test345.docx</file>
<file>test.zip</file>
</FilesInBucket>

JSON

{
"FilesInBucket": {
"file": [
"test123.docx",
"test345.docx",
"test.zip"
]
}
}

With CSEG, list file process for Amazon S3 or IBM Cloud is as follows:

1. User provides the key name, data, Amazon S3 login credentials, region/endpoint, bucketName to the core solution.

2. CSEG requests the data listing request corresponding to the bucket from Amazon S3 or IBM Cloud.