Legacy Storage to Google Cloud Storage Migration

Google Cloud Storage is an object storage utility, with storage locations worldwide. With GCS, you will enjoy the peace of mind of low latency and 99.999999999% annual durability. You can find more details about the availability of the GCS storage classes here.

More details about Google Cloud Storage in general can be found here.

There are 3 tools we recommend to access your GCS data. The first two tools, gsutil and rclone, are CLI tools, while Cyberduck is a GUI tool.

• gsutil - Installation instructions can be found here.
• rclone - Installation instructions can be found here.
• Cyberduck - Installation instructions can be found here.

Some key features that differ between our legacy storage and GCS include the following:

• Our legacy storage does not have an option to restrict read access as it was built for internal communication from our CDN when pulling content. However, GCS allows users to create private buckets, where the public cannot read content without authorization. To access a private GCS bucket, you can use AWS V4 signing, which is discussed further in this article. More information about signing your GCS dataset can be found here.
• FTP/SFTP and RSYNC access are not available with GCS. This document contains other options to access GCS.

Your data will remain intact until January 31, 2021. StackPath is offering assistance with the migration of data from legacy platforms to GCS buckets for qualifying datasets. If desired, we will transfer your data automatically to GCS buckets.

You will need to adjust your integrations to leverage tools compatible with GCS, such as gsutil or rclone. These tools are discussed more in this article. You will also need to migrate any CDN origin configurations to use your GCS buckets. Assistance in executing CDN origin migrations will be provided and can be executed with no downtime.

You can log into StrikeTracker to view and manage your GCS buckets, service accounts, and associated service account credentials. If you choose to opt into migration assistance, buckets will be created for you and data populated from your existing storage. This information can be found in "How do I create a GCS bucket?" below.

Yes, you may choose a location that is best suited to your needs.

Currently we offer the following locations.

• europe-west-4
• us-east-4
• us-west-4

By default, GCS does pass through a Cache-Control header which will assign all objects a Cache-Control: public, max-age=3600 header. If you would like to change this header, there are a couple of options available, which are outlined below.

• Use StrikeTracker to set the Cache Expiration Method to Relative to Ingest. This will apply to all newly ingest objects.
• Set the Cache-Control metadata value on objects. This is done on an object-by-object basis, so it's recommended for objects that should have varying Cache-Control headers.

We will cover these options in "How do I configure the Cache-Control header?" below.

No, you will need to migrate your tooling to gsutil, rclone, Cyberduck, or a similar tool.

Yes. You can find more information about GCS and S3 interoperability here.

Your cost structure will not change. You will continue to be billed the same per GB-month and operation costs.

You should have received a survey asking if you would like to be migrated, or if you would like to migrate yourself. If you have opted to be migrated by Stackpath and meet our criteria for an automated migration, we will create your GCS bucket and service account, then we will migrate your data. While the migration is in progress, we ask that you upload any new data to both your legacy storage and GCS. Once the automated migration is complete, you will no longer need to upload any new data to your legacy storage, and we will work with you on switching your origin in StrikeTracker to the new GCS bucket.

If you wish to migrate yourself, or do not meet our criteria for having an automated migration, this article provides details and options for you to migrate your data. As always, our support is available 24/7 to assist with any questions or concerns you may have when migrating.

If the upload speed is known for the machine you will be using to initiate the migration from legacy storage to GCS, you can use the transfer time calculator here.

We will not be able to directly re-write it, but our Support team will be here to help with assistance.

Overall, you should see improved performance with GCS with faster origin pulls and more stable storage.

You can view your legacy storage content via FTP. If you need any assistance accessing your legacy storage, please reach out to our 24/7 support team.

Yes, you will be able to use the GCS API using HMAC keys. When using the API with HMAC keys, you will only be able to use the XML API, not the JSON API. Below is a link to Google's XML API documentation.

• https://cloud.google.com/storage/docs/xml-api/overview

Each service account create will have full access to the entirety of the GCS bucket. These service accounts will also be the only way to authenticate to GCS.

You will not be able to manage your GCS buckets contents within StrikeTracker. The only way to manage your GCS files will be with gsutil, rclone, Cyberduck, or similar tools.

GCS does not have users, but you can create a service account from StrikeTracker that will permit access to your GCS buckets.

To create a service account, you will need to go the menu and select "Object Storage".

From there, you will want to navigate to "Service Accounts" and click "Add Service Account".

Now that you have a service account created, you can click on that service account to either generate a key or HMAC key, depending on how you access your content. If you are using rclone, you will want to generate a key. If you use gsutil, you will need to generate a HMAC key. We will discuss authenticating these tools in more detail.

Creating a GCS bucket can be done within StrikeTracker. To create a new bucket you will need to click the menu and select "Object Storage".

From here you will want to select "Object Storage" from the left-hand menu (if you're not already there), and click "Add Bucket".

Here, you'll have the option to name your bucket and select the region you wish to use for that bucket.

Bucket name only supports (a-z, 0-9, -) and must start/end with a letter or number.

Once your bucket is created, you can click on it and view more details about the bucket, as well as modify it from private to public.

You will need to configure gsutil to access your bucket using your service account and your HMAC key. You can find installation instructions for gsutil here if you have not yet installed this tool.

Once gsutil is installed, you will want to run the following command and follow the instructions to grant access to your service account.

        	gsutil config -a
        

By default, gsutil will try to authenticate with OAuth2 credentials from the Cloud SDK, but this is not support by Stackpath. So you will need to run the following command to ensure the HMAC credentials are used to authenticate.

        	gcloud config set pass_credentials_to_gsutil false
        

You will need to configure rclone to access your bucket using the key file for your service account. You can find downloads and installation instructions for rclone here.

Below are instructions to configure a common GCS profile using rclone. 

• First, run rclone config
• Enter n to create a new config
• Use the defaults for most variables, except for the following:

Cyberduck provides a GUI interface to navigate, upload, and delete your data. When using Cyberduck, you will need to use HMAC keys to connect. Please see "How do I create a GCS user (service account)?" above for instructions to generate an HMAC key.

• Click "Open Connection"
• Select "Amazon S3" for the service
• Update the "Server" to the storage.googleapis.com
• The "Access Key ID" and "Secret Access Key" are the values provided when your HMAC key was generated in StrikeTracker.

Once logged in, you will see all of your GCS buckets. You can double click the bucket you wish to view or modify to enter into it.

The migration speed is dependent on the upload speeds of the machine initiating the transfer. This limitation is due to how rclone migrates this data, as it acts as a proxy between the two platforms with the transferring machine acting as the bridge between the two platforms.

• • sync - Sync the source to the destination, changing the destination only. Doesn't transfer unchanged files, testing by size and modification time or MD5SUM. Destination is updated to match source, including deleting files if necessary.
  • copy - Copy the source to the destination. Doesn't transfer unchanged files, testing by size and modification time or MD5SUM. Doesn't delete files from the destination.

It is always the contents of the directory that are synced, not the directory. When source:path is a directory, it's the contents of that source directory that are copied, not the directory name and contents. If dest:path doesn't exist, it is created and the source:path contents are copied there.

There are some limitations with rclone when migrating millions of files which could result in high memory consumption. You can view more information about limitations with rclone here.

Below are instructions to configure a common legacy storage proflie.

• First, run rclone config
• Enter n to create a new config
• Use the defaults for most variables, except for the following:

You will need to configure rclone to access your bucket using the key file for your service account. You can find downloads and installation instructions for rclone here.

Below are instructions to configure a common GCS profile using rclone. 

• First, run rclone config
• Enter n to create a new config
• Use the defaults for most variables, except for the following:

When performing a migration, you can either use the copy or sync commands. With both commands, we recommend using the -P flag to show the progress. The <BUCKET_ID> is the Bucket Name seen in StrikeTracker in the Bucket Details. See examples below:

• Copy

          rclone copy \
          <source_legacy_storage_name_in_rclone_config> \
          <destination_GCS_name_in_rclone_config>:<BUCKET_ID> -P
        

More details about this command can be found here.

• Sync

When performing a sync, we strongly recommend performing a dry run using the -n flag, first.

          rclone sync \
          <source_legacy_storage_name_in_rclone_config> \
          <destination_GCS_name_in_rclone_config>:<BUCKET_ID> -n
        

Once you've confirmed the dry run is good, you can perform the sync.

          rclone sync \
          <source_legacy_storage_name_in_rclone_config> \
          <destination_GCS_name_in_rclone_config>:<BUCKET_ID> -P
        

More details about this command can be found here.

By default, rclone will not copy or sync empty directories from legacy storage. To create the empty directories from your legacy storage in your GCS bucket, you will need to use the following flag during the copy or sync:

          --create-empty-src-dirs
        

Below are some rclone flags we feel will be helpful in your data transfer and future uploads. You can find more available flags here.

• Before your cutover, we recommend that over 50% of your dataset is migrated to your GCS container.
• For a graceful cutover, you will need to configure the origins for your site(s) so that GCS is the primary origin and legacy storage is the backup origin.
• With this configuration, all origin pull attempts will start by trying to request content from GCS. If the content does not yet exist on GCS, it will then fall back to legacy storage for the missing content. You should not see any decreased performance during origin pulls.

The primary origin is the first origin we will attempt to pull content from to be cached on the CDN. If we fail to retrieve the content from your primary origin (i.e. a 404 HTTP response), we will attempt to pull the content from the backup origin.

First, you will need to add your GCS bucket to your origins in StrikeTracker. This can be done by going to Origins→ Add Origin.

When adding your GCS bucket as an origin, you will want to use the External origin type and your endpoint URL as the hostname (i.e. mybucket.storage.googleapis.com). The endpoint URL can be found in the bucket details within StrikeTracker.

Then you can update your origin connections by going to Origin → Origin Selection → Origin Connections in your site editor, and select your GCS bucket as the primary origin and your legacy storage as your backup origin.

Additionally, if it's not already configured, you should use HTTPS as the origin pull protocol.

Please note that by default, new GCS buckets will be private and will require additional signing for successful origin pulls using HMAC keys from a service account. You will also need to configure the AWS Signed OriginPull V4 policy. This will sign URLs during origin pulls so the content can be pulled and cached on our CDN. This policy can be configured by going to your site editor and navigating to Origin → Uncategorized → AWS Signed OriginPull V4. Below is the configuration template you will need to use to allow access from the CDN to GCS to pull and cache content.

You will also need to add the following signed headers in this policy:

• host
• x-amz-content-sha256
• x-amz-date

You can find Google documentation on the V4 signing here.

You should continue to upload new content to both platforms until you perform the graceful cutover outlined above. Once the cutover is completed, you will no longer need to upload new content to your legacy storage.

You will be able to create service accounts and interact directly with the GCS API to upload data, modify permissions, etc. To create a service account, please see "How do I create a GCS bucket?" above.

Once a bucket is created, you can click on that bucket and modify the permissions so that the bucket is either private, or public under the "Visibility" policy.

Listing files in your GCS bucket can be done using gsutil, rclone, or by logging into the bucket via Cyberduck. In this section, we will cover listing files via gsutil and rclone.

            	gsutil ls gs://<BUCKET_ID>
            

            	gsutil ls gs://<BUCKET_ID>/<SUBDIRECTORY>/
            

            	gsutil ls gs://<BUCKET_ID>/example*/
            

            	gsutil ls gs://<BUCKET_ID>/*.txt
            

            	gsutil ls -r gs://<BUCKET_ID>
            

            	gsutil ls -r gs://<BUCKET_ID>/**
            

            	gsutil ls -r gs://<BUCKET_ID>/<SUBDIRECTORY>/**
            

            	gsutil ls -l gs://<BUCKET_ID>/<FILENAME>
            

            	gsutil ls -l gs://<BUCKET_ID>/*.txt
            

            	gsutil ls -L gs://<BUCKET_ID>/<SUBDIRECTORY>
            

            	rclone ls <GCS_name_in_rclone_config>:<BUCKET_ID>
            

            	rclone ls --max-depth 1 <GCS_name_in_rclone_config>:<BUCKET_ID>/<SUBDIRECTORY>
            

            	rclone lsl <GCS_name_in_rclone_config>:<BUCKET_ID>/<SUBDIRECTORY>
            

            	rclone lsd <GCS_name_in_rclone_config>:<BUCKET_ID>
            

            	rclone lsf -R <GCS_name_in_rclone_config>:<BUCKET_ID>
            

            	rclone lsjson <GCS_name_in_rclone_config>:<BUCKET_ID>/<SUBDIRECTORY>
            

The Cache-Control header can be customized using StrikeTracker or metadata. If your dataset should have the same Cache-Control values, you can use StrikeTracker to configure the header. However, if your dataset requires varying Cache-Control values for your files, you will need to utilize metadata on GCS to configure the values using gsutil.

         		gsutil setmeta -r -h "Cache-control:public, max-age=3600" \
         		gs://<BUCKET_ID>
          

         		gsutil setmeta -r -h "Cache-control:public, max-age=3600" \
        		gs://<BUCKET_ID>/object
      	  

         		gsutil setmeta -r -h "Cache-control:public, max-age=3600" \
         		gs://<BUCKET_ID>/*.html