how to use google cloud storage.
*this is a guide that was created from lecture videos and is used to help you gain an understanding of how to use google cloud storage.
Understanding Google Cloud Storage
Google Cloud Platform- App Engine, Compute Engine, Cloud Storage, Cloud Datastore, Cloud SQL, BigQuery
Cloud Storage- Global regional hosting of assets and data, edge caching = reduced latency = faster access, guaranteed up-time of 99.5% backup and restore options, no cap on storage limits, enhanced security, OAuth 2.0 authentication, group-based access controls such as ACL’s
Cloud Storage is great for files.
Cloud SQL is best for data not contained in a discrete file- relational database based on MySQL that can be hosted in US, Europe or Asia with up to 100GB storage and 16GB of RAM.
Cloud SQL has automatic database replication to multiple locations and point in time backup recovery. Mysqldump, MySQL Wire Protocol, JBDC and database instances can be spun up as needed. You access the database instances through Google Cloud Console interface, Standard PHP/MySQL connections, MySQL Client through command line or the JSON API
Cloud Datastore is for non relational data based off NoSQL. It has automatic replication.Supports ACID (Atomicity, Consistency, Isolation and Durability) transactions for reliable processing. You can access Cloud Datastore with Google Cloud Console interface, gcd command line tool or JSON API
Google App Engine is a PaaS compared to Google Compute Engine which has the infrastructure as a service (IaaS) model.
App Engine can be used to build, deploy and maintain web and mobile applications and games
App Engine provides complete server side management and automatically scales to unlimited users. Auto scaling will allow instances as needed, databases shared on demand and bandwidth expanded when necessary
Google Cloud Storage is ideal for storing assets and data as a resource to Google App Engine. You can store, retrieve, display and delete assets with Cloud Storage. You can also read and write flat datasheets that will be kept in cloud storage.
App Engine has full connectivity to Cloud SQL and Cloud Datastore and they both can be queried. You can manage your data with the Cloud SQL relational database or with Cloud Datastore non relational database.
App Engine can access BigQuery wia SQL dialect and you can utilize the analysis prowess of BigQuery through the language specific API REST interfaces.
Google Compute Engine is an IaaS for large scale computing needs. It has the same infrastructure as Google Search, Gmail and Ads. Compute Engine provides virtual machines on demand that give fast computations, has efficient scaling and you can specify how many core instances (1, 2, 4, 8, 16) Each core has up to 3.75 GB memory per core.
Difference between App Engine vs Compute Engine
App Engine gives developers the ability to control Compute Engine cores, and provide a web front end for Google Compute Engine data processing applications.
Compute Engine offers direct and complete operating system management for your virtual machines
Signup for Google Cloud -> Sign in or Create a Google account -> cloud.google.com -> get started -> enter information needed and payment method -> Create project -> Project ID is a unique identifier that Google provides but you can change the name.
Enabled APIs under API & Auth -> API -> Enabled: BigQuery, Debuglet Controller, Cloud SQL, Cloud Storage, Cloud Storage JSON API -> you can enable any API as needed that are provided for Google Cloud. If you need more information on getting the foundations of how to use google cloud storage, visit our article laying the foundations of Google Cloud.
Google Cloud Storage Architecture
Unlimited file storage, unlimited file size, different pricing levels and are assigned on the bucket level. Google Cloud Storage uses Google’s infrastructure, with high security and globally placed datacenters and fast retrieval due to edge caching for minimal latency
Google Cloud Storage is used for storage of digital content, retrieval from any capable computer/device, secure sharing that supports access control lists and OAuth 2.0 Authentication.
You can perform data analysis of stored data with Google Cloud integration with BigQuery.
You can integrate App Engine for mobile gaming and app development.
You can also host static website content in Google Cloud Storage.
Google Cloud Storage Access from the Google Cloud Storage Manager web interface, HTTP standard methods such as get, put and post, the REST interface or gsutil, a command line interface.
Google Cloud Storage is project based and have buckets within projects, And within buckets, there are objects. Projects -> Buckets -> Objects
Google Cloud Storage buckets are the main container, they provide access level control and are project specific. You cannot share buckets from one project to another. And you cannot NEST buckets. You cannot have a bucket inside of a bucket.
Google Cloud Storage objects are the content that resides in a bucket. Objects are bucket specific and cannot be shared between buckets. An object is made up of the object and the objects metadata, which is a series of name/value pairs that describe the object. Objects are verified when uploaded and they must be readable to be acceptable Once read, the objects are available. Once deleted, they are gone.
Google Cloud Storage supports browser authenticated file transfer, pause and resume uploading and downloading, and automatic pause/resume when necessary. Google Cloud Storage can be programmatically user controlled.
Installing Google Cloud SDK on a Mac- cloud.google.com/sdk -> ensure you have Python on your machine -> follow installation instructions
Google Cloud Storage: Buckets
Ensure API for Google Cloud Storage is enabled -> Google Cloud Storage -> create bucket -> bucket name must be unique to Cloud Storage namespace and conform to DNS naming conventions (3 to 63 characters long) ->upload files -> select file you want to upload to the created object -> you can also upload folders
Manipulating objects in buckets
Cloud Storage: Storage Browser -> select bucket you want to utilize objects for -> edit metadata ->
-> save -> edit permissions will allow permissions for user account management -> to access object go to storage.cloud.google.com/bucket name/object name -> ‘shared publicly’ will allow you to publish through storage.googleapis.com/bucketname/objectname
To get a better understanding of Google Cloud Buckets,, go to the googlecloudplatform on github.
Google Cloud Storage: gsutil
Authenticate with Google Cloud SDK -> specify project with gsutil: gcloud config set project (project id) -> gsutil ls will list buckets in project -> gsutil ls -L will give you the details about how many buckets you have in a JSON format, including who can access it, the access is the ACL
To see what is inside of a bucket with gsutil: gsutil ls gs://name of bucket -> you can use gsutil ls -l to see object information
Gsutil command to create a bucket- gsutil mb gs://unique bucket name -> check in Cloud Storage to ensure bucket was created
Files are moved from local to remote folders with cp command -> gsutil cp name-of-file gs://destination-of-file gs://unique-bucket-name -> will copy and upload files to buckets
Rearrange objects from bucket to bucket -> mv gs://source-bucket source-object gs://destination bucket
cloud.google.com/storage/docs/gsutil for full list of commands and documentation
Build static website with Cloud Storage (not using server side language/not database driven)
Verify you are owner of website through google search console -> setup CNAME alias that points to the domain/subdomain to a Google API Service, you can do this under your current hosting provider in the DNS Zone file -> Add Record: CNAME and fill out information -> ensure you have created a new CNAME alias -> Cloud Storage: Storage Browser -> Add new bucket -> bucket name MUST be the same name of the subdomain/domain that you are going to use ex mind.mindmovementmachine.com -> Create bucket -> add website contents/files to your root -> open command line of root -> gsutil -m cp -r name-of-file–path gs://bucketname to move assets to Google Cloud Storage bucket -> ensure your files have downloaded into the appropriate Google Cloud Storage bucket -> gsutil -m acl set -r public-read gs://bucketname to make files publicly available -> specify which page is default (which should be homepage for example) command: gsutil web set -m homepage -e error-page gs://bucket-name to set the main homepage and the error page
Use gsutil Create new bucket copy files to bucket -> move files to gallery bucket -> delete empty bucket
Google Cloud SQL
Fully managed relational database service that uses MySQL Data automatically replicated across multiple datacenters.
MySQL is widely in used and you can import databases via mysqldump or export to mysqldump format
You can create multiple database instances with 100GB per instance and 16GB of RAM per instance
Cloud SQL Access can be done with Java, Python, the command line tool or the SQL prompt in the Google Cloud Console
Cloud SQL only supports MySQL 5.5 or higher, the instance size is limited to 500GB.
MySQL Statements not supported to Cloud SQL- Load Data in file, Select into outfile/dumpfile, install/uninstall plugin, create function, load file
Cloud Storage for import/export and is accessible from App Engine and Compute Engine
Managing Google Cloud SQL Instances
Cloud SQL -> Create an instance -> fill out information as needed -> show advanced options for more option such as billing, backup, MySQL flags and app engine -> click on newly created database instance
Importing and exporting data in Google Cloud SQL
Everything goes through Cloud Storage.
Import- Cloud Storage: storage browser -> add bucket -> upload files -> select .sql file, ensure it is supported to import into Cloud SQL -> click import -> add cloud storage path with gs:// bucket-name imported-sql-file -> go to operations to see success of importing
Export- Google Cloud SQL -> Export -> enter gs://bucket-name sql-database-exporting
Working with Google Cloud SQL Prompt- Enable Google Cloud SQL API -> access Google Cloud SQL with listed instances -> click on database instance -> Access SQL Prompt -> choose database to query
Scheduling backups on Google Cloud SQL- Google Cloud SQL -> database instance -> edit Google Cloud instance -> enable backups
Asynchronous could possibly result in data loss
Create database instance -> add sql file to new bucket -> query data to see first 200 rows
If you want to work with Google Cloud SQL and WordPress, go to Geekflare.
Google Cloud Storage: Datastore
For non relational data, schema less – no predefined columns, supports transactions, supports robust queries, fully managed.
Based on App Engine Replication Datastore (HRD). Cloud Datastore offers exceptional performance. There is over 1 petabyte for storage and has 4.5 trillion transactions per month and a 99.5 uptime.
Cloud Datastore Access by development environments: Java, Python, Node.js, SDK includes development development server and datastore simulator, and the command line tool gcd
Entities are the primary Datastore object and they are similar to a relational database table. An entity can have one or more properties. A property is like a relational database column. A property can have one or more values (values are similar to relational database rows). You cannot have multiple values with a relational database.
Properties can be different types such as integer, string etc. Each entity is a particular kind, and a kind is a category. Kinds are used for queries for sorting and filtering. Entities must have a unique ID, either a string or number. Entities can be hierarchical (referred to as ancestor path)
An app gets results from Cloud Datastore through a series of queries. Queries can include the entity kind, a filter to search entity property values, ID’s or ancestors, and one or more sort orders for sequencing results.
Indexes help queries be more efficient and quicker. Each index is a table of entities and it is ordered by properties. It can be ordered by ancestors. When entities are modified, the index is automatically updated.
Defining data in Google Cloud Datastore
Google Cloud Datastore -> create entity -> create kind (kind like a table) -> create properties (like a database column) -> you can easily filter on properties that are indexed
Setting up for local datastore
Enable you have Java Version 7 or 8 -> download gcd tools from cloud.google.com/datastore/docs/downloads -> place gcd tool in root folder -> go to command line and unzip gcd tools
GCD command line
Command line -> ./gcd.sh create dataset-id-or-project-id -> ./gcd.sh start project-id
-> *indexes can be created automatically based on your data or programmatically via XML -> place XML file into appropriate folder -> ./gcd.sh updateindexes project-id
Need more tips for Google Cloud Datastore? Visit Hackernoon.
Google Cloud Storage: BigQuery
Analyze massive datasets interactively, data storage scales to hundreds of TB, uses SQL like queries.
BigQuery can be accessed with the web interface, command line interface: bq or programmatically via a REST or an Excel connector
BigQuery is a fast data analyzer that is based on OLAP principles. It is CSV data based and reliant on small number of non related tables
BigQuery is not a relational database. It is not a database because you can’t insert, update or delete records, there is not support for indexes and BigQuery supports joins only when one side is smaller than the other.
Ensure you enable the BigQuery API and enable billing to work with BigQuery
To learn more in depth regarding Google Cloud BigQuery a go to medium.
Loading data into BigQuery
BigQuery -> Create new dataset -> create and import: fill out destination, source data, schema -> check details of schema
Querying data in BigQuery
BigQuery -> select table -> compose query OR query table -> New Query
Exporting data with BigQuery
BigQuery -> run query -> you can download as csv or save as table
Python runtime environment
Download Python if not installed -> install Google API client library for Python -> download your notebook or IDE such as PyCharm. Go to JetBrains to download PyCharm
Authorizing requests to Cloud Storage
Setup consent screen- credentials ->create new Client ID for OAuth -> Download JSON file of file -> paste into google cloud python folder
To get started with Google Cloud Storage visit Google or reach out in the comments.