Introduction to scaling Large Model training and inference using DeepSpeed
DeepSpeed library optimizes training and inference for foundational models. Learn more about its features.
Amazon AWS
Amazon S3 lifecycle policies
Why S3 lifecycle policies S3 lifecycle policies allow you to do two things: Reduce cost by deleting data that is not longer required. Implement your security policies by : Retaining data for the required duration and reducing cost by moving it to low cost storage. Deleting data that you are not allowed to retain for … Read more
Host a static website on Amazon S3
In this blog I will show you how to deploy a static website on Amazon S3, along with an HTTPS endpoint and content delivered by CloudFront CDN. Here’s a video of the steps Six steps to deploying a static website on Amazon S3 Step 1. Create a Bucket As the first step I create a … Read more
Amazon S3 encryption
Encryption allows you to store objects in such a way that only an entity that has the encryption key, or access to the encryption key can access that object. S3 Encryption types There are five ways in which you can encrypt objects that you write to S3. S3 Server Side Encryption using Amazon S3 Managed … Read more
Amazon S3 access control and permissions
There are three ways to control access to s3 bucket and its objects Using bucket policies. Using bucket Access Control Lists (ACL) Using User policies ACL is used only in cases where Objects are not owned by the bucket owner. I.e. objects are uploaded by another account and the bucket owner does not own these … Read more
Amazon S3 pricing
In this blog we will talk about S3 pricing. Important points to consider are: Pricing is based on region. In this blog we will look at pricing for Sydney region. Pricing varies based on storage used. The more storage you use, the lesser is the price per GB. Here’s the storage cost for 1000G in … Read more
Create/drop Athena table using python in Sagemaker
Amazon Athena is a serverless query engine that allows you to query data in Amazon S3 using standard SQL. You can also use the query result to create another table. This table can then be used as a source of further analytics or reporting. For example : You can use Athena to perform feature engineering … Read more
CloudFormation for SageMaker instance
Amazon SageMaker helps data scientists and Machine Learning developers build, train and deploy machine learning models. It includes Jupyter notebook to build and train model as well SageMaker API to train and deploy model with a few lines of code. Amazon CloudFormation helps in provisioning AWS resources using code. It automates provisioning and configuring resources … Read more
Amazon DynamoDB
DynamoDB Basics DynamoDB Overview DynamoDb is an AWS managed NoSQL database. It offers the following: Encryption at rest Scale throughput without downtime On demand backup and point in time recovery Data stored in SSD and replicated accross multiple AZs. DynamoDB Components table –> items –> attributes DynamoDB tables table is a collection of data items. … Read more
AWS VPC Tutorial – Part II subnets
In this AWS VPC tutorial, we will look at how to create the VPC, public and private subnets, route table, and an internet gateway. AWS VPC Tutorial – Recap In the previous tutorial we saw an introduction to Amazon VPC and also looked at the key concepts in VPC. Here’s the diagram of what we … Read more