Amazon Managed Streaming for Apache Kafka (Amazon MSK)
- for Apache Kafka (alternative for Amazon Kinesis Data Streams)
- Fully managed, with data stored in EBS for as long as needed (longer than 1 year)
- Or Amazon MSK serverless, which no capacity concerns
- Differences with Amazon Kinesis Data Streams
- Message size is not limited in 1MB(default)
- Topics with Partitions in Kafka = Shards in Data Streams
- MSK can only add Partition, but Data Streams can split and merge Shards
- PlainText or In-flight TLS encryption; Data Streams can only in-flight TLS
- Kinesis provides “at-least-once” message delivery -> duplicate records might occur
- MSK provides “exactly-once” delivery -> risk to lost message
| Kinesis Data Streams | Firehose | Managed Service for Apache Flink | |
|---|---|---|---|
| Usage | Capture stream log and event data, run real-time analytics, and build event-driven applications | Load data streams into AWS data stores | Analyze data streams with Managed Service for Apache Flink Studio and Apache Flink |
| Data sources | Mobile apps, application logs, web clickstream/social, IoT sensors, connected products, smart buildings | Connected devices such as consumer appliances, embedded sensors, TV set-top boxes, clickstream data, application logs | Analyze streaming data from Kinesis Data Streams, Amazon MSK, Amazon MQ, custom connectors |
| Stream ingestion | AWS SDKs, Kinesis Producer Library, AWS Mobile SDKs, Kinesis Agent, AWS IoT, Amazon CloudWatch Events, Amazon DynamoDB, AWS DMS | AWS SDKs, Kinesis Producer Library, Kinesis Data Streams, Kinesis Agent, AWS IoT, Amazon CloudWatch Events | Analyze streaming data from Kinesis Data Streams, Amazon MSK, Amazon MQ, custom connectors |
Amazon OpenSearch Service (previous “Amazon ElasticSearch”)
- Previous, “AWS ElasticSearch”
- Can search on “any” field, even partial match.
- Two cluster modes, “managed” and “serverless”.
- Not natively support SQL.
- OpenSearch Dashboards for visualisation.
AWS SES
- Simple E-mail Service for sending marketing e-mails (like a Marketo or ConstantContact)
Scalability
- an application / system can handle greater loads by adapting.
- Vertical, to increase instance size
- Horizontal, also called as “elasticity”, to increase number of instances
High Availability
- running your application / system in at least 2 data centers (== Availability Zones)
Server Name Indication (SNI)
- allow multiple SSL certificates onto one web server (to serve multiple websites), only works for ALB & NLB and CloudFront
API Rate Limits
- DescribeInstances API for EC2 has a limit of 100 calls per seconds
- GetObject on S3 has a limit of 5500 GET per second per prefix
- For Intermittent Errors: implement Exponential Backoff
- For Consistent Errors: request an API throttling limit increase
Service Quotas (Service Limits)
- Running On-Demand Standard Instances: 1152 vCPU
- You can request a service limit increase by opening a ticket
- You can request a service quota increase by using the Service Quotas API
Exponential Backoff
- If get ThrottlingException intermittently
- For on 5xx server errors and throttling, not on the 4xx client errors
Signing AWS API requests
- using Signature v4 (SigV4) to send credentials (access key & secret key)
- HTTP Header option (signature in Authorization header)
- Query String option, ex: S3 pre-signed URLs (signature in X-Amz-Signature)
AWS Fault Injection Simulator
- a managed service that is commonly used in chaos engineering, and not for application development. It enables you to perform fault injection experiments on your AWS workloads to improve the performance and resiliency of your applications
AWS Batch
- used to efficiently run hundreds of thousands of batch computing jobs in AWS; mostly used for ML model training, simulation, and analysis at any scale
AWS Simple Workflow Service (SWF)
- is for executing tasks. Helps developers build, run, and scale background jobs; but it does not provide serverless orchestration to multiple AWS resources
Amazon Pinpoint
- allows you to engage with your customers across multiple messaging channels.
- primarily used to send push notifications, emails, SMS text messages, and voice messages.
AWS CloudShell
- simply a command-line interface used for managing AWS resources from a terminal
AWS IoT Greengrass
- enable connected devices to run AWS Lambda functions, execute predictions based on machine learning models, keep device data in sync, and communicate with other devices securely even without an Internet connection.
Amazon DynamoDB point-in-time recovery (PITR)
- provides automatic continuous backups of your DynamoDB table data. Point-in-time recovery (PITR) backups are fully managed by DynamoDB and provide up to 35 days of recovery points at a per second granularity.
Disaster Recovery (DR)
- DR approaches
- Backup and restore = lowest cost, just create backups
- Pilot Light = small part of core services that is running and syncing data or documents
- Warm Standby = scaled down version of a fully functional environment that is actively running
- Multi-site = on-prem and in AWS in an active-active configuration
- For disaster recovery in a different region, create a AMI from your EC2 instance and copy it into a 2nd region.
AWS Global Accelerator
- increases availability and performance improve the performance of your network traffic by utilizing the AWS global infrastructure instead of the public Internet.
- can be expensive
- runs over AWS global network
- directs traffic to optimal endpoints across multiple regions
- By default, provides you with 2 static IP addresses that are anycast from the AWS edge network. You can migrate existing IPv4 (/24) IPs rather than creating new.
AWS Cost Management
- gain visibility into their cloud spending, identify cost-saving opportunities, and make informed decisions about resource allocation and optimization
- Steps of Cost Allocation
- STEP ONE: create user-defined tags with key-value pairs that reflect attributes such as project names or departments to ensure proper categorization of resources
- STEP TWO: apply these tags to the relevant resources to enable tracking
- STEP THREE: enable the cost allocation tags in the Billing console
- (AFTER) STEP FOUR: Configure tag-based cost and usage reports (AWS Cost Allocation Reports) for detailed analysis in Cost Explorer
- Step of Budgeting
- Set up automated alerts based on budget thresholds for expenses
- AWS Budgets is typically used for setting cost and usage limits and receiving alerts; more about monitoring and controlling costs rather than detailed tracking and reporting
Amazon Managed Service for Prometheus (AMP)
- a fully managed, Prometheus-compatible monitoring and alerting service that allows users to monitor containerized applications at scale on AWS. It simplifies the process of collecting, storing, and querying Prometheus metrics by handling the underlying infrastructure and scaling needs. AMP is compatible with the open-source Prometheus project and supports the Prometheus Query Language (PromQL).
- works with container clusters that run on Amazon Elastic Kubernetes Service and self-managed Kubernetes environments.
- AMP offers agentless observability, meaning users don’t need to manage any agents within their clusters
- AMP utilizes Amazon S3 for reliable and scalable long-term storage of metrics.
| Serverless Component | Max Timeout | Comments |
| API Gateway | 50 milliseconds – 29 seconds | Configurable |
| Lambda Function | 900 seconds (15 minutes) | Also limited to 1,000 concurrent executions. If not handled, can lead to throttling issues. |
| DynamoDB Streams | 40,000 write capacity units per table | |
| S3 | No timeout by default, can be configured to 5-10 seconds | Unlimited objects per bucket |
- AWS App Mesh = for application networking for microservices applications
- AWS Resource Access Manager = share a Transit Gateway connection (only?) with other AWS accounts
- AWS Server Migration Service (SMS) is for migrating virtual machines
AWS CodeStar quickly develop, build and deploy applications on AWS- AWS Import/Export send HDDs with data to AWS and they import the data into S3
- Amazon Neptune: interactive graphs of DBs
- Amazon AppStream: streaming service
- Amazon Elastic Transcoder: convert video and audio files into versions that play on phones, tablets and PCs
- CloudSearch: search engine for your site
- AWS LightSail: easy alternative to setting up a VPC. Product set includes virtual servers (instances), MySQL DBs, HA storage and load balance
- AWS IoT Core: connected devices interact securely with cloud applications