10. Miscellanies

Amazon OpenSearch Service (previous “Amazon ElasticSearch”)

  • Previous, “AWS ElasticSearch”
  • Can search on “any” field, even partial match.
  • Two cluster modes, “managed” and “serverless”.
  • Not natively support SQL.
  • OpenSearch Dashboards for visualisation.

Amazon Athena

  • Serverless
  • Analysis on S3
  • Using SQL, with Presto engine
  • Support JSON, CSV, ORC, Avro and Parquet
  • with Amazon QuickSight for reporting/dashboards
  • Performance Improve
    • Using columnar data, for less scan (cost-saving)
    • Data type: ORC or Apache Parquet
    • Using Glue for data conversion
    • Compression for smaller retrievals
    • Partition datasets (using year/month/date in S3 path/folders)
    • Large files (> 128MB) to lower overhead
  • Federated Query
    • any data sources
    • using AWS Data Source Connectors(Lambda func)
    • results stored in Amazon S3

Amazon MSK

  • for Apache Kafka (alternative for Amazon Kinesis Data Streams)
  • Fully managed, with data stored in EBS for as long as needed (longer than 1 year)
  • Or Amazon MSK serverless, which no capacity concerns
  • Differences with Amazon Kinesis Data Streams
    • Message size is not limited in 1MB(default)
    • Topics with Partitions in Kafka = Shards in Data Streams
    • MSK can only add Partition, but Data Streams can split and merge Shards
    • PlainText or In-flight TLS encryption; Data Streams can only in-flight TLS

AWS Step Functions:

  • Workflow in JSON
  • Start with SDK, API Gateway call, or Event Bridge
  • Task: Invoke 1 AWS service or Run 1 Activity
  • States: Choice, Fail-or-Succeed, Pass, Wait, Map, Parallel
  • Error Handling should be in Step Functions, not in Task; using Retry and Catch, running from top to bottom but not sequentially (“OR”)
  • Wait for Task token: append .waitForTaskToken in Resource, pause the running until receiving a SendTaskSuccess or SendTaskFailure API call. (PUSH mechanism)
  • Activity Task: Activity Worker on EC2/Lambda/.., using GetTaskActivity API call for poll, sending response with SendTaskSuccess or SendTaskFailure API call (PULL mechanism), with SendTaskHeartBeat + HeartBeatSeconds
  • Standard vs Express (asynchronous and synchronous)

—–

OTHERS

  • AWS SES: Simple E-mail Service for sending marketing e-mails (like a Marketo or ConstantContact)
  • Amazon Macie: discover and protect sensitive data (Personal Identifiable Information, PII) by machine learning and pattern matching, on S3 bucket
  • AWS AppConfig: deploy dynamic configuration change without code deployment; validate with JSON Schema or Lambda Function
  • CloudWatch Evidently:
    • validate/serve new features to specified % of users only
    • Launches (= feature flags) and Experiments (= A/B testing), and Overrides (specific variants assigned to specific user-id)
    • evaluation events stored in CloudWatch Logs or S3
  • AWS AppSync:
    • managed service of GraphQL, combining multiples data sources
    • retrieve data in “realtime” or “MQTT” of WebSocket
    • for mobile apps: local data access and data sync
    • Security: API_KEY, AWS_IAM, OPENID_CONNECT, AMAZON_COGNITO_USER_POOLS
  • AWS Amplify:
    • create mobile and web applications (aka. ElasticBeans for mobile and web application)
    • Authentication (Cognito) + Data Store (AppSync + DynamoDB)
    • End-to-End (E2E) test, using Cypress
  • —–
  • AWS App Mesh = for application networking for microservices applications
  • AWS Resource Access Manager = share a Transit Gateway connection (only?) with other AWS accounts
  • AWS Server Migration Service (SMS) is for migrating virtual machines
  • AWS Elastic Beanstalk is a PaaS service for describing and provisioning resources. Can be used to quickly deploy and manage applications in AWS. Developers upload applications and Beanstalk handles the deployment details. Note that it’s not serverless, it relies on EC2 instances.
  • AWS Simple Workflow Service (SWF) is for executing tasks. Helps developers build, run, and scale background jobs
  • AWS CodeStar quickly develop, build and deploy applications on AWS
  • AWS Config manage the config of AWS resources
  • AWS Batch batch processing of computing jobs
  • Amazon Lex builds conversational interfaces into an application using voice and text
  • AWS X-Ray analyze, debug and improve performance of serverless applications
  • Amazon EMR allows researchers, scientists, businesses, etc. to process vast amounts of data using hosted Hadoop running on EC2 and S3
  • AWS Import/Export send HDDs with data to AWS and they import the data into S3
  • Amazon Connect: call centre
  • Amazon QuickSight: BI
  • Amazon Neptune: interactive graphs of DBs
  • AWS Config: tracks resource inventory, config history and config change notifications for the purpose of security and compliance. Assess, audit and evaluate the configurations of AWS resources.
  • Amazon AppStream: streaming service
  • Amazon Kinesis: collect and process streaming data
  • Amazon Elastic Transcoder: convert video and audio files into versions that play on phones, tablets and PCs
  • CloudSearch: search engine for your site
  • CLI: command-line interface. To use, it requires an IAM access key ID & secret access key
  • AWS LightSail: easy alternative to setting up a VPC. Product set includes virtual servers (instances), MySQL DBs, HA storage and load balance 
  • AWS IoT Core: connected devices interact securely with cloud applications