Access Amazon S3 Tables from Local using Apache Spark Docker Container | Demo Recently, Amazon S3 has introduced S3 Tables, a fully managed Apache Iceberg tables for storing tabular data at scale. It is specifically optimized for analytics workloads to deliver up to 3x faster query throughput and up to 10x higher transactions per second compared to self-managed tables (Iceber Abhijit Ghosh 22 Jan 2025 · 8 min read
Effortless Automation: Rapidly Generate AWS CloudFormation Templates and CDK Apps for Your Existing Resources in Minutes! Abhijit Ghosh 7 Feb 2024 · 1 min read
Idempotency Support for Route Table and Network ACL Creation in Amazon VPC! Abhijit Ghosh 5 Feb 2024 · 1 min read
Explore the Power of Account-Level Subscription Filters in Amazon CloudWatch Logs! Abhijit Ghosh 1 Feb 2024 · 1 min read
[new capability] Amazon EC2 supports disabling Amazon Machine Image (or AMI) Housekeeping is always an important activity. Otherwise, soon, it will become an operational and management overhead. Fortunately, Amazon EC2 now supports disabling an Amazon Machine Image (or AMI). Let’s take a closer look. Imagine there are hundreds of AMIs in your Catalog. Some are outdated, som Abhijit Ghosh 23 Oct 2023 · 2 min read
AWS Lambda How to Run Bash Scripts on AWS Lambda? If you want to run bash scripts on AWS Lambda, create a Lambda function with a custom runtime. Here are the steps to follow: Create a Lambda function with the "Author from scratch" option and "Use default bootstrap on Amazon Linux 1" as runtime. 🏖️The custom runtime option - "Provide your own boo Abhijit Ghosh 25 Sept 2023 · 4 min read
Amazon S3 Multivalue answer (MVA) support for DNS queries of Amazon S3 endpoints With multivalue answer (MVA) support for DNS queries of Amazon S3 endpoints, you can achieve enhanced reliability and higher aggregate throughput for your connections to S3. It is available at no cost to you. A single DNS query of an S3 endpoint (e.g. s3.us-east-1.amazonaws.com) will provide 8 S3 I Abhijit Ghosh 19 Sept 2023 · 1 min read
Amazon RDS Automatically set up connectivity between Amazon RDS or Aurora database and AWS Lambda in minutes The Amazon RDS console now offers the feature of automatically setting up connectivity between Amazon RDS or Aurora database and AWS Lambda in minutes. This feature helps to eliminate manual tasks such as configuring VPC, security groups, subnets, ingress/egress rules, and RDS proxy and boost produc Abhijit Ghosh 25 Aug 2023 · 2 min read
Amazon S3 How to Access Amazon S3 Bucket like a Local File System Mountpoint for Amazon S3 is now generally available and ready for production workloads. This open-source simple file client is optimized for read-heavy workloads that require high throughput and focused on high-performance access to large data sets. In this post, we'll see how to effortlessly acces Abhijit Ghosh 21 Aug 2023 · 3 min read
Amazon EventBridge Easy housekeeping with the new capability of Amazon EventBridge Scheduler To create, trigger, and manage scheduled events and tasks, we use Amazon EventBridge Scheduler (serverless offering). Over time, we accumulate a large number of unused finished schedules, which count towards your account's quota (1 million schedules by default). To meet the demands of many enterpris Abhijit Ghosh 18 Aug 2023 · 2 min read
Amazon VPC Use "Public IP Insights" [new feature] to optimize usage of chargeable public IPv4 address Public IPv4 addresses will no longer be free. Effective February 1, 2024, Amazon will charge you 0.005$ per public IPv4 address per hour. You will likely soon use "Public IP insights" (a new feature) to optimize the usage of public IPv4 addresses and associated costs to prevent unexpected bills. It Abhijit Ghosh 16 Aug 2023 · 2 min read
AWS Lambda Better, Faster and Cheaper with AWS Lambda's new Python 3.11 runtime If you are still running your AWS Lambda workloads on Python 3.10 or earlier versions, migrate to Python 3.11 for better performance and lower cost. AWS Lambda has added support for Python 3.11. On an average, Python 3.11 is 25% faster than 3.10 Faster Startup and Faster Runtime are the two main f Abhijit Ghosh 31 Jul 2023 · 1 min read
AWS Lambda Detect recursive loop and stop run-away AWS Lambda function A run-away AWS Lambda function may hog all available concurrency of your account and cause unexpected charges in the AWS bill. But Lambda can now detect and stop certain types of recursive or infinite loops. In this post, we'll explore more on this feature with a video demo. 🖋️Currently, this fea Abhijit Ghosh 21 Jul 2023 · 3 min read
AWS CodeBuild How to use GitHub Action with AWS CodeBuild ? With the release of GitHub Action support, AWS CodeBuild has expanded its scope of possibilities. You can leverage pre-built actions or third-party solutions available in GitHub’s marketplace. There is no additional cost for running GitHub Action Runner. What is GitHub Action? "Automate, custo Abhijit Ghosh 18 Jul 2023 · 22 min read
AWS CodeBuild Test and Debug AWS CodeBuild Builds Locally using AWS CodeBuild agent If you are still using AWS CodeBuild managed service for testing and debugging, then you have the scope to save cost and expedite the build-test-debug cycle loop using AWS CodeBuild local agent. Troubleshooting and fixing issues in BuildSpec file is so easy on the local development machine. In this Abhijit Ghosh 17 Jul 2023 · 7 min read
AWS CodeBuild AWS CodeBuild - GitHub Actions: COMMAND_EXECUTION_ERROR Message: Error while executing command | [error]Docker pull failed with exit code 1 For GitHub Docker container actions, "build project must have privileged mode enabled and be run by the default Docker user (root)." Otherwise, build will fail with following error logs. [Container] 2023/07/11 02:09:11 Entering phase PRE_BUILD [Container] 2023/07/11 02:09:11 Preparing action comma Abhijit Ghosh 10 Jul 2023 · 2 min read
Amazon DynamoDB How to Install and Configure Amazon DynamoDB local version 2.0 (downloadable)? Amazon DynamoDB local not only speeds up development and testing iteration loop, it also saves cost of using DynamoDB webservices. Free downloadable version of Amazon DynamoDB (DynamoDBLocal.jar) can run on any operating systems or platform having JRE (Java Runtime Environment). "This latest ver Abhijit Ghosh 7 Jul 2023 · 8 min read
Amazon S3 AWS S3: The difference between the request time and the current time is too large If system clock is out of sync, we may encounter error message like "An error occurred (RequestTimeTooSkewed) when calling the ListBuckets operation: The difference between the request time and the current time is too large." ag@aj:~$ aws s3 ls An error occurred (RequestTimeTooSkewed) when calling Abhijit Ghosh 4 Jul 2023 · 1 min read
AWS CLI How to uninstall AWS CLI version 2? If you already have AWS Command Line Interface (AWS CLI 2) and want to uninstall it from your system, then you will find all the steps to remove it. Find symbolic link for aws program. This will give you the --bin-dir abhijit@awsjunkie:~$ which aws /usr/local/bin/aws Find installation director Abhijit Ghosh 4 Jul 2023 · 1 min read
Amazon S3 Mountpoint for Amazon S3 supports sequential write operations for creating new files Mountpoint for Amazon S3 (an open source simple file client for mounting S3 bucket as a local file system) was launched on AWS Pi Day 2023 with no support for write operations. Now, it has added the support for sequential write operations to create new files. Let's quickly take a look. Installer i Abhijit Ghosh 4 Jul 2023 · 1 min read
Amazon S3 Amazon S3 dual-layer server-side encryption with AWS KMS (DSSE-KMS) Amazon S3 has recently added support for two independent layers of server-side encryption with AWS KMS (DSSE-KMS) for top-secret workloads. With this launch, Amazon S3 is the only cloud object storage service that supports multi layers of server-side encryption at object level. "DSSE-KMS is desi Abhijit Ghosh 20 Jun 2023 · 2 min read
Amazon CodeWhisperer How to setup CodeWhisperer with JupyterLab? Amazon CodeWhisperer (AI coding companion) is now available as an extension for JupyterLab. If you outline the coding task as a comment, this generative AI extension will auto-suggest single-line, multiline or full-function code right within the JupyterLab. Here's a guide on how to set up the Cod Abhijit Ghosh 21 May 2023 · 7 min read
AWS Lambda AWS Lambda SnapStart - Turbocharge Java Application Startup by up to 10x at no extra cost Spring Framework 5.0 introduced Spring Cloud Function to aid the implementation of business logic as a function and to enable Spring Boot features on different serverless cloud providers (AWS Lambda, Azure, Google Cloud Platform, Oracle Fn platform and Apache OpenWhisk). Its "functional" style of be Abhijit Ghosh 21 Apr 2023 · 4 min read
Amazon Linux How to run Amazon Linux 2023 (AL2023) locally on WSL? Amazon Linux 2023 (AL2023) is the next generation of Amazon Linux for developing and running cloud applications in high-performance and security enhanced execution environment. This rpm-based Linux distribution is optimized for Amazon EC2. Amazon Linux 2023 is not available in the Microsoft Store. Abhijit Ghosh 14 Apr 2023 · 7 min read
WSL Fixed: Temporary failure in name resolution in WSL2 DNS resolution not working in WSL2? Getting messages like "Temporary failure in name resolution" or "Temporary failure resolving..."? Issue: abhijit@AwsJunkie:~$ ping www.AwsJunkie.com ping: www.AwsJunkie.com: Temporary failure in name resolution abhijit@AwsJunkie:~$ sudo apt update Err:1 http Abhijit Ghosh 12 Apr 2023 · 2 min read
AWS Lambda What is AWS Lambda's new Response Payload streaming feature? AWS Lambda's new feature, Response Streaming, can enhance user experience, responsiveness, and search engine rankings of your web applications by lowering Time to First Byte (TTFB). Additionally, it supports a larger payload (soft limit of 20 MB) compared to a traditional buffered response (max 6 MB Abhijit Ghosh 8 Apr 2023 · 4 min read
AWS Management Console AWS Console Toolbar - a new feature that brings both AWS Management Console and AWS CloudShell in a single view Last week, AWS launched a new feature called Console Toolbar that brings both AWS Management Console and AWS CloudShell in a single view. There is no additional charge for using CloudShell and Console Toolbar :-) What is AWS CloudShell? AWS CloudShell is a pre-authenticated ready to use browser- Abhijit Ghosh 3 Apr 2023 · 2 min read
LocalStack How to install awslocal ( LocalStack AWS CLI) on Ubuntu running on WSL2 (Windows Subsystem for Linux)? If you are already enjoying LocalStack ("A fully functional local cloud stack") for building awesome AWS cloud and serverless applications locally, then awslocal will add extra Saccharin to sweeten your development experience for sure. What is awslocal? awslocal is a wrapper around the AWS CLI to Abhijit Ghosh 2 Apr 2023 · 4 min read
LocalStack How to install LocalStack (AWS cloud service emulator) on WSL2 to develop cloud and serverless apps offline? LocalStack is an AWS cloud service emulator that can be used as a local cloud sandbox for development, prototyping and testing. It is "A fully functional local cloud stack". You can efficiently build AWS cloud and serverless applications locally without connecting to AWS cloud. Therefore, it will he Abhijit Ghosh 27 Mar 2023 · 5 min read
Node.js How to install Node.js on Ubuntu 22.04 distro running on WSL2 (Windows Subsystem for Linux)? In this post, we'll see how to install Node.js (open-source, cross-platform JavaScript runtime environment) on Ubuntu 22.04 distro running in WSL2 (Windows Subsystem for Linux). Option #1 - Install Node.js with APT package Manager from the Official Repositories NodeJs is already available in defa Abhijit Ghosh 26 Mar 2023 · 3 min read
Amazon S3 Mountpoint for Amazon S3 -Build & Demo On AWS Pi Day 2023, Amazon announced Mountpoint for Amazon S3, an open-source simple file client for mounting S3 buckets as a local file system. It is optimized for read-heavy workloads that require high throughput and is mainly focused on high-performance access to large data sets. So naturally, it Abhijit Ghosh 16 Mar 2023 · 8 min read
Amazon EC2 Does a "stopped" EC2 instance cost any money? There is no charge for a stopped EC2 instance or any data transfer fees. Associated Elastic IP (EIP) addresses and attached Amazon Elastic Block Store (EBS) volumes will incur charges. If you don't need Elastic IP addresses, disassociate and release. As an EBS snapshot is relatively cheaper than a Abhijit Ghosh 14 Mar 2023 · 1 min read
AWS CLI Install AWS Command Line Interface (AWS CLI) on Windows Subsystem for Linux (WSL 2) AWS Command Line Interface (AWS CLI) is used to interact with different AWS (Amazon Web Services) product and services in command-line shell. AWS CLI version 2 is the latest one. In this article, we'll see how to download and install AWS CLI on Windows Subsystem for Linux (WSL 2). Video: Update Abhijit Ghosh 23 Feb 2023 · 1 min read
AWS SAM Install AWS Serverless Application Model Command Line Interface (AWS SAM CLI) on Windows Subsystem for Linux (WSL 2) AWS SAM command line interface (AWS SAM CLI) tool is must have for building AWS serverless applications. In this post, we'll see how to install latest version of AWS SAM CLI on Windows Subsystem for Linux (WSL 2). Video: Prerequisites: AWS Command Line Interface (AWS CLI 2) is recommended for S Abhijit Ghosh 23 Feb 2023 · 2 min read
Docker Fix: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running? Issue: sudo service docker start says "Starting Docker: docker" but sudo service docker status shows "Docker is not running" and most docker commands throw following message "Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?". abhijit@AwsJunkie:~$ sud Abhijit Ghosh 4 Feb 2023 · 3 min read
Docker Install Docker Engine and Docker Compose on Windows Subsystem for Linux (WSL 2) If you already have Docker Desktop installed or want to use docker on both Windows and WSL2, then Docker Desktop is better choice. But I personally prefer to install and run the "most-loved Tool" docker inside Windows Subsystem for Linux (WSL 2). This keeps the development environment isolated and t Abhijit Ghosh 3 Feb 2023 · 5 min read
Python Install JupyterLab or Jupyter Notebook in Windows Subsystem for Linux (WSL2) Web-based interactive development environment JupyterLab (the next generation of Jupyter Notebook ) is very popular in the field of data science and machine learning. Windows Subsystem for Linux (WSL2) is well known for super-fast boot time, less resource consumption and seamless integration of Linu Abhijit Ghosh 1 Feb 2023 · 6 min read
WSL Check or Upgrade/Downgrade Windows Subsystem for Linux (WSL) Version This is a short post on how to check or upgrade/downgrade the version of Windows Subsystem for Linux (WSL) running on Windows 10 or Windows 11 host. To check WSL version set to installed Linux distro, execute wsl -l -v or wsl --list --verbose C:\Users\AwsJunkie>wsl --list --verbose NAME Abhijit Ghosh 2 Oct 2022 · 1 min read
WSL Setup Proxy for CURL in Ubuntu running on WSL (Windows Subsystem for Linux) If Windows host machine is running behind any proxy (e.g. corporate proxy in office network), then it is required to pass/configure Proxy server and Proxy Authentication for cURL commands. Let's explore multiple ways to do it. CURL option "proxy" and "proxy-user" cURL command has option -x, --pr Abhijit Ghosh 11 Sept 2022 · 4 min read
WSL Update WSL2 (Windows Subsystem for Linux) kernel on Windows Windows Build version 20150 has introduced command line options to update WSL2 (Windows Subsystem for Linux) Kernel very conveniently. Let's try that. Run wmic os get BuildNumber in command line to check the build version of your Windows. C:\Users\AwsJunkie>wmic os get BuildNumber BuildNumber 220 Abhijit Ghosh 11 Sept 2022 · 2 min read
WSL How to Backup and Restore Linux Distro on WSL (Windows Subsystem for Linux)? Nobody can deny the importance of backup & restore process for any environment. Specially, when multi-steps are involved in setup and configuration and we keep modifying the environment with time. Basically, if anything goes wrong for any reason, we don't want to end up wasting time by setting up th Abhijit Ghosh 3 Sept 2022 · 4 min read
WSL How to Uninstall or Remove Linux Distro Installed on WSL (Windows Subsystem for Linux)? Uninstallation of redundant or unused Linux Distros on Windows Subsystem for Linux (WSL) is an important activity to reclaim space and maintain hygiene of the system. It is just one command to remove a distro from WSL. Video: * Run the following command in Windows Command Prompt to find the lis Abhijit Ghosh 2 Sept 2022 · 1 min read
WSL How to Install Windows Subsystem for Linux (WSL2) on Windows 11? Steps to install Windows Subsystem for Linux (WSL2) on Windows 10 works for Windows 11 too. But Microsoft made it super simple in Windows 11. Just one command that takes minutes to install a Linux distribution (Ubuntu by default). If you are curious about other 3 different options that I mentioned Abhijit Ghosh 30 Aug 2022 · 5 min read
WSL How to Install Windows Subsystem for Linux (WSL2) on Windows 10? Windows Subsystem for Linux (WSL) is an awesome feature of Windows 10. No more VMware or Oracle VirtualBox is required to run Linux on Windows. WSL is faster and less resource consuming, zero or less configuration compared to any other traditional Virtual Machines (VM). Installation and setup of Lin Abhijit Ghosh 26 Aug 2021 · 8 min read
WSL How to change hostname on Ubuntu running on Windows WSL? If you are running Ubuntu on Windows WSL (Windows Subsystem for Linux) and wondering how to change hostname permanently then follow all steps mentioned below. Once hostname is changed, if you get "unable to resolve host <hostname>: Name or service not known" then follow the step #4 for the fix. V Abhijit Ghosh 13 Jul 2021 · 3 min read
IoT Kid's Eye and Energy Safe Smart TV - Part 2 Sometimes I feel, kids do this just to irritate us. Or may be, we always have a prioritized list of problems in our subconscious mind. When we solve the top one, the next one pops up and gets highlighted. It is indeed never ending. Few extra hours of running TV will hardly make any difference in m Abhijit Ghosh 9 Jul 2020 · 5 min read
IoT Kid's Eye Safe Smart TV - Part 1 Outside of home COVID-19. Inside at home COVID-16. Whole world is fighting against COVID-19. But for COVID-16, I am all alone. I am tired of saying "please stay away from the screen". So thought of this concept project to avoid brain infection at home. By the way, I call two of my kids (age 1 and 6 Abhijit Ghosh 19 Jun 2020 · 4 min read
IoT Internet enabled (IOT) smoke alarm with SMS notification using esp8266, MQ2, ThingSpeak and Twilio ESP8266 based Internet enabled (IOT) smoke and fire alarm to send SMS notification (Twilio) and to trigger alarm with flashing red LED if it detects smoke and leakage of combustible gas using MQ2 gas sensor. Also ThingSpeak analytics is a great bonus. Abhijit Ghosh 4 Sept 2017 · 7 min read
IoT IOT - Home Automation project using Geofencing, MQTT, ESP8266 and MycroPython A home automation project that maps switching on/off of appliances with your entry/exit of a geofence (virtual perimeter for a real-world geographic area). Step into a cool room without having to wait for the air conditioning to start functioning? Watch the garage door open as you drive in. Abhijit Ghosh 8 May 2017 · 7 min read
IoT Home Automation - Internet of Things - MQTT (MQ Telemetry Transport), ESP8266 and MycroPython This is my first post on home automation. As a 'Hello World' project, switching on/off a bulb is a common use case. This post is not an exception. But to spice it up, we'll use MQTT (MQ Telemetry Transport) - a machine-to-machine (M2M)/Internet of Things (IOT) connectivity protocol and an android ap Abhijit Ghosh 22 Mar 2017 · 6 min read
IoT Configure STA_IF interface of ESP8266 (MycroPython firmware) to connect with WIFI network automatically on boot ESP8266 can act as both Station / WIFI client (STA_IF interface) and Access Point (AP_IF interface). AP_IF interface is active by default in MycroPython firmware on ESP826 . In the earlier post, we connected to the access point (AP mode) of ESP8266 and used WebREPL to remotely control switch on/off Abhijit Ghosh 14 Mar 2017 · 4 min read
IoT Example: Remotely switch on/off and control brightness of an LED using MicroPython WebREPL So far, we have flashed MicroPython firmware onto an ESP8266 and setup WebREPL to run programs over WIFI on the chip. Now it's time for a hands-on excercise. We'll remotely switch on/off and control brightness of an LED using MicroPython WebREPL. Abhijit Ghosh 8 Mar 2017 · 5 min read
IoT How to setup WebREPL to connect to Python prompt (REPL) of ESP8266 over WIFI network? WebREPL daemon auto-start is disabled by default in MycroPython on ESP8266 for security reason. Here, we'll enable the same, connect to the REPL over WIFI using locally installed WebREPL client and run a "Hello World" program remotely on the chip. Abhijit Ghosh 23 Feb 2017 · 3 min read
IoT How to flash MicroPython firmware onto an ESP8266 ESP-12E chip using esptool? By default, ESP8226 module comes with AT Firmware. In this post, we'll see how to flash MicroPython firmware onto an ESP8266 ESP-12E chip using esptool. Tools,Technologies and Components used in this article 1. ESPlorer IDE 2. MicroPython 3. macOS Sierra 4. Components: * ESP8266 ESP Abhijit Ghosh 4 Feb 2017 · 6 min read
Raspberry Pi How to uninstall OpenCV (Open Source Computer Vision) from Raspberry Pi - Raspbian Jessie? To uninstall OpenCV (Open Source Computer Vision) from Raspberry Pi (Raspbian Jessie), follow the steps mentioned below in this article. Abhijit Ghosh 10 Jul 2016 · 3 min read
IoT Install and Configure Oracle Java ME Embedded on Raspberry Pi Raspberry Pi is a low cost, credit-card sized and fastest selling personal computer. In this article, we'll install and configure Oracle Java ME Embedded on a Raspberry Pi. What is a Raspberry Pi? The Raspberry Pi is a low cost, credit-card sized computer that plugs into a computer monitor or Abhijit Ghosh 17 Mar 2015 · 6 min read