DevDojo

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

A Laravel Starter Kit that includes Authentication, User Dashboard, Edit Profile, and a set of UI Components. Learn more about the DevDojo sponsorship program and see your logo here to get your brand in front of thousands of developers.

docker offload stop

🧠 Why This Is a Big Deal

If you're on an underpowered laptop: now you can run LLMs, big builds, and heavy services without blowing up your laptop.

If your team has mixed hardware: with Docker Offload everyone gets a consistent, fast environment.

If you're building agentic apps or running CI jobs: this makes local + cloud hybrid dev actually work.

💡 What You Can Do With Docker Offload

Docker Offload brings the power of the cloud into your local Docker workflow. Here are a few things you can do right away:

Build containers faster by offloading docker build to remote cloud machines with smart caching
Run GPU-heavy apps like machine learning pipelines, video processing, or LLM inference using NVIDIA L4 GPUs
Use Docker Compose to spin up full-stack apps in the cloud just like you do locally
Develop on low-powered machines or virtual desktops without worrying about performance limits
Run AI demos like Jupyter Lab, Hugging Face Transformers, or multi-agent systems without setting up any infra
Share the same cloud environment across your team, no matter what hardware they use
No special setup. No infra to manage. Just faster, smoother development with tools you already know.

🧵 Final Thoughts

Docker Offload makes serious development easier for everyone, whether you're solo or working in a team. You keep your local flow, but your containers run on machines that don't melt under pressure.

You can also check out the pricing details here.

Try it and let me know what you think in the comments!

How to Write Unit Tests for Bash Scripts (The Simple Way)

Bobby Iliev — Fri, 11 Apr 2025 22:21:48 -0700

Introduction

If you enjoy writing Bash scripts to automate tasks, then you might ask yourself: "Can I write tests for my Bash scripts?".

The answer is: Yes, you can! And it's not hard at all.

🛠️ What You Need

We will use a tool called bats. It lets you write tests for Bash scripts in a clean and simple way.

To install it:

brew install bats-core  # On macOS
# OR
sudo apt install bats  # On Ubuntu/Debian

🧪 A Simple Bash Script Example

Let’s say you have this function in script.sh:

#!/bin/bash

say_hello() {
  echo "Hello, $1!"
}

✅ How to Test It

Create a file called test_script.bats:

#!/usr/bin/env bats

load './script.sh'

@test "say_hello prints the correct message" {
  run say_hello "Bobby"
  [ "$status" -eq 0 ]
  [ "$output" = "Hello, Bobby!" ]
}

Then run the tests with:

bats test_script.bats

🤖 Run Tests with GitHub Actions

You can also run the tests automatically using GitHub Actions.

Create a .github/workflows/test.yml file:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

name: Bash Script Tests

on:
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - name: Install BATS
        run: sudo apt-get update && sudo apt-get install -y bats
      - name: Run tests
        run: bats test_script.bats

Every time you push changes or open a pull request, your tests will run automatically. This helps you catch problems early.

📘 Learn More About Bash

If you are just starting with Bash scripting, I wrote a free and open-source book that explains the basics step by step:

Introduction to Bash Scripting

It covers loops, variables, functions, and more. Perfect for beginners.

If you want to try this on a cloud server, I use and recommend DigitalOcean. You'll get $200 in free credits for 60 days. Great for running your projects, testing scripts, or learning Linux.

If you have questions or want to chat more about Bash or DevOps, feel free to reach out on Twitter/X: @bobbyiliev_

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Thanks for reading and happy scripting!

– Bobby

7 Best SQL Books for Beginners

Bobby Iliev — Sat, 29 Mar 2025 08:11:22 -0700

Learning SQL doesn't have to be boring or overwhelming. With the right book, it can actually be fun and super rewarding.

Whether you're trying to level up your backend skills, work with data more efficiently, or build your first app with a real database, t'ese beginner-friendlSQL books will help you get started. I've included both free and paid options, so there's something for every budget.

1. Introduction to SQL (Free)

This free ebook is perfect for anyone starting their SQL journey. It covers the basics of SELECT statements, filtering data, using joins, and more. If you're brand new to SQL and want to get started today, this is a great resource.

2. SQL for Data Analysis by Cathy Tanimura (Paid)

If you're into data, this book is for you. It teaches SQL in the context of real-world data analysis tasks. It's beginner-friendly, with practical examples that make it easy to follow along.

3. Head First SQL by Lynn Beighley (Paid)

This is a super engaging and visual book that breaks down SQL concepts in a fun, approachable way. Perfect for beginners who want to avoid dry technical writing and learn by doing.

4. Learning SQL by Alan Beaulieu (Paid)

A classic. It starts with the basics and takes you through everything from creating tables to writing complex queries. Clear explanations and practical exercises make it a solid resource for SQL newcomers.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

5. SQLBolt (Free)

Not a book, but a fantastic interactive tutorial that walks you through SQL concepts step by step. You can run queries right in the browser, g'eat for hands-olearners.

👉 Try SQLBolt

6. The Art of SQL by Stéphane Faroult (Paid)

Once you've got the basics down, this book will help you think like a SQL pro. It's less about syntax and more about strategy, h'w to write clean, efficienqueries.

7. SQL Official Documentation (Free)

Not exactly a book, but if you ever get stuck, the docs are your best friend. Whether you're using PostgreSQL, MySQL, or SQLite, their official docs are gold mines of information.

👉 PostgreSQL Docs 👉 MySQL Docs 👉 SQLite Docs

8. SQL Courses by Aaron Francis (Paid)

If you're ready to take your SQL skills to the next level, Aaron Francis has two phenomenal courses that go beyond the basics:

🚀 High Performance SQLite: Learn how to get the most out of SQLite, especially in production.
🐘 Mastering PostgreSQL: A deep dive into advanced PostgreSQL features like indexing, performance tuning, and query optimization.

Aaron's teaching style is clear, practical, and full of insights you can apply immediately. Follow him on X for even more great tips.

Conclusion

SQL is one of the most valuable skills you can learn as a developer or data enthusiast. With these beginner-friendly books and resources, you'll be writing queries and analyzing data in no time.

And if you want to practice your SQL skills on a real database, you can spin up a server with my DigitalOcean referral link and get $200 in free credit.

Already reading one of these books or have another favorite? Drop it in the comments, I'd love to check it out. Happy querying! 📊

Timeless Developer Skills: Building a Rock-Solid Foundation for Long-Term Success

Bobby Iliev — Sun, 02 Mar 2025 11:49:21 -0800

As a developer who has worked with countless technologies over the years, I've realized something important: while frameworks and libraries come and go, certain fundamental skills remain invaluable throughout your career. Instead of constantly chasing the latest trend, mastering these timeless skills provides a strong foundation that will serve you regardless of industry changes.

1. ⚡ Algorithms and Data Structures

Why It Matters

Every piece of software processes data, and understanding how to efficiently organize and manipulate that data is essential. Mastering algorithms and data structures helps you develop a problem-solving mindset that transcends specific programming languages.

How to Learn

Implement fundamental structures like linked lists, stacks, and trees from scratch.
Solve classic problems on platforms like LeetCode and HackerRank, gradually increasing difficulty.
Analyze algorithms in your daily work - ask, "How would this scale with 10x more data?"
Build a small project that requires thoughtful data structure selection.
For more in-depth learning, check out the Algorithms book by Robert Sedgewick and Kevin Wayne
In addition to the above book, check out this GitHub repo: Awesome Algorithms

2. 🏗️ System Design and Scalability

Why It Matters

The ability to design scalable systems separates junior developers from architects. Understanding system design ensures your applications won’t collapse under growing workloads.

How to Learn

Study the architecture of successful open-source projects.
Practice designing systems on paper before implementing them, focusing on bottlenecks.
Explore comprehensive resources like System Design: The Complete Course.
Try scaling a simple app from handling 10 requests per second to 1000 and document what breaks first.

3. 🔄 Version Control and Git

Why It Matters

Version control is more than just tracking changes - it enables collaboration, rollback safety, and a clear project history. Mastering Git can significantly improve your workflow.

How to Learn

Move beyond commit, push, and pull - learn how to use branches effectively.
Master Git’s advanced features like interactive rebasing, cherry-picking, and bisect.
Check out this free ebook: Introduction to Git and GitHub.
Contribute to open-source projects to experience real-world Git collaboration.

4. 🗄️ SQL and Database Design

Why It Matters

Despite the rise of NoSQL databases, SQL remains the backbone of most applications. Understanding how to design efficient databases and optimize queries is an invaluable skill.

How to Learn

Design a schema for a real-world application, like an e-commerce platform.
Write increasingly complex queries, focusing on joins and aggregations.
Learn to read and interpret query execution plans for performance tuning.
Build a small CRUD application that emphasizes efficient database interactions.
Explore this free Introduction to SQL ebook.

5. 🖥️ Linux and Shell Scripting

Why It Matters

Most servers, cloud instances, and containers run Linux. Being proficient with Linux commands and shell scripting improves productivity and troubleshooting abilities.

How to Learn

Set up a Linux environment for daily development.
Perform common operations via the terminal instead of relying on GUIs.
Automate repetitive tasks with simple Bash scripts.
Check out this Introduction to Bash Scripting ebook.
Dive deeper with this Introduction to Linux book.

6. 🌍 Networking

Why It Matters

Understanding how data flows across networks helps with debugging, optimization, and building resilient applications.

How to Learn

Study the OSI model and where HTTP fits in.
Use tools like curl, Wireshark, and browser network debugging tools.
Build a simple REST API and analyze its network traffic.
Experiment with HTTP caching headers to observe their performance effects.
Check out this Networking Fundamentals book.

7. ✨ Writing Clean Code and Debugging

Why It Matters

Most development time is spent reading code, not writing it. Writing clean, readable code saves hours of maintenance and reduces bugs.

How to Learn

Review and refactor your old code for readability.
Master debugging tools specific to your tech stack.
Study design patterns but focus on when to apply them.
Practice pair programming to get immediate feedback on code clarity.
Check out the Clean Code book by Robert C. Martin.

8. 🔐 Security Best Practices

Why It Matters

Security must be a priority, not an afterthought. Understanding vulnerabilities and how to prevent them is essential for protecting applications and user data.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

How to Learn

Study the OWASP Top 10 vulnerabilities with real-world examples.
Set up a deliberately vulnerable app (like DVWA) and practice exploiting and fixing issues.
Integrate security checks into your development workflow.
Perform regular security reviews of your code and applications.
Check out Ross Anderson's Security Engineering book. It's free!

9. ☁️ Cloud and Infrastructure as Code

Why It Matters

Infrastructure as Code (IaC) has revolutionized application deployment and scaling. Mastering these skills enables reproducible environments and eliminates configuration drift.

How to Learn

Start a project on AWS, GCP, or Azure using their free tiers.
Learn Docker fundamentals with this Introduction to Docker.
Practice automation with this Introduction to Terraform book.
Build a CI/CD pipeline for automated deployments.

10. 🗣️ Soft Skills and Communication

Why It Matters

Technical expertise alone isn’t enough - strong communication and collaboration skills accelerate career growth and make teamwork more effective.

How to Learn

Practice explaining technical concepts to non-technical people.
Contribute to open-source projects to experience diverse collaboration.
Write technical blog posts to improve written communication skills on platforms like DEV.to and DevDojo
Learn how to give and receive constructive feedback effectively.

📍 Structured Learning Paths

If you prefer structured guidance on skill development, check out roadmap.sh. Their curated roadmaps for different specializations complement the timeless skills in this guide.

🎯 Final Thoughts

Developers who master these foundational skills adapt quickly to new technologies and frameworks. They recognize patterns over syntax, principles over commands, and solutions over isolated code snippets.

The key is to apply these skills through practical projects. Start small, build gradually, and learn from experienced developers. Mastery takes time, so focus on deep understanding and hands-on experience rather than surface-level knowledge.

📩 Want to Connect?

If you have questions or want to share your learning journey, reach out to me on Twitter @bobbyiliev_ or visit my blog at bobbyiliev.com!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Need a server for practice? Get a free $200 credit on DigitalOcean with my referral link: Free $200 Credit for DigitalOcean. Perfect for experimenting with Linux, Docker, and networking!

How to Build Windows Docker Images with GitHub Actions

Bobby Iliev — Wed, 05 Feb 2025 10:11:49 -0800

Have you ever needed to build a Windows Docker image but don't have access to a Windows machine? In this guide, I'll show you how to leverage GitHub Actions to build Windows container images without needing a local Windows environment.

Why Build Windows Docker Images?

While Linux containers are more common, Windows containers are essential for:

.NET Framework applications
Windows-specific applications
Legacy Windows applications that need containerization
Applications that require Windows-specific features

Prerequisites

Before we start, you'll need:

A GitHub account
A Docker Hub account
A repository with your Dockerfile and application code
Docker Hub credentials configured as GitHub secrets

You can take a look at my example repository for reference.

The Dockerfile

Let's look at a sample Dockerfile that sets up a Python environment on Windows:

# Use a more recent Windows Server Core image
FROM mcr.microsoft.com/windows/servercore:ltsc2022

# Set shell to PowerShell
SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

# Download and install Python
RUN [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; \
    Invoke-WebRequest -Uri https://www.python.org/ftp/python/3.9.5/python-3.9.5-amd64.exe -OutFile python-3.9.5-amd64.exe ; \
    Start-Process python-3.9.5-amd64.exe -ArgumentList '/quiet InstallAllUsers=1 PrependPath=1' -Wait ; \
    Remove-Item python-3.9.5-amd64.exe

# Verify Python installation
RUN python --version

# Set the working directory
WORKDIR /app

# Copy application files
COPY . /app

# Set the startup command
CMD ["python", "app.py"]

This Dockerfile:

Uses the mcr.microsoft.com/windows/servercore:ltsc2022 base image
Installs Python 3.9.5
Sets the working directory to /app
Copies the application files to the container
Specifies the startup command as python app.py

Setting Up GitHub Actions

Create a .github/workflows/build.yml file in your repository:

name: Build Windows Docker Image

on:
  push:
    branches: [ main ]
  workflow_dispatch:

jobs:
  build:
    runs-on: windows-latest
    environment: prod
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Login to Docker Hub
        uses: docker/login-action@v3
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and Push
        shell: powershell
        run: |
          docker build . -t ${{ secrets.DOCKERHUB_USERNAME }}/windows-python:latest
          docker push ${{ secrets.DOCKERHUB_USERNAME }}/windows-python:latest

How It Works

The workflow runs on windows-latest runner provided by GitHub Actions
It checks out your code
Logs into Docker Hub using your credentials
Builds the Windows Docker image
Pushes the image to Docker Hub
Displays the image digest for verification

Note that in this example, for simplicity, we're using the latest tag. In a production environment, you should use versioned tags and instead of hardcoding the image name, you can use environment variables.

Important Considerations

Image Size: Windows containers are typically larger than Linux containers. Plan your registry storage accordingly.
Build Time: Windows container builds usually take longer than Linux builds.
Compatibility: Ensure your base image version matches your deployment environment.

Resources

Want to learn more about Docker? Check out these resources:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Conclusion

GitHub Actions makes it possible to build Windows Docker images without a local Windows environment. This approach is particularly useful for cross-platform teams or developers primarily working on non-Windows systems.

If you're new to Docker and want to learn more, I've written a comprehensive free Docker eBook that covers all the basics and more.

Want to try this out but don't have a server? You can use my DigitalOcean referral link to get a free $200 credit!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Have questions or want to share your experience? Feel free to reach out to me on Twitter @bobbyiliev_.

Docker Networking - A Quick Guide to Get You Started

Bobby Iliev — Fri, 31 Jan 2025 06:29:58 -0800

In this post, I'll show you how Docker networking works in simple terms. If you've ever wondered how containers talk to each other or connect to the internet, this guide is for you!

If you want to try this out but don't have a server, you can use my DigitalOcean referral link to get a free $200 credit: Free $200 Credit For DigitalOcean. This is more than enough to spin up a few Droplets and practice Docker networking!

And if you're new to Docker, I've written a free comprehensive Docker ebook that covers all the basics - you can check it out at Introduction to Docker eBook. The ebook will help you get started with Docker before diving into networking!

Docker Network Types

Docker comes with 5 network types out of the box:

Bridge (default)
Host
None
Overlay
MacVlan

Check out the documentation for the network drivers here.

Let's look at each type and see how they work in real-world scenarios!

Bridge Network - The Default Choice

When you install Docker, it creates a default bridge network automatically. All containers use this network unless you tell them otherwise. Think of a bridge network like a virtual switch - it lets containers on the same network talk to each other.

Let's first check what networks we already have. Run this command:

docker network ls

You'll see something like this:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

NETWORK ID     NAME      DRIVER    SCOPE
abc123def456   bridge    bridge    local
xyz789uvw321   host      host      local
123abc456def   none      null      local

Creating Your Own Bridge Network

While the default bridge network works, creating your own is better for a few reasons:

Your containers can find each other by name (like 'laravel-app' or 'mysql-db')
You get better isolation from other containers on your system
You can connect and disconnect containers while they're running

Here's how to create your own bridge network. We'll call it 'my-network':

docker network create my-network

Now let's try something practical - we'll run an Nginx container and a test container to see if they can talk to each other:

# Start a Nginx container
docker run -d --name web --network my-network nginx

# Start an Ubuntu container
docker run -it --name ubuntu --network my-network ubuntu bash

From inside the Ubuntu container, you can now ping the Nginx container by name. This is super useful when setting up connections between your app and database:

ping web

Host Network - Direct Access

The host network is different - it removes the network isolation between the container and your Docker host. It's like running the program directly on your machine. This can be useful for specific cases where you need the best network performance.

Here's how to use host networking:

docker run -d --network host nginx

Now Nginx is available directly on port 80 of your host - you don't need to map any ports!

Warning: Be careful with host networking. Only use it when you really need to because it gives containers full access to your host's network.

Port Mapping

When you're using bridge networking (which is most of the time), you need to map container ports to host ports if you want to access them from outside. Let me show you how this works:

docker run -d -p 8080:80 nginx

This command tells Docker: "Take port 80 inside the container and make it available as port 8080 on my host machine." Now you can access Nginx at http://localhost:8080

Real World Example: Running Laravel with MySQL

Let's set up a more realistic example - a Laravel application with MySQL. We'll create a network for them to communicate and set up everything properly.

First, let's create a network for our Laravel app:

# Create a network for our Laravel application
docker network create laravel-net

Now, let's start MySQL. We'll give it a specific name so our Laravel app can find it:

# Start MySQL with specific settings for Laravel
docker run -d \
    --name mysql-db \
    --network laravel-net \
    -e MYSQL_ROOT_PASSWORD=secret \
    -e MYSQL_DATABASE=laravel \
    -e MYSQL_USER=laravel \
    -e MYSQL_PASSWORD=secret \
    mysql:5.7

Let's break down what this does:

Creates a container named 'mysql-db'
Connects it to our 'laravel-net' network
Sets up a root password
Creates a database called 'laravel'
Creates a user 'laravel' with password 'secret'

Now for our Laravel application:

# Start the Laravel application
docker run -d \
    --name laravel-app \
    --network laravel-net \
    -p 8000:80 \
    -e DB_HOST=mysql-db \
    -e DB_DATABASE=laravel \
    -e DB_USERNAME=laravel \
    -e DB_PASSWORD=secret \
    your-laravel-image

Here's what's happening:

The app is named 'laravel-app'
It's connected to the same network as MySQL
Port 80 in the container is mapped to 8000 on your host
Environment variables are set to connect to MySQL
Notice we use 'mysql-db' as the database host - that's the container name!

Checking Container Networks

Here are some helpful commands to check your container networks. I use these all the time when debugging:

Want to see what network a container is using? Try this:

docker inspect -f '{{range $key, $value := .NetworkSettings.Networks}}{{$key}} {{end}}' laravel-app

Need a list of all your networks?

docker network ls

Want detailed info about a network? This shows all containers connected to it:

docker network inspect laravel-net

Troubleshooting Tips

Here are some common issues and how to fix them:

If your Laravel app can't connect to MySQL:
- Check if both containers are on the same network
- Try docker exec laravel-app ping mysql-db
- Look at the environment variables in your Laravel app
If you can't access your Laravel app from your browser:
- Check if the ports are mapped correctly with docker ps
- Make sure no other service is using port 8000 on your host
- Try curl localhost:8000 from your host machine
Network seems slow?
- Consider using host networking for development
- Check container logs for any error messages
- Monitor container resources with docker stats

Conclusion

That's the basics of Docker networking! Remember these key points:

Use bridge networks for connecting your containers
Create your own networks instead of using the default bridge
Use container names as hostnames when containers need to talk
Map ports when you need to access containers from outside

Now you can set up proper networking for your Laravel applications in Docker. Try creating different networks and connecting containers to see how it all works together.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Have questions? You can reach me on Twitter @bobbyiliev_ or check out my blog at bobbyiliev.com!

5 Terraform Best Practices I Wish I Knew When I Started

Bobby Iliev — Wed, 29 Jan 2025 07:06:23 -0800

Introduction

Hey! I'm Bobby, a DevOps engineer and the author of the Introduction to Terraform ebook.

In this article, I'll share five Terraform best practices that I wish I knew when I first started using Terraform. These tips will help you write cleaner, more maintainable infrastructure as code and avoid common mistakes that can lead to deployment headaches.

Prerequisites

To follow along, you should have:

Basic knowledge of Terraform
Terraform installed on your system

If you're new to Terraform, I highly recommend checking out my Introduction to Terraform ebook, where I cover everything from the basics to managing infrastructure at scale.

Step 1 — Always Use Remote State Storage

By default, Terraform stores its state file (terraform.tfstate) locally. However, in real-world projects, this is not ideal because:

Local state files can be lost, leading to data inconsistencies.
Teams working together need a shared state to prevent conflicts.
Sensitive data may be exposed if the state file is not secured properly.

A better approach is to store your state remotely using Terraform Cloud. S3, or HashiCorp Consul.

Example: Storing Terraform State in AWS S3

terraform {
  backend "s3" {
    bucket         = "my-terraform-state"
    key            = "state/terraform.tfstate"
    region         = "us-east-1"
    encrypt        = true
    dynamodb_table = "terraform-lock"
  }
}

This setup:

Stores the state file in AWS S3.
Encrypts the state file for security.
Uses DynamoDB for state locking to prevent simultaneous updates.

💡 Tip: If you're using Terraform Cloud, it provides a built-in remote state storage and locking mechanism.

Step 2 — Use Modules to Keep Code DRY

One of the biggest Terraform mistakes is copy-pasting 🍝 infrastructure code across different projects. Instead, you should use Terraform modules to keep your configurations reusable and maintainable.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

For example, instead of writing the same code for every EC2 instance, you can create a reusable module:

modules/
  ec2-instance/
    main.tf
    variables.tf
    outputs.tf

Example: A Simple EC2 Module (`main.tf`)

resource "aws_instance" "web" {
  ami           = var.ami_id
  instance_type = var.instance_type
  tags = {
    Name = var.name
  }
}

Then, in your main Terraform configuration, you can call the module:

module "web_server" {
  source        = "./modules/ec2-instance"
  ami_id        = "ami-12345678"
  instance_type = "t2.micro"
  name          = "web-server"
}

Benefits of using modules:

Define infrastructure once and reuse it.
Modify a single module rather than updating multiple files.
Keeps your main configuration files clean.

Step 3 — Implement Terraform Workspaces for Multi-Environment Deployments

When managing multiple environments (e.g., development, staging, production), many people initially copy-paste Terraform files. This leads to configuration drift and inconsistency.

A better approach is to use Terraform workspaces, which allow you to manage multiple environments with the same Terraform code.

Example: Switching Workspaces

terraform workspace new staging
terraform workspace list
terraform workspace select staging

Inside your Terraform configuration, use terraform.workspace:

resource "aws_s3_bucket" "example" {
  bucket = "my-app-${terraform.workspace}"
}

This automatically creates different S3 buckets based on the workspace (my-app-dev, my-app-staging, etc.).

💡 Tip: If you need more complex environment configurations, consider using separate state files rather than workspaces.

Step 4 — Lock Provider Versions to Avoid Unexpected Breakages

Terraform providers are updated frequently, and sometimes these updates introduce breaking changes. If you don’t lock provider versions, your infrastructure might suddenly stop working when running terraform apply.

Example: Locking Provider Versions

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.0" # Locks to major version 5
    }
  }

  required_version = ">= 1.5.0"
}

By doing this:

You prevent unexpected changes when running Terraform commands.
Your infrastructure remains stable across deployments.

💡 Tip: Always test updates in a separate branch before upgrading provider versions in production.

Step 5 — Use Terraform Validate and Format Before Applying Changes

Before running terraform apply, it's good practice to validate and format your Terraform configuration to catch issues early.

Example: Checking Syntax with `terraform fmt` and `terraform validate`

terraform fmt  # Automatically formats code
terraform validate  # Checks for syntax errors

You can also automate this in a CI/CD pipeline to ensure code consistency:

name: Terraform CI

on: [push]

jobs:
  validate:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Setup Terraform
      uses: hashicorp/setup-terraform@v1
    - name: Validate Terraform Code
      run: terraform validate

Using these commands ensures:

Your Terraform code follows best practices.
You catch issues before applying changes.
Your team follows a consistent formatting style.

Bonus Tip: Keep Secrets Secure with Environment Variables

Terraform configurations often require API keys, passwords, and database credentials. Hardcoding secrets in your .tf files is a major security risk.

Instead, use environment variables or a secrets manager.

Example: Using Environment Variables

export TF_VAR_db_password="supersecretpassword"
terraform apply

In Terraform:

variable "db_password" {}

resource "aws_db_instance" "example" {
  password = var.db_password
}

Better yet, use HashiCorp Vault or some other managed secrets managers for managing secrets securely.

Conclusion

These five best practices: using remote state, leveraging modules, implementing workspaces, locking provider versions, and validating Terraform code, will help you manage your infrastructure more efficiently.

If you're looking to dive deeper into Terraform, check out my ebook:

👉 Introduction to Terraform

And if you're setting up your Terraform infrastructure on DigitalOcean, you can get $200 in free credits to get started!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Happy Terraforming! 🚀

7 Best Terraform Books for Beginners

Bobby Iliev — Fri, 24 Jan 2025 06:42:47 -0800

Terraform is a great tool for managing infrastructure as code, but getting started can be overwhelming without the right resources. Whether you're a developer, sysadmin, or cloud enthusiast, these beginner-friendly books and guides will help you take your first steps with Terraform.

I've included a mix of free and paid options to suit every learning style. Let's dive in!

1. Terraform Up & Running by Yevgeniy Brikman (Paid)

This is one of the most popular books for learning Terraform. It provides a comprehensive introduction to Terraform and progresses into advanced concepts like managing state, creating reusable modules, and scaling multi-cloud environments. The hands-on examples make it perfect for beginners and intermediate users.

2. Introduction to Terraform (Paid)

This ebook is an excellent resource for developers, especially those working with DigitalOcean. It's beginner-friendly and focuses on using Terraform to manage infrastructure in a simple and cost-effective way. With clear examples for DigitalOcean users, it's perfect for learning how to automate and scale resources effortlessly.

Key benefits of learning with DigitalOcean:

Ease of use: Terraform and DigitalOcean are a great match for beginners.
Affordability: DigitalOcean's low-cost infrastructure makes it ideal for experimentation.
Practical examples: Includes step-by-step guides to create and manage resources like droplets and Kubernetes clusters.

👉 Get the ebook on Leanpub

3. Terraform in Action by Scott Winkler (Paid)

This hands-on guide covers everything from Terraform basics to advanced topics like testing, CI/CD, and custom providers. It's a great resource for anyone looking to take their Terraform skills to the next level, with real-world examples that make complex concepts easier to grasp.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

4. HashiCorp's Terraform Documentation (Free)

While not a book, the official Terraform documentation is an incredible free resource. It's beginner-friendly and regularly updated, covering everything from basic usage to advanced features like modules, backends, and provider configuration.

👉 Check out the official docs

5. The Terraform Book by James Turnbull (Paid)

This practical guide walks you through setting up your Terraform environment, creating reusable configurations, and managing infrastructure as code. It's straightforward and perfect for beginners who want to build a strong foundation in Terraform.

6. Automating Multi-Cloud Infrastructure with Terraform by Mikael Krief (Paid)

If you're interested in multi-cloud environments, this book is a must-read. It explains how to use Terraform to manage resources across AWS, Azure, and Google Cloud, while emphasizing best practices for scalability and maintainability.

7. Terraform Best Practices (Free)

This free online resource is a must-read for anyone looking to adopt best practices when working with Terraform. Whether you're a beginner or an experienced user, this guide provides actionable advice to help you write clean, efficient, and maintainable Terraform code.

👉 Get the book on here

Conclusion

Terraform makes managing infrastructure simple and efficient, and the right resources can make learning it much easier. Whether you start with the comprehensive Terraform Up & Running, dive into Introduction to Terraform for a DigitalOcean-focused approach, the key is to get started and experiment.

If you're looking for a cloud provider to practice with, DigitalOcean is a fantastic option. Use my DigitalOcean referral link to get $200 in free credit, perfect for building and testing your Terraform skills.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

What are your favorite Terraform resources? Let me know in the comments below! 🌍✨

9 Best Linux Books for Beginners

Bobby Iliev — Wed, 08 Jan 2025 00:13:07 -0800

Learning Linux can feel like a daunting task, especially if you're just starting out. But the right resources can make all the difference.

Whether you’re a developer, system administrator, or simply someone curious about Linux, these beginner-friendly books will guide you through the essentials. I've included a mix of free and paid options, so there’s something for everyone!

1. 101 Linux Commands (Free)

This free ebook is perfect for Linux beginners who want to build a strong foundation. It covers 101 essential Linux commands with practical examples, making it a must-have guide for anyone getting started with the command line.

2. Introduction to Linux (Paid)

If you want to dive deeper into Linux basics, this ebook is for you. It covers the fundamentals of Linux, including installation, file systems, and permissions. It's ideal for beginners looking to build a solid understanding of Linux.

👉 Get the book on Leanpub

3. Introduction to Bash Scripting (Free)

Learning Linux often starts with learning Bash scripting. This free ebook covers the basics of Bash scripting, from writing your first script to automating tasks. It’s beginner-friendly and great for anyone looking to become more efficient with Linux.

4. The Linux Command Line by William Shotts (Free)

This classic guide is free to download and covers the Linux command line in detail. It starts with the basics and progresses to more advanced topics like scripting. Perfect for anyone who wants to become proficient in using the Linux terminal.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

👉 Download it here

5. How Linux Works by Brian Ward (Paid)

This highly recommended book explains the inner workings of Linux. It covers everything from boot processes to file systems and is an excellent resource for those who want to understand how Linux operates behind the scenes.

6. Linux for Beginners by Jason Cannon (Paid)

Aimed at beginners, this book breaks down complex Linux concepts into easy-to-understand lessons. It covers basic commands, file management, and shell scripting, making it a great resource for those just starting out.

7. Linux Bible by Christopher Negus (Paid)

The Linux Bible is a comprehensive guide that’s great for beginners and intermediate users alike. It covers everything from basic commands to advanced system administration tasks. This book is a trusted resource for mastering Linux.

8. GNU/Linux Advanced Administration by Remo Suppi Boldrito (Free)

This comprehensive guide is ideal for those who already have some Linux knowledge and want to level up their skills. It’s packed with practical tips and is freely available online.

👉 Download it here

9. Linux Official Documentation (Free)

Not exactly a book, but the Linux Documentation Project (LDP) offers free, high-quality guides for every skill level. From beginner tutorials to advanced topics, the LDP is a go-to resource for Linux learners.

👉 Check out the documentation

Conclusion

Linux is an incredibly powerful operating system, and with the right resources, anyone can learn to use it effectively. Whether you’re starting with free guides like 101 Linux Commands and The Linux Command Line or diving into paid guides like Linux Bible or How Linux Works, there’s something here for every skill level.

Remember, the best way to learn Linux is by experimenting. If you need a server to test on, you can use my DigitalOcean referral link to get $200 in free credit, perfect for setting up a Linux environment.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Have you read any of these books or know of other great Linux resources? Let me know in the comments. Happy learning! 🐧

5 Best Docker Books for Beginners

Bobby Iliev — Wed, 27 Nov 2024 06:02:37 -0800

Getting started with Docker can feel overwhelming, but having the right resources makes all the difference.

Whether you're a developer, sysadmin, or just curious about containerization, these beginner-friendly books and guides will help you take your first steps with Docker.

I’ve included a mix of free and paid options to fit different budgets, so there’s something for everyone!

1. Introduction to Docker (Free)

This free ebook is written specifically for beginners and covers Docker from the ground up. It starts with the basics—like what containers are—and walks you through creating and managing your first Docker containers. Since it’s available as a free download on GitHub, it’s a great starting point for anyone exploring Docker.

2. The Docker Handbook by Farhan Hasin Chowdhury (Free)

Published by FreeCodeCamp, this is one of the best free resources for beginners. It’s straightforward and filled with practical examples. The book explains core Docker concepts and takes you through building and running containers, creating images, and even touches on Docker Compose.

👉 Read it on FreeCodeCamp

3. Docker Deep Dive by Nigel Poulton (Paid)

If you’re serious about learning Docker, this is one of the most comprehensive books available. Nigel Poulton explains Docker concepts in plain language and includes hands-on exercises to help you practice. Whether you’re learning about containers, volumes, or Docker Swarm, this book has you covered.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

4. The Docker Book by James Turnbull (Paid)

James Turnbull’s The Docker Book is a classic for Docker beginners. It offers a deep dive into the Docker ecosystem, from setting up your environment to deploying applications in containers. The book also introduces advanced concepts like networking and scaling containers.

5. Docker in Action by Jeff Nickoloff and Stephen Kuenzli (Paid)

If you’re looking to go beyond the basics, Docker in Action is an excellent resource. The book combines beginner-friendly explanations with more advanced examples, like building CI/CD pipelines with Docker. It’s perfect for those who want to build practical Docker skills.

6. Docker Official Documentation (Free)

Not exactly a book, but the Docker Docs are one of the best free resources available. They’re beginner-friendly, regularly updated, and include tutorials on everything from installation to advanced use cases. If you’re ever stuck, the official docs are a reliable place to look for answers.

👉 Check out the Docker Docs

Conclusion

Docker is a game-changer for developers and sysadmins, and learning it is easier than ever with the right resources. Whether you prefer free guides like Introduction to Docker and The Docker Handbook or want a deeper dive with books like Docker Deep Dive, the key is to just start!

The best way to learn Docker is by doing. If you don’t have a server to experiment with, you can use my DigitalOcean referral link to get $200 in free credit. This is perfect for spinning up a droplet and trying out Docker in a real-world environment.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

If you’ve read any of these or have other Docker books to recommend, let me know in the comments. Happy containerizing! 🚀

How To Consume an External API with Express.js

Bobby Iliev — Mon, 18 Nov 2024 06:13:26 -0800

Introduction

In this guide, we'll learn how to use Node.js, Express, and Neon Postgres to build a simple application that consumes the QuizAPI to fetch tech-related questions and store them in a PostgreSQL database.

By the end of this guide, you'll be able to:

Use the axios library to make HTTP requests in Node.js.
Set up an Express server to handle routing.
Connect to a Neon Postgres database to store and retrieve data.
Deploy your application to a DigitalOcean Droplet.

We'll even include storing the correct answers, so you can build on this project to create a full-fledged quiz application.

Prerequisites

Before starting, make sure you have:

Node.js and npm installed on your system. You can download them here.
A Neon Postgres database. Sign up for a free account here.
A QuizAPI account with an API key. You can sign up here.
Optionally, a DigitalOcean Droplet for deployment. Use my affiliate link to get $200 in free credit: DigitalOcean Free Credit.

If you're new to setting up a Node.js application on a server, check out this guide: How to Deploy a Node.js App on DigitalOcean.

Project Setup

Let's get started by creating a new Node.js project and setting up the necessary dependencies.

Step 1: Initialize the Project

Open your terminal and create a new directory for your project:

mkdir node-quiz-api
cd node-quiz-api
npm init -y

The -y flag automatically fills in the default settings in your package.json file.

Step 2: Install Dependencies

We need a few libraries to make our project work:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

npm install express axios pg dotenv

Here's a quick overview of the packages:

express: A minimalist web framework for Node.js.
axios: A popular HTTP client for making API requests.
pg: PostgreSQL client to connect to our Neon database.
dotenv: Loads environment variables from a .env file.

Step 3: Setting Up the Database

We'll use Neon Postgres to store our quiz data. Make sure you’ve created your database and have the connection details ready.

Create a `.env` File

In the root of your project, create a file named .env:

DATABASE_URL=postgres://<username>:<password>@<host>:<port>/<database>
QUIZ_API_KEY=YOUR_QUIZ_API_KEY

Replace , , , , and with your actual Neon credentials. Don't forget to replace YOUR_QUIZ_API_KEY with your QuizAPI key.

What is the `.env` file?

The .env file allows you to securely store sensitive information like your database credentials and API keys. It prevents hardcoding these details in your code, reducing security risks.

Step 4: Creating the Database Table

To store questions and their answers, we'll create a table in our Neon database.

Connect to Your Database

You can connect to your Neon Postgres database using a tool like psql, DBeaver, or any other Postgres client.

Create the `questions` Table

Run the following SQL command:

CREATE TABLE questions (
    id SERIAL PRIMARY KEY,
    question TEXT NOT NULL,
    answer_a TEXT,
    answer_b TEXT,
    answer_c TEXT,
    answer_d TEXT,
    correct_a BOOLEAN,
    correct_b BOOLEAN,
    correct_c BOOLEAN,
    correct_d BOOLEAN,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

Explanation of the Table Structure:

id: A unique identifier for each question.
question: The quiz question text.
answer_a, answer_b, answer_c, answer_d: Possible answer options.
correct_a, correct_b, correct_c, correct_d: Boolean values indicating whether each answer is correct.
created_at: A timestamp for when the question was added.

Step 5: Setting Up the Project Files

Let's create the file structure for our application:

node-quiz-api
├── .env
├── app.js
├── db.js
└── package.json

`db.js`

This file handles our connection to the Neon Postgres database:

const { Pool } = require('pg');
require('dotenv').config();

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  ssl: { rejectUnauthorized: false },
});

module.exports = {
  query: (text, params) => pool.query(text, params),
};

Explanation:

We’re using the pg package to connect to our database.
The Pool class manages multiple connections efficiently.
We use dotenv to load environment variables from our .env file.

`app.js`

This file sets up our Express server, handles requests to the QuizAPI, and stores the results in our database.

const express = require('express');
const axios = require('axios');
const db = require('./db');
require('dotenv').config();

const app = express();
const PORT = process.env.PORT || 3000;

// Fetch and store questions from QuizAPI
app.get('/fetch', async (req, res) => {
  try {
    const response = await axios.get('https://quizapi.io/api/v1/questions', {
      params: {
        apiKey: process.env.QUIZ_API_KEY,
        limit: 10,
      },
    });

    const quizzes = response.data;

    for (const quiz of quizzes) {
      const { question, answers, correct_answers } = quiz;

      await db.query(
        `INSERT INTO questions 
          (question, answer_a, answer_b, answer_c, answer_d, 
           correct_a, correct_b, correct_c, correct_d)
         VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9)`,
        [
          question,
          answers.answer_a,
          answers.answer_b,
          answers.answer_c,
          answers.answer_d,
          correct_answers.answer_a_correct === 'true',
          correct_answers.answer_b_correct === 'true',
          correct_answers.answer_c_correct === 'true',
          correct_answers.answer_d_correct === 'true',
        ]
      );
    }

    res.send('Questions fetched and stored successfully!');
  } catch (error) {
    console.error('Error:', error.message);
    res.status(500).send('Failed to fetch questions.');
  }
});

// Retrieve all stored questions
app.get('/questions', async (req, res) => {
  try {
    const result = await db.query('SELECT * FROM questions ORDER BY id DESC');
    res.json(result.rows);
  } catch (error) {
    console.error('Error:', error.message);
    res.status(500).send('Failed to fetch questions.');
  }
});

app.listen(PORT, () => {
  console.log(`Server running on http://localhost:${PORT}`);
});

Explanation:

Fetching Data: We use axios to make a request to the QuizAPI.
Storing Data: We loop through the questions and store them in our Postgres database.
Retrieving Data: The /questions endpoint returns all stored questions.

Step 6: Running the Application

To run your application:

node app.js

Visit:

http://localhost:3000/fetch to fetch and store questions.
http://localhost:3000/questions to view the stored questions.

Step 7: Deploying Your Application to DigitalOcean

Now that your application is working locally, let's deploy it to a server so it's accessible to others. We'll use a DigitalOcean Droplet for this purpose.

If you're new to deploying Node.js applications, check out this comprehensive guide:
How To Set Up a Node.js Application for Production on Ubuntu 16.04.

Quick Deployment Steps:

Create a Droplet: Use my DigitalOcean link to get $200 in free credit and spin up an Ubuntu Droplet.
SSH into Your Droplet:
```
ssh root@your_droplet_ip
```
Install Node.js:

Follow the steps in the linked guide to install Node.js, Nginx, and set up a firewall.

Clone Your Project:

git clone https://github.com/your-username/node-quiz-api.git
cd node-quiz-api

Install Dependencies:
```
npm install
```
Set Up Environment Variables:

Create a .env file on the server and copy your local .env content into it.
Start Your Application:
```
node app.js
```
Use PM2 for Process Management:

To keep your Node.js app running in the background, install pm2:
```
npm install -g pm2
pm2 start app.js
```
Set Up Nginx:

Use Nginx as a reverse proxy to route incoming requests to your Node.js app. Follow the configuration steps in the DigitalOcean guide.
Access Your App:

Once everything is set up, you can access your application via your server's IP address.

Use my DigitalOcean link to get $200 in free credit!

Conclusion

Here is how our Node.js application consumes the QuizAPI and stores the data in the Neon Postgres database:

                +------------------------------------+
                |          QuizAPI Service           |
                | https://quizapi.io/api/v1/questions|
                +------------------------------------+
                                ^
                                |
                      (HTTP GET Request)
                                |
                                v
   +------------------------------------------------------+
   |                    Node.js Application               |
   |------------------------------------------------------|
   | 1. Express server receives request at '/fetch' route |
   | 2. Uses Axios to fetch questions from QuizAPI        |
   | 3. Processes the response data                       |
   | 4. Stores questions and answers in the database      |
   +------------------------------------------------------+
                                |
                                |
                      (Database Query)
                                |
                                v
        +----------------------------------------+
        |         Neon Postgres Database          |
        |----------------------------------------|
        | Stores:                                |
        | - Question text                        |
        | - Answer options (A, B, C, D)          |
        | - Correct answer flags (true/false)    |
        +----------------------------------------+
                                ^
                                |
                      (Database Query)
                                |
                                v
   +----------------------------------------------------------+
   |  Node.js Application retrieves questions via '/questions'|
   +----------------------------------------------------------+
                                |
                                |
                                v
                  +-------------------------------+
                  |       Client Browser          |
                  |   (Displays stored data)      |
                  +-------------------------------+

With that we've built a Node.js application that fetches data from an external API and stores it in a Neon Postgres database. You can now extend this project by adding a frontend, implementing authentication, or even creating a full quiz game.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

How to Create Multi-Platform Docker Images

Bobby Iliev — Mon, 18 Nov 2024 05:29:25 -0800

Building Docker images that work on different CPU architectures like ARM and x86 is becoming more important, especially with the rise of ARM-based machines like M1/M2 Macs and Raspberry Pis. In this guide, I'll show you how to build images that run everywhere!

What are Multi-Platform Images?

Multi-platform images are Docker images that can run on different CPU architectures. Instead of building separate images for each architecture, we can create a single image that works across:

AMD64 (regular Intel/AMD computers)
ARM64 (M1/M2 Macs, newer Raspberry Pis)
ARMv7 (older Raspberry Pis)
And more!

Prerequisites

Before we start, you'll need:

Docker installed on your machine
Docker BuildKit enabled (it's enabled by default in newer Docker versions)
Docker Hub account (or another registry) to push your images

Setting Up BuildX

BuildX is Docker's tool for building multi-platform images. Let's set it up:

Create a new builder instance:

docker buildx create --name mybuilder --use

Start the builder:

docker buildx inspect --bootstrap

Creating a Simple Multi-Platform Image

Let's create a basic web server as an example. First, make a new directory:

mkdir multi-arch-demo
cd multi-arch-demo

Create a simple index.html:

echo "<h1>Hello from $(uname -m)h1>" > index.html

Now create a Dockerfile:

FROM nginx:alpine
COPY index.html /usr/share/nginx/html/index.html

Building for Multiple Platforms

Here's the cool part! Build and push the image for multiple platforms in one command:

docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 \
  -t yourusername/multi-arch-demo:latest \
  --push .

This command:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Builds for three different architectures
Tags the image with your username
Pushes it to Docker Hub

Testing Your Multi-Platform Image

To test your image, you can pull and run it on different devices:

# On a regular computer (AMD64)
docker run -p 80:80 yourusername/multi-arch-demo:latest

# On a Raspberry Pi (ARM)
docker run -p 80:80 yourusername/multi-arch-demo:latest

# On an M1/M2 Mac (ARM64)
docker run -p 80:80 yourusername/multi-arch-demo:latest

Docker automatically picks the right version for each device!

Real-World Example: Python Application

Let's create a more practical example with a Python application:

Create a simple app.py:

from flask import Flask
import platform

app = Flask(__name__)

@app.route('/')
def hello():
    return f"Hello from {platform.machine()}!"

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=5000)

Create a requirements.txt:

flask==2.0.1

Create a multi-platform Dockerfile:

FROM --platform=$BUILDPLATFORM python:3.9-slim

WORKDIR /app
COPY requirements.txt .
RUN pip install -r requirements.txt

COPY app.py .
EXPOSE 5000

CMD ["python", "app.py"]

Build and push for multiple platforms:

docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7 \
  -t yourusername/python-multi-arch:latest \
  --push .

Tips and Tricks

Cache Management

docker buildx build --platform linux/amd64,linux/arm64 \
  --cache-from type=registry,ref=yourusername/myapp:cache \
  --cache-to type=registry,ref=yourusername/myapp:cache \
  -t yourusername/myapp:latest \
  --push .

Checking Available Platforms

docker buildx ls

Inspecting Multi-Platform Images

docker buildx imagetools inspect yourusername/multi-arch-demo:latest

Common Issues and Solutions

No space left on device Solution: Clean up your builder:

docker buildx prune

QEMU errors Solution: Install QEMU support:

docker run --privileged --rm tonistiigi/binfmt --install all

Build taking too long Solution: Build only the platforms you need:

docker buildx build --platform linux/amd64,linux/arm64 \
  -t yourusername/myapp:latest \
  --push .

Conclusion

Multi-platform Docker images are super helpful when you need your application to run on different types of computers. With BuildX, it's pretty straightforward to create images that work everywhere. The key things to remember are:

Use BuildX for multi-platform builds
Test your images on different architectures
Only build for the platforms you actually need
Keep your images small and efficient

Now you can build Docker images that run on any device, from your laptop to a Raspberry Pi!

And if you're new to Docker, I've written a free Docker ebook that covers all the basics - you can check it out at Introduction to Docker eBook. The ebook will help you get started with Docker before diving into multi-architecture builds!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Give it a try and let me know if you have any questions. You can reach me on Twitter @bobbyiliev_.

How To Use Pan for Easy Analytics in Your Laravel App

Bobby Iliev — Thu, 17 Oct 2024 05:19:59 -0700

Introduction

When you're building a website or an app, knowing how users interact with it is key to improving the experience. However, many tracking tools are complicated, costly, or don't care about privacy. That's where Pan steps in. Pan is a simple, privacy-friendly tool for tracking user behavior on your Laravel app, created by Nuno Maduro and David Hill from the Laravel team.

In this guide, I'll show you how to integrate Pan into your Laravel project and track things like views, hovers, and clicks without collecting personal data from your users.

What You'll Need

Before we jump in, make sure you have the following ready:

PHP 8.3 or higher
A Laravel 11+ project set up
Composer installed (helps manage your project dependencies)

Check out this intro video by Josh Cirre:

Step 1 — Installing Pan

First up, let's add Pan to your Laravel project. You’ll do this using Composer:

Open your terminal.
Navigate to your Laravel project folder.
Run this command:

composer require panphp/pan

This will install Pan into your project.

Now, run this command to set up Pan:

php artisan install:pan

This command will configure everything for you!

With Pan installed, you can now track parts of your website. Pan uses a data-pan attribute to track specific elements.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

For example, to track when someone clicks or hovers over a "Sign Up" button, you’d add this:

<button data-pan="sign-up-button">Sign Upbutton>

Here are more examples:

<div data-pan="hero-section">Welcome to our site!div>
<a data-pan="learn-more-link" href="/about">Learn Morea>
<form data-pan="contact-form">
  
form>

Pan will automatically start tracking impressions, hovers, and clicks for these elements!

Step 3 — View Your Tracking Data

Now that Pan is tracking your elements, you can view the data it collects using Artisan:

Open your terminal.
Go to your project folder.
Run this command:

php artisan pan

You'll get a table showing how many times each element has been viewed, hovered over, or clicked.

If you want to check a specific element, use:

php artisan pan --filter=sign-up-button

Replace sign-up-button with the element you’re tracking.

Step 4 — Customize Pan (Optional)

By default, Pan tracks up to 50 elements, but you can change this if needed. To customize Pan, open the AppServiceProvider.php file and add:

use Pan\PanConfiguration;

public function register(): void
{
    // Track specific elements
    PanConfiguration::allowedAnalytics([
        'sign-up-button',
        'contact-form',
    ]);

    // Or increase the limit
    // PanConfiguration::maxAnalytics(10000);

    // Or remove the limit entirely
    // PanConfiguration::unlimitedAnalytics();
}

Step 5 — What Pan Tracks

Here’s what Pan tracks:

Impressions: Counts how many times an element is seen.
Hovers: Counts how many times users hover over an element.
Clicks: Counts how many times users click on an element.

Pan is completely privacy-friendly—it doesn’t collect any personal data about your users.

Step 6 — Use the Data to Improve

With Pan tracking how users interact with your site, you can use that data to improve your site. For example:

If a button isn’t clicked much, maybe it needs more attention or clearer text.
If users are hovering over an element but not clicking, maybe it’s confusing.
If some sections aren’t being viewed, think about moving them to more visible areas.

Step 7 — Clear Tracking Data

If you ever need to clear Pan's tracking data, it’s easy:

Open your terminal.
Navigate to your project folder.
Run this command:

php artisan pan:flush

This will reset Pan's tracking data, so you can start fresh.

Conclusion

Pan makes it easy to understand how users interact with your Laravel app without compromising their privacy.

By adding the data-pan attribute and using Artisan commands, you can track user behavior and make informed improvements to your website or app.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Lockdown Your Containers: 11 Essential Docker Security Tips

Bobby Iliev — Sat, 05 Oct 2024 06:58:14 -0700

Introduction

Hey! I'm Bobby, a Docker Captain and the author of the free Introduction to Docker eBook. In this article, we'll cover 11 essential Docker security tips to help you protect your containerized applications.

Prerequisites

To follow this guide, you should have:

Docker installed on your system
Basic knowledge of Docker commands and concepts

1. Keep Docker Updated

Regularly updating your Docker engine is crucial for maintaining the security of your containerized environment. Each new release of Docker often includes security patches and fixes for vulnerabilities discovered in previous versions.

To update Docker on a Ubuntu system, you can use the following commands:

sudo apt-get update
sudo apt-get upgrade docker-ce

After updating, it's a good practice to restart the Docker daemon:

sudo systemctl restart docker

Make sure to test your applications after updating Docker to ensure compatibility with the new version.

2. Use Official Images

Official images from Docker Hub are maintained by the Docker team and the original software maintainers. They follow best practices for Docker image creation and are regularly updated with security patches.

When using official images in your docker-compose.yml file, always specify a specific version tag rather than using latest:

version: '3.8'
services:
  web:
    image: nginx:1.21.3  # Specific version of the official Nginx image

Using specific version tags ensures reproducibility and prevents unexpected changes when rebuilding your containers.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

3. Scan Images for Vulnerabilities

Regularly scanning your Docker images for vulnerabilities is essential for maintaining a secure environment. Docker Scout is a powerful tool integrated into Docker Desktop and the Docker CLI for this purpose.

To scan an image using Docker Scout:

docker scout cve :

For example:

docker scout cve nginx:1.21.3

This command will provide a detailed report of any known vulnerabilities in the image, including their severity levels and available fixes.

You can also integrate Docker Scout into your CI/CD pipeline. Here's an example of how you might do this in a GitHub Actions workflow:

name: Docker Image CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Build the Docker image
      run: docker build . --file Dockerfile --tag myapp:${GITHUB_SHA}
    - name: Scan the Docker image
      run: docker scout cve myapp:${GITHUB_SHA}

This workflow builds your Docker image and then scans it for vulnerabilities on every push to the repository.

4. Limit Container Resources

Setting Docker resource limits on your containers is crucial for preventing Denial of Service (DoS) attacks and ensuring fair resource allocation in multi-container environments.

In your docker-compose.yml file, you can set these limits as follows:

version: '3.8'
services:
  web:
    image: nginx:1.21.3
    deploy:
      resources:
        limits:
          cpus: '0.50'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 256M

This configuration limits the web service to use a maximum of 0.5 CPU cores and 512MB of memory, while reserving a minimum of 0.25 CPU cores and 256MB of memory.

You can also set these limits when running a container using the Docker CLI:

docker run -d --name myapp --cpus=0.5 --memory=512m nginx:1.21.3

By setting these limits, you prevent a single container from consuming all available resources on the host, which could affect other containers or the host system itself.

5. Use Non-Root Users

Running containers as root is a significant security risk. If an attacker manages to break out of the container, they would have root access to the host system. To mitigate this risk, create a non-root user in your Dockerfile and switch to this user.

Here's an example Dockerfile that creates a non-root user:

FROM node:18
# Create app directory
WORKDIR /usr/src/app

# Install app dependencies
COPY package*.json ./
RUN npm ci --only=production

# Bundle app source
COPY . .

# Create a non-root user and switch to it
RUN groupadd -r myapp && useradd -r -g myapp myuser
USER myuser

EXPOSE 8080
CMD [ "node", "server.js" ]

In this Dockerfile, we create a new user myuser and a group myapp, then switch to this user before running the application. This ensures that the application runs with limited permissions.

When running the container, you can also use the --user flag to specify a non-root user:

docker run -d --name myapp --user 1000:1000 myimage

This runs the container as the user with UID 1000 and GID 1000.

6. Implement Secret Management

Storing sensitive information like passwords, API keys, and other secrets directly in your Dockerfile or Docker Compose file is a security risk. Instead, use Docker secrets for managing this sensitive data.

First, create a secret:

echo "mysecretpassword" | docker secret create db_password -

Then, in your docker-compose.yml file, you can use this secret:

version: '3.8'
services:
  db:
    image: mysql:8.0
    secrets:
      - db_password
    environment:
      MYSQL_ROOT_PASSWORD_FILE: /run/secrets/db_password
secrets:
  db_password:
    external: true

In this example, MySQL will read the root password from the file /run/secrets/db_password inside the container, which contains the secret we created.

For non-swarm mode, you can use environment variables and a .env file:

version: '3.8'
services:
  db:
    image: mysql:8.0
    environment:
      MYSQL_ROOT_PASSWORD: ${DB_PASSWORD}

And in your .env file (which should be added to .gitignore):

DB_PASSWORD=mysecretpassword

This approach keeps secrets out of your version-controlled files while still allowing easy configuration.

7. Enable Content Trust

Docker Content Trust (DCT) allows you to verify the integrity and the publisher of all the data received from a registry over any channel. When DCT is enabled, you can be sure that the image you're pulling is the one that was pushed, without any tampering.

To enable DCT, set the DOCKER_CONTENT_TRUST environment variable:

export DOCKER_CONTENT_TRUST=1

With DCT enabled, when you push an image, Docker signs it:

docker push myrepo/myimage:latest

When pulling images with DCT enabled, Docker will only pull signed images:

docker pull myrepo/myimage:latest

If the image isn't signed or the signature doesn't match, the pull will fail.

You can also enable DCT in your Docker daemon configuration file (/etc/docker/daemon.json):

{
  "content-trust": {
    "mode": "enforced"
  }
}

This enforces DCT for all operations on the Docker daemon.

8. Use Read-Only Containers

Running containers in read-only mode prevents attackers from making changes to the container's filesystem, even if they manage to execute arbitrary code within the container.

In your docker-compose.yml file, you can set a container to read-only mode like this:

version: '3.8'
services:
  web:
    image: nginx:1.21.3
    read_only: true
    tmpfs:
      - /tmp
      - /var/cache/nginx

The tmpfs mounts are necessary for Nginx to function correctly, as it needs to write to these directories. These are temporary filesystems that exist only in memory.

When using the Docker CLI, you can use the --read-only flag:

docker run -d --name myapp --read-only nginx:1.21.3

For applications that need to write data, you can use volumes or bind mounts for specific directories that need write access, while keeping the rest of the filesystem read-only.

9. Implement Network Segmentation

Network segmentation in Docker helps to isolate containers and reduce the attack surface. By default, all containers on a Docker network can communicate with each other. However, you can create separate networks for different groups of containers.

Here's an example docker-compose.yml file that implements network segmentation:

version: '3.8'
services:
  frontend:
    image: nginx:1.21.3
    networks:
      - frontend
  backend:
    image: app:latest
    networks:
      - backend
  db:
    image: mysql:8.0
    networks:
      - backend
networks:
  frontend:
    driver: bridge
  backend:
    driver: bridge

In this setup, the frontend service cannot communicate directly with the db service, as they're on different networks. The backend service acts as a bridge between the two.

You can also create and manage networks using the Docker CLI:

# Create networks
docker network create frontend
docker network create backend

# Run containers on specific networks
docker run -d --name nginx --network frontend nginx:1.21.3
docker run -d --name app --network frontend --network backend app:latest
docker run -d --name mysql --network backend mysql:8.0

This network segmentation adds an extra layer of security by limiting the potential spread of an attack if one container is compromised.

10. Regular Security Audits

Regular security audits are crucial for maintaining the security of your Docker environment. Docker Bench for Security is a script that checks for dozens of common best practices around deploying Docker containers in production.

To run Docker Bench for Security:

docker run -it --net host --pid host --userns host --cap-add audit_control \
    -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
    -v /var/lib:/var/lib \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /usr/lib/systemd:/usr/lib/systemd \
    -v /etc:/etc --label docker_bench_security \
    docker/docker-bench-security

This command runs a series of checks and provides a report with recommendations for improving your Docker security posture.

It's a good practice to run this audit regularly, such as weekly or after any significant changes to your Docker environment. You can automate this process by integrating it into your CI/CD pipeline or setting up a cron job.

Here's an example of how you might integrate Docker Bench into a GitHub Actions workflow:

name: Docker Security Audit

on:
  schedule:
    - cron: '0 0 * * 0'  # Run weekly on Sunday at midnight

jobs:
  audit:
    runs-on: ubuntu-latest
    steps:
    - name: Check out code
      uses: actions/checkout@v2
    - name: Run Docker Bench for Security
      run: |
        docker run --net host --pid host --userns host --cap-add audit_control \
          -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
          -v /var/lib:/var/lib \
          -v /var/run/docker.sock:/var/run/docker.sock \
          -v /usr/lib/systemd:/usr/lib/systemd \
          -v /etc:/etc --label docker_bench_security \
          docker/docker-bench-security

This workflow runs Docker Bench for Security every week and provides a report in the GitHub Actions log.

11. Implement Logging and Monitoring

Proper logging and monitoring are essential for detecting and responding to security incidents in your Docker environment. Docker provides built-in logging mechanisms that you can configure for each container:

Docker Logging Drivers

In your docker-compose.yml file, you can set up logging like this:

version: '3.8'
services:
  web:
    image: nginx:1.21.3
    logging:
      driver: "json-file"
      options:
        max-size: "200m"
        max-file: "10"

This configuration uses the json-file logging driver, which is the default. It sets a maximum size of 200MB for each log file and keeps up to 10 log files.

For more advanced logging, you can use the syslog driver to send logs to a central syslog server:

logging:
  driver: syslog
  options:
    syslog-address: "udp://192.168.0.42:1234"

For monitoring, consider using tools like Prometheus and Grafana. Here's an example docker-compose.yml that sets up a basic monitoring stack:

version: '3.8'
services:
  prometheus:
    image: prom/prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"
  grafana:
    image: grafana/grafana
    depends_on:
      - prometheus
    ports:
      - "3000:3000"

You'll need to configure Prometheus to scrape metrics from your Docker daemon and containers. Here's a basic prometheus.yml configuration:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'docker'
    static_configs:
      - targets: ['docker.for.mac.localhost:9323']

To enable Docker metrics, you need to configure your Docker daemon. Add the following to /etc/docker/daemon.json:

{
  "metrics-addr" : "0.0.0.0:9323",
  "experimental" : true
}

Remember to restart the Docker daemon after making these changes.

With this setup, you can create dashboards in Grafana to visualize your Docker metrics and set up alerts for any suspicious activity.

Conclusion

Security is an ongoing process that requires regular attention and updates!

These 11 security tips will significantly help you with your container security and Docker environment but you should always keep an eye on the latest security best practices and updates.

For more information on Docker, check out my free Introduction to Docker eBook.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

If you're looking for a reliable platform to practice these security tips, consider using DigitalOcean. They offer a $200 free credit to get you started.

Optimizing Docker Image Sizes: Advanced Techniques and Tools

Bobby Iliev — Mon, 16 Sep 2024 01:51:52 -0700

Introduction

I'm Bobby, a Docker Captain and the author of the free Introduction to Docker eBook. Today, we're diving deep into a crucial aspect of Docker that can significantly impact your containerized applications' performance and resource utilization: optimizing Docker image sizes.

The efficiency of our Docker images is increasingly important, especially in cloud-native and microservices architectures where we might be deploying hundreds or thousands of containers. Every megabyte counts when you're scaling up!

Prerequisites

Before we embark on this image-slimming journey, make sure you have:

Docker installed on your system
Basic knowledge of Dockerfiles and Docker commands
Some experience building and running Docker containers

If you're new to Docker or need a refresher, I highly recommend checking out my free Introduction to Docker eBook. It covers all the fundamentals you'll need to follow along with this guide and will give you a solid foundation for understanding the optimization techniques we'll discuss.

Why Optimize Docker Image Sizes?

Before we dive into the specific aproaches, let's take a moment to understand why smaller Docker images are so beneficial:

Smaller images mean less data to transfer. This translates to quicker pulls from registries, faster deployments, and reduced time to scale up your applications.
In cloud environments, storage isn't free. Smaller images consume less storage space, which can lead to significant cost savings, especially when you're dealing with multiple images and frequent updates.
A smaller image means a reduced attack surface. With fewer packages and files, there's less opportunity for vulnerabilities to creep in. It's easier to audit and maintain the security of a lean image.

Our Amazing Sponsors

View Website
DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.
View Website
Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.
View Website
A Laravel Starter Kit that includes Authentication, User Dashboard, Edit Profile, and a set of UI Components. Learn more about the DevDojo sponsorship program and see your logo here to get your brand in front of thousands of developers.
Smaller images often lead to faster container startup times. In orchestrated environments like Kubernetes, where containers might be frequently started and stopped, this can make a big difference in overall system responsiveness.
In environments with limited resources, such as edge computing or IoT devices, every byte counts. Smaller images allow you to run more containers on the same hardware.

Now that we understand the importance of optimizing our Docker images, let's dive into the techniques that can help us achieve this goal.

1. Use Minimal Base Images

One of the most effective ways to reduce your Docker image size is to start with a minimal base image. The base image is the foundation of your Docker image, and choosing the right one can make a significant difference.

Instead of using a full-fledged operating system image like:

FROM ubuntu

Consider using a more lightweight alternative:

FROM alpine

Alpine Linux is a security-oriented, lightweight Linux distribution that's only 5MB in size. It's perfect for creating small, secure Docker images. However, it uses musl libc instead of glibc, which can occasionally cause compatibility issues with some applications.

For even more minimalism, especially for compiled languages, consider using distroless images:

FROM gcr.io/distroless/static-debian11

Distroless images contain only your application and its runtime dependencies. They don't contain package managers, shells, or any other programs you would expect to find in a standard Linux distribution. This makes them extremely small and secure.

Here's a comparison of base image sizes:

Ubuntu: ~72MB
Alpine: ~5MB
Distroless: ~2MB

To learn more about distroless images, check out this blog post: Is Your Container Image Really Distroless?.

By choosing the right base image, you can reduce your starting point from hundreds of MBs to just a few MBs. This sets the stage for a much smaller final image.

2. Multi-stage Builds: The Secret Sauce

Multi-stage builds are a game-changer for creating efficient Docker images. They allow you to use one image for building your application and another for running it. This technique is particularly powerful for compiled languages but can be useful for interpreted languages as well.

Here's an expanded example for a Go application:

# Build stage
FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
# Install any necessary dependencies
RUN go mod download
# Build the application
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .

# Final stage
FROM alpine:3.18
RUN apk --no-cache add ca-certificates
WORKDIR /root/
# Copy the pre-built binary file from the previous stage
COPY --from=builder /app/main .
# Command to run the executable
CMD ["./main"]

Let's break this down:

The first stage uses the golang:1.21-alpine image, which includes the Go compiler and tools. We build our application in this stage.
We use CGO_ENABLED=0 and GOOS=linux to create a statically linked binary that doesn't depend on C libraries. This allows us to use a very minimal runtime image.
The second stage starts from the Alpine image, which is much smaller than the Go image.
We only copy the compiled binary from the build stage to the final stage. All the Go development tools and source code are left behind.
We add ca-certificates to ensure our application can make HTTPS connections if needed.

This approach separates your build environment from your runtime environment, resulting in a much smaller final image. It's not uncommon to see image sizes reduce from 300-400MB to 10-20MB using this technique.

Multi-stage builds can be adapted for other languages too. For example, in a Node.js application, you might use one stage to install all dependencies and build your application, and another stage with only production dependencies for the final image. The same principle applies for Python, Java, and other languages.

3. Layer Optimization: Every Line Counts

In Docker, each instruction in your Dockerfile creates a new layer. These layers are cached, which can speed up builds, but they also contribute to the final image size. To minimize layers and optimize your image:

Combine RUN commands: Use && to chain commands and \ for line breaks to make your Dockerfile more readable. This reduces the number of layers and can significantly decrease your image size.

Instead of:
```
RUN apt-get update
RUN apt-get install -y package1
RUN apt-get install -y package2
RUN apt-get install -y package3
RUN rm -rf /var/lib/apt/lists/*
```
Use:
```
RUN apt-get update && apt-get install -y \
    package1 \
    package2 \
    package3 \
 && rm -rf /var/lib/apt/lists/*
```
This not only reduces the number of layers but also ensures that the package lists are updated and cleaned up in the same layer, preventing outdated package lists from persisting in your image.
Use COPY instead of ADD: COPY is more transparent in its behavior. Use ADD only when you specifically need its tar auto-extraction feature or copying remote files from URLs. For more information check out the documentation here.
Order instructions from least to most frequently changing: Docker caches layers, and this cache is invalidated when the content used in an instruction changes. By putting instructions that change frequently (like copying your application code) towards the end of your Dockerfile, you can take better advantage of the build cache for the layers that change less frequently.

For example:
```
FROM node:14-alpine
WORKDIR /app

# These layers change less frequently
COPY package*.json ./
RUN npm ci --only=production

# This layer changes most frequently
COPY . .

CMD ["node", "server.js"]
```
Use .dockerignore: This file works like .gitignore and prevents unnecessary files from being added to your build context. This can significantly speed up the build process and reduce the final image size.

Example .dockerignore:
```
.git
*.md
node_modules
npm-debug.log
```
This prevents the .git directory, Markdown files, the node_modules directory, and npm debug logs from being copied into your image.
Clean up in the same layer: When installing packages or downloading files, make sure to clean up in the same RUN instruction:
```
RUN wget https://example.com/big-file.tar.gz \
    && tar -xzf big-file.tar.gz \
    && make -C big-file \
    && rm -rf big-file big-file.tar.gz
```
This ensures that the downloaded and extracted files don't persist in the final image.

By carefully considering each instruction in your Dockerfile and how it affects layering, you can create more efficient, smaller images that are faster to build and deploy.

4. Use `.dockerignore`

The .dockerignore file is a powerful tool for optimizing your Docker builds and reducing image sizes. It works similarly to .gitignore, allowing you to specify which files and directories should be excluded from the Docker build context.

Here's why .dockerignore is crucial:

The build context is sent to the Docker daemon before the build starts. A smaller context means faster uploads and builds.
By excluding files that change frequently but aren't needed in the image, you can prevent unnecessary cache invalidation and speed up builds.
It helps prevent sensitive files (like .env files or ssh keys) from accidentally being included in your image.

Here's an example of a comprehensive .dockerignore file:

# Version control
.git
.gitignore

# Documentation
*.md
docs/

# Development artifacts
node_modules/
npm-debug.log
yarn-error.log
*.log
*.bak

# Build output
dist/
build/
*.exe
*.dll
*.so
*.dylib

# Editor and IDE files
.vscode/
.idea/
*.swp
*.swo

# OS generated files
.DS_Store
Thumbs.db

# Test files
test/
__tests__/
*.test.js

# Configuration files that shouldn't be in the image
.env
.env.*
config.local.js

# Docker files
Dockerfile
docker-compose.yml

By carefully crafting your .dockerignore file, you can significantly reduce your build context size and ensure that only the necessary files are included in your Docker image. This not only speeds up your build process but also helps in creating smaller, more focused images.

Remember to review and update your .dockerignore file regularly, especially as your project structure changes or grows.

5. Use Build Arguments for Flexibility

Build arguments provide a powerful way to create flexible and reusable Dockerfiles. They allow you to pass variables at build-time, enabling you to customize your image building process without maintaining multiple Dockerfiles.

Here's an in-depth look at how to use build arguments effectively:

Defining build arguments: In your Dockerfile, you can define build arguments using the ARG instruction:
```
ARG VERSION=latest
FROM base:${VERSION}
```
This sets a default value of "latest" for the VERSION argument, which can be overridden at build time.

Using build arguments: You can use build arguments in various ways throughout your Dockerfile:

ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}

ARG USER=nobody
USER ${USER}

ARG PORT=8080
EXPOSE ${PORT}

Passing build arguments: When building your image, you can pass values for your build arguments:
```
docker build --build-arg VERSION=18.04 --build-arg NODE_ENV=development -t myapp:custom .
```
Multi-stage builds with arguments: Build arguments can be particularly powerful in multi-stage builds:
```
ARG GO_VERSION=1.21
FROM golang:${GO_VERSION}-alpine AS builder
# ... build stage instructions ...

ARG ALPINE_VERSION=3.18
FROM alpine:${ALPINE_VERSION}
# ... final stage instructions ...
```
This allows you to easily update the versions of your base images by passing different build arguments.

Scoping build arguments: It's important to note that each FROM instruction in a multi-stage Dockerfile clears all ARGs defined before it. If you need an ARG in multiple stages, you need to redefine it:

ARG VERSION=latest
FROM base:${VERSION} AS builder
ARG VERSION  # Redefine to use in this stage
RUN echo "Building version: ${VERSION}"

FROM base:${VERSION} AS final
ARG VERSION  # Redefine again for this stage
LABEL version="${VERSION}"

Security considerations: Be cautious with sensitive information in build arguments. While they're not persisted in the final image like ENV variables, they are visible in the image history. For sensitive data, consider using Docker secrets or environment variables at runtime instead.

By using build arguments, you can create more flexible and maintainable Dockerfiles. This approach allows you to use the same Dockerfile for different environments (development, staging, production) or to easily update versions of base images and dependencies without modifying the Dockerfile itself.

6. Use Docker's BuildKit

BuildKit is Docker's next-generation build engine, offering improved performance, better caching, and more advanced features. It's designed to be faster and more efficient than the legacy builder, especially for complex build scenarios.

Here's a deep dive into using BuildKit for optimizing your Docker builds:

Enabling BuildKit: You can enable BuildKit by setting an environment variable:
```
export DOCKER_BUILDKIT=1
```
Or, for a single build:
```
DOCKER_BUILDKIT=1 docker build .
```
You can also enable it by default in the Docker daemon configuration file (/etc/docker/daemon.json):
```
{
  "features": {
    "buildkit": true
  }
}
```
Syntax directive: To use BuildKit-specific features, add this line at the top of your Dockerfile:
```
# syntax=docker/dockerfile:1.4
```
This enables the use of the latest Dockerfile syntax and BuildKit features.
Improved caching with --mount=type=cache: BuildKit introduces a new caching mechanism that can significantly speed up builds:
```
RUN --mount=type=cache,target=/root/.cache/go-build \
    go build -o myapp .
```
This caches the Go build cache between builds, greatly speeding up subsequent builds.
Secret mounting: BuildKit allows you to securely use secrets during build without them being stored in the final image:
```
RUN --mount=type=secret,id=mysecret cat /run/secrets/mysecret
```
You can then pass the secret at build time:
```
docker build --secret id=mysecret,src=path/to/secret.txt .
```
SSH agent forwarding: BuildKit can forward your SSH agent, allowing secure cloning of private repositories during build:
```
RUN --mount=type=ssh ssh-add -l && \
    git clone git@github.com:myorg/myrepo.git
```
Build with:
```
docker build --ssh default .
```
Parallel execution: BuildKit can execute independent build stages in parallel, potentially speeding up complex builds.
Better output formatting: BuildKit provides cleaner, more informative build output, making it easier to understand what's happening during the build process.
Inline cache storage: BuildKit can store cache metadata in the image itself, allowing you to push and pull cached layers along with your image:
```
docker build --cache-from myimage:latest --cache-to type=inline .
```
Multi-platform builds: BuildKit simplifies creating multi-architecture images:
```
docker buildx build --platform linux/amd64,linux/arm64 -t myimage:latest .
```

BuildKit is the future of Docker builds and offers many advanced features that can help optimize your build process. You can check out the official BuildKit documentation for more details.

7. Optimize for Your Specific Language

Different programming languages have different optimization techniques when it comes to Docker images. Here's an in-depth look at optimizing for some popular languages:

Node.js

Use npm ci instead of npm install:
```
COPY package*.json ./
RUN npm ci --only=production
```
npm ci is faster and more reliable for CI environments. It installs exact versions from package-lock.json.
Prune development dependencies:
```
RUN npm prune --production
```
This removes dev dependencies after your build step, significantly reducing image size.
Use the official Node.js Alpine image:
```
FROM node:14-alpine
```
Alpine-based images are much smaller than the default Node.js images.

Use multi-stage builds for front-end applications:

FROM node:14-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npm run build

FROM nginx:alpine
COPY --from=builder /app/build /usr/share/nginx/html

This builds your application in one stage and copies only the built assets to the final nginx image.

Python

Use pip install --no-cache-dir:
```
RUN pip install --no-cache-dir -r requirements.txt
```
This prevents pip from caching downloaded packages, reducing the image size.

Use virtual environments:

RUN python -m venv /opt/venv
ENV PATH="/opt/venv/bin:$PATH"
RUN pip install --no-cache-dir -r requirements.txt

This keeps your Python environment isolated and can help reduce size.

Use Python Alpine images:
```
FROM python:3.9-alpine
```
Alpine-based Python images are much smaller than the standard ones.
Compile Python bytecode:
```
RUN python -m compileall .
```
This can slightly reduce size and improve startup time.

Java

Use JDK for build, JRE for runtime:

FROM openjdk:11 AS builder
COPY . /app
RUN javac /app/Main.java

FROM openjdk:11-jre
COPY --from=builder /app/Main.class /app/
CMD ["java", "-cp", "/app", "Main"]

This uses the full JDK to compile, but only the JRE to run.

Use jlink to create a custom JRE:

FROM openjdk:11 AS builder
WORKDIR /app
COPY . .
RUN jlink --add-modules java.base,java.logging --output /javaruntime

FROM alpine:3.18
COPY --from=builder /javaruntime /opt/java
COPY --from=builder /app/Main.class /app/
ENV PATH="/opt/java/bin:${PATH}"
CMD ["java", "-cp", "/app", "Main"]

This creates a minimal JRE with only the modules your application needs.

Use Spring Boot's layered jars: For Spring Boot applications, use the layered jar feature to create more efficient Docker images:

FROM openjdk:11-jre as builder
WORKDIR application
COPY target/*.jar application.jar
RUN java -Djarmode=layertools -jar application.jar extract

FROM openjdk:11-jre
WORKDIR application
COPY --from=builder application/dependencies/ ./
COPY --from=builder application/spring-boot-loader/ ./
COPY --from=builder application/snapshot-dependencies/ ./
COPY --from=builder application/application/ ./
ENTRYPOINT ["java", "org.springframework.boot.loader.JarLauncher"]

Go

Build statically linked binaries:

FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
RUN CGO_ENABLED=0 GOOS=linux go build -a -installsuffix cgo -o main .

FROM scratch
COPY --from=builder /app/main /main
ENTRYPOINT ["/main"]

This creates a binary with no external dependencies, allowing the use of a scratch image.

Use Alpine for non-static binaries: If you need CGO or can't use static linking:

FROM golang:1.21-alpine AS builder
RUN apk add --no-cache git
WORKDIR /app
COPY . .
RUN go build -o main .

FROM alpine:3.18
RUN apk add --no-cache ca-certificates
COPY --from=builder /app/main /main
ENTRYPOINT ["/main"]

By optimizing your Dockerfile to the specific needs of your programming language and application, you can create highly optimized Docker images that are both small in size and efficient in runtime.

8. Regular Audits and Updates

Regularly auditing and updating your Docker images is crucial for maintaining efficiency, security, and performance. Here's a detailed look at how to approach this:

Use Docker Scout for vulnerability scanning: Docker Scout is a powerful tool for scanning your images for vulnerabilities:
```
docker scout cve your-image:tag
```
This command provides a detailed report of known vulnerabilities in your image.

Integrate scanning into your CI/CD pipeline: Here's an example of how you might integrate Docker Scout into a GitHub Actions workflow:

name: Docker Image CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Build the Docker image
      run: docker build . --file Dockerfile --tag my-image:$(date +%s)
    - name: Scan the Docker image
      run: docker scout cve my-image:$(date +%s)

Regularly update base images: Set up a process to regularly check for and test updates to your base images. You can automate this with tools like Dependabot for GitHub.
Monitor image size over time: Keep track of your image sizes over time. You can do this manually or set up a script to record image sizes after each build:
```
docker images --format "{{.Size}}\t{{.Repository}}:{{.Tag}}" | sort -h
```
Analyze image layers: Regularly inspect your image layers to identify any unexpected growth:
```
docker history your-image:tag
```
Use image digests for immutability: Instead of relying on tags, which can be overwritten, use image digests for important images:
```
docker pull your-image@sha256:a1b2c3...
```
Implement a retention policy: Regularly clean up old and unused images to save storage space:
```
docker image prune -a --filter "until=240h"
```
This removes images older than 10 days.
Stay informed about Docker updates: Keep your Docker engine and tools up to date, and stay informed about new features that might help optimize your workflows.
Perform regular security audits: Beyond just scanning for vulnerabilities, regularly review your Dockerfiles and build processes for security best practices.
Test optimized images thoroughly: After making optimizations, always thoroughly test your images to ensure they still function as expected in all required environments.

By implementing these practices, you can ensure that your Docker images remain efficient, secure, and up-to-date over time.

Conclusion

Optimizing Docker image sizes is an ongoing process that requires attention to detail and regular reviews. By implementing these techniques, you can significantly reduce your Docker image sizes, leading to faster deployments, reduced costs, and improved security.

Remember, the goal is to find the right balance between image size and functionality. Don't sacrifice necessary features just to make your image smaller!

If you're looking to dive deeper into Docker and container optimization, don't forget to check out my free Introduction to Docker eBook. It provides a solid foundation for working with Docker and will help you understand the concepts we've covered here in more depth.

Also, if you're setting up your Docker environment and need a reliable host, consider using DigitalOcean. You can get a $200 free credit to get started!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Happy Dockerizing! 🐳

Laravel Announces Major Updates at Laracon US 2024: Cloud, Inertia 2.0, and More

Bobby Iliev — Fri, 30 Aug 2024 01:23:39 -0700

Hello Laravel community! 👋

The Laracon US 2024 keynote by Taylor Otwell just wrapped up, and there are some exciting new features coming to the Laravel ecosystem. Let's break down the key announcements!

You can watch the video here if you prefer too:

1. Laravel Cloud ☁️

Laravel Cloud is set to change how we deploy Laravel applications:

🚀 Deploy from GitHub to a live URL in under a minute
🔄 Automatic scaling and load balancing
🐘 Serverless PostgreSQL databases that auto-scale
💤 App and database hibernation to reduce costs
🌿 Branch-based preview environments
🧑‍🔧 Integrated queue workers and job processing

This platform aims to make deployment fast and worry-free for Laravel developers.

Larave Forge which we all love, will still remain if this is your thing!

2. Inertia 2.0 ✨

Inertia.js is getting a major upgrade with version 2.0:

🏎️ Async requests for better performance
🔄 Easy data polling
👀 "When visible" lazy loading
🛝 Helpers for infinite scrolling
🏃‍♂️ Page prefetching for faster navigation
🚚 Deferred props to optimize initial page loads

These features will help create more responsive and faster-loading applications with Inertia.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

3. Official VS Code Extension 💻

I don't know about you but this is a huge thing for me! I wrote a post on the VS Code extensions that I am using and it will be great to update that post and just include that official VS Code extension!

8 Awesome VS Code Extensions for Laravel Developers

Laravel is releasing an official Visual Studio Code extension:

🧠 Enhanced autocompletion for Laravel code
🕵️ Improved error detection
🗺️ Easy navigation between related files
🧪 Built-in testing tools

This extension will make Laravel development in VS Code more efficient and user-friendly.

4. Framework Updates 🛠️

The improvements to the Framework are worth of a totally separate post themselves! But some of the the Laravel framework new features that were announce were:

🎯 Container attributes for simpler dependency injection
🏃‍♂️ Deferred functions for background processing without queues
🚦 Concurrency facade for parallel processing
🧊 Improved caching with flexible time-to-live options
🗄️ Local private file serving

I love the new deffered functions! A game changer for simple tasks where you don't want to introduce jobs and queues.

These updates focus on improving performance and simplifying common tasks.

5. Design Improvements 🎨

The Laravel team is also focusing on design:

Refreshing the Laravel.com website
Creating a new brand resource site
Improving the documentation experience

These changes aim to enhance the overall Laravel brand and make resources more accessible.

Conclusion

Laracon US 2024 has shown that Laravel continues to evolve and improve. From Laravel Cloud to Inertia 2.0, the VS Code extension, and core framework updates, there's a lot to look forward to in the Laravel ecosystem.

Keep an eye on the official Laravel channels for more information about release dates and how to get early access to Laravel Cloud.

What new feature are you most excited about? We'd love to hear your thoughts!

Containerizing a Microservices Architecture with Docker and Redpanda

Bobby Iliev — Wed, 28 Aug 2024 06:30:53 -0700

Introduction

Hello there!

I'm Bobby, a Docker Captain and the author of the free Introduction to Docker eBook. In this guide, we'll dive deep into containerizing a microservices architecture using Docker and Redpanda, a Kafka-compatible event streaming platform.

As we progress through 2024, microservices and event-driven architectures continue to be at the leading approach of modern application development, and Docker remains an indispensable tool for deploying and managing these complex systems.

Prerequisites

Before we go any further, make sure that you have:

Docker and Docker Compose installed on your system
Basic knowledge of Docker, microservices concepts, and event streaming

If you're new to Docker or need a refresher, I highly recommend checking out my free Introduction to Docker eBook. It covers all the fundamentals you'll need to follow along with this guide.

Our Example Architecture

For this article, we'll build a simple e-commerce platform with the following microservices:

Product Service (Go)
User Service (Python)
Order Service (Node.js)
API Gateway (Nginx)
Event Streaming Platform (Redpanda)
Database (PostgreSQL)

This architecture demonstrates a typical microservices setup, with each service responsible for a specific domain and communicating through both REST APIs and event streaming.

An example diagram of the architecture is shown below. The client apps (web, mobile, etc.) interact with the API Gateway, which routes requests to the appropriate microservices. The services communicate with a PostgreSQL database for persistent storage and with Redpanda for event streaming.

                  +-------------------+
                  |    Client Apps    |
                  | (Web, Mobile, etc)|
                  +---------+---------+
                            |
                            | HTTP/HTTPS
                            |
                  +---------v---------+
                  |                   |
                  |    API Gateway    |
                  |     (Nginx)       |
                  |                   |
                  +----+------+-------+
                       |      |
           +-----------+      +-----------+
           |                              |
     +-----v------+  +-----------+  +-----v------+
     |            |  |           |  |            |
     |  Product   |  |   User    |  |   Order    |
     |  Service   |  |  Service  |  |  Service   |
     |   (Go)     |  | (Python)  |  | (Node.js)  |
     |            |  |           |  |            |
     +-----+------+  +-----+-----+  +-----+------+
           |              |              |
           |              |              |
    +------v--------------v--------------v------+
    |                                           |
    |              PostgreSQL                   |
    |              Database                     |
    |                                           |
    +-------------------------------------------+
                       |
                       | (Event Streaming)
                       |
               +-------v-------+
               |               |
               |   Redpanda    |
               |               |
               +-------+-------+
                       |
                       | (External Systems)
                       |
               +-------v-------+
               | Notifications |
               |   Analytics   |
               | Materialize   |
               | Other Services|
               +---------------+

Step 1: Dockerizing Individual Services

Let's start by creating Dockerfiles for each service and include some basic code examples. We'll go through each service, explaining the Dockerfile and the code in detail.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Product Service (Go)

Dockerfile:

FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o main .

FROM alpine:3.18
COPY --from=builder /app/main /app/main
CMD ["/app/main"]

This Dockerfile uses a multi-stage build:

The first stage uses the golang:1.21-alpine image to build our Go application.
The second stage uses a minimal alpine:3.18 image and copies only the compiled binary from the first stage.
This approach significantly reduces the final image size by excluding all build tools and intermediate files.

Basic Go code (main.go):

package main

import (
    "encoding/json"
    "log"
    "net/http"
)

type Product struct {
    ID    string  `json:"id"`
    Name  string  `json:"name"`
    Price float64 `json:"price"`
}

func main() {
    http.HandleFunc("/products", func(w http.ResponseWriter, r *http.Request) {
        products := []Product{
            {ID: "1", Name: "Product 1", Price: 19.99},
            {ID: "2", Name: "Product 2", Price: 29.99},
        }
        json.NewEncoder(w).Encode(products)
    })

    log.Fatal(http.ListenAndServe(":8080", nil))
}

This Go code:

Defines a simple Product struct.
Creates an HTTP server that listens on port 8080.
Implements a /products endpoint that returns a JSON array of product data.
In a real-world scenario, this would interact with a database instead of using hardcoded data but for simplicity, we're using hardcoded data.

User Service (Python)

Dockerfile:

FROM python:3.11-slim
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY . .
CMD ["python", "app.py"]

This Dockerfile:

Uses the official Python 3.11 slim image as the base.
Sets the working directory to /app.
Copies and installs the Python dependencies first (for better caching).
Copies the rest of the application code.
Specifies the command to run the Python application.

Basic Python code (app.py):

from flask import Flask, jsonify

app = Flask(__name__)

@app.route('/users')
def get_users():
    users = [
        {"id": "1", "name": "John Doe"},
        {"id": "2", "name": "Jane Smith"}
    ]
    return jsonify(users)

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8081)

This Python code:

Uses Flask to create a simple web server.
Defines a /users endpoint that returns a JSON array of user data.
Runs the server on port 8081 and binds to all available network interfaces (0.0.0.0).
Like the product service, in a real-world scenario, this would interact with a database.

Order Service (Node.js)

Dockerfile:

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --only=production
COPY . .
CMD ["node", "server.js"]

This Dockerfile:

Uses the official Node.js 20 Alpine image as the base.
Sets the working directory to /app.
Copies package.json and package-lock.json files first.
Installs production dependencies using npm ci for faster, more reliable builds.
Copies the rest of the application code.
Specifies the command to run the Node.js application.

Basic Node.js code (server.js):

const express = require('express');
const { Kafka } = require('kafkajs');

const app = express();
const kafka = new Kafka({
  clientId: 'order-service',
  brokers: ['redpanda:9092']
});

const producer = kafka.producer();

app.use(express.json());

app.post('/orders', async (req, res) => {
  const order = req.body;
  await producer.connect();
  await producer.send({
    topic: 'orders',
    messages: [{ value: JSON.stringify(order) }],
  });
  await producer.disconnect();
  res.status(201).json({ message: 'Order created' });
});

app.listen(8082, () => console.log('Order service listening on port 8082'));

This Node.js code:

Uses Express to create a web server.
Configures a Kafka producer to connect to our Redpanda service.
Implements a POST /orders endpoint that:
- Receives order data in the request body.
- Sends the order data to a Kafka topic named 'orders'.
- Responds with a success message.
This service demonstrates how to integrate with Redpanda for event streaming.

Step 2: Creating the Docker Compose File

Now, let's create a compose.yaml file to orchestrate our microservices. We'll go through each section of the Compose file in detail:

version: '3.9'

services:
  product-service:
    build: ./product-service
    ports:
      - "8080:8080"
    environment:
      - DB_HOST=postgres
    depends_on:
      - postgres

  user-service:
    build: ./user-service
    ports:
      - "8081:8081"
    environment:
      - DB_HOST=postgres
    depends_on:
      - postgres

  order-service:
    build: ./order-service
    ports:
      - "8082:8082"
    environment:
      - DB_HOST=postgres
      - REDPANDA_HOST=redpanda
    depends_on:
      - postgres
      - redpanda

  api-gateway:
    image: nginx:alpine
    ports:
      - "80:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
    depends_on:
      - product-service
      - user-service
      - order-service

  postgres:
    image: postgres:13-alpine
    environment:
      POSTGRES_PASSWORD: mysecretpassword
    volumes:
      - pgdata:/var/lib/postgresql/data

  redpanda:
    container_name: redpanda
    image: docker.vectorized.io/vectorized/redpanda:v23.3.5
    depends_on:
      - postgres
    command:
      - redpanda start
      - --overprovisioned
      - --smp 1
      - --memory 1G
      - --reserve-memory 0M
      - --node-id 0
      - --check=false
      - --kafka-addr 0.0.0.0:9092
      - --advertise-kafka-addr redpanda:9092
      - --pandaproxy-addr 0.0.0.0:8082
      - --advertise-pandaproxy-addr redpanda:8082
      - --set redpanda.enable_transactions=true
      - --set redpanda.enable_idempotence=true
      - --set redpanda.auto_create_topics_enabled=true
      - --set redpanda.default_topic_partitions=1
    ports:
      - 9092:9092
      - 8081:8081
      - 8082:8082
    healthcheck:
      test: curl -f localhost:9644/v1/status/ready
      interval: 1s
      start_period: 30s

volumes:
  pgdata:

Let's break down the key components of this Compose file:

Version: We're using version 3.9 of the Compose file format, which is compatible with Docker Engine 19.03.0+.
Services:
- Each service is defined under the services key.
- For our custom services (product, user, order), we use the build directive to specify the location of the Dockerfile.
- We map ports from the container to the host for each service.
- Environment variables are set using the environment key.
- Dependencies between services are specified using depends_on.
API Gateway:
- Uses the official Nginx Alpine image.
- Mounts a custom Nginx configuration file.
- Depends on all other services to ensure they're running before the gateway starts.
PostgreSQL:
- Uses the official PostgreSQL Alpine image.
- Sets a password using an environment variable.
- Uses a named volume for data persistence.
Redpanda:
- Uses the official Redpanda image.
- Configures various Redpanda-specific options.
- Exposes necessary ports for Kafka protocol and admin interface.
- Includes a health check to ensure the service is ready.
Volumes:
- Defines a named volume pgdata for PostgreSQL data persistence. This volume is automatically created by Docker and will persist even if the container is removed or recreated.

Step 3: Configuring the API Gateway

The API Gateway uses Nginx to route requests to the appropriate microservices. Here's a detailed explanation of the nginx.conf file:

events {
    worker_connections 1024;
}

http {
    upstream product-service {
        server product-service:8080;
    }

    upstream user-service {
        server user-service:8081;
    }

    upstream order-service {
        server order-service:8082;
    }

    server {
        listen 80;

        location /products {
            proxy_pass http://product-service;
        }

        location /users {
            proxy_pass http://user-service;
        }

        location /orders {
            proxy_pass http://order-service;
        }
    }
}

This Nginx configuration:

Defines upstream servers for each of our microservices.
Creates a server block that listens on port 80.
Uses location blocks to route requests to the appropriate service based on the URL path.
This setup allows us to access all our microservices through a single entry point.

Step 4: Running the Microservices

To start all services, use the following command:

docker compose up -d

This command:

Builds any images that aren't cached (on the first run or when changes are made).
Creates and starts containers for all services defined in the Compose file.
The -d flag runs the containers in detached mode (in the background).

To view the logs of all running containers:

docker compose logs -f

The -f flag "follows" the log output, showing logs in real-time.

To stop all services:

docker compose down

This stops and removes all containers defined in the Compose file. Add the -v flag to also remove volumes.

Step 5: Testing the Services

You can test the services using curl or a tool like Postman. Here are some example curl commands:

Get products:
```
curl http://localhost/products
```
This should return a JSON array of products.
Get users:
```
curl http://localhost/users
```
This should return a JSON array of users.

Create an order:

curl -X POST http://localhost/orders -H "Content-Type: application/json" -d '{"productId": "1", "userId": "1", "quantity": 2}'

This should return a success message and send the order data to Redpanda.

Step 6: Scaling Services

One of the advantages of this architecture is the ability to scale services independently. To scale a service, use:

docker compose up -d --scale product-service=3

This command:

Increases the number of product-service containers to 3.
The API Gateway (Nginx) will automatically load balance requests between these instances.

You can verify the running containers with:

docker compose ps

Step 7: Monitoring and Logging

For a production environment, you'd want to add monitoring and centralized logging. Here's a basic example of adding Prometheus for monitoring:

Add this to your compose.yaml:

  prometheus:
    image: prom/prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

Create a prometheus.yml file:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'redpanda'
    static_configs:
      - targets: ['redpanda:9644']

This setup will allow Prometheus to scrape metrics from itself and Redpanda.

Conclusion

Docker Compose is awesome for local development environments and cross-team collaboration by providing a unified, easily shareable configuration that ensures consistent setup across different machines, allowing developers to quickly spin up the entire microservices ecosystem with a single command, regardless of their individual development environments. For production deployments, you'd typically use a container orchestration platform like Kubernetes or Docker Swarm.

If you're looking to dive deeper into Docker and container orchestration, don't forget to check out my free Introduction to Docker eBook. It provides a solid foundation for working with Docker and will help you understand the concepts we've covered here in more depth.

Also, if you're setting up your containerized microservices and need a reliable host, consider using DigitalOcean. You can get a $200 free credit to get started!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Happy Dockerizing, and may your microservices always be scalable, resilient, and secure!

5 Docker Best Practices I Wish I Knew When I Started

Bobby Iliev — Fri, 23 Aug 2024 08:32:07 -0700

Introduction

Hey! I'm Bobby, a Docker Captain and the author of the free Introduction to Docker eBook.

In this article, I'll share five Docker best practices that I wish I knew when I first started using Docker. These tips will help you avoid common mistakes and improve your Docker workflows, making your containerization journey smoother and more efficient.

Prerequisites

To follow along, you should have:

Basic knowledge of Docker
Docker installed on your system

If you're new to Docker, I encourage you to check out my free Introduction to Docker eBook. It covers all the basics you need to get started with Docker and will provide a solid foundation for understanding the best practices we'll discuss in this article.

Step 1 — Use Multi-Stage Builds for Smaller Images

Multi-stage builds are a powerful feature in Docker that help you create smaller, more efficient Docker images. This is crucial because smaller images offer several advantages:

They're faster to push and pull from registries, reducing deployment times.
They use less storage space, which can lead to cost savings in cloud environments.
They have a smaller attack surface, potentially improving security.

Here's an example of a multi-stage Dockerfile for a Go application:

# Build stage
FROM golang:1.16 AS builder
WORKDIR /app
COPY . .
RUN go build -o main .

# Final stage
FROM alpine:latest
WORKDIR /root/
COPY --from=builder /app/main .
CMD ["./main"]

Let's break this down:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

The first stage, labeled as builder, uses the official Go image to compile our application. This stage includes all the necessary build tools and dependencies.
The second stage starts from a minimal Alpine Linux image. We copy only the compiled binary from the builder stage using the COPY --from=builder command.
The final image contains only the compiled application and the minimal runtime requirements, resulting in a much smaller image.

By using this approach, you can significantly reduce your image size. For example, a Go application image could shrink from several hundred megabytes to just a few megabytes. This not only saves space but also reduces the time it takes to deploy your application.

Step 2 — Use `.dockerignore` Files

A .dockerignore file is to Docker what a .gitignore file is to Git. It helps you exclude files and directories from your Docker build context. This is important for several reasons:

It speeds up the build process by reducing the amount of data sent to the Docker daemon.
It prevents sensitive or unnecessary files from being inadvertently included in your Docker image.
It helps reduce the final size of your Docker image.

Here's an example of a .dockerignore file:

.git
*.md
*.log
node_modules
test

Let's go through each line:

.git: Excludes the entire Git repository, which is usually not needed in the Docker image.
*.md: Ignores all Markdown files, typically documentation that's not required for running the application.
*.log: Prevents any log files from being included in the image.
node_modules: For Node.js projects, this excludes all dependencies, which should be installed fresh during the build process.
test: Excludes the test directory, as tests typically aren't needed in production images.

By using a .dockerignore file, your Docker builds will be faster and your images will be cleaner and more secure. It's a simple step that can make a big difference in your Docker workflow.

For more information check out the official .dockerignore documentation.

Step 3 — Implement Health Checks in Your `Dockerfiles`

Health checks are an important feature in Docker that help you make sure that your containers are not only running, but actually working as expected. They allow Docker to regularly check if your application is functioning correctly.

Here's an example of how to add a health check to a Dockerfile for an Nginx server:

FROM nginx:latest
HEALTHCHECK --interval=30s --timeout=3s --start-period=5s --retries=3 \
  CMD curl -f http://localhost/ || exit 1

Let's break down the health check options:

--interval=30s: Docker will run the health check every 30 seconds.
--timeout=3s: The health check must complete within 3 seconds, or it's considered failed.
--start-period=5s: Docker will wait 5 seconds before running the first health check, giving the application time to start up.
--retries=3: If the health check fails 3 times in a row, the container is considered unhealthy.

The actual health check command (curl -f http://localhost/ || exit 1) attempts to make an HTTP request to the server. If the request fails, the health check fails.

Implementing health checks offers several benefits:

It allows Docker to automatically restart containers that have become unhealthy.
In a swarm or orchestrated environment, it enables automatic rerouting of traffic away from unhealthy containers.
It provides a clear indicator of application health, making it easier to monitor and troubleshoot issues.

By implementing health checks, you're adding an extra layer of reliability to your containerized applications.

Step 4 — Use Docker Compose for Local Development

Docker Compose is a tool for defining and running multi-container Docker applications. It's especially useful for local development environments where you might need to run several interconnected services.

As of recent updates to Docker, the default path for a Compose file is compose.yaml (preferred) or compose.yml in the working directory. Docker Compose also supports docker-compose.yaml and docker-compose.yml for backwards compatibility. If both files exist, Compose prefers the canonical compose.yaml.

Here's an example of a compose.yaml file:

version: '3'
services:
  web:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - .:/code
    environment:
      - DEBUG=True
  db:
    image: postgres:13
    environment:
      - POSTGRES_DB=myapp
      - POSTGRES_PASSWORD=secret

This compose.yaml file defines two services:

web: This is your application. It builds from the current directory (.), maps port 8000, mounts the current directory as a volume for live code reloading, and sets an environment variable.
db: This is a PostgreSQL database using the official image. It sets up a database named myapp with a password.

To run this multi-container setup, you simply use:

docker compose up

Docker Compose offers several advantages for local development:

It allows you to define your entire application stack in a single file.
You can start all services with a single command.
It provides an isolated environment that closely mimics production.
It's easy to add or remove services as your application evolves.

By using Docker Compose, you can significantly simplify your development workflow and ensure consistency across your team.

Step 5 — Be Cautious with the Latest Tag

While using the latest tag might seem convenient, it can lead to unexpected issues and make your builds less reproducible. Here's why you should be cautious with the latest tag:

Unpredictability: The latest tag doesn't refer to a specific version. It usually points to the most recent version, which can change without warning.
Lack of consistency: Different team members or environments might pull the latest image at different times, potentially getting different versions.
Difficulties in debugging: If an issue arises, it's harder to pinpoint which exact version of the image is causing the problem.

Instead of using latest, it's better to specify exact versions in your Dockerfile or compose file. For example:

FROM node:18.20.4

Or in your compose.yaml:

services:
  db:
    image: postgres:13.3

By specifying exact versions:

You ensure that everyone on your team is using the same version.
You can easily reproduce builds and deployments.
You have more control over when and how you update your dependencies.

When you're ready to update to a new version, you can do so intentionally by changing the version number in your Dockerfile or compose file. This gives you the opportunity to test the new version thoroughly before deploying it to production.

Bonus Tip: Implement Regular Security Scans

Security should be a top priority when working with Docker images. Implementing regular security scans can help you identify and address vulnerabilities in your containers before they become a problem.

One powerful tool for this purpose is Docker Scout, which is integrated directly into Docker Desktop and the Docker CLI. Here's how you can use it:

First, ensure you have the latest version of Docker installed.

To scan an image, use the following command:

docker scout cve <image_name>

For example:

docker scout cve nginx:latest

This command will provide a detailed report of any known vulnerabilities in the image, including severity levels and available fixes.

You can also integrate Docker Scout into your CI/CD pipeline to automatically scan images before deployment. Here's an example of how you might do this in a GitHub Actions workflow:

name: Docker Image CI

on: [push]

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - name: Build the Docker image
      run: docker build . --file Dockerfile --tag my-image:$(date +%s)
    - name: Scan the Docker image
      run: docker scout cve my-image:$(date +%s)

By implementing regular security scans, you can:

Identify vulnerabilities early in your development process
Keep your production environments more secure
Stay informed about the security status of your Docker images
Make informed decisions about when to update your base images or dependencies

Regular scans, combined with prompt updates and patches, will help keep your Docker environments secure.

Conclusion

These tips - using multi-stage builds, leveraging .dockerignore files, implementing health checks, using Docker Compose for local development, being cautious with the latest tag and running security scans - will help you create more efficient, reliable, and maintainable Docker workflows.

If you're looking to dive deeper into Docker, I encourage you to check out my free Introduction to Docker eBook.

Also, if you're setting up your Docker environment and need a reliable host, consider using DigitalOcean. You can get a $200 free credit to get started!

Happy Dockerizing, and may your containers always run smoothly!

- Bobby

How to Remove Arrow on Input type Number with Tailwind CSS

Bobby Iliev — Wed, 10 Jul 2024 06:32:57 -0700

When designing forms with Tailwind CSS, you might want to remove the default arrows (also known as spinners) from number input fields. These arrows can interfere with custom designs and are challenging to style consistently across different browsers.

In this tutorial, we'll explore how to achieve this using Tailwind CSS, both with inline styles and through a global CSS approach. We'll also address accessibility concerns and provide examples for styling custom buttons.

The Problem

By default, browsers add increment and decrement arrows to elements. While functional, these arrows often clash with custom designs and can be difficult to style uniformly across various browsers.

The Solution

We'll use Tailwind CSS utility classes to remove these arrows and create clean, customized number inputs. We'll also look at how to apply this styling globally for larger projects and how to add accessible custom controls.

Inline Approach

Let's start with an example that uses inline Tailwind classes:

class="max-w-md mx-auto mt-8 p-6 bg-white rounded-lg shadow-md">
  <div class="mb-4">
    <label for="quantity" class="block text-sm font-medium text-gray-700 mb-2">Quantitylabel>
    <input type="number" id="quantity" name="quantity" 
      class="block w-full px-3 py-2 bg-white border border-gray-300 rounded-md text-sm shadow-sm
      focus:outline-none focus:border-sky-500 focus:ring-1 focus:ring-sky-500
      [appearance:textfield] [&::-webkit-outer-spin-button]:appearance-none [&::-webkit-inner-spin-button]:appearance-none">
  div>
  
  <div class="mb-4">
    <label for="price" class="block text-sm font-medium text-gray-700 mb-2">Pricelabel>
    <input type="number" id="price" name="price" step="0.01"
      class="block w-full px-3 py-2 bg-white border border-gray-300 rounded-md text-sm shadow-sm
      focus:outline-none focus:border-sky-500 focus:ring-1 focus:ring-sky-500
      [appearance:textfield] [&::-webkit-outer-spin-button]:appearance-none [&::-webkit-inner-spin-button]:appearance-none">
  div>
  
  <button type="submit" class="w-full bg-blue-500 text-white py-2 px-4 rounded-md hover:bg-blue-600 transition-colors">
    Submit
  button>
form>

The key classes for removing the arrows are:

[appearance:textfield]: Removes default styling in Firefox.
[&::-webkit-outer-spin-button]:appearance-none: Removes outer spin button in WebKit browsers.
[&::-webkit-inner-spin-button]:appearance-none: Removes inner spin button in WebKit browsers.

Global Approach

For larger projects, you might want to apply this styling to all number inputs. You can do this by adding styles to your global CSS file:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Open your global.css file (or equivalent, like app.css or styles.css) depending on your framework and setup.
Add the following CSS:

/* In your global.css file */
@layer utilities {
  input[type="number"]::-webkit-inner-spin-button,
  input[type="number"]::-webkit-outer-spin-button {
    -webkit-appearance: none;
    margin: 0;
  }

  input[type="number"] {
    -moz-appearance: textfield;
  }
}

Ensure this CSS file is imported in your main Tailwind CSS file or included in your HTML.

After adding these global styles, you can simplify your HTML:

class="max-w-md mx-auto mt-8 p-6 bg-white rounded-lg shadow-md">
  <div class="mb-4">
    <label for="quantity" class="block text-sm font-medium text-gray-700 mb-2">Quantitylabel>
    <input type="number" id="quantity" name="quantity" 
      class="block w-full px-3 py-2 bg-white border border-gray-300 rounded-md text-sm shadow-sm
      focus:outline-none focus:border-sky-500 focus:ring-1 focus:ring-sky-500">
  div>
  
  <div class="mb-4">
    <label for="price" class="block text-sm font-medium text-gray-700 mb-2">Pricelabel>
    <input type="number" id="price" name="price" step="0.01"
      class="block w-full px-3 py-2 bg-white border border-gray-300 rounded-md text-sm shadow-sm
      focus:outline-none focus:border-sky-500 focus:ring-1 focus:ring-sky-500">
  div>
  
  <button type="submit" class="w-full bg-blue-500 text-white py-2 px-4 rounded-md hover:bg-blue-600 transition-colors">
    Submit
  button>
form>

Notice that we've removed the arrow-removing classes from individual inputs, as they're now handled by the global CSS.

Adding Custom Arrows

While removing default arrows improves design consistency, it's crucial to maintain accessibility. Here's how to create custom, accessible increment/decrement buttons that match our form's design:

class="mb-4">
  <label for="quantity" class="block text-sm font-medium text-gray-700 mb-2">Quantitylabel>
  <div class="relative flex items-center">
    <input type="number" id="quantity" name="quantity" 
      class="block w-full px-3 py-2 bg-white border border-gray-300 rounded-md text-sm shadow-sm
      focus:outline-none focus:border-sky-500 focus:ring-1 focus:ring-sky-500
      [appearance:textfield] [&::-webkit-outer-spin-button]:appearance-none [&::-webkit-inner-spin-button]:appearance-none">
    <div class="absolute right-0 flex items-center">
      <button type="button" 
              class="px-2 h-full border-l border-gray-300 text-gray-500 hover:text-sky-500 focus:outline-none focus:ring-2 focus:ring-sky-500 focus:ring-offset-2"
              onclick="document.getElementById('quantity').stepUp()"
              aria-label="Increase quantity">
        <svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke-width="1.5" stroke="currentColor" class="w-4 h-4">
          <path stroke-linecap="round" stroke-linejoin="round" d="M4.5 15.75l7.5-7.5 7.5 7.5" />
        svg>
      button>
      <button type="button" 
              class="px-2 h-full border-l border-gray-300 text-gray-500 hover:text-sky-500 focus:outline-none focus:ring-2 focus:ring-sky-500 focus:ring-offset-2"
              onclick="document.getElementById('quantity').stepDown()"
              aria-label="Decrease quantity">
        <svg xmlns="http://www.w3.org/2000/svg" fill="none" viewBox="0 0 24 24" stroke-width="1.5" stroke="currentColor" class="w-4 h-4">
          <path stroke-linecap="round" stroke-linejoin="round" d="M19.5 8.25l-7.5 7.5-7.5-7.5" />
        svg>
      button>
    div>
  div>
div>

Key accessibility and styling improvements:

We've added aria-label attributes to the buttons, providing context for screen readers.
The focus states now include a visible ring (focus:ring-2 focus:ring-sky-500 focus:ring-offset-2) to improve keyboard navigation visibility.
The buttons are styled consistently with the input field, maintaining the form's overall design.

Important Considerations

There are a few things that you should consider:

Accessibility: While removing default arrows can improve visual design, it's crucial to maintain accessibility. Always provide alternative methods for incrementing/decrementing values, such as custom buttons with proper ARIA labels.
Keyboard Navigation: Ensure that your custom controls are fully accessible via keyboard. This includes proper focus states and the ability to activate buttons using the keyboard.
Cross-browser Compatibility: This solution works in modern browsers. Older browsers may require additional CSS or JavaScript.

Conclusion

By implementing these techniques, you can effectively remove the default arrows from number inputs across your project while maintaining accessibility and consistent styling. Remember to always test your forms with various input methods (mouse, keyboard, touch) and assistive technologies to ensure a great user experience for all.

For those looking to improve their Tailwind CSS development process further, check out the DevDojo Tails page builder, which can help you create amazing designs with ease.

Happy coding!

How to Make Parent Div Activate Styling of Child Div for Hover and Active States

Bobby Iliev — Fri, 05 Jul 2024 08:39:57 -0700

Hey there developers! 👋

Today, we're diving into a Tailwind CSS trick! We'll explore how to make a parent div control the styling of its child elements on hover and active states. Let's jump right in!

The Problem

You've probably encountered situations where you want an entire component to respond to user interactions, not just individual elements.

For example, you might want a card to change its appearance when hovered, including all its child elements. Tailwind CSS has a elegant solution for this: the group and group-hover utilities.

The Solution: Group and Group-Hover

Tailwind CSS provides a powerful feature called "group hover" that allows us to style child elements based on the state of a parent element. Here's how it works:

Add the group class to your parent element.
Use group-hover: prefix on child elements to apply styles when the parent is hovered.

Let's see this in action with a cool example:

<div class="group p-6 bg-gray-100 rounded-lg transition-all duration-300 hover:bg-blue-100">
  <h2 class="text-2xl font-bold text-gray-800 group-hover:text-blue-600">Awesome Featureh2>
  <p class="mt-2 text-gray-600 group-hover:text-blue-500">This feature will blow your mind!p>
  <button class="mt-4 px-4 py-2 bg-blue-500 text-white rounded group-hover:bg-blue-600">
    Learn More
  button>
div>

In this example, when you hover over the parent div, the heading, paragraph, and button all change color simultaneously. Pretty cool, right?

Taking It Further: Group-Active

But wait, there's more! We can also use the group-active variant to style child elements when the parent is in an active state (e.g., being clicked). Here's an enhanced version of our previous example:

<div class="group p-6 bg-gray-100 rounded-lg transition-all duration-300 hover:bg-blue-100 active:bg-blue-200">
  <h2 class="text-2xl font-bold text-gray-800 group-hover:text-blue-600 group-active:text-blue-800">Awesome Featureh2>
  <p class="mt-2 text-gray-600 group-hover:text-blue-500 group-active:text-blue-700">This feature will blow your mind!p>
  <button class="mt-4 px-4 py-2 bg-blue-500 text-white rounded group-hover:bg-blue-600 group-active:bg-blue-800">
    Learn More
  button>
div>

Now, when you click on the component, you'll see an additional style change. It's like magic, but it's just the power of Tailwind CSS! 🎩✨

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Pro Tip: Extending Core Plugins

It's worth noting that not every property supports group-hover or group-active out of the box. In some cases, you might need to extend Tailwind's core plugins. You can do this in your tailwind.config.js file:

module.exports = {
  // ...other config
  plugins: [
    plugin(({ addVariant }) => {
      addVariant('group-hover', ':merge(.group):hover &')
      addVariant('group-active', ':merge(.group):active &')
    })
  ],
}

This will allow you to use group-hover and group-active with any utility in Tailwind CSS.

Wrapping Up

And there you have it! You've just learned how to make parent divs control the styling of their child elements for hover and active states using Tailwind CSS.

As a next step, check out DevDojo's Tails page builder at https://devdojo.com/tails/. It's a fantastic visual builder that lets you create stunning Tailwind CSS powered pages with ease. Give it a spin and see how it can supercharge your development workflow!

Keep coding, stay curious, and until next time, may your builds be bug-free and your coffee strong! 💻☕️

How to Access Direct Children of a Div in Tailwind CSS v3

Bobby Iliev — Wed, 03 Jul 2024 04:43:23 -0700

In this tutorial, we'll explore how to target and style the direct children of a div using Tailwind CSS v3's powerful arbitrary value syntax. This feature allows for more flexible and precise styling, especially when dealing with nested layouts.

The Problem

Consider the following HTML structure:

<div class="section">
   <div class="footer">footerdiv>
   <div class="content">
      <div>sub contents 1div>              
      <div>sub contents 2div>
   div>
div>

We want to style only the direct children of the div with the class "section", which are the divs with classes "footer" and "content". In regular CSS, we'd use the child combinator selector like this: div.section > div. But how do we achieve this in Tailwind CSS v3?

The Solution: Arbitrary Value Syntax

Tailwind CSS v3 introduced the arbitrary value syntax, which allows us to use the & character to reference the current selector. This feature provides a powerful way to target direct children. Here's how you can do it:

Syntax: `[&>*]:{class}`

& represents the current element
> is the child combinator
* selects all direct children
:{class} applies the specified Tailwind class

Let's see this in action with some examples.

Example 1: Adding padding to all direct children

<div class="section [&>*]:p-4">
   <div class="footer">footerdiv>
   <div class="content">
      <div>sub contents 1div>              
      <div>sub contents 2div>
   div>
div>

In this example, [&>*]:p-4 adds padding of 1rem (p-4) to all direct children of the "section" div.

Example 2: Targeting specific child elements

<div class="section [&>div]:bg-gray-100 [&>.footer]:text-red-500">
   <div class="footer">footerdiv>
   <div class="content">
      <div>sub contents 1div>              
      <div>sub contents 2div>
   div>
div>

Here, we're applying different styles to different direct children:

[&>div]:bg-gray-100 adds a light gray background to all direct div children.
[&>.footer]:text-red-500 makes the text red only for the direct child with the "footer" class.

Example 3: Combining multiple styles

<div class="section [&>*]:p-4 [&>*]:mb-2 [&>.content]:bg-blue-100">
   <div class="footer">footerdiv>
   <div class="content">
      <div>sub contents 1div>              
      <div>sub contents 2div>
   div>
div>

This example combines multiple arbitrary value selectors:

[&>*]:p-4 adds padding to all direct children.
[&>*]:mb-2 adds margin-bottom to all direct children.
[&>.content]:bg-blue-100 applies a light blue background only to the "content" div.

Practical Example

Let's create a footer with multiple columns, social media links, and a newsletter signup form. We'll use the arbitrary value syntax to style the direct children of our footer container.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

<footer class="bg-gray-800 text-white py-12">
  <div class="container mx-auto px-4">
    <div class="[&>div]:mb-8 [&>div:last-child]:mb-0 md:[&>div]:mb-0 md:flex md:justify-between">
      
      <div class="md:w-1/4">
        <h2 class="text-2xl font-bold mb-4">TechCorph2>
        <p class="[&>a]:text-blue-400 [&>a]:hover:underline">
          123 Tech Street, San Francisco, CA 94122<br>
          <a href="mailto:info@techcorp.com">info@techcorp.coma><br>
          <a href="tel:+14155551234">(415) 555-1234a>
        p>
      div>

      
      <div class="md:w-1/4">
        <h3 class="text-lg font-semibold mb-4">Quick Linksh3>
        <ul class="[&>li]:mb-2 [&>li>a]:text-gray-300 [&>li>a]:hover:text-white [&>li>a]:transition-colors">
          <li><a href="#">About Usa>li>
          <li><a href="#">Servicesa>li>
          <li><a href="#">Bloga>li>
          <li><a href="#">Contacta>li>
        ul>
      div>

      
      <div class="md:w-1/4">
        <h3 class="text-lg font-semibold mb-4">Connect With Ush3>
        <div class="flex [&>a]:mr-4 [&>a]:text-2xl [&>a]:text-gray-300 [&>a]:hover:text-white [&>a]:transition-colors">
          <a href="#" aria-label="Facebook"><i class="fab fa-facebook">i>a>
          <a href="#" aria-label="Twitter"><i class="fab fa-twitter">i>a>
          <a href="#" aria-label="LinkedIn"><i class="fab fa-linkedin">i>a>
          <a href="#" aria-label="Instagram"><i class="fab fa-instagram">i>a>
        div>
      div>

      
      <div class="md:w-1/4">
        <h3 class="text-lg font-semibold mb-4">Subscribe to Our Newsletterh3>
        <form class="[&>*]:mb-2 [&>*:last-child]:mb-0">
          <input 
            type="email" 
            placeholder="Enter your email" 
            class="w-full px-4 py-2 bg-gray-700 text-white rounded focus:outline-none focus:ring-2 focus:ring-blue-500"
          >
          <button 
            type="submit" 
            class="w-full bg-blue-500 hover:bg-blue-600 text-white font-bold py-2 px-4 rounded transition-colors"
          >
            Subscribe
          button>
        form>
      div>
    div>
  div>
footer>

![https://imgur.com/B6XHhRx.png]

In this example, we've used several instances of the arbitrary value syntax to style our footer:

For the main container:
- [&>div]:mb-8: Adds margin-bottom to all direct child divs.
- [&>div:last-child]:mb-0: Removes margin-bottom from the last child div.
- md:[&>div]:mb-0: Removes margin-bottom from all child divs on medium screens and up.
For the company info section:
- [&>a]:text-blue-400: Makes all direct child links blue.
- [&>a]:hover:underline: Underlines links on hover.
For the quick links section:
- [&>li]:mb-2: Adds margin-bottom to all list items.
- [&>li>a]:text-gray-300: Sets the text color of links within list items.
- [&>li>a]:hover:text-white: Changes link color on hover.
- [&>li>a]:transition-colors: Adds a smooth transition effect for color changes.
For the social media section:
- [&>a]:mr-4: Adds margin-right to all links.
- [&>a]:text-2xl: Sets the font size for social icons.
- [&>a]:text-gray-300: Sets the initial color for social icons.
- [&>a]:hover:text-white: Changes icon color on hover.
For the newsletter form:
- [&>*]:mb-2: Adds margin-bottom to all direct children of the form.
- [&>*:last-child]:mb-0: Removes margin-bottom from the last child of the form.

Conclusion

Tailwind CSS v3's arbitrary value syntax provides a powerful and flexible way to target and style direct children of an element. This approach allows you to:

Apply styles to all direct children using [&>*]:{class}.
Target specific direct children using class or element selectors like [&>.classname]:{class} or [&>element]:{class}.
Combine multiple selectors for more complex styling needs.

If you're looking to improve your Tailwind CSS development process even further, check out Tails by DevDojo. This powerful page builder allows you to visually create stunning, responsive Tailwind CSS websites with drag-and-drop ease, making it an excellent tool for both beginners and experienced developers alike.

Happy coding!

How to Vertically Align Content with Tailwind CSS Across a Full-Screen Div

Bobby Iliev — Sun, 30 Jun 2024 13:19:57 -0700

Vertical alignment can often be a challenge in web design, but with Tailwind CSS, you can easily align elements in the center of the screen.

This quick guide will walk you through the steps to vertically align content within a full-screen div using Tailwind CSS, complete with nicely styled examples.

Step 1: Setting Up Tailwind CSS

First, make sure you have Tailwind CSS set up in your project. If you're starting from scratch, you can use the following CDN link in your HTML file:

html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Vertical Alignment with Tailwind CSStitle>
    <link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.19/dist/tailwind.min.css" rel="stylesheet">
head>
<body>
    
body>
html>

If you're using a build tool like Webpack or a framework like Next.js, refer to the Tailwind CSS installation guide for the appropriate setup.

Step 2: Creating the Full-Screen Div

To create a full-screen div, we'll use Tailwind's utility classes. We'll start by creating a div that spans the full viewport height and width. Here's a simple example:

<div class="min-h-screen flex items-center justify-center bg-gray-100">
    
div>

min-h-screen: This class sets the minimum height of the div to the full height of the viewport.
flex: This makes the div a flex container.
items-center: This vertically centers the content inside the flex container.
justify-center: This horizontally centers the content inside the flex container.
bg-gray-100: This adds a light gray background color to the div.

Step 3: Adding Content

Now, let's add some content inside our full-screen div. We'll use a simple card component as our example:

<div class="min-h-screen flex items-center justify-center bg-gray-100">
    <div class="bg-white p-8 rounded-lg shadow-lg">
        <h1 class="text-2xl font-bold mb-4">Vertically Aligned Contenth1>
        <p class="text-gray-700">This content is centered both vertically and horizontally using Tailwind CSS.p>
    div>
div>

bg-white: This sets the background color of the card to white.
p-8: This adds padding to the card.
rounded-lg: This rounds the corners of the card.
shadow-lg: This adds a large shadow to the card.
text-2xl: This sets the font size of the heading to 2xl.
font-bold: This makes the heading bold.
mb-4: This adds a bottom margin to the heading.
text-gray-700: This sets the color of the paragraph text to a dark gray.

Step 4: Styling the Content

To make our example more visually appealing, we can add some additional styling. Let's enhance the card with a more polished look:

<div class="min-h-screen flex items-center justify-center bg-gradient-to-r from-blue-500 to-purple-600">
    <div class="bg-white p-8 rounded-lg shadow-2xl transform hover:scale-105 transition-transform duration-300">
        <h1 class="text-3xl font-extrabold mb-6 text-transparent bg-clip-text bg-gradient-to-r from-green-400 to-blue-500">
            Vertically Aligned Content
        h1>
        <p class="text-gray-800 text-lg">
            This content is centered both vertically and horizontally using Tailwind CSS.
        p>
    div>
div>

bg-gradient-to-r from-blue-500 to-purple-600: This creates a background gradient for the full-screen div.
shadow-2xl: This adds a larger shadow to the card.
transform hover:scale-105 transition-transform duration-300: This adds a scaling effect when the card is hovered over, with a smooth transition.
text-3xl: This sets the font size of the heading to 3xl.
font-extrabold: This makes the heading extra bold.
text-transparent bg-clip-text bg-gradient-to-r from-green-400 to-blue-500: This creates a gradient text effect for the heading.
text-lg: This sets the font size of the paragraph text to large.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Conclusion

By using Tailwind CSS's utility classes, you can easily vertically align content within a full-screen div. The flexbox utilities provided by Tailwind make it simple to center content both vertically and horizontally with just a few classes.

For even more styling options and to create beautiful designs effortlessly, check out the DevDojo Tails Tailwind CSS builder. It's a fantastic tool to help you with your workflow and create stunning designs with Tailwind CSS.

How to Get the Directory Where a Bash Script is Located

Bobby Iliev — Wed, 19 Jun 2024 06:49:13 -0700

Hello everyone!

When you're working with Bash scripts, it's often useful to know the directory where the script itself resides. This can be essential for referencing relative paths, ensuring the script works correctly regardless of where it is executed from. Let’s dive into how you can achieve this.

Method 1: Using `dirname` and `$0`

The simplest way to get the directory of the script is to use the dirname command in combination with $0. Here's a step-by-step explanation:

$0: This special variable contains the path used to invoke the script. It might be a relative path, an absolute path, or just the script name.
dirname: This command removes the last component from a path, effectively giving you the directory part.

Here’s a small snippet to demonstrate this:

#!/bin/bash

# Get the directory of the script
SCRIPT_DIR=$(dirname "$0")

echo "The script is located in: $SCRIPT_DIR"

Method 2: Resolving the Full Path

If you need the absolute path, especially if the script is run from a relative path, you can combine dirname with readlink:

#!/bin/bash

# Resolve the full path of the script
SCRIPT_DIR=$(dirname "$(readlink -f "$0")")

echo "The script is located in: $SCRIPT_DIR"

Breaking Down the Command

readlink -f "$0": This command follows all symlinks and returns the absolute path of the script.
dirname "$(readlink -f "$0")": This gives the absolute directory path of the script.

Method 3: Handling Edge Cases

For more complex scenarios, such as when the script is sourced or if it's part of a symlink chain, you might need additional logic to ensure you always get the correct directory.

Here’s an advanced version that handles more edge cases:

#!/bin/bash

# Function to resolve the script path
get_script_dir() {
    local SOURCE=$0
    while [ -h "$SOURCE" ]; do # Resolve $SOURCE until the file is no longer a symlink
        DIR=$(cd -P "$(dirname "$SOURCE")" && pwd)
        SOURCE=$(readlink "$SOURCE")
        [[ $SOURCE != /* ]] && SOURCE=$DIR/$SOURCE # If $SOURCE was a relative symlink, resolve it relative to the symlink base directory
    done
    DIR=$(cd -P "$(dirname "$SOURCE")" && pwd)
    echo "$DIR"
}

# Get the directory
SCRIPT_DIR=$(get_script_dir)

echo "The script is located in: $SCRIPT_DIR"

Conclusion

Getting the directory of the Bash script from within the script itself is a common requirement and can be easily achieved using dirname and $0. For more robust solutions, combining these with readlink can ensure you handle absolute paths and symlinks correctly. By incorporating these methods into your scripts, you can make them more portable and easier to manage.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

If you’re interested in learning more about Bash scripting, be sure to check out my free eBook on Bash scripting. It's packed with useful information and examples to help you become a Bash scripting pro!

Happy scripting!

Running Laravel Wave with Sail: A Step-by-Step Guide

Bobby Iliev — Tue, 26 Dec 2023 05:59:04 -0800

Introduction

This tutorial will guide you through the process of running Laravel Wave using Laravel Sail, a light-weight command-line interface for managing Dockerized Laravel applications. This tutorial is perfect for those who prefer a Docker-based environment for their Laravel applications. Let's dive in!

Prerequisites

Before we begin, ensure you have:

PHP 8.1
Composer installed on your system

These are essential for running any Laravel application.

Getting Started with Laravel Wave

First things first, let's get the Laravel Wave repository. Fork and clone the repository from GitHub:

git clone git@github.com:thedevdojo/wave.git

Note: if you've cloned the repository change thedevdojo with your GitHub username

Cloning the repository provides you with the necessary files to get started.

Setting Up Laravel Sail

Laravel Sail is a simple solution for running your Laravel project using Docker. Follow these steps to set it up:

Install Dependencies

Navigate to your project directory and install the Composer dependencies:

composer install

This command pulls in all the necessary PHP packages required by Laravel Wave.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Environment Configuration

Next, you need to set up your environment file. Copy the .env.example file to create your .env file:

cp .env.example .env

After copying the file, update the database details in your .env file to match the following:

DB_CONNECTION=mysql
DB_HOST=mysql
DB_PORT=3306
DB_DATABASE=wave
DB_USERNAME=sail
DB_PASSWORD=password

These settings configure your application to connect to the MySQL database provided by Sail.

Start Sail

Now, it's time to start your Dockerized environment with Sail:

bash ./vendor/bin/sail up -d

This command starts all the necessary Docker containers in the background.

Database Migrations and Seeding

With your environment up and running, proceed to set up your database:

./vendor/bin/sail artisan migrate 
./vendor/bin/sail artisan db:seed
./vendor/bin/sail artisan storage:link

These commands create the necessary database tables, seed them with data, and set up storage links.

Accessing Your Application

Finally, after completing the setup, you can access your Laravel Wave application by visiting http://localhost in your browser.

Conclusion

Setting up Laravel Wave with Sail offers a seamless and straightforward way to manage your local Laravel Wave dev app in a Docker environment. This approach not only simplifies the development process but also ensures consistency across different development environments.

For production, I could suggest using DigitalOcean. If you wish, you can use my affiliate code to get free $200 DigitalOcean credit to spin up your own servers!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Happy coding, and as always, feel free to reach out if you have any questions or need further assistance!

Introduction to Linux eBook: A Comprehensive Guide for Beginners

Bobby Iliev — Sun, 01 Oct 2023 09:55:59 -0700

Introduction

Hello to all passionate learners, tech enthusiasts, and budding system administrators!

I'm thrilled to announce the release of my latest eBook: "Introduction to Linux". After months of dedicated research, writing, and refining, I've put together a guide that aims to help beginners and provide a solid foundation for those looking to gain a deeper understanding.

Why Linux?

Linux is a powerful operating system that has become a cornerstone of modern computing. It's a versatile platform that can be used for everything from running web servers to powering mobile devices. Linux is also open-source, meaning anyone can contribute to its development and use it free of charge.

Gaining proficiency in Linux not only opens the door to understanding this powerful OS but also paves the way for numerous career opportunities.

What Can You Expect from the eBook?

Foundational Concepts: Delve into the history and philosophy of Linux, its various distributions, and its place in the modern tech ecosystem.
Hands-on Learning: Engage with practical exercises and real-world examples that guide you through navigating the Linux filesystem, working with the command line, and much more.
Deep Dives: Explore topics like process management, user permissions, networking basics, and even bash scripting. Perfect for those who wish to transition from beginners to adept users.

Who is this Book For?

Whether you're an absolute beginner curious about Linux, a developer looking to hone your system administration skills, or an IT professional aiming to broaden your knowledge base, "Introduction to Linux" is for you.

A Glimpse into My Journey

Writing this eBook was not just about compiling information; it was a journey of revisiting the foundational concepts, engaging with the vast Linux community, and reflecting on my experiences as a Linux system administrator. I've always believed in the power of shared knowledge, and this book is my humble contribution to the ever-growing repository of Linux wisdom.

Ready to Get Started?

I invite you to dive into the world of Linux with this comprehensive guide. It's designed to be both a learning tool and a reference, serving you well as you progress in your journey.

Introduction to Linux eBook

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Happy learning!

Introduction to Terraform: From Zero to Hero eBook

Bobby Iliev — Mon, 24 Jul 2023 08:01:49 -0700

Introduction

Hey folks, I'm super stoked to announce my brand-new eBook: "Introduction to Terraform: From Zero to Hero". This project is something I'm genuinely excited about, born from countless late-night coding sessions and my growing fascination with Infrastructure as Code. I've come to realize that Terraform is an absolute game-changer in the DevOps world, and I can't wait to share everything I've learned with you!

A sneak peek into what's inside:

Introduction to Infrastructure as Code (IaC)
Introduction to Terraform
Installation and Setup
Terraform Basics
Terraform Configuration Language (HCL)
Terraform State
Modules in Terraform
Terraform Cloud
Best Practices
Intermediate Terraform Language Features
Importing Existing Infrastructure into Terraform
Advanced State Management
Terraform with Multiple Providers
Terraform for Serverless Architecture
Terraform for Container Orchestration
Terraform with Continuous Integration/Continuous Deployment (CI/CD)
Security and Compliance with Terraform
Managing Secrets with Vault
Troubleshooting and Debugging Terraform
Terraform with Ansible
Advanced Topics and Conclusion

Why "From Zero to Hero"?

The philosophy behind this title is simple. We all start our learning journeys from square one, with a spark of curiosity and an urge to get better at something new. Consider this book your handy roadmap that'll guide you from being a newbie in the world of Infrastructure as Code to really understanding the ins and outs of Terraform. This guide is loaded with in-depth chapters, real-world examples, and hands-on exercises designed to help you in the awesome world of Terraform. By the end of it, I promise you'll be bossing around your own projects like a champ!

Who is it for?

Whether you're a beginner just starting out in the DevOps world or a professional looking to expand your skill set, this book has something for everyone. It's written in a way so that both newcomers and those seeking advanced knowledge could benefit from.

Conclusion

In a rapidly evolving tech world, staying updated and continuously learning is the key. "Introduction to Terraform: From Zero to Hero" isn't just a book; it's your next step in the DevOps world. So, don't hesitate and get your copy today from Leanpub. Let's start this exciting adventure together!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

I genuinely hope you find value in every page of this book.

Until next time, happy learning and terraforming!

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

How to use DigitalOcean Spaces with Laravel Voyager?

Bobby Iliev — Tue, 30 May 2023 23:39:03 -0700

Introduction

DigitalOcean Spaces is an object storage service that allows you to store and serve large amounts of data. It is a reliable and flexible solution for developers, especially when integrated with Laravel Voyager - a Laravel package that provides an admin interface with BREAD (Browse, Read, Edit, Add, Delete) operations, media manager, menu builder, and much more.

This tutorial will guide you step-by-step on how to integrate DigitalOcean Spaces with Laravel Voyager.

Before we begin, ensure you have a working Laravel application with Voyager installed. We're also assuming that you have already created your DigitalOcean Space and have the necessary credentials: Spaces Key, Spaces Secret, Spaces region, Spaces Bucket and Spaces endpoint.

Laravel Filesystem

Laravel provides a clean, simple API over the popular Flysystem PHP package by Frank de Jonge. The Laravel Flysystem integration provides simple to use drivers for working with local filesystems, Amazon S3, and even FTP servers. To utilize this feature for our purpose, we need to install a specific package.

Step 1: Installing the `flysystem-aws-s3-v3` Package

The DigitalOcean Spaces is compatible with the Amazon S3 API. So, we will use the flysystem-aws-s3-v3 package to interact with the DigitalOcean Spaces.

Open your terminal and navigate to the root directory of your Laravel project. Now, run the following command:

composer require league/flysystem-aws-s3-v3

This command will install the flysystem-aws-s3-v3 package which will allow us to interact with the DigitalOcean Spaces.

Step 2: Configuring the File System

In the root directory of your Laravel application, open the config/filesystems.php file. Here, we will add a new disk definition for 'spaces':

'spaces' => [
    'driver' => 's3',
    'key' => env('DO_SPACES_KEY'),
    'secret' => env('DO_SPACES_SECRET'),
    'region' => env('DO_SPACES_REGION'),
    'bucket' => env('DO_SPACES_BUCKET'),
    'endpoint' => env('DO_SPACES_ENDPOINT'),
    'visibility' => 'public',
],

The 'visibility' => 'public' setting is needed to ensure that the images uploaded via the media manager are visible to the public.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Next, open the .env file and add the following lines:

DO_SPACES_KEY=your_spaces_key           # Example: 1234567890ABCDEF
DO_SPACES_SECRET=your_spaces_secret     # Example: 1234567890ABCDEF1234567890ABCDEF1234567890ABCDEF
DO_SPACES_REGION=your_spaces_region     # Example: nyc3
DO_SPACES_BUCKET=your_spaces_bucket     # Example: my-bucket
DO_SPACES_ENDPOINT=your_spaces_endpoint # Example: https://nyc3.digitaloceanspaces.com

Note that we didn't add the bucket name to the endpoint. This is because the flysystem-aws-s3-v3 package will automatically append the bucket name to the endpoint.

Step 3: Configuring Voyager

Now, let's open the config/voyager.php file and update the storage disk to 'spaces':

'storage' => [
    'disk' => 'spaces',
],

Step 4: Overriding Default Voyager Blade View (Optional)

In some older versions of Voyager, the media manager view had some issues with the file path.

To verify if you are still affected by this issue, visit the Voyager admin and if you see this error:

Found 1 error while validating the input provided for the GetObject operation: [Key] expected string length to be >= 1, but found string length of 0

The following steps will help you resolve this issue.

NOTE: If you don't see the above error, you can skip this step.

First, create a new directory for the view:

mkdir -p resources/views/vendor/voyager/media

Next, copy the Voyager media manager view to your new directory:

cp vendor/tcg/voyager/resources/views/media/manager.blade.php resources/views/vendor/voyager/media

Finally, open the copied manager.blade.php file and update this line:

<div class="img_icon" :style="imgIcon('{{ Storage::disk(config('voyager.storage.disk'))->url('/') }}'+file)">div>

To:

<div class="img_icon" :style="imgIcon('{{ Storage::disk(config('voyager.storage.disk'))->url('./') }}'+file)">div>

Note that the only change is the addition of a ./ before the file variable.

Conclusion

After completing these steps, your Laravel Voyager application should now be set up to use DigitalOcean Spaces for file storage. This integration unlocks new possibilities for you, allowing you to scale your applications efficiently.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

As a next step, you can try to upload a file using the media manager and check if it is uploaded to your DigitalOcean Spaces bucket.

Happy coding!

Difference between scaling horizontally and vertically for databases?

Bobby Iliev — Sun, 14 May 2023 12:30:39 -0700

Introduction

When your application or website starts to receive more traffic, you'll need to think about how to scale your database to handle the additional load. There are two main ways to scale a database: vertically and horizontally. Let's dive in and understand the differences between these two approaches.

1. Vertical Scaling (Scale-up)**

Vertical scaling, also known as scaling up, involves adding more resources to your existing server to enhance its computational power. This could mean increasing the CPU power, RAM, SSD, or other hardware resources on your existing machine.

Advantages of Vertical Scaling:

Simplicity: Vertical scaling is usually simpler as it involves just increasing the server's power. There's no need for code changes or complex architecture designs.
Data Consistency: Since there's only one database, there's no issue of data consistency.

Disadvantages of Vertical Scaling:

Limited Growth: There's a limit to how much you can scale up a single server. At some point, you won't be able to add any more resources.
Downtime: To upgrade the server, you might need to face some downtime unless your system supports hot-swappable components.
Cost: High-end servers can become very expensive.

2. Horizontal Scaling (Scale-out)

Horizontal scaling, also known as scaling out, involves adding more servers to your existing pool of servers. The load is distributed across multiple servers, thereby increasing the ability to handle more requests.

Advantages of Horizontal Scaling:

Greater Capacity: Horizontal scaling allows for virtually limitless scaling, as you can always add more servers to handle more traffic.
Redundancy: Having multiple servers can increase the availability of your application. If one server fails, the others can take over.
Cost-effective: Rather than investing in a high-end server, you can distribute the load across multiple cost-effective servers.

Disadvantages of Horizontal Scaling:

Complexity: Horizontal scaling introduces complexity into your system. Your application code needs to handle distributing data across multiple servers, and this could require significant re-architecting of your application.
Data Consistency: When data is distributed across multiple servers, it's harder to maintain consistency. This is especially important for databases, where you want to ensure that all users are seeing the same data.

Scaling Databases

When it comes to databases, both scaling methods have their place.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Vertical scaling can be a good option for databases with high data integrity requirements, such as financial systems. However, at a certain point, the cost and limitations of hardware may necessitate a move to horizontal scaling.
Horizontal scaling is great for databases with large amounts of data that can be easily partitioned. For example, a database for a social media site could partition data based on user ID, with different users' data stored on different servers.

Conclusion

In conclusion, the choice between vertical and horizontal scaling depends on the specific needs of your database and application. You may also use a combination of both methods for different parts of your application, using a technique known as hybrid scaling.

For more information on database scaling, I would recommend checking out this post here:

Database Sharding Explained

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

How to Create a Sticky NavBar using Tailwind CSS

Bobby Iliev — Sun, 23 Apr 2023 02:50:37 -0700

Introduction

When building a modern, responsive website, navigation plays a crucial role in user experience. A sticky or affix navigation bar remains visible at the top of the screen as users scroll, making it easy for them to access the main menu items without having to scroll back to the top. Tailwind CSS, a popular utility-first CSS framework, makes it simple to create elegant, functional sticky navigation bars. In this tutorial, we'll walk you through the process of creating a sticky navigation bar using Tailwind CSS and showcase some stylish designs to inspire your project.

Step 1: Setting Up the Project

Before we start, make sure you have Tailwind CSS installed in your project. You can either use the CLI, include it from a CDN, or set up a custom build. Visit the official Tailwind CSS documentation to learn how to set up Tailwind CSS for your specific project.

First, let's create a simple navigation bar using Tailwind CSS. Add the following HTML code to your project:

html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.16/dist/tailwind.min.css" rel="stylesheet">
    <title>Sticky NavBar using Tailwind CSStitle>
head>
<body class="bg-gray-100">
    <header class="bg-white">
        <nav class="container mx-auto px-6 py-3">
            <div class="flex justify-between items-center">
                <a href="#" class="text-2xl font-bold text-gray-800">MyWebsitea>
                <div class="flex space-x-4">
                    <a href="#" class="text-gray-800">Homea>
                    <a href="#" class="text-gray-800">Abouta>
                    <a href="#" class="text-gray-800">Servicesa>
                    <a href="#" class="text-gray-800">Contacta>
                div>
            div>
        nav>
    header>
    
body>
html>

This code sets up a basic navigation bar with a container, logo, and menu links.

To make the navigation bar sticky, we need to add the fixed, top-0, and w-full classes to the

element. This will fix the header at the top of the viewport and span the full width of the screen. Update your

tag as follows:

class="bg-white fixed top-0 w-full">

Now, when you scroll down the page, the navigation bar will remain at the top.

Step 4: Adding a Shadow

To create a subtle separation between the sticky navigation bar and the page content, add a shadow using the shadow-md class:

class="bg-white fixed top-0 w-full shadow-md">

Step 5: Design Inspirations

Now that you've created a basic sticky navigation bar using Tailwind CSS, you can customize its design by applying various utility classes or creating your own. Here are a few design ideas to inspire you:

Transparent background with a color change on scroll: Add a transparent background to the navigation bar and change the background color when the user scrolls down, creating a smooth transition effect. You can achieve this by using JavaScript to toggle a class when the user scrolls.

Our Amazing Sponsors

View Website
DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.
View Website
Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.
View Website
A Laravel Starter Kit that includes Authentication, User Dashboard, Edit Profile, and a set of UI Components. Learn more about the DevDojo sponsorship program and see your logo here to get your brand in front of thousands of developers.
Hover effect on menu items: Enhance user experience by adding a hover effect to the menu items. Use the hover: prefix to change the text color or background color of the menu items when users hover over them. For example:

<a href="#" class="text-gray-800 hover:text-blue-600">Homea>

Add a call-to-action button: Encourage users to take a specific action, such as signing up or contacting you, by adding a CTA button to the navigation bar. Use the bg-, text-, and rounded- classes to style the button:

<a href="#" class="bg-blue-600 text-white px-4 py-2 rounded-md">Sign Upa>

Responsive design with a hamburger menu: For mobile devices, you can create a responsive design by hiding the menu items and displaying a hamburger menu instead. Use Tailwind CSS's responsive classes (e.g., lg:hidden and lg:flex) and JavaScript to toggle the mobile menu.
Add a search bar: Enhance your sticky navigation bar by integrating a search bar. Use the border-, rounded-, and focus: classes to style the input field and search button:

class="flex items-center space-x-2">
    <input type="search" placeholder="Search" class="border border-gray-300 rounded-md px-3 py-2 focus:outline-none focus:border-blue-600">
    <button class="bg-blue-600 text-white px-4 py-2 rounded-md">Searchbutton>
div>

Complete example

Here is the complete HTML code for a basic sticky navigation bar with some additional design elements, including hover effects, a CTA button, and a search bar:

html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <link href="https://cdn.jsdelivr.net/npm/tailwindcss@2.2.16/dist/tailwind.min.css" rel="stylesheet">
    <title>Sticky NavBar using Tailwind CSStitle>
head>
<body class="bg-gray-100">
    <header class="bg-white fixed top-0 w-full shadow-md">
        <nav class="container mx-auto px-6 py-3">
            <div class="flex justify-between items-center">
                <a href="#" class="text-2xl font-bold text-gray-800">MyWebsitea>
                <div class="hidden md:flex items-center space-x-4">
                    <a href="#" class="text-gray-800 hover:text-blue-600">Homea>
                    <a href="#" class="text-gray-800 hover:text-blue-600">Abouta>
                    <a href="#" class="text-gray-800 hover:text-blue-600">Servicesa>
                    <a href="#" class="text-gray-800 hover:text-blue-600">Contacta>
                    <a href="#" class="bg-blue-600 text-white px-4 py-2 rounded-md">Sign Upa>
                div>
                <div class="md:hidden flex items-center">
                    <button class="text-gray-800 focus:outline-none"> 
                        <svg class="w-6 h-6" fill="none" stroke="currentColor" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg">
                            <path stroke-linecap="round" stroke-linejoin="round" stroke-width="2" d="M4 6h16M4 12h16M4 18h16">path>
                        svg>
                    button>
                div>
            div>
            <div class="mt-4">
                <div class="flex items-center space-x-2">
                    <input type="search" placeholder="Search" class="border border-gray-300 rounded-md px-3 py-2 focus:outline-none focus:border-blue-600 w-full">
                    <button class="bg-blue-600 text-white px-4 py-2 rounded-md">Searchbutton>
                div>
            div>
        nav>
    header>
    
body>
html>

This example includes hover effects for menu items, a CTA button, and a search bar. You can further customize the design by applying different utility classes or creating your own. Remember to test your designs on various devices and screen sizes to ensure a seamless experience for all users.

Conclusion

In this tutorial, we demonstrated how to create a sticky navigation bar using Tailwind CSS and provided some design ideas to help you customize the look and feel of your NavBar.

By implementing a sticky navigation bar on your website, you can improve user experience and ensure that essential menu items are always easily accessible. To make the design process even more efficient and enjoyable, you can use the Tails Tailwind CSS page builder, which offers a comprehensive library of pre-built UI components, all created with Tailwind CSS.

This intuitive, user-friendly page builder allows you to effortlessly create and customize your website designs using a visual interface, without having to write a single line of code. Remember to test your designs on various devices and screen sizes to ensure a seamless experience for all users.

7 Best VS Code Extensions for Python Developers

Bobby Iliev — Mon, 10 Apr 2023 05:46:45 -0700

Introduction

Visual Studio Code (VS Code) has become a popular choice for Python developers due to its flexibility, extensibility, and user-friendly interface. One of the key features of VS Code is its extensive collection of extensions that can significantly improve your development experience.

In this post, we will highlight the 7 best VS Code extensions for Python developers to help you write better code, increase productivity, and enhance your overall coding experience.

1. Python (by Microsoft)

The Python extension by Microsoft is a must-have for any Python developer using VS Code. It provides a rich set of features, including:

IntelliSense (code completion)
Linting (error checking)
Debugging
Code navigation
Unit testing
Jupyter Notebooks support
Code refactoring

2. Pylance

Pylance is a fast and feature-rich language server for Python that provides excellent IntelliSense capabilities. It enhances your development experience with:

Type checking
Autocompletion
Type inference
Automatic imports
Signature help
Code navigation

Pylance works best when used alongside the Python extension by Microsoft.

Visual Studio Live Share is an extension that allows real-time collaboration with other developers, enabling you to share your code, workspace, and even terminal instances with others. This powerful tool is excellent for remote pair programming, code reviews, and group debugging sessions.

With Live Share, you can:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Share your workspace with read or write access
Co-edit and co-debug code
Share a local server and terminal instances
Communicate with collaborators using integrated audio and text chat

Live Share supports Python development and seamlessly integrates with the Python extension by Microsoft, making it an invaluable tool for Python developers working in teams or on collaborative projects.

4. Code Runner

Code Runner is a lightweight extension that allows you to run code snippets in various languages, including Python, with just a simple click or keyboard shortcut. It is extremely helpful for quickly testing your code without having to switch to the terminal or execute the entire script.

5. Better Comments

Better Comments is an extension that helps you create more human-friendly and visually distinguishable comments in your code. It supports different comment categories like queries, alerts, and highlights, making it easier to navigate through your comments and understand the purpose behind each one.

6. GitLens — Git supercharged

GitLens is a powerful extension that provides Git integration right inside VS Code. It offers various features to enhance your workflow, such as:

Inline Git blame annotations
Commit search and comparison
Git history explorer
Branch and commit visualization
Support for remote repositories (GitHub, GitLab, and more)

GitLens can be a game-changer for Python developers working on large projects or collaborating with a team.

7. Python Docstring Generator

Python Docstring Generator is a helpful extension that automatically generates docstrings for your Python functions and methods following popular docstring conventions, such as Google, NumPy, and reStructuredText. This extension saves time and ensures consistent documentation across your codebase.

To use this extension, simply place the cursor within a function or method definition and press the default keybinding Ctrl+Alt+D followed by Ctrl+Alt+S. The generated docstring will appear, and you can easily fill in the necessary details.

Conclusion

The 7 extensions mentioned in this post are essential for any Python developer using VS Code. They will help you write better code, increase productivity, and enhance your overall coding experience. By installing these extensions, you'll be well-equipped to tackle a wide range of Python development tasks more efficiently.

Remember, the VS Code marketplace offers countless other extensions that cater to specific needs and preferences. Feel free to explore the available extensions to further tailor your development environment to your unique requirements. Happy coding!

Introduction to Structs and Implementations in Rust

Bobby Iliev — Mon, 10 Apr 2023 04:10:22 -0700

Introduction

Structs in Rust are a way to create custom data types that group together related pieces of data. This tutorial will introduce you to the basics of structs in Rust, including defining, instantiating, and accessing their data. By the end of this tutorial, you'll have a solid understanding of how to use structs in your Rust projects.

Structs

To define a struct, use the struct keyword followed by the name of the struct and its fields within curly braces {}. Each field has a name and a type, separated by a colon:

struct User {
    username: String,
    email: String,
    age: u8,
    active: bool,
}

In this example, we've defined a User struct with four fields: username, email, age, and active.

Creating Instances of a Struct

To create an instance of a struct, specify the name of the struct followed by the field values inside curly braces {}:

let user1 = User {
    username: String::from("Alice"),
    email: String::from("alice@example.com"),
    age: 30,
    active: true,
};

Here, we've created an instance of the User struct called user1 with the specified field values.

Accessing Struct Fields

To access the value of a field in a struct, use the dot notation:

println!("Username: {}", user1.username);
println!("Email: {}", user1.email);
println!("Age: {}", user1.age);
println!("Active: {}", user1.active);

This will output the following:

Username: Alice
Email: alice@example.com
Age: 30
Active: true

Mutable Struct Instances

By default, struct instances are immutable. To make a mutable instance, use the mut keyword:

let mut user2 = User {
    username: String::from("Bob"),
    email: String::from("bob@example.com"),
    age: 25,
    active: false,
};

To update a field's value in a mutable struct, use the dot notation:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

user2.age = 26;
println!("Updated Age: {}", user2.age);

Output:

Updated Age: 26

Using Functions with Structs

Functions can accept and return struct instances. Here's an example of a function that takes a User struct and returns a String:

fn format_user(user: &User) -> String {
    format!("{} ({}) - {}", user.username, user.age, user.email)
}

let user_info = format_user(&user1);
println!("{}", user_info);

This will output:

Alice (30) - alice@example.com

Implementations

Implementations allow you to define methods and associated functions for a struct. Use the impl keyword followed by the name of the struct and a block containing the method and associated function definitions.

struct User {
    username: String,
    email: String,
    age: u8,
    active: bool,
}

impl User {
    // Method and associated function definitions go here
}

Methods:

Methods are functions associated with instances of a struct. To define a method, write a function within the impl block. The first parameter of a method is always self, which represents the instance the method is called on. Here's an example of a method that returns a formatted string for a User instance:

impl User {
    fn format_info(&self) -> String {
        format!("{} ({}) - {}", self.username, self.age, self.email)
    }
}

To call a method, use the dot notation:

let user1 = User {
    username: String::from("Alice"),
    email: String::from("alice@example.com"),
    age: 30,
    active: true,
};

let user_info = user1.format_info();
println!("{}", user_info);

This will output:

Alice (30) - alice@example.com

Associated Functions

Associated functions are similar to methods, but they don't have an instance of the struct as their first parameter. They are often used as constructors to create new instances of the struct. To define an associated function, write a function within the impl block without the self parameter:

impl User {
    fn new(username: String, email: String, age: u8) -> Self {
        User {
            username,
            email,
            age,
            active: true,
        }
    }
}

To call an associated function, use the double colon notation:

let user2 = User::new(
    String::from("Bob"),
    String::from("bob@example.com"),
    25,
);

println!("{}", user2.format_info());

This will output:

Bob (25) - bob@example.com

Implementing Traits for Structs

Traits are a way to define shared behavior between different structs in Rust. By implementing a trait for a struct, you can provide a common interface for different data types. Here's an example of implementing the std::fmt::Display trait for the User struct:

use std::fmt;

impl fmt::Display for User {
    fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
        write!(f, "{} ({}) - {}", self.username, self.age, self.email)
    }
}

Now you can use the User struct with any function or macro that requires the Display trait, like println!():

println!("{}", user1);

This will output:

Alice (30) - alice@example.com

Conclusion

In this tutorial, you've learned the basics of working with structs in Rust, including defining, creating, and modifying struct instances, as well as using functions with structs. Structs are a powerful way to group related data together and make your code more organized and maintainable. As you continue to explore Rust, you'll find that structs are a fundamental building block for creating complex data structures and handling real-world problems.

You've also learned how to extend structs in Rust with implementations, methods, and associated functions. Additionally, you've discovered how to implement traits for structs, allowing you to define shared behavior between different data types. These techniques will enable you to create more complex and powerful data structures in Rust while keeping your code organized and maintainable.

Happy coding!

8 Awesome VS Code Extensions for JavaScript Developers

Bobby Iliev — Mon, 03 Apr 2023 05:00:51 -0700

Introduction

Visual Studio Code (VS Code) is a popular, lightweight, and powerful source code editor developed by Microsoft. It has extensive support for JavaScript and TypeScript, making it the go-to choice for many developers. One of the most significant features of VS Code is its extensibility, allowing you to add custom extensions to enhance your development experience.

In this tutorial, we will explore 8 awesome VS Code extensions for JavaScript developers.

ESLint

ESLint is a widely used linting tool for JavaScript that analyzes your code and identifies potential problems, such as syntax errors, performance issues, and coding standard violations. This extension integrates ESLint into VS Code, providing real-time feedback and highlighting issues in your code as you type.

Prettier - Code formatted

Prettier is an opinionated code formatter that supports JavaScript, TypeScript, and many other languages. The Prettier extension for VS Code formats your code automatically on save or manually via a command, ensuring consistent code style across your project.

Debugger for Chrome

Debugger for Chrome connects the Google Chrome debugger to VS Code, allowing you to debug your JavaScript code directly within the editor. With this extension, you can set breakpoints, step through code, inspect variables, and more without leaving VS Code.

npm Intellisense

npm Intellisense is a must-have for developers who work with Node.js and npm packages. This extension provides autocompletion for npm modules in your code, making it easy to import and require packages without having to memorize package names.

GitLens — Git supercharged

GitLens is an extension that supercharges Git functionality in VS Code. It provides features like inline blame annotations, commit searching, and file history explorers. GitLens is especially useful for JavaScript developers working in large teams or on open-source projects, as it helps you navigate and understand codebases faster.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Import Cost

Import Cost is a useful extension that displays the size of imported packages inline, helping you keep track of your application's bundle size. With this extension, you can make informed decisions about adding or removing dependencies based on their impact on the overall size of your project.

Live Server

Live Server is a simple but powerful extension that launches a local development server with live reloading for static and dynamic pages. This is especially useful for front-end JavaScript developers, as it allows you to see changes in real-time without manually refreshing the browser.

REST Client

REST Client is a convenient VS Code extension that lets you send HTTP requests and view responses directly within the editor. It's great for JavaScript developers working with APIs, as it allows you to test and debug endpoints without needing a separate tool like Postman.

Conclusion

These 8 awesome VS Code extensions can significantly improve your productivity and development experience as a JavaScript developer. By leveraging these tools, you'll be able to write cleaner, more efficient code, debug your applications more effectively, and more.

Happy coding!

Simulating Streaming Data for Fraud Detection with Datagen CLI

Bobby Iliev — Mon, 03 Apr 2023 04:28:54 -0700

Introduction

Building and testing a real-time fraud detection application requires a continuous stream of realistic data. But generating that data can be a challenge. That's why we recently created the Datagen CLI, a simple tool that helps you create believable fake data using the FakerJS API.

In this blog post, we'll explore how to use the Datagen CLI to simulate a streaming data use-case like a fraud detection app. I'll show you how to install and configure the tool, create a schema for the data, and send the data to a Kafka topic. By the end of this tutorial, you'll be able to generate your own realistic streaming data for testing and development purposes.

Prerequisites

Basic knowledge of Kafka and streaming data
Node.js and npm installed on your machine
A Kafka cluster set up and running

Installation and Setup

First, install the Datagen CLI using npm:

npm install -g @materializeinc/datagen

Create a .env file in your working directory with the necessary Kafka and Schema Registry environment variables. Replace the placeholder values with your actual settings:

# Kafka Brokers
KAFKA_BROKERS=

# For Kafka SASL Authentication:
SASL_USERNAME=
SASL_PASSWORD=
SASL_MECHANISM=

Creating a Schema for Fraud Detection Data

To generate realistic data for a fraud detection app, we need to define a schema that includes relevant fields like transaction ID, user ID, timestamp, and transaction amount. Let's create a JSON schema called transactions.json:

[
  {
    "_meta": {
      "topic": "transactions"
    },
    "transaction_id": "faker.datatype.uuid()",
    "user_id": "faker.datatype.number({min: 1, max: 10000})",
    "timestamp": "faker.date.between('2023-01-01', '2023-12-31')",
    "amount": "faker.finance.amount(0, 10000, 2)",
    "is_fraud": "faker.datatype.boolean()"
  }
]

This schema generates a stream of transaction data with random transaction IDs, user IDs, timestamps, and amounts. We've also added a field called is_fraud that randomly the transactions as fraudulent.

Generating and Sending Data to Kafka

Now that we have our schema, we can use the Datagen CLI to generate data and send it to a Kafka topic. Use the following command to generate an infinite stream of transactions in JSON format:

datagen \
  -s transactions.json \
  -f json \
  -n -1 \
  -dr

The -n flag specifies the number of messages to generate. We've set it to -1 to generate an infinite stream of data. The -dr flag enables dry run mode, which prints the data to the console instead of sending it to Kafka. This is useful for testing and debugging.

Example output:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

✔  Dry run: Skipping record production...  
  Topic: transactions 
  Record key: null 
  Payload: {"transaction_id":"b86d1d57-a650-4680-843d-06179f1c4c2e","user_id":5127,"timestamp":"2023-09-02T03:26:28.194Z","amount":"6904.40","is_fraud":false}


✔  Dry run: Skipping record production...  
  Topic: transactions 
  Record key: null 
  Payload: {"transaction_id":"719fe62a-322c-4b58-89f9-e380e2f3552d","user_id":2757,"timestamp":"2023-09-30T06:40:37.378Z","amount":"3375.15","is_fraud":true}

Press Ctrl+C to stop producing data.

Enriching the Data

To enrich the datagen schema example, we can add more fields related to geolocation and other attributes that can be useful for fraud detection. Update the JSON input schema as follows:

[
  {
    "_meta": {
      "topic": "transactions",
      "key": "id"
    },
    "id": "faker.datatype.uuid()",
    "user_id": "faker.datatype.number({min: 1, max: 1000})",
    "amount": "faker.finance.amount(1, 5000, 2)",
    "currency": "faker.finance.currencyCode()",
    "timestamp": "faker.date.past(1, '2023-01-01').getTime()",
    "is_fraud": "faker.datatype.boolean({likelihood: 5})",
    "ip_address": "faker.internet.ip()",
    "location": {
      "latitude": "faker.address.latitude()",
      "longitude": "faker.address.longitude()"
    },
    "device": {
      "id": "faker.datatype.uuid()",
      "type": "faker.helpers.arrayElement(['mobile', 'tablet', 'desktop'])",
      "os": "faker.helpers.arrayElement(['ios', 'android', 'windows', 'macos', 'linux', 'other'])"
    },
    "merchant_id": "faker.datatype.number({min: 1, max: 500})"
  }
]

In this enriched schema, we've added:

ip_address: An IP address related to the transaction.
location: An object containing latitude and longitude.
device: An object containing device information such as ID, type, and operating system.
merchant_id: A unique ID representing the merchant involved in the transaction.

Relationship between Transactions and Users

The Datagen CLI can also generate data for related entities.

For example, we can extend the schema to include a users topic that contains user information. We can then use the user_id field in the transactions topic to join the two topics together:

[
  {
    "_meta": {
      "topic": "users",
      "key": "id",
      "relationships": [
        {
          "topic": "transactions",
          "parent_field": "id",
          "child_field": "user_id",
          "records_per": 10
        }
      ]
    },
    "id": "faker.datatype.number({min: 1, max: 1000})",
    "name": "faker.name.fullName()",
    "email": "faker.internet.email()",
    "registered_at": "faker.date.past(5, '2023-01-01').getTime()"
  },
  {
    "_meta": {
      "topic": "transactions",
      "key": "id"
    },
    "id": "faker.datatype.uuid()",
    "user_id": "faker.datatype.number(100)",
    "amount": "faker.finance.amount(1, 5000, 2)",
    "currency": "faker.finance.currencyCode()",
    "timestamp": "faker.date.between('relationship.registered_at', new Date()).getTime()",
    "is_fraud": "faker.datatype.boolean({likelihood: 5})",
    "ip_address": "faker.internet.ip()",
    "location": {
      "latitude": "faker.address.latitude()",
      "longitude": "faker.address.longitude()"
    },
    "device": {
      "id": "faker.datatype.uuid()",
      "type": "faker.helpers.arrayElement(['mobile', 'tablet', 'desktop'])",
      "os": "faker.helpers.arrayElement(['ios', 'android', 'windows', 'macos', 'linux', 'other'])"
    },
    "merchant_id": "faker.datatype.number({min: 1, max: 500})"
  }
]

The data will be produced to the users and transactions topics. The transactions topic will contain a user_id field that references the id field in the users topic. The transactions topic will also contain a registered_at field that references the registered_at field in the users topic.

An example of the data produced:

...
  Topic: users 
  Record key: 602 
  Payload: {"id":602,"name":"Mr. Jennie Prohaska","email":"Natalie60@hotmail.com","registered_at":1591058898886}


  Topic: transactions 
  Record key: 417f6a6b-d7c5-47a0-a013-93f2aee94941 
  Payload: {"user_id":602,"id":"417f6a6b-d7c5-47a0-a013-93f2aee94941","amount":"1760.29","currency":"MZN","timestamp":1680516946423,"is_fraud":true,"ip_address":"240.254.28.18","location":{"latitude":"60.3920","longitude":"11.9718"},"device":{"id":"1e57d9d2-de48-4bf2-9131-e70ceb5f4fee","type":"mobile","os":"macos"},"merchant_id":20}
...

Testing Your Fraud Detection App

With realistic streaming data available, you can now test your fraud detection app using the data generated by the Datagen CLI. Consume the data from the transactions Kafka topic and implement your fraud detection logic, which may involve analyzing transaction patterns, comparing with historical data, or applying machine learning models.

As a next step you can use Materialize to create a materialized view of the transactions data. This will allow you to query the data in real-time and build a fraud detection dashboard.

Conclusion

The Datagen CLI is a simple tool for generating realistic streaming data for testing and development purposes. In this tutorial, we showcased how to use Datagen CLI to simulate a fraud detection app use-case. With this knowledge, you can create your own schemas and generate data for various streaming data applications.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Useful Links:

Reset Local Repository Branch to be Just Like Remote Repository HEAD: A Step-by-Step Guide

Bobby Iliev — Thu, 30 Mar 2023 12:23:52 -0700

Introduction

In the world of Git, it's not uncommon for developers to find themselves in a situation where their local repository has diverged from the remote repository. This can happen due to accidental changes, conflicts, or outdated branches.

To get your local branch back in sync with the remote repository HEAD, you'll need to reset it. In this blog post, we'll walk you through the steps to reset your local repository branch to be just like the remote repository HEAD, making sure your Git workflow stays efficient and up-to-date.

Step 1: Fetch the Latest Changes from the Remote Repository

Before resetting your local branch, you'll first need to fetch the latest changes from the remote repository. To do this, open your terminal or command prompt and navigate to the root directory of your local Git repository. Then, execute the following command:

git fetch origin

This command fetches the latest changes from the remote repository (usually called "origin") without merging them into your local branch.

Step 2: Reset Your Local Branch to the Remote Repository HEAD

Now that you've fetched the latest changes, it's time to reset your local branch to match the remote repository HEAD. Execute the following command, replacing "branch-name" with the name of the branch you want to reset:

git reset --hard origin/branch-name

This command resets your local branch to the specified remote branch, discarding any local changes that were made since the last commit.

Step 3: Verify Your Local Branch is in Sync with the Remote Repository HEAD

To ensure that your local branch has been successfully reset and is now in sync with the remote repository HEAD, execute the following command:

git status

If your local branch has been successfully reset, you should see a message stating that your branch is up-to-date with the remote branch.

Conclusion

Resetting your local repository branch to match the remote repository HEAD is a simple but essential skill when working with Git.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

By following these steps, you'll ensure that your Git workflow remains efficient and up-to-date.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

For even more helpful Git tips and tricks, be sure to download our free Introduction to Git and GitHub eBook!

Happy coding!

Introduction to Rust

Bobby Iliev — Fri, 24 Mar 2023 09:17:30 -0700

What is Rust?

Rust is a systems programming language. It is designed to be fast, reliable, and maintainable.

Rust was started in 2006 by Graydon Hoare at Mozilla Research. It was originally designed to be a safer alternative to C++. It has since become a popular language for writing low-level code, such as operating systems and device drivers. Version 1.0 was released in 2015.

Firefox started sponsoring Rust and Firefox Quantum was rewritten in Rust in 2017, which boosted Firefox's performance.

Installing Rust

To install Rust, you can use

rustup

. Rustup is the official tool for installing Rust. It will install the Rust compiler and Cargo, Rust's package manager, and build tool.

curl --proto '=https' --tlsv1.2 -sSf //sh.rustup.rs> | sh

What is Cargo?

Cargo is Rust's package manager and build tool. It is used to build, test, and package Rust projects.

If you are coming from a JavaScript background, Cargo is similar to npm. Or if you are coming from a Python background, Cargo is similar to pip.

The compiler that Cargo uses is called rustc. You can use rustc to compile a single file.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Hello World

To create your first Rust project, you can use Cargo's new command. This will create a new project with the name hello_world.

bash cargo new hello_world

Output:

     Created binary (application) `hello_world` package

This will create a new directory called hello_world with the following structure:

hello_world
├── Cargo.toml
└── src
   └── main.rs

The Cargo.toml file contains the project's metadata and is the configuration file for your project. Rust uses semantic versioning, so you can specify the version of Rust that your project requires.

The src directory contains the source code for the project.

The main.rs file is the entry point for the project.

To run the project, you can use the run command which also builds the project.

bash cargo run

Output:

   Compiling
   hello_world v0.1.0 (hello_world)
   Finished dev [unoptimized + debuginfo] target(s) in 0.82s
   Running `target/debug/hello_world`
   Hello, world!

The target directory contains the compiled project. The debug directory contains the unoptimized version of the project.

You can add --release to the run command to build the project in release mode.

cargo run --release

The release directory contains the optimized version of the project.

Variables

To declare a variable, you can use the let keyword. The let keyword is used to create a variable that is immutable by default. This means that the variable cannot be changed once it is created this improves the safety and the performance.

let x = 5;

Rust is a strongly typed language, so you must specify the type of the variable. The type of the variable is inferred from the value that is assigned to the variable.

let x: i32 = 5;

You can initialize multiple variables on the same line.

let (x, y) = (1, 2);

To initialize a mutable variable, you can use the mut keyword.

let mut x = 5;

As in almost all programming languages, you can declare constants with the const keyword.

const MAX_POINTS: u32 = 100_000;

The const keyword is used to create a constant that is valid for the entire time a program runs. Constants are always immutable. The convention for constants is to use all uppercase with underscores between words.

Scope

Variables are scoped to the block that they are declared in. This means that variables declared in a block are only accessible in that block.

fn main() {
    let x = 1;
    let y = 10;
    println!("x = {}, y = {}", x, y);
}

The y variable is only accessible in the block that it is declared in. If you try to access the y variable outside of the block, you will get a compiler error.

error[E0425]: cannot find value `y` in this scope  --> src/main.rs:6:35
| 6 |     println!("x = {}, y = {}", x, y);     |
                    ^ not found in this scope

Shadowing

You can declare a new variable with the same name as a previous variable. This is called shadowing. The new variable shadows the previous variable. The previous variable is no longer accessible.

fn main() {
    let x = 1;
    let x = x + 1;
    println!("x = {}", x);
}

The x variable is shadowed by the x variable on the second line.

The first x variable is no longer accessible.

Memory Safety

Rust is a memory safe language. This means that Rust prevents memory errors such as null pointer dereference, use after free, and double free.

Uninitialized memory is a common source of memory errors. Rust prevents uninitialized memory by requiring you to initialize all variables.

fn main() {
    let x: i32;
    println!("x = {}", x);
}

If you try to use an uninitialized variable, you will get a compiler error.

error[E0381]: use of possibly uninitialized variable: `x`  --> src/main.rs:3:25
| 3 |     println!("x = {}", x);     |
                    ^ use of possibly uninitialized `x`

Functions

Functions are declared with the fn keyword.

The fn keyword is used to declare a function that is immutable by default. This means that the function cannot be changed once it is created.

fn hello_world() {
    println!("Hello, world!");
}

The Rust guidelines recommend using snake_case for function names.

To call a function, you can use the function name followed by parentheses.

fn hello_world() {
    println!("Hello, world!");
}
fn main() {
    hello_world();
}

Functions do not have to appear in the same order as they are called. You can call a function before it is declared.

fn main() {
    hello_world();
}
fn hello_world() {
    println!("Hello, world!");
}

Parameters

Functions can have parameters. Parameters are declared in the function signature.

fn hello_world(name: &str) {
    println!("Hello, {}!", name);
}
fn main() {
    hello_world("World");
}

The &str type is a string slice. A string slice is a reference to a string.

String slices are immutable by default. This means that the string slice cannot be changed once it is created.

Return Values

Functions can return values. The return type is declared in the function signature.

fn add(x: i32, y: i32) -> i32 {
    x + y
}
fn main() {
    let sum = add(1, 2);
    println!("sum = {}", sum);
}

The -> i32 part of the function signature declares the return type of the function. The -> i32 part of the function signature is optional.

If the function does not return a value, you can omit the -> i32 part of the function signature.

Modules System

Rust uses a module system to organize code. Modules are declared with the mod keyword.

mod hello_world {
    pub fn hello_world() {
        println!("Hello, world!");
    }
}
fn main() {
    hello_world::hello_world();
}

The use keyword is used to bring a module into scope.

mod hello_world {
    pub fn hello_world() {
        println!("Hello, world!");
    }
}
use hello_world::hello_world;
fn main() {
    hello_world();
}

Scalar Types

There are 4 scalar types in Rust: integers, floating-point numbers, Booleans, and characters.

Types of integers:

Signed integers: i8, i16, i32, i64, i128, isize
Unsigned integers: u8, u16, u32, u64, u128, usize

The isize and usize types depend on the kind of computer that your program is running on: 64 bits if you’re on a 64-bit architecture and 32 bits if you’re on a 32-bit architecture.

The default integer type is i32. This is usually the fastest, even on 64-bit systems. The default integer type can be changed with the #![feature(default_int_type)] attribute.

#![feature(default_int_type)]
fn main() {
    let x = 1;
    println!("x = {}", x);
}

Floating-point numbers:

f32
f64

The default floating-point type is f64. This type is usually the fastest, even on 32-bit systems. The default floating-point type can be changed with the #![feature(default_float_type)] attribute.

#![feature(default_float_type)]
fn main() {
    let x = 1.0;
    println!("x = {}", x);
}

Booleans:

true
false

Characters are always 4 bytes in size and represent a Unicode Scalar Value. Literal characters are specified with single quotes.

Strings do not use the character type. Strings are a collection of characters.

Compound Types

There are 2 compound types in Rust: tuples and arrays.

Tuples

Tuples are a collection of values of different types. Tuples are declared with parentheses.

fn main() {
    let x = (1, 2.0, "three");
    println!("x = {:?}", x);
}

The {:?} format specifier is used to print the tuple.

You can use a dot to access a tuple element.

fn main() {
    let x = (1, 2.0, "three");
    println!("x.0 = {}", x.0);
    println!("x.1 = {}", x.1);
    println!("x.2 = {}", x.2);
}

Arrays

Arrays are a collection of values of the same type. Arrays are declared with square brackets.

fn main() {
    let x = [1, 2, 3];
    println!("x = {:?}", x);
}

The {:?} format specifier is used to print the array.

Arrays are limited to a size of 32 elements. If you need a collection of more than 32 elements, you should use a vector.

You can use a dot to access an array element.

fn main() {
    let x = [1, 2, 3];
    println!("x[0] = {}", x[0]);
    println!("x[1] = {}", x[1]);
    println!("x[2] = {}", x[2]);
}

Control Flow

`if` Expressions

The if keyword is used to create an if expression.

fn main() {
    let x = 1;
    if x == 1 {
        println!("x is 1");
    }
}

`if`-`else` Expressions

The else keyword is used to create an if-else expression.

fn main() {
    let x = 1;
    if x == 1 {
        println!("x is 1");
    } else {
        println!("x is not 1");
    }
}

`if`-`else if`-`else` Expressions

The else if keyword is used to create an if-else if-else expression.

fn main() {
    let x = 1;
    if x == 1 {
        println!("x is 1");
    } else if x == 2 {
        println!("x is 2");
    } else {
        println!("x is not 1 or 2");
    }
}

You can assign the result of an if expression to a variable.

fn main() {
    let x = 1;
    let y = if x == 1 {
        2
    } else {
        3
    };
    println!("y = {}", y);
}

Note that there is a semicolon after the if expression when it is assigned to a variable.

We can't use return in an if expression.

There is no ternary operator in Rust. The if expression is the closest thing to a ternary operator.

fn main() {
    let x = 1;
    let y = if x == 1 { 2 } else { 3 };
    println!("y = {}", y);
}

`loop` Expressions

The loop keyword is used to create a loop expression.

fn main() {
    loop {
        println!("Hello, world!");
    }
}

The break keyword is used to break out of a loop expression.

fn main() {
    loop {
        println!("Hello, world!");
        break;
    }
}

To break a nested loop expression, you can use a label.

fn main() {
    'outer: loop {
        'inner: loop {
            println!("Hello, world!");
            break 'outer;
        }
    }
}

`while` Expressions

The while keyword is used to create a while expression.

fn main() {
    let mut x = 1;
    while x <= 3 {
        println!("x = {}", x);
        x += 1;
    }
}

There is no do-while loop in Rust. But you can use a loop expression with a break expression.

fn main() {
    let mut x = 1;
    loop {
        println!("x = {}", x);
        x += 1;
        if x > 3 {
            break;
        }
    }
}

`for` Expressions

The for keyword is used to create a for expression.

fn main() {
    for x in 1..4 {
        println!("x = {}", x);
    }
}

The iter method is used to iterate over a collection.

fn main() {
    let x = [1, 2, 3];
    for y in x.iter() {
        println!("y = {}", y);
    }
}

The enumerate method is used to iterate over a collection and get the index of each element.

fn main() {
    let x = [1, 2, 3];
    for (i, y) in x.iter().enumerate() {
        println!("i = {}, y = {}", i, y);
    }
}

The for can use a range.

fn main() {
    for x in 1..4 {
        println!("x = {}", x);
    }
}

The for can use a range with a step.

fn main() {
    for x in (1..4).step_by(2) {
        println!("x = {}", x);
    }
}

Strings

There are at least 6 different string types in Rust. But we will focus on the most common string types.

String slices

String slices are a reference to a string. String slices are declared with double quotes.

fn main() {
    let x = "Hello, world!";
    println!("x = {}", x);
}

A literal string always a borrowed string slice.

fn main() {
    let x = "Hello, world!";
    let y = "Hello, world!";
    println!("x = {}", x);
    println!("y = {}", y);
    println!("x == y: {}", x == y);
}

`String` type

The String type is a heap-allocated string. The String type is declared with the String::new function.

fn main() {
    let x = String::new();
    println!("x = {}", x);
}

The String type is declared with the to_string method.

fn main() {
    let x = "Hello, world!".to_string();
    println!("x = {}", x);
}

The String type is declared with the String::from function.

fn main() {
    let x = String::from("Hello, world!");
    println!("x = {}", x);
}

A borrowed string slice can be converted to a String type.

fn main() {
    let x = "Hello, world!".to_string();
    println!("x = {}", x);
}

`&str` type

The &str type is a borrowed string slice. The &str type is declared with the & operator.

fn main() {
    let x = "Hello, world!";
    println!("x = {}", x);
}

A String type can be converted to a borrowed string slice.

fn main() {
    let x = "Hello, world!".to_string();
    let y = &x;
    println!("x = {}", x);
    println!("y = {}", y);
}

A vector of String types example.

fn main() {
    let x = vec!["Hello, world!".to_string(), "Hello, world!".to_string()];
    println!("x = {:?}", x);
}

Ownership

Ownership is a Rust feature that allows you to manage memory. The Rust compiler ensures that memory is always valid.

Each value in Rust has a variable that's called its owner. There can only be one owner at a time. When the owner goes out of scope, the value will be dropped.

There is only one owner of a value. The value is moved to the new owner.

When the owner goes out of scope, the value will be dropped.

fn main() {
    let x = 1;
    let y = x;
    println!("x = {}", x); // error here
    println!("y = {}", y); // value moved here
}

Stack and Heap

Stack:

Fast access
Fixed size
Last in, first out
In order

Heap:

Slow access
Dynamic size
First in, first out
Out of order

Clone

The clone method is used to clone a value.

fn main() {
    let x = 1;
    let y = x.clone();
    println!("x = {}", x);
    println!("y = {}", y);
}

References and Borrowing

References allow you to refer to some value without taking ownership of it.

fn main() {
    let x = 1;
    let y = &x;
    println!("x = {}", x);
    println!("y = {}", y);
}

The & operator is used to create a reference.

Under the hood, the & operator is a borrow method.

References are immutable by default. The &mut operator is used to create a mutable reference.

fn main() {
    let mut x = 1;
    let y = &mut x;
    *y += 1;
    println!("x = {}", x);
    println!("y = {}", y);
}

To dereference a reference, use the * operator.

fn main() {
    let mut x = 1;
    let y = &mut x;
    *y += 1;
    println!("x = {}", x);
    println!("y = {}", y);
}

Structs

A struct is a custom data type that you can name and shape however you want.

struct Point {
    x: i32,
    y: i32,
}

Note that there is a comma after the last field.

Instanciating a struct.

```rs
fn main() {
    let x = Point { x: 1, y: 2 };
    println!("x = {:?}", x);
}

An associated function is a function that is associated with a struct.

impl Point {
    fn new(x: i32, y: i32) -> Self {
        Self { x, y }
    }
}

fn main() {
    let x = Point::new(1, 2);
    println!("x = {:?}", x);
}

The Self type is an alias for the type that the impl block is for.

Methods

Methods are functions that are associated with a struct.

impl Point {
    fn new(x: i32, y: i32) -> Self {
        Self { x, y }
    }

    fn distance(&self) -> f64 {
        ((self.x.pow(2) + self.y.pow(2)) as f64).sqrt()
    }
}

fn main() {
    let x = Point::new(1, 2);
    println!("x = {:?}", x);
    println!("x.distance() = {}", x.distance());
}

Traits

Traits are similar to interfaces in other languages.

trait HasArea {
    fn area(&self) -> f64;
}

impl HasArea for Point {
    fn area(&self) -> f64 {
        0.0
    }
}

fn main() {
    let x = Point::new(1, 2);
    println!("x = {:?}", x);
    println!("x.area() = {}", x.area());
}

Rundown of the code above:

The HasArea trait is defined.
The HasArea trait is implemented for the Point struct.
The area method is defined.
The area method is implemented for the Point struct.

A hands on example of traits.

trait Car {
    fn new() -> Self;
    fn drive(&self);
}

struct Toyota {
    model: String,
}

impl Car for Toyota {
    fn new() -> Self {
        Self {
            model: "Corolla".to_string(),
        }
    }

    fn drive(&self) {
        println!("The {} is driving.", self.model);
    }
}

fn main() {
    let x = Toyota::new();
    x.drive();
}

Copy

The Copy trait is used to indicate that a type can be copied.

#[derive(Copy, Clone)]
struct Point {
    x: i32,
    y: i32,
}

fn main() {
    let x = Point { x: 1, y: 2 };
    let y = x;
    println!("x = {:?}", x);
    println!("y = {:?}", y);
}

Debug

The Debug trait is used to indicate that a type can be printed.

#[derive(Debug)]
struct Point {
    x: i32,
    y: i32,
}

fn main() {
    let x = Point { x: 1, y: 2 };
    println!("x = {:?}", x);
}

Collections

Collections are data structures that can group multiple values into a single type.

Vectors

Vectors are resizable arrays. They can grow or shrink in size.

fn main() {
    let mut x = vec![1, 2, 3];
    x.push(4);
    println!("x = {:?}", x);
}

The push and pop methods are used to add and remove elements from the end of a vector.

fn main() {
    let mut x = vec![1, 2, 3];
    x.push(4);
    x.pop();
    println!("x = {:?}", x);
}

The !vec macro is used to create a vector.

Hash Maps

Hash maps are key-value stores. In some languages, they are also known as dictionaries or associative arrays.

fn main() {
    let mut x = HashMap::new();
    x.insert("one", 1);
    x.insert("two", 2);
    println!("x = {:?}", x);
}

Common methods on hash maps:

insert: Insert a key-value pair into a hash map.
get: Get the value associated with a key.
remove: Remove a key-value pair from a hash map.
len: Get the number of key-value pairs in a hash map.

fn main() {
    let mut x = HashMap::new();
    x.insert("one", 1);
    x.insert("two", 2);
    println!("x = {:?}", x);
    println!("x.get(\"one\") = {:?}", x.get("one"));
    x.remove("one");
    println!("x = {:?}", x);
    println!("x.len() = {:?}", x.len());
}

Enums

Enums are types that have a few definite values.

enum Color {
    Red,
    Green,
    Blue,
}

fn main() {
    let x = Color::Red;
    println!("x = {:?}", x);
}

Enums can also have data associated with them.

enum Color {
    Red,
    Green,
    Blue,
    RgbColor(u8, u8, u8),
    CmykColor { cyan: u8, magenta: u8, yellow: u8, black: u8 },
}

fn main() {
    let x = Color::RgbColor(0, 0, 0);
    println!("x = {:?}", x);
}

Option

The Option enum is used to indicate that a value may or may not be present.

fn main() {
    let x = Some(1);
    println!("x = {:?}", x);
}

The match expression is used to handle the different cases of an enum.

fn main() {
    let x = Some(1);
    match x {
        Some(i) => println!("x = {:?}", i),
        None => println!("x is empty"),
    }
}

To create a None value, use the None keyword.

fn main() {
    let x: Option = None;
    println!("x = {:?}", x);
}

Result

The Result enum is used to indicate that a function may or may not succeed.

#[must_use]
enum Result<T, E> {
    Ok(T),
    Err(E),
}

The Result enum has two variants: Ok and Err. The Ok variant indicates that the function succeeded. The Err variant indicates that the function failed.

use std::fs::File;

fn main() {
    let x = File::open("hello.txt");
    println!("x = {:?}", x);
}

You can unwrap a Result to get the value inside the Ok variant.

use std::fs::File;

fn main() {
    let x = File::open("hello.txt").unwrap();
    println!("x = {:?}", x);
}

You can also expect a Result to get the value inside the Ok variant.

use std::fs::File;

fn main() {
    let x = File::open("hello.txt").expect("Failed to open hello.txt");
    println!("x = {:?}", x);
}

Closures

A closure is a function that can capture its environment.

The type and the arguments of a closure are defined in the same way as a function.

fn main() {
    let x = |a: i32, b: i32| -> i32 { a + b };
    println!("x(1, 2) = {}", x(1, 2));
}

A closure will borrow a reference:

fn main() {
    let x = 1;
    let y = |a: i32| -> i32 { a + x };
    println!("y(2) = {}", y(2));
}

Here is an example on how to use a closure to sort a vector.

fn main() {
    let mut x = vec![1, 3, 2];
    x.sort_by(|a, b| a.cmp(b));
    println!("x = {:?}", x);
}

Rundown of the code above:

We create a vector with the values 1, 3, and 2.
We sort the vector using the sort_by method.
The sort_by method takes a closure as an argument.
The closure takes two arguments: a and b.
The closure returns a std::cmp::Ordering value.
The cmp method compares two values and returns an Ordering value.
The sort_by method sorts the vector based on the Ordering value returned by the closure.

Threads

Threads are used to run multiple tasks simultaneously.

use std::thread;

fn main() {
    thread::spawn(|| {
        println!("Hello from a thread!");
    });
    println!("Hello from the main thread!");
}

The thread::spawn function takes a closure as an argument and returns a JoinHandle. A JoinHandle is a handle to a thread that can be used to wait for the thread to finish.

The more threads you have, the more context-switching you have to do. This can slow down your program if you have too many threads.

Conclusion

This is the end of the Rust tutorial. I hope you enjoyed it. If you have any questions, feel free to ask them in the community forum.

As the next steps, I recommend you to:

Read the Rust book.
Read the Rust reference.
Read the Rust by example.
Read the Rustonomicon.

How to increase the size of the file upload in Laravel Voyager Media Manager?

Bobby Iliev — Thu, 05 Jan 2023 01:11:44 -0800

Introduction

Laravel Voyager is a great open-source admin panel for Laravel. It is also the admin panel currently used in Laravel Wave which is a Laravel SaaS starter kit.

In this article, you will learn how to increase the size of the file upload in Laravel Voyager Media Manager.

PHP changes

First, you will need to increase the size of the file upload in the PHP configuration file. You can do this by editing the php.ini file and changing the upload_max_filesize and post_max_size values.

If you don't know where your php.ini file is located, you can create a new file called info.php and add the following code:


phpinfo();

Then, you can access the info.php file in your browser and it will show you the location of your php.ini file.

After you have increased the size of the file upload in the PHP configuration file, you will need to restart your web server. And then visit the info.php file again to make sure that the changes have been applied.

Dropzone.js changes

The Laravel Voyager Media Manager uses the Dropzone.js library to handle the file upload. So, you will need to increase the size of the file upload in the Dropzone definition.

The default size of the file upload in the Laravel Voyager Media Manager is 256MB.

For this, we will copy the media manager view file from the Laravel Voyager package to our project. You can do this by running the following command:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Create the resources/views/vendor/voyager/media directory if it doesn't exist:

mkdir -p resources/views/vendor/voyager/media

Copy the manager.blade.php file from the Laravel Voyager package to our project:

cp vendor/tcg/voyager/resources/views/media/manager.blade.php resources/views/vendor/voyager/media/manager.blade.php

Now, you can open the resources/views/vendor/voyager/media/manager.blade.php file and set the maxFilesize value to the desired size.

Open the file and go to line 847:

nano +847 resources/views/vendor/voyager/media/manager.blade.php

And inside the dropzone function add a new line and set the maxFilesize value to the desired size:

...
            dropzone.dropzone({
                // Add the line below
                maxFilesize: 500,
                timeout: 180000,
                url: '{{ route('voyager.media.upload') }}',
...

Save the file and you are done. After that go to the media manager page and do a hard refresh to make sure that the cache is cleared and try to upload a file that is larger than the default size.

Conclusion

In this article, you have learned how to increase the size of the file upload in Laravel Voyager Media Manager.

If you are building a SaaS application, you can use the Laravel Wave starter kit which will let you build a SaaS application quickly as it comes with a lot of features out of the box!

If you have any questions, feel free to ask them in the community forum at https://devdojo.com/forum.

How to find a process locking port 3000 on Mac?

Bobby Iliev — Wed, 21 Dec 2022 03:17:37 -0800

Introduction

Are you having trouble with a process locking port 3000 (or essentially any other port) on your Mac? Here's how you can find and kill the process!

Finding the process

To find the process, first open the Terminal app on your Mac.

You can find it in the Utilities folder within your Applications folder.

Then ype the following command and press Enter:

sudo lsof -i :3000

This command will list all the processes that are currently using port 3000. The output will include the process ID (PID), the user that owns the process, and the name of the process.

Kill the process

Once you've identified the process that you want to kill and note down its PID.

To kill the process, use the following command:

kill -9 <PID>

Note: Replace with the actual PID of the process.

The -9 option kills the process immediately, without giving it the opportunity to clean up after itself. This can potentially cause issues. As an alternative, you may want to consider using the -15 (TERM) or -3 (QUIT) option for a gentler termination, which allows the process to perform any necessary cleanup before it is terminated.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Conclusion

If the process was successfully killed, you should no longer see it listed when you run the lsof command again.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

I hope this helps! Let me know if you have any questions.

How to setup Laravel Wave on Windows using Laragon?

Bobby Iliev — Sat, 10 Dec 2022 06:14:31 -0800

Introduction

Laravel is a free, open-source PHP web framework, created by Taylor Otwell and intended for the development of web applications following the model–view–controller (MVC) architectural pattern and based on Symfony. Some of the features of Laravel are a modular packaging system with a dedicated dependency manager, different ways for accessing relational databases, utilities that aid in application deployment and maintenance, and its orientation toward syntactic sugar.

Laravel Wave is a SaaS starter-kit built with Laravel and Tailwind CSS. It is a fully functional SaaS application that you can use to build your own SaaS application. It comes with a lot of features that you can use to build your own SaaS application.

Laragon is a Windows application that allows you to run multiple PHP applications on your computer. Laragon is one of the best solutions for running Laravel on Windows!

In this tutorial, we will be installing Wave on Laragon.

Download Laragon

Visit https://laragon.org/download/ and download the full Laragon version as it comes with PHP 8.x.

Then run the Laragon exe file and follow the steps.

Laragon Installation Steps

Once you run the Laragon exe file, follow the installation guide and make changes based on your needs:

Select destination location: using the default C:\laragon is sufficient, but feel free to change that according to your personal preferences.
Feel free to disable ‘Run Laragon when Windows starts’
Enable the ‘Auto virtual hosts’ feature
Finally, click Install and wait for the installation to complete

After the installation, restart your computer.

After the restart, you should see the Laragon icon in your system tray. Click on it and then click on Start All Services:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Create a new Laravel project

After the Laragon installation is complete, create a new Laravel project by following these steps:

Click on the Menu button
Click on Quick app
Select Blank
Choose a name for the project
And click on OK

Then click on the Terminal icon and cd into the C:\laragon\www\wave directory and clone the Wave project:

git clone https://github.com/thedevdojo/wave.git .

Note: make sure to change the thedevdojo username with your repo where you've cloned the Wave project

Next, change the document root to the public folder of the project:

Click on the Menu button
Click on Apache
Click on sites-enabled
Click on the wave.test file
Change the DocumentRoot to C:\laragon\www\wave\public and the Directory to C:\laragon\www\wave\public
Click on Save

After that, click on the Menu button and then click on Apache and then click on Reload Apache to apply the changes.

Enable PHP Sodium

The PHP sodium extension is not enabled by default. To enable it, follow these steps:

Click on the Menu button
Click on PHP
Click on PHP Extensions
Enable the sodium extension
Click on Save

Installing Wave

After the Laragon installation is complete, clone the Wave project into the C:\laragon\www\wave directory or the one location that you’ve installed Laragon in. You can do that by clicking the Terminal icon and then running the following command:

cd C:\laragon\www\wave

Then install the dependencies:

composer install --ignore-platform-req=ext-redis

Update your database details

After the installation, update your .env file and add the necessary database details. First copy the .env.example file and rename it to .env:

copy .env.example .env

Then open the .env file and update the database details.

The default details for Laragon should be:

DB_CONNECTION=mysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_DATABASE=wave
DB_USERNAME=root
DB_PASSWORD=

Note: change the DB_DATABASE=wave with the name of your project, eg. use the same name as you did when you created the project in Laragon.

While you are in the .env file, you can also update the APP_URL to match your local URL. For example, if you are using the default Laragon URL, you can update it to:

APP_URL=http://wave.test

Then run the Laravel migrations:

php artisan migrate

Then run the Laravel seeders:

php artisan db:seed

Then visit http://wave.test in your browser and you should see the Wave application!

Conclusion

Wave is just a standard Laravel application, so you can use any of the Laravel deployment methods to deploy it. In this tutorial, we’ve used Laragon to install Wave. You can also use Laragon to install other Laravel applications.

For more information about Laragon, visit https://laragon.org/.

For more information about Wave, visit https://wave.devdojo.com/.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

If you have any questions, post them on the DevDojo community forum https://devdojo.com/questions.

How to show all tables in PostgreSQL?

Bobby Iliev — Mon, 07 Nov 2022 04:39:41 -0800

Introduction

PostgreSQL is a powerful, open-source database system. It is a relational database management system (RDBMS) based on the SQL language. PostgreSQL is one of the most popular database systems in the world. It is the go-to database for many developers and companies.

In this tutorial, we will show you how to list all tables in PostgreSQL using the psql command-line tool.

Prerequisites

To follow along with this tutorial, you will need:

A PostgreSQL database server installed on your computer. You can download and install PostgreSQL from the official website. Or you can use a cloud-based PostgreSQL database such as DigitalOcean's managed PostgreSQL.
psql command-line tool installed on your computer.

Step 1 — Connect to the PostgreSQL database

To connect to the PostgreSQL database, run the following command:

psql -U postgres -h localhost -p 5432

The -U flag specifies the username to connect to the database. In this case, we are connecting to the default postgres user.
The -h flag specifies the hostname of the database server. In this case, we are connecting to the database server running on the same computer, so we use localhost.
The -p flag specifies the port number of the database server. In this case, we are connecting to the default port number 5432.

Step 2 — Switch to a specific database

To check the list of all databases, run the following command:

\l

That is the equivalent of running SHOW DATABASES; in MySQL.

To switch to a specific database, run the following command:

\c database_name

The \c command is used to switch to a specific database.

The \c command is similar to the USE command in MySQL.

Step 3 — List all tables in the database

To list all tables in the database, run the following command:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

\dt

The \dt command is used to list all tables in the database in the public schema.

The \dt command is similar to the SHOW TABLES; command in MySQL.

To list all tables in the database in all schemas, run the following command:

\dt *.*

The *.* specifies that we want to list all tables in all schemas.

Conclusion

In this tutorial, you learned how to list all tables in PostgreSQL using the psql command-line tool.

To learn more about SQL, check out this free SQL eBook:

Introduction to SQL eBook

As a next step, I would recommend checking out Materialize. It is a streaming database that makes it easy to build real-time applications. It is the perfect database for building data-driven applications:

Materialize

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

How to flatten JSON array in SQL using jsonb_array_elements?

Bobby Iliev — Wed, 19 Oct 2022 05:07:45 -0700

Introduction

The jsonb_array_elements function returns a set of rows, one for each element of the input array. The input array can be a top-level array or a nested array.

In this tutorial, we will learn how to flatten JSON array in SQL using jsonb_array_elements.

`jsonb_array_elements` Example

Let's start by creating a table with a JSON column:

CREATE TABLE users (
    id INT,
    name TEXT,
    data JSONB
);

Next, let's insert some data into the table:

INSERT INTO users VALUES
    (1, 'John', '{"age": 30, "hobbies": ["gaming", "coding"]}'::JSONB),
    (2, 'Jane', '{"age": 25, "hobbies": ["reading", "cooking"]}'::JSONB);

After that query the table as normal:

SELECT * FROM users;

Output:

 id | name |          data
----+------+-----------------------
  1 | John | {"age": 30, "hobbies": ["gaming", "coding"]}
  2 | Jane | {"age": 25, "hobbies": ["reading", "cooking"]}
(2 rows)

Let's use the jsonb_array_elements function to flatten the hobbies array:

SELECT
    id,
    name,
    data->>'age' AS age,
    hobby
FROM users, jsonb_array_elements(data->'hobbies') hobby;

Output:

 id | name | age | hobby
----+------+-----+-------
  1 | John |  30 | gaming
  1 | John |  30 | coding
  2 | Jane |  25 | reading
  2 | Jane |  25 | cooking
(4 rows)

Creating a Materialized View

Now that we have our query, let's create a materialized view with the jsonb_array_elements function to store the flattened data:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

How to Submit Your First Pull Request on GitHub

CREATE MATERIALIZED VIEW users_hobbies AS
    SELECT
        id,
        name,
        data->>'age' AS age,
        hobby
    FROM users, jsonb_array_elements(data->'hobbies') hobby;

Let's query the materialized view:

SELECT * FROM users_hobbies;

Output:

 id | name | age | hobby
----+------+-----+-------
  1 | John |  30 | gaming
  1 | John |  30 | coding
  2 | Jane |  25 | reading
  2 | Jane |  25 | cooking

With PostgreSQL, when creating a materialized view, your query is executed and the results are stored in the materialized view. This means that the materialized view is not updated when the underlying table is updated. To update the materialized view, you need to run the REFRESH MATERIALIZED VIEW command.

If you want to have a live materialized view, that is incrementally updated in milliseconds when the underlying table is updated, you can use Materialize. Materialize is a streaming SQL database that supports incremental updates. You can learn more about Materialize here.

Conclusion

This tutorial showed you how to flatten JSON array in SQL using jsonb_array_elements.

To learn more about SQL in general, check out this free SQL eBook:

Free Introduction to SQL eBook

To learn more about Materialize, check out the official documentation:

Materialize Documentation

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

9 Open Source Projects to Contribute to - Hacktoberfest 2022

Bobby Iliev — Sun, 02 Oct 2022 11:43:13 -0700

Introduction

Hacktoberfest is a month-long event where people are awarded for contributing to open source projects 🙌.

In order to participate you need to sign up here and then Submit four pull requests (PRs) to a GitHub repo tagged with the hacktoberfest label and redeem a Hacktoberfest shirt.

For more information, you can find the official rules here.

How to Submit Your First Pull Request on GitHub

If you are new to Git and GitHub, make sure to go through this tutorial here on how to submit your first pull request:

Also, you can watch this Git and GitHub crash course on YouTube here:

Introduction to Git and GitHub video series

Or you can also read this free eBook here:

Introduction to Git and GitHub eBook

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Downloading and installing Node.js and npm

Once you have the necessary Git and GitHub knowledge you are ready to start hacking!

Here is a list with a few open-source projects that I maintain which you can contribute to!

TailwindCSS Components

The Tails Open Source Components are a (no-config) copy'n paste free collection of hand-crafted templates and components built in TailwindCSS:

thedevdojo/tails

LaraSail

LaraSail is a CLI tool, written in Bash, for Laravel to help you do all of the necessary initial server setup on DigitalOcean:

thedevdojo/larasail

Laravel Wave

Wave is a Software as a Service Starter Kit that can help you build your next great idea 💰. Wave is built with Laravel, Voyager, TailwindCSS, and a few other awesome technologies. Here are some of the awesome features ✨:

thedevdojo/wave

Docker eBook

This is an open-source introduction to Docker guide that will help you learn the basics of Docker and how to start using containers for your SysOps, DevOps, and Dev projects. No matter if you are a DevOps/SysOps engineer, developer, or just a Linux enthusiast, you will most likely have to use Docker at some point in your career:

bobbyiliev/introduction-to-docker-ebook

Git and GitHub eBook

This is an open-source introduction to Git and GitHub guide that will help you learn the basics of version control and start using Git for your SysOps, DevOps, and Dev projects. No matter if you are a DevOps/SysOps engineer, developer, or just a Linux enthusiast, you can use Git to track your code changes and collaborate with other members of your team or open source maintainers:

bobbyiliev/introduction-to-git-and-github-ebook

SQL Basics eBook

This is an open-source introduction to SQL guide that will help you learn the basics of SQL and start using relational databases for your SysOps, DevOps, and Dev projects. No matter if you are a DevOps/SysOps engineer, developer, or just a Linux enthusiast, you will most likely have to use SQL at some point in your career:

bobbyiliev/introduction-to-sql

Bash Scripting eBook

This is an open-source introduction to Bash scripting guide/ebook that will help you learn the basics of Bash scripting and start writing awesome Bash scripts that will help you automate your daily SysOps, DevOps, and Dev tasks. No matter if you are a DevOps/SysOps engineer, developer, or just a Linux enthusiast, you can use Bash scripts to combine different Linux commands and automate boring and repetitive daily tasks, so that you can focus on more productive and fun things.

bobbyiliev/introduction-to-bash-scripting

Laravel Tips eBook

This is an open-source Laravel Tips and Tricks eBook that is a collection of my own notes that I've put together for myself throughout the years. You would more likely than not need many of those tips at some point in your career as a Laravel Developer:

bobbyiliev/laravel-tips-and-tricks-ebook

101 Linux commands eBook

This is an open-source eBook with 101 Linux commands that everyone should know.

This is a brand new eBook that I have been planning to write.

I have added a list of commands but feel free to add new suggestions too!

bobbyiliev/101-linux-commands-ebook

Conclusion

If you have any questions make sure to check out the FAQ page here or reach out to me on Twitter: @bobbyiliev_

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

If you are a project maintainer share your project in the comments below! 🙌

How to find the version of an installed npm package?

Bobby Iliev — Tue, 09 Aug 2022 02:44:10 -0700

Introduction

npm is a package manager for Node.js. It is used to install and manage packages for your JavaScript projects. npm is open source and was initially released in 2009. The npm repository is a collection of open-source packages and tools for Node.js.

Check `npm` version

Let's quickly check the version of npm that we are using:

npm -v

# Output:
8.11.0

If you don't see the version number, you need to install npm first:

Check the version of the installed `npm` packages

Once you have installed npm, you can check the versions of the installed packages for your project:

npm list

The output will look something like this depending on the installed packages:

├── adonis-preset-ts@2.1.0
├── luxon@2.1.1
├── pg@8.7.1
├── pino-pretty@7.2.0
├── proxy-addr@2.0.7
├── reflect-metadata@0.1.13
├── source-map-support@0.5.20
├── typescript@4.2.4
├── youch-terminal@1.1.1

Check the version of the installed `npm` package globally

The -g flag is used to check the global packages. The output will be the same as the output of the npm list command, except that it will only show the global packages.

npm list -g

Example output:

├── @cloudflare/wrangler@1.19.8
├── @stackbit/cli@0.2.30
├── npm@8.12.1
├── pg@8.7.3
└── typescript@4.6.4

Check the versions of the dependencies of your packages

If you want to check not only the versions of your packages but also their dependencies, you can use the --depth option:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

npm list --depth=1

This will show you the versions of your packages and their dependencies.

Check the version of a specific package

You can also check the version of a specific package by adding the package name to the command:

npm list pg

Output:

└─┬ pg@8.7.1
  └─┬ pg-pool@3.4.1
    └── pg@8.7.1 deduped

Conclusion

Using the npm list command is a good way to check the versions of the installed packages.

For more information on what the latest version of a package is, you can use the npm info command.

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

For more information, you can visit the official documentation at docs.npmjs.com.

How to pass command line arguments to a Node.js app?

Bobby Iliev — Sat, 23 Jul 2022 12:42:08 -0700

Introduction

Similar to a Bash script where you can pass arguments to a script using the $1 syntax, you can also pass arguments to a Node.js app.

In this quick tutorial, you will learn how to pass arguments to a Node.js app using the process.argv array.

Prerequisites

Before you get started, you will need to have Node.js installed:

Download Node.js

Pass arguments to a Node.js app

Let's start by creating a new file called script.js and adding the following code to it:

const process = require('process');

console.log(process.argv[2]);

A quick rundown of the process.argv array:

process.argv[0] is the path to the Node.js executable
process.argv[1] is the path to the script file
process.argv[2] is the first argument passed to the script
process.argv[3] is the second argument passed to the script and so on

Let's run the script with the following command:

node script.js DevDojo

# Output:
DevDojo

Printing all arguments

To print all arguments, you can use a forEach loop just as you would with a standard array:

const process = require('process');

process.argv.forEach((val, index) => {
  console.log(`${index}: ${val}`);
});

Let's run the script with the following command:

node script.js hi there devs

We are now passing 3 arguments to the script and in this case, the output of this script will be:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

How To Write Your First Node.js Script

0: /opt/homebrew/Cellar/node@16/16.16.0/bin/node
1: /Users/bobby/dev/script.js
2: hi
3: there
4: devs

Conclusion

This is pretty much it! I hope that you find this useful!

In case you are new to Node.js I could suggest the following tutorial on how to get started:

To learn more about arguments in Bash scripts, you can read the following article:

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Bash Scripting Tutorial

Quick introduction to WebSockets with Node.js

Bobby Iliev — Sat, 23 Jul 2022 07:30:09 -0700

Introduction

WebSockets allow you to send and receive data over a network without having to use a traditional HTTP protocol. Using WebSockets, you can build a real-time application. For example, you can send messages to other users without having to refresh the page to see the new messages.

We will be using the ws library. It is a Node.js module that allows you to create WebSockets servers and clients.

Prerequisites

Before you get started, make sure you have Node.js installed:

Download Node.js

Setup

In order to install the ws library you need to run the following command:

npm install ws

Next, create a new file called server.js and open it in your favorite text editor.

Creating a WebSocket server

In the server.js file, create the following code:

const WebSocket = require('ws')

const wss = new WebSocket.Server({ port: 8081 })

wss.on('connection', function connection(ws) {
    console.log('Client connected')
    const interval = setInterval(() => {
        ws.send('hello world')
    }, 1000)
    ws.on("close", () => {
        console.log("Client disconnected");
    });
    ws.onerror = function () {
        console.log("Some Error occurred");
    }
});

Rundown of the code:

const WebSocket = require('ws'): Import the the ws module.
const wss = new WebSocket.Server({ port: 8081 }): Create a new WebSocket server on port 8081.
wss.on('connection', function connection(ws) {: When a new client connects to the server, the connection event is emitted.
const interval = setInterval(() => {: Create a new interval that sends a message to the client every second.
ws.send('hello world'): Send a message to the client.
ws.on("close", () => {: When the client disconnects, the close event is emitted.
ws.onerror = function () {: When an error occurs, the onerror event is emitted.

That way we will simulate a streaming application where the server sends a message to the client every second.

Then to start the server, run the following command:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

node server.js

Next, leave the server running and open a new terminal window where we will prepare our client.

Creating a WebSocket client

In a new file called client.js file, add the following code:

const WebSocket = require('ws')
const ws = new WebSocket('ws://localhost:8081')
ws.on('open', function open() {
    ws.on('message', function message(data) {
      console.log(`${data}`);
    });
});

The above code creates a new WebSocket client and connects to the server on port 8081.

When a message is received from the server, the message event is emitted.

To test the client, run the following command:

node client.js

You should see the following output:

hello world
hello world
hello world
...

Example:

Conclusion

This is a simple example of how to use WebSockets with Node.js.

If you want to learn more about WebSockets, check out the official documentation.

If you want to see how WebSockets work with Laravel, check out the following tutorial:

Laravel WebSockets

For further reading, I could also suggest taking a look at the following tutorial on what the difference between SSE and WebSockets is:

SSE vs WebSockets

If you are working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Building a live chart with Deno, WebSockets, Chart.js and Materialize

Bobby Iliev — Thu, 09 Jun 2022 07:05:18 -0700

Introduction

❗️ This demo includes examples for an unsupported version of Materialize (0.26.x) ❗️

This is a self-contained example of a real-time chart powered by Deno, Web Sockets, Chart.js, and Materialize.

Deno is a simple and secure runtime for JavaScript and TypeScript that uses V8. Deno, just like Materialize, is also written in Rust.

In this demo, we will build a simple live dashboard app that displays real-time data from a Deno Web Socket server. Deno will then connect to Materialize and TAIL our live materialized view to get the latest data and display it in a real-time chart using Chart.js.

Overview

Here is a quick overview of the project:

A mock service to continually generate user score events.
Redpanda instance to store the user score events in a topic.
Materialize instance that is connected to the Redpanda instance and ingests the data from the topic in a live materialized view which we can query in real-time using just SQL.
A Deno backend service that connects to Materialize and TAIL the live materialized view to get the latest data and display it in a real-time chart.
Frontend service that connects to the Deno app via a web socket and displays the data in a real-time chart using Chart.js.

Here is a diagram of the project:

Prerequisites

To run this demo, you need to have the following installed.

Running the demo

To get started, clone the repository:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Serverless Functions Demo code

git clone git clone https://github.com/bobbyiliev/materialize-tutorials.git

Then you can access the directory:

cd materialize-tutorials
git checkout lts
cd mz-deno-live-dashboard

With that you can then build the images:

docker-compose build

And finally, you can run all the containers:

docker-compose up -d

It might take a couple of minutes to start the containers and generate the demo data.

After that, you can visit http://localhost in your browser to see the demo:

Next, let's review the Materialize setup and the Deno backend setup.

Materialize setup

The Deno service will execute the following DDL statements on boot so that we don't have to run them manually:

Create a Kafka source: Creating a source in Materialize does not actually start the data ingestion. You can think of a non-materialized source as just the metadata needed for Materialize to connect to your source but not process any data:

CREATE SOURCE score
FROM KAFKA BROKER 'redpanda:9092' TOPIC 'score_topic'
FORMAT BYTES;

Create a create non-materialized view, that essentially only provides us with an alias for the SELECT statements they include:

CREATE VIEW score_view AS
    SELECT
        *
    FROM (
        SELECT
            (data->>'user_id')::int AS user_id,
            (data->>'score')::int AS score,
            (data->>'created_at')::double AS created_at
        FROM (
            SELECT CAST(data AS jsonb) AS data
            FROM (
                SELECT convert_from(data, 'utf8') AS data
                FROM score
            )
        )
    );

Create a materialized view:

CREATE MATERIALIZED VIEW score_view_mz AS
    SELECT
        (SUM(score))::int AS user_score,
        user_id
    FROM score_view GROUP BY user_id;

To check if the views and the sources were created, launch the Materialize CLI:

docker-compose run mzcli

This is just a shortcut to a docker container with postgres-client pre-installed, if you already have psql you could run psql -U materialize -h localhost -p 6875 materialize.

Then check the views and the sources:

SHOW VIEWS;
-- Output:
-- +-----------------+
-- | score_view      |
-- | score_view_mz   |
-- +-----------------+

SHOW sources;
-- Output:
-- +-----------------+
-- | score           |
-- +-----------------+

Using `TAIL`

Next, to see the results in real-time we can use TAIL:

COPY ( TAIL score_view_mz ) TO STDOUT;

You will see a flow of the new user score that was generated in real-time.

We can also start a TAIL without a snapshot, which means that you will only see the latest records after the query is run:

COPY ( TAIL score_view_mz WITH (SNAPSHOT = false) ) TO STDOUT;

This is what we will use in our Deno application to get the top user scores and display them in a real-time chart.

For more information on how the TAIL function works, see the Materialize documentation.

Deno

Now that we have Materialize ready, let's review the Deno setup.

We would use two Deno modules:

The Postgres module to connect to Materialize.
The Web Sockets module to create a Web Socket connection to our Frontend service.

You can find the code in the backend directory.

import { WebSocketClient, WebSocketServer } from "https://deno.land/x/websocket@v0.1.4/mod.ts";
import { Client } from "https://deno.land/x/postgres/mod.ts";

// Specify your Materialize connection details
const client = new Client({
  user: "materialize",
  database: "materialize",
  hostname: "materialized",
  port: 6875,
});

await client.connect();
console.log("Connected to Postgres");

// Start a transaction
await client.queryObject('BEGIN');
// Declare a cursor without a snapshot
await client.queryObject(`DECLARE c CURSOR FOR TAIL score_view_mz WITH (SNAPSHOT = false)`);

const wss = new WebSocketServer(8080);

wss.on("connection", async function (ws: WebSocketClient) {
  console.log("Client connected");
  setInterval(async () => {
    const result = await client.queryObject<{ mz_timestamp: string; mz_diff: number, user_id: number, user_score: number}>(`FETCH ALL c`);
    for (const row of result.rows) {
      let message = { user_id: row.user_id, user_score: row.user_score };
      broadcastEvent(message);
    }
  } , 1000);

});

// Broadcast a message to all clients
const broadcastEvent = (message: any) => {
  wss.clients.forEach((ws: WebSocketClient) => {
    ws.send(JSON.stringify(message));
  });
}

Rundown of the code:

As Materialize is Postgres wire compatible, first we import the Client class from the https://deno.land/x/postgres/mod.ts module. This is the class that we will use to connect to the Materialize instance.
Next, we create a new Client instance and pass it the credentials for Materialize.
Then we call the connect() method on the client instance to connect to Materialize.
Next, we call the queryObject() method on the client instance to start a transaction and also call the queryObject() method on the client instance to declare a cursor without a snapshot.
Finally, we create a new WebSocketServer instance and pass it the port to listen on.
We then define a connection event handler on the WebSocketServer instance, which is called when a client connects.
We then set an interval to fetch the latest data from Materialize and broadcast it to all clients.

Frontend setup

For the frontend, we will not be using any JavaScript framework, but just the Chart.js library.

Thanks to the web sockets connection, we can now receive the latest data from Materialize and display it in a real-time chart.

html>
<html lang="en">
    <head>
        <script src="https://cdn.jsdelivr.net/npm/chart.js">script>
    head>
    <body>
        <div class="w-full mt-10">
            <canvas id="myChart">canvas>
        div>
    <script>
      const ctx = document.getElementById("myChart");
      const myChart = new Chart(ctx, {
        type: "bar",
        data: {
          labels: [ "Player 1", "Player 2", "Player 3", "Player 4", "Player 5", "Player 6" ],
          datasets: [
            {
              label: "# of points",
              data: [0, 0, 0, 0, 0, 0],
              backgroundColor: [
                "rgba(255, 99, 132, 0.2)",
                "rgba(54, 162, 235, 0.2)",
                "rgba(255, 206, 86, 0.2)",
                "rgba(75, 192, 192, 0.2)",
                "rgba(153, 102, 255, 0.2)",
                "rgba(255, 159, 64, 0.2)",
              ],
              borderColor: [
                "rgba(255, 99, 132, 1)",
                "rgba(54, 162, 235, 1)",
                "rgba(255, 206, 86, 1)",
                "rgba(75, 192, 192, 1)",
                "rgba(153, 102, 255, 1)",
                "rgba(255, 159, 64, 1)",
              ],
              borderWidth: 1,
            },
          ],
        },
        options: {
          scales: {
            y: {
              beginAtZero: true,
            },
          },
        },
      });

      webSocket = new WebSocket("ws://127.0.0.1:8080");
      webSocket.onmessage = function (message) {
        const data = message.data;
        const dataObj = JSON.parse(data);
        const dataArray = Object.values(dataObj);
        console.log(dataArray);
        index = dataArray[0] - 1;
        myChart.data.datasets[0].data[index] = dataArray[1];
        myChart.update();
      };
    script>
  body>
html>

Rundown of the code:

We first define the new chart using the Chart.js library: new Chart() and pass the different configuration options.
Then we create a new WebSocket instance and pass it the URL of the Web Socket server with webSocket = new WebSocket("ws://backend:8080");
Finally, we define an onmessage event handler on the WebSocket instance, which is called when a message is received and updates the chart.

You can find the code in the frontend directory.

Conclusion

You can leave the Deno application running so that it would be subscribed to the Materialize instance and update the chart in real-time.

As a next step you can check out the Materialize + dbt + Redpanda demo which is based on the same user reviews mock data:

Materialize + dbt + Redpanda demo

Helpful resources:

Community

If you have any questions or comments, please join the Materialize Slack Community!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

How to use Materialize with DigitalOcean Serverless Functions

Bobby Iliev — Fri, 27 May 2022 10:09:32 -0700

Introduction

❗️ This demo includes examples for an unsupported version of Materialize (0.26.x) ❗️

Materialize is a streaming database for real-time analytics. It was launched in 2019 to address the growing need for the ability to build real-time applications easily and efficiently on streaming data so that businesses can obtain actionable intelligence from streaming data.

This is a simple example of how to use the DigitalOcean serverless functions to query Materialize.

Prerequisites

DigitalOcean Account
doctl CLI installed
A running Materialize instance

You can also find the code from this tutorial here:

Configure Materialize

Once you have your Materialize instance running, let's quickly add some data in there.

You can use psql to access Materialize:

psql -U materialize -h localhost -p 6875

For the sake of simplicity, let's start by creating a simple table:

CREATE TABLE my_view (
  id INT(11),
  name VARCHAR(255) NOT NULL,
);

After that insert some data:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

INSERT INTO my_view (id, name) VALUES (1, 'Bobby'), (2, 'John'), (3, 'Jane'), (4, 'Jack');

In a real-world scenario, you would probably want to add a source like Kafka and create a materialized view. For this example, we will use a simple table, but you can refer to the Materialize documentation for more information.

Setup `doctl`

Once you have doctl installed as per the DigitalOcean documentation, you can go ahead and follow the steps here to create a new serverless function:

Start by installing the sandbox support:

doctl serverless install

Connect to the cloud portion of your sandbox:

doctl sandbox connect

Initialize a local file system directory for the serverless function:

doctl serverless init --language php materialize-php

Then rename the sample function to materialize-php:

mv materialize-php/packages/sample/ materialize-php/packages/materialize-php/

Note that you can do all this via the DigitalOcean Control Panel rather than the CLI.

Create a `.env` file

As you will need to pass your Materialize login details securely, you should not define them in the project.yaml file. Instead, you should create a .env file at the root of your project.

Start by creating a .env file at the root of your project.

touch materialize-php/.env

Then add the following contents to the .env file:

# Your Materialize Host
MATERIALIZE_HOST=""
# Your Materialize App Specific Password
MATERIALIZE_PASSWORD=""
# Your Materialize App Username
MATERIALIZE_USER=""
# Materialize Port
MATERIALIZE_PORT="6875"
# Materialize Database Name
MATERIALIZE_DB="materialize"

Make sure to replace the values with your own.

Update the `project.yml`

The project.yml contains the information for your serverless function. In there you can specify things like the name of the function, your environment variables, and more.

With your favorite text editor, open the project.yml file and add the following contents:

targetNamespace: ''
parameters: {}
environment:
  MATERIALIZE_HOST: "${MATERIALIZE_HOST}"
  MATERIALIZE_PORT: "${MATERIALIZE_PORT}"
  MATERIALIZE_DB: "${MATERIALIZE_DB}"
  MATERIALIZE_USER: "${MATERIALIZE_USER}"
  MATERIALIZE_PASSWORD: "${MATERIALIZE_PASSWORD}"
packages:
  - name: materialize-php
    environment: {}
    parameters: {}
    annotations: {}
    actions:
      - name: hello
        binary: false
        main: ''
        runtime: 'php:default'
        web: true
        parameters: {}
        environment: {}
        annotations: {}
        limits: {}

The only thing that we've changed is the environment, where we've specified the database credentials. And also the name of the function: materialize-php.

Create the serverless function

This is more or less all the configuration that we have to do. Next, we can create the function.

Let's use the default hello.php file that was automatically created for us when we ran the doctl serverless init command:

materialize-php/packages/materialize-php/hello/hello.php

Edit the file and add the following contents:


// Function that connects to Materialize Cloud and returns the response
function connect(string $host, int $port, string $db, string $user, string $password): PDO
{
    try {
        $dsn = "pgsql:host=$host;port=$port;dbname=$db;";
        return new PDO(
            $dsn,
            $user,
            $password,
            [PDO::ATTR_ERRMODE => PDO::ERRMODE_EXCEPTION]
        );
    } catch (PDOException $e) {
        die($e->getMessage());
    }
}

function main() : array
{
    $host = getenv('MATERIALIZE_HOST');
    $port = getenv('MATERIALIZE_PORT');
    $db = getenv('MATERIALIZE_DB');
    $user = getenv('MATERIALIZE_USER');
    $password = getenv('MATERIALIZE_PASSWORD');

    $connection = connect($host, $port, $db, $user, $password);

    $sql = 'SELECT * FROM my_view';

    $statement = $connection->query($sql);
    $results = [];
    while (($row = $statement->fetch(PDO::FETCH_ASSOC)) !== false) {
        $results[] = $row;
    }

    return ["body" => $results];
}

A rundown of the function:

function connect(): We start by creating a function that we can reuse to connect to Materialize. As Materialize is a PostgreSQL wire-compatible, we just use the default PHP PDO class to connect.
function main(): This is the function that will be called when we want to run the function. As you can see we don't call the function directly, but instead we call it from the doctl serverless invoke command or access it via the URL provided by the DigitalOcean.
Inside the main() function we run a simple query to get the data from Materialize and return it.

Deploy the function

Finally, to deploy the serverless function, we can run:

doctl serverless deploy materialize-php --env materialize-php/.env

This will deploy the function to DigitalOcean and create a URL that we can access to invoke the function.

To get the URL, you can run:

doctl sbx fn get materialize-php/hello --url

Alternatively, you can also invoke the function directly with the following command:

doctl serverless functions invoke materialize-php/hello

You will get the following response:

{
    "body": [
        {
            "id": 1,
            "name": "Bobby"
        },
        {
            "id": 2,
            "name": "Jane"
        },
        {
            "id": 3,
            "name": "John"
        },
        {
            "id": 4,
            "name": "Jack"
        }
    ]
}

As a further step, we can extend the function to accept different parameters and return different data like sorting and filtering.

Conclusion

The new serverless functions allow you to quickly deploy your code without having to worry about the details of the infrastructure. Being able to query Materialize from your serverless function is a great way to get real-time data for your functions.

I am planning to add more examples of serverless functions with different languages. You will be able to find them at:

https://github.com/bobbyiliev/materialize-tutorials/tree/main/mz-digitalocean-serverless

Helpful resources:

Community

If you have any questions or comments, please join the Materialize Slack Community!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Python Script to Copy The Content of a Materialized View to S3

Bobby Iliev — Fri, 27 May 2022 05:51:32 -0700

Introduction

❗️ This demo includes examples for an unsupported version of Materialize (0.26.x) ❗️

This is a simple example of how to copy the content of a Materialize view into a local file and upload it to S3.

Prerequisites

Before you can run this script, you need to have the following prerequisites:

A running Materialize instance
Python 3.6 or later
AWS S3 Bucket and credentials

Install the Python Dependencies

As Materialize is Postgres wire-compatible, we can use the psycopg2 Python library to connect and execute queries just like we would with a Postgres database.

Start by installing the psycopg2 Python library:

pip3 install psycopg2

If you are also planning to upload the data to S3, you will need to install the boto3 Python library:

pip3 install boto3

With that done, you can now run the script!

If you are going to upload the data to S3, you will need to create an AWS S3 Bucket and credentials in ~/.aws/credentials:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

[default]
aws_access_key_id = <your access key>
aws_secret_access_key = <your secret key>

Creating the Script

Create a new Python script file and paste the following code into it:

#!/usr/bin/env python3

import psycopg2
import sys
import datetime
import boto3
from botocore.client import Config

# Connect to the Materialize database
# Replace the values with your own
dsn = "postgresql://materialize@localhost:6875/materialize?sslmode=disable"
conn = psycopg2.connect(dsn)

# If arg supply don't ask for input
if len(sys.argv) > 1:
    mz_view = sys.argv[1]
else:
    mz_view = input("Enter your view name: ")
    # While empty ask for input
    while mz_view == "":
        mz_view = input("Enter your view name: ")

# Check if view exists
print("Checking if view exists...")
with conn.cursor() as cur:
    try:
        cur.execute("SELECT * FROM " + mz_view + " LIMIT 1")
    except psycopg2.Error as e:
        print("View " + mz_view + " doesn't exist")
        sys.exit(1)

# File name with timestamp without spaces
mz_file = mz_view + "_" + str(datetime.datetime.now()).replace(" ", "_") + ".txt"

print("Dumping view " + mz_view + " to file " + mz_file)
with conn.cursor() as cur:
    cur.execute("SELECT * FROM " + mz_view)
    for row in cur:
        with open(mz_file, 'a') as f:
            f.write(str(row) + '\n')

# Ask if want to upload to S3
if input("Upload to S3? (y/n): ") == "y":
    if len(sys.argv) > 2:
        BUCKET = sys.argv[2]
    else:
        BUCKET = input("Enter your S3 bucket name: ")
        # While empty ask for input
        while BUCKET == "":
            BUCKET = input("Enter your S3 bucket name: ")

    s3 = boto3.resource('s3')
    # MinIo Example:
    # s3 = boto3.resource('s3',
    #                 endpoint_url='http://localhost:9000',
    #                 aws_access_key_id='YOUR-ACCESSKEYID',
    #                 aws_secret_access_key='YOUR-SECRETACCESSKEY',
    #                 config=Config(signature_version='s3v4'),
    #                 region_name='us-east-1')

    print("Uploading file " + mz_file + " to S3 bucket " + BUCKET)
    try:
        s3.Bucket(BUCKET).upload_file(mz_file, "mz_dump/" + mz_file)
    except Exception as e:
        print("Error uploading file to S3: " + str(e))
        sys.exit(1)

You can also get the script from GitHub

Running the Script

To run the script, you need to supply the name of the view you want to copy.

python3 mz-copy.py

You will be asked for the name of the view and the name of the S3 bucket in case you want to upload the data to S3:

Enter your view name: my_view
Checking if view exists...
Dumping view test to file my_view_2022-05-27_14:52:02.737658.txt
Upload to S3? (y/n): y
Enter your S3 bucket name: my_bucker

You can also specify the name of the view and the S3 bucket as arguments:

python3 mz-copy.py my_view my_bucket

This will generate a file named my_view_2022-05-27_14:52:02.737658.txt and upload it to the S3 bucket my_bucket.

Using MinIO

To use MinIO instead of S3, you can change the s3 = boto3.resource('s3') line to:

s3 = boto3.resource('s3',
    endpoint_url='http://localhost:9000',
    aws_access_key_id='YOUR-ACCESSKEYID',
    aws_secret_access_key='YOUR-SECRETACCESSKEY',
    config=Config(signature_version='s3v4'),
    region_name='us-east-1')

For more information on MinIO, please refer to this article.

Conclusion

This is a simple example of how to copy the content of a Materialize view into a local file and upload it to S3.

There are plans to add is something similar to Postgres's COPY command for exactly this purpose, but in the meantime, you can use this script to dump the data into a local file and upload it to S3.

At the moment you can use COPY send the output to STDOUT example:

COPY (SELECT * FROM some_view) TO STDOUT;

Helpful resources:

Community

If you have any questions or comments, please join the Materialize Slack Community!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Using Deno and Materialize to send notifications in real-time

Bobby Iliev — Fri, 13 May 2022 08:47:16 -0700

Introduction

❗️ This demo includes examples for an unsupported version of Materialize (0.26.x) ❗️

This is a self-contained example of a real-time notification app with Deno, Materialize and Discord.

Deno is a simple and secure runtime for JavaScript and TypeScript that uses V8. Deno, just like Materialize, is also written in Rust.

In this demo, we will build a simple notifications app. The notifications are going to be triggered when a user creates a bad review for a product. The reviews are generated by a mock service and stored in a Kafka/Redpanda topic. The data is then ingested into a live Materialize view which is kept up to date in real-time.

We then use Deno to connect to Materialize and send notifications to a Discord channel when a user creates a bad review.

That way we can send real-time notifications to a Discord channel and take action immediately to help the user with the issue they are facing with the product.

Overview

Here is a quick overview of the project:

A mock service to continually generate reviews and users.
Redpanda instance to store the reviews and users in topics.
Materialize instance that is connected to the Redpanda and joins the reviews and users topics in a live materialized view which we can query in real-time using just SQL.
A Deno backend service that connects to Materialize and sends notifications to a Discord channel for each bad review submitted by a user with a role of vip.

Here is a diagram of the project:

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

Prerequisites

To run this demo, you need to have the following installed.

Optionally, if you want to use the Discord webhook integration, you need to create a Discord channel and get a webhook integration URL. You can skip that step if you don't want to use it, and you can just see the notifications in the console.

Running the demo

To get started, clone the repository:

git clone git clone https://github.com/bobbyiliev/materialize-tutorials.git

Then you can access the directory:

cd materialize-tutorials
git checkout lts
cd mz-deno-discord-live-notifications

With that you can then build the images:

docker-compose build

And finally, you can run all the containers:

docker-compose up -d

It might take a couple of minutes to start the containers and generate the demo data.

After that, we are ready to complete the Materialize setup.

Materialize setup

Now that we have the containers running, we need to set up the Materialize instance.

First, launch the Materialize CLI:

docker-compose run mzcli

This is just a shortcut to a docker container with postgres-client pre-installed, if you already have psql you could run psql -U materialize -h localhost -p 6875 materialize.

Once you are in the Materialize CLI, let's start by creating the Kafka sources!

Create the Kafka sources

By using the CREATE SOURCE statement, we can connect Materialize to an external data source and lets us interact with its data as if the data were in a SQL table.

To create the sources, execute the following statements:

CREATE SOURCE users
FROM KAFKA BROKER 'redpanda:9092' TOPIC 'users_topic'
FORMAT BYTES;

CREATE SOURCE roles
FROM KAFKA BROKER 'redpanda:9092' TOPIC 'roles_topic'
FORMAT BYTES;

CREATE SOURCE reviews
FROM KAFKA BROKER 'redpanda:9092' TOPIC 'reviews_topic'
FORMAT BYTES;

Creating the sources in Materialize does not actually start the data ingestion. You can think of a non-materialized source as just the metadata needed for Materialize to connect to your source but not process any data.

Create non-materialized views

Once we have our sources ready, let's create non-materialized views, that essentially only provide us with an alias for the SELECT statements they include:

Create a users view:

CREATE VIEW users_view AS
    SELECT
        *
    FROM (
        SELECT
            (data->>'id')::int AS id,
            (data->>'username')::text AS username,
            (data->>'email')::text AS email,
            (data->>'role_id')::int AS role_id
        FROM (
            SELECT CAST(data AS jsonb) AS data
            FROM (
                SELECT convert_from(data, 'utf8') AS data
                FROM users
            )
        )
    );

Create the roles view:

CREATE VIEW roles_view AS
    SELECT
        *
    FROM (
        SELECT
            (data->>'id')::int AS id,
            (data->>'name')::text AS name
        FROM (
            SELECT CAST(data AS jsonb) AS data
            FROM (
                SELECT convert_from(data, 'utf8') AS data
                FROM roles
            )
        )
    );

Create the reviews view:

CREATE VIEW reviews_view AS
    SELECT
        *
    FROM (
        SELECT
            (data->>'user_id')::int AS user_id,
            (data->>'rating')::int AS rating,
            (data->>'review_text')::text AS review_text,
            (data->>'created_at')::double AS created_at
        FROM (
            SELECT CAST(data AS jsonb) AS data
            FROM (
                SELECT convert_from(data, 'utf8') AS data
                FROM reviews
            )
        )
    );

Essentially, as the data in our Redpanda topics is JSON, we can use the CAST function to convert the data to a JSONB object and then use the ->> operator to access the different fields. That way we can access the data in a more convenient way. For more information, check out the Materialize documentation.

Create the materialized views

Now that we have our sources ready, let's create the materialized views!

The CREATE MATERIALIZED VIEW statement creates a materialized view that computes and maintains the results of a SELECT query in memory. The results of the query are incrementally updated in real-time as new data is added to the source.

To create the materialized views we need to execute the following statements:

Join users and roles and only get the users with role vip:

CREATE MATERIALIZED VIEW vip_users AS
    SELECT
        u.id,
        u.username,
        u.email,
        u.role_id,
        r.name
    FROM users_view u
    JOIN roles_view r ON u.role_id = r.id
    WHERE r.name = 'vip';

And for the reviews, we would want to only materialize the reviews that were generated in the last 10 minutes and only get the reviews for the users with role vip:

CREATE MATERIALIZED VIEW bad_vip_reviews AS
    SELECT
        r.user_id,
        r.rating,
        r.review_text,
        r.created_at,
        u.username,
        u.email
    FROM reviews_view r
    JOIN vip_users u ON r.user_id = u.id
    WHERE r.rating < 3
    AND
        mz_logical_timestamp() < (created_at*1000 + 100000)::numeric;

For more information on how the mz_logical_timestamp() function works, see the Materialize documentation.

You can now query the data in the materialized views using standard SQL!

SELECT * FROM bad_vip_reviews LIMIT 10;

You can run the query multiple times to see the results change.

Using `TAIL`

Next, to see the results in real-time we can use TAIL:

COPY ( TAIL bad_vip_reviews ) TO STDOUT;

You will see a flow of the reviews that were generated in real-time. As we've used the mz_logical_timestamp() function, you will see that the records which are older than 10 minutes are being filtered out, which is indicated with a -1.

We can also start a TAIL without a snapshot, which means that you will only see the latest records after the query is run:

COPY ( TAIL bad_vip_reviews WITH (SNAPSHOT = false) ) TO STDOUT;

This is what we will see in our Deno application to get the latest bad reviews and send them to Discord as notifications when they are generated.

For more information on how the TAIL function works, see the Materialize documentation.

Using Deno

Now that we have Materialize ready, we can use Deno to get the latest reviews and send them to Discord as notifications when they are generated.

Start by checking if Deno is installed:

deno --version

If you don't have Deno installed, follow the steps here:

Install Deno

Next, create a new file called index.ts:

touch index.ts

And add the following code:

import { Client } from "https://deno.land/x/postgres/mod.ts";

// Specify your Discord webhook URL
const discord_webhook_url = "";

// Specify your Materialize connection details
const client = new Client({
    user: "materialize",
    database: "materialize",
    hostname: "127.0.0.1",
    port: 6875,
});

const tail = async () => {
    // Connect to Materialize
    await client.connect();

    // Start a transaction
    await client.queryObject('BEGIN');
    // Declare a cursor without a snapshot
    await client.queryObject('DECLARE c CURSOR FOR TAIL bad_vip_reviews WITH (SNAPSHOT = false)');

    // Start a loop to get the latest records
    while (true) {
        // Get the next record
        const result = await client.queryObject<{ mz_timestamp: string; mz_diff: number, user_id: number, 
rating: number, review_text: number, created_at: string, username: string, email: string }>('FETCH ALL c');

        // Loop through the records and send them to Discord
        for (const row of result.rows) {
            if (row.mz_diff > 0) {
                console.log(`${row.username} has left a bad review with rating ${row.rating}. Email: ${row.email}`);
                // Make an HTTP request to post to the Discord webhook
                if (discord_webhook_url) {
                    await fetch(discord_webhook_url, {
                        method: "POST",
                        headers: {
                            "Content-Type": "application/json",
                        },
                        body: JSON.stringify({
                            content: `${row.username} has left a bad review!\nRating ${row.rating}.\nEmail: ${row.email}`,
                        }),
                    });
                }
            }
        }
    }

    await client.end();
}

tail();

Rundown of the code:

As Materialize is Postgres wire compatible, first we import the Client class from the https://deno.land/x/postgres/mod.ts module. This is the class that we will use to connect to the Materialize instance.
Then, you can specify the Discord webhook URL which you want to send the notifications to. If you don't want to send notifications, you can leave this empty and the code will just print the reviews to the console.
Next, we create a new Client instance and pass it the credentials for Materialize.
Then we call the connect() method on the client instance to connect to Materialize.
Next, we call the queryObject() method on the client instance to start a transaction and also call the queryObject() method on the client instance to declare a cursor without a snapshot.
Finally, we start a loop to get the latest records.
In the loop, we call the queryObject() method on the client instance to get the next record.
In the loop, we loop through the records and send them to Discord.

Then run the application:

deno run --allow-net --allow-read --allow-env --unstable index.ts

Sample output:

user7872 has left a bad review with rating 2. Email: user7872@example.com
user7881 has left a bad review with rating 2. Email: user7881@example.com
user7887 has left a bad review with rating 2. Email: user7887@example.com
user7890 has left a bad review with rating 1. Email: user7890@example.com
user7894 has left a bad review with rating 1. Email: user7894@example.com
...

The same information will also be sent to Discord as a notification:

Conclusion

You can leave the Deno application running so that it would be subscribed to the Materialize instance and send the latest reviews to Discord as notifications.

As a next step, you can implement some error handling for the application so that it would gracefully handle errors and send the error to Discord as a notification.

As a next step you can check out the Materialize + dbt + Redpanda demo which is based on the same user reviews mock data:

Materialize + dbt + Redpanda demo

As an extra step for this tutorial, you can try using the TrustPilot API to get accurate ratings rather than the mock data.

Helpful resources:

Community

If you have any questions or comments, please join the Materialize Slack Community!

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Scaling Laravel with Serverless Redis

Bobby Iliev — Mon, 09 May 2022 09:05:30 -0700

Introduction

Laravel is a popular PHP framework for building scalable, high-performance web applications.

In this article, we will learn how to use serverless Redis to scale Laravel applications by storing the Laravel session and cache data in a serverless Redis instance.

Prerequisites

Before you get started, you'll need to have the following:

Upstash account: In case that you don't have one, you can sign up for free, no credit card required.
If you don't have Laravel installed, you can follow the steps on how to do that here: Install Laravel with 1-click

Architecture Overview

Rather than running Laravel on a single server, let's consider the following scenario:

A Laravel application running on two web servers.
A single Load Balancer is responsible for routing requests to the two web servers.
A MySQL database server, used to store the application's data.
Upstash Serverless Redis cluster is responsible for caching data and storing user sessions.

Diagram:

What is Serverless Redis?

Serverless Redis is a fully managed database as a service product where the pricing is based on per command, so you are only charged what you actually use.

That way you don't have to over provision your servers, and you can scale your application as needed.

Why Serverless Redis?

By default, Laravel would store the user sessions in files on the web server's disk. That way if the load balancer forwards the user request to a different server, the user session would be lost.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.

This is why it is important to have a centralized place to store the user sessions and application cache, so that they can be shared between requests and across multiple servers, and not be lost each time the load balancer forwards the request to a different server.

Of course, you can also use your database to store the user sessions and cache data, but for better performance, it is recommended to use Redis for better performance. If you want to learn more about the performance benefits of the different options, check out this great article here: Which is the best Laravel cache driver for performance?.

Horizontal Scaling vs. Vertical Scaling

Just a few words about the difference between horizontal and vertical scaling:

When you have a single server, you can scale it vertically by adding more resources to it. For example, you can add more CPU cores, RAM, or disk space to scale-up.
Horizontal scaling on the other side, is when you add more servers that are responsible for serving requests to scale-out.

Here is a simple example of horizontal scaling vs. vertical scaling:

When horizontally scaling an application, it is important to handle your user sessions and cache data in a scalable way.

Creating a serverless Redis cluster

With Upstash, you can create a serverless Redis cluster in 30 seconds by following these steps:

Log in to your Upstash account.
Click on the Create Database button.
Enter the name of your Redis cluster and choose a region.
Click on the Create button.

That's it! You now have a serverless Redis cluster ready to use.

Make sure to note down the endpoint of your Redis cluster along with the password and the port.

Configuring Laravel with Serverless Redis

Now that you have a serverless Redis cluster, you can configure Laravel to use it just as you would any other Redis instance.

Install the Predis package

In the past, you would use the PHP Redis extension to connect to your Redis cluster. However, now you can use the Predis package instead.

To install the Predis package, run the following command:

composer require predis/predis

Next, head over to your Laravel project's .env file and update the following lines:

REDIS_HOST=your_upstash_redis_endpoint
REDIS_PASSWORD=your_upstash_redis_password
REDIS_PORT=your_upstash_redis_port

While changing the Redis details, make sure to also change the cache driver and the session driver to redis:

CACHE_DRIVER=redis
SESSION_DRIVER=redis

Finally, clear your config cache by running the following command:

php artisan config:clear

That way your Laravel application will use the serverless Redis cluster to store its cache and session data.

Conclusion

Using Laravel with Serverless Redis is a great way to scale your application. Even if you are running Laravel on a Kubernetes cluster, you can still use a serverless Redis cluster to store your user sessions and cache data in a scalable way.

For more information on Upstash, check their documentation.

For more information on how to scale your Laravel application check out the following article:

How to Set Up a Scalable Laravel 6 Application using Managed Databases and Object Storage

If you are already working as a DevOps engineer, check out this DevOps Scorecard and evaluate your DevOps skills across 8 key areas and discover your strengths and growth opportunities

Using LogicLoop and Materialize with dbt and Redpanda/Kafka

Bobby Iliev — Fri, 06 May 2022 08:53:49 -0700

Introduction

❗️ This demo includes examples for an unsupported version of Materialize (0.26.x) ❗️

LogicLoop allows you to write rules in SQL and then run them against your data and trigger different actions based on the results. LogicLoop also allows you to create and share dashboards and visualizations easily via their web interface.

Materialize is a source-available streaming database that takes data coming from sources like Kafka/Redpanda, Postgres, S3, and more, and allows users to write views that aggregate data on your event stream. The magic is that the views are translated to dataflows which allows Materialize to maintain the views incrementally in real-time. A normal materialized view would do a full scan of the data every time it needs to be updated, Materialize only does the work to maintain the view based on events that come in, so it is much faster and more efficient.

In this tutorial, we will walk through how to use LogicLoop with Materialize.

Prerequisites

Make sure to sign up for a LogicLoop account first.

Also, for this tutorial, we will extend upon the previous article on how to use dbt with Materialize and Redpanda.

The architecture of the previous article is as follows:

If you want to follow along, make sure to read the previous article first and have the project up and running on your server.

Our Amazing Sponsors

DigitalOcean offers a simple and reliable cloud hosting solution that enables developers to get their website or application up and running quickly.

Laravel News keeps you up to date with everything Laravel. Everything from framework news to new community packages, Laravel tutorials, and more.