Catatan Perjalanan - Nicolas Julian

How to Recover Kubernetes Cluster After Cilium Network Policy Lockout - etcd Disaster Recovery Guide

Sat, 13 Sep 2025 13:39:31 GMT

TL;DR: Learn how to recover from a complete Kubernetes cluster lockout caused by misconfigured Cilium network policies. This guide covers etcd disaster recovery using --force-new-cluster, fixing lost quorum, and preventing similar incidents.

In this comprehensive guide, I'll show you how one Cilium network policy configuration mistake completely locked me out of my production Kubernetes cluster, and the exact step-by-step process I used to recover without any data loss. If you're running Kubernetes with Cilium CNI and working with network policies, this post will help you avoid and recover from similar disasters.

Background: Our Kubernetes Cluster Setup

I run a 3-node Kubernetes cluster with the following configuration:

Deployment tool: Kubespray
Infrastructure: VPS with public IPs only (no private networking)
CNI: Cilium
etcd: External etcd cluster (systemd service)
Use case: Production workloads requiring strict network security

Because our cluster runs on VPS with only public IPs exposed, network security through Cilium network policies is critical. My colleague was implementing security hardening and needed to restrict node-exporter metrics endpoint (port 9100) access using Cilium's clusterwide network policies.

What seemed like a simple security improvement turned into a complete cluster outage within seconds.

The Problem: Kubernetes Cluster Completely Locked Out

What Went Wrong With Cilium Network Policy

Here's the exact Cilium clusterwide network policy that caused the complete cluster lockout:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: restrict-node-exporter-hostport-9100
spec:
  nodeSelector:
    matchLabels: {}  # Applied to ALL nodes - this was the problem
  ingress:
  - fromCIDR:
    - 103.xxx.xxx.xxx/32
    toPorts: 
    - ports:
      - port: "9100"
        protocol: TCP
  - fromEntities:
    - cluster
    toPorts:
    - ports:
      - port: "9100"
        protocol: TCP

Understanding Cilium Clusterwide Network Policy Behavior

The critical mistake: Cilium clusterwide network policies with empty nodeSelector operate in whitelist mode, not blacklist mode. This is a common misunderstanding that causes many Kubernetes cluster outages.

By only specifying port 9100 in the ingress rules, we unintentionally created a default-deny policy for ALL other ports across the entire cluster, including:

Port 6443 - Kubernetes API server (no kubectl access)
Port 2379/2380 - etcd cluster communication (quorum lost)
Port 10250 - kubelet API (node communication broken)
Port 53 - DNS (service discovery failed)

This created a perfect deadlock:

Cannot delete the network policy → need API server access
Cannot restore API server → need etcd working
Cannot fix etcd → need cluster communication (blocked by Cilium)
Cannot disable Cilium → need etcd to delete DaemonSet

Kubernetes Cluster Troubleshooting: Identifying the Issue

Symptom 1: API Server Unreachable

$ kubectl get nodes
Unable to connect to the server: dial tcp 103.xxx.xxx.xxx:6443: connect: connection refused

$ curl -k https://127.0.0.1:6443/healthz
curl: (7) Failed to connect to 127.0.0.1 port 6443: Connection refused

Symptom 2: etcd Lost Quorum - No Leader

$ ETCDCTL_API=3 etcdctl endpoint status -w table
+----------------+------------------+---------+---------+-----------+------------+
|    ENDPOINT    |        ID        | VERSION | DB SIZE | IS LEADER | RAFT TERM  |
+----------------+------------------+---------+---------+-----------+------------+
| 127.0.0.1:2379 | 760112fd749b508a |   3.5.6 |  497 MB |   false   |   13384    |
+----------------+------------------+---------+---------+-----------+------------+

ERROR: etcdserver: no leader

The etcd cluster lost quorum because nodes couldn't communicate on ports 2379/2380. Without a Raft leader, etcd refuses all write operations - meaning we cannot delete the problematic network policy even if we had API access.

Complete Kubernetes Disaster Recovery Process

Step 1: Stop Cilium Agent to Disable Network Policies

First attempt to disable Cilium network policy enforcement:

# Find cilium-agent container using nerdctl
$ nerdctl ps | grep cilium-agent | grep -v pause

# Stop the cilium-agent container
$ nerdctl stop <cilium-container-id>

Result: Cilium stopped but API server was already crashed. etcd still had no leader because the damage was already done.

Step 2: Attempt Cilium Uninstall

$ cilium uninstall
🔥 Deleting agent DaemonSet...
🔥 Deleting operator Deployment...
✅ Cilium was successfully uninstalled.

Problem: This command requires etcd to work, creating a circular dependency. We needed a different approach.

Step 3: Backup etcd Before Recovery (Critical Step)

Always backup etcd before attempting any disaster recovery operations:

ETCDCTL_API=3 etcdctl snapshot save /root/etcd-backup-$(date +%Y%m%d-%H%M%S).db \
  --endpoints=127.0.0.1:2379 \
  --cacert=/etc/ssl/etcd/ssl/ca.pem \
  --cert=/etc/ssl/etcd/ssl/node-argocd-1.pem \
  --key=/etc/ssl/etcd/ssl/node-argocd-1-key.pem

Output:

Snapshot saved at /root/etcd-backup-20251013-110002.db

This backup is your safety net. If the recovery process fails, you can restore from this snapshot.

Step 4: etcd Disaster Recovery - Force New Cluster

This is the nuclear option for etcd disaster recovery. The --force-new-cluster flag forces a node to become a single-member cluster with itself as the leader.

What --force-new-cluster does:

Removes all other members from the cluster configuration
Keeps all existing data intact (does NOT delete data)
Forces this node to become the Raft leader
Allows the cluster to operate as single-node temporarily

# Stop etcd systemd service
$ systemctl stop etcd

# Start etcd with force-new-cluster flag
$ etcd --force-new-cluster \
  --data-dir=/var/lib/etcd \
  --name=etcd2 \
  --advertise-client-urls=https://103.xxx.xxx.xxx:2379 \
  --listen-client-urls=https://103.xxx.xxx.xxx:2379,https://127.0.0.1:2379 \
  --cert-file=/etc/ssl/etcd/ssl/member-argocd-1.pem \
  --key-file=/etc/ssl/etcd/ssl/member-argocd-1-key.pem \
  --trusted-ca-file=/etc/ssl/etcd/ssl/ca.pem \
  --client-cert-auth=true &

Success! etcd became leader:

{"level":"info","ts":"2025-10-13T11:28:12.212+0700","msg":"760112fd749b508a became leader at term 13385"}
{"level":"info","ts":"2025-10-13T11:28:12.215+0700","msg":"published local member to cluster through raft"}
{"level":"info","ts":"2025-10-13T11:28:12.218+0700","msg":"serving client traffic securely"}

Now etcd had a leader, and the Kubernetes API server automatically came back online!

Step 5: Delete the Problematic Cilium Network Policy

With API access restored, we could finally delete the bad network policy:

$ kubectl get ciliumclusterwidenetworkpolicies
NAME                                   AGE
restrict-node-exporter-hostport-9100   94m

$ kubectl delete ciliumclusterwidenetworkpolicies/restrict-node-exporter-hostport-9100
ciliumclusterwidenetworkpolicy.cilium.io "restrict-node-exporter-hostport-9100" deleted

Step 6: Restore etcd Cluster - Add Members Back

Running with single-node etcd is risky. We needed to restore the 3-node cluster for high availability.

Add first member back to etcd cluster:

ETCDCTL_API=3 etcdctl member add etcd1 --peer-urls=https://103.xxx.xxx.xxx:2380 \
  --endpoints=127.0.0.1:2379 \
  --cacert=/etc/ssl/etcd/ssl/ca.pem \
  --cert=/etc/ssl/etcd/ssl/node-argocd-1.pem \
  --key=/etc/ssl/etcd/ssl/node-argocd-1-key.pem

Output:

Member d25f65265e3477c7 added to cluster ed92624dcc0aa007
ETCD_NAME="etcd1"
ETCD_INITIAL_CLUSTER="etcd2=https://103.xxx.xxx.xxx:2380,etcd1=https://103.xxx.xxx.xxx:2380"
ETCD_INITIAL_CLUSTER_STATE="existing"

On the second node, rejoin the cluster:

$ ssh root@103.xxx.xxx.xxx

# Stop etcd and backup old data
$ systemctl stop etcd
$ mv /var/lib/etcd /var/lib/etcd.old

# Update etcd configuration to join existing cluster
$ sed -i 's/ETCD_INITIAL_CLUSTER_STATE=.*/ETCD_INITIAL_CLUSTER_STATE=existing/' /etc/etcd.env
$ sed -i 's/ETCD_INITIAL_CLUSTER=.*/ETCD_INITIAL_CLUSTER=etcd2=https:\/\/103.xxx.xxx.xxx:2380,etcd1=https:\/\/103.xxx.xxx.xxx:2380/' /etc/etcd.env

# Start etcd to rejoin cluster
$ systemctl start etcd

Repeat this process for the third node.

Step 7: Verify Complete Kubernetes Cluster Recovery

Check etcd cluster members:

$ ETCDCTL_API=3 etcdctl member list -w table
+------------------+---------+-------+------------------------------+------------------------------+------------+
|        ID        | STATUS  | NAME  |          PEER ADDRS          |         CLIENT ADDRS         | IS LEARNER |
+------------------+---------+-------+------------------------------+------------------------------+------------+
| 760112fd749b508a | started | etcd2 | https://103.xxx.xxx.xxx:2380 | https://103.xxx.xxx.xxx:2379 |      false |
| d25f65265e3477c7 | started | etcd1 | https://103.xxx.xxx.xxx:2380 | https://103.xxx.xxx.xxx:2379 |      false |
| da70198d6c2536f2 | started | etcd3 | https://103.xxx.xxx.xxx:2380 | https://103.xxx.xxx.xxx:2379 |      false |
+------------------+---------+-------+------------------------------+------------------------------+------------+

Verify etcd health and leader election:

$ ETCDCTL_API=3 etcdctl endpoint status -w table
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
|       ENDPOINT       |        ID        | VERSION | DB SIZE | IS LEADER | IS LEARNER | RAFT TERM | RAFT INDEX | RAFT APPLIED INDEX | ERRORS |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+
| 103.xxx.xxx.xxx:2379 | 760112fd749b508a |   3.5.6 |  497 MB |      true |      false |     13388 | 2079555217 |         2079555217 |        |
| 103.xxx.xxx.xxx:2379 | d25f65265e3477c7 |   3.5.6 |  497 MB |     false |      false |     13388 | 2079555218 |         2079555218 |        |
| 103.xxx.xxx.xxx:2379 | da70198d6c2536f2 |   3.5.6 |  497 MB |     false |      false |     13388 | 2079555220 |         2079555220 |        |
+----------------------+------------------+---------+---------+-----------+------------+-----------+------------+--------------------+--------+

Perfect! All three etcd members are healthy, synchronized (same RAFT term), with proper leader election. Kubernetes cluster fully recovered with zero data loss!

The Correct Cilium Network Policy Configuration

Here's how to properly configure Cilium clusterwide network policy without locking yourself out:

apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
  name: restrict-node-exporter-hostport
spec:
  nodeSelector:
    matchLabels: {}
  ingress:
  # Allow node-exporter access from monitoring server only
  - fromCIDR:
    - 103.xxx.xxx.xxx/32
    toPorts: 
    - ports:
      - port: "9100"
        protocol: TCP
  
  # CRITICAL: Allow all cluster-internal communication
  - fromEntities:
    - cluster
    - host
  
  # CRITICAL: Explicitly allow Kubernetes control plane ports
  - toPorts:
    - ports:
      - port: "6443"  # Kubernetes API server
        protocol: TCP
      - port: "2379"  # etcd client port
        protocol: TCP
      - port: "2380"  # etcd peer port
        protocol: TCP
      - port: "10250" # kubelet API
        protocol: TCP
      - port: "53"    # CoreDNS
        protocol: TCP
      - port: "53"    # CoreDNS
        protocol: UDP

Key differences:

Added fromEntities: [cluster, host] to allow all internal cluster communication
Explicitly whitelisted all critical Kubernetes ports
Maintained the port 9100 restriction to specific monitoring IP

Kubernetes Network Policy Best Practices

1. Understand Cilium Policy Modes

Cilium clusterwide network policies with nodeSelector work as whitelist mode:

Only explicitly allowed traffic is permitted
All other traffic is denied by default
Very different from traditional firewall deny rules

2. Always Whitelist Critical Kubernetes Ports

When using any network policies, ALWAYS allow:

6443 - Kubernetes API server (kubectl access)
2379/2380 - etcd cluster communication (critical for consensus)
10250 - kubelet API (node management)
53 - CoreDNS (service discovery)

3. Test Network Policies in Staging Environment

Never test Cilium clusterwide network policies directly in production. Always:

Create identical staging cluster
Apply policies in staging first
Verify all critical services remain accessible
Monitor for 24 hours before production rollout

4. Prefer Namespaced Network Policies

Instead of CiliumClusterwideNetworkPolicy, use namespaced CiliumNetworkPolicy when possible:

apiVersion: cilium.io/v2
kind: CiliumNetworkPolicy  # Namespaced - safer
metadata:
  name: restrict-node-exporter
  namespace: monitoring
spec:
  endpointSelector:
    matchLabels:
      app: node-exporter
  # Only affects pods in monitoring namespace

Namespaced policies reduce blast radius and prevent cluster-wide outages.

5. Implement Kubernetes Monitoring and Alerts

Set up monitoring for:

etcd cluster health (alert on "no leader")
etcd member count changes
API server availability
Network policy changes (audit log)

These alerts would have caught the issue within seconds instead of after the lockout occurred.

6. Document etcd Disaster Recovery Procedures

Every Kubernetes administrator should know:

How to take etcd snapshots
How to restore from etcd backup
When and how to use etcd --force-new-cluster
How to rejoin etcd members after recovery

Emergency Kubernetes Recovery Commands

Save these commands for disaster recovery situations:

etcd Snapshot Backup

ETCDCTL_API=3 etcdctl snapshot save /root/etcd-backup-$(date +%Y%m%d-%H%M%S).db \
  --endpoints=127.0.0.1:2379 \
  --cacert=/etc/ssl/etcd/ssl/ca.pem \
  --cert=/etc/ssl/etcd/ssl/node-argocd-1.pem \
  --key=/etc/ssl/etcd/ssl/node-argocd-1-key.pem

etcd Health Check

ETCDCTL_API=3 etcdctl endpoint health \
  --endpoints=127.0.0.1:2379 \
  --cacert=/etc/ssl/etcd/ssl/ca.pem \
  --cert=/etc/ssl/etcd/ssl/node-argocd-1.pem \
  --key=/etc/ssl/etcd/ssl/node-argocd-1-key.pem

etcd Force New Cluster (Last Resort)

systemctl stop etcd
etcd --force-new-cluster \
  --data-dir=/var/lib/etcd \
  --name=<node-name> \
  --advertise-client-urls=https://<ip>:2379 \
  --listen-client-urls=https://<ip>:2379,https://127.0.0.1:2379 \
  [additional TLS flags] &

Add etcd Member

ETCDCTL_API=3 etcdctl member add <name> --peer-urls=https://<ip>:2380

Restore etcd from Snapshot

ETCDCTL_API=3 etcdctl snapshot restore <backup-file> \
  --data-dir=/var/lib/etcd-restore

Frequently Asked Questions (FAQ)

Q: Will `etcd --force-new-cluster` delete my data?

No. The --force-new-cluster flag only modifies cluster membership configuration. All your Kubernetes resources, configurations, and data remain intact in the etcd data directory.

Q: Can I recover without `--force-new-cluster`?

If you have lost etcd quorum and cannot restore communication between nodes, --force-new-cluster is the only option. It's specifically designed for disaster recovery when the cluster cannot elect a leader.

Q: Should I use Cilium clusterwide policies at all?

Yes, but carefully. Clusterwide policies are powerful for enforcing security across your entire cluster, but they require thorough testing and understanding. For most use cases, namespaced policies are safer.

Q: What if I don't have an etcd backup?

The --force-new-cluster method works without a backup because it uses the existing data directory. However, always maintain regular etcd backups for other disaster scenarios like disk failure or data corruption.

Conclusion: Key Takeaways for Kubernetes Administrators

One misconfigured Cilium network policy brought down our entire production Kubernetes cluster. But with proper understanding of etcd operations and disaster recovery procedures, we recovered completely without any data loss.

Critical lessons:

Cilium clusterwide policies with nodeSelector create whitelist mode - understand this before applying
Always explicitly allow critical Kubernetes ports (6443, 2379, 2380, 10250, 53)
Test network policies in staging environments, never directly in production
Know how to use etcd --force-new-cluster before you need it
Maintain regular etcd backups and document recovery procedures
Implement monitoring and alerting for etcd cluster health

The etcd --force-new-cluster command saved us from rebuilding the entire cluster from scratch. Every Kubernetes administrator should understand this critical disaster recovery tool.

[Personal Takes] Network Storage: Principal

Wed, 11 Dec 2024 06:44:58 GMT

Warning: This post is mostly about Ceph, but the principles apply to most distributed storage systems. Your mileage may vary with other solutions.

The Reality Check

After years of building, breaking, and rebuilding network storage systems, I've learned that most documentation glosses over the harsh realities. The vendor presentations show perfect scenarios with uniform hardware and infinite budgets. The real world? Well, that's messier. Here are the hard-earned lessons that'll save you some sleepless nights.

1. Why Busy the Network if We Don't Have To?

This might sound obvious, but you'd be amazed how often people overcomplicate their storage architecture. Every byte that crosses the network is a byte that could cause latency, congestion, or worse - complete failure during peak loads.

The principle is simple: keep data local whenever possible. If your application can work with local storage, don't force it through the network just because you have a fancy distributed storage system. Reserve network storage for when you actually need the redundancy, accessibility, or shared state.

I've seen perfectly good applications hobbled by unnecessary network hops. Sometimes the best network storage solution is no network storage at all.

2. How to Move Data Fast? Don't Move It at All

The fastest data transfer is the one that never happens. This isn't just philosophical - it's practical architecture.

Instead of designing systems that constantly shuffle data around, design them to minimize data movement from the start. Use techniques like:

Data locality: Keep related data on the same nodes
Smart caching: Cache frequently accessed data locally
Lazy replication: Only replicate when necessary, not because you can

When you absolutely must move data, batch it. Move large chunks during off-peak hours rather than constant tiny transfers that create network chatter.

3. Fast Local Storage is Easy and Cheap. Fast Shared Storage is Hard and Expensive

This is the fundamental economics of storage that marketing teams don't want you to understand. You can build blazing-fast local storage with consumer SSDs for a fraction of what enterprise shared storage costs.

But the moment you want that performance shared across multiple nodes? Welcome to complexity hell. Now you need:

High-speed networking (expensive)
Specialized protocols (complex)
Redundancy mechanisms (more expensive)
Synchronization overhead (performance killer)

Don't build shared storage unless you genuinely need it. And if you do need it, budget accordingly - both in money and complexity.

4. Your Cluster Storage System is Only as Good as Its Network

I cannot stress this enough: your storage cluster will perform like your worst network link. It doesn't matter if you have the fastest SSDs money can buy if they're connected with gigabit Ethernet from 2010.

Network considerations for storage clusters:

Bandwidth: More is always better, but consistency matters more
Latency: Low latency beats high bandwidth for small operations
Reliability: One flaky switch can bring down your entire cluster

Invest in your network infrastructure. It's not glamorous, but it's the foundation everything else depends on.

5. Mixing Drive Capacities: Proceed with Extreme Caution

"It's okay to mix drive capacities, but keep them as uniform as possible." This sounds contradictory, but here's the reality:

When you mix drive sizes, several problems emerge:

Uneven distribution: Smaller disks fill up faster, creating hotspots
Placement group imbalance: Larger disks handle more placement groups
Uneven wear: Some disks work harder than others, failing sooner
Performance degradation: The cluster performance drops to accommodate the differences

If you must mix capacities (and sometimes you must), try to keep ratios reasonable. Don't put 1TB drives next to 10TB drives in the same pool. Your future self will thank you.

6. Don't Exceed 80% Capacity if You Want Nice Weekend Trips

This is the "golden rule" of storage administration, and it exists for good reasons:

Performance degradation: Most storage systems slow down significantly above 80% capacity
Rebalancing headaches: Less free space means longer recovery times
Emergency buffer: You need space for temporary operations and emergency growth
Peace of mind: Nothing ruins a weekend like a full storage cluster

I've learned this the hard way. That extra 15% capacity seems wasteful until you're frantically trying to free up space at 2 AM because someone decided to upload their entire video collection to the company share.

7. We'll Figure It Out on the Fly

This isn't defeatism - it's pragmatism. No matter how much you plan, real-world usage will surprise you. Applications will behave differently than expected. Users will find creative ways to stress your system. Hardware will fail in ways the vendor said were impossible.

The key is building systems that are observable and adjustable. You need:

Good monitoring: Know what's happening in real-time
Flexible architecture: Be able to adapt without complete rebuilds
Documentation: Write down what you learn for next time

Don't aim for perfection on day one. Aim for "good enough to start" and "flexible enough to improve."

8. Don't Sabotage Your Beautiful Cluster with Terrible Frontend Networking

This is the classic mistake: spending thousands on high-performance storage nodes, then connecting clients through whatever network infrastructure was lying around.

Your storage cluster might be capable of hundreds of thousands of IOPS, but if clients connect through old gigabit switches with oversubscribed uplinks, users will experience the performance of those old gigabit switches.

The client experience is what matters. Build your network architecture from the user backward, not from storage forward.

The Bottom Line

Network storage is about trade-offs, not absolutes. Every decision has consequences, and those consequences compound over time. The best storage architecture is the one that:

Meets your actual needs (not your imagined ones)
Fits your budget (including operational costs)
Can evolve as requirements change
Doesn't keep you up at night

Remember: perfect is the enemy of good, but "good enough" is the enemy of excellent. Find your balance, document your decisions, and always keep learning.

Because in six months, you'll need to explain to your future self why you made these choices. Make sure it's a conversation you'll enjoy having.

Expose Service In Your Baremetal without Public IP in Kubernetes with Cloudflare Tunnel

Mon, 04 Mar 2024 11:54:36 GMT

Thumbnail: "Indonesian Students in Prague, Czech Republic During the Graduation of One of Their Friends in 1968. Photo from Soegeng Soejono's Archive." - Gestapu Menghapus Satu Generasi Intelektual Indonesia

So if we are hosting our own Kubernetes, one of problem is expose your service, how to do that? NodePort? Loadbalancer, or what?. This problem is make metallb.universe.tf born.

But even white these tools, we still need to have deal with IPs pools, route, etc. It's good if when you need it, but what if your goals is just get ingress-nginx, and point your DNS to IP from cloudflare, and enable Cloudflare Proxy? then this is the right blog post for you. You can just utilize Cloudflare Tunnel.

Lets get into the details.

What is Cloudflare Tunnel

Cloudflare Tunnel provides you with a secure way to connect your resources to Cloudflare without a publicly routable IP address. With Tunnel, you do not send traffic to an external IP — instead, a lightweight daemon in your infrastructure (‘cloudflared’) creates outbound-only connections to Cloudflare’s global network.

This way, your origins can serve traffic through Cloudflare without being vulnerable to attacks that bypass Cloudflare.

Let's just set it up!

1. Get your cloudflare tunnel token

2. Create a daemonset in your Kubernetes

Put your token value inside TUNNEL_TOKEN key, and then you ready to deploy it.

---
apiVersion: v1
stringData:
  TUNNEL_TOKEN: eyJhXXXXxXXxXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXTM3OWJXXXXXXxxxxXXXXXxxxxxxxXXXXXXXXXXXXXXXXFNCJ9
kind: Secret
metadata:
  name: cf-tunel-production
  namespace: cloudflared
type: Opaque
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  labels:
    app: cloudflared
  name: cloudflared-daemonset
  namespace: cloudflared
spec:
  selector:
    matchLabels:
      pod: cloudflared
  template:
    metadata:
      labels:
        pod: cloudflared
    spec:
      containers:
        - command:
            - cloudflared
            - tunnel
            - --metrics
            - 0.0.0.0:2000
            - run
          image: cloudflare/cloudflared:latest
          env:
            - name: NO_AUTOUPDATE
              value: "true"
          envFrom:
            - secretRef:
                name: cf-tunel-production
          imagePullPolicy: Always
          livenessProbe:
            failureThreshold: 1
            httpGet:
              path: /ready
              port: 2000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 1
          name: cloudflared

kubectl create ns cloudflared
kubectl create -f deployment-cloudflare-tunnel.yaml # above yaml files.

3. Point your service

Let's say you wanna expose the Ingress-Nginx, so you can just point everything at the same tunnel.

Also if your goals is just make HTTPS from user to your site, then just ignore the verify so you dont have to deal with Failed termination or ininitfy loops.

To edit, just click on "Additional application settings"

4. Lets try expose an app

This is example of my apps running in kubernetes withs service name nginx-test-expose

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: nginx
spec:
  ingressClassName: nginx
  rules:
  - host: "not-actual-my-domain.io"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx-test-expose
            port:
              number: 8080
  tls:
  - hosts:
    - not-actual-my-domain.io

Final, results

Pros and Cons

If you want to implement this architecture, you have to consider did your egress traffic is free? if yes, the you are lucky. If not you should consire just using Public IP just like usual. Cause traffic cost is very scary!

The pros is you have your simple apps easy quck behind cloudflare, anti DDOS, etc. But the cons is you are depens on cloudflare Proxy (But is actualy the same like you are using cloudflare with Proxied).

The choice is yours!

Configuring WordPress Behind Ingress Nginx in Kubernetes with Cloudflare

Tue, 26 Sep 2023 21:48:36 GMT

This setup will help you configure WordPress behind an Ingress Nginx in a Kubernetes cluster while using Cloudflare for SSL/TLS termination and security.

1. Set Home Page

mysql> UPDATE wp_options SET option_value = 'https://migrate.nicolas.io/' WHERE option_name IN ('siteurl', 'home');
Query OK, 2 rows affected (0.00 sec)
Rows matched: 2  Changed: 2  Warnings: 0

2. WordPress Configuration

Add the following logic to your wp-config.php file:

// If we're behind a proxy server and using HTTPS, we need to alert WordPress of that fact
// see also https://wordpress.org/support/article/administration-over-ssl/#using-a-reverse-proxy
//

if( strpos( $_SERVER['HTTP_X_FORWARDED_PROTO'], 'https') !== false )
    $_SERVER['HTTPS'] = 'on';

// (we include this by default because reverse proxying is extremely common in container environments)

3. Set Cloudflare SSL/TLS to Full/Strict

Navigate to your Cloudflare account and update your SSL/TLS settings.

4. Ingress Configuration

Create your Ingress configuration YAML file:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wordpress-migrate
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: 500m
    nginx.ingress.kubernetes.io/proxy-read-timeout: 1000s
    nginx.org/proxy-read-timeout: 1000s
    nginx.ingress.kubernetes.io/configuration-snippet: |
      more_set_headers   "X-Forwarded-Proto https";
spec:
  rules:
    - host: "migrate.nicolas.io"
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: wp-migrate
                port:
                  number: 80
  tls:
    - hosts:
        - migrate.nicolas.io
      secretName: wildcard-nicolas-io

5. DNS Configuration

Finally, point your DNS to the IP address of your Ingress Nginx.

Getting Secrets from HashiCorp Vault in Ansible Playbooks

Fri, 15 Sep 2023 15:03:57 GMT

In this blog post, I'll explain how to securely manage secret dependencies in your infrastructure using HashiCorp Vault and Ansible. Storing keys inside your repositories can be risky, and I'll show you a better way.

Prerequisites

Before we get started, make sure you have the following prerequisites in place:

Install the Latest Version of Ansible:
- Ensure you have the most up-to-date version of Ansible installed on your system.
Install the Ansible Community HashiCorp Vault Collection:
- You can install it by running the following command:
```
ansible-galaxy collection install community.hashi_vault
```
- To confirm the installation, run:
```
ansible-galaxy collection list | grep  community.hashi_vault
```
Install the Python Module hvac:
- Install the hvac module using pip:
```
pip install hvac
```

Create Your Ansible Playbooks

Here's an example of an Ansible playbook that retrieves secrets from HashiCorp Vault:

---
- name: Return all kv v2 secrets from a path
  hosts: test
  tasks:
    - name: Get required secret from Vault
      ansible.builtin.set_fact:
        apm: "{{ lookup('community.hashi_vault.vault_kv2_get', 'apm', url='https://your.vault-server.service') }}"

    - name: Display the APM Results
      ansible.builtin.debug:
        msg:
          - "Activeate : {{ apm.secret.ELASTIC_APM_ACTIVE }}"
          - "Environment : {{ apm.secret.ELASTIC_APM_ENVIRONMENT }}"
          - "The Secret: {{ apm.secret.ELASTIC_APM_SECRET_TOKEN }}"
          - "The URL : {{ apm.secret.ELASTIC_APM_SERVER_URL }}"

Run Your Ansible Playbooks

To run your Ansible playbooks securely with HashiCorp Vault, follow these steps:

Set the environment variables ANSIBLE_HASHI_VAULT_TOKEN and ANSIBLE_HASHI_VAULT_AUTH_METHOD:

export ANSIBLE_HASHI_VAULT_TOKEN=YOURSECRETKEYYYYYYYYYYYY
export ANSIBLE_HASHI_VAULT_AUTH_METHOD=token

Execute your playbook:

ansible-playbook -i inventory.ini vault.yml

After running your playbook, you'll securely retrieve secrets from HashiCorp Vault and be able to use them in your Ansible tasks.

Menjalankan Windows Server Dengan Kubevirt

Sat, 15 Apr 2023 21:59:36 GMT

Pada artikel ini, saya akan menunjukkan bagaimaan cara memenuhi kebutuhan server Windows Kita dengan KubeVirt. Saya juga akan menjelaskan beberapa kasus di mana Kita bisa memilih Kubevirt sebagai pilihan yang tepat.

Latar Belakang

Saya bekerja untuk perusahaan yang memiliki applikasi yang berjalan di Server Windows, dan biaya menggunakan windows di Public Cloud cukup tinggi. Untuk mengatasi tantangan ini, saya mengusulkan solusi untuk menjalankan server Windows pada BMAAS yang sudah kita sewa (sudah menjadi cluster kubernetes). Pendekatan ini tidak hanya memberikan latensi yang lebih baik karena server ada di Indonesia, tetapi juga signifikan mengurangi biaya.

Persyaratan

1. Virtualisasi Diaktifkan

Pastikan node kubernetes Kita sudah mengaktifkan mode virtualisasi.

2. Penyimpanan Backend Yang Persisten

Kita membutuhkan storageclass kubernetes dengan penyimpanan backend persisten seperti CEPH, Longhorn, vSAN, atau apa pun.

3. Pahami Apa yang Dilakukan Kubevirt

Jika Anda belum memahami apa yang dilakukan kubevirt, kubevirt pada dasarnya melakukan automasi terhadap operasional kvm dan qemu dengan YAML kubernetes. Lihat kubevirt.io

4. Menginstal KubeVirt dan Container Data Importer Engine

Ikuti panduan dari dokumen resmi, atau baca artikel saya sebelumnya tentang kubevirt.

5. Plugin Virt Kubectl

Kita memerlukan plugin ini untuk memudahkan hidup kita, instal dengan krew.sigs.k8s.io

$ kubectl krew install virt

Mari Buat Windows Kita

Dalam blog ini, saya akan menggunakan image base dari cloudbase.it, karena image tersebut sudah terdapat cloudinit atau beberapa orang mungkin mengatakan "Cloud Native".

1. Unduh Image Base

Unduh image base dari tempat di mana kita dapat mengakses cluster kita dengan kubectl.

2. Forward Service CDI Upload Proxy ke Local

Container data Importer Engine mempunyai proxy yang dapat kita gunakan untuk mengunggah file image base kita mejadi PVC. Di sini saya melakukan forwarding port 6969 di localhost saya menuju service cdi-uploadproxy

kubectl port-forward svc/cdi-uploadproxy -n cdi 6969:443

3. Unggah Image Base Dengan Virt Plugin

kubectl virt image-upload \
--image-path=./windows_server_2012_kvm.qcow2 \
--pvc-name=iso-winserver-r12 \
--access-mode=ReadWriteOnce \
--pvc-size=20G \
--storage-class=kind-local \
--uploadproxy-url=https://localhost:6969 \
--insecure

4. Buat Spesifikasi Perangkat Windows Kita

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  creationTimestamp: 2023-04-10T15:03:08Z
  generation: 1
  labels:
    kubevirt.io/os: windows
  name: windowsvm1
spec:
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: windowsvm1
    spec:
      domain:
        cpu:
          cores: 4
        devices:
          disks:
          - disk:
              bus: virtio
            name: disk0
          - cdrom:
              bus: sata
              readonly: true
            name: cloudinitdisk
        machine:
          type: q35
        resources:
          requests:
            memory: 4096M
      volumes:
      - name: disk0
        persistentVolumeClaim:
          claimName: iso-winserver-r12
      - cloudInitConfigDrive:
          userData: |
            #ps1
            net user Administrator THizisMypzzzowrd
        name: cloudinitdisk

5. Verifikasi VM Running State

Running Windows Server With Kubevirt

Sat, 15 Apr 2023 21:59:36 GMT

In this post i will show you how to run your windows server workloads and needs in KubeVirt. Also probably what usecase when you need this.

Background

I work for a company that has workloads running on Windows Servers, and the cost of using public cloud windows is quite high. To address this challenge, I proposed a solution to host the Windows servers on bare metal as service servers (already bootstrapped with Kubespray). This approach not only provides better latency since the servers are in our country (Indonesia), but it also significantly reduces costs.

Prerequisite

1. Virualization Enabled

Make sure your kubernetes nodes already enable Virtualization

2. Presistent Backend Storage

You need storageclass kubernetes with presistent backend storage like CEPH, Longhorn, vSAN or whatever.

3. Understand What Kubevirt Do

If you has't understand what kubevirt do, is basicly do what kvm and qemu automation with YAML kubernetes. Go checkout kubevirt.io

4. Installed the KubeVirt and Container Data Importer Engine

Follow the guide from official docs, or here my previouse introduction about kubevirt

5. Virt Plugin Kubectl

We need this plugin to make our life easier, install it with krew.sigs.k8s.io

Let's Deploy Your Windows

In this blog i will use cloudbase.it image, bacause the image is already cloudinit-able or some people may say "Cloud Native"

1. Download the Image

Download the image where ever you can access your your cluster with kubectl

2. Forward Your CDI Upload Proxy

kubectl port-forward svc/cdi-uploadproxy -n cdi 6969:443

3. Upload With Virt Plugin

kubectl virt image-upload \
--image-path=./windows_server_2012_kvm.qcow2 \
--pvc-name=iso-winserver-r12 \
--access-mode=ReadWriteOnce \
--pvc-size=20G \
--storage-class=kind-local \
--uploadproxy-url=https://localhost:6969 \
--insecure

4. Specifiying Your VM

apiVersion: kubevirt.io/v1alpha3
kind: VirtualMachine
metadata:
  creationTimestamp: 2023-04-10T15:03:08Z
  generation: 1
  labels:
    kubevirt.io/os: windows
  name: windowsvm1
spec:
  running: true
  template:
    metadata:
      creationTimestamp: null
      labels:
        kubevirt.io/domain: windowsvm1
    spec:
      domain:
        cpu:
          cores: 4
        devices:
          disks:
          - disk:
              bus: virtio
            name: disk0
          - cdrom:
              bus: sata
              readonly: true
            name: cloudinitdisk
        machine:
          type: q35
        resources:
          requests:
            memory: 4096M
      volumes:
      - name: disk0
        persistentVolumeClaim:
          claimName: iso-winserver-r12
      - cloudInitConfigDrive:
          userData: |
            #ps1
            net user Administrator THizisMypzzzowrd
        name: cloudinitdisk

5. Verify VM Running State

[Sebuah Cerita Tentang Implementasi GitOps] - Management Secret Dengan Vault dan ArgoCD

Thu, 02 Feb 2023 16:10:00 GMT

Ketika menggunakan repository sebagai media penyimpanan deployment files, maka tantangan yang dihadapi adalah management sensitif data pada deployment tersebut

Latar belakang

Ketika akan melakukan implementasi GitOps di kubernetes (Menggunakan Git sebagai tempat penyimpanan Manifest dari resources Kubernetes) maka melakukan management sensitif data perlu dilakukan dengan rapihh~~~, menuliskan sensitif data secara telanjang pada Git Repo yang digunakan sebagai pusat operation kita adalah sesuatu yang kurang etis.

Masalah

Untuk dapat melakukan management resource Kind=Secret pada kubernetes bisa dihandle dengan sangat cantik oleh CustomResoureDefintion dari External Secrets Operator, namun terkadang konfidensial data perlu juga diletakkan pada resources lain selain seperti ConfigMaps atau resources lainnya, sayangnya hal tersebut tidak disupport oleh external-secrets operator.

Statement dari design-crd-spec.md

Contoh keperluan confidential data di resources selain secret.

External Secret Operator

Pros	Cons
There's some schedule `RefreshInterval` by design	Only support `Kind=Secret` resources
More mature project also more wide comunity
Multiple namespace management secret by design

ArgoCD Vault Plugin

Pros	Cons
Handle all resources k8s	Required hard refresh the application, if we want to pull new values from vault
ArgoCD Ecosystem	Required third party tools like `relector` or create multi application in all wanted namespace where resources going to deploy

Pilih Solusi Sesuai Kebutuhan

Apabila tidak ada keperluan konfidensial data di luar Kind=Secret maka Externalsecret.io operator adalah piliahan paling cantik.

External Secret Operator

Penggunaan External Secret secara sederhana seperti menifest dibawah. Secret tersebut akan terbuat di namespace yang memiliki label eso=test

ArgoCD Vault Plugin

Untuk argocd-vault-plugin, tools tersebut akan dipasang sebagai plugin dari argocd. Setiap saat sebuah manifests akan di-render maka command tersebut bisa dijalankan.

Lakukan penyesuaian pada Deployment argocd-repo-server kita pada beberapa bagian.

Menambahkan initContainer.
Menambahkan sidecar container.
Melakukan Mounting ServiceAccountToken.
Menambah clusterrolebinding di untuk SA argocd-repo-server.

1. Create backend configruation

Di sini backend saya menggunakan Vault dari Hascicorp dan menggunakan metode autentifikasi github token.

apiVersion: v1
stringData:
  VAULT_ADDR: http://vault.myvaultserver.com
  AVP_AUTH_TYPE: github
  AVP_GITHUB_TOKEN: t0ke3nsecret
  AVP_TYPE: vault
kind: Secret
metadata:
  name: argocd-vault-plugin-credentials
  namespace: argocd
type: Opaque

2. InitContainer - Download binary argocd-vault-plugin

# Download tools
initContainers:
- name: download-tools
image: registry.access.redhat.com/ubi8
env:
  - name: AVP_VERSION
    value: 1.11.0
command: [sh, -c]
args:
  - >-
    curl -L https://github.com/argoproj-labs/argocd-vault-plugin/releases/download/v$(AVP_VERSION)/argocd-vault-plugin_$(AVP_VERSION)_linux_amd64 -o argocd-vault-plugin &&
    chmod +x argocd-vault-plugin &&
    mv argocd-vault-plugin /custom-tools/
volumeMounts:
  - mountPath: /custom-tools
    name: custom-tools
volumes:
- configMap:
    name: cmp-plugin
  name: cmp-plugin
- name: custom-tools
  emptyDir: {}

3. Configure argocd-repo-server Container

Add backend Secret to Environment Variables in argocd-repo-server container, and mount argocd-vault-plugin.

containers:
- name: argocd-repo-server
volumeMounts:
- name: custom-tools
  mountPath: /usr/local/bin/argocd-vault-plugin
  subPath: argocd-vault-plugin        
envFrom:
  - secretRef:
      name: argocd-vault-plugin-credentials

4. Add ClusterRoleBinding

Untuk dapat membaca konfigurasi backend vault yang sudah kita masukkan ke secret argocd-vault-plugin-credentials maka token dari service account argocd-repo-server perlu mendapatkan akses read ke secrets.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  # "namespace" omitted since ClusterRoles are not namespaced
  name: argocd-vault
rules:
- apiGroups: [""]
  #
  # at the HTTP level, the name of the resource for accessing Secret
  # objects is "secrets"
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
# This cluster role binding allows anyone in the "manager" group to read secrets in any namespace.
kind: ClusterRoleBinding
metadata:
  name: read-secrets-argocd-repo-server
subjects:
- kind: ServiceAccount
  name: argocd-repo-server
  namespace: argocd
roleRef:
  kind: ClusterRole
  name: argocd-vault
  apiGroup: rbac.authorization.k8s.io

5. Mounting ServiceAccountToken

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      serviceAccount: argocd-repo-server
      serviceAccountName: argocd-repo-server
      automountServiceAccountToken: true

6. Buat konfigurasi plugin argo-cd

YAML berikut bukanlah resources dari kubernetes, namun syntax dari Plugin argo-cd yang mengikuti rules dari Kubernetes YAML.

Plugin ini hanya akan berjalan apabila terdapat file ./kustomization.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: cmp-plugin
data:
  avp-kustomize.yaml: |
    ---
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: argocd-vault-plugin-kustomize
    spec:
      allowConcurrency: true

      # Note: this command is run _before_ anything is done, therefore the logic is to check
      # if this looks like a Kustomize bundle
      discover:
        fileName: "./kustomization.yaml"
      generate:
        command:
          - sh
          - "-c"
          - "kustomize build . |  argocd-vault-plugin generate --secret-name argocd:argocd-vault-plugin-credentials -"
      lockRepo: false
---

7. Tambahkan sidecar container.

Sidecar container ini ditambahkan pada deployment argocd-repo-server.

# argocd-vault-plugin with Kustomize
- name: avp-kustomize
command: [/var/run/argocd/argocd-cmp-server]
image: quay.io/argoproj/argocd:v2.4.0
securityContext:
  runAsNonRoot: true
  runAsUser: 999
volumeMounts:
  - mountPath: /var/run/argocd
    name: var-files
  - mountPath: /home/argocd/cmp-server/plugins
    name: plugins
  - mountPath: /tmp
    name: tmp
  # Register plugins into sidecar
  - mountPath: /home/argocd/cmp-server/config/plugin.yaml
    subPath: avp-kustomize.yaml
    name: cmp-plugin
  # Important: Mount tools into $PATH
  - name: custom-tools
    subPath: argocd-vault-plugin
    mountPath: /usr/local/bin/argocd-vault-plugin

Contoh Penggunaan

Buat Secret baru di hasci vault

Buat resources secret, dengan annotation avp.kubernets.io/path: "secret/data/test-secret", annotation tersebut menunjukan alamat dari path secret yang akan diambil. Lalu <supper-secret> pada isi stringData secret tersebut menunjukan property key yang harus diambil dari path yang di-define pada annotation.

Deploy dengan Argocd

Lihat final Secret

Kekurangan Saat ini

Jika kita menggunakan ArgoCD Plugin dan juga argocd-image-updater maka ada sedikit penyesuaian yang perlu dilakukan pada setiap YAML application, ya itu penambahan annotation writeback target

annotation:
  argocd-image-updater.argoproj.io/write-back-target: kustomization

My Talk at Kubernetes Meetup X BukaMeetup

Referensi

Melakukan Orkestrasi Virtual Machine Dengan KubeVirt

Mon, 28 Nov 2022 17:39:00 GMT

Cerita tentang bermain (lagi) di Baremetal. KubeVirt, sebuah operator virtual machine yang didesain untuk melakukan management VM melalui Kubernetes CRD.

Latar belakang

Selayaknya hari mager seperti biasa (dikarenakan semuanya sudah beres), saya scroll twitter dengan santuy ... mata saya sekejap melihat akun CNCF Official meberikan re-twitt terhadap post @kubevirt yang berisikan release terbaru dari mereka, dengan rasa penasaran "Apaan nih kubevirt ?" sambil memberikan dugaan awal, saya mengujungi akun tersebut.

Seperti yang saya duga sejak awal, KubeVirt "Paling ~~cuman~~ jalanin VM di container pake virt, manage pake CRD kube". Setelah saya membaca sedikit, saya kembali ke timeline dan melanjutkan scroll ....

Kewajiban pun datang. Seperti kata saudara se-tongkrongan Bekasi JustHumanz satu satunya hal sulit jika bermain di public cloud adalah Cost Optimization.

Saya mendapatkan sebuah situasi dimana ada baremetal yang hanya berisikan 1 VM akan segera Expired (this is cost optimization desicion moment), di sisi lain saya memiliki kubernetes cluster yang berjalan di atas baremetal tanpa ada overlay VMs. Seketika KubeVirt terpanggil di benak saya.

Masalah

Tidak ada masalah, hanya ada "Ini kalau bisa dipindahin ke sini, yang ini bisa di-terminate", Cost Optimization Moment~~

Saya sangat bisa memahami hal tersebut, dan untuk hal ini saya sangat setuju. Maka dari itu saya akan mencari ~~MASALAH~~ solusi memindahkan VM tersebut ke Kubernetes Cluster dengan bantuan KubeVirt.

Solusi

Memahami Arsitektur dari KubeVirt

Dikarenakan bermain dengan virt adalah hal yang sudah pernah saya lakukan di kehidupan sebelumnya (maksudnya di kantor sebelumnya ) hal ini merupakan "gitu gitu doang moment ~~"

Dengan bahasa anak tongkrongan yang "semoga" mudah dimengerti,

KubeVirt adalah operator dengan CRD kubernetes yang nantinya akan melakukan spawn container berisikan kvm dan segala utility-nya seperti libvirtd, virsh, dkk.

Untuk persistent disk, qemu image-nya akan disupply melalui PVC, atau bisa menggunakan epehemeral volume untuk disk yang bersifat temporary, seperti SWAP.

Dikarenakan VM tersebut berada di overlay container, maka networking dari VM akan menggunakan network CNI kubernetes. Sehingga VM dan juga Pod App kita berada di Network yang sama.

Penjelasan di atas hanya "how it works", dan sependek pengetahuan saya untuk lebih detail, lengkap dan akurat silahkan merujuk ke https://kubevirt.io/user-guide/architecture/

Kembali ke LABtop

Seperti kebiasaan lama, sebelum membawa sesuatu untuk dihidangkan, saya selalu memasak matang - matang hidangan tersebut di Lab pribadi.

Install KubeVirt Operator

# Point at latest release
$ export RELEASE=$(curl https://storage.googleapis.com/kubevirt-prow/release/kubevirt/kubevirt/stable.txt)
# Deploy the KubeVirt operator
$ kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-operator.yaml
# Create the KubeVirt CR (instance deployment request) which triggers the actual installation
$ kubectl apply -f https://github.com/kubevirt/kubevirt/releases/download/${RELEASE}/kubevirt-cr.yaml
# wait until all KubeVirt components are up
$ kubectl -n kubevirt wait kv kubevirt --for condition=Available

Install CDI Operator

Container Data Importer, salah satu komponen yang dibutuhkan untuk management disk PVC pada VM nantinya.

# Point at latest release
$ export TAG=$(curl -s -w %{redirect_url} https://github.com/kubevirt/containerized-data-importer/releases/latest)
$ export VERSION=$(echo ${TAG##*/})
# Deploy CDI operator
$ kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-operator.yaml
$ kubectl create -f https://github.com/kubevirt/containerized-data-importer/releases/download/$VERSION/cdi-cr.yaml

Install Kubectl Virt Plugin

Hal ini Opsional, namun untuk hidup yang lebih baik sebaiknya dilakukan, dikarenakan akan digunakan untuk melakukan forwarding ke pod virt-launcer, dan akan otomatis membuka vnc client pada device kita.

krew install virt

Halangan

Backend Storage

Di sini lah sebuah keputusan yang sangat besar perlu ditentukan. Keputusan yang akan berdampak pada kehidupan setelah VM terdeploy.

"choose your backend storage wisely" pertimbangkan dengan seksama. Apa kebutuhannya, dan bagaimana kondisinya.

Pada kasus ini kebutuhan saya hanya memindahkan VM tersebut ke worker kubernetes dengan KubeVirt, lalu kemudahan ketika scaling up disk, cpu, dan ram jika dibutuhkan. Namun keadaannya adalah tidak ada block storage kosong yang dapat digunakan.

Hostpath Operator

Pilihan saya jatuh pada hostpath operator

1. Install Hostpath Provisoner

# Deploy cert-manager, do not required if already deployed (1.7.x up version).
$ kubectl create -f https://github.com/cert-manager/cert-manager/releases/download/v1.7.1/cert-manager.yaml
# Create hostpath-provisioner namespace
$ kubectl create -f https://raw.githubusercontent.com/kubevirt/hostpath-provisioner-operator/main/deploy/namespace.yaml
# Deploy Webhook 
$ kubectl create -f https://raw.githubusercontent.com/kubevirt/hostpath-provisioner-operator/main/deploy/webhook.yaml -n hostpath-provisioner
# Deploy Operator
$ kubectl create -f https://raw.githubusercontent.com/kubevirt/hostpath-provisioner-operator/main/deploy/operator.yaml -n hostpath-provisioner

2. Create Pool and StorageClass

Data dari PVC nanti akan tersimpan pada node2 di folder /data-kubevirt.

apiVersion: hostpathprovisioner.kubevirt.io/v1beta1
kind: HostPathProvisioner
metadata:
  name: hostpath-provisioner-node2
spec:
  imagePullPolicy: Always
  storagePools:
  - name: node-2
    path: /data-kubevirt
  workload:
    nodeSelector:
      kubernetes.io/hostname: node2
---
allowVolumeExpansion: true
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: hostpath-csi-node2
parameters:
  storagePool: node-2
provisioner: kubevirt.io.hostpath-provisioner
reclaimPolicy: Retain
volumeBindingMode: WaitForFirstConsumer

Kembali ke LAB ( Mari Membuat VMs )

Download Image to PVC with DataVolume

kind: DataVolume
apiVersion: cdi.kubevirt.io/v1beta1
metadata:
  name: ubuntu-image
  namespace: default
spec:
  source:
    http:
      url: https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.img
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 3Gi
    storageClassName: hostpath-csi-node2

Create Virtual Machine

kind: DataVolume
apiVersion: cdi.kubevirt.io/v1beta1
metadata:
  name: myvm-volume-data
spec:
  source:
    pvc:
      namespace: default
      name: ubuntu-image
  pvc:
    accessModes:
      - ReadWriteOnce
    resources:
      requests:
        storage: 30Gi
    storageClassName: hostpath-csi-node2

apiVersion: kubevirt.io/v1
kind: VirtualMachine
metadata:
  name: my-ubuntu
  labels:
    vm: my-ubuntu
spec:
  runStrategy: Always
  template:
    metadata:
      labels:
        vm: my-ubuntu
    spec:
      terminationGracePeriodSeconds: 30
      domain:
        resources:
          requests:
            cpu: 2
            memory: 4092M
        devices:
          disks:
          - name: datavolume-disk1
            disk:
              bus: virtio
          - disk:
              bus: virtio
            name: cloudinitdisk
      volumes:
      - dataVolume:
          name: myvm-volume-data
        name: datavolume-disk1
      - name: cloudinitdisk
        cloudInitNoCloud:
          userData: |-
            #cloud-config
            password: Hayomaucarileaksdigithubgwya???ngerixxxxxxxxxAdaHeker
            chpasswd: { expire: False }

Operasional VM

Get in to Console

Get to the console with kubectl virt vnc your-vm-name. All YAML that used here is on my github repos here

Resize CPU, and RAM

Change the CPU, and memory section with resource you want, and make sure your worker is have that space.

$ kubectl delete vm/my-ubuntu
$ vim virtualmachine.yaml
[...]
        resources:
          requests:
            cpu: 2
            memory: 4092M
[...]
$ kubectl apply -f virtualmachine.yaml

Scale Up Disk

In this case, resize disk become tricky, because hostpath-provisioner is different than real storage cluster like Rook, longhorn. So we need to scale image manualy with qemu-img comand

$ kubectl delete vm/my-ubuntu
$ kubectl virt guestfs myvm-volume-data --root --image=quay.io/kubevirt/libguestfs-tools:v0.57.1
[...]
bash-4.4 # qemu-img resize disk.img 10G
[...]
$ kubectl apply -f virtualmachine.yaml

Expose VM Port with Service

You can expose the what you want trough service kubernetes.

apiVersion: v1
kind: Service
metadata:
  name: vmiservice
spec:
  ports:
  - port: 27017
    protocol: TCP
    targetPort: 22
  selector:
    vm: my-ubuntu
  type: LoadBalancer
---    
apiVersion: v1
kind: Service
metadata:
  name: ubuntu-domain
spec:
  ports:
  - port: 9100
    protocol: TCP
    targetPort: 9100
    name: metrics
  - port: 8080
    protocol: TCP
    targetPort: 8080
    name: app
  selector:
    vm: my-ubuntu
  type: ClusterIP

Akhiran (Opinion are my own)

Singkat cerita setelah mengimplementasi hal tersebut saya berhasil memindahkan 1 VM tersebut ke Managed by KubeVirt yang berdampak baik karena tentu saja mengurangi biaya server.

TLDR; Saya menghadiri Kubernetes Comunity Day, dan di sana saya sempat berbincang di booth RedHat, karena saya melihat mereka melakukan demostrasi terhadap provisioning Virtual Machine di Openshift mereka.

Pada Cluster Openshift mereka saya melihat bagaimana KubeVirt yang sebenarnya terintegrasi dengan backend storage yang proper. Tanpa tricky tricks seperti pada kasus saya.

Saya dan sobat nongs warga Bekasi kabupaten Ammarun tentu saja membredel berbagai pertanyaan kepada sales Redhat (karena rasa penasaran) mereka juga menjawab dengan senang hati. Lalu warlo(c)k Tambun ini mengajukan pertanyaan terkait "Mengapa yang di show nga ada Openstack ? kan monitornya 2" they said "Kita sudah lebih fokus ke CloudNative, development Openstack saat ini lebih ke Telco". Bahkan kata mereka, Redhat melakukan stop development terhadap RHV (No new feature, just fix bugs. Link announcement ada di referensi).

So where the world of cloud will moving ?, will everything is Webhook Kubernetes in Future ?

Well my opinions everything should be back to what you have, and what you need. That's it. Running openshift for Small company with small scale of loads is overkill, probably what you need is just couple of VMs in Digital Ocean or something.

But if you have your own Data Center, or you Running a Banking system that used by millions of people dependen on your Infrastructure, considering have something that make your life is more easier is something that you have to contemplate. It's does't have to be openshift, or kubernetes in general.

So the Kubernetes-ing is just an option, i guess. Option to manage your repetitive task and automating it. The cloud providers will have their own option to choose what they need with what they have. BUT IF I WAS THE ONE WHO CHOOSE WHAT STACK. DEFINITELY KUBERNETES ON OF TOP BAREMETAL IS THE ONE!

My Talk at Open Infra - Bekasi, Jawa Barat, Indonesia

Saya juga mebawakan materi tersebut pada acara meetup reguler Open Infra Indonesia, pada bulan februari 2023 kemarin.

Live Stream

Google Slides

Referensi

nicolasjulian/KubeVirt: Bundle to getting start with KubeVirt - https://github.com/nicolasjulian/KubeVirt
KubeVirt User-Guide - https://kubevirt.io/user-guide/
Red Hat Virtualization Life Cycle - Red Hat Customer Portal

https://access.redhat.com/support/policy/updates/rhev

OpenShift virtualization: Not as scary as it seems

https://www.redhat.com/en/blog/openshift-virtualization-not-scary-it-seems

Menghalang Serangan - Serangan Dasar Web Dari Sisi Ingress Nginx (Infrastruktur)

Tue, 20 Sep 2022 00:45:00 GMT

Menghalangi serangan pada aplikasi berbasis web merupakan sebuah PR yang sangat panjang. Di tulisan ini saya akan mencoretkan beberapa langkah awal melakukan pertahanan terhadap serangan-serangan yang "biasa" terjadi. Penghalang ini diterapkan di sisi Infrastruktur.

Latar belakang

Pada dasarnya hampir semua serangan pada aplikasi berbasis web menggunakan HTTP Request, sehingga hal pertama yang bisa dilakukan adalah melakukan filtering terhadap request yang diterima applikasi tersebut.

Filtering terhadap request yang masuk tersebut, disebut WAF (Web Application Firewall).

Sudah banyak berbagai penyedia WAF ternama seperti Imperva, Cloudflare, hingga untuk skala bisnis lebih besar seperti Akamai.

Untuk saat ini Cloudflare merupakan penyedia WAF yang memberikan layanannya secara cuma-cuma kepada para penggunanya. Hanya cukup menggunakan NS milik Cloudflare, pengguna sudah bisa mengaktifkan WAF secara Gratis.

Masalah

Bypass Cloudflare

Ketika pengguna mengaktifkan WAF dari cloudflare, maka semua request yang masuk ke domain tersebut akan diarahkan menuju reverse proxy milik Cloudflare yang sudah diberikan filtering untuk menyaring semua request yang masuk. Namun apabila seseorang menemukan IP dibalik dari WAF tersebut. Maka dia bisa leluasa melakukan pointing langsung dari /etc/hosts miliknya pribadi untuk melakukan bypass terhadap WAF cloudflare.

Pendekatan Untuk Dilakukan

Pada dasarnya tidak ada tolak ukur yang pasti untuk mengukur sebuah sistem "Aman", namun seperti kata Voltaire "Perfect is the enemy of good".

Penggambaran di bawah dapat digunakan juga dalam hal Security Design. Tolak ukur untuk melakukan hardening yang "secure" mutlak selamanya adalah tak terhingga.

Tujuan kita ada di titik "Something" agar mengurangi point of attack hingga seminim mungkin. Untuk mencapai itu ada beberapa pendekatan yang dapat dilakukan.

1. Jangan Pernah Mematikan WAF

Jangan mematikan WAF dari penyedia seperti Cloudflare. Dikarenakan dapat meng-ekspose alamat IP origin server dibalik WAF tersebut, dan tersimpan selamanya di record list seperti Security Trails, dnsdumpster.

2. Aktifkan ModSecurity & OWASP ModSecurity CRS (Ingress Nginx)

Mengaktifkan ModSecurity Pada Ingress

ConfigMaps

apiVersion: v1
data:
  allow-snippet-annotations: "true"
  enable-modsecurity: "true"
  enable-owasp-modsecurity-crs: "true"
kind: ConfigMap
metadata:
  name: ingress-nginx-controller

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: bwapp-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/modsecurity-snippet: |
      SecRuleEngine On

Perbandingan ketika mengaktifkan ModSecurity tersebut.

Untuk melakukan testing terhadap efisiensi dari ModSecurity di sini saya melakukan spawning bwapp. Full deployment dapat dilihat di https://github.com/nicolasjulian/bwapp-k8s

Melakukan serangan terhadap paramter rentan title, berjalan dengan sangat mudah.

Melakukan serangan terhadap paramter rentan title, dengan metode yang sama. Namun tidak berhasil seperti sebelumnya.

Dapat juga terlihat ada 249 request yang mendapatkan balikan 403 (Forbidden) dapat dipastikan penghalang yang dipasang bekerja dengan baik.

Penampakan logs yang masuk ke ingress ketika aplikasi web tersebut mendapat percobaan serangan SQL Injection. Semua request yang dianggap abusive diblok oleh ModSecurity.

Percobaan Bypass ModSecurity dengan Tampering SQLMAP

Saya mencari berbagai bypass yang untuk filtering dari ModSecurity tersebut, salah satunya yang saya coba seperti gambar di bawah. Namun metode tersebut tidak berhasil.

python3 sqlmap.py  -u "https://bwapp.staging.nicolas.my.id/sqli_1.php?title=aa&action=search"  --cookie='PHPSESSID=kt0q1vf9dfdushriq9kdd5vj23; security_level=0' --tables --level 5 --random-agent --batch

Secara garis besar kita sudah berada pada titik lebih baik daripada sebelumnya. SQL Injection tidak bisa dilakukan dengan sangat mudah.

ModSecurity only is not enough to protect your website You need to fix the BUG.

3. Ketahui Aset Kita Yang Terekspos

Lakukan aset discovery terhadap infrastruktur kita sendiri. Dengan melakukan subdomain enumeration menggunakan simple tools seperti Amass, Subfinder, knockpy, atau gunakan Security Trails.

Lakukan pencarian sensitif informasi seperti environment variable, password, ip address. Menggunakan tools seperti Truffle Hog.

Jika kita menemukan sesuatu yang semestinya tidak ter-ekspose ke publik, maka sebaiknya segera kita lakukan mitigasi pada temuan tersebut.

4. (Cloudflare as Front Liner) Hanya Izinkan Subnet Cloudflare

Salah satu pencegahan yang mudah dilakukan untuk menghindari bypass cloudflare adalah, dengan mengizinkan hanya subnet cloudflare saja yang boleh mengakses port dari service kita. Informasi tersebut dapat ditemukan di https://www.cloudflare.com/ips. Hal ini seharusnya dapat dilakukan juga pada sistem WAF yang lain.

Referensi

NGINX Tutorial: Protect Kubernetes Apps from SQL Injection - NGINX https://www.nginx.com/blog/microservices-march-protect-kubernetes-apps-from-sql-injection/
ModSecurity with ingress-nginx https://containerinfra.com/blog/cloud/2021-07-30-using-nginx-ingress-waf/
OWASP ModSecurity Core Rule Set | OWASP Foundation https://owasp.org/www-project-modsecurity-core-rule-set/
How To Use ModSecurity and OWASP CRS For Web App Firewall (WAF) To Secure Your Website https://blog.runcloud.io/modsecurity-firewall/#what-is-owasp-crs
Annotations - NGINX Ingress Controller https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#modsecurity
https://github.com/sqlmapproject/sqlmap/issues/4529 [CRITICAL] WAF/IPS identified as ‘’modsecurity (Trustwave)’ · Issue #4529 · sqlmapproject/sqlmap

[Production Case] Menelusuri Misteri Kubernetes DNS Penyebab "ENOTFOUND"

Sat, 20 Aug 2022 15:19:00 GMT

Sepengal kisah tentang jejak penelusuran pada salah satu komponen utama Kubernetes Cluster.

Latar belakang

Dikisahkan di sebuah startup terdapat urban legend mengenai Apps yang berjalan di atas Kubernetes namun sering mendapatkan logs error dikarenakan gagal melakukan resolve ke DNS. Apps tersebut dibangun dengan menggunakan Java Scripts berjalan dengan Runtime node.js.

getaddrinfo ENOTFOUND disini.namadomain.com

Dikarenakan gagalnya resolve ke DNS tersebut, membuat apps sering bermasalah.

Penelusuran

Pada malam jum'at pukul 23:11, Kota Bekasi, Jawa Barat. Di sebuah kamar temenung sosok pemuda yang sedang menatapi langit-langit kamar. Dengan suara latar lagu NDX AKA - Sido Sayang Ora. Pemuda tersebut dalam hati berkata "Si A███ lagi ngopo yo ?, jane ki kepiye to karep e ki ?" pemuda itu adalah ~~saya~~ Sobat Ambyar. Untuk kedepannya akan kita sebut singkat sebagai "Sobyar".

Sobyar seketika terbangun dari ranjang tidurnya. Karena ingin melepaskan penat memikirkan sang pujaan hati yang entah menghilang kemana. Ia ingin menghibur diri. CTRL + ENTER, terbukalah Terminal dengan mode fullscreen. Dirinya teringat akan getaddrinfo ENOTFOUND yang terjadi di Cluster Kubernetes tempat Sobyar bekerja.

Sobyar kemudian membuka nvim sambil membuka browser.

Pencarian Solusi

Setelah membaca berbagai sumber, sobyar mengkrucutkan ke beberapa pendekatan solusi. Diataranya sebagai berikut

1. Masalah Node.js Library

Dikarenakan apps berjalan menggunakan runtime node.js, maka Sobyar mencari - cari apakah ada keterkaitan dengan Bug di NodeJS. Hingga dia menemukan artikel terkait salah satu bug di node/lib/dns.js.

function errnoException(err, syscall, hostname) {
  // FIXME(bnoordhuis) Remove this backwards compatibility nonsense and pass
  // the true error to the user. ENOTFOUND is not even a proper POSIX error!
  if (err === uv.UV_EAI_MEMORY ||
      err === uv.UV_EAI_NODATA ||
      err === uv.UV_EAI_NONAME) {
    err = 'ENOTFOUND';
  }

Namun ternyata issue tersebut sudah di-fix pada node v8.x, dan node di production sudah menggunakan v12.x. Dapat disimpulkan bahwa masalah yang terjadi bukan dari bug Runtime.

2. Memasang Pod CoreDNS Cache Dengan Node Local DNS

Selain masalah ETNOTFOUND ada masalah lain yang terjadi di kantor Sobyar, yaitu waktu resolving yang cukup lama ketika traffic sedang tinggi-tingginya.

Seakan Qos coredns termakan HABIS!, maka pada malam itu Sobyar mencari juga apakah ada "tune-up" untuk masalah performa coredns tersebut, dan ia menemukan; KubeCon Talk Going from 5s to 5ms: Benefits of a Node-Local DNSCache - Pavithra Ramesh & Blake Barnett .

Sebenarnya Sobyar nga ngerti-ngerti amat secara detil mendalamnya namun dia paham gambaran gampangnya.

Penjelasan Singat dan Hasil Implementasi

NodeLocalDNS akan membuat pods di setiap nodes kubernetes yang berperan sebagai coredns cache (Daemonset).

Pod tersebut akan menggunakan hostNetwork dan membuat interface baru, dengan IP address yang sama seperti IP dari service kube-dns. Sehingga ketika pods di setiap nodes akan melakukan resolve dns maka akan di-routing ke pods LocalDNS di masing-masing nodes. CMIIW

Hari semakin larut, jam dinding menujukan waktu pukul dua dini hari. Sobyar bergegas tidur agar besok tidak mengantuk di KRL.

Siang hari di jam kerja, Sobyar melakukan diskusi dan meminta izin kepada petuah di startup tempat ia bekerja untuk melakukan ritual pemasangan nodelocaldns. Dengan segera para petuah pun menyetujui.

Ritual pemasangan nodelocaldns berjalan dengan lancar, namun masih belum terlihat jelas perbedaan yang signifikan.

Hingga hari event tiba, beberapa cluster mendadak stuck ketika melakukan resolving dns. Namun tidak terjadi pada cluser yang sudah dilakukan ritual nodelocaldns. Sehingga sekarang dapat terlihat jelas perbandingannya.

Setelah itu semua cluster critical Kubernetes dilakukan ritual nodelocaldns., dan benar saja seketia cluster yang stuck langsung berjalan normal kembali.

Sobyar pun membuat HELM Template untuk mempermudah ritualnya.

helm repo add auto-node-local-dns https://nicolasjulian.github.io/node-local-dns/
helm install my-node-local-dns auto-node-local-dns/node-local-dns --version 0.1.0

Namun, apakah sudah tidak ditemukan logs ENOTFOUND ? jawabannya masih, namun tidak sebanyak sebelumnya.

3. Efisiensi resolv.conf

Di malam hari Soybar menyalakan rokok, dan membuat secangkir kopi dengan mug yang bertuliskan "1000 hari mengenang bapak █████████" saat itu malam minggu. Sobyar tidak keluar malam itu, suasana hatinya masih sama "ra karu - karu an, pokok e".

Dinyalakannya Laptop, dan seketika terminal terbuka. Ia bergumam, "opo aku kurang sat set sat set yo ?, dek'e wis karo wong liyo koyone.., jan██████"

Untuk menghilangkan penat. Maka sobyar memilih membaca die.net dikarenakan Ia menemukan salah satu artikel tentang penurunan performa coredns ketika menggunakan ndots:5

resolv.conf akan menentukan bagaimana perilaku ketika melakukan resolving ke DNS Server.

Ada dua hal yang perlu dipahami

search

search Search list for host-name lookup.

The search list is normally determined from the local domain name; by default, it contains only the local domain name. This may be changed by listing the desired domain search path following the search keyword with spaces or tabs separating the names. Resolver queries having fewer than ndots dots (default is 1) in them will be attempted using each component of the search path in turn until a match is found. For environments with multiple subdomains please read options ndots:n below to avoid man-in-the-middle attacks and unnecessary traffic for the root-dns-servers. Note that this process may be slow and will generate a lot of network traffic if the servers for the listed domains are not local, and that queries will time out if no server is available for one of the domains.

The search list is currently limited to six domains with a total of 256 characters.

ndots

ndots:n

sets a threshold for the number of dots which must appear in a name given to res_query(3) (see resolver(3)) before an initial absolute query will be made. The default for n is 1, meaning that if there are any dots in a name, the name will be tried first as an absolute name before any search list elements are appended to it. The value for this option is silently capped to 15.

Default Kubernetes Config resolv.conf

root@nginx:/# cat /etc/resolv.conf 
nameserver 10.96.0.10
search <namespace>.svc.cluster.local svc.cluster.local cluster.local
options ndots:5

Rangkuman search & ndots

Search memiliki 4 list value, seperti yang dilihat di atas, list tersebut berisikan local domain name default dari kubernetes. Sedangkan ndots menentukan berapa jumlah titik yang diperlukan sebuah domain dapat disebut "FQDN (Fully qualified domain name)" dan dilakukan resolv pertama kali.

Bagaimana kerja default resolv.conf Kubernetes

Supaya mempercepat latency, ndots diberikan value 5 (default). Sehingga domain yang pertama kali dilakukan resolv adalah nama-service.<namespace>.svc.cluster.local.Sangat masuk akal performanya akan sangat baik, jika domain name yang diresolve adalah pada level service Kubernetes Cluster. Namun jika yang di-resolve adalah public domain maka hal tersebut hanya akan menimbulkan becana.

Berikut adalah contoh melakukan resolve public domain, jika menggunakan default ndots values. Empat kali proses pathing nama domain dengan local domain name search supaya mencapai path yang benar.

Berikut contoh hasil jika menggunakan ndots:1, dikarenakan hanya perlu satu titik di dalam nama domain untuk menjadi FQDN maka nama dns langsung menuju ke google.com

Malam itu berjalan dengan cepat. Singkat certia hari Senin tiba, Sobyar menyampaikan temuannya kepada petuah, dan meminta izin untuk melakukan ritual perubahan Deployment.

  dnsConfig:
    options:
      - name: ndots
        value: "1"

Petuah memberi izin,

"minggu depan yaa ..."

"Wokeee ..." jawab Sobyar. Minggu yang dimaksud telah tiba, pada hari itu developer Apps tersebut juga menanyakan perihal solusi untuk masalah tersebut. Momen yang tepat. Deployment pun dilakukan perubahan di hari itu.

Benar saja. Logs ENOTFOUND berangsur menghilang untuk service tersebut, dan 1 jam kemudian semua error ENOTFOUND sudah tidak terlihat. Everybody happy!

Berikut logs selama 2 minggu setelah melakukan tuning.

Satu chart spike terakhir di atas adalah percobaan apabila deployment tidak menggunakan efisien resolv.conf

Penyesuaian Lainnya

Perubahan Upstream DNS

Hal ini dilakukan karena ternyata /etc/resolv.conf bawaan dari cloud provider, mengarah ke DNS yang berada di belahan dunia lain. Sobyar mengangap hal seperti itu tidak best practice.

Mengaktifkan AutoPath

Menjadi tidak berguna apabila sudah melakukan efisiensi resolv.conf

If the autopath plugin sees a query that matches the first element of the configured search path, it will follow the chain of search path elements and return the first reply that is not NXDOMAIN

Berikut adalah full configMaps pada bagian Corefile.

  Corefile: |-
    .:53 {
        log
        errors
        autopath @kubernetes
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods verified
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . 8.8.8.8 8.8.4.4 {
          prefer_udp
          policy sequential
        }
        cache 30
        loop
        reload
        loadbalance
    }

Referensi

nicolasjulian/node-local-dns: Deploy Helm Chart for Node Local DNS with GitOps Way https://github.com/nicolasjulian/node-local-dns
What is DNS? | How DNS works | Cloudflare https://www.cloudflare.com/learning/dns/what-is-dns/
autopath https://coredns.io/plugins/autopath/#:~:text=Description,the%20original%20reply%20is%20returned.
Kubernetes_DNS_Horror_Stories.pdf https://static.sched.com/hosted_files/kccnceu20/de/Kubernetes_DNS_Horror_Stories.pdf
Embracing high request loads on Kubernetes | by Nitin Bisht | Blocket Engineering | Medium https://medium.com/blocket-engineering/embracing-high-request-loads-on-kubernetes-1c9d4e84c508
DNS Lookups in Kubernetes https://mrkaran.dev/posts/ndots-kubernetes/
Kubernetes pods /etc/resolv.conf ndots:5 option and why it may negatively affect your application performances https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html
How to fix nodejs DNS issues?. Networking issues are always most… | by Aleksandar Mirilovic | Medium https://medium.com/@amirilovic/how-to-fix-node-dns-issues-5d4ec2e12e95
Setting up NodeLocal DNS Cache | Yandex Cloud - Documentation https://cloud.yandex.com/en/docs/managed-kubernetes/tutorials/node-local-dns
AWS Elastic Kubernetes Service: load-testing and high-load tuning – problems and solutions https://rtfm.co.ua/en/kubernetes-load-testing-and-high-load-tuning-problems-and-solutions/

[Personal Project] Menghapus Bloatware Pada Ponsel Xiaomi

Sun, 03 Jul 2022 14:30:00 GMT

Latar Belakang

Sudah lama saya merasa bahwa ponsel yang saya gunakan, semakin lama penyimpanan yang tersisa semakin sedikit. Maka dari itu, saya merencanakan akan melakukan pemeliharaan dengan membersihkan files yang sekiranya sudah jarang saya akses secara mobile.

Masalah

Namun di tengan jalan ketika melakukan pembersihan saya merasa bahwa ada sesuatu yang sangat menggangu yaitu iklan dari aplikasi-aplikasi bawaan Xiaomi yang tidak pernah saya gunakan.

Solusi

Hapus Bloatware. Saya mencoba meng-Uninstall aplikasi - aplikasi tersebut namun tidak terdapat opsi Uninstall. Saya sempat berfikir mungkin diperlukan akses root untuk melakukan tindakan tersebut. Tapi ternyata ada trik untuk melakukan hal tersebut tanpa memerlukan akses root.
Pindahkan data ke HDD.

Hapus Bloatware Dengan ADB (Android Debug Bridge)

Untuk meng-Uninstall Aplikasi yang tidak berguna bawaaan dari OS tersebut kita hanya perlu melakukannya via SHELL tanpa harus memiliki priviledge root.

Install ADB (Ubuntu)

$ sudo apt install -y adb                             		      
$ sudo apt list | grep adb

Hubungkan ADB Ke Ponsel Xiaomi

$ adb devices                                                              
* daemon not running; starting now at tcp:5037
* daemon started successfully
List of devices attached

$ adb devices                                                             
List of devices attached
HU7HQ4PZCY9TFY75	device

Lis Semua Aplikasi

$ adb shell                                                                
angelica:/ $
angelica:/ $ pm list packages -s                                                       
package:com.miui.screenrecorder
package:com.mediatek.ims
package:com.google.android.apps.subscriptions.red
package:com.mediatek.op01.phone.plugin
package:com.android.cts.priv.ctsshim
package:com.google.android.youtube
package:com.android.internal.display.cutout.emulation.corner
package:com.google.android.ext.services
package:com.android.internal.display.cutout.emulation.double
package:com.android.providers.telephony
package:com.android.dynsystem
package:com.miui.powerkeeper
package:com.xiaomi.miplay_client
package:com.google.android.googlequicksearchbox
package:com.miui.fm
package:com.xiaomi.calendar
package:com.xiaomi.bluetooth.overlay
package:com.xiaomi.discover
[--------------]

Hapus Aplikasi Bloatware

Namun ada pengecualian untuk beberapa aplikasi yang jika dihapus akan menyebabkan permasalahan seperti boot-loop, dll. Berikut daftarnya: (dikutip dari referensi di bawah)

Apps	Result
com.xiaomi.finddevice	endless boot loop, some time after it will ask to erase the phone and start over. Guess how I learned that?
com.miui.securitycenter	phone reboots only in recovery mode
com.android.contacts	you lose the phone icon
com.mi.android.globalminusscreen	Xiaomi App Vault. if you are logged in with Mi account, device becomes locked, to unlock you should bring it to Xiaomi Services Center. Settings -> Home Screen crashes Settings app.

Tambahan, pengalaman pribadi; jika kita menghapus com.miui.miwallpaper wallpapaer lockscreen tidak akan bisa dirubah.

Hapus aplikasi bloatware sesuai yang diinginkan. DWYOR!

angelica:/ $ pm uninstall --user 0 com.miui.analytics                                  
Success
angelica:/ $ pm uninstall --user 0 com.xiaomi.mipicks                                  
Success
angelica:/ $ pm uninstall --user 0 com.miui.msa.global                                 
Success
angelica:/ $ pm uninstall --user 0 com.miui.videoplayer                                
Success
angelica:/ $ pm uninstall --user 0 com.miui.player                                     
Success
angelica:/ $ pm uninstall --user 0 com.mi.globalbrowser                                
Success
angelica:/ $ pm uninstall -k --user 0 com.android.thememanager
Success
angelica:/ $ pm uninstall -k --user 0 com.miui.miwallpaper
Success
angelica:/ $ pm uninstall -k --user 0 com.xiaomi.midrop
Success
angelica:/ $ pm uninstall -k --user 0 com.xiaomi.calendar

Pindahkan Data Dari Ponsel Ke HDD

Kelemahan ketika melakukan transfer menggunakan MTP (Media Transfer Protocol) adalah performa dari transfer file yang sangat lamban, meskipun sudah menggunakan USB 3.0.

Alternatif lain untuk melakukan pemindahan data dengan performa yang lebih cepat adalah menggunakan ADB pull command.

$ cd /hdd/
$ adb pull /sdcard/ ./sdcard

Akhir

Iklan pada Lock Screen akan hilang, dan experience untuk penggunaan sehari-hari terasa lebih nyaman.

Home Screen	Lock Screen

References

[Q] Faster MTP transfer solution? | XDA Forums

https://forum.xda-developers.com/t/q-faster-mtp-transfer-solution.1954616/

Android: adb pull file on desktop - Stack Overflow

https://stackoverflow.com/questions/17629889/android-adb-pull-file-on-desktop?answertab=trending#tab-top

How to make adb shell ps list all processes in Android O? - Stack Overflow

https://stackoverflow.com/questions/44732749/how-to-make-adb-shell-ps-list-all-processes-in-android-o

Adb useful commands list

https://gist.github.com/Pulimet/5013acf2cd5b28e55036c82c91bd56d8

Removing all unnecessary bloatware from Xiaomi MIUI 11/12 (Android 9/10) without root

https://selivan.github.io/2020/02/25/removing-bloatware-from-xiaomi-miui-android.html

How to free up internal storage on Android, including 'other' storage

https://nerdschalk.com/how-to-free-up-storage-on-android/

[Personal Project] Utas Tentang Laptop Yang Melawan Untuk Rehat

Sun, 19 Jun 2022 11:00:00 GMT

Laptop dengan spesifikasi super, port utilitas yang lengkap, dan bandel sejak dahulu.

Latar belakang

Jarum pendek menunjuk antara angka 2 dan 3. Jarum panjang menunjuk angka 6. Jam menunjuk pukul dua lebih tiga puluh menit, sudah dini hari. Tidak ada angin, tidak ada hujan saya merasa perlu sebuah perangkat yang ready to go, dan nga rewel.

Langsung seketika saya buka marketplace langganan, dan menuliskan "Laptop" dengan filter harga tertinggi 1 juta rupiah. Seketika saya teringat dengan Thinkpad laptop murah dengan daleman yang mundah di-oprek.

Riset

Setelah melihat-lihat marketplace kesayangan, saya mulai jatuh cinta dengan Thinkpad T-60 dimana laptop tersebut masih terpajang logo IBM yang sangat terasa retro. Saya mbatin di dalam hati "mantep iki retroo.. poooll.." belum lagi ditambah keyboard yang diisikan berbagai bentuk kanji bahasa jepang yang pasti saya tidak tahu artinya, dan nga bakal kepake juga.

Dalam beberapa hari setiap kali meimiliki waktu luang. Kerjaan saya hanya ngeliat-in marketplace..., apa ada barang bagus baru hari?. Saya juga menyempatkan untuk bergabung ke dalam group Telegram "Cari Laptop Lenovo Thinkpad". Tidak cukup sampai di situ, bahkan saya menghubungi kolega lama, yang saya ketahui beliau adalah pemilik Laptop Thinkpad segede gaban. Untuk menggali informasi lebih dalam mengenai harga pasaran, dll.

Kemudian saya meluncur ke situs berbagai video "Youtube". Referensi dari berbagai permasalahan hidup. Saya menemukan beberapa video yang saya rasa sangat informatif mengenai Laptop Thinkpad ini, diantarnya :

Best Value ThinkPads in 2020 https://www.youtube.com/watch?v=x-c-_A4AdSw
Which ThinkPad Should You Get? (And Libreboot notes) https://www.youtube.com/watch?v=La3sb5y7e-k

Dari video tersebut saya mendapatkan informasi bahwasannya salah satu laptop yang masih memiliki design warisan dari IBM dan memiliki kapasitas upgradeable yang paling mumpuni adalah Thinkpad X220. Seketika saya campakkan keinginan untuk meminang T-60 yang saya pikir retro dan keren itu. Dikarenakan hal-hal berikut..

Menurut sumber pelbagai wiki yang saya baca salah satunya thinkwiki.com. Laptop ini adalah salah satu laptop dengan hardware specification yang paling tangguh pada zamannya. CPU tertinggi yang dapat dipasang pada papan utama laptop ini adalah Intel® Core™ i7-2640M (2.8GHz, 4MB L3 cache) dengan kapasitas RAM Up to 8GB PC3-10600 DDR3 memory at 1333MHz, bahkan di BIOS versi 1.28 batasan kecepatan RAM yang bisa dicapai ada di 1866MHz. Saya mbatin "Muuuaanteepp ikii.."

Akhir Riset

Dengan hati yang mantap setelah melakukan pelbagai riset. Saatnya saya beraksi. Semua marketplace pun saya jelajahi berharap mendapat harga "musuh"(murah~~).

Di salah satu group Facebook, laptop tipe ini dibandrol dengan harga kisaran 2,5 hingga 3,9 Juta rupiah. Saya pikir "loh kok iseh larangg.." bahkan saya berfikir untuk mencoba International Marketplace Ebay.com namun harga yang ditawarkan masih relatif mahal dikarenakan jika saya membeli barang dari luar negeri, maka saya harus membayar pajak barang impor.

Beberapa hari berlalu, kerjaan saya jika sedang gabut hanya mantengin marketplace (di KRL, sebelum tidur, di Stasiun). Hingga saya jatuh cinta dengan salah satu produk yang terpajangan di marketplace kesayangan. Di situ tertuliskan "Thinkpad X220 - Intel Core i7" dibandrol dengan harga 2.200.000 rupiah. Toko tersebut sudah menjual lebih dari 10 unit laptop dengan tipe tersebut. Kemudian saya mencoba memulai obrolan melalui fitur chat untuk memastikan bahwa penjual laptop tersebut legit.

Setelah mengobrol dengan penjual saya langsung melakukan pembelian dengan metode pengiriman Same Day Delivery untuk menghindari kerusakan apabila menggunakan pengiriman cargo.

Kedatangan

Laptop diantar kurir sekitar pukul 22:00 WIB. Dengan semangat saya membuka pintu dan wajah saya difoto oleh kurir dengan senyum merek sikat gigi.

Ketika unboxing saya dikejutkan dengan isi dari kotak kardus tersebut, karena barang yang saya terima terasa BNIB (Brand New in Packet) tidak terasa bahwa saya membeli laptop bekas berumur lebih dari 10 Tahun. Di dalam kotak tersebut berisi tas, kertas brosur, charger, dan juga plastik barang elektronik.

Kesan Pertama

Kesan paling pertama adalah "keyboardnya enak bettt" berbeda dengan keyboard butterfly dari MacBook Pro 2017 yang biasa saya gunakan sehari-hari ~~rasanya kek batu~~.

Segera saya mencoba semua fungsionaltias dari laptop tersebut, mulai dari wifi hingga semua tombol-tombol keyboard. Semua normal, hanya saja tombol switch wifi card tidak terlalu klik. Tapi saya rasa itu sesuai dengan harga yang saya bayar.

Hardware Upgrade

Sebelum meminang laptop ini, salah satu hal yang saya pertimbangkan adalah upgradeable-nya. Beberapa perangkat keras yang sangat mempengaruhi performa di laptop ini saya rasa mudah dilakukan upgrade.

Perangkat keras yang saya lakukan peningkatan diantaranya adalah RAM, dan SSD, namun hal yang termasuk krusial untuk mendapatkan performa yang lebih baik saya rasa adalah pindah ke OS dengan Kernel Linux.

Upgrade RAM

Saya membeli 2 keping RAM dari marketplace dengan kapasitas 8GB setiap kepingnya.

Upgrade SSD

Untuk SSD saya membeli m-Sata dengan kapasitas 128GB. Nantinya Kernel dan juga semua Home directory dari sistem operasi akan dipasang di dalam m-Sata.

Sedangkan HDD sebesar 350GB bawaan dari laptop hanya akan dimounting menjadi External Partisi untuk menyimpan data yang memang jarang diakses.

Memasang Sistem Operasi

Kemudian saya memilih memasang Sistem Operasi Xubuntu. Sistem operasi yang ringan karena minimnya perangkat lunak bawaan, dan menggunakan Xfce desktop environment.

Total Biaya

Barang	Harga
Laptop	Rp 2.200.000
SSD MSATA	Rp 234.000
RAM DDR3 16GB (8X2)	Rp 593.200
Total	Rp 3.027.200

Akhir

Dengan biaya 3.027.200 rupiah saya bisa mendapatkan performa laptop Core i7-2640M di generasi ke-2 (Max Supported X220) disandingkan dengan RAM 16 GB, juga didukung SSD sebagai operasional sistem operasi, belum lagi HDD 350GB. Saya rasa performa laptop ini jauh melampaui performa MacBook yang saya gunakan sehari hari.

Monitor ZFS for Better Sleep

Mon, 16 May 2022 06:44:58 GMT

Monitoring is the key to happy consumer.

Monitoring with Grafana and Prometheus

ZFS Exporter Setup:

# Install zfs_exporter for Prometheus metrics
cd /opt
git clone https://github.com/pdf/zfs_exporter.git
cd zfs_exporter
go build # to get most latest metrics, or just simply wget

# Create systemd service
cat > /etc/systemd/system/zfs-exporter.service << 'EOF'
[Unit]
Description=ZFS Prometheus Exporter
After=network.target

[Service]
Type=simple
User=root
ExecStart=/opt/zfs_exporter/zfs_exporter --web.listen-address=:9254
Restart=always

[Install]
WantedBy=multi-user.target
EOF

systemctl enable --now zfs-exporter

Prometheus Configuration:

# Add to prometheus.yml
- job_name: 'zfs'
  static_configs:
    - targets: ['localhost:9254']
  scrape_interval: 30s

Critical Grafana Dashboard Panels:

# Pool Health Status
zfs_pool_state{pool="$pool"} != 0

# Pool Capacity Utilization (Alert at 80%)
(zfs_pool_allocated_bytes{pool="$pool"} / zfs_pool_size_bytes{pool="$pool"}) * 100

# ARC Hit Ratio (Alert below 85%)
rate(zfs_arc_hits_total[5m]) / (rate(zfs_arc_hits_total[5m]) + rate(zfs_arc_misses_total[5m])) * 100

# Checksum Errors (Alert on any)
increase(zfs_pool_checksum_errors_total{pool="$pool"}[5m])

# Fragmentation Percentage
zfs_pool_fragmentation_percent{pool="$pool"}

# Read/Write IOPS
rate(zfs_pool_read_ops_total{pool="$pool"}[5m])
rate(zfs_pool_write_ops_total{pool="$pool"}[5m])

# Read/Write Latency
zfs_pool_read_latency_seconds{pool="$pool"}
zfs_pool_write_latency_seconds{pool="$pool"}

Alerting Rules (alerts.yml):

groups:
- name: zfs-alerts
  rules:
  - alert: ZFSPoolDegraded
    expr: zfs_pool_state != 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "ZFS pool {{ $labels.pool }} is degraded"
      description: "Pool state: {{ $value }}"

  - alert: ZFSPoolCapacityHigh
    expr: (zfs_pool_allocated_bytes / zfs_pool_size_bytes) * 100 > 80
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "ZFS pool {{ $labels.pool }} capacity is {{ $value }}%"

  - alert: ZFSARCHitRateLow
    expr: (rate(zfs_arc_hits_total[5m]) / (rate(zfs_arc_hits_total[5m]) + rate(zfs_arc_misses_total[5m]))) * 100 < 85
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "ZFS ARC hit ratio is {{ $value }}%"

  - alert: ZFSChecksumErrors
    expr: increase(zfs_pool_checksum_errors_total[5m]) > 0
    for: 0m
    labels:
      severity: critical
    annotations:
      summary: "ZFS checksum errors detected on pool {{ $labels.pool }}"

  - alert: ZFSFragmentationHigh
    expr: zfs_pool_fragmentation_percent > 50
    for: 30m
    labels:
      severity: warning
    annotations:
      summary: "ZFS pool {{ $labels.pool }} fragmentation is {{ $value }}%"

Reference Dashboards:

Smashing ZFS Storage

Mon, 11 Apr 2022 06:44:58 GMT

ZFS is reliable, and rock solid. Here's why, and how to test smashh it!

1. Understanding ZFS Architecture

1.1. Core Components

ZFS operates through three main layers:

SPA (Storage Pool Allocator): Manages physical storage and presents unified address space.
DMU (Data Management Unit): Handles transactions and ensures ACID properties.
ARC (Adaptive Replacement Cache): Smart caching that adapts to workload patterns.

1.2. ARC Operation

The ARC uses four lists for intelligent caching:

T1: Recently used blocks (accessed once)
T2: Frequently used blocks (accessed multiple times)
B1/B2: Ghost lists tracking evicted entries

Check your current ARC status:

# View ARC statistics
cat /proc/spl/kstat/zfs/arcstats

# Check ARC size limits
echo "ARC max: $(cat /sys/module/zfs/parameters/zfs_arc_max)"
echo "ARC min: $(cat /sys/module/zfs/parameters/zfs_arc_min)"

# arc stat command (monitor L2arc)
arcstat -f read,hits,miss,hit%,l2read,l2hits,l2miss,l2hit%,arcsz,l2size 5

# very detail 
arc_summary

1.3. L2ARC vs ARC

L2ARC extends ARC to SSDs but operates as a simple FIFO ring buffer, not adaptive replacement. It caches blocks that are about to be evicted from ARC.

# Monitor L2ARC if configured
zpool iostat -v 1

2. ZFS Data Ingestion for Proxmox/KVM

2.1. VM Storage Options

For Proxmox/KVM workloads, you have two main options:

ZVOLs (Block devices) - Manual Creation:

# Create zvol for VM disk - these WON'T appear in Proxmox UI
# Using default volblocksize (16K since OpenZFS v2.2)
zfs create -V 50G tank/vm-101-disk-0

# For specific workloads, adjust volblocksize as needed
zfs create -V 50G -o volblocksize=16K tank/vm-102-disk-0  # Database VMs

# Manual ZVOLs appear as block devices
ls -la /dev/zvol/tank/vm-*
# /dev/zvol/tank/vm-101-disk-0
# /dev/zvol/tank/vm-102-disk-0

Important: Manual ZVOLs created via command line won't show in Proxmox GUI storage management. You must manage them entirely via CLI or use Proxmox's ZFS storage integration.

Proxmox-managed ZFS storage:

# Add ZFS storage in Proxmox (this goes in /etc/pve/storage.cfg)
# Proxmox will manage ZVOLs automatically through the GUI
pvesm add zfspool local-zfs -pool tank -content images,rootdir

Raw disk images on datasets:

# Create dataset for VM images - these appear in Proxmox GUI
zfs create tank/vmimages
pvesm add dir vmstore -path /tank/vmimages -content images,iso,vztmpl
# VMs stored as files: /tank/vmimages/101/vm-101-disk-0.raw

2.4. Managing Manual ZVOLs with VMs

When you create ZVOLs manually, you need to configure VMs to use them:

# Create ZVOL for VM
zfs create -V 50G -o volblocksize=16K tank/vm-103-disk-0

# Add to VM configuration manually
echo "scsi0: /dev/zvol/tank/vm-103-disk-0,cache=none,discard=on" >> /etc/pve/qemu-server/103.conf

# Or use qm command
qm set 103 --scsi0 /dev/zvol/tank/vm-103-disk-0,cache=none,discard=on

# ZVOL won't appear in Proxmox GUI storage management
# Manage snapshots, clones, etc. via ZFS commands:
zfs snapshot tank/vm-103-disk-0@backup-$(date +%Y%m%d)
zfs clone tank/vm-103-disk-0@backup-20241201 tank/vm-104-disk-0

Recent testing shows raw disk images often outperform ZVOLs by 6x for small random I/O:

# Optimize dataset for VM images
zfs set recordsize=64K tank/vmimages
zfs set compression=lz4 tank/vmimages
zfs set xattr=sa tank/vmimages
zfs set logbias=throughput tank/vmimages

# For zvols, optimize at creation
zfs create -V 50G -o volblocksize=8K -o compression=lz4 tank/vm-disk

2.3. Data Flow in VM Workloads

VM I/O flows: Guest OS → QEMU → Host Kernel → ZFS

Key considerations:

Async writes: Buffered in ARC, flushed every 5 seconds via TXG
Sync writes: Must hit stable storage before acknowledgment
Cache modes: Use cache=none in QEMU to let ZFS handle all caching

# Monitor TXG sync times
watch 'zpool iostat -v 1 1'

# Check for sync write pressure
zpool iostat -v 1

3. Creating Production Test Pools

3.1. Production Disk Pool Creation

WARNING: This will destroy all data on specified disks. Only use spare/test disks.

# List available disks
lsblk
fdisk -l

# Create various topologies for testing with real disks
# ENSURE these are spare disks - this destroys all data!
zpool create testmirror mirror /dev/sdc /dev/sdd
zpool create testraidz1 raidz /dev/sde /dev/sdf /dev/sdg  
zpool create testraidz2 raidz2 /dev/sdh /dev/sdi /dev/sdj /dev/sdk

# Verify pool creation
zpool status
zpool list

3.2. Add Cache and Log Devices

# Add NVMe SSD as cache device (L2ARC)
zpool add testmirror cache /dev/nvme0n1

# Add smaller SSD as log device (SLOG) - only for sync-heavy workloads
# SLOG should be mirrored in production
zpool add testmirror log mirror /dev/sdb1 /dev/sdb2

# Verify topology
zpool status testmirror

4. Simulating ZFS Failures

4.1. Simulating Disk Failures

Like Ceph, there are hard and soft failure scenarios. ZFS handles both gracefully with proper redundancy.

Prerequisites:

Redundant ZFS pool (mirror or raidz) with spare disks
Root access to the system
Test data with known checksums
CRITICAL: Ensure you're using test/spare disks only

4.2. Soft Failure Simulation (Clean Offline)

Procedure:

Take real device offline cleanly:

# Offline physical device in mirror
zpool offline testmirror /dev/sdc

# Check degraded status  
zpool status testmirror

Expected output:

  pool: testmirror
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
config:
        NAME         STATE     READ WRITE CKSUM
        testmirror   DEGRADED     0     0     0
          mirror-0   DEGRADED     0     0     0
            /dev/sdc OFFLINE      0     0     0
            /dev/sdd ONLINE       0     0     0

Bring device back online:

zpool online testmirror /dev/sdc
zpool status testmirror  # Should show resilvering

4.3. Hard Failure Simulation (Physical Removal)

Procedure:

Create test VM with active workload:

# Create ZVOL for test VM (using default 16K volblocksize)
zfs create -V 20G testmirror/vm-999-disk-0

# Create test VM configuration
qm create 999 --name zfs-test-vm --memory 2048 --cores 2 --net0 virtio,bridge=vmbr0
qm set 999 --scsi0 /dev/zvol/testmirror/vm-999-disk-0,cache=none,discard=on
qm set 999 --cdrom local:iso/ubuntu-22.04-live-server-amd64.iso
qm set 999 --boot order=scsi0

# Start VM and install/configure it with test workload
qm start 999

# Once VM is running, create continuous I/O workload inside VM
# SSH into VM and run:
# nohup dd if=/dev/zero of=/home/test-large-file bs=1M count=1000 &
# nohup find / -type f -exec wc -l {} \; > /dev/null 2>&1 &

# Monitor VM status and ZFS pool
qm status 999  # Should show 'running'
zpool iostat testmirror 1 5  # Observe I/O activity

Simulate physical disk failure during VM operation:

# Remove device from system while VM is running (simulates hardware failure)
echo 1 > /sys/block/sdc/device/delete

# Check pool status - should detect failure but continue operation
zpool status testmirror

# Verify VM continues running without interruption
qm status 999  # Should still show 'running'

# Check VM console for any I/O errors (there should be none)
qm terminal 999

Monitor detection and recovery:

# Watch pool operating in degraded mode while VM continues
watch 'zpool status testmirror'

# Expected output shows DEGRADED state but ONLINE operation:
#   pool: testmirror
#  state: DEGRADED
# config:
#         NAME         STATE     READ WRITE CKSUM
#         testmirror   DEGRADED     0     0     0
#           mirror-0   DEGRADED     0     0     0
#             /dev/sdc UNAVAIL      0     0     0  cannot open
#             /dev/sdd ONLINE       0     0     0

# Verify VM performance during degraded operation
qm monitor 999
# (qm) info block
# Check for I/O statistics and verify no errors

# Test VM responsiveness
ping VM_IP  # Should respond normally
ssh user@VM_IP "uptime"  # Should show continued operation

# Monitor VM I/O during degraded state
zpool iostat -v testmirror 1 10  # Watch I/O patterns

Expected Behavior:

VM continues running without interruption
No I/O errors in VM guest OS
Pool operates in DEGRADED state but remains functional
Performance may be slightly reduced but still acceptable
ZFS transparently handles the failure without VM awareness

4.4. Disk Replacement Procedure

Procedure:

Replace failed disk while VM continues operation:

# After physical disk replacement, rescan SCSI bus
echo "- - -" > /sys/class/scsi_host/host0/scan

# Replace failed device with new one
zpool replace testmirror /dev/sdc /dev/sdl

# Monitor resilver progress while VM remains active
watch 'zpool status -v testmirror'

# Expected output during resilver:
#   pool: testmirror
#  state: DEGRADED
# status: One or more devices is currently being resilvered.
#  scan: resilver in progress since [timestamp]
#        X.XXG scanned at X.XXG/s, X.XXG issued at X.XXG/s, X.XXG total
#        X.X% done, X.XXh to go

# Verify VM performance during resilver
qm status 999  # Should remain 'running'

# Test VM I/O performance during resilver (expect some impact)
ssh user@VM_IP "dd if=/dev/zero of=/tmp/test bs=1M count=100 oflag=direct"

# Monitor resilver completion
while zpool status testmirror | grep -q "resilver"; do 
    echo "Resilver progress: $(zpool status testmirror | grep resilver)"
    echo "VM status: $(qm status 999)"
    sleep 60
done

echo "Resilver complete - verifying VM integrity"
qm status 999  # Should show 'running'
zpool status testmirror  # Should show ONLINE

# Clean up test VM after successful test
qm stop 999
qm destroy 999
zfs destroy testmirror/vm-999-disk-0

Production Considerations:

VMs experience minimal impact during single disk failure
Resilver operations may cause 10-30% performance degradation
Monitor VM response times during resilver operations
Schedule resilvers during maintenance windows for critical systems

4.6. L2ARC Cache Device Failure Simulation

Procedure: Test L2ARC device failure scenarios - cache failures should not affect pool integrity:

# Prerequisites: Pool with L2ARC device configured
zpool status testmirror  # Verify cache device present

# Create workload to populate L2ARC
echo "Populating L2ARC cache..."
for i in {1..50}; do
    dd if=/dev/urandom of=/testmirror/testfs/cachefile$i bs=1M count=20
done

# Verify L2ARC is populated
zpool iostat -v testmirror 1 5
cat /proc/spl/kstat/zfs/arcstats | grep l2_

# Test read performance with L2ARC active
echo "Testing read performance with L2ARC..."
fio --direct=1 --iodepth=32 --rw=randread --ioengine=libaio --bs=64k \
    --size=500M --numjobs=2 --runtime=60 --group_reporting \
    --filename=/testmirror/testfs/l2arc-test --name=L2ARC_Active_Test

# Simulate L2ARC device failure
echo "Simulating L2ARC device failure..."
if [[ -b /dev/nvme0n1 ]]; then
    # Physical device failure simulation
    echo 1 > /sys/block/nvme0n1/device/delete
else
    # File-based cache failure simulation
    rm /mnt/zfs-test/cache.img 2>/dev/null || true
fi

# Check pool status - should remain ONLINE with cache device UNAVAIL
zpool status testmirror

# Expected output shows:
# pool: testmirror
# state: ONLINE
# config:
#         NAME                      STATE     READ WRITE CKSUM
#         testmirror                ONLINE       0     0     0
#           mirror-0                ONLINE       0     0     0
#             /dev/sdc              ONLINE       0     0     0
#             /dev/sdd              ONLINE       0     0     0
#         cache
#           /dev/nvme0n1            UNAVAIL      0     0     0

# Test read performance without L2ARC
echo "Testing read performance without L2ARC..."
fio --direct=1 --iodepth=32 --rw=randread --ioengine=libaio --bs=64k \
    --size=500M --numjobs=2 --runtime=60 --group_reporting \
    --filename=/testmirror/testfs/l2arc-test --name=L2ARC_Failed_Test

# Remove failed cache device from pool
zpool remove testmirror /dev/nvme0n1

# Verify pool operates normally without cache
zpool status testmirror  # Should show no cache devices
zpool iostat -v testmirror

# Add replacement cache device if available
if [[ -b /dev/nvme1n1 ]]; then
    zpool add testmirror cache /dev/nvme1n1
    echo "Replacement cache device added"
fi

echo "L2ARC failure test complete - pool integrity maintained"

Expected Behavior:

Pool remains ONLINE despite cache device failure
No data loss occurs
Read performance may degrade temporarily
Cache device can be safely removed and replaced
Critical: L2ARC failures never affect pool integrity

4.7. Multiple Device Failures

Procedure: Test RAIDZ resilience with real disks:

# RAIDZ1 can survive 1 disk failure
echo 1 > /sys/block/sde/device/delete  # Pool remains online

# Simulate second failure - pool becomes unavailable
echo 1 > /sys/block/sdf/device/delete
zpool status testraidz1  # Will show FAULTED state

Procedure: Test RAIDZ resilience with real disks:

# RAIDZ1 can survive 1 disk failure
echo 1 > /sys/block/sde/device/delete  # Pool remains online

# Simulate second failure - pool becomes unavailable
echo 1 > /sys/block/sdf/device/delete
zpool status testraidz1  # Will show FAULTED state

5. Production Monitoring

5.1. Critical Metrics to Monitor

# Pool health check
zpool status -v

# Capacity monitoring (alert at 80%)
zpool list

# I/O latency tracking  
zpool iostat -v 1

# ARC efficiency
cat /proc/spl/kstat/zfs/arcstats | grep -E "hits|misses|hit_rate"

5.2. Automated Monitoring Script

#!/bin/bash
# /usr/local/bin/zfs-monitor.sh

# Check pool status
for pool in $(zpool list -H -o name); do
    status=$(zpool list -H -o health $pool)
    if [ "$status" != "ONLINE" ]; then
        echo "ALERT: Pool $pool status: $status" | mail -s "ZFS Alert" admin@company.com
    fi
    
    # Check capacity
    cap=$(zpool list -H -o capacity $pool | tr -d '%')
    if [ $cap -gt 80 ]; then
        echo "WARNING: Pool $pool at ${cap}% capacity" | mail -s "ZFS Capacity" admin@company.com
    fi
done

5.3. ZED Configuration

Only if your mail command can send email to the world, just configure the email address in the zed.rc file as needed, and you should start getting email.

Configure ZFS Event Daemon for automated responses:

# Edit ZED configuration
vim /etc/zfs/zed.d/zed.rc

# Key settings:
ZED_EMAIL_ADDR="admin@company.com"
ZED_EMAIL_PROG="mail"
ZED_NOTIFY_VERBOSE=1
ZED_SPARE_ON_IO_ERRORS=1
ZED_SPARE_ON_CHECKSUM_ERRORS=10

# Enable and start ZED
systemctl enable zfs-zed
systemctl start zfs-zed

Configure ZFS Event Daemon for automated responses:

# Edit ZED configuration
vim /etc/zfs/zed.d/zed.rc

# Key settings:
ZED_EMAIL_ADDR="admin@company.com"
ZED_EMAIL_PROG="mail"
ZED_NOTIFY_VERBOSE=1
ZED_SPARE_ON_IO_ERRORS=1
ZED_SPARE_ON_CHECKSUM_ERRORS=10

# Enable and start ZED
systemctl enable zfs-zed
systemctl start zfs-zed

6. Performance Tuning for VM Workloads

6.1. ARC Sizing for Virtualization Hosts

# Conservative: 25% of RAM (adjust based on VM memory requirements)
echo $(($(grep MemTotal /proc/meminfo | awk '{print $2}') * 1024 * 25 / 100)) > /sys/module/zfs/parameters/zfs_arc_max

# Check current ARC usage
grep "size" /proc/spl/kstat/zfs/arcstats

6.2. Recordsize Optimization

# For VM datasets (guest filesystem dependent)
zfs set recordsize=16K tank/databases     # Database and general VMs
zfs set recordsize=1M tank/backups        # Large sequential files

# Monitor record size effectiveness
zfs get recordsize,written,referenced tank/vmimages

7. Recovery Verification

7.1. VM-Based Data Integrity Testing

#!/bin/bash
# Create multiple test VMs for comprehensive failure testing

# Function to create test VM with known data
create_test_vm() {
    local vmid=$1
    local pool=$2
    
    # Create ZVOL for VM (using default 16K volblocksize)
    zfs create -V 10G ${pool}/vm-${vmid}-disk-0
    
    # Create VM
    qm create $vmid --name "zfs-test-vm-$vmid" --memory 1024 --cores 1 \
        --net0 virtio,bridge=vmbr0 --onboot 0
    qm set $vmid --scsi0 /dev/zvol/${pool}/vm-${vmid}-disk-0,cache=none,discard=on
    qm set $vmid --cdrom local:iso/ubuntu-22.04-live-server-amd64.iso
    
    echo "Created test VM $vmid on pool $pool"
}

# Create test VMs before failure scenarios
create_test_vm 990 testmirror
create_test_vm 991 testmirror
create_test_vm 992 testmirror

# Start VMs and create unique test data in each
for vmid in 990 991 992; do
    qm start $vmid
    
    # Wait for VM to boot and create test data
    sleep 30
    
    # SSH into each VM and create unique test files with known checksums
    ssh user@VM_${vmid}_IP "
        echo 'Creating test data in VM $vmid'
        dd if=/dev/urandom of=/home/testfile-$vmid bs=1M count=100
        md5sum /home/testfile-$vmid > /home/checksum-$vmid.txt
        sync
    "
done

# After failure scenarios, verify VM data integrity
verify_vm_integrity() {
    local vmid=$1
    
    echo "Verifying integrity of VM $vmid"
    qm status $vmid
    
    if [[ $(qm status $vmid) == *"running"* ]]; then
        ssh user@VM_${vmid}_IP "
            echo 'Verifying checksum in VM $vmid'
            md5sum -c /home/checksum-$vmid.txt
            if [ \$? -eq 0 ]; then
                echo 'VM $vmid: Data integrity PASSED'
            else
                echo 'VM $vmid: Data integrity FAILED'
            fi
        "
    else
        echo "VM $vmid is not running - attempting to start"
        qm start $vmid
        sleep 30
        ssh user@VM_${vmid}_IP "md5sum -c /home/checksum-$vmid.txt"
    fi
}

# Verify all VMs after recovery
echo "=== VM Data Integrity Verification ==="
for vmid in 990 991 992; do
    verify_vm_integrity $vmid
done

# Cleanup test VMs
cleanup_test_vms() {
    for vmid in 990 991 992; do
        qm stop $vmid 2>/dev/null
        qm destroy $vmid
        zfs destroy testmirror/vm-${vmid}-disk-0
    done
}

# Uncomment to cleanup after testing
# cleanup_test_vms

7.2. Performance Impact Assessment - VM Workload (Cheat Sheet)

# Cheat Sheet - ZFS Performance 
# This mimics real VM I/O patterns with 4K blocks and high queue depths

# Login to the VM
TESTDIR="/testmirror"
mkdir ${TESTDIR}

# Baseline tests before failure
echo "=== BASELINE PERFORMANCE TESTS ==="

# Random write test (VM boot, application startup)
fio --direct=1 --iodepth=128 --rw=randwrite --ioengine=libaio --bs=4k \
    --size=1G --numjobs=1 --runtime=300 --group_reporting \
    --filename=${TESTDIR}/iotest-randwrite --name=Baseline_Random_Write
rm ${TESTDIR}/iotest-*; sync

# Random read test (database queries, file access)
fio --direct=1 --iodepth=128 --rw=randread --ioengine=libaio --bs=4k \
    --size=1G --numjobs=1 --runtime=300 --group_reporting \
    --filename=${TESTDIR}/iotest-randread --name=Baseline_Random_Read
rm ${TESTDIR}/iotest-*; sync

# Mixed random I/O (typical VM workload - 70% read, 30% write)
fio --direct=1 --iodepth=128 --rw=randrw --rwmixread=70 --ioengine=libaio \
    --bs=4k --size=1G --numjobs=1 --runtime=300 --group_reporting \
    --filename=${TESTDIR}/iotest-randmix --name=Baseline_Mixed_Random
rm ${TESTDIR}/iotest-*; sync

# Sequential write test (large file transfers, backups)
fio --direct=1 --iodepth=128 --rw=write --ioengine=libaio --bs=4k \
    --size=1G --numjobs=1 --runtime=300 --group_reporting \
    --filename=${TESTDIR}/iotest-seqwrite --name=Baseline_Sequential_Write
rm ${TESTDIR}/iotest-*; sync

# Sequential read test (streaming, large file access)
fio --direct=1 --iodepth=128 --rw=read --ioengine=libaio --bs=4k \
    --size=1G --numjobs=1 --runtime=300 --group_reporting \
    --filename=${TESTDIR}/iotest-seqread --name=Baseline_Sequential_Read
rm ${TESTDIR}/iotest-*; sync

# Test during degraded operation (one disk offline)
echo "=== DEGRADED POOL PERFORMANCE TESTS ==="
zpool offline testmirror /dev/sdc

# Critical test: Random write performance during degraded operation
fio --direct=1 --iodepth=128 --rw=randwrite --ioengine=libaio --bs=4k \
    --size=1G --numjobs=1 --runtime=300 --group_reporting \
    --filename=${TESTDIR}/iotest-degraded-randwrite --name=Degraded_Random_Write
rm ${TESTDIR}/iotest-*; sync

# Mixed workload during degraded state
fio --direct=1 --iodepth=128 --rw=randrw --rwmixread=70 --ioengine=libaio \
    --bs=4k --size=1G --numjobs=1 --runtime=300 --group_reporting \
    --filename=${TESTDIR}/iotest-degraded-mixed --name=Degraded_Mixed_Random
rm ${TESTDIR}/iotest-*; sync

# Bring pool back online and test during resilver
echo "=== RESILVER PERFORMANCE TESTS ==="
zpool online testmirror /dev/sdc

# Test performance during resilver (most critical for production)
fio --direct=1 --iodepth=64 --rw=randrw --rwmixread=70 --ioengine=libaio \
    --bs=4k --size=512M --numjobs=1 --runtime=120 --group_reporting \
    --filename=${TESTDIR}/iotest-resilver-mixed --name=Resilver_Mixed_Random &

# Monitor resilver progress
while zpool status testmirror | grep -q "resilver"; do 
    echo "Resilver in progress: $(zpool status testmirror | grep resilver)"
    sleep 30
done
wait  # Wait for fio to complete

rm ${TESTDIR}/iotest-*; sync

# Final performance test after recovery
echo "=== POST-RECOVERY PERFORMANCE TESTS ==="
fio --direct=1 --iodepth=128 --rw=randrw --rwmixread=70 --ioengine=libaio \
    --bs=4k --size=1G --numjobs=1 --runtime=300 --group_reporting \
    --filename=${TESTDIR}/iotest-recovered-mixed --name=Recovered_Mixed_Random
rm ${TESTDIR}/iotest-*; sync

8. Production Notes

8.1. Disk Identification

CRITICAL: Always verify disk identification before testing:

# Use multiple methods to identify disks
lsblk -o NAME,SIZE,MODEL,SERIAL
smartctl -i /dev/sdc
ls -la /dev/disk/by-id/ | grep sdc

# Check if disk contains important data
blkid /dev/sdc
mount | grep sdc
lsof | grep sdc

8.2. Safe Testing Environment

# Create isolated test environment
# Use dedicated test disks - never production storage
# Label test pools clearly
zpool create -o comment="TEST POOL - SAFE TO DESTROY" testmirror mirror /dev/sdc /dev/sdd

This hands-on approach lets you thoroughly test ZFS failure scenarios using realistic Proxmox VM workloads without risking production data. The key is understanding that ZFS self-healing only works with proper redundancy - never test these scenarios on single-device pools in production.

Key of good sleep:

VMs continue running transparently during single disk failures
Resilver operations cause measurable but acceptable performance impact
Data integrity is automatically maintained without VM awareness
L2ARC failures have no impact on VM operation or data safety

Hard Notes: ZFS can protect against drive failures but not against operator errors.

Using Strace to Debug Your Stuck Commands

Sun, 23 Jan 2022 18:04:00 GMT

Strace is a useful diagnostic, instructional, and debugging tool. System administrators, diagnosticians and trouble-shooters will find it invaluable for solving problems with programs for which the source is not readily available since they do not need to be recompiled in order to trace them.

Dalam tulisan saya kali ini, saya akan menceritakan sebuah "mejik~~" tool yang sudah menyelamatkan Production cluster berisikan lebih dari 1000 VMs

TL;DR

Di sebuah pagi ketika saya baru saja sampai di kantor untuk on-site. Saya melakukan pengecekan terhadap CEPH cluster (dikarenakan sebelumnya sedang proses recovery slow ops osd) dengan menggunakan perintah :

ubuntu@xxxxxxxxxxxx ~$ ceph -s
  cluster:
    id:     xxxxx-xx-xxx-xxxx-xxxxxxx
    health: HEALTH_WARN
            500011 slow ops, oldest one blocked for 104335 sec, [mon.xxxxx-01,mon.xxxxx-02,mon.xxxxx-03] has slow ops

Saya melihat adanya slow ops hanya ada pada ceph-mon, "berarti slow ops osd sebelumnya, sudah resolved" pikir saya.

Kemudian saya memeriksa systemd di setiap host ceph monitor, dan menemukan bahwa systemd ceph-mon pada host xxxxx-02 inactive (Dead).

ubuntu@xxxxx-02:~$ systemctl status ceph-mon@xxxxx-02
● ceph-mon@xxxxx-02.service - Ceph cluster monitor daemon
   Loaded: loaded (/lib/systemd/system/ceph-mon@.service; disabled; vendor preset: enabled)
   Active: inactive (dead) since Mon 2022-01-24 02:32:16 WIB; 15 hours ago
 Main PID: 875486 (ceph-mon)
    Tasks: 27
   CGroup: /system.slice/system-ceph\x2dmon.slice/ceph-mon@xxxxx-02.service
           └─875486 /usr/bin/ceph-mon -f --cluster ceph --id xxxxx-02 --setuser ceph --setgroup ceph

Namun ketika dilakukan pengecekan menggunakan ps aux | grep ceph-mon binary /usr/bin/ceph-mon running. Dapat disimpulkan bahwa ceph-mon pada host xxxxx-02 running, namun tidak di bawah pantauan systemd.

Sehingga saya putuskan untuk kill pid ceph-mon yang running tersebut

sudo kill -9 {pid-ceph-mon}

Seketia saya tekan tombol ENTER, "Klakk...." tiba-tiba command ceph tidak bisa dieksekusi alias stuck.

Dalam hati saya berkata "Oh mungkin leader mon-nya pindah tungguin aja... Sambil saya meng-aktifkan systemd ceph-mon pada host xxxxx-02"

Setelah menunggu 10-30 menit, command ceph masih tidak bisa dijalankan alias masih stuck.

Dalam hati saya berkata "Wadoohh, napa ni yakkk!!"

Troubleshoot

1. Check Systemd Status

Langkah pertama yang saya lakukan adalah melakukan pengecekan pada systemd ceph-mon di semua Host.

for i in xxxxx-01 xxxxx-02 xxxxx-03 ; do ssh "systemctl status ceph-mon@$i" ; done

Hasilnya semua running well.

2. Check Leader via admin socket ceph-mon

Pada dasarnya ceph perlu memiliki 2 ceph-mon yang qorum agar bisa berjalan normal.

root@xxxxx-03:/home/ubuntu# ceph--admin-daemon/var/run/ceph/ceph-mon.xxxxx-03.asok mon_status
{
    "name": "xxxxx-03".
    "rank": 1,
    "state": "leader"
    "election_epoch": 15370,
    "quorum": [
        1,
        2
    ],
}

Dapat terlihat bahwa terdapat 2 ceph-monitor yang qorum, sehingga secara teori, ceph seharusnya berjalan sebagaimana mustinya.

Mengumpat di dalam hati "█████ kenapa kagak bisa dahhh..."

3. Ceph command dengan Sepesifik Monitor

Salah satu kolega saya Aji Arya menyarankan untuk melakukan percobaan menjalalankan command dengan argumen -m yang akan membuat ceph menembak ke sepesifik monitor.

ceph -s -m 172.x.238.56

172.x.238.56 merupakan IP segment ceph monitor milik xxxxx-03, seharusnya jika memang monitor tersebut sebagai leader, ceph command bisa dijalankan. Namun hasilnya teteap stuck.

4. Despreate

Setelah 8 jam berdiskusi dengan para kolega, dan lain-lain. Saya merehatkan diri "toh VM exsisting tidak ada yang ter-impact saat itu" dan saya masih berfikiran bahwa itu hanya kasus di mana storedb ceph-mon yang tinggi sedang compacting istilah kasarnya "lagi ngejar (nge-sync) ituu, storedb ceph-mon nyaa.."

"Troubleshoot dilanjutkan oleh para kolega lain"

Hingga saya bagun di pagi hari, dan melanjutkan hidup, dikarenakan hari itu adalah hari sabtu., dan saya adalah manusia normal yang selayaknya perlu malam mingguan.

"Namun semesta berkata lain, malam minggu saya harus berada di depan laptop..."

5. Strace & Debug mode

"Saya join troubleshoot lagi karena ~~disuruh~~ penasaran, bersama CEO saya."

Se-dari awal, saya sudah mencoba menjalalankan command ceph -s --debug-mode=99 yang mana artinya menggunakan mode debug paling "nge-poooll.." namum tidak muncul apa-apa.

"Kalo mode debug aja nga muncul apa-apa gimana mau debug dah..."

Hingga saya menemukan kasus yang sama di Mailing Lists - Ceph, dan dalam kasus tersebut beliau menyarankan melakukan debug menggunakan strace.

"Meskipun di hari sebelumnya saya sudah mencoba menggunakan tools tersebut, namun tidak saya perhatikan dengan teliti, karena output-nya yang musing-innn.."

Kemudian saya memikiran skenario apa yang harus dicoba, dan saya mendapatkan sebuah ide.

$ sudo strace -f ceph -s --debug-ms=99 -m 172.x.238.56
[....]
pid 3052239] connect(12, {sa_family=AF_INET sin_port=ntons(3300),sin_addr-inet_addr("172.xx.238.56")}, 16 sunfinished
pid 3052240] <... fentl resumed» ) = 0x2 (flags O_RDAR)
pid 3052239] <... connect resumed ) -1 ECONNREFUSED (Connection refused)
pid 3052240] fcntl(13, F_SETFL, O_RDWRIO_NONBLOCK) = 0
pid 3052239] futex(0x7fcb9805d8e4, FUTEX_MAKE_PRIVATE, 2147483647 «unfinished
pid 3052240] setsockopt(13, SOL_TCP, TCP NODELAY, [1], 4 «unfinished
pid 3052241] <.... futex resumed)=0
pid 3052240] <... setsockopt resumed> ) = 0
pid 3052241] futex(0x7fcb9805d890, FUTEX_WAIT_PRIVATE, 2, NULL ‹unfinished ...>
pid 3052240] connect(13, {sa_family=AF_INET, sin_port-htons(6789), sin_addr-inet_addr("172.xx.238.56")}, 16 sunfinished
pid 3052239] <... futex resumed ) = 1
pid 3052239] futex(0x7fcb9805d890, FUTEX_WAKE_PRIVATE, 1 «unfinished ...>
pid 3052240] <..connect resumed)= -1 EINPROGRESS (Operation now in progress)
pid 3052241] <... futex resumed )
pid 3052240] poll_ctl(9, EPOLL_CTL_ADD, 13, {EPOLLINIEPOLLET, {u32-13, 464=13}} cunfinished
pid 3052241] futex(0x7fcb9805d890, FUTEX_WAKE_PRIVATE, 1 unfinished .
pid 3052240] <..• epol1_ctl resumedo ) = 0
pid 3052241] <... futex resumed )= 0
pid 3052240] connect (13, [sa_family-AF_INET,sin_port-htons(6789),sin_addrainet_addr("172.xx.238.56")}, 16 unfinished
pid 3052241] futex(0x7fcb9805d8e0, FUTEX_WAIT_PRIVATE, 0, NULL cunfinished
pid 3052240] <.connect resumed)= -1 ECONNREFUSED (Connection refused)
pid 30522391 <..futex resumed )
[....]

Dan saya notice sesuatu yang janggal, "Lohh... kok refused", buru-buru saya check apakah semua ceph-mon hidup, dan running pada port tersebut.

# Check LISTEN Port ceph-mon
for i in xxxxx-01 xxxxx-02 xxxxx-03; do echo $i;ssh $i "sudo lsof -i -P -n | grep LISTEN|grep 6789";done
for i in xxxxx-01 xxxxx-02 xxxxx-03; do echo $i;ssh $i "sudo lsof -i -P -n | grep LISTEN|grep 3300";done

Hasil pengecekan menunjukkan bahwa ternyata ceph-mon running bukan di segment IP yang sebagaimana mustinya.

Posisi Binding Port Ceph Mon
- xxxxx-01  =>  172.x.251.54
- xxxxx-02  =>  172.x.238.55
- xxxxx-03  =>  172.x.251.56

Setelah itu saya seperti mendapat pencerahan, "owalahhh...." dan saya coba ubah skenarionya.

$ sudo strace -f ceph -s --debug-ms=99 -m 172.x.251.56
[...]
futex(Oxa745cc, FUTEX_WAKE_PRIVATE, 1) = 1
futex(Oxa74560, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x218db20, FUTEX_WAIT_BITSET_PRIVATE I FUTEX_CLOCK_REALTIME, O, NULL, Oxffffffff) = 0
futexCoxa745c8, FUTEX_WAIT_BITSET_PRIVATE I FUTEX_CLOCK_REALTIME, 0, {tv_sec=1642852349, tv_nsec=778854000}, Oxffffffff) = -1 EAGAIN (Resource temporarily unavailable)
futexCoxa74560, FUTEX_WAKE_PRIVATE, 1) = 0
futex(@x7ff4c0000e70, FUTEX_WAIT_BITSET_PRIVATE I FUTEX_CLOCK_REALTIME, 0, {tv_sec=3790335996, tv_nsec=773920000}, Oxffffffff) = 0
write(1, " cluster:\n id: 3938d835-"..., 862 cluster:
    id: xxxxx-xx-xxx-xxxx-xxxxxxx
    health: HEALTH_WARN
            no active mgr
            mons xxxxx-03, xxxxx-01 are using a lot of disk space
            1981 slow ops, oldest one blocked for 77 sec, daemons [mon.xxxxx-03, mon.xxxxx-01] have slow ops.
    services:
      mon: 2 daemons, quorum xxxxx-03, xxxxx-01 (age 19s)
      mgr: no daemons active (since 32h)
      osd: 836 osds: 826 up (since 15h), 816 in (since 15h); 154 remapped pgs
    data:
      pools:    10 pools, 12513 pgs
      objects:  346.45M objects, 1.3 PiB
      usage:    2.0 PiB used, 1.8 PiB / 3.8 PiB avail
      pgs:      12513 active+clean
    io:
      client:    13 MiB/s rd, 46 MiB/s wr, 813 op/s rd, 1.54k op/s wr
      cache:     32 MiB/s flush, 627 MiB/s evict, 52 op/s promote

)= 862
munmapC0x7ff4c459b000, 262144)
munmap(Øx7ff4c455b000, 262144)
munmap@x7ff4c44db000, 262144)
munmap(@x7ff4c465b000, 262144)
clone(child_stack=0x7ff4bf59dfbo, flags=CLONE_VMI CLONE_FS ICLONE_FILES I CLONE_SIGHAND I CLONE_THREAD I CLONE_SYSVSEMI CLONE_SETTLS I CLONE_PARENT_SETTIDICLONE_CHILD_CLEARTID, parent_tidptr=0x7ff4bf59e9do, tls=0x7ff4bf59e700, child_tidptr=0x7ff4bf59e9d0
- 490490
futexCoxa745cc, FUTEX_WAKE_PRIVATE, 1) = 1
futexCoxa74560, FUTEX_WAKE_PRIVATE, 1) = 1
futex(0x218db20, FUTEX_WAIT_BITSET_PRIVATE I FUTEX_CLOCK_REALTIME, O, NULL, Oxffffffff) = 0
[...]
+++ exited with 0 +++

Conclusion

Ternyata menurut kisah yang beredar, ceph-mon tersebut berubah segment IP dikarenakan pernah terjadi proses recreate ceph-mon menggunakan segment yang berbeda. Itu penyebab ketika saya kill ceph-mon pada host xxxxx-02, membuat ceph command stuck.

Untuk me-resolve hal tersebut hanya cukup mengubah config /etc/ceph/ceph.conf ke segment .251 (hal tersebut juga menyelesaikan masalah no active mgr), kemudian melakukan recreate ceph-mon, kembali ke .238. Baru setelah itu mengubah file konfigurasi seperti yang seharusnya.

Setelah Root Cause dari isu ini akhirnya selesai, *"saya akhirnya bisa melanjutkan malam minggu meskipun jatuhnya minggu-subuhan, bukan malam mingguan"*🥱.

Notes

Proses troubleshooting tidak semudah 5 langkah di atas, banyak sekali faktor/problem yang dalam prosesnya dilakukan. Namun tidak dituliskan. Tulisan di atas merupakan sebagian pendekatan yang saya lakukan sehingga bisa mengerucutkan proses troubleshooting.

"Everything will be fine if you just keep working on it, and take some cigarettes or coffee to meanwhile break also give a moment think clearly."

Reference

strace(1) - Linux manual page https://man7.org/linux/man-pages/man1/strace.1.html
Bug #14340: client command stuck forever - Ceph

https://ceph-users.ceph.narkive.com/xiTpiO1U/fwd-rbd-map-command-hangs

Troubleshooting Monitors - Ceph Documentation

https://docs.ceph.com/en/latest/rados/troubleshooting/troubleshooting-mon/

ceph: how to fix if monitor IP changes - 尚码园 https://www.shangmayuan.com/a/05cc4d6ab2e44b6cb71978d5.html

Resize Root Volume in Openstack Without Downtime

Sat, 11 Dec 2021 17:35:11 GMT

Terkadang kita perlu scale up volume pada partisi root (biasanya /dev/sda), yang menjadi masalah adalah volume tersebut tidak bisa di-deatach. Lalu bagaimana cara me-resize-nya ? berikut hal yang dapat dilakukan dengan syarat dan ketentuan yang berlaku..

Syarat dan Ketentuan

Pastikan VM tidak menggunakan Ephemeral storage, dalam kasus ini Saya menggunakan Backend Storage CEPH.

Resize From Openstack Side

## set status to available to a volume
openstack volume set --state available {volume id}

## resize the volume
openstack volume set --size 40 {volume id}

## check size and status again
openstack volume show {volume id}

## set status to in-use again
openstack volume set --state in-use {volume id}

Resize from Inside VM Side

## Grow part (Contoh akan resize root partisi)
sudo growpart /dev/vda 1

## Check apakah volume sudah grow sesuai size baru
lsblk

## Resize ‘/’ partition to fill all space
sudo resize2fs /dev/vda1

## (option) If your filesystem is XFS, it can be grown while mounted using the xfs_growfs command:

sudo xfs_growfs /

## Verify
sudo df -H

Reference

How To resize an ext2/3/4 and XFS root partition without LVM | ComputingForGeeks

https://computingforgeeks.com/resize-ext-and-xfs-root-partition-without-lvm/

How to resize a volume in an instance on OpenStack? - Server Fault

https://serverfault.com/questions/1006906/how-to-resize-a-volume-in-an-instance-on-openstack

Extend a Linux file system after resizing a volume - Amazon Elastic Compute Cloud

https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html

Using TCPDump & Wireshark to Debug Your Intermitten Interconnection

Sat, 27 Nov 2021 19:00:51 GMT

tcpdump is a powerful command-line packet analyzer; and libpcap, a portable C/C++ library for network traffic capture.

Dalam tulisan Saya kali ini, Saya akan menceritakan sebuah Production Case yang cukup me-musingkan, sekaligus membuat bingung Saya dan Team yang menangani Case tersebut.

TL;DR

Pada siang hari di saat Saya dan para kolega sedang menikmati coffe break, tiba-tiba kita mendapatkan pesan dari salah satu client yang kita manage service openstack-nya. Dalam pesannya mereka menyampaikan bahwa, terdapat salah satu Tenant yang Floating IP-nya tidak bisa diakses dari manapun.

Pesan tersebut seketika membuat Saya yang tadinya sedang menikmati sebatang Sampoernah Mild, buru-buru untuk membuka laptop.

Quick Check

Segera setelah itu Saya melakukan pengecekan, apakah benar terjadi hal seperti dalam pesan yang client sampaikan.

1. Testing ICMP Connection

Ketika Saya mencoba melakukan testing ICMP koneksi menggunakan ping, dari Host controller menuju VM pada tenant yang bermasalah, hasilnya semua running well.

Di dalam hati Saya berkata "oh, harusnya aman ini, orang di-ping aja bisa... Paling cuman Security Groups..."

2. Testing Access SSH

Dapat terlihat bahwa ketika melakukan akses ssh ke FIP tersebut menggunakan argument debug ( -vvv ), connection established. Namun session SSH tidak terbuka-buka.

Di dalam hati Saya berkata lagi, "Loh,, kok kagak bisa yak!". Kemudian Saya dan Team sedikit berfikir, mungkin karena MTU yang berbeda.

3. Testing Ping Jumbo Frames

Setelah dilakukan percobaan ping ke vRouter pada tenant tersebut dengan Jumbo Frame, hasilnya memang tidak bisa dilakukan. Namun ternyata vRouter pada tenant tersebut memang menggunakan MTU 1500, sehingga hal tersebut wajar terjadi.

Saya berkata lagi, namun kali ini tidak dalam hati "INI KENAPA OYYYY!!!". Teman Saya yang sedang handle pekerjaan yang lebih santai, tertawa melihat Saya yang terlihat puyeng.

Saya mencoba memutar otak; skenario apa yang perlu Saya coba lagi, hingga Saya mendapat ide...

4. Testing SSH Access ( VM -> Controller)

Jika akses SSH dari [Controller -> VM] tidak bisa, "seharusnya" SSH dari [VM -> Controller], juga tidak bisa.

Ternyata hasilnya sebaliknya, SSH dari [VM -> Controller] bisa dilakukan.

Dalam hati Saya berkata "Loh kok aneh.....!"

In-depth Check

1. Service Network Openstack

Saya melakukan pemeriksaan di semua service network openstack. Namun tidak ditemukan adanya error yang menyebabkan hal semacam ini terjadi.

2. Firewall

Pemeriksaan Firewall di semua Host dan VM untuk memastikan bahwa tidak ada firewall yang menghadang paket. Namun tidak ditemukan adanya tanda - tanda firewall penghadang.

3. TCPdump & Wireshark

Di saat-saat despreate seperti ini TCPdump dan Wireshark merupakan perfect combination

TCPDump DARI CONTROLLER

Di sini Saya menggunakan Tmux window 1 dan window 2. Keduanya melakukan eksekusi command yang berbeda.

### Window 1
tcpdump -ni team0.1102 -s 65535 host 172.16.100.228 -w debug-ssh.pcap

### Window 2
ssh -vvv 172.16.100.228

Keduanya Saya biarkan berjalan sekitar ±5 menit.

IMPORT KE Whireshark

Dari visualisasi pada Wireshark tersebut sudah dapat terlihat pokok ujung dari permasalahan ini.

Mac-address fa:16:3e:3c:42:16, merupakan mac dari vRouter tenant yang bermasalah. Jika dilihat pattern yang terjadi; semua TCP Retransmission, Dup ACK terus menerus terjadi dan tidak ada paket yang berhasil melewati vRouter tenant tersebut.

Final Action

Setelah melakukan diskusi dengan Team dan Client, kita memutuskan untuk melakukan recreate vRouter pada tenant tersebut, dan melakukan konfigurasi Gateway yang lebih proper.

After Recreate vRouter

Berikut tampilan visualisasi TCPDump setelah dilakukan Action Recreate vRouter.

Dapat dilihat hasil visualisasi TCPDump dari akses SSH yang menggunakan vRouter baru & Gateway yang lebih proper tidak menunjukkan adanya TCP Retransmission ataupun Dup ACK .

Akhirnya setelah semuanya selesai, baru Saya bakar lagi sebatang Sampoerna Mild yang baru saja buka bungkus.

Reference

D.3. tcpdump: Capturing with “tcpdump” for viewing with Wireshark

https://www.wireshark.org/docs/wsug_html_chunked/AppToolstcpdump.html

tcpdump(1) man page | TCPDUMP & LIBPCAP

https://www.tcpdump.org/manpages/tcpdump.1.html

TCP Retransmission Timeout (RTO): Causes & Performance

https://www.extrahop.com/company/blog/2016/retransmission-timeouts-rtos-application-performance-degradation/#:~:text=What%20Is%20TCP%20Retransmission%3F,expires%2C%20the%20segment%20is%20retransmitted.

TCP Series #3: Network Packet Loss, Retransmissions, and Dup Acks

https://accedian.com/blog/network-packet-loss-retransmissions-and-duplicate-acknowledgements/

Memahami Message [watchdog: BUG: soft lockup - CPU#X stuck for XXs!]

Sat, 25 Sep 2021 10:50:58 GMT

This 'soft lockup' can happen if the kernel is busy, working on a huge amount of objects which need to be scanned, freed, or allocated, respectively.

Berdasarkan pengalaman, kernel message watchdog: BUG: soft lockup - CPU#X stuck for XXs! dalam beberapa kasus tidak akan terjadi problem (Hang) jika stuck hanya beberapa detik. Namun jika sudah stuck time -nya semakin lama, dan message tersebut sering muncul mengindikasikan kernel space busy tiada henti.

A soft lockup is a situation usually caused by a bug, when a task is executing in kernel space on a CPU without rescheduling. The task also does not allow any other task to execute on that particular CPU. As a result, a warning is displayed to a user through the system console. This problem is also referred to as the soft lockup firing.

Soft lockup merupakan situasi yang disebabkan oleh bug, ketika sebuah task dieksekusi oleh kernel space di CPU tanpa rescheduling. Task tersebut juga tidak mengizinkan task yang lain dieksekusi di CPU tertentu. Sehingga muncul Warning message dari kernel, watchdog: BUG: soft lockup - CPU#X stuck for XXs!

Untuk indikasi lebih dalam perlu pengecekan Host dari VM tersebut.

Beberapa Opsi yang bisa dicoba untuk menangani hal tersebut

Disable kerenel panic; By default RHEL 8 sudah melakukan hal tersebut. Parameters ini berguna supaya tidak terjadi kernel panic ketika ada soft lockup; Sehingga VM Tidak Hang ketika terjadi soft lockup.

 echo "kernel.softlockup_panic=0" >> /etc/sysctl.conf
 sysctl --system

Incrase watchdog_thresh, agar message soft lockup muncul hanya ketika cpu stuck lebih dari 30s.

   echo "kernel.watchdog_thresh=30" >> /etc/sysctl.conf                
   sysctl --system

Upgrade Kernel to latest. Kembali lagi tindakan yang bisa dilakukan untuk bug kernel tidak banyak, opsi terbaik adalah patch the bug.
Check VM Host, issue seperti ini sering terjadi pada teknologi virtualisasi seperti VMware, KVM. Tidak menutup kemungkinan Host VM tersebut memang sudah exhausted.

Reference

What are all these "Bug: soft lockup" messages about? | Support | SUSE

https://www.suse.com/support/kb/doc/?id=000018705

kernel - watchdog: BUG: soft lockup - CPU#6 stuck for 23s - Ask Ubuntu

https://askubuntu.com/questions/1264859/watchdog-bug-soft-lockup-cpu6-stuck-for-23s

Error message: "NMI watchdog: BUG: soft lockup - CPU#X stuck for XXs!" on console (67623)

https://kb.vmware.com/s/article/67623

VMware virtual machine guest soft lockups in mpt_put_msg_frame - Red Hat Customer Portal

https://access.redhat.com/solutions/3176741

Virtualization lags and hypervisor overcommitment - Red Hat Customer Portal

https://access.redhat.com/articles/5008811

Chapter 7. Keeping kernel panic parameters disabled in virtualized environments Red Hat Enterprise Linux 8 | Red Hat Customer Portal

https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/managing_monitoring_and_updating_the_kernel/keeping-kernel-panic-parameters-disabled-in-virtualized-environments_managing-monitoring-and-updating-the-kernel

linux - What does "kernel:NMI watchdog: BUG: soft lockup" followed by other errors mean? - Unix & Linux Stack Exchange

https://unix.stackexchange.com/questions/216959/what-does-kernelnmi-watchdog-bug-soft-lockup-followed-by-other-errors-mean

"NMI watchdog: BUG: soft lockup - CPU # stuck for #s! [X:3005]" in message log file after a Flame application crash | Flame Products 2020 | Autodesk Knowledge Network

https://knowledge.autodesk.com/support/flame-products/troubleshooting/caas/sfdcarticles/sfdcarticles/NMI-watchdog-BUG-soft-lockup-CPU-stuck-for-s-X-3005-in-message-log-file-after-a-Flame-application-crash.html

OpenStack Multi-Tenancy Architecture: Glance, Keystone, and Cross-Tenant Resource Sharing

Tue, 22 Dec 2020 10:00:00 GMT

Multi-tenancy is one of OpenStack's fundamental design principles, enabling secure resource isolation while allowing controlled sharing when needed. Understanding how this works at the architectural level, particularly the interaction between Keystone's authentication layer and Glance's image management is crucial for operating production OpenStack deployments.

OpenStack Multi-Tenancy Architecture

Keystone: Projects, Domains, and RBAC

Keystone implements multi-tenancy through a hierarchical model:

# tree
Domain
  └── Projects (Tenants)
       └── Users + Roles

Projects (formerly called tenants) are the fundamental isolation boundary. Each project has its own quota, resource ownership, and access control. Domains provide an additional layer of organization, allowing multiple projects to be grouped, useful for large organizations with multiple departments.

Role-Based Access Control (RBAC) determines what users can do within a project. Common roles include:

admin: Full control within a project
member: Standard user operations
reader: Read-only access

These roles are enforced through policy.json files in each OpenStack service.

Token Scoping Mechanisms

Keystone tokens can be scoped in different ways, fundamentally changing what resources they can access:

# Unscoped token - can list projects but access no resources
openstack token issue

# Project-scoped token - access resources in specific project
openstack --os-project-name production token issue

# Domain-scoped token - administrative operations across projects
openstack --os-domain-name engineering token issue

When a user authenticates, Keystone returns a token containing:

User identity and roles
Scope (project/domain)
Service catalog (API endpoints)
Expiration timestamp

This token is presented to every OpenStack service (Nova, Glance, Cinder, etc.), which validates it against Keystone and enforces authorization based on the scope and roles.

Control Plane vs Data Plane Authorization

OpenStack separates two authorization layers:

Control Plane: API-level authorization enforced by policy.json rules. Example from Glance:

{
  "get_image": "role:admin or role:member",
  "add_member": "role:admin",
  "modify_member": "role:admin"
}

Data Plane: Database-level ownership and visibility. Even if you have API access, you can only see resources you own or that have been explicitly shared with you.

Glance Image Management Architecture

Glance is the OpenStack Image Service, responsible for discovering, registering, and delivering virtual machine images. Its architecture cleanly separates concerns.

Image Registry vs Image Store

Glance uses a two-component architecture:

glance-registry: Manages image metadata in a database (MySQL). Stores:

Image properties (name, size, format, container format)
Ownership (project_id)
Visibility (public/private/shared/community)
Custom metadata (key-value pairs)
Image members (sharing relationships)

glance-api: Handles API requests and interfaces with storage backends. Acts as the gateway between clients and actual image data.

This separation allows metadata operations (listing images, updating properties) to be fast database queries, while image data transfers happen through optimized storage backends.

Backend Storage Drivers

Glance supports multiple storage backends, each with different characteristics:

# In /etc/glance/glance-api.conf

[glance_store]
stores = file,http,swift,rbd
default_store = rbd

[glance_store:rbd]
rbd_store_pool = images
rbd_store_user = glance
rbd_store_ceph_conf = /etc/ceph/ceph.conf

Storage backend comparison:

Ceph (RBD): High performance, distributed, supports CoW (copy-on-write) for fast cloning
Swift: Object storage, highly scalable, geographically distributed
Filesystem: Simple, limited scalability, useful for small deployments
HTTP: Read-only store for importing images from HTTP/HTTPS URLs

Image Metadata Schema

Every image has core metadata and supports custom properties:

$ openstack image create \
  --disk-format qcow2 \
  --container-format bare \
  --min-disk 10 \
  --min-ram 1024 \
  --property os_type=linux \
  --property os_distro=ubuntu \
  --property os_version=20.04 \
  --file ubuntu-20.04.qcow2 \
  ubuntu-base

+------------------+------------------------------------------------------+
| Field            | Value                                                |
+------------------+------------------------------------------------------+
| checksum         | 4d8b3f50b6e7c8a2d9f1e3a5c6b8d9e1                     |
| container_format | bare                                                 |
| created_at       | 2020-12-15T10:30:00Z                                 |
| disk_format      | qcow2                                                |
| file             | /v2/images/a3c7b4d1-2e8f-4a9b-b2c1-5d6e7f8a9b0c/file |
| id               | a3c7b4d1-2e8f-4a9b-b2c1-5d6e7f8a9b0c                 |
| min_disk         | 10                                                   |
| min_ram          | 1024                                                 |
| name             | ubuntu-base                                          |
| os_distro        | ubuntu                                               |
| os_type          | linux                                                |
| os_version       | 20.04                                                |
| owner            | f4e8d2c1a3b5e7d9c8b6a4f2e1d3c5b7                     |
| protected        | False                                                |
| schema           | /v2/schemas/image                                    |
| size             | 2361393152                                           |
| status           | active                                               |
| tags             |                                                      |
| updated_at       | 2020-12-15T10:31:45Z                                 |
| virtual_size     | None                                                 |
| visibility       | private                                              |
+------------------+------------------------------------------------------+

Custom properties enable advanced functionality:

hw_scsi_model=virtio-scsi: Use virtio SCSI for better performance
hw_disk_bus=scsi: Set disk bus type
os_secure_boot=required: Enable UEFI Secure Boot
img_signature: GPG signature for image verification

Image Visibility States

Glance defines four visibility levels:

public: Visible to all projects (requires admin to create)
private: Visible only to owning project (default)
shared: Shared with specific projects via image members
community: Shared with all, but not in default listings

The state machine:

private → shared → community
   │         ↑              ↑
   └─────────┴──────────────┘
          (admin only)

Image Membership Model

The glance.image_members table implements sharing (see Glance API v2 Image Members implementation):

CREATE TABLE image_members (
  id INT PRIMARY KEY AUTO_INCREMENT,
  image_id VARCHAR(36) NOT NULL,
  member VARCHAR(255) NOT NULL,  -- target project_id
  can_share BOOLEAN DEFAULT FALSE,
  status ENUM('pending', 'accepted', 'rejected') DEFAULT 'pending',
  created_at DATETIME,
  updated_at DATETIME,
  deleted_at DATETIME,
  deleted BOOLEAN DEFAULT FALSE,
  UNIQUE KEY (image_id, member)
);

The status field implements a three-way handshake for sharing, preventing spam.

Image Lifecycle State Machine

Images transition through several states:

queued → saving → active → deactivated → deleted
                    ↓
                 pending_delete

queued: Metadata created, no data uploaded yet
saving: Data upload in progress
active: Ready for use
deactivated: Temporarily disabled (admin action)
pending_delete: Scheduled for removal
deleted: Soft-deleted (recoverable)

Cross-Tenant Image Sharing

The complete workflow for sharing an image between projects involves several steps and database changes.

The Complete API Workflow

Step 1: Image owner creates member

# As project A (owner), share image with project B
$ openstack image add project \
  my-ubuntu-image \
  <project-b-id>

# Low-level API call
$ curl -X POST \
  -H "X-Auth-Token: $TOKEN_PROJECT_A" \
  -H "Content-Type: application/json" \
  http://glance:9292/v2/images/<image-id>/members \
  -d '{"member": "<project-b-id>"}'

The query that happens in database:

INSERT INTO image_members
(image_id, member, status, created_at)
VALUES
('<image-id>', '<project-b-id>', 'pending', NOW());

At this point, project B cannot see the image yet.

Step 2: Project B accepts the share

# As project B, list pending shares
$ openstack --os-project-name project-b image member list <image-id>

# Accept the share
$ openstack --os-project-name project-b image set \
  --accept \
  <image-id>

# Low-level API call
$ curl -X PUT \
  -H "X-Auth-Token: $TOKEN_PROJECT_B" \
  -H "Content-Type: application/json" \
  http://glance:9292/v2/images/<image-id>/members/<project-b-id> \
  -d '{"status": "accepted"}'

The query that happens in database:

UPDATE image_members
SET status = 'accepted', updated_at = NOW()
WHERE image_id = '<image-id>' AND member = '<project-b-id>';

Step 3: Verification

# Project B can now see and use the image
$ openstack --os-project-name project-b image list

$ openstack --os-project-name project-b server create \
  --image my-ubuntu-image \
  --flavor m1.small \
  test-instance

Policy.json Rules

The sharing workflow is governed by policies in /etc/glance/policy.json:

{
  "add_member": "role:admin or (role:member and project_id:%(owner)s)",
  "get_member": "role:admin or project_id:%(owner)s or project_id:%(member)s",
  "modify_member": "role:admin or (role:member and project_id:%(member)s)",
  "delete_member": "role:admin or (role:member and project_id:%(owner)s)"
}

Key points:

Only the image owner can add members
Only the target project can accept/reject membership
Both owner and member can view membership status
Only owner can revoke sharing

Keystone Token Validation Integration

When project B makes an API call to Glance:

Glance receives the request with project B's token
Glance validates token with Keystone (cached for performance)
Glance extracts project_id from token context

Glance queries database:

SELECT * FROM images i
LEFT JOIN image_members m ON i.id = m.image_id
WHERE i.owner = '<project-b-id>'
   OR i.visibility = 'public'
   OR (m.member = '<project-b-id>' AND m.status = 'accepted');

Policy.json rules are evaluated with context variables
If authorized, the operation proceeds

Sharing vs Cloning

Two approaches to make an image available across projects:

Image Sharing (membership):

Single image, multiple viewers
No storage duplication
Owner controls lifecycle (can delete/modify)
Changes propagate to all members
Uses image_members table

Image Cloning:

# Download from project A
openstack image save --file /tmp/image.qcow2 <image-id>

# Upload to project B
openstack --os-project-name project-b image create \
  --file /tmp/image.qcow2 \
  --disk-format qcow2 \
  cloned-image

Complete copy, independent lifecycle
Double storage consumption
Full ownership by target project
Immune to source image changes
Better for long-term independence

With Ceph RBD backend, cloning is more efficient:

# Create copy-on-write clone (instant, no data copy)
rbd clone images/<image-id>@snap images/<new-image-id>

Architectural Tradeoffs

Sharing vs Cloning: When to Use Each

Use sharing when:

Image is large (100GB+ VM templates)
Centralized image management desired
Regular updates need to propagate
Storage efficiency is critical
Short-term collaboration

Use cloning when:

Projects need full control
Image stability required (prod vs dev)
Compliance requires data isolation
Projects might outlive image owner
Network transfer is acceptable

Security Boundaries and Tenant Isolation

OpenStack provides strong isolation guarantees:

Network isolation: Neutron ensures projects cannot see each other's traffic
Storage isolation: Ceph uses project-specific pools/namespaces
Compute isolation: Hypervisor (KVM/QEMU) provides process isolation
API isolation: All cross-tenant operations require explicit authorization

However, shared images introduce a trust boundary:

Malicious image owner could include backdoors
Image updates affect all consumers
Metadata visibility (image size, format) is shared

Best practices:

Use image signature verification
Clone images for production workloads
Implement image scanning pipelines
Restrict public image creation to admins

Advanced Topics

Image Deactivation and Reactivation

Admins can temporarily disable images without deletion:

# Deactivate image (prevents new instance creation)
openstack image set --deactivate <image-id>

# Existing instances continue running
# New instance creation fails:
# ERROR: Image <image-id> is deactivated

# Reactivate when issue resolved
openstack image set --activate <image-id>

Database transition:

UPDATE images SET status = 'deactivated' WHERE id = '<image-id>';
-- Later...
UPDATE images SET status = 'active' WHERE id = '<image-id>';

Use cases:

Security vulnerability discovered
Licensing issues
Corrupted image data
Maintenance windows

Multi-Backend Configuration

Glance can use different backends for different images:

[glance_store]
stores = ceph_ssd,ceph_hdd,swift_backup
default_store = ceph_ssd

[glance_store:ceph_ssd]
rbd_store_pool = images-fast

[glance_store:ceph_hdd]
rbd_store_pool = images-archive

[glance_store:swift_backup]
swift_store_auth_address = https://swift.example.com/auth/v1.0

Migrating images between backends:

# Copy image to different backend
glance --os-image-api-version 2 image-import <image-id> \
  --method copy-image \
  --store ceph_hdd

Image Import Methods

Glance v2 supports three import methods:

1. glance-direct (default): Upload directly to Glance

openstack image create --file image.qcow2 my-image

2. web-download: Glance downloads from URL

openstack image create my-image
openstack image set my-image --property hw_disk_bus=scsi

# Trigger import
curl -X POST \
  -H "X-Auth-Token: $TOKEN" \
  -H "Content-Type: application/json" \
  http://glance:9292/v2/images/<image-id>/import \
  -d '{
    "method": {
      "name": "web-download",
      "uri": "https://cloud-images.ubuntu.com/focal/current/focal-server-cloudimg-amd64.img"
    }
  }'

3. copy-image: Copy between Glance stores

glance image-import <image-id> --method copy-image --store swift_backup

Image Signature Verification

For trusted image pipelines:

# Generate GPG signature
gpg --detach-sign --armor image.qcow2

# Upload with signature
openstack image create \
  --file image.qcow2 \
  --property img_signature=$(cat image.qcow2.asc | base64 -w0) \
  --property img_signature_hash_method=SHA-256 \
  --property img_signature_key_type=GPG \
  --property img_signature_certificate_uuid=$KEY_UUID \
  trusted-ubuntu

Nova can be configured to reject unsigned images:

[glance]
verify_glance_signatures = true

Glance Integration with Cinder and Nova

Cinder (volumes from images):

# Create bootable volume from image
openstack volume create \
  --image ubuntu-20.04 \
  --size 20 \
  --bootable \
  my-boot-volume

Backend flow:

Cinder calls Glance API to get image location
Cinder uses qemu-img convert or Ceph RBD clone
Volume marked as bootable with image metadata

Nova (ephemeral disk creation):

openstack server create \
  --image ubuntu-20.04 \
  --flavor m1.small \
  instance-1

Backend flow:

Nova queries Glance for image details
Nova creates ephemeral disk (Ceph RBD clone or qcow2 backing file)
Nova boots VM with disk attached

With Ceph RBD:

# Nova creates instant CoW clone
rbd clone images/ubuntu-20.04@snap vms/instance-1_disk

Ceph RBD Cloning: Flatten vs Not-Flatten

When Cinder creates volumes from Glance images stored in Ceph RBD, it can either create a copy-on-write (CoW) clone or flatten the volume. This choice has significant performance implications for production clusters.

The Clone Hierarchy Problem:

Glance Image (parent)
    └── Volume Clone (child)
         └── Snapshot
              └── Another Clone (grandchild)

Each clone maintains a dependency on its parent, creating a chain that Ceph must traverse on every read operation that hasn't been overwritten yet.

Performance Impact Analysis

Flattening (decoupling from parent):

When you flatten a volume, Ceph copies all data from the parent image into the child volume, making it independent:

# Flatten a volume manually
rbd flatten volumes/volume-abc123

Impact on OSDs during flattening:

Massive sequential I/O: The entire parent image is copied to the volume
OSD saturation: Sequential writes can consume hundreds of MB/s per OSD
Immediate Slow Ops: Other VMs experience degraded performance during the operation
Duration: Can take 10+ minutes for a 100GB volume depending on cluster load

Not Flattening (keeping CoW clone):

Volumes remain as thin clones, sharing data with the parent image:

# Check clone relationship
rbd info volumes/volume-abc123
# Shows: parent: images/ubuntu-20.04@snap

Impact on ongoing operations:

Random I/O spikes: Every first read of a block triggers a "copy-up" operation
Increased metadata lookups: Ceph must check parent chain for unchanged blocks
Chronic Slow Ops: Sustained performance degradation during normal VM operations
Higher read latency: 2-10x increase for reads hitting parent blocks
Deep clone chains: >5 levels can severely impact performance

Cinder Configuration Options

In /etc/cinder/cinder.conf:

[DEFAULT]
# Clone from Glance image (fast initial creation)
rbd_flatten_volume_from_snapshot = false

# Enable copy-on-read to gradually flatten on-demand
rbd_clone_copy_on_read = true

# Maximum clone depth before forcing flatten
rbd_max_clone_depth = 5

Blueprint reference: Cinder Flatten Volume from Image

Real-World Impact Summary

Scenario	Impact on OSDs	Consequence
Flattening	Massive, sustained sequential I/O while decoupling	Immediate Slow Ops due to OSD saturation during the move
Not Flattening	Random I/O spikes (copy-ups) and increased metadata lookups	Chronic Slow Ops during normal VM operations and high read latency

Production Recommendations

1. Use rbd_clone_copy_on_read = true:

This enables Copy-on-Read, which moves data to the clone upon first read, effectively "flattening" the image on-demand:

[DEFAULT]
rbd_clone_copy_on_read = true

Benefits:

Gradual flattening during normal operations
No massive I/O spike
Parent blocks copied only when accessed
Eventually achieves independent volume

2. Avoid Long Clone Chains:

Limit clone depth to prevent performance degradation:

[DEFAULT]
rbd_max_clone_depth = 5

If your Glance configuration allows deep hierarchies (e.g., snapshot → clone → snapshot → clone), ensure you're not exceeding 5-10 levels. Each level adds latency to read operations.

3. Flatten During Low Traffic:

If you must flatten to decouple from a parent (e.g., before deleting a "golden image"), schedule it during off-peak hours:

# Flatten all volumes derived from an image before deletion
for vol in $(rbd ls volumes | grep parent-image); do
    rbd flatten volumes/$vol &
done
wait

This avoids competing with production VM traffic and prevents widespread Slow Ops warnings.

4. Monitor Clone Depth:

Track clone chain depth in your monitoring:

# Check clone depth for a volume
rbd info volumes/volume-abc123 | grep parent

# Recursive script to find depth
function clone_depth() {
    local img=$1
    local depth=0
    while rbd info $img 2>/dev/null | grep -q parent; do
        img=$(rbd info $img | grep parent | awk '{print $2}')
        ((depth++))
    done
    echo $depth
}

clone_depth volumes/volume-abc123

Alert when depth exceeds 5 to trigger manual flattening or investigation.

5. Pre-flatten Critical Workload Volumes:

For performance-sensitive workloads (databases, high-IOPS applications), flatten volumes immediately after creation:

# Create volume from image
openstack volume create --image ubuntu-db --size 100 db-volume

# Get volume ID and flatten
VOL_ID=$(openstack volume show db-volume -f value -c id)
rbd flatten volumes/volume-$VOL_ID

This ensures predictable, low-latency I/O without parent lookups.

Debugging Clone-Related Performance Issues

Symptom: VMs experience high I/O latency and Ceph shows "slow ops"

# Check for slow ops in Ceph
ceph health detail
# HEALTH_WARN 15 slow ops, oldest one blocked for 32 sec

# Identify which volumes have deep clone chains
for vol in $(rbd ls volumes); do
    depth=$(clone_depth volumes/$vol)
    if [ $depth -gt 3 ]; then
        echo "Volume $vol has depth $depth"
        rbd info volumes/$vol
    fi
done

Solution: Flatten volumes with deep chains during maintenance window or enable rbd_clone_copy_on_read to gradually resolve the issue.

Technical Troubleshooting

Keystone Role Assignment Issues

Issue: Member acceptance fails with 403 Forbidden.

$ glance member-update <image-id> <project-id> accepted
403 Forbidden: Policy doesn't allow modify_member to be performed

Diagnosis: Verify user has correct role in target project:

openstack role assignment list \
  --user $USER_ID \
  --project $PROJECT_ID \
  --names

Fix:

openstack role add \
  --user john.doe \
  --project project-b \
  member

Image Member State Machine Errors

Issue: Image shows as shared but isn't visible in target project.

Diagnosis: Check member status in database:

SELECT
  i.name,
  m.member AS project_id,
  m.status,
  m.created_at
FROM images i
JOIN image_members m ON i.id = m.image_id
WHERE i.id = '<image-id>';

Expected output:

+------------+------------------+----------+---------------------+
| name       | project_id       | status   | created_at          |
+------------+------------------+----------+---------------------+
| my-ubuntu  | abc123...        | accepted | 2020-01-22 10:30:00 |
+------------+------------------+----------+---------------------+

If status = 'pending', the target project needs to accept:

glance member-update <image-id> <project-id> accepted

Storage Backend Connectivity Problems

Issue: Image uploads hang or fail with timeout.

Diagnosis: Test Ceph connectivity:

# Check Ceph cluster health
ceph health detail

# Test RBD access with Glance credentials
rbd --id glance --keyring /etc/ceph/ceph.client.glance.keyring \
  ls images

# Monitor Glance API logs
sudo tail -f /var/log/glance/glance-api.log | grep -i error

Common errors:

ERROR glance.common.client [req-...] Connection timeout to Ceph cluster

Fix: Verify network connectivity and credentials:

# Test network to Ceph monitors
telnet ceph-mon-1 6789

# Verify keyring permissions
ls -la /etc/ceph/ceph.client.glance.keyring
# Should be readable by glance user

Debugging with Glance Logs and SQL Queries

Enable debug logging in /etc/glance/glance-api.conf:

[DEFAULT]
debug = True
log_file = /var/log/glance/glance-api.log

Monitor API requests:

sudo tail -f /var/log/glance/glance-api.log | \
  grep -E "(POST|PUT|DELETE|ERROR)"

Useful database queries:

-- Find all images shared with project
SELECT
  i.id,
  i.name,
  i.owner,
  i.visibility,
  m.status AS share_status
FROM images i
LEFT JOIN image_members m ON i.id = m.image_id
WHERE m.member = '<project-id>' OR i.owner = '<project-id>'
ORDER BY i.created_at DESC;

-- Find orphaned image members (target project deleted)
SELECT m.* FROM image_members m
LEFT JOIN keystone.project p ON m.member = p.id
WHERE p.id IS NULL AND m.deleted = 0;

-- Check image storage usage by project
SELECT
  owner AS project_id,
  COUNT(*) AS image_count,
  SUM(size) / 1024 / 1024 / 1024 AS total_gb
FROM images
WHERE deleted = 0
GROUP BY owner
ORDER BY total_gb DESC;

Conclusion

OpenStack's multi-tenancy architecture demonstrates sophisticated separation of concerns: Keystone handles authentication and authorization, Glance manages image lifecycle and metadata, and the cross-tenant sharing mechanism provides controlled collaboration without sacrificing isolation.

Understanding these internals, from token scoping to database schema to policy enforcement, enables operators to troubleshoot issues quickly, implement custom workflows, and optimize for specific use cases. The key insight is that multi-tenancy isn't just API-level filtering; it's a comprehensive system spanning authentication tokens, database constraints, policy rules, and storage backends working together to provide both isolation and controlled sharing.

For production deployments, remember:

Use image signature verification for security
Clone images for critical workloads to avoid dependency on other projects
Monitor quota usage and enforce limits at both API and database layers
Implement proper RBAC through Keystone and policy.json
Choose storage backends based on your performance and scale requirements
Maintain detailed logs for audit and troubleshooting

The multi-tenancy model is what makes OpenStack suitable for large-scale cloud deployments, enabling thousands of projects to safely coexist while sharing infrastructure efficiently.

References

Special thanks to my mentor whose blog posts on OpenStack provided foundational knowledge for this deep dive:

Layanan Keystone untuk Autentikasi Identitas di OpenStack (Bagian 2) - Comprehensive guide to Keystone authentication and identity services
OpenStack Image Service: Managing Snapshots & Backups - Detailed coverage of Glance image management
OpenStack Glance Official Documentation - Glance architecture and API
Cinder Flatten Volume from Image Blueprint - Specification for Ceph RBD volume flattening feature

Centralized Logging Dengan Graylog

Wed, 16 Sep 2020 06:44:58 GMT

Apa itu Centralized Logging ?

Centralized Loging, jika diBahasa-kan (Pencatatan yang tersentralisasi). Sentralisasi logingg di sini dapat diartikan dimana terdapat berbagai sumber log, yang di dorong ke satu pusat antarmuka. Di antarmuka tersebut kita dapat melakukan penguraian, penambahan, dan pemantauan waktu nyata terhadap semua data-data log tersebut.

Pada tulisan Saya kali ini, Saya akan memberikan ringkasan langkah-langkah dalam installasi Graylog kluster sederhana sebagai Centralized Logging.

Topologi

Topologi di atas adalah infrastruktur yang akan kita coba untuk bangun.

Dependency

Langkah pertama adalah melakukan pemasangan Dependency yang dibutuhkan oleh graylog :

Install Dependency

$ sudo apt-get update && sudo apt-get upgrade
$ sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen

The Stack

Untuk membangun Graylog Server yang siap digunakan, ada graylog-server memberlukan integrasi dengan beberapa perangkat lunak lain diantaranya Elasticsearch dan MongoDB :

Install MongoDB

$ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 9DA31620334BD75D9DCB49F368818C72E52529D4
$ echo "deb [ arch=amd64 ] https://repo.mongodb.org/apt/ubuntu bionic/mongodb-org/4.0 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-4.0.list
$ sudo apt-get update
$ sudo apt-get install -y mongodb-org
$ sudo systemctl daemon-reload
$ sudo systemctl enable mongod.service
$ sudo systemctl restart mongod.service
$ sudo systemctl --type=service --state=active | grep mongod

Install Elasticsearch

$ wget -q https://artifacts.elastic.co/GPG-KEY-elasticsearch -O myKey
$ sudo apt-key add myKey
$ echo "deb https://artifacts.elastic.co/packages/oss-6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
$ sudo apt-get update && sudo apt-get install elasticsearch-oss
$ sudo tee -a /etc/elasticsearch/elasticsearch.yml > /dev/null <<EOT
cluster.name: graylog
action.auto_create_index: false
EOT
$ sudo systemctl daemon-reload
$ sudo systemctl enable elasticsearch.service
$ sudo systemctl restart elasticsearch.service
$ sudo systemctl --type=service --state=active | grep elasticsearch

Install Graylog Server

$ wget https://packages.graylog2.org/repo/packages/graylog-3.3-repository_latest.deb
$ sudo dpkg -i graylog-3.3-repository_latest.deb
$ sudo apt-get update && sudo apt-get install graylog-server graylog-enterprise-plugins graylog-integrations-plugins graylog-enterprise-integrations-plugins
$ sudo -E sed -i -e "s/password_secret =.*/password_secret = $(pwgen -s 128 1)/" /etc/graylog/server/server.conf
$ sudo sed -i -e "s/root_password_sha2 =.*/root_password_sha2 = $(echo -n 'InP8TP@AsssowrdKamudiSini' | shasum -a 256 | cut -d' ' -f1)/" /etc/graylog/server/server.conf
$ sudo sed -i -e "s/#http_bind_address\s=\s127.0.0.1:9000/http_bind_address = 10.252.10.81:9000/g" /etc/graylog/server/server.conf
$ sudo systemctl daemon-reload
$ sudo systemctl enable graylog-server.service
$ sudo systemctl start graylog-server.service
$ sudo systemctl --type=service --state=active | grep graylog

Cek apakah graylog-server sudah berasil bind port 9000

Web interface Graylog dapat diakses sekitar 2-3 menit setelah service graylog-server di-start
```
curl 10.252.10.81:9000
netstat -nputw|grep 9000
```

Integrasi Inputs Logs

Untuk melakukan input log ke graylog kita memberlukan sebuah forwarder log message yang berfungsi sebagai pengirim logs ke graylog server. Cukup banyak Forwarder yang dapat digunakan diantaranya Ryslog, Filebeat, Fluentd, dan Khusus untuk Docker Gelf(Graylog Extended Format logging driver)

Ryslogs

Install rsyslog

$ sudo apt install ryslog -y
$ sudo vi /etc/rsyslog.d/graylog.conf
  *.* @10.252.10.81:1514;RSYSLOG_SyslogProtocol23Format

10.252.10.81, Merupakan alamat IP graylog server.

1514, Meruapakan port yang listent input dengan format syslog

Gelf
```
docker run -d -p 8889:80 --log-driver=gelf --log-opt gelf-address=udp://10.252.10.81:18312 httpd
```
--log-driver=gelf, Adalah yang kita butuhkan untuk menggunakan gelf sebagai forwarder logs menuju graylog server.

--log-opt gelf-address=udp://10.252.10.81:18312, dibutuhkan untuk memberitahukan alamat dan port yang me-listent gelf input pada graylog server.

Bug Hunting di Platform Tokopedia

Mon, 07 Sep 2020 06:44:58 GMT

5 Februari 2020, Bulan dimana saya memasuki bulan ke-2 program apprentice di salah satu IT Konsultan di Bogor.

Semua berawal ketika saya duduk di kursi kerja, dan merasa bosan, karena sudah menyelesaikan pekerjaan saya. Seketika langsung saja saya ketikkan di terminal kesayangan amass enum -d tokopedia.net,tokopedia.com -json tokopedia-subdomains.json.

Menunggu dengan mata yang sayup-sayup mengantuk saat tengah hari, di bawah AC yang sangat dingin. Saya lihat hasil output dari amass cat tokopeda-subdomains.json | jq '.name' | tr -d '"'.

Saya mencoba melihat baris demi baris, dari nama-nama subdomain yang didapat, tapi tak ada yang menarik. Kemudian saya lanjutkan melakukan request ke semua subdomain yang didapat untuk mengetahui protokol yang tersedia http/https cat tokopeda-subdomains.json | jq '.name' | tr -d '"'| httprobe | tee -a results.

Lalu saya melakukan request lagi menggunakan meg untuk mendapatkan semua content, yang ada di setiap subdomain, meg / results all-front-index

Argumen / digunakan untuk men-spesifikasikan path apa yang akan coba diakses, dalam kasus ini saya menggunakan /, contoh request yang akan terbuat adalah tokopedia.com/yang mana seharusnya akan mengarah ke front page index halaman domain tersebut.
Argumen results merupakan nama file yang berisikan list domain lengkap dengan protokol yang terdeteksi oleh httprope.
Argumen all-front-index, merupakan nama folder yang akan terbuat dan menyimpan semua content hasil dari request yang dilakukan oleh meg.

Setelah proses request menggunakan meg selesai, langsung saja saya berpindah direktori cd all-front-index, dan eksekusi gf urls, maka munculah semua string yang match dengan pattern regex url

Dengan mata yang semakin mengantuk, saya melihat ke baris demi baris berharap dewi fortuna berpihak, dan terlihatlah sesuatu yang menarik perhatian saya.

Dengan sedikit harapan, percobaan eksekusi XSS dimulai :

https://www.tokopedia.com/amp/find/<script>alert(1)</script> Fail
https://www.tokopedia.com/amp/find/"><img src onerror=alert(1)"</script> Fail
https://www.tokopedia.com/amp/find/{><img src onerror=alert(document.domain)} XSS Fired

Dengan menambahkan cruly brackets, filter content yang telah diterapkan di input form tersebut di-render, menjadi bagian dari halamat tersebut.

Seketika mata saya yang sebelumnya mengantuk berat, jadi bagaikan terkena boncabai..

Dari saat itu saya cukup rajin untuk melakukan uji coba ke beberapa scope tokopedia.

Untuk soal "Reward" di tokopedia memang tidak sebesar big company di hackerone dan Bugcrowd. But, money is the not only priority, maybe money's the motivation; but not one and only.

Catatan Perjalanan - Nicolas Julian

How to Recover Kubernetes Cluster After Cilium Network Policy Lockout - etcd Disaster Recovery Guide

Background: Our Kubernetes Cluster Setup

The Problem: Kubernetes Cluster Completely Locked Out

What Went Wrong With Cilium Network Policy

Understanding Cilium Clusterwide Network Policy Behavior

Kubernetes Cluster Troubleshooting: Identifying the Issue

Symptom 1: API Server Unreachable

Symptom 2: etcd Lost Quorum - No Leader

Complete Kubernetes Disaster Recovery Process

Step 1: Stop Cilium Agent to Disable Network Policies

Step 2: Attempt Cilium Uninstall

Step 3: Backup etcd Before Recovery (Critical Step)

Step 4: etcd Disaster Recovery - Force New Cluster

Step 5: Delete the Problematic Cilium Network Policy

Step 6: Restore etcd Cluster - Add Members Back

Step 7: Verify Complete Kubernetes Cluster Recovery

The Correct Cilium Network Policy Configuration

Kubernetes Network Policy Best Practices

1. Understand Cilium Policy Modes

2. Always Whitelist Critical Kubernetes Ports

3. Test Network Policies in Staging Environment

4. Prefer Namespaced Network Policies

5. Implement Kubernetes Monitoring and Alerts

6. Document etcd Disaster Recovery Procedures

Emergency Kubernetes Recovery Commands

etcd Snapshot Backup

etcd Health Check

etcd Force New Cluster (Last Resort)

Add etcd Member

Restore etcd from Snapshot

Frequently Asked Questions (FAQ)

Q: Will etcd --force-new-cluster delete my data?

Q: Can I recover without --force-new-cluster?

Q: Should I use Cilium clusterwide policies at all?

Q: What if I don't have an etcd backup?

Conclusion: Key Takeaways for Kubernetes Administrators

[Personal Takes] Network Storage: Principal

The Reality Check

1. Why Busy the Network if We Don't Have To?

2. How to Move Data Fast? Don't Move It at All

3. Fast Local Storage is Easy and Cheap. Fast Shared Storage is Hard and Expensive

4. Your Cluster Storage System is Only as Good as Its Network

5. Mixing Drive Capacities: Proceed with Extreme Caution

6. Don't Exceed 80% Capacity if You Want Nice Weekend Trips

7. We'll Figure It Out on the Fly

8. Don't Sabotage Your Beautiful Cluster with Terrible Frontend Networking

The Bottom Line

Expose Service In Your Baremetal without Public IP in Kubernetes with Cloudflare Tunnel

What is Cloudflare Tunnel

Let's just set it up!

1. Get your cloudflare tunnel token

2. Create a daemonset in your Kubernetes

3. Point your service

4. Lets try expose an app

Pros and Cons

Configuring WordPress Behind Ingress Nginx in Kubernetes with Cloudflare

1. Set Home Page

2. WordPress Configuration

3. Set Cloudflare SSL/TLS to Full/Strict

4. Ingress Configuration

5. DNS Configuration

Getting Secrets from HashiCorp Vault in Ansible Playbooks

Prerequisites

Create Your Ansible Playbooks

Run Your Ansible Playbooks

Menjalankan Windows Server Dengan Kubevirt

Latar Belakang

Persyaratan

1. Virtualisasi Diaktifkan

2. Penyimpanan Backend Yang Persisten

3. Pahami Apa yang Dilakukan Kubevirt

4. Menginstal KubeVirt dan Container Data Importer Engine

5. Plugin Virt Kubectl

Mari Buat Windows Kita

1. Unduh Image Base

2. Forward Service CDI Upload Proxy ke Local

3. Unggah Image Base Dengan Virt Plugin

4. Buat Spesifikasi Perangkat Windows Kita

5. Verifikasi VM Running State

Q: Will `etcd --force-new-cluster` delete my data?

Q: Can I recover without `--force-new-cluster`?