setting of Kafka

-

  • Number of Partitions >= Number of Instance
  • Transmission Warranty - Adjust to suit the Client's requirements
    • At Most Once Retransmission X, Duplicate deletion X
    • At Least Once: Retransmission O, Duplicate deletion X
    • Exactly Once: Retransmission O, Duplicate deletion O
    • Offset Commit: Adjust to suit the Client's requirements
      • Auto Offset Commit
      • Manual Offset Commit
      • commitSync
      • commitAsync
    • Offset is managed by Consumer Group
      • one message can be consumed to the same value in another Consumer group, but only once within the same Consumer Group.
    • Producer processing method
      • Synchronous
      • Asynchronous
    • server.properties
      • unclean.leader.election.enable
        • True: Expelled from the ISR, but automatically becomes the leader when it comes to life first.
        • False: Waiting for the last leader to come to alive
      • Enable connectivity only from internal clients
        • listeners = PLAINTEXT://{private IP}:9092
        • advertised.listeners= PLAINTEXT://{private IP}:9092
      • Enable connectivity only from external clients
        • listeners=PLAINTEXT://0.0.0.0:9092
        • advertised.listeners= PLAINTEXT://{public IP}:9092
        • need to change bootstrap server address for consumer.properties and producer.properties

Comments

Post a Comment

Popular posts from this blog

[Kubernetes] Node Overcommitted

-
Image
Recap on Requests and Limits Requests Pods are allocated to nodes based solely on the specific CPU and Memory requirements. In other words, a pod is assigned for a node if the available capacity on that node can accommodate the additional CPU and Memory requested by the pod.  Limits Pod Scheduling does not take limits into account, but limits play a crucial role for different purposes. They establish an upper limit on the resources a pod can acquire before facing throttling or running out of memory. This precautionary measure prevents a single pod from monopolizing all resources on a node due to any deficiencies.  it is crucial to understand that reaching the CPU limit does not result in pod eviction; instead, it leads to throttling, potentially causing application failure. However, surpassing the Memory limit results in an out-of-memory (OOM) condition, necessitating pod eviction.  [example of overcommit] Now, if the node is anywhere close to reservation capacity then naturally the su
Read more

[AWS] What is the Cloud Formation?

-
Image
  What is AWS CloudFormation? AWS CloudFormation is a service that helps you model and set up your Amazon Web Services resources so that you can spend less time managing those resources and more time focusing on your applications that run in AWS. You create a template that describes all the AWS resources that you want (like Amazon EC2 instances or Amazon RDS DB instances), and AWS CloudFormation takes care of provisioning and configuring those resources for you. AWS CloudFormation Basics: In CloudFormation, you work with Templates and Stacks. Template:  JSON or YAML formatted text file which is a blueprint of your AWS resources. Stack:  In CloudFormation, you manage resources as a single unit called a Stack. All the resources in a stack are defined by the stack’s Template. Hence the AWS CloudFormation workflow can be summarized as the below image  reference: docs.aws.amazon.com. example of cloud formation configuration which is I made it by myself.
Read more

How to Install jenkins using Helm chart on instance

-
How to install Jenkins using Helm Chart on instance Prerequisites To follow along with this post you need to use a Kubernetes cluster and Helm chart. All major cloud provides offer hosted Kubernetes clusters: 1. AWS: EKS 2. Azure: AKS 3. Google Cloud: GKE you must also have the Helm Client installed. The Helm documentation provides install instructions. [Helm documentation: https://helm.sh/docs/intro/install/ ] we also have a way to deploy Jenkins on your local machine, but we will not explain it on this post.  Adding the Jenkins Chart repository $ helm repo add jenkins https://charts.jenkins.io $ helm repo update $ helm repo list Deploying a simple Jenkins instance To deploy a jenkins instance with the default settings, run this command: $ helm upgrade --install jenkins(jenkins_name) jenkins/jenkins  its the easiest way to deploy jenkins with default values. but I will introduce how to deploy Jenkins to LoadBalancer type. I will share the Helm-Jenkins yaml file: ----------------------
Read more