Apache Kafka® with custom configurations
Deploy an Apache Kafka® node to Aiven for Apache Kafka® with custom configurations defined in Terraform. A part of Aiven's Terraform Cookbook.
This example deploys an Aiven for Apache Kafka® service with some custom configurations, as well as a Kafka topic, a Kafka user, and an access control list (ACL) to allow fine-grained permissions about which topic that user can access, using the Aiven Terraform Provider
Before looking at the Terraform script, let's visualize the resources:
Be sure to check out the getting started guide to learn about the common files required to execute the following recipe. For example, you'll need to declare the variables for project_name
, api_token
, and kafka_user_name
.
Common files
Navigate to a new folder and add the following files.
Add the following to a new provider.tf
file:
terraform { required_providers { aiven = { source = "aiven/aiven" version = ">=4.0.0, < 5.0.0" } } } provider "aiven" { api_token = var.aiven_api_token }
You can also set the environment variable TF_VAR_aiven_api_token
for the api_token
property. With this, you don't need to pass the -var-file
flag when executing Terraform commands.
To avoid including sensitive information in source control, the variables are defined here in the variables.tf
file. You can then use a *.tfvars
file with the actual values so that Terraform receives the values during runtime, and exclude it.
The variables.tf
file defines the API token, the project name to use, and the prefix for the service name:
variable "aiven_api_token" { description = "Aiven console API token" type = string } variable "project_name" { description = "Aiven console project name" type = string } variable "kafka_user_name" { description = "Username for Aiven for Apache Kafka user" type = string }
The var-values.tfvars
file holds the actual values and is passed to Terraform using the -var-file=
flag.
var-values.tfvars
file:
aiven_api_token = "<YOUR-AIVEN-AUTHENTICATION-TOKEN-GOES-HERE>" project_name = "<YOUR-AIVEN-CONSOLE-PROJECT-NAME-GOES-HERE>" kafka_user_name = "<A-SAMPLE-USERNAME>"
Services.tf file
Here is the sample Terraform script to stand-up Aiven for Apache Kafka and related resources. The script also performs some custom configurations on these resources.
services.tf
file:
resource "aiven_kafka" "demo-kafka" { project = var.project_name cloud_name = "google-europe-west1" plan = "business-4" service_name = "demo-kafka" maintenance_window_dow = "sunday" maintenance_window_time = "01:00:00" default_acl = false kafka_user_config { kafka_rest = true kafka_connect = false schema_registry = true kafka_version = "3.4" kafka { auto_create_topics_enable = true num_partitions = 3 default_replication_factor = 2 min_insync_replicas = 2 } kafka_authentication_methods { certificate = true } public_access { kafka_rest = true } } } resource "aiven_kafka_topic" "demo-kafka-topic" { project = var.project_name service_name = aiven_kafka.demo-kafka.service_name topic_name = "demo-kafka-topic" partitions = 5 replication = 3 } resource "aiven_kafka_user" "demo-kafka-user" { project = var.project_name service_name = aiven_kafka.demo-kafka.service_name username = var.kafka_user_name } resource "aiven_kafka_acl" "demo-kafka-user-acl" { project = var.project_name service_name = aiven_kafka.demo-kafka.service_name username = var.kafka_user_name permission = "read" topic = aiven_kafka_topic.demo-kafka-topic.topic_name }
Execute the files
The init
command performs several different initialization steps in order to prepare the current working directory for use with Terraform. In our case, this command automatically finds, downloads, and installs the necessary Aiven Terraform provider plugins.
terraform init
The plan
command creates an execution plan and shows you the resources that will be created (or modified) for you. This command does not actually create any resource; this is more like a preview.
terraform plan -var-file=var-values.tfvars
If you're satisfied with the output of terraform plan
, go ahead and run the terraform apply
command which actually does the task or creating (or modifying) your infrastructure resources.
terraform apply -var-file=var-values.tfvars
Let's go over a few of these configurations and understand their functions:
aiven_kafka
resource configurations:
-
default_acl
parameter, when set to true, creates default wildcard Kafka ACL. This example sets this parameter to false and prevents the default wildcard ACL for resources. -
For
kafka_user_config
,schema_registry
is set to true, which enables theKarapace Schema Registry <https://aiven.io/blog/what-is-karapace>
_ andkafka_rest
allows you to view the messages in the topics from the Aiven web console when set to true. -
auto_create_topics_enable
underkafka
nested configurations enables the auto creation of topics when set to true. This means that a topic doesn't need to exist before sending a message. -
num_partitions
will set the number of partitions for the automatically created topics. -
By default, the replication factor is 2. This example sets
default_replication_factor
to 2 and thus requires a minimum of two brokers. For production environments, a replication factor of 3 is recommended. -
min_insync_replicas
indicates that at least 2 replicas (brokers) should respond back if all replicas(brokers) are not functioning properly. When all replicas are functioning properly, this setting has no effect. -
The
certificate
parameter under thekafka_authentication_methods
nested configurations, when set to true, enables certificate/SSL authentication.
aiven_kafka_topic
resource configurations:
partitions
denotes the number of partitions to create in the topic, andreplication
sets the replication factor for the topic.
aiven_kafka_user
resource configurations:
- We are passing a preset username using
var.kafka_user_name
aiven_kafka_acl
resource configurations:
- This ACL allows read access to the
demo-kafka-topic
topic for thevar.kafka_user_name
user.
Warning
By default, Aiven adds anavnadmin
account to every new service and adds admin
permission for all topics to that user. When you create your own ACLs to restrict access, you probably want to remove this ACL entry.Note
When using the Aiven Terraform Provider, you can add thedefault_acl
key to your resource
and set it to false
if you do not want to create the admin user with wildcard permissions.More resources
Keep in mind that some parameters and configurations will vary for your case. Some related resources are provided below: