Originally post in CNCF.

The CNCF Technical Oversight Committee (TOC) has voted to accept KubeVela as a CNCF incubating project.

KubeVela is an application delivery engine built with the Kubernetes control plane that makes deploying and operating applications across hybrid and multi-cloud environments easier, faster, and more reliable. KubeVela can orchestrate, deploy, and operate application components and cloud resources with a workflow-based application delivery model. The application delivery abstraction of KubeVela is powered by the Open Application Model (OAM).

The KubeVela project evolved from oam-kubernetes-runtime and developed with bootstrap contributions from more than eight different organizations, including Alibaba Cloud, Microsoft, Upbound and more. It was publicly announced as an open source project in November 2020, released as v1.0 in April 2021, and accepted as a CNCF sandbox project in June 2021. The project has more than 260 contributors and committers across multiple continents from organizations like Didi, JD.com, JiHu GitLab, SHEIN, and more.

“KubeVela pioneered a path for delivering applications across multi-cloud/multi-cluster environments with unified yet extensible abstractions,” said Lei Zhang, CNCF TOC sponsor. “This innovation unlocked the next-generation software delivery experience and filled the ‘last mile’ gap in existing practices which focus on the ‘deploy’ stage rather than ‘orchestrating’. We are excited to welcome more app-centric tools/platforms in the CNCF community and look forward to watching the adoption of KubeVela grow to a new level in the fast-growing application delivery ecosystem.”

KubeVela is used in production today by multiple companies across all major public clouds and on-premises deployments. Most users are adopting KubeVela as their internal “PaaS”, as part of their CI/CD pipeline, or as an extensible DevOps kernel for building their own IDP. Public adopters include Alibaba, which uses KubeVela as the core to deliver and manage applications across hybrid environments; Bytedance, which uses KubeVela and Crossplane to provide advanced Gaming PaaS abilities; China Merchants Bank, which uses KubeVela to build a hybrid cloud application platform to unify the whole process from build, ship and run; and many more.

“We’ve found capabilities such as gateway, security, and observability are being standardized with the emergence of corresponding open source tools and cloud services,” said Fang SITU, leader of aPaaS and Serverless team at Alibaba Cloud. “These capabilities can be integrated so that developers can self-serve, easily configure, automatically trigger, and get faster feedback, thereby greatly improving the application development efficiency. KubeVela is very suitable for the integration and customization of this kind of application delivery process, and it is the best practice of Platform Engineering.”

“KubeVela enables China Merchants Bank to quickly establish a large-scale unified OAM cloud native application management platform, reducing the complexity of cloud for FinTech and accelerating the standardized development and delivery of modern applications,” said Jiahang Xu, Senior Architect at China Merchants Bank and KubeVela maintainer. “KubeVela provides an application model and programmable CRD, workflow orchestration application delivery, observability, and configuration capabilities. KubeVela empowers cloud native applications along with the CNCF ecosystem.”

Main Components:

• Vela Core is the main component of KubeVela, also known as the KubeVela Control Plane. It provides operators and webhooks for rendering, orchestrating and delivering OAM applications.
• The Vela Workflow engine translates CUE-based steps and executes them. It’s a common library that can work as a standalone engine or run inside a KubeVela application.
• KubeVela CLI provides various commands that help you to operate applications, such as managing definitions, viewing resources, restarting workflow, and rolling versions.
• VelaUX is the Web UI for KubeVela. It incorporates business logic into fundamental APIs and provides out-of-box user experiences for non-k8s-expert users.
• The Terraform Controller in KubeVela allows users to use Terraform to manage cloud resources through Kubernetes Custom Resources.
• The Cluster Gateway provides a unified multi-cluster access interface.
• KubeVela also has a growing catalog with more than 50 community add-ons for integrations, including ArgoCD, FluxCD, Backstage, OpenKruise, Dapr, Crossplane, Terraform, OpenYurt and more.

Notable Milestones:

• >4.7k GitHub Stars
• >3.5k pull requests
• >1.6k issues
• >290 contributors
• >150 Releases

Since joining the CNCF Sandbox, KubeVela has released seven minor versions to v1.7 and added five new components, including standalone workflow, VelaUX, ClusterGateway, VelaD, and Vela Prism. The number of contributors has tripled from 90+ to 290+, GitHub stars have doubled from 1900+ to 4700+, and contributing organizations have tripled from 20+ to 70+.

“KubeVela improved the developer experience when it comes to complex multi-cloud environments with their modern open source software delivery control plane,” said Chris Aniszczyk, CNCF CTO. “We look forward to supporting the community in its growth and maturity towards a graduated project.”

Looking forward, the community plans to improve the user experience for cloud resource provisioning and consumption with delivery workflow, enhance the security for the whole CI/CD delivery process in hybrid/multi-cluster scenarios, support the KubeVela Dynamic API that allows users to make integration with third-party APIs easily, and more. Visit the roadmap to learn more.

“We’re humbled and grateful for the trust and support from users and contributors,” said Jianbo Sun, Staff Engineer at Alibaba Cloud and KubeVela maintainer. “KubeVela’s highly extensible design is very suitable for the diverse user scenarios of the community, bringing us a powerful engine to the ecosystem for application delivery. Thanks to the support and endorsement from CNCF, I believe that reaching incubation status is an important milestone for the project. The KubeVela maintainers look forward to collaborating with CNCF in our goals to make deploying and operating applications across today’s hybrid environments easier, faster and more reliable.”

“Thanks to Kubevela, the way we deploy and manage applications in Kubernetes has now become more accessible,” Daniel Higuero, CTO at Napptive and KubeVela maintainer. “The use of Application and Workflow as top-level entities greatly simplifies common processes on top of Kubernetes. The strength of the approach lies in its ability to simplify basic use cases while enabling complex ones with multi-tenant and multi-cluster support. This is combined with an inherently extensible system which supports community add-ons, allowing it to integrate with other tools and add custom definitions to tailor the experience to your use case.”

“The CNCF community has incubated a large number of cloud native operation and atomic management capabilities,” said QingGuo Zeng, Technical Expert at Alibaba Cloud and KubeVela maintainer. “We hope to integrate various capabilities through a unified, application-centric concept and help more and more platform developers easily implement standardized applications in enterprises. KubeVela is growing into a powerful helper for enterprises to practice Platform Engineering.”

“KubeVela aims to provide convenience and advancements of the rich cloud native infrastructures to various industries,” said Da Yin, Senior Engineer at Alibaba Cloud and KubeVela maintainer. “To meet the modern application delivery demands, KubeVela is always exploring extensible and flexible architecture and adding pioneering ideas, including multi-cluster delivery, programmable workflow, and automated observability. KubeVela also continuously cares for the security and stability of the control plane, which builds up production confidence for community adopters. We expect the openness of KubeVela could make it a frontier explorer in the cloud native era.”

As a CNCF-hosted project, KubeVela is part of a neutral foundation aligned with its technical interests and the larger Linux Foundation, which provides governance, marketing support, and community outreach. The project joins 35 other incubating technologies, including Backstage, Cilium, Istio, Knative, OpenTelemetry, and more. For more information on maturity requirements for each level, please visit the CNCF Graduation Criteria.

Serverless Application Engine (SAE) is a Kubernetes-based cloud product that combines the Serverless architecture and the microservice model. As an iterative cloud product, it has encountered many challenges in the process of rapid development. How can we solve these challenges in the booming cloud-native era and perform reliable and fast upgrades for architecture? The SAE team and the KubeVela community worked closely to address these challenges and came up with a replicable open-source solution, KubeVela Workflow.

This article describes how to use KubeVela Workflow to upgrade the architecture of SAE and interprets multiple practice scenarios.

Challenges in the Serverless Era​

SAE is an application hosting platform for business application architecture and microservices. It is a Kubernetes-based cloud product that combines the Serverless architecture and the microservice model.

As shown in the preceding architecture diagram, SAE users can host multiple types of applications on SAE. At the underlying layer of SAE, the Java business layer processes the relevant business logic and interacts with Kubernetes resources. At the bottom, it relies on highly available, O&M-free, and pay-as-you-go elastic resource pools.

In this architecture, SAE mainly relies on its Java business layer to provide users with capabilities. This architecture helps users easily deploy applications. Still, at the same time, it brings some challenges as well.

With the continuous development of Serverless, SAE has encountered three major challenges:

1. Engineers in SAE have some complex and non-standardized operations. How can we automate these complex operations to reduce development consumption?
2. With the development of business, the application-delivering capability of SAE is favored by a large number of users. The growth of users also brings efficiency challenges. How can we optimize the existing delivery capability and improve efficiency in high concurrency?
3. As Serverless architecture continues to be implemented in enterprises, numerous of cloud vendors are making their product systems serverless. In such a wave, how can SAE quickly connect with internal Serverless capabilities and reduce development costs while launching new features?

The preceding three challenges show that SAE needs some kind of orchestration engine to upgrade the delivery process, integrate with internal capabilities, and automate operations.

So, what does this orchestration engine need to meet to solve these challenges?

1. High Scalability: The steps in the process need to be highly scalable for this orchestration engine. Only in this way can the originally non-standardized and complex operations be standardized with the steps. In the meanwhile, the steps can combine with the process control capability of the orchestration engine, thus reducing the consumption of development resources.
2. Lightweight and Efficient: This orchestration engine must be efficient and production-ready, so as to meet the high concurrency requirements of SAE in large-scale user scenarios.
3. Strong Integration and Process Control Capabilities: This orchestration engine needs to be able to quickly integrate with the atomic functions in the company. If the glue code that connects upstream and downstream capabilities can be transformed into the process in the orchestration engine, development costs can be reduced.

Based on these challenges and considerations, the SAE team and the KubeVela community have conducted in-depth cooperation and launched the KubeVela Workflow project as an orchestration engine.

Why Use Kubevela Workflow?​

Thanks to the booming ecological of cloud-native, there are already many mature workflow projects in the community (such as Tekton, Argo, etc.). There are also some orchestration engines within Alibaba Cloud. So, why reinvent the wheel instead of using existing technology?

This is because KubeVela Workflow has a fundamental difference in design. The steps in the workflow are designed for the cloud-native IaC system and support abstract encapsulation and reuse, which means that you can use atomic capabilities like a function call in every step, instead of just creating pods or containers.

In KubeVela Workflow, each step has a step type, and each step type corresponds to the resource callend WorkflowStepDefinition. You can use the CUE language (an IaC language, which is a superset of JSON) to write this step definition or directly use the step type defined in the community.

You can simply regard a WorkflowStepDefinition as a function declaration. Each time a new step type is defined, a new function is defined. The function requires some input parameters, and the step definition is the same. In the step definition, you can define the input parameters required for this step type in the parameter field. When the workflow is running, the workflow controller executes the CUE code in the corresponding step definition using the actual parameter values input by the user, just as it executes your custom function.

With such a layer of abstraction of the steps, it adds huge possibilities to the steps:

• If you want to customize the step type (just like writing a new function), you can import the official code packages in the step definition, thus introducing other atomic capabilities into the step, including HTTP requests, CRUD resources in multiple clusters, conditional waiting, etc. With such a programmable step type, you can easily integrate with any system. For example, in the SAE scenario, the step definition is integrate with other internal atomic capabilities (such as MSE, ACR, ALB, SLS, etc.). Then, the workflow orchestration capability is used to control the process.

• If you only want to use defined steps, just like calling a packaged third-party function, the only thing you need to care about is your input parameters and the step type you want. For example, in a typical image-building scenario, first, specify the step type as build-push-image and then specify your input parameters: the code source and branch of the built image, the name of the built image, and the secret key of the image repository to push.

apiVersion: core.oam.dev/v1alpha1
kind: WorkflowRun
metadata:
  name: build-push-image
  namespace: default
spec:
  workflowSpec:
   steps:
    - name: build-push
      type: build-push-image
      properties:
        context:
          git: github.com/FogDong/simple-web-demo
          branch: main
        image: fogdong/simple-web-demo:v1
        credentials:
          image:
            name: image-secret

In such an architecture, the abstraction of the step brings infinite possibilities to the workflow. When you need to add a step to the workflow, you no longer need to compile-build-package the business code and then use the pod to execute the code. You only need to modify the configuration code in the step definition (together with the workflow engine's orchestration and control capability) to integrate with new features.

This is also the main reason why SAE chooses Kubevela Workflow. Based on scalability, we can fully leverage the power of ecology and accelerate product upgrades.

Cases​

Next, let's go deeper into the user cases in SAE.

Case 1: Automated Operations​

The first scenario is an automated operations scenario for SREs in SAE.

In SAE, we write and update some base images for users. We need to preload these images to multiple clusters in different regions to provide a better experience for users who use these base images.

The original operation process is very complicated. It involves building images and pushing them across multiple regions using ACR, as well as creating image cache templates and managing those image caches. Regions here include Shanghai, US West, Shenzhen, and Singapore. These operations are non-standardized and time-consuming. This is because when an SRE needs to push these images from the local to foreign clusters, it is likely to fail due to network problems. Therefore, he needs to disperse his energy on these operations that could otherwise be automated.

This is also the scenario that KubeVela Workflow is suitable for: each step in these operations can be converted into a step in the workflow programmatically, so that these steps can be orchestrated and reused. In addition, KubeVela Workflow provides a visual dashboard. SREs only need to configure a pipeline template once, and can automate the process by triggering execution or by entering specific runtime parameters each time the execution is triggered.

The simplified steps is like below:

1. Use the HTTP request step type to build an image by requesting the service of ACR and pass the image ID to the next step through inputs/outputs. In this step definition, you need to wait until the ACR service is done before ending the execution of the current step.
2. If the first step fails, execute the error handle step.
3. If the first step is successfully built, use the HTTP request step to build image cache, at the same time, the service logs are used as the source of the current step. Here you can view the logs of the steps directly in the Dashboard to troubleshoot problems.
4. Use a step group with deploy step type to preload images for clusters in the China (Shanghai) region and the US (West) region. The multi-cluster management and control capabilities of KubeVela Workflow are used to distribute the ImagePullJob workload to multiple clusters to preload images.

In the preceding process, if you do not use KubeVela Workflow, you may need to write a bunch of business code to connect multiple services and clusters. Let’s take the last step of distributing the ImagePullJob workload to multiple clusters as an example: Not only do you need to manage the configuration of multiple clusters, but also need to watch the status of the workload (CRD) until the status of the workload becomes Ready before proceeding to the next step. This process actually corresponds to a simple Kubernetes Operator's reconcile logic: first create or update a resource, if the status of the resource is as expected, then end the reconcile, if not, continue to wait.

Do we need to implement an Operator for every new resource management in our operations? Is there any convenient way to free us from the complicated Operator development?

It is precisely because of the programmability of steps in KubeVela Workflow that it can completely cover these operations and resource management in SAE scenarios, which can help engineers reduce manpower consumption. Similar to the above logic, the step definition corresponding to KubeVela Workflow is very simple. No matter what kind of resources (or a HTTP interface request), it can be covered by a similar step template like:

template: {
  // First, read resources from the specified cluster
    read: op.#Read & {
    value: {
        apiVersion: parameter.apiVersion
        kind: parameter.kind
        metadata: {
        name: parameter.name
        namespace: parameter.namespace
        }
    }
    cluster: parameter.cluster
    }
    // Second, wait until the resource is Ready. Otherwise, the step keeps waiting
    wait: op.#ConditionalWait & {
    continue: read.value.status != _|_ && read.value.status.phase == "Ready"
    }
    // Third(optional), if the resource is Ready, then...
    // Custom Logic...
    
    // Users must input the defined parameter when using the step type
    parameter: {
    apiVersion: string
    kind: string
    name: string
    namespace: *context.namespace | string
    cluster: *"" | string
    }
}

Corresponding to the current case is:

• First: Read the status of ImagePullJob in a specified cluster, such as the cluster in the China (Shanghai) region.
• Second: If the ImagePullJob is ready and the image has been preloaded, continue to execute.
• Third(Optional): After the ImagePullJob is ready, clean up the ImagePullJob in the cluster.

Like this, no matter how many Region clusters or new type resources are added in the subsequent O&M scenarios, you can let KubeVela Workflow manage the cluster's KubeConfig, and use the defined step types with different cluster names or resource types to achieve a simplified Kubernetes Operator Reconcile process to reduce development costs.

Case 2: Optimize the Existing Delivery Process​

In addition to automating internal O&M operations, upgrading the original SAE product architecture to improve delivery efficiency for users is also an important reason of choosing KubeVela Workflow.

Original Architecture​

In the delivery scenario of SAE, a delivery process corresponds to a series of tasks, such as: initializing the environment, building images, releasing in batches, etc. This series of tasks corresponds to the SAE Tasks in the figure below.

These tasks are sequentially thrown to Java Executor for business logic in the original architecture of SAE, such as creating resources in Kubernetes, and synchronizing the status of current tasks with the MySQL database, etc.

After the current task is completed, JAVA Executor will get the next task from SAE's original orchestration engine, and at the same time, the orchestration engine will continuously put new tasks into the initial task list.

The biggest problem in this old architecture is the polling call. JAVA Executor will get it from the SAE task list every second to check whether there are new tasks; at the same time, after JAVA Executor creates Kubernetes resources, it will attempts to get the status of resources from the cluster every second.

The original architecture of SAE is not the controller model in the Kubernetes ecosystem, but the polling model. If the orchestration engine layer is upgraded to the controller model for event watching, it can better integrate with the entire Kubernetes ecosystem and improve efficiency.

However, the logic coupling of business is deep in the original architecture. If the traditional container-based cloud-native workflow is used, SAE needs to package the original business logic into images and maintain and update a large number of images, which is not a sustainable path. We hope the upgraded workflow engine can be easily integrated with the task orchestration, business execution layer, and Kubernetes cluster.

New Architecture​

With the high scalability of KubeVela Workflow, SAE engineers do not need to repackage the original capabilities into images or make large-scale modifications.

The new process is shown in the preceding figure. After a delivery paln is created on the SAE product side, the business side writes the model to the database, converts the model, and generates a KubeVele workflow, which corresponds to YAML on the right side.

At the same time, SAE's original Java Executor provides the original business capabilities as microservice APIs. When KubeVela Workflow is executed, each step is IaC-based, and the underlying implementation is in the CUE language. Some of these steps will call the business microservice API of SAE, while others will directly interact with the underlying Kubernetes resources. Data can be transferred between steps. If there is an error in the call, you can use the conditional judgment of the step to handle the error.

Such optimization is scalable and fully reuses the Kubernetes ecosystem. It extends workflow processes and atomic capabilities and is oriented to the final state. This combination of scalability and process control can cover the original business functions and reduce the amount of development. At the same time, the state update latency is reduced from the minute level to the millisecond level, which is agile and native. It has the YAML-level description capability but also improves the development efficiency from 1d to 1h.

Case 3: Launch New Features Quickly​

In addition to automated O&M and upgrading the original architecture, what else can KubeVela Workflow provide?

The reusability of steps and the ease of integration with other ecosystems bring more surprises to SAE in addition to upgrades: from writing business code to orchestrating different steps, so as to launch new product features quickly!

SAE has accumulated a large amount of JAVA foundation and supports a wealth of functions, such as: supporting single-batch release, multi-batch release, and canary release for JAVA microservices. However, with the increase of customers, some customers have put forward new requirements, and they hope to have the ability to publish multilingual north-south traffic in canary release.

There are also many mature open-source products for canary releases, such as Kruise Rollout. After investigation, SAE engineers found that Kruise Rollout can be used to complete the ability of canary release, and it can cooperate with the internal ingress controller(ALB) of Alibaba Cloud to split different traffic.

Such a solution is shown in the architecture diagram above. SAE distributes a KubeVela Workflow, and the steps in the Workflow will integrate with Alibaba Cloud ALB, open-source Kruise Rollout, and SAE's business components at the same time. Batch management is completed in the steps, thus completing the rapid launch of features.

In fact, after using KubeVela Workflow, it is no longer necessary to write new business codes for this feature. It is only necessary to write a step type for updating canary batches.

Due to the programmability of step types, we can easily use different patch policies in the definition to update the release batches of Rollout objects in the different clusters. Moreover, in the workflow, the step type of each step is reusable. This means when you develop a new step type, you are laying the foundation for the next time a new feature is launched. This reuse allows you to launch features quickly and reduce development costs.

Summary​

After SAE upgraded the KubeVela architecture, it improved delivery efficiency and reduced development costs. Based on the advantages of its underlying reliance on Serverless infrastructure, it can give full play to the advantages of the product in application hosting.

In addition, the architecture upgrade solution of KubeVela Workflow in SAE is an open-source solution that can be replicated. The community provides more than 50 built-in step types (including image building, image pushing, image scanning, and multi-cluster deployment, using Terraform to manage infrastructure, condition wait, message notification, etc) to help you open up CI/CD easily.

You can refer to the following documents for more usage scenarios:

• Build images, push images, and deploy resources
• Orchestrate multiple KubeVela Applications
• Initialize the Environment with One Click, use Terraform to pull up clusters, manage clusters, and distribute resources to new clusters
• Call the specified service and send a notification of the returned result through data passing
• Use different context parameters to control the deployment of resources

The End​

You can learn more about KubeVela and the OAM project through the following materials:

• Project Code Library: https://github.com/kubevela/kubevela Welcome to Star/Watch/Fork!
• Workflow Code Library: https://github.com/kubevela/workflow Welcome to Star/Watch/Fork!
• Official Project Homepage and Documents: kubevela.io
• Slack：CNCF #kubevela Channel

This article will focus on KubeVela and OpenYurt (two open-source projects of CNCF) and introduce the solution of cloud-edge collaboration in a practical Helm application delivery scenario.

Background​

With the popularization of the Internet of Everything scenario, the computing power of edge devices is increasing. It is a new technological challenge to use the advantages of cloud computing to meet complex and diversified edge application scenarios and extend cloud-native technology to the end and edge. Cloud-Edge Collaboration is becoming a new technological focus. This article will focus on KubeVela and OpenYurt (two open-source projects of CNCF) and introduce the solution of cloud-edge collaboration in a practical Helm application delivery scenario.

OpenYurt focuses on extending Kubernetes to edge computing in a non-intrusive manner. Based on the container orchestration and scheduling capabilities of native Kubernetes, OpenYurt integrates edge computing power into the Kubernetes infrastructure for unified management. It provides capabilities (such as edge autonomy, efficient O&M channels, unitized edge management, edge traffic topology, secure containers, and edge Serverless/FaaS) and support for heterogeneous resources. In short, OpenYurt builds a unified infrastructure for cloud-edge collaboration in a Kubernetes-native manner.

Incubated in the OAM model, KubeVela focuses on helping enterprises build unified application delivery and management capabilities. It shields the complexity of underlying infrastructure for developers and provides flexible scaling capabilities. It also provides out-of-the-box microservice container management, cloud resource management, versioning and canary release, scaling, observability, resource dependency orchestration and data delivery, multi-cluster, CI docking, and GitOps. Maximize the R&D performance of developer self-service application management, which also meets the extensibility demands of the long-term evolution of the platform.

OpenYurt + KubeVela - What Problems Can be Solved?​

As mentioned before, OpenYurt supports the access of edge nodes, allowing users to manage edge nodes by operating native Kubernetes. "Edge nodes" are used to represent computing resources closer to users (such as virtual machines or physical servers in a nearby data center). After you add them through OpenYurt, these edge nodes are converted into nodes that can be used in Kubernetes. OpenYurt uses NodePool to describe a group of edge nodes in the same region. After basic resource management is met, we have the following core requirements for how to orchestrate and deploy applications to different NodePools in a cluster.

1. Unified Configuration: If you manually modify each resource to be distributed, there are many manual interventions, which are prone to errors. We need a unified way to configure parameters, which can not only facilitate batch management operations, but also connect security and audit to meet enterprise risk control and compliance requirements.
2. Differential Deployment: Most workloads deployed to different NodePools have the same attributes, but there are still personalized configuration differences. The key is how to set the parameters related to node selection. For example, NodeSelector can instruct the Kubernetes scheduler to schedule workloads to different NodePools.
3. Scalability: The cloud-native ecosystem is booming. Both workload types and different O&M functions are growing. We need the overall application architecture to be scalable so that we can fully benefit from the dividends of the cloud-native ecosystem and meet business needs flexibly.

KubeVela and OpenYurt can complement each other at the application layer to meet the preceding three core requirements. Next, I will show these functions with the operation process.

Deploy an Application to the Edge​

We will use the Ingress controller as an example to show how to use KubeVela to deploy applications to the edge. We want to deploy the Nginx Ingress controller to multiple NodePools to access the services provided by the specified NodePool through the edge Ingress. An Ingress can only be handled by the Ingress controller in the NodePool.

The cluster in the schematic diagram contains two NodePools: Beijing and Shanghai. The networks between them are not interconnected. We want to deploy an Nginx Ingress Controller, which can act as the network traffic ingress for each NodePool. A client close to Beijing can access the services provided in the Beijing NodePool by accessing the Ingress Controller of the Beijing NodePool and does not access the services provided in the Shanghai NodePool.

Basic Environment of Demo​

We will use Kubernetes clusters to simulate edge scenarios. The cluster has three nodes, and their roles are:

• Node 1: master node, cloud node
• Node 2: worker node, edge node, in NodePool Beijing
• Node 3: worker node, edge node, in NodePool Shanghai

Preparations​

1. Install YurtAppManager​

YurtAppManager is the core component of OpenYurt. It provides NodePool CRD and controller. There are other components in OpenYurt, but we only need YurtAppManager in this tutorial.

git clone https://github.com/openyurtio/yurt-app-managercd yurt-app-manager && helm install yurt-app-manager -n kube-system ./charts/yurt-app-manager/

2. Install KubeVela and Enable the FluxCD Addon​

Install the Vela command-line tool and install KubeVela in the cluster:

curl -fsSl https://kubevela.net/script/install.sh | bash
vela install

We want to reuse the mature Helm charts provided by the community, so we use Helm-type components to install the Nginx Ingress Controller. In KubeVela with microkernel design, Helm components are provided by the FluxCD addon. The following enables the FluxCD addon.

vela addon enable fluxcd

3. Prepare NodePool​

Create two NodePools: Beijing and Shanghai. Dividing NodePools by region is a common pattern in actual edge scenarios. Different groups of nodes often have obvious isolation attributes (such as network disconnection, no resource sharing, resource heterogeneity, and application independence). This is the origin of the NodePool concept. In OpenYurt, features (such as NodePools and service topologies) are used to help users deal with the preceding issues. In today's example, we will use NodePools to describe and manage nodes:

kubectl apply -f - <<EOF
apiVersion: apps.openyurt.io/v1beta1
kind: NodePool
metadata:
  name: beijing
spec:
  type: Edge
  annotations:
    apps.openyurt.io/example: test-beijing
  taints:
    - key: apps.openyurt.io/example
      value: beijing
      effect: NoSchedule
---
apiVersion: apps.openyurt.io/v1beta1
kind: NodePool
metadata:
  name: shanghai
spec:
  type: Edge
  annotations:
    apps.openyurt.io/example: test-shanghai
  taints:
    - key: apps.openyurt.io/example
      value: shanghai
      effect: NoSchedule
EOF

Add edge nodes to their respective NodePools. Please see how OpenYurt nodes are added for more information about how edge nodes are added.

kubectl label node <node1> apps.openyurt.io/desired-nodepool=Beijing
kubectl label node <node2> apps.openyurt.io/desired-nodepool=shanghai

kubectl get nodepool

Expected Output

NAME       TYPE   READYNODES   NOTREADYNODES   AGE
beijing    Edge   1            0               6m2s
shanghai   Edge   1            0               6m1s

Deploy Edge Applications in Batches​

Before we get into the details, let's look at how KubeVela describes and deploys applications to the edge. With the following application, we can deploy multiple Nginx Ingress Controller to their respective edge NodePools. Using the same application to uniformly configure the Nginx Ingress can eliminate duplication and reduce the management burden. It facilitates unified operations (such as the release and other O&M of components in the cluster).

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: edge-ingress
spec:
  components:
    - name: ingress-nginx
      type: helm
      properties:
        chart: ingress-nginx
        url: https://kubernetes.github.io/ingress-nginx
        repoType: helm
        version: 4.3.0
        values:
          controller:
            service:
              type: NodePort
            admissionWebhooks:
              enabled: false
      traits:
        - type: edge-nginx
  policies:
    - name: replication
      type: replication
      properties:
        selector: [ "ingress-nginx" ]
        keys: [ "beijing","shanghai" ]
  workflow:
    steps:
      - name: deploy
        type: deploy
        properties:
          policies: ["replication"]

A KubeVela application has three parts.

1. Helm Type Component: It describes the version of the Helm package that we want to install into the cluster. In addition, we have attached an trait to this component: edge-nginx. We'll show the details of this trait later. You can regard it as a patch that contains the properties of different NodePools.
2. The Strategy of Replication: It describes how to copy components to different NodePools. The field selector is used to select the components to be copied. Its keys field will convert one component to two components with different keys ("Beijing" and "Shanghai").
3. Deploy Workflow Steps: It describes how to deploy an application. It specifies the replication strategy to perform replication work.

Now, we can send the application to the cluster.

vela up -f app.yaml

Check the application status and resources created by KubeVela.

vela status edge-ingress --tree --detail

Expected Output

CLUSTER       NAMESPACE     RESOURCE                           STATUS    APPLY_TIME          DETAIL
local  ─── default─┬─ HelmRelease/ingress-nginx-beijing  updated   2022-11-02 12:00:24 Ready: True  Status: Release reconciliation succeeded  Age: 153m
                   ├─ HelmRelease/ingress-nginx-shanghai updated   2022-11-02 12:00:24 Ready: True  Status: Release reconciliation succeeded  Age: 153m
                   └─ HelmRepository/ingress-nginx       updated   2022-11-02 12:00:24 URL: https://kubernetes.github.io/ingress-nginx  Age: 153m
                                                                                         Ready: True
                                                                                         Status: stored artifact for revision '7bce426c58aee962d479ca84e5c
                                                                                         fc6931c19c8995e31638668cb958d4a3486c2'

Vela CLI can stand on a higher level to uniformly show the health status of applications. When it's needed, Vela CLI can help you penetrate applications and direct access to the underlying workloads. Also, it offers a rich selection of observability and Debug capability. For example, you can print the application's log through vela logs. You can forward the port that deploys applications locally through vela port-forward. You can use the vela exec command to go deep into the container at the edge and run Shell commands to troubleshoot the problem.

If you need an intuitive understanding of the application, KubeVela officials provide a Web console addon, VelaUX. You can view detailed resource topologies with the VelaUX addon .

vela addon enable velaux

Visit the Resource Topology page of VelaUX:

As you can see, KubeVela creates two HelmRelease to deliver the Nginx Ingress Controller Helm chart to two NodePools. HelmRelease resources are processed by the preceding FluxCD addon, and NGINX Ingress is installed in the two NodePools of the cluster. Run the following command to check whether pods of the Ingress controller are created in the Beijing NodePool. The same is true for the shanghai NodePool.

$ kubectl get node -l  apps.openyurt.io/nodepool=beijing                               
NAME                      STATUS   ROLES    AGE   VERSION
iz0xi0r2pe51he3z8pz1ekz   Ready    <none>   23h   v1.24.7+k3s1

$ kubectl get pod ingress-nginx-beijing-controller-c4c7cbf64-xthlp -oyaml|grep iz0xi0r2pe51he3z8pz1ekz
  nodeName: iz0xi0r2pe51he3z8pz1ekz

Differentiated Deployment​

How can a differentiated deployment of the same component be implemented during KubeVela application delivery? Let's continue to learn about the Trait and Policy that support the application. As mentioned above, we use KubeVela's built-in component replication policy in the workflow to attach a custom edge-nginx trait to ingress-nginx components.

• The component splitting policy splits the component into two components with a different context.replicaKey.
• edge-nginx Trait uses different context.replicaKey to deliver Helm charts with different configuration values to the cluster. Let the two Nginx Ingress Controller run in different NodePools and monitor Ingress resources with different ingressClasses. Patch the values of the Helm chart and modify the fields related to Node Selection, Affinity, and ingressClass.
• Different patch strategies is used when patching different fields. For example, a retainKeys strategy can overwrite the original value, while a jsonMergePatch strategy is merged with the original value.

"edge-nginx": {
  type: "trait"
  annotations: {}
  attributes: {
    podDisruptive: true
    appliesToWorkloads: ["helm"]
  }
}
template: {
  patch: {
    // +patchStrategy=retainKeys
    metadata: {
      name: "\(context.name)-\(context.replicaKey)"
    }
    // +patchStrategy=jsonMergePatch
    spec: values: {
      ingressClassByName: true
      controller: {
        ingressClassResource: {
          name:            "nginx-" + context.replicaKey
          controllerValue: "openyurt.io/" + context.replicaKey
        }
        _selector
      }
      defaultBackend: {
        _selector
      }
    }
  }
  _selector: {
    tolerations: [
      {
        key:      "apps.openyurt.io/example"
        operator: "Equal"
        value:    context.replicaKey
      },
    ]
    nodeSelector: {
      "apps.openyurt.io/nodepool": context.replicaKey
    }
  }
  parameter: null
}

Let More Types of Applications Go to the Edge​

As you can see, we only customize a trait with more than 40 rows and make full use of the built-in capabilities of KubeVela to deploy Nginx Ingress to different NodePools. More applications are likely to be deployed at the edge with the prosperous cloud-native ecosystem and cloud-edge collaboration trend. When a new application needs to be deployed in the edge NodePool in the new scenario, there is no need to worry. KubeVela makes it easy to expand a new edge application deployment trait following this pattern without writing code.

For example, we hope to deploy the implementation of the Kubernetes community's recent evolution hotspot Gateway API to the edge as well. We can use the Gateway API to enhance the expressive capability and extensibility of the edge NodePool to expose services and use role-based network APIs on edge nodes. For this scenario, we can easily complete the deployment task based on the preceding extension method. You only need to define a new trait (as shown below):

"gateway-nginx": {
  type: "trait"
  annotations: {}
  attributes: {
    podDisruptive: true
    appliesToWorkloads: ["helm"]
  }
}

template: {
  patch: {
    // +patchStrategy=retainKeys
    metadata: {
      name: "\(context.name)-\(context.replicaKey)"
    }
    // +patchStrategy=jsonMergePatch
    spec: values: {
      _selector
      fullnameOverride: "nginx-gateway-nginx-" + context.replicaKey
      gatewayClass: {
        name:           "nginx" + context.replicaKey
        controllerName: "k8s-gateway-nginx.nginx.org/nginx-gateway-nginx-controller-" + context.replicaKey
      }
    }
  }
  _selector: {
    tolerations: [
      {
        key:      "apps.openyurt.io/example"
        operator: "Equal"
        value:    context.replicaKey
      },
    ]
    nodeSelector: {
      "apps.openyurt.io/nodepool": context.replicaKey
    }
  }
  parameter: null
}

This Trait is similar to the one used to deploy the Nginx Ingress mentioned earlier. We also made some similar patch for Nginx Gateway chart values, including node selection, affinity, and resource name. The difference between the former trait is that this Trait specifies the gatewayClass instead of the IngressClass. Please see the GitHub repository for more information about the trait and application files. We extend the ability of the cluster to deploy a new application to the edge by customizing such a Trait. If we cannot predict application deployment requirements brought about by the development of edge computing in the future, at least we can adapt to new scenarios through this scalability way.

How KubeVela Solves Edge Deployment Puzzles​

Let's review how KubeVela addressed the key issues raised at the beginning of the article.

1. Unified Configuration::A component is used to describe the common properties of the ingress-nginx Helm chart to be deployed (such as Helm repository, chart name, version, and other unified configurations).
2. Attribute Differences: KubeVela uses a user-defined trait that describes the differences in Helm configurations sent to different NodePools. This trait can be repeatedly used to deploy the same Helm Chart.
3. Scalability: KubeVela can be programmable for common workloads (such as Deployment/StatefulSet) or other packaging methods (such as Helm or Kustomize). It requires only a few lines of code to push a new application to the edge.

It benefits from the powerful functions provided by KubeVela in the application delivery and management field. In addition to solving application definition, delivery, O&M, and observability issues within a single cluster, KubeVela natively supports application release and management in the multi-cluster mode. Currently, the Kubernetes deployment mode suitable for edge computing scenarios is not fixed. KubeVela can manage application tasks regardless of the architecture of single cluster + edge NodePool or multi-edge cluster architecture.

With OpenYurt and KubeVela, cloud-edge applications are deployed in a unified manner and share the same abstraction, O&M, and observability, which avoids the experience of being separated in different scenarios. In addition, cloud applications and edge applications can take advantage of KubeVela's excellent practices of a continuously integrated cloud-native ecosystem in the form of addons. In the future, the KubeVela community will continue to enrich out-of-the-box system addons and deliver better and easier-to-use application delivery and management capabilities.

If you want to know more about the capabilities of application deployment and management, you can read KubeVela’s official documents. If you want to know the latest development in the KubeVela community, you are welcome to participate in the discussion! If you are interested in OpenYurt, you are welcome to join the OpenYurt community!

You can learn more about KubeVela and the OAM project through the following materials:

• Project Code Base: https://github.com/oam-dev/kubevela Welcome Star/Watch/Fork!
• Project Official Homepage and Documents: kubevela.io

This document aims to explain the integration of Kubevela and ArgoCD. We have two approaches to integrate this flow. This doc is trying to explain the pros and cons of two different approaches. Before diving deep into details, we can describe Kubevela and ArgoCD.

KubeVela is a modern software delivery platform that makes deploying and operating applications across multi environments easier, faster, and more reliable.

KubeVela is infrastructure agnostic and application-centric. It allows you to build robust software and deliver them anywhere! Kubevela provides an Open Application Model (OAM) based abstraction approach to ship applications and any resource across multiple environments.

Open Application Model (OAM) is a set of standard yet higher-level abstractions for modeling cloud-native applications on top of today’s hybrid and multi-cloud environments. You can find more conceptual details here.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: first-vela-app
spec:
  components:
    - name: express-server
      type: webservice
      properties:
        image: oamdev/hello-world
        ports:
         - port: 8000
           expose: true
      traits:
        - type: scaler
          properties:
            replicas: 1

ArgoCD​

Argo CD is a declarative continuous delivery tool for Kubernetes applications. It uses the GitOps style to create and manage Kubernetes clusters. When any changes are made to the application configuration in Git, Argo CD will compare it with the configurations of the running application and notify users to bring the desired and live state into sync.

Flux​

By default, Kubevela works with Flux addon to complete the Gitops journey for applications. You need to enable the flux addon and perform git operations quickly. https://kubevela.io/docs/end-user/gitops/fluxcd.

vela addon enable fluxcd

Integration with ArgoCD​

If you want to use Kubevela with ArgoCD, there are two options to enable it.

• Kubevela dry-run option
• Kubevela + ArgoCD Gitops syncer mode

1. Vela dry-run approach

First approach is that without using Kubevela GitOps Controller, we can use ArgoCD as the GitOps Controller, which means ArgoCD applies a raw manifest to the target cluster. Kubevela provides us to export oam application components and traits to native Kubernetes resource therefore we can still use OAM based model and ArgoCD can manage resources with custom argo-repo-server plugin. Behind this approach that KubeVela and OAM can also works as a client side tool with the feature dry-run.

This part describe how ArgoCD apply dry-run mode on Kubevela resources. Firstly, ArgoCD and Kubevela same should be installed.

ArgoCD version v2.5.4

Kubevela version v1.6.4

Prerequisites​

Argo CD allows integrating more config management tools using config management plugins.

1. Write the plugin configuration file​

Plugins will be configured via a ConfigManagementPlugin manifest located inside the plugin container.

apiVersion: v1
kind: ConfigMap
metadata:
  name: vela-cmp
  namespace: argocd
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: vela
    spec:
      version: v1.0
      init:
        command: ["vela", "traits"]
      generate:
        command: ["sh"]
        args:
          - -c
          - |
            vela dry-run -f test-argo-oam.yml
      discover:
        # fileName: "-oam.yml*"
        find:
          # This does the same thing as fileName, but it supports double-start (nested directory) glob patterns.
          glob: "**/*oam*.yml"

kubectl apply -f vela-cmp.yml

This plugin is responsible for converting Kubevela OAM manifest definition to raw kubernetes object and ArgoCD should be responsible to deploy.

2. Register the plugin sidecar​

To install a plugin, patch argocd-repo-server to run the plugin container as a sidecar, with argocd-cmp-server as its entrypoint.

Vela plugin runs the vela command to export manifest when the plugin is discovered on git manifest. Before initializing the plugin, we need to install Kubevela CLI to run the order successfully. The below configuration adds an init container to download the necessary CLI.

initContainers:
 - name: kubevela
   image: nginx:1.21.6
   command:
    - bash
    - '-c'
    - |
     #!/usr/bin/env bash
     set -eo pipefail
     curl -fsSl https://kubevela.io/script/install.sh | bash -s 1.6.4
   env:
    - name: VELA_INSTALL_DIR
      value: /custom-tools
   resources:
    limits:
     cpu: 50m
     memory: 64Mi
    requests:
     cpu: 10m
     memory: 32Mi
   volumeMounts:
    - name: custom-tools
      mountPath: /custom-tools
   terminationMessagePath: /dev/termination-log
   terminationMessagePolicy: File
   imagePullPolicy: IfNotPresent

after adding init container, we need to add our custom sidecar binary to run plugins properly. To use configmap plugin configuration in our sidecar, we need to mount configmap plugin to our pods

containers:
        - name: vela
          image: nginx
          command:
            - /var/run/argocd/argocd-cmp-server
          env:
            - name: KUBECONFIG
              value: /kubeconfig
          resources: {}
          volumeMounts:
            - name: var-files
              mountPath: /var/run/argocd
            - name: vela-cli-dir
              mountPath: /.vela
            - name: kubeconfig
              mountPath: /kubeconfig
              subPath: kubeconfig
            - name: plugins
              mountPath: /home/argocd/cmp-server/plugins
            - name: vela-cmp
              mountPath: /home/argocd/cmp-server/config/plugin.yaml
              subPath: plugin.yaml
            - name: cmp-tmp
              mountPath: /tmp
            - name: custom-tools
              mountPath: /usr/local/bin/vela
              subPath: vela

3. Configure Argo CD to watch this repo for Git pushes.​

Create git repository and put oam manifest to target directory after you can create a simple ArgoCD application that watch git repository and apply manifests to local kubernetes cluster.

apiVersion: core.oam.dev/v1beta1
kind: Application
metadata:
  name: test-2
spec:
  components:
    - name: wf-poc-app-2
      type: webservice
      properties:
        image: oamdev/helloworld-python:v1
        env:
          - name: "TARGET"
            value: "ArgoCD"
        port: 8080

4. Final Output​

ArgoCD synced applications successfully and rendered kubernetes resources.

2. Kubevela Controller + ArgoCD Gitops syncer

Second approach is that we can use Kubevela gitops controller way as the server side and argocd can be our gitops syncer. This approach is flexible to use native kubevela feature set without using a custom plugin or dry run module. We just need to add below annotations to our manifest repository to ignore outofsync.

argocd.argoproj.io/compare-options: IgnoreExtraneous

In this example, ArgoCD tracks native Kubevela application resources and its revision.

Conclusion​

1. Vela Dry-run approach

• Kubevela can be installed different cluster from ArgoCD.
• Limited Kubevela features, we just use component and traits definition.
• Workflow and policy features don’t supported by this way.
• Depend on Kubevela dry run feature.
• No needed rbac, ui or any different experience from ARGO.

2. Kubevela Controller + ArgoCD Gitops syncer

• Kubevela must be installed to ArgoCD cluster.
• We can use all the features of Kubevela.
• No need depends on anything.

Reference: https://gokhan-karadas1992.medium.com/argocd-kubevela-integration-eb88dc0484e0

Big thanks to Erkan Zileli 🎉🎉🎉

Since Open Application Model invented in 2020, KubeVela has experienced tens of version changes and evolves advanced features towards modern application delivery. Recently, KubeVela has proposed to become a CNCF incubation project and delivered several public talks in the community. As a memorandum, this article will look back into the starting points and give a comprehensive introduction to the state of KubeVela in 2022.

What is KubeVela?

KubeVela is a modern software platform that makes delivering and operating applications across today's hybrid, multi-cloud environments easier, faster and more reliable. It has three main features:

• Infrastructure agnotic: KubeVela is able to deploy your cloud-native application into various destinations, such as Kubernetes multi-clusters, cloud provider runtimes (like Alibaba Cloud, AWS or Azure) and edge devices.
• Programmable: KubeVela has abstraction layers for modeling applications and delivery process. The abstraction layers allow users to use programmable ways to build higher level reusable modules for application delivery and integrate arbitrary third-party projects (like FluxCD, Crossplane, Istio, Prometheus) in the KubeVela system.
• Application-centric: There are rich tools and eco-systems designed around the KubeVela applications, which add extra capabilities for deliverying and operating the applications, including CLI, UI, GitOps, Observability, etc.

KubeVela cares the whole lifecycle of the applications, including both the Day-1 Delivery and the Day-2 Operating stages. It is able to connect with a wide range of Continuous Integration tools, like Jenkins or GitLab CI, and help users deliver and operate applications across hybrid environments.

Why KubeVela?

Challenges and Difficulties​

Nowadays, the fast growing of the cloud native infrastructures has given more and more capabilities for users to deploying applications, such as High Availability and Security, but also exposes an increase number of complexities directly to application developers. For example, the Ingress resource on Kubernetes enables users to expose their applications easily, but developers need to handle the Ingress upgrades when the underlying Kubernetes version shifts, which requires knowledges for the Ingress resource. The hybrid deployment across various cloud provides can make this problem even harder. These difficulties are caused by the lack of operational input in the application definition and developers must face the infrastructure details directly if they want to enjoy the benefits brought by the rich cloud-native community.

Open Application Model​

To tackle the above challenges and bridge the gap between the use of applications and the understanding of infrasturcture details, Open Application Model (OAM) is jointly proposed by Alibaba Cloud and Microsoft Azure in 2020. The aim is to define a consistent application model for application delivery, irrelevant with the platforms and implementations. The defined application model describes an interface for developers on what an application consists of and how it should work. The former one is known as Component in OAM, which is usually used to model the workloads of the application. The latter one is defined as Trait in OAM, which attaches extra capabilities to Components.

KubeVela as OAM​

KubeVela is one of the implementations for the Open Application Model. In KubeVela, the abstraction layer is powered by CUE, a novel configuration programming language which can describe complex rendering logics and work as a superset of JSON. The abstraction layer simplifies the configuration of resources in Kubernetes, which hides the details of implementations and exposes limited parameters to the front developers. With KubeVela application, it is easy for developers to focus on the centric logic of applications, like what container image should be used and how the service should be made accessible. To achieve that, best practices of using Kubernetes native resources are summarized into KubeVela X-Definitions, which provide rendering templates of resources using CUE. These templates can be accessed from various sources, including official repositories, community addons or even self customized implementations by system operators. The templates are mostly infrastructure implemetation agnostic, in other words, not necessarily bond to specific infrastructures. The developers do not need to be aware of the underlying infra when using these templates.

Components & Traits​

The application model divides the abstraction of infra into two different aspects. The Component describes the main workload, which particularly in Kubernetes can be Deployments, StatefulSets, Helm Releases, etc. The Trait on the other hands, describes the added capability for the main workload, such as the scaler trait specifying the number of replicas and the gateway trait aggregates the endpoints for access. The separation of concerns in the design of Component and Trait give high extensibility and reusability to the abstraction. For example, the gateway trait could be backended by different infrastructures like Ingress or HTTPRoute. The application developer who uses the trait only needs to care about the exposed parameters, including the path, port and domain. The trait can be attached to various types of workloads, abstracted by different types of components, such as Deployment, StatefulSet, CloneSet, etc. In the cases where application developers and SRE are in the different teams, KubeVela makes clear division for their responsibilities.

• The platform team providing infrastructures, are responsible to build up X-Definitions where they enforce best practices and deployment confidence.
• The end users only need to choose the Component and Trait provided by the platform team and use them to assemble applications. They can simply enjoy PaaS-like experiences instead of directly interacting with the infra behind.

These are made possible thanks to the flexible, extensible and programmable system of KubeVela and can be applied under varying environments.

Unified Delivery​

Application delivery could happen everywhere. Therefore, another goal for KubeVela application is to build up unified delivery and provide consistent usage for users under various scenarios.

Hybrid-Cloud & Multi-Cluster​

In addition to the abstraction layer, KubeVela also supports hybrid-cloud or multi-cluster architecture natively as modern cloud native applications are not only about containers but involves lots of cloud resources as well. Besides, more and more users and teams start facing the difficulties of deliverying applications to various environments or multi-clusters for different purposes, such as testing or high availability. The KubeVela application allows user to define delivery targets and differentiated configurations through policies. The abstraction helps hide the details of how clusters are registered and connected and provide runtime-agnostic usages to app developers.

Addon Integration​

To enrich the delivery capability, users can leverage KubeVela addons to make extensions to their system. The addons are discoverable, reusable and easy-to-install capability bundles. They usually contain capability providers, including a wide range of third-party projects, like FluxCD, ClickHouse, Crossplane, etc. Addons not only install those projects into the system but create corresponding definitions for the integration concurrently, which extends the types of Component and Trait that application developers are able to use. The KubeVela community currently have 50+ addons already. Platform builders could enjoy these out-of-box integrations in systems depending on their customized demands.

With addons enabled in the system, it would be possible for end users to assemble applications in more customized ways, such as deploying cloud resources or using advanced workloads.

KubeVela Workflow​

While the Open Application Model defines the composition of an application, in real cases, the delivery process of the compositions could still vary a lot. For example, the different components in one application could have inter dependencies or data passing where delivery steps must be executed in specific order. Furthermore, the delivery process sometimes also involves more actions apart from the delivery of resources, such as rollouts or notifications. An extensible workflow is therefore designed to fulfill the needs of the process customization in KubeVela. Similar to Component and Trait, KubeVela workflow also leverages CUE to define workflow steps, providing flexibility, extensibility and programmability. It can be seen as another form of Infrastructure as Code (IaC). A bunch of build-in workflow steps has already provided rich out-of-box capabilities in KubeVela, such as making multi-cluster deployments and sending notifications through slack or email. The lightweight engine ensures the high performance and safety of step executions, compared to other types of engines involving running extra containers. Differ from the Component and Trait definitions in KubeVela, the WorkflowStep definition does not render templates into resources. Instead, it describes the actions to be executed in the step, which calls underlying atomic functions in various providers. With the use of workflow and addons, users are able to build arbitrary delivery process and make customized integrations. For example, it is possible to let the Continuous Integration tools to trigger the delivery of KubeVela applications and implement the GitOps solutions combining FluxCD and other addons.

Day-2 Management​

KubeVela cares more other than Day-1 Delivery. It also provides a unified Day-2 application management capability for all it's extensibility. The day-2 management is necessary for system operators and application developers to make continuous operation for the delivered applications and ensure the applications are always under control.

Resource Management​

The basic capabilities for application management are for its resources. KubeVela's core controller continuously watches the difference between the current state and the desired state of delivered resources. It makes sure that the live spec is accord with the declared spec recorded in the delivery process and therefore effectively prevents any configuration drifts. Besides, the automated garbage collection help recycle the resources that are not in-use during upgrades or deletion. There are also times resources need to be shared across multiple applications. These are all made possible in KubeVela application through the use of policies.

Version Control​

KubeVela application keeps history records for deliveries. These snapshots are useful when new version publish are out of expectations. The change inspectation could be used to diagnose the possible error changes and the rollback allows fast recovery to the previous successful states.

Observability​

KubeVela treats observability as first class citizen. It is the eyes to users for monitoring the state of applications and observing exceptions. There are multiple tools and methods in KubeVela to do the observation job. One of the most straightforward way is to use the CLI tool of KubeVela. The Vela CLI is able to provide in-time status info for the application in fine-grain or aggregated level. For users that prefer web interfaces, VelaUX provides an alternative way to view application status. In the cases applications are monitored through third-party projects, such as Grafana, Prometheus or Loki, KubeVela further provides addons for bootstrapping the observability infrastructures and empower users to customize the observing rules as codes in applications, through the abstraction layer. A series of out-of-box metrics and dashboards give users the basic capability of automated system observability. These can be used to diagnose system level exceptions and help improve the overall performance.

Eco-system​

In addition to the above mentioned tools, KubeVela also has several other tools in the eco-systems to facilitate application delivery.

• Vela CLI: KubeVela CLI provides various commands that helps you to operate applications, such as managing definitions, viewing resources, restarting workflow, rolling versions.
• VelaUX: VelaUX is the Web UI for KubeVela. Besides, it incorporates business logics into fundamental APIs and provides out-of-box user experiences for non-k8s-expert users.
• Terraform Controller: The terraform controller in KubeVela allows users to use Terraform to manage cloud resources through Kubernetes Custom Resources.
• Cluster Gateway: The gateway that provides unified multi-cluster access interface. Working as Kubernetes Aggregated API Server, the gateway leverages the native Authentication and Authorization modules and enforces secure and transparent access to managed clusters.
• VelaD: Building on top of k3s & k3d, VelaD integrates KubeVela with Kubernetes cores, which can be extremely helpful for building dev/test environment.
• Vela Prism: The extension API server for KubeVela built upon the Kubernetes Aggregated API Server. It projects native APIs like creating dashboards on Grafana into Kubernetes resource APIs, so that users can manage 3rd-party resources as Kubernetes native resources.
• Vela Workflow: The workflow engine translates CUE-based steps and executes them. It works as a pure delivery tool and can be used aside by the KubeVela application. Compared to Tekton, it mainly organize the process in CUE style, instead of using Pods and Jobs directly.

Stability​

To ensure KubeVela is able to handle certain amount of applications under limited resources, multiple load testings have been conducted under various circumstances. The experiments have demonstrated that the performance of KubeVela system is capable of dealing thousands of applications in an ordinary-sized cluster. The observability infrastructure further exposes the bottleneck of KubeVela and guides system operators to do customized tunning to improve the performance in specific use environments.

In a nutshell

Currently, KubeVela has already been applied in production by a number of adopters from various areas. Some mainly use KubeVela's abstraction capability to simplify the use and deploy of applications. Some build application-centric management system upon KubeVela. Some use the customized workflow to orchestrate the delivery process. It is especially welcomed in high-tech industries and shown to be helpful for delivering and managing enourmous applications.

The KubeVela community has attracted world-wide contributors and continuously evolves over the past two years. Nowadays, there are over 200 contributors from various contries have participated in the developing of KubeVela. Thousands of issues have been raised and 85% of them are already solved. There are also bi-weekly community meetings held in both English and Chinese community.

With more and more people coming into the community, KubeVela is consistently upgrading itself to fit into more complex, varying use cases and scenarios.

Application Delivery on Kubernetes​

The cloud-native landscape is formed by a fast-growing ecosystem of tools with the aim of improving the development of modern applications in a cloud environment. Kubernetes has become the de facto standard to deploy enterprise workloads by improving development speed, and accommodating the needs of a dynamic environment.

Kubernetes offers a comprehensive set of entities that enables any potential application to be deployed into it, independent of its complexity. This however has a significant impact from the point of view of its adoption. Kubernetes is becoming as complex as it is powerful, and that translates into a steep learning curve for newcomers into the ecosystem. Thus, this has generated a new trend focused on providing developers with tools that improve their day-to-day activities without losing the underlying capabilities of the underlying system.

Napptive Application Platform​

The NAPPTIVE Playground offers a cloud native application platform focused on providing a simplified method to operate on Kubernetes clusters and deploy complex applications without needing to work with low-level Kubernetes entities. This is especially important to be able to accommodate non-developer users' personas as the computing resources of a company are being consolidated in multi-purpose, multi-tenant clusters. Data scientists, business analysts, modeling experts and many more can benefit from simple solutions that do not require any type of knowledge of cloud computing infrastructures, or orchestration systems such as Kubernetes, which enable them to run their applications with ease in the existing infrastructure.

Any tool that works in this space must start by analyzing the existing abstractions. In particular, at Napptive, we focus on the Applications. This quite understood abstraction is not present in Kubernetes, requiring users to manually tag, identify and reason about the different components involved in an application. The Open Application Model provides an excellent abstraction to represent any type of application independently of the cloud provider, containerization technology, or deployment framework. The model is highly customizable by means of adding Traits, Policies, or new Component Definitions.

KubeVela in Napptive​

The Napptive Playground leverages Kubevela as the OAM runtime for Kubernetes deployments. Our Playground provides an environment abstraction with multi-tenant guarantees that is equivalent to partitioning a shared cluster by means of differentiated namespaces. The benefit of our approach is the transparency of its configuration using a higher abstraction level that does not involve any Kubernetes knowledge.

The following diagram describes the overall architecture of Napptive and Kubevela deployed in a Kubernetes cluster.

The user has the ability to interact with the cluster by using the Napptive user interface (CLI or Web UI), or by means of using the standard Kubernetes API with kubectl. Isolated environments can be easily created to establish the logical separations such as type of environment (e.g., deployment, staging, production), or purpose (e.g., projectA, projectB), or any other approach to differentiate where to deploy an application. Once the OAM application is deployed on Kubernetes, Kubevela is in charge of managing the low-level application lifecycle creating the appropriate Kubernetes entities as a result. One of the main differences with other adopters of Kubevela, is the fact that we use Kubevela in a multi-tenant environment. The following figure shows the specifics of an application deployment in this type of scenario.

The Napptive Playground is integrated with Kubernetes RBAC and offers a native user management layer that works in both on-premise and cloud deployments. User identity is associated with each environment, and Kubevela is able to take that information to ensure that users can only access their allowed resources. Once a user deploys an OAM application, Kubevela will intercept the call and attach the user identity as an annotation. After that, the rendering process will ensure that the user has access to the entities (e.g., trait, component, policy, workflow step definitions) and that the target namespaces are also accessible by the user. Once the application render is ready, the workflow orchestrator will create the different entities in the Kubernetes namespace.

Napptive in the Community​

In terms of our involvement with the OAM/Kubevela community, it has evolved overtime from being passive members of the community simply exploring the possibilities of OAM in Kubernetes, to becoming active contributor members in different areas. In particular, we have closely worked with the core Kubevela development team to test and overcome the different challenges related to using a multi-tenant, RBAC compliance installation of Kubevela. Security in this type of installation is critical and it is one of our main focuses within the Kubevela community. We are committed to continue working with the community not only to ensure that multi-tenancy is maintained throughout the different releases, but also to add our own perspective representing our customer use cases.

The Kubevela community is growing at a fast pace, and we try to contribute our adaptations, features, feedback, and thoughts back to the community. This type of framework is deployed in a multitude of environments. The most common one is probably where one or more clusters are inside a company. In our particular application, we are interested in exploring methods to offer computing capabilities for a variety of user personas in a shared cluster, and we contribute our view and experience in the community.

Future Evolution​

In terms of future evolution, we believe the Napptive Playground is a great tool to experience working with the Open Application Model without the need of installing new clusters and frameworks. From the point of view of our contributions in the community, we are internally working in exploring QA mechanisms that ensure that customer applications remain working after upgrades, and identifying potential incompatibilities between releases. Once that work is ready, we plan to contribute our testing environment setup back to the community so that it can be adopted in the main branch. We are also excited with new functionalities that have been recently added to Kubevela such as multi-cluster support, and are exploring methods to adopt it inside the Playground. Moreover, we are actively working on an OAM-compatible application catalog that will simplify the way organizations store and make accessible application definitions so that they can be deployed into a cluster from a central repository. The catalog focuses on the OAM entities and relies on existing container registries for storing the image.

The community has released the new milestone release v1.6 of KubeVela during the 2022 Apsara Conference. This release is a qualitative change in KubeVela from application delivery to application management. It also creates a precedent in the industry to build an application platform with delivery and management integrated based on a extensible model.

As we all know, KubeVela is a highly extensible platform, on which users can build their own customizations using Definitions. KubeVela addons are a convenient way to pack all these customizations and their dependencies together as a bundle, to extend your own platform capabilities!

This blog introduces the main concepts of an addon and guides you to quickly start building one. Finally, we'll show you the experience of the end user, and how the capabilities provided are glued in a consistent user-friendly experience.

KubeVela 1.5 was released recently. This release brings more convenient application delivery capabilities to the community, including system observability, CloudShell terminals that move the Vela CLI to the browser, enhanced canary releases, and optimized multi-environment application delivery workflows. It also improved KubeVela's high extensibility as an application delivery platform. The community has started to promote the project to the CNCF Incubation stage. It has absorbed the practice sharing of multiple benchmark users in many community meetings, which proves the community's healthy development. The project is now mature to some extent, and its adoption has made periodical achievements, thanks to the contributions of more than 200 developers in the community.

Background​

Helm is an application packaging and deployment tool of client side widely used in the cloud-native field. Its simple design and easy-to-use features have been recognized by users and formed its ecosystem. Up to now, thousands applications have been packaged using Helm Chart. Helm's design concept is very concise and can be summarized in the following two aspects:

1. Packaging and templating complex Kubernetes APIs and then abstracting and simplifying them into small number of parameters

2. Giving Application Lifecycle Solutions: Production, upload (hosting), versioning, distribution (discovery), and deployment.