How to rock at Serverless - from beginner to Knative and Kubeless

Tags: #<Tag:0x00007f76fd14bd80> #<Tag:0x00007f76fd14bbc8> #<Tag:0x00007f76fd14ba10> #<Tag:0x00007f76fd14b7e0> #<Tag:0x00007f76fd14b6c8> #<Tag:0x00007f76fd14ac28> #<Tag:0x00007f76fd14a728> #<Tag:0x00007f76fd14a458> #<Tag:0x00007f76fd14a318> #<Tag:0x00007f76fd149f58> #<Tag:0x00007f76fd149cd8> #<Tag:0x00007f76fd149af8> #<Tag:0x00007f76fd149508> #<Tag:0x00007f76fd149260> #<Tag:0x00007f76fd149030>

This is a hands-on lab guide for engineers. It assumes some experience but there are a lot of links and annotations.


How to rock at Serverless - from beginner to Knative and Kubeless

Motivation: building services in a cloud-native fashion

Serverless is like an open invitation to idea-building without high investments upfront. Focus on business logic, and what’s behind the curtains shall scale based on demand. Financially, as well as technically. Ideal for Startups, Incubators, traction-based funding, … Unicorns.

Serverless does not mean that the code runs on thin air. On the contrary: the following lab guide starts at a barebones Ubuntu VM (for this lab you are going to need the equivalent of an AWS a1.xlarge - 4 cores and 8 GB RAM). If you use Cloud-providers that is not going to be your problem, because a provider is going to consolidate that to keep things affordable.

Summary:

  • executive motivation and vision of opportunities enabled through Serverless :heavy_check_mark:

Details on the technical expose

This guide is going to remain cloud-agnostic. It will feature an on-premise setup. I hope that this way the in-depth (and low-level) knowledge will percolate through shared hands-on experience.

One key element of a Serverless strategy is to maintain an exit strategy by avoiding potential vendor-lock. I think with an on-prem cloud-agnostic setup this is achievable. And maybe this also helps with mixed- or hybrid-cloud scenarios when some parts of the architecture have to remain in a colocation site?

In this guide, I chose to use frameworks that lower the “time to market” and that are considered to be sane for Enterprise adoption by the majority of developers.

I am going to distinguish between Serverless and Function As A Service (FaaS). This terminology is scoped for this essay and does not apply in general.

  • Knative - we will use it with the Spring framework[1]
  • Kubeless - we can adopt Kubeless[2], but it’s more FaaS-like

It’s much better to experience this subtle difference. – With Knative we need to push Docker containers to a registry, with Kubeless that is not required. This has got advantages, but also disadvantages. Do you think that there are limitations to be aware of?
– FaaS workloads in Kubeless can also be called from within Knative services.

Summary:

  • refined terminology regarding Serverless and Function As A Service :heavy_check_mark:
  • the technical expose is centered around real-world Enterprise / Startup needs :heavy_check_mark:

Motivation: highly demanded technical exertise

While reading this guide you’ll gain familiarity with cutting edge DevOps tech and get some security insights.

I am assuming that you didn’t have a lot of exposure to the following tech stack. Either because you work in InfoSec or because it’s all relatively new. Or both :slight_smile:

With this you will have an up to date lab with a lot of components that you will encounter during real-world assignments. The versions are documented because technology rapidly progresses.

Platform engineering lab (Ubuntu Linux and Kubernetes):

  • setup of Linux VMs and networks
  • Kubernetes (1.18) - Operators, Deployments, Services, Pods, …
  • Istio (1.5.1) - Service Mesh networks, Sidecars, Traces and Debugging, advanced deployments
  • Knative (0.13.0)
  • Kubeless (1.0.6)
  • Ubuntu Server (18.04.4 LTS) - Docker 19 CE, systemd, coreutils, GNU parallel (20161222)
  • minikube (1.92) - plugins, CNI, Ingress, Load Balancer

Software development lab (full-stack):

  • OpenJDK Java 11, SpringBoot 2.2.6 (current release), Maven 3.6, Java 11, Kotlin 1.3
  • basic (Anaconda) Python 3.7, JupterLab 6.0.3, Tensorflow 2.1
  • basic Node.js 12.x, React 16.x, Gatsby 2.2

Continouus Integration lab (DevOps):

  • Jenkins 2.x
  • SonarQube CE 8.x
  • diverse security-testing tools (up to date, Git)

The reason why I decided to feature multiple programming languages is that MicroService development invites to polyglot coding. This way you can combine the best elements. For example auto-scaling REST API servers with Java and intuitive Machine Learning with Python. No one likes reimplementing things or having unnecessary barriers between code and production.

If there is some interest a future guide may shed some light on the security concerns and AppArmor and SELinux especially.

Don't be shy!

Maybe you can join the “self-help group” at the Because-Security Slack and message /me directly… Or use Twitter.

The new channels collect cool feedback, new impressions, your opinions, and other nice comments. Please keep in mind that this is just my personal website and that good vibes matter. Expertise, in reality, is a perpetual learning experience.

  • #serverless-security - Serverless security topics focused on development and productivity
  • #cloud-security - topics surrounding Enterprise Cloud Security
  • #kube-security - Kubernetes security topics focused on operations and performance
  • #jenkins-security - Jenkins and general CI / CD security topics focused on build, test, release, deploy.
  • #spring-security - security in the Java ecosystem focused on Spring and Spring Security

Summary:

  • preview of full-stack development technology fit for modern Serverless engineering labs :heavy_check_mark:
  • provided contact point to reach out in case of questions :heavy_check_mark:

Istio Service Mesh network visualization

The following Gif features a live demo of the aforementioned components. You can see a hello-world Go service answering to HTTP requests. This is a good start point.

Kiali_Knative_Istio_Kubernetes_Ubuntu18

We are going to see auto-scaling in action, live, on-prem and agnostic to a cloud provider. I made a short video and added it to a later section of the essay. The little circles are an animation of web requests coming in, but I think it’s rather intuitive.

Dev env setup


Needless to say: the following setup steps are for demo environments only. Don’t do this in prod.


Minikube on Ubuntu

My personal Ubuntu lab VM is on an ESXi (on-prem) within the 192.168.1.0/24 net (a NAT’ed virtual network). The VM got a route to the internet via DHCP (on my pfSense), which is why installing minikube is straight forward[3].

This can resemble production environments a little, but you may also use KVM (qemu) and the usual dnsmasq and iptables approach.

image

For this lab we will (still) use Docker[4]. Today, in April 2020, this is a stable and standard approach. Canonical’s microk8s already uses a different container engine, and that’s the way to go in the future.

The $USER needs to belong to the appropriate groups:

sudo usermod -aG docker $USER
su - $USER

The second command drops you into a session where the user group change is effective. No need to restart.

Clean slate for the lab

image
sudo iptables -F
sudo iptables -F -t nat
sudo minikube delete
sudo docker system prune -a
sudo reboot

The iptables NAT chain gets flushed separately here. The following may be a got fit for the lab’s ~/.bash_aliases:

alias minikube-kill = `docker rm $(docker kill $(docker ps -a --filter="name=k8s_" --format="{{.ID}}"))`
alias minikube-stop = `docker stop $(docker ps -a --filter="name=k8s_" --format="{{.ID}}")`

Lab VM DNS and systemd

Systemd likes to reinvent how Linux handles things. This does not exclude DNS resolvers:

sudo mv /etc/resolv.conf /etc/resolv.conf.old
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf

Without doing this you may get problems when you are starting the minikube env. I had many issues with coredns[5].

Lab VM hostname

pidof systemd && echo "yes" || echo "no"
ping $(hostname -f) && echo "yes" || echo "no"

You need to be able to resolve the hostname of the VM internally.
And of course, a systemd process needs to run.

Install DevOps tools via Snap

Install snap packages and add the bins to $PATH.

snap install helm --classic
snap install kubectl --classic
export PATH=$PATH:/snap/bin

kubectl is the management CLI tool for Kubernetes’ orchestration. helm is a package manager, and actually, I don’t recommend to use it. I only needed it to create a YAML template here.

Generally, I don’t like dropping binaries into Linux system folders. Rather than that, I prefer to add custom folders to the environment $PATH and to have these utilities managed by a package-manager.

Start minikube

Installing an on-premise single-node Kubernetes system isn’t easy. Doing this from upstream Kubernetes is a waste of time. You would need to redefine the roles of the node and override a lot of configs. I did this once, and it took hours. – Lesson learned: don’t do that.

sudo minikube start --driver=none \
--extra-config=apiserver.enable-admission-plugins="LimitRanger,NamespaceExists,NamespaceLifecycle,ResourceQuota,ServiceAccount,DefaultStorageClass,MutatingAdmissionWebhook" \
--memory 9000 \
--cpus 4

sudo chown -R $USER ~/.kube
sudo chown -R $USER ~/.minikube

You may be tempted to think, that the --memory and --cpus parameters can be omitted because this installs the services natively. I found out that these variables are used to calculate quotas and limits. If you do not specify them, you have to edit the ConfigMaps and Namespaces later. If you know where they are…

Now to the computing backbone: our simple single-node Kubernetes “cluster” is ready. Please be patient with that little 4 core system:

kubectl get nodes
kubectl get services -A --watch

watch -n 0.3 \
kubectl get po,services,deployment,rc,rs,ds,no,job,cm,ing -A # or similar

If installed with minikube's native driver Kube API is reachable from the network. Therefore you can copy ~/.kube and ~/.minikube to your Dev box, Jumphost, … any host in the same net. Or to a test env CI / CD server, maybe you have Test Jenkins?

Are you ready to understand the tech not just as a theory?

Summary:

  • installed Lab Kube environment as “computing backbone” :heavy_check_mark:

Basics: what is all this stuff? Recap on Kubernetes

The idea to place the theory here is to allow a pragmatic and simplified reflection of the Kubernetes related concepts. At this point, it can be combined with a systematic exploration.

If you are a seasoned Kubernetes administrator you may find the following illustration to be inaccurate. That is because this is just teaching material from a freely available essay. :innocent:

  • Pods[6] are a core concept of Kubernetes
  • Within a Deployment you specify that you want a certain amount of instances to be available (or let an Operator manage the scaling). In our case, we will deploy an application that gets managed within the cluster by a Serverless operator.
  • A Service[7] is the result of a Deployment. This way it becomes available. In our case, we will create an API server endpoint that clients can connect to.
  • If persistent storage is required, it’s usually better to use network filesystems and to mount these external resources into the transient instances.
  • Labels and Selectors are important if you have many Services. You may have different settings for different environments (test, prod, staging, … lab)

Summary:

  • short summary of elements within a basic Kubernetes architecture :heavy_check_mark:

On-premise Serverless stack - Istio and Knative

Install Istio with the observability tools

Istio is like an extension to Kubernetes. It adds a Service Mesh. In my opinion, this is the future of how we are going to run and think about services in Cloud-Native environments.

From a security perspective, I like having an mTLS (mutual Transport Layer Security) enabled Service Mesh network between Pods. If you make a Threat Model of your env you may find out that the MicroServices exchange confidential data containing PII (Personally Identifiable Information).
Encryption can help to develop means within the MicroService architecture that treat the trust boundaries within third-party networks. Like between AWS EC2 instances.

If you are only interested in in-transit encryption for Kubernetes, you may also like to learn more about Cilium[8]. I only used CNI and for a single-node minikube lab that is fine.

Other use-cases of Istio are Blue-Green deployments, Canary deployments or Dark Deployments. Tese happen with the Mixer component.

cd ~ 
curl -L https://istio.io/downloadIstio | sh - 
cd istio-1.5.1 
export PATH=$PWD/bin:$PATH 
istioctl manifest apply --set profile=demo 
kubectl label namespace default istio-injection=enabled

export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
export SECURE_INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="https")].nodePort}')
export INGRESS_HOST=$(minikube ip)

Keep these variables (or the commands). We are going to need them to test the routes and reachability of our Egress endpoints of the Service Mesh (with curl).

Summary:

  • demo installation of Istio to extend the basic Kubernetes architecture for Serverless :heavy_check_mark:

Connect to Kiali with the browser from your Dev system without a tunnel

Kiali is a wonderful way to gain instant familiarity with a Micro Service Mesh Network. It uses Prometheus (a metrics collector) and Jaeger (a tracer).

With both systems combined, you can gain insights into the auto-scaling behavior of Knative services. This relates to Istio in so far, that its Sidecar containers get injected into the Knative service Pods, and that the metrics are provided this way.
– But let’s focus on one thing at a time. Before we are ready to observe all of this, we need to continue with the setup. Keep in mind: you don’t just want the front-row seats for this. You want to be a player!

Although minikube has tunneling and kubectl has got proxyfication features, it may be more comfortable to map Kiali to the VM’s IP. This could also be done with Ingress or MetalLB, but the simplest form of mapping out the service is this:

kubectl patch service kiali -n istio-system -p '{"spec": {"type": "LoadBalancer", "externalIPs":["'$(minikube ip)'"]}}'

Consider this specific step replaceable, but keep the one-liner in mind unless you prefer Ingress or MetalLB.

Add the gateway - only in a Dev env

Knative needs this because its routing by default will assume a real Kube cluster.

cd ~/istio-1.5.1 
helm template --namespace=istio-system \
  --set gateways.custom-gateway.autoscaleMin=1 \
  --set gateways.custom-gateway.autoscaleMax=2 \
  --set gateways.custom-gateway.cpu.targetAverageUtilization=60 \
  --set gateways.custom-gateway.labels.app='cluster-local-gateway' \
  --set gateways.custom-gateway.labels.istio='cluster-local-gateway' \
  --set gateways.custom-gateway.type='ClusterIP' \
  --set gateways.istio-ingressgateway.enabled=false \
  --set gateways.istio-egressgateway.enabled=false \
  --set gateways.istio-ilbgateway.enabled=false \
  --set global.mtls.auto=false \
  install/kubernetes/helm/istio \
  -f install/kubernetes/helm/istio/example-values/values-istio-gateways.yaml \
  | sed -e "s/custom-gateway/cluster-local-gateway/g" -e "s/customgateway/clusterlocalgateway/g" \
  > ./istio-local-gateway.yaml

kubectl apply -f istio-local-gateway.yaml 

Patch the Istio Ingress-gateway

Again this is just for the lab. Naturally, if you were to use OpenShift this would be a little different.

kubectl patch service istio-ingressgateway -n istio-system -p '{"spec": {"type": "LoadBalancer", "externalIPs":["'$(minikube ip)'"]}}' 

Deploy Knative Serving and Eventing operators

kubectl apply -f https://github.com/knative/serving-operator/releases/download/v0.13.0/serving-operator.yaml 

cat <<-EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
 name: knative-serving
---
apiVersion: operator.knative.dev/v1alpha1
kind: KnativeServing
metadata:
  name: knative-serving
  namespace: knative-serving
EOF

kubectl get deployment knative-serving-operator

kubectl apply -f https://github.com/knative/eventing-operator/releases/download/v0.13.0/eventing-operator.yaml

cat <<-EOF | kubectl apply -f -
apiVersion: v1
kind: Namespace
metadata:
 name: knative-eventing
---
apiVersion: operator.knative.dev/v1alpha1
kind: KnativeEventing
metadata:
  name: knative-eventing
  namespace: knative-eventing
EOF

kubectl get deployment -n knative-serving

Summary:

  • installed Istio and Knative in a Lab Kube environment for Serverless :heavy_check_mark:
  • we are ready for test deployments :heavy_check_mark:

Recap: what are Operators, ConfigMaps and Routes?

At this point in time (Q2 2020), there are different kinds of Operators that can extend the Kubernetes architecture. The topic is vast and involves Custom Resource Definitions (CRDs) that can extend the set of functions of the Kube API. Knative brings an operator, that will initialize an “Autoscaler”. Keep in mind that in production you may not want your API endpoints to scale down to 0 instances and that you can configure all of that. For demonstration purposes I refrained from doing this here. It’s super easy to do, via a ConfigMap object.

ConfigMaps simply hold the configuration similar to Linux’s /etc/ directoy. They can be edited, and the respective Operators observe them. So in case you want to change Knative’s domain or the autoscaling behavior, you will find documented variables in the ConfigMap.

Routes are how Services are bridged out, either via custom LoadBalancers, Ingress, or other Service objects. Knative changes the Routes for the autoscaling to distribute the requests. This is visualized in a video in this essay.

The good news is, that you’ll pick this up in great detail over time. I find very little use for encyclopedic summaries about Kubernetes. This is an area of rapid progress and in a couple of months, everything here will be different again. If you are a Linux user, you will many opportunities to harness your command-line skills to pick things up on the fly.

Summary:

  • brief mentioning of advanced Kubernetes concepts as much as they relate to Istio and Knative :heavy_check_mark:

Time for Hello World - Knative Go

Here we build, tag and push the local image. You can use Docker or Podman to do so.

cd ~/Source/knative-docs/docs/serving/samples/hello-world/helloworld-go
docker build .
docker login
docker tag e...6 wishi/knative-go-hello

docker push wishi/knative-go-hello
The push refers to repository [docker.io/wishi/knative-go-hello]
...

Time for 1, 2, 3, many curls

Let’s make a few HTTP requests to the service in parallel:

export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="http2")].nodePort}')
export INGRESS_HOST=$(minikube ip)

while true; \
do curl http://$INGRESS_HOST:$INGRESS_PORT  \
--header 'Host: helloworld-go.default.example.com' \
&& sleep 0.3; \
done
Hello Go Sample v1!
Hello Go Sample v1!
Hello Go Sample v1!

sudo apt-get install parallel
seq 1000 | parallel -n0 \
-j10 "curl -s http://INGRESS_HOST:$INGRESS_PORT \
-H 'Host: helloworldgo.default.example.com'"
Hello Go Sample v1!
Hello Go Sample v1!
Hello Go Sample v1!
...

Summary:

  • deployed Go-based sample app for Knative into Kube cluster :heavy_check_mark:
  • used curl and parallel for some basic testing :heavy_check_mark:

Development of a Kotlin SpringBoot REST server into your Knative cluster

Let’s deliver Chuck Norris jokes at scale, via a REST service endpoint developed with Spring Boot, written in Kotlin.
The advantage of this is that the code is easy to read and that the service behavior will be compliant to the standards so that we can do HTTP load-testing. This way we can see auto-scaling Serverless workloads in action.

I use a Maven package that provides the joke texts via Spring Boot Actuator package[9]. This way I do not need a DBMS.

The application is rather primitive. It offers two endpoints:

  • / - returns HTML, via a Thymeleaf template
  • /joke - returns JSON, via a library and Spring

I published the image to DockerHub[10]. You can also build it yourself and push it into your own registry.

I would like to highlight that no code changes are required to run this application in Knative (or Docker). The code remains untouched.

Java 11 (versions higher than 8) is aware of Linux’s cgroups limits within container environments. This way the container environments memory limits are respected by the Java runtime. And you do not need to add mystical /dev/urandom mounts anymore[11].

(base) [email protected]:~/Source/chucknorris-kotlin-spring$ kubectl apply -f service.yaml 
service.serving.knative.dev/chuckjokes-java-spring created

Now get the endpoint:

(base) [email protected]:~/ curl -i http://$INGRESS_HOST:$INGRESS_PORT/joke \
-H "Accept: application/json" \
-H 'Host: chuckjokes-java-spring.default.example.com' \
-w "\n"
HTTP/1.1 200 OK
content-length: 55
content-type: application/json
date: Sun, 19 Apr 2020 06:56:19 GMT
server: istio-envoy
x-envoy-upstream-service-time: 7031

{"joke #1":"Chuck Norris's first program was kill -9."}

Initially, it will be slow because it’s instantiating the application on-demand. You can use this to warm up the cluster :fire:

I tried to get an Ubuntu VM to make more than 200 curl requests using GNU parallel. Although I unset the limits, I reached up to 162 requests per second, which is not enough to put the respective cluster under enough stress (the default requests per second to trigger the Knative autoscaler is 200:

(base) [email protected]:~/$ kubectl -n knative-serving describe cm config-autoscaler

# The requests per second (RPS) target default is what the Autoscaler will
# try to maintain ...
requests-per-second-target-default: "200"

Instead, I used wrk, also because its invocation is similar to curl.

(base) [email protected]:~/$ wrk -t12 -c400 -d300s \
http://$INGRESS_HOST:$INGRESS_PORT/joke \
-H 'Host: chuckjokes-java-spring.default.example.com'

On the right, you can see the Pods serving Chuck Norris jokes. On the left, you can see the curl command to bring up the initial instance. This is followed by the wrk command to put the Service under enough load.

Summary:

  • deployed auto-scaling on-premise cloud-agnostic Serverless workload with Knative :heavy_check_mark:
  • serving a REST API SpringBoot and Kotlin :heavy_check_mark:
  • consuming a REST endpoint with curl :heavy_check_mark:
  • load-testing a REST API with wrk and live-observation of the system :heavy_check_mark:

Kubeless on Kube - Rapidly connect Python DataScience flows into Serverless SpringBoot apps through FaaS

Python is very popular for DataScience because you can design idiomatic code that intuitively uses data and statistics to augment control-flow decisions. In my opinion Python code can host Machine Learning more easily than Java or Kotlin.

It should naturally to mesh code from JupyterLab / Notebook[12] cells together by calling their respective Kubeless API endpoints.

Here’s a simple code fragment that represents a Jupyter cell:

image

Kubeless and Knative can coexist on a Kubernetes cluster

The Kubeless installation is short in comparison[13]. Finishing with the initial steps should lead to Pods in the Kubernetes cluster with the following state:

(base) [email protected]:~$ kubectl get pods -n kubeless
NAME                                           READY   STATUS    RESTARTS   AGE
kubeless-controller-manager-54cc78b665-p9vwl   3/3     Running   0          9m55s

This section features a Function As A Service (FaaS) workflow, rather than a Serverless workflow with direct dependencies like containers. Keep in mind that heavy Machine Learning / Deep Learning workflows may need a custom runtime, and that this here is a limited example by intention.

(base) [email protected]:~$ kubeless get-server-config
INFO[0000] Current Server Config:                       
INFO[0000] Supported Runtimes are: ballerina0.981.0, dotnetcore2.0, dotnetcore2.1, dotnetcore2.2, go1.10, go1.11, go1.12, java1.8, java11, nodejs6, nodejs8, nodejs10, nodejs12, php7.2, php7.3, python2.7, python3.4, python3.6, python3.7, ruby2.3, ruby2.4, ruby2.5, ruby2.6, jvm1.8, nodejs_distroless8, nodejsCE8, vertx1.8 

Now that looks polyglot to me… let’s keep things simple and adopt the Serverless framework, which is made for FaaS deployments. :slight_smile:

For Python you manage the usual requirements.txt for your deployable[14], and Serverless framework expects them in the function directory.

(base) [email protected]:~/Source/kubeless-tensorflow-py3-srvless$ cat requirements.txt 
tensorflow==2.1

After running serverless deploy on the checked out Git project you can run the info command:

(base) [email protected]:~/Source/kubeless-tensorflow-py3-srvless$ serverless info

Service Information "hello"
Cluster IP:  10.102.235.227
Type:  ClusterIP
Ports: 
  Name:  http-function-port
  Protocol:  TCP
  Port:  8080
  Target Port:  8080
Function Info
Labels:
  created-by: kubeless
  function: hello
Handler:  handler.hello
Runtime:  python3.7
Dependencies:  tensorflow==2.1

As you can see this object doesn’t have a local LAN egress, but only a cluster-internal IP. That is my intention because it’s for internal use. You may, of course, define your own Ingress service for this.

The kubeless CLI uses kubectl's proxy functions. Therefore this just works out of the box from an external host with access to the Kube API:

(base) [email protected]:~/$ kubeless function ls
NAME 	NAMESPACE	HANDLER      	RUNTIME  	DEPENDENCIES   	STATUS   
hello	default  	handler.hello	python3.7	tensorflow==2.1	1/1 READY

(base) [email protected]:~/$ kubeless function call hello 
Hello, TensorFlow!

You can also deploy the function (as a service) with a one-liner without Serverless framework:

 kubeless function deploy hello --runtime python3.7 \
--from-file handler.py \
--dependencies requirements.txt \
--handler handler.hello
INFO[0000] Deploying function...                        
INFO[0000] Function hello submitted for deployment      
INFO[0000] Check the deployment status executing 'kubeless function ls hello' 

I have not found a better way to get the Ingress endpoint without Serverless framework. The respective runtime Pods with the Python runtime get initialized in the default namespace…

[email protected]:~$ kubectl describe pod hello-69688c9dd9-zqr6b | grep -i IP | head -n 1
IP:           172.17.0.36

[email protected]:~$ curl http://172.17.0.36:8080 -H 'Host: hello_tensorflow.hello' -w '\n'
Hello, TensorFlow!

[email protected]:~$ kubectl describe service hello
Name:              hello
Namespace:         default
Labels:            created-by=kubeless
                   function=hello
Annotations:       <none>
Selector:          function=hello
Type:              ClusterIP
IP:                10.102.235.227
Port:              http-function-port  8080/TCP
TargetPort:        8080/TCP
Endpoints:         172.17.0.34:8080
Session Affinity:  None
Events:            <none>


# or use the Service IP from Serverless
[email protected]:~$ curl http://10.102.235.227:8080 -H 'Host: hello.handler' -w '\n'
Hello, TensorFlow!

Granted, in this particular case, what the Tensorflow function does is not very exciting. – The main point is to illustrate the flexibility of a deployment that does CI / CD directly within the Kube cluster. That is where Kubeless’s magic is. Of course, there is a Pod on your cluster with the Python runtime…

Summary:

  • deployed FaaS workload with on-prem Kubeless using Serverless framework :heavy_check_mark:
    • this asserts compatibility to big Serverless clouds such as AWS’s Lambda.
  • handled a popular Python Machine Learning runtime dependency in a FaaS deployable :heavy_check_mark:

Mesh this into a React frontend - no boilerplate code

As you know React isn’t a framework but a library to build virtual DOMs. Gatsby[15] just adds some lightweight components to React. One of the reasons to mention it here is, that it moves Wordpress to the intranet, where it belongs. Wordpress’s security history[16] accounts for that.

Instead of a Content Management System (CMS) today you build a content-mesh. Such a content mesh that calls into our new cluster mesh-network. :nerd_face: Logically.

As a side-note: it’s certainly possible to build Gatsby React-based Micro-frontends[17] to segment development concerns, but this small-scale project would not benefit from this. Furthermore, you’d serve the API with GraphQL, to have more control over the data structure and the content.

Let’s keep things basic: we want to get a frontend app out, now, without thinking… of routing and without doing much boilerplate coding. Quick and easy.

We call our Knative REST endpoint. The Knative service may call into on-prem Kubeless or wherever our hypothetical Machine Learning pipeline may be hosted.

It's about React's component lifecycle

For the REST call, I used axios, which is a Node.js module.

componentDidMount() {
    axios.get(API_ENDPOINT, config)
      .then(result => {

        console.log(result.data)
        return result.data
      })
      .then(result => {
        let joke = result["joke #1"]
        console.log(joke)

        this.setState({ jokeText: joke })
      })
  }

config holds the HTTP host header. I wanted to keep things as similar to the curl commands as possible. Of course, you can use Ingress, an map the service to a public domain. That’s out of scope here, and I wanted to share how you’d start in a lab.

The functionality here can certainly be implemented without JavaScript, APIs, and Markup (JAMstack[18]). But you’d pick up skills this way that aren’t in demand. You don’t get to rock at Serverless with Wordpress templates :slight_smile:

And here our Full-Stack Serverless and FaaS journey comes to an end.

Summary:

  • developed a basic JAMstack frontend with React and Gatsby that consumes a REST API :heavy_check_mark:
  • learned how to adopt React Components for development and testing :heavy_check_mark:
  • understood that modern JAMstack frameworks are suitable for the integration with Serverless backends :heavy_check_mark:

Knative - flying with CI / CD using Bash & Groovy in Jenkins

Last but not least we need to implement a build, test, release, deploy lifecycle to support our Agile-minded Software Development Lifecycle (SDLC). Obviously, security-testing has to be part of the continuous automation processes.

Static Container-image security scans with Clair fit for Continuous Integration (CI)

How to use Clair

This requires an up to date installation of Go and a valid GOPATH. You also may need to extend the environment PATH (for the env of the Jenkins user).

Example (incl. the installation of dependencies, assuming a user in the docker user group):

docker run -p 5433:5433 -d --name clair_db arminc/clair-db:latest
docker run -p 6060:6060 --link clair_db:postgres -d --name my_clair arminc/clair-local-scan
mkdir ~/go
export GOPATH=$HOME/go
export PATH=$PATH:$GOPATH/bin/
curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh
cd ~/go/src/github.com/arminc/clair-scanner
make ensure && make installLocal
# for Vmware ESXi / for KVM use eth0 for example
$IP=$(ip addr show ens160 | awk '/inet/ {print $2}' | cut -d/ -f1 | head -n 1)

Results (assume set variables, not whitelisted CVEs):

clair-scanner --ip $IP wishi/chuckjokes:latest
2020/04/21 11:16:26 [INFO] ▶ Start clair-scanner
2020/04/21 11:16:30 [INFO] ▶ Server listening on port 9279
2020/04/21 11:16:30 [INFO] ▶ Analyzing 9...6
...
2020/04/21 11:16:46 [INFO] ▶ Image [wishi/chuckjokes:latest] contains NO unapproved vulnerabilities

We are good! :partying_face:

Static code-analysis of Kotlin code for security issues technical debt

For those issues that are more like low-hanging fruits, I still use SonarQube[19] on my personal lab. It has got support for Kotlin. In the following screenshot I filted down the available rules to those, that are tagged as “Vulnerability”.

PMD - Static Analysis Security Testing

PMD[20] is one of the more traditional code-quality tools, which can analyze the Abstract Syntax Tree (AST) and which is not just a grep-like Linter[21].

For code-duplication:

pmd/pmd-dist/target/pmd-bin-6.23.0-SNAPSHOT/bin$ ./run.sh cpd --minimum-tokens 10 --files ~/Source/chucknorris-kotlin-spring/src/

For Continuous Integration testing consider PMD’s --failOnViolation. I haven’t found it to be as helpful as ktlint and detekt for Kotlin CI. Comerically I have used some expensive SAST tools, that also haven’t provided better results.

Detekt - debt

(base) [email protected]:~/Source/chucknorris-kotlin-spring$ java -jar detekt-cli-1.8.0-all.jar 
empty-blocks - 5min debt
	EmptyFunctionBlock - [contextLoads] at /home/ww/Source/chucknorris-kotlin-spring/src/test/kotlin/com/becausesec/chuckjokes/ChuckjokesApplicationTests.kt:13:24
...
	NewLineAtEndOfFile - [JokesRestController.kt] at /home/ww/Source/chucknorris-kotlin-spring/src/main/kotlin/com/becausesec/chuckjokes/controllers/JokesRestController.kt:24:1

Overall debt: 45min

Detekt will also work with SonarQube[22], via Maven or Gradle. I prefer having these steps with a nightly Jenkins job in my lab instead of having them as part of my builds. For larger DevOps teams this needs to be evaluated.

OWASP dependency checker

(base) [email protected]:~/Source/DependencyCheck/CLI/dependency-check$ ./bin/dependency-check.sh --out . --scan ~/Source/chucknorris-kotlin-spring/target/chuckjokes-0.0.1-SNAPSHOT.jar
[INFO] Checking for updates
[INFO] Skipping NVD check since last check was within 4 hours.
[INFO] Skipping RetireJS update since last update was within 24 hours.
[INFO] Check for updates complete (86 ms)
[INFO] 
...
[INFO] Analysis Started
[INFO] Finished Archive Analyzer (1 seconds)
[INFO] Finished File Name Analyzer (0 seconds)
[INFO] Finished Jar Analyzer (0 seconds)
[INFO] Finished Central Analyzer (0 seconds)
[INFO] Finished Dependency Merging Analyzer (0 seconds)
[INFO] Finished Version Filter Analyzer (0 seconds)
[INFO] Finished Hint Analyzer (0 seconds)
[INFO] Created CPE Index (1 seconds)
[INFO] Finished CPE Analyzer (2 seconds)
[INFO] Finished False Positive Analyzer (0 seconds)
[INFO] Finished NVD CVE Analyzer (0 seconds)
[INFO] Finished Sonatype OSS Index Analyzer (0 seconds)
[INFO] Finished Vulnerability Suppression Analyzer (0 seconds)
[INFO] Finished Dependency Bundling Analyzer (0 seconds)
[INFO] Analysis Complete (4 seconds)

There are Maven and Gradle plugins for this. Or Jenkins plugins…

Building a Jenkins pipeline - security and the grain of salt

I am playing around with this, and this is another project. – There are multiple challenges here:

  • sometimes you need to add plugins to your build tools (Maven or Gradle) but I like to keep build files slim. Builds need to be fast, and build files need to be simple. Maven and Gradle need a long time to download all the dependencies. We don’t need to add more.
  • there are better ways to test these things than using an overloaded Jenkins, that produces tons of logs no one is reading…
  • Jenkins is feature-rich but it’s messy as well. I don’t necessarily think of Jenkins as a good place to run CodeQL or other advanced tools that support a Secure Software Development Lifecycle. These tools need a lot of computation resources.

Homework: add security to this without making it complex

node {

    def app

       stage('Clone repository') {
            /* Let's make sure we have the repository cloned to our workspace */

            checkout scm
        }

       stage('Build image') {
           /* This builds the actual image; synonymous to
            * docker build on the command line */

                app = docker.build("wishi/chuckjokes")
       }

       stage('Test image') {
                /* Ideally, we would run a test framework against our image.
                * For this example, we're using a Volkswagen-type approach ;-) */

               app.inside {
                        sh 'echo "Tests passed"'
                }
            }
}
image

We can see projects like Tekton[23] that consolidate the CI / CD into Kubernetes. So you’d target your build to be created within a Kube cluster that’s located in a staging environment. – Bringing security into Serverless has to go into that direction as well.

Summary:

  • showcased some software security testing tools, but running them doesn’t yield results unless they are integrated well. :x:
  • demonstration of static security tests of docker containers :heavy_check_mark:
  • demonstration of some fancy code-scanners for Kotlin that do something, but who knows what … :slight_smile:
  • defined why Serverless and MicroServices may develop into a direction where security testing needs to be integrated differently :heavy_check_mark:

Summary & wrap up

I produced this essay during COVID19. Here in Germany, working for an eCommerce company, I remain mostly unaffected so far. In our sector we should certainly see this with some humility.

Maybe this essay helps to convey an understanding. It’s a little long, but the website here has got a good search function. No one will read it in its entirety :wink:

Wrap up

  • executive motivation and vision of opportunities enabled through Serverless :heavy_check_mark:
  • refined terminology regarding Serverless and Function As A Service :heavy_check_mark:
  • the technical expose is centered around real-world Enterprise / Startup needs :heavy_check_mark:
  • preview of full-stack development technology fit for modern Serverless engineering labs :heavy_check_mark:
  • provided contact point to reach out in case of questions :heavy_check_mark:
  • installed Lab Kube environment as “computing backbone” :heavy_check_mark:
  • short summary of elements within a basic Kubernetes architecture :heavy_check_mark:
  • demo installation of Istio to extend the basic Kubernetes architecture for Serverless :heavy_check_mark:
  • installed Istio and Knative in a Lab Kube environment for Serverless :heavy_check_mark:
  • we are ready for test deployments :heavy_check_mark:
  • brief mentioning of advanced Kubernetes concepts as much as they relate to Istio and Knative :heavy_check_mark:
  • deployed Go-based sample app for Knative into Kube cluster :heavy_check_mark:
  • used curl and parallel for some basic testing :heavy_check_mark:
  • deployed auto-scaling on-premise cloud-agnostic Serverless workload with Knative :heavy_check_mark:
  • serving a REST API using SpringBoot and Kotlin :heavy_check_mark:
  • consuming a REST endpoint with curl :heavy_check_mark:
  • load-testing a REST API with wrk and live-observation of the system :heavy_check_mark:
  • deployed FaaS workload with on-prem Kubeless using Serverless framework :heavy_check_mark:
    • this asserts compatibility to big Serverless clouds such as AWS’s Lambda.
  • handled a popular Python Machine Learning runtime dependency in a FaaS deployable :heavy_check_mark:
  • developed a basic JAMstack frontend with React and Gatsby that consumes a REST API :heavy_check_mark:
  • learned how to adopt React Components for development and testing :heavy_check_mark:
  • understood that modern JAMstack frameworks are suitable for the integration with Serverless backends :heavy_check_mark:
  • showcased some software security testing tools, but running them doesn’t yield results unless they are integrated well. :x:
  • demonstration of static security tests of docker containers :heavy_check_mark:
  • demonstration of some fancy code-scanners for Kotlin that do something, but who knows what … :slight_smile:
  • defined why Serverless and MicroServices may develop into a direction where security testing needs to be integrated differently :heavy_check_mark:

Bringing OpenSource security testing into CI / CD without blowing it up, with the right emphasis on Serverless and MicroServices, is a future project. There are some approaches, of course. I also have some cool ideas.


  1. https://spring.io/projects/spring-boot ↩︎

  2. https://kubeless.io/ ↩︎

  3. https://minikube.sigs.k8s.io/docs/start/linux/ ↩︎

  4. https://docs.docker.com/engine/install/ubuntu/ ↩︎

  5. https://kubernetes.io/docs/tasks/administer-cluster/coredns/ ↩︎

  6. https://kubernetes.io/docs/concepts/workloads/pods/pod/ ↩︎

  7. https://kubernetes.io/docs/concepts/services-networking/service/ ↩︎

  8. https://cilium.io/ ↩︎

  9. https://github.com/springframeworkguru/chuck-norris-for-actuator ↩︎

  10. https://hub.docker.com/r/wishi/chuckjokes ↩︎

  11. https://docs.oracle.com/javase/8/docs/technotes/guides/security/enhancements-8.html ↩︎

  12. https://jupyter-notebook.readthedocs.io/en/stable/index.html ↩︎

  13. https://kubeless.io/docs/quick-start/ ↩︎

  14. https://serverless.com/framework/docs/providers/kubeless/guide/functions/ ↩︎

  15. https://www.gatsbyjs.org/ ↩︎

  16. https://www.cvedetails.com/product/4096/Wordpress-Wordpress.html?vendor_id=2337 ↩︎

  17. https://www.youtube.com/watch?v=0Ta-awtLZTs ↩︎

  18. https://jamstack.org/ ↩︎

  19. https://www.sonarqube.org ↩︎

  20. https://pmd.github.io/latest/index.html ↩︎

  21. https://ktlint.github.io/#getting-started ↩︎

  22. https://github.com/detekt/sonar-kotlin ↩︎

  23. https://cloud.google.com/tekton ↩︎