• Integrating Logstash with Splunk to forward and analyze Kubernetes Logs:
    Basic Overview:
    1. Infra Logs are generated and pushed to a centralised server, from there it has to be forwarded.
    2. We can use logstash as a way to forward the logs (events) to Splunk.  (In this case Kubernetes logs)
    3. Filebeat, logstash, ElasticSearch, they are all in the flow of taking the  logs and persisting to a database.
    4. So we have many points of integration between these components and Splunk.
    Enabling HTTP Event Collector in Splunk:
    1. Before we configure logstash, we need to enable Splunk to receive HTTP Events.
    2. Follow this page to enable HTTP Event Collector (HEC) and generate a token
    Configure logstash to send events to Splunk:
          1.Run the following command to retrieve the logstash pipeline configuration:
                   kubectl get cm logstash-pipeline -n kube-system -o yaml > logstash-pipeline.yaml
           2. Now open the file logstash-pipeline.yaml with your favorite editor, as we need to update it with your Splunk token.
                   Search for the following section in the YAML file:

                      output {
                          elasticsearch {
                                    index => "logstash-%{+YYYY.MM.dd}"
                                   hosts => "elasticsearch:9200"
                        }headers => ["Authorization", "Splunk <your token>"]

            3.  Add the following output in this section, replacing <your token> with the Splunk token generated in the previous step:

                         http {
                           http_method => "post"
                            url => "http://splunkenterprise:8088/services/collector/event/1.0"
                           headers => ["Authorization", "Splunk <your token>"]
                              mapping => {
                                 "event" => "%{log}"


    Deploy the new configuration:

          1. We are now ready to deploy the logstash pipeline configuration.

          2.  Run the following command to replace the current configuration:
                     kubectl replace cm logstash-pipeline -f logstash-pipeline.yaml

          3. Then we need to recycle the logstash Pod. Run the following command to find the existing Pod:
                    kubectl get po -n kube-system | grep logstash

          4. You will see an output like this, noticing the Pod ID:

                   patro:tmp edu$ kubectl get po -n kube-system | grep logstash
                   logstash-5c8c4954d9-gzkdt                                 1/1       Running            0          2h

    Now delete the Pod:

                  kubectl delete po -n kube-system <pod-id>

                Kubernetes will start a new Pod with the refreshed configuration (from the ConfigMap). You can see the output by running the following command:

                   kubectl logs -f $(kubectl get po -n kube-system | grep logstash | awk '{print $1}')
    Set up and use HTTP Event Collector in Splunk Web - Splunk Documentation
  • great does anyone offer alienvalut or log rhythm as service ?

  • low cost MSSP can be a good evaluate !

  • thanks daniel singh ! will RELEARN my info security basics as per your advise.

  • You might want to look at how each one is compared to: Metrices and dashboards, Ability to create custom dashboards, Real time updates, Compliance packages to aid in compliance reporting,log management,Event Detection, Analytics & Visualization, CISO's Incident dashboard,Alerts and notifications ... & more

    Here is a quick comparison of Splunk vs Alien Vault vs Logrhythm:Click Here for Comparisons

    Of course you could compare other 30+ vendors who offer similar solutions.

  • ELK Stack is a low cost option that works well.

  • Hello,

    Are you looking for an MSSP or a self-setup?

  • Hi,

    I would suggest search around for Managed Detection & Response vendors. I don't want name a vendor but there are many around such as Alienvault, Arctic Wolf, Channel SOC etc.

    A SIEM tool (such as splunk) is big component of SOC capability. In itself, such a tool is only a technology piece but for an effective SOC, the other components - people (eg: 24x7 monitoring) and processes (eg: incident response) need to be defined as well. (consider all three - People/Process/Technology)

    Good luck!

  • Hello, sorry but did you say SPLUNK?

    If you already have splunk and still ask this question then please you need to RELEARN your info security basics.

    Sorry to sound harsh but splunk does the job pretty well and much more, your team should not need anything else at all.

This reply was deleted.