Elk Stack Setup on Amazon Linux

Image Courtesy of blog.eagerelk.com

How to Setup ELK on Amazon Linux

A few months ago I needed to set up an ELK Stack on Amazon Linux, so I looked for a guide to walk me through it but I couldn’t find one. So I found myself following a guide meant for Ubuntu and figure out difference as I went ahead. It wasn’t an easy process, so if you’re looking to do the same hopefully this can save you some time.

What’s ELK Stack?

ELK stack is a term used in DevOps community to refer to 3 tools that go together very well: Elastic Search, Logstash and Kibana.

  • Elastic Search is a tool for data analytics and deep search
  • Logstash is for centralized logging, log enrichment and parsing
  • Kibana is for powerful and beautiful data visualizations

The ELK stack together provides you all you need to understand exactly what is happening in your apps on every server you have.

ELK Stack on Amazon Linux Elastic Search

Install Elastic Search

We are going to use yum to install all the packages today. Unfortunately none of the ELK Satck is available directly through standard yum repositories, so in order to be able to install them with yum you need to add repositories to your yum repos. But first you need to add the GPG key to your rpm so it will trust the repos.

rpm --import https://packages.elastic.co/GPG-KEY-elasticsearch

Next you need to add the repo to /etc/yum.repos.d/elasticsearch.repo. Use your favorite editor to create that file and add this to it

[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=https://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

The final step is to install the Elastic Search and now you can use yum to do that:

yum install elasticsearch

Start Elastic Search

To start Elastic Search you can use service command

service elasticsearch start

You probably want the Elastic Search to start on startup, to do that you can run this command

chkconfig --add elasticsearch

Now that you have Elastic Search running you should be able to use curl to see it:

curl localhost:9200

 

ELK Stack on Amazon Linux Logstash

Install Logstash

Similar to Elastic Search you need to add the repo to /etc/yum.repos.d/logstash.repo. Note that we are not importing the GPG key with rpm now because we’ve already done that in previous step. If you only want to install Logstash you need to do that too.

[logstash-2.3]
name=Logstash repository for 2.3.x packages
baseurl=https://packages.elastic.co/logstash/2.3/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1" > /etc/yum.repos.d/logstash.repo

Similarly you can now install the Logstash with yum:

yum install logstash

Start Logstash

We are going to keep it consistent. So you can use service to start Logstash.

service logstash start
service logstash status

Configure Logstash and Elastic Search

This is the tricky part. Here I’m going to show you how to have a very simple configuration and push all the syslog from /var/log/messages from Logstash to Elastic Search, but you are probably going to need to have a more complicated/sophisticated configuration later. Try this to make sure your setup works, then use the Logstash’s documentation to configure your Logstash/Elastic Search.

You can have multiple configurations for Logstash. Here we create one in this path: /etc/logstash/conf.d/logstash-syslog.conf.

input {
  file {
    type => syslog
    path => [ "/var/log/messages", "/var/log/*.log" ]
  }
}

output {
  stdout {
    codec => rubydebug
  }
  elasticsearch {
    host => "localhost" # If you are running elasticsearch in different instance, use #the private ip instead of localhost.
  }
}

In order for your configuration to take effect you need to restart your Logstash

service logstash restart

 

ELK Stack on Amazon Linux Kibana

Install Kibana

You have your Logstash, you have your Elastic Search, now you need to be able to visually see all the data and search them, etc. You need Kibana. Similar to Logstash and Elastic Search we use yum for that and you need to add the repo to /etc/yum.repos.d/kibana.repo

[kibana-4.5]
name=Kibana repository for 4.5.x packages
baseurl=http://packages.elastic.co/kibana/4.5/centos
gpgcheck=1
gpgkey=http://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

Start Kibana

Now you can install Kibana and and add it to the auto-start set of processes.

yum install kibana
chkconfig kibana

And voilà! You’re done. Just visit your website on port 5601 and you have everything ready to go.

Amin Yazdani

Amin Yazdani

Director of Technology at A.Y. Technologies
Founder of A.Y. Technologies, Amin is formerly Senior Software Architect at Dun & Bradstreet and Solutions Engineer at Indicee. He has a wealth of experience and understanding of the software and applications industry, particularly as it pertains to cloud computing, and web application solutions.
Amin Yazdani

Latest posts by Amin Yazdani (see all)

0 Comments

Leave a reply

Your email address will not be published. Required fields are marked *

©2019 A.Y.Technologies

Log in with your credentials

Forgot your details?