In this article we will parse our apache logs from previous topic by logstash. In a nutshell about logstash config format...    Logstash config consist of three main part:   input {} filter {} output {}   Input field can contain few inputs. It could be stdin, tcp, udp, relp or file. For tests you could use tcp input and stdout output like below:   input { tcp { port => 3333 } } output { stdout { codec => json debug => true } }   Start your logstash with this config and do next on another machine:   nc -v your_logstash_machine_addres...
Published ago by ochirkov

Hi everyone. In this topic I put grok's regexes. Actually you could find them on GitHub but it is comfortably when you read about grok and could look to regexes at this moment. So, list of regexes: USERNAME [a-zA-Z0-9_-]+ USER %{USERNAME} INT (?:[+-]?(?:[0-9]+)) BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))) NUMBER (?:%{BASE10NUM}) BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+)) BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b POSINT \b(?:[1-9][0-9]*)\b NONNEGINT \b(?:[0-9]+)\b WORD \b\w+\b NOTSPACE \S+ SPACE \s* ...
Published ago by ochirkov

Beaver is a python based shipper messages to Logstash. Let's install it. rpm -ivh http://dl.fedoraproject.org/pub/epel/6/i386/epel-release-6-8.noarch.rpm yum install -y python-pip pip install beaver==31 Beaver can read configuration from conf file. It's file in ini format. Let's create it: mkdir -p /etc/beaver.d/ vi conf.ini [beaver] logstash_version: 1 [/var/log/{messages,secure}] type: syslog tags: sys Run shipper. Specify path to config and type of output: beaver -c /etc/beaver.d/conf.ini -t stdout Check it: echo "foo" >> /var/log/messages You should see: {"tags": ["sys"], "@versio...
Published ago by ochirkov

Python
Web development
Review Linux
Administration
Provisioning CVS
Windows AWS
Continuous Integration
NoSql Logging Web Servers FileSystems Scala
accurev ansible apach2 apache api application approaches architecture archivation argparse arguments artifactory automation awk aws backup bash batch-scripting beaver block build built-in caching call captcha cartridge case certificates cgi chain client cloud cloudformation cluster cmd coding collections command-line commands compression conditionals conversion convert cookbook copy counter cron crud css cut cygwin daemon datatype date dd debpackage decorator delattr deploy deque dict distributed-file-systems django DNS domain driver dropbox dump elasticcache elasticsearch encryption exit ext4 extra-tags failover file filename filters for form format freetds functions gerrit getattribute gid git globals glusterfs hardware hook iam indexing inheritance init install job-interview jquery ldap linux list locals logging logs logstash lookup magicmethods mail main-menu metadata metalogger mongodb moosefs mount mssql multiplatform multithreading mysql netcat nginx nosql open-ssh openldap openshift os packaging parse partition path pattern patterns permissions pid pil pip pipe playbook pool post post-commit processes production provisioning proxy putty python python-mysql recursively redirection redis register replication repr restore return review rotation scala script search selenium server setattr settings setup shipper signals singleton slots snapshot socket splunk ssh ssh-key ssl storage str string style subprocess sugar super switch syntactic syntaxhighlighter systeminfo tail tar templatetags time tls tune2fs tuple ubuntu unicode unique unix unixodbc usage usecases uuid uwsgi variable vars version vi virtualenv volume web web-server windows with_items __getattr__