fire 

siren 

female 

finally 

apple 

Airport 

 

 

 

F다음에 [ie] -> i아니면 e 다음에 온다 

 

 

 

abcd.txt 

abce.txt 

abc0.txt 

abcf.txt 

abc1.txt 

abc2.txt 

abc3.txt 

 

 

보면 ...으로 3자리를 나타내고 그다음 0부터 9까지 아무 숫자나 들어가며 찾는다. 

 

좀더 쉽게 하기 위해 하이픈 -  이용 

 

[a-z]도가능 [a-ZA-Z0-9] 이렇게 이어서도 가능 

 

그럼 포함하지안흔것들은 어떻게 적용케럿 이용 ^ 

 

 

3자리이면서 그다음0-9 포함하지않고 그다음 글자를 찾았다. 

 

 

 


'오락기 > 정규표현식' 카테고리의 다른 글

정규표현식 하위 표현  (0) 2018.05.11
정규표현식 수량자  (0) 2018.05.11
정규표현 연속문자  (0) 2018.05.11
정규표현식 메타문자  (0) 2018.05.11
정규 표현식 - 시작  (0) 2018.05.11

ABCText Text ITexty 

 

ioTextm Textll abcdety 

 

 

대소문자  I플래그 

 

 

 

G 글로벌 그러니깐 처음찾는거 1개인가 아니면 전체문서인가 

 

 

 

 

I was fifty years old to-day. Half a century has hurried by since I first lay in my mother's wondering arms. To be sure, I am not old; but I can no longer deceive myself into believing that I am still young. After all, the illusion of youth is a mental habit consciously encouraged to defy and face down the reality of age. If, at twenty, one feels that he has reached man's estate he, nevertheless, tests his strength and abilities, his early successes or failures, by the temporary and fictitious standards of youth. 

 

 문장은 

 

 

점을 늘리면 

 

 

 

 

늘릴때마다 자릿수 와일드카드 느낌으로 간다. 

 

 

 

하지만 이렇게 되엇을때 . 끝나는 문장은 어떻게 표현해야할까면 

 

 

역슬래시로 표현가능하다 \. 일반 문장 


'오락기 > 정규표현식' 카테고리의 다른 글

정규표현식 하위 표현  (0) 2018.05.11
정규표현식 수량자  (0) 2018.05.11
정규표현 연속문자  (0) 2018.05.11
정규표현식 메타문자  (0) 2018.05.11
정규 표현식 문자집합  (0) 2018.05.11

#-------------------------- Elasticsearch output ------------------------------ 
#output.elasticsearch: 
  # Array of hosts to connect to. 
  #hosts: ["localhost:9200"] 
 

# Optional protocol and basic auth credentials. 
  #protocol: "https" 
  #username: "elastic" 
  #password: "changeme" 
 

#----------------------------- Logstash output -------------------------------- 
output.logstash: 
  # The Logstash hosts 
  hosts: ["localhost:5044"] 
 

# Optional SSL. By default is off. 
  # List of root certificates for HTTPS server verifications 
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"] 
 

# Certificate for SSL client authentication 
  #ssl.certificate: "/etc/pki/client/cert.pem" 
 

# Client Certificate Key 
  #ssl.key: "/etc/pki/client/cert.key" 
 

  • 읽어들일 파일이 있는 위치를 지정한다. 

paths: 
    - /home/yongho/test/*.log 
 

  • 실행 

$ ./filebeat -e -c filebeat.yml 


'오락기 > ELK' 카테고리의 다른 글

은전한닢 한글 형태소 분석기 적용  (0) 2018.05.21
Logstash multilLine java stackTrace  (0) 2018.05.11
logstash 아파치 엑세스 로그 필터  (0) 2018.05.11
logstash grok  (0) 2018.05.11
logstash data input  (0) 2018.05.11

익셉션의 멀티 라인을 

파싱하는 작업이 필요하여 해봤다



 

# stacktrace java as one message 

 

 

multiline { 

 

#type => "all" # no type means for all inputs 

 

pattern => "(^.+Exception: .+)|(^\s+at .+)|(^\s+... \d+ more)|(^\s*Caused by:.+)" 

 

what => "previous" 

 

} 

 

com.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Empty key 

 

 

at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71) 

 

at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:55) 

 

at com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:83) 

 

at com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:46) 

 

at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:238) 

 

at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170) 

 

at com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:776) 

 

at com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:564) 

 

at com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:732) 

 

at com.base2services.jenkins.SqsProfile.createQueue(SqsProfile.java:72) 

 

at com.base2services.jenkins.SqsProfile.getQueueUrl(SqsProfile.java:62) 

 

at com.base2services.jenkins.SqsQueueHandler.doRun(SqsQueueHandler.java:37) 

 

at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) 

 

at java.util.TimerThread.mainLoop(Timer.java:555) 

 

at java.util.TimerThread.run(Timer.java:505) 

 

Caused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Empty key 

 

at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90) 

 

at com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68) 

 

... 14 more 

 

Caused by: java.lang.IllegalArgumentException: Empty key 

 

at javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96) 

 

at com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:87) 

 

... 15 more 

 

juil. 25, 2012 10:49:54 AM hudson.slaves.SlaveComputer tryReconnect 

 

Infos: Attempting to reconnect CentosVagrant 


 

결과 

 

{"hits":[ 
 

{"_index":"logstash-2012.07.26","_type":"dummy","_id":"u5S8Bg2RRo-YutkXuSCoKQ","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log" 
,"@type":"dummy" 
,"@tags":[] 
,"@fields":{} 
,"@timestamp":"2012-07-26T13:35:04.457000Z" 
,"@source_host":"centos-6-vagrant.vagrantup.com" 
,"@source_path":"/vagrant_projet/logstash/stacktrace.log" 
,"@message":"juil. 25, 2012 10:49:46 AM hudson.triggers.SafeTimerTask run\r"},"sort":[1343309704457]}, 
 

{"_index":"logstash-2012.07.26","_type":"dummy","_id":"zyeCfYboStC1oofKFMW8-w","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log" 
,"@type":"dummy" 
,"@tags":["multiline"] 
,"@fields":{} 
,"@timestamp":"2012-07-26T13:35:04.474000Z" 
,"@source_host":"centos-6-vagrant.vagrantup.com" 
,"@source_path":"/vagrant_projet/logstash/stacktrace.log" 
,"@message":"Grave: Timer task com.base2services.jenkins.SqsQueueHandler@32eea79d failed\r\ncom.amazonaws.AmazonClientException: Unable to calculate a request signature: Unable to calculate a request signature: Empty key\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:71)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:55)\r\n\tat com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:83)\r\n\tat com.amazonaws.auth.QueryStringSigner.sign(QueryStringSigner.java:46)\r\n\tat com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:238)\r\n\tat com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:170)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.invoke(AmazonSQSClient.java:776)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:564)\r\n\tat com.amazonaws.services.sqs.AmazonSQSClient.listQueues(AmazonSQSClient.java:732)\r\n\tat com.base2services.jenkins.SqsProfile.createQueue(SqsProfile.java:72)\r\n\tat com.base2services.jenkins.SqsProfile.getQueueUrl(SqsProfile.java:62)\r\n\tat com.base2services.jenkins.SqsQueueHandler.doRun(SqsQueueHandler.java:37)\r\n\tat hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54)\r\n\tat java.util.TimerThread.mainLoop(Timer.java:555)\r\n\tat java.util.TimerThread.run(Timer.java:505)\r\nCaused by: com.amazonaws.AmazonClientException: Unable to calculate a request signature: Empty key\r\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:90)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.signAndBase64Encode(AbstractAWSSigner.java:68)\r\n\t... 14 more\r\nCaused by: java.lang.IllegalArgumentException: Empty key\r\n\tat javax.crypto.spec.SecretKeySpec.<init>(SecretKeySpec.java:96)\r\n\tat com.amazonaws.auth.AbstractAWSSigner.sign(AbstractAWSSigner.java:87)\r\n\t... 15 more\r"},"sort":[1343309704474]}, 
 

{"_index":"logstash-2012.07.26","_type":"dummy","_id":"efd9s2rgTFeNiatd__pofg","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log" 
,"@type":"dummy" 
,"@tags":[] 
,"@fields":{} 
,"@timestamp":"2012-07-26T13:35:04.599000Z" 
,"@source_host":"centos-6-vagrant.vagrantup.com" 
,"@source_path":"/vagrant_projet/logstash/stacktrace.log" 
,"@message":"\r"},"sort":[1343309704599]}, 
 

{"_index":"logstash-2012.07.26","_type":"dummy","_id":"xcxgNxRUTb2syet--n3CsQ","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log" 
,"@type":"dummy" 
,"@tags":[] 
,"@fields":{} 
,"@timestamp":"2012-07-26T13:35:04.610000Z" 
,"@source_host":"centos-6-vagrant.vagrantup.com" 
,"@source_path":"/vagrant_projet/logstash/stacktrace.log" 
,"@message":"juil. 25, 2012 10:49:54 AM hudson.slaves.SlaveComputer tryReconnect\r"},"sort":[1343309704610]}, 
 

{"_index":"logstash-2012.07.26","_type":"dummy","_id":"WwoynUlNSl6iAMzFw-4gZg","_score":null, "_source" : {"@source":"file://centos-6-vagrant.vagrantup.com/vagrant_projet/logstash/stacktrace.log" 
,"@type":"dummy" 
,"@tags":[] 
,"@fields":{} 
,"@timestamp":"2012-07-26T13:35:04.631000Z" 
,"@source_host":"centos-6-vagrant.vagrantup.com" 
,"@source_path":"/vagrant_projet/logstash/stacktrace.log" 
,"@message":"Infos: Attempting to reconnect CentosVagrant\r"},"sort":[1343309704631]} 
 

]} 

 

 

 

 


'오락기 > ELK' 카테고리의 다른 글

은전한닢 한글 형태소 분석기 적용  (0) 2018.05.21
file beat 번외  (0) 2018.05.11
logstash 아파치 엑세스 로그 필터  (0) 2018.05.11
logstash grok  (0) 2018.05.11
logstash data input  (0) 2018.05.11

input {  

  beats { 

    port => 5044 

  } 

} 

 

 

filter {   

  grok { 

    match => { "message" => ["%{COMBINEDAPACHELOG} %{NUMBER:time}"]} 

  } 

  grok { 

    match => { "request" => ["%{URIPATH:uripath}"] } 

  } 

  grok { 

    match => { "referrer" => ["%{URIPROTO}://%{IPORHOST:referrer_host}(?::%{POSINT})?%{URIPATH:referrer_uripath}"] } 

  } 

  date { 

    match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ] 

    target => "@timestamp" 

  } 

  kv { 

    source => "request" 

    field_split => "&?" 

    include_keys => ["name"] 

  } 

  kv { 

    source => "cookie" 

    field_split => "; " 

    include_keys => [ "id"]    

  } 

  useragent { 

    source => "agent" 

    prefix => "a_" 

  } 

  

  ruby { 

    code => "event['index_day'] = event['@timestamp'].time.localtime.strftime('%Y%m%d')" 

  } 

  mutate { 

    convert => { 

       "time" => "integer" 

       "bytes" => "integer"}     

  } 

} 

 

 

output { 

        

                elasticsearch { 

                        hosts => "localhost:9200" 

                        manage_template => false 

                        index => "acc_test2" 

                        document_type => "%{[@metadata][type]}" 

} 

} 


결과

접속 카운트와 ip정보로 가져온 위치 정보






'오락기 > ELK' 카테고리의 다른 글

file beat 번외  (0) 2018.05.11
Logstash multilLine java stackTrace  (0) 2018.05.11
logstash grok  (0) 2018.05.11
logstash data input  (0) 2018.05.11
kibana managment  (0) 2018.05.11

USERNAME [a-zA-Z0-9._-]+ 
USER %{USERNAME} 
INT (?:[+-]?(?:[0-9]+)) 
BASE10NUM (?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))) 
NUMBER (?:%{BASE10NUM}) 
BASE16NUM (?<![0-9A-Fa-f])(?:[+-]?(?:0x)?(?:[0-9A-Fa-f]+)) 
BASE16FLOAT \b(?<![0-9A-Fa-f.])(?:[+-]?(?:0x)?(?:(?:[0-9A-Fa-f]+(?:\.[0-9A-Fa-f]*)?)|(?:\.[0-9A-Fa-f]+)))\b 
 

POSINT \b(?:[1-9][0-9]*)\b 
NONNEGINT \b(?:[0-9]+)\b 
WORD \b\w+\b 
NOTSPACE \S+ 
SPACE \s* 
DATA .*? 
GREEDYDATA .* 
QUOTEDSTRING (?>(?<!\\)(?>"(?>\\.|[^\\"]+)+"|""|(?>'(?>\\.|[^\\']+)+')|''|(?>`(?>\\.|[^\\`]+)+`)|``)) 
UUID [A-Fa-f0-9]{8}-(?:[A-Fa-f0-9]{4}-){3}[A-Fa-f0-9]{12} 
 

# Networking 
MAC (?:%{CISCOMAC}|%{WINDOWSMAC}|%{COMMONMAC}) 
CISCOMAC (?:(?:[A-Fa-f0-9]{4}\.){2}[A-Fa-f0-9]{4}) 
WINDOWSMAC (?:(?:[A-Fa-f0-9]{2}-){5}[A-Fa-f0-9]{2}) 
COMMONMAC (?:(?:[A-Fa-f0-9]{2}:){5}[A-Fa-f0-9]{2}) 
IPV6 ((([0-9A-Fa-f]{1,4}:){7}([0-9A-Fa-f]{1,4}|:))|(([0-9A-Fa-f]{1,4}:){6}(:[0-9A-Fa-f]{1,4}|((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){5}(((:[0-9A-Fa-f]{1,4}){1,2})|:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3})|:))|(([0-9A-Fa-f]{1,4}:){4}(((:[0-9A-Fa-f]{1,4}){1,3})|((:[0-9A-Fa-f]{1,4})?:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){3}(((:[0-9A-Fa-f]{1,4}){1,4})|((:[0-9A-Fa-f]{1,4}){0,2}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){2}(((:[0-9A-Fa-f]{1,4}){1,5})|((:[0-9A-Fa-f]{1,4}){0,3}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(([0-9A-Fa-f]{1,4}:){1}(((:[0-9A-Fa-f]{1,4}){1,6})|((:[0-9A-Fa-f]{1,4}){0,4}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:))|(:(((:[0-9A-Fa-f]{1,4}){1,7})|((:[0-9A-Fa-f]{1,4}){0,5}:((25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)(\.(25[0-5]|2[0-4]\d|1\d\d|[1-9]?\d)){3}))|:)))(%.+)? 
IPV4 (?<![0-9])(?:(?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2})[.](?:25[0-5]|2[0-4][0-9]|[0-1]?[0-9]{1,2}))(?![0-9]) 
IP (?:%{IPV6}|%{IPV4}) 
HOSTNAME \b(?:[0-9A-Za-z][0-9A-Za-z-]{0,62})(?:\.(?:[0-9A-Za-z][0-9A-Za-z-]{0,62}))*(\.?|\b) 
HOST %{HOSTNAME} 
IPORHOST (?:%{HOSTNAME}|%{IP}) 
HOSTPORT %{IPORHOST}:%{POSINT} 
 

# paths 
PATH (?:%{UNIXPATH}|%{WINPATH}) 
UNIXPATH (?>/(?>[\w_%!$@:.,-]+|\\.)*)+ 
TTY (?:/dev/(pts|tty([pq])?)(\w+)?/?(?:[0-9]+)) 
WINPATH (?>[A-Za-z]+:|\\)(?:\\[^\\?*]*)+ 
URIPROTO [A-Za-z]+(\+[A-Za-z+]+)? 
URIHOST %{IPORHOST}(?::%{POSINT:port})? 
# uripath comes loosely from RFC1738, but mostly from what Firefox 
# doesn't turn into %XX 
URIPATH (?:/[A-Za-z0-9$.+!*'(){},~:;=@#%_\-]*)+ 
#URIPARAM \?(?:[A-Za-z0-9]+(?:=(?:[^&]*))?(?:&(?:[A-Za-z0-9]+(?:=(?:[^&]*))?)?)*)? 
URIPARAM \?[A-Za-z0-9$.+!*'|(){},~@#%&/=:;_?\-\[\]]* 
URIPATHPARAM %{URIPATH}(?:%{URIPARAM})? 
URI %{URIPROTO}://(?:%{USER}(?::[^@]*)?@)?(?:%{URIHOST})?(?:%{URIPATHPARAM})? 
 

# Months: January, Feb, 3, 03, 12, December 
MONTH \b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b 
MONTHNUM (?:0?[1-9]|1[0-2]) 
MONTHNUM2 (?:0[1-9]|1[0-2]) 
MONTHDAY (?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) 
 

# Days: Monday, Tue, Thu, etc... 
DAY (?:Mon(?:day)?|Tue(?:sday)?|Wed(?:nesday)?|Thu(?:rsday)?|Fri(?:day)?|Sat(?:urday)?|Sun(?:day)?) 
 

# Years? 
YEAR (?>\d\d){1,2} 
HOUR (?:2[0123]|[01]?[0-9]) 
MINUTE (?:[0-5][0-9]) 
# '60' is a leap second in most time standards and thus is valid. 
SECOND (?:(?:[0-5]?[0-9]|60)(?:[:.,][0-9]+)?) 
TIME (?!<[0-9])%{HOUR}:%{MINUTE}(?::%{SECOND})(?![0-9]) 
# datestamp is YYYY/MM/DD-HH:MM:SS.UUUU (or something like it) 
DATE_US %{MONTHNUM}[/-]%{MONTHDAY}[/-]%{YEAR} 
DATE_EU %{MONTHDAY}[./-]%{MONTHNUM}[./-]%{YEAR} 
ISO8601_TIMEZONE (?:Z|[+-]%{HOUR}(?::?%{MINUTE})) 
ISO8601_SECOND (?:%{SECOND}|60) 
TIMESTAMP_ISO8601 %{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE}? 
DATE %{DATE_US}|%{DATE_EU} 
DATESTAMP %{DATE}[- ]%{TIME} 
TZ (?:[PMCE][SD]T|UTC) 
DATESTAMP_RFC822 %{DAY} %{MONTH} %{MONTHDAY} %{YEAR} %{TIME} %{TZ} 
DATESTAMP_RFC2822 %{DAY}, %{MONTHDAY} %{MONTH} %{YEAR} %{TIME} %{ISO8601_TIMEZONE} 
DATESTAMP_OTHER %{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{TZ} %{YEAR} 
DATESTAMP_EVENTLOG %{YEAR}%{MONTHNUM2}%{MONTHDAY}%{HOUR}%{MINUTE}%{SECOND} 
 

# Syslog Dates: Month Day HH:MM:SS 
SYSLOGTIMESTAMP %{MONTH} +%{MONTHDAY} %{TIME} 
PROG (?:[\w._/%-]+) 
SYSLOGPROG %{PROG:program}(?:\[%{POSINT:pid}\])? 
SYSLOGHOST %{IPORHOST} 
SYSLOGFACILITY <%{NONNEGINT:facility}.%{NONNEGINT:priority}> 
HTTPDATE %{MONTHDAY}/%{MONTH}/%{YEAR}:%{TIME} %{INT} 
 

# Shortcuts 
QS %{QUOTEDSTRING} 
 

# Log formats 
SYSLOGBASE %{SYSLOGTIMESTAMP:timestamp} (?:%{SYSLOGFACILITY} )?%{SYSLOGHOST:logsource} %{SYSLOGPROG}: 
COMMONAPACHELOG %{IPORHOST:clientip} %{USER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-) 
COMBINEDAPACHELOG %{COMMONAPACHELOG} %{QS:referrer} %{QS:agent} 
 

# Log Levels 
LOGLEVEL ([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?) 

 


'오락기 > ELK' 카테고리의 다른 글

Logstash multilLine java stackTrace  (0) 2018.05.11
logstash 아파치 엑세스 로그 필터  (0) 2018.05.11
logstash data input  (0) 2018.05.11
kibana managment  (0) 2018.05.11
elasticsearch aggregation  (0) 2018.05.11

Country,1980,1981,1982,1983,1984,1985,1986,1987,1988,1989,1990,1991,1992,1993,1994,1995,1996,1997,1998,1999,2000,2001,2002,2003,2004,2005,2006,2007,2008,2009,2010 

North America,320.27638,324.44694,328.62014,332.72487,336.72143,340.74811,344.89548,349.07829,353.2939,357.68457,362.4468,367.70684,373.29069,378.74233,383.9166,388.97216,393.9428,398.97205,403.85585,408.60296,413.3245,417.83236,422.05268,426.06238,430.26938,434.47232,438.82964,443.3473,447.67394,451.83698,456.59331 

이러한 데이터가 있다. 

 

이걸 로그스테이시로 수집해본다 

 

input { 

  file { 

    path => "/usr/elk/logstash/p1.csv" <--경로 

    start_position => "beginning"   <--처음부터 

    sincedb_path => "/dev/null"   <-- 조사해봐야함 

  } 

} 

filter { 

  csv { 

      separator => ","  <--구분자 

      columns => ["Country","1980","1981","1982","1983","1984","1985","1986","1987","1988","1989","1990","1991","1992","1993","1994","1995","1996","1997","1998","1999","2000","2001","2002","2003","2004","2005","2006","2007","2008","2009","2010"] <--컬럼 

  } 

  mutate {convert => ["1980", "float"]} <--데이터 컨버팅 

  mutate {convert => ["1981", "float"]} 

  mutate {convert => ["1982", "float"]} 

  mutate {convert => ["1983", "float"]} 

  mutate {convert => ["1984", "float"]} 

  mutate {convert => ["1985", "float"]} 

  mutate {convert => ["1986", "float"]} 

  mutate {convert => ["1987", "float"]} 

  mutate {convert => ["1988", "float"]} 

  mutate {convert => ["1989", "float"]} 

  mutate {convert => ["1990", "float"]} 

  mutate {convert => ["1991", "float"]} 

  mutate {convert => ["1992", "float"]} 

  mutate {convert => ["1993", "float"]} 

  mutate {convert => ["1994", "float"]} 

  mutate {convert => ["1995", "float"]} 

  mutate {convert => ["1996", "float"]} 

  mutate {convert => ["1997", "float"]} 

  mutate {convert => ["1998", "float"]} 

  mutate {convert => ["1999", "float"]} 

  mutate {convert => ["2000", "float"]} 

  mutate {convert => ["2001", "float"]} 

  mutate {convert => ["2002", "float"]} 

  mutate {convert => ["2003", "float"]} 

  mutate {convert => ["2004", "float"]} 

  mutate {convert => ["2005", "float"]} 

  mutate {convert => ["2006", "float"]} 

  mutate {convert => ["2007", "float"]} 

  mutate {convert => ["2008", "float"]} 

  mutate {convert => ["2009", "float"]} 

  mutate {convert => ["2010", "float"]} 

} 

output {   

    elasticsearch { 

        hosts => "localhost" <--- 엘라스틱서치 호스트 

        index => "population" <-- 인덱스 이름 

    } 

 

 

sudo ./logstash -f /usr/elk/logstash/logstash.conf <-- 관리자로 해야 되더라 권한때문인거같다 

하면 데이터가 넣어지게되고 

 

 

이런식으로 시각화 가능함 

 

 

 


'오락기 > ELK' 카테고리의 다른 글

logstash 아파치 엑세스 로그 필터  (0) 2018.05.11
logstash grok  (0) 2018.05.11
kibana managment  (0) 2018.05.11
elasticsearch aggregation  (0) 2018.05.11
elasticsearch search  (0) 2018.05.11

데이터를 삽입해주고 

 

curl -XPOST 'localhost:9200/_bulk?preety' -H 'Content-Type:application/json' --data-binary @bulk_basketball.json 

 

키바나에서 인덱스를 생성한다 

 

 

 

 

 

시간필터 지정 

 

 

결과 

 

 

 


'오락기 > ELK' 카테고리의 다른 글

logstash grok  (0) 2018.05.11
logstash data input  (0) 2018.05.11
elasticsearch aggregation  (0) 2018.05.11
elasticsearch search  (0) 2018.05.11
elasticsearch mapping  (0) 2018.05.11

수학적인 평균 최소 최대  구할수잇는 함수 

 

{ 

 

 

"size" : 0, 

 

"aggs" : { 

 

"avg_score" : { 

 

"avg" : { 

 

"field" : "points" 

 

} 

 

} 

 

} 

 

} 

 

json으로 만들고 

 

[js@localhost elasticsearch]$ curl -XGET localhost:9200/_search?pretty -H 'Content-Type:application/json' --data-binary @avg_points_aggs.json 

{ 

  "took" : 314, 

  "timed_out" : false, 

  "_shards" : { 

    "total" : 16, 

    "successful" : 16, 

    "skipped" : 0, 

    "failed" : 0 

  }, 

  "hits" : { 

    "total" : 31, 

    "max_score" : 0.0, 

    "hits" : [ ] 

  }, 

  "aggregations" : { 

    "avg_score" : { 

      "value" : 25.0 

    } 

  } 

} 

 

 

평균값을 구할수있다. 

 

이처럼 max min 기타 등등 구할  있다. 

 

 

 

 

 


'오락기 > ELK' 카테고리의 다른 글

logstash data input  (0) 2018.05.11
kibana managment  (0) 2018.05.11
elasticsearch search  (0) 2018.05.11
elasticsearch mapping  (0) 2018.05.11
elasticsearch bulk  (0) 2018.05.11

이런데이터가 있다고하자 

 

[js@localhost elasticsearch]$ curl -XGET localhost:9200/basketball/?pretty 

{ 

  "basketball" : { 

    "aliases" : { }, 

    "mappings" : { 

      "record" : { 

        "properties" : { 

          "assists" : { 

            "type" : "long" 

          }, 

          "name" : { 

            "type" : "text", 

            "fields" : { 

              "keyword" : { 

                "type" : "keyword", 

                "ignore_above" : 256 

              } 

            } 

          }, 

          "points" : { 

            "type" : "long" 

          }, 

          "rebounds" : { 

            "type" : "long" 

          }, 

          "submit_date" : { 

            "type" : "date" 

          }, 

          "team" : { 

            "type" : "text", 

            "fields" : { 

              "keyword" : { 

                "type" : "keyword", 

                "ignore_above" : 256 

              } 

            } 

          } 

        } 

      } 

    }, 

    "settings" : { 

      "index" : { 

        "creation_date" : "1519972187852", 

        "number_of_shards" : "5", 

        "number_of_replicas" : "1", 

        "uuid" : "cxRCcCVDTTCHwW75DOElXg", 

        "version" : { 

          "created" : "6020299" 

        }, 

        "provided_name" : "basketball" 

      } 

    } 

  } 

} 

 

실제데이터는 

[js@localhost elasticsearch]$ curl -XGET localhost:9200/basketball/record/1/?pretty 

{ 

  "_index" : "basketball", 

  "_type" : "record", 

  "_id" : "1", 

  "_version" : 3, 

  "found" : true, 

  "_source" : { 

    "team" : "Chicago Bulls", 

    "name" : "Michael Jordan", 

    "points" : 30, 

    "rebounds" : 3, 

    "assists" : 4, 

    "submit_date" : "1996-10-11" 

  } 

} 

[js@localhost elasticsearch]$ curl -XGET localhost:9200/basketball/record/2/?pretty 

{ 

  "_index" : "basketball", 

  "_type" : "record", 

  "_id" : "2", 

  "_version" : 3, 

  "found" : true, 

  "_source" : { 

    "team" : "Chicago Bulls", 

    "name" : "Michael Jordan", 

    "points" : 20, 

    "rebounds" : 5, 

    "assists" : 8, 

    "submit_date" : "1996-10-11" 

  } 

} 

[js@localhost elasticsearch]$ 

 

 

 

 

Record 서칭을 하게되면 

 

[js@localhost elasticsearch]$ curl -XGET localhost:9200/basketball/record/_search?pretty 

{ 

  "took" : 372, 

  "timed_out" : false, 

  "_shards" : { 

    "total" : 5, 

    "successful" : 5, 

    "skipped" : 0, 

    "failed" : 0 

  }, 

  "hits" : { 

    "total" : 2, 

    "max_score" : 1.0, 

    "hits" : [ 

      { 

        "_index" : "basketball", 

        "_type" : "record", 

        "_id" : "2", 

        "_score" : 1.0, 

        "_source" : { 

          "team" : "Chicago Bulls", 

          "name" : "Michael Jordan", 

          "points" : 20, 

          "rebounds" : 5, 

          "assists" : 8, 

          "submit_date" : "1996-10-11" 

        } 

      }, 

      { 

        "_index" : "basketball", 

        "_type" : "record", 

        "_id" : "1", 

        "_score" : 1.0, 

        "_source" : { 

          "team" : "Chicago Bulls", 

          "name" : "Michael Jordan", 

          "points" : 30, 

          "rebounds" : 3, 

          "assists" : 4, 

          "submit_date" : "1996-10-11" 

        } 

      } 

    ] 

  } 

} 

 

이렇게  인덱스를 몰라도 각각 항목에 대해서 한꺼번 조회가능함  

Uri 파라미터를 넣어서 확인도 가능함 

 

여기서 q 쿼리 

 

[js@localhost elasticsearch]$ curl -XGET 'localhost:9200/basketball/record/_search?pretty&q=points:30' 

{ 

  "took" : 72, 

  "timed_out" : false, 

  "_shards" : { 

    "total" : 5, 

    "successful" : 5, 

    "skipped" : 0, 

    "failed" : 0 

  }, 

  "hits" : { 

    "total" : 1, 

    "max_score" : 1.0, 

    "hits" : [ 

      { 

        "_index" : "basketball", 

        "_type" : "record", 

        "_id" : "1", 

        "_score" : 1.0, 

        "_source" : { 

          "team" : "Chicago Bulls", 

          "name" : "Michael Jordan", 

          "points" : 30, 

          "rebounds" : 3, 

          "assists" : 4, 

          "submit_date" : "1996-10-11" 

        } 

      } 

    ] 

  } 

} 

 

-d 이용하여 리퀘스트 바디도 가능함 

 

 


'오락기 > ELK' 카테고리의 다른 글

kibana managment  (0) 2018.05.11
elasticsearch aggregation  (0) 2018.05.11
elasticsearch mapping  (0) 2018.05.11
elasticsearch bulk  (0) 2018.05.11
elasticsearch update  (0) 2018.05.11

+ Recent posts