Do It For You

https://streamsets.com/ 로그인

Deployments 메뉴를 이용하여 deployment task진행

: A deployment is a group of identical engine instances deployed within an environment.

intall script를 로컬환경에서 실행하면

streamSets collector를 다운로드 받음.

위 스크립트가 수행되면 아래 경로의 파일을 다운로드 함. (오류가 있으면 브라우저를 통해 다운로드 진행)

Starting download of 

https://archives.streamsets.com/datacollector/4.1.0/tarball/streamsets-datacollector-core-4.1.0.tgz

dcans@dcans:~/.streamsets/download/dc/collector$ sudo bin/streamsets dc

Java 1.8 detected; adding $SDC_JAVA8_OPTS of "-XX:+UseConcMarkSweepGC -XX:+UseParNewGC -Djdk.nio.maxCachedBufferSize=262144" to $SDC_JAVA_OPTS
INFO - Starting engine
Logging initialized @5466ms to org.eclipse.jetty.util.log.Slf4jLog
Running on URI : 'http://dcans:18630'

-------------------------------------------------------

StreamSets collector를 구동하기 위해 jdk 1.8이상 필요.

버전이 맞지 않아 기존 jdk 삭제후 재설치 함

jdk삭제하는 법

Remove OpenJDK,

sudo apt remove openjdk*

Remove OpenJDK along with dependencies,

sudo apt remove --auto-remove openjdk*

Remove OpenJDK and the configuration files

sudo apt purge openjdk*

kafka@dcans:~/logs$ sudo systemctl status kafka
● kafka.service
     Loaded: loaded (/etc/systemd/system/kafka.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-11-21 18:44:01 KST; 4min 14s ago
   Main PID: 7383 (sh)
      Tasks: 72 (limit: 8176)
     Memory: 331.1M
     CGroup: /system.slice/kafka.service
             ├─7383 /bin/sh -c /home/kafka/kafka/bin/kafka-server-start.sh /home/kafka/kafka/config/server.properties > /home/kafka/kafka/kafka.log 2>&1
             └─7384 java -Xmx1G -Xms1G -server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -XX:MaxInlineLevel=15 -Djava.awt.headless=true >

11월 21 18:44:01 dcans systemd[1]: Started kafka.service.
lines 1-11/11 (END)

kafka start 시 아래 에러 발생시 "meta.properties파일 삭제후  카프카 재시작한다".

kafka.common.InconsistentClusterIdException: The Cluster ID WX_SndaJRfmYqLQRDavlVg doesn't match stored clusterId Some(uJ8xz_r_SuKKvuQOCK0zgg) in meta.properties.

/home/kafka/kafka/config/server.properties

의 로그 폴더 확인

############################# Log Basics #############################
# A comma separated list of directories under which to store log files
log.dirs=/home/kafka/logs

meta.properties파일 삭제후

카프카 재실행

kafka 설치 및 테스트 참고url)

https://www.digitalocean.com/community/tutorials/how-to-install-apache-kafka-on-ubuntu-20-04

1. cpu 수만큼의 cpu 상세정보

dcans@dcans:~$ cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 22
model : 48
model name : AMD A8-6410 APU with AMD Radeon R5 Graphics
stepping : 1
microcode : 0x7030105
cpu MHz : 1271.893
cache size : 2048 KB
physical id : 0
siblings : 4
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13


2. lscpu 명령 


dcans@dcans:~$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   40 bits physical, 48 bits virtual
CPU(s):                          4
On-line CPU(s) list:             0-3
Thread(s) per core:              1
Core(s) per socket:              4
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       AuthenticAMD
CPU family:                      22
Model:                           48
Model name:                      AMD A8-6410 APU with AMD Radeon R5 Graphics
Stepping:                        1
Frequency boost:                 enabled
CPU MHz:                         1400.000
CPU max MHz:                     2000.0000
CPU min MHz:                     1000.0000
BogoMIPS:                        3992.33
Virtualization:                  AMD-V
L1d cache:                       128 KiB
L1i cache:                       128 KiB
L2 cache:                        2 MiB
NUMA node0 CPU(s):               0-3
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:        Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:        Mitigation; Full AMD retpoline, STIBP disabled, RSB filling
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected