Apache Nifi Setup

Here we will give details about what is Docker file, it’s content along the deployment steps to deploy the application into AWS.

1. Login in you AWS Account.

2. At first you need to create an instance in AWS or GCP. I have some articles on how you can create instance on AWS here.

3. Next, we need to connect to that instance. There are many ways to connect to an instance like Putty / Terminal or Console form AWS or GCP. If you want to connect from Putty then follow my other article here.

4. Docker file: Pull the latest(1.15.3) nifi docker image from https://hub.docker.com/r/apache/nifi/ in your local system.

   docker pull apache/nifi:1.15.3 

5. Run the pulled docker image

   1. Secure Mode(with https)

      docker run -p 8443:8443 -e SINGLE_USER_CREDENTIALS_USERNAME=admin -e SINGLE_USER_CREDENTIALS_PASSWORD='admin@123' -e NIFI_WEB_HTTPS_PORT='8443' apache/nifi:1.15.3

      1. When we will use secure mode then few useful environment variables are

         1. custom login username and password set SINGLE_USER_CREDENTIALS_USERNAME and SINGLE_USER_CREDENTIALS_PASSWORD.

         2. If username/password not set executed in command then It will create some auto-generated credentials and we'll see them in the nifi-app.log.

            Generated Username [USERNAME]
            Generated Password [PASSWORD] 

             

         3. NIFI_WEB_HTTPS_PORT secure port no, NIFI_WEB_HTTPS_HOST secure host ip or domain name.

         4. open nifi url in browser https://instance-host:8443/nifi
            

   2. Unsecure Mode(with http)

      docker run -p 8080:8080  -e NIFI_WEB_HTTP_PORT='8080' apache/nifi:1.15.3

       

      1. NIFI_WEB_HTTP_PORT unsecure port no, NIFI_WEB_HTTP_HOST secure host ip or domain name.

      2. open nifi url in browser http://instance-host:8080/nifi

Upload AtaGlancee template in NIFI

1. Open the Nifi Url http://instance-host:8080/nifi and Login(If required)

2. Create 2 Global Variable

   1. ADDON_KEY = datapluginweb

   2. VMH_MONGO_URI = MongoDB server url

      1. Like : mongodb://admin:admin12345@instance-host:27017/master?authSource=admin

3. Next, import the all ataglance template here.

4. Now drag all the templates on nifi dashboard.

5. And select all the template and starrt.

Optional Setting

• Set Log Size in nifi through SSH Login

1. Login the aws server using ssh(open cmd terminal)

   ssh -i nifibetaintancepair.pem  ec2-user@18.222.63.48

2. find the running docker image container docker ps

3. copy the container id and execute the docker bash docker exec --user="root" -it <container_name> bash

4. Check logs file size /nifi-current/logs$ ls -lh

5. Remove Logs /nifi-current/logs$ rm nifi-app_*.log

6. Set maximum log size go to on /nifi-current$ cd conf

   1. find the logback.xml file /nifi-current/conf$ ls

   2. edit the file /nifi-current/conf$ vi logback.xml if vi command not exist then install first /nifi-current/conf$ apt-get install vim -y

   3. press +i for edit file. add totalSizeCap value on logs section

       <totalSizeCap>10GB</totalSizeCap>

   4. 

      after edit save the change. press ESC then type :wq

   5. check your changes using by /nifi-current/conf$ cat logback.xml