Deployment Best Practices

Best practices for production deployment

Configuration Best Practices for Production

  • Increase the number of TCP socket ports available. This is particularly important if the flow will be setting up and tearing down a large number of sockets in a short time.
sudo sysctl -w net.ipv4.ip_local_port_range="10000 65000"
  • Tell Linux you never want DC/OS NiFi to swap. Swapping is fantastic for some applications. It is not good for something like DC/OS NiFi that always wants to be running.

To set swapping off you can edit /etc/sysctl.conf to add the following line

vm.swappiness = 0

For the partitions handling the various DC/OS NiFi repos turn off things like atime. Doing so can cause a surprising bump in throughput. Edit the /etc/fstab file and for the partition(s) of interest add the noatime option.

Hardware Sizing Recommendations

The following image shows the recommended hardware for a DC/OS NiFi production installation:

Hardware Recommendation

Figure 1. Hardware sizing recommendations for DC/OS NiFi

Disk Recommendations

DC/OS NiFi performs best when using disks with fast read and write patterns.

We recommend the following:

  • Always prefer locally-attached storage. Remote storage adds points of failure, add latency/overhead to block requests and are more complicated to troubleshoot.
  • For better performance, use solid-state disks vs. spinning disks, or allocate more memory to cache more data, reducing the use of disks.