DataIQ Installation and Implementation - Participant Guide

DATAIQ INSTALLATION
AND IMPLEMENTATION
PARTICIPANT GUIDE
PARTICIPANT GUIDE
DataIQ Installation and Implementation-SSP
© Copyright 2021 Dell Inc. Page i

Table of Contents
Requirements ............................................................................................................. 2
Installation Requirements for DataIQ v2.1.x ......................................................................... 3
Other Requirements............................................................................................................. 4
Target Volume ..................................................................................................................... 6
Sizing Guidelines ................................................................................................................. 7
DataIQ Installation...................................................................................................... 9
Installation Considerations ................................................................................................. 10
DataIQ Installer - Things to Know ...................................................................................... 11
Types of Installation - Virtual Host...................................................................................... 12
DataIQ Installation - Physical Host ..................................................................................... 15
Upgrade ............................................................................................................................. 16
DataIQ Plugins Installation ...................................................................................... 17

Plugin Installation ............................................................................................................... 18
Data Mover Plugin ............................................................................................................. 20
Data Mover Configuration .................................................................................................. 21
Data Mover Process Anatomy............................................................................................ 23
Data Mover Considerations ................................................................................................ 25
DataIQ Secure Remote Services ............................................................................. 26

Secure Remote Services Overview .................................................................................... 27
DataIQ SRS Integration ..................................................................................................... 28
Telemetry Data .................................................................................................................. 29
DataIQ SRS Integration Architecture.................................................................................. 30
DataIQ SRS Integration Installation ................................................................................... 31
Page ii © Copyright 2021 Dell Inc.

Requirements
© Copyright 2021 Dell Inc. Page 1

Requirements
Requirements
Page 2 © Copyright 2021 Dell Inc.

Requirements
Installation Requirements for DataIQ v2.1.x
Enterprises must meet software and security requirements in their environment,

which determines the platform on which DataIQ is installed. Install DataIQ either on
a standard system or a virtual appliance.
Component Requirement
DataIQ Host Operating Open Virtual Appliance (OVA): VMware ESXI

System versions 6.7.
Red Hat Enterprise Linux® or CentOS:
• 64-bit CentOS 7.6, 7.7, or 7.8
• 64-bit Red Hat Enterprise Linux (RHEL) Server,
v7.6, 7.7, or 7.8
DataIQ 2.1 lets users use Red Hat Package

Manager (RPM) to install DataIQ.
DataIQ Host CPU Performance (recommended): 12 CPUs

Minimum: 8 CPUs
DataIQ Host RAM 64 GB
DataIQ Host Capacity Minimum of 15% total drive free space + 40 GB free
space for installation
DataIQ SSD Minimum of 512 GB

At least 250 GB for every 1 billion files and folders to
scan
The table shows the installation requirements.

Requirements
Other Requirements
Drive
A combination of both solid-state drives (SSD) and traditional hard disk drives
(HDD) is recommended. Set up drives as logical drives1 to support future
expansion.
Directory Description
Operating system Hosted on a traditional HDD or an SSD.
/opt/dataiq The /opt/dataiq directory stores the DataIQ

application, the storage monitoring database, the
keycloak database, and the claritynow configuration
database.
/var/lib/docker The /var/lib/docker directory stores the

Docker application and DataIQ pods.
1Logical volumes can be expanded later onto new drives when more storage is
needed.

Requirements
/mnt/ssd In DataIQ v2.1, data management database no

longer requires a specific directory (/mnt/ssd).
This directory is still the default directory, but you
may now change its location during installation by
using the DataIQ --ssd_path <path> option. The
default backup path is
/mnt/ssd/data/claritynow/ssd/backup.
Unlike other directories, it is recommended to host
/mnt/ssd on an SSD.
Browser
You can use the following browsers with DataIQ:
• Google Chrome
• Microsoft Edge
• Safari
• Mozilla Firefox (standard edition)
• Internet Explorer 11+
Network
DataIQ does not require an Internet connection. However, some features such as
proactive support do require an Internet connection.
The DataIQ host and each worker node require a static IP address. All nodes can
access each other hence fast, and stable networking between nodes improves
performance and scalability.
Check the following before installing:

• Appropriate default gateway
• Appropriate DNS entries
• No IP address conflicts

Requirements
Target Volume
DataIQ can manage storage across multiple types of unstructured storage

products, and can monitor storage on Dell EMC PowerScale and Isilon systems.
Data Management
DataIQ manages, scans, indexes, and reports on network-attached file systems

including NFS and SMB. DataIQ requires root access to clusters that it is directly
managing.
Examples of volumes you can connect:

• Dell EMC PowerScale and Isilon systems
• Amazon Web Services (AWS) S3
• Google Cloud Platform (GCP)
• Dell EMC ECS
• Dell EMC Unity
Storage Monitoring
DataIQ can manage and monitor storage on Dell EMC PowerScale and Isilon
systems using OneFS 8.0.0.0 and later, including Gen 4, Gen 5, Gen 6, and Gen
6.5 product models.
DataIQ can monitor storage on up to 70 Dell EMC PowerScale and Isilon clusters
up to 2,000 nodes.
DataIQ requires access to a user account with read permission to the clusters that
it is monitoring.

Requirements
Sizing Guidelines
Sizing estimates depend on the complexity of the DataIQ solution and

requirements2. The general rule for sizing is to add additional disk resources to the
planned capacity.
Disk-space requirements depend on various factors that are used for data
management or storage monitoring.
Data management
• The data management portion of the software writes to an IndexDB which is

installed and hosted on a specific mount point3, such as /mnt/ssd.
• The underlying disk platform4 is recommended to be SSD.
• When sizing a DataIQ solution, consider the complexity of the customer storage
environment and the total number of files, folders, and objects per volume.
• Workloads can be distributed across the external data mover nodes in a scale-
out fashion.
2There is no one size that can fit all hardware resources that are planned for
DataIQ because every environment is different. The CPU, memory, and network
are shared resources for dataset management and storage monitoring.
3 This mount point (or subdirectory or folder) must exist before installation either in
the form of a simple folder or as a mounted partition.
4This can be in the form of a single large SSD VMDK for the entire operating
system and DataIQ application (such as the OVA by itself), or an SSD partition or
VMDK mounted to /mnt/ssd.

Requirements
Storage monitoring
The general considerations for storage monitoring disk space are as follows:
• Load rate on the cluster is beyond the scope of DataIQ.

• DataIQ uses different data collection strategies for different dashboards.
• DataIQ provides a data retention feature and can only delete older monitoring
data.
• DataIQ provides a data backup feature for storage monitoring.
• Always add additional disk resources to the planned capacity to allow for
monitoring-data growth.
• Never undersize the estimated size of the disk requirements for monitoring.
• The disk-space requirement depends on various factors5.
Resource: For more details about sizing guidelines, refer to DataIQ

Best Practices Guide
5Various factors, such as active OneFS features, load rate, data backup strategy,
and data-retention settings. DataIQ regularly backs up data automatically, but the
backup can be disabled or defined as needed. The data-retention policy is disabled
by default, but it can be enabled and configured.

DataIQ Installation
DataIQ Installation

DataIQ Installation
Installation Considerations
Listed are areas to consider when installing DataIQ:

• DataIQ supports HTML5 and does not require Java.
• The DataIQ host should not have Docker, or Kubernetes preinstalled as the
DataIQ install program installs these components.
• DataIQ may need additional storage for systems using plugins, or if the DataIQ
scan environment has many files and folders. As a rule of thumb, at least 250
MB should be allotted for scanning every million files and folders.
• The administrator is given a temporary password for the DataIQ UI when the
installation completes.
• DataIQ works well on systems that have up to 20 billion files and folders.
Environments with more than 2000 nodes or 70 clusters may need multiple
DataIQ instances.
• It is recommended to use a dedicated server host for each instance of

DataIQ (server or data mover host) to avoid potential network port or
resource conflicts. There is no way to combine data from multiple DataIQ
instances.

DataIQ Installation
DataIQ Installer - Things to Know
The DataIQ installer is the installation software bundle.
The details of the Installer are:

• Bundle size is about 4 GB.
• Bundle is self-extracting and executable.
• A single installation can take up to 10 minutes.
• Installer displays a successful message when install succeeds.
• Successful or failed installations are shown in log file.

DataIQ Installation
Types of Installation - Virtual Host
The installation process has been simplified in DataIQ v2.1.
The tabs show the two ways to install DataIQ.
Install DataIQ onto a new VM created from an OVA
To install DataIQ deploy a new VM using an OVA installer6, use a virtualization

platform such as VMware vSphere.
Click each installation step to learn more.

1) Download installation and upgrade packages7
2) Prepare a VM to host DataIQ8
3) Create and mount logical volumes9
6The OVA installer prepares a VM with CentOS 7.6, DataIQ, Docker, and
Kubernetes installed, with many settings preconfigured.
7Download the installer and optionally, a DataIQ license file. A DataIQ license can
be added either during installation or after installation through the UI.
8You can host DataIQ and its features entirely on an SSD, or you can split it across
drives and use a less expensive HDD.
9 Prepare logical volumes using logical volume manager (LVM) for your storage
locations before installing DataIQ.

DataIQ Installation
4) Set up networking10
5) Initialize DataIQ11
Install DataIQ on a server or existing VM
Install DataIQ onto a server or VM that already has CentOS or RHEL installed.
Click each installation step to learn more.

• Install and register the operating system12
• Download installation and upgrade packages
• Protect authenticity and integrity13
• Create and mount logical volumes
• Set up NetworkManager14
• Set up networking
10 Perform this step even if you are installing DataIQ on a host that is not connected
to the Internet.
11Run the DataIQ installer, receive initial passwords, and optionally set up Secure
Remote Services.
12You can install DataIQ on supported Red Hat Enterprise Linux or CentOS
systems. If you use a Red Hat Enterprise Linux, it must be registered.
13Files that have a .sign file extension can be verified using standard OpenSSL
tools.
14 Enable and start Linux NetworkManager tools.

DataIQ Installation
• Extract files from the installer packages15

• Initialize DataIQ
Note: Use the uninstall.sh script to uninstall DataIQ from a host.

The uninstall script completely removes all the Docker, Kubernetes,
and nfs-utils packages. An uninstall can take up to 5 minutes to
complete.
15 Extract the DataIQ files from the installer and dependency packages.

DataIQ Installation
DataIQ Installation - Physical Host
1:
Download the DataIQ installer package from Dell Support. If the DataIQ host has
Internet access, download the online package. If the DataIQ host does not have
Internet access, download the offline package to a client and then transfer it to the
DataIQ host.
2: Change the hostname from the default localhost to a unique name.
Example: hostnamect1 set-hostname dataiq-server where dataiq-server

is a unique name.
3: Run the installer ./dataiq_installer_<version>.sh
The installer prompts the user for entries such as the IP address and the IP
addresses of worker nodes. Upon completion, the user will have:
• The DataIQ admin URL or IP address

• A temporary password for the DataIQ UI administrator user

DataIQ Installation
Upgrade
DataIQ supports upgrade to a higher version and getting new features and fixes.
But you should have a supported version of DataIQ installed to upgrade it to 2.x.
The host must meet all the requirements that are mentioned in the requirement
section of the course.
DataIQ version 2.x requires more CPUs, drive space, and memory (RAM) than
version 1.x. Hence ensure that sufficient resources are configured on the DataIQ
host.
The upgrade process completes within 20 minutes. During this time, users are
logged out and storage is not monitored.
After the upgrade completes, Data Management searches will not show any results
until after the first full volume scan is complete.
From/To DataIQ 1.0 DataIQ 2.0 DataIQ 2.1
DataIQ 1.0 - Supported Supported
DataIQ 2.0 - - Supported
DataIQ 2.1 - - -
Uninstall DataIQ and reinstall the original version to roll back to the original version
of DataIQ.

DataIQ Plugins Installation

Plugin Installation
The steps for installing a plug-in can vary slightly. Refer the DataIQ Admin Guide
for comprehensive details.
The high-level steps are:

1. Download the plug-in package from Dell Support.
• If available, get a copy of its proof of authenticity16.
2. Transfer the plugin tar.gz file to a folder on the DataIQ Linux host system.
You can use SCP or a shared mount point to transfer.
3. From the DataIQ host on the /root directory, run:
# cp /<package location>/<plugin package>-
installer.tar.gz /root
# cd /root
# tar -zxvf <plug-in package>-installer.tar.gz
# cd /root/<plug-in package>/
# python <plug-in package>.py install
4. Optional: Check the status by running kubectl get pods -ndataiq
5. Enable the plugin on the DataIQ WebUI > Settings > Data management
configuration page:
16Files that have a .sign file extension can be verified using standard OpenSSL
tools. The RPM uses a GPG signature for signing. The OVA can be verified by
using the Open Virtualization Format Tool (ovftool) utility that VMware provides.

Enable, disable, and manage plugin
Note: If you have previously installed a plugin, check to see if it still

exists on the system. If the plugin does exist, either remove or
upgrade17 the plugin.
17 Use the plugin installation package to update your plugins.

Data Mover Plugin
DataIQ Data Mover is a paid plugin that enables the movement of files and folders
more efficiently between file systems. Use Data Mover to transfer files and folders
between OneFS systems and external object storage such as Amazon Simple
Storage Service (S3), Google Cloud Platform (GCP), and ECS.
DataIQ logs file transfers to assist with audits and supports transfer policies to
allow or restrict the use of Data Mover based on user group and by source and
destination visibility. DataIQ is kept updated by reindexing files after transfers.
Data Mover enables you to choose a single file, or an entire folder, for a Copy-
Move operation from the folder tree UI.
The Data Mover has three components. Click each component for an overview:
• UI plugin18
• Service plugin19
• Workers plugin20
18 The user interface that is installed on the DataIQ host.
19The service plug-in is the job manager that is installed on the DataIQ host. This
service accepts requests from the Data Mover UI Plugin and services the work
assignment requests to/from Data Mover Workers.
20 The workers plug-in is the transfer agent service, responsible for transferring
files. Install the Data Mover Worker plug-in on dedicated transfer hosts or the
DataIQ host. Administrators can deploy multiple Data Mover Worker hosts. If on a
dedicated host, it requires RHEL/CentOS 7.6 or 7.7.

Data Mover Configuration
Configure Data Mover, including adding required volume information. If you use
Data Mover External Workers, modify the configuration on each External Worker to
match.
You can edit the Data Mover configuration file from the DataIQ UI. If there are no
active and paused transfer jobs, DataIQ applies changes and resets the Data
Mover service. The reset takes about one minute to complete.
By default, the Data Mover will refuse to transfer certain file paths and file types.
Data Movers for 2.1 have added support for these files21 in the plugin configuration.
21 1) support_extended_characters ––#1 & #2
2) convert_system_files_to_stubs ––#3

The graphic shows opening the Data Mover configuration file from the WebUI > Settings > Data
management configuration page.

Data Mover Process Anatomy
The graphic shows an overview of the end-to-end process for moving data from a
PowerScale cluster to a target. The target can be an S3 object target, ECS, Isilon,
or a supported third-party platform.
The image shows a Data Mover Worker on a separate node. Worker nodes do not
require a database disk or an SSD.
1. The DataIQ server scans and indexes a mounted volume. The Data Mover UI
plugin and the Data Mover service plugin is installed on the DataIQ host.
2. The network share is also mounted to the DataIQ Data Mover worker.
3. A file or folder is manually selected using the DataIQ WebUI.
4. The Data Mover service receives requests from the Data Mover UI and sends
instructions to the Data Mover worker.
5. After validating the file or folder path, the Data Mover heavy-worker begins the
copy job to the target. Only the data, not ACL and DACL information is sent. No file
stubs are used. The full file is either copied to target or copied with source deleted.
Light workers perform pre-transfer job validation and path preparation. Heavy
workers perform pre-allocations and data transfer. Administrators can configure the
number of workers in the
/usr/local/data_mover_workers/etc/workers.cfg file.
Allocating too many heavy workers may negatively impact the stability of the host.

6. If the target is S3, the Data Mover uses embedded credentials and the Data
Mover worker logs in to the cloud subscription service. For ECS, the service opens
an SSL tunnel to the recommended load balancer. Data copy-move operation is
complete.

Data Mover Considerations
Consider the following when planning to move data from a source to a target
volume.
• File to file data movement with a scale-out worker node architecture.

• Moving data from file to S3 with a scale-out worker node architecture preserves
data independence by leaving a standard S3 object in the destination.
Administrators can retrieve and consume objects independent of any DataIQ
component.
• Data Mover does not support moving data directly between two external object
storage (S3) buckets.
• Cannot move files based on policies. For example, data not accessed in the last
30 days.
• Does not copy Windows file permissions. Data Mover uses NFS protocol when
moving data.
• It is not tested with petabyte-scale data movement.
• Does not support automatic archiving, and does not have incremental archiving
capability.

DataIQ Secure Remote Services

Secure Remote Services Overview
The Secure Remote Services (SRS) application architecture is a secure,

asynchronous messaging system that is designed to support the functions of
secure encrypted file transfer, monitoring of device status, and remote execution of
diagnostic activities.
SRS delivers a secure, IP-based, distributed remote service support solution that
provides command, control, and visibility of remote services access.
Secure Remote Services Architecture.

DataIQ SRS Integration
Dell EMC software products must be able to debug and resolve any issues with
minimal input from customers. DataIQ can communicate with Dell Support using
SRS. DataIQ can send logs, configurations, and managed device status to Dell
Support, either periodically for proactive support, or on-demand as needed.
SRS integration features are:

• SRS proactively and periodically sends telemetry data to Dell EMC after getting
confirmation from administrators.
• DataIQ launches a process that hides all sensitive information before sending
the telemetry data.
• Users can disable SRS at any time.
• SRS does not upload anything on disconnected sites (dark or on-premise sites)
although telemetry data collection continues.
With DataIQ v2, you can create an account for Dell Support to use a secure tunnel
into your system to investigate issues and support requests.
For more information about SRS, refer to your Secure Serviceability

Customer Letter.

Telemetry Data
The table shows the DataIQ handling of telemetry data:
Proactively push debug information to engineering and support when necessary to achieve an adequate level of
Upload Telemetry application quality monitoring and instrumentation.
Data
Customer confirmation required (Yes/No, Default: No).
Gather Telemetry Telemetry data is always gathered without regard to the internet connection status.
Data Customer confirmation required (Yes/No, Default: No).
Hide Sensitive
Automatically hides sensitive information such as username, password, directory names, and so on.
Information

DataIQ SRS Integration Architecture
DataIQ services communicate with SRS in customer network. SRS communicates

with Dell EMC backend.
Web Services
Access Servers
Certification Authority
Dell EMC Support Analyst
Resource: For assistance with setup and maintenance with SRS,

contact your Dell EMC Global Services representatives.

DataIQ SRS Integration Installation
Set up SRS, and then configure it to work with DataIQ. SRS and DataIQ must be
installed on separate hosts or VMs. DataIQ can be attached to the external SRS
gateways based on Docker Edition (DE) or Virtual Edition (VE). SRS in DE can be
installed on DataIQ as embedded.
Prerequisite
Listed are the prerequisites for SRS installation:

• Specific assigned ports must be available.
• Docker supported Linux distribution (x64 bit).
• Docker Engine (Docker runtime).
• Disk space availability must be at least 64 GB.
• Valid IPv4 address.
Procedure
Listed is the procedure to perform SRS installation in DataIQ:

1. During the DataIQ installation, SRS installation is prompted with two options,
either to install SRS or not (default: No).
2. If user chooses to install SRS, then the installer requests password to be set for
the root account of SRS.
3. At the end of a successful SRS installation, the installer displays a message
stating that the SRS services are up and running and provides the URL to
continue to provision SRS in a web browser.
Configuration
The administrator must configure external network and/or firewalls to allow traffic
over specific ports and ensure that the default gateway is properly configured.
TCP Port or Protocol Description
Passive FTP 21 Sends Connect home messages

SMTP 25 Processes Connect home files, sends email

notifications
HTTPS 443 Service notification, setup, and so on
HTTPS 443 AND 8443 Remote support22
HTTP 9443 Manage File Transfer (MFT)23
Note: SRS upgrade and uninstall are subprocesses of DataIQ

upgrade and uninstall processes. It is NOT recommended to upgrade
or uninstall SRS separately.
22The Dell EMC remote services strategy delivers immediate, secure response to
product event reports such as error alerts, which can greatly increase the
availability of your information infrastructure.
23 MFT is a bidirectional file transfer mechanism that is provided as part of SRSv3.

You can use MFT to send or receive large files, such as log files, microcode,
firmware, scripts, or large installation files between the managed device and Dell
EMC. A distribution "locker" is used for bidirectional file staging.


DataIQ Installation and Implementation - Participant Guide

Uploaded by

Copyright:

Available Formats

You might also like

DataIQ Installation and Implementation - Participant Guide

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

DataIQ Installation and Implementation - Participant Guide

Uploaded by

Copyright:

Available Formats

DATAIQ INSTALLATION

© Copyright 2021 Dell Inc. Page i

DataIQ Plugins Installation ...................................................................................... 17

DataIQ Secure Remote Services ............................................................................. 26

DataIQ Installation and Implementation-SSP

Page ii © Copyright 2021 Dell Inc.

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 1

DataIQ Installation and Implementation-SSP

Page 2 © Copyright 2021 Dell Inc.

Installation Requirements for DataIQ v2.1.x

Enterprises must meet software and security requirements in their environment,

DataIQ Host Operating Open Virtual Appliance (OVA): VMware ESXI

DataIQ 2.1 lets users use Red Hat Package

DataIQ Host CPU Performance (recommended): 12 CPUs

DataIQ Host RAM 64 GB

DataIQ SSD Minimum of 512 GB

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 3

Operating system Hosted on a traditional HDD or an SSD.

/opt/dataiq The /opt/dataiq directory stores the DataIQ

/var/lib/docker The /var/lib/docker directory stores the

DataIQ Installation and Implementation-SSP

Page 4 © Copyright 2021 Dell Inc.

/mnt/ssd In DataIQ v2.1, data management database no

You can use the following browsers with DataIQ:

Check the following before installing:

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 5

DataIQ can manage storage across multiple types of unstructured storage

DataIQ manages, scans, indexes, and reports on network-attached file systems

Examples of volumes you can connect:

DataIQ Installation and Implementation-SSP

Page 6 © Copyright 2021 Dell Inc.

Sizing estimates depend on the complexity of the DataIQ solution and

• The data management portion of the software writes to an IndexDB which is

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 7

• Load rate on the cluster is beyond the scope of DataIQ.

Resource: For more details about sizing guidelines, refer to DataIQ

DataIQ Installation and Implementation-SSP

Page 8 © Copyright 2021 Dell Inc.

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 9

Listed are areas to consider when installing DataIQ:

• It is recommended to use a dedicated server host for each instance of

DataIQ Installation and Implementation-SSP

Page 10 © Copyright 2021 Dell Inc.

DataIQ Installer - Things to Know

The DataIQ installer is the installation software bundle.

The details of the Installer are:

DataIQ Installation and Implementation-SSP

© Copyright 2021 Dell Inc. Page 11

Types of Installation - Virtual Host

The installation process has been simplified in DataIQ v2.1.

The tabs show the two ways to install DataIQ.

Install DataIQ onto a new VM created from an OVA

To install DataIQ deploy a new VM using an OVA installer6, use a virtualization