Opentext™ Documentum™ Server Certificate Based SSL Configuration and Troubleshooting

Product: OpenText™ Documentum™ Server
Version: 7.2, 7.3, 16.4, 16.7, 16.7.1, 20.2, 20.3, 20.4

Task/Topic: Deployment
Audience: Administrator
Platform: Windows, Linux
Document ID: 100028
Updated: October 14, 2020
Application Note
OpenText™ Documentum™ Server
Certificate Based SSL Configuration and
Troubleshooting
Documentum™ Server SSL Configuration and Troubleshooting
Contents
Executive summary .................................................................................................... 3
Audience ................................................................................................................. 3
Configuration............................................................................................................... 4
Docbroker ............................................................................................................... 4
Create Docbroker Keystore ............................................................................. 4
Create Docbroker Keystore's password file ..................................................... 5
Docbroker configuration ................................................................................... 6
Start Docbroker ................................................................................................ 6
Server ..................................................................................................................... 6
Create Server Keystore ................................................................................... 6
Create Server Keystore's password file ........................................................... 7
Create Server Trust-Store ................................................................................ 7
Server configuration ......................................................................................... 8
Start Server ...................................................................................................... 8
DFC ........................................................................................................................ 8
Create DFC Trust-Store ................................................................................... 8
DFC configuration ............................................................................................ 9
Troubleshooting ........................................................................................................ 10
Docbroker startup fails.......................................................................................... 10
Using OpenSSL: ...................................................................................... 10
Using Keytool: ......................................................................................... 10
Server startup fails................................................................................................ 10
Server not able to connect to Docbroker .............................................................. 11
Using OpenSSL: ...................................................................................... 11
Using Keytool: ......................................................................................... 11
Clients not able to connect to Docbroker ............................................................. 11
Clients not able to connect to Server ................................................................... 12
Multiple Docbase .................................................................................................. 12
Miscellaneous ....................................................................................................... 12
Conclusion................................................................................................................. 13
References ................................................................................................................. 14
Appendix .................................................................................................................... 15
Documentum Server Error Messages Quick Reference ............................... 15
The Information Company™ 2

Executive summary
OpenText™ Documentum™ Server and Connection Broker support connections in
both native and secure modes. For secure connections, Anonymous SSL is used by
default. Support for Non Anonymous or Certificate based SSL for communication is
new feature introduced in Documentum Server to further enhance communication
security. For using Certificate based SSL, Documentum Server, Docbroker and
Clients needs to be configured after installation. This paper explains in detail
configuration and troubleshooting steps.
All the components (Server, Docbroker and Clients) should use same mode for
secure communication. Mixed environments are not supported, i.e. either all the
components have to use Anonymous SSL or Non Anonymous SSL for
communication. For Certificate based SSL, only supported cipher is AES128-SHA for
maintaining backward compatibility with RSA libraries.
Audience
The audience for this white paper comprises personnel responsible for the
Installation, configuration, and deployment of Documentum Server in production
environments. This document is intended for internal Opentext teams, partners, and
customers.

Configuration
This section describes the steps required to configure different components of the
system (Docbroker, Documentum Server, Clients) to use certificate based SSL for
communication. Following needs to be configured:
• Docbroker as SSL server (Documentum Server & DFC as SSL clients)
• Documentum Server as SSL server (DFC as SSL client) and SSL Client
(Docbroker as SSL Server)
• DFC as SSL client
To enable clients to use Certificate based SSL communication for secure connections
to Docbroker and Documentum Server, we need to install Docbroker and
Documentum Server in secure or 'native & secure' mode. After installation is
complete, all the services needs to be stopped (Docbroker, Documentum Server &
Method Server) and then system needs to be configured for Certificate based SSL
manually. Currently Documentum Server installer do not support automatic
configuration of Certificate based SSL, so this configuration needs to be done
manually after installation.
Docbroker
Create Docbroker Keystore

For configuring Docbroker as SSL server, a private key needs to be created for
Docbroker and Certificate Signing Request (CSR) needs to be generated using
private key. This CSR then needs to be presented to and signed by Certificate
Authority (CA) for generating Docbroker's public certificate. Finally, we need to create
a Keystore for Docbroker to store its private key and public certificate.
All the above steps are described here in detail. In this paper OpenSSL tool is used
for managing Public Key Infrastructure (PKI). Below command can be used to
generate Docbroker's private key and CSR. It will generate private key
(brokerkey.pem) and CSR (brokercsr.pem) in PEM (text) format. CSR
(brokercsr.pem) can then be presented to some CA for generating Docbroker's
public certificate.
openssl req -newkey rsa:2048 -days 365 -subj

"/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=broker.dctm.emc.com" -
keyout
brokerkey.pem -out brokercsr.pem
Loading 'screen' into random state - done
Generating a 2048 bit RSA private key
....................................................+++
............+++
writing new private key to 'brokerkey.pem'

Enter PEM pass phrase:

Verifying - Enter PEM pass phrase:
-----
For the purpose of this paper, we will show the configuration using self-signed
certificates instead of CA signed certificates. Below command will generate
Docbroker's private key (brokerkey.pem) and self-signed public certificate
(brokercrt.pem), both in PEM format. Option -x509 is used to self-sign the public
certificate.
openssl req -newkey rsa:2048 -x509 -days 365 -subj

"/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=broker.dctm.emc.com" -
keyout brokerkey.pem -out brokercrt.pem
Docbroker will look for private key and public certificate in Keystore which should be
in PKCS #12 format. There is no restriction on Keystore's filename or extension. It
should be placed in $DOCUMENTUM/dba/secure as Docbroker will look in this
directory for locating its keystore. To generate Docbroker's keystore (broker.p12)
by storing key and self-signed public certificate, use below command:
openssl pkcs12 -export -out broker.p12 -inkey brokerkey.pem -

in brokercrt.pem -name broker
Enter pass phrase for brokerkey.pem:
Enter Export Password:
Verifying - Enter Export Password:
When executed, it will ask for key’s and Keystore's password. Option -name is
specified to provide an alias for keys in keystore.
Create Docbroker Keystore's password file

Keystores are secured by making them password protected. Docbroker Keystore's
password is encrypted and stored in a file which will be used by Docbroker for
accessing Keystore. Docbroker will look in directory $DOCUMENTUM/dba/secure for
locating password file. There is no restriction on password file's name or extension.
Below command generates password file with keystore's password:
dm_encrypt_password -encrypt <password> -file <file>

e.g dm_encrypt_password -encrypt broker -file broker.pwd

dm_encrypt_password utility is installed with Documentum Server and will work on

machines on which Documentum Server is installed.
Docbroker configuration
Docbroker will use properties in Docbroker’s configuration file (<Docbroker>.ini)
for resolving Keystore and password file names. Property keystore_file is used to
specify Keystore file name and keystore_pwd_file is used to specify file name in
which Keystore's password is stored.
Put below mandatory properties in Docbroker’s configuration file
(<Docbroker>.ini) in $DOCUMENTUM/dba directory, for the files that we generated
in previous steps.
keystore_file=broker.p12 keystore_pwd_file=broker.pwd
Start Docbroker
Docbroker should start successfully on secure port (1490).
Server
Create Server Keystore

Similar to Docbroker, private key and public certificates needs to be generated for
Server and stored in Server's Keystore from where Server will access them. Below
command can be used to generate Server's private key and CSR.
openssl req -newkey rsa:2048 -days 365 -subj

"/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=server.dctm.emc.com" -
keyout serverkey.pem -out servercsr.pem
For generating private key (serverkey.pem) and self-signed public certificate

(servercrt.pem), use above command with –x509 option as below:
openssl req -newkey rsa:2048 -x509 -days 365 -subj

"/C=IN/ST=Kar/L=BLR/O=emc/OU=iig/CN=server.dctm.emc.com" -
keyout
serverkey.pem -out servercrt.pem
Generating a 2048 bit RSA private key
..........+++ .+++
writing new private key to 'serverkey.pem'
Enter PEM pass phrase:

Verifying - Enter PEM pass phrase:

-----
These keys generated should now be stored in Keystore. Server Keystore should be
in PKCS #12 format. There is no restriction on Keystore's filename or extension. It
should be placed in $DOCUMENTUM/dba/secure where server will look to locate its
Keystore. Generate Server's Keystore (server.p12) containing private key and
selfsigned public certificate using below command:
openssl pkcs12 -export -out server.p12 -inkey serverkey.pem -

in servercrt.pem -name server
Enter pass phrase for serverkey.pem:
Enter Export Password:
Verifying - Enter Export Password:
Create Server Keystore's password file

Server Keystore's password is encrypted and stored in a file in directory
$DOCUMENTUM/dba/secure. There is no restriction on password file's name or
extension. Below command generates password file with Keystore's password:
dm_encrypt_password -encrypt server -file server.pwd
Create Server Trust-Store

When acting as SSL client to Docbroker, Server needs to verify public certificate sent
by Docbroker. For verifying Docbroker's public certificate, Docbroker’s public
certificate or CA certificate which was used to sign Docbroker's public certificate
needs to be stored in a Keystore. This Keystore will act as Server's trust-store and
server will verify Certificates sent by Docbroker against certificates in this keystore.
Server expects trust-store to be in PKCS #7 binary format. There is no restriction on
trust-store's name or extension. It should be placed in location
$DOCUMENTUM/dba/secure. Below command generates trust-store by importing
Docbroker's certificate.
openssl crl2pkcs7 -nocrl -certfile brokercrt.pem -outform der

-out server-trust.p7b

Server configuration
Server resolves file names of Server Keystore, trust-store and Keystore’s password
files by reading properties from Server configuration file (server.ini). Property
keystore_file is used to specify the name of Server's keystore,
keystore_pwd_file to specify password file name and truststore_file to
specify trust-store name. All the below properties are mandatory.
Put below properties in Server Configuration file (server.ini):
keystore_file=server.p12 keystore_pwd_file=server.pwd
truststore_file=server-trust.p7b
Start Server
Server should start successfully, and errors related to connection to Docbroker
should not be displayed.
DFC
Create DFC Trust-Store

When DFC will be acting SSL client to Docbroker or Server, it will use a trust-store to
store trusted certificates. Certificates sent by Docbroker or Server will be validated
against the certificates in DFC trust-store. DFC trust store should be in JKS (Java
Keystore) format. There is no restriction on store's name or extension as well as
location.
We will convert the Docbroker & Server public certificates that were generated in text
format to binary format using OpenSSL. Then public certificates of Docbroker and
Server in binary format will be imported to DFC trust-store.
Below commands converts certificates from PEM to DER format:
openssl x509 -outform der -in brokercrt.pem -out brokercrt.der

openssl x509 -outform der -in servercrt.pem -out servercrt.der
Following commands create DFC trust-store (dfc.keystore) by importing

Docbroker's & Server's public certificates:
keytool -importcert -keystore dfc.keystore -file brokercrt.der

-alias broker
keytool -importcert -keystore dfc.keystore -file servercrt.der
-alias server

DFC configuration
DFC will look in dfc.properties file to resolve trust-store location, name and
password.
Property dfc.security.ssl.truststore is used to specify trust-store's path
and dfc.security.ssl.truststore_password to specify trust-store's password. Trust-
store's password can be given in plain text or encrypted format. In dfc.properties, put
entries for DFC trust-store and trust-store's password:
dfc.security.ssl.truststore = c\:/secure/dfc.keystore
dfc.security.ssl.truststore_password = password
DFC trust-store password can be encrypted using DFC utility as below:

java com.documentum.fc.tools.RegistryPasswordUtils <password>
e.g.
java com.documentum.fc.tools.RegistryPasswordUtils password

MmKZP8KOlvrE6ndUolyl6Q==
The encrypted password generated by this command can be copied and pasted to
properties file. For this command to execute successfully, dfc.jar should be in java
classpath.
There is an additional property dfc.security.ssl.use_existing_truststore
that is mutually exclusive to above two properties. When this property is specified,
there is no need to put above two properties. In this case, Java Keystore will act as
DFC truststore or trust-store can be specified using JVM parameter
javax.net.ssl.trustStore.
For secure connections to Server and Docbroker, put the value of property
dfc.session.secure_connect_default as secure. Same property controls
behavior of connections to both Docbroker and Server.
Put below property in dfc.properties file:
secure.dfc.session.secure_connect_default = secure

Troubleshooting
This section describes some of the common issues faced during configuration and
steps to troubleshoot.
Docbroker startup fails

1. Check if Docbroker Keystore and Keystore's password files are present in
$DOCUMENTUM/dba/secure
2. Check if entry for above two files is there in Docbroker configuration file
(<Docbroker>.ini)
3. Check if the format of Docbroker Keystore is PKCS #12.

Below commands should list the keys in Keystore if it’s in PKCS #12 format, else
error will be displayed.
Using OpenSSL:
openssl pkcs12 -info -in <keystore>
Using Keytool:
keytool -list –v -storetype pkcs12 -keystore <keystore>
4. Check if password in Keystore password file is correct. For testing, password can
be put in plain-text (without encryption).
Server startup fails

1. Check if Server Keystore, Server Keystore password and Server trust-store files
are present in $DOCUMENTUM/dba/secure
2. Check if entry for above three files is there in Server configuration file
(server.ini)
3. Check if the format of Server Keystore is PKCS #12. See commands in
Docbroker startup fails section to verify.
4. Check if server trust-store is in PKCS #7 binary (der) format.
For verifying, check if below command dumps it successfully:
openssl pkcs7 -in <Keystore> -inform der
e.g.
openssl pkcs7 -in server-trust.p7b -inform der

Server not able to connect to Docbroker

1. Check whether Docbroker Keystore has proper key and public certificate. Use
below command to print the keys in keystore:
Using OpenSSL:
openssl pkcs12 -info -in <keystore>
Using Keytool:
keytool -list -storetype pkcs12 -keystore <keystore>
2. Check whether Docbroker is sending proper certificates.

Below command will start a simple client which tries to connect to SSL Server
and displays certificate chain sent by Server:
openssl s_client -showcerts -debug -connect
<SSL_Server_IP>:<SSL_Server_Port>
e.g.
openssl s_client -showcerts -debug -connect 10.8.53.24:1490
3. Check if Server's trust-store contains Docbroker's public certificate or CA

certificate chain used to sign Docbroker's public certificate Use below command
to display all certificates in trust store:
openssl pkcs7 -in <trust-store> -inform der -print_certs -
text
Clients not able to connect to Docbroker

1. Check if proper entries are present in dfc.properties and trust-store file exists.
2. Check if DFC trust-store contains Docbroker's public certificate or CA certificate
used to sign Docbroker’s public certificate.
3. To dump Trust-store's contents (no need to specify storetype as default type is
JKS):
keytool -list -keystore <keystore> -storepass <storepass>

Clients not able to connect to Server
1. Check if proper entries are made in dfc.properties and trust-store file exists.
2. Check if DFC trust-store contains Server's public certificate or CA certificate used
to sign Server’s public certificate.
3. To dump Trust-store's contents (no need to specify storetype as default type is
JKS):
keytool -list -keystore <keystore> -storepass <storepass>
4. Check if dfc.properties has right value for trust-store password. For
verification, password can be given in plain-text.
Multiple Docbase
If Sever and Docbroker are configured in secure mode with Certificate based SSL
and second Docbase is created, installer will hang for some time in middle during
step where docbase tries to connect to existing Docbroker. This issue occurs as
Docbase will try to connect using Anonymous SSL by default and since mixed mode
is not supported, it won’t be able to connect. Follow below steps to resolve this issue:
1. While the installer is hanging, open server.ini of the new Server
2. Add the proper options for Certificate based SSL
3. Start/restart the new Server
Miscellaneous
1. To verify that connection is secure and check encryption algorithm used, use jvm
parameter -Djavax.net.debug=all, when starting the client.

Conclusion
Documentum support for Certificate based SSL communication to Connection Broker
and Documentum Server improves security in Documentum products. This paper
describes how to configure deployment to use this feature. For further information on
installation, Documentum Server Enterprise Edition Installation Guide can be
referenced.

References
1. Documentum Server Enterprise Edition Installation Guide
2. OpenSSL Documentation: http://www.openssl.org/docs/apps/openssl.html
3. Keytool Documentation (Java SE8):

http://docs.oracle.com/javase/8/docs/technotes/tools/windows/keytool.html

Appendix
Documentum Server Error Messages Quick Reference
Table 1 Documentum Server Error Messages

Error Message Possible Cause
[DM_SERVER_E_START_NETWISE]error: "The server failed to start • Server Keystore missing
due to an error returned by the Netwise networking subsystem. • Server Keystore different
Failed API: nl_init(). Error (705) SSL error(10100) in call to format than PKCS #12
SSL_CTX_new(). Winsock error: 10100." • Server Keystore corrupt
• Server Keystore password
file contains wrong
[DM_SERVER_E_START_NETWISE]error: "The server failed to start password
due to an error returned by the Netwise networking subsystem.
Failed API: nl_open(). Error (501) Network Library not • Server Keystore password
initialized. file missing
Extended network error: 0."
[ERROR] [AGENTEXEC 3088] Detected during program initialization: • Server Keystore empty
Command Failed:
connect,<server_name.docbase_name>,<user>,'',,,try_native_first,
status: 0, with error message [DM_SESSION_E_RPC_ERROR]error:
"Server communication failure"
javax.net.ssl.SSLHandshakeException: Received fatal alert:
handshake_failure
[DM_SERVER_E_START_NETWISE]error: "The server failed to start • Server trust-store missing

due to an error returned by the Netwise networking subsystem. • Server trust-store corrupt
Failed API: nl_init(). Error Unknown error code 10008 (_nl_error_
=
0). Extended network error: 0."
[DM_SERVER_E_START_NETWISE]error: "The server failed to start
Failed API: nl_open(). Error (501) Network Library not
initialized.


[DM_SERVER_E_START_NETWISE]error: "The server failed to start • Server trust-store wrong
due to an error returned by the Netwise networking subsystem. format
Failed API: nl_init(). Error Unknown error code 1795 (_nl_error_
=
0). Extended network error: 0."
[DM_SERVER_E_START_NETWISE]error: "The server failed to start

Failed API: nl_open(). Error (501) Network Library not
initialized.
[DM_DOCBROKER_E_NETWORK_ERROR]error: "An error occured • Server trust-store empty

performing a network operation: (Unknown error code 112 • Docbroker's keystore empty
(_nl_error_ = 0)). Network specific error: (Extended network
error: 0)."
[DM_DOCBROKER_E_CONNECT_FAILED_EX]error: "Unable to connect to

DocBroker. Clients please check your dfc.properties file for a
correct host. Server please check your server.ini or target
attributes in server config. Network address: (INET_ADDR:
family: 2, port: <port>, host: <host_name> (<host_ip>,
1835080a))."
[ERROR] [AGENTEXEC 2692] Detected during program initialization: • DFC trust-store missing
Command Failed:
status: 0, with error message [DFC_DOCBROKER_REQUEST_FAILED]
Request to Docbroker “<docbroker_name>:<port>“ failed
[DM_SESSION_E_RPC_ERROR]error: "Server communication failure"
javax.net.ssl.SSLException: java.lang.RuntimeException:
Unexpected error:
java.security.InvalidAlgorithmParameterException: the
trustAnchors parameter must be non-empty
java.lang.RuntimeException: Unexpected error:



Command Failed:
Docbroker's Certificate

javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building
failed:
sun.security.provider.certpath.SunCertPathBuilderException:
unable to find valid certification path to requested target

failed:
Command Failed:
Server's Certificate
status: 0, with error message [DM_SESSION_E_RPC_ERROR]error:
"Server communication failure"
javax.net.ssl.SSLHandshakeException:
failed:

failed:


[ERROR] [AGENTEXEC 1672] Detected during program initialization: • DFC trust-store different
Command Failed:
format
java.net.SocketException: java.security.NoSuchAlgorithmException:
Error constructing implementation (algorithm: Default, provider:
SunJSSE, class:
com.sun.net.ssl.internal.ssl.DefaultSSLContextImpl)
java.security.NoSuchAlgorithmException: Error constructing

implementation (algorithm: Default, provider: SunJSSE, class:
java.io.IOException: Invalid keystore format
[ERROR] [AGENTEXEC 2608] Detected during program initialization: • DFC trust-store corrupt
Command Failed:
java.net.SocketException:

java.security.cert.CertificateException: Unable to initialize,

java.io.IOException: DerInputStream.getLength(): lengthTag=127,
too big.
java.io.IOException: DerInputStream.getLength(): lengthTag=127,

too big.


[ERROR] [AGENTEXEC 5720] Detected during program initialization: • DFC trust-store wrong
Command Failed:
password
java.net.SocketException:
java.io.IOException: Keystore was tampered with, or password was
incorrect
java.security.UnrecoverableKeyException: Password verification
failed

About OpenText
OpenText enables the digital world, creating a better way for organizations to work with information, on premises or in the
cloud. For more information about OpenText (NASDAQ: OTEX, TSX: OTC) visit opentext.com.
Connect with us:
OpenText CEO Mark Barrenechea’s blog

Twitter | LinkedIn
20
www.opentext.com/contact
Copyright © 2020 Open Text SA or Open Text ULC (in Canada).
All rights reserved. Trademarks owned by Open Text SA or Open Text ULC (in Canada).

Opentext™ Documentum™ Server Certificate Based SSL Configuration and Troubleshooting

Uploaded by

Copyright:

Available Formats

You might also like

Opentext™ Documentum™ Server Certificate Based SSL Configuration and Troubleshooting

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Opentext™ Documentum™ Server Certificate Based SSL Configuration and Troubleshooting

Uploaded by

Copyright:

Available Formats

Product: OpenText™ Documentum™ Server

Version: 7.2, 7.3, 16.4, 16.7, 16.7.1, 20.2, 20.3, 20.4

The Information Company™ 2

The Information Company™ 3

Create Docbroker Keystore

openssl req -newkey rsa:2048 -days 365 -subj

The Information Company™ 4

Enter PEM pass phrase:

openssl req -newkey rsa:2048 -x509 -days 365 -subj

openssl pkcs12 -export -out broker.p12 -inkey brokerkey.pem -

Create Docbroker Keystore's password file

dm_encrypt_password -encrypt <password> -file <file>

The Information Company™ 5

dm_encrypt_password utility is installed with Documentum Server and will work on

Create Server Keystore

openssl req -newkey rsa:2048 -days 365 -subj

For generating private key (serverkey.pem) and self-signed public certificate

openssl req -newkey rsa:2048 -x509 -days 365 -subj

The Information Company™ 6

Verifying - Enter PEM pass phrase:

openssl pkcs12 -export -out server.p12 -inkey serverkey.pem -

Create Server Keystore's password file

dm_encrypt_password -encrypt server -file server.pwd

Create Server Trust-Store

openssl crl2pkcs7 -nocrl -certfile brokercrt.pem -outform der

The Information Company™ 7

Create DFC Trust-Store

openssl x509 -outform der -in brokercrt.pem -out brokercrt.der

Following commands create DFC trust-store (dfc.keystore) by importing

keytool -importcert -keystore dfc.keystore -file brokercrt.der

The Information Company™ 8

DFC trust-store password can be encrypted using DFC utility as below:

java com.documentum.fc.tools.RegistryPasswordUtils password

The Information Company™ 9

Docbroker startup fails

3. Check if the format of Docbroker Keystore is PKCS #12.

openssl pkcs12 -info -in <keystore>

keytool -list –v -storetype pkcs12 -keystore <keystore>

Server startup fails

The Information Company™ 10

Server not able to connect to Docbroker

2. Check whether Docbroker is sending proper certificates.

3. Check if Server's trust-store contains Docbroker's public certificate or CA

Clients not able to connect to Docbroker

The Information Company™ 11

Clients not able to connect to Server

The Information Company™ 12

The Information Company™ 13

2. OpenSSL Documentation: http://www.openssl.org/docs/apps/openssl.html

3. Keytool Documentation (Java SE8):

The Information Company™ 14

Table 1 Documentum Server Error Messages

[DM_SERVER_E_START_NETWISE]error: "The server failed to start • Server trust-store missing

The Information Company™ 15

Error Message Possible Cause

[DM_SERVER_E_START_NETWISE]error: "The server failed to start

[DM_DOCBROKER_E_NETWORK_ERROR]error: "An error occured • Server trust-store empty

[DM_DOCBROKER_E_CONNECT_FAILED_EX]error: "Unable to connect to