IBM TechXchange

watsonx Community Webinar

Scaling generative AI with watsonx- The
power of choice and flexibility

May 2, 2024 (11AM EST)

Steven Sawyer Luv Aggarwal Angela Jamerson

Senior Product Manager Global Sales Leader Program Director, Product Mgmt
watsonx.ai watsonx.ai watsonx.ai

Join the watsonx community

2
Agenda
watsonx.ai Model POV

• Enterprise client needs

• IBM Generative AI Technology stack incl. watsonx

• IBM's differentiated approach to delivering enterprise-grade foundation models

• watsonx model library : What's new?

• Granite models

watsonx.ai Client Stories

• Use Cases

• Win Stories

watsonx.ai Platform Updates

• New tooling releases

• A look head

3
watsonx.ai Model PoV

4
 Surpassed Driver of growth Governance, Risk
2024, the experimental and competitive and Compliance
breakout-year phase differentiation are key concerns
for enterprise
Generative AI 55% 75% 58%
adoption of organizations are
already in piloting or production
of CEOs believe generative AI is
a source of competitive
of business executives think
major ethical risks abound with
mode with generative AI, advantage and 50%2 are now generative AI3
reveals recent Gartner poll1 integrating the tech into their
products and services.
The Challenge

Scale and operationalize ... while mitigating foundation

generative AI faster… model related risk and
governance issues

CEOs say they feel over 6x more pressure from their 79% say of business executive say AI ethics is important to
boards and investors to accelerate generative AI their enterprise-wide AI approach.
adoption rather than to slow it down.
Client Needs
As clients move from
exploration to investigation and
production with generative AI,
they are looking for the right
model choices, robust platform
to infuse AI into applications,
and a reliable partner who can
help scale and operationalize AI
with minimal risks.
Enterprise-grade
foundation models
‘Trust’ in AI models
Hinges on transparency in data
management, methodical training
procedures and rigorous evaluation
standards.
‘Performance’
Deliver optimal performance
measures for accuracy and latency
with targeted enterprise business
domains and use cases.
'Cost-Effective'
Achieve lower inferencing costs and
total cost of ownership while
meeting performance requirements.
Trusted platform to scale
AI with confidence
Model customization
Tailor models with proprietary data and
expertise to unique use cases, company
and industry domain, with easy
integrations to ready-to-use AI apps.
Robust governance
Make AI safe and secure at scale with
AI guardrails, continuous risk
monitoring and integrated governance.
Flexible deployment
Work with the infrastructure of choice
with hybrid multi-cloud and on-prem
options to avoid vendor lock-in and
reduce total cost of ownership.
Reliable partner with
deep AI expertise
Enterprise AI leader
A partner who has a successful
track-record of bringing AI
technologies and solutions to
Fortune 500 clients and partners.
Champion of Responsible AI
Thoughtfully applies AI Ethics,
privacy and regulatory preparedness
across the generative AI lifecycle.
IBM's differentiated approach to delivering enterprise-grade foundation models
Open
→ Bring best-in-class IBM and
popular open-source models
to our watsonx foundation
models library.
→ IBM is committed to open
innovation, supporting and
contributing to open
communities.
Trusted
→ Train models on trusted and
governed data for confident
applications downstream with
minimal risks.
→ IBM Granite models are
trained in accordance with
IBM AI Ethics code and
integrated governance
approach.
→ Clients benefit from IP
indemnification for IBM-
developed models.
Targeted
→ Designed for the enterprise
and optimized for targeted
business domains and use
cases.
→ IBM Granite models that are
trained on domain-specific,
enterprise relevant data
performs on-par with 3-5x
larger models in accuracy
measures at lower latencies.
Empowering
→ Empower clients with
competitively priced model
choices to build AI that best
suits their unique business
needs and risk profiles.
→ IBM empowers AI builders to
scale generative AI faster with
tools for training, validating,
tuning, and bring your own
models (BYOM).
IBM's differentiated approach to delivering enterprise-grade foundation models

Models developed by IBM Research, open source, 3rd

party through open collaborations and partnerships.
OPEN MODEL INNOVATION
IBM Granite models are trained on enterprise relevant
content with data transparency per IBM AI Ethics code.

IBM Research LAB Alignment technique to instruct tune

TRUSTED MODEL ALIGNMENT models with new skills and knowledge.

Models are tested and benchmarked with IBM

TARGETED PERFORMANCE
FM_EVAL datasets that simulate real-world enterprise
BENCHMARKING
Gen AI applications for specific domains /use cases.

Empower clients to put Gen AI to work with watsonx

ENTERPRISE-
EMPOWERING GRADE MODELS
Access to 'Trusted, Performant and Cost-effective'
models purpose built for their enterprise.

Feedback from clients and ENTERPRISE Scale generative AI for targeted business domains
AI communities rolls up PLATFORM
and use cases on watsonx with trust and confidence.
9
IBM watsonx.ai Foundation Models Library – available today
IBM Granite Llama 3 models LAB Aligned models

Model in deprecation
What is IBM Granite ? Trusted, Performant, Cost-effective AI foundation models purpose built for enterprises.

➢ Granite is IBM's flagship series granite-13b-v2 ( English LLM ) granite-20b-multilingual

of LLM foundation models based -chat-v2.1, -instruct-v2
on decoder-only transformer
architecture. 13B parameters in size 20B parameters in size
2.5T tokens of data 2.6 T tokens of Data
➢ Granite language models are
trained on trusted enterprise
data spanning internet,
academic, code, legal and
finance.

(v1 breakdown)

➢ Chat derivative model is optimized for granite-8b-japanese

dialogue use cases and works well with
virtual agent and chat applications. 8B parameters in size
1.6T tokens of Data
➢ Instruct derivative model was designed to
perform well on natural language tasks and
can be customized for specific industries
and domains via prompt-tuning.

11
Trusted

IBM Granite models

were trained to adhere
to AI Ethics code and
governance approach
and optimized with
alignment techniques.
Alignment Bias Harmlessness

Granite-13b-V2.1 outperformed Granite-13b-V2.1 demonstrates granite-13b-v2.1 consistently

much larger models in MT Bench
Measure, a set of challenging multi-
turn questions that evaluate model
alignment with human judgement [2]
some of the least biased behavior
of studied models, according to
Bias measures in open-ended
Language Generation Dataset
(BOLD) evaluation [1]
scores high on harmlessness across
attack domains, according to AttaQ
dataset evaluation [1]

[1] Granite Report

[2]Based on IBM Internal Evaluation
Performant
According to IBM's internal benchmark, Granite-13b-chat has been consistently
IBM Research demonstrating improved performance in the past few months.

developed LLM
alignment technique
→ Large-scale Alignment of
chatBots, it is a new training
paradigm for LLMs where IBM
does very large-scale targeted
alignment on granite LLMs

Impact:
→ dramatic improvement in
Granite RAG performance

[1] Internal benchmarks

Performant
ConFinQA[1] is a standard benchmark measuring accuracy of multi-turn numeric
IBM Granite models are reasoning in financial services Q&A.
highly performant for
targeted industry domains
like finance and legal
ConFinQA Evaluation
0.4

0.3

Accuracy
0.2

0.1

[1] ConFinQA Benchmark, Granite Report

Cost Effective Base FM - Inference
Estimated # of Inference Price/1K
Our competitive prices Model Offerings Parameters tokens
watsonx granite-13B 13B $0.0006
help clients keep
granite-20b-multilingual 20B $0.0006
inferencing costs under
granite-8b-japanese 8B $0.0006
control.
llama 2-13B 13B $0.0006
llama 2-70B 70B $0.0018
flan-ul2-20b 20B $0.0050
mt0-xxl-13b 13B $0.0018
mixtral 8X7b 46.7B $0.0006
flan-t5-xxl-11b 11B $0.0018

Pricing as of March 2, 2024

watsonx.ai Use Cases + Client Stories

16
IBM is actively
Customer Service
engaging with
enterprise clients Customer service
across a broad
Empower customers to find
solutions with easy,
set of business compelling experiences
Automate answers
domains with 95% accuracy
Non-exhaustive
Marketing
Increase personalization,
improve efficiency across
the content supply chain
Reduce content creation
costs by up to 40%
Content creation
watsonx Use Cases Ex. Enhance digital sports
viewing with auto-generated
spoken AI commentary
Scale live viewing
experiences cost effectively
Find more Client Knowledge worker
Stories & Use-Cases on Enable higher value work,
improve decision making,
Seismic! and increase productivity
Source: IBM internal data Reduce 90% of text reading
and analysis work
HR, Finance,
and Supply Chain
HR automation
Reduce manual work and
automate recruiting, sourcing
and nurturing job candidates
Reduce employee mobility
processing time by 50%
Supply chain
Automate source to pay
processes, reduce resource
needs and improve cycle times
Reduce cost per
invoice by up to 50%
Planning and analysis
Make smarter decisions, focus
on higher value tasks with
automated workflows and A.
Process planning data
up to 80% faster
Regulatory compliance
Support compliance based on
requirements / risks, proactively
respond to regulatory changes
Reduce time spent
responding to issues
IT
App modernization, migration
Generate code, tune code
generation response
in real time
Deliver faster
development output
IT automation
Identify deployment issues,
avoiding incidents, optimize
application demand to supply
Reduce mean time to
repair (MTTR) by 50%+
AIOps
Assure continuous, cost-
effective performance and
connectivity across applications
Reduce application
support tickets by 70%
Data platform engineering
Redesign the approach for data
integration using generative AI
Reduce data integration
time by 30%+
Operations
Threat management
Reduce incident response
times from hours to
minutes or seconds
Contain potential
threats 8x faster
Asset management
Optimize critical asset performance
and operations while delivering
sustainable outcomes
Reduce unplanned
downtime by 43%
Product development
Ex. Expedite drug discovery by
inferring structure with AI from
simple molecular representations
Faster and less
expensive drug discovery
Environmental intelligence
Provide intelligence to proactively
plan and manage impact of
severe weather and climate
Increase manufacturing
output by 25%
Use case: App Modernization
Edger Finance: Accelerating
the collection and analysis of
investment information with
generative AI
Challenge
A fintech startup and IBM Business Partner
headquartered in Sweden, Edger Finance aims
to be the go-to solution that investors can use
to navigate the stock market and make better
investment decisions.
In 2023, Edger joined the IBM® Fintechx
program and began collaborating with IBM
Client Engineering and the IBM Innovation
Studio. The goal for the engagement was to
strengthen the firm’s processes and platform
by piloting generative AI (gen AI).
Solution
The collaboration resulted in the creation of
three AI-assisted processes that are offered
in Swedish and English and were explored
during a four-week minimum viable product
(MVP) pilot:
– The first accelerates and simplifies
the creation of a CEO summary from
corporations’ quarterly reports.
– The second automates the extraction
of data points that are within each report. 90%
– The third allows investors to interact with
improvement in the turnaround time
the data in the report through a question-
answer chat flow. for quarterly report data extracts
Each assistant relies on IBM watsonx.ai , ~96%
an integrated suite of AI tools designed
for security-rich, collaborative data improvement in time to summarize
management and process automation. The quarterly reports
third assistant also utilizes IBM watsonx
Assistant, a conversational AI platform that
delivers automated self-service support.
Case study 18
Use case: App Modernization
Dun & Bradstreet and IBM collaborate
to bring trustworthy business insights
to fuel responsible generative AI
solutions powered by watsonx
Challenge
Dun & Bradstreet, a leading global provider
of business decisioning data and analytics,
seeks to build AI use cases, implement
watsonx, and develop applications that
help address employee productivity,
enhance customer experiences, mitigate
business-to-business risks, automate
workflows, and optimize efficiency.
Above copy is an excerpt from
the press release and video.
Solution
Dun & Bradstreet and IBM, have
announced a strategic collaboration that
will bring together Dun & Bradstreet’s
Data Cloud and IBM’s watsonx to help
organizations responsibly expand their
use of generative AI. Dun & Bradstreet
also intends to leverage watsonx for its
workflows and solutions, supported by
IBM Consulting.
Minutes
instead of days procurement
process using Ask Procurement
Press release
Video
1
Use case: Customer Service
Sicredi: Using generative
AI to support improved
customer service and
employee satisfaction
Challenge
Each support representative at Sicredi is
responsible for answering questions on a
wide range of products. When a member
reaches out in person or over the phone,
the support rep is accountable for promptly
and thoroughly resolving their query. Given
the wide range of products they support,
these representatives rely on a digital
assistant to compile information to answer
each member query. Given the previous
configuration of the assistant, support
representatives often needed to escalate a
query to a product specialist in order to get
it fully resolved. This contributed to longer
wait times for members and a frustrating
experience for support reps.
Above copy is an excerpt
from the case study.
Solution
Sicredi chose to partner with IBM® Client
Engineering to augment its support
representatives’ efforts using generative AI.
Sicredi spent three weeks co-creating the
new assistant with IBM, and then spent 20
days testing it. Because the new assistant
is enabled by the IBM watsonx.ai , IBM
Watson® Discovery and IBM watsonx
Assistant solutions, Sicredi’s team can
submit a wide range of questions (varying
in complexity) in natural language. Then 10%–12% Seconds
the assistant will query Sicredi’s support improvement in query for new assistant to generate
documentation and generate an answer
resolution without escalation an answer
within a matter of seconds.
8%
Case study
decrease in abandoned
support calls
Client testimonial
LinkedIn with video 20
watsonx.ai Platform Updates

21
watsonx.ai
Train, validate, tune, and deploy AI models A next generation enterprise studio for AI builders
to train, validate, tune, and deploy generative AI,
foundation models, and machine learning capabilities.

The watsonx.ai components include:

• Foundation Model Library with

IBM and open-source models

• Prompt Lab to experiment with

foundation models and build prompts
for various use cases and tasks

• Tuning Studio to tune your foundation

models with labeled data

• Data Science and MLOps to build

machine learning models automatically
with model training, development, and
visual modeling

Learn What’s New with watsonx.ai

watsonx.ai Supported Model Architecture Types
Bring your own custom foundation model
Model Criteria
Provides greater flexibility in how generative AI solutions are created.
• Must be TGI compatible and tested by IBM
Thousands of potential models.
• Must have a config.json file
Common use cases that will benefit:
• Additional language support Supports
Model Supported
• Leverage a fine-tuned model that caters to industry or business parallel
architecture quantization
domain tensors
type method
(multiGpu)

Note: Available in software as of 4/28, targeted for SaaS in June bloom N/A Yes
codegen N/A No
falcon N/A Yes

Simple process Learning resources gpt_bigcode gptq Yes

• Technical blog gpt_neox N/A Yes

• Tutorial video gptj N/A No
• Product documentation llama2 gptq Yes
mixtral gptq No
Usage limits mistral N/A No
mt5 N/A No
• will not appear in the Tuning Studio mpt N/A No
(target Q3) t5 N/A Yes
• prompt templates will not be
evaluated/tracked by
watsonx.governance (target Q3) 23
watsonx.ai Embeddings API API spec preview for generating embeddings

• Generate embeddings endpoint to produce vector embeddings

based on input strings (learn more)
• REST API and Python SDK accessible
• Initial release on SaaS (4/18/2024), Software in June release
1
IBM Slate models available today (knowledge distilled):
Model Origin Context Dim Price per 1k
length tokens
slate.125m.english.rtrvr IBM 512 768 $0.0001 2
slate.30m.english.rtrvr IBM 512 384 $0.0001
5

Coming soon 3
• Additional API endpoints for similarity search and reranking
• LangChain and additional orchestration framework 4
integration support
• Multilingual Slate embeddings models (Q3), and other model
additions
• Fine-tuning and BYOM support for embedding models 1. IAM authorization token for IBM Cloud
2. String inputs within the request body
Model Origin Context length Dim Price per 1k tokens 3. Embeddings model ID specification
4. Watsonx.ai Project/Spaces ID for resource association
bge-large-en-v1.5 Open Source 512 1024 $0.0001
5. Response:
multilingual-e5-large Open Source 512 1024 $0.0001 a) Embeddings for each input string
b) Token count for consumption tracking
all-MiniLM-L12-v2 Open Source 512 384 $0.0001
watsonx.ai Chat Mode

• New tab in prompt lab

• A chat application to demo and

explore art of possible

• Experiment with different models to

help narrow down selection for your
use case

• Save a notebook to jump start your

application

• Also available as a standalone trial

from the watsonx.ai product page

25
watsonx.ai Taxonomy Explorer

• Displays skills and knowledge

taxonomy in a hierarchical
format

• Easy navigation - click on any

skill to see details of what was
used to train the base model.

• Provides a degree of
transparency to the model’s
training and potential behavior

• Available on the model card for

IBM granite 13b chat and other
LAB trained models

26
 Looking ahead…

• IBM opensource models (more on the way!)

• Tuning enhancements

• Additional RAG support (e.g. chat with documents)

• Enhanced AI-builder / developer experience (e.g. node.js support, additional SDK integrations)

• Exciting partnerships for multi-modal and additional multi-lingual models

• Exciting new LAB features from IBM Research – Announcements coming on May 6th and at THINK

• Data center expansions

IBM Software / © 2023 IBM Corporation / IBM

Confidential 27
watsonx.ai
Demo
Try the chat demo,
explore foundation models

https://www.ibm.com/events/think

https://www.ibm.com/community/ibm-
techxchange-conference/

28