[EXTERNAL] AI Trailblazers Workshop - Industry (5th Sep)

Industry - Cohort 3
5th September 2023

Opening & Welcome Program Leader from
9:00 - 9:15 SNDGO, EDB
And Google team
AI Trailblazers Agenda
Time Topics Descriptions Presentor
9.00 - 9.30 Registration & Refreshments Relax and Ready for the Workshop N/A
9:30 - 9:45 Opening & Welcome Agenda briefing, Program Journey and Goal Setting Glen Francis
9:45 - 10: 10 Generative AI: A Primer What and How Gen AI works, Model Adoption Dambo Ren
10:10 - 11:00 Google Gen AI Intro & Demo Overall offering, Model Garden, AI Studio, App Builder and Use Cases Demo Dambo Ren
BIO BREAK (11:00 - 11:15)
11:15 - 11:55 Custom Model & Integration Prompt Tune/Custom Open Source LLM & LangChain Integration Saurabh Mangal
Identify the challenges or problems in daily operation, and inspire the ideas
11:55 - 12:30 Use Case Brainstorming Trainee, CE, Partner Eng
for new use cases
LUNCH (12:30 - 13:00)
[All] Cloud Skills Boost Lab - Generative AI Studio
For non-technical attendees (CSB) For technical attendees
Wan Qi Ang
13:00 - 15:30 Hands On Lab / Coding Lab 1: Finetuning LLM Model
Lab 1: Vertex AI - Prompt Design Saurabh Mangal
Lab 2: LangChain Integration
Lab 2: Gen AI with Vertex AI - Getting Started
Lab 3: LangChain -QA RAG
BIO BREAK (15:30 - 15:45)
15:45 - 16:45 Solution Whiteboarding & Prototyping Group attendees based on use case, conduct the solutioning Trainee, CE, Partner Eng
16:45 - 17:00 Summary and Follow-Up Summarize the workshop and plan for the follow-ups Dambo Ren
4 Pillars as Foundation for National AI Strategy
AI Infrastructure AI Governance AI Talent AI Startups
Build Essential Advance Singapore’s Develop Sustainable A ract and Cultivate

Foundational AI Thought Leadership in Pipeline of Talent Best-in-class AI
Infrastructure to AI Governance Across Various Startups
Enable Other Personas for Future AI
Strategies Economy
Drive Awareness and Adoption of AI Among

AI for All
Key Stakeholders
Aligning Our Approach with Singapore AI Vision
AI Leader & Global AI Hub

Tech
& Capa
bility | S
cale &
Adop
tion |
Talen
AI Hu t& Edu |
b&M Respo
odel M nsible
AI Pla a rketp AI
tform lace
& Com
Data mon S
Archi ervic
tectu es
re & E
AI Inf xchan
rastru ge
cture
(GPU
| TPU
| CPU
)
Confidential + Proprietary
AI Trailblazer on the global stage of Cloud Nex
Deputy CE of
GovTech
Thanks Caroline Yap for the opportunity

to speak on Singapore’ government‘s AI
efforts, at Google Cloud Next 2023! I
spoke on the AI Trailblazer, our
collaboration with Google to bring AI
to public and private sector
organisations and the various use
cases GovTech are working on. Also
great to meet with Hamidou Dia and his
sharings during the session!
Program
Welcome
● Welcome
● Program Format
Proprietary + Confidential
Launch of Trailblazer Program
Launch of Trailblazer 24th July
Building of projects from workshops

conducted
Aug Sep Oct Nov Dec
Jul
Government / Industry Capstone Projects
Gen AI Workshops
Teams to build on their
Scale up workshops to get projects
more industries to try out
Gen AI and create use cases
AI Trailblazers Programme Journey (Industry)
Use case refinement & prototyping Capstone MVP Building
Week 0 Week 2 Week 4 Week 8 Week 8+
Tech Review
Workshop Demo Day with Capstone 🎓
with Google
Day EDB & Google MVP Graduation
Engineers
● Gen AI Hands-On Session Prep Session Prep Working MVP ready Work towards
● Submit use case(s) ● Use case(s) ● Refined use case(s) production
● Decide on ● Architecture diagram
engagement model ● Working prototype
By end of Week 4:
● Move out from sandbox to organisation
1 month sandbox environment access
© 2021 Google LLC. All rights reserved.

Trailblazer Workshops Schedules
24th July 16th August 28th September
10 Government Teams 20 Government Teams 20 Government Teams

6 Industry Teams 50
Government
Teams
Aug Sep
Jul
50
14th August 5th September 29th September 6th Oct Industry
Teams
11 Industry 12 Industry 11 Industry 11 Industry
Teams Teams Teams Teams
Gen AI/LLM Intro
9:45 - 10:10
● AI Foundation
● Generative AI/LLM Intro
● New Opportunities
A.I.
Transformative Potential
The Path to Generative Language Models
What is a Language Model?
● Natural Language Processing (NLP)
● Natural Language Generation (NLG)
A language model predicts the probability of a sequence of words in a language.
P(books) = 0.5
P(laptops) = 0.3
P(exams) = 0.1
P(minds) = 0.1
What is a Neural Network?
What is a Transformer?
Attention Is All You Need (2017)

● Encoder-Decoder
● Parallelism
○ Word Embedding
○ Positional Encoding
○ Self-Attention
○ Multi-Head Attention
○ Masked Multi-Head Attention
The Life of Large Language Model
Task Specific Model
Adapting to
downstream task
Pre-Training
Vertical Specific Model
Open Data Foundation Model ● Instruction Training

or
Internet Data ● Supervised Fine Tuning
Pre-Trained Model
● RLHF
● Domain Training Domain Specific Model
● Safety Training
Different Tuning Method
Instruction
Prompt Engineering
New Model
Fine Tuning
New Layers
Parameter-Efficient Fine Tuning
Proprietary + Confidential
Pros & Cons
Prompt Engineering Fine Tuning Parameter Efficiency Fine

Tuning
Open Data + Context + Examples Open Data + Private Data Open Data + Private Data
● End user interface to use LLM. ● Strongest performance, higher ● Frozen pre-trained model
Pros
● Guide model to output expect quality (accuracy, domain ● Comparable performance wrt FT
speci city, smaller student models,
results by writing prompt. on larger model.
fewer hallucinations)
● Zero to few-shot learning ● Unlimited training data
● Meet data residency requirements
● Limited prompt length. ● Each tuning task is a fork of ● Limited use cases
● Inference only. Can not bring your pre-train model ● Poor performance on smaller and
Cons
own model. ● Computing resources intensive foundation models

● Serve N big downstream model is
challenging
Capabilities
Google
AI Studio Model Garden Adapter Tuning

1. Be concise
2. Be specific, and well-defined
Prompt Engineering
Best Practices
3. Ask one task at a time

4. Improve response quality by
including examples
5. Turn generative tasks to
classification tasks to improve safety
Generative AI is driving new opportunities (Industry)

Operational efficiencies, cost savings, and value creation
Customer Engagement HealthCare
● Interactive Humanlike Chatbots ● Voice enabled natural language based interactions

● Conversation summarization for agents ● Domain specific entity extraction
● Sentiment Analysis and Entity extraction ● Case and audio/video call documentation summary
Technology Retail
● Generating Code Snippets from description ● Natural language prompted search and buy experiences
● Multi-modal search, personalization and recommendations
● Code Translation between languages
● Auto generated product descriptions and summarized product reviews to
● Auto Generated Documentation from Code power user experiences
Financial Services Media and Gaming
● Auto Generated summary of documents ● Auto Generated content creation for blogs, images, videos.
● Entity Extraction from KYC documents ● Trailer, script, video generation
● Summarize and rephrase investment news articles, earnings reports. ● Personalized news and content generation
Generative AI is driving new opportunities
Operational efficiencies, cost savings, and value creation
Complex data, Online interactions Content generation at Customize

intuitively accessible made conversational the click of a button Foundational models
Improve time-to-value to Improve customer Generate text, code, image, Customize large models and
search, navigate, and experience, reaching larger video, or music quickly and incorporate state of the art
extract insights and client bases by making multi-modally, speeding up generative capabilities
understanding from large online interactions more every business process and natively into your own
amounts of complex data natural, conversational, and maximizing employee internal ML operational
rewarding productivity platforms
Customer
Analyst Service AI
Creative
Data Employee Practitioner
Consumer Engagement
Make data more
accessible and useful
with Gen App Builder
Many organizations have

huge knowledge bases, and
want to leverage generative
AI capabilities for fast,
effective Q&A with their
data. Let’s take a look at how
generative AI can make
complex data more
accessible, and more useful.
Reimagine
conversational
experiences with Gen
App Builder
When online interactions

become more
conversational and
human-like, it creates an
opportunity for enterprises
to better connect with
customers, employees and
partners. Let’s look at how
generative AI can transform
the customer experience.
Gen AI Intro & Demo
10:10 - 11:00
● Gen AI on Vertex AI
● Use Cases Demo
Dambo Ren
● Enterprise Readiness
Head of
Public Sector
This revolution started at Google and we continue to innovate

Language Model Sizes - Feb 2023
1E+10 Transformer
invented
Google invented the Transformer Switch
Model Parameters (in 1000s)

architecture that powers Large Language 1E+9
MT-NLG PaLM Bard
Models in 2017. Gopher
GPT-3
1E+8
LaMDA
ChatGPT
Large Models already power Cloud AI Sparrow
1E+7
products, including Contact Center AI, T5
Document AI, NL Content Classification, and
Google Search 1E+6 GPT-2
BERT
1E+5 GPT-1
3,000+ Google / Deepmind researchers and
7,000+ scientific publications continue to 2018 2019 2020 2021 2022 2023
drive our research to product pipeline YEAR
Google / DeepMind OpenAI NVIDIA
Beeswarm/bubble plot, sizes linear to scale. Selected highlights only. Original: Alan D. Thompson, August 2022, https://lifearchitect.ai/
AI ecosystem
Vertex AI Duet AI
For builders For users
Google Cloud Next ‘23 Proprietary 041

Google Cloud Generative AI
Empower enterprises innovate faster with enterprise-ready generative AI
Build your own generative AI-powered agents
Vertex AI
AI Solutions
Contact Center | Document
Broad Ecosystem Search Conversation Responsible

of Partners AI
AI Platform
Extension | Connectors | Grounding Prompt | Serve | Tune | Distill | Eval
Model Garden
Google | OSS | Partner Models
Google Cloud Infrastructure (GPU/TPU) | Google Data Cloud

Alt with GCE remaining
Google Cloud Next

Announcing - New versions on Vertex AI GA
Foundation Model Improvements

with new versions and tuning
Quality, performance, and tuning improvements across PaLM, Imagen and Codey
PaLM New version Embeddings New version
32k context windows support longer form content Improved semantic matching for retrieval
Language expansion (38 in public available) Improved i8nl support
Imagen New version Expanded Tuning Availability

Improvement in visual appeal for images GA of adapter tuning for Imagen and Codey
Latency reductions Public preview of reinforcement learning from

human feedback (RLHF)
Experimental launch of digital watermarking for Imagen
Private Preview of Style Tuning for Imagen
Codey New version
Up to 25% quality improvement in major

supported languages
Text2SQL for better SQL performance on top of

the code models
32K context windows for longer form code

Announcing - Generally Available on Vertex AI GA
New Models in Vertex AI Model Garden

Vertex Model Garden is home to 100+ Google 1P & open-source models,
with enterprise MLOps tooling built in
Customize & deploy Llama 2 on Vertex AI

One-click deployment of all the Llama 2 model variants (7B, 13B, 70B), or tune
the model with your own business datasets
Pre-announcing Anthropic Claude 2 model

Access the next-generation Claude 2 LLM from Anthropic on Vertex AI
Tap into advanced enterprise MLOps tooling

Open-source models in Vertex Model Garden come with GCP’s enterprise MLOps
tooling, including built-in capabilities such as one-click deployment, parameter
efficient fine tuning (PEFT), reinforcement learning with human feedback (RLHF),
and responsible AI safety checks

Announcing - Private Preview on Vertex AI Preview
Extensions
Enable foundation models to take action
Connect models to real-time data and real-world actions Use Cases: Factuality on private data, retrieval
augmented generation, user personalized
Extensions provides a set of fully-managed tools for building and managing
responses and code interpretation
extensions that connect models to proprietary data sources or 3rd party services
Access pre-built extensions

Developers can also access pre-built extensions from 3rd party partners and
upload their own extensions for internal or external use
Built-in integrations with LangChain

Developers can continue to use open source frameworks like LangChain with
easy deployment and tooling through Vertex AI
Enterprise-grade tooling
Will support data transparency, data residency, customer-managed encryption
keys, and virtual private cloud security controls
Build using the single extension standard across Google

Announcing - Private Preview on Vertex AI Preview
Grounding for PaLM API

and Vertex AI Search
Ground model responses with your own enterprise data
Ground PALM API responses with enterprise data

The only generative AI platform with grounding capabilities on your enterprise data
Responses augmented with even better quality and factuality
API in now in Private Preview
Ground Vertex AI Search with enterprise data

Integration out-of-the-box without any development work
Testing in Private Preview with a small set of partners before
making it available more broadly

Vertex AI Search
Deliver relevant, personalized search experiences in minutes, for enterprise apps
or consumer-facing websites
Quickly set up Google quality multi-modal,
multi-turn search experiences for customers and employees
Ground search results in enterprise data or combine
with LLM knowledge
Enterprise access controls ensure information is surfaced
only to users who have access
Data connectors and extensions connect to backend
systems to read data and take actions with 3rd party applications
like Salesforce, etc.
Vector Search for highly scalable vector embedding search across
different types of data
Helpful features like summarization, citations, relevance scores,
flexible LLM style prompting help in knowledge discovery and
customize search results

Vertex AI Conversation
Build custom chat and voice bots powered by generative AI and grounded in
your enterprise data
Combine deterministic workflows with generative AI to make
conversations dynamic and personalized
Multi-modal support image and audio in addition to text
Train on a variety of data from websites, documents, FAQs, email
and agent conversation history
Enable transactions to connect to backend systems or 3rd party
applications for complete transactions (bookings, payments etc)
Generators and generative fallbacks inject actions and
pre-programmed conversations into workflows
Helpful features for customer service workflows summarization,
generative FAQ, citations and agent assist
Fully integrates with existing

Contact Center AI offerings

Pre-Announcing - Generally Available on Vertex AI GA
Colab Enterprise on Vertex AI

Colab Enterprise combines the ease of use of Google Colab notebooks with
the enterprise-level security and compliance capabilities of Google Cloud
Use Cases: Data science, data analysis,

data engineering, ML engineering
Collaboration & Productivity Enterprise Ready
IAM based notebook sharing Will support a wide range of security and
management capabilities including:
Automatic Versioning
● VPC-SC
Commenting (coming soon!)
● CMEK
Co-editing (coming soon!)
● Regionalization
Generative AI powered code completion
and generation ● Cloud Monitoring
Zero-Config & Flexible Compute ● Cloud Logging
Provides both zero-config compute options, as Available across Google Cloud

well as access to a wide range of machine-shapes
Available in BigQuery and Vertex AI (Dataproc
and compute
and Dataflow coming soon), making it easy to
work across data and AI workloads

Enterprise Readiness for Foundation Models
Your organization’s Built-in support for The right tools to

data is yours, safe model outputs future-proof your AI
& its secure investment
Organizations can leverage
technical tools to detect sensitive
Organizations always have control We continually innovate across our ML
content and take appropriate platform with tooling that is upgraded to
over where and how data is used. action fulfill the large needs of foundation
Tune & customize FMs with your data models (eg. 500GB+ models)
for your use cases, while maintaining
privacy.
Enterprise Architecture Your OnPrem or

Alternative Cloud
Your Google Cloud Perimeter
Generative AI App Builder
Your content
Google Cloud
Central Hosting
Conversation Enterprise
AI search
Chatbot, API, etc. Your data
Internet/Intranet
Vertex AI
Your inference
Large Base Model

Your Users Adapter Layers
(Frozen)
Your security
CMEK DRZ AxT VPC-SC
Use Case Demo
Enterprise Search Summarization & Conversational

Create a search application based Classification Engagement
on non-structure data with
Generate summary and thesis Create a chat experiences with
no-code experiences
base on the context, and provide the prompt design and context
the classification with few shots
prompt
Time for a break!
We’ll start at 11:00
15 mins break
Custom Model &
LangChain
Integration
11:15 - 11:55
● Model Tuning
● LangChain Integration
● Fine Tuning with Google Model
● Fine Tuning with OSS Mode
Model Tuning
Adapting a PLM to a downstream task
(Full) Fine
Tuning
PLM (24B) Task Specific Model (~24B)
Parameter
Efficient Fine
Tuning
PLM (24B) PLM (~24B+) with task
specific tuned parameters
Prompting Task Specific

Prompt
Methods
PLM (24B) PLM (24B)

Parameter-Efficient Tuning Methods
(PETM):
Methods for tuning an LLM on your own custom
data without duplicating the model. The base model
itself is not altered. Instead, a small number of
add-on layers are tuned, which can be swapped in
and out at inference time.
Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuning Confidential + Proprietary

Adapter Tuning:
Adapter tuning is a method of transfer learning for
natural language processing (NLP) that involves
adding small, task-specific "adapter" modules to a
pre-trained language model.
How does it work?
The adapter modules add a small number of new

parameters to a model to be trained while tuning.
The modules allow the pre-trained model to be

fine-tuned for the task without having to retrain the
entire model.
During training, the adapters may then be activated/

deactivate to change the distribution of activations
throughout the network.
Parameter-Efficient Transfer Learning for NLP

Prompt Tuning:
This method freezes the parameters of the LLM and
then adding a small number of trainable parameters
to the input of the model.
These trainable parameters are called "soft

prompts". Those prompts are used to condition the
LLM on the specific task at hand.
How does it work?
Who wrote The Lord of the Rings?

Similar to prompt engineering, but it prepends
embeddings as “soft prompts” instead of
human-readable examples aka “hard prompts”.
Soft prompts are fed together with the input Tunable Soft Prompt
prompt to get the tuned response for the
downstream tasks.
The pre-trained model stays frozen while tunable

soft prompt part learns the embedding of prompt Pre-trained Lanaguge Model
tokens via back propagation.
The Power of Scale for Parameter-Efficient Prompt Tuning Confidential + Proprietary

Low-Rank Adaptation (LoRA):
Adds pairs of rank-decomposition weight matrices
(called update matrices) to existing weights, and
only trains those newly added weights
LoRA: Low-Rank Adaptation of Large Language Models

BitFit:
Freeze everything, except the bias
BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models Confidential + Proprietary
Reinforcement Learning from Human Feedback Proprietary + Confidential
Open Issues: LLM are not aligned with human assessment

● do not follow user instructions
● exhibit unintended behaviors (e.g. makeup facts, generate biased or toxic text)
● raise broad and diverse problems for enterprise customers
Reason: Misalignment between language training objectives and human/business needs.
Approach
1. Take Human Feedback
Human demonstration Human preference
Prompt: write a story about a robot and a young boy. Prompt: write a short story about a robot and a young boy in 30 words.
Human demonstration: A boy named Timmy becomes inseparable from a Response 1: A young boy befriends an abandoned robot in his town. They
new robot his parents brought home. Their friendship grows stronger as become close friends, but as the boy grows up and moves away, he
they share new adventures together. remembers their bond.
Human Preference (👍)
Response 2: A robot became a boy's best friend.
Human Preference (👎)
Expensive to collect; lack of “negative signal”
→ supervised fine-tuning Cheaper to collect; “rich comparative signal”
→ How to incorporate them into the model?
Reinforcement Learning from Human Feedback Proprietary + Confidential
Approach
2. Training language models on objectives that are linked with human feedback.
Reinforcement Learning (RL):
● Unique way of taking valuable signals
● Directly optimizes a reward defined by human preference.
Key idea
Agent 1. Prompt Environment ● Language model as a policy.

● Contextual-bandits
environment
Context ● Prompt: input (context/state) to
the policy.
● Response: action output by the
Policy 2. Response
policy.
Reward model
[Language model] ● Reward: function of the prompt
3. Reward and the response, reflecting
human preference
● Policy is optimized using reward
as a feedback.
LangChain
Typical usage of LLMs
Tasks
● Classification
● Summarization
● Q&A
API
LLM
Common Patterns
1 Webhook
2 LLM
Google
Search
LLM
LLM LLM
SQL LLM
Code
LLM
3 LLM LLM LLM
… and more
LangChain 🦜🔗
Open source developer framework for
developing LLM applications built around LLMs
● Chatbots
● Generative QA
● Summarization
● LLM agents
Python and TypeScript packages

Focused on composition and modularity
Value adds:
● Modular components + implementation
● Common ways to combine those
components together
LangChain 🦜🔗
Google Drive
Google Search
Google PaLM2
Vertex AI PaLM API APIs & webhooks

🦜🔗 LangChain
Knowledge Repositories
Databases
Code Repositories
LangChain Concepts
Agents
Core Idea: Agents is using LLMs as a reasoning engine to determine how to interact with the outside
world based on user input.
● Use LLM as the “reasoning engine”

● Non-deterministic sequence of actions
Webhook
Why use Agents?

Search
● Connect LLM to external data sources or computation LLM
○ Search, APIs, Database, Calculators, run code … SQL
● Recover from errors, handle multi-hop tasks
Code
Agents: Implementation
Pseudo implementation: LLM

● Select the Tool to use
● Observe output of the Tool 2. Q with agent prompt 3. Return agent instructions
to answer or use tools to
● Repeat until stopping condition get additional info
met
1. Response 5. Get additional info
Agent
Asks question 4. Run tool (Action)
with inputs Tools
User
7. Final answer 6. Send additional

context
LLM
LangChain + Vertex AI
Vertex AI PaLM — Text, Chat, and Embeddings — and Vertex AI Matching Engine are officially
integrated with the LangChain Python SDK, making it convenient to build applications on top of
Vertex AI PaLM models.
● LangChain with Vertex AI PaLM for LLMs
● LangChain with Vertex AI PaLM for Chat
● LangChain with Vertex AI Embedding API for Text
● LangChain with Vertex AI Matching Engine
● Langhchain with Ent. Search

Retrieval
Augmented
Generation
Retrieval Augmented Generation (RAG)
Meta - RAG Sept, 2020

High-Level Architecture Proprietary + Confidential
Use case: Max is a developer looking to search for code snippets, confluence design
documentation or kick starting their role with easy access assets. The architecture
leverages a retrieval augmented generation (RAG) architecture to search huge corpus of
data, urls and other disparate assets. Components Description
Breakdown query: From the user input, leveraging

1 chain-of-thought methods, the query will be break down into its
atomic components and be processed sequentially. In this case, if
multiple searches are needed to fulfill the query, then each search
will be done sequentially, retaining the information from previous
2 searches in context.
2 Search URLs and Documents: Based on the search queries,

Enterprise Search could query indexed documents and URLs for the
Extract Question 1 Search URLs and 2 Search Similar 33 Extract Image 4 most relevant responses and rank them accordingly.
Text
Query Documents Embeddings
New business
prompt 3 Search for similar business/locale: We call the Vertex Matching
Engine match function, and pass it the text embeddings. Vertex
Matching Engine
Vertex Enterprise Vertex Matching Engine will return top ~n nearest neighbor text matches.
Embedding Search Embeddings
Matching Engine text index created.
Summarize relevant 5 Extract embeddings from text block documents: We extract the
information with 4
chat context text embeddings with Vertex AI Embedding Extractor. We will use
these embeddings to search for similar text passages across long
User Input
form documents
PaLM
Summarize relevant information with chat context: Not only does
5 the search operations return multiple documents, but also if more
Public/Intranet
6 Relevant long form than one search is needed, then there could potentially be multiple
Websites
Using chat documents documents. With the objective to condense that information, a
memory and map-reduce style of summarization will be done.
Chat relevant results
response from search,
return results with Return result: Using the contents of the search summarization and
URLs linking to 6 previous chat iterations, a new relevant response will be
websites or constructed using the PaLM API.
documents.
Retrieval QA Chain: Ingestion

from langchain.embeddings import VertexAIEmbeddings
from langchain.document_loaders import GCSFileLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.vectorstores import MatchingEngine
Ingestion
# Define Text Embeddings model
embedding = VertexAIEmbeddings()
# Define Matching Engine as Vector Store

🦜🔗 Ingest me = MatchingEngine.from_components(
Documents project_id=PROJECT_ID,
region=ME_REGION,
🦜🔗 Split in gcs_bucket_name=f'gs://{ME_BUCKET_NAME}',
Chunks embedding=embedding,
index_id=ME_INDEX_ID,
endpoint_id=ME_INDEX_ENDPOINT_ID)
Chunk 1 Chunk 2 … Chunk n
# Define Cloud Storage file loader to read a document

loader = GCSFileLoader(project_name=PROJECT_ID,
Embedding 1 Embedding 2 … Embedding n
bucket=bucket, blob=prefix)
Build
# Split document into chunks
Vector Store
Index Streaming Inserts Matching Engine text_splitter = RecursiveCharacterTextSplitter(
Fetch actual text chunk_size=1000, chunk_overlap=0)
based on doc ids doc_splits = text_splitter.split_documents(document)
Doc Chunks (Text)
Cloud Storage
# Add embeddings of document chunks to Matching Engine
texts = [doc.page_content for doc in doc_splits]
me.add_texts(texts=texts)
Retrieval QA Chain: Query Time from langchain.chains import RetrievalQA

from langchain.llms import VertexAI
RAG Chain # Define Matching Engine as Vector Store

LangChain 🦜🔗 Retrieval QA Chain
me = MatchingEngine.from_components(
"How was Alphabet’s net income in 2022?"
project_id=PROJECT_ID,
"What contributes to this company’s growth?" region=ME_REGION,
gcs_bucket_name=f'gs://{ME_BUCKET_NAME}',
Multi-turn embedding=embedding,
conversation
index_id=ME_INDEX_ID,
Question Answer endpoint_id=ME_INDEX_ENDPOINT_ID)
Query # Expose Matching Engine index as a retriever interface

LLM to
Embedding generate retriever = me.as_retriever(search_type="similarity",
response search_kwargs={"k":NUM_OF_RESULTS})
Semantic
Search
Retrieve top-k relevant documents
# Define LLM to generate response
Approx. Nearest llm = VertexAI(model_name='text-bison@001')
Neighbors Search
# Create QA chain to respond to user query along with source documents

Vector Store qa = RetrievalQA.from_chain_type(llm=llm,
Matching Engine
chain_type="stuff",
Fetch actual text
based on doc ids
retriever=retriever,
return_source_documents=True)
Doc Chunks (Text)
Cloud Storage
# Run QA chain
result = qa({"query": query})
Common Use Cases with LangChain
● Question & Answering over private corpus

● Large document Summarization
● Personal assistants & Chatbots
● Querying tabular data
● Code understanding
● API interactions
● Information Extraction
● Evaluation
● …
Gen AI Use Cases
Brainstorming Trainees, CEs and
11:55 - 12:30 Engineers from Partners
AI is a top and urgent priority for our customers*
Therefore, we’re putting an AI lens on enablement plans this year
Top technology priorities for 2024 Urgency to adopt Gen AI
Adopting AI 89% 42%
Securing against threats 48%

Evolving legacy
41%
to optimized data
26%
Enabling better collab 30%
20%
Updating infra 26% 9%
3%
Other 16%
No 2 3 4 We’re
urgency - 1 late - 5
*Based on pre-event survey of 113 of Next’23 Leaders Circle attendees

Top use cases: bots, coding and efficiency
Our generative AI announcements at NEXT support our customer priorities
2024 Priority use cases
73%
65%
58%
52% 47%
27% 27% 25% 24%

21%
Customer Software General Data Content Marketing Security Quality Forecasting Sales
service, development efficiency analysis creation assurance manage supply process
chat bots (automation)
*Based on pre-event survey of 113 of Next’23 Leaders Circle attendees

Design Thinking
Workshop for
AI Trailblazers
Confidential & need to know - Do not share or forward

Design Thinking: Team Activity 1
1. Identify and assign roles within the team (5 mins) Make a copy of the
○ Roles: Leading the discussion and follow-up, Design Doc ⬇!
create slides and take notes
○ Google CE & Partner Eng will facilitate the session
2. Make a copy of the Design Doc here ➡

○ Use Google account (if you have) or Gmail account
3. Craft or Refine the use cases - (15 mins)

tinyurl.com/ai-tb-
4. Frame problem statements - (15 mins) design-doc
Hands On Lab
13:00 - 13:30
● Cloud Skills Boost Env Walkthrough
● Hands On Lab
○ Vertex AI: Qwik Start
○ Getting Started with
Generative AI Studio
Hands-on Session: Time breakdown
Start / End Session Name
13:00 - 13:30 [All] Cloud Skills Boost Lab - Get Started with Generative AI Studio
Track 1: Prompting with PaLM API Track 2: Fine-tuning and LangChain

Meeting Rooms Main Room (Developer Space)
● Fine-tuning LLMs
● Zero/few-shot prompting
13:30 - 15:30 ● LangChain with Vertex PaLM API
● PaLM API
● LangChain - QA with Documents
Get your Hands-on Lab materials here → tinyurl.com/ai-tb-labs-5sep

Hands-on Lab:
Cloud Skills Boost
Cloud Skills Boost: Lab Prerequisites

Follow these instructions if you don’t have a Cloud Skills Boost account else skip to Step 4
Step 1: Open a new window or new tab and go to

https://www.cloudskillsboost.google/catalog?qlcampaign=6f-trailblazers-27
Step 2: Click on Join at the top

right and choose to either sign in
with Google or fill up your info
Step 3: Once your account is

created, log in to your account and
skip to Step 8

Follow these instructions if you already have a Cloud Skills Boost account else go back to Step 1
Step 4: Open a new window or new tab and go to www.cloudskillsboost.google
Step 5: If you see that you’re signed in to your account, please sign out
Step 6: Once signed out, in the same tab or window, go to

https://www.cloudskillsboost.google/catalog?qlcampaign=6f-trailblazers-27
Step 7: Now, sign in to your Cloud Skills Boost account and continue to Step 8

Step 8: Once signed in, click on your profile at top right corner and you should see
that you have 9 credits loaded into your account.
If you don’t see the credits, please try signing out and signing in again.

Step 9: In the top menu, click on Explore to open up the labs catalog and search
for the lab: “Get Started with Generative AI Studio” then click into the lab.
Step 10: In search bar, type Get Started with Generative AI Studio and search.
In the search results, click into the lab.
Cloud Skills Boost: Starting the lab

Step 11: Click Start Lab. Step 12: The lab username and password are
When prompted, select provided to you on the left. Right click on
‘Launch with 1 credit’ “Open Google Console” and open it using
an incognito tab.
Cloud Skills Boost: Signing in and completing the lab
Step 13: In the incognito window, sign

in to the lab environment using the
username and password provided.
Step 14: Once signed in, follow the lab

instructions to complete the tasks.
Step 15: Once all tasks are completed,

click on End Lab at top left panel.
Ending this hands-on

section at 1.50pm
Breakout Session: Choose your track!
Track 1: Prompting Track 2: Fine-tuning and LangChain
● Zero/few-shot prompting ● Fine-tuning LLMs

● PaLM API ● LangChain with Vertex PaLM API
Please move to the Please stay put in the

meeting rooms main room
Track 1 (Cloud Skills Boost): Prompting with PaLM API
Once you’re done with the first lab: Vertex AI: Qwik Start, you can try out other
labs as listed below:
● Generative AI with Vertex AI: Getting Started
● Generative AI with Vertex AI: Prompt Design
Note: These labs will be available to you for 30 days starting from today
Hands On:
Custom LLM &
LangChain Integration
13:30 - 15:30
Hands-on Lab:
Google Cloud
Environment
Cloud Environment Setup: Industry Learners
User Account
Email ai-sb-user-##@ai-trailblazer-sandbox.com
Password Default password (to be changed upon login)
Individual Projects (No GPUs) Shared Project (With GPUs)
Project ID ai-sandbox-<company> Project ID trailblazer-innovation-sandbox
● 1 company → 1 project ● Only for using the GPU

● Try out all GCP products/services except ● More details will be shared later
GPUs
## represents the number that the attendee is assigned to

Getting Started: Get your accounts ready!
Step 1: Go to tinyurl.com/ai-tb-userid-5sep to
reserve User IDs
Step 2: Open an incognito Chrome window and go

to console.cloud.google.com.
Step 3: Log in with your User ID and default password.
Step 4: When prompted, change the password. Do

remember your new password as you’ll be using this
account to access the sandbox environment.
Step 5: Once in console.cloud.google.com, check

the Terms of Service box and click on AGREE AND
CONTINUE when prompted.
Getting Started: Check that you have access to GCP projects

Check that you can access your company’s sandbox project - raise your hand if you encounter issues.
Step 6: Click on the Select a Step 7: Select AI-TRAILBLAZER-SANDBOX.COM

project dropdown menu. as organisation and click on the project
created for your company.
Project name:
ai-sandbox-<your-company>
Getting Started: Check that you have access to GPU project

Check that you can access the GPU sandbox project - raise your hand if you encounter issues.
Step 8: Click on the Select a Step 9: Select AI-TRAILBLAZER-SANDBOX.COM

project dropdown menu. as organisation and click on the shared
GPU project
Project name: Trailblazer

Innovation Sandbox
Hands-on Lab:
Fine Tuning LLMs with
A100 80GB GPU
IMPORTANT: Use it only with Compute Engine
IMPORTANT
GPU Project for Hands-on Lab
● Each team has been allocated 3 GPUs each for the hands-on lab.
● You will find them named in this format:
○ <company>-gpu-1
○ <company>-gpu-2
○ <company>-gpu-3
● Please consider sharing the screen with each other or take turns to try the
fine-tuning lab
Hands-on Fine Tuning: A100 80GB GPU VM instance
Step 0: In the Google Cloud console, ensure

that the GPU project is selected (Trailblazer
Innovation Sandbox).
Step 1: Click on the three bar Menu icon on

the top left corner to open up the Menu side
panel.
Step 2: Click on Compute Engine > VM instances

Hands-on Fine Tuning: SSH into your A100 80GB GPU VM instance
Step 3: Look for the GPUs assigned to your company and choose one of it
Step 4: Click on the SSH button to SSH into your VM instance

Hands-on Fine Tuning: SSH into your A100 80GB GPU VM instance
Step 5: In the SSH window, click on Authorize

Hands-on Fine Tuning: Set up your A100 80GB GPU VM instance
Step 6: When prompted, type “y” to Nvidia Driver Installation. Once done, you
should see Nvidia driver installed printed in your terminal
Step 7: In your SSH terminal, run the following commands:

sudo rm -f '/root/google_vm_config.lock'
sudo chmod -R 777 '/root'
This to provide permission required for installing packages (& compiling with CUDA)
Hands-on Fine Tuning: Copy files to your VM instance
Step 8: Upgrade pytorch to 2.0 with the following command:
pip install --upgrade torch
If you see this error → ERROR: Could not install packages due to an OSError: [Errno 13]
Permission denied: 'autocast_mode.cpython-310.pyc'
Consider using the --user option or check the permissions.
Use this command instead → pip install --upgrade torch --user
Step 9: Copy the Python notebooks from Cloud Storage bucket with:
gsutil cp -r gs://llm-models-base-bucket .
Step 10: Run ls and check that you copied llm-models-base-bucket successfully
Hands-on Fine Tuning: Starting Jupyter Lab
Step 11: To start the Jupyter Lab, run the following command:
jupyter lab --ip=0.0.0.0 --port=8888
Step 12: Copy the URL that starts with http://127.0.0.1:8888/lab?token=...
Step 13: Head back to Compute Engine page and copy your VM’s External IP
Step 14: Replace 127.0.0.1 with your External IP and paste it in a new window/tab
Example
http://35.186.147.47:8888/lab?token=b6099de8e953f40a39092d892c3ac5397a251cd0599cd1ec
Hands-on Fine Tuning: Fine Tuning Llama-2-7b
Step 15: Click into the

Llama_2_Fine_Tuning_using_QLora_main
Python notebook
Step 16: Uncomment the Setup section to install the packages. Once installed,
restart the kernel by clicking Kernel > Restart Kernel.
Step 17: Once your packages are installed, restart the kernel then run the
remaining cells to start fine-tuning Llama-2-7b.
Potential issues you might face:
If you see the following error message:

AttributeError: module ‘importlib’ has no attribute ‘metadata’.
Replace import importlib with import importlib.metadata
If you see the following error message:

Repo card metadata block was not found. Setting CardData to empty.
Ignore it and continue on.

Hands-on Lab:
LangChain with
Vertex AI Integration
Hands-on LangChain: Starting Vertex Workbench
ai-sandbox-<>
Step 0: In the Google Cloud console

(console.cloud.google.com), ensure that the
non-GPU project is selected (ai-sandbox-#).
Step 1: Click on the three bar Menu icon on

the top left corner to open up the Menu panel.
Step 2: Under ARTIFICIAL INTELLIGENCE, click

on Vertex AI > Workbench.
Step 3: Click on USER-MANAGED NOTEBOOKS.

Step 4: You should see that there are notebooks created for your team.
Step 5: Choose one of it and click on OPEN JUPYTERLAB. If your notebook

was in the ‘Stopped’ state, it will take a few minutes to start up.
Step 6: Once a green tick appears, click OPEN JUPYTERLAB to open the notebook
Step 7: Verify that Jupyter Notebook opens up in a new tab.

Hands-on LangChain: Clone Generative AI GitHub Repo
Step 8: In the Jupyter Notebook, click on Terminal to open a new terminal tab.
Step 9: In the terminal, run the command

git clone https://github.com/GoogleCloudPlatform/generative-ai.git
Hands-on LangChain: LangChain + Vertex AI PaLM API Notebook
Step 10: Navigate to generative-ai > language > examples > langchain-intro
Step 11: Click on the Python notebook named intro_langchain_palm_api.ipynb
Step 12: Run all the cells to see LangChain + Vertex AI PaLM API in action.
Hands-on LangChain: Wrap Up
All notebooks can be found on our official GitHub

⬇
goo.gle/gen-ai-github
💻 Try out the other Generative AI notebooks!

Hands-on Lab:
Vertex AI Search
Hands-on Vertex AI Search: Search Apps
In this lab, you will learn how to quickly create search apps with 3 types of data:
● Website content - cloud.google.com
● Structured data - NDJSON-formatted dataset
● Unstructured data - PDFs ingested from a Cloud Storage bucket
Step 1: Go to Vertex AI Search tutorial for step by step guide

Step 2: Follow the instructions to create the search apps in your individual
company project ai-sandbox-<company>
IMPORTANT
Dos and Don’ts: GCP AI Sandbox Environment

● Shared GPU AI Sandbox
○ Only use the A100 80GB GPUs that were created for you
○ By default, 1 company gets 1 GPU instance
■ If you require > 1, please email Praizel <Praizel_TAY@edb.gov.sg> and the request will
be looked at depending on availability and use case.
○ Do not use resources that don’t belong to you
○ Resource names: use your org name as prefix e.g. <org>-bucket for easy identification
● Individual Organisation Project
○ No GPUs in these projects
○ Free to use any other services within this project to do testing / experimentation
○ Treat the sandbox like your own → shut down any resources that you do not need
○ Do not create new projects - billable services will not be available.
IMPORTANT
Data in GCP AI Sandbox Environment
● It’s Sandbox not Production env
● Use Dummy data, Open data, Non-sensitive data
● Special use cases which need PII or Sensitive data, please
deidentify/tokenize/mask data
● Consult with Google tech team if you are not sure

Solution Whiteboarding
& Prototyping Trainees, CEs and
15:45 - 16:45 Engineers from Partners
Design Thinking
Workshop for
AI Trailblazers
Confidential & need to know - Do not share or forward

Design Thinking: Team Activity 2
1. Brainstorm ideas and tech approaches - (25 mins)
2. Conduct ideas prioritization if need - (5 mins)
3. Whiteboarding and prototyping - (30 mins)
4. Deliverables
○ Draft version of solution design or prototype artifacts
○ Action plan to build the prototype
○ Action plan to verify the values/benefits from target users
Summary and
Follow-Up All
16:45 - 17:00
Use Case Development & Deployment Services (self-funding)
Inhouse Engineers Google SI Partners Google PSO
Service Category Unit Price(USD) During Description
Google PSO Development Service $25,000.00 2 Weeks Use case evaluation, feasibility study, and prototyping
Google SI Partner $20,000.00 2 Weeks Use case evaluation, feasibility study, and prototyping
Production Deployment It's highly depends on agencies env and requirements, and expect agency/organization has to
Complex Solution apply their own budget to cover the production deployment
The Machine Learning Advanced Solutions Lab (ASL) is
focused on significant Google Cloud
$200,000.00 4 weeks + customers/prospects to work with core Google
engineering and the GCP professional services team
ASL Immersive Training on hard, real-world ML problems with client data.
With Product Strategy partnership Google will help you….
Understand “what”, “who”, “why” and Plan “how”, “when”
UX Research Plan
UX Study Report Roadmap
Product Story CUJs/Personas Milestones
UX Design Prioritization
Success Metrics
Product Req Document
Tech Design
Sharad
GenAI offerings
Align Jumpstart Implement

Implement Maintain
Maintain Optimize
Understand your Build a GenAI

GenAI objectives Minimum Viable Operationalize GenAI Resident Cloud Optimize
and high-priority Product (MVP) for
Fine tune, test, and AI Specialist GenAI on GCP
use case(s) each prioritized
productionize your MVP model Get expert advisory
use case Implement, maintain,
optimize on Infra and Security
Create
Resident Cloud AI
Specialist
Summarize Resident Cloud
Implement, maintain, optimize AI Specialist
Implement, maintain,
Discover
optimize
AI Readiness
Automate Set your GenAI roadmap
Custom AI
Build your production Custom
AI solution
Summary and Actionable Follow-up (Industry)
1 Prioritize and categorize use cases

● Categorize use cases in magic quadrant of two dimension: Complexity and Values
● Prioritize the top 1-3 Use Cases
2 Decide on the implementation resources

● Inhouse Engineers, Service Partner or Google PSO?
3 Update use case slide and submit to organizing team

● Fill up the Google Forms (bit.ly/ai-tb-workshop-ind) → 1 response = 1 use case
● EDB POC: Praizel Tay <Praizel_TAY@edb.gov.sg>, Wei Yi Ow <OW_Wei_Yi@edb.gov.sg>
4 Book a tech review session with Google Engineers

● Come prepared to discuss use case & architecture with our ML specialist
● Book slots on 25th / 26th Sep → bit.ly/ai-tb-techreview
Next Steps Form/Finalize the team members, define R&R
(Industry) Setup communication & follow-up cadence
AI Trailblazer is the first step of Attend the WEEK + 2 Checkpoint (Tech Review)
8 weeks of engagements,
Build a prototype to test your ideas

Suggested Next steps are
listed:
Attend the WEEK + 4 Checkpoint (Demo Day)
Refine the use case development and build a MVP
Validate the MVP with end users
Attend the WEEK + 8 Checkpoint (Capstone Project & Graduation )
© 2021 Google LLC. All rights reserved.

We hope you enjoyed the

session today 😃
Please share your feedback

with us!
bit.ly/AI-Trailblazer-Survey
Join Google Gen AI
Community:
Industry Customers
Join the Google Gen AI

Community on WhatsApp!
Groupie Time !!
THANK YOU!!

[EXTERNAL] AI Trailblazers Workshop - Industry (5th Sep)

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

[EXTERNAL] AI Trailblazers Workshop - Industry (5th Sep)

Uploaded by

Copyright:

Available Formats

Industry - Cohort 3

5th September 2023

AI Infrastructure AI Governance AI Talent AI Startups

Build Essential Advance Singapore’s Develop Sustainable A ract and Cultivate

Drive Awareness and Adoption of AI Among

AI Leader & Global AI Hub

Thanks Caroline Yap for the opportunity

Launch of Trailblazer Program

Launch of Trailblazer 24th July

Building of projects from workshops

Aug Sep Oct Nov Dec

Use case refinement & prototyping Capstone MVP Building

Week 0 Week 2 Week 4 Week 8 Week 8+

1 month sandbox environment access

© 2021 Google LLC. All rights reserved.

Trailblazer Workshops Schedules

24th July 16th August 28th September

10 Government Teams 20 Government Teams 20 Government Teams

A language model predicts the probability of a sequence of words in a language.

Attention Is All You Need (2017)

Task Specific Model

Vertical Specific Model

Open Data Foundation Model ● Instruction Training

Pros & Cons

Prompt Engineering Fine Tuning Parameter Efficiency Fine

own model. ● Computing resources intensive foundation models

AI Studio Model Garden Adapter Tuning

3. Ask one task at a time

Generative AI is driving new opportunities (Industry)

Customer Engagement HealthCare

● Interactive Humanlike Chatbots ● Voice enabled natural language based interactions

Financial Services Media and Gaming

Complex data, Online interactions Content generation at Customize

Many organizations have

When online interactions

This revolution started at Google and we continue to innovate

Model Parameters (in 1000s)

For builders For users

Google Cloud Next ‘23 Proprietary 041

Build your own generative AI-powered agents

Broad Ecosystem Search Conversation Responsible

Google Cloud Infrastructure (GPU/TPU) | Google Data Cloud

Google Cloud Next ‘23 Proprietary 042

Google Cloud Next

Foundation Model Improvements

PaLM New version Embeddings New version

Language expansion (38 in public available) Improved i8nl support

Imagen New version Expanded Tuning Availability

Latency reductions Public preview of reinforcement learning from

Up to 25% quality improvement in major

Text2SQL for better SQL performance on top of

32K context windows for longer form code

Google Cloud Next ‘23 Proprietary 045

New Models in Vertex AI Model Garden

Customize & deploy Llama 2 on Vertex AI

Pre-announcing Anthropic Claude 2 model

Tap into advanced enterprise MLOps tooling

Google Cloud Next ‘23 Proprietary 046

Access pre-built extensions

Built-in integrations with LangChain

Google Cloud Next ‘23 Proprietary 047

Grounding for PaLM API

Ground PALM API responses with enterprise data

Ground Vertex AI Search with enterprise data