迁移学习 - 迁移学习

第二章:迁移学习
m
2.1 迁移学习理论 he
i ma
. co
i t
w.
ww
学习目标程
序
员
马
了解迁移学习中的有关概念
黑 .
掌握迁移学习的两种迁移方式.
迁移学习中的有关概念:
预训练模型
微调
微调脚本 . co
m
m a
h ei
it
w.
预训练模型(Pretrained model): ww
员
一般情况下预训练模型都是大型模型，具备复杂的网络结构，众多的参数量，以及在足
程
序
够大的数据集下进行训练而产生的模型黑
马 . 在NLP领域，预训练模型往往是语言模型，因
为语言模型的训练是无监督的，可以获得大规模语料，同时语言模型又是许多典型NLP
任务的基础，如机器翻译，文本生成，阅读理解等，常见的预训练模型有BERT, GPT,
roBERTa, transformer-XL等.
微调(Fine-tuning):
根据给定的预训练模型，改变它的部分参数或者为其新增部分输出结构后，通过在小部
分数据集上训练，来使整个模型更好的适应特定任务. .com
a
e im
h
. it
微调脚本(Fine-tuning script): www
员
实现微调过程的代码文件。这些脚本文件中，应包括对预训练模型的调用，对微调参数
程
序
的选定以及对微调结构的更改等，同时，因为微调是一个训练过程，它同样需要一些超
黑
马
参数的设定，以及损失函数和优化器的选取等, 因此微调脚本往往也包含了整个迁移学
习的过程.
关于微调脚本的说明:
一般情况下，微调脚本应该由不同的任务类型开发者自己编写，但是由于目前研究的
NLP任务类型（分类，提取，生成）以及对应的微调输出结构都是有限的，有些微调方
式已经在很多数据集上被验证是有效的，因此微调脚本也可以使用已经完成的规范脚本.
m
. co
ma
两种迁移方式: the
i
i
直接使用预训练模型，进行相同任务的处理，不需要调整参数或模型结构，这些模型开
ww w.
箱即用。但是这种情况一般只适用于普适任务
员 , 如：fasttest工具包中预训练的词向量模
型。另外，很多预训练模型开发者为了达到开箱即用的效果，将模型结构分各个部分保
程
序
存为不同的预训练模型，提供对应的加载方法来完成特定目标
黑
马 .
更加主流的迁移学习方式是发挥预训练模型特征抽象的能力，然后再通过微调的方式，
通过训练更新小部分参数以此来适应不同的任务。这种迁移方式需要提供小部分的标注
数据来进行监督学习.
关于迁移方式的说明:
直接使用预训练模型的方式, 已经在fasttext的词向量迁移中学习om
. 接下来的迁移学习实践
将主要讲解通过微调的方式进行迁移学习. ma
. c
h ei
it
w.
ww
2.2 NLP 中的标准数据集马程序员
黑
学习目标
了解NLP中GLUE标准数据集合的相关知识.
掌握GLUE标准数据集合的下载方式, 数据样式及其对应的任务类型.
GLUE 数据集合的介绍: . co
m
a
GLUE由纽约大学, 华盛顿大学, Google联合推出, 涵盖不同 e im NLP任务类型, 截止至2020年1
h
月其中包括11个子任务数据集, 成为衡量NLP研究发展的衡量标准
w . it .
ww
员
序
程
数据集合包含以下数据集
GLUE 黑
马
CoLA 数据集
SST-2 数据集
MRPC 数据集
STS-B 数据集
QQP 数据集
MNLI 数据集
m
SNLI 数据集 . co
a
QNLI 数据集 eim
h
it
RTE 数据集 w .
ww
WNLI 数据集员
序
diagnostics数据集(官方未完善
程
马 )
黑
GLUE 数据集合的下载方式:
下载脚本代码:
''' Script for downloading all GLUE data.'''

import os m
import sys . co
a
import shutil eim
h
import argparse
. it
import tempfile w
import urllib.request
ww
import zipfile
员
序
程
马
TASKS = ["CoLA", "SST", "MRPC", "QQP", "STS", "MNLI", "SNLI", "QNLI",
黑
"RTE", "WNLI", "diagnostic"]
TASK2PATH = {"CoLA":'https://firebasestorage.googleapis.com/v0/b/mtl-
sentence-representations.appspot.com/o/data%2FCoLA.zip?
alt=media&token=46d5e637-3411-4188-bc44-5809b5bfb5f4',
"SST":'https://firebasestorage.googleapis.com/v0/b/mtl-
sentence-representations.appspot.com/o/data%2FSST-2.zip?
alt=media&token=aabc5f6b-e466-44a2-b9b4-cf6337f84ac8',
"MRPC":'https://firebasestorage.googleapis.com/v0/b/mtl-
sentence-representations.appspot.com/o/data%2Fmrpc_dev_ids.tsv?
alt=media&token=ec5c0836-31d5-48f4-b431-7480817f1adc',
m
co
"QQP":'https://firebasestorage.googleapis.com/v0/b/mtl-
.
sentence-representations.appspot.com/o/data%2FQQP.zip? a
e
alt=media&token=700c6acf-160d-4d89-81d1-de4191d02cb5', im
h
it
"STS":'https://firebasestorage.googleapis.com/v0/b/mtl-
.
w
sentence-representations.appspot.com/o/data%2FSTS-B.zip?
ww
alt=media&token=bddb94a7-8706-4e0d-a694-1109e12273b5',
员
"MNLI":'https://firebasestorage.googleapis.com/v0/b/mtl-
序
程
sentence-representations.appspot.com/o/data%2FMNLI.zip?
马
alt=media&token=50329ea1-e339-40e2-809c-10c40afff3ce',
黑
"SNLI":'https://firebasestorage.googleapis.com/v0/b/mtl-
sentence-representations.appspot.com/o/data%2FSNLI.zip?
alt=media&token=4afcfbb2-ff0c-4b2d-a09a-dbf07926f4df',
"QNLI": 'https://firebasestorage.googleapis.com/v0/b/mtl-
sentence-representations.appspot.com/o/data%2FQNLIv2.zip?
alt=media&token=6fdcf570-0fc5-4631-8456-9505272d1601',
"RTE":'https://firebasestorage.googleapis.com/v0/b/mtl-
sentence-representations.appspot.com/o/data%2FRTE.zip?
alt=media&token=5efa7e85-a0bb-4f19-8ea2-9e1840f077fb',
"WNLI":'https://firebasestorage.googleapis.com/v0/b/mtl-
sentence-representations.appspot.com/o/data%2FWNLI.zip?
m
co
alt=media&token=068ad0a0-ded7-4bd7-99a5-5e00222e0faf',
.
a
im
"diagnostic":'https://storage.googleapis.com/mtl-sentence-
e
h
representations.appspot.com/tsvsWithoutLabels%2FAX.tsv?
.it
GoogleAccessId=firebase-adminsdk-0khhl@mtl-sentence-
w
ww
representations.iam.gserviceaccount.com&Expires=2498860800&Signature=DuQ2CSPt2
员
序
程
MRPC_TRAIN = 马
黑
'https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.tx
MRPC_TEST =
'https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_test.txt
def download_and_extract(task, data_dir):

print("Downloading and extracting %s..." % task)
data_file = "%s.zip" % task
urllib.request.urlretrieve(TASK2PATH[task], data_file)
with zipfile.ZipFile(data_file) as zip_ref: m
zip_ref.extractall(data_dir) . co
a
os.remove(data_file)
eim
print("\tCompleted!") th i
w w.
def w
format_mrpc(data_dir, path_to_data):
print("Processing MRPC...") 员
序
mrpc_dir = os.path.join(data_dir, "MRPC")
程
马
if not os.path.isdir(mrpc_dir):
黑
os.mkdir(mrpc_dir)
if path_to_data:
mrpc_train_file = os.path.join(path_to_data,
"msr_paraphrase_train.txt")
mrpc_test_file = os.path.join(path_to_data,
"msr_paraphrase_test.txt")
else:
print("Local MRPC data not specified, downloading data from %s" %
MRPC_TRAIN)
mrpc_train_file = os.path.join(mrpc_dir,
"msr_paraphrase_train.txt") m
co
mrpc_test_file = os.path.join(mrpc_dir, "msr_paraphrase_test.txt")
.
a
im
urllib.request.urlretrieve(MRPC_TRAIN, mrpc_train_file)
e
urllib.request.urlretrieve(MRPC_TEST, t h
mrpc_test_file)
i
assert os.path.isfile(mrpc_train_file),w."Train data not found at %s" %
mrpc_train_file ww
员 "Test data not found at %s" %
assert os.path.isfile(mrpc_test_file),
mrpc_test_file 序
程
马
urllib.request.urlretrieve(TASK2PATH["MRPC"], os.path.join(mrpc_dir,
"dev_ids.tsv")) 黑
dev_ids = []
with open(os.path.join(mrpc_dir, "dev_ids.tsv"), encoding="utf8") as
ids_fh:
for row in ids_fh:
dev_ids.append(row.strip().split('\t'))
with open(mrpc_train_file, encoding="utf8") as data_fh, \

open(os.path.join(mrpc_dir, "train.tsv"), 'w', encoding="utf8") as
train_fh, \
open(os.path.join(mrpc_dir, "dev.tsv"), 'w', encoding="utf8") as
m
dev_fh:
. co
a
header = data_fh.readline()
eim
train_fh.write(header) h
dev_fh.write(header) .it
w
for row in data_fh: ww
员 s2 = row.strip().split('\t')
label, id1, id2, s1,
if [id1, id2] in序dev_ids:
程
马
dev_fh.write("%s\t%s\t%s\t%s\t%s\n" % (label, id1, id2, s1,
s2)) 黑
else:
train_fh.write("%s\t%s\t%s\t%s\t%s\n" % (label, id1, id2,
s1, s2))
with open(mrpc_test_file, encoding="utf8") as data_fh, \

open(os.path.join(mrpc_dir, "test.tsv"), 'w', encoding="utf8")
as test_fh:
header = data_fh.readline()
test_fh.write("index\t#1 ID\t#2 ID\t#1 String\t#2 String\n")
for idx, row in enumerate(data_fh): m
co
label, id1, id2, s1, s2 = row.strip().split('\t')
.
a
im % (idx, id1, id2, s1, s2))
test_fh.write("%d\t%s\t%s\t%s\t%s\n"
e
print("\tCompleted!") th i
w w.
def download_diagnostic(data_dir): w
员 diagnostic...")
print("Downloading and extracting
序
if not os.path.isdir(os.path.join(data_dir, "diagnostic")):
程
马
os.mkdir(os.path.join(data_dir, "diagnostic"))
黑
data_file = os.path.join(data_dir, "diagnostic", "diagnostic.tsv")
urllib.request.urlretrieve(TASK2PATH["diagnostic"], data_file)
print("\tCompleted!")
return
def get_tasks(task_names):
task_names = task_names.split(',')
if "all" in task_names:
tasks = TASKS
else:
tasks = [] m
for task_name in task_names: . co
a
assert task_name in TASKS, "Task %s not
e im found!" % task_name
tasks.append(task_name) h
return tasks . it
w
ww
def main(arguments): 员
序
parser = argparse.ArgumentParser()
程
马
parser.add_argument('--data_dir', help='directory to save data to',
type=str, default='glue_data')黑
parser.add_argument('--tasks', help='tasks to download data for as a
comma separated string',
type=str, default='all')
parser.add_argument('--path_to_mrpc', help='path to directory
containing extracted MRPC data, msr_paraphrase_train.txt and
msr_paraphrase_text.txt',
type=str, default='')
args = parser.parse_args(arguments)
if not os.path.isdir(args.data_dir):
os.mkdir(args.data_dir)
m
tasks = get_tasks(args.tasks)
. co
ima
e
for task in tasks:
i th
if task == 'MRPC':
w.
ww
format_mrpc(args.data_dir, args.path_to_mrpc)
员
elif task == 'diagnostic':
序
download_diagnostic(args.data_dir)
程
else: 马
黑
download_and_extract(task, args.data_dir)
if __name__ == '__main__':
sys.exit(main(sys.argv[1:]))
运行脚本下载所有数据集:
m
. co
ma
假设你已经将以上代码到 e i文件中
#
运行这个脚本你将同目录下得到一个 i th
copy download_glue_data.py
文件夹
# python ,
w.
glue
python download_glue_data.py ww
员
序
程
马
黑
输出效果:
Downloading and extracting CoLA...

Completed!
Downloading and extracting SST...
Completed!
Processing MRPC...
Local MRPC data not specified, downloading data from
om
https://dl.fbaipublicfiles.com/senteval/senteval_data/msr_paraphrase_train.txt
.c
ma
Completed!
hei
Downloading and extracting QQP...
. it
w
Completed!
Downloading and extracting STS...
ww
员
Completed! 序
Downloading and 程
extracting MNLI...
马
Completed! 黑
Downloading and extracting SNLI...
Completed!
Downloading and extracting QNLI...
Completed!
Downloading and extracting RTE...
Completed!
Downloading and extracting WNLI...
Completed!
Downloading and extracting diagnostic...
Completed!
m
. co
i ma
he
GLUE 数据集合中子数据集的样式及其任务类型
ww
w.
i t
数据集文件样式
CoLA 序
员
程
马
- CoLA/ 黑
- dev.tsv
- original/
- test.tsv
- train.tsv
文件样式说明:
m
在使用中常用到的文件是train.tsv, dev.tsv, test.tsv,a.分别代表训练集
co , 验证集和测试集.
其中train.tsv与dev.tsv数据样式相同, 都是带有标签的数据
he
i m , 其中test.tsv是不带有标签
的数据. w.
i t
ww
员
序
程
马
黑
train.tsv数据样式:
...
gj04 1 She coughed herself awake as the leaf landed on her nose.
gj04 1 The worm wriggled onto the carpet.
gj04 1 The chocolate melted onto the carpet.
gj04 0 * The ball wriggled itself loose.
gj04 1 Bill wriggled himself loose.
bc01 1 m
The sinking of the ship to collect the insurance
o was very
devious.
a .c
bc01 1 The ship's sinking was very devious.im
bc01 0 * The ship's sinking to collect t he insurance was very
the
i
devious.
w w.
bc01 1 The testing of such drugsw on oneself is too risky.
bc01 0 * This drug's testing on员oneself is too risky.
序
... 程
马
黑
train.tsv数据样式说明:
中的数据内容共分为4列, 第一列数据, 如gj04, bc01等代表每条文本数据的来
train.tsv
源即出版物代号; 第二列数据, 0或1, 代表每条文本数据的语法是否正确, 0代表不正确,
1代表正确; 第三列数据, '', 是作者最初的正负样本标记, 与第二列意义相同, ''表示不正
确; 第四列即是被标注的语法使用是否正确的文本句子.
m
. co
i ma
the
test.tsv 数据样式: w.
i
ww
员
序
index sentence 程
马
0 黑the house.
Bill whistled past
1 The car honked its way down the road.
2 Bill pushed Harry off the sofa.
3 the kittens yawned awake and played.
4 I demand that the more John eats, the more he pay.
5 If John eats more, keep your mouth shut tighter, OK?
6 His expectations are always lower than mine are.
7 The sooner you call, the more carefully I will word the letter.
8 The more timid he feels, the more people he interviews without asking
questions of.
9 Once Janet left, Fred became a lot crazier.
... om
a .c
eim
th
i
w w.
w
数据样式说明:
test.tsv 员
序
test.tsv中的数据内容共分为
马 2列, 第一列数据代表每条文本数据的索引; 第二列数据代
程
表用于测试的句子. 黑
CoLA 数据集的任务类型:
二分类任务
评估指标为: MCC(马修斯相关系数, 在正负样本分布十分不均衡的情况下使用的二分
类评估指标) .c
om
a
e im
h
. it
www
SST-2
序
员
程
马
- SST-2/ 黑
- dev.tsv
- original/
- test.tsv
- train.tsv
文件样式说明:
在使用中常用到的文件是train.tsv, dev.tsv, test.tsv, 分别代表训练集, 验证集和测试集.
其中train.tsv与dev.tsv数据样式相同, 都是带有标签的数据, 其中test.tsv是不带有标签
的数据. .c
om
ma
h ei
it
w.
ww
员
train.tsv 数据样式: 程
序
马
黑
sentence label
hide new secretions from the parental units 0
contains no wit , only labored gags 0
that loves its characters and communicates something rather beautiful about
human nature 1
remains utterly satisfied to remain the same throughout 0
on the worst revenge-of-the-nerds clichés the filmmakers could dredge up
0
that 's far too tragic to merit such superficial treatment 0
m
demonstrates that the director of such hollywood blockbusters as patriot
games can still turn out a small , personal film .with co an emotional wallop .
a
1
eim
h
of saucy 1
. it
a depressed fifteen-year-old 's suicidalw poetry 0
... ww
员
序
程
马
黑
数据样式说明:
train.tsv
train.tsv中的数据内容共分为2列, 第一列数据代表具有感情色彩的评论文本; 第二列数

据, 0或1, 代表每条文本数据是积极或者消极的评论, 0代表消极, 1代表积极.
test.tsv数据样式: com
a.
eim
ith
.
index
0
sentence
uneasy mishmash of styles and genres w.
ww
1 员
this film 's relationship to actual
序 istension is the same as what
程
christmas-tree flocking in a spray can to actual snow : a poor -- if
durable -- imitation . 马
2
黑
by the end of no such thing the audience , like beatrice , has a
watchful affection for the monster .
3 director rob marshall went out gunning to make a great one .
4 lathan and diggs have considerable personal charm , and their screen
rapport makes the old story seem new .
5 a well-made and often lovely depiction of the mysteries of friendship .
6 none of this violates the letter of behan 's book , but missing is its
spirit , its ribald , full-throated humor .
7 although it bangs a very cliched drum at times , this crowd-pleaser 's
fresh dialogue , energetic music , and good-natured spunk are often
infectious .
8 it is not a mass-market entertainment but an uncompromising attempt by
m
one artist to think about another .
. co
a
9
im .
this is junk food cinema at its greasiest
e
... h t
w .i
ww
员
序
程
数据样式说明黑: *马test.tsv
test.tsv 中的数据内容共分为2列, 第一列数据代表每条文本数据的
索引; 第二列数据代表用于测试的句子.
SST-2数据集的任务类型:
二分类任务
评估指标为: ACC m
. co
ma
h ei
i t
数据集文件样式 w w.
MRPC w
员
序
- MRPC/ 程
马
- dev.tsv 黑
- test.tsv
- train.tsv
- dev_ids.tsv
- msr_paraphrase_test.txt
- msr_paraphrase_train.txt
文件样式说明: m
. co
在使用中常用到的文件是train.tsv, dev.tsv, test.tsv, i分别代表训练集 , 验证集和测试集.
ma
其中train.tsv与dev.tsv数据样式相同, 都是带有标签的数据th
e , 其中test.tsv是不带有标签
的数据. ww
. i
w
员
序
程
马
黑
train.tsv 数据样式:
Quality #1 ID #2 ID #1 String #2 String
1 702876 702977 Amrozi accused his brother , whom he called " the
witness " , of deliberately distorting his evidence . Referring to him as
only " the witness " , Amrozi accused his brother of deliberately
distorting his evidence .
0 2108705 2108831 Yucaipa owned Dominick 's before selling the chain to
m
Safeway in 1998 for $ 2.5 billion .
co
Yucaipa bought Dominick 's in 1995
for $ 693 million and sold it to Safeway fora.$ 1.8 billion in 1998 .
1
m
1330381 1330521 They had published aneiadvertisement on the Internet on
June 10 , offering the cargo for salei, th
w . heonadded . On June 10 , the ship
explosives for sale .

ww
's owners had published an advertisement the Internet , offering the
员
0 序 GMT , Tab shares were up 19 cents , or 4.4
3344667 3344648 Around 0335
程
% , at A $ 4.56 , having earlier set a record high of A $ 4.57 . Tab shares
马% , to set
黑
jumped 20 cents , or 4.6 a record closing high at A $ 4.57 .
1 1236820 1236712 The stock rose $ 2.11 , or about 11 percent , to close
Friday at $ 21.51 on the New York Stock Exchange . PG & E Corp. shares
jumped $ 1.63 or 8 percent to $ 21.03 on the New York Stock Exchange on
Friday .
1 738533 737951 Revenue in the first quarter of the year dropped 15
percent from the same period a year earlier . With the scandal hanging
over Stewart 's company , revenue the first quarter of the year dropped 15
percent from the same period a year earlier .
0 264589 264502 The Nasdaq had a weekly gain of 17.27 , or 1.2 percent
, closing at 1,520.15 on Friday . The tech-laced Nasdaq Composite .IXIC
m
rallied 30.46 points , or 2.04 percent , to 1,520.15
1
co .
579975 579810 The DVD-CCA then appealed toa.the state Supreme Court .
m
The DVD CCA appealed that decision to the U.S.
h ei Supreme Court .
... it
w.
ww
员
序
程
马
数据样式说明: 黑
train.tsv
train.tsv中的数据内容共分为5列, 第一列数据, 0或1, 代表每对句子是否具有相同的含

义, 0代表含义不相同, 1代表含义相同. 第二列和第三列分别代表每对句子的id, 第四列
和第五列分别具有相同/不同含义的句子对.
test.tsv数据样式: com
ma.
h ei
t
index #1 ID #2 ID #1 String #2 String
w .i
0 ww officer , Mike Butcher , and
1089874 1089925 PCCW 's chief operating
Alex Arena , the chief financial officer
员 , will report directly to Mr So .
Current Chief Operating Officer Mike序Butcher and Group Chief Financial
to 程
Officer Alex Arena will report 马 So .
1 3019446 3019327 The world黑's two largest automakers said their U.S.
sales declined more than predicted last month as a late summer sales frenzy
caused more of an industry backlash than expected . Domestic sales at both
GM and No. 2 Ford Motor Co. declined more than predicted as a late summer
sales frenzy prompted a larger-than-expected industry backlash .
2 1945605 1945824 According to the federal Centers for Disease Control
and Prevention ( news - web sites ) , there were 19 reported cases of
measles in the United States in 2002 . The Centers for Disease Control
and Prevention said there were 19 reported cases of measles in the United
States in 2002 .
3 1430402 1430329 A tropical storm rapidly developed in the Gulf of
Mexico Sunday and was expected to hit somewhere along the Texas or
om rapidly developed in
Louisiana coasts by Monday night . A tropical cstorm
.
the Gulf of Mexico on Sunday and could havemahurricane-force winds when it
ei Monday night .
hits land somewhere along the Louisiana hcoast
t
i detail the costs of the replacement
4 3354381 3354396 The company didn.'t
w w
and repairs . w
But company officials expect the costs of the replacement
work to run into the millions 员of dollars .
5 序 companies would also assign their possible
1390995 1391183 The settling
程
马
claims against the underwriters to the investor plaintiffs , he added .
Under the agreement , 黑the settling companies will also assign their
potential claims against the underwriters to the investors , he added .
6 2201401 2201285 Air Commodore Quaife said the Hornets remained on
three-minute alert throughout the operation . Air Commodore John Quaife
said the security operation was unprecedented .
7 2453843 2453998 A Washington County man may have the countys first
human case of West Nile virus , the health department said Friday . The
countys first and only human case of West Nile this year was confirmed by
health officials on Sept . 8 .
...
m
. co
ma
h ei
t
数据样式说明: * test.tsv中的数据内容共分为
test.tsv
w w.
i 5列, 第一列数据代表每条文本数据的
索引; 其余列的含义与train.tsv中相同员. w
序
程
马
黑
MRPC 数据集的任务类型:
句子对二分类任务
评估指标为: ACC和F1
STS-B c om
a.
eim
- STS-B/ ith
w .
- dev.tsv
- test.tsv
ww
员
- train.tsv 序
- LICENSE.txt 程
马
- readme.txt 黑
- original/
文件样式说明:
的数据.
m
. co
ima
the
i
train.tsv数据样式: ww
w.
员
序
程
index genre filename马 year old_index source1 source2 sentence1
sentence2 score 黑
0 main-captions MSRvid 2012test 0001 none none A plane is
taking off. An air plane is taking off. 5.000
1 main-captions MSRvid 2012test 0004 none none A man is
playing a large flute. A man is playing a flute. 3.800
spreading shreded cheese on a pizza. A man is spreading shredded cheese
on an uncooked pizza. 3.800
3 main-captions MSRvid 2012test 0006 none none Three men
are playing chess.Two men are playing chess. 2.600
4 main-captions MSRvid 2012test 0009 nonem none A man is
playing the cello.A man seated is playing the cello. co 4.250
.
5 main-captions MSRvid 2012test 0011 imanone none Some men
are fighting. Two men are fighting. 4.250 h e
6 main-captions MSRvid 2012test it
.0012 none none A man is
w
smoking. A man is skating. 0.500 ww
7 main-captions MSRvid 2012test员 0013 none none The man is
playing the piano. The man is 序 playing the guitar. 1.600
程
8 main-captions MSRvid 马 2012test 0014 none none A man is
黑
playing on a guitar and singing. A woman is playing an acoustic guitar
and singing. 2.200
9 main-captions MSRvid 2012test 0016 none none A person is
throwing a cat on to the ceiling. A person throws a cat on the ceiling.
5.000
...
数据样式说明:
train.tsv
co
m
.
train.tsv中的数据内容共分为10列, 第一列数据是数据索引 im ; 第二列代表每对句子的来
a
e
源, 如main-captions表示来自字幕; 第三列代表来源的具体保存文件名
. it
h , 第四列代表出
现时间(年); 第五列代表原始数据的索引; w第六列和第七列分别代表句子对原始来源
w w ;第
八列和第九列代表相似程度不同的句子对员 ; 第十列代表句子对的相似程度由低到高, 值
域范围是[0, 5]. 马
程
序
黑
test.tsv数据样式:
index genre filename year old_index source1 source2 sentence1

sentence2
0 main-captions MSRvid 2012test 0024 none none A girl is
styling her hair. A girl is brushing her hair. om
1 main-captions MSRvid 2012test 0033a.
c none none A group of
men play soccer on the beach. A group eof
m
i boys are playing soccer on the
h
beach.
.it
w
2 main-captions
ww A 0045
MSRvid 2012test
is measuring another woman's ankle.
none none One woman
woman measures another woman's
员
ankle. 序
3 main-captions MSRvid程 2012test
马man is slicing 0063 none none A man is
黑
cutting up a cucumber. A a cucumber.
playing a harp. A man is playing a keyboard.
5 main-captions MSRvid 2012test 0074 none none A woman is
cutting onions. A woman is cutting tofu.
riding an electric bicycle. A man is riding a bicycle.
playing the drums. A man is playing the guitar.
playing guitar. A lady is playing the guitar.
m none
9 main-captions MSRvid 2012test 0095
playing a guitar. A man is playing a trumpet. a.
co
none A man is
10 main-captions MSRvid 2012test

m
0096 ei none none A man is
t h
.i
playing a guitar. A man is playing a trumpet.
... ww
w
员
序
程
马
黑
数据样式说明:
test.tsv
test.tsv中的数据内容共分为9列, 含义与train.tsv前9列相同.
STS-B数据集的任务类型:
句子对多分类任务/句子对回归任务 com
评估指标为: Pearson-Spearman Corr a.
eim
ith
w .
ww
员
序
QQP
马
程
黑
- QQP/
- dev.tsv
- original/
- test.tsv
- train.tsv
co
在使用中常用到的文件是train.tsv, dev.tsv,eitest.tsv,
ma
. 分别代表训练集, 验证集和测试集.
其中train.tsv与dev.tsv数据样式相同.i, 都是带有标签的数据
th , 其中test.tsv是不带有标签
的数据. ww
w
员
序
程
马
黑
id qid1 qid2 question1 question2 is_duplicate

133273 213221 213222 How is the life of a math student? Could you
describe your own experiences?Which level of prepration is enough for the
exam jlpt5? 0
402555 536040 536041 How do I control my horny emotions? How do you
control your horniness? 1 m
360472 364011 490273 What causes stool color to . cochange to yellow?
a 0
im
What can cause stool to come out as little balls?
e
150662 155721 7256 h
What can one do after
MBBS ? 1 . it MBBS? What do i do after my
w
183004 279958 279959 Where can I find ww a power outlet for my laptop at
Melbourne Airport? Would a second 员airport in Sydney, Australia be needed if
序 between Melbourne and Sydney? 0
a high-speed rail link was created
程
119056 193387 193388 How 马 not to feel guilty since I am Muslim and I'm
黑
conscious we won't have sex together? I don't beleive I am bulimic, but I
force throw up atleast once a day after I eat something and feel guilty.
Should I tell somebody, and if so who? 0
356863 422862 96457 How is air traffic controlled? How do you become
an air traffic controller?0
106969 147570 787 What is the best self help book you have read? Why? How
did it change your life? What are the top self help books I should read?
1
...
com
a.
eim
数据样式说明:
train.tsv
. ith
w
train.tsv中的数据内容共分为6列, 第一列代表文本数据索引
ww ; 第二列和第三列数据分别
代表问题1和问题2的id; 第四列和第五列代表需要进行
序
员 '是否重复'判定的句子对; 第六列
代表上述问题是/不是重复性问题的标签马
程 , 0代表不重复, 1代表重复.
黑
test.tsv数据样式:
id question1 question2
0 Would the idea of Trump and Putin in bed together scare you, given the
geopolitical implications? Do you think that if Donald Trump were elected
m
President, he would be able to restore relations
. co with Putin and Russia as
i ma
he said he could, based on the rocky relationship Putin had with Obama and
Bush? e
1 i th
What are the top ten Consumer-to-Consumer
w .E-commerceE-commerce online? What
2
ww
are the top ten Consumer-to-Business online?
Why don't people simply 'Google' instead of asking questions on Quora?
员 instead of just searching google?
序
Why do people ask Quora questions
3 程
Is it safe to invest in
马 socialthen trade biz? Is social trade geniune?
4 黑
If the universe is expanding does matter also expand? If universe
and space is expanding? Does that mean anything that occupies space is also
expanding?
5 What is the plural of hypothesis? What is the plural of thesis?
6 What is the application form you need for launching a company? What is
the application form you need for launching a company in Austria?
7 What is Big Theta? When should I use Big Theta as opposed to big O? Is
O(Log n) close to O(n) or O(1)?
8 What are the health implications of accidentally eating a small
quantity of aluminium foil?What are the implications of not eating
vegetables?
... om
a.c
eim
t h
i
w w.
w
数据样式说明:
test.tsv 员
序
test.tsv中的数据内容共分为
马 3列, 第一列数据代表每条文本数据的索引; 第二列和第三
程
列数据代表用于测试的问题句子对
黑 .
QQP 数据集的任务类型:
句子对二分类任务
评估指标为: ACC/F1
com
a.
e im
ith
.
(MNLI/SNLI) 数据集文件样式 ww
w
员
序
- (MNLI/SNLI)/ 程
- dev_matched.tsv 马
- dev_mismatched.tsv
黑
- original/
- test_matched.tsv
- test_mismatched.tsv
- train.tsv
co
在使用中常用到的文件是train.tsv, dev_matched.tsv, im
a. dev_mismatched.tsv,
it
h
test_matched.tsv, test_mismatched.tsv分别代表训练集
e , 与训练集一同采集的验证集,
与训练集不是一同采集验证集, w与训练集一同采集的测试集 ww
. , 与训练集不是一同采集测
试集. 其中train.tsv与dev_matched.tsv
员和dev_mismatched.tsv数据样式相同, 都是带
有标签的数据, 其中test_matched.tsv
程
序与test_mismatched.tsv数据样式相同, 都是不带
有标签的数据. 黑马
index promptID pairID genre sentence1_binary_parse

sentence2_binary_parse sentence1_parse sentence2_parse sentence1
sentence2 label1 gold_label com
.
0 31193 31193n government ( ( Conceptually i ma ( cream skimming ) ) ( (
has ( ( ( two ( basic dimensions ) ) - ) (t( heproduct and ) geography ) ) )
i
. ) ) ( ( ( Product and ) geography ) w(. ( are ( what ( make ( cream (
skimming work ) ) ) ) ) . ) ) (ROOTww(S (NP (JJ Conceptually) (NN cream)
员 (CD two) (JJ basic) (NNS dimensions))
(NN skimming)) (VP (VBZ has) (NP (NP
(: -) (NP (NN product) (CC and) 序(NN geography)))) (. .))) (ROOT (S (NP (NN
程
马
Product) (CC and) (NN geography)) (VP (VBP are) (SBAR (WHNP (WP what)) (S
黑cream)) (VP (VBG skimming) (NP (NN work))))))))
(VP (VBP make) (NP (NP (NN
(. .))) Conceptually cream skimming has two basic dimensions - product
and geography. Product and geography are what make cream skimming work.
neutral neutral
1 101457 101457e telephone ( you ( ( know ( during ( ( ( the season )
and ) ( i guess ) ) ) ) ( at ( at ( ( your level ) ( uh ( you ( ( ( lose
them ) ( to ( the ( next level ) ) ) ) ( if ( ( if ( they ( decide ( to (
recall ( the ( the ( parent team ) ) ) ) ) ) ) ) ( ( the Braves ) ( decide
( to ( call ( to ( ( recall ( a guy ) ) ( from ( ( triple A ) ( ( ( then (
( a ( double ( A guy ) ) ) ( ( goes up ) ( to ( replace him ) ) ) ) ) and )
( ( a ( single ( A guy ) ) ) ( ( goes up ) ( to ( replace m him ) ) ) ) ) ) )
) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ) ( You ( ( ( ( lose ( .the co things ) ) ( to (
the ( following level ) ) ) ) ( if ( ( the peoplei) marecall ) ) ) . ) )
(ROOT (S (NP (PRP you)) (VP (VBP know) (PP (INth
e
during) (NP (NP (DT the) (NN
season)) (CC and) (NP (FW i) (FW guess)))) w(PP.i (IN at) (IN at) (NP (NP
(PRP$ your) (NN level)) (SBAR (S (INTJ (UH ww uh)) (NP (PRP you)) (VP (VBP
lose) (NP (PRP them)) (PP (TO to) (NP 员 (DT the) (JJ next) (NN level))) (SBAR
(IN if) (S (SBAR (IN if) (S (NP (PRP
序they)) (VP (VBP decide) (S (VP (TO to)
程
马the) (NN parent) (NN team)))))))) (NP (DT
(VP (VB recall) (NP (DT the) (DT
黑
the) (NNPS Braves)) (VP (VBP decide) (S (VP (TO to) (VP (VB call) (S (VP
(TO to) (VP (VB recall) (NP (DT a) (NN guy)) (PP (IN from) (NP (NP (RB
triple) (DT A)) (SBAR (S (S (ADVP (RB then)) (NP (DT a) (JJ double) (NNP A)
(NN guy)) (VP (VBZ goes) (PRT (RP up)) (S (VP (TO to) (VP (VB replace) (NP
(PRP him))))))) (CC and) (S (NP (DT a) (JJ single) (NNP A) (NN guy)) (VP
(VBZ goes) (PRT (RP up)) (S (VP (TO to) (VP (VB replace) (NP (PRP
him)))))))))))))))))))))))))))) (ROOT (S (NP (PRP You)) (VP (VBP lose) (NP
(DT the) (NNS things)) (PP (TO to) (NP (DT the) (JJ following) (NN level)))
(SBAR (IN if) (S (NP (DT the) (NNS people)) (VP (VBP recall))))) (. .)))
you know during the season and i guess at at your level uh you lose them to
the next level if if they decide to recall the the parent team the Braves
decide to call to recall a guy from triple A then a double A guy goes up to
m
replace him and a single A guy goes up to replace
. co him You lose the things
a entailment entailment
to the following level if the people recall.
e im
2 134793 134793e fiction ( ( One ( of h ( our number ) ) ) ( ( will ( ( (
carry out ) ( your instructions ) ) . it
minutely ) ) . ) ) ( ( ( A member ) (
w
ww
of ( my team ) ) ) ( ( will ( ( execute ( your orders ) ) ( with ( immense
precision ) ) ) ) . ) ) 员 (S (NP (NP (CD One)) (PP (IN of) (NP (PRP$
(ROOT
our) (NN number)))) (VP (MD 序will) (VP (VB carry) (PRT (RP out)) (NP (PRP$
程
马 (ADVP (RB minutely)))) (. .))) (ROOT (S (NP (NP
your) (NNS instructions))
黑 (IN of) (NP (PRP$ my) (NN team)))) (VP (MD will)
(DT A) (NN member)) (PP
(VP (VB execute) (NP (PRP$ your) (NNS orders)) (PP (IN with) (NP (JJ
immense) (NN precision))))) (. .))) One of our number will carry out your
instructions minutely. A member of my team will execute your orders with
immense precision. entailment entailment
3 37397 37397e fiction ( ( How ( ( ( do you ) know ) ? ) ) ( ( All
this ) ( ( ( is ( their information ) ) again ) . ) ) ) ( ( This
information ) ( ( belongs ( to them ) ) . ) ) (ROOT (S (SBARQ (WHADVP (WRB
How)) (SQ (VBP do) (NP (PRP you)) (VP (VB know))) (. ?)) (NP (PDT All) (DT
this)) (VP (VBZ is) (NP (PRP$ their) (NN information)) (ADVP (RB again)))
(. .))) (ROOT (S (NP (DT This) (NN information)) (VP (VBZ belongs) (PP
(TO to) (NP (PRP them)))) (. .))) How do you know? com All this is their
.
information again. This information belongs ito ma them. entailment
e
entailment
i th
... w.
ww
员
序
程
马
数据样式说明: 黑
train.tsv
train.tsv中的数据内容共分为12列, 第一列代表文本数据索引; 第二列和第三列数据分

别代表句子对的不同类型id; 第四列代表句子对的来源; 第五列和第六列代表具有句法
结构分析的句子对表示; 第七列和第八列代表具有句法结构和词性标注的句子对表示,
第九列和第十列代表原始的句子对, 第十一和第十二列代表不同标准的标注方法产生
的标签, 在这里，他们始终相同, 一共有三种类型的标签, neutral代表两个句子既不矛
盾也不蕴含, entailment代表两个句子具有蕴含关系, contradiction代表两个句子观点
矛盾. om
.c
ma
hei
. it
w
ww
test_matched.tsv数据样式: 序
员
程
马
黑
index promptID pairID genre sentence1_binary_parse
sentence2_binary_parse sentence1_parse sentence2_parse sentence1
sentence2
0 31493 31493 travel ( ( ( ( ( ( ( ( Hierbas , ) ( ans seco ) ) , )
( ans dulce ) ) , ) and ) frigola ) ( ( ( are just ) ( ( a ( few names ) )
( worth ( ( keeping ( a look-out ) ) for ) ) ) ) . ) ) ( Hierbas ( ( is
( ( a name ) ( worth ( ( looking out ) for ) ) ) ) . ) ) (ROOT (S (NP (NP
(NNS Hierbas)) (, ,) (NP (NN ans) (NN seco)) (, ,) (NP (NN ans) (NN dulce))
(, ,) (CC and) (NP (NN frigola))) (VP (VBP are) (ADVP (RB just)) (NP (NP
(DT a) (JJ few) (NNS names)) (PP (JJ worth) (S (VP (VBG keeping) (NP (DT a)
(NN look-out)) (PP (IN for))))))) (. .))) (ROOT (S (NP (NNS Hierbas)) (VP
m
(VBZ is) (NP (NP (DT a) (NN name)) (PP (JJ worth)
. co (S (VP (VBG looking) (PRT
a
(RP out)) (PP (IN for))))))) (. .)))
im
Hierbas,
e
ans seco, ans dulce, and
frigola are just a few names worth keepingh a look-out for. Hierbas is a
name worth looking out for. .it
w
1 92164 92164 government (ww ( ( The extent ) ( of ( the ( behavioral
effects ) ) ) ) ( ( would ( ( 员depend ( in ( part ( on ( ( the structure ) (
of ( ( ( the ( individual ( 序account program ) ) ) and ) ( any limits ) ) )
程
马 ( the funds ) ) ) ) ) . ) )
) ) ) ) ) ( on ( accessing ( ( Many people )
( ( would ( be ( very 黑
( unhappy ( to ( ( loose control ) ( over ( their (
own money ) ) ) ) ) ) ) ) ) . ) ) (ROOT (S (NP (NP (DT The) (NN extent))
(PP (IN of) (NP (DT the) (JJ behavioral) (NNS effects)))) (VP (MD would)
(VP (VB depend) (PP (IN in) (NP (NP (NN part)) (PP (IN on) (NP (NP (DT the)
(NN structure)) (PP (IN of) (NP (NP (DT the) (JJ individual) (NN account)
(NN program)) (CC and) (NP (DT any) (NNS limits)))))))) (PP (IN on) (S (VP
(VBG accessing) (NP (DT the) (NNS funds))))))) (. .))) (ROOT (S (NP (JJ
Many) (NNS people)) (VP (MD would) (VP (VB be) (ADJP (RB very) (JJ unhappy)
(PP (TO to) (NP (NP (JJ loose) (NN control)) (PP (IN over) (NP (PRP$ their)
(JJ own) (NN money)))))))) (. .))) The extent of the behavioral effects
would depend in part on the structure of the individual account program and
any limits on accessing the funds. Many people would com be very unhappy to
.
loose control over their own money. i ma
e
2 9662 9662 government ( ( ( Timely
i th access ) ( to information ) )
( ( is ( in ( ( the ( best interests ) w). ( of ( ( ( both GAO ) and ) ( the
agencies ) ) ) ) ) ) . ) ) ( It ( w(w ( is ( in ( ( everyone 's ) ( best
员 ) ( to ( information ( in ( a (
interest ) ) ) ) ( to ( ( have access
timely manner ) ) ) ) ) ) ) ) . 序) ) (ROOT (S (NP (NP (JJ Timely) (NN
程
access)) (PP (TO to) (NP (NN马information)))) (VP (VBZ is) (PP (IN in) (NP
黑 interests)) (PP (IN of) (NP (NP (DT both) (NNP
(NP (DT the) (JJS best) (NNS
GAO)) (CC and) (NP (DT the) (NNS agencies))))))) (. .))) (ROOT (S (NP (PRP
It)) (VP (VBZ is) (PP (IN in) (NP (NP (NN everyone) (POS 's)) (JJS best)
(NN interest))) (S (VP (TO to) (VP (VB have) (NP (NN access)) (PP (TO to)
(NP (NP (NN information)) (PP (IN in) (NP (DT a) (JJ timely) (NN
manner))))))))) (. .))) Timely access to information is in the best
interests of both GAO and the agencies. It is in everyone's best interest
to have access to information in a timely manner.
3 5991 5991 travel ( ( Based ( in ( ( the ( Auvergnat ( spa town )
) ) ( of Vichy ) ) ) ) ( , ( ( the ( French government ) ) ( often ( ( ( (
proved ( more zealous ) ) ( than ( its masters ) ) ) ( in ( ( ( suppressing
( civil liberties ) ) and ) ( ( drawing up ) ( anti-Jewish c om legislation ) )
.
) ) ) . ) ) ) ) ) ( ( The ( French government ) )i( ma( passed ( ( anti-
e
Jewish laws ) ( aimed ( at ( helping ( the Nazi
i th) ) ) ) ) ) . ) ) (ROOT
(S (PP (VBN Based) (PP (IN in) (NP (NP (DT wthe) . (NNP Auvergnat) (NN spa)
(NN town)) (PP (IN of) (NP (NNP Vichy)))))) ww (, ,) (NP (DT the) (JJ French)
(NN government)) (ADVP (RB often)) (VP员(VBD proved) (NP (JJR more) (NNS
zealous)) (PP (IN than) (NP (PRP$ 程序 (NNS masters))) (PP (IN in) (S (VP
its)
马
(VP (VBG suppressing) (NP (JJ civil) (NNS liberties))) (CC and) (VP (VBG
drawing) (PRT (RP up)) (NP (JJ黑anti-Jewish) (NN legislation))))))) (. .)))
(ROOT (S (NP (DT The) (JJ French) (NN government)) (VP (VBD passed) (NP (NP
(JJ anti-Jewish) (NNS laws)) (VP (VBN aimed) (PP (IN at) (S (VP (VBG
helping) (NP (DT the) (JJ Nazi)))))))) (. .))) Based in the Auvergnat spa
town of Vichy, the French government often proved more zealous than its
masters in suppressing civil liberties and drawing up anti-Jewish
legislation. The French government passed anti-Jewish laws aimed at
helping the Nazi.
...
m
. co
数据样式说明:
test_matched.tsv i ma
he
test_matched.tsv中的数据内容共分为.i 10列, 与train.tsv的前10列含义相同.
t
ww w
员
序
程
马
黑
数据集的任务类型:
(MNLI/SNLI)
句子对多分类任务
评估指标为: ACC
(QNLI/RTE/WNLI)
m
. co
a
* QNLI, RTE, WNLI 三个数据集的样式基本相同. h ei
m
it
w w.
- (QNLI/RTE/WNLI)/ w
- dev.tsv 员
序
- test.tsv 程
- train.tsv 马
黑
文件样式说明:
的数据.
com
a.
e im
ith
w.
QNLI 中的train.tsv数据样式: ww
员
序
程
index question
马
sentence 黑 label
0 When did the third Digimon series begin? Unlike the two seasons
before it and most of the seasons that followed, Digimon Tamers takes a
darker and more realistic approach to its story featuring Digimon who do
not reincarnate after their deaths and more complex character development
in the original Japanese. not_entailment
1 Which missile batteries often have individual launchers several
kilometres from one another? When MANPADS is operated by specialists,
batteries may have several dozen teams deploying separately in small
sections; self-propelled air defence guns may deploy in pairs.
not_entailment
2 What two things does Popper argue Tarski's theory involves in an
m
evaluation of truth? He bases this interpretation
. co on the fact that
examples such as the one described above referm a to two things: assertions
i
and the facts to which they refer. he
entailment
t
3 What is the name of the village . i
w w9 miles north of Calafat where the
w
Ottoman forces attacked the Russians? On 31 December 1853, the Ottoman
员 the Russian force at Chetatea or Cetate, a
forces at Calafat moved against
序 of Calafat, and engaged them on 6 January
small village nine miles north
程
1854. entailment 马
4 黑is located in London?
What famous palace London contains four World
Heritage Sites: the Tower of London; Kew Gardens; the site comprising the
Palace of Westminster, Westminster Abbey, and St Margaret's Church; and the
historic settlement of Greenwich (in which the Royal Observatory, Greenwich
marks the Prime Meridian, 0° longitude, and GMT). not_entailment
5 When is the term 'German dialects' used in regard to the German
language? When talking about the German language, the term German
dialects is only used for the traditional regional varieties. entailment
6 What was the name of the island the English traded to the Dutch in
return for New Amsterdam? At the end of the Second Anglo-Dutch War, the
English gained New Amsterdam (New York) in North America in exchange for
Dutch control of Run, an Indonesian island. entailment com
.
7 How were the Portuguese expelled from Myanmar?i ma From the 1720s onward,
e
the kingdom was beset with repeated Meithei
i thraids into Upper Myanmar and a
w.
nagging rebellion in Lan Na. not_entailment
8 ww
What does the word 'customer' properly apply to? The bill also
员
required rotation of principal maintenance inspectors and stipulated that
序 to the flying public, not those
the word "customer" properly applies
程
马 entailment
entities regulated by the FAA.
... 黑
RTE中的train.tsv数据样式:
index sentence1 sentence2 label

0 No Weapons of Mass Destruction Found in Iraq Yet. Weapons of Mass
Destruction Found in Iraq. not_entailment om
c
1 A place of sorrow, after Pope John Paul II died, m a.became a place of
i
celebration, as Roman Catholic faithful gathered
t hein downtown Chicago to
i
mark the installation of new Pope Benedict XVI.Pope Benedict XVI is the new
w w.
leader of the Roman Catholic Church. entailment
w
2 Herceptin was already approved to 员
treat the sickest breast cancer
序 it will discuss with federal
patients, and the company said, Monday,
程
regulators the possibility of prescribing the drug for more breast cancer
马
黑 to treat breast cancer. entailment
patients. Herceptin can be used
3 Judie Vivian, chief executive at ProMedica, a medical service company
that helps sustain the 2-year-old Vietnam Heart Institute in Ho Chi Minh
City (formerly Saigon), said that so far about 1,500 children have received
treatment. The previous name of Ho Chi Minh City was Saigon.entailment
4 A man is due in court later charged with the murder 26 years ago of a
teenager whose case was the first to be featured on BBC One's Crimewatch.
Colette Aram, 16, was walking to her boyfriend's house in Keyworth,
Nottinghamshire, on 30 October 1983 when she disappeared. Her body was
later found in a field close to her home. Paul Stewart Hutchinson, 50, has
been charged with murder and is due before Nottingham magistrates later.
Paul Stewart Hutchinson is accused of having stabbed a girl.
m
not_entailment
. co
a
5 Britain said, Friday, that it has barred
eim cleric, Omar Bakri, from
h
returning to the country from Lebanon, where he was released by police
.it was briefly detained, but was
after being detained for 24 hours. Bakri
w
released. entailment ww
6 Nearly 4 million children 员
who have at least one parent who entered the
U.S. illegally were born in 序
the United States and are U.S. citizens as a
程
result, according to the马study conducted by the Pew Hispanic Center. That's
黑 the estimated 5.5 million children of illegal
about three quarters of
immigrants inside the United States, according to the study. About 1.8
million children of undocumented immigrants live in poverty, the study
found. Three quarters of U.S. illegal immigrants have children.
not_entailment
7 Like the United States, U.N. officials are also dismayed that Aristide
killed a conference called by Prime Minister Robert Malval in Port-au-
Prince in hopes of bringing all the feuding parties together. Aristide had
Prime Minister Robert Malval murdered in Port-au-Prince. not_entailment
8 WASHINGTON -- A newly declassified narrative of the Bush
administration's advice to the CIA on harsh interrogations shows that the
small group of Justice Department lawyers who wrote commemos authorizing
.
i ma
controversial interrogation techniques were operating not on their own but
h e
with direction from top administration officials, including then-Vice
.
President Dick Cheney and national security it adviser Condoleezza Rice. At
w
the same time, the narrative suggestswwthat then-Defense Secretary Donald H.
员 Colin Powell were largely left out of
Rumsfeld and then-Secretary of State
序 Cheney was the Vice President of Bush.
the decision-making process. Dick
程
entailment 马
黑
WNLI 中的train.tsv数据样式:
index sentence1 sentence2 label

0 I stuck a pin through a carrot. When I pulled the pin out, it had a
hole. The carrot had a hole. 1
1 John couldn't see the stage with Billy in front ofcohim m because he is so
.
short. John is so short. 1 ma
iThey
2 The police arrested all of the gang members. e were trying to stop
the drug trade in the neighborhood. i
The policeth were trying to stop the
w.
drug trade in the neighborhood. 1 ww
3 Steve follows Fred's example in everything.
员 He influences him hugely.
Steve influences him hugely. 0 序
4 程
When Tatyana reached the cabin,
马 her mother was sleeping. She was
黑
careful not to disturb her, undressing and climbing back into her berth.
mother was careful not to disturb her, undressing and climbing back into
her berth. 0
5 George got free tickets to the play, but he gave them to Eric, because
he was particularly eager to see it. George was particularly eager to see
it. 0
6 John was jogging through the park when he saw a man juggling
watermelons. He was very impressive. John was very impressive. 0
7 I couldn't put the pot on the shelf because it was too tall. The pot
was too tall. 1
8 We had hoped to place copies of our newsletter on all the chairs in the
auditorium, but there were simply not enough of them. There were simply
m
not enough copies of the newsletter. 1 co
ma.
h ei
it
w.
ww
(QNLI/RTE/WNLI) 中的train.tsv序数据样式说明
员 :
train.tsv中的数据内容共分为
程 4列, 第一列代表文本数据索引; 第二列和第三列数据代表
马
需要进行'是否蕴含黑'判定的句子对; 第四列数据代表两个句子是否具有蕴含关系,
0/not_entailment代表不是蕴含关系, 1/entailment代表蕴含关系.
QNLI中的test.tsv数据样式:
index question sentence

om
c
0 What organization is devoted to Jihad against m a. Israel? For some
i
decades prior to the First Palestine Intifada
t he in 1987, the Muslim
i
w. stance towards Israel, focusing
Brotherhood in Palestine took a "quiescent"
w
w
on preaching, education and social services, and benefiting from Israel's
"indulgence" to build up a network员of mosques and charitable organizations.
1
序
In what century was the Yarrow-Schlick-Tweedy balancing system used?
程
马 Yarrow-Schlick-Tweedy balancing 'system' was
In the late 19th century, the
黑
used on some marine triple expansion engines.
2 The largest brand of what store in the UK is located in Kingston Park?
Close to Newcastle, the largest indoor shopping centre in Europe, the
MetroCentre, is located in Gateshead.
3 What does the IPCC rely on for research? In principle, this means
that any significant new evidence or events that change our understanding
of climate science between this deadline and publication of an IPCC report
cannot be included.
4 What is the principle about relating spin and space variables? Thus in
the case of two fermions there is a strictly negative correlation between
spatial and spin variables, whereas for two bosons (e.g.m quanta of
o
electromagnetic waves, photons) the correlation is strictly
a .c positive.
m
5 Which network broadcasted Super Bowl 50 in the
h eiof $5 million
U.S.? CBS broadcast
Super Bowl 50 in the U.S., and charged an average
. it for a 30-
second commercial during the game.
6
ww
What did the museum acquire from the wRoyal College of Science? To link
this to the rest of the museum, a new 员entrance building was constructed on
序the
程
the site of the former boiler house, intended site of the Spiral,
between 1978 and 1982. 马
7
黑
What is the name of the old north branch of the Rhine? From Wijk bij
Duurstede, the old north branch of the Rhine is called Kromme Rijn ("Bent
Rhine") past Utrecht, first Leidse Rijn ("Rhine of Leiden") and then, Oude
Rijn ("Old Rhine").
8 What was one of Luther's most personal writings? It remains in use
today, along with Luther's hymns and his translation of the Bible.
...
(RTE/WNLI) 中的test.tsv数据样式:
m
. co
index sentence1 sentence2
ima
0 he
Maude and Dora had seen the trains rushing
t across the prairie, with
.i
long, rolling puffs of black smoke streaming back from the engine. Their
w
ww could be heard from far away. Horses
roars and their wild, clear whistles
员
ran away when they came in sight. Horses ran away when Maude and Dora
came in sight. 序
程
1 马 the trains rushing across the prairie, with
Maude and Dora had seen
黑black smoke streaming back from the engine. Their
long, rolling puffs of
roars and their wild, clear whistles could be heard from far away. Horses
ran away when they came in sight. Horses ran away when the trains came
in sight.
2 Maude and Dora had seen the trains rushing across the prairie, with
ran away when they came in sight. Horses ran away when the puffs came in
sight.
m
roars and their wild, clear whistles could be heard
. co from far away. Horses
ran away when they came in sight. Horses ranmaaway when the roars came in
i
sight.
t he
4 .i
Maude and Dora had seen the trains rushing
w across the prairie, with
ww
员 could be heard from far away. Horses
roars and their wild, clear whistles
序 Horses ran away when the whistles came
ran away when they came in sight.
程
in sight. 马
5 黑 the trains rushing across the prairie, with
Maude and Dora had seen
ran away when they came in sight. Horses ran away when the horses came
in sight.
ran away when they saw a train coming. Maude and Dora saw a train coming.
roars and their wild, clear whistles could be heard from c om far away. Horses
.
ran away when they saw a train coming. The trains
imasaw a train coming.
8 Maude and Dora had seen the trains rushing h e
it across the prairie, with
long, rolling puffs of black smoke streaming. back from the engine. Their
w
roars and their wild, clear whistles could ww be heard from far away. Horses
ran away when they saw a train coming.员 The puffs saw a train coming.
... 序
程
马
黑
(QNLI/RTE/WNLI) 中的test.tsv数据样式说明:
中的数据内容共分为3列, 第一列数据代表每条文本数据的索引; 第二列和第三
test.tsv
列数据代表需要进行'是否蕴含'判定的句子对.
o m
数据集的任务类型:
(QNLI/RTE/WNLI)
im
a.c
句子对二分类任务 .i
th e
评估指标为: ACC ww
w
员
序
程
马
黑
小节总结
学习了GLUE数据集合的介绍:
GLUE由纽约大学, 华盛顿大学, Google联合推出, 涵盖不同NLP任务类型, 截止至2020年1
月其中包括11个子任务数据集, 成为衡量NLP研究发展的衡量标准.
om
GLUE数据集合包含以下数据集: i m a.
c
CoLA 数据集 t he
i
w.
SST-2 数据集 ww
员
MRPC 数据集序
程
STS-B 数据集
马
黑
QQP 数据集
MNLI 数据集
SNLI 数据集
QNLI 数据集
RTE 数据集
WNLI 数据集
com
a.
e im
ith
.
2.3 NLP 中的常用预训练模型序员 ww
w
程
马
黑
学习目标
了解当下NLP中流行的预训练模型.
掌握如何加载和使用预训练模型.
当下NLP中流行的预训练模型
m
BERT
. co
ima
GPT
the
i
GPT-2 w.
ww
Transformer-XL 员
序
XLNet
程
马
黑
XLM
RoBERTa
DistilBERT
ALBERT
T5
XLM-RoBERTa
m
. co
ma
ei
BERT 及其变体: t
i
h
w.
bert-base-uncased: 编码器具有12个隐层 w , 输出768维张量, 12个自注意力头, 共110M参
w
数量, 在小写的英文文本上进行训练而得到序
员 .
程
bert-large-uncased: 编码器具有
马 24个隐层, 输出1024维张量, 16个自注意力头, 共340M参
黑
数量, 在小写的英文文本上进行训练而得到.
bert-base-cased: 编码器具有12个隐层, 输出768维张量, 12个自注意力头, 共110M参数
量, 在不区分大小写的英文文本上进行训练而得到.
bert-large-cased: 编码器具有24个隐层, 输出1024维张量, 16个自注意力头, 共340M参数
量, 在不区分大小写的英文文本上进行训练而得到.
bert-base-multilingual-uncased: 编码器具有12个隐层, 输出768维张量, 12个自注意力头,
共110M参数量, 在小写的102种语言文本上进行训练而得到. m
o
bert-large-multilingual-uncased: 编码器具有24个隐层, 输出 a .c1024维张量, 16个自注意力
m
头, 共340M参数量, 在小写的102种语言文本上进行训练而得到 t h ei .
i
bert-base-chinese: 编码器具有12个隐层, 输出 . 维张量, 12个自注意力头, 共110M参数
w768
ww
量, 在简体和繁体中文文本上进行训练而得到员 .
序
程
马
黑
GPT:
编码器具有12个隐层, 输出768维张量, 12个自注意力头, 共110M参数量, 由
openai-gpt:
OpenAI在英文语料上进行训练而得到.
GPT-2及其变体:
om
gpt2: 编码器具有12个隐层, 输出768维张量, 12a个自注意力头 .c , 共117M参数量, 在OpenAI
m
GPT-2英文语料上进行训练而得到. ei
i th
gpt2-xl: 编码器具有48个隐层, 输出1600
ww 维张量, 25个自注意力头, 共1558M参数量, 在大
w.
型的OpenAI GPT-2英文语料上进行训练而得到
序
员 .
程
马
黑
Transformer-XL:
编码器具有18个隐层, 输出1024维张量, 16个自注意力头, 共257M参数

transfo-xl-wt103:
量, 在wikitext-103英文语料进行训练而得到.
XLNet 及其变体:
xlnet-base-cased: 编码器具有12个隐层, 输出768维张量 co, 12个自注意力头, 共110M参数
m
量, 在英文语料上进行训练而得到. eim
a .
xlnet-large-cased: 编码器具有24个隐层, 输出 it1024维张量, 16个自注意力头, 共240参数
h
.
量, 在英文语料上进行训练而得到. www
员
序
程
马
黑
XLM:
xlm-mlm-en-2048: 编码器具有12个隐层, 输出2048维张量, 16个自注意力头, 在英文文本

上进行训练而得到.
RoBERTa 及其变体:
roberta-base: 编码器具有12个隐层, 输出768维张量, 12个自注意力头, 共125M参数量,
在英文文本上进行训练而得到. . co
m
roberta-large: 编码器具有24个隐层, 输出1024维张量
ma个自注意力头, 共355M参数量,
i16
h e,
在英文文本上进行训练而得到. ww
. it
w
员
序
程
DistilBERT 及其变体: 黑
马
基于bert-base-uncased的蒸馏(压缩)模型, 编码器具有6个隐层,
distilbert-base-uncased:
输出768维张量, 12个自注意力头, 共66M参数量.
基于bert-base-multilingual-uncased的蒸馏(压缩)模
distilbert-base-multilingual-cased:
型, 编码器具有6个隐层, 输出768维张量, 12个自注意力头, 共66M参数量.
ALBERT:
m
编码器具有12个隐层, 输出768im维张量 co
albert-base-v1: a. , 12个自注意力头, 共125M参数量,
在英文文本上进行训练而得到. th
e
i
albert-base-v2: 编码器具有12个隐层
ww , 输出768维张量, 12个自注意力头, 共125M参数量,
w.
在英文文本上进行训练而得到序
, 相比v1使用了更多的数据量, 花费更长的训练时间.
员
程
马
黑
T5及其变体:
t5-small: 编码器具有6个隐层, 输出512维张量, 8个自注意力头, 共60M参数量, 在C4语料
上进行训练而得到.
t5-base: 编码器具有12个隐层, 输出768维张量, 12个自注意力头, 共220M参数量, 在C4语
料上进行训练而得到.
t5-large: 编码器具有24个隐层, 输出1024维张量, 16个自注意力头, 共770M参数量, 在C4
语料上进行训练而得到. co
m
a.
im
t he
i
及其变体: w w.
XLM-RoBERTa w
员
xlm-roberta-base: 编码器具有12序个隐层, 输出768维张量, 8个自注意力头, 共125M参数
程
量, 在2.5TB的100种语言文本上进行训练而得到
黑
马 .
xlm-roberta-large: 编码器具有24个隐层, 输出1027维张量, 16个自注意力头, 共355M参
数量, 在2.5TB的100种语言文本上进行训练而得到.
预训练模型说明:
所有上述预训练模型及其变体都是以transformer为基础，只是在模型结构如神经元连接
方式，编码器隐层数，多头注意力的头数等发生改变，这些改变方式的大部分依据都是
由在标准数据集上的表现而定，因此，对于我们使用者而言，不需要从理论上深度探究
. co
m
这些预训练模型的结构设计的优劣，只需要在自己处理的目标数据上，尽量遍历所有可
e im
a
用的模型对比得到最优效果即可. .i
th
w
ww
员
序
程
小节总结黑
马
当下NLP中流行的预训练模型:
BERT
GPT
GPT-2
Transformer-XL
XLNet
m
XLM . co
i ma
RoBERTa
the
i
w.
DistilBERT
ww
ALBERT 员
序
程
T5 马
黑
XLM-RoBERTa
2.4 加载和使用预训练模型
学习目标 m a . co
m
ei
了解加载和使用预训练模型的工具. it h
w.
掌握加载和使用预训练模型的过程. ww
员
序
程
马
加载和使用预训练模型的工具
黑
在这里我们使用torch.hub工具进行模型的加载和使用.
这些预训练模型由世界先进的NLP研发团队huggingface提供.
注意: 下面使用的代码需要国外服务器的资源, 在国内使用的时候, 国内的网站下载可能会出
现在原地卡死不动, 或是网络连接超时等一些网络报错, 均是网络问题, 不是代码问题, 这个可
以先行跳过, 把主要逻辑梳理完成即可
com
a.
im
加载和使用预训练模型的步骤 w. ith
e
ww
第一步: 确定需要加载的预训练模型并安装依赖包员 .
序
第二步: 加载预训练模型的映射器tokenizer.
马
程
黑
第三步: 加载带/不带头的预训练模型 .
第四步: 使用模型获得输出结果.
第一步: 确定需要加载的预训练模型并安装依赖包
能够加载哪些模型可以参考2.3 NLP中的常用预训练模型
这里假设我们处理的是中文文本任务, 需要加载的模型是BERT的中文模型: bert-base-chinese
在使用工具加载模型前需要安装必备的依赖包: m o
a.c
m
ei
pip install tqdm boto3 requests regex sentencepiece
h
sacremoses
.it
w
ww
第二步: 加载预训练模型的映射器tokenizer
程
序
员
马
import torch
黑
# 预训练模型来源
source = 'huggingface/pytorch-transformers'
# 选定加载模型的哪一部分这里是模型的映射器
,
part = 'tokenizer'
# 加载的预训练模型的名字
model_name = 'bert-base-chinese'
tokenizer = torch.hub.load(source, part, model_name)
m
. co
ma
第三步: 加载带/不带头的预训练模型 t h ei
.i
加载预训练模型时我们可以选择带头或者不带头的模型 ww
w
这里的'头'是指模型的任务输出层,程选择加载不带头的模型序
员 , 相当于使用模型对输入文本进行特
征表示. 黑
马
选择加载带头的模型时, 有三种类型的'头'可供选择, modelWithLMHead(语言模型头),
modelForSequenceClassification(分类模型头), modelForQuestionAnswering(问答模型头)
不同类型的'头', 可以使预训练模型输出指定的张量维度. 如使用'分类模型头', 则输出尺寸为

(1,2)的张量, 用于进行分类任务判定结果.
# 加载不带头的预训练模型
part = 'model'
model = torch.hub.load(source, part, model_name)
m
加载带有语言模型头的预训练模型 . co
#
ima
part = 'modelWithLMHead'
he
it
lm_model = torch.hub.load(source, part, model_name)
w.
# 加载带有类模型头的预训练模型 ww
员
序
part = 'modelForSequenceClassification'
程
classification_model = torch.hub.load(source, part, model_name)
马
黑
# 加载带有问答模型头的预训练模型
part = 'modelForQuestionAnswering'
qa_model = torch.hub.load(source, part, model_name)
第四步: 使用模型获得输出结果
使用不带头的模型进行输出:
# 输入的中文文本 m
input_text = " 人生该如何起头" . co
ma
使用tokenizer进行数值映射 ei
#
i th
w .
indexed_tokens = tokenizer.encode(input_text)
ww
# 打印映射后的结构序
员
程
print("indexed_tokens:", indexed_tokens)
马
#
黑
将映射结构转化为张量输送给不带头的预训练模型
tokens_tensor = torch.tensor([indexed_tokens])
# 使用不带头的预训练模型获得结果
with torch.no_grad():
encoded_layers, _ = model(tokens_tensor)
print("不带头的模型输出结果:", encoded_layers)
print("不带头的模型输出结果的尺寸:", encoded_layers.shape)
m
. co
m a
h ei
it
输出效果: ww w.
员
序
映射后的结果
# tokenizer 和程是起止符
, 101 马102 ,
# 中间的每个数字对应人生该如何起头的每个字
" 黑 " .
indexed_tokens: [101, 782, 4495, 6421, 1963, 862, 6629, 1928, 102]
不带头的模型输出结果: tensor([[[ 0.5421, 0.4526, -0.0179, ..., 1.0447,

-0.1140, 0.0068],
[-0.1343, 0.2785, 0.1602, ..., -0.0345, -0.1646, -0.2186],
[ 0.9960, -0.5121, -0.6229, ..., 1.4173, 0.5533, -0.2681],
...,
[ 0.0115, 0.2150, -0.0163, ..., 0.6445, 0.2452, -0.3749],
[ 0.8649, 0.4337, -0.1867, ..., 0.7397, -0.2636,
m 0.2144],
[-0.6207, 0.1668, 0.1561, ..., co
1.1218, -0.0985,
. -0.0937]]])
ma
h ei
t
#输出尺寸为1x9x768, 即每个字已经使用768维的向量进行了表示
w .i ,
#我们可以基于此编码结果进行接下来的自定义操作, w如w: 编写自己的微调网络进行最终输出.
不带头的模型输出结果的尺寸: torch.Size([1,
序
员9, 768])
程
马
黑
使用带有语言模型头的模型进行输出:
# 使用带有语言模型头的预训练模型获得结果
lm_output = lm_model(tokens_tensor)
print("带语言模型头的模型输出结果:", lm_output)
print("带语言模型头的模型输出结果的尺寸:", lm_output[0].shape)
m
co
m a.
h ei
it
w.
ww
输出效果: 员
序
程
马
带语言模型头的模型输出结果黑: (tensor([[[ -7.9706, -7.9119, -7.9317, ...,
-7.2174, -7.0263, -7.3746],
[ -8.2097, -8.1810, -8.0645, ..., -7.2349, -6.9283,
-6.9856],
[-13.7458, -13.5978, -12.6076, ..., -7.6817, -9.5642,
-11.9928],
...,
[ -9.0928, -8.6857, -8.4648, ..., -8.2368, -7.5684,
-10.2419],
[ -8.9458, -8.5784, -8.6325, ..., -7.0547, -5.3288,
-7.8077], m
[ -8.4154, -8.5217, -8.5379, ..., -6.7102,. co -5.9782,
m a
-7.6909]]]),)
e i
i th
# 输出尺寸为1x9x21128, 即每个字已经使用21128 w维的向量进行了表示
. ,
ww
# 同不带头的模型一样, 我们可以基于此编码结果进行接下来的自定义操作 , 如: 编写自己的微调网
络进行最终输出. 序
员
带语言模型头的模型输出结果的尺寸: torch.Size([1,
程 9, 21128])
马
黑
使用带有分类模型头的模型进行输出:
# 使用带有分类模型头的预训练模型获得结果
classification_output = classification_model(tokens_tensor)
带分类模型头的模型输出结果:", classification_output) m
print("
. co
a
im
print("带分类模型头的模型输出结果的尺寸:", classification_output[0].shape)
e
th
w .i
ww
员
序
程
输出效果: 黑
马
带分类模型头的模型输出结果: (tensor([[-0.0649, -0.1593]]),)

# 输出尺寸为1x2, 可直接用于文本二分问题的输出
带分类模型头的模型输出结果的尺寸: torch.Size([1, 2])
使用带有问答模型头的模型进行输出:
m
. co
使用带有问答模型头的模型进行输出时, 需要使输入的形式为句子对 ma
#
h ei
# 第一条句子是对客观事物的陈述 i t
# 第二条句子是针对第一条句子提出的问题 www.
# 问答模型最终将得到两个张量, 员
# 每个张量中最大值对应索引的分别代表答案的在文本中的起始位置和终止位置
序 .
我家的小狗是黑色的
input_text1 = "
马
程 "
我家的小狗是什么颜色的呢
黑
input_text2 = " ?"
# 映射两个句子
indexed_tokens = tokenizer.encode(input_text1, input_text2)
句子对的
print(" indexed_tokens:", indexed_tokens)
# 输出结果: [101, 2769, 2157, 4638, 2207, 4318, 3221, 7946, 5682, 4638, 102,
2769, 2157, 4638, 2207, 4318, 3221, 784, 720, 7582, 5682, 4638, 1450, 136,
102]
# 用，来区分第一条和第二条句子
0 1
. co
m
segments_ids = [0]*11 + [1]*14
ma
h ei
# 转化张量形式 w.
ti
w
segments_tensors = torch.tensor([segments_ids])
w
tokens_tensor = torch.tensor([indexed_tokens])
员
序
# 使用带有问答模型头的预训练模型获得结果马
程
with torch.no_grad(): 黑
start_logits, end_logits = qa_model(tokens_tensor,
token_type_ids=segments_tensors)
print("带问答模型头的模型输出结果:", (start_logits, end_logits))

print("带问答模型头的模型输出结果的尺寸:", (start_logits.shape,
end_logits.shape))
c om
输出效果: e im
a.
ith
.
ww
句子对的indexed_tokens: [101, 2769, 2157, w
4638, 2207, 4318, 3221, 7946, 5682,
员 3221, 784, 720, 7582, 5682, 4638,
4638, 102, 2769, 2157, 4638, 2207, 4318,
序
1450, 136, 102] 程
马
黑
带问答模型头的模型输出结果: (tensor([[ 0.2574, -0.0293, -0.8337, -0.5135,
-0.3645, -0.2216, -0.1625, -0.2768,
-0.8368, -0.2581, 0.0131, -0.1736, -0.5908, -0.4104, -0.2155,
-0.0307,
-0.1639, -0.2691, -0.4640, -0.1696, -0.4943, -0.0976, -0.6693,
0.2426,
0.0131]]), tensor([[-0.3788, -0.2393, -0.5264, -0.4911, -0.7277,
-0.5425, -0.6280, -0.9800,
-0.6109, -0.2379, -0.0042, -0.2309, -0.4894, -0.5438, -0.6717,
-0.5371,
-0.1701, 0.0826, 0.1411, -0.1180, -0.4732, -0.1541, 0.2543,
m
0.2163,
. co
-0.0042]])) ma
i
the
i
w.
#输出为两个形状1x25的张量, 他们是两条句子合并长度的概率分布
ww ,
#第一个张量中最大值所在的索引代表答案出现的起始索引
员 ,
#第二个张量中最大值所在的索引代表答案出现的终止索引
程
序 .
带问答模型头的模型输出结果的尺寸马 : (torch.Size([1, 25]), torch.Size([1, 25]))
黑
小节总结
加载和使用预训练模型的工具:
在这里我们使用torch.hub工具进行模型的加载和使用. 这些预训练模型由世界先进的
NLP研发团队huggingface提供.
m
. co
ma
ei
加载和使用预训练模型的步骤: . it
h
ww
第一步: 确定需要加载的预训练模型并安装依赖包
员
w .
第二步: 加载预训练模型的映射器序
程 tokenizer.
马
第三步: 加载带/不带头的预训练模型
黑 .
第四步: 使用模型获得输出结果.
2.5 迁移学习实践
om
学习目标 eim
a.
c
th
了解并掌握指定任务类型的微调脚本使用方法. www . i
了解并掌握通过微调脚本微调后模型的使用方法
序
员 .
掌握通过微调方式进行迁移学习的两种类型实现过程
马
程 .
黑
指定任务类型的微调脚本:
研究机构向我们提供了针对GLUE数据集合任务类型的微调脚本, 这些微调脚
huggingface
本的核心都是微调模型的最后一个全连接层.
通过简单的参数配置来指定GLUE中存在任务类型(如: CoLA对应文本二分类, MRPC对应
句子对文本二分类, STS-B对应句子对文本多分类), 以及指定需要微调的预训练模型.
m
. co
i ma
指定任务类型的微调脚本使用步骤 w.
i the
w w
第一步: 下载微调脚本文件员
第二步: 配置微调脚本参数马程序
第三步: 运行并检验效果黑
第一步: 下载微调脚本文件
# 克隆的
huggingface transfomers 文件
git clone https://github.com/huggingface/transformers.git
# 进行
transformers 文件夹 m
cd transformers
. co
ma
# 安装的
python transformer
i .
工具包, 因为微调脚本是tpyhe文件
pip install . i
w w.
w
# 当前的版本可能跟我们教学的版本并不相同，你还需要执行：
员
序
pip install transformers==2.3.0
程
马
黑
# 进入微调脚本所在路径并查看
cd examples
ls
# 其中run_glue.py就是针对GLUE数据集合任务类型的微调脚本
注意：
对于run_glue.py，由于版本变更导致，请通过该地址http://git.itcast.cn/Stephen/AI-
key-file/blob/master/run_glue.py复制里面的代码，覆盖原有内容。
m
co .
ma
h ei
. it
第二步: 配置微调脚本参数 ww
w
员
在run_glue.py同级目录下创建run_glue.sh
程
序文件 , 写入内容如下:
马
黑
# 定义
DATA_DIR: 微调数据所在路径这里我们使用
, 中的数据作为微调数据
glue_data
export DATA_DIR="../../glue_data"
# 定义
SAVE_DIR: 模型的保存路径我们将模型保存在当前目录的
, 文件中
bert_finetuning_test
export SAVE_DIR="./bert_finetuning_test/"
# 使用 python运行微调脚本
# --model_type: 选择需要微调的模型类型, 这里可以选择BERT, XLNET, XLM, roBERTa,
distilBERT, ALBERT
# --model_name_or_path: 选择具体的模型或者变体, 这里是在英文语料上微调, 因此选择
bert-base-uncased
# --task_name: 它将代表对应的任务类型, 如MRPC代表句子对二分类任务 m
# --do_train: 使用微调脚本进行训练 . co
# --do_eval: 使用微调脚本进行验证 im
a
# --data_dir: 训练集及其验证集所在路径, 将自动寻找该路径下的
it
h e train.tsv和dev.tsv作为训
练集和验证集 w .
# --max_seq_length: 输入句子的最大长度 ww , 超过则截断, 不足则补齐
# --learning_rate: 学习率员
# --num_train_epochs: 训练轮数序
程
马训练后的模型保存路径
# --output_dir $SAVE_DIR:
黑再次训练时将清空之前的保存路径内容重新写入
# --overwrite_output_dir:
python run_glue.py \
--model_type BERT \
--model_name_or_path bert-base-uncased \
--task_name MRPC \
--do_train \
--do_eval \
--data_dir $DATA_DIR/MRPC/ \
--max_seq_length 128 \
--learning_rate 2e-5 \
--num_train_epochs 1.0 \ m
--output_dir $SAVE_DIR \ . co
--overwrite_output_dir ma
h ei
it
w w.
w
第三步: 运行并检验效果序
员
程
马
# 使用命令运行
sh 黑
sh run_glue.sh
输出效果:
# 最终打印模型的验证结果 :
m
01/05/2020 23:59:53 - INFO - __main__ - Saving features
. co into cached file
a
../../glue_data/MRPC/cached_dev_bert-base-uncased_128_mrpc
01/05/2020 23:59:53 - INFO - __main__ - *****he im
Running evaluation *****
01/05/2020 23:59:53 - INFO - __main__ - i
Numt examples = 408
.
01/05/2020 23:59:53 - INFO - __main__ - www Batch size = 8
Evaluating: 100%|█| 51/51 [00:23<00:00,
员 2.20it/s]
序 - ***** Eval results *****
01/06/2020 00:00:16 - INFO - __main__
程 -
01/06/2020 00:00:16 - INFO - __main__ acc = 0.7671568627450981
马
01/06/2020 00:00:16 - INFO - 黑
__main__ - acc_and_f1 = 0.8073344506341863
01/06/2020 00:00:16 - INFO - __main__ - f1 = 0.8475120385232745
查看$SAVE_DIR的文件内容:
added_tokens.json
checkpoint-450
checkpoint-400
m
checkpoint-350
. co
checkpoint-200
ima
checkpoint-300
the
checkpoint-250 i
w.
checkpoint-200
checkpoint-150
ww
员
checkpoint-100 序
checkpoint-50 程
马
pytorch_model.bin 黑
training_args.bin
config.json
special_tokens_map.json
vocab.txt
eval_results.txt
tokenizer_config.json
文件解释: m
co
pytorch_model.bin代表模型参数，可以使用torch.load a 加载查看；
.
eim
traning_args.bin代表模型训练时的超参，如ibatch_size
th ，epoch等，仍可使用torch.load
.
查看； ww
w
config.json是模型配置文件，如多头注意力的头数，编码器的层数等，代表典型的模型
序
员
结构，如bert，xlnet，一般不更改；马
程
黑
added_token.json记录在训练时通过代码添加的自定义token对应的数值，即在代码中使
用add_token方法添加的自定义词汇；
special_token_map.json当添加的token具有特殊含义时，如分隔符，该文件存储特殊字
符的及其对应的含义，使文本中出现的特殊字符先映射成其含义，之后特殊字符的含义
仍然使用add_token方法映射。
checkpoint: 若干步骤保存的模型参数文件(也叫检测点文件)。
com
a.
通过微调脚本微调后模型的使用步骤 . ith
eim
w
第一步: 在https://huggingface.co/join上创建一个帐户
员
ww
第二步: 在服务器终端使用transformers-cli 程
序登陆
马
第三步: 使用transformers-cli上传模型并查看
黑
第四步: 使用pytorch.hub加载模型进行使用
第一步: 在https://huggingface.co/join上创建一个帐户
# 如果由于网络原因无法访问, 我们已经为你提供了默认账户
username: ItcastAI
password: ItcastAI
m
. co
i ma
the
i
w.
ww
员
序
程
马
黑
m
. co
m a
h ei
it
w w.
w
员
序
程
马
黑
第二步: 在服务器终端使用transformers-cli登陆 com

a.
在微调模型的服务器上登陆 e im
#
使用刚刚注册的用户名和密码 ith
#
w.
# 默认 username: ItcastAI ww
# 默认 password: ItcastAI 员
$ transformers-cli login 序
程
马
黑
# 使用
transformers-cli upload 命令上传模型
# 选择正确的微调模型路径
$ transformers-cli upload ./bert_finetuning_test/
# 查看上传结果
$ transformers-cli ls
m
co
.
Filename
ETag Size ei
ma LastModified
h
.it
----------------------------------------------------- ---------------------
w
ww
--- ----------------------------------
bert_finetuning_test/added_tokens.json
---------
2020-01-
员
序
05T17:39:57.000Z "99914b932bd37a50b983c5e7c90ae93b" 2
程
bert_finetuning_test/checkpoint-400/config.json 2020-01-
马
黑
05T17:26:49.000Z "74d53ea41e5acb6d60496bc195d82a42" 684
bert_finetuning_test/checkpoint-400/training_args.bin 2020-01-
05T17:26:47.000Z "b3273519c2b2b1cb2349937279880f50" 1207
bert_finetuning_test/checkpoint-450/config.json 2020-01-
05T17:15:42.000Z "74d53ea41e5acb6d60496bc195d82a42" 684
bert_finetuning_test/checkpoint-450/pytorch_model.bin 2020-01-
05T17:15:58.000Z "077cc0289c90b90d6b662cce104fe4ef" 437982584
bert_finetuning_test/checkpoint-450/training_args.bin 2020-01-
05T17:15:40.000Z "b3273519c2b2b1cb2349937279880f50" 1207
bert_finetuning_test/config.json 2020-01-
05T17:28:50.000Z "74d53ea41e5acb6d60496bc195d82a42" 684
bert_finetuning_test/eval_results.txt o m 2020-01-
05T17:28:56.000Z "67d2d49a96afc4308d33bfcddda8a7c5" a .c 81
i m
bert_finetuning_test/pytorch_model.bin e 2020-01-
i th
05T17:28:59.000Z "d46a8ccfb8f5ba9ecee70cef8306679e" 437982584
w.
ww
bert_finetuning_test/special_tokens_map.json
05T17:28:54.000Z "8b3fb1023167bb4ab9d70708eb05f6ec"
2020-01-
112
员
序
bert_finetuning_test/tokenizer_config.json 2020-01-
程
05T17:28:52.000Z "0d7f03e00ecb582be52818743b50e6af" 59
马
黑
bert_finetuning_test/training_args.bin 2020-01-
05T17:28:48.000Z "b3273519c2b2b1cb2349937279880f50" 1207
bert_finetuning_test/vocab.txt 2020-01-
05T17:39:55.000Z "64800d5d8528ce344256daf115d4965e" 231508
第四步: 使用pytorch.hub加载模型进行使用, 更多信息请参考2.4 加载和使用预训练模型

# 若之前使用过huggingface的transformers, 请清除~/.cache
m
. co
import torch
如： ima
# ItcastAI/bert_finetuning_test
source = 'huggingface/pytorch-transformers' it
he
选定加载模型的哪一部分这里是模型的映射器 .
# ,
w ww
part = 'tokenizer'
员
序
程
#############################################
加载的预训练模型的名字马
# 黑
# 使用自己的模型名字 "username/model_name"
# 如：
'ItcastAI/bert_finetuning_test'
model_name = 'ItcastAI/bert_finetuning_test'
#############################################
tokenizer = torch.hub.load('huggingface/pytorch-transformers', 'tokenizer',
model_name)
model = torch.hub.load('huggingface/pytorch-transformers',
'modelForSequenceClassification', model_name)
index = tokenizer.encode("Talk is cheap", "Please show me your code!")
是
# 102 bert模型中的间隔结束符号的数值映射
( )
m
mark = 102 co.
i ma
# 找到第一个102的索引, 即句子对的间隔符号 e
i th
k = index.index(mark)
w.
ww
# 句子对分割id列表, 由0，1组成, 0的位置代表第一个句子
员 , 1的位置代表第二个句子
segments_ids = [0]*(k + 1) +序[1]*(len(index) - k - 1)
# 转化为tensor
程
马
黑
tokens_tensor = torch.tensor([index])
segments_tensors = torch.tensor([segments_ids])
# 使用评估模式
使用模型预测获得结果
#
result = model(tokens_tensor, token_type_ids=segments_tensors)
打印预测结果以及张量尺寸
#
print(result)
print(result[0].shape)
m
. co
ma
h ei
t
输出效果: w w.
i
w
员
序
(tensor([[-0.0181, 程
0.0263]]),)
马
torch.Size([1, 2]) 黑
通过微调方式进行迁移学习的两种类型
类型一: 使用指定任务类型的微调脚本微调预训练模型, 后接带有输出头的预定义网络输出结
果.
类型二: 直接加载预训练模型进行输入文本的特征表示, 后接自定义网络进行微调输出结果
om
.
.c
ma
h ei
说明: 所有类型的实战演示, 都将针对中文文本进行ww. .it
w
员
序
类型一实战演示黑
马
程
使用文本二分类的任务类型SST-2的微调脚本微调中文预训练模型, 后接带有分类输出头的预
定义网络输出结果. 目标是判断句子的情感倾向.
准备中文酒店评论的情感分析语料, 语料样式与SST-2数据集相同, 标签0代表差评, 标签1好评.
语料存放在与glue_data/同级目录cn_data/下, 其中的SST-2目录包含train.tsv和dev.tsv
train.tsv
m
. co
sentence label i ma
e
早餐不好服务不到位,晚餐无西餐,早餐晚餐相同.,i房间条件不好
, th ,餐厅不分吸烟区.房间不分有无烟房.
0 w
去的时候酒店大厅和餐厅在装修,感觉大厅有点挤
, ww .由于餐厅装修本来该享受的早饭,也没有享受(他
们是点开始每个房间送,但是我时间来不及了
8 序
员 )不过前台服务员态度好! 1
有很长时间没有在西藏大厦住了，以前去北京在这里住的较多。这次住进来发现换了液晶电视，但网络
程
不是很好，他们自己说是收费的原因造成的。其它还好。
黑
马 1
非常好的地理位置，住的是豪华海景房，打开窗户就可以看见栈桥和海景。记得很早以前也住过，现在
重新装修了。总的来说比较满意，以后还会住 1
交通很方便，房间小了一点，但是干净整洁，很有香港的特色，性价比较高，推荐一下哦 1
酒店的装修比较陈旧，房间的隔音，主要是卫生间的隔音非常差，只能算是一般的 0
酒店有点旧，房间比较小，但酒店的位子不错，就在海边，可以直接去游泳。8楼的海景打开窗户就是
海。如果想住在热闹的地带，这里不是一个很好的选择，不过威海城市真的比较小，打车还是相当便宜
的。晚上酒店门口出租车比较少。 1
位置很好，走路到文庙、清凉寺5分钟都用不了，周边公交车很多很方便，就是出租车不太爱去（老城
区路窄爱堵车），因为是老宾馆所以设施要陈旧些， 1
酒店设备一般，套房里卧室的不能上网，要到客厅去。 0
m
. co
m a
h ei
dev.tsv it
w w.
w
sentence label 员
序
房间里有电脑，虽然房间的条件略显简陋，但环境、服务还有饭菜都还是很不错的。如果下次去无锡，
程
我还是会选择这里的。 1 黑
马
我们是5月1日通过携程网入住的，条件是太差了，根本达不到四星级的标准，所有的东西都很陈旧，卫
生间水龙头用完竟关不上，浴缸的漆面都掉了，估计是十年前的四星级吧，总之下次是不会入住了。
0
离火车站很近很方便。住在东楼标间，相比较在九江住的另一家酒店，房间比较大。卫生间设施略旧。
服务还好。10元中式早餐也不错，很丰富，居然还有青菜肉片汤。 1
坐落在香港的老城区，可以体验香港居民生活，门口交通很方便，如果时间不紧，坐叮当车很好呀！周
围有很多小餐馆，早餐就在中远后面的南北嚼吃的，东西很不错。我们定的大床房，挺安静的，总体来
说不错。前台结账没有银联！ 1
酒店前台服务差，对待客人不热情。号称携程没有预定。感觉是客人在求他们，我们一定得住。这样的
宾馆下次不会入住！ 0
价格确实比较高，而且还没有早餐提供。 1 co
m
是一家很实惠的酒店，交通方便，房间也宽敞，晚上没有电话骚扰，住了两次，有一次住５０１房间，
a .
洗澡间排水不畅通，也许是个别问题．服务质量很好，刚入住时没有调好宽带，服务员很快就帮忙解决
e im
了． 1 it
h
位置非常好，就在西街的街口，但是却闹中取静，环境很清新优雅。
w .
ww,设备也不错. 1
1
房间应该超出30平米,是HK同级酒店中少有的大;重装之后
员
序
程
马
黑
在run_glue.py同级目录下创建run_cn.sh文件, 写入内容如下:
# 定义
DATA_DIR: 微调数据所在路径
export DATA_DIR="../../cn_data"
# 定义
SAVE_DIR: 模型的保存路径我们将模型保存在当前目录的bert_finetuning文件中
,
export SAVE_DIR="./bert_cn_finetuning/"
# 使用python运行微调脚本
--model_type: 选择BERT m
#
--model_name_or_path: 选择bert-base-chinese . co
# a
# --task_name: 句子二分类任务SST-2 eim
--do_train: 使用微调脚本进行训练
h
#
.it
--do_eval: 使用微调脚本进行验证 w
#
#
ww
--data_dir: "./cn_data/SST-2/", 将自动寻找该路径下的train.tsv和dev.tsv作为训练
集和验证集员
序
# --max_seq_length: 128，输入句子的最大长度
程
马 "./bert_finetuning/", 训练后的模型保存路径
黑
# --output_dir $SAVE_DIR:
python run_glue.py \
--model_type BERT \
--model_name_or_path bert-base-chinese \
--task_name SST-2 \
--do_train \
--do_eval \
--data_dir $DATA_DIR/SST-2/ \
--max_seq_length 128 \
--learning_rate 2e-5 \
--num_train_epochs 1.0 \
m
--output_dir $SAVE_DIR \
. co
ma
h ei
it
w w.
w
运行并检验效果员
序
程
马
# 使用命令运行
sh
黑
sh run_cn.sh
输出效果:
# 最终打印模型的验证结果准确率高达
, 0.88.
m
01/06/2020 14:22:36 - INFO - __main__ - Saving features
. co into cached file
a
01/06/2020 14:22:36 - INFO - __main__ - *****he
im
../../cn_data/SST-2/cached_dev_bert-base-chinese_128_sst-2
Running evaluation *****
01/06/2020 14:22:36 - INFO - __main__ - it examples = 1000
.Num
w
01/06/2020 14:22:36 - INFO - __main__ - ww Batch size = 8
Evaluating: 100%|████████████| 125/125员[00:56<00:00, 2.20it/s]
序
01/06/2020 14:23:33 - INFO - __main__
程 -- ***** Eval results *****
马
01/06/2020 14:23:33 - INFO - __main__ acc = 0.88
黑
查看$SAVE_DIR的文件内容:
added_tokens.json
checkpoint-350
checkpoint-200
m
checkpoint-300
. co
checkpoint-250
i ma
checkpoint-200
the
checkpoint-150 i
w.
checkpoint-100
checkpoint-50
ww
员
pytorch_model.bin 序
training_args.bin 程
马
config.json 黑
special_tokens_map.json
vocab.txt
eval_results.txt
tokenizer_config.json
使用transformers-cli上传模型:
# 默认 username: ItcastAI
co
m
# 默认 password: ItcastAI
m a .
$ transformers-cli login
h ei
t
使用命令上传模型 w .i
#
#
transformers-cli upload
选择正确的微调模型路径 ww
员
序
$ transformers-cli upload ./bert_cn_finetuning/
程
马
黑
通过pytorch.hub加载模型进行使用:
import torch
# 模型名字为 'ItcastAI/bert_cn_finetuning'
model_name = 'ItcastAI/bert_cn_finetuning'
m
co
tokenizer = torch.hub.load(source, 'tokenizer', model_name)
.
ma
model = torch.hub.load(source, 'modelForSequenceClassification',
model_name) ei h
. it
w
def get_label(text):
index = tokenizer.encode(text)
ww
员
序
tokens_tensor = torch.tensor([index])
# 使用评估模式马
程
with torch.no_grad(): 黑
#使用模型预测获得结果
result = model(tokens_tensor)
predicted_label = torch.argmax(result[0]).item()
return predicted_label
if __name__ == "__main__":
# text = " 早餐不好服务不到位晚餐无西餐早餐晚餐相同,房间条件不好"
, , ,
text = " 房间应该超出平米是同级酒店中少有的大;重装之后,设备也不错."
30 , HK
print(" 输入文本为:", text)
print(" 预测标签为:", get_label(text))
m
. co
i ma
the
输出效果: w.
i
ww
员
序
输入文本为: 早餐不好,服务不到位
马
程,晚餐无西餐,早餐晚餐相同,房间条件不好
预测标签为: 0 黑
输入文本为: 房间应该超出30平米,是HK同级酒店中少有的大;重装之后,设备也不错.
预测标签为: 1
类型二实战演示
直接加载预训练模型进行输入文本的特征表示, 后接自定义网络进行微调输出结果.
使用语料和完成的目标与类型一实战相同. a . co
m
m
h ei
it
直接加载预训练模型进行输入文本的特征表示 w.
ww :
员
序
import torch 程
# 进行句子的截断补齐规范长度 ( )黑
马
from keras.preprocessing import sequence
# 直接使用预训练的 bert中文模型
model_name = 'bert-base-chinese'
# 通过 torch.hub 获得已经训练好的 bert-base-chinese 模型

model = torch.hub.load(source, 'model', model_name)
# 获得对应的字符映射器它将把中文的每个字映射成一个数字
,
co
m
a .
tokenizer = torch.hub.load(source, 'tokenizer', model_name)
e im
# 句子规范长度 ith
w.
cutlen = 32
ww
员
def get_bert_encode(text): 序
""" 程
使用编码中文文本马
description: 黑
bert-chinese
要进行编码的文本
:param text:
使用编码后的文本张量表示
:return: bert
"""
# 首先使用字符映射器对每个汉字进行映射
# 这里需要注意, bert的tokenizer映射后会为结果前后添加开始和结束标记即101和102
# 这对于多段文本的编码是有意义的, 但在我们这里没有意义, 因此使用[1:-1]对头和尾进行
切片
indexed_tokens = tokenizer.encode(text[:cutlen])[1:-1]
# 对映射后的句子进行截断补齐
indexed_tokens = sequence.pad_sequences([indexed_tokens], cutlen)
# 之后将列表结构转化为 tensor
m
co
tokens_tensor = torch.LongTensor(indexed_tokens)
.
# 使模型不自动计算梯度 eim
a
with torch.no_grad(): h
# 调用模型获得隐层输出 .it
w
ww
encoded_layers, _ = model(tokens_tensor)
# 输出的隐层是一个三维张量最外层一维是我们使用降去它
, 员 1, [0] .
序
encoded_layers = encoded_layers[0]
程
马
return encoded_layers
黑
调用:
if __name__ == "__main__":
text = " 早餐不好服务不到位晚餐无西餐早餐晚餐相同,房间条件不好"
, , ,
encoded_layers = get_bert_encode(text)
m
print(encoded_layers)
. co
print(encoded_layers.shape)
ma
h ei
it
w w.
w
员
输出效果: 程
序
马
黑
tensor([[-1.2282, 1.0551, -0.7953, ..., 2.3363, -0.6413, 0.4174],
[-0.9769, 0.8361, -0.4328, ..., 2.1668, -0.5845, 0.4836],
[-0.7990, 0.6181, -0.1424, ..., 2.2845, -0.6079, 0.5288],
...,
[ 0.9514, 0.5972, 0.3120, ..., 1.8408, -0.1362, -0.1206],
[ 0.1250, 0.1984, 0.0484, ..., 1.2302, -0.1905, 0.3205],
[ 0.2651, 0.0228, 0.1534, ..., 1.0159, -0.3544, 0.1479]])
torch.Size([32, 768])
c om
a.
e im
自定义单层的全连接网络作为微调网络: ith
w .
ww 0.5倍的训练数据量, 小于10倍的训练
根据实际经验, 自定义的微调网络参数总数应大于
员
数据量, 这样有助于模型在合理的时间范围内收敛
程
序 .
马
黑
import torch.nn as nn
import torch.nn.functional as F
class Net(nn.Module):
""" 定义微调网络的类 """
def __init__(self, char_size=32, embedding_size=768):
"""
:param char_size: 输入句子中的字符数量即输入句子规范后的长度128.
,
:param embedding_size: 字嵌入的维度因为使用的中文模型嵌入维度是768,
, bert
因此为
embedding_size 768
m
"""
. co
a
super(Net, self).__init__()
im
# 将和
char_size embedding_size 传入其中
th
e
self.char_size = char_size .i
w
ww
self.embedding_size = embedding_size
# 实例化一个全连接层员
序
self.fc1 = nn.Linear(char_size*embedding_size, 2)
程
马
def forward(self, 黑
x):
# 对输入的张量形状进行变换以满足接下来层的输入要求
,
x = x.view(-1, self.char_size*self.embedding_size)
# 使用一个全连接层
x = self.fc1(x)
return x
调用: co
m
a .
m
h ei
if __name__ == "__main__": it
# 随机初始化一个输入参数 w w.
w
x = torch.randn(1, 32, 768)
实例化网络结构所有参数使用默认值员
# , 序
net = Net() 程
马
nr = net(x) 黑
print(nr)
输出效果:
tensor([[0.3279, 0.2519]], grad_fn=<ReluBackward0>)
c om
a.
eim
构建训练与验证数据批次生成器: ith
w.
ww
import pandas as pd 员
序
from collections import Counter 程
from functools import reduce 马
黑
from sklearn.utils import shuffle
def data_loader(train_data_path, valid_data_path, batch_size):

"""
从持久化文件中加载数据
description:
训练数据路径
:param train_data_path:
验证数据路径
:param valid_data_path:
训练和验证数据集的批次大小
:param batch_size:
训练数据生成器验证数据生成器训练数据数量验证数据数量
:return: , , ,
"""
# 使用进行数据的读取并去除第一行的列名
pd csv ,
om
train_data = pd.read_csv(train_data_path, cheader=None,
a.
sep="\t").drop([0])
eim
th
valid_data = pd.read_csv(valid_data_path, header=None,
sep="\t").drop([0]) .i w
ww
# 打印训练集和验证集上的正负样本数量员
训练数据集的正负样本数量
print("
程
序 :")
马
print(dict(Counter(train_data[1].values)))
验证数据集的正负样本数量
print(" 黑 :")
print(dict(Counter(valid_data[1].values)))
# 验证数据集中的数据总数至少能够满足一个批次
if len(valid_data) < batch_size:
raise("Batch size or split not match!")
def _loader_generator(data):
"""
获得训练集验证集的每个批次数据的生成器
description: /
训练数据或验证数据
:param data: m
一个批次的训练数据或验证数据的生成器
:return:
a . co
""" im
# 以每个批次的间隔遍历数据集 th
e
ibatch_size):
for batch in range(0, len(data), w.
#定义数据的张量列表
batch ww
batch_encoded = [] 员
batch_labels = [] 序
程
# 将一个马
bitch_size
黑
大小的数据转换成列表形式并进行逐条遍历 ,
for item in shuffle(data.values.tolist())[batch:
batch+batch_size]:
# 使用中文模型进行编码
bert
encoded = get_bert_encode(item[0])
# 将编码后的每条数据装进预先定义好的列表中
batch_encoded.append(encoded)
# 同样将对应的该的标签装进
batch 列表中 labels
batch_labels.append([int(item[1])])
# 使用高阶函数将列表中的数据转换成模型需要的张量形式
reduce
的形状是
# encoded (batch_size*max_len, embedding_size)
encoded = reduce(lambda x, y: torch.cat((x, my), dim=0),
o
batch_encoded)
a .c
labels = torch.tensor(reduce(lambda x, my: x + y, batch_labels))
# 以生成器的方式返回数据和标签 h ei
it
yield (encoded, labels) w.
ww
# 对训练集和验证集分别使用员
_loader_generator , 函数返回对应的生成器
# 最后还要返回训练集和验证集的样本数量程
序
马
return _loader_generator(train_data), _loader_generator(valid_data),
黑
len(train_data), len(valid_data)
调用:
if __name__ == "__main__":
train_data_path = "./cn_data/SST-2/train.tsv"
valid_data_path = "./cn_data/SST-2/dev.tsv"
m
batch_size = 16
train_data_labels, valid_data_labels, \ a.
co
m
ei
train_data_len, valid_data_len = data_loader(train_data_path,
h
t
valid_data_path, batch_size)
print(next(train_data_labels))ww
.i
w
print(next(valid_data_labels))
员
print("train_data_len:",序train_data_len)
程 valid_data_len)
print("valid_data_len:",
马
黑
输出效果:
训练数据集的正负样本数量:
{'0': 1518, '1': 1442}
验证数据集的正负样本数量:
{'1': 518, '0': 482} m
(tensor([[[-0.8328, 0.9376, -1.2489, ..., 1.8594, . co -0.4636, -0.1682],
a
[-0.9798, 0.5113, -0.9868, ..., 1.5500,
eim -0.1934, 0.2521],
h
[-0.7574, 0.3086, -0.6031, ..., t1.8467, -0.2507, 0.3916],
. i
...,
ww
[ 0.0064, 0.2321, 0.3785, w..., 0.3376, 0.4748, -0.1272],
[-0.3175, 0.4018, -0.0377,员
序 ..., 0.6030, 0.2916, -0.4172],
程
[-0.6154, 1.0439, 0.2921, ..., 0.5048, -0.0983, 0.0061]]]),
马1, 0, 0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1,
tensor([0, 1, 1, 1, 1, 0, 1,
0, 0,
黑
1, 0, 1, 1, 1, 1, 0, 0]))
(tensor([[[-0.1611, 0.9182, -0.3419, ..., 0.6323, -0.2013, 0.0184],
[-0.1224, 0.7706, -0.2386, ..., 0.7925, 0.0444, 0.2160],
[-0.0301, 0.6867, -0.1510, ..., 0.9140, 0.0308, 0.2611],
...,
[ 0.3662, -0.4925, 1.2332, ..., 0.7741, -0.1007, -0.3099],
[-0.0932, -0.8494, 0.6586, ..., 0.1235, -0.3152, -0.1635],
[ 0.5306, -0.5510, 0.3105, ..., 1.2631, -0.5882, -0.1133]]]),
tensor([1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0,
0, 0, m
1, 0, 0, 1, 1, 1, 0, 0])) . co
a
train_data_len: 2960
e im
valid_data_len: 1000 th
w.i
ww
员
序
编写训练和验证函数: 马
程
黑
import torch.optim as optim
def train(train_data_labels):
"""
训练函数在这个过程中将更新模型参数并收集准确率和损失
description: , ,
训练数据和标签的生成器对象
:param train_data_labels:
整个训练过程的平均损失之和以及正确标签的累加数
:return:
"""
# 定义训练过程的初始损失和准确率累加数
train_running_loss = 0.0
train_running_acc = 0.0
m
# 循环遍历训练数据和标签生成器每个批次更新一次模型参数
,
a. co
im
for train_tensor, train_labels in train_data_labels:
初始化该批次的优化器
#
it
h e
optimizer.zero_grad()
w .
使用微调网络获得输出
# ww
员
train_outputs = net(train_tensor)
得到该批次下的平均损失
#
程
序
马
train_loss = criterion(train_outputs, train_labels)
将该批次的平均损失加到
# 黑中
train_running_loss
train_running_loss += train_loss.item()
损失反向传播
#
train_loss.backward()
优化器更新模型参数
#
optimizer.step()
将该批次中正确的标签数量进行累加以便之后计算准确率
# ,
train_running_acc += (train_outputs.argmax(1) ==
train_labels).sum().item()
return train_running_loss, train_running_acc
m
. co
def valid(valid_data_labels): ma
""" h ei
t
验证函数在这个过程中将验证模型的在新数据集上的标签
description: , w.
i , 收集损失和准确
率 ww
:param valid_data_labels: 验证数据和标签的生成器对象
员
:return: 整个验证过程的平均损失之和以及正确标签的累加数
序
程
""" 马
# 定义训练过程的初始损失和准确率累加数黑
valid_running_loss = 0.0
valid_running_acc = 0.0
# 循环遍历验证数据和标签生成器
for valid_tensor, valid_labels in valid_data_labels:
不自动更新梯度
#
使用微调网络获得输出
#
valid_outputs = net(valid_tensor)
得到该批次下的平均损失
#
valid_loss = criterion(valid_outputs, valid_labels)
m
将该批次的平均损失加到
# 中
valid_running_loss .co
a
valid_running_loss += valid_loss.item() im
将该批次中正确的标签数量进行累加以便之后计算准确率
# ,
it
h e
w .
valid_running_acc += (valid_outputs.argmax(1) ==
valid_labels).sum().item() ww
员
return valid_running_loss, valid_running_acc
序
程
马
黑
调用并保存模型:
if __name__ == "__main__":
# 设定数据路径
train_data_path = "./cn_data/SST-2/train.tsv"
valid_data_path = "./cn_data/SST-2/dev.tsv"
# 定义交叉熵损失函数
criterion = nn.CrossEntropyLoss()
# 定义优化方法
SGD om
.c
optimizer = optim.SGD(net.parameters(), alr=0.001, momentum=0.9)
定义训练轮数 i m
# e
epochs = 4 i th
定义批次样本数量 .
#
www
batch_size = 16
进行指定轮次的训练员
# 序
程
for epoch in range(epochs):
打印轮次马
# 黑
print("Epoch:", epoch + 1)
通过数据加载器获得训练数据和验证数据生成器以及对应的样本数量
# ,
train_data_labels, valid_data_labels, train_data_len, \
valid_data_len = data_loader(train_data_path, valid_data_path,
batch_size)
调用训练函数进行训练
#
train_running_loss, train_running_acc = train(train_data_labels)
调用验证函数进行验证
#
valid_running_loss, valid_running_acc = valid(valid_data_labels)
计算每一轮的平均损失
# 和是每个批次
, train_running_loss valid_running_loss
的平均损失之和 o m
# 因此将它们乘以batch_size就得到了该轮的总损失 a .,c除以样本数即该轮次的平均损失
m i
train_average_loss = train_running_loss
t he * batch_size /
train_data_len i
w w.
valid_average_loss = valid_running_loss * batch_size /
valid_data_len
w
员
序
和程是每个批次的正确标签累加和
# train_running_acc valid_running_acc
马 ,
# 因此只需除以对应样本总数即是该轮次的准确率
黑
train_average_acc = train_running_acc / train_data_len
valid_average_acc = valid_running_acc / valid_data_len
# 打印该轮次下的训练损失和准确率以及验证损失和准确率
print("Train Loss:", train_average_loss, "|", "Train Acc:",
train_average_acc)
print("Valid Loss:", valid_average_loss, "|", "Valid Acc:",
valid_average_acc)
print('Finished Training')
# 保存路径 co
m
MODEL_PATH = './BERT_net.pth' a .
# 保存模型参数 e im
torch.save(net.state_dict(), MODEL_PATH) it
h
.
print('Finished Saving') www
员
序
程
马
黑
输出效果:
Epoch: 1
Train Loss: 2.144986984236597 | Train Acc: 0.7347972972972973
Valid Loss: 2.1898122818128902 | Valid Acc: 0.704
Epoch: 2
Train Loss: 1.3592962406135032 | Train Acc: 0.8435810810810811
Epoch: 3 om
.c
Train Loss: 1.5507876996199943 | Train Acc: a0.8439189189189189
m
Valid Loss: 1.8626576719331536 | Valid Acc:
h ei 0.795
t
Epoch: 4
w .i Acc: 0.9081081081081082
ww
Train Loss: 0.7825378059198299 | Train
员
Finished Training 序
Finished Saving 程
马
黑
加载模型进行使用:
if __name__ == "__main__":
MODEL_PATH = './BERT_net.pth'
# 加载模型参数
net.load_state_dict(torch.load(MODEL_PATH)) m
. co
酒店设备一般，套房里卧室的不能上网，要到客厅去。 a
m
# text = "
房间应该超出平米是同级酒店中少有的大重装之后设备也不错."h ei ; "
text = "
输入文本为
30 , HK
. it ,
print(" :", text) w
with torch.no_grad(): ww
员
output = net(get_bert_encode(text))
从中取出最大值对应的索引序
# output 程
print("预测标签为黑 :", 马
torch.argmax(output).item())
输出效果:
输入文本为: 房间应该超出30平米,是HK同级酒店中少有的大;重装之后,设备也不错.
预测标签为: 1
输入文本为: 酒店设备一般，套房里卧室的不能上网，要到客厅去。
预测标签为: 0
c om
a.
eim
小节总结 ww
w . ith
学习了指定任务类型的微调脚本: 程序员
huggingface研究机构向我们提供了针对
黑
马 GLUE数据集合任务类型的微调脚本, 这些微调脚
本的核心都是微调模型的最后一个全连接层.
通过简单的参数配置来指定GLUE中存在任务类型(如: CoLA对应文本二分类, MRPC对应
句子对文本二分类, STS-B对应句子对文本多分类), 以及指定需要微调的预训练模型.
学习了指定任务类型的微调脚本使用步骤:
第一步: 下载微调脚本文件
第二步: 配置微调脚本参数
第三步: 运行并检验效果 . co
m
i ma
the
i
学习了通过微调脚本微调后模型的使用步骤 w.
ww :
第一步: 在https://huggingface.co/join
序
员上创建一个帐户
程
第二步: 在服务器终端使用黑 transformers-cli登陆
马
第四步: 使用pytorch.hub加载模型进行使用
学习了通过微调方式进行迁移学习的两种类型:
类型一: 使用指定任务类型的微调脚本微调预训练模型, 后接带有输出头的预定义网络输
出结果.
om
类型二: 直接加载预训练模型进行输入文本的特征表示 a . c, 后接自定义网络进行微调输出结
果. eim
th i
w w.
w
员
学习了类型一实战演示: 程
序
使用文本二分类的任务类型黑
马
SST-2的微调脚本微调中文预训练模型, 后接带有分类输出头
的预定义网络输出结果. 目标是判断句子的情感倾向.
准备中文酒店评论的情感分析语料, 语料样式与SST-2数据集相同, 标签0代表差评, 标签1
好评.
语料存放在与glue_data/同级目录cn_data/下, 其中的SST-2目录包含train.tsv和dev.tsv
学习了类型二实战演示: m
. co
直接加载预训练模型进行输入文本的特征表示, 后接自定义网络进行微调输出结果.
ma
ei
th
w.i
ww
员
序
程
马
黑

迁移学习 - 迁移学习

Uploaded by

Copyright:

Available Formats

You might also like

迁移学习 - 迁移学习

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

迁移学习 - 迁移学习

Uploaded by

Copyright:

Available Formats

第二章:迁移学习

''' Script for downloading all GLUE data.'''

def download_and_extract(task, data_dir):

with open(mrpc_train_file, encoding="utf8") as data_fh, \

with open(mrpc_test_file, encoding="utf8") as data_fh, \

Downloading and extracting CoLA...

train.tsv中的数据内容共分为2列, 第一列数据代表具有感情色彩的评论文本; 第二列数

explosives for sale .

train.tsv中的数据内容共分为5列, 第一列数据, 0或1, 代表每对句子是否具有相同的含

index genre filename year old_index source1 source2 sentence1

10 main-captions MSRvid 2012test

id qid1 qid2 question1 question2 is_duplicate

index promptID pairID genre sentence1_binary_parse

train.tsv中的数据内容共分为12列, 第一列代表文本数据索引; 第二列和第三列数据分

index sentence1 sentence2 label

index sentence1 sentence2 label

index question sentence

编码器具有18个隐层, 输出1024维张量, 16个自注意力头, 共257M参数

xlm-mlm-en-2048: 编码器具有12个隐层, 输出2048维张量, 16个自注意力头, 在英文文本

不同类型的'头', 可以使预训练模型输出指定的张量维度. 如使用'分类模型头', 则输出尺寸为

不带头的模型输出结果: tensor([[[ 0.5421, 0.4526, -0.0179, ..., 1.0447,

带分类模型头的模型输出结果: (tensor([[-0.0649, -0.1593]]),)

print("带问答模型头的模型输出结果:", (start_logits, end_logits))

第二步: 在服务器终端使用transformers-cli登陆 com

第四步: 使用pytorch.hub加载模型进行使用, 更多信息请参考2.4 加载和使用预训练模型

# 通过 torch.hub 获得已经训练好的 bert-base-chinese 模型

tensor([[0.3279, 0.2519]], grad_fn=<ReluBackward0>)

def data_loader(train_data_path, valid_data_path, batch_size):

You might also like