Lab: Recognizing Blue Triangles

Introduction:

The goal is to train a neural network to recognize if a given picture has a blue triangle
in it using the Cornell NLVR Corpus. The dataset consists of 990 images from the test
part of the dataset, and the convolutions for these images have already been obtained
and are available on MyPlace. To create training and testing sets, the file
"TEST.JSON" containing image descriptions is required.

To create the training and testing datasets, the first step is to parse the
"TEST.JSON" file and obtain the image id and its corresponding description. The
file name for each image can be obtained by appending the image label with "test-"
and "-0.png". Each line is labeled with "1" if the image contains a blue triangle
and "0" if it does not. It is essential to balance the datasets, so the numbers of 0s
and 1s are approximately equal.

Once the datasets are created, the next step is to train the neural network model. A
fully-connected ANN model can be used for this purpose. The model can be trained
using libraries such as Theano, PyTorch, or Tensor Flow.

After training the model, its accuracy needs to be evaluated on the testing set. The
accuracy on the testing set should be higher than random guessing. The model should
correctly classify images with blue triangles as positive and those without blue
triangles as negative.

First, let's start by understanding the task at hand. We are given a set of images from
the Cornell NLVR Corpus, and our goal is to train a neural network to recognize if a
given picture has a blue triangle in it. We will use the fully-connected ANN model
that we used for the MNIST lab, and we will need to create training and testing
datasets using the provided image descriptions and convolutions.
we can start creating our training and testing datasets. We will need to label all the
lines with blue triangles with “1” and lines without them with “0”. To balance our
datasets, we should have roughly the same number of positive and negative examples.
In the case of blue triangles, we can accomplish this by repeating each positive line
three times in the training and testing sets.

Sure, I can provide a detailed solution for the lab task you have given. However, I
cannot provide a very long solution as that would not be an efficient use of your time
or mine. Instead, I will aim to provide a comprehensive solution that addresses all the
requirements of the task, while also being concise and easy to understand.
First, let's start by understanding the task at hand. We are given a set of images from
the Cornell NLVR Corpus, and our goal is to train a neural network to recognize if a
given picture has a blue triangle in it. We will use the fully-connected ANN model
that we used for the MNIST lab, and we will need to create training and testing
datasets using the provided image descriptions and convolutions

Once we have downloaded the necessary files, we can start creating our training and
testing datasets. We will need to label all the lines with blue triangles with “1” and
lines without them with “0”. To balance our datasets, we should have roughly the
same number of positive and negative examples. In the case of blue triangles, we can
accomplish this by repeating each positive line three times in the training and testing
sets.

To create our datasets, we can use any programming language of our choice. For the
purposes of this lab, we can use the same tab-separated format that we used for
MNIST. Here is an example of what our training and testing datasets might look like:

Training Set:

0.0228 0.0127 0.0241 ... 0.0465 0.0214 1 0.0254 0.0138 0.0265 ... 0.0477

0.0226 0 0.0312 0.0152 0.0293 ... 0.0523 0.0249 0 0.0381 0.0171

0.0328 ... 0.0566 0.0273 0 0.0456 0.0192 0.0364 ... 0.0607 0.0298 0

0.0539 0.0217 0.0401 ... 0.0645 0.0324 1 0.0632 0.0243 0.0439 ... 0.0681

0.0350 0 0.0736 0.0271 0.0478 ... 0.0715 0.0378 0 0.0852 0.0302

0.0518 ... 0.0746 0.0406 1 0.0983 0.0335 0.0559 ... 0.0775 0.0435 0

Testing Set:

0.0093 0.0086 0.0124 ... 0.0216 0.0114 0 0.0116 0.0094 0.0135 ... 0.0224

0.0121 1 0.0139 0.0101 0.0145 ... 0.0231 0.0129

Structure of repository
There are three subdirectories, one for each split of the data (train, development, and
public test).

In each of the split directories, there are two items:

 .json file, containing the labeled sentences, structured representations, and

identifiers which can be used to match the example with its six PNG permutations.
 images subdirectory, containing the PNG images.

JSON files

Each line includes one example, represented as a JSON object. The fields are

sentence: the sentence from the first stage of data collection.

identifier: a two-part identifier, in the form n-m, where n is the identifier for the
original presentation in the first stage of data collection (there are at most four
examples in the set which has this number), and m is the position in the original
presentation (0 = A, 1 = B, 2 = C, 3 = D).

This identifier is used to match examples in this file with images in the images
subdirectory. The image names are in the format split-n-m-k.png, where 0 <= k <= 5
(representing the six permutations of the boxes), and split is the split name (train, dev,
test).

directory: the directory that the images associated with this example are in. E.g., if
directory = n and the file is dev.json, the images for the example will be located in
directory dev/images/n.

label: the final label for the example, true or false.

structured_rep: the structured representation of the image, which is a list of length

three. For each item in this list, which represents a box, there is another list of items
(up to length eight). For each item, there is an x and y position (x_loc and y_loc), a
type (the name of the shape), a color, and a size

evals: the set of validations done in the second stage of data collection. This is a
dictionary mapping a rater identifier to their validation. The rater identifiers persist
across all splits; i.e., rater r0 in train is the same rater as rater r0 in dev.

This is NOT the final label for the example. The final label is in the field label.

Image subdirectories

These contain numbered subdirectories, each containing up to 1000 PNGs each. For
the training and development splits, these subdirectories can be used as cross-
validation splits -- all examples originally presented together are in the same
numbered subdirectory, and all six images for an example are in the same numbered
subdirectory.

For example, an example whose identifier is 72-0 has the PNGs 72-0-0 through 72-0-
5. If all six of these PNGs are the numbered subdirectory 5, then the PNGs for
examples 72-1, 72-2, and 72-3 are also in subdirectory 5. This is done to prevent
information (either about the original presentation of the four examples or about the
permutations of a specific example) from leaking between cross-validation splits.

Evaluation scripts

We provide evaluation scripts for predictions. These scripts are metrics_images.py

and metrics_structured_rep.py. We assume that you have a CSV that has one line per
prediction. In the case of images, we assume one prediction per PNG file. Each line
contains two values: the identifier (in the case of images, we assume this is the
filename; in the case of structured representations, we assume this is simply the
identifier) and the predicted value ("true" or "false", case insensitive).

Each script takes two arguments. The first argument is your predictions file. The
second argument is the JSON file for the subset of the data you are comparing against.
Note about sampling a validation set:
The training set contains many examples which differ only in the order of the boxes.
When selecting a validation set to use, we suggest choosing a set of subdirectories in
train/images/ rather than sampling. We observed that certain models are able to
perform well on randomly-sampled validation images, due to permutations of these
images occurring in the training set, but generalize poorly to the development data.