Scribd Logo
Upload

Welcome to Scribd!

  • FALLSEM2021-22 MDI4001 ETH VL2021220104135 Reference Material II 09-Aug-2021 Data2 2

    Uploaded by

    Chidvi Reddy
    3 views8 pages

    Original Title

    FALLSEM2021-22_MDI4001_ETH_VL2021220104135_Reference_Material_II_09-Aug-2021_Data2_2

    Copyright

    © © All Rights Reserved

    Available Formats

    PDF or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    Download as pdf
    3 views8 pages

    FALLSEM2021-22 MDI4001 ETH VL2021220104135 Reference Material II 09-Aug-2021 Data2 2

    Uploaded by

    Chidvi Reddy

    Copyright:

    © All Rights Reserved
    Download as PDF or read online from Scribd
    Flag for inappropriate content
    Download as pdf
    of 8
    Data Cleaning ‘Most Machine Learning algorithms cannot work with missing features, so let’ create a few functions to take care of them. You noticed earlier that the total_bedroons attribute has some missing values, so le’ fix this. You have three options: + Get rid ofthe corresponding districts + Get rid of the whole attribute + Set the values to some value (zero, the mean, the median, et). You can accomplish these easily using DataFrame’s dropna(), drop(), and FULLna() methods: housing. dropna(subset=["total_bedroons"]) # option 1 housing. drap("total_bedroons", axis=i) # option 2 rnedian ~ houstngf"total_becroons"}.medtan() # option 2 noustngl total_bedroons"].FtUIna(medtan, tnplace=True) Ifyou choose option 3, you should compute the median value on the training set, and use it to fil the missing values in the training set, but also dont forget to save the ‘median value that you have computed. You will need it later to replace missing values in the test set when you want to evaluate your system, and also once the system goes live to replace missing values in new data. Scikit-Learn provides @ handy class to take care of missing values: Sinpleinputer. Here is how to use it. First, you need to create a SimpleInputer instance, specifying that you want to replace each attributes missing values with the median of that attribute: fron sklesrn.tnpute tnport Stnplelnputer Lnputer = Sinpletnputerstrategy="nedian") Since the median can only be computed on numerical attributes, we need to create a copy of the data without the text attribute ocean_proxinity housing.nun = hovstng.drop("ocean_ proximity", axis=1) Now you can fit the Unputer instance to the training data using the F.t() method: ‘puter FLe¢housina_nun) ‘The Anputer has simply computed the median of each attribute and stored the result in its statistics_instance variable. Only the total_bedroons attribute had missing values, but we cannot be sure that there won't be any missing values in new data after the system goes live, so its safer to apply the tnputer to all the numerical attributes: a> Lnputer. statistics, array([ 118.51, 34.25, 29. , 2119.5, 433. , 1164. , 408. , 3.5409]) Prepare the Datafor Machine Leaning Agortims | 67 >>> housing_nun.nedian() values array([ 118.51, 34.25, 29. , 2119.5 , 433. , 1164. , 408. , 3.5469]) Now you can use this “trained” imputer to transform the training set by replacing ‘missing values by the learned medians: X « tmputer.transformthousing_nun) ‘The result is a plain NumPy array containing the transformed features. If you want to put it back into a Pandas DataFrame, it’ simple: housing tr d.ataFrane(X, column 2_pur.colurns) Scikit-Learn Design Scikit-Learnis API is remarkably well designed. The main design principles are: + Consistency. All objects share a consistent and simple interface: Estimators. Any object that can estimate some parameters based on a dataset is called an estimator (eg, an inputer is an estimator). The estimation itself is performed by the F1t() method, and it takes only a dataset as a parameter (or two for supervised learning algorithms; the second dataset contains the labels). Any other parameter needed to guide the estimation process is con- sidered a hyperparameter (such as an inputer's strategy), and it must be set as an instance variable (generally via a constructor parameter) — Transformers. Some estimators (such as an imputer) can also transform a dataset; these are called transformers. Once again, the API is quite simple: the transformation is performed by the transform() method with the dataset to transform as a parameter. It returns the transformed dataset, This transforma: tion generally relies on the learned parameters, as is the case for an inputer All transformers also have a convenience method called Fut_transform() that is equivalent to calling Fit() and then transforn() (but sometimes Fit_transforn() is optimized and runs much faster) — Predictors. Finally, some estimators are capable of making predictions given a dataset; they are called predictors. For example, the LinearRegression model in the previous chapter was a predictor: it predicted life satisfaction given a country’s GDP per capita. A predictor has a predict() method that takes a dataset of new instances and returns a dataset of corresponding predictions. t also has a score() method that measures the quality of the predictions given For more detail onthe design principles, se "API design for machin learning software: experiences from the slam projet” Bultic, G. Louppe, M. Blondel, FPedeeoss, A Miller etal. (2013) (68 | Gapter2 End to End Machine Learning Projet a test set (and the corresponding labels in the case of supervised learning algorithms." + Inspection, All the estimator’s hyperparameters are accessible directly via public Instance variables (eg, inputer.strategy), and all the estimator’s learned parameters are also accessible via public instance variables with an underscore suffix (eg., inputer statistics). + Nonproliferation of classes, Datasets are represented as NumPy arrays or SciPy sparse matrices, instead of homemade classes. Hyperparameters are just regular Python strings or numbers + Composition, Existing building blocks are reused as much as possible. For example, itis easy to create a Pipeline estimator from an arbitrary sequence of, transformers followed by 2 final estimator, as we will se. + Sensible defaults. Scikit-Learn provides reasonable default values for most parameters, making it easy to create a baseline working system quickly. Handling Text and Categorical Attributes Earlier we left out the categorical attribute ocean_proximity because it is a text attribute so we cannot compute its median: >>> housing_cat = housing[["ocean_proximity”]] >>> housing_cat.head(i0) ‘ocean proxinty 17606 1H OCEAN 18632 fran sklesrn.preprocessing tmport ordinalencoder de> ordinal_encoder ~ Ordinalencoder() 18 Some predictors alo provide methods to measure the confidence of ther predictions ei avalable since Sik Le Series. factorze() method, 19 This 020. 1f you us an eater version, please consider upgrading, or use Panda Prepare the Datafor Machine Leaning Agortims | 69 >>> housing_cat_encoded = ordinal_encoder.fit_transforn(houstng_cat) >>> housing_cat_encoded[ 19] array({{0-1, 8.1, 1% 7 cn I, a You can get the list of categories using the categories_instance variable. It isa list containing a 1D array of categories for each categorical attribute (in this case, alist containing a single array since there is just one categorical attribute): >>> ordinal_encoder categories [array(('sti OCEAN’, "INLAND", "TSLAND', "NEAR BAY", NEAR OCEAN'], ‘type=object] One issue with this representation is that ML algorithms will assume that two nearby values are more similar than two distant values. This may be fine in some cases (eg, for ordered categories such as “bad’, ‘average’, “good, “excellent’), but itis obviously not the case for the ocean_proxinity column (for example, categories 0 and 4 are clearly more similar than categories 0 and 1). To fix this issue, a common solution is to create one binary attribute per category: one attribute equal to 1 when the category is “<1H OCEAN” (and 0 otherwise), another attribute equal to 1 when the category is, “INLAND” (and 0 otherwise), and so on. This is called one-hot encoding, because only one attribute will be equal to I (hot), while the others will be 0 (cold), The new attributes are sometimes called dummy attributes. Scikit-Learn provides a OneHotEn coder class to convert categorical values into one-hot vectors* a2 from sklearn.preprocessing tmport OneHotEncode’ da cat encoder ~ OneHotEncoder() >>> housing_cat_thot = cat_encoder fit s >>> housing_eat_thot 1651245 sparse matrix of type ‘" with 16512 stored elenents in Conpressed Sparse Row Fornat> ansforn(housing_cat) Notice that the output is a SciPy sparse matrix, instead of a NumPy array. This is very useful when you have categorical attributes with thousands of categories. After one hot encoding we get a matrix with thousands of columns, and the matrix is ull of, zeros except for a single 1 per row. Using up tons of memory mostly to store zeros would be very wasteful, so instead a sparse matrix only stores the location of the non- 2 Before Skt earn 0.2, ould only encode integer categoria values, bu since 0.20 it can also handle other types of inputs, ncuding text categorical apt. 70 | Gapter: End to-End Machine Learning Projet zero elements. You can use it mostly like a normal 2D array. but if you really want to convert it to a (dense) NumPy array, just call the toarray() method: >>> housing_cat_thot.toarray() array([[e1 Bs Bre Bre 8] A, 8, 2, 8, 8.1, 01, 8., 8,8, 1), ble Oe Oe : eo, Je 1, a) Once again, you can get the list of categories using the encoder’s categories_ instance variable: b>> cat_encoder catecories_ [aeray({'

    You might also like