Scribd Logo
Upload

Welcome to Scribd!

  • Data Goodness: Mostly in Black and White by Dom

    Uploaded by

    Syed Zulqarnain Haider
    6 views7 pages
    The document discusses the importance of properly storing and managing research data. It notes that lost data results in no research, publications, jobs or PhDs. It recommends storing large data on servers which provide backups, mirroring and parity protection to prevent data loss from desktop computer failures. The document emphasizes curating data as work is done by using meaningful folder names, README files and keeping data tidy. It stresses testing software and using versioning tools to ensure scripts work properly and data is safely stored.

    Original Description:

    Original Title

    Data Goodness

    Copyright

    © Attribution Non-Commercial (BY-NC)

    Available Formats

    PPT, PDF, TXT or read online from Scribd

    Did you find this document useful?

    Is this content inappropriate?

    Report this Document
    The document discusses the importance of properly storing and managing research data. It notes that lost data results in no research, publications, jobs or PhDs. It recommends storing large data on servers which provide backups, mirroring and parity protection to prevent data loss from desktop computer failures. The document emphasizes curating data as work is done by using meaningful folder names, README files and keeping data tidy. It stresses testing software and using versioning tools to ensure scripts work properly and data is safely stored.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PPT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as ppt, pdf, or txt
    6 views7 pages

    Data Goodness: Mostly in Black and White by Dom

    Uploaded by

    Syed Zulqarnain Haider
    The document discusses the importance of properly storing and managing research data. It notes that lost data results in no research, publications, jobs or PhDs. It recommends storing large data on servers which provide backups, mirroring and parity protection to prevent data loss from desktop computer failures. The document emphasizes curating data as work is done by using meaningful folder names, README files and keeping data tidy. It stresses testing software and using versioning tools to ensure scripts work properly and data is safely stored.

    Copyright:

    Attribution Non-Commercial (BY-NC)
    Download as PPT, PDF, TXT or read online from Scribd
    Flag for inappropriate content
    Download as ppt, pdf, or txt
    of 7

    Data goodness

    Mostly in black and white By Dom

    You must love your data!


    Lost data :
    Current imaging data in BRIC cost ~5.1M, just for scanning costs! (2011)

    no research
    no publications
    no jobs no PhDs! Sad Dom

    Look after your data!


    It looks after you
    Happy Dom

    Data Storage
    Home directories:
    ISIS home, U Home Not for large amounts of imaging data

    Projects directory
    ISIS, V: Big stuff goes here

    If you require large amounts of space


    E.g. > 50 GB
    LET ME KNOW IN ADVANCE!

    Server goodness
    Why is the server a good place to store data?
    Mirror and parity - some errors - data can be easily recovered

    BACKUPS:
    Tape backups, daily - 1 month retention if you have funding, processed data can be mirrored off site raw data is always mirrored offsite (ECDF) by default

    Desktop PC's
    not reliable - no mirroring, no parity - some errors - data is lost (Often all of it) Network backups often fail
    Machines turned off, Network busy moving to a new system when I get time!

    Data love
    Curation: Do this as you work!
    Plan your data use

    Use meaningful folder names Make 'README.txt' files with dates, names of students/employees involved, references to software, scripts and versions, purpose of experiment/processing. Be tidy with your data - tidy up occasionally Friday afternoon - quick tidy up Big tidy up at end of experiment/ project/ phase/ year
    BE CAREFUL, dont rush

    Data, spreadsheets, databases


    Anonymisation *** Repatriation keys***

    Code and Scripts


    Coding:
    Testing
    Make sure that the software you are using does exactly what you think it does! Check every step for every image!

    Do not use hard coded paths


    Use versioning software (ECDF)

    Safe data is Happy data!

    You might also like