开源软件名称：

开源软件地址：

开源编程语言：

开源软件介绍：

Parsons School of Design, The New School
School of Art, Media and Technology
Master of Science, Data Visualization

Data Structures

PGDV 5110 Syllabus, Fall 2021

Wednesdays, 12:10pm - 2:50pm
66 5th Ave | Room: 710
Nearest designated eating area: 2 W. 13th Street, Room 406

Faculty: Aaron Hill | Office Hours: Fridays 3:00pm - 4:00pm: no appointment needed; contact info and Zoom info in Canvas

Teaching Assistant: Jessie Han | Office Hours: Mondays 11:00am - 12:00pm and Thursdays 4:00pm - 5:00pm: no appointment needed; contact info and Zoom info in Canvas

I encourage use of the Canvas discussion board for collaboration and work on assignments, including the 10 weekly and 3 final assignments. All questions/issues about the assignments should be posted here, building a shared knowledge base in the process. Questions/issues raised in office hours should also be posed here prior to the office hour meeting and the answer/resolution will be posted by the person who originally posed the question.

Course description: Curating raw data presents issues of technology, speed, and efficiency, as well as broader ethical considerations of what it means to represent and make conclusions about groups and individuals from their data. This course covers the database, semi-structured data, and unstructured data. Students will gain familiarity with underlying data structures; techniques and tools, including acquisition, augmentation, and restructuring; data storage and aggregation; access to parallel and distributed computing; high-volume data, disparate sources, and performance; and streaming data, real time, and dynamic queries.

Prerequisites: Basic knowledge of JavaScript, HTML/CSS, Linux command line, Git, and GitHub.

Course Overview

Course Outline

September 1 (Week 1)

Introduction to data structures.

Weekly Assignment 1, due Tuesday 9/7 at 6:00pm:
You will "get" some HTML files, save them, and study their structure.

Documentation:

• What is npm?
• Node got module
• Node fs module

September 8 (Week 2)

The web as a database. "Raw data."

Read:

• Gitelman, Introduction and Color Plates
• Hills, Part One (pp. 17-42)
• Express/Node Introduction (read carefully the "Node" introduction and quickly skim the "Express" introduction)

Weekly Assignment 2, due Tuesday 9/14 at 6:00pm:
You will parse one of the HTML files you saved last week and log essential data to the console.

Documentation:

• Node cheerio module
• Introduction to the DOM

September 15 (Week 3)

Fundamental data structures. Data augmentation and integration. GitHub for open source collaboration.

Read:

• Gitelman, Chapter 5
• Hills, Chapters 10, 11, and 12
• Javascript data types and structures
• Working with objects (JavaScript)

Weekly Assignment 3, due Tuesday 9/21 at 6:00pm:
Using data you parsed last week, you will interface with TAMU GeoServices to augment the data you collected.

Documentation:

• TAMU GeoServices
• TAMU GeoServices Geocoding APIS
• Node Async module
• Tutorial: creating and using environment variables in Linux.

September 22 (Week 4)

Data models I: SQL database.

Read:

• Gitelman, Chapter 2
• Hills, Chapters 13, 14, 15, and 16

Weekly Assignment 4.1, due Tuesday 9/28 at 6:00pm:
You will design a schema and populate a representation of your database (in Excel) with the first three rows of your AA data. (2.5 points)

Documentation:

• SQL tutorial
• PostgreSQL data types
• PostgreSQL documentation
• Node pg module

September 29 (Week 5)

Data models I: NoSQL database.

Read:

• Gitelman, Chapter 6
• Hills, Chapter 17

Weekly Assignment 5.1, due Tuesday 10/5 at 6:00pm:
You will design a (non)schema and populate a representation of your database (in Word) with the first three "documents" of your Process Blog data. (2.5 points)

Final Assignment 1 distributed: Map of AA meetings in Manhattan.

Final Assignment 2 distributed: "Process Blog," a source of semi-structured, qualitative data.

October 6 (Week 6)

The Query. Indexes, joins, query optimization, and data restructuring. The E-R model. Data provision.

Read:

• Hills, Chapter 5
• Database as Symbolic Form, Lev Manovich
• Gitelman, Data Flakes: An Afterword to "Raw Data" Is an Oxymoron

Weekly Assignments 4.2 and 5.2, due Tuesday 10/19 at 6:00pm:
4.2: You will create a new managed PostgreSQL database, and write SQL code to populate your database with your AA data. (2.5 points)

5.2: You will create a new managed DynamoDB database, and write code to populate your database with your "Process Blog" data. (2.5 points)

October 13 (Week 7)

Data cleaning. Logging issues.

Data cleaning workshop.

Weekly Assignment 6, due Tuesday 10/19 at 6:00pm:
You will write queries for both the SQL and NoSQL databases you have created, filtering, aggregating, and/or restructuring the data in the process.

Read:

• Gitelman, Chapter 4
• For Big-Data Scientists, 'Janitor Work' Is Key Hurdle to Insights, The New York Times, August 17, 2014

Documentation:

• GitHub issues

October 20 (Week 8)

Note: IEEE VIS virtual conference is October 24-29

Sensors. Data gathering and surveillance.

Weekly Assignment 7, due Tuesday 11/2 at 6:00pm:
Finish parsing and cleaning the rest of the AA data in all ten Manhattan "zones", and update/replace the data in your Postgres table(s).

Read:

• Gitelman, Chapter 7

Documentation:

• Particle documentation [ Photon | Electron ]

October 27 (Week 9)

[slides]

IoT.

Note: IEEE VIS virtual conference is October 24-29

DUE Wednesday 11/4 at 12:00pm: IEEE VIS conference presentation summary

Read:

• Gitelman, Chapter 3
• Klepman, Chapter 2

November 3 (Week 10)

From database to visual representation. Interface design. Workshop by Jessie Han: How to create filters on the front end.

Weekly Assignment 8, due Tuesday 11/9 at 6:00pm:
Switching gears from previous assignments, you will use an IoT development board and sensors to collect and log data to your local computer. You will also sketch your first iteration of the design of an interactive visualization of the data you will collect from the sensor.

Final Assignment 3 distributed: Surveilling your life.

Debate: SQL vs. NoSQL 1/2 | 3/4]

Read:

• Gitelman, Chapter 1
• New York Times: Slump in Air Travel Hindered Weather Forecasting, Study Shows

November 10 (Week 11)

Serverless computing. Workshop by Jessie Han: Use python + flask to host a website.

Weekly Assignment 9, due Tuesday 11/16 at 6:00pm:
You will begin writing sensor data to a database.

Read:

• REDESIGNING DESIGN / JOS DE MUL

Documentation:

• What is Node.js and Express?
• Express
• Handlebars

November 17 (Week 12)

Discussion: Reliability. Lab.

Coding demo: making a map in mapbox

Weekly Assignment 10, due Tuesday 11/30 at 6:00pm:
You will submit the final designs for the interfaces for each of your three data sources.

Read:

• Klepmann, Chapter 1. Read from the beginning through the "Reliability Section."
• How the internet works

Groups for Reliability, Scalability, and Maintainability Discussions:

• Reliability (Dec. 1): Yang, Molly, Gisli, Wan, Wenqin \
• Scalability (Dec. 8): Yoana, Juliet, Omar, Yann \
• Maintainability (Dec. 15): Jeremy, Chay, Esther, Joo, Kemi

November 24: No class, Thanksgiving holiday

December 1 (Week 13)

Discussion: Scalability. When do you need a database(s)? When do you not? Lab.

Read:

• Klepmann, Chapter 1. Read the "Scalability Section."
• MapReduce: Simplified Data Processing on Large Clusters, Jeffrey Dean and Sanjay Ghemawat
• immens paper

December 8 (Week 14)

Discussion: Maintainability. Lab.

Read:

• Klepmann, Chapter 1. Read the "Maintainability Section" and through the end of the chapter.
• Gitelman, Chapter 8
• What is code?

Final Assignments 1, 2, and 3 due Tuesday 12/14 at 6:00pm:

December 15 (Week 15)

In the final class meeting, you will present your work in groups, by theme. Group and theme assignments:

• Reliability
• Scalability
• Maintainability

Terminating and cleaning up your AWS resources and credentials. [checklist]

Learning Outcomes

By the successful completion of this course, students will be able to demonstrate:

• Proficiency with various techniques to acquire, process, restructure, and analyze raw data.
• Ability to program and work within structured and unstructured databases to deliver and exchange data.
• Knowledge of methods for web development, hosting, and processing of data for visualizations, often from very large data sources.
• An understanding of computer science principles that support computational efficiency and speed.
• An understanding of the limitations and ethical considerations when using data to represent individuals and groups.

Final Grade Calculation

Weekly assignments are worth 5 points each, unless otherwise indicated. They are due 6:00pm the Tuesday after they are assigned, unless otherwise indicated. The quality of your documentation is a part of the grade for every assignment, including all the weekly and final assignments. All documentation must be kept in your (well organized!) data-structures repository, using markdown.

Late assignments:

If an assignment is less than 48 hours late, the maximum possible grade is 75%.
If an assignment is 48-96 hours late, the maximum possible grade is 50%.
If an assignment is 96-144 hours late, the maximum possible grade is 25%.
If an assignment is more than 144 hours late, no credit will be given.

Required Reading

"Raw Data" Is an Oxymoron, Lisa Gitelman, 2013. This book is available electronically through course reserves.
NoSQL and SQL Data Modeling: Bringing Together Data, Semantics, and Software, Ted Hills, 2016. This book is not available through course reserves and should be purchased (~$27 on Amazon).
Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems, Martin Kleppmann, 2017. This book is available electronically through course reserves.

Recommended/Optional Reading

Database Aesthetics : Art in the Age of Information Overflow, Victoria Vesna. This book is available electronically through course reserves.
The Stuff of Bits : An Essay on the Materialities of Information, Paul Dourish. This book is available electronically through course reserves.

Office Hours (Zoom; no appointment needed)

Fridays, 3:00pm - 4:00pm

Materials and Supplies

In this course we will rely exclusively on open source software and we will work with the software by writing code: JavaScript, SQL, and some basic Linux.

We will also rely on cloud-based services to provide infrastructure, platforms, and computational power:

• Amazon Web Services (or similar) for computing infrastructure, platforms, and computational power.
• AWS Cloud 9 for a working environment to use for prototyping and experimentation.

Required:

• A modern web browser. Preferred: Google Chrome. Acceptable: Firefox. Not acceptable: Internet Explorer, Safari.
• Bring your laptop to every class.
• A (free) GitHub account
• A (free) Amazon Web Services (AWS) account
• A (free) AWS Educate account
• A microcontroller and sensors (as assigned; you will be given a list of the things you'll need to purchase, ~$50)
• It is also helpful to have a text editor installed locally. Sublime and Atom are popular choices. Atom integrates well with GitHub.
• If you decide to work locally (instead of in Cloud9), you may need to install and maintain additional software such as Git (installed with Xcode in OSX), Node.js, npm (which is installed with Node.js), and other software.

Resources

The university provides many resources to help students achieve academic and artistic excellence. These resources include:

• The University (and associated) Libraries: http://library.newschool.edu
• The University Learning Center: http://www.newschool.edu/learning-center
• University Disabilities Service: http://www.newschool.edu/student-disability-services/

In keeping with the university's policy of providing equal access for students with disabilities, any student with a disability who needs academic accommodations must contact SDS. There are several ways for students to contact the office: via email at [email protected], through the Starfish service catalog, or by calling the office at 212.229.5626. A self-ID form can also be completed on the SDS webpage at www.newschool.edu/student-disability-services. Once you contact the office, SDS staff will arrange an intake appointment to discuss your concerns and, if appropriate, provide you with accommodation notices to give to me. Please note that faculty will not work unilaterally with students to provide accommodations. If you inform me of a disability but do not provide any official notification, I must refer you to SDS.

Making Center

The Making Center is a constellation of shops, labs, and open workspaces that are situated across the New School to help students express their ideas in a variety of materials and methods. We have resources to help support woodworking, metalworking, ceramics and pottery work, photography and film, textiles, printmaking, 3D printing, manual and CNC machining, and more. A staff of technicians and student workers provide expertise and maintain the different shops and labs. Safety is a primary concern, so each area has policies for access, training, and etiquette with which students and faculty should be familiar. Many areas require specific orientations or trainings before access is granted.

Grading Standards

A Work of exceptional quality
A- Work of high quality
B+ Very good work
B Good work; satisfies course requirements Satisfactory completion of a course is considered to be a grade of B or higher.
B- Below-average work
C+ Less than adequate work
C Well below average work
C- Poor work; lowest possible passing grade
F Failure
GM Grade missing for an individual

Grades of D are not used in graduate level courses.

Grade of W:
The grade of W may be issued by the Office of the Registrar to a student who officially withdraws from a course within the applicable deadline. There is no academic penalty, but the grade will appear on the student transcript.

Unofficial Withdrawal (Grade of Z):
This grade is to be assigned to students who have never attended or stopped attending classes. Exceptions can be made if the student has completed enough work to warrant a grade (including a failing grade), and arrangements have been made with the instructor(s) and the Dean’s Office prior to grade submission. The Z grade does not calculate into the student’s GPA. Though a Z grade does not have a failing penalty it still carries a myriad of consequences for students on visas or receiving financial aid. Only issue the Z grade when a student meets the above criteria.

Grades of Incomplete:
The grade of I, or temporary incomplete, may be granted to a student under unusual and extenuating circumstances, such as when the student's academic life is interrupted by a medical or personal emergency. This mark is not given automatically but only upon the student's request and at the discretion of the instructor. A Request for Incomplete form must be completed and signed by student and instructor. The time allowed for completion of the work and removal of the I mark will be set by the instructor with the following limitations:

Work must be completed no later than one year following the end of the class. Grades of I not revised in the prescribed time will be recorded as a final grade of N by the Registrar’s Office.

College, School, Program and Class Policies:

A comprehensive overview of policy may be found under Policies: A to Z. Students are also encouraged to consult the Academic Catalog for Parsons.

Canvas

Use of Canvas may be an important resource for this class. Students should check it for announcements before coming to class each week.

Electronic Devices

The use of electronic devices (phones, tablets, laptops, cameras, etc.) is permitted when the device is being used in relation to the course's work. All other uses are prohibited in the classroom and devices should be turned off before class starts.

Responsibility

Students are responsible for all assignments, even if they are absent. Late assignments, failure to complete the assignments for class discussion and/or critique, and lack of preparedness for in-class discussions, presentations and/or critiques will jeopardize your successful completion of this course.

Active Participation and Attendance

Class participation is an essential part of class and includes: keeping up with reading, assignments, projects, contributing meaningfully to class discussions, active participation in group work, and coming to class regularly and on time.

Parsons' attendance guidelines were developed to encourage students’ success in all aspects of their academic programs. Full participation is essential to the successful completion of coursework and enhances the quality of the educational experience for all, particularly in courses where group work is integral; thus, Parsons promotes high levels of attendance. Students are expected to attend classes regularly and promptly and in compliance with the standards stated in this course syllabus.

While attendance is just one aspect of active participation, absence from a significant portion of class time may prevent the successful attainment of course objectives. A significant portion of class time is generally defined as the equivalent of three weeks, or 20%, of class time. Lateness or early departure from class may be recorded as one full absence. Students may be asked to withdraw from a course if habitual absenteeism or tardiness has a negative impact on the class environment.

I will assess each student’s performance against all of the assessment criteria in determining your final grade.