Course Hours: Mon 10.00-12.00, Thu 12.00-14.00
Room: H.208
Weekly Lab: Fri 18.00-20.00, Room H.208
Office Hours: Available upon request
Big Data requires the storage, organization, and processing of data at a scale and efficiency -typically of heterogeneous nature and in streaming flow- that go well beyond the capabilities of conventional information technologies. Such requirements have been first introduced for processing the web, and they are today a common place in many industries. In this respect many traditional assumptions break, new query and programming interfaces are required (Map/Reduce), and new computing models will emerge (Cloud Computing). This course aims to introduce parallel/distributed data processing using the MapReduce (M/R) paradigm and provide insights for developing applications on top of the Hadoop platform.
Big data raises also new challenges in data mining. Given the scale and speed of data that needs to be processed as well the variety of parameters to be taken into account, state of the art machine learning algorithms working offline and expecting homogeneous and clean data are also challenged. There is on ongoing effort to design Big Data Mining algorithms accommodating a parallel/distributed or even a streaming evaluation. Of course such kind of incremental, partial evaluation impacts the quality of obtained statistical models and thus algorithms compromise between quality of the learning and computation time. The course will adopt an algorithmic viewpoint: data mining is about applying algorithms to data, rather than using data to “train” a machine-learning engine of some sort.
The course will consist of lectures based both on textbook material (freely-available for download on the Web) and scientific papers. It will also include programming assignments that will provide students with hands-on experience on building data-intensive applications using existing Big Data tools and platforms. The intended audience of this course is MSc and PhD students but also practitioners who plan to design or develop state-of-the-art algorithms available today for Big Data analysis.
Lecture 1 :[Thursday:08/02 [12:00-14:00] ] Intro on Big Data & Data Science
[pdf]Lecture 2:Tuesday 13/02 [12:00 -14:00]
Lecture 3:Thursday 15/02 [ 12:00 -14:00 ]
Introduction to Scalable Data Analytics using Apache Spark [pdf]
Lab 1 (16/02): MapReduce Programming Fri 18.00-20.00 [pdf]
Assignment 1 Spark Intro (Deadline 01/03)
Finding Similar Sets [pdf]
Introduction to Scalable Data Analytics using Apache Spark
Lab 2 (23/02): Programming in Spark Fri 18.00-20.00
Relational Data Processing [pdf]
Frequent Itemsets & Association Rules [pdf]
Lab 4 (08/03): Intro to Spark Streaming [pdf]
Analysing Data Streams
Assignment 2 (Deadline : 29/03)
Lab 5 (15/03): Intro to DataFrames and Spark SQL [pdf]
Zoom Meeting https://us04web.zoom.us/j/78394266232?pwd=2rj2hiPcU4lcTDBm6QkCVRimYJX5qW.1 Meeting ID: 783 9426 6232 Passcode: 1234
Data Ethics in Algorithmic Decision Making
Data Ethics in Algorithmic Decision Making
Presentations