Cloud Data Engineering Technologies

Data is displayed for the academic year: 2024./2025.

Lecturers

Asst. Prof.
Marko Horvat
PhD

Course Description

Students will learn the basic processes of data engineering projects, such as data entry, storage, transfer, transformation, cleaning, validation and transfer. Also source code management procedures, serverless programming environments and continuous code integration and delivery procedures which are essential on today’s cloud data engineering projects will be addressed. Through assignments students will gain practical skills in working with the GO programming language, Docker containers and the Kubernetes cluster, MongoDB NoSQL database, GIT Jankins and Terraform.

Prerequisites

Knowledge of basic programming and algorithms and data structures. Construction of a web computer application that includes a client and a server part.

Study Programmes

University undergraduate
[FER3-HR] Computing - study
Skills (3. semester) (5. semester)
[FER3-HR] Electrical Engineering and Information Technology - study
Skills (3. semester) (5. semester)
University graduate
[FER3-HR] Computing - study
Skills (1. semester) (3. semester)
[FER3-HR] Electrical Engineering and Information Technology - study
Skills (1. semester) (3. semester)
[FER3-HR] Information and Communication Technology - study
Skills (1. semester) (3. semester)

Learning Outcomes

  1. Explain the application of cloud data engineering technologies
  2. Apply the basic principles of cloud data engineering technologies
  3. Design, implement and test simpler cloud services and discover errors
  4. Use program code continuous integration and continuous delivery procedures
  5. Use containerization systems in cloud environment

Forms of Teaching

Lectures

Lectures will be held in 8 cycles lasting 2 hours per week in laboratory classrooms at the faculty.

Laboratory

Laboratory exercises will be held in 8 cycles of 2 hours per week in the laboratory classrooms at the Faculty.

Grading Method

Continuous Assessment Exam
Type Threshold Percent of Grade Threshold Percent of Grade
Laboratory Exercises 50 % 40 % 50 % 40 %
Seminar/Project 50 % 40 % 50 % 40 %
Attendance 50 % 20 % 50 % 20 %

Week by Week Schedule

  1. What is a data platform, cloud platforms and building data pipelines
  2. Data sources, transfer of local data to the cloud, retrieval from relational databases, program interfaces and public sources, data ingestion
  3. Using Pub / sub agent for message forwarding and Auro message format, metadata and data origin
  4. Data consolidation and transformation, message schema control, schema catalog construction and how to solve the problem of message schema evolution
  5. Using software containers, managing the Kubernetes cluster, and using the MongoDB NoSQL database
  6. Development of programming interfaces in GO programming language (REST API, Swagger, oAuth authentication), serving through message forwarding agents, data democratization
  7. Code development applications aligned with the DevOps concept, using GIT, Jankins, Terraform, establishing development, test, presentation and production environments in the cloud
  8. Final project
  9. No classes
  10. No classes
  11. No classes
  12. No classes
  13. No classes
  14. No classes
  15. No classes

Literature

Martin Kleppmann (2017.), Designing Data-intensive Applications, Oreilly & Associates Incorporated
John Arundel, Justin Domingus (2019.), Cloud Native DevOps with Kubernetes, O'Reilly Media
Manoj Kukreja, Danil Zburivsky (2021.), Data Engineering with Apache Spark, Delta Lake, and Lakehouse, Packt Publishing Ltd
Alan A. A. Donovan, Brian W. Kernighan (2015.), The Go Programming Language, Addison-Wesley Professional

General

ID 234065
  Winter semester
3 ECTS
L0 English Level
L1 e-Learning
15 Lectures
0 Seminar
0 Exercises
15 Laboratory exercises
0 Project laboratory
0 Physical education excercises

Grading System

90 Excellent
80 Very Good
65 Good
50 Sufficient

Similar Courses