Introduction to Big Data · End Term · Jan 2023 · FN

What differentiates “streaming processing” from “batch processing” in the context of big data?

What are the core components for a Big Data Streaming application?

A big data streaming application that reads from using Kafka as source is observed to be really slow. The Kafka cluster has 2 broker nodes and this application is reading from 1 topic that has 10 partitions. On closer investigation, it was found that Kafka is not scaling to the velocity of input data coming in. How will you scale Kafka further?

You are given the task of improving the performance of a Spark SQL program. You suspect that the culprit is the main transformation job in the program. When you run EXPLAIN on that SQL, you see that Spark wrongly estimates that there are only 10 values for the key being aggregated, whereas in reality the underlying data has a million values for that key. What actions would you perform from the below to ensure that the right estimates are used?

Which of the following statements are true?

What happens when a Spark Structured Streaming pipeline operating with Kafka as source is subject to a machine failure?

A company with headquarters (HQ) in the Middle East operates on a Sunday-Thursday weekday schedule with Friday & Saturday as weekend days. It computes end of week revenue numbers by first computing sales for each day at 1AM local time of the next day, and then summing up the weekly sales every Sunday early morning at 3AM local time. This number gets reported to leadership every Sunday morning 9AM local time. As a result of management change, it has decided to relocate its HQ to India. Which of the following changes will need to be done to its ETL pipeline?

You are appointed as a Data Engineer in a company that has a legacy reporting application written in Java which suffers from both performance and maintainability problems. The reporting application plots dashboards with near real-time refresh (once every minute) of key business indicators to help management take live decisions. The application reads data directly from the source database of MongoDB, aggregates using simple counts and shows them visually in a UI. The performance problem of this application comes because the source database is at times overloaded and therefore the dashboard is not able to refresh fast enough. The maintainability issue is because every time a new KPI summary is required, the source has to be changed to include it. Choose from the options below the option that best tackles both problems:

Dhoni is on the crease with a bat in hand that has sensors embedded throughout. The sensors talk to the spider cam every second. The spider cam is itself a powerful ARM-based computer which has connectivity to the cloud through the wire on which it hangs. Using this connectivity, it can send as much or as little data as required and also receive instructions from the cloud. There is a machine learning model which suggests to the batsman to loosen the grip on his bat or tighten it based on the shots played using the sensor measurements. The way the suggestion happens is using dynamic vibration intensity communicated to the sensors embedded in the bat handle. Your task is to design the data pipeline that enables such feedback to Dhoni ideally before every ball with as much accuracy as possible throughout the match. Which of the following options best satisfies the requirements?

In the class, we saw the UDF for mobilenet_v2. By definition, UDFs are scalar. In Spark, there is another class of user defined routines called UDAFs, which stand for User Defined Aggregator Functions. UDAFs are meant to provide a means to write a custom aggregation function which aggregates over a grouping of values to arrive at a single value. UDAF structure differs saliently from UDFs in that it exposes the notion of a buffer as a way of maintaining intermediate state before finalizing aggregate output. Why is a buffer required for UDAF and not for a UDF?

In the class, we saw the UDF for mobilenet_v2. Specifically, the predict() function contained the below lines of code

Why are these lines important in that pipeline of categorizing flowers?

You are given a Spark Streaming pipeline that invokes a pre-trained DL model for every image it receives as input and produces the classification result in quick time. The model with the best recall rate from the PyTorch library takes 3.1 seconds on an average to execute on a Quad CPU Spark worker machine. However, the output is expected to be produced consistently within 3 seconds from having received the input. What is the best option to meet the expectations without compromising on false negatives and minimizing the amount of money & effort that is spent further?

Which of the following code snippets will give a runtime error? (Note: df is a spark dataframe. It has a column called content which has images represented as byte array)

Consider a Structured Streaming application running on Google Dataproc firing up every 10 seconds, consuming any number of records from Kafka available since last read, and emitting some computed answers to another Kafka topic. Consider also that apart from the functional logic, the same application is also emitting into a file the start time and end time of every batch invocation for audit purposes. Assume there is a failure in one of the Dataproc machines that results in a failure of a specific run. For the external world (i.e. anybody consuming the outputs of this application), will they see the effect of the failure at all, or will it be as though there was no failure? Select the ones from the list below that answer this question.

Let us say we are using structured streaming for continuously reading data from Kafka and storing the results back into a Kafka topic. Now, instead, we decide that we need to just perform a one-time batch operation, where there is a need to read specific data from Kafka (i.e. using pre- determined offsets). How will we need to modify the code to make it work?

Consider a Kafka system that has two brokers with the exact same specifications. Let’s consider a topic A with a single partition, whose two copies are being maintained by Kafka. Consider the following cases: Case 1: Both the copies reside in the same broker Case 2: Each broker has one copy of the partition Two of the key promises of Kafka are: (i) Availability (ii) Throughput Regarding which of the above promises, case 1 is at a disadvantage as compared to case 2?

Kubernetes is an open-source system for automating deployment, scaling and management of containerized applications. Google Datastore and HBase are both highly-scalable NoSQL database systems for interactive, real-time applications. Consider the following pipeline choices for effecting the same outcome: (i) Java producer on VM on GCP → Kafka VM on GCP → Spark Streaming on Hadoop VMs on GCP → HBase on same Hadoop VMs on GCP (ii) Java producer in Google Cloud Function → Pub/Sub → Dataflow → Datastore (iii) Java producer in Kubernetes on GCP → Pub/Sub → Spark Streaming on Google Dataproc → Datastore (iv) Java producer on VM on GCP → Pub/Sub → Dataflow → Datastore (v) Java producer in Kubernetes on GCP → Kafka VM on GCP → Spark Streaming on Hadoop VMs on GCP → HBase on same Hadoop VMs on GCP Which option below represents the correct order of pipeline options that has the “most IaaS” entry to the left and the “most PaaS” entry to the right?

Since it is the onset of summer, there is a surge in railway ticket bookings. The business head at IRCTC is interested in a real-time view of what are the top 10 destinations being booked in descending order of count of tickets purchased where port of origin is one of the tier-1 urban centres such as Mumbai, Delhi, Chennai etc. She wants to see this be presented in a monitor mounted in her office wall that refreshes with the latest info on an India map every 1 minute along with the time of update so that she gets the confirmation that this is the latest data, so that she can plan for new summer-special trains as required. What solution option below best solves for the need?

Observe the below image and select the options that are true.

A Spark Streaming application is configured to execute once per minute. However, each run takes 10+ minutes consistently resulting in a never-ending backlog of work. The code in a nutshell looks as follows:

You are given a Spark program that runs on a Google Dataproc cluster on a daily schedule from 1AM-12PM to produce as output the total amount of purchases made by every customer the previous day. The data is coming into GCS every minute from a variety of sources as standalone files. Therefore, the business leader now feels that having to wait till 12PM the next day is no longer acceptable and instead wants approximate purchase information for each customer at least every hour. What’s more, she wants to control the computation of the outcome completely with flexibility to change the schedule as she wishes without the need for you or any other developer to be involved. Which amongst the below represents the best option to achieve the above?