Which of the following are best practices associated with Streaming applications?
6406532052540.
Use a message store that supports message replay so that no data is lost in
processing.
6406532052541.
“Hot potato” principle is when the streaming application operates on data from
cache (i.e. the hot area of memory) and therefore is able to produce very high throughput
6406532052542.
Hadoop is best suited for executing Streaming applications
6406532052543.
Use checkpointing when faced with mission-critical workloads that require
100% accuracy.
6406532052544.
Handle state pollution by restarting the persistent store software periodically.
A published solution is not available for this question yet.
You are given the task of improving the performance of a Spark SQL program. You suspect that
the culprit is the main transformation job in the program. When you run EXPLAIN on that SQL, you
see that Spark wrongly estimates that there are only 10 values for the key being aggregated,
whereas in reality the underlying data has a million values for that key. What actions would you
perform from the below to ensure that the right estimates are used?
6406532052555.
Create all tables as external tables.
6406532052556.
Ensure cost based optimizer (CBO) is ON.
6406532052557.
Partition all tables on the same key on which the aggregate is happening.
6406532052558.
Run ANALYZE on all tables.
6406532052559.
Cache the table in a step with actions ahead of the SQL statement that is the
culprit.
A published solution is not available for this question yet.
What happens when a Spark Structured Streaming pipeline operating with Kafka as source is
subject to a failure of a machine in either of the Kafka cluster or the Spark cluster?
6406532052545.
Failure of a machine in the Kafka cluster will result in an Exception in the Spark
pipeline which will then fail and halt.
6406532052546.
The Spark pipeline will not be able to start again from previously committed
offset by restarting itself, resulting in at least-once processing semantics
6406532052547.
Irrespective of whatever machine fails, Spark will throw an error and halt.
6406532052548.
The pipeline will be restarted automatically by Spark which is able to pick up
the exact data from Kafka which was being processed at the time of error, resulting in exactly-
once semantics.
6406532052549.
Data that is being processed will not be processed again, resulting in atmost-
once semantics.
A published solution is not available for this question yet.
You are given a Spark program that runs on a Google Dataproc cluster on a daily schedule from
1AM-12PM to produce as output the total amount of purchases made by every customer the
previous day. The data is coming into GCS every minute from a variety of sources as standalone
files. Therefore, the business leader now feels that having to wait till 12PM the next day is no
longer acceptable and instead wants approximate purchase information for each customer at
least every 5 minutes. What’s more, she wants to be able to change this time window later as she
pleases without involving you.
Which amongst the below represents the best option to achieve the above?
6406532052619.
Change the code to run every 5 minutes, no other change required
6406532052620.
Change the code to leverage Spark Streaming with streaming window as “5
minutes”, & let her manage the execution of the code on Dataproc
6406532052621.
Write a Cloud Function to move all incoming per-minute standalone files from
GCS to Pub/Sub, change the code to leverage Spark Streaming with streaming window as “5 mins”,
convert from Dataproc to Dataflow, point source to Pub/Sub, & let her manage the execution of
the code on Dataflow
6406532052622.
Change the code to leverage Spark Streaming with streaming window as “5
minutes”, convert from Dataproc to Dataflow, & let her manage the execution of the code on
Dataflow
A published solution is not available for this question yet.
A big data streaming application that uses Kafka as source is observed to be really slow. The Kafka
cluster has 2 broker nodes and this application is reading from 1 topic that has 10 partitions. On
closer investigation, it was found that Kafka is not scaling to the velocity of input data coming in.
What can you first try to do to scale Kafka further while incurring minimal costs?
6406532052550.
Add disks to each broker in the cluster, and disks are the cheapest computer
component
6406532052551.
Add new brokers to the cluster, even though this is more expensive than the
other options this is the only foolproof way to scale.
6406532052552.
Increase memory in each of the brokers in the cluster. While cost of memory is
more than cost of disks, it is still cheaper than adding brokers and helps to scale.
6406532052553.
Create more topics and change input application to reroute data to all topics to
be able to spread input data better. This is nearly the least expensive since only developer effort is
required to change application.
6406532052554.
Double the number of partitions for this single topic to be able to spread input
data better. This is the least expensive since only administrator effort is required without changing
application.
A published solution is not available for this question yet.
A company with headquarters (HQ) in the Middle East operates on a Sunday-Thursday weekday
schedule with Friday & Saturday as weekend days. It computes end of week revenue numbers
using an ETL pipeline by first computing sales for each day at 1AM local time of the next day, and
then summing up the weekly sales every Sunday early morning at 3AM local time. This number
gets reported to leadership every Sunday morning 9AM local time, so the ETL pipeline is scheduled
to run every Sunday morning at 8AM local time. As a result of management change, it has decided
to relocate its HQ to India. Which of the following changes will need to be done to its ETL pipeline
to ensure the correct output continues to be produced?
6406532052560.
The business time for the final weekly sum operation needs to be changed to
that of Monday 3AM India time instead of Sunday 3AM Middle East time.
6406532052561.
Since time zone has changed as well as week definition too, the definition of
business time has changed. So, the ETL has to be rewritten entirely.
6406532052562.
Nothing needs to change since daily sales is available at 1AM Middle East time
which is anyway behind India time and so the numbers will be available before leadership comes
in at 9AM.
6406532052563.
Event time has changed since the event of week ending has changed in
definition, and so the ETL needs to be changed to consider the new event in the data.
6406532052564.
No change required since neither event time nor business time is changing
whereas only the operational time is changing.
A published solution is not available for this question yet.
You are appointed as a Data Engineer in a company that has a legacy reporting application written
in Java which suffers from performance problems. The reporting application plots dashboards
with near real-time refresh (once every minute) of key business indicators to help management
take live decisions. The application reads data directly from the source database of MongoDB,
aggregates using simple counts and shows them visually in a UI. The performance problem of this
application comes because the source database is at times overloaded and therefore the
dashboard is not able to refresh fast enough. Choose from the options below the option that best
addresses performance with minimal maintenance effort:
6406532052565.
Since MongoDB is OLTP, it is not able to support business reporting. So,
replace it with Hadoop which supports OLAP better
6406532052566.
Convert the application from using plain Java to using Spark Streaming in Java
6406532052567.
Query MongoDB every 1 minute for new data using a check on document
inserted timestamp, use Spark Streaming to compute the KPIs with the queried data, and then
populate into a NoSQL DB like Redis for the UI to consume.
6406532052568.
Extract raw data from MongoDB using Change Data Capture (CDC) once every
minute into Kafka, and then use Spark Streaming to compute the KPIs and then populate into a
NoSQL DB like Redis for the UI to consume.
6406532052569.
Convert application to using Python along with a NoSQL database for storing
and retrieving the aggregated counts.
A published solution is not available for this question yet.
Dhoni is on the crease with a bat in hand that has sensors embedded throughout. The sensors talk
to the spider cam every second. The spider cam is itself a powerful ARM-based computer which
has connectivity to the cloud through the wire on which it hangs. Using this connectivity, it can
send as much or as little data as required and also receive instructions from the cloud. There is a
machine learning model which suggests to the batsman to loosen the grip on his bat or tighten it
based on the shots played using the sensor measurements. The way the suggestion happens is
using dynamic vibration intensity communicated to the sensors embedded in the bat handle. Your
task is to design the data pipeline that enables such feedback to Dhoni ideally before every ball
with as much accuracy as possible throughout the match. Which of the following options best
satisfies the requirements?
6406532052570.
Ingest all data into Pub/Sub, process using Google Cloud Dataflow, invoke the
ML model, and then write back output from Cloud to the spider cam to relay to the bat.
6406532052571.
Compress the ML model to fit into the spider cam’s available resource, and
write pipelines to execute the model in the spider cam itself
6406532052572.
Compress the data in the spider cam every 5 seconds, write to Pub/Sub the
compressed data, invoke the ML model and then write back output from Cloud to the spider cam
to relay to the bat.
6406532052573.
Compress the ML model to fit into the spider cam’s available resource, and
write pipelines to execute in the spider cam itself, with periodically data being sent to the cloud,
retrain the model using Google Cloud ML and then redeploy the model to the spider cam.
A published solution is not available for this question yet.
In the class, we saw the UDF for mobilenet_v2. By definition, UDFs are scalar. In Spark, there is
another class of user defined routines called UDAFs, which stand for User Defined Aggregator
Functions. UDAFs are meant to provide a means to write a custom aggregation function which
aggregates over a grouping of values to arrive at a single value. UDAF structure differs saliently
from UDFs in that it exposes the notion of a buffer as a way of maintaining intermediate state
before finalizing aggregate output. Why is a buffer required for UDAF and not for a UDF?
6406532052574.
A UDF is a scalar operation executing on 1 row at a time and producing output
immediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operates
on multiple rows which will require multiple passes for the final output thereby requiring a buffer.
6406532052575.
A UDF is a scalar operation executing on many rows at a time, grouped by a
key and producing a single output, and therefore there is no intermediate output necessary.
Whereas, a UDAF operates on multiple rows which will require multiple passes for the final output
thereby requiring a buffer.
6406532052576.
A UDF is a scalar operation executing on 1 row at a time and producing output
immediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operates
on multiple rows of unbounded size requiring a divide-and-conquer approach for computing
aggregates, which uses the intermediate buffer to store partial values before finalizing the result
aggregate.
6406532052577.
A UDF is a scalar operation executing on many rows at a time, grouped by a
key and producing a single output, and therefore there is no intermediate output necessary.
Whereas, a UDAF operates on multiple rows of unbounded size requiring a divide-and-conquer
approach for computing aggregates, which uses the intermediate buffer to store partial values
before finalizing the result aggregate.
A published solution is not available for this question yet.
Yes, since they check the syntax of the model function so that there are no
errors
6406532052579.
They invoke PyTorch libraries that have already been setup for model scoring
on GCP using APIs embedded within the function
6406532052580.
No, since the primary function of these lines of code is to eliminate repeated
DL model loads as DL models are large in size but with XGboost this isn’t required since model
load time is negligible
6406532052581.
They are Map-style UDFs that make it an embarrassingly parallel computation thus making the
execution parallelized and fast.
A published solution is not available for this question yet.
You are given a Spark Streaming pipeline that invokes a pre-trained DL model for every image it
receives as input and produces the classification result in quick time. The model with the best
recall rate from the PyTorch library already runs in under 3 seconds on an average executing on a
single GPU Spark worker machine. However, your management has instructed you to reduce the
cost of AI projects significantly. What is the best option to explore to meet the expectations
without compromising on false negatives while also being within 10-20% of the average execution
time?
6406532052582.
Use a different DL model from PyTorch that is already compressed to half the
size.
6406532052583.
Build a custom model that compresses the highest recall rate model just
enough to be able to execute within the stipulated time, and measure recall.
6406532052584.
Remove complexity associated with Spark Streaming and convert the model
execution pipeline into a single threaded Python application running on the same GPU machine.
6406532052585.
Change Spark machine to use CPUs and train a fresh pipeline to achieve
objectives.
A published solution is not available for this question yet.
Which of the following code snippets will give a runtime error? (Note: df is a spark dataframe. It
has a column called content which has images represented as byte array)
Consider a Structured Streaming application running on Google Dataproc firing up every 10
seconds, consuming any number of records from Kafka available since last read, and emitting
some computed answers to another Kafka topic. Consider also that apart from the functional
logic, the same application is also emitting into a file the start time and end time of every batch
invocation for audit purposes.
Assume there is a failure in one of the Dataproc machines that results in a failure of a specific run.
For the external world (i.e. anybody consuming the outputs of this application), will they see any
change in output as a result of the failure at all, or will the only impact of failure be one of slower
performance for the failed-and-retried run? select the one from the list below that answer this
question.
6406532052598.
No, the failure is not visible. The only visible effect for the external world would
be in the form of a slowdown in runtime for completion of that batch as Structured Streaming
retries the batch that failed thus taking twice as much time as normal.
6406532052599.
No, no failure is visible since Structured Streaming uses transactions and
idempotence to achieve exactly-once processing.
6406532052600.
Yes, the failure is visible because the side effect of emitting timestamps in a
batch will be visible as 2 consecutive Start timestamps without any end timestamp as Structured
Streaming retries the failed batch.
6406532052601.
No, no failure is visible since Structured Streaming can process the same data
in a retry resulting in the same outputs again.
A published solution is not available for this question yet.
Let us say we are using structured streaming for continuously reading data from Kafka and
storing the results back into a Kafka topic using window function aggregates. Now, instead, we
decide that we need to just perform a one-time batch operation using the same logic, where there
is a need to read specific data from Kafka (i.e. using pre-determined offsets). How will we need to
modify the code to make it work?
6406532052602.
The read and write commands will remain the same, but the remaining code
will need to be modified, as operations on streaming dataframes are not supported on static
dataframes.
6406532052603.
Only the read and write commands need to be modified to specify that it's a
batch operation.
6406532052604.
The entire code will need to be modified as the APIs for stream and batch
processing are completely different.
6406532052605.
The read and write commands need to be modified to specify that it’s a batch
operation. Further, the specific logic of window functions will also need to be modified since there
are no time windows anymore in batch processing.
A published solution is not available for this question yet.
Consider a Kafka system that has two brokers with the exact same resources. Let’s consider a topic
A with a single partition, whose two copies are being maintained in sync by Kafka. Consider the
following cases:
Case 1: Both the copies reside in the same broker
Case 2: Each broker has one copy of the partition
Two of the key promises of Kafka are:
(i) Availability
(ii) Throughput
Regarding which of the above promises, case 2 is at an advantage as compared to case 1?
6406532052606.
Only (i)
6406532052607.
Only (ii)
6406532052608.
Both (i) and (ii)
6406532052609.
Neither (i) nor (ii)
A published solution is not available for this question yet.
Kubernetes is an open-source system for automating deployment, scaling and management of
containerized applications. Google Datastore and HBase are both highly-scalable NoSQL database
systems for interactive, real-time applications. Consider the following pipeline choices for effecting
the same outcome:
(i) Shell producer on VM on GCP → Pub/Sub → Spark Streaming on Hadoop VMs on GCP → HBase
on same Hadoop VMs on GCP
(ii) Shell producer in Google Cloud Function → Kafka VM on GCP → Dataflow → Datastore
(iii) Shell producer in Kubernetes on GCP → Pub/Sub → Spark Streaming on Google Dataproc →
Datastore
(iv) Shell producer on VM on GCP → Pub/Sub → Dataflow → Datastore
(v) Shell producer in Kubernetes on GCP → Pub/Sub → Spark Streaming on Hadoop VMs on GCP →
HBase on same Hadoop VMs on GCP
Which option below represents the correct order of pipeline options that has the “most IaaS” entry
to the left and the “most PaaS” entry to the right?
6406532052610.
(i), (v), (iii), (iv), (ii)
6406532052611.
(i), (ii), (iii), (iv), (v)
6406532052612.
(ii), (iii), (i), (iv), (v)
6406532052613.
(v), (iii), (i), (iv), (ii)
6406532052614.
All are equally PaaS / IaaS
A published solution is not available for this question yet.
Since it is the onset of summer, there is a surge in railway ticket bookings. The business head at
IRCTC is interested in a real-time view of all of her stations irrespective of whether there were
bookings or not. She wants to see this be presented in a monitor mounted in her office wall that
refreshes with the latest info on an India map every 1 minute along with the time of update so
that she gets the confirmation that this is the latest data, so that she can plan for new summer-
special trains as required. What solution option below best solves for the need?
6406532052623.
Route a copy of the ticket purchase to a Kafka topic, use Spark Structured
Streaming to continuously read from this topic and update the aggregates by destinations, and
emit using output mode “Update”.
6406532052624.
Route a copy of the ticket purchase to a Kafka topic, use Spark Structured
Streaming to periodically read from this topic every 1 minute and update the aggregates by
destinations, and emit all aggregates using the output mode “Complete”.
6406532052625.
Route a copy of the ticket purchase to a Kafka topic, use Spark Structured
Streaming to periodically read from this topic every 1 minute and count the destinations in that
batch, and emit only all aggregates in that batch using the output mode “Append”.
**Fin Forensics**
**Section Id :** 64065341446
**Section Number :** 8
**Section type :** Online
**Mandatory or Optional :** Mandatory
**Number of Questions :** 41
**Number of Questions to be attempted :** 41
**Section Marks :** 40
**Display Number Panel :** Yes
**Section Negative Marks :** 0
**Group All Questions :** No
**Enable Mark as Answered Mark for Review and**
Yes
**Clear Response :**
**Maximum Instruction Time :** 0
A published solution is not available for this question yet.
