MauryaHub PYQ Practice

Introduction to Big Data · End Term · Jan 2024 · FN

Introduction to Big Data · End Term · Jan 2024 FN

← Course papers · Start practice / exam

This page contains the reliably extracted subset, not the complete original paper.

Questions and published explanations below are available without starting a test. Some questions may not have a published solution yet.

0 / 30 published solutions

Question 166 MCQ · 2.0 marks

What best describes "big data"?
6406532755068.
Big data is used to refer to the set of technologies built around Hadoop
6406532755069.
Big data refers to the cloud-native design principle
6406532755070.
Big data is about All data, Any Time, Any Method
6406532755071.
Big data is about volume, velocity, variety

A published solution is not available for this question yet.

Report a correction for Q166

Question 167 MCQ · 2.0 marks

Which of these are implementations of the divide-and-conquer paradigm?
6406532755072.
Spark and Map
6406532755073.
Serverless and Message Broker
6406532755074.
Hadoop and Spark
6406532755075.
MapReduce and Google Cloud Functions

A published solution is not available for this question yet.

Report a correction for Q167

Question 168 MCQ · 2.0 marks

A website sees 1 Billion hits every month. The website owner wants to count average hits per customer in the latest month, where a customer is denoted by the IP address of the device from which the customer is accessing the website. The owner has at his disposal a Hadoop cluster of 5 workers and 2 masters each with 1GB of RAM. Which of the following methods is the most likely to finish fastest?
6406532755081.
Write a MapReduce program where the Map does nothing useful, Combine computes the aggregated hits per customer, Shuffle combined data based on IP address across workers, and in the Reduce, build hash table on each machine with hash key = IP address and hash value = counter, followed by another Reduce that finally computes the avg on top of all hash values.
6406532755082.
Write a Spark program that forms a Dataframe as grouping by IP address with count as aggregate, followed by a take into a list in the Spark driver which further computes the average of all the individual counts in the list
6406532755083.
Write a Spark program that forms a Dataframe as grouping by IP address with count as aggregate, followed by another stage that computes the avg on top of the Dataframe of the first stage
6406532755084.
Write a MapReduce program where 2 pairs of Map Reduce are chained together: 1\(^{st}\) pair is where the Map does nothing useful, Shuffle data based on IP address across workers, and in the Reduce, build hash table on each machine with hash key = IP address and hash value = counter, while the 2\(^{nd}\) pair is another Map that does nothing useful followed by a Reduce that finally computes avg on top of all hash values.
6406532755085.
All will finish in approximately the same time.

A published solution is not available for this question yet.

Report a correction for Q168

Question 169 MCQ · 2.0 marks

An enterprise software designer wants to leverage the best of Google cloud to minimize the number of administrative overheads associated with her big data pipeline while also getting on- demand scalability without sacrificing flexibility. What option should she choose to best serve these needs?
6406532755086.
Build the data pipeline using VMs – one for Python & one for storing files
6406532755087.
Build the data pipeline using Python running on Google Cloud Functions where the data is stored on GCS
6406532755088.
Build the data pipeline using MapReduce on, data storage on HDFS, and deploy both on Dataproc
6406532755089.
Build the data pipeline using Dataflow on top of data stored on GCS

A published solution is not available for this question yet.

Report a correction for Q169

Question 170 MCQ · 2.0 marks

Consider an application that can scale from handling 1000 users to handling 100 million users by simply making copies of itself, logs information about its workings on a central logger backed by Kafka, but has no automation to detect task failures and retry. Which of the following statements is true?
6406532755090.
This application has adopted cloud-native design
6406532755091.
This application cannot be called as cloud-native since it is not observable at all times
6406532755092.
This application cannot be called as cloud-native since it is not manageable easily
6406532755093.
This application cannot be called as cloud-native since it is not resilient

A published solution is not available for this question yet.

Report a correction for Q170

Question 171 MCQ · 2.0 marks

Consider a file “data.bin” which is formatted as follows: every data record has 10 key-value pairs of the format “key:value”, with each pair separated by a comma. Every data record occurs in its own line. You are asked to write a data processing script using Python that scales with big data. Which of the following represents your approach?
6406532755098.
Since data.bin is compliant with the RFC 4180, use PySpark’s read_csv() to read the data as is.
6406532755099.
Rename the file data.bin to data.csv to make it compliant with RFC 4180 and then use PySpark’s read_csv() to read the data
6406532755100.
The problem cannot be solved since the file cannot be converted to a valid format for reading consistently without additional information
6406532755101.
Write PySpark code to read all lines in data.bin, use string split on “,” as delimiter, and then collect all column names and corresponding values into a RDD for further processing

A published solution is not available for this question yet.

Report a correction for Q171

Question 172 MCQ · 2.0 marks

Consider the program outline as below running on a Spark cluster of 1 driver and 4 worker nodes with 2 executors per worker node:
Source diagram or notation
Consider the below mutually-exclusive characterisations about the program: i. Program will not run since RDD transformation operation map() runs on Executors that is referencing a variable train_master which is declared in the Spark driver ii. Program will run successfully and produce the count of bookings that match for trains present in the train_master For either of the 2 characterizations, consider the following (one or more) actions you could carry out in order to improve on the characterizations: 1. Program will need to be changed to bring the train_master into the Closure for executors to pick it up 2. Program will perform more poorly as the number of executors per worker node increases. Broadcast of train_master will need to be used to make it perform better. 3. Program will perform more poorly as the number of worker nodes increases (keeping number of executors per worker node constant). Broadcast of train_master will need to be used to make it perform better. Which of the following combinations of statements are correct?
6406532755102.
i and 1
6406532755103.
i and 1 followed by 2
6406532755104.
i and 1 followed by 2 & 3
6406532755105.
ii and both 2 & 3
6406532755106.
ii and 2
6406532755107.
ii and 3

A published solution is not available for this question yet.

Report a correction for Q172

Question 173 MCQ · 2.0 marks

What happens when a Spark Structured Streaming pipeline operating with Kafka as source is subject to a failure of a machine in either of the Kafka cluster or the Spark cluster?
6406532755130.
Failure of a machine in the Kafka cluster will result in an Exception in the Spark pipeline which will then fail and halt.
6406532755131.
The Spark pipeline will not be able to start again from previously committed offset by restarting itself, resulting in at least-once processing semantics
6406532755132.
Irrespective of whatever machine fails, Spark will throw an error and halt.
6406532755133.
Data that is being processed will not be processed again, resulting in atmost- once semantics.
6406532755134.
The pipeline will be restarted automatically by Spark which is able to pick up the exact data from Kafka which was being processed at the time of error, resulting in exactly- once semantics.

A published solution is not available for this question yet.

Report a correction for Q173

Question 174 MCQ · 2.0 marks

A big data streaming application that uses Kafka as source is observed to be really lagging behind currently live data. The Kafka cluster has 2 broker nodes and this application is reading from 1 topic that has 10 partitions. On closer investigation, it was found that Kafka is not scaling to the velocity of input data coming in. What can you first try to do to scale Kafka further while incurring minimal overall costs?
6406532755135.
Add disks to each broker in the cluster, and disks are the cheapest computer component
6406532755136.
Increase memory in each of the brokers in the cluster. While cost of memory is more than cost of disks, it is still cheaper than adding brokers and helps to scale.
6406532755137.
Add new brokers to the cluster, even though this is more expensive than the other options this is the only foolproof way to scale.
6406532755138.
Create more topics and change input application to reroute data to all topics to be able to spread input data better. This is nearly the least expensive since only developer effort is required to change application.
6406532755139.
Double the number of partitions for this single topic to be able to spread input data better. This is the least expensive since only administrator effort is required without changing application.

A published solution is not available for this question yet.

Report a correction for Q174

Question 175 MCQ · 2.0 marks

A company with headquarters (HQ) in the Middle East operates on a Sunday-Thursday weekday schedule with Friday & Saturday as weekend days. It computes end of week revenue numbers using an ETL pipeline by first computing sales for each day at 1AM local time of the next day, and then summing up the weekly sales every Sunday early morning at 3AM local time. This number gets reported to leadership every Sunday morning 9AM local time, so the ETL pipeline is scheduled to run every Sunday morning at 8AM local time. As a result of management change, it has decided to relocate its HQ to India. Which of the following changes will need to be done to its ETL pipeline to ensure the correct output continues to be produced?
6406532755145.
The business time for the final weekly sum operation needs to be changed to that of Monday 3AM India time instead of Sunday 3AM Middle East time.
6406532755146.
There is zero change needed since neither event time nor business time is changing whereas only the operational time is changing.
6406532755147.
Since time zone has changed as well as week definition too, the definition of business time has changed. So, the ETL has to be rewritten entirely.
6406532755148.
Nothing needs to change since daily sales is available at 1AM Middle East time which is anyway behind India time and so the numbers will be available before leadership comes in at 9AM.
6406532755149.
Event time has changed since the event of week ending has changed in definition, and so the ETL needs to be changed to consider the new event in the data.

A published solution is not available for this question yet.

Report a correction for Q175

Question 176 MCQ · 2.0 marks

You are appointed as a Data Engineer in a company that has a legacy reporting application written in Java which suffers from performance problems. The reporting application plots dashboards with near real-time refresh (once every minute) of key business indicators to help management take live decisions. The application reads data directly from the source database of MongoDB, aggregates using simple counts and shows them visually in a UI. The performance problem of this application comes because the source database is at times overloaded and therefore the dashboard is not able to refresh fast enough. Choose the best option that gives the best performance with minimal maintenance effort:
6406532755150.
Since MongoDB is OLTP, it is not able to support business reporting. So, replace it with Hadoop which supports OLAP better
6406532755151.
Convert the application from using plain Java to using Spark Streaming in Java
6406532755152.
Extract raw data from MongoDB using Change Data Capture (CDC) once every minute into Kafka, and then use Spark Streaming to compute the KPIs and then populate into a NoSQL DB like Redis for the UI to consume.
6406532755153.
Query MongoDB every 1 minute for new data using a check on document inserted timestamp, use Spark Streaming to compute the KPIs with the queried data, and then populate into a NoSQL DB like Redis for the UI to consume.
6406532755154.
Convert application to using Python along with a NoSQL database for storing and retrieving the aggregated counts.

A published solution is not available for this question yet.

Report a correction for Q176

Question 177 MCQ · 2.0 marks

Dhoni is on the crease with a bat in hand that has sensors embedded throughout. The sensors talk to the spider cam every second. The spider cam is itself a powerful ARM based computer which has connectivity to the cloud through the wire on which it hangs. Using this connectivity, it can send as much or as little data as required and also receive instructions from the cloud. There is a machine learning model which suggests to the batsman to loosen the grip on his bat or tighten it based on the shots played using the sensor measurements. The way the suggestion happens is using dynamic vibration intensity communicated to the sensors embedded in the bat handle. Your task is to design the data pipeline that enables such feedback to Dhoni ideally before every ball with as much accuracy as possible throughout the match. Which of the following options best satisfies the requirements?
6406532755155.
Ingest all data into Pub/Sub, process using Google Cloud Dataflow, invoke the ML model, and then write back output from Cloud to the spider cam to relay to the bat.
6406532755156.
Compress the ML model to fit into the spider cam’s available resource, and write pipelines to execute the model in the spider cam itself
6406532755157.
Compress the ML model to fit into the spider cam’s available resource, and write pipelines to execute in the spider cam itself, with periodically data being sent to the cloud, retrain the model using Google Cloud ML and then redeploy the model to the spider cam.
6406532755158.
Compress the data in the spider cam every 5 seconds, write to Pub/Sub the compressed data, invoke the ML model and then write back output from Cloud to the spider cam to relay to the bat.

A published solution is not available for this question yet.

Report a correction for Q177

Question 178 MCQ · 2.0 marks

In the class, we saw the UDF for mobilenet_v2. By definition, UDFs are scalar. In Spark, there is another class of user defined routines called UDAFs, which stand for User Defined Aggregator Functions. UDAFs are meant to provide a means to write a custom aggregation function which aggregates over a grouping of values to arrive at a single value. UDAF structure differs saliently from UDFs in that it provides a buffer to keep track of intermediate state before finalizing aggregate output. Why is a buffer required for UDAF and not for a UDF?
6406532755159.
A UDF is a scalar operation executing on 1 row at a time and producing output immediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows which will require multiple passes for the final output thereby requiring a buffer.
6406532755160.
A UDF is a scalar operation executing on 1 row at a time and producing output immediately, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows of unbounded size requiring a divide-and-conquer approach for computing aggregates, which uses the intermediate buffer to store partial values before finalizing the result aggregate.
6406532755161.
A UDF is a scalar operation executing on many rows at a time, grouped by a key and producing a single output, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows which will require multiple passes for the final output thereby requiring a buffer.
6406532755162.
A UDF is a scalar operation executing on many rows at a time, grouped by a key and producing a single output, and therefore there is no intermediate output necessary. Whereas, a UDAF operates on multiple rows of unbounded size requiring a divide-and-conquer approach for computing aggregates, which uses the intermediate buffer to store partial values before finalizing the result aggregate.

A published solution is not available for this question yet.

Report a correction for Q178

Question 179 MCQ · 2.0 marks

In the class, we saw the UDF for mobilenet_v2. Specifically, the predict() function contained the below lines of code
Source diagram or notation
Suppose there’s a new transformer model from Facebook that is orders of magnitude bigger than mobilenet_v2 but that is able to provide better F1 score. Will these lines of code still be required in the pipeline?
6406532755163.
Yes, since they check the syntax of the model function so that there are no errors
6406532755164.Source diagram or notation
Yes, they invoke PyTorch libraries that have already been setup for model scoring on GCP using APIs embedded within the function
6406532755165.
No, since the primary function of these lines of code is to eliminate repeated DL model loads as DL models are large in size
6406532755166.
Yes, they are Map-style UDFs that make it an embarrassingly parallel computation thus making the execution parallelized and fast.
6406532755167.
Yes, since the new model will potentially have large load times.

A published solution is not available for this question yet.

Report a correction for Q179

Question 180 MCQ · 2.0 marks

You are given a Spark Streaming pipeline that invokes a pre-trained DL model for every image it receives as input and produces the classification result in quick time. The model with the best recall rate from the PyTorch library already runs in under 3 seconds on an average, when executing on a single GPU Spark worker machine. However, your management has instructed you to reduce the cost of AI projects significantly. What is the best option to explore to meet the expectations without compromising on false negatives while also being within 10-20% of the average execution time?
6406532755168.
Build a custom model that compresses the highest recall rate model just enough to be able to execute within the stipulated time, and measure recall.
6406532755169.
Use a different DL model from PyTorch that is already compressed to half the size.
6406532755170.
Remove complexity associated with Spark Streaming and convert the model execution pipeline into a single threaded Python application running on the same GPU machine.
6406532755171.
Change Spark machine to use CPUs and train a fresh pipeline to achieve objectives.

A published solution is not available for this question yet.

Report a correction for Q180

Question 181 MCQ · 2.0 marks

Consider a Structured Streaming application running on Google Dataproc firing up every 10 seconds, consuming any number of records from Kafka available since last read, and emitting some computed answers to another Kafka topic. Consider also that apart from the functional logic, the same application is also emitting into a file the start time and end time of every batch invocation for audit purposes. Assume there is a failure in one of the Dataproc machines that results in a failure of a specific run. For the external world (i.e. anybody consuming the outputs of this application), will they see any change in output as a result of the failure at all, or will the only visible impact of failure be of slower performance for the failed-and-retried run?
6406532755176.
No, the failure is not visible. The only visible effect for the external world would be in the form of a slowdown in runtime for completion of that mini-batch as Structured Streaming retries the mini-batch that failed thus taking twice as much time as normal.
6406532755177.
No, the failure is not visible since Structured Streaming uses transactions and idempotence to achieve exactly-once processing.
6406532755178.
No, the failure is not visible since Structured Streaming can process the same data in a retry resulting in the same outputs again.
6406532755179.
Yes, the failure is visible because the side effect of emitting timestamps in a batch will be visible as 2 consecutive Start timestamps without any end timestamp as Structured Streaming retries the failed batch.
6406532755180.
Yes, the failure is visible since the logs in the backend of Spark Structured Streaming are also logging the state of the machine.

A published solution is not available for this question yet.

Report a correction for Q181

Question 182 MCQ · 2.0 marks

Let us say we are using structured streaming for continuously reading data from Kafka and storing the results back into a Kafka topic using window function aggregates. Now, instead, we decide that we need to just perform a one-time batch operation using the same logic, where there is a need to read specific data from Kafka (i.e. using pre-determined offsets). How will we need to modify the code to make it work?
6406532755181.
The read and write commands will remain the same, but the remaining code will need to be modified, as operations on streaming dataframes are not supported on static dataframes.
6406532755182.
The entire code will need to be modified as the APIs for stream and batch processing are completely different.
6406532755183.
The read and write commands need to be modified to specify that it’s a batch operation. Further, the specific logic of window functions will also need to be modified since there are no time windows anymore in batch processing.
6406532755184.
Only the read and write commands need to be modified to specify that it's a batch operation.

A published solution is not available for this question yet.

Report a correction for Q182

Question 183 MCQ · 2.0 marks

Consider a Kafka system that has two brokers with the exact same resources. Let’s consider a topic A with a single partition, whose two copies are being maintained in sync by Kafka. Consider the following cases: Case 1: Each broker has one copy of the partition Case 2: Both the copies reside in the same broker Two of the key promises of Kafka are: (i) Availability (ii) Throughput Regarding which of the above promises, case 1 is at an advantage as compared to case 2?
6406532755185.
Only (ii)
6406532755186.
Only (i)
6406532755187.
Both (i) and (ii)
6406532755188.
Neither (i) nor (ii)

A published solution is not available for this question yet.

Report a correction for Q183

Question 184 MCQ · 2.0 marks

Kubernetes is an open-source system for automating deployment, scaling and management of containerized applications. Google Datastore and HBase are both highly-scalable NoSQL database systems for interactive, real-time applications. Consider the following pipeline choices for effecting the same outcome: (i) Shell producer on VM on GCP **→** Pub/Sub **→** Spark Streaming on Hadoop VMs on GCP **→** HBase on same Hadoop VMs on GCP (ii) Shell producer in Google Cloud Function **→** Kafka VM on GCP **→** Dataflow **→** Datastore (iii) Shell producer in Kubernetes on GCP **→** Pub/Sub **→** Spark Streaming on Google Dataproc **→** Datastore (iv) Shell producer on VM on GCP**→** Pub/Sub **→** Dataflow**→** Datastore (v) Shell producer in Kubernetes on GCP **→** Pub/Sub **→** Spark Streaming on Hadoop VMs on GCP **→** HBase on same Hadoop VMs on GCP Which option below represents the correct order of pipeline options that has the “most IaaS” entry to the left and the “most PaaS” entry to the right?
6406532755189.
(i), (v), (iii), (iv), (ii)
6406532755190.
(i), (ii), (iii), (iv), (v)
6406532755191.
(ii), (iii), (i), (iv), (v)
6406532755192.
(v), (iii), (i), (iv), (ii)
6406532755193.
All are equally PaaS / IaaS

A published solution is not available for this question yet.

Report a correction for Q184

Question 185 MCQ · 2.0 marks

A Spark Streaming application is configured to execute once per minute. However, each run takes 10+ minutes consistently resulting in a never-ending backlog of work. The code in a nutshell looks as follows:
Source diagram or notation
Your goal is to optimize the code to bring down execution of each iteration within 1 minute. Which of the following represent options that will help in this mission?
6406532755194.Source diagram or notation
6406532755195.Source diagram or notation
6406532755196.Source diagram or notation
6406532755197.Source diagram or notation

A published solution is not available for this question yet.

Report a correction for Q185

Question 186 MCQ · 2.0 marks

You are given a Spark program that runs on a Google Dataproc cluster on a daily schedule from 8PM-10PM to produce as output the total amount of purchases made by every customer the previous day. The data is coming into GCS every minute from a variety of sources as standalone files. Therefore, the business leader now feels that having to wait till 10PM the next day is no longer acceptable and instead wants approximate purchase information for each customer at least every 5 minutes. What’s more, she wants to be able to change this time window later as she pleases without involving you. Which amongst the below represents the best option to achieve the above?
6406532755198.
Change the code to run every 5 minutes, no other change required
6406532755199.
Change the code to leverage Spark Streaming with streaming window as “5 minutes”, & let her manage the execution of the code on Dataproc
6406532755200.
Change the code to leverage Spark Streaming with streaming window as “5 minutes”, convert from Dataproc to Dataflow, & let her manage the execution of the code on Dataflow
6406532755201.
Write a Cloud Function to move all incoming per-minute standalone files from GCS to Pub/Sub, change the code to leverage Spark Streaming with streaming window as “5 mins”, convert from Dataproc to Dataflow, point source to Pub/Sub, & let her manage the execution of the code on Dataflow

A published solution is not available for this question yet.

Report a correction for Q186

Question 187 MCQ · 2.0 marks

Since it is the onset of summer, there is a surge in railway ticket bookings. The business head at IRCTC is interested in a real-time view of all of her stations irrespective of whether there were bookings or not. She wants to see this be presented in a monitor mounted in her office wall that refreshes with the latest info on an India map every 1 minute. Along with the info, there also needs to be the current time so that she gets a visual confirmation that this is the latest data. This dashboard allows her to plan for new summer-special trains as required. What solution option below best solves for the need?
6406532755202.
Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to continuously read from this topic and update the aggregates by destinations, and emit using output mode “Update”.
6406532755203.
Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to periodically read from this topic every 1 minute and update the aggregates by destinations, and emit all aggregates using the output mode “Complete”.
6406532755204.
Route a copy of the ticket purchase to a Kafka topic, use Spark Structured Streaming to periodically read from this topic every 1 minute and count the destinations in that batch, and emit only all aggregates in that batch using the output mode “Append”.

A published solution is not available for this question yet.

Report a correction for Q187

Question 188 MSQ · 2.0 marks

What option(s) best describe the differences between MapReduce and Spark?
6406532755076.Source diagram or notation
MapReduce leverages memory heavily, while Spark optimizes for disk-based computations
6406532755077.
MapReduce forces barrier synchronization after every step, while Spark uses directed acyclic graphs to execute as many steps as possible in parallel
6406532755078.
MapReduce leverages disk heavily, while Spark optimizes for memory-based computations
6406532755079.
MapReduce enables massively parallel computation, while Spark's driver program sequentially executes each worker
6406532755080.
MapReduce is restricted in flexibility since only Map and Reduce are possible steps, while Spark has a variety of Actions possible making it highly flexible

A published solution is not available for this question yet.

Report a correction for Q188

Question 189 MSQ · 2.0 marks

You are provided with a Spark program that picks out a list of suspicious transactions. It’s logic is based on both the financial value of the transaction and the geographic location of the transaction. If the financial value is higher than a threshold and the geographic location is from a set of suspected locations (provided as a 1MB file), then the program deems the transaction as suspicious. The version of the Spark program given to you is written in such a way that it pulls all the transactions from the Workers to the Driver and then applies the logic. Which 2 changes from the list below will get you the most benefit in performance?
6406532755094.
Use broadcast variables for the 1MB file
6406532755095.
Hardcode threshold value as a filter condition in the Driver program
6406532755096.
Reorder operations on the Driver such that geographic location is checked first before filtering high value transactions
6406532755097.
Hardcode threshold value as a filter condition in the Workers itself

A published solution is not available for this question yet.

Report a correction for Q189

Question 190 MSQ · 2.0 marks

Which of the following types of data sources can you read successfully without missing data using a program that extracts once every day? If no data is missed, then all data in source will match data extracted successfully.
6406532755108.
A MySQL table that is modelled as a Type II SCD where all Creates are new rows and all Updates are appended only, and these can happen anytime during the day.
6406532755109.
A Linux machine’s system.log file into which all processes append their events
6406532755110.
Share transactions stored as facts & share prices updated in-place in a PostgreSQL database
6406532755111.
A weather API that provides the temperature & rainfall readings for all its weather stations in India for the specific time instant when being queried
6406532755112.
Google drive account that stores all historical census data for India

A published solution is not available for this question yet.

Report a correction for Q190

Question 191 MSQ · 2.0 marks

Which of the following is true?
6406532755113.
Snapshots of source systems can be created using an event capture tool like CDC and then replaying the events in sequence for that time period.
6406532755114.
A data lake is a collection of data to be provided as input for data science algorithms.
6406532755115.
Zookeeper is a library for ensuring that critical services used in big data are able to stay in sync with each other as to the health of those services.
6406532755116.
A single Spark cluster can have multiple “leader” master nodes.
6406532755117.
Spark is optimized for in-memory computation.
6406532755118.
Given the RDD underlying a Dataframe, you can recreate the same Dataframe provided you know the schema

A published solution is not available for this question yet.

Report a correction for Q191

Question 192 MSQ · 2.0 marks

What are some capabilities common to both “streaming processing” & “batch processing” when using Spark for big data?
6406532755119.
Batch operates on a set of data elements taken together while streaming can also operate on a set of data elements as determined by the window
6406532755120.
Batch operates on data that is static while streaming operates on data that is dynamically changing
6406532755121.
Batch processing can assume data as fully specified and complete while streaming cannot make that assumption
6406532755122.
Batch processing is typically high latency while streaming processing is necessary for real-time latencies
6406532755123.
Both Streaming & Batch processing can operate on massively large data sets
6406532755124.
Both Streaming and Batch processing in Spark can use the same syntax for programming the main functional logic of the application

A published solution is not available for this question yet.

Report a correction for Q192

Question 193 MSQ · 2.0 marks

Which of the following are best practices associated with Streaming applications?
6406532755125.
Use a message store that supports message replay so that no data is lost in processing.
6406532755126.
“Hot potato” principle is when the streaming application operates on the minimum amount of processing to produce valid output, which then becomes input for the next minimal processor, and so on, thus ensuring that no single process is doing too many things in one go.
6406532755127.
Hadoop is best suited for executing Streaming applications.
6406532755128.
Use checkpointing when faced with mission-critical workloads that require 100% accuracy.
6406532755129.
Handle state pollution by restarting the persistent store software periodically.

A published solution is not available for this question yet.

Report a correction for Q193

Question 194 MSQ · 2.0 marks

You are given the task of improving the performance of a Spark SQL program that is doing a simple count after a series of transformations. You suspect that the culprit is the main transformation job in the program. When you run EXPLAIN on that SQL, you see that Spark wrongly estimates that there are only 10 values for the key on which the main transformation job is hinged upon, whereas in reality the underlying data has a million values for that key. What actions would you perform from the below to improve performance of the program?
6406532755140.
Create all tables as external tables.
6406532755141.
Ensure cost based optimizer (CBO) is ON.
6406532755142.
Partition all tables on the same key on which the aggregate is happening.
6406532755143.
Run ANALYZE on all tables to ensure the right estimates are available to the optimizer.
6406532755144.
Cache the table in a step with actions ahead of the SQL statement that is the culprit.

A published solution is not available for this question yet.

Report a correction for Q194

Question 195 MSQ · 2.0 marks

Observe the below image and select the options that are true.
Source diagram or notation
6406532755172.
Each subscriber gets one third of all messages published into Topic 1.
6406532755173.
Publisher 1 can safely send data only for Subscriber 1’s consumption while restricting access to Subscribers 2 & 3.
6406532755174.
Subscriber 2 gets all messages published into Topic 1 by Publisher 1 but only half of Publisher 2’s messages.
6406532755175.
Each subscriber gets all messages published into Topic 1.       **LLM** **Section Id :** 64065357830 **Section Number :** 9 **Section type :** Online **Mandatory or Optional :** Mandatory **Number of Questions :** 11 **Number of Questions to be attempted :** 11 **Section Marks :** 50 **Display Number Panel :** Yes **Section Negative Marks :** 0 **Group All Questions :** No **Enable Mark as Answered Mark for Review and** Yes **Clear Response :** **Maximum Instruction Time :** 0

A published solution is not available for this question yet.

Report a correction for Q195