How Do You Know When a Spark Program Is Broken

"The most difficult matter is finding out why your chore is failing, which parameters to change. Most of the fourth dimension, information technology's OOM errors…" Jagat Singh, Quora

Spark has go one of the nearly important tools for processing information – peculiarly non-relational data – and deriving value from it. And Spark serves as a platform for the creation and delivery of analytics, AI, and machine learning applications, amidst others. Only troubleshooting Spark applications is difficult – and we're here to help.

In this blog post, we'll draw ten challenges that ascend ofttimes in troubleshooting Spark applications. We'll start with bug at the task level, encountered by most people on the data team – operations people/administrators, information engineers, and data scientists, as well equally analysts. Then, we'll look at problems that apply beyond a cluster. These problems are usually handled past operations people/administrators and data engineers.

For more on Spark and its use, please meet this piece in Infoworld. And for more depth about the problems that arise in creating and running Spark jobs, at both the job level and the cluster level, please come across the links below. At that place is also a expert introductory guide hither.

V Reasons Why Troubleshooting Spark Applications is Hard

Some of the things that make Spark great also get in hard to troubleshoot. Here are some key Spark features, and some of the problems that arise in relation to them:

1. Retentivity-resident. Spark gets much of its speed and power by using memory, rather than disk, for interim storage of source information and results. However, this tin cost a lot of resources and money, which is especially visible in the cloud. It can likewise make it like shooting fish in a barrel for jobs to crash due to lack of sufficient available memory. And it makes problems hard to diagnose – only traces written to disk survive later on crashes.
2. Parallel processing. Spark takes your task and applies it, in parallel, to all the information partitions assigned to your job. (You specify the data partitions, some other tough and important decision.) But when a processing workstream runs into trouble, it tin can be difficult to find and understand the problem among the multiple workstreams running at once.
3. Variants. Spark is open source, so information technology can be tweaked and revised in innumerable ways. There are major differences among the Spark 1 series, Spark 2.10, and the newer Spark 3. And Spark works somewhat differently across platforms – on-premises; on cloud-specific platforms such as AWS EMR, Azure HDInsight, and Google Dataproc; and on Databricks, which is bachelor beyond the major public clouds. Each variant offers some of its ain challenges, and a somewhat unlike set of tools for solving them.
4. Configuration options. Spark has hundreds of configuration options. And Spark interacts with the hardware and software environment it'south running in, each component of which has its own configuration options. Getting ane or ii critical settings right is difficult; when several related settings take to exist correct, guesswork becomes the norm, and over-allotment of resources, especially memory and CPUs (encounter below) becomes the safe strategy.
5. Trial and error approach. With so many configuration options, how to optimize? Well, if a job currently takes vi hours, yous can modify one, or a few, options, and run it again. That takes six hours, plus or minus. Repeat this three or four times, and information technology'southward the end of the week. Yous may have improved the configuration, but y'all probably won't have exhausted the possibilities as to what the best settings are.Sparkitecture diagram – the Spark application is the Driver Process, and the chore is split up across executors. (Source: Apache Spark for the Impatient on DZone.)

Iii Bug with Spark Jobs, On-Bounds and in the Cloud

Spark jobs can crave troubleshooting against three main kinds of problems:

• Failure. Spark jobs tin can simply fail. Sometimes a job volition fail on 1 try, and so work again after a restart. Just finding out that the job failed can be hard; finding out why can be harder. (Since the job is retentiveness-resident, failure makes the evidence disappear.)
• Poor performance. A Spark chore can run slower than you lot would similar it to; slower than an external service level agreement (SLA); or slower than it would practice if it were optimized. Information technology's very difficult to know how long a job "should" have, or where to start in optimizing a job or a cluster.
• Excessive cost or resources use. The resource use or, peculiarly in the cloud, the hard dollar cost of a job may heighten concern. As with operation, it'due south difficult to know how much the resource use and cost "should" exist, until you put piece of work into optimizing and run across where you've gotten to.

All of the issues and challenges described here apply to Spark across all platforms, whether it's running on-premises, in Amazon EMR, or on Databricks (across AWS, Azure, or GCP). However, there are a few subtle differences:

• Movement to cloud. At that place is a big movement of big data workloads from on-premises (largely running Spark on Hadoop) to the cloud (largely running Spark on Amazon EMR or Databricks). Moving to cloud provides greater flexibility and faster time to market place, too equally admission to built-in services found on each platform.
• Move to on-premises. At that place is a small motion of workloads from the cloud back to on-premises environments. When a cloud workload "settles downwards," such that flexibility is less important, then it may become significantly cheaper to run it on-premises instead.
• On-premises concerns. Resources (and costs) on-premises tend to exist relatively fixed; there can be a leadtime of months to years to significantly expand on-bounds resources. So the main business concern on-premises is maximizing the existing manor: making more jobs run in existing resources, and getting jobs to consummate reliably and on-fourth dimension, to maximize the pay-off from the existing estate.
• Cloud concerns. Resource in the deject are flexible and "pay equally you become" – but equally you lot get, you lot pay. So the master business organization in the cloud is managing costs. (Equally AWS puts it, "When running big data pipelines on the cloud, operational toll optimization is the name of the game.") This concern increases considering reliability concerns in the cloud can oftentimes be addressed past "throwing hardware at the problem" – increasing reliability, but at greater cost.
• On-premises Spark vs Amazon EMR. When moving to Amazon EMR, it'southward easy to do a "elevator and shift" from on-premises Spark to EMR. This saves time and coin on the cloud migration try, but any inefficiencies in the on-premises surround are reproduced in the deject, increasing costs. It'southward also fully possible to refactor earlier moving to EMR, just equally with Databricks.
• On-premises Spark vs Databricks. When moving to Databricks, about companies take advantage of Databricks' capabilities, such every bit ease of starting/shutting down clusters, and do at to the lowest degree some refactoring every bit role of the cloud migration effort. This costs time and coin in the cloud migration attempt, only results in lower costs and, potentially, greater reliability for the refactored chore in the cloud.

All of these concerns are accompanied past a distinct lack of needed data. Companies oftentimes make crucial decisions – on-premises vs. cloud, EMR vs. Databricks, "elevator and shift" vs. refactoring – with only guesses available as to what different options will cost in fourth dimension, resource, and money.

Ten Spark Challenges

Many Spark challenges relate to configuration, including the number of executors to assign, retentivity usage (at the driver level, and per executor), and what kind of hardware/auto instances to use. You make configuration choices per job, and also for the overall cluster in which jobs run, and these are interdependent – so things become complicated, fast.

Some challenges occur at the job level; these challenges are shared right beyond the data team. They include:
1. How many executors should each job use?
two. How much memory should I allocate for each chore?
3. How do I find and eliminate information skew?
4. How do I make my pipelines work improve?
5. How do I know if a specific job is optimized?

Other challenges come upwardly at the cluster level, or fifty-fifty at the stack level, as you lot decide what jobs to run on what clusters. These bug tend to exist the remit of operations people and data engineers. They include:
half-dozen. How practise I size my nodes, and match them to the right servers/instance types?
vii. How practice I see what's going on across the Spark stack and apps?
eight. Is my data partitioned correctly for my SQL queries?
ix. When do I take advantage of auto-scaling?
x. How do I get insights into jobs that have issues?

For easy access, these challenges are listed in the table below, linked to the appropriate section.

Job-Level Challenges	Cluster-Level Challenges
one. Executor and core resource allotment	6. Resource allocation
2. Retentiveness allocation	vii. Observability
three. Data skew / small files	8. Data partition vs. SQL queries / inefficiency
4. Pipeline optimization	9. Use of auto-scaling
5. Finding out whether a job is optimized	10. Troubleshooting
Impacts: Resources for a given job (at the cluster level) or across clusters tend to exist significantly under-allocated (causes crashes, hurting business results) or over-allocated (wastes resources and can cause other jobs to crash, both of which hurt business results).

See how Unravel can solve your Spark problems.

Try Unravel for free.

Section 1: Five Job-Level Challenges

These challenges occur at the level of private jobs. Fixing them can exist the responsibleness of the programmer or data scientist who created the job, or of operations people or data engineers who work on both private jobs and at the cluster level.

However, task-level challenges, taken together, have massive implications for clusters, and for the entire information estate. Ane of our Unravel Data customers has undertaken a right-sizing plan for resource-intensive jobs that has clawed back nearly one-half the space in their clusters, even though data processing book and jobs in product take been increasing.

For these challenges, we'll assume that the cluster your chore is running in is relatively well-designed (see adjacent department); that other jobs in the cluster are non resource hogs that will knock your job out of the running; and that you have the tools you need to troubleshoot individual jobs.

1. How many executors and cores should a job use?

1 of the cardinal advantages of Spark is parallelization – you run your job's code against different information partitions in parallel workstreams, as in the diagram below. The number of workstreams that run at once is the number of executors, times the number of cores per executor. So how many executors should your job use, and how many cores per executor – that is, how many workstreams do you want running at once?A Spark task using iii cores to parallelize output. Up to 3 tasks run simultaneously, and seven tasks are completed in a stock-still period of time. (Source: Lisa Hua, Spark Overview, Slideshare.)

You lot want high usage of cores, high usage of memory per cadre, and data partitioning appropriate to the task. (Usually, partitioning on the field or fields you're querying on.) This beginner's guide for Hadoop suggests two-three cores per executor, but not more than v; this expert's guide to Spark tuning on AWS suggests that you use 3 executors per node, with five cores per executor, every bit your starting point for all jobs. (!)

You lot are likely to have your ain sensible starting signal for your on-bounds or cloud platform, the servers or instances bachelor, and experience your team has had with similar workloads. Once your task runs successfully a few times, you can either leave information technology alone, or optimize it. We recommend that you optimize it, because optimization:

• Helps you save resource and coin (not over-allocating)
• Helps prevent crashes, because you right-size the resource (not under-allocating)
• Helps you fix crashes fast, because allocations are roughly correct, and because yous empathize the job meliorate

2. How much memory should I classify for each job?

Memory allocation is per executor, and the most you can allocate is the total bachelor in the node. If y'all're in the cloud, this is governed past your instance type; on-premises, past your concrete server or virtual auto. Some retention is needed for your cluster manager and arrangement resources (16GB may be a typical corporeality), and the remainder is bachelor for jobs.

If you have three executors in a 128GB cluster, and 16GB is taken upward by the cluster, that leaves 37GB per executor. However, a few GB volition exist required for executor overhead; the remainder is your per-executor memory. You will desire to division your data and so it can be processed efficiently in the bachelor memory.

This is just a starting point, however. You may need to be using a different instance type, or a different number of executors, to make the most efficient use of your node's resources against the task you lot're running. As with the number of executors (run into previous section), optimizing your job will help you know whether yous are over- or under-allocating memory, reduce the likelihood of crashes, and get you ready for troubleshooting when the need arises.

For more on retentivity management, see this widely read article, Spark Memory Management, by our own Rishitesh Mishra.

See how Unravel simplifies Spark Memory Management.

Endeavour Unravel for gratis.

iii. How do I handle data skew and pocket-size files?

Data skew and small files are complementary problems. Data skew tends to draw big files – where one key value, or a few, have a big share of the total data associated with them. This tin can force Spark, as it's processing the data, to move data around in the cluster, which tin can slow downwardly your job, cause low utilization of CPU capacity, and cause out-of-retention errors which abort your job. Several techniques for handling very large files which announced equally a effect of information skew are given in the pop article, Data Skew and Garbage Drove, past Rishitesh Mishra of Unravel.

Small files are partly the other end of data skew – a share of partitions will tend to be small. And Spark, since it is a parallel processing organisation, may generate many small files from parallel processes. Besides, some processes you use, such as file compression, may cause a large number of small files to appear, causing inefficiencies. You may need to reduce parallelism (undercutting i of the advantages of Spark), repartition (an expensive functioning you should minimize), or beginning adjusting your parameters, your data, or both (see details hither).

Both data skew and modest files incur a meta-problem that's common across Spark – when a job slows down or crashes, how do you know what the problem was? We will mention this over again, but it can be particularly difficult to know this for data-related bug, equally an otherwise well-constructed chore tin can accept seemingly random slowdowns or halts, caused by difficult-to-predict and hard-to-detect inconsistencies across different information sets.

4. How do I optimize at the pipeline level?

Spark pipelines are fabricated up of dataframes, connected by transformers (which calculate new data from existing information), and Estimators. Pipelines are widely used for all sorts of processing, including extract, transform, and load (ETL) jobs and machine learning. Spark makes information technology like shooting fish in a barrel to combine jobs into pipelines, just it does non go far piece of cake to monitor and manage jobs at the pipeline level. So information technology'south easy for monitoring, managing, and optimizing pipelines to appear every bit an exponentially more hard version of optimizing private Spark jobs.Existing Transformers create new Dataframes, with an Estimator producing the final model. (Source: Spark Pipelines: Elegant Withal Powerful, InsightDataScience.)

Many pipeline components are "tried and trusted" individually, and are thereby less likely to cause problems than new components you create yourself. However, interactions betwixt pipeline steps can cause novel problems.

Just as job issues roll up to the cluster level, they likewise roll up to the pipeline level. Pipelines are increasingly the unit of work for DataOps, simply it takes truly deep knowledge of your jobs and your cluster(due south) for you to work effectively at the pipeline level. This commodity, which tackles the issues involved in some depth, describes pipeline debugging as an "art."

v. How do I know if a specific task is optimized?

Neither Spark nor, for that matter, SQL are designed for ease of optimization. Spark comes with a monitoring and management interface, Spark UI, which can aid. But Spark UI can be challenging to use, especially for the types of comparisons – over time, across jobs, and across a big, decorated cluster – that you need to really optimize a job. And there is no "SQL UI" that specifically tells you how to optimize your SQL queries.

There are some full general rules. For example, a "bad" – inefficient – bring together can take hours. Merely information technology'southward very hard to discover where your app is spending its fourth dimension, let alone whether a specific SQL command is taking a long time, and whether it can indeed be optimized.

Spark's Catalyst optimizer, described here, does its best to optimize your queries for y'all. But when information sizes grow big enough, and processing gets complex plenty, yous have to help it forth if you want your resource usage, costs, and runtimes to stay on the adequate side.

Section 2: Cluster-Level Challenges

Cluster-level challenges are those that arise for a cluster that runs many (perchance hundreds or thousands) of jobs, in cluster design (how to get the well-nigh out of a specific cluster), cluster distribution (how to create a set of clusters that all-time meets your needs), and allocation across on-premises resources and one or more than public, private, or hybrid cloud resources.

The first step toward meeting cluster-level challenges is to meet chore-level challenges effectively, as described above. A cluster that'due south running unoptimized, poorly understood, slowdown-prone and crash-prone jobs is impossible to optimize. Merely if your jobs are right-sized, cluster-level challenges become much easier to meet. (Note that Unravel Information, equally mentioned in the previous department, helps yous find your resources-heavy Spark jobs, and optimize those kickoff. It also does much of the piece of work of troubleshooting and optimization for you.)

Meeting cluster-level challenges for Spark may exist a topic amend suited for a graduate-level informatics seminar than for a weblog post, just here are some of the problems that come up up, and a few comments on each:

six. Are Nodes Matched Upwardly to Servers or Cloud Instances?

A Spark node – a physical server or a cloud instance – will have an allotment of CPUs and physical retentiveness. (The whole point of Spark is to run things in bodily memory, so this is crucial.) Y'all have to fit your executors and retentiveness allocations into nodes that are advisedly matched to existing resources, on-premises or in the cloud. (You can classify more or fewer Spark cores than there are bachelor CPUs, but matching them makes things more than predictable, uses resources better, and may make troubleshooting easier.)

On-premises, poor matching betwixt nodes, physical servers, executors and memory results in inefficiencies, just these may non exist very visible; as long as the total concrete resource is sufficient for the jobs running, in that location's no obvious problem. Withal, bug like this can cause datacenters to be very poorly utilized, meaning there'south big overspending going on – it's only non noticed. (Ironically, the impending prospect of cloud migration may crusade an organization to freeze on-prem spending, shining a spotlight on costs and efficiency.)

In the cloud, "pay every bit you get" pricing shines a unlike type of spotlight on efficient utilise of resources – inefficiency shows upward in each calendar month's bill. You need to match nodes, deject instances, and job CPU and memory allocations very closely indeed, or incur what might amount to massive overspending. This article gives you some guidelines for running Apache Spark price-effectively on AWS EC2 instances, and is worth a read fifty-fifty if yous're running on-bounds, or on a different cloud provider.

You still have big problems here. In the cloud, with costs both visible and variable, price allocation is a big issue. It's hard to know who'south spending what, let solitary what the concern results that become with each unit of spending are. But tuning workloads against server resources and/or instances is the starting time step in gaining control of your spending, across all your information estates.

seven. How Practise I See What'due south Going on in My Cluster?

"Spark is notoriously difficult to tune and maintain," according to an article in The New Stack. Clusters need to be "expertly managed" to perform well, or all the good characteristics of Spark can come crashing downwardly in a heap of frustration and high costs. (In people'southward time and in business organisation losses, as well as direct, hard dollar costs.)

Key Spark advantages include accessibility to a wide range of users and the ability to run in memory. Just the nearly popular tool for Spark monitoring and direction, Spark UI, doesn't really assistance much at the cluster level. You tin can't, for case, easily tell which jobs consume the most resources over time. So it'due south difficult to know where to focus your optimization efforts. And Spark UI doesn't support more avant-garde functionality – such as comparing the current task run to previous runs, issuing warnings, or making recommendations, for example.

Logs on cloud clusters are lost when a cluster is terminated, so issues that occur in short-running clusters can be that much harder to debug. More than generally, managing log files is itself a big information management and data accessibility issue, making debugging and governance harder. This occurs in both on-premises and deject environments. And, when workloads are moved to the cloud, you lot no longer have a fixed-cost data estate, nor the "tribal noesis" accrued from years of running a gradually changing set of workloads on-bounds. Instead, you accept new technologies and pay-as-you lot-go billing. So cluster-level management, hard equally it is, becomes disquisitional.

Run across how Unravel helps manage clusters.

Try Unravel for free.

8. Is my data partitioned correctly for my SQL queries? (and other inefficiencies)

Operators tin can get quite upset, and rightly so, over "bad" or "rogue" queries that can cost style more than, in resource or price, than they need to. 1 colleague describes a team he worked on that went through more than $100,000 of cloud costs in a weekend of crash-testing a new application – a discovery made after the fact. (But before the job was put into product, where it would take really run up some bills.)

SQL is not designed to tell you how much a query is likely to cost, and more elegant-looking SQL queries (ie, fewer statements) may well be more expensive. The same is truthful of all kinds of code you lot have running.

And so you have to do some or all of three things:

• Learn something nearly SQL, and about coding languages you use, especially how they piece of work at runtime
• Understand how to optimize your code and partition your data for expert price/functioning
• Experiment with your app to understand where the resource use/cost "hot spots" are, and reduce them where possible

All this fits in the "optimize" recommendations from 1. and 2. above. We'll talk more nigh how to carry out optimization in Part two of this blog post series.

ix. When exercise I take advantage of automobile-scaling?

The ability to auto-scale – to assign resources to a chore just while it'south running, or to increment resources smoothly to meet processing peaks – is one of the well-nigh enticing features of the cloud. It'due south likewise one of the almost dangerous; there is no practical limit to how much yous tin can spend. You lot need some course of guardrails, and some form of alerting, to remove the run a risk of truly gigantic bills.

The demand for motorcar-scaling might, for instance, decide whether you lot move a given workload to the deject, or leave it running, unchanged, in your on-premises data center. But to help an awarding benefit from machine-scaling, you take to profile information technology, then cause resources to be allocated and de-allocated to match the peaks and valleys. And you have some calculations to make, considering cloud providers charge you more for spot resources – those you lot grab and let go of, as needed – than for persistent resources that y'all keep running for a long time. Spot resources may cost two or three times every bit much every bit defended ones.

The first step, as you might take guessed, is to optimize your application, every bit in the previous sections. Auto-scaling is a price/functioning optimization, and a potentially resource-intensive one. Yous should exercise other optimizations first.

Then profile your optimized application. You need to calculate ongoing and peak memory and processor usage, effigy out how long you demand each, and the resources needs and cost for each country. Then determine whether it'southward worth auto-scaling the job, whenever it runs, and how to do that. You may besides need to observe quiet times on a cluster to run some jobs, so the task'southward peaks don't overwhelm the cluster's resource.

To help, Databricks has two types of clusters, and the 2nd type works well with auto-scaling. Most jobs start out in an interactive cluster, which is similar an on-bounds cluster; multiple people use a fix a shared resources. It is, by definition, very difficult to avoid seriously underusing the capacity of an interactive cluster.

So you lot are meant to move each of your repeated, resources-intensive, and well-understood jobs off to its ain, dedicated, job-specific cluster. A job-specific cluster spins upwards, runs its job, and spins down. This is a form of machine-scaling already, and you can also scale the cluster's resources to friction match job peaks, if appropriate. But note that y'all desire your application profiled and optimized before moving it to a job-specific cluster.

10. How Do I Find and Fix Problems?

Just every bit it's difficult to fix an private Spark job, there's no like shooting fish in a barrel manner to know where to await for bug across a Spark cluster. And once you do find a trouble, at that place's very little guidance on how to set information technology. Is the problem with the chore itself, or the surround it's running in? For instance, over-allocating memory or CPUs for some Spark jobs tin can starve others. In the cloud, the noisy neighbors problem can slow down a Spark job run to the extent that information technology causes concern problems on one outing – only leaves the aforementioned job to finish in good time on the next run.

The better y'all handle the other challenges listed in this blog post, the fewer issues you'll have, but information technology's still very difficult to know how to most productively spend Spark operations time. For instance, a slow Spark job on one run may be worth fixing in its own right, and may be warning you of crashes on time to come runs. But information technology's very difficult but to see what the trend is for a Spark task in performance, permit lonely to go some idea of what the task is accomplishing vs. its resource apply and boilerplate time to complete. So Spark troubleshooting ends up being reactive, with all too many furry, blind little heads popping up for operators to play Whack-a-Mole with.

Impacts of these Challenges

If y'all run across the higher up challenges finer, you'll use your resource efficiently and cost-finer. However, our ascertainment hither at Unravel Data is that about Spark clusters are not run efficiently.

What we tend to run across nigh are the post-obit problems – at a job level, within a cluster, or across all clusters:

• Under-resource allotment. It tin be tricky to allocate your resources efficiently on your cluster, partition your datasets effectively, and determine the right level of resources for each job. If you nether-classify (either for a job's driver or the executors), a job is likely to run too slowly, or to crash. As a result, many developers and operators resort to…
• Over-allocation. If you assign as well many resources to your job, you lot're wasting resources (on-premises) or money (cloud). We hear about jobs that need, for instance, 2GB of memory, but are allocated much more – in i example, 85GB.

Applications can run slowly, because they're under-allocated – or because some apps are over-allocated, causing others to run slowly. Data teams then spend much of their time burn down-fighting bug that may come and go, depending on the item combination of jobs running that day. With every level of resource in shortage, new, business-critical apps are held up, then the cash needed to invest against these bug doesn't show upwards. It becomes an organizational headache, rather than a source of business capability.

Conclusion

To jump ahead to the end of this series a scrap, our customers here at Unravel are easily able to spot and prepare over-allocation and inefficiencies. They tin then monitor their jobs in product, finding and fixing bug as they arise. Developers even become on board, checking their jobs before moving them to production, then teaming up with Operations to go on them tuned and humming.

One Unravel customer, Mastercard, has been able to reduce usage of their clusters by roughly one-half, fifty-fifty as data sizes and application density has moved steadily upward during the global pandemic. And everyone gets along better, and has more fun at piece of work, while achieving these previously unimagined results.

So, whether y'all choose to utilise Unravel or not, develop a civilisation of right-sizing and efficiency in your work with Spark. It volition seem to be a hassle at outset, simply your team volition become much stronger, and y'all'll enjoy your work life more, as a event.

Y'all need a sort of X-ray of your Spark jobs, ameliorate cluster-level monitoring, surroundings information, and to correlate all of these sources into recommendations. In Troubleshooting Spark Applications, Part 2: Solutions, we volition describe the most widely used tools for Spark troubleshooting – including the Spark Web UI and our own offering, Unravel Data – and how to get together and correlate the information y'all need.

We hope you have enjoyed, and learned from, reading this blog post. You tin can download your copy of this blog mail here. If you would similar to know more than about Unravel Data at present, y'all can download a free trial or contact Unravel.

johnsonthable1952.blogspot.com

Source: https://www.unraveldata.com/resources/spark-troubleshooting-part-1-ten-challenges/