Blog posts about Spark

Co-authors: Venkata Krishnan Sowrirajan and Min Shen We are excited to announce that push-based shuffle (codenamed Pr...

Co-authors: Andy Li and Hongbin Wu Indexing plays the key role in modern search engines for fast and accurate informa...

Co-authors: Preetam Nandy, Yunsong Meng, Cyrus DiCiccio, Heloise Logan, Amir Sepehri, Divya Venugopalan, Kinjal Basu,...

Co-authors: Min Shen, Chandni Singh, Ye Zhou, and Sunitha Beeram At LinkedIn, we rely heavily on offline data analyti...

Co-authors: Sriram Vasudevan, Cyrus DiCiccio, and Kinjal Basu At LinkedIn, our imperative is to create economic oppor...

Co-authors: Jun Shi, Mingzhou Zhou Introduction In the machine learning community, Apache Spark is widely used for da...

Co-authors: Walaa Eldin Moustafa, Wenye Zhang, Adwait Tumbde, Ratandeep Ratti Introduction Over the years, the popula...

One of the most common ways to store results from a Spark job is by writing the results to a Hive table stored on HDF...

Jeremy Smith, Jonathan Indig, Faisal Siddiqi

There is often a hidden performance cost tied to the complexity of data pipelines — Overhead. In this post we will ex...

Co-authors: Xuhong Zhang, Chenya Zhang, and Yiming Ma Today, we are announcing a new open source project called Avro2...

How we scaled Spark streaming with a novel balanced Kafka reader for ingesting massive amount of logging events from ...

Next Tuesday marks the start of the Spark Summit Conference in San Francisco. This year, LinkedIn engineers and data ...

This post has been updated to note the release of Pepperdata's Application Profiler, a commercial project based on Dr...

Machine learning is a key component of LinkedIn’s relevance-driven products. We use machine learning to train the ran...

We are proud to announce today that we are open sourcing Dr. Elephant, a powerful tool that helps users of Hadoop and...