Closed Bug 1282758 Opened 8 years ago Closed 8 years ago

Fix main_summary compatibility with partition discovery

Categories

(Cloud Services Graveyard :: Metrics: Pipeline, defect, P1)

Product:
Cloud Services Graveyard
Component:
Metrics: Pipeline
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: mreid, Assigned: mreid)

References

Details

Attachments

(1 file)

Remove metadata files
55 bytes, text/x-github-pull-request
rvitillo
: review+
Details | Review 
The v3 main_summary code currently generates extra _metadata and _SUCCESS files in non-leaf directories. This causes Spark partition discovery to fail.

We should stop generating these extra files, at least until we upgrade Spark to a version that fixes the issue.  See [1], [2], and [3].

[1] https://issues.apache.org/jira/browse/SPARK-13207
[2] https://issues.apache.org/jira/browse/SPARK-15454
[3] https://issues.apache.org/jira/browse/SPARK-15895
Assignee: nobody → mreid
Points: --- → 1
Priority: -- → P1
Blocks: 1275889
Attached file Remove metadata files — 

        Attachment #8765881 -
        Flags: review?(rvitillo)
Status: NEW → RESOLVED
Closed: 8 years ago
Resolution: --- → FIXED

        Attachment #8765881 -
        Flags: review?(rvitillo) → review+

    
  
Product: Cloud Services → Cloud Services Graveyard

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: