Closed Bug 1296659 Opened 8 years ago Closed 8 years ago

Intermittent [taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"QO|Ws|U9|Uz|EN|Hf"} has failed: HTTP code is 500 which indicates error: server error - unknown parent image ID sha256:

Categories

(Taskcluster :: General, defect)

Product:
Taskcluster
Component:
General
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 1302596

People

(Reporter: intermittent-bug-filer, Assigned: garndt)

References

Details

(Keywords: intermittent-failure, Whiteboard: [docker-error-pulling])

Attachments

(1 file)

PR 247
53 bytes, text/x-github-pull-request
wcosta
: review+
Details | Review 
The interesting fact is that we downloaded the docker image successfully but failed afterward. 

[taskcluster 2016-08-23 18:42:00.897Z] Downloaded artifact successfully.
[taskcluster 2016-08-23 18:42:00.897Z] Downloaded 3914.719 mb
[taskcluster 2016-08-23 18:42:00.898Z] Loading docker image from downloaded archive.

[taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"cGMJA6AjSPmujXSViDZK-A"} has failed: HTTP code is 500 which indicates error: server error - unknown parent image ID sha256:dc5c3df9b3763c47fc2099ecaf756f863faecbb38b4177947a60756f8b6ad45a
Looking back at the last dozen or so of these recorded, the workers are removing an image during a garbage collection cycle at the same time as the worker is importing an image.  I'm guessing that during import of an image tarball there are less locks on shared layers than when doing a pull from docker hub.

hat tip to dustin for mentioning this before.  I thought I saw some things in the logs that ruled out that hypothesis but it just might be true based on the few instances I looked at today.
This is spiking horribly on Aurora at the moment :(
Flags: needinfo?(garndt)
At least the one error that I've been pointed to is different than the error here.  What I think is causing this bug, does not seem to be the same thing causing bugs on aurora.
(In reply to Greg Arndt [:garndt] from comment #7)
> At least the one error that I've been pointed to is different than the error
> here.  What I think is causing this bug, does not seem to be the same thing
> causing bugs on aurora.

Sorry about that. Spun off to bug 1298488 per our IRC discussion.
Flags: needinfo?(garndt)
Summary: Intermittent [taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"FMxRAVRZSjyPtgYEfK9DvA"} has failed: HTTP code is 500 which indicates error: server error - unknown parent image ID sha256:1b19496cc6b2007074e74 → Intermittent [taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"Uz|EN|Hf|U9"} has failed: HTTP code is 500 which indicates error: server error - unknown parent image ID sha256:
Summary: Intermittent [taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"Uz|EN|Hf|U9"} has failed: HTTP code is 500 which indicates error: server error - unknown parent image ID sha256: → Intermittent [taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"Uz|EN|Hf|U9|QO"} has failed: HTTP code is 500 which indicates error: server error - unknown parent image ID sha256:
Summary: Intermittent [taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"Uz|EN|Hf|U9|QO"} has failed: HTTP code is 500 which indicates error: server error - unknown parent image ID sha256: → Intermittent [taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"QO|U9|Uz|EN|Hf"} has failed: HTTP code is 500 which indicates error: server error - unknown parent image ID sha256:
Summary: Intermittent [taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"QO|U9|Uz|EN|Hf"} has failed: HTTP code is 500 which indicates error: server error - unknown parent image ID sha256: → Intermittent [taskcluster:error] Pulling docker image {"path":"public/image.tar","type":"task-image","taskId":"QO|Ws|U9|Uz|EN|Hf"} has failed: HTTP code is 500 which indicates error: server error - unknown parent image ID sha256:
Attached file PR 247 — 

        Attachment #8789074 -
        Flags: review?(wcosta)
Assignee: nobody → garndt
Status: NEW → ASSIGNED

    
    

    
  
Comment on attachment 8789074 [details] [review]
PR 247

One small nit but lgtm overall

        Attachment #8789074 -
        Flags: review?(wcosta) → review+
Whiteboard: [docker-error-pulling]
Status: ASSIGNED → RESOLVED
Closed: 8 years ago
Resolution: --- → DUPLICATE

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: